From 3135661b8451d33f6f83cdc8061ab2f52d8c53f1 Mon Sep 17 00:00:00 2001 From: Eric Rodrigues Pires Date: Sun, 30 Nov 2025 08:33:59 -0300 Subject: [PATCH] Prepare for duperq release --- duper/CHANGELOG.md | 2 +- duper_website/docs/duperfmt.md | 2 + duper_website/docs/duperq.md | 2 + duperfmt/README.md | 2 +- duperfmt/src/duperfmt.md | 17 ++++ duperfmt/src/lib.rs | 2 +- duperq/CHANGELOG.md | 5 + duperq/README.md | 2 +- duperq/src/duperq.md | 165 +++++++++++++++++++++++++++++++++ duperq/src/lib.rs | 2 +- 10 files changed, 196 insertions(+), 5 deletions(-) create mode 100644 duperfmt/src/duperfmt.md create mode 100644 duperq/CHANGELOG.md create mode 100644 duperq/src/duperq.md diff --git a/duper/CHANGELOG.md b/duper/CHANGELOG.md index 61cfdef..5abe59c 100644 --- a/duper/CHANGELOG.md +++ b/duper/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## Unreleased +## 0.4.3 (2025-11-30) ### Changed diff --git a/duper_website/docs/duperfmt.md b/duper_website/docs/duperfmt.md index ad87761..e2fefc0 100644 --- a/duper_website/docs/duperfmt.md +++ b/duper_website/docs/duperfmt.md @@ -4,6 +4,8 @@ ## Installation +`duperfmt` is currently only available as a Crates.IO binary. To compile from source: + ```bash cargo install --locked duperfmt ``` diff --git a/duper_website/docs/duperq.md b/duper_website/docs/duperq.md index 45c79e0..db41225 100644 --- a/duper_website/docs/duperq.md +++ b/duper_website/docs/duperq.md @@ -4,6 +4,8 @@ ## Installation +`duperq` is currently only available as a Crates.IO binary. To compile from source: + ```bash cargo install --locked duperq ``` diff --git a/duperfmt/README.md b/duperfmt/README.md index 92a2053..f096339 100644 --- a/duperfmt/README.md +++ b/duperfmt/README.md @@ -10,4 +10,4 @@ The official Duper formatting library and CLI. -[Check out the official website for Duper.](https://duper.dev.br) +[Check out the documentation on the Duper website.](https://duper.dev.br/duperfmt.html) diff --git a/duperfmt/src/duperfmt.md b/duperfmt/src/duperfmt.md new file mode 100644 index 0000000..ad87761 --- /dev/null +++ b/duperfmt/src/duperfmt.md @@ -0,0 +1,17 @@ +# duperfmt + +`duperfmt` is a Duper formatter based on tree-sitter and Topiary. It powers `duper_lsp`'s own formatting engine. + +## Installation + +```bash +cargo install --locked duperfmt +``` + +## Basic usage + +```bash +duperfmt -f input.duper -o output.duper +``` + +Run `duperfmt --help` for more details. diff --git a/duperfmt/src/lib.rs b/duperfmt/src/lib.rs index 6ef337d..4fed9f3 100644 --- a/duperfmt/src/lib.rs +++ b/duperfmt/src/lib.rs @@ -1,6 +1,6 @@ #![doc(html_logo_url = "https://duper.dev.br/logos/duper-100-100.png")] //! -#![doc = include_str!("../../duper_website/docs/duperfmt.md")] +#![doc = include_str!("./duperfmt.md")] //! use std::io::Write; diff --git a/duperq/CHANGELOG.md b/duperq/CHANGELOG.md new file mode 100644 index 0000000..a1c10f4 --- /dev/null +++ b/duperq/CHANGELOG.md @@ -0,0 +1,5 @@ +# Changelog + +## 0.1.0 (2025-11-30) + +Initial release. diff --git a/duperq/README.md b/duperq/README.md index 5632561..78dffc5 100644 --- a/duperq/README.md +++ b/duperq/README.md @@ -10,4 +10,4 @@ A fast Duper and JSON filter/processor. -[Check out the official website for Duper.](https://duper.dev.br) +[Check out the documentation on the Duper website.](https://duper.dev.br/duperq.html) diff --git a/duperq/src/duperq.md b/duperq/src/duperq.md new file mode 100644 index 0000000..45c79e0 --- /dev/null +++ b/duperq/src/duperq.md @@ -0,0 +1,165 @@ +# duperq + +`duperq` is a fast filter and processor of Duper files and logs, which also works with JSON. + +## Installation + +```bash +cargo install --locked duperq +``` + +## Basic usage + +As an example, we'll assume the following data from a log format: + +```duper +{ + traceId: UUID("a2ce2f29-84cf-47c9-a877-381855d59e77"), + spanId: UUID("78ee3c78-c090-43f2-8568-f4542dc10ea5"), + timestamp: "2025-11-29T22:21:45.133Z", + level: "INFO", + service: "store-webapp", + development: false, + http: { + method: "GET", + url: "/shopping-cart", + statusCode: 200, + address: ("192.168.1.100", 14567), + userAgent: "Mozilla/5.0", + duration: Duration('PT0.14567S'), + history: ["/", "/search?q=headphones+", "/products/42"], + }, +} +``` + +You can read files by passing them after the `duperq` filter: + +```bash +duperq "filter ." path/to/**/*.duper +``` + +You can also read lines of Duper values from stdin. + +```bash +tail -f path/to/app.log | duperq "filter ." +``` + +### Filtering + +To filter results, use the `filter` param in the query. You can bypass filtering by passing an empty query to `duperq`. + +```bash +duperq "" log.duper +``` + +To access fields in objects or arrays of objects, use `.fieldName`. For complex keys, you can use quotes and Duper escaping, i.e. `."special key"`. + +```bash +duperq "filter .level == \"INFO\"" log.duper +# ... equivalent to ... +duperq "filter .\"level\" == \"INFO\"" log.duper +``` + +You can concatenate fields to access nested objects: + +```bash +duperq "filter .http.method == \"GET\"" log.duper +``` + +You can also combine filters with `and`/`&&` or `or`/`||`. To check if a field simply exists, use `exists(...)`. + +```bash +duperq "filter (.http.url = \"/admin\" || .level = \"INFO\") && exists(.spanId)" log.duper +``` + +You can use comparison operators (`==` or `=` for equality; `!=` or `<>` for inequality; `<`, `<=`, `>`, `>=`) as you'd expect. For sized values (objects, arrays, tuples, strings and bytes), you can use the `len(...)` function. + +```bash +duperq "filter .http.statusCode >= 400" log.duper +duperq "filter len(.http.history) > 2" log.duper +duperq "filter .http.duration < Duration('PT1S')" log.duper +``` + +You can also match strings/bytes with [Rust regexes](https://docs.rs/regex/) via `=~ "regex"`, or access the current element with a sole `.`. To filter elements in an array, add a `[selector operator value]` to the fields. For example, we can filter history values that contain "headphones" by putting all of these together: + +```bash +duperq "filter .http.history[. =~ \"headphones\"]" log.duper +``` + +To check if a value is truthy, simply use the selector without an operator. You can also negate the result of a filter with `!`: + +```bash +duperq "filter !.development" log.duper +``` + +To validate that a value is of a certain type, use the `is` operator. The valid right-handside operands are: + +- `Object` +- `Array` +- `Tuple` +- `String` +- `Bytes` +- `Instant` +- `ZonedDateTime` +- `PlainDate` +- `PlainTime` +- `PlainDateTime` +- `PlainYearMonth` +- `PlainMonthDay` +- `Duration` +- `Temporal` +- `Integer` +- `Float` +- `Number` +- `Boolean` +- `Null` + +```bash +duperq "filter .http.address is Tuple" log.duper +``` + +You can index into an array/tuple with `[index]`. Negative indexes also work, but they do not wrap around. To filter over an identifier, use `identifier(...)`. + +```bash +duperq "filter identifier(.http.address[0]) == \"IPv4Address\"" log.duper +# We can use a regex instead +duperq "filter identifier(.http.address[0]) =~ \"(?i)^ipv\\\\daddress\$\"" log.duper +# To check if there is NO identifier +duperq "filter identifier(.http.userAgent) == null" log.duper +# To check if there is ANY identifier +duperq "filter identifier(.traceId) <> null" log.duper +``` + +You can use [ranges](https://doc.rust-lang.org/std/ops/struct.Range.html) over array values. + +```bash +duperq "filter .http.history[..2] == \"/\"" log.duper +``` + +Last but not least, you can cast values into different types with `cast(..., type)`, with the same possible types from the `is` operator. This can be useful when dealing with JSON data, where there are no tuples or Temporal values, or to treat a tuple as an array. In our example, we can transform the string-only timestamp into a filterable value: + +```bash +duperq "filter cast(.timestamp, Instant) > Instant('2025-11-01T00:00:00-03:00')" log.duper +``` + +### Manipulation + +Other than filtering data, you may also skip the first values with `skip X`, or limit the number of filtered values you take with `take X`. These operations can be combined with pipes `|`: + +```bash +duperq "filter .development | skip 3 | filter .level = \"ERROR\" | take 10" path/to/**/*.duper +``` + +### Output + +By default, `duperq` serializes output data into a single-line format. You can change this by piping the output of your query to: + +- `| ansi`: Prints with ANSI colors. +- `| pretty-print`: Pretty-prints values over multiple lines with indentation. +- `| format`: Allows you to print arbitrary strings, replacing `${...}` blocks with the selector inside. String values will have their quotes stripped. Missing values will be printed as ``. + +```bash +duperq "filter . | format \"[\${.level}] \${.http.statusCode} - \${.http.method} \${.http.url}\"" log.duper +``` + +Formats must always be the last block in your query workflow. diff --git a/duperq/src/lib.rs b/duperq/src/lib.rs index c3c0505..f5e1854 100644 --- a/duperq/src/lib.rs +++ b/duperq/src/lib.rs @@ -1,6 +1,6 @@ #![doc(html_logo_url = "https://duper.dev.br/logos/duper-100-100.png")] //! -#![doc = include_str!("../../duper_website/docs/duperq.md")] +#![doc = include_str!("./duperq.md")] //! mod accessor; -- 2.51.2