# tools/ Developer tooling. Its own cargo workspace, listed under `exclude` in the root manifest, so nothing here is built by `cargo build --workspace`, tested by `cargo test --workspace` or linted by `scripts/lint.sh`. Lint it on its own: cargo clippy --manifest-path tools/Cargo.toml --all-targets -- -D warnings `oauth-interop/` is a Node program rather than a member of that workspace: `crates/didbot-serve/tests/oauth_interop.rs` runs it, and `docs/testing.md` says how. ## xray — the reference index `git grep` is a text search over a language with modules, `use` re-exports, trait impls, macros and `cfg`. It cannot tell a definition from a word in a comment, it cannot follow a re-export, and it cannot tell two methods named `cursor` apart. `xray` builds the index a compiler-accurate answer needs, and answers four questions from it: | Query | Answer | | --- | --- | | `dead-pub` | Public items that nothing outside their crate and no test reaches, each marked `unreferenced`, `file-local` or `crate-local`. This is what `dead_code` will not tell you about a library crate. | | `refs ` | Every reference to an item, with file and line, the crate and target each comes from, and the item it sits inside. | | `crate-edges` | Every declared workspace dependency with the number of item-level references crossing it. A dependency at zero is one nothing uses. | | `file-items ` | Every item in a file with its span, kind, visibility and reference count. | Paths are the ones rustdoc reports: `didbot_name::lists::bundled_names`, `didbot_name::fragments::Template::slots`. `refs` also takes any suffix of one. ### Running it scripts/xray.sh dead-pub scripts/xray.sh refs didbot_name::lists::bundled_names scripts/xray.sh crate-edges scripts/xray.sh file-items crates/didbot-serve/src/routes.rs scripts/xray.sh build # rebuild and stop The index lands in `.xray/index.db`, which is gitignored along with the two dumps it is built from. A rebuild takes about two minutes and 4 GB. **An index is only as current as the revision it was built from.** `scripts/xray.sh` rebuilds when `HEAD` has moved since the last build, and `scripts/xray.sh info` prints the revision the current index holds. It does not rebuild for uncommitted edits: after editing, run `scripts/xray.sh build`. ### Where the data comes from Two sources, because neither knows everything: - `rust-analyzer scip` for the reference edges. It resolves names the way the compiler does. SCIP rather than the LSIF the same binary also writes, because `scip` takes `--config-path` and `lsif` takes nothing: the dump has to be taken with all cargo features on, or everything behind an off feature reads as unreferenced. It is also faster and a quarter the size. - `cargo doc --output-format json` for the item inventory — kind, visibility, module path and source span, none of which SCIP carries. Nightly only, and run once for libs and once for bins, because rustdoc names its output after the crate and five packages here have a bin sharing their lib's name. `xray build` joins them: a SCIP definition sits on the first line of a rustdoc span, and an item with no source position of its own is matched by the path its SCIP symbol spells. ### Schema `sqlite3 .xray/index.db` if a question needs SQL rather than a subcommand. item(id, crate, module_path, name, kind, visibility, file, line_start, line_end, target_kind, exported) ref(from_item, to_item, file, line, crate, target_kind, cfg_gated) dep(from_crate, to_crate, kind) meta(key, value) `target_kind` is lib, bin, test, bench, example or build. `exported` marks an item every module between it and the crate root is `pub`. `cfg_gated` marks a reference from code rustdoc does not see — a `#[cfg(test)]` module, or a feature that is off — which `dead-pub` counts as a use. ### What it cannot see - Code in doc comments. A doctest is the one caller a `dead-pub` hit may still have. - Callers outside this repository. Everything here is `publish = false`, so in practice that means vibescrobble.com and anything built against a running server rather than against the crates. - A trait impl reached only through `dyn` dispatch is recorded against the trait method, not the impl's. - Items generated by a macro carry the span of the macro call, so `file-items` reports them where they are invoked. # Developer tooling A workspace of its own, excluded from the product workspace at the repo root, so that a tool's dependencies never enter the graph that ships. | | | | --- | --- | | [`xray-clones`](xray-clones/README.md) | Functions that are the same **shape** as other functions | | [`xray-literals`](xray-literals/README.md) | Constant **values** that more than one file holds | | `xray-common` | Which files both read, and the reviewed-finding list both answer to | ``` cargo run --release --manifest-path tools/xray-clones/Cargo.toml -- --root . cargo run --release --manifest-path tools/xray-literals/Cargo.toml -- --root . ``` Each exits 1 when it reports something, 0 when it reports nothing, and 2 on an error. ## Two measures, because one was not enough This workspace grew four duplications that a person found weeks later by reading code. `xray-clones` was measured against all four and found one: it finds a shape that was copied, and not a rule that was reimplemented. What the other three had in common was a repeated *literal* — the same `100_000` under one name in two crates, the same NSID string in four files, the same loopback address in three guards. `xray-literals` indexes those. It is the dumber measure, and it catches more of them. | | `xray-clones` | `xray-literals` | | --- | --- | --- | | Two bounded replay caches, `REPLAY_CAPACITY = 100_000` twice | no | rank 171 of 364; **43 of 48** with `--declared-twice` | | Four readers of `com.atproto.repo.getRecord` | no | **rank 4 of 363** | | Two SSRF address guards | yes, via `--include-tests` | rank 55 of 364; **1 of 1** with `--kind ip` | | A hand-rolled JWS parser beside `jacquard_common::service_auth` | no | rank 168, and only with `--scan` | Neither is a gate. Both are things to run when a change lands in an area that already had a way of doing this. ## What neither of them catches - **A rule reimplemented with neither a shared shape nor a shared value.** Two validators that spell the same constraint in different words — one a range check, one a regex — share no token stream and no literal. Nothing here will see them. - **A duplicate that a dependency already holds.** Both tools read the workspace; `--scan ` adds a dependency's sources, but only to compare against, and only for what those two measures can see. - **A value one site computes and the other writes down.** `1 << 20` beside `1048576`, or `Duration::from_secs(60 * 60)` beside `3600`, are one fact in two places that neither the shapes nor the literals match on. - **Duplication the language spans already.** Copies behind a generic, a trait default or a macro do not exist in source, so there is nothing to read. - **Whether the two copies agree.** Both tools say two places hold one thing. Neither says which is right, and the address guards are the case that matters: the two lists were found by their shared `127.0.0.1`, and the bug was the thirteen CIDR blocks only one of them had. Read both reports as questions. Roughly two in five of what either reports is repetition you look at and keep.