permadeath.com / didbot
Identities for entities did.bot
agent llm did
Star
Fork 0
Watch
didbot tools
5 folders · 3 files
xray
xray-clones
xray-clusters
xray-common
xray-literals
Cargo.lock
Cargo.toml
README.md
README.md

tools/ #

Developer tooling. Its own cargo workspace, listed under exclude in the root manifest, so nothing here is built by cargo build --workspace, tested by cargo test --workspace or linted by scripts/lint.sh. Lint it on its own:

cargo clippy --manifest-path tools/Cargo.toml --all-targets -- -D warnings

xray — the reference index #

git grep is a text search over a language with modules, use re-exports, trait impls, macros and cfg. It cannot tell a definition from a word in a comment, it cannot follow a re-export, and it cannot tell two methods named cursor apart. xray builds the index a compiler-accurate answer needs, and answers four questions from it:

Query Answer
dead-pub Public items that nothing outside their crate and no test reaches, each marked unreferenced, file-local or crate-local. This is what dead_code will not tell you about a library crate.
refs <path> Every reference to an item, with file and line, the crate and target each comes from, and the item it sits inside.
crate-edges Every declared workspace dependency with the number of item-level references crossing it. A dependency at zero is one nothing uses.
file-items <file> Every item in a file with its span, kind, visibility and reference count.

Paths are the ones rustdoc reports: didbot_name::lists::bundled_names, didbot_name::fragments::Template::slots. refs also takes any suffix of one.

Running it #

scripts/xray.sh dead-pub
scripts/xray.sh refs didbot_name::lists::bundled_names
scripts/xray.sh crate-edges
scripts/xray.sh file-items crates/didbot-serve/src/routes.rs
scripts/xray.sh build      # rebuild and stop

The index lands in .xray/index.db, which is gitignored along with the two dumps it is built from. A rebuild takes about two minutes and 4 GB.

An index is only as current as the revision it was built from. scripts/xray.sh rebuilds when HEAD has moved since the last build, and scripts/xray.sh info prints the revision the current index holds. It does not rebuild for uncommitted edits: after editing, run scripts/xray.sh build.

Where the data comes from #

Two sources, because neither knows everything:

  • rust-analyzer scip for the reference edges. It resolves names the way the compiler does. SCIP rather than the LSIF the same binary also writes, because scip takes --config-path and lsif takes nothing: the dump has to be taken with all cargo features on, or everything behind an off feature reads as unreferenced. It is also faster and a quarter the size.
  • cargo doc --output-format json for the item inventory — kind, visibility, module path and source span, none of which SCIP carries. Nightly only, and run once for libs and once for bins, because rustdoc names its output after the crate and five packages here have a bin sharing their lib's name.

xray build joins them: a SCIP definition sits on the first line of a rustdoc span, and an item with no source position of its own is matched by the path its SCIP symbol spells.

Schema #

sqlite3 .xray/index.db if a question needs SQL rather than a subcommand.

item(id, crate, module_path, name, kind, visibility,
     file, line_start, line_end, target_kind, exported)
ref(from_item, to_item, file, line, crate, target_kind, cfg_gated)
dep(from_crate, to_crate, kind)
meta(key, value)

target_kind is lib, bin, test, bench, example or build. exported marks an item every module between it and the crate root is pub. cfg_gated marks a reference from code rustdoc does not see — a #[cfg(test)] module, or a feature that is off — which dead-pub counts as a use.

What it cannot see #

  • Code in doc comments. A doctest is the one caller a dead-pub hit may still have.
  • Callers outside this repository. Everything here is publish = false, so in practice that means vibescrobble.com and anything built against a running server rather than against the crates.
  • A trait impl reached only through dyn dispatch is recorded against the trait method, not the impl's.
  • Items generated by a macro carry the span of the macro call, so file-items reports them where they are invoked.

Developer tooling #

A workspace of its own, excluded from the product workspace at the repo root, so that a tool's dependencies never enter the graph that ships.
xray-clones Functions that are the same shape as other functions
xray-literals Constant values that more than one file holds
xray-common Which files both read, and the reviewed-finding list both answer to 
cargo run --release --manifest-path tools/xray-clones/Cargo.toml   -- --root .
cargo run --release --manifest-path tools/xray-literals/Cargo.toml -- --root .

Each exits 1 when it reports something, 0 when it reports nothing, and 2 on an error.

Two measures, because one was not enough #

This workspace grew four duplications that a person found weeks later by reading code. xray-clones was measured against all four and found one: it finds a shape that was copied, and not a rule that was reimplemented.

What the other three had in common was a repeated literal — the same 100_000 under one name in two crates, the same NSID string in four files, the same loopback address in three guards. xray-literals indexes those. It is the dumber measure, and it catches more of them.

xray-clones xray-literals
Two bounded replay caches, REPLAY_CAPACITY = 100_000 twice no rank 171 of 364; 43 of 48 with --declared-twice
Four readers of com.atproto.repo.getRecord no rank 4 of 363
Two SSRF address guards yes, via --include-tests rank 55 of 364; 1 of 1 with --kind ip
A hand-rolled JWS parser beside jacquard_common::service_auth no rank 168, and only with --scan

Neither is a gate. Both are things to run when a change lands in an area that already had a way of doing this.

What neither of them catches #

  • A rule reimplemented with neither a shared shape nor a shared value. Two validators that spell the same constraint in different words — one a range check, one a regex — share no token stream and no literal. Nothing here will see them.
  • A duplicate that a dependency already holds. Both tools read the workspace; --scan <dir> adds a dependency's sources, but only to compare against, and only for what those two measures can see.
  • A value one site computes and the other writes down. 1 << 20 beside 1048576, or Duration::from_secs(60 * 60) beside 3600, are one fact in two places that neither the shapes nor the literals match on.
  • Duplication the language spans already. Copies behind a generic, a trait default or a macro do not exist in source, so there is nothing to read.
  • Whether the two copies agree. Both tools say two places hold one thing. Neither says which is right, and the address guards are the case that matters: the two lists were found by their shared 127.0.0.1, and the bug was the thirteen CIDR blocks only one of them had.

Read both reports as questions. Roughly two in five of what either reports is repetition you look at and keep.