diff --git a/docs/design-plans/2026-04-13-test-labeler.md b/docs/design-plans/2026-04-13-test-labeler.md new file mode 100644 index 0000000..9da2cae --- /dev/null +++ b/docs/design-plans/2026-04-13-test-labeler.md @@ -0,0 +1,398 @@ +# `atproto-devtool test labeler` design + +## Summary + +The implementation is organized as a **layered four-stage pipeline** (identity → HTTP → subscription → crypto) driving a single `LabelerReport`. Each stage is an async function with a typed input and a typed output (`Outcome`), so the Rust type system enforces the dependency graph at compile time: the crypto stage literally cannot be called without the `IdentityFacts` and label slice produced by earlier stages. The pipeline driver is non-fail-fast — it runs every stage whose inputs are available regardless of prior failures — so a single invocation always produces a complete diagnostic picture rather than stopping at the first problem. + +Diagnostics are first-class throughout. Every non-passing result carries a `miette::Diagnostic` whose `NamedSource` attaches the offending raw bytes (JSON document, CBOR frame, serialized record) and whose `#[label]` spans highlight the exact field that caused the violation — making the tool useful for debugging rather than merely detecting. The subscription stage uses a two-connection strategy with a configurable time budget, inferring backfill completion from an idle gap heuristic. The crypto stage is designed around the reality of key rotation: it verifies label signatures against the currently declared key first, then falls back lazily to the PLC operation audit log to accept labels signed by a legitimately rotated-out key, attaching an `Advisory` rather than a `Fail` in that case. Stages are coded against narrow async traits (`DidResolver`, `HttpClient`, `WebSocketClient`, `PlcLogFetcher`) so unit tests can replay recorded fixtures from real labelers without any network access; rendered output is locked down with `insta` snapshots to catch diagnostic drift automatically. + +## Definition of Done + +**Primary deliverable:** A Rust binary `atproto-devtool` with a modular subcommand architecture (single crate, clap-derived, async/tokio), designed so new feature modules can be added with minimal ceremony. + +**First feature:** `atproto-devtool test labeler [--did ]` that runs a conformance suite across four layers: + +- **Identity:** DID document contains an `atproto_labeler` service entry of type `AtprotoLabeler` and a `#atproto_label` signing key; the labeler's PDS holds a valid `app.bsky.labeler.service/self` record with policies. +- **HTTP:** `com.atproto.label.queryLabels` responds with well-formed labels and handles pagination/schema correctly. +- **Subscription:** `com.atproto.label.subscribeLabels` WebSocket firehose emits valid frames and supports `cursor=0` backfill. +- **Crypto:** Label signatures verify against the declared `#atproto_label` key. + +Skipped checks are clearly reported when inputs don't unlock them (e.g., endpoint-only invocation without `--did` skips identity/crypto checks). + +**Output:** Rich human-readable diagnostics via `miette`; non-zero exit on any check failure. + +**Out of scope (initial):** JSON output, Ozone (`tools.ozone.*`) moderation endpoints, authenticated check flows, additional subcommands beyond `test labeler`. + +## Acceptance Criteria + +### test-labeler.AC1: CLI skeleton and invocation modes + +- **test-labeler.AC1.1 Success:** `atproto-devtool test labeler ` accepts an atproto handle, resolves it to a DID, and runs all four stages. +- **test-labeler.AC1.2 Success:** `atproto-devtool test labeler ` accepts a DID directly (did:plc or did:web) and runs all four stages without a prior handle resolution. +- **test-labeler.AC1.3 Success:** `atproto-devtool test labeler ` accepts a raw endpoint URL and runs the HTTP and subscription stages while marking identity and crypto checks `Skipped` with a clear reason. +- **test-labeler.AC1.4 Success:** `atproto-devtool test labeler --did ` unlocks identity checks (and therefore the crypto stage) against an endpoint URL, and also cross-checks that the DID's declared labeler service endpoint matches the provided URL — reporting a `SpecViolation` on mismatch. +- **test-labeler.AC1.5 Failure:** A target argument that is neither a valid handle, DID, nor URL produces a clap parsing error with a helpful message and exit code `2`. +- **test-labeler.AC1.6 Edge:** `atproto-devtool test labeler --help` renders the subcommand's help text including all stage-related flags (`--subscribe-timeout`, `--did`, `--verbose`, `--no-color`). + +### test-labeler.AC2: Identity-layer checks + +- **test-labeler.AC2.1 Success:** A DID document containing a `#atproto_labeler` service entry of type `AtprotoLabeler` with a valid `serviceEndpoint`, plus a signing-key verification method parseable as k256 or p256, passes all identity checks. +- **test-labeler.AC2.2 Success:** A `app.bsky.labeler.service/self` record with a non-empty `policies.labelValues` list passes the labeler-record checks. +- **test-labeler.AC2.3 Failure:** A DID document missing the `atproto_labeler` service entry produces a `SpecViolation` `CheckResult` whose diagnostic's `NamedSource` is the DID-document JSON and whose `#[label]` span highlights the `service` array. +- **test-labeler.AC2.4 Failure:** A DID document missing the labeler signing key produces a `SpecViolation` whose diagnostic highlights the `verificationMethod` array. +- **test-labeler.AC2.5 Failure:** A labeler `serviceEndpoint` that is not a valid HTTPS URL produces a `SpecViolation` with the offending endpoint value highlighted in the DID doc. +- **test-labeler.AC2.6 Failure:** A missing `app.bsky.labeler.service/self` record (PDS returns 404) produces a `SpecViolation` distinct from a PDS transport failure. +- **test-labeler.AC2.7 Failure:** An `app.bsky.labeler.service/self` record with an empty `policies.labelValues` list produces a `SpecViolation` whose diagnostic highlights the `policies` field in the re-serialized record. +- **test-labeler.AC2.8 NetworkError:** A DNS failure resolving a handle, an unreachable `plc.directory`, or an unreachable PDS produces a `NetworkError` result that is called out separately in the summary and does not by itself fail the run. + +### test-labeler.AC3: HTTP-layer checks (`queryLabels`) + +- **test-labeler.AC3.1 Success:** A labeler endpoint that responds to `com.atproto.label.queryLabels` with a well-formed lexicon response decodes into typed labels and passes the schema check. +- **test-labeler.AC3.2 Success:** A labeler that honors the `cursor` parameter — returning a distinct page when called with a cursor from the first page — passes the pagination round-trip check. +- **test-labeler.AC3.3 Success:** A labeler that returns an empty labels array passes the schema and pagination checks and contributes a distinct `Advisory` ("labeler has no published labels") to inform downstream stages. +- **test-labeler.AC3.4 Failure:** A `queryLabels` response that omits required fields or otherwise fails lexicon decoding produces a `SpecViolation` whose diagnostic carries the response JSON as its `NamedSource`. +- **test-labeler.AC3.5 Failure:** A labeler that ignores the `cursor` parameter (returns the first page again) produces a `SpecViolation` on the pagination check. +- **test-labeler.AC3.6 NetworkError:** An unreachable or TLS-failing labeler endpoint produces a `NetworkError` that does not cascade into `queryLabels` schema failures. + +### test-labeler.AC4: Subscription-layer checks (`subscribeLabels`) + +- **test-labeler.AC4.1 Success (backfill completes within budget):** A labeler whose backfill flushes within `--subscribe-timeout` followed by a ≥500ms idle gap produces a `Pass` backfill check and an implicit `Pass` live-tail check ("live tail observed after backfill completed"). +- **test-labeler.AC4.2 Success (backfill exceeds budget):** A labeler whose backfill is still producing frames at the end of the budget produces an `Advisory` backfill check and triggers a second connection for the live-tail check. A clean live-tail connection (any frames decoded, or connection held open with no decode errors) produces a `Pass` live-tail check. +- **test-labeler.AC4.3 Success (empty labeler):** A labeler producing no frames at all during the budget produces an `Advisory` backfill check ("labeler has no published labels") and a `Skipped` live-tail check with the same reason. +- **test-labeler.AC4.4 Failure:** A frame that fails the two-CBOR-block decode, or a `#labels` payload that fails lexicon decoding, produces a `SpecViolation` `CheckResult` independent of the backfill/live-tail pass/fail dimension. The diagnostic carries the offending frame bytes as `NamedSource`. +- **test-labeler.AC4.5 Failure:** A WebSocket handshake that succeeds but whose first frame has `op: -1` (error) with a malformed `#info` payload produces a `SpecViolation` on decode. +- **test-labeler.AC4.6 NetworkError:** An unreachable WebSocket endpoint or a handshake TLS failure produces a `NetworkError` rather than a subscription `Fail`. +- **test-labeler.AC4.7 Edge:** Passing `--subscribe-timeout 0` (or another invalid duration) produces a clap parse error; values below a reasonable floor (e.g., 1s) are rejected with a helpful message. + +### test-labeler.AC5: Crypto-layer checks (signature verification with key rotation) + +- **test-labeler.AC5.1 Success:** Every label in `HttpFacts::first_page` whose signature verifies against the current declared signing key produces a `Pass` on the crypto rollup with no historic-key fetch performed. +- **test-labeler.AC5.2 Success (rotated-out key, did:plc):** When at least one label fails against the current key, the crypto stage fetches the PLC audit log and retries verification against each historic verification-method entry for the labeler's signing-key slot. Labels that verify against a historic key are accepted, the rollup is `Pass`, and an `Advisory` is attached listing the count and key ids involved. +- **test-labeler.AC5.3 Success (empty labeler):** A labeler with zero published labels results in the crypto stage being `Skipped("labeler published no labels; nothing to verify")` and does not affect the exit code. +- **test-labeler.AC5.4 Failure (current-key mismatch, no history):** A did:web labeler whose labels do not verify against the current key produces a `Fail` rollup with a diagnostic listing the current key id and stating that did:web provides no rotation history. +- **test-labeler.AC5.5 Failure (current and historic mismatch, did:plc):** A did:plc labeler whose labels verify against neither the current nor any historic key produces a `Fail` rollup with a diagnostic listing every key id that was tried. +- **test-labeler.AC5.6 Failure (canonicalization mismatch surfaced):** A label whose serialized-for-signing bytes cannot be produced (e.g., invalid CBOR in the fetched record) produces a per-label `Fail` with a distinct diagnostic code from signature mismatch, so misbehaviour of the canonicalizer is distinguishable from a genuine signature problem. +- **test-labeler.AC5.7 NetworkError:** A failure to fetch the PLC audit log (`plc.directory` unreachable) during the historic-key path produces a `NetworkError` result and prevents the stage from issuing a false `Fail`; the labels that failed against the current key are reported as `Fail` only if history could not be consulted. + +### test-labeler.AC6: Cross-cutting reporting and exit semantics + +- **test-labeler.AC6.1 Success:** A labeler that passes every stage produces a `LabelerReport` whose rendered output shows each stage with `[OK]` glyphs, a header listing the target and resolved DID/PDS, a summary with all-zero failure/advisory counts, and exits `0`. +- **test-labeler.AC6.2 Success:** A run containing at least one `SpecViolation` exits `1`, and the summary footer shows the count broken down by severity. +- **test-labeler.AC6.3 Success:** A run containing only `NetworkError` results (no `SpecViolation`s) exits `0`, with the network-error count called out separately in the summary. +- **test-labeler.AC6.4 Success:** A run containing only `Advisory` results exits `0`. +- **test-labeler.AC6.5 Success:** `Skipped` checks are rendered with a reason string taken from the stage's `Outcome::Skipped` variant, so users can see why a check was not run (missing DID, upstream failure, empty labeler, not-yet-implemented). +- **test-labeler.AC6.6 Success:** Setting `NO_COLOR=1` suppresses ANSI color codes in the rendered output while keeping ASCII glyphs (`[OK]`/`[FAIL]`/`[SKIP]`/`[WARN]`) and miette diagnostic layout intact. +- **test-labeler.AC6.7 Success:** `--verbose` raises the tracing filter, causing stage IO (HTTP requests, WebSocket frames, PLC log fetch) to be logged at DEBUG to stderr without affecting the rendered report or exit code. +- **test-labeler.AC6.8 Edge:** The tool's own unrecoverable bootstrap failures (invalid CLI args, panics caught by the miette handler, tokio runtime failure) exit `2` — distinct from a `SpecViolation`-driven `1`. + +## Glossary + +- **atproto (AT Protocol)**: The open, federated social-network protocol developed by Bluesky. It defines the identity, data, and communication standards that this tool validates against. +- **labeler**: An atproto service that publishes moderation labels — structured judgments that clients use to filter or annotate content. A labeler is identified by its DID and exposes two XRPC endpoints for querying and streaming its labels. +- **DID (Decentralized Identifier)**: A W3C standard for self-sovereign identity. In atproto, every account and service is identified by a DID; the DID resolves to a DID document that declares the entity's public keys and service endpoints. +- **did:plc**: An atproto-specific DID method maintained by a centralized-but-auditable directory (`plc.directory`). Key rotations and endpoint changes are recorded as a signed operation log, which this tool consults to accept labels signed by rotated-out keys. +- **did:web**: A DID method that derives identity from a domain's HTTPS server (e.g., `did:web:example.com` resolves via `https://example.com/.well-known/did.json`). Unlike did:plc, did:web provides no rotation history, so historic-key fallback is unavailable. +- **PLC audit log**: The append-only sequence of signed operations at `https://plc.directory/{did}/log/audit` recording every key rotation and service change for a did:plc identity. The crypto stage walks this log newest-to-oldest to find historic signing keys. +- **handle**: A human-readable atproto username (e.g., `alice.bsky.social`). Handles are resolved to DIDs via a DNS TXT record or an HTTPS `.well-known` path before any protocol checks can proceed. +- **PDS (Personal Data Server)**: The server that hosts a user's or labeler's repository of records. The identity stage fetches the `app.bsky.labeler.service/self` record from the labeler's declared PDS endpoint. +- **XRPC**: The HTTP-based RPC layer used by atproto. Procedure names like `com.atproto.label.queryLabels` are lexicon-defined endpoints served under the `/xrpc/` path prefix. +- **`com.atproto.label.queryLabels`**: The XRPC HTTP endpoint that returns a paginated list of labels published by a labeler. The HTTP stage calls this endpoint to fetch labels and verify schema conformance and pagination behavior. +- **`com.atproto.label.subscribeLabels`**: The XRPC WebSocket endpoint that streams label events as they are published, and replays historical labels when called with `cursor=0`. The subscription stage tests both the backfill replay and the live-tail behavior. +- **`app.bsky.labeler.service/self`**: The well-known record in a labeler's PDS repository that declares the labeler's policy configuration, including the set of label values it may apply. Absence of this record or an empty `labelValues` list is a spec violation. +- **`#atproto_labeler` service entry**: The entry in a DID document's `service` array with the fragment `#atproto_labeler` and type `AtprotoLabeler`, declaring the labeler's XRPC endpoint URL. The identity stage checks that this entry is present and well-formed. +- **DRISL-CBOR**: The deterministic CBOR serialization required by atproto for data that is cryptographically signed. It mandates strict map key ordering, minimal integer encoding, and no floats or indefinite-length items. The crypto stage canonicalizes each label to this format before hashing and verifying its signature. +- **multikey**: The atproto encoding for public keys in DID documents — a multibase-prefixed byte string that encodes both the curve identifier and the raw key bytes. This tool decodes multikeys to obtain `k256` (secp256k1) or `p256` (NIST P-256) verifying keys. +- **miette**: A Rust diagnostics library that provides rich, human-readable error reporting with source code spans, labels, and help text. This tool uses it to attach the offending JSON or CBOR bytes to every check failure, so users see exactly which field caused a violation. +- **`NamedSource` / `#[label]` spans**: The miette types that attach a named source document (e.g., a DID document's raw JSON) and highlight specific byte ranges within it in the rendered output. Each `CheckResult` failure uses these to point to the offending field. +- **`insta` snapshot testing**: A Rust testing library that records the text output of a function on first run and then asserts it matches on subsequent runs. Used here to lock down rendered `LabelerReport` output so any change in diagnostic formatting is caught automatically. +- **clap**: The standard Rust CLI argument-parsing library, used in derive mode here to generate the subcommand tree and argument validation declaratively. + +## Architecture + +### High-level shape + +`atproto-devtool` is a single-crate async Rust binary. The top-level CLI is a tree of clap-derived subcommands: the root `Command` enum has one initial variant `Test`, which itself is an enum whose initial variant `Labeler` holds the `test labeler` arguments. Adding a new feature is a two-line enum edit plus a new sibling module — the same pattern used by `zcash-devtool`. + +The first feature, `test labeler`, is a **layered pipeline** of four typed stages (identity → HTTP → subscription → crypto). Each stage is an async function whose signature consumes the outputs the stage actually needs, so the type system enforces the dependency graph: crypto verification cannot be called without a resolved signing key and a set of labels to verify. The pipeline driver runs every stage whose inputs are available — no fail-fast — so a single invocation produces a complete diagnostic picture. + +Each stage emits one or more `CheckResult`s tagged with a named check ID, a status (`Pass | Fail | Skipped | Advisory`), and — for non-`Pass` results — a `miette::Diagnostic` with source spans pointing into the JSON/CBOR document that caused the violation. The collected `LabelerReport` is rendered through `miette`'s `GraphicalReportHandler` at the end of the run. + +### Crate layout + +Sibling-file module pattern throughout (no `mod.rs`): + +```text +atproto-devtool/ +├── Cargo.toml +├── src/ +│ ├── main.rs # tokio bootstrap +│ ├── cli.rs # root clap Parser + error reporting +│ ├── commands.rs # top-level Command enum +│ ├── commands/ +│ │ ├── test.rs # Test subcommand enum { Labeler(..) } +│ │ └── test/ +│ │ ├── labeler.rs # clap Args + pipeline entry point +│ │ └── labeler/ +│ │ ├── pipeline.rs # drives the four stages, assembles Report +│ │ ├── identity.rs # stage 1: DID doc + labeler record +│ │ ├── http.rs # stage 2: queryLabels +│ │ ├── subscription.rs # stage 3: subscribeLabels WS +│ │ ├── crypto.rs # stage 4: signature verification +│ │ └── report.rs # LabelerReport + miette rendering +│ ├── common.rs # cross-feature primitives +│ └── common/ +│ ├── identity.rs # handle→DID, DID doc fetch, service lookup +│ └── diagnostics.rs # miette helpers shared by all features +``` + +### Pipeline contracts + +The pipeline operates on two input shapes that reflect the two invocation modes: + +```rust +pub enum LabelerTarget { + Identified { identifier: AtIdentifier }, // handle or DID + Endpoint { url: Url, did: Option }, // raw endpoint, optional DID +} +``` + +Each stage returns an `Outcome`: + +```rust +pub enum Outcome { + Pass(T), + Fail(Diagnostic), + Skipped(&'static str), +} +``` + +Stage signatures: + +```rust +identity::run(&LabelerTarget, &IdentityOptions) + -> Outcome; + +http::run(&Url /* labeler endpoint */, &HttpOptions) + -> Outcome; + +subscription::run(&Url, &SubscriptionOptions) + -> Outcome; + +crypto::run(&IdentityFacts, &[Label]) + -> Outcome; + +pub struct IdentityFacts { + pub did: Did, + pub did_doc: DidDocument, // carries source bytes for span-tagging + pub labeler_endpoint: Url, + pub pds_endpoint: Url, + pub signing_key: VerifyingKey, // k256 or p256, parsed from multikey + pub service_record: LabelerServiceRecord, +} + +pub struct HttpFacts { + pub first_page: Vec