From b120c162d0236a1934623b4c72aef90d5fed9a01 Mon Sep 17 00:00:00 2001 From: Patrick Singletary Date: Tue, 18 Aug 2026 11:32:07 -0400 Subject: [PATCH] Add atproto-cid, atproto-repository, atproto-attestation, atproto-oauth skills (ported from ngerakines/atproto-skills, MIT) - Language-neutral specs + TypeScript guides kept verbatim in references/ - Rust/Go guides omitted (not in user stack) - Validator scripts: validate_cid.py, validate_client_metadata.py - atproto-development: cross-link to atproto-oauth for DPoP/PAR depth --- .../atproto-attestation/SKILL.md | 128 +++++++ .../references/shared/cid-computation.md | 150 ++++++++ .../references/shared/divergence-matrix.md | 175 +++++++++ .../references/shared/inline-attestation.md | 106 ++++++ .../references/shared/remote-attestation.md | 132 +++++++ .../shared/signature-normalization.md | 75 ++++ .../references/shared/spec.md | 201 ++++++++++ .../references/shared/test-vectors.md | 81 ++++ .../references/typescript/README.md | 198 ++++++++++ .../references/typescript/creating.md | 360 ++++++++++++++++++ .../references/typescript/signatures.md | 189 +++++++++ .../references/typescript/verifying.md | 287 ++++++++++++++ .../software-development/atproto-cid/SKILL.md | 120 ++++++ .../references/shared/binary-layout.md | 136 +++++++ .../references/shared/divergence-matrix.md | 93 +++++ .../atproto-cid/references/shared/spec.md | 94 +++++ .../references/shared/test-vectors.md | 128 +++++++ .../references/typescript/README.md | 122 ++++++ .../references/typescript/codecs.md | 121 ++++++ .../references/typescript/construction.md | 153 ++++++++ .../references/typescript/parsing.md | 167 ++++++++ .../atproto-cid/scripts/validate_cid.py | 57 +++ .../atproto-development/SKILL.md | 7 + .../atproto-oauth/SKILL.md | 104 +++++ .../references/shared/client-metadata.md | 234 ++++++++++++ .../references/shared/divergence-matrix.md | 170 +++++++++ .../atproto-oauth/references/shared/dpop.md | 162 ++++++++ .../atproto-oauth/references/shared/flows.md | 312 +++++++++++++++ .../atproto-oauth/references/shared/scopes.md | 203 ++++++++++ .../shared/security-requirements.md | 188 +++++++++ .../references/shared/sessions.md | 198 ++++++++++ .../atproto-oauth/references/shared/spec.md | 155 ++++++++ .../references/shared/test-vectors.md | 253 ++++++++++++ .../references/shared/troubleshooting.md | 231 +++++++++++ .../references/typescript/README.md | 202 ++++++++++ .../references/typescript/client-metadata.md | 161 ++++++++ .../references/typescript/dpop.md | 133 +++++++ .../references/typescript/flows.md | 217 +++++++++++ .../references/typescript/sessions.md | 312 +++++++++++++++ .../scripts/validate_client_metadata.py | 221 +++++++++++ .../atproto-repository/SKILL.md | 103 +++++ .../references/shared/car-v1.md | 220 +++++++++++ .../references/shared/commit-and-signing.md | 185 +++++++++ .../references/shared/data-model.md | 211 ++++++++++ .../references/shared/divergence-matrix.md | 145 +++++++ .../references/shared/drisl.md | 127 ++++++ .../references/shared/mst.md | 253 ++++++++++++ .../references/shared/test-vectors.md | 356 +++++++++++++++++ .../references/typescript/README.md | 124 ++++++ .../references/typescript/car.md | 209 ++++++++++ .../references/typescript/commit.md | 247 ++++++++++++ .../references/typescript/drisl.md | 142 +++++++ .../references/typescript/mst.md | 241 ++++++++++++ 53 files changed, 9299 insertions(+) create mode 100644 skills/software-development/atproto-attestation/SKILL.md create mode 100644 skills/software-development/atproto-attestation/references/shared/cid-computation.md create mode 100644 skills/software-development/atproto-attestation/references/shared/divergence-matrix.md create mode 100644 skills/software-development/atproto-attestation/references/shared/inline-attestation.md create mode 100644 skills/software-development/atproto-attestation/references/shared/remote-attestation.md create mode 100644 skills/software-development/atproto-attestation/references/shared/signature-normalization.md create mode 100644 skills/software-development/atproto-attestation/references/shared/spec.md create mode 100644 skills/software-development/atproto-attestation/references/shared/test-vectors.md create mode 100644 skills/software-development/atproto-attestation/references/typescript/README.md create mode 100644 skills/software-development/atproto-attestation/references/typescript/creating.md create mode 100644 skills/software-development/atproto-attestation/references/typescript/signatures.md create mode 100644 skills/software-development/atproto-attestation/references/typescript/verifying.md create mode 100644 skills/software-development/atproto-cid/SKILL.md create mode 100644 skills/software-development/atproto-cid/references/shared/binary-layout.md create mode 100644 skills/software-development/atproto-cid/references/shared/divergence-matrix.md create mode 100644 skills/software-development/atproto-cid/references/shared/spec.md create mode 100644 skills/software-development/atproto-cid/references/shared/test-vectors.md create mode 100644 skills/software-development/atproto-cid/references/typescript/README.md create mode 100644 skills/software-development/atproto-cid/references/typescript/codecs.md create mode 100644 skills/software-development/atproto-cid/references/typescript/construction.md create mode 100644 skills/software-development/atproto-cid/references/typescript/parsing.md create mode 100755 skills/software-development/atproto-cid/scripts/validate_cid.py create mode 100644 skills/software-development/atproto-oauth/SKILL.md create mode 100644 skills/software-development/atproto-oauth/references/shared/client-metadata.md create mode 100644 skills/software-development/atproto-oauth/references/shared/divergence-matrix.md create mode 100644 skills/software-development/atproto-oauth/references/shared/dpop.md create mode 100644 skills/software-development/atproto-oauth/references/shared/flows.md create mode 100644 skills/software-development/atproto-oauth/references/shared/scopes.md create mode 100644 skills/software-development/atproto-oauth/references/shared/security-requirements.md create mode 100644 skills/software-development/atproto-oauth/references/shared/sessions.md create mode 100644 skills/software-development/atproto-oauth/references/shared/spec.md create mode 100644 skills/software-development/atproto-oauth/references/shared/test-vectors.md create mode 100644 skills/software-development/atproto-oauth/references/shared/troubleshooting.md create mode 100644 skills/software-development/atproto-oauth/references/typescript/README.md create mode 100644 skills/software-development/atproto-oauth/references/typescript/client-metadata.md create mode 100644 skills/software-development/atproto-oauth/references/typescript/dpop.md create mode 100644 skills/software-development/atproto-oauth/references/typescript/flows.md create mode 100644 skills/software-development/atproto-oauth/references/typescript/sessions.md create mode 100644 skills/software-development/atproto-oauth/scripts/validate_client_metadata.py create mode 100644 skills/software-development/atproto-repository/SKILL.md create mode 100644 skills/software-development/atproto-repository/references/shared/car-v1.md create mode 100644 skills/software-development/atproto-repository/references/shared/commit-and-signing.md create mode 100644 skills/software-development/atproto-repository/references/shared/data-model.md create mode 100644 skills/software-development/atproto-repository/references/shared/divergence-matrix.md create mode 100644 skills/software-development/atproto-repository/references/shared/drisl.md create mode 100644 skills/software-development/atproto-repository/references/shared/mst.md create mode 100644 skills/software-development/atproto-repository/references/shared/test-vectors.md create mode 100644 skills/software-development/atproto-repository/references/typescript/README.md create mode 100644 skills/software-development/atproto-repository/references/typescript/car.md create mode 100644 skills/software-development/atproto-repository/references/typescript/commit.md create mode 100644 skills/software-development/atproto-repository/references/typescript/drisl.md create mode 100644 skills/software-development/atproto-repository/references/typescript/mst.md diff --git a/skills/software-development/atproto-attestation/SKILL.md b/skills/software-development/atproto-attestation/SKILL.md new file mode 100644 index 0000000..0b87d17 --- /dev/null +++ b/skills/software-development/atproto-attestation/SKILL.md @@ -0,0 +1,128 @@ +--- +name: atproto-attestation +description: "Use for badge.blue record attestations: inline/remote, ECDSA signing." +version: 1.0.0 +author: Hermes Agent (ported from ngerakines/atproto-skills, MIT) +license: MIT +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [ATProto, Attestation, badge.blue, ECDSA, Signature, strongRef] +--- + +# AT Protocol record attestations (badge.blue) + +CID-first record attestations per (reference impl: the +`atproto-attestation` Rust crate). An attestation binds a cryptographic or +content-addressed claim to a specific record in a specific repository. + +Ported from `ngerakines/atproto-skills` (MIT). Spec + TypeScript guides kept +verbatim under `references/`; Rust/Go omitted (fetch upstream if needed). + +## Defaults + +- **Content CID**: CIDv1, codec `0x71` (dag-cbor), hash `0x12` (SHA-256), + 32-byte digest, 36-byte binary form. Computed over a canonical merge of + `record` (without `signatures`) + `$sig` metadata (without `cid`/`signature`, + with `repository` inserted). +- **Inline attestation**: the 36-byte CID bytes are ECDSA-signed (P-256 or + K-256), normalized to low-S, base64-encoded (standard alphabet + padding), + embedded in `record.signatures[]` with the metadata. +- **Remote attestation**: the content CID is written into a separate *proof + record* in the attestor's repo; the subject record's `signatures[]` carries a + `com.atproto.repo.strongRef` pointing at the proof record. + +## What gets signed + +Neither the raw record nor the metadata alone — the signature is over the +**36-byte binary content CID** of a canonical merge: + +``` +record' = record without `signatures` +meta' = metadata without `cid` / `signature`, with `repository` inserted +merged = record' ∪ { "$sig": meta' } + +content_cid = CIDv1(dag-cbor, SHA-256(DAG-CBOR(merged))) +signature = ECDSA_sign(private_key, content_cid.bytes) +``` + +`repository` participates in the CID but NOT in the stored attestation — replay +protection: the same record + metadata signed for `did:plc:A` yields a +different CID than for `did:plc:B`. + +## Inline vs remote + +| Trait | Inline | Remote | +| --------------- | --------------------------------------- | --------------------------------------------- | +| Cryptographic | Yes — ECDSA signature in the record | No — integrity by content-address + strongRef | +| Records involved| 1 (subject only) | 2 (subject + proof, usually different repos) | +| `signatures[]` | metadata object with `cid` + `signature.$bytes` | `com.atproto.repo.strongRef` with `uri` + `cid` | +| Verifier needs | public key resolution | record resolver (AT-URI fetch) | +| Revocation | immutable once signed | delete proof record → unreachable | + +## Two CIDs in remote attestations (most common bug) + +- **Content CID** — inside the proof record's `cid` field; identifies the + signing payload (record + `$sig`). +- **Proof CID** — inside the strongRef's `cid` field; the proof record itself + (plain DAG-CBOR). + +Always verify both. + +## Curve support + +- P-256: full support everywhere. +- K-256: full support everywhere (dcrec defaults low-S; noble secp256k1 too). +- **P-384: the reference crate's `normalize_signature` returns + `UnsupportedKeyType` — interop broken. Use P-256 or K-256 only.** + +## Common pitfalls + +- **Signing the CID string (`bafyrei…`) instead of the 36-byte binary CID** — + silent interop break. +- **Signing the 32-byte digest instead of the 36-byte CID bytes** — same. +- **Including `repository` in the stored attestation** — transient input only. +- **Forgetting to strip `signatures` from the record before CID computation** — + every new signature re-signs the stripped version so prior ones stay valid; + skip the strip and all signatures invalidate each other. +- **Forgetting to strip `cid`/`signature` from metadata before CID computation** + — these are outputs, not inputs. +- **DER-encoded signatures** — ECDSA libs often return DER (70–72 bytes); spec + requires IEEE P1363 `r‖s` (64 bytes for P-256/K-256). Convert if needed. +- **Skipping low-S normalization** — normalize explicitly (Go stdlib, OpenSSL, + noble p256 default don't). +- **URL-safe base64 for `signature.$bytes`** — spec uses standard base64 + (`+`/`/` + padding). +- **Publishing the attested record before the proof record** — dangling + strongRef on network hiccup. Publish proof first. +- **Non-canonical CBOR** — use strict DAG-CBOR libraries only. + +## Decision rules + +- **Inline or remote?** Inline if the attestor holds their own key and signs + in-process. Remote if the attestor is a separate service or the attestation + must be independently deletable. +- **Multiple attestations?** Fully supported — each signature is computed over + the record with `signatures` stripped. +- **`repository` at verify time** — the DID of the repo you fetched the record + from. Hardcoding or stale value silently invalidates every inline signature. +- **Validation vs verification** — validation = structurally well-formed; + verification = recompute the signature/CID and compare. Policy checks + (issuer authorization, freshness) live above the crypto. + +## References + +- `references/shared/spec.md` — normative rules (read first) +- `references/shared/cid-computation.md` — bit-exact content-CID procedure + (most implementation bugs live here) +- `references/shared/inline-attestation.md`, `remote-attestation.md` +- `references/shared/signature-normalization.md` — low-S rules, P-384 gap +- `references/shared/test-vectors.md`, `divergence-matrix.md` +- `references/typescript/README.md`, `creating.md`, `verifying.md`, + `signatures.md` — @noble/curves + @ipld/dag-cbor stack + +## Related skills + +- `atproto-cid` — the CID primitive the whole scheme signs over +- `atproto-repository` — DAG-CBOR/DRISL canonicalization for the merge +- `atproto-development` / `atproto-python` — publishing the records diff --git a/skills/software-development/atproto-attestation/references/shared/cid-computation.md b/skills/software-development/atproto-attestation/references/shared/cid-computation.md new file mode 100644 index 0000000..66b3024 --- /dev/null +++ b/skills/software-development/atproto-attestation/references/shared/cid-computation.md @@ -0,0 +1,150 @@ +# Content CID computation + +The content CID is the thing that gets signed (inline) or referenced (remote). Every implementation must compute it bit-exactly the same way or signatures won't verify across languages. This file is the procedure, annotated. + +## Inputs + +- `record_obj` — the subject record as a JSON object. Must have a `$type` field. +- `metadata_obj` — the attestation metadata as a JSON object. Must have a `$type` field. +- `repository` — the DID string of the repo where the subject record lives. + +## Procedure + +### Step 1. Validate inputs + +- Both `record_obj` and `metadata_obj` must be JSON objects (not arrays, not scalars). Reject otherwise. +- Both must have a non-empty string `$type` field. Reject otherwise. + +Reference crate raises `RecordMustBeObject`, `MetadataMustBeObject`, `RecordMissingType`, `MetadataMissingSigType` respectively. + +### Step 2. Strip the record + +From `record_obj`, remove the `signatures` field if present. All other fields pass through unchanged. + +Rationale: the record must canonicalize to the same thing regardless of what's already in `signatures`. Signing a record twice (two inline attestations, or inline + remote) requires each signature to compute over a record that *excludes* all existing signatures. + +### Step 3. Prepare `$sig` metadata + +Starting from `metadata_obj`: + +1. Remove `cid` if present. (This is a computed output, not an input.) +2. Remove `signature` if present. (Inline attestations write this field *after* CID computation.) +3. Insert `repository: ` as a string value. + +All other metadata fields pass through unchanged — **they participate in the CID**. Adding, removing, or changing any custom field (`issuer`, `purpose`, `issuedAt`, anything else) invalidates all signatures. + +### Step 4. Merge + +Insert the prepared metadata into the stripped record under key `$sig`: + +``` +record_obj["$sig"] = metadata_obj +``` + +This is a single top-level field addition. The record should now have every original field (minus `signatures`), plus `$sig`. + +### Step 5. Encode + +Serialize the merged object to **DAG-CBOR**. + +DAG-CBOR is a canonical subset of CBOR. Key rules the encoder must follow: + +- Map keys sorted lexicographically by their UTF-8 byte sequence. +- Integers encoded in their shortest form (no leading zero bytes in multi-byte integers). +- Floats always encoded as 64-bit (8-byte) IEEE 754. +- Strings are definite-length. No indefinite-length strings, arrays, or maps. +- Tags: only tag 42 (CID link) is permitted. No other tags. +- No duplicate keys. + +Every major language has a DAG-CBOR library — use it, don't hand-roll CBOR. See per-language guides for recommended libraries. + +### Step 6. Hash + +SHA-256 the DAG-CBOR bytes. 32-byte digest. + +### Step 7. Wrap as CIDv1 + +Build a CIDv1: + +- Version: `1` (byte `0x01`). +- Codec: `0x71` (dag-cbor). +- Multihash code: `0x12` (SHA-256). +- Multihash length: `0x20` (32). +- Digest: the 32 bytes from step 6. + +Binary form: `01 71 12 20 <32 bytes>` = 36 bytes total. + +String form: `b` + base32lower(binary). Always starts with `bafyrei…` for this codec+hash pair. + +This is the **content CID**. It is what inline attestations sign and what remote attestations reference. + +## What "sign the CID bytes" means + +For inline attestations, step 2 of signing is: + +``` +signature = ECDSA_sign(private_key, content_cid_bytes) +``` + +Where `content_cid_bytes` is the **36-byte binary form** of the content CID (`01 71 12 20 `). **Not** the string form, **not** just the digest. The reference crate uses `cid.to_bytes()` which returns the 36-byte binary form. + +This is important: it means the signature covers the full CID header (including codec and hash algorithm identifiers), not just the 32-byte hash. A substitution attack that tried to reinterpret the digest under a different hash function would produce different signed bytes. + +## Worked micro-example + +Record: + +```json +{"$type": "app.example.post", "text": "hi"} +``` + +Metadata: + +```json +{"$type": "com.example.sig", "key": "did:key:zEXAMPLE", "purpose": "demo"} +``` + +Repository: `did:plc:abc123` + +After step 2–4, the object to encode is: + +```json +{ + "$sig": { + "$type": "com.example.sig", + "key": "did:key:zEXAMPLE", + "purpose": "demo", + "repository": "did:plc:abc123" + }, + "$type": "app.example.post", + "text": "hi" +} +``` + +(Note the sort order: `$sig` before `$type` because `$` (0x24) is the same, then `s` (0x73) < `t` (0x74).) + +DAG-CBOR encode → SHA-256 → wrap as CIDv1. The exact byte output is deterministic; see `test-vectors.md` for runnable fixtures. + +## What is NOT in the signed payload + +- The `signatures` array (stripped in step 2). +- Fields named `cid` or `signature` inside the metadata (stripped in step 3). +- Any whitespace, key ordering, or formatting from the JSON you were handed — DAG-CBOR re-encodes from the object model. +- The private key, the public key, the issuer identity as a separate input. The only identity input to the CID is `repository`. + +## Common mistakes + +- **Forgetting to strip `signatures` before encoding.** Produces a different CID than what the reference implementation generates. Every new signature would be invalid. +- **Leaving `cid` / `signature` inside the metadata before merge.** Same problem: the reference strips them and yours doesn't. +- **Forgetting to insert `repository`.** Kills replay protection and produces a CID that won't match the reference's output. +- **Signing the digest instead of the full 36-byte CID.** Silent interop break — implementations that sign `cid.to_bytes()` will not verify signatures that sign `cid.hash().digest()`. +- **Using non-canonical CBOR.** Indefinite-length strings, unsorted keys, or inefficient integer encoding all produce different bytes. Use a DAG-CBOR library, not generic CBOR. +- **Copying the stored attestation back as metadata on re-verify without stripping `cid` and `signature`.** The verifier must re-apply step 3 to recover the CID-time view. +- **Serializing the outer object as JSON and then CBOR-encoding that string.** Double encoding. The object must go straight into the DAG-CBOR encoder. + +## See also + +- `spec.md` §4 — the high-level overview this file expands. +- `inline-attestation.md` — how the content CID feeds into ECDSA signing. +- `remote-attestation.md` — how the content CID sits inside the proof record. +- `../rust/signatures.md`, `../typescript/signatures.md`, `../go/signatures.md` — library-specific DAG-CBOR + SHA-256 + CID assembly. diff --git a/skills/software-development/atproto-attestation/references/shared/divergence-matrix.md b/skills/software-development/atproto-attestation/references/shared/divergence-matrix.md new file mode 100644 index 0000000..374f93c --- /dev/null +++ b/skills/software-development/atproto-attestation/references/shared/divergence-matrix.md @@ -0,0 +1,175 @@ +# Divergence matrix + +A head-to-head comparison of Rust, TypeScript, and Go implementations of badge.blue attestations. This file exists so that porting work and interop reviews have one authoritative place to check "does X differ across languages?" + +The Rust reference crate (`atproto-attestation` in the `ngerakines.me/atproto-crates` workspace) is treated as canonical. + +## Library coverage + +| Concern | Rust | TypeScript | Go | +| -------------------- | --------------------------------------- | ----------------------------------- | ----------------------------------------------- | +| Canonical library | `atproto-attestation` crate | **none** — assemble from primitives | **none** — assemble from primitives | +| DAG-CBOR | `atproto-dasl` (internal) | `@ipld/dag-cbor` | `github.com/ipld/go-ipld-prime/codec/dagcbor` | +| CID | `cid` crate | `multiformats/cid` | `github.com/ipfs/go-cid` | +| ECDSA P-256 | `p256` | `@noble/curves/p256` | stdlib `crypto/ecdsa` | +| ECDSA K-256 | `k256` | `@noble/curves/secp256k1` | `github.com/decred/dcrd/dcrec/secp256k1/v4` | +| ECDSA P-384 | `atproto-identity` (sign/validate only) | `@noble/curves/p384` | stdlib `crypto/ecdsa` | +| Low-S normalization | `normalize_signature` | noble `{ lowS: true }` / `normalizeS()` | hand-rolled with `big.Int` | +| TID generation | `atproto_record::tid::Tid` | hand-rolled or `@atproto/common` | `indigo/atproto/syntax` or hand-rolled | +| Remote record fetch | `atproto_client::RecordResolver` | caller-provided `RecordResolver` | caller-provided `RecordResolver` | +| DID key parsing | `atproto_identity::identify_key` | `multiformats/bases/base58` + varint | `multiformats/go-multibase` + `go-varint` | + +## Curve support + +| Curve | Rust crate | TypeScript (`@noble/curves`) | Go stdlib / dcrec | +| ----- | -------------------------------------------- | ----------------------------------------------- | ------------------------------------------------------- | +| P-256 | ✅ sign, verify, normalize | ✅ sign, verify, normalize (`{lowS: true}`) | ✅ sign, verify — manual low-S | +| K-256 | ✅ sign, verify, normalize | ✅ sign (low-S default), verify, normalize | ✅ sign (dcrec low-S by default), verify | +| P-384 | ⚠️ sign/verify via `atproto-identity`; **normalize returns `UnsupportedKeyType`** | ✅ sign, verify, normalize — but interop broken (no Rust partner) | ✅ sign, verify — manual low-S; same interop break | + +**Interop rule**: use P-256 or K-256 only. P-384 does not round-trip through the reference crate's signing/append flow today. + +## Signature wire format + +All three languages settle on: + +- IEEE P1363 `r‖s`, **not** DER. +- 64 bytes (P-256/K-256); 96 bytes (P-384). +- Big-endian, zero-padded. + +| Language | Getting P1363 | Getting DER | +| ---------- | ----------------------------------------------------- | ------------------------------------ | +| Rust | `p256::ecdsa::Signature::to_vec()` / `k256::…` | `.to_der()` (deliberately unused) | +| TypeScript | `sig.toCompactRawBytes()` | `sig.toDERRawBytes()` (don't use) | +| Go | hand-assemble from `r.Bytes()` + `s.Bytes()` | `ecdsa.SignASN1` / `asn1.Marshal` | + +### Base64 + +All three use **standard** base64 with `=` padding. None use URL-safe. + +| Language | Encoder | +| ---------- | ---------------------------------------------------------- | +| Rust | `base64::engine::general_purpose::STANDARD` | +| TypeScript | `Buffer.toString("base64")` / `btoa(String.fromCharCode…)` | +| Go | `base64.StdEncoding` | + +## CID computation + +All three must produce identical CIDv1 bytes given the same inputs. The pipeline is: + +1. Strip `signatures` from record. +2. Strip `cid`/`signature` from metadata, insert `repository`. +3. Merge metadata under `$sig` key. +4. DAG-CBOR encode (canonical: sorted keys, minimal ints, 64-bit floats, definite lengths). +5. SHA-256. +6. CIDv1, codec `0x71`, multihash `0x12`. + +### Canonical DAG-CBOR — per-language caveats + +- **Rust**: `atproto-dasl` is strict DAG-CBOR; no configuration needed. +- **TypeScript**: `@ipld/dag-cbor` is strict DAG-CBOR. +- **Go**: `go-ipld-prime` is strict DAG-CBOR. `fxamacker/cbor` with `CoreDetEncOptions` is close but does not handle CBOR tag 42 (CID links) without custom registration. For attestation *metadata* (strings and basic types), both work; for records that embed CIDs, use go-ipld-prime. + +### Float handling + +DAG-CBOR encodes all floats as 64-bit IEEE 754. JS's `number` type is always a 64-bit float — but `1` and `1.0` round-trip as the same value, so encoding is deterministic. Rust / Go distinguish `i64` and `f64` at the type level; be careful when shaping records that include numbers — an `i64` intent encoded as `f64` produces a different CID. + +Practical rule: avoid floats in attestation records and metadata. If you must use them, fix types and test vectors. + +### Integer minimization + +All three libraries emit CBOR major type 0/1 with the shortest possible encoding (1-byte for 0–23, 2-byte for 24–255, etc.). No divergence. + +## Hashing algorithms + +| Step | Rust | TypeScript | Go | +| -------------------------- | ---------------------------- | -------------------------------------- | ----------------------------------- | +| DAG-CBOR body → SHA-256 | `sha2::Sha256` | `multiformats/hashes/sha2` (`SubtleCrypto.digest` or `@noble/hashes`) | `crypto/sha256` | +| ECDSA digest (internal) | RustCrypto library internal | `@noble/curves` internal SHA-256 | `crypto/sha256` explicit | + +Go is the odd one here: callers pass the **digest** to `ecdsa.Sign`, while Rust and TS pass the **message** (hashing happens inside the library). The outputs are equivalent because the internal hash is also SHA-256. + +## Verify permissiveness + +| Behavior | Rust reference | TS (suggested) | Go (suggested) | +| -------------------------------------------- | -------------- | -------------------- | -------------------- | +| Accepts high-S inline signatures | ✅ yes | ✅ yes by default | ✅ yes by default | +| Optional strict low-S | ❌ not exposed | ✅ `strictLowS: true` | ✅ `StrictLowS: true` | +| Verifies proof record's DAG-CBOR CID match | ❌ no | ✅ default on | ✅ default on | +| Verifies content CID inside proof record | ✅ yes | ✅ yes | ✅ yes | + +The TS and Go implementations *default* to a stricter posture (verify proof record CID) than the Rust reference. This is a defensible improvement — callers who want byte-for-byte Rust behavior pass `verifyProofCid: false` / `VerifyProofCid: false`. + +## Async surface + +| Flow | Rust | TypeScript | Go | +| ------------------------------------- | ----------------------------- | -------------- | ------------------------------------ | +| `create_inline_attestation` | sync | async (SHA256) | sync | +| `create_remote_attestation` | sync | async (SHA256) | sync | +| `append_inline_attestation` | **async** (key resolution) | async | async (context.Context threaded) | +| `verify_record` | **async** (fetches + resolve) | async | async | + +TypeScript is async everywhere because `SubtleCrypto.digest` is async. The Rust and Go paths are sync for create (pure CPU) and async only where I/O happens (remote resolution). + +## TID generation + +| Concern | Rust | TypeScript | Go | +| -------------------------- | ----------------------------- | --------------------------------- | ----------------------------------- | +| Library | `atproto_record::tid::Tid::new()` | hand-roll or `@atproto/common` | `indigo/atproto/syntax.NewTID()` or hand-roll | +| Format | 13-char base32-sortable | same | same | +| Clock skew protection | ✅ monotonic with last seen | must be implemented in hand-roll | must be implemented in hand-roll | + +All three must produce `syntax`-valid TIDs (base32, 13 chars, high bit clear). Use the library if possible. + +## `RecordResolver` / `KeyResolver` trait shapes + +| Trait | Rust | TypeScript | Go | +| -------------- | --------------------------------------------------------------- | ------------------------------------------------------- | ----------------------------------------------------------- | +| KeyResolver | `async fn resolve(&self, key: &str) -> Result` | `resolveKey(key: string) => Promise<{curve, publicKey}>` | `ResolveKey(ctx, keyRef) (Curve, any, error)` | +| RecordResolver | `async fn resolve(&self, uri: &str)` | `resolveRecord(uri: string) => Promise>` | `ResolveRecord(ctx, uri) (map[string]any, error)` | + +Shapes are equivalent in intent. The Rust variant is generic over `T` (you get typed records out); TS/Go return dynamic maps. For proof records (usually string-typed metadata) this isn't consequential. + +## CLI tooling + +| Language | CLI available? | +| ---------- | ------------------------------------------------------------ | +| Rust | ✅ `atproto-attestation-sign` / `…-verify` in the crate | +| TypeScript | ❌ none published; easy to wrap `createInlineAttestation` with `yargs` | +| Go | ❌ none published; easy to wrap with `cobra` / flag | + +The Rust CLIs are the quickest path to producing a signed record for cross-language verification. See `test-vectors.md`. + +## Error taxonomies + +Rust has a single numbered enum (`AttestationError`) with codes `error-atproto-attestation-1` through `…-30`. TS and Go have no standard scheme — suggest mapping to similarly-structured error types in your port: + +| Rust variant | Meaning | +| ------------------------------------ | --------------------------------------------------------- | +| `RecordMustBeObject` | input was null/array/scalar | +| `MetadataMissingType` | `$type` missing from metadata | +| `RemoteAttestationCidMismatch` | proof record's claimed content CID ≠ computed | +| `SignatureValidationFailed` | ECDSA verify returned false | +| `UnsupportedKeyType` | non-P-256/K-256 passed to `normalize_signature` | +| `KeyResolutionFailed` | `KeyResolver` returned an error | +| `RemoteAttestationFetchFailed` | `RecordResolver` returned an error | +| `SignatureLengthInvalid` | signature bytes not exactly 64 (or expected length) | + +Port these as distinct error types in TS/Go so consumer code can pattern-match rather than parsing strings. + +## Known porting hazards + +1. **P-384 normalization** — the Rust crate doesn't implement it. TS and Go *can* but produce signatures the Rust crate can't renormalize on append. Block P-384 for interop. +2. **Go non-deterministic signing** — `crypto/ecdsa.Sign` uses `rand.Reader`. Test vectors generated in Go won't match Rust / TS bit-for-bit without a deterministic signer. Use `dcrec` for K-256 (RFC 6979 default) and `cloudflare/circl` for P-256/P-384 with deterministic mode. +3. **TypeScript `1` vs `1.0`** — if a metadata value is a JS `number` that happens to be integer-valued, it encodes as a CBOR integer, not float. A Rust caller may have typed it as `f64` and encoded as a float. If metadata contains numbers, align types carefully. +4. **Float-valued metadata in general** — avoid. Use strings, bools, ints, and nested objects. +5. **Proof record transmogrification** — the Rust crate sends proof records through `atproto-dasl`'s canonical encoder. TS/Go using `@ipld/dag-cbor` or `go-ipld-prime` get canonical output too. But if your proof record contains a CID (`$link`), that needs tag-42 encoding — none of the naive paths handle this without special handling. +6. **Base64 alphabet drift** — easy to slip on `URL_SAFE` or `NO_PAD`. Double-check: standard alphabet with padding. +7. **Key bytes format for `did:key:`** — all three curves use **compressed** SEC1 (33 bytes for P-256/K-256, 49 bytes for P-384). Uncompressed form (65/97 bytes) won't round-trip. + +## See also + +- Each per-language `README.md` for stack choices. +- `../rust/signatures.md`, `../typescript/signatures.md`, `../go/signatures.md` — per-language crypto details. +- `cid-computation.md`, `signature-normalization.md` — the spec that everyone has to match. +- `test-vectors.md` — current interop fixtures. diff --git a/skills/software-development/atproto-attestation/references/shared/inline-attestation.md b/skills/software-development/atproto-attestation/references/shared/inline-attestation.md new file mode 100644 index 0000000..6ccc807 --- /dev/null +++ b/skills/software-development/atproto-attestation/references/shared/inline-attestation.md @@ -0,0 +1,106 @@ +# Inline attestations + +Cryptographic signatures embedded directly in a record's `signatures` array. Self-contained — a verifier needs only the record itself and the issuer's public key. + +## Record shape + +Final record (after signing): + +```json +{ + "$type": "app.bsky.feed.post", + "text": "Hello world!", + "createdAt": "2024-01-01T00:00:00.000Z", + "signatures": [ + { + "$type": "com.example.inlineSignature", + "key": "did:key:zQ3shNzMp4oaaQ1gQRzCxMGXFrSW3NEM1M9T6KCY9eA7HhyEA", + "issuer": "did:plc:issuer123", + "issuedAt": "2024-01-01T00:00:00.000Z", + "cid": "bafyrei...", + "signature": { "$bytes": "MEQCIA..." } + } + ] +} +``` + +Required attestation fields: + +| Field | Type | Notes | +| ------------------ | ------ | ----------------------------------------------------------------------------------------- | +| `$type` | string | Attestor-chosen NSID. | +| `key` | string | `did:key:…` or other resolvable key reference. | +| `cid` | string | Content CID, base32 string form (`bafyrei…`). | +| `signature.$bytes` | string | Base64 (standard alphabet, with padding) of the 64-byte low-S normalized ECDSA signature. | + +Optional attestation fields: any — common choices are `issuer` (DID), `issuedAt` (RFC 3339 datetime), `purpose`. All optional fields **participate in the CID**, so changing them invalidates the signature. + +## Create — procedure + +Given: `record`, `metadata` (without `cid` / `signature`), `repository` DID, `private_key`. + +1. Compute the content CID per `cid-computation.md`. Input: `record`, `metadata`, `repository`. +2. Sign the **36-byte binary CID** with ECDSA: `raw_signature = ECDSA_sign(private_key, content_cid.to_bytes())`. +3. Normalize `raw_signature` to low-S form (see `signature-normalization.md`). +4. Base64-encode the normalized signature (standard alphabet, with `=` padding). +5. Build the final attestation object by starting from `metadata` and adding: + - `cid`: content CID string form. + - `signature`: `{ "$bytes": }`. + - **Do not** include `repository` — it's only used during CID computation. +6. Append the attestation object to `record["signatures"]` (creating the array if needed). + +Output: the record with one new entry in `signatures`. + +## Verify — procedure + +Given: `record` (with `signatures[]`), `repository` DID, a way to resolve `key` → public key. + +For each entry in `signatures` whose `$type` is **not** `com.atproto.repo.strongRef`: + +1. Let `attestation = signatures[i]`. +2. Rebuild the signing-time metadata: strip `cid` and `signature` from `attestation`. +3. Compute the content CID per `cid-computation.md` using the stripped metadata. +4. Compare to `attestation.cid`. Must match (compare by binary form). If not, **reject**. +5. Resolve `attestation.key` to a public key. +6. Base64-decode `attestation.signature.$bytes`. +7. Verify the signature against the **36-byte binary content CID** using the public key. If ECDSA verification fails, **reject**. + +If all signatures pass, the record is verified. Note: verification does not check the *semantic* meaning of the attestation (who is allowed to attest to what) — that's application policy. + +## Multiple inline attestations on one record + +A record can have multiple inline attestations (e.g., authorship + third-party endorsement). Each signature is computed over the record with `signatures` stripped, so the order in which they were added does not matter. All past signatures remain valid when a new one is appended — the old ones were computed with `signatures` removed, and the new one is too. + +## The `$bytes` wrapper + +Attestation `signature` is an object with a single `$bytes` key whose value is base64. This is AT Protocol's standard way to embed binary in JSON. When the record is DAG-CBOR encoded (e.g., to store in a PDS), the `$bytes` form is replaced with a raw CBOR byte string; the JSON representation is for interchange. + +Consequence: when computing a CID on a signed record (e.g., for the PDS to store), the `$bytes` wrapper encodes to a CBOR byte string, not a map. This is different from computing the content CID for signing, where the attestation hasn't been added yet and `$bytes` doesn't appear. + +## Curve support + +| Curve | Reference crate | Notes | +| -------- | --------------- | -------------------------------------------------------------------------------------- | +| P-256 | ✅ full | 64-byte signatures (r‖s). Low-S normalization implemented. | +| K-256 | ✅ full | 64-byte signatures (r‖s). Low-S normalization implemented. | +| P-384 | ⚠️ partial | Signing/verification via `atproto-identity::sign/validate` works. Low-S normalization is **not** implemented in the reference crate — `normalize_signature` returns `UnsupportedKeyType` for P-384. Avoid P-384 for interop until this is resolved. | + +See `../rust/signatures.md` for exact behavior and `divergence-matrix.md` for cross-language coverage. + +## Common mistakes + +- **Signing the CID string instead of the binary CID.** The string is 59 characters; the binary is 36 bytes. These are not interchangeable. ECDSA signs bytes. +- **Forgetting to normalize to low-S before base64.** The verifier (per the reference) may reject high-S signatures. Even if it doesn't, you've introduced a malleability opportunity. +- **DER-encoding the signature.** ECDSA libraries often return DER by default. This spec requires raw `r‖s` (IEEE P1363) form. Convert if needed. +- **Including `repository` in the stored attestation.** It must not appear in the final object — only in the transient `$sig` during CID computation. +- **Appending to `signatures` before computing the CID.** The CID is over the record *without* `signatures`. Compute first, then append. +- **Using URL-safe base64 for `signature.$bytes`.** Spec uses standard base64 (alphabet + `=` padding). URL-safe (`-_` instead of `+/`) decodes to different bytes for 62/63 characters. + +## See also + +- `cid-computation.md` — step-by-step CID build. +- `signature-normalization.md` — low-S rules. +- `remote-attestation.md` — the signature-less counterpart. +- `test-vectors.md` — current fixtures. +- `../rust/creating.md`, `../typescript/creating.md`, `../go/creating.md` — per-language create flow. +- `../rust/verifying.md`, `../typescript/verifying.md`, `../go/verifying.md` — per-language verify flow. diff --git a/skills/software-development/atproto-attestation/references/shared/remote-attestation.md b/skills/software-development/atproto-attestation/references/shared/remote-attestation.md new file mode 100644 index 0000000..7f3837b --- /dev/null +++ b/skills/software-development/atproto-attestation/references/shared/remote-attestation.md @@ -0,0 +1,132 @@ +# Remote attestations + +Content-addressed, signature-less attestations. The proof of attestation lives in a *separate* record in the attestor's own repo; the subject record only contains a `com.atproto.repo.strongRef` pointing at it. + +Use remote attestations when: + +- The attestor doesn't want to hand their private key to the subject's publisher. +- The attestation should be independently rotatable or revocable (delete the proof record → attestation is unreachable). +- The attestation needs to live under a different access-control boundary than the subject record. + +## The two records + +Remote attestations create **two** records, usually in **two** repos: + +1. **Proof record** — in the attestor's repo. Collection = attestor-chosen NSID (the `$type` of the metadata). Rkey = a TID. Contains the content CID. +2. **Subject record** — in the subject's repo. Same as before, but with a strongRef appended to its `signatures` array. + +### Subject record (after attestation) + +```json +{ + "$type": "app.bsky.feed.post", + "text": "Hello world!", + "signatures": [ + { + "$type": "com.atproto.repo.strongRef", + "uri": "at://did:plc:attestor/com.example.attestation/3kxh2f...", + "cid": "bafyrei" + } + ] +} +``` + +The strongRef type is fixed: `com.atproto.repo.strongRef`. This is how verifiers distinguish remote from inline entries in the `signatures` array. + +### Proof record (in attestor's repo) + +```json +{ + "$type": "com.example.attestation", + "issuer": "did:plc:issuer123", + "purpose": "verification", + "cid": "bafyrei" +} +``` + +Required fields: + +| Field | Type | Notes | +| -------- | ------ | --------------------------------------------------------------------------------------- | +| `$type` | string | Attestor-chosen NSID. Must match the collection the proof record is stored under. | +| `cid` | string | The **content CID** — computed from the subject record + this metadata + subject repo DID. | + +Optional: any other metadata fields the attestor wants. + +## Two CIDs — do not confuse them + +Remote attestations involve two distinct CIDs: + +| CID | Where stored | What it identifies | +| ---------------- | ---------------------------------------- | -------------------------------------------------------- | +| **Content CID** | Inside the proof record, `cid` field | The signed-content payload (record + `$sig` + repository) | +| **Proof CID** | Inside the strongRef, `cid` field | The proof record itself, as stored in the attestor's repo | + +Both must be verified. The proof CID guarantees the strongRef points at the exact record bytes you expect; the content CID guarantees the proof record is bound to the subject record. + +## Create — procedure + +Given: `record`, `metadata` (without `cid`), `subject_repository` DID, `attestor_repository` DID. + +1. Compute the **content CID** per `cid-computation.md`. Input: `record`, `metadata`, `subject_repository`. +2. Build the proof record: start from `metadata`, insert `cid: `. +3. Serialize the proof record to DAG-CBOR and compute its **proof CID** (CIDv1, codec 0x71, SHA-256). This is *not* an attestation-CID call — no `$sig` merge, no `repository` field. It's just the raw DAG-CBOR CID of the proof record bytes. +4. Pick a rkey for the proof record. Convention: a TID (atproto's time-ordered identifier). +5. Build the strongRef: + - `$type`: `com.atproto.repo.strongRef` + - `uri`: `at:////` + - `cid`: the proof CID from step 3 (string form) +6. Append the strongRef to `record["signatures"]`. +7. **Actually publish** the proof record to the attestor's repo via `com.atproto.repo.putRecord`. (The in-memory returned proof record is not an attestation until it's published.) + +Output: the attested subject record (with strongRef) and the proof record (for publishing). + +## Append vs create + +The reference crate distinguishes two flows: + +- `create_remote_attestation` — generates a new proof record in memory, returns both records. You publish the proof record yourself. +- `append_remote_attestation` — you already have a proof record (perhaps created and stored elsewhere); this function takes the proof metadata + the AT-URI it was stored under, verifies the content CID matches, and appends the strongRef. + +The second flow matters when an attestation workflow spans services — e.g., the attestor creates and stores the proof record, then hands the URI back to the publisher to append to their subject record. + +## Verify — procedure + +Given: subject `record` (with `signatures[]`), `subject_repository` DID, a record resolver that can fetch by AT-URI. + +For each entry in `signatures` whose `$type == com.atproto.repo.strongRef`: + +1. Let `strongRef = signatures[i]`. +2. Parse `strongRef.uri` — `at:////`. +3. Fetch the proof record at that URI. (Any `com.atproto.repo.getRecord` XRPC call works.) +4. Compute the DAG-CBOR CID of the fetched proof record. Compare to `strongRef.cid`. If it doesn't match, **reject** — the strongRef points at a different record than expected (tampering or stale cache). +5. Extract `proof.cid` — the claimed content CID. +6. Rebuild the signing-time metadata from the proof record: strip `cid`. +7. Compute the content CID per `cid-computation.md`: `record`, stripped metadata, `subject_repository`. +8. Compare to the claimed content CID. If mismatch, **reject** — the attestation isn't bound to this record in this repo. + +All match → the remote attestation is valid. No cryptographic signature is checked; integrity is content-addressed through the two CID matches. + +## What remote attestations do NOT provide + +- **No cryptographic proof of the issuer.** Anyone can create a proof record claiming any `issuer`. Trust in the attestor comes from knowing whose repo the proof record lives in (via the `uri` field), not from a signature. Applications that need cryptographic provenance should combine a remote attestation with an inline one, or only use inline. +- **No revocation semantics.** Deleting the proof record makes the remote attestation unreachable *to new verifiers*, but anyone who cached the proof record bytes can still verify. The strongRef would return 404 on fresh resolves. Treat deletion as soft revocation. +- **No freshness signal.** The proof record can be published long after the subject record; verifiers can't tell when the attestation was created unless the proof record carries an explicit timestamp field. + +## Common mistakes + +- **Putting the content CID in the strongRef's `cid` field.** That field must be the *proof record*'s CID. The content CID lives inside the proof record. +- **Forgetting to publish the proof record.** The strongRef is useless without the record it points at. Verifiers will get 404. +- **Using a non-TID rkey.** Technically any valid rkey works, but TIDs are the atproto convention for records that don't have a natural key. +- **Storing the proof record in the subject's repo.** That's fine if the subject and attestor are the same DID, but then the whole "remote" part is degenerate — use inline instead. +- **Deleting the subject record but leaving the proof record.** Verifiers can still fetch the proof record, but the strongRef resolves from the subject side, which is gone. The attestation is logically dead but the proof record is dead weight. +- **Computing the proof-record CID with `$sig` merge.** That's the content-CID algorithm. The proof-record CID is just the DAG-CBOR CID of the proof record as published — no `$sig`, no `repository`. + +## See also + +- `cid-computation.md` — the content-CID algorithm. +- `inline-attestation.md` — the signature-bearing counterpart. +- `spec.md` §6 — the top-level overview. +- `test-vectors.md` — fixtures. +- `../rust/creating.md`, `../typescript/creating.md`, `../go/creating.md` — per-language create flow. +- `../rust/verifying.md`, `../typescript/verifying.md`, `../go/verifying.md` — per-language verify flow. diff --git a/skills/software-development/atproto-attestation/references/shared/signature-normalization.md b/skills/software-development/atproto-attestation/references/shared/signature-normalization.md new file mode 100644 index 0000000..fd7ac5d --- /dev/null +++ b/skills/software-development/atproto-attestation/references/shared/signature-normalization.md @@ -0,0 +1,75 @@ +# ECDSA signature normalization (low-S) + +ECDSA signatures are malleable by default: for any valid signature `(r, s)`, the value `(r, n − s)` is also a valid signature for the same message and key (where `n` is the curve order). This creates two distinct byte strings that verify against the same content, which is a problem when the signature bytes are themselves content-addressed or used as an identifier. + +The badge.blue spec requires signatures to be in **low-S form** — the canonical form where `s ≤ n/2`. This file documents what that means and how to implement it per curve. + +## The rule + +After signing, check whether `s > n/2`: + +- If yes: replace with `(r, n − s)`. +- If no: leave unchanged. + +Both `(r, s)` and `(r, n − s)` verify correctly; picking the low-S representative gives every message exactly one valid signature per key. + +## Curve orders + +| Curve | Order `n` (hex, high bits) | `n/2` (for comparison) | +| ------ | ----------------------------------------------------------------- | -------------------------------------------------------------- | +| P-256 | `ffffffff00000000ffffffffffffffffbce6faada7179e84f3b9cac2fc632551` | `7fffffff80000000 7fffffffffffffff de73fd56d38bcf4279dce5617e3192a8` | +| K-256 | `fffffffffffffffffffffffffffffffebaaedce6af48a03bbfd25e8cd0364141` | `7fffffffffffffff ffffffffffffffff 5d576e7357a4501ddfe92f46681b20a0` | +| P-384 | `ffffffffffffffffffffffffffffffffffffffffffffffffc7634d81f4372ddf 581a0db248b0a77aecec196accc52973` | (384-bit, see RFC 5639 or similar) | + +Implementations don't hand-compute this — they use a crypto library that provides a `normalize_s` primitive. The reference Rust crate uses `k256::ecdsa::Signature::normalize_s()` and `p256::ecdsa::Signature::normalize_s()`. + +## Curve coverage in the reference crate + +| Curve | Signing (via `atproto-identity::sign`) | Verification (via `atproto-identity::validate`) | Normalization (`normalize_signature`) | +| ----- | -------------------------------------- | ----------------------------------------------- | ------------------------------------- | +| P-256 | ✅ | ✅ | ✅ | +| K-256 | ✅ | ✅ | ✅ | +| P-384 | ✅ | ✅ | ❌ — `UnsupportedKeyType` error | + +This is a real gap. The reference crate's `normalize_signature` function explicitly returns `AttestationError::UnsupportedKeyType` for anything other than P-256 / K-256 variants. `create_inline_attestation` and `create_signature` both call `normalize_signature` unconditionally, so a P-384 key will fail signing at the normalization step even though raw signing would succeed. + +**Implication**: interop across implementations should stick to P-256 or K-256 until P-384 normalization is implemented everywhere. If you need P-384, plan to contribute the normalization code to the reference crate and keep your own implementation consistent until then. + +See `divergence-matrix.md` §curve-support for a per-language summary. + +## Signature wire format + +After normalization, the signature is emitted as **64 bytes**: `r ‖ s`, each zero-padded to 32 bytes big-endian (for P-256 and K-256). **Not** DER-encoded. + +| Format | Bytes | Used by | +| ------ | ----- | ------- | +| DER (ASN.1) | 70–72 variable | OpenSSL, Go's `ecdsa.Sign`, many defaults | +| IEEE P1363 (`r‖s` fixed) | 64 | badge.blue, Web Crypto API, Rust's `k256`/`p256` (`.to_vec()`) | + +If your language's crypto library returns DER, you must convert to P1363 before normalization and base64 encoding. See per-language signatures guides. + +For P-384 the P1363 form is 96 bytes (48 + 48), but see above re: normalization coverage. + +## Detecting high-S signatures + +During verification, the spec does not mandate that implementations reject high-S signatures — only that created signatures be low-S. In practice: + +- The reference Rust crate's `validate()` (from `atproto-identity`) is permissive: it accepts both low-S and high-S signatures. +- Strict implementations may reject high-S to enforce canonicalization. This is the safer default for new verifiers. + +For interop, always *produce* low-S. For safety, always *accept* both on verify unless your threat model says otherwise. + +## Common mistakes + +- **Skipping normalization.** Some ECDSA libraries produce low-S by default (Web Crypto does; `@noble/curves` has an option); others don't (OpenSSL, older Node). If you don't know, normalize explicitly. The cost is one comparison and one subtraction. +- **DER vs P1363 confusion.** Normalizing a DER signature byte-by-byte produces garbage. Convert to P1363 first. +- **Using the wrong curve's `n`.** Copy-pasting P-256's order into a K-256 normalization (or vice versa) produces invalid signatures that happen to look superficially valid. Use library primitives. +- **Re-normalizing an already-low-S signature.** Idempotent — no harm, but don't assume it's a no-op if your library mutates the signature in place. +- **Assuming P-384 works end-to-end with the reference crate.** It doesn't — normalization is not implemented. You'll hit `UnsupportedKeyType` at create time. + +## See also + +- `spec.md` §8 — curve list and wire-format rule. +- `inline-attestation.md` — where normalized signatures land. +- `divergence-matrix.md` §curve-support and §signature-encoding. +- `../rust/signatures.md`, `../typescript/signatures.md`, `../go/signatures.md` — library-specific ECDSA + normalization. diff --git a/skills/software-development/atproto-attestation/references/shared/spec.md b/skills/software-development/atproto-attestation/references/shared/spec.md new file mode 100644 index 0000000..d1559c9 --- /dev/null +++ b/skills/software-development/atproto-attestation/references/shared/spec.md @@ -0,0 +1,201 @@ +# CID-first attestation — normative spec + +This file is the authoritative, language-neutral specification for AT Protocol attestations per the [badge.blue](https://badge.blue/) reference. Language-specific guides live in `../rust/`, `../typescript/`, `../go/`. + +## 1. What an attestation is + +An attestation is a cryptographically verifiable statement bound to a specific record, by a specific key, in a specific repository. + +There are two kinds: + +- **Inline attestation** — an ECDSA signature embedded in the record's `signatures` array. Self-contained; verifiable with only the record and the issuer's public key. +- **Remote attestation** — a `com.atproto.repo.strongRef` entry in the record's `signatures` array pointing at a separate *proof record* stored in another repository. The proof record carries the CID; no cryptographic signature. + +Both kinds bind to the same *content CID*, and the content CID is bound to a specific repository DID. That last binding is the replay-protection. + +## 2. Terminology + +| Term | Meaning | +| ---- | ------- | +| **Subject record** | The record being attested. Lives in the `repository` DID's repo. | +| **Attestor** | The party creating an inline or remote attestation. Holds the private key (inline) or controls the proof record's repo (remote). | +| **Issuer** | Optional metadata field identifying the attesting entity (typically the attestor's DID). | +| **`signatures` array** | Top-level array on the subject record holding inline or remote attestation entries. | +| **`$sig` metadata** | Transient object merged into the record during CID computation. Not persisted on the final record. | +| **Content CID** | The CIDv1 computed over `(record − signatures, $sig merged with repository)`. The thing that is signed (inline) or referenced (remote). | +| **Proof record** | For remote attestations: a separate record in the attestor's repo containing the content CID and attestation metadata. | +| **Repository binding** | Injection of the subject repo's DID into `$sig.repository` before CID computation. | +| **Low-S normalization** | ECDSA malleability defense: if `s > n/2`, replace with `(r, n − s)`. | +| **strongRef** | `com.atproto.repo.strongRef` — a typed `{uri, cid}` reference. Used for remote attestations. | + +## 3. Lexicon boundaries + +The badge.blue spec is a **framework**, not a lexicon. The attestation metadata's `$type` is user-defined. The spec uses `com.example.inlineSignature`, `com.example.attestation`, etc. as placeholders. Real-world publishers pick their own NSIDs (`blue.badge.approval`, `sh.tangled.attestation`, …). Consumers dispatch on `$type` like any other `$type` union. + +What's **fixed** across implementations: + +- The strongRef type for remote attestations: `com.atproto.repo.strongRef` (standard atproto reference type). +- The field names in attestation metadata: `$type`, `key`, `cid`, `signature`, `repository` (transient). +- The `signatures` field name on the subject record. +- The `$sig` key name used during CID computation (transient). + +## 4. The content CID — the core of the spec + +Everything hinges on the content CID. See `cid-computation.md` for the bit-exact procedure. Summary: + +1. Start with the subject record as a JSON object. +2. Remove its `signatures` field if present. +3. Take the attestation metadata object. +4. Remove `cid`, `signature` from the metadata (these are outputs, not inputs). +5. Insert `repository: ` into the metadata. +6. Insert the modified metadata into the record under key `$sig`. +7. Serialize the result to DAG-CBOR. +8. SHA-256 the bytes. +9. Wrap as CIDv1: version 1, codec `0x71` (dag-cbor), multihash `0x12` (SHA-256), 32-byte digest. 36 bytes binary; `bafyrei…` string form. + +Determinism flows from DAG-CBOR's canonical rules (sorted keys, shortest integer encoding, etc.). Two encoders that both conform to DAG-CBOR produce byte-identical output for the same logical input, so any conforming implementation agrees on the CID. + +## 5. Inline attestation — wire shape + +The subject record after signing: + +```json +{ + "$type": "app.bsky.feed.post", + "text": "Hello world!", + "createdAt": "2024-01-01T00:00:00.000Z", + "signatures": [ + { + "$type": "com.example.inlineSignature", + "key": "did:key:zQ3sh...", + "issuer": "did:plc:issuer123", + "issuedAt": "2024-01-01T00:00:00.000Z", + "cid": "bafyrei...", + "signature": { "$bytes": "" } + } + ] +} +``` + +Required attestation fields (what the PDS/verifier will check): + +- `$type` — any valid NSID chosen by the attestor. +- `key` — a `did:key:` reference to the verification key (or any DID the key resolver understands). +- `cid` — the content CID as a base32 string (`bafyrei…`). +- `signature.$bytes` — base64 of the 64-byte low-S normalized ECDSA signature (IEEE P1363 `r‖s` form, NOT DER). + +Optional attestation fields: + +- `issuer`, `issuedAt`, `purpose`, or any other metadata the attestor wants carried. Custom fields *participate in the CID* — if they change, the signature is invalid. See `cid-computation.md` §4. + +The `repository` field is **never** present on the stored attestation. It's injected only during CID computation. Implementations that leak it into the stored object will still verify, but this violates the spec. + +See `inline-attestation.md` for the full create/verify procedures. + +## 6. Remote attestation — wire shape + +Two records: one in the subject's repo, one in the attestor's repo. + +**Subject record** (after attestation) — strongRef in `signatures`: + +```json +{ + "$type": "app.bsky.feed.post", + "text": "Hello world!", + "signatures": [ + { + "$type": "com.atproto.repo.strongRef", + "uri": "at://did:plc:attestor/com.example.attestation/", + "cid": "bafyrei" + } + ] +} +``` + +**Proof record** (in attestor's repo, collection = attestor-chosen NSID, rkey = TID): + +```json +{ + "$type": "com.example.attestation", + "issuer": "did:plc:issuer123", + "purpose": "verification", + "cid": "bafyrei" +} +``` + +Two CIDs are in play — do not confuse them: + +- **Content CID** — the CID computed from the subject record + proof metadata + subject repo DID. Stored inside the proof record's `cid` field. This is what binds the attestation to the record. +- **Proof record CID** — the DAG-CBOR CID of the proof record itself. Stored in the strongRef's `cid` field. This is what binds the strongRef to the specific proof record revision. + +Verification must check both. See `remote-attestation.md`. + +## 7. Verification + +Every entry in `signatures` must be validated. Entries with `$type = com.atproto.repo.strongRef` are remote; anything else is inline (the metadata type is attestor-chosen). + +### Inline + +1. Extract the attestation object from `signatures[i]`. +2. Rebuild the `$sig` input: strip `cid` and `signature`, insert `repository = `. +3. Recompute the content CID per §4. +4. Compare to the `cid` field in the attestation. Must match byte-for-byte (compare 36-byte binary forms). +5. Resolve the `key` field to a public key (out of scope for this spec — use `atproto-identity-resolution` or a DID-key parser). +6. Base64-decode `signature.$bytes`. +7. Verify the 64-byte ECDSA signature against the **content CID bytes** (36 bytes, the binary CID form) using the public key. + +If any step fails, the attestation is invalid. + +### Remote + +1. Extract the strongRef from `signatures[i]`. +2. Fetch the record at `strongRef.uri` — this is the proof record. Out-of-scope: how you fetch (any XRPC `com.atproto.repo.getRecord` client). +3. Compute the DAG-CBOR CID of the fetched proof record. Compare to `strongRef.cid`. Must match. +4. Extract the proof record's `cid` field — this is the claimed content CID. +5. Rebuild `$sig` from the proof record: remove `cid`, insert `repository = `. +6. Recompute the content CID per §4. +7. Compare to the claimed content CID. Must match. + +Remote verification has no cryptographic signature step — integrity is content-addressed through two CID matches. + +## 8. Signatures and curves + +ECDSA over **CID bytes** (the 36-byte binary form of the content CID, not the string form). + +Supported curves in the reference implementation: + +- P-256 (secp256r1) — low-S normalization implemented. +- K-256 (secp256k1) — low-S normalization implemented. +- P-384 — signing/verification supported, **but low-S normalization is not implemented in the reference Rust crate**. See `signature-normalization.md` and `divergence-matrix.md`. + +Wire format: raw `r‖s` concatenation, 64 bytes for P-256/K-256, 96 bytes for P-384. **Not DER-encoded.** Every implementation must strip ASN.1 if its crypto library returns DER. + +## 9. Replay protection + +The `repository` field in `$sig` makes every content CID repo-specific. Copying a signed record from one repo to another invalidates the signature: + +- A replay-copied inline record's attestation will recompute a different CID (because the verifier uses the *new* repo DID), and ECDSA verification against the wrong CID fails. +- A replay-copied remote record's strongRef still points at the original proof record, but the proof record's `cid` field encodes the *original* repo. A verifier supplying the new repo DID recomputes a different content CID, and the match in step 7 fails. + +Verifiers **must** use the actual repo DID where the record lives (e.g., the DID in the AT-URI they fetched it from). Accepting a caller-supplied repo DID without sanity-checking it defeats replay protection. + +## 10. Known gaps in the spec + +These are not spec ambiguities — they're things the spec delegates to implementations or leaves to the attestor: + +- **Expiration / freshness.** Attestations may include `issuedAt` in custom metadata, but the spec does not define time-based validity. Verifiers that care about freshness must implement it themselves. +- **Revocation.** There is no spec-level revocation. A compromised attestation can only be invalidated by rotating the issuer key (and updating the DID document) or by removing the proof record (remote only). +- **Key rotation.** The `key` field is a static reference; if it becomes a DID verification-method ID and the key rotates, existing attestations remain valid for the old key. This is by design but worth surfacing. +- **Canonical test vectors.** The reference Rust crate has determinism tests but no signed/public test-vector set. Cross-implementation verification currently goes through the Rust crate's CLI tools (`atproto-attestation-sign`, `atproto-attestation-verify`) or the `/verify` page at https://badge.blue/verify. + +## 11. See also + +- `cid-computation.md` — the bit-exact `$sig` merge + DAG-CBOR + SHA-256 procedure. +- `inline-attestation.md` — create/verify procedures for inline. +- `remote-attestation.md` — create/verify procedures for remote (both records, both CIDs). +- `signature-normalization.md` — low-S rules and curve coverage. +- `test-vectors.md` — current fixtures, their provenance, and gaps. +- `divergence-matrix.md` — cross-language differences. +- `../rust/README.md`, `../typescript/README.md`, `../go/README.md` — per-language entry points. +- Upstream spec: +- Reference implementation: `atproto-attestation` crate, source at . diff --git a/skills/software-development/atproto-attestation/references/shared/test-vectors.md b/skills/software-development/atproto-attestation/references/shared/test-vectors.md new file mode 100644 index 0000000..740c4f3 --- /dev/null +++ b/skills/software-development/atproto-attestation/references/shared/test-vectors.md @@ -0,0 +1,81 @@ +# Test vectors + +There is no published canonical test-vector set for badge.blue attestations at this time. This file catalogs the fixtures that do exist, their provenance, and what they prove. + +## What we have + +### Reference crate unit tests + +The Rust `atproto-attestation` crate ships with unit tests that verify: + +- CID determinism: identical `(record, metadata, repository)` inputs produce identical CIDs. + Source: `attestation.rs` — `test_create_attestation_cid_deterministic`. +- Repository binding: different `repository` DIDs produce different CIDs for the same record + metadata. + Source: `attestation.rs` — `test_create_attestation_cid_different_repositories`. +- Signature uniqueness: different messages or different repositories produce different signatures. + Source: `attestation.rs` — `create_signature_different_inputs_produce_different_signatures`, `create_signature_different_repositories_produce_different_signatures`. +- Round-trip: a signature produced by `create_signature` validates against the same computed CID and public key. + Source: `attestation.rs` — `create_signature_returns_valid_bytes`. +- CID format: produced CIDs are CIDv1, codec `0x71`, 32-byte SHA-256 digest, 36 bytes total. + Source: `cid.rs` — `test_create_attestation_cid`, `test_validate_dagcbor_cid`. +- P-256 / K-256 low-S normalization rejects invalid lengths. + Source: `signature.rs` — `reject_invalid_signature_length`. + +These tests do not carry stable reference byte-strings (signatures are produced from random keys each run). They prove *properties*, not specific values. + +### CLI round-trip + +The reference crate's binaries (`atproto-attestation-sign`, `atproto-attestation-verify`) can be used to create a signature with one build and verify it with another. This is the closest thing to an interop vector today: + +``` +# Terminal A: produce +echo '{"$type":"app.example.post","text":"hi"}' \ + | cargo run -p atproto-attestation --features clap,tokio --bin atproto-attestation-sign \ + -- inline - did:key:zQ3sh... '{"$type":"com.example.sig","key":"did:key:zQ3sh..."}' \ + > signed.json + +# Terminal B: consume +cargo run -p atproto-attestation --features clap,tokio --bin atproto-attestation-verify \ + -- ./signed.json did:plc:test123 +``` + +When writing a new implementation, producing a signed record via the CLI and verifying it via your implementation (and vice versa) is the primary interop test. See `divergence-matrix.md` for known interop hazards. + +### badge.blue /verify page + +The tool at verifies any published AT-URI's attestations client-side. It's useful for confirming a published record validates against the spec but does not expose raw intermediate values (content CID, signed bytes). + +## What we need (gaps) + +A complete test vector set should include, for each curve (P-256, K-256): + +- A fixed `(record, metadata, repository)` triple. +- The expected content CID (bytes + string form). +- A fixed key pair (private + public) — this is the tricky part, as baking a private key into a fixture has security-hygiene implications but is standard for test vectors. +- The expected raw signature, the expected normalized signature, the expected base64. +- For remote: the expected proof record bytes, the expected proof-record CID. +- A counter-example showing that changing the repository DID by one character produces a different CID. + +None of this exists upstream yet. A reasonable approach for a new implementation: + +1. Write your own fixture generator using a deterministic private key (e.g., one derived from `SHA256("atproto-attestation-test-vector-1")`). +2. Use it to self-check determinism within your implementation. +3. Run the same generator against the Rust reference crate (via the CLI or a small wrapper) and compare outputs. +4. Contribute the resulting vectors upstream once cross-validated. + +## Golden values (empty — placeholder) + +Reserved for canonical test vectors when they land: + +- [ ] `vector-1-inline-p256.json` — a complete inline attestation with P-256. +- [ ] `vector-2-inline-k256.json` — a complete inline attestation with K-256. +- [ ] `vector-3-remote.json` — a complete remote attestation (subject record + proof record). +- [ ] `vector-4-replay-fail.json` — a cross-repo replay showing the verifier rejects. + +Update this file when they're added. + +## See also + +- `spec.md` §10 — gaps in the spec. +- `divergence-matrix.md` — interop-hazardous implementation differences. +- Reference crate source: `/Users/nick/conductor/workspaces/atproto-crates-studious-guide/delhi-v2/crates/atproto-attestation/src/` (also at ). diff --git a/skills/software-development/atproto-attestation/references/typescript/README.md b/skills/software-development/atproto-attestation/references/typescript/README.md new file mode 100644 index 0000000..1270407 --- /dev/null +++ b/skills/software-development/atproto-attestation/references/typescript/README.md @@ -0,0 +1,198 @@ +# TypeScript — setup & idioms + +There is no canonical TypeScript crate for badge.blue attestations at the time of writing. This file documents the recommended library stack and the shape an implementation takes; see `creating.md` and `verifying.md` for concrete flows. + +## Library stack + +Every primitive badge.blue needs has a well-maintained TS library: + +| Concern | Recommended library | Why | +| ------------------ | -------------------------------- | ----------------------------------------------------------------------------------- | +| DAG-CBOR encoding | `@ipld/dag-cbor` | IPLD-official canonical DAG-CBOR. Works in Node and browsers. | +| CID construction | `multiformats` (+ `multiformats/cid`, `multiformats/hashes/sha2`) | Official IPLD multiformats implementation. | +| SHA-256 | `multiformats/hashes/sha2` (re-exports browser/Node crypto) or `@noble/hashes` | Pure JS fallback via `@noble/hashes` if you need full browser support without deps. | +| ECDSA P-256, K-256 | `@noble/curves` (`p256`, `secp256k1`) | Audited, zero-dep, works everywhere. Supports both curves; has explicit low-S helpers. | +| Base64 | built-in (`btoa`/`Buffer`) or `uint8arrays`/from-`multiformats/bases/base64` | Either works — the spec uses standard base64 with padding. | + +Install: + +```bash +pnpm add @ipld/dag-cbor multiformats @noble/curves @noble/hashes +``` + +Or npm / yarn equivalents. None of these require native extensions. + +### Why not the `@atproto` / `@bsky` packages + +The official `@atproto/*` suite does not yet expose a badge.blue attestation API. `@atproto/common` has some DAG-CBOR and CID helpers, but they're not public API. Using it would couple you to implementation details that can break across minor versions. Stick with the IPLD primitives. + +## AT Protocol record shape in TS + +Records are plain JS objects. There is **no** special `$bytes` type you need to model for signing — you serialize the record as-is, and `$bytes` wrappers appear only in the final output (see below). For DAG-CBOR encoding, byte strings become `Uint8Array` / `CID` in IPLD; JSON's `$bytes` / `$link` wrappers are AT Protocol's JSON encoding of those. + +```ts +interface AttestedRecord { + $type: string; + // … your record fields + signatures?: Array; +} + +interface InlineAttestation { + $type: string; // attestor-chosen NSID, NOT com.atproto.repo.strongRef + key: string; // did:key:z… + cid: string; // content CID, base32 string + signature: { $bytes: string }; // base64 of 64-byte normalized signature + [k: string]: unknown; // other metadata fields participate in the CID +} + +interface RemoteAttestation { + $type: "com.atproto.repo.strongRef"; + uri: string; // at://did/collection/rkey + cid: string; // the proof record's DAG-CBOR CID +} +``` + +## DAG-CBOR encoding + +`@ipld/dag-cbor` handles canonical encoding out of the box: + +```ts +import * as dagCbor from "@ipld/dag-cbor"; + +const bytes = dagCbor.encode(obj); // Uint8Array, canonical form +``` + +Under the hood: + +- Map keys sorted by UTF-8 byte sequence. +- Definite-length strings/arrays/maps. +- Integers minimal-width. +- Floats always 64-bit. +- `CID` instances → CBOR tag 42. + +You pass JS values; the encoder canonicalizes. Do **not** pre-sort keys yourself or JSON-stringify first — that's double-encoding (`../shared/cid-computation.md` §common-mistakes). + +### What about `$link` / `$bytes`? + +AT Protocol's JSON → DAG-CBOR convention: + +- `{ "$link": "bafy…" }` → CBOR tag 42 (CID). +- `{ "$bytes": "…base64…" }` → CBOR byte string. + +If you encode attestation *metadata* before the signature is added (which is what CID computation does), there are no `$link`/`$bytes` entries yet. But if you later encode a *signed* record (say, for the PDS to store), you must convert `{ "$bytes": base64 }` to a raw `Uint8Array` before `dagCbor.encode`. Implementations typically walk the tree and transmogrify; see the `atpmcp.transmogrify_record` MCP for a server-side solution. + +## CID construction + +```ts +import { CID } from "multiformats/cid"; +import { sha256 } from "multiformats/hashes/sha2"; +import * as raw from "multiformats/codecs/raw"; // not used here; for reference + +import * as dagCbor from "@ipld/dag-cbor"; + +const bytes = dagCbor.encode(obj); +const digest = await sha256.digest(bytes); // Multihash-wrapped +const cid = CID.createV1(dagCbor.code, digest); // code 0x71 +// cid.toString() → "bafyrei…" +// cid.bytes → Uint8Array, 36 bytes (binary CID) +``` + +`dagCbor.code` is `0x71`. `sha256.code` is `0x12`. This gives you a CIDv1 with the exact parameters badge.blue requires. + +### What to sign + +For **inline** attestations, sign `cid.bytes` — the 36-byte binary form. **Not** `cid.toString()`, **not** `digest.digest` (the bare 32-byte hash). See `signatures.md`. + +## ECDSA with `@noble/curves` + +```ts +import { p256 } from "@noble/curves/p256"; +import { secp256k1 } from "@noble/curves/secp256k1"; + +// P-256 (NIST prime256v1) +const privP = p256.utils.randomPrivateKey(); // Uint8Array(32) +const pubP = p256.getPublicKey(privP, /*compressed*/ true); +const sigP = p256.sign(cidBytes, privP, { lowS: true }); // Signature object +const rsP = sigP.toCompactRawBytes(); // Uint8Array(64) - r‖s + +// K-256 (secp256k1) +const privK = secp256k1.utils.randomPrivateKey(); +const pubK = secp256k1.getPublicKey(privK, true); +const sigK = secp256k1.sign(cidBytes, privK, { lowS: true }); +const rsK = sigK.toCompactRawBytes(); +``` + +Key details: + +- `sign(msg, priv, { lowS: true })` — `lowS: true` performs normalization for you. If omitted, noble defaults to `lowS: true` on secp256k1 but you should pass it explicitly for clarity. +- `toCompactRawBytes()` gives 64-byte IEEE P1363 (`r‖s`). **Use this**, not `toDERRawBytes()`. +- `@noble/curves` expects the raw message bytes — no double-hashing. Because `sign` hashes with SHA-256 internally by default, you'd double-hash if you pass the CID bytes without `{ prehash: false }`. **Actually**, for attestations we sign the CID bytes themselves (which is a hash, but we treat it as the message). See below. + +### Pre-hash vs not + +`@noble/curves` defaults to hashing the input with SHA-256 before signing: this is what you usually want. But for badge.blue attestations, the **message being signed is the 36-byte CID bytes** — not a hash-preimage. The Rust reference does `ecdsa_sign(cid.to_bytes())` which internally hashes those 36 bytes with SHA-256 and signs the resulting digest. + +So you have two choices, both yielding identical results: + +```ts +// A. Let noble hash — same as Rust reference behavior. +const sig = p256.sign(cidBytes, priv); // hashes SHA-256 → signs digest + +// B. Pre-hash and pass { prehash: true } (noble 1.x) or +// pass the already-hashed 32-byte digest with the low-level API. +// NOT recommended — it's the same bytes, just fiddlier. +``` + +**Use option A.** The Rust crate does: `ecdsa.sign(cid.to_bytes())` → the underlying ECDSA sign routine hashes with SHA-256 internally. Noble matches. + +### Verification + +```ts +const ok = p256.verify(rsP, cidBytes, pubP); // boolean +``` + +Noble's `verify` is permissive by default — accepts both high-S and low-S. To reject high-S explicitly: + +```ts +const ok = p256.verify(rsP, cidBytes, pubP, { lowS: true }); +``` + +## Key serialization to `did:key:` + +`did:key:` encoding = multibase(base58btc) of multicodec-prefixed compressed public key: + +| Curve | Multicodec prefix (varint) | Key bytes | +| ------ | -------------------------- | ----------------------------- | +| P-256 | `0x1200` (`0x80 0x24` varint) | 33-byte compressed SEC1 | +| K-256 | `0xe7` (varint `0xe7 0x01`) | 33-byte compressed SEC1 | + +There's no single canonical TS lib that does this for both curves. Options: + +- `@atproto/identity` — has `Did.Key.formatDidKey` / `parseDidKey`. Works but couples you to atproto. +- `did-resolver` + `key-did-resolver` — general DID library. +- Hand-roll with `multiformats/bases/base58` + `varint` — ~30 LOC per curve. + +For just signing and verifying in a self-contained app, you often don't need `did:key:` — you can pass pubkey bytes around directly and render `did:key:` only at record-serialization time. + +## Async / sync + +All noble operations are sync (they're pure-JS crypto over bignum). `dagCbor.encode` is sync. `sha256.digest` is async (returns a Promise because Web Crypto's SubtleCrypto is async on browsers). + +So attestation creation is a single `await` for the SHA-256 step; everything else is sync. + +## Module / environment support + +- **Node 18+**: everything works. `Buffer.from(…, "base64")` / `.toString("base64")` for base64. Web Crypto available. +- **Deno**: same as Node, import from esm.sh or npm specifiers. +- **Bun**: fine; Bun ships Web Crypto and all the JS libs work. +- **Browsers**: all these libs are pure JS / tree-shakeable. `multiformats/hashes/sha2` uses `SubtleCrypto.digest` where available; `@noble/hashes` is a pure-JS fallback. + +No native modules; no compile step beyond your normal bundler. + +## See also + +- `creating.md` — inline + remote worked examples. +- `verifying.md` — verification loop + resolvers. +- `signatures.md` — ECDSA details and noble gotchas. +- `../shared/spec.md` — normative spec. +- `../shared/divergence-matrix.md` — how TS compares to Rust/Go, especially around missing canonical library. diff --git a/skills/software-development/atproto-attestation/references/typescript/creating.md b/skills/software-development/atproto-attestation/references/typescript/creating.md new file mode 100644 index 0000000..59428b5 --- /dev/null +++ b/skills/software-development/atproto-attestation/references/typescript/creating.md @@ -0,0 +1,360 @@ +# TypeScript — creating attestations + +No off-the-shelf library; you assemble from primitives. These snippets show the full flow end-to-end and are meant to be copied into an application (or the seed of an NPM package). + +## Helper: compute the content CID + +```ts +import * as dagCbor from "@ipld/dag-cbor"; +import { CID } from "multiformats/cid"; +import { sha256 } from "multiformats/hashes/sha2"; + +/** + * Compute the content CID per badge.blue spec: + * record' = record without `signatures` + * meta' = metadata without `cid`/`signature`, with `repository` added + * record'[$sig] = meta' + * CID = CIDv1(dag-cbor, SHA-256(dagCbor.encode(record'))) + */ +export async function computeContentCid( + record: Record, + metadata: Record, + repository: string +): Promise { + if (typeof record !== "object" || record === null || Array.isArray(record)) { + throw new Error("record must be a JSON object"); + } + if (typeof metadata !== "object" || metadata === null || Array.isArray(metadata)) { + throw new Error("metadata must be a JSON object"); + } + + // Strip `signatures` from record + const { signatures: _s, ...strippedRecord } = record as Record & { + signatures?: unknown; + }; + + // Prepare metadata: drop cid/signature, add repository + const { cid: _c, signature: _sig, ...strippedMeta } = metadata as Record & { + cid?: unknown; + signature?: unknown; + }; + const sigMetadata = { ...strippedMeta, repository }; + + // Merge: record[$sig] = meta + const merged = { ...strippedRecord, $sig: sigMetadata }; + + // DAG-CBOR encode → SHA-256 → CIDv1(0x71) + const bytes = dagCbor.encode(merged); + const digest = await sha256.digest(bytes); + return CID.createV1(dagCbor.code, digest); +} +``` + +This is the heart of every flow below. Test it with `shared/test-vectors.md` once canonical vectors land. + +## Helper: standard base64 + +```ts +// Node +function toBase64(bytes: Uint8Array): string { + return Buffer.from(bytes).toString("base64"); +} +function fromBase64(str: string): Uint8Array { + return new Uint8Array(Buffer.from(str, "base64")); +} + +// Browser / Deno +function toBase64Browser(bytes: Uint8Array): string { + let s = ""; + for (const b of bytes) s += String.fromCharCode(b); + return btoa(s); // standard alphabet with padding +} +function fromBase64Browser(str: string): Uint8Array { + const raw = atob(str); + return Uint8Array.from(raw, (c) => c.charCodeAt(0)); +} +``` + +## Inline attestation — create + +```ts +import { p256 } from "@noble/curves/p256"; +import { secp256k1 } from "@noble/curves/secp256k1"; + +type Curve = "p256" | "k256"; + +export interface InlineCreateArgs { + record: Record; + metadata: Record; // must include $type and key (did:key:) + repository: string; // did:plc:… of subject repo + privateKey: Uint8Array; // 32-byte scalar + curve: Curve; +} + +export async function createInlineAttestation(args: InlineCreateArgs): Promise> { + const cid = await computeContentCid(args.record, args.metadata, args.repository); + + const cidBytes = cid.bytes; // 36 bytes — THIS is what we sign + + const curve = args.curve === "p256" ? p256 : secp256k1; + const sig = curve.sign(cidBytes, args.privateKey, { lowS: true }); + const rs = sig.toCompactRawBytes(); + // rs is 64 bytes (r‖s), low-S normalized + + const attestation = { + ...args.metadata, + cid: cid.toString(), + signature: { $bytes: toBase64(rs) }, + }; + + // Strip transient `repository` if present in metadata (paranoid — it shouldn't be): + delete (attestation as { repository?: unknown }).repository; + + const existing = Array.isArray(args.record.signatures) ? args.record.signatures : []; + return { + ...args.record, + signatures: [...existing, attestation], + }; +} +``` + +Usage: + +```ts +const signed = await createInlineAttestation({ + record: { $type: "app.bsky.feed.post", text: "hi", createdAt: new Date().toISOString() }, + metadata: { + $type: "com.example.inlineSignature", + key: "did:key:zDnaeR...", + issuer: "did:plc:issuer123", + issuedAt: new Date().toISOString(), + purpose: "authorship", + }, + repository: "did:plc:publisher456", + privateKey, + curve: "p256", +}); +``` + +`signed` is a plain JS object; send it to your PDS via `com.atproto.repo.putRecord`. + +### Gotcha: `signatures` field in metadata + +If your `metadata` accidentally contains a `signatures` field, it stays there — the stripping rule applies to **record**, not metadata. Don't put `signatures` in metadata; it's a record-level field. + +### Gotcha: ordering + +You do **not** need to sort keys on `metadata` or `record` before passing them — `dagCbor.encode` canonicalizes. But be aware that if you round-trip through `JSON.stringify`/`JSON.parse` somewhere, numeric precision can drift (`1.0` vs `1`). DAG-CBOR treats these differently (`1` is an integer, `1.0` is a float). Keep your data types stable. + +## Remote attestation — create + +```ts +import { tidGenerate } from "./tid"; // see below + +export interface RemoteCreateArgs { + record: Record; + metadata: Record; // must include $type + subjectRepository: string; + attestorRepository: string; +} + +export interface RemoteCreateResult { + attestedRecord: Record; // record with strongRef appended — publish to subjectRepository + proofRecord: Record; // proof record — publish to attestorRepository + proofUri: string; // at:// URI the caller must publish the proof to +} + +export async function createRemoteAttestation(args: RemoteCreateArgs): Promise { + const metaType = args.metadata.$type; + if (typeof metaType !== "string") throw new Error("metadata must have $type"); + + // 1. Content CID (binds record + metadata + subject repo) + const contentCid = await computeContentCid(args.record, args.metadata, args.subjectRepository); + + // 2. Build proof record (metadata + cid field) + const proofRecord: Record = { + ...args.metadata, + cid: contentCid.toString(), + }; + + // 3. Compute proof record's DAG-CBOR CID (NO $sig merge — plain DAG-CBOR CID) + const proofBytes = dagCbor.encode(proofRecord); + const proofDigest = await sha256.digest(proofBytes); + const proofCid = CID.createV1(dagCbor.code, proofDigest); + + // 4. Pick a TID for the proof record's rkey + const rkey = tidGenerate(); // e.g., "3kxh2f4jabc2s" + + // 5. Build strongRef + const proofUri = `at://${args.attestorRepository}/${metaType}/${rkey}`; + const strongRef = { + $type: "com.atproto.repo.strongRef", + uri: proofUri, + cid: proofCid.toString(), + }; + + // 6. Append to record.signatures + const existing = Array.isArray(args.record.signatures) ? args.record.signatures : []; + const attestedRecord = { + ...args.record, + signatures: [...existing, strongRef], + }; + + return { attestedRecord, proofRecord, proofUri }; +} +``` + +### TID generation + +TIDs are atproto's 13-character base32-sortable time identifiers. Minimal implementation: + +```ts +const TID_ALPHABET = "234567abcdefghijklmnopqrstuvwxyz"; + +let lastTime = 0n; +let lastClock = 0n; + +export function tidGenerate(): string { + let time = BigInt(Date.now()) * 1000n; + if (time <= lastTime) time = lastTime + 1n; + lastTime = time; + // Random 10-bit clock id + const clock = BigInt(Math.floor(Math.random() * 1024)); + // top bit 0, 53 bits time microseconds, 10 bits clock + const combined = (time << 10n) | clock; + + let s = ""; + let v = combined; + for (let i = 0; i < 13; i++) { + s = TID_ALPHABET[Number(v & 31n)] + s; + v >>= 5n; + } + return s; +} +``` + +Or use `@atproto/common` / `@atproto/syntax`'s `TID` class if you're already pulling in atproto packages. + +### Publishing sequence + +After `createRemoteAttestation` returns: + +```ts +const { attestedRecord, proofRecord, proofUri } = await createRemoteAttestation(...); + +const [, , , , collection, rkey] = proofUri.split("/"); // or a real parser +// 1. Publish the proof record first. +await xrpc.call("com.atproto.repo.putRecord", { + repo: attestorRepository, + collection, + rkey, + record: proofRecord, +}); + +// 2. Then publish the attested record. +await xrpc.call("com.atproto.repo.putRecord", { + repo: subjectRepository, + collection: subjectRecordCollection, + rkey: subjectRecordRkey, + record: attestedRecord, +}); +``` + +Publishing in the other order leaves a dangling strongRef if step 2 succeeds before step 1. Prefer proof-first. + +## Append flows + +### Append an existing inline attestation + +```ts +export interface AppendInlineArgs { + record: Record; + attestation: Record; // untrusted — will be validated + repository: string; + resolveKey: (keyRef: string) => Promise<{ curve: Curve; publicKey: Uint8Array }>; +} + +export async function appendInlineAttestation(args: AppendInlineArgs): Promise> { + // Strip cid / signature from attestation to rebuild metadata + const { cid: claimedCid, signature, ...meta } = args.attestation as Record & { + cid?: unknown; + signature?: { $bytes?: string } | unknown; + }; + + if (typeof claimedCid !== "string") throw new Error("attestation missing cid"); + + const computedCid = await computeContentCid(args.record, meta as Record, args.repository); + if (computedCid.toString() !== claimedCid) { + throw new Error(`attestation cid mismatch: expected ${claimedCid} computed ${computedCid}`); + } + + const sigBytesB64 = (signature as { $bytes?: string })?.$bytes; + if (typeof sigBytesB64 !== "string") throw new Error("signature.$bytes missing"); + const sigBytes = fromBase64(sigBytesB64); + + const keyRef = args.attestation.key; + if (typeof keyRef !== "string") throw new Error("attestation.key missing"); + const { curve, publicKey } = await args.resolveKey(keyRef); + + const ok = (curve === "p256" ? p256 : secp256k1).verify(sigBytes, computedCid.bytes, publicKey); + if (!ok) throw new Error("signature verification failed"); + + const existing = Array.isArray(args.record.signatures) ? args.record.signatures : []; + return { ...args.record, signatures: [...existing, args.attestation] }; +} +``` + +### Append a remote strongRef to an already-stored proof + +```ts +export interface AppendRemoteArgs { + record: Record; + proofMetadata: Record; // has $type, cid, and any attestation fields + repository: string; + attestationUri: string; +} + +export async function appendRemoteAttestation(args: AppendRemoteArgs): Promise> { + const claimedCid = args.proofMetadata.cid; + if (typeof claimedCid !== "string") throw new Error("proofMetadata.cid missing"); + + // Strip cid for CID computation + const { cid: _, ...stripped } = args.proofMetadata; + const computed = await computeContentCid(args.record, stripped, args.repository); + if (computed.toString() !== claimedCid) { + throw new Error("proof metadata cid does not match computed content cid"); + } + + // Proof record's DAG-CBOR CID (NB: proofMetadata here should match what's published, + // including the `cid` field — we compute the stored record's CID.) + const proofBytes = dagCbor.encode(args.proofMetadata); + const proofDigest = await sha256.digest(proofBytes); + const proofCid = CID.createV1(dagCbor.code, proofDigest); + + const strongRef = { + $type: "com.atproto.repo.strongRef", + uri: args.attestationUri, + cid: proofCid.toString(), + }; + + const existing = Array.isArray(args.record.signatures) ? args.record.signatures : []; + return { ...args.record, signatures: [...existing, strongRef] }; +} +``` + +## Common mistakes + +- **Double-hashing the CID.** `sign(cidBytes, priv)` hashes the 36 bytes with SHA-256 internally; don't pre-hash. (The *message* in ECDSA parlance is `cidBytes`; the digest is an internal implementation detail.) +- **`toDERRawBytes()` instead of `toCompactRawBytes()`.** 70–72 bytes vs 64 bytes — the first is DER, spec requires P1363. +- **Forgetting `{ lowS: true }`.** P-256 defaults may not low-S normalize; pass it explicitly. K-256 low-S is default-on in noble but be explicit. +- **Encoding a record that still has `$bytes` JSON wrappers to DAG-CBOR.** For CID computation this won't happen (we strip metadata `signature` first), but if you're computing CIDs on signed records post-hoc, transmogrify `{$bytes}` → `Uint8Array` first. +- **URL-safe base64.** Use `btoa` / `Buffer.toString("base64")` — both are standard alphabet. +- **Publishing attested record before proof record.** Leaves a dangling strongRef on network hiccup. +- **Using `canonicalize` or `JSON.stringify` as a CID pre-step.** You feed the JS *object* to `dagCbor.encode`, not a string. Stringify never appears. + +## See also + +- `verifying.md` — verification loop. +- `signatures.md` — ECDSA details. +- `../shared/inline-attestation.md`, `../shared/remote-attestation.md` — language-neutral specs. +- `../rust/creating.md`, `../go/creating.md` — sibling flows for interop. diff --git a/skills/software-development/atproto-attestation/references/typescript/signatures.md b/skills/software-development/atproto-attestation/references/typescript/signatures.md new file mode 100644 index 0000000..81e2f5c --- /dev/null +++ b/skills/software-development/atproto-attestation/references/typescript/signatures.md @@ -0,0 +1,189 @@ +# TypeScript — ECDSA signing & normalization + +Recommended library: `@noble/curves`. It's audited, zero-dependency, and covers all three curves the spec mentions. The spec requires IEEE P1363 `r‖s` output and low-S normalization — both are one option away in noble. + +## Curve imports + +```ts +import { p256 } from "@noble/curves/p256"; // NIST P-256 / secp256r1 +import { secp256k1 } from "@noble/curves/secp256k1"; // K-256 / Bitcoin curve +import { p384 } from "@noble/curves/p384"; // spec mentions it; avoid for interop +``` + +All three expose the same interface: `sign`, `verify`, `getPublicKey`, `utils.randomPrivateKey`, and a `Signature` class with `toCompactRawBytes()` / `toDERRawBytes()`. + +## Signing + +```ts +import { p256 } from "@noble/curves/p256"; + +const privateKey = p256.utils.randomPrivateKey(); // Uint8Array(32) +const cidBytes = contentCid.bytes; // Uint8Array(36) + +const sig = p256.sign(cidBytes, privateKey, { lowS: true }); +// sig is a `SignatureType` (r/s big ints) + +const rs = sig.toCompactRawBytes(); +// rs: Uint8Array(64) — 32-byte r ‖ 32-byte s, already low-S +``` + +### What `sign` does internally + +1. Hashes `cidBytes` with SHA-256 → 32-byte digest. +2. Generates a deterministic `k` per RFC 6979 (default for noble; no external RNG). +3. Computes `(r, s)`. +4. If `{ lowS: true }` (or default-on for secp256k1), replaces `s` with `n - s` when `s > n/2`. +5. Returns a `Signature` object. + +Step 1 matters: the **message you pass is the 36-byte CID**, not a 32-byte digest. Noble hashes it internally. The Rust reference does the same thing via its underlying ECDSA library. Cross-language compat: ✓. + +### `{ lowS: true }` — when to set it + +| Curve | Default `lowS` | What to pass | +| ----------- | -------------- | ------------ | +| secp256k1 | `true` | `{ lowS: true }` (explicit is good) | +| p256 | `false` | **Must** pass `{ lowS: true }` | +| p384 | `false` | **Must** pass `{ lowS: true }` — but see gap | + +For attestations, always pass `{ lowS: true }` explicitly. Cost is nil; silent high-S sigs are hard to debug later. + +### Normalizing an already-produced signature + +If you get a raw 64-byte signature from somewhere else and want to low-S-normalize it: + +```ts +const parsed = p256.Signature.fromCompact(rs); +const normalized = parsed.normalizeS(); // returns a (possibly-new) Signature +const rsLowS = normalized.toCompactRawBytes(); +``` + +`normalizeS()` is idempotent. If it was already low-S it returns the same signature. + +## Verifying + +```ts +const ok = p256.verify(rs, cidBytes, publicKey); +``` + +`verify`: + +1. Parses `rs` as two big ints (from 64-byte compact form). +2. Hashes `cidBytes` with SHA-256. +3. Runs ECDSA verify. +4. Returns `boolean`. + +Optional `{ lowS: true }` makes verify *reject* high-S: + +```ts +const okStrict = p256.verify(rs, cidBytes, publicKey, { lowS: true }); +``` + +Match to your threat model. The badge.blue spec doesn't require strict low-S on verify. + +### Public key formats + +`verify` accepts: +- 33-byte compressed SEC1 (`0x02`/`0x03` prefix + 32 bytes X). +- 65-byte uncompressed SEC1 (`0x04` prefix + 32 bytes X + 32 bytes Y). + +`did:key:z…` decodes to the 33-byte compressed form. Pass through directly. + +## DER ↔ P1363 conversion (when you need it) + +Most TS paths don't need this — you get P1363 directly from noble. But if you're consuming signatures from, say, a Java or Go ECDSA stdlib signer: + +```ts +// DER → P1363 +const sig = p256.Signature.fromDER(derBytes); +const rs = sig.toCompactRawBytes(); + +// P1363 → DER +const sig2 = p256.Signature.fromCompact(rs); +const der = sig2.toDERRawBytes(); +``` + +DER is variable length (~70–72 bytes); P1363 is always `r‖s` at curve width (64 for P-256/K-256, 96 for P-384). + +## Key generation and `did:key:` + +```ts +const priv = p256.utils.randomPrivateKey(); // 32 bytes, secure +const pub = p256.getPublicKey(priv, true); // 33-byte compressed + +// Compose a did:key: string +import { base58btc } from "multiformats/bases/base58"; +import * as varint from "uint8-varint"; + +// Multicodec prefix for p256-pub is 0x1200 (2 bytes varint: 0x80 0x24) +const prefix = varint.encodingLength(0x1200); +const prefixed = new Uint8Array(prefix + pub.length); +varint.encodeTo(0x1200, prefixed, 0); +prefixed.set(pub, prefix); +const didKey = "did:key:" + base58btc.encode(prefixed); // includes leading 'z' +``` + +For K-256 replace `0x1200` with `0xe7`. For P-384 use `0x1201` — though normalization is still broken upstream, see below. + +## The P-384 situation + +- `@noble/curves/p384` works fine for signing and verifying. +- The reference Rust crate does **not** implement low-S for P-384 — `normalize_signature` returns `UnsupportedKeyType`. +- So if you produce a P-384 attestation in TypeScript and try to verify it with the Rust crate, *verification itself* works (permissive), but *re-signing* or *append*-style flows that renormalize will fail. +- For interop: avoid P-384 for attestations until the Rust crate gains support. + +TS-local workflows (TS signer + TS verifier) work fine, but you're off the spec's interop guarantees. + +## Deterministic (test-vector) signing + +Both noble and the Rust crate use RFC 6979 deterministic nonces, so a given `(priv, msg)` pair produces the same signature bytes every time. This is useful for test vectors: + +```ts +const sig1 = p256.sign(cid.bytes, priv, { lowS: true }).toCompactRawBytes(); +const sig2 = p256.sign(cid.bytes, priv, { lowS: true }).toCompactRawBytes(); +// sig1 byte-equal sig2 +``` + +If you generate vectors with a fixed `priv` from a deterministic source (e.g., `sha256("atproto-attestation-test-vector-1")`), TS and Rust produce identical bytes. + +## Verifying the Rust reference crate's output in TS + +You can drive the Rust crate's `atproto-attestation-sign` binary, then verify in TS: + +```bash +echo '{"$type":"test","x":1}' \ + | cargo run -p atproto-attestation --features clap,tokio --bin atproto-attestation-sign \ + -- inline - did:plc:test did:key:zQ3sh... '{"$type":"com.example.sig","key":"did:key:zQ3sh..."}' \ + > signed.json +``` + +In TS: + +```ts +import { verifyRecord } from "./verify"; +const signed = JSON.parse(fs.readFileSync("signed.json", "utf8")); +await verifyRecord({ + record: signed, + repository: "did:plc:test", + keyResolver: localKeyResolver, + recordResolver: { resolveRecord: () => { throw new Error("no remotes"); } }, +}); +``` + +Full trip works if the TS implementation tracks `../shared/cid-computation.md` exactly. + +## Common mistakes + +- **Forgetting `{ lowS: true }` on P-256.** Produces high-S sigs half the time, verifies against permissive verifiers, fails against strict ones. +- **Using `toDERRawBytes`.** The spec mandates 64-byte `r‖s`. DER is 70–72. +- **Using browser `SubtleCrypto.sign` for attestations.** SubtleCrypto's ECDSA output is P1363 (good!) but you can't control low-S without a post-process step. Noble is simpler. +- **Using WebCrypto for K-256.** WebCrypto doesn't support secp256k1 in most browsers. Noble does. +- **Hand-rolling the multicodec prefix.** Easy to get the varint wrong. Use `uint8-varint` or an existing DID library. +- **Mixing up `p256.sign(msg, priv)` and `p256.sign(priv, msg)`.** Noble is `(msg, priv)`. Be careful. +- **Assuming P-384 round-trips.** See above. + +## See also + +- `creating.md`, `verifying.md` — the flows these primitives power. +- `../shared/signature-normalization.md` — curve orders, cross-language coverage. +- `../rust/signatures.md`, `../go/signatures.md` — peer comparisons. +- `@noble/curves` docs: . diff --git a/skills/software-development/atproto-attestation/references/typescript/verifying.md b/skills/software-development/atproto-attestation/references/typescript/verifying.md new file mode 100644 index 0000000..607f12e --- /dev/null +++ b/skills/software-development/atproto-attestation/references/typescript/verifying.md @@ -0,0 +1,287 @@ +# TypeScript — verifying attestations + +Verification is a loop over `record.signatures`. Each entry is either an inline attestation (signature check) or a remote strongRef (fetch + CID check). This file gives a reference implementation and discusses the resolver plumbing. + +## The full verifier + +```ts +import * as dagCbor from "@ipld/dag-cbor"; +import { CID } from "multiformats/cid"; +import { sha256 } from "multiformats/hashes/sha2"; +import { p256 } from "@noble/curves/p256"; +import { secp256k1 } from "@noble/curves/secp256k1"; + +const STRONG_REF = "com.atproto.repo.strongRef"; + +export type Curve = "p256" | "k256"; + +export interface KeyResolver { + resolveKey(keyRef: string): Promise<{ curve: Curve; publicKey: Uint8Array }>; +} + +export interface RecordResolver { + resolveRecord(atUri: string): Promise>; +} + +export interface VerifyArgs { + record: Record; + repository: string; + keyResolver: KeyResolver; + recordResolver: RecordResolver; + strictLowS?: boolean; // default false — match reference behavior + verifyProofCid?: boolean; // default true — check proof record CID on remote +} + +export async function verifyRecord(args: VerifyArgs): Promise { + const signatures = Array.isArray(args.record.signatures) ? args.record.signatures : []; + if (signatures.length === 0) return; // no signatures to verify + + for (const entry of signatures) { + if (!entry || typeof entry !== "object") throw new Error("signature entry must be an object"); + const $type = (entry as { $type?: unknown }).$type; + if (typeof $type !== "string") throw new Error("signature entry missing $type"); + + if ($type === STRONG_REF) { + await verifyRemoteEntry(entry as RemoteEntry, args); + } else { + await verifyInlineEntry(entry as InlineEntry, args); + } + } +} + +interface InlineEntry { + $type: string; + key: string; + cid: string; + signature: { $bytes: string }; + [k: string]: unknown; +} + +interface RemoteEntry { + $type: "com.atproto.repo.strongRef"; + uri: string; + cid: string; +} + +async function verifyInlineEntry(entry: InlineEntry, args: VerifyArgs): Promise { + // Rebuild signing-time metadata: drop cid + signature + const { cid: claimedCid, signature, ...meta } = entry; + if (typeof claimedCid !== "string") throw new Error("inline: cid missing"); + + const computed = await computeContentCid(args.record, meta as Record, args.repository); + if (computed.toString() !== claimedCid) { + throw new Error(`inline: cid mismatch (claimed=${claimedCid} computed=${computed})`); + } + + const sigBytesB64 = signature?.$bytes; + if (typeof sigBytesB64 !== "string") throw new Error("inline: signature.$bytes missing"); + const sigBytes = fromBase64(sigBytesB64); + if (sigBytes.length !== 64) throw new Error(`inline: signature must be 64 bytes (got ${sigBytes.length})`); + + const { curve, publicKey } = await args.keyResolver.resolveKey(entry.key); + const lib = curve === "p256" ? p256 : secp256k1; + const ok = lib.verify(sigBytes, computed.bytes, publicKey, { + lowS: args.strictLowS ?? false, + }); + if (!ok) throw new Error("inline: signature verification failed"); +} + +async function verifyRemoteEntry(entry: RemoteEntry, args: VerifyArgs): Promise { + // Fetch the proof record + const proof = await args.recordResolver.resolveRecord(entry.uri); + + // Option: verify the proof record's DAG-CBOR CID matches entry.cid + if (args.verifyProofCid !== false) { + const proofBytes = dagCbor.encode(proof); + const proofDigest = await sha256.digest(proofBytes); + const proofCid = CID.createV1(dagCbor.code, proofDigest); + if (proofCid.toString() !== entry.cid) { + throw new Error( + `remote: proof record CID mismatch (strongRef=${entry.cid} fetched=${proofCid})` + ); + } + } + + // Extract claimed content CID from proof record + const claimedContentCid = (proof as { cid?: unknown }).cid; + if (typeof claimedContentCid !== "string") throw new Error("remote: proof record missing cid"); + + // Rebuild signing-time metadata: proof record minus its `cid` field + const { cid: _, ...metaForCid } = proof as Record & { cid?: unknown }; + + const computed = await computeContentCid( + args.record, + metaForCid as Record, + args.repository + ); + if (computed.toString() !== claimedContentCid) { + throw new Error( + `remote: content CID mismatch (claimed=${claimedContentCid} computed=${computed})` + ); + } +} +``` + +## What the reference Rust crate does vs what this adds + +The Rust `verify_record` does *not* verify the proof record's CID against the strongRef — it only checks the **content CID** inside the proof record against what it computes. An attacker who controls the resolver could swap the proof record bytes for something with the same content CID inside but different outer bytes, and verification would still pass. + +This TS implementation defaults `verifyProofCid` to `true` — it's a one-extra-hash check that closes that hole. Set to `false` only if you want strict byte-compat with the Rust crate's current verify semantics. + +## `KeyResolver` implementations + +### Plain `did:key:` parser + +```ts +import { base58btc } from "multiformats/bases/base58"; +import * as varint from "uint8-varint"; + +export const localKeyResolver: KeyResolver = { + async resolveKey(keyRef: string) { + if (!keyRef.startsWith("did:key:")) throw new Error(`not a did:key: ${keyRef}`); + const multibase = keyRef.slice("did:key:".length); + const bytes = base58btc.decode(multibase); // first char 'z' is the multibase prefix + + const [code, codeLen] = varint.decode(bytes); + const keyBytes = bytes.subarray(codeLen); + + if (code === 0x1200) return { curve: "p256" as const, publicKey: keyBytes }; + if (code === 0xe7) return { curve: "k256" as const, publicKey: keyBytes }; + throw new Error(`unsupported did:key multicodec: 0x${code.toString(16)}`); + }, +}; +``` + +`keyBytes` is 33 bytes compressed SEC1 for both curves. `@noble/curves`' `verify` accepts compressed or uncompressed keys transparently. + +### DID document lookup + +If your attestations reference `did:plc:…#signingKey` or `did:web:example.com#atproto`: + +```ts +export class DidResolver implements KeyResolver { + constructor(private httpClient: typeof fetch) {} + + async resolveKey(keyRef: string): Promise<{ curve: Curve; publicKey: Uint8Array }> { + const hashIdx = keyRef.indexOf("#"); + if (hashIdx < 0) return localKeyResolver.resolveKey(keyRef); // pure did:key + + const did = keyRef.slice(0, hashIdx); + const fragment = keyRef.slice(hashIdx + 1); + const doc = await this.fetchDidDocument(did); + + const vm = doc.verificationMethod?.find((v: { id: string }) => + v.id === keyRef || v.id === `#${fragment}` + ); + if (!vm) throw new Error(`no verification method ${fragment} on ${did}`); + + if (vm.publicKeyMultibase) { + return localKeyResolver.resolveKey(`did:key:${vm.publicKeyMultibase}`); + } + throw new Error("only publicKeyMultibase supported in this resolver"); + } + + private async fetchDidDocument(did: string): Promise<{ verificationMethod?: Array<{ id: string; publicKeyMultibase?: string }> }> { + // did:plc: -> https://plc.directory/ + // did:web:: -> https:///.well-known/did.json + // Full resolution belongs in the atproto-identity-resolution skill. + throw new Error("implement me — see atproto-identity-resolution"); + } +} +``` + +Cross-reference the `atproto-identity-resolution` skill for the full DID resolution algorithm — this skill stops at "given a DID doc fragment, extract the key". + +## `RecordResolver` implementations + +### Naive XRPC client + +```ts +export class XrpcRecordResolver implements RecordResolver { + constructor(private pdsFor: (did: string) => Promise) {} + + async resolveRecord(atUri: string): Promise> { + const m = atUri.match(/^at:\/\/([^/]+)\/([^/]+)\/(.+)$/); + if (!m) throw new Error(`invalid AT-URI: ${atUri}`); + const [, did, collection, rkey] = m; + + const pds = await this.pdsFor(did); + const url = new URL(`${pds}/xrpc/com.atproto.repo.getRecord`); + url.searchParams.set("repo", did); + url.searchParams.set("collection", collection); + url.searchParams.set("rkey", rkey); + + const res = await fetch(url); + if (!res.ok) throw new Error(`getRecord ${atUri}: ${res.status}`); + const body = await res.json(); + return body.value as Record; + } +} +``` + +Note: the PDS-returned record has already been DAG-CBOR-decoded and re-serialized as JSON with `$bytes` / `$link` wrappers for any binary fields. For proof records this doesn't matter (they typically contain only strings); if you ever attach blobs to proof records, you need to transmogrify before encoding. + +### Caching + +`RecordResolver.resolveRecord` is called once per strongRef per `verifyRecord` call. If you verify many records that share attestors, cache by `atUri`: + +```ts +export class CachingResolver implements RecordResolver { + private cache = new Map>(); + constructor(private inner: RecordResolver) {} + async resolveRecord(uri: string) { + let v = this.cache.get(uri); + if (!v) { + v = await this.inner.resolveRecord(uri); + this.cache.set(uri, v); + } + return v; + } +} +``` + +## Strict vs permissive verification + +Reference behavior (matches Rust): + +- Accept both low-S and high-S signatures. +- Do not check `issuedAt` freshness. +- Do not check `issuer` authorization. +- Trust `recordResolver` to return the canonical proof record (does not verify its outer CID matches the strongRef's `cid`). + +Stricter optional modes this impl supports: + +```ts +await verifyRecord({ + record, repository, keyResolver, recordResolver, + strictLowS: true, // reject high-S inline signatures + verifyProofCid: true // verify proof record outer CID (default) +}); +``` + +## Partial verification + +`verifyRecord` is all-or-nothing. If you need to verify just one entry, slice the array: + +```ts +const only = (record.signatures as Array>) + .filter((s) => s.key === targetKey); +await verifyRecord({ ...args, record: { ...record, signatures: only } }); +``` + +The CID is computed with `signatures` stripped, so removing other entries doesn't affect verification. + +## Common mistakes + +- **Wrong `repository`.** All signatures are bound to the repo the record lives in. If you verify a record fetched from `did:plc:A` with `repository: "did:plc:B"`, every inline signature fails — as designed (replay protection). +- **Assuming `recordResolver` returns the exact byte-for-byte proof record.** It returns the PDS's JSON shape. If your `verifyProofCid: true` check fails, the most common cause is that the PDS re-ordered fields or your JSON-parse → DAG-CBOR-encode path canonicalizes differently. Feed the result through DAG-CBOR and it should re-sort. +- **Forgetting to handle the `no signatures` case.** A record with an empty / missing `signatures` array passes `verifyRecord` trivially (the Rust crate does the same). If your policy needs "at least one attestation", check length explicitly before calling. +- **Handling `RecordResolver` errors as fatal app errors.** A deleted proof record is expected; treat it as "attestation not provable" rather than crashing. +- **Using `p256.verify` without checking signature length first.** Noble throws on malformed signatures; treat 64-byte exact as a precondition. + +## See also + +- `creating.md` — the inverse flow. +- `signatures.md` — `verify` details. +- `../shared/inline-attestation.md` §Verify, `../shared/remote-attestation.md` §Verify. +- `../rust/verifying.md`, `../go/verifying.md` — sibling flows. diff --git a/skills/software-development/atproto-cid/SKILL.md b/skills/software-development/atproto-cid/SKILL.md new file mode 100644 index 0000000..9b2575e --- /dev/null +++ b/skills/software-development/atproto-cid/SKILL.md @@ -0,0 +1,120 @@ +--- +name: atproto-cid +description: "Use when working with ATProto/DASL CIDs: parse, construct, validate, debug." +version: 1.0.0 +author: Hermes Agent (ported from ngerakines/atproto-skills, MIT) +license: MIT +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [ATProto, CID, DASL, BDASL, multiformats, DAG-CBOR] +--- + +# AT Protocol / DASL CIDs + +Content Identifiers (CIDs) are the content-addressed hashes that bind every +record, blob, and commit in AT Protocol to its exact bytes. This is the +CID-specialist skill: parsing, construction, validation, verification, and +debugging (CID mismatch, tag 42, multibase prefixes, BLAKE3/BDASL). + +Ported from `ngerakines/atproto-skills` (MIT). The language-neutral spec and +TypeScript guides are kept verbatim under `references/`; Rust/Go guides were +omitted (not in our stack) — fetch them from the upstream repo if ever needed. + +## Defaults + +A DASL CID is always a **CIDv1** with codec `raw` (`0x55`) or `dag-cbor` +(`0x71`), hash SHA-256 (`0x12`), 32-byte digest, base32lower string form with a +leading `b`, 36-byte binary form. The BDASL extension permits BLAKE3 (`0x1e`) +for large-file blobs. Anything outside that set is rejected — DASL is a strict +subset of multiformats CIDs, not an alias. + +- String form: `b` → 59 chars total; `bafyrei…` = + dag-cbor record/MST node, `bafkrei…` = raw blob. First 7 chars are a fast + sniff test (`01 71 12 20` → bafyrei, `01 55 12 20` → bafkrei). +- Binary form: 36 bytes flat `version(1) || codec(1) || hash_code(1) || + digest_length(1) || digest(32)`. No tag, no length prefix. +- DAG-CBOR wire form: CBOR **tag 42** wrapping a byte string whose first byte + is the identity multibase prefix `0x00` + the 36 binary CID bytes = **37 + bytes** inside the byte string. Encoders do this automatically; hand-rolling + is a footgun. +- JSON form (AT Protocol): `{"$link": "bafyreihunttf7a3uvtzrgbnyu2rzv24w4zx7xjwqgk4x5w7n5yvq7u7aua"}` + — a bare string in a CID-typed field is invalid. The DASL spec itself does + not define a JSON encoding; `$link` is an AT Protocol convention. + +## Validation vs verification + +- **Validation**: parse succeeded into the DASL subset (version 1, codec ∈ + {0x55, 0x71}, hash ∈ {0x12, 0x1e-if-BDASL}, digest length 32). +- **Verification**: re-hash the content, rebuild the CID, compare byte-for-byte. + Always a separate step; no library does it for you. + +Conflating these is the most common bug in a new CID implementation. + +## Common pitfalls + +- **Missing `0x00` identity multibase prefix** inside DAG-CBOR tag-42 wrapping + — produces a CID other implementations silently reject. +- **CIDv0 `Qm…`** accepted by permissive libraries (TypeScript and Go in + particular). Always re-validate against the DASL subset after parsing. +- **dag-pb (`0x70`) vs dag-cbor (`0x71`)** — one codec byte apart; dag-pb is + IPFS-only and not a DASL codec. +- **base58btc (`z…`) or base64 (`m…`) prefixes** — valid multibase CIDs, not + DASL. Reject. +- **JS-specific: async hashing propagates** — you cannot collapse + `await sha256.digest(bytes)` into a sync helper; every caller building a CID + from bytes becomes async. +- **Codec constants shipped only in Go** — Rust and TS require importing or + hand-rolling. +- **TypeScript uses `cid.bytes` (property), not `cid.toBytes()`** — port-and- + paste hazard. +- **Unpadded base32 decode** — multibase base32lower is unpadded; configure the + stdlib decoder or valid DASL strings get rejected. + +## Decision rules + +- **DASL vs BDASL?** DASL (SHA-256) everywhere in the AT Protocol repo graph. + BDASL (BLAKE3) only when the platform explicitly opts in for large-file + content (video blobs). +- **raw vs dag-cbor?** `dag-cbor` for structured records (vast majority); `raw` + for opaque binary blobs referenced from records. +- **Accept a CIDv0?** No. Reject with a clear error — no lossy upgrade. +- **Store string or binary?** Binary in DAG-CBOR and CAR; string in JSON and + logs. Convert at boundaries. +- **Equality?** Compare the 36-byte binary forms — string forms can drift in + case/padding even when conformant. + +## Quick local validation + +```bash +python3 scripts/validate_cid.py [--bdasl] +``` + +Exit 0 = valid DASL string, 1 = rejected with reason on stdout, 2 = usage +error. Stdlib-only, no network, no hashing — parses the 4-byte header + digest +length against the allowed constants. Does NOT verify content. + +## References (read the relevant one before writing CID code) + +- `references/shared/spec.md` — normative DASL rules (read first; short) +- `references/shared/binary-layout.md` — byte-level diagrams +- `references/shared/test-vectors.md` — fixtures +- `references/shared/divergence-matrix.md` — cross-language differences +- `references/typescript/README.md` — multiformats + @ipld/dag-cbor setup +- `references/typescript/parsing.md` — `CID.parse` / decode + DASL gate +- `references/typescript/construction.md` — async sha256 + `CID.createV1` +- `references/typescript/codecs.md` — per-codec packages + +Always prefer the official library (`multiformats` in TS) over hand-rolling. +Never guess function names — read the reference file, then fetch live docs if a +detail is missing. + +## Related skills + +- `atproto-repository` — CAR v1 framing, MST traversal, DRISL canonical CBOR, + commit signing (CIDs are the glue between these layers) +- `atproto-blob-lifecycle` — CID-based blob scanning/cleanup (client-side + hydration gotcha: API clients convert `{$link}` refs into CID objects; + `ref.toString()` yields the bafk… string) +- `atproto-development` — JS SDK basics; `atproto-python` — Python SDK + (`atproto_core` has CID/CAR/DAG-CBOR utilities) diff --git a/skills/software-development/atproto-cid/references/shared/binary-layout.md b/skills/software-development/atproto-cid/references/shared/binary-layout.md new file mode 100644 index 0000000..4ebfe65 --- /dev/null +++ b/skills/software-development/atproto-cid/references/shared/binary-layout.md @@ -0,0 +1,136 @@ +# Binary and Wire Layout + +This document walks through the on-the-wire byte representation of a DASL CID in every place it appears in AT Protocol: standalone, embedded in DAG-CBOR, and embedded in JSON. It also derives the human-recognisable string prefixes (`bafyrei…` and `bafkrei…`) bit by bit so implementers can reproduce them. + +## 1. The 36-byte binary CID + +``` +offset 0 1 2 3 4 ..................................... 35 + ┌────┬────┬────┬────┬─────────────────────────────────────────┐ + │0x01│cdc │hsh │0x20│ digest │ + │ │ │ │ │ (32 bytes) │ + └────┴────┴────┴────┴─────────────────────────────────────────┘ + ver cdc hsh len +``` + +- `ver` (offset 0) = `0x01` — CIDv1. +- `cdc` (offset 1) = `0x55` (raw) or `0x71` (dag-cbor). +- `hsh` (offset 2) = `0x12` (SHA-256) or, under BDASL, `0x1e` (BLAKE3). +- `len` (offset 3) = `0x20` — 32-byte digest length, always. +- `digest` (offsets 4–35) — the 32-byte SHA-256 or BLAKE3 output of the content. + +Example (dag-cbor + SHA-256, first record in Bluesky's `app.bsky.actor.profile` lexicon): + +``` +01 71 12 20 b1 a5 62 d4 71 a3 6d 7a 9f e4 2b 63 87 c1 5e 8d + d3 c4 7e f2 90 16 88 a4 05 7b 19 cc fa 7d 4e 22 +``` + +(Digest bytes are illustrative; the header is exact.) + +## 2. DAG-CBOR wrapping (CBOR tag 42 + identity multibase) + +When a CID appears as a value inside a DAG-CBOR object, it is not written as those 36 bytes directly. It is wrapped: + +``` +CBOR byte sequence for a 36-byte CID: + + d8 2a ; tag(42) — the IPLD "this is a link" marker + 58 25 ; bytes(37) — major type 2 (byte string), length 37 + 00 ; identity multibase prefix + 01 71 12 20 ; CID header + <32 bytes> ; digest +``` + +Key facts: + +- Tag 42 (`0xd82a`) is what DAG-CBOR uses to signal "this byte string is a CID, not arbitrary bytes". Omit it and consumers will see a 37-byte blob. +- The **inner byte string is 37 bytes**, not 36. The extra leading `0x00` is the multibase "identity" prefix — it says "the following bytes are literal binary, not a text-encoded representation". IPLD requires it. Implementations that forget to emit it produce payloads other readers will reject (this is one of the most common interop bugs). +- The CBOR length header for a 37-byte string is `58 25` (major type 2, one-byte length = `0x25` = 37). This is the canonical shortest form per DAG-CBOR deterministic encoding. A non-canonical length prefix (for example a two-byte length `59 00 25`, or an indefinite-length byte string `5f … ff`) must be rejected by a strict decoder — even though such encodings are legal in generic CBOR, they are forbidden in DAG-CBOR and would let the same logical CID have multiple on-wire representations. + +Round-trip pseudocode: + +``` +function encode_cid_dag_cbor(cid_bytes_36): + inner = bytes([0x00]) + cid_bytes_36 # 37 bytes + return cbor_tag(42, cbor_byte_string(inner)) # d8 2a 58 25 ... + +function decode_cid_dag_cbor(cbor_payload): + tag, payload = cbor_read_tag(cbor_payload) + assert tag == 42 # else: not a CID + inner = cbor_read_byte_string(payload) + assert inner[0] == 0x00 # else: missing identity prefix + cid_bytes = inner[1:] # 36 bytes + return cid_bytes +``` + +## 3. JSON wrapping (`$link`) + +In AT Protocol JSON, a CID-typed field is an object with a single `$link` key whose value is the string form: + +```json +{"$link": "bafyreihunttf7a3uvtzrgbnyu2rzv24w4zx7xjwqgk4x5w7n5yvq7u7aua"} +``` + +- This format is defined by AT Protocol, not by DASL or IPLD. +- Bare strings and bare binary values are not acceptable — reject them. +- Round-tripping JSON ↔ DAG-CBOR must convert between `{"$link": "b…"}` and the tag-42 + identity-prefix wrapping above. + +## 4. String form: deriving the fixed prefixes + +The first four bytes of the binary CID are fixed by the header. Base32 encodes five bits per character, so the first 30 bits of the binary — all of bytes 0, 1, 2 (24 bits) plus the top 6 bits of byte 3 — map onto the first **six** data characters of the string form (characters 2 through 7, after the `b` multibase prefix). Those six characters are therefore **entirely determined** by the codec and hash choice. The seventh data character (the eighth character of the full string) depends on the digest. + +Base32 encodes five bits per character. The RFC 4648 lowercase alphabet: `a`=0, `b`=1, …, `z`=25, `2`=26, `3`=27, `4`=28, `5`=29, `6`=30, `7`=31. + +### Dag-cbor + SHA-256 + 32 (`01 71 12 20 …`) + +``` +byte layout : 00000001 01110001 00010010 00100000 .... +bit stream : 0000000101110001000100100010 0000 .... +split by 5 : 00000 00101 11000 10001 00100 01000 0000.... + 0 5 24 17 4 8 +chars : a f y r e i +string prefix : "b" + "afyrei…" = "bafyrei…" +``` + +So every record / MST node / commit CID in AT Protocol starts with **`bafyrei`**. The 8th character onward depends on the digest. + +### Raw + SHA-256 + 32 (`01 55 12 20 …`) + +``` +byte layout : 00000001 01010101 00010010 00100000 .... +bit stream : 0000000101010101000100100010 0000 .... +split by 5 : 00000 00101 01010 10001 00100 01000 0000.... + 0 5 10 17 4 8 +chars : a f k r e i +string prefix : "b" + "afkrei…" = "bafkrei…" +``` + +So every blob CID in AT Protocol starts with **`bafkrei`**. The 8th character onward depends on the digest. + +### Prefix sniff test + +| First 7 chars | Meaning | +| --------------- | ------------------------------------------ | +| `bafyrei…` | DASL dag-cbor record (valid) | +| `bafkrei…` | DASL raw blob (valid) | +| `Qm…` | IPFS CIDv0 — **reject** (not DASL) | +| `bafybei…` etc. | IPFS CIDv1 with non-DASL codec — **reject** (e.g. `dag-pb`) | +| `z…`, `f…`, `m…` | IPFS CIDv1 with non-base32lower multibase — **reject** | + +For BDASL CIDs (BLAKE3 in place of SHA-256) the derivation changes because byte 2 becomes `0x1e` instead of `0x12`; work through the bits the same way to derive the fixed prefix characters for your deployment. + +## 5. Base32 encoding / decoding mechanics + +- Alphabet: `abcdefghijklmnopqrstuvwxyz234567` (32 chars). +- Each group of 5 input bits → 1 output character. +- 36 bytes = 288 bits → 288 / 5 = 57.6 → 58 characters (final character carries 2 unused trailing bits set to zero). +- No padding (`=`) even though RFC 4648 permits it — multibase base32lower is the **unpadded** variant. + +Total string length for a DASL CID is therefore always: + +``` +1 (multibase prefix "b") + 58 (base32 chars) = 59 characters +``` + +If a candidate string is not exactly 59 characters long, something is wrong. That is a cheap length check to run before attempting a full base32 decode. diff --git a/skills/software-development/atproto-cid/references/shared/divergence-matrix.md b/skills/software-development/atproto-cid/references/shared/divergence-matrix.md new file mode 100644 index 0000000..f638f2f --- /dev/null +++ b/skills/software-development/atproto-cid/references/shared/divergence-matrix.md @@ -0,0 +1,93 @@ +# Cross-Language Divergence Matrix (DASL CID) + +This file is language-neutral. It captures the real behavioural differences between the Rust, TypeScript, and Go CID ecosystems that any skill user porting code or operating cross-stack needs to know about. + +Every per-language file (`rust/*.md`, `typescript/*.md`, `go/*.md`) links back here instead of restating the matrix. + +## Library choice + +| Ecosystem | Base library | DAG-CBOR | Hashing | DASL-strict wrapper | +| --- | --- | --- | --- | --- | +| Rust | [`cid`](https://docs.rs/cid) 0.11 + [`multihash-codetable`](https://docs.rs/multihash-codetable) | [`atproto-dasl`](https://docs.rs/atproto-dasl) (DRISL-strict) or [`serde_ipld_dagcbor`](https://docs.rs/serde_ipld_dagcbor) | `sha2` crate (`Sha256::digest`) | [`atproto-dasl::DaslCid`](https://docs.rs/atproto-dasl) — rejects non-DASL at construction | +| TypeScript | [`multiformats`](https://www.npmjs.com/package/multiformats) 13 | [`@ipld/dag-cbor`](https://www.npmjs.com/package/@ipld/dag-cbor) | `multiformats/hashes/sha2` (SubtleCrypto-backed) | None shipped — DASL validation is caller-owned | +| Go | [`github.com/ipfs/go-cid`](https://pkg.go.dev/github.com/ipfs/go-cid) + [`github.com/multiformats/go-multihash`](https://pkg.go.dev/github.com/multiformats/go-multihash) | [`github.com/ipld/go-ipld-prime/codec/dagcbor`](https://pkg.go.dev/github.com/ipld/go-ipld-prime/codec/dagcbor) | `crypto/sha256` or via `go-multihash` | None shipped — DASL validation is caller-owned | + +Only Rust has a shipped DASL-strict wrapper. In TypeScript and Go, the libraries are permissive — they will happily parse a CIDv0 `Qm…` or a `dag-pb` (0x70) CID without complaining. **DASL validation is always the caller's job in TypeScript and Go.** See each language's `parsing.md` for the gate code. + +## Operation-level divergence + +| Operation | Rust | TypeScript | Go | +| --- | --- | --- | --- | +| Parse string | `Cid::try_from(s)` / `atproto_dasl::DaslCid::from_str(s)` | `CID.parse(str)` (base32lower default; explicit base decoder needed for others) | `cid.Decode(str)` | +| Parse 36 bytes | `Cid::try_from(&[u8])` / `DaslCid::try_from(&[u8])` | `CID.decode(bytes)` (strict) or `CID.decodeFirst(bytes)` (returns remainder) | `cid.Cast(bytes)` | +| Streaming read | `atproto_dasl::cid::read_cid(reader)` | `CID.decodeFirst(bytes)` (sync, works on any `Uint8Array`) | `cid.CidFromReader(reader)` | +| Construct from (codec, digest) | `Cid::new_v1(codec_u64, mh)` | `CID.createV1(code, multihashDigest)` | `cid.NewCidV1(codec, mhBytes)` | +| Construct from content | `atproto_dasl::compute_cid(bytes)` (dag-cbor default) or manual via `Sha256::digest` + `Cid::new_v1` | `const digest = await sha256.digest(bytes); CID.createV1(codec, digest)` | `p := cid.Prefix{Version:1, Codec:cid.DagCBOR, MhType:multihash.SHA2_256, MhLength:32}; p.Sum(bytes)` | +| Hash call | `Sha256::digest(bytes)` — **sync** | `await sha256.digest(bytes)` — **async** (SubtleCrypto) | `multihash.Sum(bytes, multihash.SHA2_256, 32)` — **sync** | +| Codec constants | Re-exported: `atproto_dasl::DAG_CBOR_CODEC = 0x71`; bare `cid` crate has no enum (use `u64` constants) | Import from per-codec: `import * as dagCbor from '@ipld/dag-cbor'` → `dagCbor.code`; `import * as raw from 'multiformats/codecs/raw'` → `raw.code` | **Shipped**: `cid.DagCBOR`, `cid.Raw`, `cid.DagPB`, etc. — free off the `cid` package | +| Byte output (36 bytes) | `cid.to_bytes() -> Vec` (method) | `cid.bytes` (**property**, `Uint8Array`) | `c.Bytes() []byte` (method) | +| String output | `cid.to_string()` (Display impl → base32lower for v1) | `cid.toString()` (base32lower for v1) | `c.String()` | +| DAG-CBOR encode a CID | Automatic via `atproto_dasl::to_vec(value)` — serde emits tag 42 + `0x00` identity multibase prefix | Automatic via `dagCbor.encode(value)` — emits tag 42 + `0x00` | Automatic via `go-ipld-prime/codec/dagcbor` — emits tag 42 + `0x00` | +| JSON `$link` form | `atproto_dasl::Cid` round-trips `{"$link": "…"}` via serde | Hand-roll: `{ $link: cid.toString() }`, or use [`@ipld/dag-json`](https://www.npmjs.com/package/@ipld/dag-json) | Hand-roll: `map[string]string{"$link": c.String()}`, or use `go-ipld-prime/codec/dagjson` | +| Errors | Typed enum: `cid::Error`, `atproto_dasl::DaslCidError` with variants per failure class | Plain `Error` (sometimes `TypeError`) — inspect `.message` strings | Sentinel values: `cid.ErrCidTooShort`, `cid.ErrInvalidCid`, `cid.ErrVarintBuffSmall`, `multihash.ErrUnknownCode`, etc. | +| BLAKE3 (BDASL) support | `multihash-codetable` with the `blake3` feature; `atproto-dasl` recognizes `0x1e` | No first-party; use `@noble/hashes/blake3` or a third-party multihash adapter | `multihash.Register(multihash.BLAKE3, …)` is available; codec constant is `multihash.BLAKE3 = 0x1e` | + +## Divergences worth highlighting in prose + +### 1. JavaScript hashing is async — everything "compute a CID" becomes async + +Every `await sha256.digest(bytes)` in a TypeScript codebase propagates `async` through the call chain: the function that builds a record CID is async, which makes its callers async, and so on. Rust and Go stay synchronous because `Sha256::digest` and `crypto/sha256` are synchronous. + +Don't try to hide this with `.then()` chains or `Promise.resolve()` wrappers. The `await` is there because `crypto.subtle.digest` is the underlying primitive in the browser, and the Node implementation mirrors it for isomorphism. `typescript/construction.md` shows the canonical pattern. + +### 2. Codec constants are inconsistently shipped + +- **Go** ships `cid.Raw = 0x55`, `cid.DagCBOR = 0x71`, `cid.DagPB = 0x70` as top-level constants. This is the easiest ecosystem to write strict validation in. +- **Rust**'s bare `cid` 0.11 crate removed its `Codec` enum; you use `u64` literals (or re-export from `multicodec` / `libipld`). `atproto-dasl` gives you `DAG_CBOR_CODEC` as a const, but no enum. +- **TypeScript**'s `multiformats` does not ship codec constants centrally. Each codec package (`@ipld/dag-cbor`, `@ipld/dag-json`, `multiformats/codecs/raw`) exports its own `.code`. When you want to validate "is this codec 0x55 or 0x71?", you either hand-write the constants or import two packages for the side effect of their `.code` exports. + +When a skill user asks for "the DAG-CBOR codec constant," check their language — the answer is different. + +### 3. Byte output: method vs property + +TypeScript's `cid.bytes` is a property (returns `Uint8Array`); Rust's `cid.to_bytes()` and Go's `c.Bytes()` are methods. This is the single most common port-and-paste bug — an `await` or `()` in the wrong place. + +All three return the **36-byte binary form** (no identity multibase prefix). That prefix is only present inside DAG-CBOR tag-42 wrapping, where it is added automatically by the canonical encoder. + +### 4. DASL strictness is a caller-owned gate in two of three languages + +The DASL CID spec is a strict *subset* of multiformats CIDs. None of the bare libraries enforces this: + +- `CID.parse("QmHash…")` in TypeScript silently returns a CIDv0. +- `cid.Decode("QmHash…")` in Go silently returns a CIDv0. +- `Cid::try_from("QmHash…")` in Rust silently returns a CIDv0. + +Only `atproto_dasl::DaslCid` rejects non-DASL at parse time. Every TypeScript and Go call site that expects DASL must re-validate the output (`cid.version === 1`, `cid.code === 0x55 || cid.code === 0x71`, `cid.multihash.code === 0x12` or `0x1e`, `cid.multihash.size === 32`). See `{lang}/parsing.md` for the exact gate. + +### 5. Error handling shapes differ enough to change code structure + +- Rust: `match err` on a typed enum, exhaustive. +- Go: `errors.Is(err, cid.ErrCidTooShort)` against sentinel values; no enum-like exhaustiveness. +- TypeScript: `err instanceof TypeError` or string-match on `err.message`. Brittle. + +A skill user porting error-path logic should not assume `.message` strings carry across — Rust error variants have names (`DaslCidError::InvalidCodec { codec }`) that are genuinely more informative than the TS string `"Unsupported codec: 0x70"`. + +### 6. Validation has no standalone `validate()` — it's "parse with strict expectations" + +None of the three libraries expose a `validate(cid)` function. Validation means "parse successfully into the expected subset." Verification (re-hashing content to confirm the digest matches) is always a separate, caller-owned step. Every `{lang}/parsing.md` shows both: + +- "Parse succeeded with DASL-compatible shape" → input is a well-formed DASL CID. +- "Re-hash content, rebuild CID, compare byte-for-byte" → content matches this CID. + +Collapsing those two steps into one call is a common misunderstanding; keep them separate. + +## When in doubt, lean on the reference implementations + +- Rust: [`atproto-dasl`](https://docs.rs/atproto-dasl) is the ATProtocol-maintained reference. `DaslCid` enforces every rule in `shared/spec.md` at construction. +- Go / TypeScript: no single maintained strict wrapper; the gate code in `parsing.md` is the skill's reference. Port it verbatim. + +## Related + +- `shared/spec.md` — normative rules. +- `shared/binary-layout.md` — byte-level diagrams that the per-language code must produce. +- `shared/test-vectors.md` — fixtures for cross-language agreement testing. diff --git a/skills/software-development/atproto-cid/references/shared/spec.md b/skills/software-development/atproto-cid/references/shared/spec.md new file mode 100644 index 0000000..d639f47 --- /dev/null +++ b/skills/software-development/atproto-cid/references/shared/spec.md @@ -0,0 +1,94 @@ +# DASL CID Specification (Reference) + +Source of truth: https://dasl.ing/cid.html + +This file restates the normative rules so implementers can work from a single document without leaving the skill. The paraphrase is lossless with respect to the spec's rejection conditions; if this file and https://dasl.ing/cid.html ever disagree, the upstream spec wins. + +## 1. Version + +A DASL CID is **CIDv1 only**. Parsing must reject any CID whose version byte is not `0x01`. Legacy CIDv0 CIDs are not accepted — there is no silent upgrade. + +## 2. Codec (multicodec) + +Exactly two codecs are permitted: + +| Codec | Hex | Purpose | +| ---------- | ------ | ----------------------------------------- | +| `raw` | `0x55` | Opaque / unstructured content (blobs). | +| `dag-cbor` | `0x71` | DRISL-conformant DAG-CBOR structures (records, commits, MST nodes). | + +Any other codec — including `dag-pb` (`0x70`), `dag-json` (`0x0129`), `identity` (`0x00`), or anything else — must be rejected. + +## 3. Hash function (multihash) + +DASL permits **SHA-256 only**, with hash code `0x12`. Any other hash code must be rejected. + +**BDASL extension.** The BDASL profile (used for large-file content) additionally permits BLAKE3 (`0x1e`). When a codec explicitly opts into BDASL it must accept both `0x12` and `0x1e`; otherwise only `0x12`. + +## 4. Digest length + +The digest is always exactly **32 bytes**. The multihash length byte must equal `0x20` (32). Any other length must be rejected, including a CID whose trailing bytes are shorter than the declared length. + +## 5. String form (multibase) + +The text representation of a DASL CID is: + +``` +b +``` + +- `b` is the multibase prefix indicating base32 with the lowercase alphabet (RFC 4648). +- The encoding is unpadded (no trailing `=`). +- No other multibase is accepted. Specifically, `z` (base58btc), `m` (base64), `f` (base16), and uppercase `B` are all rejected. + +## 6. Binary form + +The binary representation is the concatenation of: + +``` +0x01 || codec (1 byte, 0x55 or 0x71) + || hash-code (1 byte, 0x12 [or 0x1e for BDASL]) + || digest-length (1 byte, 0x20) + || digest (32 bytes) +``` + +Total length is always 36 bytes. No varint-encoded fields, no optional tags — the layout is fixed. Although multicodec and multihash codes are in principle varints, every code used in a DASL CID (`0x01`, `0x55`, `0x71`, `0x12`, `0x1e`, `0x20`) fits in a single byte, so no multi-byte varint decoding is ever required; implementers should not reach for a varint helper to parse DASL CIDs. + +## 7. Validation order + +Implementations should check in this order so error messages point at the first thing that is wrong: + +1. String form: first character is `b`. If not, reject immediately. +2. Base32 decode succeeds and produces exactly 36 bytes. +3. Version byte = `0x01`. +4. Codec byte ∈ { `0x55`, `0x71` }. +5. Hash code byte = `0x12` (or `0x1e` if BDASL is enabled). +6. Digest length byte = `0x20`. +7. Digest is exactly 32 bytes. + +Each rejection is a hard error. The spec explicitly forbids fallbacks, warnings, or "best-effort" acceptance. + +## 8. Content addressing policy + +The DASL project recommends **not** chunking content into a DAG or Merkle tree. Each resource is hashed as a whole and content-addressed directly. This removes canonicalization ambiguity introduced by recursive chunking strategies like UnixFS. AT Protocol follows this policy: record CIDs are computed over the whole canonical DAG-CBOR payload; blob CIDs are computed over the whole blob. + +## 9. Relationship to multiformats / IPFS + +Every DASL CID is a valid IPFS CIDv1, but the converse is not true. DASL removes: + +- CIDv0 support. +- Multibases other than base32lower. +- Codecs other than raw and dag-cbor. +- Hash functions other than SHA-256 (DASL) and BLAKE3 (BDASL). +- Digest lengths other than 32 bytes. +- Chunked / UnixFS-style DAGs. + +Interoperability goes one way: an AT Protocol service can always hand a DASL CID to an IPFS tool, but may receive IPFS CIDs that are not valid DASL and must reject them. + +## 10. What this spec does *not* define + +The DASL spec is deliberately narrow. It does **not** cover: + +- A JSON encoding. AT Protocol defines `{"$link": ""}` separately; see the SKILL body. +- The CBOR tag 42 wrapping used when a CID is embedded inside DAG-CBOR. That comes from the IPLD DAG-CBOR codec spec. See `binary-layout.md`. +- Block / link framing, CAR files, or signing. Those belong to adjacent specs. diff --git a/skills/software-development/atproto-cid/references/shared/test-vectors.md b/skills/software-development/atproto-cid/references/shared/test-vectors.md new file mode 100644 index 0000000..5bfa2a2 --- /dev/null +++ b/skills/software-development/atproto-cid/references/shared/test-vectors.md @@ -0,0 +1,128 @@ +# Test Vectors for DASL / AT Protocol CIDs + +The DASL CID spec itself provides no normative test vectors (see https://dasl.ing/cid.html). The vectors below are derived from the reference Rust implementation at `atproto-dasl/src/cid` and are suitable for cross-implementation fixtures. Reproduce them with your own implementation by following the procedure; if you disagree with a value, run the Rust tests to see which side is wrong. + +## How to reproduce any vector + +For any input byte sequence `D`: + +1. Compute `H = SHA-256(D)`. (For BDASL, use `BLAKE3(D)` and substitute `0x1e` below.) +2. Assemble the 36-byte binary CID: + - For dag-cbor content: `01 71 12 20 || H` + - For raw content: `01 55 12 20 || H` +3. String form: `"b" + base32lower(binary)`, unpadded. +4. DAG-CBOR form: `d8 2a 58 25 00 || binary` (tag 42 → byte string of 37 → identity prefix → binary). +5. JSON form: `{"$link": ""}`. + +## Vector 1 — empty bytes, dag-cbor codec + +Purpose: the simplest possible record CID (sha-256 of zero bytes). + +| Field | Value | +| ----------------- | ------------------------------------------------------------------------------------------- | +| Input | `b""` (zero bytes) | +| SHA-256 digest | `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` | +| Binary CID (hex) | `01 71 12 20 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` | +| String form | `bafyreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku` | +| Binary length | 36 | +| String length | 59 | + +Use this as a smoke test that your encoder rejects zero-length input in the caller where that is not meaningful (a record is never empty) but handles it deterministically where it is. + +## Vector 2 — empty bytes, raw codec + +Purpose: the same digest under the raw codec — shows the only bit that changes is byte 1 (codec). + +| Field | Value | +| ----------------- | ------------------------------------------------------------------------------------------- | +| Input | `b""` (zero bytes) | +| SHA-256 digest | `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` | +| Binary CID (hex) | `01 55 12 20 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` | +| String form | `bafkreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku` | + +Vectors 1 and 2 differ only by the codec byte (`0x71` vs `0x55`) and the corresponding fourth string character (`y` vs `k`). The remaining 55 characters are identical because the digest is identical. + +## Vector 3 — ASCII string, dag-cbor codec + +Purpose: a non-trivial input that is easy to copy/paste into any language. + +| Field | Value | +| ----------------- | ------------------------------------------------------------------------------------------- | +| Input | `b"hello world"` (11 ASCII bytes) | +| SHA-256 digest | `b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9` | +| Binary CID (hex) | `01 71 12 20 b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9` | +| String form | `bafyreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e` | + +Use this to verify your `compute_cid(dag_cbor_bytes)` path end-to-end without involving a DAG-CBOR encoder: just feed the literal bytes `hello world` through your hashing pipeline. + +## Vector 4 — ASCII string, raw codec + +Purpose: a raw blob CID with a non-trivial, easy-to-reproduce input. + +| Field | Value | +| ----------------- | ------------------------------------------------------------------------------------------- | +| Input | `b"raw content"` (11 ASCII bytes) | +| SHA-256 digest | `a6e5d15bf571ca7a23fd704caad6c4c071210ba8d38ea0296dc58c3ce0a0e514` | +| Binary CID (hex) | `01 55 12 20 a6e5d15bf571ca7a23fd704caad6c4c071210ba8d38ea0296dc58c3ce0a0e514` | +| String form | `bafkreifg4xivx5lrzj5ch7lqjsvnnrgaoeqqxkgtr2qcs3ofrq6obihfcq` | + +## Vector 5 — rejection cases + +Every one of these MUST be rejected by a conformant DASL CID parser: + +| Input | Reason | +| ------------------------------------------------------------ | ---------------------------------------------------------------- | +| `"QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG"` | CIDv0 (no multibase, legacy). | +| `"zdj7WfN9c4GJ…"` | CIDv1 but multibase is `z` (base58btc), not `b`. | +| `"BAFYREI…" ` | Uppercase `B` — multibase is case-sensitive; base32lower only. | +| `"bafybeid7t3x7…"` (CIDv1 dag-pb codec `0x70`) | Codec `0x70` is not in the DASL set. | +| 36-byte CID whose byte 0 is `0x00` instead of `0x01` | CIDv0 or malformed. | +| 36-byte CID whose byte 2 is `0x1e` in a DASL-only context | BLAKE3 is BDASL, not DASL. Reject unless BDASL is explicitly on. | +| 36-byte CID whose byte 3 is `0x10` (16) | Wrong digest length. | +| 35-byte total length | Truncated digest. | +| DAG-CBOR tag-42 byte string whose first inner byte is `0x01` | Missing identity multibase prefix `0x00`. | + +## Vector 6 — DAG-CBOR wire round-trip + +Purpose: confirm your CBOR encoder emits exactly the right bytes around a CID. + +Input: any 36-byte binary CID, say `B`. + +Expected CBOR encoding of a single-CID value: + +``` +d8 2a ; tag(42) +58 25 ; bytes(37) +00 ; identity multibase + ; 36 bytes +``` + +Total encoded length: 2 + 2 + 1 + 36 = 41 bytes. + +Concrete example, using the binary CID from Vector 1 (empty-input dag-cbor): + +``` +d82a58250001711220e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +``` + +(41 bytes / 82 hex characters.) If your encoder produces any other byte sequence for this input — a different tag (`d8 2a`), a different length framing (`58 25`), a missing identity prefix (`00`), or a byte re-order — the output is non-conformant. + +## Vector 7 — JSON round-trip + +Purpose: confirm your JSON codec serializes and parses the `$link` form. + +Serialization input: any `Cid` value `C`. + +Expected JSON output: + +```json +{"$link": ""} +``` + +Parse test: `{"$link": "bafyrei…"}` should parse to a valid CID; `"bafyrei…"` (a bare string) should NOT, and `{"cid": "bafyrei…"}` should NOT. + +Concrete example, pairing with Vector 1: + +```json +{"$link": "bafyreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku"} +``` diff --git a/skills/software-development/atproto-cid/references/typescript/README.md b/skills/software-development/atproto-cid/references/typescript/README.md new file mode 100644 index 0000000..1f02870 --- /dev/null +++ b/skills/software-development/atproto-cid/references/typescript/README.md @@ -0,0 +1,122 @@ +# TypeScript (multiformats / @ipld/dag-cbor) + +The TypeScript ecosystem for CIDs is the [`multiformats`](https://www.npmjs.com/package/multiformats) package — the reference IPLD library that Bluesky's own `@atproto/*` packages depend on transitively. For DAG-CBOR encoding/decoding, pair it with [`@ipld/dag-cbor`](https://www.npmjs.com/package/@ipld/dag-cbor). + +Unlike Rust's `atproto-dasl`, there is **no shipped DASL-strict wrapper** in TypeScript. DASL validation is always a caller-owned gate on top of the permissive `multiformats` CID. + +## Dependencies + +```bash +npm install multiformats @ipld/dag-cbor +``` + +Both packages are ESM-only. Set `"type": "module"` in `package.json` (or use a bundler that handles ESM — Vite, esbuild, modern Webpack). If you are stuck on CommonJS, import via dynamic `import()` or pin to older versions (not recommended — the old multiformats 9.x API is different). + +TypeScript tsconfig hints: + +```json +{ + "compilerOptions": { + "module": "Node16", + "moduleResolution": "Node16", + "target": "ES2022", + "strict": true + } +} +``` + +`ES2022` gives you `Uint8Array`'s methods out of the box and avoids polyfill surface for `crypto.subtle`. + +## Core imports + +```ts +import { CID } from 'multiformats/cid' +import { sha256 } from 'multiformats/hashes/sha2' +import * as dagCbor from '@ipld/dag-cbor' +import * as raw from 'multiformats/codecs/raw' +``` + +That is the full surface you need for DASL CIDs. No `multiformats/bases/base32` import needed — base32lower is the default for v1 string output and for parsing `b…` prefixes. + +## The DASL gate — your validator + +Because `multiformats` accepts any valid multiformats CID, you need a tiny gate function: + +```ts +import { CID } from 'multiformats/cid' + +export type DaslCid = CID & { readonly __dasl: true } + +const DAG_CBOR = 0x71 as const +const RAW = 0x55 as const +const SHA256 = 0x12 as const +const BLAKE3 = 0x1e as const + +export function assertDasl(cid: CID, { allowBlake3 = false } = {}): asserts cid is DaslCid { + if (cid.version !== 1) throw new TypeError(`CID version ${cid.version} not allowed (DASL requires v1)`) + if (cid.code !== DAG_CBOR && cid.code !== RAW) { + throw new TypeError(`Codec 0x${cid.code.toString(16)} not allowed (expected dag-cbor or raw)`) + } + const hashCode = cid.multihash.code + const hashOk = hashCode === SHA256 || (allowBlake3 && hashCode === BLAKE3) + if (!hashOk) throw new TypeError(`Hash 0x${hashCode.toString(16)} not allowed (expected SHA-256${allowBlake3 ? ' or BLAKE3' : ''})`) + if (cid.multihash.size !== 32) throw new TypeError(`Digest length ${cid.multihash.size} not 32`) +} +``` + +Call `assertDasl(cid)` immediately after every `CID.parse` / `CID.decode` / `CID.asCID` on CIDs from untrusted sources. The branded `DaslCid` type propagates the guarantee through your codebase so internal functions can take `DaslCid` instead of `CID` and trust the shape. + +This is the single most important adapter to write in a TypeScript codebase. Copy-paste it; test it against the fixtures in `../shared/test-vectors.md`. + +## Block helpers — the ergonomic option + +`multiformats/block` wraps hashing + encoding + CID creation into a single async call: + +```ts +import * as Block from 'multiformats/block' +import * as dagCbor from '@ipld/dag-cbor' +import { sha256 } from 'multiformats/hashes/sha2' + +const block = await Block.encode({ + value: { $type: 'app.bsky.actor.profile', displayName: 'Alice' }, + codec: dagCbor, + hasher: sha256, +}) +// block.cid → CID (validate with assertDasl) +// block.bytes → Uint8Array (canonical CBOR) +// block.value → original value, round-tripped +``` + +Use `Block.encode` for the "record → CID + bytes" flow and `Block.decode` for the reverse. It is a thin wrapper but handles the three-step dance (encode, hash, assemble CID) you would otherwise write inline. See `construction.md` for when to use it vs raw `CID.createV1`. + +## Canonical encoding + +`@ipld/dag-cbor`'s `encode(value)` is DRISL-compliant: keys sorted bytewise, integers shortest form, no indefinite-length items, tag 42 for CIDs. Do **not** use the general-purpose `cbor-x`, `cbor`, or `borc` packages — they emit non-canonical output and your CIDs will not match other AT Protocol implementations. + +```ts +import * as dagCbor from '@ipld/dag-cbor' + +const bytes: Uint8Array = dagCbor.encode({ b: 2, a: 1 }) +// → a2 61 61 01 61 62 02 (keys sorted to a, b — not b, a) +``` + +If you need to double-check a record against the reference implementation, call the `lexicon-garden` MCP tool's `create_record_cid` — that's ground truth. + +## Async is unavoidable + +`sha256.digest(bytes)` returns a `Promise` because it delegates to `crypto.subtle.digest` under the hood. Every function that builds a CID from bytes ends up `async`. See `construction.md` for the propagation pattern and why a synchronous shim is a bad idea. + +## Idioms TypeScript engineers expect + +- **Errors are thrown, not returned.** The `assertDasl` example throws `TypeError`. Your codebase might wrap these in a typed `Result` or `neverthrow`-style abstraction — the pattern is yours to choose, but the underlying multiformats API throws. +- **CIDs are compared with `cid.equals(other)`** — not `===` (object identity) and not `cid.toString() === other.toString()` (string comparison is fragile if the string forms drift). `cid.equals` compares the binary bytes. +- **`cid.bytes` is a property**, not a method. Reading `cid.bytes()` is a runtime error (`cid.bytes is not a function`). +- **String parsing is synchronous.** Only the hashing path is async. `CID.parse(str)` and `CID.decode(bytes)` are synchronous; you can call them at module load time. +- **`JSON.stringify(cid)` emits a string with `{"/": "bafyrei…"}` shape** — that is the dag-json convention, *not* AT Protocol's `{"$link": "…"}`. For AT Protocol JSON, serialize CIDs manually or via [`@ipld/dag-json`](https://www.npmjs.com/package/@ipld/dag-json) with a post-process rename. Reference implementations typically hand-roll: `{ $link: cid.toString() }`. + +## Next + +- Parsing paths → `parsing.md` +- Construction (async) → `construction.md` +- Codec constants and per-codec imports → `codecs.md` +- Cross-language differences → `../shared/divergence-matrix.md` diff --git a/skills/software-development/atproto-cid/references/typescript/codecs.md b/skills/software-development/atproto-cid/references/typescript/codecs.md new file mode 100644 index 0000000..e79256f --- /dev/null +++ b/skills/software-development/atproto-cid/references/typescript/codecs.md @@ -0,0 +1,121 @@ +# TypeScript — Codecs, Hash Codes, and BLAKE3 + +TypeScript's `multiformats` package **does not ship a central registry** of codec constants. Codec codes live on the per-codec package, and you import them alongside the codec's encode/decode functions. + +## Where the numbers live + +```ts +import * as dagCbor from '@ipld/dag-cbor' // dagCbor.code === 0x71 +import * as raw from 'multiformats/codecs/raw' // raw.code === 0x55 +import * as dagJson from '@ipld/dag-json' // 0x0129 — not DASL +import * as dagPb from '@ipld/dag-pb' // 0x70 — not DASL, IPFS only +import { sha256 } from 'multiformats/hashes/sha2' // sha256.code === 0x12 +``` + +Each codec module exposes `{ code, name, encode, decode }`. That's the object you pass as the `codec:` argument to `Block.encode` / `Block.decode`, or the `.code` you feed to `CID.createV1`. The number is never typed nominally — it's just a `number` — so your DASL gate is the only thing between a stray `0x70` and acceptance. + +**Do not hand-write literals in call sites.** Use `dagCbor.code` / `raw.code`. If you find yourself writing `0x71` inline, import `dagCbor` instead. + +## DASL-acceptable values (mirror Rust and Go) + +```ts +const DAG_CBOR = 0x71 as const // dagCbor.code +const RAW = 0x55 as const // raw.code +const SHA256 = 0x12 as const // sha256.code +const BLAKE3 = 0x1e as const // see "BLAKE3" below — not in multiformats core + +const DASL_CODECS = new Set([DAG_CBOR, RAW]) +const DASL_HASHES = new Set([SHA256]) +const BDASL_HASHES = new Set([SHA256, BLAKE3]) +``` + +These live next to the `assertDasl` gate from `README.md`. Inline them there, not in every consumer. + +## Choosing the codec + +Same rules as every language: + +| Content | Codec | Why | +| ------------------------- | ----------- | --------------------------------------------------- | +| ATProto record | `dagCbor` | Structured data, DRISL-canonical CBOR. | +| MST node | `dagCbor` | Map of entries, canonical CBOR. | +| Commit | `dagCbor` | Small structured record. | +| Image, video, attachment | `raw` | Opaque bytes, no structural interpretation. | + +A record's CID must have codec `dag-cbor`. A blob's CID must have codec `raw`. A `raw` CID on a structured-record field is malformed even if the digest is correct. + +## Canonical DAG-CBOR encoder + +`@ipld/dag-cbor` is DRISL-compliant: keys sorted bytewise, integers in shortest form, no indefinite-length items, tag 42 for CIDs. **Do not substitute `cbor-x`, `cbor`, or `borc`** — they emit non-canonical output and your CIDs will not match the AT Protocol reference implementations. + +```ts +import * as dagCbor from '@ipld/dag-cbor' + +const bytes = dagCbor.encode({ b: 2, a: 1 }) +// bytes starts with a2 (map, 2 pairs) 61 61 01 61 62 02 +// keys sorted to "a" before "b" regardless of insertion order +``` + +If you need to double-check a record against ground truth, call `lexicon-garden`'s `create_record_cid`. + +## BLAKE3 (BDASL) + +`multiformats` does **not** ship a BLAKE3 hasher. To emit BDASL CIDs from TypeScript you bring a third-party BLAKE3: + +```ts +import { blake3 } from '@noble/hashes/blake3' // one option — synchronous +import { from } from 'multiformats/hashes/hasher' + +const blake3Hasher = from({ + name: 'blake3', + code: 0x1e, + encode: (input: Uint8Array) => blake3(input, { dkLen: 32 }), +}) + +// Use with Block.encode: +const block = await Block.encode({ value, codec: dagCbor, hasher: blake3Hasher }) +``` + +`from()` adapts any sync or async byte-in/byte-out function into a `MultihashHasher`. Keep BLAKE3 scoped to blob contexts: records, MST nodes, and commits are always SHA-256 even in a BDASL-enabled platform. + +Parsing a BLAKE3 CID does not require the hasher to be registered — `CID.parse` / `CID.decode` do not verify; they only read structure. The hasher is needed only when you are *computing* a CID from bytes, or when you are *verifying* one (re-hashing the content and comparing). + +## Hash code discipline + +- `sha256.code` = `0x12`, digest length 32. +- BLAKE3 = `0x1e`, digest length 32 (same size, different function). +- `0x13` (SHA-512), `0x17` (SHA3-256), `0x00` (identity) — all must be rejected by `assertDasl`. +- Do not trust a 32-byte digest alone; many hash codes produce 32 bytes. Match the `code` explicitly. + +## Multibase constants + +The multibase layer governs *string* output, not binary. Import only when you need to decode a non-default prefix: + +```ts +import { base32 } from 'multiformats/bases/base32' +import { base58btc } from 'multiformats/bases/base58' +import { base64 } from 'multiformats/bases/base64' +``` + +For DASL output you need none of these — `cid.toString()` defaults to base32lower (`b…`) for v1. For DASL input, `CID.parse(str)` recognizes the `b` prefix automatically. Import a base decoder only when you must parse a producer that sends non-default prefixes (and then reject them via the gate anyway). + +## Cross-language note + +Codec constants are only shipped as named values in **Go** (`cid.DagCBOR`, `cid.Raw`, `cid.DagPB`). In **Rust**, `atproto-dasl` ships constants; otherwise you hand-write. In **TypeScript**, each codec package provides its own `.code`. When porting a fixture, the numeric values are the language-neutral reference: + +| Codec | Hex | Import (TS) | Import (Rust) | Import (Go) | +| ------- | ------ | ----------------------------------------- | ----------------------------------- | ----------------- | +| dag-cbor| 0x71 | `@ipld/dag-cbor` → `dagCbor.code` | `atproto_dasl::cid::DAG_CBOR_CODEC` | `cid.DagCBOR` | +| raw | 0x55 | `multiformats/codecs/raw` → `raw.code` | `atproto_dasl::cid::RAW_CODEC` | `cid.Raw` | +| dag-pb | 0x70 | `@ipld/dag-pb` → `dagPb.code` *(reject)* | *(not shipped)* | `cid.DagProtobuf` | +| sha-256 | 0x12 | `multiformats/hashes/sha2` → `sha256.code`| `atproto_dasl::cid::SHA256_CODE` | `multihash.SHA2_256` | +| blake3 | 0x1e | *(third-party adapter)* | `atproto_dasl::cid::BLAKE3_CODE` | `multihash.BLAKE3` | + +See `../shared/divergence-matrix.md` §codec-constants for the full table. + +## See also + +- `../shared/spec.md` — the normative list of allowed codecs and hashes. +- `../shared/binary-layout.md` — where the codec byte sits in the 36-byte layout. +- `construction.md` — async CID creation using these codecs. +- `parsing.md` — error surfaces when inputs fall outside the allowed set. diff --git a/skills/software-development/atproto-cid/references/typescript/construction.md b/skills/software-development/atproto-cid/references/typescript/construction.md new file mode 100644 index 0000000..384464b --- /dev/null +++ b/skills/software-development/atproto-cid/references/typescript/construction.md @@ -0,0 +1,153 @@ +# TypeScript — Constructing a CID (async) + +Every construction path in this file is `async` because `sha256.digest` returns a `Promise`. Do not try to hide this — the propagation is structural, not cosmetic. + +## 1. DAG-CBOR record → CID + +Use `multiformats/block` — it wraps encode + hash + CID creation into one async call. + +```ts +import * as Block from 'multiformats/block' +import * as dagCbor from '@ipld/dag-cbor' +import { sha256 } from 'multiformats/hashes/sha2' +import { assertDasl } from './dasl-gate' + +const block = await Block.encode({ + value: { $type: 'app.bsky.actor.profile', displayName: 'Alice' }, + codec: dagCbor, + hasher: sha256, +}) + +assertDasl(block.cid) +// block.cid : CID (DaslCid after assert) +// block.bytes : Uint8Array — canonical DRISL CBOR +// block.value : the original value +``` + +The three pieces (`cid`, `bytes`, `value`) are what you want for most AT Protocol flows — you almost always need both the CID and the encoded bytes (for CAR framing, for storage, for re-emission). + +## 2. Pre-encoded DAG-CBOR bytes → CID + +If you already have DRISL-canonical bytes (received from the wire, already-encoded record), hash them directly: + +```ts +import { CID } from 'multiformats/cid' +import { sha256 } from 'multiformats/hashes/sha2' +import * as dagCbor from '@ipld/dag-cbor' +import { assertDasl } from './dasl-gate' + +const bytes: Uint8Array = /* canonical CBOR */ +const digest = await sha256.digest(bytes) +const cid = CID.createV1(dagCbor.code, digest) // 0x71 +assertDasl(cid) +``` + +`CID.createV1` is synchronous; the hashing step is the only async surface. + +## 3. Raw blob → CID + +For opaque binary content (images, video, arbitrary attachments): + +```ts +import { CID } from 'multiformats/cid' +import { sha256 } from 'multiformats/hashes/sha2' +import * as raw from 'multiformats/codecs/raw' +import { assertDasl } from './dasl-gate' + +const blob: Uint8Array = await file.arrayBuffer().then(b => new Uint8Array(b)) +const digest = await sha256.digest(blob) +const cid = CID.createV1(raw.code, digest) // 0x55 +assertDasl(cid) +``` + +Or use `Block.encode` with `codec: raw`: + +```ts +const block = await Block.encode({ value: blob, codec: raw, hasher: sha256 }) +``` + +For raw bytes, `block.value` is the bytes themselves; the round-trip is a no-op. + +## 4. Assemble manually from (codec, digest) + +Rare — most commonly when you have a pre-computed SHA-256 from a trusted source: + +```ts +import { CID } from 'multiformats/cid' +import { create as createMultihash } from 'multiformats/hashes/digest' + +const digestBytes = new Uint8Array(32) // pre-computed SHA-256 +const mh = createMultihash(0x12, digestBytes) // wrap as multihash, sync +const cid = CID.createV1(0x71, mh) +``` + +`createMultihash` is synchronous — it wraps existing bytes, does not hash. Reach for this path only when you have a strong reason not to re-hash (trusted upstream, deterministic fixture). + +## Why async propagates + +`sha256.digest` in `multiformats/hashes/sha2` delegates to the platform's SubtleCrypto: + +- In the browser: `crypto.subtle.digest('SHA-256', bytes)` — returns a `Promise`. +- In Node 20+: `crypto.subtle` mirrors the Web API, also returns a `Promise`. + +A "synchronous SHA-256" exists in Node (`crypto.createHash('sha256').update(bytes).digest()`), but using it defeats the point of `multiformats`'s platform-isomorphic design and breaks browser targets. Accept the `await`. + +A few caller-side consequences: + +- **Your "compute record CID" helper is async.** So every caller becomes async. +- **Hot paths can batch.** `Promise.all(chunks.map(c => sha256.digest(c)))` parallelises hashing across cores (in Node) or uses the event loop efficiently (in browsers). +- **Synchronous caches still work.** Once you have the `Promise`, you can memoize. Cache keys by the input bytes (or a fingerprint thereof), not by async identity. + +## JSON `$link` emission + +When emitting AT Protocol JSON, wrap the CID string yourself: + +```ts +type BlobRef = { + $type: 'blob' + ref: { $link: string } + mimeType: string + size: number +} + +const ref: BlobRef = { + $type: 'blob', + ref: { $link: cid.toString() }, // base32lower, b-prefixed + mimeType: 'image/png', + size: bytes.byteLength, +} +``` + +`cid.toString()` defaults to base32lower for v1 — exactly the DASL form. No extra configuration needed. + +## Round-trip test + +This should hold regardless of path: + +```ts +const value = { $type: 'app.bsky.feed.post', text: 'hi' } +const block = await Block.encode({ value, codec: dagCbor, hasher: sha256 }) +const redecoded = dagCbor.decode(block.bytes) +const rebuilt = await Block.encode({ value: redecoded, codec: dagCbor, hasher: sha256 }) +if (!block.cid.equals(rebuilt.cid)) throw new Error('encoder is non-canonical') +``` + +Failure means either `@ipld/dag-cbor` was replaced by a non-canonical encoder, or the value contained a non-deterministic field (e.g., a `Map` with insertion-order keys). DRISL sorts map keys regardless, so this should only fail if the CBOR library itself was swapped. + +## Common construction mistakes + +| Symptom | Cause | +| --- | --- | +| `cid.bytes is not a function` | `.bytes` is a **property**, not a method. Write `cid.bytes`, not `cid.bytes()`. | +| "Same record produces different CIDs on two machines" | You used `JSON.stringify` + `TextEncoder.encode` as a CBOR replacement. Not canonical, not CBOR. Use `dagCbor.encode`. | +| "Await is not allowed here" | Caller is sync. Make it async, or handle the `Promise` with `.then` — do not downgrade to synchronous hashing. | +| CID round-trips but doesn't match Rust / Go output | Your CBOR input had `undefined` values (DAG-CBOR disallows), or a `Map` with unordered keys, or bigints encoded as strings. Match the canonical form exactly. | +| `Block.encode` accepts a value but the CID doesn't match your server | The value's field order is fine (DRISL sorts), but any `Date` / `BigInt` / function was silently dropped or coerced. DAG-CBOR supports: string, number, boolean, null, Uint8Array, Array, Map, plain objects, and CIDs (tag 42). Nothing else. | + +## See also + +- `parsing.md` — the reverse direction. +- `codecs.md` — where `dagCbor.code` and `raw.code` come from. +- `../shared/binary-layout.md` — the 36-byte layout `CID.createV1` is producing. +- `../shared/test-vectors.md` — expected CIDs for given inputs. +- `../shared/divergence-matrix.md` — why async propagation is TypeScript-specific. diff --git a/skills/software-development/atproto-cid/references/typescript/parsing.md b/skills/software-development/atproto-cid/references/typescript/parsing.md new file mode 100644 index 0000000..36a1326 --- /dev/null +++ b/skills/software-development/atproto-cid/references/typescript/parsing.md @@ -0,0 +1,167 @@ +# TypeScript — Parsing a CID + +Always pair `CID.parse` / `CID.decode` with the `assertDasl` gate from `README.md`. The bare parse accepts non-DASL inputs (CIDv0, dag-pb, wrong hash) — the gate narrows the type to `DaslCid`. + +## From a string + +```ts +import { CID } from 'multiformats/cid' +import { assertDasl } from './dasl-gate' + +const s = 'bafyreihunttf7a3uvtzrgbnyu2rzv24w4zx7xjwqgk4x5w7n5yvq7u7aua' +const cid = CID.parse(s) // CID (may be v0 or non-DASL) +assertDasl(cid) // narrows to DaslCid +// cid.version === 1, cid.code ∈ {0x71, 0x55}, cid.multihash.code === 0x12, .size === 32 +``` + +`CID.parse` sniffs the multibase prefix: `b` → base32lower (DASL default), `z` → base58btc (CIDv0), `m` → base64, etc. For DASL CIDs you never want to accept `z` / `m` / `f` — `assertDasl` rejects them implicitly because CIDv0 fails the `version !== 1` check. + +### Parsing non-base32 multibase + +If a producer you don't control sends `zQm…` (base58btc), `CID.parse` needs an explicit base decoder: + +```ts +import { base58btc } from 'multiformats/bases/base58' +const cid = CID.parse('zQm…', base58btc) +// assertDasl will still reject — CIDv0 is not DASL. +``` + +For DASL-only ingestion, skip the explicit decoder and let `CID.parse` fail on anything other than `b…`. + +## From 36 raw bytes (CAR block frame) + +```ts +import { CID } from 'multiformats/cid' +import { assertDasl } from './dasl-gate' + +const bytes: Uint8Array = readExact(reader, 36) +const cid = CID.decode(bytes) // strict — rejects trailing bytes +assertDasl(cid) +``` + +`CID.decode` expects the CID bytes exactly. If there is trailing data, use `CID.decodeFirst` instead: + +```ts +const [cid, remainder] = CID.decodeFirst(buffer) +assertDasl(cid) +// remainder: Uint8Array — bytes after the CID +``` + +`decodeFirst` is the right tool for CAR block framing where the block-length varint tells you the CID + data total length but the CID byte count is only known after parsing. You read once, consume the CID, and the remainder is the payload. + +## From a DAG-CBOR byte string (tag 42) + +`@ipld/dag-cbor` handles CID unwrapping automatically during decode: + +```ts +import * as dagCbor from '@ipld/dag-cbor' +import { CID } from 'multiformats/cid' +import { assertDasl } from './dasl-gate' + +type MstEntry = { p: number; k: Uint8Array; v: CID; t?: CID } + +const node = dagCbor.decode<{ e: MstEntry[]; l?: CID }>(cborBytes) + +for (const entry of node.e) { + assertDasl(entry.v) + if (entry.t) assertDasl(entry.t) +} +``` + +Inside `dagCbor.decode`, any CBOR tag-42 byte string becomes a `CID` object. The identity multibase prefix is stripped for you. Hand-decoding CBOR and hunting for tag 42 manually is almost never the right move — use `dagCbor.decode`. + +If you must hand-decode (debugging a malformed payload), `multiformats` does not expose a public CID-from-tag-42 helper. Extract the 37-byte tag-42 byte string, drop the first byte (the `0x00` identity prefix), and pass the remaining 36 bytes to `CID.decode`. + +## From a JSON `$link` + +AT Protocol's JSON convention is `{"$link": "bafyrei…"}`. `multiformats` does not know about `$link` directly — `JSON.parse` gives you a string, and you feed that to `CID.parse`: + +```ts +type BlobRef = { + $type: 'blob' + ref: { $link: string } + mimeType: string + size: number +} + +const decoded: BlobRef = JSON.parse(responseBody) +const cid = CID.parse(decoded.ref.$link) +assertDasl(cid) +``` + +When *emitting* JSON, mirror the convention: `{ $link: cid.toString() }`. Never emit a bare string for a CID field — the DAG-JSON convention is `{"/": "…"}` but **AT Protocol uses `$link`**, not `/`. + +If the producer handed you a bare string instead of `{"$link": "…"}`, treat it as malformed and reject. Silent promotion breaks canonicalization downstream. + +## Streaming / incremental + +For CAR parsing, consume bytes in chunks and hand them to `CID.decodeFirst`: + +```ts +import { CID } from 'multiformats/cid' + +async function* parseBlocks(reader: AsyncIterable) { + let buf = new Uint8Array(0) + for await (const chunk of reader) { + buf = concat(buf, chunk) + // ... read varint length, slice block ... + const [cid, payload] = CID.decodeFirst(block) + yield { cid, payload } + } +} +``` + +You own the block-length framing; `CID.decodeFirst` handles the CID-vs-data split inside one framed block. No built-in "read a CID from a Node.js `Readable`" — you'd pull bytes as needed and call `decodeFirst` when enough are buffered. + +## Error handling + +`multiformats` throws plain `Error` / `TypeError` with string messages. There is no typed error enum. Practical pattern: + +```ts +try { + const cid = CID.parse(input) + assertDasl(cid) + return cid +} catch (err) { + if (err instanceof TypeError && err.message.includes('Unknown multihash code')) { + throw new BadCidError('unsupported hash function', { cause: err }) + } + throw new BadCidError(`invalid CID: ${input}`, { cause: err }) +} +``` + +Wrap in a typed error class for your caller's sake. The upstream `.message` strings are not stable across `multiformats` versions; do not string-match them in long-lived production code if you can avoid it. + +## Validation vs verification + +Parsing confirms *shape*. To confirm *content*: + +```ts +import { sha256 } from 'multiformats/hashes/sha2' +import { CID } from 'multiformats/cid' + +async function verifyCid(cid: CID, data: Uint8Array): Promise { + const hash = await sha256.digest(data) // async + const expected = CID.createV1(cid.code, hash) + return cid.equals(expected) +} +``` + +The `await` on `sha256.digest` is unavoidable — see `construction.md`. Use `cid.equals(other)`, never string or `===` comparison. + +## Common parse failures + +| Symptom | Cause | +| --- | --- | +| `TypeError: Unsupported codec: 0x70` | dag-pb CID; not DASL. | +| `TypeError: Unknown multihash code: 0x13` | SHA-512 or similar. | +| `RangeError` on `CID.decode` | Buffer is not exactly a CID — use `CID.decodeFirst` for framed inputs. | +| `Error: Unexpected end of data` | Truncated bytes; the 4-byte header is incomplete. | +| `assertDasl` throws "CID version 0 not allowed" | Input is `Qm…` CIDv0. Reject, don't try `cid.toV1()` — the lossless upgrade would give you dag-pb codec, still not DASL. | + +## See also + +- `construction.md` — building CIDs, all async. +- `codecs.md` — per-codec package imports. +- `../shared/spec.md` — rules the gate enforces. +- `../shared/divergence-matrix.md` — why the DASL gate exists only in TypeScript/Go but not Rust. diff --git a/skills/software-development/atproto-cid/scripts/validate_cid.py b/skills/software-development/atproto-cid/scripts/validate_cid.py new file mode 100755 index 0000000..a0c761b --- /dev/null +++ b/skills/software-development/atproto-cid/scripts/validate_cid.py @@ -0,0 +1,57 @@ +#!/usr/bin/env python3 +"""Validate a DASL CID string against the strict subset. + +Usage: + validate_cid.py [--bdasl] + +Exit codes: 0 valid, 1 invalid, 2 usage error. +Stdlib only — no pip dependencies. +""" +import base64 +import sys + +DAG_CBOR = 0x71 +RAW = 0x55 +SHA256 = 0x12 +BLAKE3 = 0x1e + + +def validate(cid_str: str, allow_blake3: bool = False) -> tuple[bool, str]: + if not cid_str.startswith("b"): + return False, f"not base32lower multibase: prefix must be 'b', got {cid_str[:1]!r}" + encoded = cid_str[1:].upper() + pad = (-len(encoded)) % 8 + try: + raw = base64.b32decode(encoded + "=" * pad) + except Exception as e: + return False, f"base32 decode failed: {e}" + if len(raw) != 36: + return False, f"decoded length {len(raw)} != 36" + if raw[0] != 0x01: + return False, f"version 0x{raw[0]:02x} != 0x01 (CIDv1 required)" + if raw[1] not in (DAG_CBOR, RAW): + return False, f"codec 0x{raw[1]:02x} not in {{0x71 dag-cbor, 0x55 raw}}" + hash_code = raw[2] + hash_ok = hash_code == SHA256 or (allow_blake3 and hash_code == BLAKE3) + if not hash_ok: + allowed = "0x12" + (" or 0x1e" if allow_blake3 else "") + return False, f"hash code 0x{hash_code:02x} not allowed (expected {allowed})" + if raw[3] != 0x20: + return False, f"digest length 0x{raw[3]:02x} != 0x20 (32)" + return True, f"ok: v1 codec=0x{raw[1]:02x} hash=0x{hash_code:02x} digest=32" + + +def main() -> int: + args = [a for a in sys.argv[1:] if not a.startswith("--")] + flags = {a for a in sys.argv[1:] if a.startswith("--")} + unknown = flags - {"--bdasl"} + if unknown or len(args) != 1: + print("usage: validate_cid.py [--bdasl]", file=sys.stderr) + return 2 + ok, msg = validate(args[0], allow_blake3="--bdasl" in flags) + print(msg) + return 0 if ok else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/skills/software-development/atproto-development/SKILL.md b/skills/software-development/atproto-development/SKILL.md index 53b6e71..74d8443 100644 --- a/skills/software-development/atproto-development/SKILL.md +++ b/skills/software-development/atproto-development/SKILL.md @@ -165,6 +165,13 @@ const agent = new Agent(session); To support multiple domains (e.g., custom domain + Tangled subdomain), add all redirect URIs to `redirect_uris` array and ensure `client-metadata.json` is served at each domain. +### Wire-protocol depth + +For DPoP proof minting/nonce retry, PAR, PKCE, `private_key_jwt`, session +storage, refresh-race mitigation, and scope/permission-set design, load the +`atproto-oauth` skill (ported from ngerakines/atproto-skills — spec + +TypeScript guides in its references/). + ## Tangled.org (ATProto Git Hosting) Tangled is AT Protocol-native git hosting. Repos are addressed by DID rather than username — the same repo has two equivalent forms: diff --git a/skills/software-development/atproto-oauth/SKILL.md b/skills/software-development/atproto-oauth/SKILL.md new file mode 100644 index 0000000..73e6926 --- /dev/null +++ b/skills/software-development/atproto-oauth/SKILL.md @@ -0,0 +1,104 @@ +--- +name: atproto-oauth +description: "Use for ATProto OAuth: DPoP, PAR, PKCE, client metadata, sessions." +version: 1.0.0 +author: Hermes Agent (ported from ngerakines/atproto-skills, MIT) +license: MIT +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [ATProto, OAuth, DPoP, PAR, PKCE, JWT, Sessions] +--- + +# AT Protocol OAuth + +AT Protocol OAuth is an **OAuth 2.1** profile with mandatory **PKCE (S256)**, +**PAR**, **DPoP**, and URL-based dynamic client registration via a published +**client metadata document**. No `client_secret` — confidential clients +authenticate to the token endpoint with a `private_key_jwt` assertion (ES256); +public/SPA/native clients authenticate by DPoP proof alone. + +Ported from `ngerakines/atproto-skills` (MIT). Spec + TypeScript guides kept +verbatim under `references/`; Rust/Go omitted (fetch upstream if needed). +This skill is the wire-protocol depth layer — for the quick-start SPA setup +see `atproto-development` (BrowserOAuthClient example) and cross-link here for +DPoP/PAR/session-race depth. + +## Defaults + +- **`client_id` is a URL.** It resolves to a JSON metadata document the AS + fetches on demand. URL path/host/protocol must match byte-for-byte between + registration, PAR, and authorize. +- **Every access token is DPoP-bound.** `dpop_bound_access_tokens: true` + required in client metadata; every resource request carries a fresh DPoP + proof with `ath = SHA-256(access_token)` and a per-origin `nonce`. +- **PAR is required.** Push the authorize request to + `pushed_authorization_request_endpoint`, redirect the user to + `{AS}/oauth/authorize?client_id=...&request_uri=urn:ietf:params:oauth:request_uri:...`. + Query parameters never hit the user-agent. +- **Scopes start with `atproto`.** Then layered: `transition:generic`, + `account:email?action=read`, `rpc:app.bsky.feed.*`, `include:`. +- **The session belongs to the DID.** `sub` is a DID; handles change, DIDs + don't. Persist by DID. +- **Identity verification is mandatory**: `sub` → DID doc → `#atproto_pds` → + matches the PDS you discovered → `authorization_servers[0]` → matches the AS + you talked to. Skip this = CSRF window. + +## Client types + +- **Confidential (BFF)** — server-side + signing key; the recommended pattern + for any app with a backend. `@atproto/oauth-client-node` (NodeOAuthClient). +- **Public (SPA)** — browser-only; `@atproto/oauth-client-browser` + (BrowserOAuthClient). Tokens land in the browser. +- **Public (native)** — custom-scheme redirects; NodeOAuthClient with + `token_endpoint_auth_method: none`. + +## High-frequency failure modes (TS-specific) + +- **Refresh race** — two concurrent refreshes; one invalidates the other's + refresh token → dead session. TS has `NodeRequestLock` built in; every + production BFF still needs a per-DID lock (Redis/Postgres advisory or + in-process mutex). +- **`htu` normalization** — strip query strings and fragments, elide default + ports before minting a DPoP proof. TS does this automatically; `invalid_dpop_proof` + with identical-looking URLs = suspect this. +- **`SameSite=Strict` kills the callback** — OAuth redirects are cross-origin + top-level navigations; Strict drops the cookie so the callback handler can't + find pre-flow state. Always `Lax` on session cookies. +- **Public clients have a 14-day refresh cap, not 180.** Silent until day 15 + when `invalid_grant` suddenly starts failing. + +## Validator script + +`scripts/validate_client_metadata.py ` — stdlib-only CI check of a served +`/oauth-client-metadata.json` for the invariants (missing +`dpop_bound_access_tokens`, wrong `token_endpoint_auth_signing_alg`, inline +`jwks` containing a private `d` field, `http://` redirect outside loopback…). +Exit 0 = pass, 1 = invariant failed (reasons on stdout), 2 = usage/network error. + +## References (read the relevant one before OAuth code) + +- `references/shared/spec.md` — OAuth 2.1 + AT Proto profile (read first) +- `references/shared/flows.md` — byte-level wire content per step +- `references/shared/client-metadata.md` — metadata fields, JWKS rules +- `references/shared/dpop.md` — RFC 9449 profile +- `references/shared/scopes.md` — scope grammar, permission sets +- `references/shared/sessions.md` — pre-flow state + post-flow session rules +- `references/shared/security-requirements.md` — cookies, keys, tokens, SSRF +- `references/shared/troubleshooting.md` — failure diagnosis +- `references/shared/test-vectors.md`, `divergence-matrix.md` +- `references/typescript/README.md`, `client-metadata.md`, `flows.md`, + `dpop.md`, `sessions.md` — @atproto/oauth-client-* usage + +Upstream normative sources: , +; RFCs 9449 (DPoP), 7636 (PKCE), +9126 (PAR), 7523 (JWT client auth), 8414 (server metadata), 9207 (`iss`); +OAuth 2.1 draft. + +## Related skills + +- `atproto-development` — BrowserOAuthClient quick start, dual-domain redirect + URIs, client-metadata hosting per domain, CRA/webpack @atproto pitfalls +- `atproto-identity-deep` — DID resolution/verification used in the identity + check +- `atproto-security-landscape` — OAuth phishing/abuse context diff --git a/skills/software-development/atproto-oauth/references/shared/client-metadata.md b/skills/software-development/atproto-oauth/references/shared/client-metadata.md new file mode 100644 index 0000000..f780d39 --- /dev/null +++ b/skills/software-development/atproto-oauth/references/shared/client-metadata.md @@ -0,0 +1,234 @@ +# Client metadata document + +The `client_id` **is** the URL of a JSON document describing the client. The Authorization Server fetches it at the start of every flow to dynamically register the client. This replaces the usual up-front static registration and is AT Proto's form of DCR. + +There is no `client_secret`. Authentication of confidential clients uses `private_key_jwt` with a key published in `jwks`/`jwks_uri`. + +## The URL + +Rules: + +- Scheme MUST be `https://`, with one exception: `http://localhost` for development. +- No explicit port (no `:443`). +- Path typically ends in `oauth-client-metadata.json` by convention. Any path is valid as long as it serves the JSON document with `Content-Type: application/json` and HTTP 200. +- The response body's `client_id` field MUST exactly match the URL the AS used to fetch the document. + +Examples of valid production client_ids: + +- `https://example.app/oauth-client-metadata.json` +- `https://example.app/client.json` +- `https://oauth.example.app/client` + +## Required fields + +```json +{ + "client_id": "https://example.app/oauth-client-metadata.json", + "application_type": "web", + "grant_types": ["authorization_code", "refresh_token"], + "scope": "atproto transition:generic", + "response_types": ["code"], + "redirect_uris": ["https://example.app/oauth/callback"], + "dpop_bound_access_tokens": true, + "token_endpoint_auth_method": "private_key_jwt", + "token_endpoint_auth_signing_alg": "ES256", + "jwks": { + "keys": [ + {"kty":"EC","crv":"P-256","x":"...","y":"...","kid":"...","use":"sig","alg":"ES256"} + ] + } +} +``` + +Field-by-field: + +- **`client_id`** — must equal the metadata URL exactly. String mismatch = rejected. +- **`application_type`** — `web` (default) or `native`. Drives how `redirect_uris` are validated. +- **`grant_types`** — must include `authorization_code`. Add `refresh_token` if you will refresh (almost always). +- **`response_types`** — must include `code`. +- **`scope`** — space-separated list of EVERY scope the client MAY request. Authorization requests may request a subset; they may NOT request scopes outside this list. `atproto` is mandatory. +- **`redirect_uris`** — list of all callback URIs. The authorize request's `redirect_uri` MUST match one of these exactly, character for character. +- **`dpop_bound_access_tokens`** — MUST be `true`. +- **`token_endpoint_auth_method`** — for confidential clients, `private_key_jwt`. For public clients, `none`. +- **`token_endpoint_auth_signing_alg`** — `ES256` currently. Never `none`. +- **`jwks`** or **`jwks_uri`** — confidential clients only. Exactly one. Contains PUBLIC keys. See §Keys. + +## Optional but recommended + +- `client_name` — human-readable name. Shown on consent screen for **trusted** clients only. +- `client_uri` — the app's home page. +- `logo_uri`, `tos_uri`, `policy_uri` — all HTTPS only. Shown on consent screen for trusted clients. +- `contacts` — list of email addresses for security contact. + +Untrusted clients will not have these fields displayed on the consent screen; ASes whitelist trusted clients explicitly. + +## Public clients + +Public clients omit `token_endpoint_auth_method` (or set it to `"none"`) and do NOT include `jwks`/`jwks_uri`: + +```json +{ + "client_id": "https://example.app/oauth-client-metadata.json", + "application_type": "web", + "grant_types": ["authorization_code", "refresh_token"], + "scope": "atproto transition:generic", + "response_types": ["code"], + "redirect_uris": ["https://example.app/oauth/callback"], + "dpop_bound_access_tokens": true, + "token_endpoint_auth_method": "none" +} +``` + +Public client tradeoffs: + +- No client assertion JWT on PAR/token/refresh. +- Refresh tokens capped at 14 days. +- Overall session capped at 14 days. +- Cannot be extended by key rotation (no keys to rotate). + +## Native clients + +`application_type: "native"` changes redirect URI rules: + +- Custom-scheme URIs allowed: `com.example.app:/callback`. The scheme MUST be the reverse-domain form of the `client_id` hostname, followed by `:/`. Not `://`. +- HTTPS URIs allowed as "Apple Universal Links" style. +- `http://127.0.0.1:*/` and `http://[::1]:*/` allowed in development for loopback flow. + +Note: the AT Proto profile does not define a loopback redirect mechanism for non-localhost `client_id`s. Loopback is specifically tied to the `http://localhost` development `client_id` exception. + +## localhost development client + +For development only: + +- `client_id = http://localhost` (or `http://localhost?scope=atproto+transition:generic&redirect_uri=http://127.0.0.1:8080/callback`). +- The AS generates virtual metadata: `application_type: native`, `token_endpoint_auth_method: none`, `dpop_bound_access_tokens: true`, `grant_types: [authorization_code, refresh_token]`, `response_types: [code]`. +- `scope` and `redirect_uri` can be passed as query parameters on the `client_id`. +- Default redirects are `http://127.0.0.1/` and `http://[::1]/` if not supplied. + +Worked example — a CLI running on port 8080 that wants the `atproto` and `transition:generic` scopes: + +``` +client_id = http://localhost?scope=atproto%20transition%3Ageneric&redirect_uri=http%3A%2F%2F127.0.0.1%3A8080%2Fcallback +``` + +The AS returns virtual metadata built from those query parameters, so there is nothing to host and nothing to serve. The exact same `client_id` string is passed to every OAuth call in the flow (PAR, token, refresh) — changing it mid-flow invalidates the grant. For anything exposed to real users, ship a real HTTPS `client_id` with a hosted metadata document; the localhost form is iteration-only. + +## Keys (`jwks` vs `jwks_uri`) + +Confidential clients MUST publish the PUBLIC half of each signing key. Keys sign the `private_key_jwt` client assertion sent to the token endpoint. + +Choose one: + +- **Inline `jwks`** — simplest. Rotate by redeploying the metadata document. +- **`jwks_uri`** — points to a separate endpoint that returns `{"keys":[…]}`. Rotate independently of metadata. + +JWK shape (P-256 example): + +```json +{ + "kty": "EC", + "crv": "P-256", + "x": "", + "y": "", + "kid": "", + "use": "sig", + "alg": "ES256" +} +``` + +**Never** include the `d` field (private part). If you do, you've leaked your signing key and must rotate immediately and revoke. + +## Key algorithms + +- `ES256` (P-256) — baseline. Every AS must accept it; every client should mint assertions with it. +- `ES384` (P-384) and `ES256K` (secp256k1) — optional; support varies. Only use if you know your AS supports it (check `token_endpoint_auth_signing_alg_values_supported` in AS metadata). +- `RS256` — not part of the AT Proto profile for client assertions. Stick to EC. + +## Key rotation (confidential clients) + +1. Generate a new keypair; append the public half to `jwks`/`jwks_uri` alongside the old key. +2. Start signing new assertions with the new key (pick it by `kid`). +3. Wait for all in-flight sessions using the old key to expire or migrate. +4. Remove the old key from `jwks`. + +The AS binds active sessions to the `kid` used at session start. Removing a `kid` prematurely will cause `invalid_client` on refresh for sessions bound to that key. Plan for a rotation period ≥ your longest refresh lifetime. + +## Caching + +The AS may cache the metadata document. Clients should emit HTTP caching headers (`Cache-Control`, `ETag`) but cannot rely on the AS honouring them. Consequences: + +- A rotated JWK may not propagate for the AS's cache TTL. Keep the OLD key in place after publishing the new one. +- A removed JWK is a **revocation signal** only after the cache TTL elapses. Leaked keys need rotation + explicit session revocation. + +No AS-side minimum or maximum TTL is currently specified by the profile. + +## Validation + +Before serving the document from your own service, run `scripts/validate_client_metadata.py` (stdlib-only). It checks: + +- HTTPS enforcement (or `http://localhost`). +- `dpop_bound_access_tokens: true`. +- Required grant types and response types. +- Auth method matches client type. +- JWKs are `kty=EC`, `crv=P-256` (or declared curve), and do NOT contain `d`. +- `scope` contains `atproto`. +- `redirect_uris` match `application_type` rules. + +Also verify live: fetch your own `client_id` over HTTPS and check that `client_id` in the body equals the URL you fetched from. + +## Worked examples + +Confidential, web, backend-for-frontend on `https://myapp.example.com`: + +```json +{ + "client_id": "https://myapp.example.com/oauth-client-metadata.json", + "client_name": "MyApp", + "client_uri": "https://myapp.example.com", + "logo_uri": "https://myapp.example.com/logo.png", + "tos_uri": "https://myapp.example.com/tos", + "policy_uri": "https://myapp.example.com/privacy", + "application_type": "web", + "grant_types": ["authorization_code", "refresh_token"], + "response_types": ["code"], + "redirect_uris": ["https://myapp.example.com/oauth/callback"], + "scope": "atproto transition:generic", + "dpop_bound_access_tokens": true, + "token_endpoint_auth_method": "private_key_jwt", + "token_endpoint_auth_signing_alg": "ES256", + "jwks_uri": "https://myapp.example.com/.well-known/jwks.json" +} +``` + +Public SPA served from `https://app.example.com`: + +```json +{ + "client_id": "https://app.example.com/oauth-client-metadata.json", + "client_name": "Example SPA", + "client_uri": "https://app.example.com", + "application_type": "web", + "grant_types": ["authorization_code", "refresh_token"], + "response_types": ["code"], + "redirect_uris": ["https://app.example.com/oauth/callback"], + "scope": "atproto transition:generic", + "dpop_bound_access_tokens": true, + "token_endpoint_auth_method": "none" +} +``` + +Native mobile, hostname `app.example.com` (reverse = `com.example.app`): + +```json +{ + "client_id": "https://app.example.com/oauth-client-metadata.json", + "client_name": "Example Mobile", + "application_type": "native", + "grant_types": ["authorization_code", "refresh_token"], + "response_types": ["code"], + "redirect_uris": ["com.example.app:/callback"], + "scope": "atproto transition:generic", + "dpop_bound_access_tokens": true, + "token_endpoint_auth_method": "none" +} +``` diff --git a/skills/software-development/atproto-oauth/references/shared/divergence-matrix.md b/skills/software-development/atproto-oauth/references/shared/divergence-matrix.md new file mode 100644 index 0000000..ee987b0 --- /dev/null +++ b/skills/software-development/atproto-oauth/references/shared/divergence-matrix.md @@ -0,0 +1,170 @@ +# Cross-Language Divergence Matrix (AT Proto OAuth) + +Language-neutral. Captures the real behavioural differences between the Rust (`atproto-oauth`), TypeScript (`@atproto/oauth-client-node` / `@atproto/oauth-client-browser`), and Go (`indigo/atproto/auth/oauth`) OAuth stacks that anyone porting code, operating cross-stack, or auditing interop needs to know about. + +Every per-language file (`rust/*.md`, `typescript/*.md`, `go/*.md`) links back here instead of restating the matrix. + +## Library map + +| Layer | Rust (`atproto-oauth`) | TypeScript (`@atproto/oauth-client-*`) | Go (`indigo/atproto/auth/oauth`) | +| -------------------------- | ---------------------------------- | --------------------------------------------- | -------------------------------------- | +| Client shape | Functions (`oauth_init`, `oauth_complete`, `oauth_refresh`) + `OAuthClient` struct as config bag | `NodeOAuthClient`, `BrowserOAuthClient` classes w/ `.authorize` / `.callback` / `.restore` | `ClientApp` struct w/ `StartAuthFlow` / `ProcessCallback` / `ResumeSession` | +| Confidential client | Supported (ES256 / ES384 / ES256K) | Supported (ES256 only in practice via `JoseKey`) | Supported (ES256 only — hard-coded) | +| Public / SPA client | Supported (caller wires) | **First-class** (`@atproto/oauth-client-browser`) | **Not supported** — BFF only | +| Native / desktop client | Supported (caller wires) | `@atproto/oauth-client-node` with `token_endpoint_auth_method: none` | **Not supported** — BFF only | +| DPoP minting (auth endpts) | `auth_dpop(key, method, url)` → `(token, header, claims)` | Hidden inside flow methods | `NewAuthDPoP(method, url, nonce, priv)` | +| DPoP minting (resource) | `request_dpop(key, method, url, access_token)` | Hidden inside `session.fetchHandler` | Hidden inside `ClientSession`'s transport | +| DPoP nonce retry | `DpopRetry` middleware (1-try budget) | Inside `fetchHandler` (1-try budget) | Inside flow methods + `ClientSession` (1-try budget) | +| Server-side DPoP validate | `validate_dpop_jwt` + `DpopValidationConfig` | **Not shipped** — roll with `jose` | **Not shipped** — roll with `jwx` | +| Pre-flow state storage | `OAuthRequestStorage` trait + `LruOAuthRequestStorage` | `StateStore` interface (user-impl) | `ClientAuthStore` interface + `MemStore` | +| Session storage | **Caller-owned** — no built-in abstraction | `SessionStore` interface (user-impl) | `ClientAuthStore.SaveSession` (same interface as state) | +| Refresh lock | **Caller-owned** | `NodeRequestLock` injected into `NodeOAuthClient` | **Not provided** — roll your own | +| Handle resolution | Via `atproto-identity` (separate crate) | Injected `handleResolver` (URL or function) | Via `atproto/identity` (plumbed in) | +| Client metadata builder | Caller writes the JSON directly (see `rust/client-metadata.md`) | `client.clientMetadata` echo-back | `cfg.ClientMetadata()` (map) | +| JWKS publisher | Caller iterates `jwk::generate` over keys | `client.jwks` (library strips private halves) | `cfg.PublicJWKS()` (map) | + +The shape of the trade-off: + +- **TypeScript ships the most out-of-the-box** — it has the only first-class SPA client, a built-in refresh lock abstraction, and a fetch handler that makes DPoP entirely invisible. +- **Rust ships the most primitives** — three function calls and a bag of types; you wire the flows yourself but get the only production-grade `validate_dpop_jwt` in any of the three stacks. +- **Go ships the least** — BFF-only, no refresh lock, no SPA, no server-side DPoP validator, and ES256 is hard-coded. But the `ClientApp` + `ClientAuthStore` + `ClientSession` trio covers the happy path cleanly. + +--- + +## §client-metadata — metadata document + JWKS + +| Aspect | Rust | TypeScript | Go | +| ------------------------------ | -------------------------------------- | ----------------------------------------- | -------------------------------------------- | +| Metadata representation | Caller writes Serde struct | `client.clientMetadata` (echo of constructor input) | `cfg.ClientMetadata()` → `map[string]any` | +| JWKS representation | `jwk::generate(&KeyData)` → `WrappedJsonWebKey` (one key at a time) | `client.jwks` → `{ keys: JsonWebKey[] }` | `cfg.PublicJWKS()` → `map[string]any` | +| Private component stripping | **Caller must `to_public(&key)` first** before `jwk::generate` — easy to forget | Automatic (library strips) | Automatic (library strips) | +| Multi-key rotation | Pass `Vec` in config; all published | Pass `keyset: JoseKey[]` — all published | **Single-key API** — merge manually at serve time | +| Supported alg values | ES256, ES384, ES256K | ES256 (practical — via `JoseKey`) | ES256 (hard-coded) | +| Loopback dev shortcut | Caller writes the metadata as usual | Supported via `clientId = 'http://127.0.0.1/...'` | **First-class**: `oauth.NewLocalhostConfig(...)` | + +**Practical bug one**: **Rust requires `to_public` before `jwk::generate`**. If you skip it, the private component `d` is serialized into your published JWKS, leaking the signing key. TS and Go guard against this by having the library strip private halves internally. + +**Practical bug two**: **Go is single-key**. Key rotation requires two `ClientConfig` instances + a manual JSON merge at the `/jwks.json` handler. TS and Rust natively accept multi-key sets. + +**Practical bug three**: `ClientMetadata()` in Go returns a mutable `map[string]any`. Overwriting a signed field (`redirect_uris`, `client_id`) after config-time silently breaks the signed PAR assertion. TS's `client.clientMetadata` is a live object that mirrors the constructor input — same risk exists if you mutate it. + +--- + +## §flows — the three verbs + +| Aspect | Rust | TypeScript | Go | +| --------------------------------------- | -------------------------------------- | ----------------------------------------- | -------------------------------------------- | +| Begin-flow method name | `oauth_init(&client, &state, issuer)` | `client.authorize(handle, options)` | `app.StartAuthFlow(ctx, identifier)` | +| Callback method name | `oauth_complete(&client, &request, params)` | `client.callback(params)` | `app.ProcessCallback(ctx, params)` | +| Refresh method name | `oauth_refresh(&client, &session)` | `client.restore(did)` (auto-refreshes) | `app.ResumeSession(ctx, did, sid)` (auto-refreshes) | +| Identity resolution inside begin | Caller passes resolved `issuer` | Library resolves `handle` internally | Library resolves `identifier` internally | +| `iss` parameter verification | Caller compares against pre-flow state | Library verifies | Library verifies | +| DID / `sub` / `aud` cross-check | Caller implements | Library verifies | Library verifies | +| Pre-flow state cleanup (single-use) | Caller calls `storage.delete_oauth_request_by_state(state)` after `oauth_complete` | Library calls `stateStore.del(key)` | Library calls `store.DeleteAuthRequestInfo(ctx, state)` | +| Refresh token rotation on refresh | Returned in `TokenResponse` — caller persists | Library persists via `sessionStore.set` | Library persists via `store.SaveSession` | +| Error representation | Typed: `TokenHttpRequestFailed`, `JsonParsingFailed`, `IssuerMismatch`, … | Typed: `OAuthResponseError`, `OAuthCallbackError`, `TokenRefreshError` | Plain `error` — inspect body/status | + +**Practical bug one**: **Rust is the only stack where the caller threads identity resolution in manually.** The other two resolve handles internally. Port a Rust BFF to TS or Go → you can drop your DID-resolution step; port TS or Go to Rust → you must add one. + +**Practical bug two**: **Rust leaves the session abstraction to the caller**. TS and Go both define the session row shape the library writes and reads. In Rust, each downstream application ships its own `SessionCookie` / `SessionRow` type — cross-project sharing is painful. + +**Practical bug three**: **Go error inspection requires body/status match**. TS and Rust let you `instanceof`/`match` on error types. Port code → Go defaults to string-matching until you factor out a custom error type. + +--- + +## §dpop — proof minting and nonce handling + +| Aspect | Rust (`atproto_oauth::dpop`) | TypeScript | Go (`indigo/atproto/auth/oauth`) | +| --------------------------------------- | -------------------------------------- | ----------------------------------------- | -------------------------------------------- | +| Mint-for-auth helper | `auth_dpop(key, method, url)` | N/A — hidden inside flow methods | `NewAuthDPoP(method, url, nonce, priv)` | +| Mint-for-resource helper | `request_dpop(key, method, url, access_token)` | N/A — hidden inside `session.fetchHandler` | **None** — use `ClientSession` or hand-build | +| Nonce retry middleware | `DpopRetry` wraps `reqwest::Client` | Inside `fetchHandler` | Inside `ClientSession`'s transport | +| Check response body for `use_dpop_nonce`| Configurable (`check_response_body: bool`) | Always on | Always on | +| Per-origin nonce cache | **Caller-managed** — `DpopRetry` is request-scoped | Library-managed, in-memory per session | Library-managed, persisted to `ClientSessionData.DPoPNonce` | +| `htu` auto-normalization | **No** — caller strips query/fragment | Yes — library normalizes | **No** — caller strips query/fragment | +| Server-side `validate_dpop_jwt` | Full implementation + `DpopValidationConfig` | **Not shipped** | **Not shipped** | +| `jti` replay protection | **Caller implements** (not built in) | N/A (client-only) | N/A (client-only) | +| Alg / curve mismatch guard | None — `auth_dpop` hard-codes `ES256` in the header even for P-384 keys | Enforced at key-construction time (JoseKey pins alg) | Enforced at config-time (`SetClientSecret` requires P-256) | + +**Practical bug one**: **Rust's `auth_dpop` hard-codes `ES256` in the JWT header regardless of key type.** Pass a P-384 key and the proof won't verify. Use `dpop::mint(...)` with a hand-built `Header` for non-P-256 keys. This is a latent bug documented in `rust/dpop.md`. + +**Practical bug two**: **`htu` normalization is caller-side in Rust and Go.** TS strips query/fragment and default ports automatically. Port TS code → Rust/Go and `invalid_dpop_proof` from the PDS becomes the #1 failure mode until you add the normalization step. + +**Practical bug three**: **Go has no resource-DPoP helper.** `NewAuthDPoP` omits `ath`. Either route everything through `ClientSession` (which handles resource DPoP automatically) or hand-build the proof with `golang-jwt/jwt/v5`. + +**Practical bug four**: **Only Rust ships `validate_dpop_jwt`.** If you're writing an AS or resource server in Go/TS, you're on your own for DPoP validation. + +--- + +## §sessions — storage and refresh-race + +| Aspect | Rust | TypeScript | Go | +| --------------------------------------- | -------------------------------------- | ----------------------------------------- | -------------------------------------------- | +| Pre-flow state interface | `OAuthRequestStorage` trait (4 methods) | `StateStore` interface (3 methods) | `ClientAuthStore` (6 methods, combined) | +| Session row interface | **None** — caller defines | `SessionStore` interface (3 methods) | Same `ClientAuthStore` (session + request methods) | +| Built-in dev/in-memory impl | `LruOAuthRequestStorage` (pre-flow only) | Caller implements | `MemStore` | +| TTL handling for pre-flow state | `clear_expired_oauth_requests()` method — **caller runs on cron** | Caller implements (Redis TTL is idiomatic) | Caller implements (cron on PG; TTL on Redis) | +| Refresh lock | **Caller-implemented** (Mutex/row lock/single-flight) | `NodeRequestLock` — library invokes around refresh | **Not invoked by library** — caller wraps `ResumeSession` | +| Distributed lock (multi-process) | Caller integrates (Redis/PG advisory) | `NodeRequestLock` user-provided | Caller wraps `ResumeSession` (PG advisory / Redlock) | +| Session-cookie abstraction | **None** — caller builds + encrypts | **None** — caller builds + encrypts | **None** — caller builds + encrypts | +| DPoP key life | Immortal for session; caller persists | Immortal for session; library persists | Immortal for session; library persists in `ClientSessionData.DPoPKey` | +| `DPoPNonce` persistence | Caller-managed | In-memory per session | Persisted in `ClientSessionData.DPoPNonce` | + +**Practical bug one**: **Rust and Go leave the refresh lock entirely to the caller.** TS accepts a `NodeRequestLock` at client-construction time and invokes it around every refresh — zero-thought correctness as long as you wire one. Rust and Go will happily send two concurrent refresh calls for the same DID; whichever writes to storage last has the winning refresh token, the loser's refresh token is dead. See each language's `sessions.md` §refresh race. + +**Practical bug two**: **Go's `ClientAuthStore` conflates state and session storage** — same interface, six methods. TS cleanly separates them, which makes it easier to put state in Redis (TTL) and sessions in Postgres (long-lived). In Go you either implement both against the same backend or compose two stores into one impl. + +**Practical bug three**: **Rust has no session interface at all.** The `OAuthRequestStorage` trait covers only pre-flow state. Post-flow session storage is entirely application-code; this is deliberate (your session is your concern) but it means no two Rust codebases structure sessions the same way. + +--- + +## §client-metadata vs §scopes interplay + +The scope declared in `clientMetadata.scope` is the **upper bound**. The per-flow `scope` parameter in `authorize` / `StartAuthFlow` / `oauth_init` selects a subset. + +- Rust: caller writes both. No validation that per-flow is a subset of metadata. +- TS: library enforces the subset check at `authorize`. +- Go: `NewPublicConfig(..., scopes)` sets metadata scope; `StartAuthFlow` uses metadata scope directly — **no per-flow override**. To request a different scope, build a different `ClientApp`. + +**Practical bug**: Go's immutable per-`ClientApp` scope means permission-set UIs (ask the user which scopes to grant) are awkward. Either stand up a `ClientApp` per scope combination, or patch the scope into the PAR request manually. TS and Rust are more flexible here. + +--- + +## Porting checklist + +Moving code from **TypeScript → Rust**: +- Wire a `handleResolver` equivalent using `atproto-identity`; library won't resolve for you. +- Bring your own refresh lock; `NodeRequestLock` has no direct analogue. +- Define your session row type; no built-in `SessionStore`. +- Strip private components from keys before `jwk::generate` (easy miss). +- `htu` normalization: strip query + fragment before `auth_dpop`. + +Moving code from **TypeScript → Go**: +- Same BFF assumptions transfer; no SPA support in Go. +- Implement `ClientAuthStore` against your DB; no split between state and session interfaces. +- Wrap `ResumeSession` in a per-DID mutex / advisory lock; library doesn't do it. +- Scope is fixed per `ClientApp`; stand up more than one if you need variable scopes. + +Moving code from **Rust → TypeScript**: +- Delete your ad-hoc session type and use `SessionStore`. +- Remove your DID/AS resolution — inject `handleResolver`. +- Remove your refresh-lock code — wire `requestLock` and let the library invoke it. +- Key alg is practically pinned to ES256 via `JoseKey` — check your deploy isn't relying on ES384/ES256K. + +Moving code from **Go → Rust**: +- Unpack `ClientApp` into the three function calls (`oauth_init` / `oauth_complete` / `oauth_refresh`). +- Bring your own identity resolver — `atproto-identity` crate. +- Support multiple alg values if your deploy uses anything besides ES256. +- Bring your own `validate_dpop_jwt` consumer if your server verifies proofs; Rust has the full validator. + +--- + +## See also + +- `spec.md` — normative rules for all entities and flows. +- `flows.md` — byte-level wire content. +- `dpop.md` — RFC 9449 profile. +- `sessions.md` — language-neutral session rules. +- `security-requirements.md` — cookie/key/token hardening checklist. +- `test-vectors.md` — shared test vectors for interop verification. +- `rust/README.md`, `typescript/README.md`, `go/README.md` — per-language entry points. diff --git a/skills/software-development/atproto-oauth/references/shared/dpop.md b/skills/software-development/atproto-oauth/references/shared/dpop.md new file mode 100644 index 0000000..a94bdbd --- /dev/null +++ b/skills/software-development/atproto-oauth/references/shared/dpop.md @@ -0,0 +1,162 @@ +# DPoP — Demonstrating Proof-of-Possession + +DPoP (RFC 9449) binds an access token to a specific keypair. Every request carrying the token also carries a fresh JWT proving possession of the key. The access token is useless without the key; stealing the token from a log doesn't let an attacker use it. + +AT Proto OAuth mandates DPoP on **every request** — to the AS (PAR, token, refresh, revoke) and to the PDS/RS (every XRPC call). It also mandates **server-issued nonces** that generic DPoP treats as optional. + +## The keypair + +One **DPoP keypair per session**. Generate it at the start of the flow (before PAR) and keep it for the life of the session. Losing the key ends the session. + +- Curve: P-256 (ES256). Required baseline. +- Optional: P-384 (ES384), secp256k1 (ES256K). Only if the AS advertises them and you're sure the whole stack supports them. +- Private key stays server-side in the BFF pattern, never in the browser. +- Never reuse a DPoP key across sessions or accounts. + +## The proof JWT + +Every request attaches one freshly-minted proof. Proofs are one-shot: never reuse across two requests, even two identical ones. + +Header: + +```json +{ + "typ": "dpop+jwt", + "alg": "ES256", + "jwk": { + "kty": "EC", + "crv": "P-256", + "x": "...", + "y": "..." + } +} +``` + +- `typ` MUST be exactly `dpop+jwt`. Not `JWT`. +- `jwk` is the **public** half of the DPoP key. No `d`. +- No `kid`, no `x5c`, no other header fields typically. + +Claims: + +| Claim | Required? | Value | +|---|---|---| +| `jti` | yes | Unique random string per proof. ULIDs, UUIDs, or 16+ random bytes hex all work. | +| `htm` | yes | HTTP method, uppercase: `GET`, `POST`, `PUT`, `DELETE`. | +| `htu` | yes | Full URL of the target. **No query string.** No fragment. | +| `iat` | yes | Issued-at, seconds since epoch. | +| `exp` | recommended | Short expiry, e.g. `iat + 30`. Servers typically cap at ~60s anyway. | +| `nonce` | conditionally | Server-issued nonce from `DPoP-Nonce` header. Required after first round-trip. | +| `ath` | only on resource requests | `base64url(SHA-256(access_token))`, no padding. Required when the request carries `Authorization: DPoP `. | + +Signed with the private DPoP key. Algorithm in header MUST match the key. + +## The nonce dance + +ASes and PDSes issue nonces via the `DPoP-Nonce` response header (case-insensitive). The rules: + +1. **First request** to a server: mint a proof WITHOUT `nonce` claim. +2. Server returns HTTP **400** or **401** with: + - JSON body: `{"error":"use_dpop_nonce", ...}` + - Header: `DPoP-Nonce: ` +3. **Retry** with a NEW proof that includes `nonce: ""` in claims. Fresh `jti`, fresh `iat`. Do not change the DPoP keypair. +4. Succeed (or get a different error — different failure mode). +5. Every subsequent response from that server includes a potentially rotated `DPoP-Nonce`. **Always copy the latest one** into your per-origin nonce store before the next request. + +Nonces rotate at least every 5 minutes (server rule). If you sit idle and come back, your nonce may be stale. Servers SHOULD accept recently-stale nonces, but don't rely on it. Treat a fresh `use_dpop_nonce` mid-session the same as the first: extract, retry once. + +**Per-origin nonces are separate.** Track: + +- One nonce for the AS (for PAR, token exchange, refresh, revoke). +- One nonce for the PDS (for all XRPC calls). + +A nonce minted for the AS is invalid at the PDS and vice versa. Mixing them up produces `invalid_dpop_proof`. + +**Retry budget = 1.** Two `use_dpop_nonce`s in a row on the same request is a bug — probably clock skew, wrong `htm`/`htu`, or nonce-origin confusion. Don't loop. + +## `ath` on resource requests + +When you send `Authorization: DPoP `, you MUST add: + +``` +ath = base64url(SHA-256(access_token)) # no padding +``` + +to the DPoP proof claims. This binds the proof to the specific access token. + +Omitting `ath` → `invalid_dpop_proof`. Including `ath` when there's no `Authorization` header (like on PAR or the token request) is not part of the profile — omit it there. + +## `htu` normalization + +`htu` is the full request URL **without query string or fragment**. A few consequences: + +- `GET /xrpc/com.atproto.repo.getRecord?repo=did:plc:x&collection=...&rkey=...` → `htu = "https://pds.example.com/xrpc/com.atproto.repo.getRecord"`. +- Strip the query deterministically. Server-side libraries also strip, but they may normalize case/trailing-slash differently. +- Scheme and host must match exactly: `https://pds.example.com/xrpc/…`, not `https://PDS.EXAMPLE.COM/`. +- Default ports are omitted (`:443` stripped), but some servers are strict — mirror what's in the metadata document's `token_endpoint` verbatim. + +If `htu` doesn't match the server's reconstructed URL, you get `invalid_dpop_proof`. Rare but real: a reverse proxy that rewrites the URL makes the server see a different `htu` than the client sent. Log both when debugging. + +## Clock skew + +`iat` is checked against server clock. Skew > 30 seconds → rejected as `invalid_dpop_proof` (or `Invalid timestamp`). Keep your servers' clocks in sync (NTP). + +Generous defaults seen in production validators: `iat` within `[now - 60, now + 30]` with the future-clamp optional. If you're minting from a mobile client with a user-settable clock, consider a TOFU probe to detect skew before the first real request. + +## Validating incoming DPoP (servers) + +If you're implementing the server side (lexicon-garden / PDS / entryway) rather than the client, the validator MUST: + +1. Parse JWT header. Check `typ == "dpop+jwt"`, `alg` in allowed set, `jwk` is a public EC key. +2. Verify the signature with the embedded JWK. +3. Check claims: `jti` (non-empty, optionally rate-limit recent values), `htm` matches the request method, `htu` matches the request URL (post-normalization), `iat` within clock-skew window, `exp` if present. +4. If the request carries `Authorization: DPoP `, check `ath == base64url(SHA-256(token))`. +5. Check `nonce` against the set of currently-valid server nonces for the origin. +6. If `nonce` absent or stale: issue a new nonce in the response header and respond with `use_dpop_nonce`. +7. If replay protection needed: track recent `jti`s for at least the proof's `exp` window. + +The Rust `atproto-oauth` crate's `validate_dpop_jwt` function ships a reference config (`DpopValidationConfig`) with all the knobs. + +## Common failure modes (cheatsheet) + +| Symptom | Likely cause | +|---|---| +| `use_dpop_nonce` on first request | expected — retry with the provided nonce | +| `use_dpop_nonce` twice in a row | clock skew, or nonce copied from wrong origin, or `htu` wrong | +| `invalid_dpop_proof` on resource request | missing `ath`, wrong `htm`/`htu`, or proof reused | +| `invalid_dpop_proof` immediately after refresh | forgot to rotate nonce, or reused old proof | +| `invalid_dpop_proof` on token endpoint | `htu` = authorization endpoint instead of token endpoint | +| "typ" error from server | sent `"JWT"` instead of `"dpop+jwt"` | +| Signature-verification failure | wrong algorithm vs key type, or public JWK in header doesn't match signing key | + +## Diagram + +``` +Client Server + │ │ + │ POST /token (DPoP: proof_1, no nonce) │ + ├───────────────────────────────────────────────►│ + │ │ + │ 400 use_dpop_nonce │ + │ DPoP-Nonce: N1 │ + │◄───────────────────────────────────────────────┤ + │ │ + │ POST /token (DPoP: proof_2 with nonce=N1) │ + ├───────────────────────────────────────────────►│ + │ │ + │ 200 { access_token, refresh_token, … } │ + │ DPoP-Nonce: N2 │ + │◄───────────────────────────────────────────────┤ + │ │ + │ (store N2 as new AS nonce) │ + │ │ + │ GET /xrpc/… (Authorization: DPoP , │ + │ DPoP: proof_3 │ + │ claims: { htu, htm='GET', │ + │ ath=SHA256(at), │ + │ nonce=N_pds_1 or −})│ + ├───────────────────────────────────────────────►│ +``` + +## Implementation note: retry middleware + +The Rust `atproto-oauth` crate ships a `DpopRetry` struct implementing the `reqwest-middleware` `Chainer` trait. It transparently handles the nonce dance once per request. TypeScript and Go libraries generally build the same thing in-house but may not externalize it as a reusable middleware — see the divergence matrix. diff --git a/skills/software-development/atproto-oauth/references/shared/flows.md b/skills/software-development/atproto-oauth/references/shared/flows.md new file mode 100644 index 0000000..2863b31 --- /dev/null +++ b/skills/software-development/atproto-oauth/references/shared/flows.md @@ -0,0 +1,312 @@ +# Flows — byte-level + +Every AT Proto OAuth flow is a sequence of HTTP requests with very specific headers, parameters, and retry rules. This file describes the wire content of each step. Language files translate these into library calls. + +All bodies are `application/x-www-form-urlencoded` unless noted. All responses are `application/json`. + +## Flow A — discovery + +**A1. Resolve identity** (if starting from handle or DID — skip if starting from PDS/AS URL). + +Given `handle`: resolve to DID via DNS `_atproto.{handle}` TXT or `https://{handle}/.well-known/atproto-did`. Bidirectionally verify the DID document's `alsoKnownAs` includes `at://{handle}`. + +Given `did`: resolve via `did:plc` directory (`https://plc.directory/{did}`) or `did:web` (`https://{domain}/.well-known/did.json`). + +Extract `service[id="#atproto_pds"].serviceEndpoint` from the DID document as the PDS URL. + +**A2. Fetch protected resource metadata.** + +``` +GET {PDS}/.well-known/oauth-protected-resource +Accept: application/json +``` + +Response (subset): +```json +{ + "resource": "https://pds.example.com", + "authorization_servers": ["https://pds.example.com"] +} +``` + +Assertions: +- `resource` equals the PDS URL. +- `authorization_servers` has exactly one entry. + +**A3. Fetch authorization server metadata.** + +``` +GET {AS}/.well-known/oauth-authorization-server +Accept: application/json +``` + +Response (subset, abridged): +```json +{ + "issuer": "https://pds.example.com", + "authorization_endpoint": "https://pds.example.com/oauth/authorize", + "token_endpoint": "https://pds.example.com/oauth/token", + "pushed_authorization_request_endpoint": "https://pds.example.com/oauth/par", + "require_pushed_authorization_requests": true, + "authorization_response_iss_parameter_supported": true, + "client_id_metadata_document_supported": true, + "dpop_signing_alg_values_supported": ["ES256"], + "code_challenge_methods_supported": ["S256"], + "grant_types_supported": ["authorization_code","refresh_token"], + "token_endpoint_auth_methods_supported": ["none","private_key_jwt"], + "token_endpoint_auth_signing_alg_values_supported": ["ES256"], + "scopes_supported": ["atproto","transition:generic"] +} +``` + +Assertions the client MUST enforce before proceeding: +- `issuer` matches the origin of the fetch URL. +- All the booleans above are `true`. +- Every `*_supported` list contains the corresponding required value. + +## Flow B — PAR (Pushed Authorization Request) + +**B1. Generate per-session state** (server-side; never in a client-visible cookie): + +- `state` — ≥16 chars, URL-safe random. Single use. +- `nonce` — ≥16 chars, URL-safe random. Stored for client-side verification only; not sent to the AS. +- PKCE pair: `verifier` = 43–128 chars from `[A-Z a-z 0-9 - . _ ~]`; `challenge = base64url(SHA-256(verifier))` with no padding. +- DPoP keypair: P-256 EC, private only on server. +- For confidential clients: remember which `kid` you'll sign the client assertion with. + +**B2. Mint client assertion** (confidential clients only). JWT with: + +- Header: `{"alg":"ES256","typ":"JWT","kid":""}` +- Claims: `{"iss":"","sub":"","aud":"","iat":,"exp":,"jti":""}` +- Signed with the private key whose public half is in your `jwks` under that `kid`. + +**B3. Mint DPoP proof** for the PAR request. JWT with: + +- Header: `{"alg":"ES256","typ":"dpop+jwt","jwk":}` +- Claims: `{"jti":"","htm":"POST","htu":"","iat":}` — NO `nonce` on the first try. +- Do NOT include a query string in `htu`. + +**B4. POST PAR**: + +``` +POST {pushed_authorization_request_endpoint} +Content-Type: application/x-www-form-urlencoded +DPoP: + +response_type=code +&client_id={client_id} +&redirect_uri={redirect_uri} +&scope={scope} # space-separated, URL-encoded +&state={state} +&code_challenge={challenge} +&code_challenge_method=S256 +&login_hint={handle or did} # optional but recommended +&client_assertion_type=urn%3Aietf%3Aparams%3Aoauth%3Aclient-assertion-type%3Ajwt-bearer # confidential only +&client_assertion={client assertion JWT} # confidential only +``` + +**B5. Handle first-try `use_dpop_nonce`:** + +First response will almost always be: + +``` +HTTP/1.1 400 Bad Request +DPoP-Nonce: +Content-Type: application/json + +{"error":"use_dpop_nonce","error_description":"Authorization server requires nonce in DPoP proof"} +``` + +Extract `DPoP-Nonce`. Mint a NEW DPoP proof with the same `jti`-new, `iat`-now, plus `nonce: ""` claim. Retry the POST. Allow one retry per request. + +**B6. Success response:** + +``` +HTTP/1.1 201 Created +DPoP-Nonce: +Content-Type: application/json + +{"request_uri":"urn:ietf:params:oauth:request_uri:...", "expires_in": 299} +``` + +Persist the server nonce for this AS origin for the next request. Persist the whole `OAuthRequest` state (state, nonce, verifier, DPoP private key, issuer, AS metadata URL, created_at, expires_at ~10min) keyed by `state`. + +## Flow C — user approval + +**C1. Build authorize URL**. The URL contains ONLY `client_id` and `request_uri`. No other parameters. + +``` +GET {authorization_endpoint}?client_id={client_id}&request_uri={request_uri} +``` + +**C2. Redirect the user.** Server sends a 302 (or `window.location = …` in a SPA). The user is now on the AS. + +The user authenticates with their PDS and approves (or denies) the scope request. + +**C3. Receive callback.** The AS redirects the user's browser back to `redirect_uri`: + +``` +GET {redirect_uri}?code=...&state=...&iss=... +``` + +Checks the handler MUST perform in order: + +1. Load the stored `OAuthRequest` by `state`. If missing, reject (replay or forged). +2. Delete the `OAuthRequest` row — single-use to prevent replay. +3. Verify `iss == stored.issuer`. If mismatch, reject. +4. Proceed to token exchange. + +If the AS sends an error instead: `redirect_uri?error=access_denied&error_description=...&state=...`. Surface to user; don't exchange. + +## Flow D — token exchange + +**D1. Mint DPoP proof for token endpoint:** + +- Header: `{"alg":"ES256","typ":"dpop+jwt","jwk":}` +- Claims: `{"jti":"","htm":"POST","htu":"","iat":,"nonce":""}` + +**D2. Mint fresh client assertion** (confidential clients). Same shape as B2 but new `iat`, `jti`, `exp`. + +**D3. POST token:** + +``` +POST {token_endpoint} +Content-Type: application/x-www-form-urlencoded +DPoP: + +grant_type=authorization_code +&code={code} +&redirect_uri={redirect_uri} # MUST be identical to PAR +&client_id={client_id} +&code_verifier={verifier} # PKCE +&client_assertion_type=urn%3Aietf%3Aparams%3Aoauth%3Aclient-assertion-type%3Ajwt-bearer # confidential +&client_assertion={client assertion JWT} # confidential +``` + +Handle `use_dpop_nonce` retries as in B5. Update stored AS nonce from `DPoP-Nonce` response header on both success and retry. + +**D4. Token response:** + +``` +HTTP/1.1 200 OK +DPoP-Nonce: +Content-Type: application/json + +{ + "access_token": "...", + "token_type": "DPoP", + "expires_in": 3600, + "refresh_token": "...", + "scope": "atproto transition:generic", + "sub": "did:plc:..." +} +``` + +Assertions: + +- `scope` contains `atproto`. If not, reject the session. +- `sub` is a valid DID. + +**D5. Identity verification** (mandatory): + +- Resolve `sub` DID → DID document. +- Extract PDS from DID document service record. +- Fetch `{PDS}/.well-known/oauth-protected-resource`; verify its `authorization_servers[0]` equals the AS `issuer` you just completed a flow with. +- If user supplied a handle as `login_hint`, verify the DID document's `alsoKnownAs` includes `at://{handle}`. + +If any check fails, discard the tokens and fail the flow. + +## Flow E — resource requests (to the PDS) + +Every call to the PDS needs its OWN DPoP proof with `ath`: + +- Header: `{"alg":"ES256","typ":"dpop+jwt","jwk":}` +- Claims: `{"jti":"","htm":"","htu":"","iat":,"nonce":"","ath":""}` + +Request: + +``` +GET {PDS}/xrpc/com.atproto.repo.getRecord?... +Authorization: DPoP +DPoP: +``` + +The PDS maintains its own DPoP nonce, separate from the AS. First request to a new PDS origin will get `use_dpop_nonce` (HTTP 401 this time, often), with a `DPoP-Nonce` header. Retry with `nonce` claim. Thereafter, use the latest nonce you've seen from that origin. + +Nonces can and do rotate. Always copy the latest `DPoP-Nonce` from every response before the next request. Maintain at minimum: `AS origin → nonce` and `PDS origin → nonce` in session state. + +## Flow F — refresh + +**F1. When to refresh.** Access-token expiry is ~minutes. Refresh when within 5 minutes of expiry. Never refresh sooner — you burn refresh tokens needlessly. + +**F2. Lock.** Only one refresh per session at a time. If two concurrent refresh calls both succeed, only one of the new refresh tokens is the "current" one; the other is dead on arrival. Use a per-session mutex (server-side) or a single-flight pattern. + +**F3. Mint DPoP for token endpoint** (as D1, with AS nonce). + +**F4. POST refresh:** + +``` +POST {token_endpoint} +Content-Type: application/x-www-form-urlencoded +DPoP: + +grant_type=refresh_token +&refresh_token={refresh_token} +&client_id={client_id} +&client_assertion_type=urn%3Aietf%3Aparams%3Aoauth%3Aclient-assertion-type%3Ajwt-bearer # confidential +&client_assertion={client assertion JWT} # confidential +``` + +Note: confidential clients MUST use the same `kid`/algorithm as the session was opened with. A rotated-out key will get `invalid_client` on refresh. + +**F5. Response** has a new `access_token` and a new `refresh_token`. Replace both atomically in your session store. Also update the AS DPoP nonce from `DPoP-Nonce` response header. + +**F6. On failure.** `invalid_grant` = session is dead; user must re-auth. `use_dpop_nonce` = retry once. Everything else = log and surface to user; session likely dead. + +## Flow G — logout / revocation + +Two levels: + +**G1. Local logout** — delete the server-side session and expire the cookie. The tokens still exist on the AS until they expire or are revoked explicitly. + +**G2. Server-side revocation** (optional but polite). Some ASes support RFC 7009 `/oauth/revoke`: + +``` +POST {revocation_endpoint} +Content-Type: application/x-www-form-urlencoded +DPoP: + +token={refresh_token} +&token_type_hint=refresh_token +&client_id={client_id} +&client_assertion_type=...&client_assertion=... # confidential +``` + +The AT Proto profile does not require ASes to implement `/oauth/revoke`; treat 404/405 as benign. If available, prefer revoking the refresh token — access tokens are short-lived. + +Always do G1 regardless of whether G2 succeeds. If G2 fails, delete the local session anyway. + +## Timing and retry budget + +| Action | Retries | Timeout per attempt | +|---|---|---| +| Discovery (metadata fetches) | 0 — fail loud | 10s | +| PAR / token `use_dpop_nonce` | 1 | 10s | +| Token exchange `invalid_grant` | 0 — session dead | — | +| Refresh | 1 on `use_dpop_nonce`; 0 on everything else | 10s | +| Resource request `use_dpop_nonce` | 1 | request-dependent | + +Every retry regenerates the DPoP proof with fresh `jti` and updated `nonce`. Never reuse a DPoP proof. + +## State to persist per session + +Minimum durable state, keyed by some session ID: + +- `did` — account subject (from token `sub`). +- `access_token`, `refresh_token`, `access_token_expires_at`. +- `dpop_private_key` — the single keypair used for the whole session. +- `issuer` — AS issuer URL. Needed for refresh. +- `as_dpop_nonce`, `pds_dpop_nonce` — most recent nonces for each origin. +- `scope` — granted scopes, for feature-gating. + +Do NOT put any of this in a client-readable cookie. In the BFF pattern, the session cookie carries only an opaque ID; the above lives in the server's DB encrypted at rest. diff --git a/skills/software-development/atproto-oauth/references/shared/scopes.md b/skills/software-development/atproto-oauth/references/shared/scopes.md new file mode 100644 index 0000000..6440cb3 --- /dev/null +++ b/skills/software-development/atproto-oauth/references/shared/scopes.md @@ -0,0 +1,203 @@ +# Scopes — permissions and parsing + +Scopes encode what an access token is allowed to do. AT Proto OAuth scopes are richer than generic OAuth: they include positional parameters, query parameters, and references to external "permission set" lexicons. + +Authoritative source: . + +## Scope string grammar + +``` +scope = resource [ ":" positional ] [ "?" params ] +resource = "atproto" | "transition" | "account" | "identity" | "blob" + | "repo" | "rpc" | "include" + | "openid" | "profile" | "email" # OIDC, rarely relevant +positional = URL-encoded bare value +params = param ("&" param)* +param = key "=" URL-encoded-value +``` + +Multiple scopes are joined with single spaces: `"atproto transition:generic repo:app.bsky.feed.post?action=create"`. + +## Required baseline + +- **`atproto`** — declares the atproto OAuth profile. Mandatory on every session. No parameters. +- The authorize request's `scope` MUST be a subset of the client metadata `scope` field. You can't request scopes you didn't declare. + +## Transitional scopes (legacy) + +Intended as a migration path from App Passwords. Treat as "everything the old App Password could do, minus account management". + +| Scope | Grants | +|---|---| +| `transition:generic` | Broad PDS permissions: write any record type, upload blobs, read/write preferences, most XRPC endpoints, service auth. Excludes account management and `chat.bsky.*`. | +| `transition:chat.bsky` | Adds `chat.bsky.*` Lexicons + service auth. Requires `transition:generic`. | +| `transition:email` | Read account email address and confirmation status via `com.atproto.server.getSession`. | + +Goal: deprecate these over time. For new apps, prefer granular scopes; transitional scopes mean "I haven't thought about permissions". + +## Granular scopes + +### `repo:*` — record writes + +``` +repo:[?action=create&action=update&action=delete] +repo:* +``` + +- `collection` is an NSID (e.g. `app.bsky.feed.post`) or `*` (all collections). +- `action` is optional; if absent, all three actions (create/update/delete) are allowed. Repeat to grant multiple actions. + +Examples: + +- `repo:app.bsky.feed.post?action=create&action=update` — post + edit posts, but not delete +- `repo:*?action=create` — write any new record in any collection +- `repo:com.example.widget` — full create/update/delete on widgets + +Partial wildcards like `repo:app.bsky.*` are NOT supported. + +### `rpc:*` — XRPC calls + +``` +rpc:?aud= +rpc:?aud=* +rpc:*?aud= +``` + +- `lxm` is the XRPC method NSID (e.g. `app.bsky.feed.searchPosts`) or `*`. +- `aud` is the target service DID, usually a service endpoint like `did:web:api.bsky.app#bsky_appview`, or `*`. +- At least one of `lxm` or `aud` MUST be a concrete value — both wildcarded is forbidden. + +Example: `rpc:app.bsky.feed.searchPosts?aud=did:web:api.bsky.app%23bsky_appview` — call the AppView's search endpoint, nothing else. + +### `blob:*` — media upload mime filters + +``` +blob:[&accept=...] +blob?accept=image/*&accept=video/mp4 +``` + +- `mime-pattern` is `*/*` (all), `type/*` (image/video/audio wildcard), or `type/subtype` (exact). No `*/subtype`. +- `accept` query param can also be used; either positional or `accept=` works, but not both for the same scope. + +### `account:*` — account hosting admin + +``` +account:[?action=read|manage] +account:email?action=manage +account:repo?action=read +``` + +- `attr` is `email`, `repo`, or `status`. +- `action` defaults to `read`; `manage` implies read. + +### `identity:*` — handle management + +``` +identity:handle +identity:* +``` + +Currently the only attribute is `handle` (or `*`). `identity:handle` is what a handle-changer UI would request. + +### `include:*` — permission set reference + +``` +include:[?aud=] +include:com.example.authBasicFeatures?aud=did:web:api.example.com%23svc_appview +``` + +`include` points at a **permission set** lexicon published elsewhere. Authorization server dereferences it (with caching) and expands it into the granular scopes it contains. + +See `permission-sets.md` on atproto.com for the lexicon shape. + +## Percent-encoding + +Scope values are percent-encoded within the scope string where needed. The canonical percent-encoding hazards: + +- `#` in an `aud` service reference MUST be `%23`: `aud=did:web:api.example.com%23svc_appview`. +- `&` and `=` within values likewise. +- Spaces between scopes stay as literal spaces (scope strings aren't URL-form-encoded; in a form-encoded body the whole `scope=...` value is URL-encoded normally). + +## Subsumption (which scope grants which) + +Not formally defined as a lattice in the spec, but in practice: + +- A more-wildcarded scope subsumes a more-specific one with the same resource. `repo:*` subsumes `repo:app.bsky.feed.post`. +- Query params narrow the grant. `repo:foo?action=create` does NOT subsume `repo:foo?action=delete`. +- If you have both `repo:*` and `repo:foo?action=create`, the specific one is redundant — the Rust crate's `parse_multiple_reduced` strips such redundancies. + +Clients should request the **narrowest** scope they actually need. Over-requesting erodes trust and may be rejected by cautious users. + +## Scope in the client metadata vs authorize request + +Client metadata declares the **possible** scope: + +```json +"scope": "atproto transition:generic repo:app.bsky.feed.post?action=create rpc:app.bsky.feed.getAuthorFeed?aud=*" +``` + +The authorize request's `scope` parameter is a **subset** of that. You can request less but not more. + +Many clients just duplicate the metadata `scope` into the authorize request. Fine, but consider step-up: request only `atproto` on first login, then start a fresh flow requesting more scopes when the user tries to use a feature that needs them. + +## Scope in the token response + +The AT Proto profile requires the AS to echo granted scopes in the token response: + +```json +"scope": "atproto transition:generic" +``` + +The client MUST: + +1. Verify `atproto` is present. If not, reject the session. +2. Use the echoed scope as the source of truth for what the session can do. The user may have ticked off items on the consent screen — you get less than you asked for. + +## Permission sets + +A permission set is a lexicon with type `permission-set` that bundles granular permissions with user-facing labels: + +```json +{ + "lexicon": 1, + "id": "com.example.authBasicFeatures", + "defs": { + "main": { + "type": "permission-set", + "title": "Basic App Functionality", + "detail": "Creation of posts and interactions", + "permissions": [ + { "type": "permission", "resource": "repo", + "collection": ["app.example.post"] }, + { "type": "permission", "resource": "rpc", + "inheritAud": true, + "lxm": ["app.example.getFeed", "app.example.getProfile"] } + ] + } + } +} +``` + +Caching: + +- AS fetches and caches; may serve stale up to 24h, must refresh within 90 days for new sessions. +- Permission set updates propagate to new sessions automatically. Existing tokens keep the scope they were minted with. + +Namespace authority: + +- A permission set may reference resources in its own NSID group or deeper (sub-domains). +- Cannot reference sibling groups or parents. `com.example.auth.basic` may grant `com.example.widget.*`, but NOT `com.other.thing.*` or `com.*`. + +## Implementation sketch (scope parsing) + +The Rust `atproto-oauth` crate's `scopes` module is the most complete reference: an `enum Scope` with variants per resource, `parse(&str) -> Scope`, `parse_multiple(&str) -> Vec`, `parse_multiple_reduced(&str)` (removes subsumed), `serialize_multiple(&[Scope]) -> String` (lexicographic sort). See `rust/client-metadata.md` for usage. + +TypeScript and Go libraries typically do not expose a rich scope parser — they treat scope strings as opaque and rely on the AS for semantic decisions. If you need to reason about scopes programmatically (e.g. feature-gating in the UI), port the Rust parser or request only the scopes you understand. + +## Common mistakes + +- **Missing `atproto`** — everyone's first bug. `scope=transition:generic` without `atproto` → AS rejects or client-side check rejects the token. +- **Partial wildcard** — `repo:app.bsky.*` is not valid. Use `repo:*` or list specific NSIDs. +- **Raw `#` in `aud`** — must be `%23`. URL libraries may do this for you inside a query-string builder but not in a scope literal. +- **Asking for more than you declared** — AS rejects with `invalid_scope`. The authorize `scope` MUST be a subset of client-metadata `scope`. +- **Assuming you got what you asked for** — the AS may grant less. Use the token response's `scope` field as truth. diff --git a/skills/software-development/atproto-oauth/references/shared/security-requirements.md b/skills/software-development/atproto-oauth/references/shared/security-requirements.md new file mode 100644 index 0000000..88ddffa --- /dev/null +++ b/skills/software-development/atproto-oauth/references/shared/security-requirements.md @@ -0,0 +1,188 @@ +# Security requirements + +AT Proto OAuth pushes more of the security burden onto the client than generic OAuth 2.1 does, because client metadata documents are world-fetchable and identity resolution involves user-supplied URLs. This file catalogues the non-negotiable hardening. + +## Hardened HTTP client (SSRF protection) + +Every HTTP call to a **user-derived URL** MUST go through an SSRF-hardened client. This includes: + +- Fetching `/.well-known/did.json` (did:web). +- Fetching `/.well-known/atproto-did` (handle → DID). +- Fetching `/.well-known/oauth-protected-resource` (PDS metadata). +- Fetching `/.well-known/oauth-authorization-server` (AS metadata). +- Fetching client metadata (if you're acting as an AS). +- Fetching permission set lexicons by NSID (likewise, AS side). +- Any DNS-resolved hostname that traces back to a user-controlled domain. + +Block on resolution: + +- **IPv4 private**: `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`. +- **IPv4 loopback**: `127.0.0.0/8`. +- **IPv4 link-local**: `169.254.0.0/16`. +- **IPv4 multicast / reserved**: `224.0.0.0/4`, `240.0.0.0/4`. +- **IPv6 unique-local**: `fc00::/7`. +- **IPv6 link-local**: `fe80::/10`. +- **IPv6 loopback**: `::1`. +- **DNS `.local` / `.localhost` / `.test` TLDs**. + +Also cap: + +- **Body size**: 256 KB or 1 MB depending on expected content. A 100 MB DID document is an attack. +- **Time**: total request budget 10s, connect 3s, idle 5s. +- **Redirects**: follow at most 2 redirects, never cross-scheme (`https://` must not redirect to `http://`). +- **TLS**: enforce valid certs. No self-signed unless explicitly enabled for dev. + +Trusted upstreams (PLC directory `https://plc.directory`, known entryways) may use a separate less-restricted client. Keep the hardened one as the default; opt into the relaxed one explicitly. + +Libraries: Rust `reqwest` + custom resolver; TypeScript `node-fetch` wrapped with an IP-check in its `agent`; Go `net/http` with a custom `Dialer.Control` that rejects disallowed IPs after DNS resolution. + +## Key management + +### DPoP keys + +- **Per session.** One keypair, full session lifetime. +- **Private key never in browser in BFF pattern.** Generate on server. +- **Private key in browser for SPA** — unavoidable; store non-exportable WebCrypto key where possible. +- **Rotation on refresh is NOT a thing.** The DPoP key is fixed for the session. Rotating keys ends the session. +- **Wipe on logout.** Zero the key material in memory (`zeroize` in Rust, etc.) when deleting the session. + +### Client assertion keys (confidential only) + +- **Private keys stored server-side** in an HSM / KMS in production. At minimum, encrypted at rest. +- **Never committed to source control.** Generated at deploy time or loaded from a secrets manager. +- **Multiple keys** supported via `kid` in JWK and client assertion header. Rotate by adding new, draining old sessions, then removing. +- **Rotation cadence:** quarterly is reasonable. Forced rotation on suspected compromise. +- **No `d` field leaks.** When publishing `jwks` from Rust's `atproto-oauth-service-token`, ensure the serializer strips the private half. + +### Cookie secret + +- Per-deployment, rotating. If you rotate, plan for a window where both old and new secrets decrypt (dual-secret scheme). +- Length ≥ 32 bytes of entropy for AEAD. +- Never log the secret or cookie plaintext. + +## Token security + +Access tokens and refresh tokens are credentials. Treat as such. + +- **In transit**: HTTPS only. DPoP provides sender-constraining but not confidentiality. +- **At rest**: encrypted. Session DB column uses `pgcrypto`, age-encrypted secret, or KMS-envelope encryption. +- **Never in client-readable cookies.** Always HttpOnly. +- **Never in logs.** Scrub request/response bodies passing through the token endpoint. +- **Lifetime caps:** + - Access token: ≤30 min. Many PDSes set 15 min. + - Refresh (public client): 14 days absolute. + - Refresh (confidential client): 180 days per token, but session rotates tokens every refresh. +- **DPoP-bound.** A leaked token alone is useless without the DPoP private key. But if the key leaks too, the session is compromised. + +## State / CSRF + +- **`state` parameter**: random, ≥16 chars, single-use. + - AS rejects duplicates (profile requirement). + - Client also verifies on callback. +- **PKCE verifier**: random, 43–128 chars from `[A-Z a-z 0-9 - . _ ~]`. Never log. +- **`nonce` (client-minted)**: optional but commonly used as a secondary CSRF check tied to the browser's session cookie. Smokesignal stores `{state → nonce}` and matches on callback. + +## Rate limiting + +Client-side: + +- **Authorize endpoint**: limit attempts per IP + per session cookie; backoff on repeated failures. +- **Token exchange**: same. +- **Refresh**: serialize per-session; client-side retries are almost always wrong. + +Server-side (if running an AS): + +- **PAR endpoint**: rate-limit per `client_id` and per client IP; PAR is expensive (validates DPoP, validates client assertion). +- **Token endpoint**: rate-limit per `client_id`. + +## HTTP headers / response hygiene + +On every HTML response from your web app: + +``` +Content-Security-Policy: + default-src 'self'; + script-src 'self'; + style-src 'self' 'unsafe-inline'; + img-src 'self' https: data:; + connect-src 'self' https://plc.directory; + frame-ancestors 'none'; + base-uri 'self'; + form-action 'self'; +Strict-Transport-Security: max-age=63072000; includeSubDomains; preload +X-Content-Type-Options: nosniff +X-Frame-Options: DENY +Referrer-Policy: strict-origin-when-cross-origin +Permissions-Policy: geolocation=(), microphone=(), camera=() +``` + +Adjust `connect-src` for any AT Proto endpoints your SPA needs to hit directly (BFFs won't need this). + +## Identity verification + +Every session MUST: + +1. On callback, verify `iss` query parameter against stored AS issuer. +2. On token response, verify `scope` contains `atproto`. +3. On token response, verify `sub` is a DID. +4. **Resolve `sub` → DID doc → PDS → AS metadata**, and confirm `authorization_servers[0]` matches the AS you used. If mismatch: session-invalid. +5. If login started from a handle, verify the DID document's `alsoKnownAs` includes `at://{handle}`. +6. Periodically re-resolve (every day or session-renewal): the DID document can change. Handle can be revoked (`handle.invalid`). + +Failures here mean the user could be authenticated but for a different account than they think. Production-grade bug. + +## Client metadata authenticity + +Client metadata is served from a URL the client controls. The AS treats the metadata as ground truth for the client's declared capabilities. + +- **Don't host client metadata on a shared domain** where another user might write to `/oauth-client-metadata.json`. Hostname-share = client-id-squat. +- **Rotate keys proactively** if someone else might have gained write access to your metadata. +- **`jwks_uri` if you can**, at a path only you control, with strict auth/access-control on the file server. + +From the AS side: cache client metadata but plan for invalidation. A rotated-out key must eventually stop working. Within the cache window, old keys remain valid — factor that into your rotation schedule. + +## DPoP / JWT validation + +Server-side: validate DPoP proofs with the Rust `atproto-oauth` `validate_dpop_jwt` as a reference: + +- `typ == "dpop+jwt"`, `alg` in allowed list. +- `jwk` is a public EC key with matching curve. +- Signature verifies. +- `htm` matches request method. +- `htu` matches request URL (normalized: no query, no fragment, canonical host/scheme). +- `iat` within `[now - 60, now + 30]` (skew). +- `exp` if present, not in the past. +- `ath` if request carries `Authorization: DPoP `. +- `nonce` matches current or recently-stale server nonce. +- `jti` not recently seen (replay prevention window ≥ proof's max lifetime). + +Client-side: you don't validate DPoP proofs since you minted them. You do validate server JWTs if you're checking service auth tokens, but that's a separate skill. + +## Bootstrapping (don't skip these) + +- **Generate signing keys at deploy time**, not on first boot. Ensure new deployments don't mint ephemeral keys that disappear on restart. +- **Publish `jwks` BEFORE first use.** An AS fetching metadata mid-flow that doesn't contain your current signing key's public half will reject the client assertion. +- **Test with `http://localhost` first**, then with your real client_id on a staging origin, before going to production. +- **Validate the client metadata JSON** on every CI run with `scripts/validate_client_metadata.py`. + +## Incident response + +- **Key compromise:** rotate signing key in `jwks` immediately; remove old key after longest refresh TTL. Optionally revoke active sessions (if you've implemented session storage you can iterate). +- **Token leak:** revoke the refresh token via `/oauth/revoke` if supported; delete the session; notify user. +- **Handle takeover:** not an OAuth concern directly, but sessions bound to a DID whose handle was transferred are still valid for that DID — the DID is the identity, not the handle. UI should re-verify handle per login. +- **AS compromise:** outside your trust boundary. Best you can do: notice `invalid_token` errors across a whole PDS, alarm on it. + +## Minimum security checklist (before shipping) + +- [ ] Hardened HTTP client for all user-supplied URL fetches (SSRF blocked). +- [ ] Client metadata validated with `scripts/validate_client_metadata.py` in CI. +- [ ] Signing keys stored in a secrets manager, not source. +- [ ] Session cookie is HttpOnly + Secure + SameSite=Lax. +- [ ] Tokens never in client-readable state. +- [ ] DPoP key generated per session, stored server-side (BFF) or non-exportable WebCrypto (SPA). +- [ ] `state`, `nonce`, PKCE verifier are random ≥16 chars and single-use. +- [ ] Refresh is serialized per session. +- [ ] Identity verification happens after token exchange (DID → PDS → AS match). +- [ ] CSP, HSTS, X-Frame-Options, Referrer-Policy set. +- [ ] `/oauth/callback` requires the stored `state` to exist, then deletes it before exchange. +- [ ] `/oauth/logout` clears cookies and deletes the session row; best-effort revokes refresh token. diff --git a/skills/software-development/atproto-oauth/references/shared/sessions.md b/skills/software-development/atproto-oauth/references/shared/sessions.md new file mode 100644 index 0000000..8972938 --- /dev/null +++ b/skills/software-development/atproto-oauth/references/shared/sessions.md @@ -0,0 +1,198 @@ +# Sessions, state, and BFF patterns + +Once the flow completes, you own a session. This file covers state lifecycle, storage, refresh scheduling, revocation, and the three architectural patterns: **backend-for-frontend (BFF)**, **pure browser SPA**, **native**. + +## Session state (what must persist) + +Per authenticated user, the server-side session stores: + +| Field | Why | +|---|---| +| `did` | Account identifier. Never changes. Use as primary key. | +| `handle` (cache) | Display only. Re-verify periodically. | +| `access_token` | Bearer token for PDS. Expires in minutes. | +| `refresh_token` | Swap for a new access token. Single-use. | +| `access_token_expires_at` | Absolute timestamp. Drives refresh scheduling. | +| `dpop_private_key` | Bound to tokens for life of session. Lose it = lose session. | +| `issuer` | AS issuer URL. Needed for refresh. | +| `pds_url` | Cached from DID document; re-fetch if expired. | +| `scope` | Echoed from token response. Gate features. | +| `as_dpop_nonce` | Latest nonce from AS. Update on every response. | +| `pds_dpop_nonce` | Latest nonce from PDS. Update on every response. | +| `created_at`, `last_active_at` | Session hygiene + UX. | + +**Never put any of this in a cookie the browser can read.** Put an opaque session ID in an HttpOnly cookie; the row lives in your server DB. + +## Pre-flow state (OAuth request) + +During the PAR → callback window, you also persist per-attempt state keyed by `state`: + +| Field | Why | +|---|---| +| `state` | Primary key. Single-use. | +| `nonce` | Local CSRF / session cookie correlate. | +| `pkce_verifier` | Needed for token exchange. | +| `dpop_private_key` | Needed for PAR + token exchange. | +| `issuer` | Match against `iss` in callback. | +| `authorization_server` | AS metadata URL for discovery. | +| `return_to` | Optional post-login redirect target. | +| `created_at`, `expires_at` | TTL ~10 minutes. | + +Delete this row as soon as you start the token exchange — single-use prevents replay. + +Clean up expired rows periodically. Both Rust and TS libraries ship a `clear_expired` hook; run it on a cron. + +## The BFF (backend-for-frontend) pattern + +Recommended for any app with a server. The browser never touches tokens. + +``` +┌────────────┐ ┌───────────┐ ┌──────────┐ +│ Browser │ │ BFF │ │ AS/PDS │ +│ │ │ (yours) │ │ │ +└────┬───────┘ └─────┬─────┘ └─────┬────┘ + │ │ │ + │ click "Sign in" │ │ + ├─────────────────────────►│ │ + │ │ resolve, PAR │ + │ ├─────────────────────────►│ + │ │◄─────────────────────────┤ + │ 302 to AS authorize │ │ + │◄─────────────────────────┤ │ + │ │ │ + │ (user approves on AS) │ + │ │ + │ GET /oauth/callback?code=…&state=…&iss=… │ + ├─────────────────────────►│ │ + │ │ token exchange │ + │ ├─────────────────────────►│ + │ │◄─────────────────────────┤ + │ │ store session, set cookie│ + │ 302 to app + cookie │ │ + │◄─────────────────────────┤ │ + │ │ │ + │ GET /api/feed (cookie) │ │ + ├─────────────────────────►│ │ + │ │ xrpc/getAuthorFeed │ + │ │ (DPoP + bearer) │ + │ ├─────────────────────────►│ +``` + +Properties: + +- **Client type: confidential.** Backend signs client assertions. +- **Tokens: server-only.** Browser gets a session cookie; nothing more. +- **Session cookie:** opaque ID, `HttpOnly; Secure; SameSite=Lax; Path=/`. `Lax` not `Strict` so that the OAuth callback redirect from the AS can carry the cookie. +- **Refresh:** server-side background or lazy-on-request. Transparent to the browser. +- **API calls from browser:** `/api/*` on the BFF; the BFF translates to PDS XRPC calls with DPoP + bearer. + +The BFF is the simplest pattern to reason about and the most robust to XSS and token theft. + +## Pure browser SPA (public client) + +No backend. Tokens live in the browser. Use when: + +- You have no server at all. +- You're shipping a dev tool or demo where "no backend" is a feature. + +Tradeoffs: + +- **Client type: public.** No client assertion, no signing key. +- **Session ≤ 14 days.** Refresh tokens also cap at 14 days. +- **Token storage:** IndexedDB (not localStorage — access from workers, more private). The `@atproto/oauth-client-browser` library persists sessions to IndexedDB automatically. +- **XSS exposure:** any XSS on your origin gets the tokens. Lock down CSP. +- **Multi-tab sync:** `@atproto/oauth-client-browser` emits events (`'updated'`, `'deleted'`) for sibling tabs to react to refresh and logout. +- **DPoP key in browser:** generated via WebCrypto `crypto.subtle.generateKey`. Non-exportable preferred, but then you can't store it across reload — the library typically stores exportable keys in IndexedDB. + +SPAs are legitimate but harder to get right. If you have a backend, use BFF. + +## Native mobile + +- **Client type: public** in almost all cases. A confidential client needs a server-resident signing key; distributing one inside the app bundle defeats the purpose. +- **Redirect URI: custom scheme** (`com.example.app:/callback`) or Apple Universal Link. +- **Token storage:** OS keystore — Keychain (iOS), Keystore (Android). +- **Consent-screen trust:** native clients can't prove ownership of the host the way a web client can; ASes treat them as public/untrusted and will not display `client_name`/`logo_uri` to users. + +Native flows are public-client flows with different redirect plumbing. Session lifetimes are the same as web SPA. + +## Hybrid (BFF-assisted mobile) + +Common pragma: mobile app authenticates the user through the BFF instead of directly with the AS. The BFF acts as the confidential client; the mobile app has a session with the BFF, not with the AS. Tokens never leave the BFF. + +Pros: longer sessions, less token handling in the client. + +Cons: now your BFF is the AS from the app's perspective. Requires careful scope design on the BFF-to-AS hop. Not covered by the AT Proto profile directly — this is "your BFF is a regular API server". + +## Refresh scheduling + +Refresh when the access token is within ~5 minutes of expiry. Two styles: + +- **Lazy**: check `access_token_expires_at` at request time; if close, refresh inline. Simple. Adds latency to user-facing requests. +- **Proactive background**: a timer wakes up before expiry and refreshes. Keeps request latency flat but needs per-session scheduling state. + +The Smoke Signal reference BFF in Rust uses lazy refresh in middleware — check on every inbound request, refresh before dispatching. See `rust/sessions.md`. + +## Concurrency: the refresh race + +Two concurrent requests both see an expired access token, both attempt to refresh, both succeed. Now the server has issued two new refresh tokens but only the last one is valid. The other session copy is dead. + +Mitigations: + +- **Per-session mutex** around refresh. One in-flight refresh at a time. +- **Single-flight pattern** with result broadcast. +- Database row-level lock on the session row during refresh. + +Concurrency footgun count: 1. This one bug has eaten weeks of engineer time across the AT Proto ecosystem. + +## Logout + +Two layers (see `flows.md` §G): + +1. **Local logout** — delete session row, expire session cookie. Always do this. +2. **Server-side revocation** — POST `/oauth/revoke` with the refresh token. Optional; the AT Proto profile does not mandate that ASes implement it. Attempt and ignore 404/405. + +Revoking the refresh token invalidates the whole session at the AS. Revoking only the access token leaves the refresh token usable — don't bother. + +## Session cookie details + +``` +Set-Cookie: session=; + Domain=example.com; + Path=/; + HttpOnly; + Secure; + SameSite=Lax; + Max-Age=31536000 # 1 year or whatever your policy is +``` + +- **`SameSite=Lax` is mandatory — `Strict` breaks OAuth.** The callback from the authorization server is a cross-origin top-level navigation. Browsers drop `SameSite=Strict` cookies on that hop, so your callback handler sees no session cookie, cannot correlate the PKCE verifier, and fails with "unknown state." `Lax` allows the cookie to ride a top-level navigation while still blocking iframe and XHR cross-origin sends. Set `Lax` explicitly — the browser default varies. +- `HttpOnly`: no JS access. The browser gets no visibility into tokens. +- `Secure`: HTTPS only. +- `Max-Age` / `Expires`: long (months to a year) is fine — the cookie is the session ID, not the token; revocation is DB-side. + +Sign or encrypt the cookie value. Axum's `PrivateCookieJar`, `cookie-session` in Node, `gorilla/sessions` in Go all provide this. + +**Two-cookie pattern** (optional): + +- `session` — HttpOnly, carries the opaque session ID for DB lookup. +- `identity` — **not** HttpOnly, carries `{did, handle, pds}` for browser-side display. Read-only metadata; no secrets. Useful for UI rendering without round-tripping to the server. + +Never put tokens in the identity cookie. + +## Multi-account + +A user who has logged into two DIDs has two sessions. Common approaches: + +- **One session cookie, user selects.** Session row has `dids: [did1, did2]` + `active_did`. UI picks. Simplest for BFFs. +- **Cookie jar per DID.** Multiple session cookies, each with its own DB row. More state, more complexity. +- **Account-switcher URL scheme.** `/app/@alice.bsky.social/feed` vs `/app/@bob.bsky.social/feed` — URL carries the active DID, session cookie resolves to the full set. + +The Rust `OAuthRequestStorage` trait and the TypeScript `SessionStore` interface both key by `sub` (the DID), so you can store multiple concurrent sessions under different DIDs without conflict. + +## Key considerations + +- **Rotate signing keys periodically** (confidential clients). See `client-metadata.md` §Key rotation. +- **Re-resolve the PDS** when the DID document TTL expires or on `invalid_token`. Accounts can migrate PDS. +- **Verify handle bidirectionally** on login and again periodically — handles can be transferred or invalidated (`handle.invalid`). +- **Log out aggressively on `invalid_grant`.** It means the session is dead at the AS. Don't retry; surface to user. +- **Never log tokens.** Not even redacted-last-N-chars. They're short-lived but while live they're the credential. diff --git a/skills/software-development/atproto-oauth/references/shared/spec.md b/skills/software-development/atproto-oauth/references/shared/spec.md new file mode 100644 index 0000000..aecc9c9 --- /dev/null +++ b/skills/software-development/atproto-oauth/references/shared/spec.md @@ -0,0 +1,155 @@ +# AT Protocol OAuth — specification + +AT Protocol OAuth is an OAuth 2.1 profile with mandatory **PKCE (S256)**, **PAR**, **DPoP**, and URL-based dynamic client registration via a published **client metadata document**. `client_secret` is never used; confidential clients authenticate to the token endpoint with a `private_key_jwt` assertion. + +This file is the language-neutral contract. Per-language files link here for every "why must I…" question. + +## Authoritative sources + +- Specs: , +- Guides: , , , , , +- Underlying standards: OAuth 2.1 (draft-ietf-oauth-v2-1), **RFC 9449** (DPoP), **RFC 7636** (PKCE), **RFC 9126** (PAR), **RFC 7523** (JWT client auth), **RFC 8414** (server metadata), **RFC 9207** (`iss` parameter), **draft-ietf-oauth-resource-metadata**, **draft-parecki-oauth-client-id-metadata-document**. + +## Entities + +| Entity | Role | AT Proto specifics | +| ------ | ---- | ------------------ | +| **Client** | The app requesting access. Identified by a URL. | `client_id` is the URL of a JSON metadata document. | +| **User / Account** | The human. Identified by a DID. | Returned as the `sub` field in token responses. | +| **Resource Server (RS)** | The PDS that holds the user's repo. | Publishes `/.well-known/oauth-protected-resource`. | +| **Authorization Server (AS)** | Issues tokens. Usually the same host as the PDS; may be a distinct entryway. | Publishes `/.well-known/oauth-authorization-server`. | + +The `sub` field is a **DID**, not a username. The session belongs to the DID for life — handles may change, DIDs do not. + +## Client types + +| | **Confidential** | **Public** | +|---|---|---| +| Has server-side component | yes | no | +| Can protect a signing key | yes (server keystore) | no | +| `token_endpoint_auth_method` | `private_key_jwt` | `none` | +| Publishes `jwks` or `jwks_uri` | yes (public half only) | no | +| Access-token lifetime | short (minutes), servers' choice | short (minutes) | +| Refresh-token lifetime | up to **180 days** | **14 days** | +| Session lifetime | unlimited (rotates keys periodically) | up to **14 days** | +| Client assertion JWT | yes, on every token request | no | + +Confidential clients are recommended whenever you run a backend. A pure browser SPA or a native mobile app without a server is a public client. + +## Application types (`application_type`) + +| `web` (default) | Browser-opening redirect URIs (`https://…` only, except localhost dev). | +| `native` | Custom-scheme redirect URIs (`com.example.app:/callback`) or Apple Universal Links. `client_id` hostname reversed to form the scheme. | + +## Mandatory features + +All AT Proto OAuth sessions **must** use all of the following. There is no opt-out. + +1. **PKCE S256.** `code_challenge_method=S256`. Verifier ≥ 43 chars, random. No `plain`. +2. **PAR** — Pushed Authorization Request. The authorize URL carries only `request_uri` + `client_id`; every other parameter is submitted server-to-server to `pushed_authorization_request_endpoint` first. +3. **DPoP.** Every request to AS and RS carries a signed DPoP proof JWT. AS- and RS-issued **server nonces** are mandatory and rotate within 5 minutes. See `dpop.md`. +4. **`iss` response parameter.** The callback URL MUST include `iss=`. Clients MUST verify it matches the AS they sent the request to. +5. **DID identity verification.** After token exchange, clients MUST verify that `sub` (a DID) resolves to a DID document whose PDS points back to the AS they just completed the flow with. For handle-initiated flows, clients MUST also bidirectionally verify that the handle resolves to that DID (per atproto handle spec). +6. **Scope response.** The token response MUST include a `scope` field; clients MUST reject the token if `atproto` is not present. + +## `client_id` + +The `client_id` **is** the URL of the metadata document. The AS fetches it over HTTPS on every new auth session (with optional caching — see §caching in `client-metadata.md`). + +- Scheme: `https://`, no port, path ends with a JSON file. Convention is `/oauth-client-metadata.json`. +- Exception: `http://localhost` (no port, no path except `/`) is allowed for development only. The AS generates virtual metadata; `redirect_uri` and `scope` may be passed as query parameters on `client_id`. + +## Discovery chain + +Given user input, a client resolves outward: + +``` +user input → DID / handle → DID document → PDS URL → + GET {PDS}/.well-known/oauth-protected-resource + → authorization_servers[0] (there MUST be exactly one) + GET {AS}/.well-known/oauth-authorization-server + → issuer, authorization_endpoint, token_endpoint, + pushed_authorization_request_endpoint, + require_pushed_authorization_requests = true, + authorization_response_iss_parameter_supported = true, + client_id_metadata_document_supported = true, + dpop_signing_alg_values_supported ⊇ [ES256], + code_challenge_methods_supported ⊇ [S256], + grant_types_supported ⊇ [authorization_code, refresh_token], + token_endpoint_auth_methods_supported ⊇ [none, private_key_jwt], + scopes_supported ⊇ [atproto] +``` + +The AS `issuer` MUST match the origin of the server metadata fetch. The client MUST fail the flow if any of those flags are missing. + +Handle-first input requires an extra step: resolve handle → DID → DID document via DNS `_atproto.{handle}` TXT or `/.well-known/atproto-did`, then verify the DID document's `alsoKnownAs` includes `at://{handle}`. Never trust a handle you didn't bidirectionally verify. See the `atproto-identity-resolution` skill. + +## The ten-step flow + +Every language implements these ten steps in order. See `flows.md` for byte-level wire content. + +1. **Resolve** user input → DID + PDS + AS metadata (as above). +2. **Generate per-session state:** random `state` (≥16 chars), random `nonce` (≥16 chars), PKCE `(verifier, challenge)` with S256, a fresh **DPoP keypair** (ES256 P-256), and store all of it. +3. **Mint client assertion** (confidential clients): JWT with `iss=sub=client_id`, `aud=`, `iat`, `exp` (short, ~1min), `jti` (random), signed with a key from the `jwks`. +4. **POST PAR** to `pushed_authorization_request_endpoint` with `response_type=code`, `client_id`, `redirect_uri`, `scope`, `state`, `code_challenge`, `code_challenge_method=S256`, `login_hint` (optional), `client_assertion_type`+`client_assertion` (confidential). Add a DPoP header. The first response will be HTTP 400 with `use_dpop_nonce` and a `DPoP-Nonce` header; re-sign with `nonce` claim and retry. Accept `request_uri` + `expires_in`. +5. **Redirect user** to `{authorization_endpoint}?client_id={client_id}&request_uri={request_uri}`. No other parameters. +6. **User authenticates + approves** on the AS. +7. **AS redirects to `redirect_uri`** with `code`, `state`, `iss`. Client verifies `state` and `iss`. +8. **POST token exchange** to `token_endpoint` with `grant_type=authorization_code`, `code`, `redirect_uri`, `client_id`, `code_verifier`, `client_assertion_type`+`client_assertion` (confidential). DPoP header (with nonce). Retry on `use_dpop_nonce`. +9. **Token response.** `access_token`, `token_type=DPoP`, `expires_in`, `refresh_token`, `scope`, `sub` (DID). Verify `scope` contains `atproto`. Verify `sub` DID's document → PDS → AS chain matches the AS you used. +10. **Resource requests.** Include `Authorization: DPoP ` plus a DPoP proof that adds `ath = base64url(SHA-256(access_token))` to its claims. Maintain a separate DPoP nonce per origin (AS vs RS). + +## Refresh + +Refresh tokens are single-use. Send `grant_type=refresh_token`, `refresh_token`, `client_id`, client assertion (confidential), DPoP header. The response is a new access token **and a new refresh token** — replace both atomically. Don't refresh unless the access token is within ~5 minutes of expiry, and serialize concurrent refreshes per-session (one in flight at a time) to avoid the race where two callers get two different new refresh tokens and one of them is immediately stale. + +## Token properties + +- **`access_token`** is opaque to the client. Treat as a string. Lifetime ≤ 30 minutes; servers that cannot revoke individual tokens may cap at 15 minutes. +- **DPoP binding.** Every token is bound to one DPoP keypair. Losing the DPoP private key invalidates the session; you cannot migrate tokens between devices. +- **Refresh lifetime.** Public: 14 days. Confidential: 180 days per token, unlimited session lifetime (rotate keys). +- **`sub`.** Always a DID. Never trust a handle — handles can change. + +## Error model + +The AS uses standard OAuth error codes in JSON bodies: `invalid_request`, `invalid_client`, `invalid_grant`, `invalid_dpop_proof`, `use_dpop_nonce`, `unsupported_grant_type`, `invalid_scope`, `access_denied`. See `troubleshooting.md` for the full catalog and recovery paths. + +Two error patterns have non-obvious recovery: + +- **`use_dpop_nonce` (400/401)** with `DPoP-Nonce` response header: expected on first request to each server. Extract the nonce, add `nonce` claim to a new DPoP proof, retry once. Max one retry per request. +- **`invalid_dpop_proof`**: the proof was rejected. Usually means wrong `htm`/`htu`, skewed clock, stale nonce, or missing `ath` on a resource request. Mint a fresh proof; do not reuse across requests. + +## Scopes summary + +`atproto` is always required. Everything else is additive. Full rules in `scopes.md`. + +| Scope pattern | Grants | +|---|---| +| `atproto` | declare atproto profile; mandatory | +| `transition:generic` | App-password-equivalent read/write (legacy) | +| `transition:chat.bsky` | chat.bsky Lexicons + service auth | +| `transition:email` | Email via `getSession` | +| `account:email?action=read\|manage` | Account email attribute | +| `account:repo?action=manage` | Repo-level hosting admin | +| `identity:handle`, `identity:*` | Handle management | +| `blob:*/*`, `blob:image/*`, `blob:video/*` | Blob upload mime filters | +| `repo:*` or `repo:?action=create\|update\|delete` | Record writes, per collection | +| `rpc:?aud=\|*&lxm=` | XRPC call access | +| `include:?aud=` | Reference a published permission-set lexicon | + +## Security invariants + +Full checklist in `security-requirements.md`. The non-negotiables: + +- Fetches to AS metadata, RS metadata, and PLC directory MUST use an **SSRF-hardened** HTTP client: block private ranges (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `127.0.0.0/8`, `169.254.0.0/16`, `fc00::/7`, `fe80::/10`), cap body size, cap time, limit redirects. +- DPoP private keys and `access_token`/`refresh_token` are **secrets**. Never expose to browser JS in the BFF pattern; store in server-side DB bound to an HttpOnly session cookie. +- `state` MUST be random and single-use. The AS must reject duplicate `state` values. +- The handler for the callback MUST reject replays: delete the OAuth-request row as soon as token exchange starts. +- Refresh-token rotation is atomic: lose a refresh response and the session is dead. Persist before ack. + +## What this skill does NOT cover + +- CID parsing (see `atproto-cid`). +- Handle ↔ DID resolution details (see `atproto-identity-resolution`). +- CAR / MST / record writing (see `atproto-repository`). +- Lexicon authoring, XRPC method invocation, and record parsing (see `atproto-lexicon`). diff --git a/skills/software-development/atproto-oauth/references/shared/test-vectors.md b/skills/software-development/atproto-oauth/references/shared/test-vectors.md new file mode 100644 index 0000000..9dd9c96 --- /dev/null +++ b/skills/software-development/atproto-oauth/references/shared/test-vectors.md @@ -0,0 +1,253 @@ +# Test vectors + +Fixtures used to validate cross-language OAuth implementations. Prefer tiny, self-contained inputs with byte-exact expected outputs. + +## Source of truth + +- **PKCE vector**: from RFC 7636 §4.2. +- **DPoP examples**: constructed against RFC 9449 §4.2 conventions. +- **JWK thumbprint**: RFC 7638 §3.1. +- **Client-metadata document** fixtures: hand-authored, round-tripped through `scripts/validate_client_metadata.py`. + +When you add a vector, name the source (spec paragraph, existing fixture file, upstream test suite). No hand-waving. + +## V1 — PKCE S256 (RFC 7636) + +**Input (verifier):** + +``` +dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW1gFWFOEjXk +``` + +**Expected challenge:** + +``` +E9Melhoa2OwvFrEMTJguCHaoeK1t8URWbuGJSstw-cM +``` + +**Steps:** + +1. `sha256(utf8(verifier))` → 32 bytes. +2. `base64url(32 bytes)` with no padding (strip trailing `=`). + +Verified by: Rust `atproto-oauth::pkce::challenge`, Go indigo `pkce` helper, TS `@atproto/oauth-client`'s runtime. + +## V2 — JWK thumbprint (RFC 7638) + +**Input JWK (P-256, from RFC 7638 §3.1):** + +```json +{ + "kty": "EC", + "crv": "P-256", + "x": "fD3LGX-TLg_UhL1trfxIiLfADwPHI6Oi0XiNqFkB2Ss", + "y": "jdeIe-uLj5j1PJ6_rShxoRmcXRqWfUjqUVXJmpEaNI4" +} +``` + +**Canonical JSON** (bytewise sort of keys `crv`, `kty`, `x`, `y`, no whitespace): + +``` +{"crv":"P-256","kty":"EC","x":"fD3LGX-TLg_UhL1trfxIiLfADwPHI6Oi0XiNqFkB2Ss","y":"jdeIe-uLj5j1PJ6_rShxoRmcXRqWfUjqUVXJmpEaNI4"} +``` + +**SHA-256 → base64url (no pad, 43 chars):** + +``` +(compute at verification time; expected length 43) +``` + +Rust's `atproto-oauth::jwk::thumbprint` implements this; Go and TS canonicalize the same way. + +Assertion: length is exactly 43 characters and contains only `[A-Za-z0-9_-]`. + +## V3 — Minimal confidential client metadata + +**Input:** `https://example.app/oauth-client-metadata.json` serving: + +```json +{ + "client_id": "https://example.app/oauth-client-metadata.json", + "application_type": "web", + "grant_types": ["authorization_code", "refresh_token"], + "scope": "atproto transition:generic", + "response_types": ["code"], + "redirect_uris": ["https://example.app/oauth/callback"], + "dpop_bound_access_tokens": true, + "token_endpoint_auth_method": "private_key_jwt", + "token_endpoint_auth_signing_alg": "ES256", + "jwks": { + "keys": [ + { + "kty": "EC", + "crv": "P-256", + "x": "MKBCTNIcKUSDii11ySs3526iDZ8AiTo7Tu6KPAqv7D4", + "y": "4Etl6SRW2YiLUrN5vfvVHuhp7x8PxltmWWlbbM4IFyM", + "kid": "key-1", + "use": "sig", + "alg": "ES256" + } + ] + } +} +``` + +**Expected validation result:** PASS. Run through `scripts/validate_client_metadata.py` (exit 0) and through each language's metadata loader. + +## V4 — Public client metadata + +```json +{ + "client_id": "https://spa.example.app/oauth-client-metadata.json", + "application_type": "web", + "grant_types": ["authorization_code", "refresh_token"], + "scope": "atproto transition:generic", + "response_types": ["code"], + "redirect_uris": ["https://spa.example.app/oauth/callback"], + "dpop_bound_access_tokens": true, + "token_endpoint_auth_method": "none" +} +``` + +Expected: PASS, no `jwks` required. + +## V5 — Invalid metadata (each case one property away from valid) + +Each should FAIL validation with a clear error message: + +```json +// V5a: dpop_bound_access_tokens must be true +{ "...": "..., \"dpop_bound_access_tokens\": false, ..." } + +// V5b: missing atproto scope +{ "...": "..., \"scope\": \"transition:generic\", ..." } + +// V5c: http redirect_uri on web client (non-localhost) +{ "...": "..., \"redirect_uris\": [\"http://example.app/callback\"], ..." } + +// V5d: JWK contains private component +{ "...": "..., \"jwks\": {\"keys\":[{\"kty\":\"EC\",\"crv\":\"P-256\",\"x\":\"...\",\"y\":\"...\",\"d\":\"LEAKED!\"}]}, ..." } + +// V5e: confidential client missing jwks +{ "...": "..., \"token_endpoint_auth_method\": \"private_key_jwt\" /* no jwks */, ..." } + +// V5f: client_id doesn't match URL +{ "...": "..., \"client_id\": \"https://different.example/metadata.json\", ..." } +``` + +## V6 — DPoP proof for PAR (no nonce) + +**Key:** P-256 private key (fixed in test harness; regenerate locally for sanity). + +**Claims:** + +```json +{ + "jti": "01JABCDEF...", + "htm": "POST", + "htu": "https://pds.example.com/oauth/par", + "iat": 1714657800 +} +``` + +**Header:** + +```json +{ + "typ": "dpop+jwt", + "alg": "ES256", + "jwk": { "kty":"EC","crv":"P-256","x":"...","y":"..." } +} +``` + +**Expected:** server responds HTTP 400 with `DPoP-Nonce: ` and `{"error":"use_dpop_nonce", ...}`. + +## V7 — DPoP proof for PAR (with nonce) + +Same as V6 plus a `nonce` claim: + +```json +{ + "jti": "01JABCDEG...", + "htm": "POST", + "htu": "https://pds.example.com/oauth/par", + "iat": 1714657801, + "nonce": "" +} +``` + +**Expected:** HTTP 201, `DPoP-Nonce: `, body `{"request_uri":"urn:...","expires_in":...}`. + +## V8 — DPoP proof for resource request (with `ath`) + +Access token: `"abcdef.ghijkl"` (obviously fake; just for hash). + +**Expected `ath`:** + +``` +ath = base64url_no_pad(sha256_bytes(utf8("abcdef.ghijkl"))) +``` + +Compute locally and check that implementations emit the same value. Fixture harnesses typically set up known tokens and assert exact `ath` bytes. + +**Claims:** + +```json +{ + "jti": "01JABCDEH...", + "htm": "GET", + "htu": "https://pds.example.com/xrpc/com.atproto.repo.getRecord", + "iat": 1714657900, + "nonce": "", + "ath": "" +} +``` + +## V9 — Scope round-trip + +**Input string:** + +``` +atproto transition:generic repo:app.bsky.feed.post?action=create&action=update rpc:app.bsky.feed.searchPosts?aud=did:web:api.bsky.app%23bsky_appview include:com.example.extra?aud=did:web:api.example.com%23svc_main +``` + +**Expected parse (sorted):** a list of `Scope` enum values — exact types per language. + +**Expected serialize:** bytewise-sorted string, same set of scope strings joined by single space. Round-trip is identity after canonicalization. + +**Expected reduced (if subsumption applied):** same, since no redundant scopes are present. + +## V10 — AS metadata minimum-conformant document + +```json +{ + "issuer": "https://pds.example.com", + "authorization_endpoint": "https://pds.example.com/oauth/authorize", + "token_endpoint": "https://pds.example.com/oauth/token", + "pushed_authorization_request_endpoint": "https://pds.example.com/oauth/par", + "response_types_supported": ["code"], + "grant_types_supported": ["authorization_code", "refresh_token"], + "code_challenge_methods_supported": ["S256"], + "token_endpoint_auth_methods_supported": ["none", "private_key_jwt"], + "token_endpoint_auth_signing_alg_values_supported": ["ES256"], + "scopes_supported": ["atproto", "transition:generic"], + "dpop_signing_alg_values_supported": ["ES256"], + "authorization_response_iss_parameter_supported": true, + "require_pushed_authorization_requests": true, + "client_id_metadata_document_supported": true +} +``` + +Expected: ALL AT Proto assertions pass. + +Removing any one of those booleans-set-true or list-membership checks flips validation to FAIL. + +## How to use + +1. Start with V1 (PKCE) and V2 (JWK thumbprint) — they're pure-function tests with no I/O. +2. Add V3–V5 to your metadata loader test suite. +3. Add V9 (scope round-trip) to your scope parser, if you have one. +4. V6–V8 (DPoP) need a harness with a test key; use them against a mock AS/PDS or, in integration, against a real staging PDS. +5. V10 (AS metadata) is the discovery-side regression test — ensure your conformance checker rejects each mutation in turn. + +When porting to a new language, round-trip V1 and V2 first. If those pass, the cryptographic primitives are hooked up correctly. The higher-level vectors follow. diff --git a/skills/software-development/atproto-oauth/references/shared/troubleshooting.md b/skills/software-development/atproto-oauth/references/shared/troubleshooting.md new file mode 100644 index 0000000..2a11e36 --- /dev/null +++ b/skills/software-development/atproto-oauth/references/shared/troubleshooting.md @@ -0,0 +1,231 @@ +# Troubleshooting + +Error catalogue for AT Proto OAuth clients. Each entry: symptom, likely cause, recovery. + +## `use_dpop_nonce` + +**Symptom:** HTTP 400 (or 401 from PDS) with body `{"error":"use_dpop_nonce",...}` and response header `DPoP-Nonce: `. + +**Cause:** Server requires a nonce in DPoP proofs. Expected on first request to each origin. + +**Recovery:** + +1. Extract `DPoP-Nonce` response header. +2. Mint a NEW DPoP proof with fresh `jti`, fresh `iat`, and the `nonce` claim set to the value. +3. Retry the request once. +4. Update your per-origin nonce cache with the value. + +Budget: 1 retry per request. Two in a row = bug (clock skew, wrong `htu`, nonce copied from wrong origin). + +## `invalid_dpop_proof` + +**Symptom:** HTTP 400/401 with `{"error":"invalid_dpop_proof", ...}`. + +**Possible causes:** + +| Cause | Fix | +|---|---| +| Missing `ath` on resource request | Add `ath = base64url(SHA-256(access_token))` | +| Wrong `htm` (e.g. request is POST, proof says GET) | Set `htm` to the actual method, uppercase | +| Wrong `htu` (query string included, or wrong host/path) | Use full URL without query or fragment | +| Stale `nonce` | Retry once with fresh nonce from `DPoP-Nonce` header | +| Clock skew | Sync NTP. Check server's accepted window | +| Wrong `typ` (`"JWT"` instead of `"dpop+jwt"`) | Fix the header | +| Proof reused across requests | Mint a new proof every request | +| Signature doesn't verify | Public JWK in header doesn't match signing key | +| Algorithm mismatch | `alg` in header doesn't match `jwk` curve | + +Always re-mint a fresh proof; never retry with the same proof. + +## `invalid_client` + +**Symptom:** HTTP 400/401 from token endpoint with `{"error":"invalid_client",...}`. + +**Possible causes:** + +- **Client assertion fails signature verification** — usually because the `kid` in the assertion header doesn't match a key in your published `jwks`. Check you're signing with the key whose public half is currently published. +- **Client metadata not fetchable** — the AS tried to fetch `client_id` URL and got 404, 500, or bad JSON. +- **Assertion expired** — `exp` in the past. Use a short but not zero `exp` (~60s). +- **Assertion's `aud` doesn't match AS issuer** — typo or stale issuer. +- **Assertion's `iss`/`sub` not equal to `client_id`.** +- **`jti` reuse** — AS tracks recent `jti`s per client; use a random one each time. +- **Key rotated out** — the `kid` was removed from `jwks` but sessions bound to it are still refreshing. Keep old keys longer. + +Fix: regenerate the client assertion with correct `aud`, fresh `jti`, and a `kid` currently in `jwks`. + +## `invalid_grant` + +**Symptom:** HTTP 400 from token endpoint on authorization_code or refresh_token grant. + +**Possible causes:** + +| Grant type | Likely cause | +|---|---| +| `authorization_code` | Code already used; user took too long; wrong `redirect_uri`; wrong `code_verifier` | +| `refresh_token` | Refresh token already used (single-use); session revoked; session expired; DPoP key changed | + +Recovery: **re-authenticate**. No retry is possible — the grant is dead. + +For refresh specifically: if you hit `invalid_grant`, the session is gone. Don't retry. Clear the server session, expire the cookie, prompt user for fresh login. + +## `invalid_token` + +**Symptom:** HTTP 401 from PDS with `WWW-Authenticate: DPoP error="invalid_token"`. + +**Causes:** + +- Access token expired. +- Access token revoked. +- DPoP key mismatch. +- PDS doesn't recognize the token (AS ↔ PDS state drift). + +Recovery: + +1. If token is near expiry: refresh and retry. +2. If refresh fails with `invalid_grant`: re-authenticate. +3. If you just refreshed and still get `invalid_token`: possible AS-PDS lag. Retry once after a short backoff (100-500ms). If still failing, re-authenticate. + +## `invalid_scope` + +**Symptom:** HTTP 400 during PAR or authorize. + +**Causes:** + +- Requested scope not in client metadata `scope` field. +- Scope syntax malformed (e.g. partial wildcard `repo:app.bsky.*`). +- Referenced `include:` not fetchable by AS. +- Required `atproto` scope missing. + +Fix: align the authorize `scope` with metadata. Always include `atproto`. + +## `access_denied` + +**Symptom:** Callback URL contains `error=access_denied&error_description=...&state=...`. + +**Cause:** User clicked "Deny" on the consent screen, or an AS policy rejected the request. + +Recovery: clear pending state, return to pre-login. Surface a friendly message. + +## `invalid_request` + +**Symptom:** HTTP 400 with `{"error":"invalid_request",...}`. + +**Causes:** malformed parameter — missing required field, bad encoding, `redirect_uri` not in metadata, `response_type` not `code`. + +Fix: read `error_description`; usually spells out which field. + +## `unsupported_grant_type` + +**Symptom:** HTTP 400 from token endpoint. + +**Cause:** `grant_type` you sent isn't in `grant_types` in your client metadata (or isn't supported by the AS at all). + +Fix: declare `refresh_token` in client metadata `grant_types`, not just `authorization_code`. + +## `server_error` / 5xx from AS + +**Symptom:** HTTP 500/502/503 from AS or PDS. + +**Cause:** AS/PDS is having a bad day. + +Recovery: retry with exponential backoff, 3 attempts max. If still failing, surface to user. Don't silently keep trying — cascading retry storms amplify outages. + +## Handle resolution failures + +**Symptoms:** + +- `handle.invalid` returned as the handle during resolution. +- DNS TXT `_atproto.{handle}` empty and `/.well-known/atproto-did` returns 404. +- DID document's `alsoKnownAs` doesn't include `at://{handle}`. + +**Recovery:** + +- Ask user for a DID directly. +- If the handle should work, point them at a handle debugger — bidirectional handle verification is the `atproto-identity-resolution` skill's territory. +- Don't proceed with an unverified handle. Spoofing risk. + +## PDS discovery failures + +**Symptoms:** + +- DID document lacks a `#atproto_pds` service entry. +- `{PDS}/.well-known/oauth-protected-resource` returns 404. +- `authorization_servers` empty or has multiple entries. + +**Recovery:** + +- PDS doesn't implement OAuth yet — some older PDSes don't. +- `authorization_servers` with multiple entries: AT Proto profile says exactly one. If you see more, it's out of spec; reject. +- Ask user to check with their PDS operator. + +## AS metadata rejection + +**Symptoms:** + +- `require_pushed_authorization_requests` missing or false. +- `authorization_response_iss_parameter_supported` missing or false. +- `client_id_metadata_document_supported` missing or false. +- `dpop_signing_alg_values_supported` doesn't include `ES256`. +- `code_challenge_methods_supported` doesn't include `S256`. + +**Recovery:** + +- Reject the flow. The AS doesn't meet the AT Proto profile. +- Log for debugging. This is a server configuration bug, not a client bug. + +## Token expiry flakiness + +**Symptom:** sporadic 401s despite "recent" refreshes. + +**Likely cause:** the refresh race. Two concurrent requests both tried to refresh, both succeeded, one of the new refresh tokens is now dead on arrival. + +**Fix:** serialize refreshes per session (mutex or single-flight). See `sessions.md`. + +## Cookie not sent on callback + +**Symptom:** callback handler can't find the OAuth-request row; you never stored it (but you did). + +**Likely cause:** `SameSite=Strict` on the session cookie means the cross-origin redirect from AS doesn't carry the cookie. + +**Fix:** use `SameSite=Lax`. Strict is too tight for OAuth. + +## Mismatch between what you asked and what you got + +**Symptom:** token response's `scope` is narrower than what you requested. + +**Cause:** user (or AS policy) granted fewer scopes. This is normal. + +**Fix:** respect the returned scope. Gate features you can't access gracefully. Don't pretend to have access you lack. + +## "It worked yesterday" + +Common culprits: + +1. **Client assertion key rotated out.** Check your `jwks` still contains the key whose `kid` you're signing with. +2. **Clock drifted.** Server time is wrong; restart NTP. +3. **DPoP nonces flushed across a deploy** — normal. Next request hits `use_dpop_nonce`; retry path handles it. +4. **Client metadata served with wrong content-type or stale cache.** AS might be caching a bad response. +5. **PDS migration.** User changed PDS; their DID doc now points elsewhere. Re-resolve. + +## Debugging workflow + +When a flow fails, collect in this order: + +1. Exact error body from the failing HTTP response. +2. Full `Set-Cookie` and `DPoP-Nonce` headers from the last successful response. +3. DPoP proof (header + claims) that was sent, decoded. +4. Client assertion (if confidential) decoded. +5. The AS metadata JSON. +6. Your client metadata JSON. +7. The `state`, `code`, `iss` from the callback (redacted to first/last 4 chars). + +90% of OAuth bugs are diagnosable from those seven items. + +## When to ask the server operator + +- `client_id_metadata_document_supported: false` — operator hasn't enabled AT Proto OAuth. +- `invalid_client` with metadata confirmed fetchable → AS caching old key; ask to flush. +- `invalid_token` storm from one PDS → AS-PDS lag; operator can check. +- `handle.invalid` for a handle that should work → handle-resolver outage; `atproto-identity-resolution` skill. + +Per-language troubleshooting (library-specific stack traces, middleware gotchas) lives in `{rust,typescript,go}/sessions.md`. diff --git a/skills/software-development/atproto-oauth/references/typescript/README.md b/skills/software-development/atproto-oauth/references/typescript/README.md new file mode 100644 index 0000000..cab4262 --- /dev/null +++ b/skills/software-development/atproto-oauth/references/typescript/README.md @@ -0,0 +1,202 @@ +# TypeScript — `@atproto/oauth-client-*` setup + +The official TypeScript stack is three companion packages, one per runtime target: + +| Package | Runs in | Client type | Use when… | +| -------------------------------- | ------------ | -------------------- | ---------------------------------------- | +| `@atproto/oauth-client-node` | Node.js | Confidential (BFF) | You have a backend. **Default choice.** | +| `@atproto/oauth-client-browser` | Browser | Public (SPA) | No backend; SPA shipping tokens to the browser. | +| `@atproto/oauth-client` | Any | Base class | Internal/shared use; rarely import directly. | + +All three share types from `@atproto/oauth-types`. The implementations live in the `bluesky-social/atproto` monorepo under `packages/oauth/`. + +## Install + +### Node (confidential BFF) + +```json +{ + "dependencies": { + "@atproto/oauth-client-node": "^0.3", + "@atproto/api": "^0.13" + } +} +``` + +### Browser (public SPA) + +```json +{ + "dependencies": { + "@atproto/oauth-client-browser": "^0.3", + "@atproto/api": "^0.13" + } +} +``` + +Bundler: Vite or webpack with `browser` condition. The browser package uses `WebCrypto` + `IndexedDB`; no polyfills required on modern evergreen browsers. + +## Public surface at a glance + +### Node + +```ts +import { + NodeOAuthClient, + NodeSavedState, + NodeSavedSession, + type StateStore, + type SessionStore, +} from '@atproto/oauth-client-node' + +// Key methods on NodeOAuthClient: +client.clientMetadata // ClientMetadata — hand to /oauth-client-metadata.json +client.jwks // { keys: [...] } — hand to /jwks.json +client.authorize(handle, { scope, state }) // → URL to redirect to +client.callback(params) // → { session, state } after AS redirect +client.restore(did) // → OAuthSession | undefined (refreshes if needed) +client.revoke(did) // best-effort revoke at AS +``` + +### Browser + +```ts +import { BrowserOAuthClient } from '@atproto/oauth-client-browser' + +const client = await BrowserOAuthClient.load({ clientId, handleResolver }) +client.signIn(handle, options) // → never (page navigates to AS) +await client.init() // on page load → { session? } or null +client.addEventListener('updated', e => ...) +client.addEventListener('deleted', e => ...) +client.signOut(did) +``` + +## Typical wiring — Node BFF + +```ts +import express from 'express' +import { NodeOAuthClient } from '@atproto/oauth-client-node' +import { JoseKey } from '@atproto/jwk-jose' + +const client = new NodeOAuthClient({ + clientMetadata: { + client_id: 'https://app.example.com/oauth-client-metadata.json', + client_name: 'Example App', + client_uri: 'https://app.example.com', + redirect_uris: ['https://app.example.com/oauth/callback'], + grant_types: ['authorization_code', 'refresh_token'], + response_types: ['code'], + scope: 'atproto transition:generic', + token_endpoint_auth_method: 'private_key_jwt', + token_endpoint_auth_signing_alg: 'ES256', + dpop_bound_access_tokens: true, + application_type: 'web', + jwks_uri: 'https://app.example.com/jwks.json', + }, + keyset: await Promise.all([ + JoseKey.fromImportable(process.env.PRIVATE_KEY_1!, 'key-1'), + JoseKey.fromImportable(process.env.PRIVATE_KEY_2!, 'key-2'), // rotation + ]), + stateStore: myStateStore, // 10-min pre-flow state (keyed by `state`) + sessionStore: mySessionStore, // per-DID session (keyed by `sub`) + requestLock: myRequestLock, // serializes refresh per DID — single most important knob +}) + +const app = express() + +app.get('/oauth-client-metadata.json', (_req, res) => + res.json(client.clientMetadata)) + +app.get('/jwks.json', (_req, res) => res.json(client.jwks)) + +app.post('/oauth/login', async (req, res) => { + const url = await client.authorize(req.body.handle, { + scope: 'atproto transition:generic', + state: /* opaque app-side state, threaded through */ + }) + res.redirect(url.toString()) +}) + +app.get('/oauth/callback', async (req, res) => { + const params = new URLSearchParams(req.url.split('?')[1]) + const { session, state } = await client.callback(params) + // session.did is the verified DID + // set HttpOnly session cookie → DID → use client.restore(did) on future requests + req.session.did = session.did + res.redirect('/') +}) + +app.get('/api/feed', async (req, res) => { + const session = await client.restore(req.session.did) // refreshes if needed + const agent = new Agent(session) + const { data } = await agent.app.bsky.feed.getTimeline() + res.json(data) +}) +``` + +The three store interfaces (`StateStore`, `SessionStore`, `NodeRequestLock`) are what you implement. Each is a tiny async interface — see `sessions.md`. + +## Typical wiring — Browser SPA + +```ts +import { BrowserOAuthClient } from '@atproto/oauth-client-browser' +import { Agent } from '@atproto/api' + +const client = await BrowserOAuthClient.load({ + clientId: 'https://spa.example.com/oauth-client-metadata.json', + handleResolver: 'https://api.bsky.app', // or a custom resolver +}) + +// On page load: +const result = await client.init() +if (result?.session) { + // Already signed in. + const agent = new Agent(result.session) + /* ... */ +} else if (window.location.pathname === '/oauth/callback') { + // In a callback tab; client.init() handled params + exchanged tokens. + // `result` is defined; its `.state` echoes what was passed to `signIn`. +} + +// When user clicks Sign in: +await client.signIn('alice.bsky.social', { + scope: 'atproto transition:generic', + prompt: 'login', + ui_locales: 'en', + state: 'opaque-app-state', +}) +// ^ never returns; window.location changes to the AS. + +// Cross-tab sync: +client.addEventListener('updated', e => { /* reload cached data */ }) +client.addEventListener('deleted', e => { /* sign out UI */ }) +``` + +Browser client stores sessions in IndexedDB automatically. Token refresh is transparent on `restore()`. + +## Idioms specific to TypeScript + +- **ESM only.** All three packages are `"type": "module"`. Node ≥18 or a bundler with ESM support. +- **Handle resolution is injected.** Both clients require a `handleResolver` (a URL to an AppView or a function). The crate doesn't ship DNS resolvers — it delegates. For SPA, use `https://api.bsky.app`; for Node, use `@atproto-labs/handle-resolver-node` or roll your own. +- **DPoP is invisible.** You never mint a DPoP proof directly. The `Agent` returned by the client's `session.fetchHandler` signs every XRPC request, handles nonce retry, and tracks per-origin nonces. +- **Refresh serialization via `requestLock`.** Node only. A callback you provide that takes a key + an async function and ensures only one runs at a time per key. The default lock is in-process; for multi-node BFF you must provide a distributed lock (Redis, database advisory lock). +- **Errors are typed.** `TokenRefreshError`, `OAuthResponseError`, `WellKnownHandleResolverError`, etc. Catch on the type, not the message. +- **Cross-tab sync in browser.** `BroadcastChannel` is used; the `updated` / `deleted` events fire on sibling tabs. Don't cache session state across tabs — subscribe to these. + +Link to `../shared/divergence-matrix.md` for comparison against Rust and Go. Highlights: TS is the only stack with a first-class SPA client; the Rust crate is the only one that exposes a scope-AST parser; Go's `indigo` is BFF-only. + +## File map + +| Task | File | +| ----------------------------------------------- | -------------------- | +| Serving `/oauth-client-metadata.json` + `/jwks.json` | `client-metadata.md` | +| `signIn` / `callback` / `restore` flow | `flows.md` | +| How DPoP is handled internally; custom fetch handlers | `dpop.md` | +| `StateStore` / `SessionStore` / `requestLock` + IndexedDB SPA store | `sessions.md` | + +## See also + +- `../shared/spec.md` — normative rules. +- `../shared/divergence-matrix.md` — differences from Rust and Go. +- Upstream docs: , package README for `@atproto/oauth-client-node`. +- Upstream example: diff --git a/skills/software-development/atproto-oauth/references/typescript/client-metadata.md b/skills/software-development/atproto-oauth/references/typescript/client-metadata.md new file mode 100644 index 0000000..8980a72 --- /dev/null +++ b/skills/software-development/atproto-oauth/references/typescript/client-metadata.md @@ -0,0 +1,161 @@ +# TypeScript — Client metadata and JWKS + +The Node and Browser clients both build their own `clientMetadata` object from your constructor input. Your job is to **serve it** at the URL you use as `client_id`, and (for confidential clients) to serve the matching `jwks.json`. This file is the Node/Express plumbing. For the rules themselves, see `../shared/client-metadata.md`. + +## Node (confidential BFF) — the two endpoints + +```ts +import express from 'express' +import { NodeOAuthClient } from '@atproto/oauth-client-node' +import { JoseKey } from '@atproto/jwk-jose' + +const client = new NodeOAuthClient({ + clientMetadata: { + client_id: 'https://app.example.com/oauth-client-metadata.json', + client_name: 'Example App', + client_uri: 'https://app.example.com', + redirect_uris: ['https://app.example.com/oauth/callback'], + grant_types: ['authorization_code', 'refresh_token'], + response_types: ['code'], + scope: 'atproto transition:generic', + application_type: 'web', + token_endpoint_auth_method: 'private_key_jwt', + token_endpoint_auth_signing_alg: 'ES256', + dpop_bound_access_tokens: true, + jwks_uri: 'https://app.example.com/jwks.json', + }, + keyset: await Promise.all([ + JoseKey.fromImportable(process.env.PRIVATE_KEY_1!, 'key-1'), + JoseKey.fromImportable(process.env.PRIVATE_KEY_2!, 'key-2'), // rotation + ]), + stateStore, + sessionStore, + requestLock, +}) + +const app = express() + +app.get('/oauth-client-metadata.json', (_req, res) => + res.type('application/json').json(client.clientMetadata)) + +app.get('/jwks.json', (_req, res) => + res.type('application/json').json(client.jwks)) +``` + +`client.clientMetadata` echoes what you passed in, plus the library-computed `jwks_uri` if not provided. `client.jwks` is `{ keys: JsonWebKey[] }` — **public halves only** (the library strips private components before exposing). + +Never build the `jwks.json` response by hand from your raw keys — use `client.jwks`. The library is the single source of truth for what gets published. + +## Key loading: `@atproto/jwk-jose` + +`JoseKey` is the wrapper around `jose`'s `KeyLike`/`CryptoKey` that `NodeOAuthClient` expects. + +```ts +import { JoseKey } from '@atproto/jwk-jose' + +// 1. Generate a new key (dev-only — store the PEM in secret storage): +const key = await JoseKey.generate(['ES256']) +console.log(await key.toPEM()) + +// 2. Load from PEM (most common in production): +const key = await JoseKey.fromImportable(process.env.PRIVATE_KEY_PEM!, 'key-id-1') +// `key.kid` = 'key-id-1' +// `key.alg` = 'ES256' + +// 3. Load from JWK: +const key = await JoseKey.fromJWK({ kty: 'EC', crv: 'P-256', ..., kid: 'key-id-1' }) +``` + +The `kid` (key id) is how the AS picks the verifying key for your client assertion. Pin it per key and keep it stable across deploys. + +## Key rotation + +Pass *all* live keys in the `keyset` array. The library publishes every key in `jwks.json` but signs with the first one unless the AS's metadata pins a `kid`: + +```ts +keyset: await Promise.all([ + JoseKey.fromImportable(currentPem, 'key-2026-q2'), // signs new assertions + JoseKey.fromImportable(previousPem, 'key-2026-q1'), // still in jwks; used for in-flight verification if needed +]) +``` + +Retention: keep the old key in `jwks` until the longest-lived refresh token issued under it has expired (180 days for confidential clients). Then remove it. The AS caches `jwks_uri`, typically for ≤1h. + +## Browser (public SPA) — no server-side endpoints + +`BrowserOAuthClient.load({ clientId, ... })` **fetches** the metadata from your `clientId` URL on first load. Someone still has to serve that file — usually your static host. + +```json +// Served as /oauth-client-metadata.json by e.g. your SPA's static hosting. +{ + "client_id": "https://spa.example.com/oauth-client-metadata.json", + "client_name": "Example SPA", + "client_uri": "https://spa.example.com", + "redirect_uris": ["https://spa.example.com/oauth/callback"], + "grant_types": ["authorization_code", "refresh_token"], + "response_types": ["code"], + "scope": "atproto transition:generic", + "application_type": "web", + "token_endpoint_auth_method":"none", + "dpop_bound_access_tokens": true +} +``` + +No `jwks`, no `token_endpoint_auth_signing_alg`. Public clients are authenticated by DPoP proof alone. + +If you're in a dev setup with `http://127.0.0.1`, the loopback-client shortcut applies — see `../shared/client-metadata.md` §loopback. + +## Validating your metadata + +Before pointing an AS at the URL, self-check: + +```ts +import { validateClientMetadata } from '@atproto/oauth-client' +// (Internal helper; if not exported, run the CLI validator.) +validateClientMetadata(metadataObject) // throws on invariants +``` + +Or use the repo-level `scripts/validate_client_metadata.py`. Run it in CI against the served URL — catches the mutations in `../shared/test-vectors.md` §V5. + +## Serving correctly + +- `Content-Type: application/json` (Express's `res.json()` handles this). +- `Cache-Control: public, max-age=300` is fine; don't cache for hours. ASes re-fetch aggressively. +- If behind CloudFront / Cloudflare, set a short TTL. After rotation you want the new `jwks` live within minutes. +- Serve over HTTPS. The AS will reject `http://` `client_id` URLs outside of loopback-dev mode. + +## Public-client variant (Node native/desktop) + +Rare but supported. Same shape as the Browser SPA metadata — `token_endpoint_auth_method: "none"`, no `jwks_uri`, redirect URIs with a custom scheme or `http://127.0.0.1`: + +```json +{ + "client_id": "https://app.example.com/native-client-metadata.json", + "application_type": "native", + "redirect_uris": ["com.example.app:/oauth/callback"], + "grant_types": ["authorization_code", "refresh_token"], + "response_types": ["code"], + "scope": "atproto transition:generic", + "token_endpoint_auth_method":"none", + "dpop_bound_access_tokens": true +} +``` + +`@atproto/oauth-client-node` supports this by setting `token_endpoint_auth_method: 'none'` and omitting `keyset`. Refresh token lifetime drops to 14 days. + +## Common pitfalls + +- **Hand-rolling `jwks.json` from raw PEMs.** The private key components leak. Use `client.jwks`. +- **`client_id` URL path mismatch.** The AS fetches the exact URL you put in `client_id`. A trailing slash, path change, or `?format=` mutation breaks validation. Commit to one URL and serve it there. +- **Mixing `jwks` (inline) and `jwks_uri` in the same document.** Pick one. The library uses `jwks_uri` if `jwks_uri` is set. +- **Forgetting `dpop_bound_access_tokens: true`.** Required in the AT Proto profile. Without it the AS rejects the registration. +- **Static-hosted metadata + dynamic `client_id`.** Don't template the `client_id` per-env if your static host uses the same file. Either deploy per-env metadata files or thread env through the build. +- **Caching JWKS behind a CDN with a long TTL.** Rotations stall. Cap at 5 min during the rotation window. + +## See also + +- `README.md` — package setup, public API surface. +- `flows.md` — how `clientMetadata` is consumed by `authorize` / `callback`. +- `../shared/client-metadata.md` — normative rules. +- `../shared/test-vectors.md` §V5 — mutation tests for metadata. +- `../shared/security-requirements.md` §Client assertion keys — rotation semantics. diff --git a/skills/software-development/atproto-oauth/references/typescript/dpop.md b/skills/software-development/atproto-oauth/references/typescript/dpop.md new file mode 100644 index 0000000..c292d84 --- /dev/null +++ b/skills/software-development/atproto-oauth/references/typescript/dpop.md @@ -0,0 +1,133 @@ +# TypeScript — DPoP (invisible by design) + +The `@atproto/oauth-client-*` packages handle DPoP entirely inside the `fetchHandler` they return. You never mint a proof directly, never manage nonces, never re-sign on retry. This file describes what they do so you can debug it, extend it (custom fetch middleware), or understand the `invalid_dpop_proof` failures you see in logs. For the RFC 9449 rules, see `../shared/dpop.md`. + +## The `fetchHandler` contract + +Both `NodeOAuthClient` and `BrowserOAuthClient` expose an `OAuthSession` object with a `fetchHandler` that is drop-in compatible with `fetch`: + +```ts +const session = await client.restore(did) + +// Usable directly: +const res = await session.fetchHandler(url, { method: 'GET' }) + +// Or wrapped in the Agent helper: +import { Agent } from '@atproto/api' +const agent = new Agent(session) +await agent.app.bsky.feed.getTimeline() +``` + +Every call through `fetchHandler`: + +1. Adds `Authorization: DPoP ` header. +2. Mints a fresh DPoP proof with `htm`, `htu`, `ath=SHA-256(access_token)`, fresh `jti`, and (if cached) `nonce` for that origin. +3. Sends the request. +4. On `401`/`400` with `DPoP-Nonce` response header or `use_dpop_nonce` body, caches the new nonce, re-mints, retries once. +5. Stores the server's latest nonce in a per-origin cache keyed by origin (AS vs PDS are separate). + +The DPoP keypair is generated at session creation and stored alongside the session data (in `SessionStore` for Node, IndexedDB for Browser). It is **immortal for the life of that session** — rotating it invalidates the access and refresh tokens. + +## Per-origin nonce cache + +The library keeps a `Map` of the most-recent nonce per origin. Two consequences: + +- The first request to a new origin pays a retry (no cached nonce → server issues one → retry with it). +- If you drop a `fetchHandler` and build a new one, you lose the cache. Reuse the session's handler across the request's lifetime rather than creating a new one per call. + +For the Browser client, the cache is in-memory per tab. Cross-tab sharing of nonces is **not** implemented — each tab pays its own first-request retry. This is acceptable because nonces rotate frequently anyway. + +For the Node client, the cache is in-memory per process. Multi-node BFFs will each pay their own warm-up retry. Don't try to sync nonces across nodes — it's not worth it. + +## Custom `fetch` middleware + +Pass a custom fetch to intercept every outbound request (for logging, metrics, timeouts): + +```ts +import { NodeOAuthClient } from '@atproto/oauth-client-node' + +const client = new NodeOAuthClient({ + // ... other options ... + fetch: async (req) => { + const start = performance.now() + const res = await fetch(req) + console.log(`${req.method} ${req.url} → ${res.status} in ${performance.now() - start}ms`) + return res + }, +}) +``` + +This wraps the underlying transport; the library still handles DPoP on top. **Don't** try to read or modify the DPoP header here — it's a single-use proof with an already-computed signature. + +## Reading the DPoP proof (debugging only) + +If you need to inspect a proof in flight, add a fetch wrapper that logs headers: + +```ts +fetch: async (req) => { + console.log('DPoP:', req.headers.get('DPoP')) + console.log('Auth:', req.headers.get('Authorization')) + return fetch(req) +} +``` + +Then decode the DPoP JWT with a JWT debugger (or `jose`'s `decodeJwt`): + +```ts +import { decodeJwt, decodeProtectedHeader } from 'jose' +const header = decodeProtectedHeader(dpopJwt) +const claims = decodeJwt(dpopJwt) +// header: { typ: 'dpop+jwt', alg: 'ES256', jwk: { kty, crv, x, y } } +// claims: { jti, htm: 'GET', htu: '...', iat, exp, ath: '...', nonce? } +``` + +Use this when you see `invalid_dpop_proof` from a PDS — compare `htu` to the actual URL the PDS thinks it received (with / without default port, query string, trailing slash). + +## Browser — `ath` and subresource integrity + +The Browser client's `ath` (access-token hash) is computed via `crypto.subtle.digest('SHA-256', accessTokenBytes)` → base64url (no padding). If you're running in a context where `crypto.subtle` is unavailable (ancient iframe, non-secure origin), the whole stack fails to initialize. Must be HTTPS or `http://localhost`. + +## Node — no custom Node fetch + +On Node 18+ the library uses global `fetch` (undici). If you pass your own `fetch` via the `fetch:` option, it must match the WHATWG fetch signature — `node-fetch` v3 works, v2 doesn't (wrong Request/Response shape). + +## Server-side DPoP validation + +`@atproto/oauth-client-*` are **client** packages. They don't validate incoming DPoP proofs. If you're building an AS or resource server in TypeScript: + +- Roll your own with `jose`: verify JWT, check `typ=dpop+jwt`, check `htm`/`htu`/`iat`/`exp`, compute thumbprint. +- The validation rules live in `../shared/dpop.md` §server-side. +- `jti` replay protection must be your own — keep a bounded TTL cache keyed by `(thumbprint, jti)`. + +No public library exports `validateDpopJwt` today; AT Proto AS implementations (the PDS) do this in Go/Python, not TypeScript. + +## What the Agent does on top + +`new Agent(session)` wraps the session's `fetchHandler` into a lexicon-typed XRPC client: + +```ts +const agent = new Agent(session) +await agent.app.bsky.feed.getTimeline({ limit: 30 }) +// ↓ compiles to: +// session.fetchHandler('https://pds/xrpc/app.bsky.feed.getTimeline?limit=30', ...) +``` + +No extra auth logic — the `Agent` is purely a codegen shell over the handler. If auth fails, you'll see typed errors from `@atproto/api` that wrap the underlying `OAuthResponseError`. + +## Common pitfalls + +- **Recreating a fresh `Agent` for each request.** Keep it alive — the nonce cache lives on the session's handler. Recreating drops cached nonces → every call pays a retry. +- **Proxying `fetchHandler` output and modifying headers.** Don't touch the DPoP header. If you need to add instrumentation headers, add them upstream of `fetchHandler` via the `fetch:` option. +- **Serving your app over plain HTTP.** Browser DPoP requires `crypto.subtle`, which requires a secure context. Dev with `localhost` works; dev with a `192.168.x.x` IP doesn't. +- **Mixing `Bearer` and `DPoP` auth.** Never send `Authorization: Bearer ` for a DPoP-bound session. The whole point is sender constraining. The library always uses `DPoP `. +- **Long-lived, shared `OAuthSession` across multiple DIDs.** One session = one DID = one DPoP keypair. Don't reuse an agent across users. +- **Hoping the library caches nonces across processes.** It doesn't. If you're autoscaling BFF instances, each pays its own warm-up. Acceptable — don't optimize prematurely. + +## See also + +- `README.md` — package surface. +- `flows.md` — where DPoP enters the flow (auth endpoints AND resource endpoints). +- `sessions.md` — DPoP key lifetime bound to the session. +- `../shared/dpop.md` — RFC 9449 rules and nonce-dance diagram. +- `../shared/test-vectors.md` §V6–V8 — proof-shape vectors. +- `../shared/troubleshooting.md` §`invalid_dpop_proof` — diagnosis checklist. diff --git a/skills/software-development/atproto-oauth/references/typescript/flows.md b/skills/software-development/atproto-oauth/references/typescript/flows.md new file mode 100644 index 0000000..5b71df7 --- /dev/null +++ b/skills/software-development/atproto-oauth/references/typescript/flows.md @@ -0,0 +1,217 @@ +# TypeScript — Flows + +`NodeOAuthClient` and `BrowserOAuthClient` wrap the multi-step OAuth dance (discovery → PAR → authorize → callback → token exchange → refresh) into three methods: `authorize`, `callback`, `restore`. This file walks through each in both Node and Browser. For the wire-level content of each step, see `../shared/flows.md`. + +## Node — Authorize (begin flow) + +```ts +app.post('/oauth/login', async (req, res) => { + const handle = req.body.handle // 'alice.bsky.social' + const url = await client.authorize(handle, { + scope: 'atproto transition:generic', + state: crypto.randomUUID(), // opaque app-side value + }) + res.redirect(url.toString()) +}) +``` + +Under the hood: + +1. Resolves `handle` → DID → PDS → AS (uses the `handleResolver` you configured). +2. Generates PKCE verifier + challenge, nonce, DPoP keypair. +3. Mints a client assertion JWT (`private_key_jwt` with your `keyset[0]`). +4. POSTs PAR to the AS's `pushed_authorization_request_endpoint`. Handles DPoP nonce retry. +5. Writes pre-flow state to your `stateStore` keyed by the opaque `state` value. +6. Returns a `URL` to `{AS}/oauth/authorize?client_id=...&request_uri=urn:ietf:params:oauth:request_uri:...`. + +Your only visible input is `handle` + `options`. Options worth passing: + +- `scope` — space-separated string. Must start with `atproto`. +- `state` — opaque value the library threads through. You'll get it back in `callback`. +- `prompt` — `'login'` to force re-auth, `'consent'` to re-show consent. +- `ui_locales` — language hints. + +## Node — Callback + +```ts +app.get('/oauth/callback', async (req, res) => { + const params = new URLSearchParams(req.url.split('?')[1]) + const { session, state } = await client.callback(params) + // session.did — verified DID (do NOT trust the redirect-back params for this) + // session.handle — current handle + // session.aud — intended audience (PDS URL) + // state — the opaque value from authorize() + + req.session.user_did = session.did // HttpOnly cookie session + res.redirect('/') +}) +``` + +Under the hood: + +1. Reads `state` + `iss` + `code` from the query string. +2. Looks up pre-flow state from your `stateStore`. Throws if not found / expired. +3. Verifies `iss` matches the AS from pre-flow state (prevents issuer-mixup). +4. POSTs the code to `token_endpoint` with PKCE verifier + fresh DPoP proof + client assertion. +5. Verifies `aud == PDS URL`, `sub` resolves to a DID whose document names the AS. +6. Writes session via `sessionStore.set(sub, sessionData)`. +7. Deletes the pre-flow state (single-use). +8. Returns `OAuthSession` (the live handle, not just data) + original `state`. + +**What you get back:** the `session` is a live object. Call `session.getFetchHandler()` or construct `new Agent(session)` — you don't ferry raw tokens. + +## Node — Restore (subsequent requests) + +```ts +app.get('/api/feed', async (req, res) => { + const did = req.session.user_did + if (!did) return res.status(401).end() + + const session = await client.restore(did) // auto-refreshes if needed + const agent = new Agent(session) + const { data } = await agent.app.bsky.feed.getTimeline() + res.json(data) +}) +``` + +Under the hood: + +1. `sessionStore.get(did)` → stored session data. +2. If `expiresAt` within the refresh window (library default ~5 min), acquires `requestLock(did, fn)` and refreshes. +3. After refresh: `sessionStore.set(did, newData)`. +4. Returns an `OAuthSession` with a `fetchHandler` that auto-signs DPoP per request. + +**`requestLock` is load-bearing.** Without it, two concurrent requests refresh in parallel, the first invalidates the refresh token for the second, the second's write lands last with a dead token, and the next request fails permanently. See `sessions.md` §refresh race. + +## Node — Revoke / logout + +```ts +app.post('/oauth/logout', async (req, res) => { + const did = req.session.user_did + if (did) { + await client.revoke(did) // best-effort POST to revocation_endpoint + req.session.destroy(() => {}) + } + res.redirect('/') +}) +``` + +`client.revoke(did)` also deletes the session from `sessionStore`. Ignore any error — the AS's revocation endpoint is optional. + +## Browser — Sign in + +```ts +import { BrowserOAuthClient } from '@atproto/oauth-client-browser' + +const client = await BrowserOAuthClient.load({ + clientId: 'https://spa.example.com/oauth-client-metadata.json', + handleResolver: 'https://api.bsky.app', +}) +``` + +`BrowserOAuthClient.load(...)` is async because it fetches `clientId` on initialization. Do it once at module load. + +```ts +// When user clicks Sign in: +await client.signIn('alice.bsky.social', { + scope: 'atproto transition:generic', + prompt: 'login', + ui_locales: 'en', + state: 'opaque-app-state', +}) +// ^ Never returns — window.location is replaced to the AS's authorize URL. +``` + +The library stores PKCE + DPoP state in IndexedDB before navigating. + +## Browser — Init (on page load) + +```ts +const result = await client.init() + +if (!result) { + // Not signed in. Show sign-in UI. + return +} + +if (result.session) { + // Either restoring an existing session OR just completed a callback. + const agent = new Agent(result.session) + // result.state is the opaque string you passed to signIn (present only on callback) +} +``` + +`client.init()` does three things at once: + +1. If `window.location.pathname === redirect path` and query has `code`+`state`, runs the callback exchange. Then replaces `history.state` to strip the query (so a refresh doesn't re-run the callback). +2. Otherwise, looks in IndexedDB for an existing session and restores it (auto-refreshes if close to expiry). +3. Returns `{ session, state? }` or `null`. + +**Call `init()` once, on every page load, before any UI depends on auth.** + +## Browser — Sign out + +```ts +await client.signOut(did) +// Removes from IndexedDB + revokes at AS (best-effort). +// Sibling tabs receive the 'deleted' event. +``` + +## Browser — Cross-tab sync + +```ts +client.addEventListener('updated', (e) => { + // Session data changed (refresh, new session). Re-read data. + const session = e.detail.session +}) + +client.addEventListener('deleted', (e) => { + // Session removed (sign-out, or refresh failed irrecoverably). + // Drop caches, show sign-in UI. +}) +``` + +Uses `BroadcastChannel` internally. Don't cache session state in local variables — always read via the event or `client.restore()`. + +## Error handling — what to catch + +The three methods throw typed errors. Catch on the constructor, not the message. + +```ts +import { + OAuthResponseError, // AS returned 4xx with structured body (e.g. access_denied) + OAuthCallbackError, // callback-specific: state unknown, code exchange failed + TokenRefreshError, // refresh failed permanently (invalid_grant etc.) +} from '@atproto/oauth-client-node' // same names in -browser + +try { + await client.callback(params) +} catch (e) { + if (e instanceof OAuthResponseError) { + // e.error == 'access_denied' | 'invalid_request' | ... + // e.errorDescription, e.status available + } else if (e instanceof OAuthCallbackError) { + // Usually: state is unknown (expired/replay) or PKCE mismatch + } + throw e +} +``` + +For refresh failures (stored session becomes dead), `restore()` throws `TokenRefreshError` — delete the session and prompt re-login. + +## Common pitfalls + +- **Trusting query-string DID over `session.did`.** `session.did` is the **verified** DID (from the AS's id-token `sub`, cross-checked against DID doc). Never read `did` from the URL. +- **Skipping `iss` check.** The library does this for you; don't write your own callback handler that bypasses it. +- **No `requestLock` on Node.** In a multi-process BFF, the default in-process lock doesn't cross instances. Use Redis (`ioredis` + `redlock`) or a Postgres advisory lock. +- **Running `client.init()` multiple times.** The callback exchange is single-use. Guard with a module-level boolean or ensure `init()` runs exactly once per page load. +- **Catching all errors as `Error`.** Loses the typed error structure; you can't distinguish "user cancelled" (`access_denied`) from "our request was malformed" (`invalid_request`). +- **Hard-coding the PDS or AS URL.** Resolution is mandatory — different users are on different PDSes, and PDSes can migrate. `handleResolver` exists precisely so you don't hard-code. + +## See also + +- `README.md` — package setup, full Node BFF sketch. +- `dpop.md` — how DPoP + nonce retry are handled internally. +- `sessions.md` — `StateStore` / `SessionStore` / `requestLock` contracts. +- `../shared/flows.md` — byte-level wire content for each step. +- `../shared/troubleshooting.md` — diagnosing the common callback failures. diff --git a/skills/software-development/atproto-oauth/references/typescript/sessions.md b/skills/software-development/atproto-oauth/references/typescript/sessions.md new file mode 100644 index 0000000..903eb59 --- /dev/null +++ b/skills/software-development/atproto-oauth/references/typescript/sessions.md @@ -0,0 +1,312 @@ +# TypeScript — State, sessions, and storage + +`NodeOAuthClient` requires three injected interfaces: `stateStore` (10-minute pre-flow state), `sessionStore` (long-lived per-DID sessions), and `requestLock` (refresh serialization). `BrowserOAuthClient` uses IndexedDB automatically — no injection needed. This file is the implementation guide for Node, plus notes on what the Browser does under the hood. For the shape and lifecycle of each, see `../shared/sessions.md`. + +## The three interfaces (Node) + +```ts +import type { + NodeSavedState, NodeSavedSession, + StateStore, SessionStore, + NodeRequestLock, +} from '@atproto/oauth-client-node' + +interface StateStore { + set(key: string, value: NodeSavedState): Promise + get(key: string): Promise + del(key: string): Promise +} + +interface SessionStore { + set(sub: string, value: NodeSavedSession): Promise + get(sub: string): Promise + del(sub: string): Promise +} + +type NodeRequestLock = (key: string, fn: () => Promise) => Promise +``` + +All values are plain objects — the library serializes to JSON internally. Your implementation's job is storage + retrieval by key. + +## In-memory (dev only) + +```ts +const stateStore: StateStore = { + store: new Map(), + async set(k, v) { this.store.set(k, v) }, + async get(k) { return this.store.get(k) }, + async del(k) { this.store.delete(k) }, +} as StateStore & { store: Map } +``` + +Good for unit tests. Do not ship. + +## Redis-backed `stateStore` + +```ts +import Redis from 'ioredis' +const redis = new Redis(process.env.REDIS_URL!) + +const stateStore: StateStore = { + async set(key, value) { + await redis.set(`oauth:state:${key}`, JSON.stringify(value), 'EX', 600) + }, + async get(key) { + const raw = await redis.get(`oauth:state:${key}`) + return raw ? JSON.parse(raw) : undefined + }, + async del(key) { + await redis.del(`oauth:state:${key}`) + }, +} +``` + +TTL of 600s (10 minutes) matches the AS's PAR expiry. Redis handles expiry for you — no cron needed. + +## Postgres-backed `sessionStore` + +```ts +import { Pool } from 'pg' +const pool = new Pool({ connectionString: process.env.DATABASE_URL }) + +// Schema: +// CREATE TABLE oauth_sessions ( +// sub TEXT PRIMARY KEY, +// data JSONB NOT NULL, +// updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +// ); + +const sessionStore: SessionStore = { + async set(sub, value) { + await pool.query( + `INSERT INTO oauth_sessions (sub, data, updated_at) + VALUES ($1, $2, NOW()) + ON CONFLICT (sub) DO UPDATE SET data = $2, updated_at = NOW()`, + [sub, value], + ) + }, + async get(sub) { + const { rows } = await pool.query( + 'SELECT data FROM oauth_sessions WHERE sub = $1', + [sub], + ) + return rows[0]?.data + }, + async del(sub) { + await pool.query('DELETE FROM oauth_sessions WHERE sub = $1', [sub]) + }, +} +``` + +Encrypt `data` at rest — it contains the refresh token and the DPoP private key JWK. Use `pgcrypto`, envelope encryption, or application-level AEAD. + +## `requestLock` — the load-bearing primitive + +```ts +type NodeRequestLock = (key: string, fn: () => Promise) => Promise +``` + +The library calls this whenever it refreshes a session. `key` is the DID. `fn` does the refresh work. Must run `fn` to completion while holding the lock for `key`; concurrent calls for the same key must serialize. + +### In-process (default) + +If you don't provide `requestLock`, the library uses an in-process promise map — fine for a single Node process: + +```ts +// What the library does internally: +const inFlight = new Map>() +const defaultLock: NodeRequestLock = async (key, fn) => { + const existing = inFlight.get(key) + if (existing) { await existing.catch(() => {}); return fn() } + // ^ simplified; real impl re-reads state to avoid double refresh. +} +``` + +### Redis / Redlock (multi-node) + +```ts +import Redlock from 'redlock' +const redlock = new Redlock([redis], { retryCount: 10, retryDelay: 200 }) + +const requestLock: NodeRequestLock = async (key, fn) => { + const lock = await redlock.acquire([`oauth:refresh-lock:${key}`], 30_000) + try { + return await fn() + } finally { + await lock.release().catch(() => {}) + } +} +``` + +TTL of 30s is a generous ceiling for refresh (normally <2s). `retryCount: 10 * retryDelay: 200` = ~2s max wait — tune for your concurrency. + +### Postgres advisory lock (if you already have PG) + +```ts +const requestLock: NodeRequestLock = async (key, fn) => { + const hash = hashToBigint(key) // pg_advisory_lock takes bigint + const client = await pool.connect() + try { + await client.query('SELECT pg_advisory_lock($1)', [hash]) + return await fn() + } finally { + await client.query('SELECT pg_advisory_unlock($1)', [hash]).catch(() => {}) + client.release() + } +} +``` + +Cheap, transactional, no extra infra. Use this if you're already on Postgres. + +## Session data contents + +What the library stores in `sessionStore.set(sub, value)`: + +```ts +interface NodeSavedSession { + dpopJwk: JWK // the client's DPoP private key + tokenSet: { + sub: string // DID + aud: string // PDS URL + iss: string // AS URL + scope: string + access_token: string + refresh_token?: string + token_type: 'DPoP' + expires_at: string // ISO-8601 + } +} +``` + +**Everything in here is secret.** The `refresh_token` + `dpopJwk` together = full account access. Encrypt at rest. Never log the full object. + +## Browser — IndexedDB (automatic) + +`BrowserOAuthClient` stores sessions in an IndexedDB database named `@atproto-oauth-client`. Two stores: + +- `state` — pre-flow state (same shape as Node's `StateStore`, TTL 10 min). +- `session` — post-flow sessions (keyed by DID). + +You can inspect it in DevTools → Application → IndexedDB. You can also clear it to force re-auth (or call `client.signOut(did)`). + +The browser library listens to `storage` events and `BroadcastChannel` to sync across tabs. Don't write to the IndexedDB store manually — use the client API. + +**IndexedDB persistence is load-bearing across reloads.** The per-origin DPoP nonces issued by the AS and PDS are stored in the same IndexedDB database as the session itself. If you clear IndexedDB — either manually in DevTools, or programmatically as part of a "reset" flow — you drop the nonces along with the session, and the next request pays a round-trip to re-prime the nonce (plus one `use_dpop_nonce` retry). Do not clear IndexedDB on logout; call `client.signOut(did)` instead, which removes the session entry but leaves the nonce cache intact for the next login. + +## Session hand-off BFF → browser + +In a BFF pattern (Node back-end, browser front-end), the browser never sees the OAuth tokens. Instead: + +1. Callback handler sets an HttpOnly cookie containing an **app-specific session ID** (not the access token). +2. Browser sends the cookie on every request. +3. BFF middleware looks up the session ID → DID, calls `client.restore(did)`, uses the resulting `Agent` server-side. + +Example middleware: + +```ts +import session from 'express-session' + +app.use(session({ + secret: process.env.SESSION_SECRET!, + resave: false, + saveUninitialized: false, + cookie: { httpOnly: true, secure: true, sameSite: 'lax', maxAge: 30 * 24 * 60 * 60 * 1000 }, +})) + +app.use(async (req, res, next) => { + if (!req.session.user_did) return next() + try { + req.oauthSession = await client.restore(req.session.user_did) + next() + } catch (e) { + if (e instanceof TokenRefreshError) { + req.session.destroy(() => res.redirect('/login')) + } else { + next(e) + } + } +}) +``` + +See `../shared/sessions.md` §BFF for the pattern rationale. + +## Two-cookie variant + +If you need a DID accessible to the front-end (for display), set two cookies: + +```ts +// Encrypted session cookie (HttpOnly) — contains session_id → DID mapping +res.cookie('session', encryptedSessionId, { + httpOnly: true, secure: true, sameSite: 'lax', + maxAge: 30 * 24 * 60 * 60 * 1000, +}) + +// Identity cookie (readable by JS) — just display data +res.cookie('identity', JSON.stringify({ did, handle, pds_url }), { + httpOnly: false, secure: true, sameSite: 'lax', + maxAge: 30 * 24 * 60 * 60 * 1000, +}) +``` + +The identity cookie holds no secrets; the session cookie is the authority. Never trust the identity cookie on the server side — always derive from the session cookie. + +## Session cookie settings + +```ts +{ + httpOnly: true, + secure: true, // HTTPS only; dev: set to false only for localhost + sameSite: 'lax', // NOT 'strict' — OAuth callback is cross-origin + maxAge: 30 * 24 * 60 * 60 * 1000, // 30 days (long enough for refresh cycle) + path: '/', +} +``` + +**`SameSite: 'lax'`, not `'strict'`.** The AS callback is a cross-origin top-level navigation; `strict` drops the cookie and your `callback` handler can't find the pre-flow state. See `../shared/troubleshooting.md` §"Cookie not sent on callback". + +## Logout + +```ts +app.post('/oauth/logout', async (req, res) => { + const did = req.session.user_did + if (did) { + try { await client.revoke(did) } catch {} // best-effort + } + req.session.destroy(() => { + res.clearCookie('session') + res.clearCookie('identity') + res.redirect('/') + }) +}) +``` + +`client.revoke(did)`: +1. POSTs to AS's revocation endpoint (if advertised) with the refresh token. +2. Calls `sessionStore.del(did)`. + +Always run 2 even if 1 fails. A cookie on a user's device persists until it expires; revocation at the AS is the only way to kill the refresh token server-side. + +## Key hygiene + +- **DPoP private key is immortal for the session.** Stored as JWK in `NodeSavedSession.dpopJwk`. Never rotate during a session. Rotation = end of session. +- **Encrypt `sessionStore.data` at rest.** Contains `refresh_token` + `dpopJwk`. A leaked row = account takeover. +- **Rotate the session cookie secret** periodically. Use a dual-secret scheme (decrypt with old+new) during transitions. +- **No tokens in logs.** Especially not `tokenSet.access_token` or `tokenSet.refresh_token`. Log `sub` only. + +## Common pitfalls + +- **In-process `requestLock` behind a load balancer.** Two Node instances refresh in parallel, one wins the rotation, the other's refresh token dies. Switch to Redis/Postgres lock the moment you scale past 1 process. +- **TTL on `stateStore` > 10 minutes.** The AS expires PAR `request_uri` at ~10 min anyway; longer TTL just leaks state. Match it. +- **Not clearing `sessionStore` on `TokenRefreshError`.** The session is permanently dead. Leaving it creates a perpetual refresh-fail loop. `del` it and force re-login. +- **Storing sessions in a plain-text file or unencrypted JSON.** Refresh token + DPoP key in one file = game over. Encrypt. +- **Cookie `SameSite: 'strict'`.** Breaks the callback. Lax is the required setting. +- **Hand-rolling the browser IndexedDB access.** Unnecessary. `BrowserOAuthClient` gives you `restore(did)`, `signOut(did)`, and events. Use them. + +## See also + +- `README.md` — Node and Browser package setup. +- `flows.md` — where `stateStore`/`sessionStore`/`requestLock` plug into each flow method. +- `dpop.md` — DPoP key lifetime bound to session. +- `../shared/sessions.md` — language-neutral rules and BFF/SPA/native patterns. +- `../shared/security-requirements.md` — cookie/key/token hardening checklist. +- `../shared/troubleshooting.md` §refresh race — diagnosing token-rotation failures. diff --git a/skills/software-development/atproto-oauth/scripts/validate_client_metadata.py b/skills/software-development/atproto-oauth/scripts/validate_client_metadata.py new file mode 100644 index 0000000..dfeaa47 --- /dev/null +++ b/skills/software-development/atproto-oauth/scripts/validate_client_metadata.py @@ -0,0 +1,221 @@ +#!/usr/bin/env python3 +""" +ATProtocol OAuth Client Metadata Validator + +Validates a client metadata document against ATProtocol OAuth requirements. +Can validate from a URL, file path, or stdin. + +Usage: + python validate_client_metadata.py https://example.com/oauth-client-metadata.json + python validate_client_metadata.py ./client-metadata.json + cat metadata.json | python validate_client_metadata.py - +""" + +import json +import sys +import urllib.request +import urllib.error +from typing import Any + + +class ValidationError: + def __init__(self, field: str, message: str, severity: str = "error"): + self.field = field + self.message = message + self.severity = severity # "error" or "warning" + + def __str__(self): + return f"[{self.severity.upper()}] {self.field}: {self.message}" + + +def validate_client_metadata(metadata: dict[str, Any]) -> list[ValidationError]: + """Validate ATProtocol OAuth client metadata document.""" + errors: list[ValidationError] = [] + + # Required fields + required_fields = [ + "client_id", + "client_name", + "redirect_uris", + "grant_types", + "response_types", + "scope", + "dpop_bound_access_tokens", + "token_endpoint_auth_method", + ] + + for field in required_fields: + if field not in metadata: + errors.append(ValidationError(field, "Required field is missing")) + + # client_id must be a URL + client_id = metadata.get("client_id", "") + if client_id and not client_id.startswith("https://"): + if not (client_id.startswith("http://localhost") or client_id.startswith("http://127.0.0.1")): + errors.append(ValidationError("client_id", "Must be an HTTPS URL (except localhost)")) + + # dpop_bound_access_tokens must be true + if metadata.get("dpop_bound_access_tokens") is not True: + errors.append(ValidationError("dpop_bound_access_tokens", "Must be true for ATProtocol")) + + # grant_types validation + grant_types = metadata.get("grant_types", []) + if "authorization_code" not in grant_types: + errors.append(ValidationError("grant_types", "Must include 'authorization_code'")) + if "refresh_token" not in grant_types: + errors.append(ValidationError("grant_types", "Should include 'refresh_token'", "warning")) + + # response_types validation + response_types = metadata.get("response_types", []) + if response_types != ["code"]: + errors.append(ValidationError("response_types", "Must be ['code']")) + + # token_endpoint_auth_method validation + auth_method = metadata.get("token_endpoint_auth_method") + if auth_method not in ("private_key_jwt", "none"): + errors.append(ValidationError( + "token_endpoint_auth_method", + "Must be 'private_key_jwt' (confidential) or 'none' (public)" + )) + + # Confidential client specific validations + if auth_method == "private_key_jwt": + # Must have jwks or jwks_uri + if "jwks" not in metadata and "jwks_uri" not in metadata: + errors.append(ValidationError("jwks", "Required for confidential clients")) + + if "jwks" in metadata and "jwks_uri" in metadata: + errors.append(ValidationError("jwks", "Cannot specify both 'jwks' and 'jwks_uri'")) + + # token_endpoint_auth_signing_alg should be ES256 + if metadata.get("token_endpoint_auth_signing_alg") != "ES256": + errors.append(ValidationError( + "token_endpoint_auth_signing_alg", + "Should be 'ES256' for ATProtocol" + )) + + # Validate jwks if present + jwks = metadata.get("jwks", {}) + keys = jwks.get("keys", []) + if jwks and not keys: + errors.append(ValidationError("jwks.keys", "Must contain at least one key")) + + for i, key in enumerate(keys): + key_errors = validate_jwk(key, f"jwks.keys[{i}]") + errors.extend(key_errors) + + # redirect_uris validation + redirect_uris = metadata.get("redirect_uris", []) + if not redirect_uris: + errors.append(ValidationError("redirect_uris", "Must contain at least one URI")) + + for i, uri in enumerate(redirect_uris): + if not uri.startswith("https://"): + if not (uri.startswith("http://localhost") or uri.startswith("http://127.0.0.1")): + errors.append(ValidationError( + f"redirect_uris[{i}]", + f"Must be HTTPS (except localhost): {uri}" + )) + + # scope validation + scope = metadata.get("scope", "") + if "atproto" not in scope: + errors.append(ValidationError("scope", "Should include 'atproto' scope", "warning")) + + # application_type validation (optional) + app_type = metadata.get("application_type") + if app_type and app_type not in ("web", "native"): + errors.append(ValidationError("application_type", "Must be 'web' or 'native'")) + + return errors + + +def validate_jwk(key: dict[str, Any], prefix: str) -> list[ValidationError]: + """Validate a JWK in the jwks array.""" + errors: list[ValidationError] = [] + + # Required fields for EC key + required = ["kty", "crv", "x", "y"] + for field in required: + if field not in key: + errors.append(ValidationError(f"{prefix}.{field}", "Required field missing")) + + # kty must be EC + if key.get("kty") != "EC": + errors.append(ValidationError(f"{prefix}.kty", "Must be 'EC' for P-256 keys")) + + # crv must be P-256 + if key.get("crv") != "P-256": + errors.append(ValidationError(f"{prefix}.crv", "Must be 'P-256'")) + + # Should not contain private key material + if "d" in key: + errors.append(ValidationError( + f"{prefix}.d", + "SECURITY: Private key material found! Remove 'd' field from public metadata" + )) + + # alg should be ES256 + if key.get("alg") and key.get("alg") != "ES256": + errors.append(ValidationError(f"{prefix}.alg", "Should be 'ES256'")) + + # use should be sig + if key.get("use") and key.get("use") != "sig": + errors.append(ValidationError(f"{prefix}.use", "Should be 'sig'")) + + # kid should be present + if "kid" not in key: + errors.append(ValidationError(f"{prefix}.kid", "Should include key ID", "warning")) + + return errors + + +def load_metadata(source: str) -> dict[str, Any]: + """Load metadata from URL, file, or stdin.""" + if source == "-": + return json.load(sys.stdin) + elif source.startswith("http://") or source.startswith("https://"): + with urllib.request.urlopen(source, timeout=10) as response: + return json.loads(response.read().decode()) + else: + with open(source) as f: + return json.load(f) + + +def main(): + if len(sys.argv) != 2: + print(__doc__) + sys.exit(1) + + source = sys.argv[1] + + try: + metadata = load_metadata(source) + except json.JSONDecodeError as e: + print(f"Invalid JSON: {e}") + sys.exit(1) + except urllib.error.URLError as e: + print(f"Failed to fetch URL: {e}") + sys.exit(1) + except FileNotFoundError: + print(f"File not found: {source}") + sys.exit(1) + + errors = validate_client_metadata(metadata) + + error_count = sum(1 for e in errors if e.severity == "error") + warning_count = sum(1 for e in errors if e.severity == "warning") + + if errors: + print(f"\nValidation results for: {source}\n") + for error in errors: + print(f" {error}") + print(f"\n{error_count} error(s), {warning_count} warning(s)") + else: + print(f"✓ Client metadata is valid: {source}") + + sys.exit(1 if error_count > 0 else 0) + + +if __name__ == "__main__": + main() diff --git a/skills/software-development/atproto-repository/SKILL.md b/skills/software-development/atproto-repository/SKILL.md new file mode 100644 index 0000000..c038208 --- /dev/null +++ b/skills/software-development/atproto-repository/SKILL.md @@ -0,0 +1,103 @@ +--- +name: atproto-repository +description: "Use when working with ATProto repos: CAR v1, MST, DRISL, commit signing." +version: 1.0.0 +author: Hermes Agent (ported from ngerakines/atproto-skills, MIT) +license: MIT +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [ATProto, Repository, CAR, MST, DRISL, DAG-CBOR, Commit-Signing] +--- + +# AT Protocol Repository + +An AT Protocol **repository** is a single account's entire record store, +addressed by its DID. Every record lives at a key in a Merkle Search Tree; the +tree's root CID is sealed by a signed commit; the commit plus every block it +transitively references are the repository. On the wire it travels as a +**CAR v1** file (`com.atproto.sync.getRepo`). + +Ported from `ngerakines/atproto-skills` (MIT). Spec files and TypeScript guides +kept verbatim under `references/`; Rust/Go guides omitted (fetch upstream if +needed). This is the deep-dive skill — for app-level record CRUD see +`atproto-development` / `atproto-python`. + +## Defaults + +- **DRISL** — canonical DAG-CBOR: bytewise map keys, shortest-form integers, no + indefinite-length framing, CIDs as tag 42 + identity multibase. +- **CAR v1** — `varint ‖ header-cbor ‖ (varint ‖ 36-byte-CID ‖ bytes)*`. Roots + list declares the commit CID. +- **MST** — SHA-256 key height with fanout 4 (count pairs of leading zero + bits). Node shape `{l?, e:[{p,k,v,t?}]}`. Keys are `/`, + max 1024 bytes. +- **Commit** — `{did, version:3, data, rev, prev, sig}`. `prev` is `null` for + genesis (though see divergence below). Signing bytes = DAG-CBOR of the commit + minus `sig`. +- **Signature** — raw `r ‖ s` ECDSA (k-256 or p-256), low-S normalized. + Verified against the `#atproto` Multikey in the signer's DID document. + +## The conceptual stack + +``` +CAR v1 file ──────▶ header: { version:1, roots } + (varint ‖ CID ‖ bytes)* +Signed commit ────▶ { did, version:3, data, rev, prev, sig } (root block) +MST node ─────────▶ { l?, e:[entry,…] }, entry = { p, k, v, t? } (v = leaf CID) +Record block ─────▶ DAG-CBOR of record { $type: "", … } +``` + +CAR just frames a sequence of `(CID, bytes)` blocks. The same block bytes +travel over `getRecord`, the firehose, and any cache. + +## Cross-language hazards (TS-focused; full detail in divergence-matrix) + +- **`prev: null` vs omitted** — Rust reference impl omits `prev` for genesis + (4-entry map); Go and TS always serialize `prev: null` (5-entry map). When + signatures fail on genesis commits only, suspect this. **Strip the signature + from raw bytes rather than re-encoding before verifying.** +- **TS MST is immutable** — `add`/`update`/`delete` return a new `MST`; the + original is unchanged. Reassign to the return value or mutations silently + no-op. +- **TS CAR reader verifies CIDs on ingest** — `verifyIncomingCarBlocks` is on by + default; a CAR that Go/Rust accept may fail TS intake on block corruption. +- **Partial trees are exceptional in TS** — a firehose event traversal throwing + `MissingBlockError` is expected for partial trees; fall back to a partial + walker. +- **TS has no end-to-end wired verification** — resolve the signing key + separately first: `@atproto/identity` → `IdResolver.did.resolveAtprotoData(did)` + → `{ signingKey }` (a didKey), then pass into `verifyCommitSig`. + +## Record-side helpers + +- TID generation, AT-URI parsing: `@atproto/syntax` (TS). +- Canonical record CIDs: `create_record_cid(record_json)` via the lexicon-garden + MCP (or atpmcp against a local PDS) — verify your encoder output against the + canonical CID without booting a PDS. `transmogrify_record` round-trips a + record to DAG-CBOR bytes + CID. + +## References (read the relevant one before repo code) + +- `references/shared/drisl.md` — canonical DAG-CBOR rules +- `references/shared/car-v1.md` — CAR v1 byte layout +- `references/shared/mst.md` — MST algorithm + invariants +- `references/shared/commit-and-signing.md` — commit shape, signing bytes, + verification +- `references/shared/data-model.md` — records, NSIDs, TIDs, AT-URIs +- `references/shared/test-vectors.md` — fixtures +- `references/shared/divergence-matrix.md` — cross-language differences +- `references/typescript/README.md`, `drisl.md`, `car.md`, `mst.md`, + `commit.md` — @atproto/repo, @atproto/lex-cbor usage + +Prefer the official library (`@atproto/repo` in TS) over hand-rolling. Debug +targets: "CID mismatch", "unknown codec in CAR", "MST node not found", +"prefix_len exceeds previous key", "signature invalid", "prev null vs absent". + +## Related skills + +- `atproto-cid` — CID parse/construct/validate (the bytes CAR frames) +- `atproto-relays-firehose` / `atproto-python` — consuming subscribeRepos + commits (CAR.decode via the Python SDK) +- `atproto-identity-deep` / `atproto-development` — DID resolution needed for + signature verification +- `atproto-blob-lifecycle` — listBlobs/listRecords enumeration against a repo diff --git a/skills/software-development/atproto-repository/references/shared/car-v1.md b/skills/software-development/atproto-repository/references/shared/car-v1.md new file mode 100644 index 0000000..5b47509 --- /dev/null +++ b/skills/software-development/atproto-repository/references/shared/car-v1.md @@ -0,0 +1,220 @@ +# CAR v1 — Content Addressable aRchive (Reference) + +Source of truth: https://dasl.ing/car.html (DASL's restatement of CAR v1, constrained for AT Protocol). + +A CAR file is the on-the-wire container for one or more IPLD blocks — a header declaring the *roots* of interest, followed by a sequence of `(CID, bytes)` blocks in any order. AT Protocol's repo export, firehose commit events, and all sync endpoints frame their payloads as CAR v1. + +CAR is transport, not semantics. The receiver still has to parse the blocks (commit → MST → records) once they're unpacked. + +## 1. Byte layout in one picture + +``` +┌──────────────────────────────────────────────────────────────────────────┐ +│ varint(header_len) │ +├──────────────────────────────────────────────────────────────────────────┤ +│ header: DAG-CBOR { version: 1, roots: [CID, …] } (header_len bytes) │ +├──────────────────────────────────────────────────────────────────────────┤ +│ varint(block_1_len) │ +├──────────────────────────────────────────────────────────────────────────┤ +│ block_1: cid_bytes_1 || data_bytes_1 (block_1_len bytes) │ +├──────────────────────────────────────────────────────────────────────────┤ +│ varint(block_2_len) │ +├──────────────────────────────────────────────────────────────────────────┤ +│ block_2: cid_bytes_2 || data_bytes_2 (block_2_len bytes) │ +├──────────────────────────────────────────────────────────────────────────┤ +│ … │ +└──────────────────────────────────────────────────────────────────────────┘ +``` + +No magic number, no checksum, no footer. The file ends when the byte stream ends. A truncated CAR is detectable only by "the last varint claimed N bytes but we got M ; dag-cbor CIDv1 + SHA-256 digest + 67 76 65 72 73 69 6f 6e ; "version" + 01 ; unsigned(1) +``` + +That's 58 bytes of header, framed by a leading varint `0x3a` (decimal 58). Breakdown: `a2` (1) + "roots" key (6) + `81` (1) + CID as tag-42 bytes (41) + "version" key (8) + `01` (1) = 58. + +### What AT Protocol constrains + +- **Exactly one root** in practice. The spec allows multiple, but the only root that matters for a repo CAR is the signed commit. Multi-root CARs show up in niche contexts (repo plus side-channel blobs) but never in repo exports. +- **`version` must be 1**. CAR v2 exists upstream but is not used anywhere in AT Protocol. +- **Roots must be present in the CAR body** — every root CID must correspond to a block later in the file. A CAR whose root isn't included is malformed. + +## 4. Block + +Each block after the header is: + +``` +varint(cid_len + data_len) || || +``` + +The `data` portion is the raw, canonical DAG-CBOR encoding of the block value. The CID you read is the claim; the decoder is expected to re-hash `data` and check against the claimed CID. + +### CID length inside a block + +CAR v1 can in principle carry any IPLD codec. AT Protocol CAR files should contain **only** DASL CIDs: + +- 37 raw bytes total when you see them inside DAG-CBOR (1-byte identity prefix + 36 bytes CID). +- But in the **CAR block framing**, there is **no identity multibase prefix** — the raw 36 bytes of the CID go into the block header directly. See `atproto-cid` for the byte layout; the short version is `01 71 12 20 <32-byte SHA-256 digest>` for dag-cbor blocks. + +The reference Rust implementation reads the CID by: + +1. Reading the multibase/version byte (`0x01`). +2. Reading the codec varint (`0x71` = dag-cbor, `0x55` = raw). +3. Reading the multihash header (`0x12 0x20` = SHA-256, 32 bytes). +4. Reading 32 bytes of digest. + +Then the remainder of the block (up to `cid_len + data_len` total) is the payload. + +### Codecs you'll see + +| Codec byte | Codec | When you'll see it | +| ---------- | -------- | -------------------------------------------------------- | +| `0x71` | dag-cbor | Every commit, every MST node, every record. | +| `0x55` | raw | Blobs — images, video referenced from records. | + +A block whose CID codec is neither of those is invalid in an AT Protocol repo export. + +## 5. CID verification on read + +For every block the reader pulls out: + +1. Decode the CID bytes → (codec, digest). +2. Compute `SHA-256(data_bytes)` → 32-byte digest. +3. Require that computed digest equals declared digest. Mismatch = `CidMismatch` error, whole CAR is suspect. +4. For dag-cbor blocks, additionally require that `data_bytes` round-trips through a DRISL-strict decoder cleanly. A block that decodes only in lenient mode is a corruption signal, not a silent success. + +Skipping the verification step is the most common "my sync works but silently corrupts" bug. Don't. + +## 6. Block ordering + +Blocks may appear in **any order** in the file. The practical orderings are: + +- **Root-first** (most common): commit block, then MST nodes in traversal order, then records. Makes streaming decoders happy — they can start validating the signed commit before the whole file is in memory. +- **Arbitrary**: the spec permits any order, including duplicates. A correct decoder deduplicates by CID (the second copy of the same CID must be byte-identical, otherwise the producer is broken). + +Streaming decoders should buffer blocks into a block store keyed by CID, then walk the tree starting from the root. A one-pass decoder that requires topological order will break on legitimate CARs. + +## 7. Streaming guidance + +For large repos (a busy Bluesky account can exceed 1 GB), stream the CAR rather than buffering. Pseudocode: + +``` +read_varint() -> header_len +read_exact(header_len) -> header_bytes +header = dag_cbor.decode_strict(header_bytes) +require header.version == 1 +require not header.roots.is_empty() + +while not eof(): + read_varint() -> block_len + start = cursor + cid = read_cid() # advances cursor + data_len = block_len - (cursor - start) + read_exact(data_len) -> data + verify cid == dag_cbor_cid(data) # or raw_cid for blobs + store.put(cid, data) +``` + +After the loop, `store` has every block. You now do the tree walk (MST + commit) against the store. + +### Backpressure + +A producer (e.g. PDS `getRepo`) streams the CAR in chunks. Consumers should process blocks as they arrive and NOT wait for EOF before starting verification — for anything over a few MB, doing so is the difference between sub-second and tens-of-seconds-of-latency perceived by the user. + +### Partial / incremental CARs + +`com.atproto.sync.subscribeRepos` delivers event payloads whose CAR carries **only the changed blocks** since the previous commit, with the new commit as the root. The framing is identical to a full CAR; only the blocks included differ. A consumer maintaining a persistent block store MUSTN'T treat "block missing from this CAR" as an error — it's expected to be reused from the store. + +## 8. Writing a CAR + +To produce a CAR: + +1. Compute the header bytes: `to_dag_cbor({version: 1, roots: [root_cid]})`. +2. Emit `varint(len(header_bytes)) || header_bytes`. +3. For each block you want to include, once: + - Encode the CID as raw CID bytes (36 bytes for dag-cbor / raw DASL CIDs). + - Emit `varint(len(cid_bytes) + len(data_bytes)) || cid_bytes || data_bytes`. +4. Flush. Done. + +Two traps: + +- **Don't include a CID twice.** Consumers tolerate it but producers shouldn't emit it — track a `HashSet` while writing. +- **Don't include a block whose CID doesn't match its bytes.** That produces a CAR that fails verification on read but looks fine if you open it in a hex editor. + +## 9. AT Protocol's CAR endpoints + +| XRPC | What the CAR contains | +| -------------------------------------- | -------------------------------------------------------------------- | +| `com.atproto.sync.getRepo` | Full repo: signed commit + all MST nodes + all records + blob CIDs. | +| `com.atproto.sync.getBlocks` | Just the requested CIDs, in a CAR framed with the commit as root. | +| `com.atproto.sync.getLatestCommit` | Sometimes wrapped as a minimal CAR; sometimes a plain JSON response.| +| `com.atproto.sync.listBlobs` | Not a CAR — a JSON cursor. | +| `com.atproto.sync.subscribeRepos` (WS) | Each event payload is a tiny CAR with the new commit as root. | + +For firehose consumers, see the parallel spec material; the CAR rules are unchanged. + +## 10. Reference implementation + +In the `atproto-dasl` crate: + +- `src/car/reader.rs` — `CarReader::new(reader)` produces a stream of `(Cid, Bytes)` pairs after parsing the header. +- `src/car/writer.rs` — `CarWriter::new(writer, roots)` accepts blocks via `.write_block(cid, data)` and handles framing. +- `src/car/varint.rs` — unsigned LEB128 varint codec. + +Both reader and writer use `DrislStrict` by default for header encoding; the block payload is passed through opaquely because the CAR layer doesn't assume codec. + +## 11. Common CAR errors + +| Symptom | Likely cause | +| ------------------------------------------------ | ------------------------------------------------------------- | +| `UnsupportedVersion` | Header says `version != 1`. Reject. | +| `InvalidHeader` / "roots missing" | Header CBOR is not DRISL-strict or missing `roots`. | +| `CidMismatch` | Block payload's SHA-256 doesn't match the declared CID. | +| Decoder hangs reading varint | Malicious or buggy producer emitting continuation bits forever; cap at 10 bytes. | +| "Reader ran out of bytes mid-block" | Truncated CAR; the last varint overshoots available data. | +| Duplicate CID appears with different payload | Producer is broken. Reject, don't pick one. | +| Root CID has no matching block in the file | Malformed CAR; every root must be present. | diff --git a/skills/software-development/atproto-repository/references/shared/commit-and-signing.md b/skills/software-development/atproto-repository/references/shared/commit-and-signing.md new file mode 100644 index 0000000..f396443 --- /dev/null +++ b/skills/software-development/atproto-repository/references/shared/commit-and-signing.md @@ -0,0 +1,185 @@ +# Repo Commit & Signing (Reference) + +Source of truth: https://atproto.com/specs/repository. + +The commit is the root of a repository: a small DAG-CBOR record binding a DID, a revision, and the current MST root CID, sealed by a signature from the account's atproto signing key. Everything else in the repo is cryptographically anchored to the commit's CID. + +This reference covers the commit record layout, the exact bytes that get signed, how a verifier checks them, and how key rotation interacts with older commits. + +## 1. Record shape + +`Commit` is a DAG-CBOR map. Fields in bytewise key order (which is how DRISL serializes them): + +| Key | Sort bytes | Type | Required on wire | Notes | +| --------- | ------------------ | ------- | ---------------- | -------------------------------------------------------- | +| `data` | `0x64 0x61 0x74 0x61` | CID | yes | MST root CID. Must be a dag-cbor CID (codec `0x71`). | +| `did` | `0x64 0x69 0x64` | string | yes | Owner DID. Must start with `did:`. | +| `prev` | `0x70 0x72 0x65 0x76` | CID? | yes (per spec) | CID of the parent commit, or `null` for the genesis commit. See §1.1. | +| `rev` | `0x72 0x65 0x76` | string | yes | Monotonic revision, in TID form (13 chars base32-sortable). | +| `sig` | `0x73 0x69 0x67` | bytes | yes | Raw ECDSA signature over `UnsignedCommit` DAG-CBOR bytes. Omitted from `UnsignedCommit` when signing. | +| `version` | `0x76 …` | integer | yes | Exactly `3`. Versions 1 and 2 are historical; never emit them. | + +Key sort order: `data` (0x64…) < `did` (0x64…) < `prev` (0x70…) < `rev` (0x72…) < `sig` (0x73…) < `version` (0x76…). + +Canonical field-by-field comparison of `data` vs `did`: byte 1 both 0x64, byte 2 `a`=0x61 vs `i`=0x69 — `data` sorts first. Exact ordering matters: emitting `did` before `data` changes the commit CID and invalidates the signature. + +### 1.1. `prev` — null vs omitted + +Per spec, `prev` is a **required** field whose value is either a CID or `null`. The reference `atproto-repo` crate elides `prev` from the serialized output when it's `None` (uses `skip_serializing_if`). This is a spec-vs-impl divergence: + +- **Spec-strict**: the map always contains a `prev` key; genesis commits have `prev: null`. +- **Reference impl**: genesis commits omit the key entirely. + +The distinction matters because the signing bytes depend on exactly which map keys are present. A verifier that reconstructs `UnsignedCommit` from a received `Commit` must do so the same way the signer did — include or omit `prev` to match. In practice, consumers must be liberal: accept both the spec-strict form (`prev: null` present) and the reference impl form (key absent), and when verifying signatures, reconstruct the signing bytes by removing only the `sig` field without imposing any other structural change. + +If you are writing a verifier, **start from the raw commit block bytes**, strip the `sig` field, and hash the result. Do not re-encode from a struct, because doing so could round-trip `null` into absent (or vice versa) and break verification. + +## 2. `UnsignedCommit` — the signing surface + +To sign or verify, isolate the commit without its `sig`: + +| Key | Present in `UnsignedCommit` | +| --------- | --------------------------- | +| `data` | yes | +| `did` | yes | +| `prev` | yes (CID or null per spec; or absent per reference impl) | +| `rev` | yes | +| `sig` | **no** | +| `version` | yes | + +The **signing bytes** are the DAG-CBOR (DRISL) encoding of `UnsignedCommit`. Every rule in `drisl.md` applies: keys sorted bytewise, integers in shortest form, CIDs as tag 42 with identity multibase prefix, no indefinite-length framing. + +Reference implementation: `Commit::signing_bytes()` at `atproto-repo/src/repo/commit.rs:93`. + +### 2.1. Canonical genesis commit walkthrough + +Fields: + +- `did = "did:plc:ewvi7nxzyoun6zhxrhs64oiz"` +- `version = 3` +- `data = ` +- `rev = "3jzfcijpj2z2a"` (a TID; see `data-model.md` §3) +- `prev = null` (genesis) + +DAG-CBOR encoding (using spec-strict form with `prev: null`): + +``` +a5 ; map(5) — data, did, prev, rev, version + 64 64 61 74 61 ; "data" + d8 2a 58 25 00 <36 bytes> ; tag 42, bytes(37), + 63 64 69 64 ; "did" + 78 20 ; text(32) + 64 69 64 3a 70 6c 63 3a … ; "did:plc:ewvi7nxzyoun6zhxrhs64oiz" + 64 70 72 65 76 ; "prev" + f6 ; null + 63 72 65 76 ; "rev" + 6d ; text(13) + 33 6a 7a 66 63 69 6a 70 6a ; "3jzfcijpj2z2a" + 32 7a 32 61 + 67 76 65 72 73 69 6f 6e ; "version" + 03 ; unsigned(3) +``` + +Total length: 118 bytes. Hash these bytes with the signing key to produce `sig`. Byte-by-byte breakdown in `test-vectors.md` §4.1. + +The reference impl's emitted form omits `prev` entirely and produces a 4-entry map (`a4`) instead of `a5`, so the signing bytes and resulting signature differ from the spec-strict form. Any two implementations signing the same logical state must agree on which form they emit. + +## 3. Signing + +1. Build `UnsignedCommit` with all required fields populated. +2. Serialize to DAG-CBOR through a DRISL-strict encoder. +3. Compute the signature. AT Protocol requires **raw ECDSA** output: + - For k256 (secp256k1): 64 bytes, `r ‖ s` concatenation. Signer must use low-S canonical form (BIP-62); a high-S signature is a verification failure. + - For p256 (secp256r1): 64 bytes, same `r ‖ s`, same low-S requirement. + - DER-wrapped ECDSA (70+ bytes) is **not** acceptable. Unwrap it. + - ed25519 is reserved for future use — not widely deployed in PDS code today. +4. Attach the signature in the `sig` field; the commit is now complete. +5. Compute the commit's own CID (dag-cbor over the signed encoding). This CID is what goes into the next commit's `prev` and into the CAR header's `roots`. + +## 4. Verifying + +Given a signed commit `C`: + +1. **Isolate the signing bytes.** Strip the `sig` field from the commit's raw bytes — ideally without re-encoding. If you must re-encode, reconstruct `UnsignedCommit` in the exact form the signer emitted (see §1.1 on `prev`). +2. **Find the signing key.** Resolve `C.did` via `atproto-identity-resolution` to get its DID document. The active key is the `verificationMethod` entry whose `id` ends with `#atproto`, `type` is `"Multikey"`, and `controller` equals the DID. Decode `publicKeyMultibase` — the multibase-decoded bytes have a multicodec prefix (k256 = `0xe7 0x01`, p256 = `0x80 0x24`, followed by the compressed curve point). +3. **Verify.** Run ECDSA verification over the signing bytes with the recovered public key and the declared curve. Require low-S form. +4. **Record the verification result.** The reference crate exposes a `SignatureVerification { valid, signer_did, key_id }` struct (`atproto-repo/src/repo/commit.rs:225`), but note: **the reference crate does not implement end-to-end verification**. The struct is a future-proofing shape; actual verification is the caller's responsibility today. + +### 4.1. Timing of DID resolution + +The signing key in the DID document is the key that is currently live. If the account has rotated its key between when the commit was signed and when you try to verify, the live key will fail to verify an older commit. + +- For recent commits (within seconds to minutes), the live key is almost certainly the right one. +- For older commits, you must resolve the DID **as of `C.rev`**: + - For `did:plc`, fetch the PLC operation log (`GET https://plc.directory//log/audit`) and find the signing key that was active at the timestamp derivable from the TID in `C.rev`. + - For `did:web`, this isn't possible — `did:web` has no on-chain history. Commits older than the current key are unverifiable. + - For `did:webvh`, walk the verifiable history log to the state at `C.rev`. + +A verifier that returns "signature invalid" immediately on the first failed check risks false negatives for rotated accounts. A robust verifier retries against the historical key before giving up. + +## 5. The `rev` field + +`rev` is a TID — a 13-char base32-sortable string encoding a 53-bit microsecond timestamp plus a 10-bit clock identifier (64 bits total, top bit always 0). Requirements: + +- **Strictly monotonic**: `rev` of a commit must be bytewise-greater than the previous commit's `rev`. Since TIDs are sortable, bytewise greater-than corresponds to later-in-time. +- **Unique per PDS**: clock-id bits prevent collisions when a single PDS rapidly produces commits; TID generation is lockless. +- **Not meaningful as an absolute time** for external consumers — use it as an ordering key, not as a clock. Nothing stops a PDS from issuing a `rev` slightly in the future; and nothing requires the TID's wall-clock portion to match any particular clock. + +See `data-model.md` §3 for TID syntax. + +## 6. Chaining commits + +Commits form a linked list via `prev`. For commit `C_n`: + +- `prev` = CID of `C_{n-1}`, or `null` for `n = 0`. +- `rev > rev_{n-1}` (bytewise). +- `data` = root of the MST *after* the operation(s) that this commit represents. + +Firehose subscribers expect each new commit's `prev` to equal the previous commit's CID they saw — if it doesn't, there's been a gap (missed events, a rewound PDS, a replayed CAR). Consumers should treat `prev` mismatches as a hard sync error and re-fetch from the current `getLatestCommit`. + +## 7. Rotation — how it interacts with verification + +AT Protocol separates the *signing* key from the *rotation* keys: + +- The `#atproto` Multikey in the DID document is the **signing key** — the one that signs commits. +- Rotation keys live in the PLC operation log (for `did:plc`) and are used to authorize changes to the DID document itself, including rotations of the signing key. They do not sign commits directly. + +When a signing key rotates: + +- Older commits remain valid under the **old** signing key. Verifiers must be able to retrieve the historic key. +- New commits signed after rotation are verifiable under the **new** signing key. +- There's no retroactive re-signing — commits aren't touched during rotation. + +For `did:web`, key rotation requires rewriting the DID document served at `/.well-known/did.json`. Historic commits become unverifiable unless the verifier has cached the old key. + +## 8. The CAR that wraps a commit + +On the wire, a signed commit travels inside a CAR v1 file: + +- **Repo export**: `com.atproto.sync.getRepo` — CAR whose root is the latest commit's CID; blocks include the commit, every MST node, every record. +- **Firehose event**: `com.atproto.sync.subscribeRepos` — each `#commit` event's payload is a tiny CAR whose root is the new commit's CID and whose blocks are **only the blocks that changed** since the previous commit (the commit itself plus the minimal subtree needed to justify the new MST root, plus any new or changed records). Consumers combine these with a persistent block store. + +The CAR framing is identical to a full repo — see `car-v1.md`. A firehose consumer that assumes every block it needs is present in each event's CAR will break the moment a commit only touches already-known subtrees. + +## 9. Record-block side: what is "inside" the repo + +Beyond the commit and the MST nodes, the repo contains: + +- **Record blocks**: one per record, DAG-CBOR encoded, addressed by dag-cbor CID. +- **Blob CIDs**: records may reference blobs (images, video) via `$link` CIDs of codec `raw` (`0x55`). Blobs themselves are usually fetched separately (`com.atproto.sync.getBlob`) and are **not** part of the CAR export from `getRepo` — the CAR carries only the record that references the blob's CID, not the blob bytes. + +A CAR consumer that tries to look up every referenced CID inside the CAR will trip over blob CIDs; treat "referenced but not present in this CAR" as a signal to fetch externally, not as a protocol violation. + +## 10. Common errors + +| Symptom | Likely cause | +| ---------------------------------------------------- | ---------------------------------------------------------------------------------------------- | +| `UnsupportedCommitVersion { version: 2 }` | Legacy v2 commit from pre-release data. Reject for any verification; lenient reads may allow. | +| `InvalidDid` on `did` field | Commit's `did` doesn't start with `did:`. Likely truncated or corrupted. | +| `MissingCommitField { field: "sig" }` | `UnsignedCommit` was submitted as a `Commit` by mistake. Sign it first. | +| Signature verifies only after stripping `prev: null` | Signer omitted the `prev` key (reference-impl style). Reconstruct the signing bytes the same way. | +| Signature verifies against an older DID document | Account rotated keys. Re-resolve as of `commit.rev` and retry. | +| DER-encoded `sig` field | Producer emitted DER ECDSA. Unwrap to raw `r ‖ s`, 64 bytes, before verifying. | +| `rev` not greater than parent's `rev` | Broken monotonicity. A PDS clock skew or TID generator bug; reject commit. | +| `prev` doesn't match local head | Missed firehose events or forked state. Re-fetch `com.atproto.sync.getLatestCommit`. | +| Two `#atproto` verification methods in DID document | Spec says first match wins. The reference impl scans `verificationMethod` in order and picks the first entry whose `id` ends with `#atproto`, `type == "Multikey"`, and `controller` matches the DID; later entries are ignored. | diff --git a/skills/software-development/atproto-repository/references/shared/data-model.md b/skills/software-development/atproto-repository/references/shared/data-model.md new file mode 100644 index 0000000..9c9268c --- /dev/null +++ b/skills/software-development/atproto-repository/references/shared/data-model.md @@ -0,0 +1,211 @@ +# Data Model — Records, TIDs, AT-URIs (Reference) + +Sources of truth: https://atproto.com/specs/data-model, https://atproto.com/specs/at-uri-scheme, https://atproto.com/specs/tid. + +The repository is a map from keys (`/`) to records (DAG-CBOR maps). This file captures the on-disk shape of a record, the rules for collection NSIDs, the TID format used for most record keys, and the AT-URI scheme that names records externally. + +## 1. Record + +A record is a DAG-CBOR map encoded using DRISL rules. Every record must carry a `$type` field — the NSID of its lexicon — and may carry any other fields permitted by that lexicon. + +### 1.1. Required field: `$type` + +- `$type` = NSID (see §2). +- In DAG-CBOR, `$type` is a regular map key — it is not a tag. The leading `$` is lexicographically significant: `$` (0x24) sorts before any letter, so `$type` almost always appears as the first key of a record in canonical form. +- The value of `$type` must match the collection NSID the record lives in (the first segment of the MST key). A mismatch is a lexicon validation failure, not just a mis-categorization. + +### 1.2. Supported value types + +DAG-CBOR (and therefore the repo) supports: + +| CBOR value | DRISL treatment | Typical use in records | +| ------------- | ------------------------ | ----------------------------------------------------- | +| unsigned int | shortest form, major type 0 | counts, ages, timestamps in some lexicons | +| negative int | shortest form, major type 1 | e.g. `delta` fields, net-negative values | +| float64 | 64-bit, finite only | lat/long, sentiment score; NaN/Infinity forbidden | +| text string | UTF-8, DRISL-strict | human text, URIs | +| byte string | raw bytes | binary payloads — but usually blobs are referenced, not inlined | +| boolean | `0xf4` / `0xf5` | flags | +| null | `0xf6` | optional absence sentinel | +| array | ordered, heterogeneous | lists of references, facets, tags | +| map | keys sorted bytewise | nested records and objects | +| tag 42 (CID) | only allowed tag | links to other records, blobs, or embedded CIDs | + +CBOR tags other than 42 are forbidden (`drisl.md` §5, §8). Undefined, simple values, and big-integer extensions are forbidden. + +### 1.3. Blob references + +Records refer to binary attachments via **blob references** — not by inlining bytes. The canonical shape (lexicon type `blob`) in JSON is: + +```json +{ + "$type": "blob", + "ref": {"$link": "bafkrei..."}, + "mimeType": "image/png", + "size": 12345 +} +``` + +In DAG-CBOR (how the record actually sits in the repo), `ref` is a CID encoded as tag 42 pointing to a `raw` codec block (`0x55`), not a dag-cbor block. The blob bytes themselves travel via `com.atproto.sync.getBlob`, not as part of the repo CAR export. + +### 1.4. Links between records + +To reference another record by its CID, use a `$link` (JSON) / tag 42 (DAG-CBOR) wrapping a dag-cbor CID. Common shapes: + +``` +{ + "$type": "app.bsky.feed.like", + "subject": {"uri": "at://did:plc:…/app.bsky.feed.post/…", "cid": "bafyrei…"}, + "createdAt": "2024-01-01T00:00:00Z" +} +``` + +The `{uri, cid}` pair (called a `com.atproto.repo.strongRef`) is the idiomatic way to reference another record with tamper-evidence: the URI tells you where the record lives, the CID tells you exactly which version. + +### 1.5. Size limits + +The spec and reference implementations impose practical bounds: + +- A single record's encoded size should stay under a few hundred KB. Bluesky's PDS caps at 100 KB per record. +- The whole repo has no hard upper bound, but tree traversal cost is O(log N) in the number of records, so walking a 1M-record repo is still feasible. +- Blobs have their own size caps (typically 1 MB per blob for images) enforced by the PDS. + +## 2. NSID — the collection identifier + +An NSID (Namespaced Identifier) is a reverse-DNS-style string: `com.example.feature.subfeature`. Used for: + +- `$type` on records. +- The collection segment of an MST key (`app.bsky.feed.post` in `app.bsky.feed.post/3k2…`). +- The method name of an XRPC endpoint (`com.atproto.sync.getRepo`). +- Lexicon IDs. + +### 2.1. Syntax + +- Dot-separated segments, 2 or more segments total. +- Each segment: `[a-zA-Z][a-zA-Z0-9-]*`. Digits or hyphens in the first position of any segment are forbidden. +- Hyphens are allowed, but only within a segment — never at the start or end of a segment. +- Total length ≤ 317 characters; each segment ≤ 63 characters; the final segment ≤ 63 characters but further restricted to `[a-zA-Z]` (no digits, no hyphens). This last restriction keeps NSIDs unambiguous when concatenated with rkeys. +- Case is preserved and meaningful: `app.bsky.feed.Post` is a different NSID than `app.bsky.feed.post` and neither lexicon ecosystem will cross-reference them. In practice, all established NSIDs are lowercase except the final segment, which is `camelCase`. + +### 2.2. Examples + +- Valid: `com.atproto.repo.putRecord`, `app.bsky.feed.post`, `app.bsky.feed.like`, `com.example.some-app.record` +- Invalid: `.com.example` (leading dot), `com..example` (empty segment), `1com.example` (segment starts with digit), `com.example-` (segment ends with hyphen), `com.example.1record` (final segment not `[a-zA-Z]`-only), `com` (single segment). + +### 2.3. Ownership + +NSIDs are namespaced by DNS ownership. Registering `com.yourcompany.feature` is conceptually like registering a DNS name — the tree of lexicons under `com.yourcompany` is yours to define. The spec doesn't enforce this cryptographically; it's social convention backed by the AppView ecosystem. + +## 3. TID — the default record key + +TIDs are timestamp-based identifiers used as record keys when the lexicon doesn't specify a fixed key. + +### 3.1. Format + +- **Exactly 13 ASCII characters**. +- **Alphabet**: `234567abcdefghijklmnopqrstuvwxyz` (base32-sortable). No `0`, `1`, `8`, `9`, no uppercase. +- **Structure** (64-bit big-endian integer, encoded 5 bits per char, highest-bit-first): + - Bit 63: always `0`. In the base32-sortable encoding this restricts the **first character** to `234567abcdefghij` (values 0–15 in the alphabet). + - Bits 62–10: 53-bit microsecond timestamp (UNIX epoch). + - Bits 9–0: 10-bit clock identifier (random or sequentially allocated per-PDS to avoid collisions within a single microsecond). + +### 3.2. Sortability + +Because characters in `234567abcdefghijklmnopqrstuvwxyz` sort bytewise in the same order as their 5-bit values, and the encoding is big-endian, TID strings sort *as strings* in the same order as their underlying 64-bit integers — which is the same order as their timestamps. That's the whole point: the PDS can emit a new TID for each new record and rely on the MST's bytewise key order to put newer records after older ones. + +### 3.3. Generation rules + +- Must be strictly greater than the previous TID the PDS has emitted for this account (monotonicity). +- When the system clock moves backwards, advance to `previous_timestamp + 1` rather than regress. +- When two TIDs would share a timestamp, bump the clock identifier (or use a random clock ID). +- Clock identifier width (10 bits = 1024 values) supports up to 1024 TIDs per microsecond per PDS — far above any realistic record-creation rate. + +### 3.4. Parsing and validation + +A consumer receiving a TID string should: + +- Check length is exactly 13. +- Check every character is in the base32-sortable alphabet. +- Check the first character is in `234567abcdefghij` (the top bit must be 0). +- Optionally decode to `(timestamp_us, clock_id)` for diagnostic purposes. + +The wall-clock time is not to be trusted — a PDS can emit a TID slightly in the future, and clock drift is real. Treat TID as an ordering key first, a timestamp second. + +### 3.5. Examples + +- Valid: `3jzfcijpj2z2a`, `7777777777777`, `2222222222222` +- Invalid: `3jzfcijpj2z2` (12 chars), `3jzfcijpj2z2aa` (14 chars), `AAAAAAAAAAAAA` (uppercase not in alphabet), `zzzzzzzzzzzzz` (decodes with top bit set; first char must be `2-7` or `a-j`). + +## 4. Record key (`rkey`) + +The rkey is the second half of an MST key. Rules: + +- Most collections use TIDs as rkeys (posts, likes, reposts, follows). +- Some lexicons specify a fixed rkey (`app.bsky.actor.profile` always uses `rkey = "self"`; `app.bsky.feed.generator` uses a custom name). +- Lexicons may accept an open alphabet; the general-purpose rkey rule is: `[A-Za-z0-9._~:-]{1,512}` with no slashes, no `%`, no spaces. +- Two special sentinels: `self` (singleton records) and explicit rkeys chosen by the lexicon. +- **Uniqueness**: `(collection, rkey)` pairs must be unique within a repo. Inserting a record at an existing key overwrites (the MST records a new `v` CID). + +## 5. AT-URI + +AT-URIs name records externally. The syntax is `at:////[/]`, though for record addressing only the three-segment form is meaningful. + +### 5.1. Authority + +- **In stored data and in repo exports, the authority is always a DID.** A record stored at `at://did:plc:xxx/app.bsky.feed.post/3k…` is unambiguous because the DID is the account's permanent identifier. +- In URIs that appear inside records (a post referencing another post's URI), either a DID or a handle is accepted by parsers at read time, but the canonical stored form is a DID. A handle-authority URI must be resolved to a DID-authority URI before indexing. +- **Handles in URIs are rejected by the reference `ATURI::from_str()` parser** — see `atproto-record/src/aturi.rs:90`. That's a deliberate choice: the parser's contract is that it returns a DID. Code that accepts handles does so at a higher layer, before handing the DID-form URI to the parser. + +Supported DID methods in the authority position: `did:plc`, `did:web`, `did:webvh`. `did:key` and all other methods are rejected. + +### 5.2. Collection + +The collection segment is an NSID (see §2). Its rules apply unchanged inside an AT-URI. + +### 5.3. Record key + +The rkey segment is the record key as described in §4. No percent-encoding is applied — rkeys use only URL-safe characters anyway. + +### 5.4. Fragments and extra path components + +- Extra path segments after the rkey are permitted by the scheme but not meaningful for record addressing. The reference parser silently ignores them: `at://did:…/app.bsky.feed.post/3k…/extra/path` gives the same three-field result as without `/extra/path`. +- Trailing slashes are rejected by the reference parser. +- Fragments (`#…`) are used by some URIs to point at sub-fields (like `#atproto` for a DID document's signing key), but `ATURI::from_str` does not parse them; they're treated as part of the rkey if present, which almost certainly isn't what the producer intended. + +### 5.5. Examples + +- Valid: `at://did:plc:ewvi7nxzyoun6zhxrhs64oiz/app.bsky.feed.post/3jui7kp54ic2i` +- Valid: `at://did:web:example.com/app.bsky.feed.post/3jui7kp54ic2i` +- Valid: `at://did:web:example.com:8080:tenant:path/app.bsky.feed.like/3k2akjh32kj` (non-strict did:web path form, reference-impl-accepted) +- Valid: `at://did:plc:abcdefghijklmnopqrstuvwx/b/c` (minimal three-segment form) +- Rejected: `at://alice.bsky.social/app.bsky.feed.post/3k…` — handle authority. Resolve first. +- Rejected: `at://did:key:z…/…` — unsupported DID method. +- Rejected: `did:plc:…/app.bsky.feed.post/3k…` — missing `at://` prefix. +- Rejected: `at://did:plc:xxx/app.bsky.feed.post/3k…/` — trailing slash. +- Rejected: `at://did:plc:xxx/` — missing collection. +- Rejected: `at://did:plc:xxx/app.bsky.feed.post` — missing rkey. +- Rejected: `at://did:plc:xxx//3k…` — empty collection. + +## 6. Putting it together + +An AT-URI like `at://did:plc:ewvi7nxzyoun6zhxrhs64oiz/app.bsky.feed.post/3jzfcijpj2z2a` corresponds to: + +- **DID** `did:plc:ewvi7nxzyoun6zhxrhs64oiz` — resolve via `atproto-identity-resolution` to get the PDS endpoint and signing key. +- **MST key** `app.bsky.feed.post/3jzfcijpj2z2a` — the path within the repo's MST. +- **Record** at that key — a DAG-CBOR map with `$type = "app.bsky.feed.post"` and whatever `app.bsky.feed.post` lexicon fields the post carries. + +To fetch the record: `com.atproto.repo.getRecord?repo=&collection=&rkey=`. To fetch just its bytes by CID: `com.atproto.sync.getBlocks?did=&cids=`. To fetch the whole repo: `com.atproto.sync.getRepo?did=` — returns a CAR. + +## 7. Common errors + +| Symptom | Likely cause | +| ------------------------------------------------------- | ------------------------------------------------------------------------------------- | +| `$type` value doesn't match the collection | Record stored in the wrong collection, or `$type` hand-edited. Lexicon validation fails. | +| "missing `$type`" when decoding a record | Upstream put a partial update into the repo or stripped `$type` for JSON display. The `$type` is non-negotiable in the stored form. | +| `ATURI::HandleNotSupported` | Caller passed a handle-authority URI to `ATURI::from_str`. Resolve the handle to a DID first (`atproto-identity-resolution`). | +| `TidError::InvalidLength` | TID has the wrong number of characters. Regenerate. | +| `TidError::InvalidCharacter` | Typo or encoded in the wrong alphabet (base32 != base32-sortable). | +| `TidError::InvalidFormat "Top bit must be 0"` | First char outside `234567abcdefghij` — the top bit of the 64-bit integer is set. | +| NSID rejected with "digit in first position of segment" | `1record`, `0abc`; segments must start with a letter. | +| Records appear in the wrong order when iterating | Comparing keys as Unicode strings instead of bytewise. Bytewise over UTF-8 is spec. | +| A record has `null` for `$type` | Producer bug — `$type` is required. Reject at lexicon validation step. | diff --git a/skills/software-development/atproto-repository/references/shared/divergence-matrix.md b/skills/software-development/atproto-repository/references/shared/divergence-matrix.md new file mode 100644 index 0000000..3cab489 --- /dev/null +++ b/skills/software-development/atproto-repository/references/shared/divergence-matrix.md @@ -0,0 +1,145 @@ +# Cross-Language Divergence Matrix (ATProto Repository) + +Language-neutral. Captures the real behavioural differences between the Rust (`atproto-repo` + `atproto-dasl` + `atproto-record`), TypeScript (`@atproto/repo`), and Go (`indigo/atproto/repo`) stacks that anyone porting code or operating cross-stack needs to know about. + +Every per-language file (`rust/*.md`, `typescript/*.md`, `go/*.md`) links back here instead of restating the matrix. + +## Library map + +| Layer | Rust | TypeScript | Go | +| ----------------- | -------------------------------------- | ------------------------------------- | ------------------------------------------- | +| Canonical DAG-CBOR | `atproto-dasl` (DRISL-strict) | `@atproto/lex-cbor` | `cbor-gen` typed + `atproto/atdata` generic | +| CID | `atproto-dasl::Cid` (DASL-strict) | `@atproto/lex-data` → `Cid` class | `github.com/ipfs/go-cid` | +| CAR | `atproto-dasl::{CarReader,CarWriter}` | `@atproto/repo/car.ts` (reader + writer) | `github.com/ipld/go-car` + `atproto/repo/car.go` (reader only) | +| MST | `atproto-repo::Mst` | `@atproto/repo::MST` | `atproto/repo/mst::Tree` | +| Commit | `atproto-repo::{Commit, UnsignedCommit}` | `@atproto/repo::Commit` | `atproto/repo::Commit` | +| Signing/verify | **Caller-owned** — k256/p256 crates | `signCommit`, `verifyCommitSig` — caller resolves didKey | **Fully wired**: `VerifyCommitSignatureFromCar` resolves DID → key → verify | +| Identity wiring | `atproto-identity` (separate crate) | `@atproto/identity` (separate package) | `atproto/identity.Directory` (plumbed into repo package) | +| Record types | `atproto-record` (separate crate) | Lex schemas in individual packages | `atproto/atdata` + typed client packages | + +The shape of the trade-off: **Go ships the most end-to-end, Rust ships the most primitive, TS sits in the middle.** Go's `VerifyCommitSignatureFromCar` will resolve the DID, pull the signing key, and check the signature in one call. Rust gives you `Commit::signing_bytes()` and expects you to bring your own everything. TS ships `verifyRepo(carBytes, did, didKey)` where the didKey is already resolved. + +--- + +## §drisl — canonical DAG-CBOR + +| Aspect | Rust (`atproto-dasl`) | TypeScript (`@atproto/lex-cbor`) | Go (`cbor-gen` + `atdata`) | +| ----------------------------------- | -------------------------------------- | ----------------------------------- | --------------------------------------------- | +| Map key sort on encode | Bytewise | Bytewise | **Struct declaration order** (cbor-gen quirk) | +| Strict canonical decode | No (permissive like go-ipld-cbor) | No | No | +| CID in memory | `atproto_dasl::Cid` (DASL-strict) | `Cid` class | `cid.Cid` or `atdata.CIDLink` wrapper | +| Typed bytes wrapper | `atproto_dasl::Bytes` | Plain `Uint8Array` | `atdata.Bytes` | +| Blob wrapper | `atproto_record::Blob` | Plain object (`{$type: "blob", ref, mimeType, size}`) | `atdata.Blob` | +| Size limits enforced at CBOR layer | Configurable in `atproto-dasl` | **None** — enforced at PDS/XRPC only | Hard-coded in `atdata/const.go` (1 MiB record, 128k container, 1 MiB string, 8 KiB key) | +| Generic `Value`-style path | `atproto_dasl::Value` | `LexValue` (from `@atproto/lex-data`) | `atdata.UnmarshalCBOR` → `map[string]any` | + +**Practical bug one**: Go's cbor-gen sorts by **struct declaration order**, not bytewise. The indigo `Commit` struct happens to have fields in an order that produces canonical output, but this is a coincidence — add a cbor-gen struct yourself and if the declaration order doesn't match bytewise, your output is non-canonical and the CID won't match other implementations. + +**Practical bug two**: Size limits aren't consistently enforced at the CBOR layer. Go rejects >1 MiB records at `atdata.UnmarshalCBOR`; TS and Rust don't. If you're reading potentially-adversarial CAR input in TS or Rust, wrap the input with a size cap before decoding. + +--- + +## §car — CAR v1 framing + +| Aspect | Rust | TypeScript | Go | +| ----------------------------------- | -------------------------------------- | ---------------------------------- | --------------------------------------------- | +| Reader | `atproto-dasl::CarReader` | `readCar` / `readCarReader` / `readCarStream` | `repo.LoadRepoFromCAR` / `LoadCommitFromCAR` + `go-car` directly | +| Writer | `atproto-dasl::CarWriter` | `writeCarStream` / `blocksToCarFile` / `blocksToCarStream` | **None shipped** — use `go-car` directly | +| CID verification on ingest | **Off** by default | **On** by default (`verifyIncomingCarBlocks`); toggle via `skipCidVerification: true` | **Off** (`// TODO: not verifying CID` in `repo.go:65`) | +| Blockstore surface | `MemoryStorage`, `SpillableBuffer` for on-disk spill | `BlockMap` (unbounded map), `MemoryBlockstore`, `SyncStorage` (composed) | `TinyBlockstore` (unbounded `map[string]blocks.Block`); swap via `RepoBlockSource` | +| Streaming support | `CarReader` is streaming; `SpillableBuffer` for memory cap | `readCarStream` streaming; `BlockMap` in-memory only | `go-car.NewCarReader` streaming; `TinyBlockstore` in-memory | +| Block framing (on the wire) | `varint(cid_len + data_len) ‖ cid ‖ bytes`, 36-byte CID | Same | Same | + +**Practical bug one**: **TS verifies CIDs on ingest. Go and Rust don't.** A CAR that passes through Go or Rust unmodified may carry corrupted blocks; TS will reject it. For trusted internal transport, TS's verification is extra CPU you can skip with `skipCidVerification: true`. For untrusted input, TS is the only implementation that protects you by default. + +**Practical bug two**: Go has no built-in CAR writer. Reaching for "how do I write a CAR in indigo" leads you to `go-car` directly. Rust (`atproto-dasl::CarWriter`) and TS (`writeCarStream` / `blocksToCarFile`) ship writers. + +**Practical bug three**: `TinyBlockstore` (Go) and `BlockMap` (TS) are **unbounded in-memory maps**. Large repo exports OOM. Rust's `SpillableBuffer` has on-disk spillover for large inputs; Go and TS leave it to the caller to implement. + +--- + +## §mst — Merkle Search Tree + +| Aspect | Rust (`atproto_repo::Mst`) | TypeScript (`MST`) | Go (`mst.Tree`) | +| ----------------------------------- | -------------------------------------- | ----------------------------------- | --------------------------------------------- | +| Mutability | Mutable in place | **Immutable** — `add`/`update`/`delete` return new `MST` | Mutable in place | +| Key validation | `MAX_KEY_BYTES = 1024`, character set | **Stricter**: `/` shape enforced, char set `[a-zA-Z0-9_~\-:.]` | `MAX_KEY_BYTES = 1024`, character set | +| Node on wire | `{l, e: [{p, k, v, t}]}` | Same | Same | +| Height (layer) computation | SHA-256 leading zero-bit pairs (fanout 4) | Same | Same (despite `// fanout: 16` comment — misleading) | +| Previous-value semantics on write | Returns `Option` on `insert`/`remove`/`delete` | Returns new `MST` (no prev value) | Returns `(prevValue, error)` on `Insert`/`Remove` | +| Partial tree semantics | `is_partial()` method; ops on missing blocks error gracefully | `MissingBlockError` thrown — no soft mode | `IsPartial()`, `Stub` flag; `ErrPartialTree` sentinel — expected for firehose | +| Cross-height insert (same call) | **Not supported** — bottom-up only | Supported (recursion handles splits) | Supported (recursion handles splits) | +| Diff API | `diff_entries` → `MstDiff::{Add,Update,Delete}` | `DataDiff.of(newTree, oldTree)` → `addList/updateList/deleteList` | `WriteToMap` + manual comparison (no first-party flat diff) | +| Structural verify | `Mst::verify` | **Not shipped** — invariants maintained by construction | `Tree.Verify` | + +**Practical bug one**: **Rust's `insert_recursive` can't cross heights in a single call.** If you're building a large tree, you must bottom-up from the leaves, not top-down from the root. TS and Go handle cross-height splits internally. + +**Practical bug two**: **TS's MST is immutable.** `let tree = await MST.create(); tree.add(k, v)` silently discards the result — you need `tree = await tree.add(k, v)`. Rust and Go mutate in place, so this pattern is foreign to TS callers. + +**Practical bug three**: Partial trees are **expected** in Go (firehose events return them normally) but **exceptional** in TS (a `MissingBlockError` throws). If you're porting Go firehose code to TS, wrap subtree traversals in try/catch or use `SyncStorage` that falls through to a prior store. + +**Practical bug four**: Go's `HeightForKey` comment says `// fanout: 16` but the algorithm counts pairs of zero bits (fanout 4). Don't trust the comment. All three implementations agree on fanout 4; cross-implementation trees interop correctly. + +--- + +## §commit — commit record and signatures + +| Aspect | Rust | TypeScript | Go | +| ----------------------------------- | -------------------------------------- | ---------------------------------- | --------------------------------------------- | +| `prev` for genesis commit | **Omitted** (`#[serde(skip_serializing_if)]`) — 4-entry map `a4` | Always present as null — 5-entry map `a5` | Always present as null — 5-entry map `a5` (comment at `commit.go:18` explicit) | +| Spec conformance | Reference-impl divergent from spec | Spec-strict | Spec-strict | +| Signing bytes API | `UnsignedCommit::signing_bytes()` | `cbor.encode(unsigned)` inside `signCommit` | `Commit::UnsignedBytes()` (re-marshals through cbor-gen) | +| Signature format | Caller-supplied (k256/p256 raw r‖s, low-S) | `Keypair.sign()` from `@atproto/crypto` — raw r‖s, low-S | `atcrypto.PrivateKey.HashAndSign` — raw r‖s, low-S | +| Signature verification API | **Caller-owned** — resolve DID + verify yourself | `verifyCommitSig(commit, didKey)` — caller resolves didKey | `commit.VerifySignature(pubkey)` + `VerifyCommitSignatureFromCar(ctx, dir, carBytes)` — fully wired | +| Inductive firehose verification | Not provided | Not provided (use `verifyDiff` + manual op compare) | `VerifyCommitMessage` — op inversion + compare to `prevData` | +| Rotation handling helpers | None | None | None | +| Version-2 legacy support | Not in `Commit` type | `LegacyV2Commit` + `ensureV3Commit` upgrade | Validates `Version == 3` only, no upgrade helper | + +**Practical bug one** — the big one: **`prev` is sometimes omitted, sometimes null.** A Rust reference-impl genesis commit has 4 fields on the wire (`prev` skipped). Go and TS always serialize `prev: null` (5 fields). **A Rust-signed genesis commit won't verify if you re-marshal it through Go's `UnsignedBytes()` or TS's `signCommit` encoder.** If you see signatures fail on genesis commits only, suspect this. The workaround is to verify against the raw commit block bytes from the CAR — strip the `sig` field without re-encoding the others — but TS's `verifyCommitSig` and Go's `commit.VerifySignature` both re-marshal, so they don't expose a raw-bytes verification path. You have to build it manually. + +See `shared/commit-and-signing.md` §1.1 for the full treatment. + +**Practical bug two**: **Go has a fully wired `VerifyCommitSignatureFromCar`; TS and Rust don't.** Go takes a CAR, resolves the DID via `identity.Directory`, pulls the signing key, and verifies — single call. TS takes a pre-resolved `didKey: string`. Rust expects you to do all of it yourself. When porting Go verification code, you'll need to add DID resolution as a separate step in TS / Rust. + +**Practical bug three**: **None of the implementations handle key rotation automatically.** All three return "signature invalid" when a commit was signed under a historical key. You must resolve the PLC operation log, find the key that was active at `commit.rev`, and retry verification. For `did:web` this is not recoverable — rotated accounts lose verifiability of old commits. + +**Practical bug four**: **TS upgrades v2 commits transparently via `ensureV3Commit`**; Go rejects anything but v3; Rust doesn't ship a `LegacyV2Commit` type. If you're reading ancient CARs, only TS copes out of the box. + +--- + +## §validation — what each implementation checks + +| Check | Rust | TypeScript | Go | +| ---------------------------------------- | --------------------------------------- | ------------------------------------- | -------------------------------------- | +| CID → content matches on CAR read | No (caller's job) | **Yes** by default | No (`// TODO: not verifying CID`) | +| Commit structure (version, DID, sig, rev) | `commit.validate()` | zod schema on decode | `commit.VerifyStructure()` | +| MST key validity | Char set + length | Char set + **`/` shape** | Char set + length | +| MST structural invariants (heights, order) | `Mst::verify` | Not shipped | `Tree.Verify` | +| Commit signature | Caller | `verifyCommitSig(commit, didKey)` | `commit.VerifySignature(pubkey)` | +| Commit signature wired to DID resolution | Caller | Caller resolves didKey first | **`VerifyCommitSignatureFromCar`** — wired | +| MST → record block presence | Caller | `verifyDiff({ ensureLeaves: true })` | Implicit via `GetRecordBytes` miss | +| Ops match committed MST | Caller | `verifyDiff` + manual op-vs-diff comparison | `VerifyCommitMessage` (full inductive) | + +**Practical takeaway**: if you need end-to-end verification of an incoming firehose event and you can only use one implementation — Go. `VerifyCommitSignature` + `VerifyCommitMessage` together cover every check that matters (CID integrity is the one gap). TS gets most of the way with `verifyDiffCar` but lacks the op-inversion check against `prevData`. + +--- + +## When porting, the order of surprises + +1. **`prev: null` vs absent** — trips everyone. Always check this first when sigs fail. +2. **TS MST is immutable** — `tree.add(k, v)` returns a new tree; you must reassign. +3. **Go cbor-gen sorts by declaration order** — if you add a struct, audit the field order. +4. **TS verifies CIDs on CAR ingest** — toggle off with `skipCidVerification` for speed or trusted input. +5. **Rust MST insertions can't cross heights** — build bottom-up, not top-down. +6. **Partial trees are normal in Go/Rust, exceptional in TS** — catch `MissingBlockError`. +7. **Go has `VerifyCommitSignatureFromCar`; others don't** — port adds DID resolution as a distinct step. +8. **Size limits at the CBOR layer are Go-only** — TS and Rust rely on higher layers. + +## Related + +- `shared/drisl.md` — normative canonical DAG-CBOR rules. +- `shared/car-v1.md` — normative CAR v1 framing. +- `shared/mst.md` — normative MST algorithm. +- `shared/commit-and-signing.md` — normative commit + signing rules, including §1.1 on `prev`. +- `shared/test-vectors.md` — fixtures for cross-language agreement testing. +- `{rust,typescript,go}/{README,drisl,car,mst,commit}.md` — per-language detail with back-references to this matrix. diff --git a/skills/software-development/atproto-repository/references/shared/drisl.md b/skills/software-development/atproto-repository/references/shared/drisl.md new file mode 100644 index 0000000..4cf75cc --- /dev/null +++ b/skills/software-development/atproto-repository/references/shared/drisl.md @@ -0,0 +1,127 @@ +# DRISL — Deterministic DAG-CBOR (Reference) + +Source of truth: https://dasl.ing/drisl.html + +DRISL is a named subset of DAG-CBOR: CBOR restricted so that any given value has **exactly one** valid encoding. Every block in an AT Protocol repository — every record, every MST node, every commit — is DRISL-encoded. Determinism is not optional; a non-canonical encoding will hash to a different CID and the repo will fail to verify. + +This file states DRISL's rules. The rest of this skill assumes a DRISL-strict encoder and a strict decoder are in place. + +## 1. The rules at a glance + +1. Map keys are sorted **bytewise lexicographically** (not by codepoint, not by length-first). +2. Map keys are **text strings** (CBOR major type 3). Integer or byte-string keys are forbidden. +3. Integers use **shortest encoding**: the smallest of the five CBOR widths (immediate 0–23, 1-byte, 2-byte, 4-byte, 8-byte) that holds the value. +4. Lengths on arrays, maps, and strings use **shortest encoding**, same rule. +5. **No indefinite-length items** — no `0x5f/0x7f/0x9f/0xbf` framed values. +6. Floats are **64-bit only** (major type 7, additional info 27). 16-bit and 32-bit floats are forbidden. +7. **No NaN, no Infinity**. Encoders must reject them before writing; decoders must reject them on read. +8. **The only CBOR tag allowed is tag 42** (CIDs). Every other tag is a decoder error. +9. Duplicate map keys are forbidden. +10. Trailing data after the top-level value is forbidden — a DRISL payload is exactly one value. + +A single bit that breaks any of these rules changes the resulting CID. + +## 2. Map key ordering in practice + +Sort keys by comparing their **UTF-8 byte representation** byte-by-byte, not codepoint by codepoint, not length-first. For ASCII keys this is equivalent to lexicographic sort. For non-ASCII keys it means higher-bit sequences sort differently than a naive code-point sort would suggest. + +Example set of keys and their correct order: + +``` +$type (0x24 0x74 0x79 0x70 0x65) +_meta (0x5f 0x6d 0x65 0x74 0x61) +author (0x61 0x75 0x74 0x68 0x6f 0x72) +createdAt (0x63 0x72 0x65 0x61 0x74 0x65 0x64 0x41 0x74) +subject (0x73 0x75 0x62 0x6a 0x65 0x63 0x74) +``` + +`$` (0x24) sorts before `_` (0x5f), which sorts before lowercase letters. A common bug is alphabetising as if `$type` came after `author`; if you see `$type` in any position other than first in your records' map keys, your encoder is non-canonical. + +## 3. Integer shortest form + +CBOR integers use major type 0 (unsigned) or major type 1 (negative). The shortest-encoding rule says: pick the narrowest "additional info" that represents the value. + +| Value | Additional info byte | Follow-up bytes | +| ------------------------------ | -------------------- | --------------- | +| 0–23 | the value itself (0x00–0x17 for positive) | none | +| 24–255 | 0x18 | 1 byte | +| 256–65535 | 0x19 | 2 bytes | +| 65536–4294967295 | 0x1a | 4 bytes | +| 4294967296–2⁶⁴−1 | 0x1b | 8 bytes | + +The exact same rule applies to negative integers under major type 1, and to the length prefix on strings, byte strings, arrays, and maps. + +**Non-canonical example**: encoding the value `5` with an 8-byte length: `1b 00 00 00 00 00 00 00 05`. The canonical form is just `05`. A strict decoder must reject the 8-byte form. + +## 4. Float handling + +Every floating-point value in DRISL is encoded as a 64-bit IEEE 754 double (8 bytes after the `0xfb` prefix). The canonical form of an integer value that also fits in an integer (e.g., `1.0` vs `1`) is the integer form — but in practice, record schemas explicitly pick one or the other; don't round-trip a JSON `1` through a float intermediate. + +NaN, +∞, −∞ are **encode-time and decode-time errors**. If your data model must represent "no value", use CBOR null (`0xf6`) or omit the field entirely. + +## 5. CIDs as tag 42 + +A CID inside DRISL is encoded as: + +``` +d8 2a ; tag 42 +58 ; byte string, shortest length form +00 ; identity multibase prefix (REQUIRED) +<36 bytes> ; raw CID bytes (CIDv1, 4-byte header + 32-byte digest) +``` + +Full rules are in the `atproto-cid` skill. Two rules matter here for determinism: + +- The inner byte string is always 37 bytes (identity prefix + 36-byte CID). Its length prefix is `58 25` — single-byte length = 37. Non-canonical length encodings (like `59 00 25` or indefinite `5f … ff`) must be rejected. +- The identity multibase prefix byte `0x00` is **required**. A decoder that sees a CID byte string whose first byte is anything else must reject. + +Every other CBOR tag — 0, 1, 2, 3, 21–24, 32, 55799, anything — is a hard error. + +## 6. What a strict decoder checks + +On every value it reads, a DRISL-strict decoder must verify all of: + +- **Additional info is shortest** for the value's width. +- **Length prefixes are shortest** for strings, byte strings, arrays, maps. +- **Map keys are strings** and appear in sorted order with no duplicates. +- **No indefinite-length framing**. +- **No forbidden tags** (only 42). +- **Floats are 64-bit and finite**. +- **Trailing data**: exactly one top-level value consumes the entire buffer. + +Any violation is a decode error. The whole block is rejected — there's no "skip and continue". Partial decodes leave the reader in a broken state; restart from the next block. + +## 7. Lenient / non-strict mode + +Non-strict decoding exists mainly to pull records out of legacy data that predates strict enforcement. It relaxes: + +- Non-shortest integer/length encodings are accepted but produce a `NonCanonicalEncoding` warning. +- Duplicate map keys are tolerated (last write wins). +- Unsorted map keys are tolerated. + +Non-strict mode **must never** be used to produce repo blocks or to verify signatures. The CID of a lenient-decoded block is not what a strict encoder would produce; re-serializing produces a different CID. Treat the lenient path as one-way: read only, do not write. + +## 8. Why this matters + +Two implications that trip up every new implementer: + +1. **The order fields are listed in your source code does not matter.** JSON serializers sometimes preserve insertion order; DRISL ignores that and imposes byte order on keys. Build your encoder around `Map` with a sort step, not `struct` field order. +2. **Re-encoding a decoded value must yield the exact same bytes.** If round-tripping changes the bytes, your encoder is non-canonical somewhere — the most common culprits are integer widening, float canonicalization, and map-key sort. + +## 9. Reference encoder behavior + +The Rust implementation (`atproto-dasl::drisl`) enforces strict mode by default: + +- `to_vec(value)` — DRISL-strict encode. Panics on NaN/Infinity by design. +- `from_slice(bytes)` — DRISL-strict decode. Returns `DecodeError::NonCanonicalEncoding` / `MapKeysNotSorted` / `UnsupportedTag` on violations. +- `from_slice_non_strict(bytes)` — lenient decode for legacy reads only. + +Key file paths: + +- `crates/atproto-dasl/src/drisl/mod.rs` — public API. +- `crates/atproto-dasl/src/drisl/cbor/encode.rs` — the shortest-form logic. +- `crates/atproto-dasl/src/drisl/ser/serializer.rs` — the map sort / buffer logic. +- `crates/atproto-dasl/src/drisl/cbor/decode.rs` — strict decode checks. +- `crates/atproto-dasl/src/drisl/config.rs` — strict vs non-strict flags. + +A handy invariant: if you can round-trip a value (decode strict → re-encode strict → byte-equal), your encoder is DRISL-conformant for that value. Add that as a fuzz target. diff --git a/skills/software-development/atproto-repository/references/shared/mst.md b/skills/software-development/atproto-repository/references/shared/mst.md new file mode 100644 index 0000000..339f8cb --- /dev/null +++ b/skills/software-development/atproto-repository/references/shared/mst.md @@ -0,0 +1,253 @@ +# Merkle Search Tree (Reference) + +Source of truth: https://atproto.com/specs/repository (§ "Merkle Search Tree"). + +An AT Protocol repository is a single Merkle Search Tree (MST) mapping string keys to record CIDs. The MST gives the repo three properties at once: ordered iteration, deterministic content addressing (each node has a stable CID), and cheap sync (unchanged subtrees share structure across commits, so deltas hit only the changed paths). + +This reference file covers the node format, the key height rule, the placement invariants, prefix compression, and tree diffing. The companion file `car-v1.md` covers how these nodes travel on the wire; `commit-and-signing.md` covers the commit that seals the root CID. + +## 1. Key shape + +Every key in the tree is exactly `/` where: + +- `` is an NSID (`app.bsky.feed.post`, `com.atproto.repo.strongRef`). +- `` is a TID (13-char base32-sortable) or a lexicon-permitted custom key (`self`, `[a-zA-Z0-9._~:-]{1,512}`). + +Keys are compared **bytewise** (not by code-point or numeric TID value). Bytewise UTF-8 comparison over these characters is identical to ASCII order — no surprises — but the invariant is a bytewise sort, not a Unicode sort. + +Full key rules live in `data-model.md`. + +## 2. Key height — fanout 4 + +The *height* of a key determines which tree level it lives on. Pseudo-code: + +``` +height(key) = leading_zero_bits(SHA-256(key_utf8)) / 2 +``` + +Dividing by 2 produces a branching factor of 4 — each layer holds roughly one quarter of the keys that the layer below has. Distribution: + +| Height | Probability | Cumulative | +| ------ | ------------ | ----------- | +| 0 | ~75% | 75% | +| 1 | ~18.75% | ~93.75% | +| 2 | ~4.69% | ~98.44% | +| 3 | ~1.17% | ~99.61% | +| ≥ 4 | ~0.39% | 100% | + +Every key is deterministically placed. Two clients that insert the same records in any order end up with the same tree, same node CIDs, same root CID. This is the whole point — the tree is content-addressed, not insertion-ordered. + +### Why SHA-256, not the key's natural position? + +Using the key directly (like a regular balanced tree would) would cluster TIDs near their creation time, producing a skewed tree that rebalances on every insert. Hashing spreads keys uniformly, and tying height to leading zero bits means the tree shape is a pure function of the key set — no rotation or rebalancing logic is ever needed. + +### Reference implementation + +See `atproto-repo/src/mst/key.rs`: + +- `key_height(&str) -> u32` — returns `count_leading_zero_bits(SHA-256(key)) / 2`. +- `count_leading_zero_bits(&[u8]) -> u32` — count zero bits byte-by-byte from MSB. + +## 3. Node shape + +Each MST node is a DAG-CBOR map serialized canonically (DRISL rules, see `drisl.md`). Bytewise key order in the map: + +| Key | Sort bytes | Type | Required | Meaning | +| --- | ------------- | -------- | -------- | ------------------------------------------------------------------------------------------ | +| `e` | `0x65` | array | yes | Entries at this node, sorted by reconstructed key ascending (bytewise). | +| `l` | `0x6c` | CID | no | Left subtree. All keys in that subtree are `<` the first entry's key. Omit when absent. | + +Each *entry* in `e` is itself a DAG-CBOR map: + +| Key | Sort bytes | Type | Required | Meaning | +| --- | ---------- | ------- | -------- | ----------------------------------------------------------------------------------------------- | +| `k` | `0x6b` | bytes | yes | Key *suffix* — bytes after the prefix this entry shares with the previous entry's reconstructed key. | +| `p` | `0x70` | integer | yes | Length of the prefix shared with the previous entry's reconstructed key. `0` for the first entry. | +| `t` | `0x74` | CID | no | Right subtree between this entry's key and the next entry's key. Omit when absent. | +| `v` | `0x76` | CID | yes | CID of the value — the record block for this entry. | + +The sort for entry fields under DRISL is `k`, `p`, `t`, `v` (bytewise: `0x6b < 0x70 < 0x74 < 0x76`). Getting that order wrong changes the node's CID. + +### Field semantics in prose + +- `l` points to a whole subtree of lower-keyed entries. There's at most one `l` per node. +- Each `t` on an entry points to a subtree that lives between that entry and the next. An entry without `t` means there are no keys strictly between this entry and the next. +- Omit `l` and `t` entirely when there is no subtree — **do not write `null`**. An encoded `null` is a different set of bytes and produces a different CID. + +## 4. Prefix compression + +Adjacent entries often share long prefixes (records in the same collection share `app.bsky.feed.post/`, for instance). MST nodes store each entry's full key as: + +- `p` — how many leading bytes are shared with the **reconstructed previous key**. +- `k` — the remaining suffix bytes. + +### Reconstruction algorithm + +To recover the full key of entry *i*: + +1. If *i = 0*, the key is exactly `k₀` and `p₀` must be 0. +2. For *i > 0*, reconstruct key *i−1* first, then: + ``` + key_i = key_{i-1}[..p_i] + k_i + ``` + +A streaming reader keeps a single "previous key" buffer and advances it each entry. See `KeyReconstructor` in `atproto-repo/src/mst/entry.rs:149`. + +### Worked example + +Keys `app.bsky.feed.post/abc`, `app.bsky.feed.post/def`, `app.bsky.feed.post/ghi` at a single node: + +| i | Full key | `p` | `k` | +| - | -------------------------- | --- | --------------------------- | +| 0 | `app.bsky.feed.post/abc` | 0 | `app.bsky.feed.post/abc` | +| 1 | `app.bsky.feed.post/def` | 19 | `def` | +| 2 | `app.bsky.feed.post/ghi` | 19 | `ghi` | + +### Invariants a strict reader must enforce + +- `p` of the first entry must equal `0`. +- `p` must never exceed the length (in bytes) of the previous reconstructed key. +- The resulting key must sort strictly greater than the previous key (duplicates are forbidden; the tree is a map, not a multimap). + +Violations raise `InvalidPrefix` or `InvalidNode` in the reference crate. + +## 5. Traversal + +To iterate in key order: + +``` +traverse(node): + if node.l: traverse(load(node.l)) + prev_key = "" + for entry in node.e: + key = reconstruct(prev_key, entry.p, entry.k) + yield (key, entry.v) + if entry.t: traverse(load(entry.t)) + prev_key = key +``` + +That visits left subtree, then each entry followed by its right subtree, which produces sorted ascending order because: + +1. Every key in `l` is `< e[0]`. +2. Every key in `e[i].t` is `> e[i]` and `< e[i+1]`. +3. Entries in `e` are themselves sorted. + +See `Mst::entries()` and `collect_entries()` in `atproto-repo/src/mst/tree.rs:432`. + +## 6. Lookup + +To look up a key *K*: + +``` +lookup(node, K): + walk entries in order, reconstructing keys: + if reconstructed_key == K: return entry.v + if reconstructed_key > K: + // K, if it exists, is in the subtree to the left of this entry + // (either node.l if this is entry 0, or prev_entry.t) + recurse accordingly + return + // fell off the right: check the last entry's .t +``` + +A complete lookup costs O(log₄ N) nodes fetched from storage — each level of the tree divides the search by ~4. For a repo with 100,000 records, that's ≤ ~10 node fetches. Cache the root. + +Reference: `Mst::get_recursive()` at `atproto-repo/src/mst/tree.rs:139`. + +## 7. Insert — the tricky one + +Inserting `(K, V_cid)`: + +1. Compute `h = key_height(K)`. +2. Walk down the tree from the root. At each node: + - If `h == node_height`, the key belongs at this node. Find the bytewise sort position, insert an entry, and **recompute prefix compression on the entry immediately after** (its `p`/`k` are now relative to a different previous key). + - If `h < node_height`, descend into the subtree that brackets `K` (either `l` or the `t` of the preceding entry). + - If `h > node_height`, a new node at height `h` must be created above this point, and the existing subtree becomes a child of the new node (split on the position of `K`). +3. Each modified node produces a fresh CID. The chain of fresh CIDs up to the root is the new MST root. +4. Previously unchanged nodes keep their old CIDs and are reused — that's the structural sharing that makes sync cheap. + +Insert is where most bugs live. The reference `atproto-repo` crate's `Mst::insert_recursive` (`src/mst/tree.rs:222`) handles only the simple within-node case cleanly; users with inserts that cross heights are expected to build the tree bottom-up from sorted records and recompute node boundaries, not rely on a recursive insert. Verify any insert implementation by checking that repeated inserts in randomized order all produce byte-identical root CIDs. + +## 8. Delete + +Deleting `K`: + +1. Lookup the node holding `K`. Remove the entry. +2. If the deleted entry had a right subtree (`t`) or the surrounding entries had bracketing subtrees, **merge** them — their contents must be stitched back into the node or promoted to replace the deleted boundary. The simplest correct implementation: collect every `(key, cid)` pair from the deleted region's subtrees, and re-insert them into the tree from scratch. +3. If the node becomes empty and has no subtrees, it's removed; its parent sheds a pointer. +4. Recompute prefix compression on the entry that now follows the gap. +5. Propagate up: each ancestor is re-serialized (its child CID changed), producing a new root CID. + +The reference `Mst::delete_recursive` handles simple cases only; the same caveat as insert applies. + +## 9. Invariants a verifier must check + +On every node loaded from storage, require: + +- **Canonical DAG-CBOR** (DRISL). A non-canonical encoding changes the CID; if the CID claimed on the block doesn't match, treat as corrupt. +- **Map keys exactly `e`, optionally `l`**. No extras, no `null` placeholders. +- **Entries sorted ascending** by reconstructed key, bytewise, strictly. +- **First entry's `p == 0`**. +- **All entry keys reconstruct successfully** — no `p` exceeding the previous key's length, all `k` valid UTF-8. +- **All referenced CIDs are dag-cbor CIDs** (SHA-256, 32-byte digest) for subtrees and values alike. No `raw`-codec CIDs for tree structure. +- **All entries in a node share the same key height.** A node's height is not stored on the wire — compute `key_height` for each reconstructed key and require they all agree. The node's implicit height is that shared value. Mixed-height entries in a single node are a bug. + +Repos that fail these checks cannot be safely synced; reject and surface the specific violation. + +## 10. Diff — the sync primitive + +Two roots `R_old` and `R_new`. Walk both trees with a merged iterator: + +- Advance through entries in sorted key order on both sides. +- At each step: + - Key only in old → `Delete`. + - Key only in new → `Add`. + - Key in both, same value CID → no-op (skip). + - Key in both, different value CID → `Update`. + +The content-addressed tree lets you short-circuit: whenever two subtree CIDs on the old and new sides are equal, the entire subtree is unchanged — you can skip descending into it entirely. For tiny diffs (one record added to a 100k-record repo), this reduces the work from O(N) to O(log N). + +The reference crate exposes this as a flat-list diff in `atproto-repo/src/mst/diff.rs`: + +- `diff_entries(old, new) -> Vec` on two sorted `(String, Cid)` slices. +- `MstDiff::{Add, Update, Delete}` variants with the key and CID(s). +- `DiffStats` for counts. + +The flat-list version is O(N) per diff but trivially correct; use it as a reference oracle when testing a CID-short-circuiting walker. + +## 11. Storage + +The tree doesn't prescribe how nodes are stored. Typical choices: + +- **In-memory**: `HashMap`. Fine for small repos and tests. +- **On-disk**: SQLite table keyed by CID, or a raw files-on-disk layout indexed by CID. +- **Remote**: the PDS holds the canonical tree; clients fetch blocks on demand via `com.atproto.sync.getBlocks`. + +The `atproto-dasl::storage` module exposes `BlockStorage` with `MemoryStorage` and `DiskStorage` implementations. Any consumer-built storage must: + +- Return the exact bytes stored (no re-encoding). +- Preserve (CID, bytes) as an immutable pair — never overwrite a CID with different bytes. +- Handle a `get` for an unknown CID by returning "not found" rather than producing bytes. + +## 12. Relationship to commits + +An MST root CID is just a CID — it says nothing about ownership or revision. That's the commit's job: + +``` +Commit { did, version: 3, data: , rev: , prev: , sig: } +``` + +The commit fixes the tree root at a point in time, binds it to a DID, and is signed by that DID's atproto signing key. See `commit-and-signing.md` for the exact signing bytes. + +A repo's identity is `(did, commit_cid)` — the tree root CID alone is not enough to identify a particular repo state, because two accounts could theoretically converge on the same set of records at the same time and produce identical tree roots. + +## 13. Common errors + +| Symptom | Likely cause | +| ------------------------------------------------ | ---------------------------------------------------------------------------------------- | +| `NodeNotFound` | The referenced subtree block isn't in the block store. CAR was incomplete; re-sync. | +| `InvalidPrefix` / `prefix_len exceeds previous` | Node was hand-constructed. Check the first entry has `p == 0` and each `p` stays within bounds. | +| Root CID differs after reinserting same records | Non-deterministic encode path — most likely unsorted map keys or non-canonical integer form. Verify `drisl.md` compliance. | +| Tree depth explodes on small repos | Missing `/ 2` in height calc — raw leading-zero-bit count gives fanout 2, which is pathological. | +| Update produces a new root CID but `entries()` returns stale data | Old root pinned somewhere; replace with new root everywhere before re-reading. | +| Diff reports Update for a record whose JSON is identical | Record encoder is non-canonical — re-encoding the "same" record produces different bytes, hence a different CID. Check field order. | diff --git a/skills/software-development/atproto-repository/references/shared/test-vectors.md b/skills/software-development/atproto-repository/references/shared/test-vectors.md new file mode 100644 index 0000000..d2da820 --- /dev/null +++ b/skills/software-development/atproto-repository/references/shared/test-vectors.md @@ -0,0 +1,356 @@ +# Test Vectors (Reference) + +Small, hand-checkable fixtures for DRISL, CAR, MST, and commit encoding. Use these to wire up unit tests for a new implementation, or to sanity-check an encoder that's already "mostly working." + +All vectors are **synthetic** — they use placeholder CIDs (32 zero bytes of digest) where real-world data would plug in a SHA-256 of something substantial. Byte hexdumps are authoritative; every implementation should produce the exact same bytes. + +## 1. DRISL encoding + +### 1.1. Empty map + +``` +a0 +``` + +One byte: major type 5 (map), length 0. + +### 1.2. `{"a": 1}` + +``` +a1 ; map(1) +61 61 ; text(1) "a" +01 ; unsigned(1) +``` + +4 bytes total. + +### 1.3. `{"b": 2, "a": 1}` — keys must be sorted on write + +Canonical output (note sort to `a` first, then `b`): + +``` +a2 ; map(2) +61 61 ; "a" +01 ; 1 +61 62 ; "b" +02 ; 2 +``` + +6 bytes. A non-canonical encoder that preserves insertion order would produce `a2 61 62 02 61 61 01` — reject on strict decode. + +### 1.4. Integer shortest-form boundary: value 23 vs 24 + +- `23` → one byte: `17` (immediate, major type 0, value 23). +- `24` → two bytes: `18 18` (major type 0, additional info `0x18` → 1-byte follow-up, value `0x18` = 24). + +A strict decoder must reject `18 17` (non-canonical one-byte form of 23), `19 00 17` (two-byte form of 23), and all wider forms. + +### 1.5. Negative integer: -5 + +``` +24 ; major type 1, immediate value 4 +``` + +CBOR encodes negative integers as `-1 - N`, where `N` is the immediate value. For `-5`, `N = 4`, so the byte is `0x20 | 4 = 0x24`. One byte total. + +### 1.6. Float 1.5 + +``` +fb 3f f8 00 00 00 00 00 00 ; major type 7, info 27 (float64), big-endian IEEE-754 1.5 +``` + +9 bytes. The same value cannot be encoded as 16-bit or 32-bit float in DRISL — always 64-bit. + +### 1.7. `null` + +``` +f6 +``` + +One byte (major type 7, simple value 22). + +### 1.8. CID (tag 42) + +A dag-cbor CID with a 32-byte all-zero digest: + +``` +d8 2a ; tag 42 +58 25 ; bytes(37) +00 ; identity multibase prefix +01 71 12 20 <32 zero bytes> ; CIDv1 + dag-cbor codec + SHA-256 multihash header + digest +``` + +41 bytes total: `d8 2a 58 25 00 01 71 12 20 00…(32 zeros)`. + +### 1.9. Non-canonical integer (reject on strict decode) + +``` +18 05 ; one-byte encoding of value 5 +``` + +Strict decode error: `NonCanonicalEncoding`. The canonical form is `05` (immediate). + +### 1.10. Forbidden tag (reject on strict decode) + +``` +c6 01 ; tag 6 (CBOR date string), value 1 +``` + +Strict decode error: `UnsupportedTag`. Only tag 42 is allowed. + +## 2. CAR v1 + +### 2.1. Minimum CAR — header only, one root, no blocks + +Let the single root CID `R` have all-zero digest. The header is the DAG-CBOR map: + +``` +a2 ; map(2) +65 72 6f 6f 74 73 ; "roots" +81 ; array(1) +d8 2a ; tag 42 +58 25 ; bytes(37) +00 01 71 12 20 <32 zero> ; identity || CID(dag-cbor, SHA-256, zeros) +67 76 65 72 73 69 6f 6e ; "version" +01 ; unsigned(1) +``` + +58 bytes of header. Frame with varint length `0x3a` (58): + +``` +3a a2 65 … 01 +``` + +Total 59 bytes. This is not a valid *repo* export (the root block isn't included, and the root CID would mismatch any real commit), but it's a valid CAR framing and a good place to test a reader's varint + header parse before exercising block handling. + +### 2.2. Block framing example + +A single block with a dag-cbor CID `R` and payload `a0` (empty map): + +``` + ; block length = cid_len(36) + data_len(1) = 37 = 0x25 +25 +01 71 12 20 <32-byte digest of 0xa0> ; CID bytes (no identity prefix in CAR block framing) +a0 ; payload +``` + +Verify: decoder reads varint → 37 bytes follow → first 36 bytes are the CID (`0x01 71 12 20` + 32-byte digest) → remaining 1 byte is the payload. The decoder recomputes `SHA-256(0xa0)`, checks it against the declared digest, and accepts the block only if they match. (Compute the expected digest in your chosen implementation to bake a real oracle value into your tests.) + +## 3. MST + +### 3.1. Key heights for "app.bsky.feed.post/" + +Using `key_height = leading_zero_bits(SHA-256(key)) / 2`: + +| Key | Approx SHA-256 leading hex | Leading zero bits | Height | +| ------------------------------------------ | -------------------------- | ----------------- | ------ | +| `app.bsky.feed.post/3jzfcijpj2z2a` | (non-zero first nibble) | 0 | 0 | +| `app.bsky.feed.post/3jzfcijpj2z2b` | (non-zero first nibble) | 0 | 0 | +| A randomly chosen key with first SHA-256 byte `0x0f` | `0f…` | 4 | 2 | +| A key with first byte `0x00` and second byte `0xff` | `00ff…` | 8 | 4 | + +Expect ~75% of arbitrary keys to land at height 0. You can smoke-test this with 1000 random keys and check the distribution matches the table in `mst.md` §2. + +### 3.2. Prefix compression + +Keys `app.bsky.feed.post/abc`, `app.bsky.feed.post/def`, all at height 0, sharing a common prefix of 19 bytes: + +Entry 0: + +``` +p = 0 +k = "app.bsky.feed.post/abc" (22 bytes) +v = +t = absent +``` + +Entry 1: + +``` +p = 19 +k = "def" (3 bytes) +v = +t = absent +``` + +Reconstructed keys: `key_0 = "app.bsky.feed.post/abc"`, `key_1 = key_0[..19] + "def" = "app.bsky.feed.post/def"`. + +Invariants: first entry's `p == 0`, `p <= len(prev_key)`, resulting key sorts strictly greater than previous. + +### 3.3. Single-node MST DAG-CBOR encoding + +A node with left subtree absent and two entries above: + +``` +a1 ; map(1) — only "e" present (no "l") +65 65 ; "e" +82 ; array(2) + + a3 ; map(3) — first entry (no "t") + 61 6b ; "k" + 56 ; bytes(22) + 61 70 70 2e 62 73 6b 79 2e 66 65 ; "app.bsky.feed.post/abc" + 65 64 2e 70 6f 73 74 2f 61 62 63 + 61 70 ; "p" + 00 ; 0 + 61 76 ; "v" + d8 2a 58 25 00 <37-byte CID> ; value CID + + a3 ; map(3) — second entry (no "t") + 61 6b ; "k" + 43 ; bytes(3) + 64 65 66 ; "def" + 61 70 ; "p" + 13 ; 19 + 61 76 ; "v" + d8 2a 58 25 00 <37-byte CID> ; value CID +``` + +Field order inside entry: `k` (0x6b) < `p` (0x70) < `v` (0x76). Omission of `t` and `l` is why those are `a3` (map-of-3) not `a4`, and the outer node is `a1` not `a2`. + +## 4. Commit + +### 4.1. Genesis commit, spec-strict form (with `prev: null`) + +Fields: + +- `did = "did:plc:ewvi7nxzyoun6zhxrhs64oiz"` (32 bytes) +- `version = 3` +- `data = ` (dag-cbor CID, 37 bytes inside tag 42) +- `rev = "3jzfcijpj2z2a"` (13 bytes) +- `prev = null` (genesis) + +DAG-CBOR (`UnsignedCommit`): + +``` +a5 ; map(5) +64 64 61 74 61 ; "data" +d8 2a 58 25 00 <37-byte CID> ; data CID + +63 64 69 64 ; "did" +78 20 ; text(32) +64 69 64 3a 70 6c 63 3a ; "did:plc:" +65 77 76 69 37 6e 78 7a ; "ewvi7nxz" +79 6f 75 6e 36 7a 68 78 ; "youn6zhx" +72 68 73 36 34 6f 69 7a ; "rhs64oiz" + +64 70 72 65 76 ; "prev" +f6 ; null + +63 72 65 76 ; "rev" +6d ; text(13) +33 6a 7a 66 63 69 6a 70 ; "3jzfcijp" +6a 32 7a 32 61 ; "j2z2a" + +67 76 65 72 73 69 6f 6e ; "version" +03 ; unsigned(3) +``` + +Total: 1 (`a5`) + 5+41=46 (`data` + tag-42 CID) + 4+34=38 (`did` + text(32)) + 5+1=6 (`prev` + `f6`) + 4+14=18 (`rev` + text(13)) + 8+1=9 (`version` + `03`) = **118 bytes**. + +(The CID inside tag 42 is 41 bytes: `d8 2a` tag + `58 25` bytes(37) header + 37 payload bytes.) + +These 118 bytes are what a signer feeds to its ECDSA function to produce the `sig`. The full signed `Commit` adds: + +``` +63 73 69 67 ; "sig" +58 40 ; bytes(64) +<64-byte signature r||s> +``` + +Signed commit total: 118 + 4+2+64 = **188 bytes**, and the outer map header changes from `a5` to `a6` (6 entries now). Map-key order is `data`, `did`, `prev`, `rev`, `sig`, `version`. + +### 4.2. Genesis commit, reference-impl form (with `prev` absent) + +The reference Rust impl elides `prev` when it's `None`. `UnsignedCommit` becomes: + +``` +a4 ; map(4) — no "prev" +(data, did, rev, version exactly as above, minus the 6-byte prev section) +``` + +Total: 118 − 6 = **112 bytes** of signing bytes. A signer in reference-impl mode and one in spec-strict mode will produce different signatures over logically identical commits. + +When verifying a commit received over the wire: + +1. Take the original commit block bytes. +2. Remove exactly the `sig` field (along with its key). Do not alter `prev` (keep it present if it was present, absent if it was absent). +3. Verify over those bytes. + +### 4.3. Signature length + +- k256 / p256 raw ECDSA signatures are exactly 64 bytes. +- The `sig` field on a valid commit encodes to `58 40 <64 bytes>` = 66 bytes total. +- If you see `59 00 40 <64 bytes>` (non-canonical 2-byte length), reject — it's a non-canonical encoding and will fail strict decode. +- If the `sig` field is longer than 64 bytes, the producer probably emitted DER-wrapped ECDSA. Unwrap to raw r||s before verifying. + +## 5. AT-URI + +| Input | Expected result | +| --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | +| `at://did:plc:ewvi7nxzyoun6zhxrhs64oiz/app.bsky.feed.post/3jui7kp54ic2i` | `authority="did:plc:ewvi7nxzyoun6zhxrhs64oiz"`, `collection="app.bsky.feed.post"`, `record_key="3jui7kp54ic2i"` | +| `at://did:web:example.com/app.bsky.feed.like/3k2akjh32kj` | valid; did:web authority accepted | +| `at://did:web:example.com:8080:tenant:users/app.bsky.feed.post/3k…` | valid *in reference impl* (non-strict did:web path); spec-strict validators may reject | +| `at://did:plc:abcdefghijklmnopqrstuvwx/b/c` | valid; minimal form | +| `at://did:plc:ewvi7nxzyoun6zhxrhs64oiz/app.bsky.feed.post/3k…/extra/parts` | valid; extras silently ignored by reference parser | +| `at://alice.bsky.social/app.bsky.feed.post/3k…` | `HandleNotSupported` — resolve handle first | +| `at://did:key:z6M…/…` | `AuthorityParsingFailed` — unsupported DID method | +| `did:plc:…/app.bsky.feed.post/3k…` | `MissingPrefix` | +| `at://did:plc:xxx/app.bsky.feed.post/3k…/` | `TrailingSlash` | +| `at://did:plc:xxx/` | `TrailingSlash` (not CollectionMissing — the slash makes it trailing first) | +| `at://did:plc:xxx` | `CollectionMissing` | +| `at://did:plc:xxx/app.bsky.feed.post` | `RecordKeyMissing` | +| `at://did:plc:xxx//3k…` | `EmptyCollection` | +| `at:// /app.bsky.feed.post/3k…` | `AuthorityParsingFailed` — whitespace authority | + +## 6. TID round-trips + +| String | `timestamp_micros` | `clock_id` | Valid? | +| --------------- | --------------------- | ---------- | ------------------------------------------------------------------- | +| `3jzfcijpj2z2a` | some microseconds ts | some clock | yes (reference-spec example) | +| `2222222222222` | 0 | 0 | yes (minimum encodable; first char `2` = value 0) | +| `7777777777777` | large | large | yes (spec example — all `7`s = all-bit-5 = value 5 repeated) | +| `jjjjjjjjjjjjj` | large | some | yes (first char `j` = value 15, which is the max with top bit 0) | +| `zzzzzzzzzzzzz` | n/a | n/a | **no** — first char `z` = value 31, which sets the top bit | +| `AAAAAAAAAAAAA` | n/a | n/a | **no** — uppercase letters not in base32-sortable alphabet | +| `3jzfcijpj2z2` | n/a | n/a | **no** — 12 chars | +| `3jzfcijpj2z2aa`| n/a | n/a | **no** — 14 chars | +| `3jzfcijpj2z2!` | n/a | n/a | **no** — `!` not in alphabet | + +## 7. End-to-end fixture + +A complete, minimal synthetic repo: + +- DID: `did:plc:abcdefghijklmnopqrstuvwx` (24-char base32lower, valid per `atproto-identity-resolution` rules) +- One record: `app.bsky.actor.profile/self` with value `{"$type": "app.bsky.actor.profile", "displayName": "Alice"}` +- MST root: a single node with one entry, `p=0`, `k="app.bsky.actor.profile/self"`, `v=` +- Commit: `{ data: , did, prev: null, rev: "3jzfcijpj2z2a", sig: <64 bytes>, version: 3 }` +- CAR: header `{version: 1, roots: []}` + commit block + MST node block + profile record block. + +Build this end-to-end in your implementation, then load the same CAR with `atproto-repo::MemoryRepository::from_car` and compare: + +- `repo.did() == "did:plc:abcdefghijklmnopqrstuvwx"` +- `repo.commit().rev == "3jzfcijpj2z2a"` +- `repo.get_record(&RecordPath::new("app.bsky.actor.profile", "self")).await.unwrap()` returns the JSON form of your record. + +If every CID matches and all lookups succeed, your encoder is at parity with the reference. + +## 8. Round-trip invariant + +For any value `v` encoded through your encoder: + +``` +encode(decode(encode(v))) == encode(v) bytewise +``` + +A canonical encoder satisfies this trivially. A non-canonical one fails somewhere, usually at integer widening or map sort. Add this as a fuzz target — it catches 90% of production DRISL bugs. + +## 9. Upstream fixtures to cross-check against + +When the synthetic vectors above pass, validate against real known-good data from the reference implementations: + +- **Go (indigo)** — `indigo/atproto/repo/testdata/` and `indigo/atproto/repo/mst/testdata/` hold real-world repo CARs, MST node fixtures, and interop vectors. +- **TypeScript (`@atproto/repo`)** — `atproto/packages/repo/tests/*.test.ts` exercises real commit signing, MST mutation sequences, and CAR round-trips with shared fixture builders. +- **Rust (`atproto-repo`)** — `atproto-identity-rs/crates/atproto-repo/tests/` carries the cross-language interop fixtures, including `prev` divergence cases. + +If your encoder disagrees with any of these at the byte level, the bug is in your encoder — they've been validated against each other across ecosystems. diff --git a/skills/software-development/atproto-repository/references/typescript/README.md b/skills/software-development/atproto-repository/references/typescript/README.md new file mode 100644 index 0000000..4340e91 --- /dev/null +++ b/skills/software-development/atproto-repository/references/typescript/README.md @@ -0,0 +1,124 @@ +# TypeScript — `@atproto/repo` overview + +The reference TypeScript implementation of AT Protocol repository format is split across four packages in the `bluesky-social/atproto` monorepo: + +| Package | Purpose | +| -------------------- | ------------------------------------------------------------------------------- | +| `@atproto/repo` | `Repo` / `ReadableRepo`, `MST`, `BlockMap`, CAR reader/writer, commit signing. | +| `@atproto/lex-cbor` | Canonical DAG-CBOR (`encode`, `decode`), `cidForLex`, `sha256RawToCid`. | +| `@atproto/lex-data` | `Cid` class, `LexMap`/`LexValue` types, `decodeCid`, `isCidForBytes`. | +| `@atproto/crypto` | `Secp256k1Keypair`, `P256Keypair`, `verifySignature`. | +| `@atproto/syntax` | DID / TID / NSID / AT-URI parsing, `RecordKeyString`, `NsidString`. | +| `@atproto/common-web`| `TID.nextStr()` for revision generation. | + +Pin via `npm install @atproto/repo @atproto/crypto @atproto/lex-data`; the others are transitive deps. Check `https://www.npmjs.com/package/@atproto/repo` for the current version — the package is actively maintained. + +## Public surface of `@atproto/repo` + +Re-exported from `index.ts`: + +- **Repo types** — `Repo`, `ReadableRepo` +- **Trees** — `MST`, `Leaf`, `NodeEntry`, `NodeData` +- **Blocks** — `BlockMap`, `CarBlock`, `CidSet` +- **Commits** — `Commit`, `UnsignedCommit`, `LegacyV2Commit`, `VersionedCommit`, `CommitData`, `RepoUpdate` +- **Writes** — `WriteOpAction`, `RecordWriteOp`, `RecordCreateOp`, `RecordUpdateOp`, `RecordDeleteOp`, `RecordWriteDescript`, `WriteLog` +- **Signing** — `signCommit`, `verifyCommitSig` +- **CAR** — `readCar`, `readCarWithRoot`, `readCarStream`, `readCarReader`, `writeCarStream`, `blocksToCarFile`, `blocksToCarStream`, `verifyIncomingCarBlocks` +- **Parsing** — `getAndParseRecord`, `getAndParseByDef`, `cborToLex`, `cborToLexRecord` +- **Storage** — `ReadableBlockstore`, `MemoryBlockstore`, `RepoStorage`, `SyncStorage` +- **Diffs** — `DataDiff` +- **Sync verification** — from `./sync`: `verifyRepoCar`, `verifyRepo`, `verifyDiffCar`, `verifyDiff`, `verifyProofs`, `verifyRecords` + +## Reading a CAR → Repo + +```typescript +import { readCarWithRoot, ReadableRepo } from '@atproto/repo' +import { MemoryBlockstore } from '@atproto/repo' + +const { root, blocks } = await readCarWithRoot(carBytes) +// blocks is a BlockMap. CIDs are verified against their bytes by default. +const storage = new MemoryBlockstore(blocks) +const repo = await ReadableRepo.load(storage, root) + +console.log(repo.did, repo.commit.rev) +const record = await repo.getRecord('app.bsky.feed.post', 'abc123') +``` + +`ReadableRepo.load` reads the commit object via `storage.readObj(root, def.versionedCommit)`, upgrades any legacy v2 commit through `ensureV3Commit`, and constructs the MST lazily (`MST.load(storage, commit.data)` doesn't touch storage — entries load on first traversal). + +## Building a Repo → writing a CAR + +```typescript +import { + Repo, + MemoryBlockstore, + WriteOpAction, + blocksToCarFile, +} from '@atproto/repo' +import { Secp256k1Keypair } from '@atproto/crypto' + +const storage = new MemoryBlockstore() +const keypair = await Secp256k1Keypair.create() +const repo = await Repo.create(storage, 'did:plc:example', keypair, []) + +const update = await repo.formatCommit( + { + action: WriteOpAction.Create, + collection: 'app.bsky.feed.post', + rkey: TID.nextStr(), + record: { $type: 'app.bsky.feed.post', text: 'hello', createdAt: now }, + }, + keypair, +) + +// update.newBlocks contains the commit + all MST + record blocks. +const carBytes = await blocksToCarFile(update.cid, update.newBlocks) +``` + +`Repo.create` signs a genesis commit via `formatInitCommit`; `Repo.formatCommit` takes one or more `RecordWriteOp`s, applies them to the MST, and returns a fresh `CommitData` with `newBlocks` holding everything that needs to be persisted. + +## Idioms to watch for + +- **Immutable MST** — every mutation returns a *new* `MST` value. The old value is still valid. Rust and Go use mutation; TS does not. See `mst.md`. +- **`prev` always serialized** — `Commit.prev` is `Cid | null`, and the field is always present in the serialized map (matches Go, diverges from the Rust reference impl). See `commit.md` §prev. +- **CAR reader verifies CIDs on ingest** — `readCar*` runs `verifyIncomingCarBlocks` by default, re-hashing every block and throwing on mismatch. Go's `LoadRepoFromCAR` does NOT verify. Rust's `CarReader` doesn't by default either. See `car.md`. +- **`LexMap` vs raw map** — records round-trip as `LexMap` (a plain object with lex-aware types for CIDs, bytes, blobs). Use `cborToLexRecord` to decode block bytes into a `LexMap`. +- **Caller owns DID → didKey resolution** — `verifyRepo`/`verifyCommitSig` take a `didKey: string` (the `did:key:…` form of the atproto signing key), not a DID document. Resolve separately (see the `atproto-identity-resolution` skill). +- **No inductive firehose verifier** — TS does not ship an equivalent of Go's `VerifyCommitMessage` (op inversion to reproduce `prevData`). Use `verifyDiff(repo, newBlocks, newRoot, did, signingKey)` against the prior repo state for equivalent guarantees. +- **async/await everywhere** — most APIs are async because the storage interface is async. Even pure-ish helpers (`MST.get`, `MST.add`) are async because MST entries load on demand. + +## When to reach for which API + +| Task | Use | +| ------------------------------------------------- | ---------------------------------------------------------------------- | +| Read a signed CAR export, check it end to end | `verifyRepoCar(carBytes, did, signingKey)` → `VerifiedRepo` | +| Verify a delta CAR relative to the prior repo | `verifyDiffCar(repo, carBytes, did, signingKey)` → `VerifiedDiff` | +| Inspect records without verification | `readCarWithRoot` + `ReadableRepo.load` + `repo.getRecord` | +| Produce a new commit for a set of writes | `Repo.formatCommit(ops, keypair)` → `RepoUpdate` | +| Produce a firehose `#commit` delta CAR | `blocksToCarFile(update.cid, update.newBlocks)` — only changed blocks | +| Seed a fresh repo | `Repo.create(storage, did, keypair, initialWrites)` | +| Check a specific record inclusion proof | `verifyProofs(carBytes, claims, did, didKey)` | +| Walk the MST directly | `repo.data.walk()` or `repo.data.walkLeavesFrom(key)` | + +## File pointers (monorepo) + +| Concern | File | +| ------------------------------- | ----------------------------------------------------- | +| Public barrel | `packages/repo/src/index.ts` | +| `Repo` / `ReadableRepo` | `packages/repo/src/repo.ts`, `readable-repo.ts` | +| `MST` | `packages/repo/src/mst/mst.ts` | +| `MST` util (hash, key-validity) | `packages/repo/src/mst/util.ts` | +| CAR reader / writer | `packages/repo/src/car.ts` | +| Block store | `packages/repo/src/block-map.ts`, `src/storage/` | +| Commit types & schema | `packages/repo/src/types.ts` | +| `signCommit` / `verifyCommitSig`| `packages/repo/src/util.ts` | +| Sync verification | `packages/repo/src/sync/consumer.ts` | +| Data diff | `packages/repo/src/data-diff.ts` | + +## See also + +- `drisl.md` — canonical DAG-CBOR via `@atproto/lex-cbor`. +- `car.md` — CAR v1 reading / writing and firehose framing. +- `mst.md` — the immutable `MST` class and its on-wire node shape. +- `commit.md` — commit signing, verification, and the `prev` divergence. +- `../shared/divergence-matrix.md` — where TS differs from Rust and Go. diff --git a/skills/software-development/atproto-repository/references/typescript/car.md b/skills/software-development/atproto-repository/references/typescript/car.md new file mode 100644 index 0000000..2c8a793 --- /dev/null +++ b/skills/software-development/atproto-repository/references/typescript/car.md @@ -0,0 +1,209 @@ +# TypeScript — CAR v1 reading and writing + +`@atproto/repo` ships a complete CAR v1 reader **and** writer — unlike Go (reader only) and Rust (writer available in `atproto-dasl`). The reader verifies CID-to-content on ingest by default, which is the most important divergence from the other implementations. + +## Reading — `readCar` and friends + +Four public entry points, all in `packages/repo/src/car.ts`: + +```typescript +readCar(bytes, opts?): Promise<{ roots: Cid[]; blocks: BlockMap }> +readCarWithRoot(bytes, opts?): Promise<{ root: Cid; blocks: BlockMap }> // asserts 1 root +readCarStream(iter, opts?): Promise<{ roots: Cid[]; blocks: CarBlockIterable }> // async-iterable input +readCarReader(reader, opts?): Promise<{ roots: Cid[]; blocks: CarBlockIterable }> // low-level +``` + +Everything flows through `readCarReader`, which: + +1. Reads a varint, interprets it as the header byte length. +2. Decodes the header as DAG-CBOR via `cbor.decode`, validates against zod schema `{ version: 1, roots: Cid[] }`. **CAR v1 only** — any other version fails the schema. +3. Returns a block iterator. Each block: varint length, 36-byte binary CID, block bytes. +4. Wraps the iterator with `verifyIncomingCarBlocks` unless `opts.skipCidVerification` is true. + +`readCar` / `readCarWithRoot` eagerly drain the iterator into a `BlockMap`. `readCarStream` / `readCarReader` give you the iterator so you can stream blocks into an arbitrary store. For large inputs prefer the streaming variants. + +## CID verification on ingest — the default + +```typescript +export async function* verifyIncomingCarBlocks( + car: AsyncIterable, +): AsyncGenerator { + for await (const block of car) { + if (!(await isCidForBytes(block.cid, block.bytes))) { + throw new Error(`Not a valid CID for bytes (${block.cid.toString()})`) + } + yield block + } +} +``` + +Source: `car.ts:177`. Every block on the way in is re-hashed via sha-256 and compared to its declared CID. + +**This is the main TS-vs-rest divergence:** + +- Go's `LoadRepoFromCAR` has `// TODO: not verifying CID` (`repo.go:65`). Content is trusted. +- Rust's `CarReader` doesn't verify by default either. +- TS does, by default. + +If you want to skip it (e.g., for trusted input, or when the CIDs are already verified by an upstream): + +```typescript +const { root, blocks } = await readCarWithRoot(carBytes, { skipCidVerification: true }) +``` + +## `BlockMap` — the in-memory block store + +```typescript +class BlockMap { + add(value: LexValue): Promise // encode, hash, set. Returns the new CID. + set(cid: Cid, bytes: Uint8Array): void + get(cid: Cid): Uint8Array | undefined + has(cid: Cid): boolean + delete(cid: Cid): void + getMany(cids: Cid[]): { blocks: Map; missing: Cid[] } + addMap(other: BlockMap): void + clear(): void + entries(): Iterable<{ cid: Cid; bytes: Uint8Array }> + keys(): Iterable + values(): Iterable + forEach(fn): void + [Symbol.iterator](): Iterator<...> + + readonly size: number + readonly byteSize: number +} +``` + +In-memory only; unbounded. For very large exports, either stream via `readCarReader` into your own persistent store (see below), or rely on the caller to limit input size. + +## Storage implementations + +From `packages/repo/src/storage/`: + +- `ReadableBlockstore` — interface used by `ReadableRepo` / `MST`. Methods: `getBytes`, `has`, `readObj`, `attemptReadRecord`, `getBlocks`. +- `MemoryBlockstore` — wraps a `BlockMap`. +- `RepoStorage` — read/write variant used by mutable `Repo`. Adds `applyCommit`, `getRoot`, `updateRoot`. +- `SyncStorage` — composes two block sources (staged + prior). Used during diff verification so lookups fall through to the prior-repo store. + +```typescript +import { MemoryBlockstore, ReadableRepo } from '@atproto/repo' + +const { root, blocks } = await readCarWithRoot(carBytes) +const storage = new MemoryBlockstore(blocks) +const repo = await ReadableRepo.load(storage, root) +``` + +## Streaming into a custom store + +```typescript +import { readCarReader } from '@atproto/repo' + +const { roots, blocks } = await readCarReader(reader) // async iterator +for await (const block of blocks) { + await customStore.put(block.cid, block.bytes) +} +// Or: await blocks.dump() to close without consuming. +``` + +`CarBlockIterable` is an `AsyncGenerator` with an extra `dump()` method that cancels the iteration and closes the underlying reader without throwing. + +## Writing + +```typescript +import { writeCarStream, blocksToCarStream, blocksToCarFile } from '@atproto/repo' + +// Stream: suitable for piping to a response or file +const stream = writeCarStream(root, asyncIterableOfBlocks) +for await (const chunk of stream) response.write(chunk) + +// From a BlockMap, streaming +const stream2 = blocksToCarStream(root, blockMap) + +// From a BlockMap, fully buffered (returns Uint8Array) +const carBytes = await blocksToCarFile(root, blockMap) +``` + +`writeCarStream` emits: + +1. Varint(header length) || header (CBOR-encoded `{ version: 1, roots: [root] }`, or `roots: []` if `root === null`). +2. For each block: varint(CID bytes + block bytes) || CID bytes (36) || block bytes. + +Matches the CAR v1 spec byte-for-byte. Consumers don't rely on block order; duplicate CIDs are *not* deduplicated — dedupe before passing in if you care (`BlockMap` dedupes automatically because `add` returns the existing CID). + +## Block framing on the wire + +Per block, same as Go / Rust: + +``` +varint(cid_len + data_len) || cid_bytes(36) || data_bytes +``` + +where `cid_bytes` is the 36-byte binary CID (`0x01 0x71 0x12 0x20 `), not the 37-byte tag-42 form. TS handles this correctly; relevant only for raw-byte debugging. + +See `../shared/car-v1.md` for the spec-level framing and `../../atproto-cid/shared/binary-layout.md` for the 36-vs-37-byte distinction. + +## Firehose framing + +Each `#commit` and `#sync` event carries a `blocks` field of type `Uint8Array`. Pass it directly to `readCarWithRoot`: + +```typescript +const { root, blocks } = await readCarWithRoot(msg.blocks) +const storage = new MemoryBlockstore(blocks) +const repo = await ReadableRepo.load(storage, root) +``` + +The CAR only contains blocks that changed in this commit — the commit block, new/changed MST nodes, and new/changed record blocks. Unchanged MST subtrees are referenced by CID but their blocks aren't in the CAR. + +**Partial-tree handling:** when the `MST` later tries to descend into a subtree whose block isn't in `storage`, it throws a `MissingBlockError`. This is different from Go and Rust, which treat partial subtrees as a normal state (`ErrPartialTree` / returning `Ok` with missing children). In TS, operations that need the missing blocks fail loudly. + +For firehose delta verification, use `verifyDiff(priorRepo, newBlocks, newRoot, did, signingKey)` — it composes a `SyncStorage` that falls through to the prior repo's store, so unchanged subtree lookups succeed against the old state. + +## End-to-end verified ingest + +```typescript +import { verifyRepoCar, verifyDiffCar } from '@atproto/repo' + +// Full repo CAR from com.atproto.sync.getRepo +const verified: VerifiedRepo = await verifyRepoCar(carBytes, did, didKey) +// verified.creates is the full record listing; verified.commit holds cid/rev/prev. + +// Delta CAR (e.g. a firehose event) +const diff: VerifiedDiff = await verifyDiffCar(priorRepo, carBytes, did, didKey) +// diff.writes is the RecordWriteDescript[]; diff.commit is the new state. +``` + +Both accept `did?: string` and `signingKey?: string` (the `did:key:…` form of the current `#atproto` signing key). When passed, they verify `commit.did` matches and the signature is valid. When omitted, they skip those checks — use only for trusted input or when you've verified elsewhere. + +Caller owns DID → didKey resolution. See `commit.md` §verification. + +## Common errors + +| Error | Cause | +| --------------------------------------------- | ---------------------------------------------------------------------- | +| `Could not parse CAR header` | Header bytes don't decode to DAG-CBOR or don't match `{ version: 1, roots: [...] }`. | +| `Not a valid CID for bytes ()` | Block content hash doesn't match the declared CID. Corruption or forgery. | +| `Expected one root, got N` | `readCarWithRoot` called on a CAR with zero or multiple roots. | +| `could not parse varint` | Truncated CAR or invalid varint framing. | +| `Invalid repo did: ` | `verifyRepo`: the commit's `did` doesn't match the expected DID. | +| `Invalid signature on commit: ` | `verifyRepo`: signature check failed under `signingKey`. | +| `missing leaf blocks: ` | `verifyDiff` with `ensureLeaves: true`: new leaves referenced in MST but block not in CAR. | + +## File pointers + +| Concern | File | +| ---------------------------------- | --------------------------------------------------- | +| `readCar*` | `packages/repo/src/car.ts` | +| `writeCarStream`, `blocksToCar*` | `packages/repo/src/car.ts` | +| `verifyIncomingCarBlocks` | `packages/repo/src/car.ts:177` | +| `BlockMap` | `packages/repo/src/block-map.ts` | +| `ReadableBlockstore` / `MemoryBlockstore` / `SyncStorage` | `packages/repo/src/storage/` | +| `verifyRepoCar` / `verifyDiffCar` | `packages/repo/src/sync/consumer.ts` | +| CAR header schema | `packages/repo/src/types.ts` (`schema.carHeader`) | + +## See also + +- `../shared/car-v1.md` — byte-level CAR v1 spec. +- `drisl.md` — canonical DAG-CBOR underlying every block. +- `mst.md` — `MST` + partial-tree semantics. +- `commit.md` — `verifyCommitSig` / `verifyRepo` / `verifyDiff` on top of a CAR. +- `../shared/divergence-matrix.md` §car — TS verifies CIDs by default; Go and Rust don't. diff --git a/skills/software-development/atproto-repository/references/typescript/commit.md b/skills/software-development/atproto-repository/references/typescript/commit.md new file mode 100644 index 0000000..8b88c2d --- /dev/null +++ b/skills/software-development/atproto-repository/references/typescript/commit.md @@ -0,0 +1,247 @@ +# TypeScript — commit record, signing, and verification + +TS sits between Rust (everything caller-owned) and Go (fully wired end-to-end). `@atproto/repo` provides `signCommit` / `verifyCommitSig` as primitives, `verifyRepo` / `verifyDiff` as bundled checks on top of CAR input, but **DID → didKey resolution is still the caller's job**. Pass the resolved `did:key:…` string into the verifier; the package does not resolve DIDs. + +Source: `packages/repo/src/types.ts`, `packages/repo/src/util.ts`, `packages/repo/src/sync/consumer.ts`. + +## Commit shape + +Defined as zod schemas in `types.ts`: + +```typescript +const unsignedCommit = z.object({ + did: z.string(), + version: z.literal(3), + data: cidSchema, + rev: z.string(), + prev: cidSchema.nullable(), +}) +export type UnsignedCommit = z.infer & { sig?: never } + +const commit = z.object({ + did: z.string(), + version: z.literal(3), + data: cidSchema, + rev: z.string(), + prev: cidSchema.nullable(), + sig: z.instanceof(Uint8Array), +}) +export type Commit = z.infer +``` + +`prev: cidSchema.nullable()` — **not optional**. `prev` is **always** present as a map key. Genesis commits have `prev: null`; non-genesis have `prev: `. + +There's also a `LegacyV2Commit` type for reading (`version: 2`, `rev` optional) and `VersionedCommit = Commit | LegacyV2Commit` as a discriminated union. `ensureV3Commit(commit)` in `util.ts` upgrades a v2 commit by filling `rev = commit.rev ?? TID.nextStr()`. + +## `prev`: null vs omitted + +**Same divergence as Go-vs-Rust.** + +TS always serializes `prev`. Genesis commit on the wire: 5-entry map (CBOR header `a5`), `prev` key with CBOR null (`0xF6`) as value. Matches the spec-strict form and matches Go's output. + +The Rust reference impl (`atproto-repo`) uses `#[serde(skip_serializing_if = "Option::is_none")]` on `prev`, so a genesis commit has 4 entries (`a4` header) and no `prev` key. **A commit signed by Rust's reference impl will not verify via `verifyCommitSig` after round-tripping through TS's `UnsignedCommit`**, because re-encoding adds `prev: null` and produces different bytes. + +Practical implication: TS-to-TS and TS-to-Go verification is fine. Rust-signed genesis commits need special handling — ideally verify against the raw commit bytes from the CAR without re-encoding, but `verifyCommitSig` doesn't expose that path. You'd have to CBOR-decode, strip the `sig` key, and re-encode manually. + +See `../shared/commit-and-signing.md` §1.1. + +## DAG-CBOR field order + +TS uses `@atproto/lex-cbor`, which sorts map keys **bytewise** on encode. Bytewise order of the commit fields: `data, did, prev, rev, sig, version`. This matches the DRISL canonical order — TS is always canonical on the wire. + +(Go's cbor-gen sorts by struct declaration order — which happens to match bytewise for the commit struct but only by luck. See `go/drisl.md`.) + +## Signing + +```typescript +import { signCommit } from '@atproto/repo' +import { Secp256k1Keypair } from '@atproto/crypto' + +const keypair = await Secp256k1Keypair.create() // or .import(privateBytes) + +const unsigned: UnsignedCommit = { + did: 'did:plc:example', + version: 3, + data: mstRootCid, + rev: TID.nextStr(), + prev: null, // or previous commit CID +} + +const commit: Commit = await signCommit(unsigned, keypair) +// commit.sig is a Uint8Array of raw r||s bytes, low-S normalized. +``` + +Source: `util.ts`: + +```typescript +export const signCommit = async ( + unsigned: UnsignedCommit, + keypair: Keypair, +): Promise => { + const encoded = cbor.encode(unsigned) + const sig = await keypair.sign(encoded) + return { ...unsigned, sig } +} +``` + +The `Keypair` interface (from `@atproto/crypto`) guarantees `sign` returns raw `r||s` bytes, low-S normalized — the atproto signature shape. + +## Verification — low-level + +```typescript +import { verifyCommitSig } from '@atproto/repo' + +const valid: boolean = await verifyCommitSig(commit, didKey) +// didKey is the did:key:... string for the current #atproto signing key. +// Returns boolean; does not throw on bad signature. +``` + +Source: `util.ts`: + +```typescript +export const verifyCommitSig = async ( + commit: Commit, + didKey: string, +): Promise => { + const { sig, ...rest } = commit + const encoded = cbor.encode(rest) + return crypto.verifySignature(didKey, encoded, sig) +} +``` + +`crypto.verifySignature` from `@atproto/crypto` dispatches to k-256 or p-256 based on the multibase prefix inside `didKey`. Low-S normalization is applied. + +**The `{ sig, ...rest }` destructuring** is what makes this work for `prev: null` commits — it re-encodes the commit-minus-sig through the same canonical encoder that signed it. Round-tripping is safe because both sign and verify go through `cbor.encode`. The failure mode is cross-implementation: a commit signed by something with different encoding rules (e.g., Rust omitting `prev`) won't verify here. + +## Verification — bundled against a CAR + +```typescript +import { verifyRepoCar, verifyRepo, verifyDiffCar, verifyDiff } from '@atproto/repo' + +// Full repo CAR, e.g. from com.atproto.sync.getRepo +const verified: VerifiedRepo = await verifyRepoCar(carBytes, did, didKey) +// verified.creates: RecordCreateDescript[] — every record in the repo +// verified.commit: CommitData +``` + +```typescript +// Delta CAR: firehose #commit event or #sync event +const verified: VerifiedDiff = await verifyDiffCar(priorRepo, carBytes, did, didKey) +// verified.writes: RecordWriteDescript[] — creates/updates/deletes vs priorRepo +// verified.commit: CommitData (cid, rev, since, prev, newBlocks, ...) +``` + +Behaviour: + +1. `readCarWithRoot(carBytes)` — CAR-level CID verification on every block (see `car.md`). +2. Load commit via `storage.readObj(root, def.commit)` — zod validates commit shape. +3. If `did` passed: assert `commit.did === did`. +4. If `didKey` passed: `verifyCommitSig(commit, didKey)` — fails with `RepoVerificationError` if invalid. +5. `DataDiff.of(newTree, priorTree)` — compute the change set. +6. Assert leaf blocks referenced by new MST nodes are present (unless `opts.ensureLeaves === false`). + +`did` and `didKey` are optional — pass both for full verification, omit for reading only. + +## Inductive firehose verification + +**Go has `VerifyCommitMessage` that inverts each op, reapplies to a copied tree, and compares the resulting root to `msg.PrevData`. TypeScript does not ship this.** + +TS's equivalent is `verifyDiff(priorRepo, newBlocks, newRoot, did, signingKey)`: it computes the diff between the new tree and the prior repo's tree (via `SyncStorage` that reads from the prior store for unchanged subtrees), producing the same `RecordWriteDescript[]` that a firehose event would declare. If the diff doesn't match the ops in the event, the caller catches the discrepancy manually by comparing `verified.writes` to `msg.ops`. There's no single-call "inductive verification" helper. + +If you're reimplementing Go's flow manually: + +```typescript +// 1. Load delta CAR into the existing repo's storage. +const { root, blocks } = await readCarWithRoot(msg.blocks) +const storage = new SyncStorage(new MemoryBlockstore(blocks), priorRepo.storage) + +// 2. Verify sig. +const newRepo = await ReadableRepo.load(storage, root) +await verifyCommitSig(newRepo.commit, didKey) + +// 3. Compute diff and compare to msg.ops. +const diff = await DataDiff.of(newRepo.data, priorRepo.data) +const writes = await diffToWriteDescripts(diff) +// compare `writes` vs `msg.ops` to catch op fabrication. + +// 4. If msg includes prevData, check priorRepo.data root equals it. +// (This is the analog of Go's "inverted tree root == msg.PrevData".) +``` + +Caller writes the glue. + +## Key rotation — verifying older commits + +`didKey` is the **current** signing key. Older commits were signed under historical keys. `verifyCommitSig` fails on rotation with `return false`. + +No TS helper wires historical-key lookup into commit verification. For `did:plc`, walk the operation log via `@atproto/identity` (or a direct HTTP call to `plc.directory//log/audit`) and retry with the historic didKey. For `did:web`, historical documents aren't retrievable; older commits are permanently unverifiable post-rotation. + +See `../shared/commit-and-signing.md` §7 and the `atproto-identity-resolution` skill. + +## Producing a new commit + +The higher-level flow, combining MST, blocks, and signing: + +```typescript +import { Repo, WriteOpAction } from '@atproto/repo' + +const repo = await Repo.load(storage) // or Repo.create(...) for genesis + +const update = await repo.formatCommit( + [ + { + action: WriteOpAction.Create, + collection: 'app.bsky.feed.post', + rkey: TID.nextStr(), + record: { $type: 'app.bsky.feed.post', text: 'hi', createdAt: now }, + }, + ], + keypair, +) +// update: RepoUpdate { cid, rev, prev, since, newBlocks, relevantBlocks, removedCids, ops } + +await storage.applyCommit(update) +// update.newBlocks now persisted; repo root updated to update.cid. +``` + +`Repo.formatCommit` does: + +1. Apply each `RecordWriteOp` to the MST (immutable returns — tracked internally). +2. Collect new MST nodes and new record blocks into `newBlocks`. +3. Build the `UnsignedCommit` with `data = newTree.getPointer()`, `prev = repo.cid`, `rev = TID.nextStr()`. +4. `signCommit(unsigned, keypair)` — produce signature. +5. Add the signed commit block to `newBlocks`; return `RepoUpdate`. + +`Repo.formatInitCommit` / `Repo.create` do the genesis equivalent with `prev: null`. + +## File pointers + +| Concern | File | +| ------------------------------------ | ---------------------------------------------------- | +| `Commit`, `UnsignedCommit`, schemas | `packages/repo/src/types.ts` | +| `signCommit`, `verifyCommitSig` | `packages/repo/src/util.ts` | +| `ensureV3Commit` | `packages/repo/src/util.ts` | +| `verifyRepo`, `verifyDiff`, `verifyProofs` | `packages/repo/src/sync/consumer.ts` | +| `RepoVerificationError` | `packages/repo/src/sync/consumer.ts` | +| `Repo.formatCommit`, `Repo.formatInitCommit` | `packages/repo/src/repo.ts` | +| `Keypair` interface | `packages/crypto/src/` (external) | +| `crypto.verifySignature` | `packages/crypto/src/` (external) | + +## Common errors + +| Error | Cause | +| ----------------------------------------------- | ---------------------------------------------------------------------- | +| zod validation error on `Commit` schema | Commit bytes don't match `{ did, version: 3, data, rev, prev, sig }`. Usually v2 (run through `ensureV3Commit`) or corruption. | +| `RepoVerificationError: Invalid repo did: ` | `verifyRepo` / `verifyProofs`: `commit.did` doesn't match the expected `did`. | +| `RepoVerificationError: Invalid signature on commit: ` | `verifyCommitSig` returned false. Rotation, or prev-null divergence, or bad didKey. | +| `verifyCommitSig` returns `false` | Sig invalid under given didKey. Diagnose: check didKey is current, or try historical keys for rotated accounts. | +| `missing leaf blocks: ` | `verifyDiff` with `ensureLeaves: true` and new MST nodes reference leaves whose blocks aren't in the CAR. | + +## See also + +- `../shared/commit-and-signing.md` — language-neutral commit & signing rules (includes §1.1 prev divergence). +- `drisl.md` — `cbor.encode(unsigned)` is what `signCommit` signs. +- `mst.md` — `tree.getPointer()` produces `commit.data`. +- `car.md` — end-to-end verification against a CAR. +- `../shared/divergence-matrix.md` §commit — TS vs Rust (`prev` omit) vs Go (fully wired via `VerifyCommitSignatureFromCar`). +- `atproto-identity-resolution` skill — resolving DIDs to `did:key:…` signing keys. diff --git a/skills/software-development/atproto-repository/references/typescript/drisl.md b/skills/software-development/atproto-repository/references/typescript/drisl.md new file mode 100644 index 0000000..171268e --- /dev/null +++ b/skills/software-development/atproto-repository/references/typescript/drisl.md @@ -0,0 +1,142 @@ +# TypeScript — DAG-CBOR encoding via `@atproto/lex-cbor` + +TypeScript does canonical DAG-CBOR through `@atproto/lex-cbor`. The term "DRISL" does not appear in the package — Bluesky calls it "lex-cbor" or just "canonical DAG-CBOR" — but the rules of `../shared/drisl.md` apply and the encoder is conformant. + +`@atproto/repo` never touches CBOR directly. Every encode/decode goes through `lex-cbor`: + +```typescript +import * as cbor from '@atproto/lex-cbor' +// packages/repo/src/util.ts, /repo.ts, /car.ts, /mst/mst.ts all import this. + +const bytes = cbor.encode(value) // canonical DAG-CBOR +const value = cbor.decode(bytes) // parses into LexValue +const cid = await cbor.cidForLex(value) // encode → sha-256 → 0x71 (dag-cbor) CID +``` + +## Public encoding API + +From `@atproto/lex-cbor`: + +- `encode(value: LexValue): Uint8Array` — canonical DAG-CBOR bytes. +- `decode(bytes: Uint8Array): LexValue` — parses the bytes into the lex data model. +- `cidForLex(value: LexValue): Promise` — encode then CID (dag-cbor, sha-256, CIDv1). +- `encodeBlock(value)` — returns `{ cid, bytes }` in one shot (used internally to build MST blocks and commit blocks). + +From `@atproto/lex-data`: + +- `Cid` — the CID class used everywhere. Not `CID` (pascal in @ipld/dag-cbor); the TS atproto stack uses `Cid`. +- `decodeCid(bytes: Uint8Array): Cid` — parse a binary CID prefix (36 bytes for standard sha-256 dag-cbor). +- `isCidForBytes(cid: Cid, bytes: Uint8Array): Promise` — re-hash `bytes` and compare to `cid`. Used by the CAR verifier. +- `LexMap`, `LexValue`, `LexArray` — the typed-value tree shapes (objects, primitives, CID links, typed bytes, blob refs). +- `ifCid(unknown): Cid | null` — narrow an unknown value to a `Cid`. + +## How records round-trip + +Raw block bytes → `LexMap` (record): + +```typescript +import { cborToLexRecord } from '@atproto/repo' + +const record = cborToLexRecord(blockBytes) // returns LexMap, i.e. plain object +// record is a typed JS object with any nested { $link, $bytes, blob } values parsed. +``` + +`LexMap` values: + +| In the DAG-CBOR | In `LexMap` | +| --------------------------------- | ------------------------------------------------- | +| CBOR tag 42 (CID link) | A `Cid` instance | +| CBOR byte string (major type 2) | `Uint8Array` | +| `{ $type: "blob", ref, mimeType, size }` | A typed blob reference object | +| Strings / numbers / booleans | Native JS values | +| Maps with bytewise-sorted string keys | Plain JS objects | + +The TS side represents CIDs as **`Cid` instances** in memory, not as `{$link: "..."}` wrappers. JSON serialization of a `Cid` produces `{"$link": "..."}`; DAG-CBOR serialization produces tag 42. You generally don't construct the `$link` wrapper by hand — pass a `Cid` object and the encoder handles the shape. + +## Canonical-encoding guarantees + +`encode` emits canonical DAG-CBOR: + +- Map keys sorted **bytewise** (not by struct declaration order — TS doesn't have struct-declaration-order the way Go's cbor-gen does). +- Integers in shortest form. +- No indefinite-length framing. +- CIDs as tag 42 wrapping the 37-byte identity-multibase-prefixed binary CID form. + +Because the encoder canonicalizes on every call, round-tripping an arbitrary object produces stable bytes. This is what signature verification depends on: `verifyCommitSig` does `cbor.encode(commitWithoutSig)` and feeds that to the crypto library — and because the commit was signed via the same encoder, the bytes match. + +On the decoder side, `decode` does NOT strict-check canonicalness. Non-canonical input will decode successfully but re-encoding may produce different bytes. For signature verification this is usually fine because you're verifying against the same encoder that produced the input. For untrusted input where re-encoded-CID comparison matters, there's no in-box strict canonical decoder; you'd re-hash the decoded bytes and compare to the claimed CID. + +## Size and validation limits + +Unlike Go (`atdata` enforces 1 MiB record size, 128k container length, 1 MiB string length) and Rust, **`@atproto/lex-cbor` does not enforce record size or container count limits at the encode/decode layer**. Those are enforced at higher layers: + +- **PDS ingest** — the PDS rejects records > 1 MiB at the XRPC layer, not at the CBOR layer. +- **Lexicon validation** — when a record is round-tripped through a lex schema (`cborToLexRecord` followed by schema validation), individual field length/type constraints are enforced by the schema. + +If you're reading potentially-adversarial CAR input, wrap `readCar` with a size cap on the input `Uint8Array` before passing it in; the CAR reader itself has no size guardrails. + +## CID shape — 36 vs 37 bytes + +- **Inside a CAR frame** (per block): the binary CID is 36 bytes (no multibase prefix): `0x01 0x71 0x12 0x20 <32-byte sha-256>`. TS handles this correctly via `decodeCid(blockBytes.subarray(0, 36))` in `car.ts`. +- **Inside a DAG-CBOR tag 42**: 37 bytes with identity-multibase prefix `0x00`. TS handles this inside the encoder. + +See `../../atproto-cid/shared/binary-layout.md`. + +## Typed bytes (`$bytes`) and blob refs + +For record fields that are raw bytes, the DAG-CBOR wire form is a CBOR byte string (major type 2). JSON representation: `{"$bytes": ""}`. The TS `LexMap` represents this as a `Uint8Array` in memory — no wrapper class is needed. `encode` converts `Uint8Array` to major-type-2 bytes automatically. + +Blob references: + +```typescript +// In a LexMap, a blob field looks like: +{ + $type: 'blob', + ref: Cid { ... }, // a Cid instance + mimeType: 'image/png', + size: 123456, +} +``` + +There's no first-class `Blob` class in `@atproto/repo` analogous to Go's `atdata.Blob` — callers work with plain objects that happen to have these keys. + +## Divergences from Rust/Go worth remembering + +| Aspect | TS (`@atproto/lex-cbor`) | Rust (`atproto-dasl`) | Go (cbor-gen + `atdata`) | +| --------------------------------- | ------------------------------------------------- | ---------------------------------------------- | ----------------------------------------------- | +| Map key ordering on encode | Bytewise | Bytewise | **Struct declaration order** (cbor-gen quirk) | +| CID in memory | `Cid` class | `atproto_dasl::Cid` | `cid.Cid` (or `CIDLink` wrapper) | +| Size limits at CBOR layer | None — enforced at PDS/XRPC | Configurable in `atproto-dasl` | Hard-coded in `atdata/const.go` | +| Strict canonical decode | No | No | No (delegates to go-ipld-cbor) | +| Typed bytes wrapper | Plain `Uint8Array` | `atproto_dasl::Bytes` | `atdata.Bytes` | + +See `../shared/divergence-matrix.md` §drisl for the full table. + +## Common errors + +| Error | Cause | +| ------------------------------------------ | ----------------------------------------------------------------------------- | +| `Not a valid CID` | `ifCid` / zod `cidSchema` rejected a field that was supposed to be a CID. | +| `Not a valid CID for bytes ()` | CAR ingest: block bytes don't hash to the claimed CID. Corruption or forgery. | +| `Could not parse CAR header` | Header CBOR doesn't match the `{ version: 1, roots: Cid[] }` zod schema. | +| `lexicon records be a json object` | `cborToLexRecord` received bytes that decoded to a non-object (primitive / array). | + +## File pointers + +| Concern | File | +| ---------------------------- | ------------------------------------------------------ | +| Commit encode/decode | `packages/repo/src/util.ts` (`signCommit`, `verifyCommitSig`) | +| MST encode/decode | `packages/repo/src/mst/mst.ts` (`serialize`, `getEntries`) | +| CAR encode/decode | `packages/repo/src/car.ts` | +| `cborToLex`, `cborToLexRecord` | `packages/repo/src/util.ts` | +| Canonical encoder | `packages/lex-cbor/src/` (external) | +| `Cid` / `LexValue` / `decodeCid` | `packages/lex-data/src/` (external) | + +## See also + +- `../shared/drisl.md` — language-neutral canonical DAG-CBOR rules. +- `car.md` — how encoded blocks flow through CAR frames. +- `mst.md` — `NodeData` / tree entry serialization. +- `commit.md` — `cbor.encode(unsigned)` is what `signCommit` signs. +- `../shared/divergence-matrix.md` §drisl — interop matrix across TS/Rust/Go. +- `../../atproto-cid/typescript/` — CID construction idioms in TS. diff --git a/skills/software-development/atproto-repository/references/typescript/mst.md b/skills/software-development/atproto-repository/references/typescript/mst.md new file mode 100644 index 0000000..8bc3e7e --- /dev/null +++ b/skills/software-development/atproto-repository/references/typescript/mst.md @@ -0,0 +1,241 @@ +# TypeScript — `MST` class (immutable Merkle Search Tree) + +The TS `MST` is **immutable**: every mutation (`add`, `update`, `delete`) returns a **new** `MST` with `outdatedPointer = true`; the old value is still valid and still reflects its pre-mutation state. This is the biggest deviation from Rust (mutable in place, fed to a `BlockStorage`) and Go (mutable `Tree` with a `Root *Node` pointer). + +Source: `packages/repo/src/mst/mst.ts`. + +## Node shape on the wire + +```typescript +type NodeData = { + l: Cid | null // left-most subtree pointer + e: TreeEntry[] // entries +} + +type TreeEntry = { + p: number // prefix length shared with previous key + k: Uint8Array // rest of key (ASCII bytes after the prefix) + v: Cid // leaf value CID + t: Cid | null // right subtree pointer for this leaf +} +``` + +The schemas live as zod types in `mst.ts:50-56`. Field names (`l`, `e`, `p`, `k`, `v`, `t`) match Rust and Go — the wire format is identical across implementations. + +Each `TreeEntry` carries a leaf *and* the subtree to its right. The leftmost subtree is the top-level `l` field. Prefix compression: `key = lastKey.slice(0, p) + asciiDecode(k)`. + +## In-memory types + +```typescript +type NodeEntry = MST | Leaf + +class Leaf { + key: string // collection/rkey, e.g. "app.bsky.feed.post/3k2..." + value: Cid + isTree(): this is MST + isLeaf(): this is Leaf +} + +class MST { + storage: ReadableBlockstore + pointer: Cid // may be outdated after mutation + entries: NodeEntry[] | null // null if lazily loaded + layer: number | null + outdatedPointer: boolean +} +``` + +An `MST` is either loaded (entries in memory) or lazy (`entries === null`, `pointer` valid). `getEntries()` resolves lazy by reading the node block from `storage` and deserializing. + +## Construction + +```typescript +// Empty tree in a fresh storage. +const tree = await MST.create(storage) + +// Tree pointing at an existing CID (lazy — no storage read). +const tree = MST.load(storage, rootCid) + +// Tree with entries already known. +const tree = await MST.fromData(storage, nodeData) +``` + +`MST.create` computes the CID for an empty entry list. `MST.load` is the usual entry point for verification — it defers the storage read until the first traversal. + +## Read API — all async + +```typescript +async get(key: string): Promise +async getEntries(): Promise +async getPointer(): Promise // re-serializes if pointer is outdated +async getLayer(): Promise + +async *walk(): AsyncIterable // depth-first +async *walkFrom(key: string): AsyncIterable +async *walkLeavesFrom(key: string): AsyncIterable +async leaves(): Promise +async leafCount(): Promise +async list(count?, after?, before?): Promise +async listWithPrefix(prefix: string, count?): Promise + +async serialize(): Promise<{ cid: Cid; bytes: Uint8Array }> +async getUnstoredBlocks(): Promise<{ root: Cid; blocks: BlockMap }> +async cidsForPath(key: string): Promise +async getCoveringProof(key: string): Promise +``` + +Everything is async because MST entries may not be loaded yet — any traversal might hit storage. `get(key)` returns `null` if the key doesn't exist, a `Cid` if it does. + +## Write API — immutable + +```typescript +async add(key: string, value: Cid, knownZeros?: number): Promise // throws if key exists +async update(key: string, value: Cid): Promise // throws if key absent +async delete(key: string): Promise // throws if key absent +``` + +Each returns a **new** `MST` instance. The `outdatedPointer` flag is set to `true`; `getPointer()` / `serialize()` will re-hash on demand. + +Usage: + +```typescript +let tree = await MST.create(storage) +tree = await tree.add('app.bsky.feed.post/abc', recordCid1) +tree = await tree.add('app.bsky.feed.post/xyz', recordCid2) +tree = await tree.update('app.bsky.feed.post/abc', recordCid3) +// At this point `tree` reflects 2 entries. The earlier tree values are also still valid. +const rootCid = await tree.getPointer() +``` + +`add`'s `knownZeros` parameter lets you skip re-hashing the key to determine its layer when you already know it (the internal recursion uses this). + +## Key format and validation + +Keys are `${collection}/${rkey}` strings. Enforced by `ensureValidMstKey`: + +```typescript +// Total length ≤ 1024 chars +// Exactly two segments separated by '/' +// Each segment non-empty +// Characters matching /^[a-zA-Z0-9_~\-:.]*$/ +``` + +Source: `mst/util.ts`. `InvalidMstKeyError` is thrown on violation. + +Go and Rust enforce `MAX_KEY_BYTES = 1024` but don't enforce the two-segment structure at the MST layer. TS is stricter — it rejects any key that isn't `/` shape. + +## Height (layer) computation + +```typescript +export const leadingZerosOnHash = async (key: string | Uint8Array) => { + const hash = await sha256(key) + let leadingZeros = 0 + for (let i = 0; i < hash.length; i++) { + const byte = hash[i] + if (byte < 64) leadingZeros++ + if (byte < 16) leadingZeros++ + if (byte < 4) leadingZeros++ + if (byte === 0) { leadingZeros++ } else { break } + } + return leadingZeros +} +``` + +Source: `mst/util.ts:24`. Counts leading **pairs of zero bits** in the sha-256 hash — fanout 4, matching Rust and Go (despite Go's misleading `// fanout: 16` comment). The key is hashed as **ASCII bytes of the string**, not UTF-8 — this matters only for keys with non-ASCII characters, which are rejected by `ensureValidMstKey` anyway. + +## Diff API + +```typescript +import { DataDiff } from '@atproto/repo' + +const diff = await DataDiff.of(newerTree, olderTree) +// Or: await DataDiff.of(newerTree, null) — diff against an empty tree (used for full-repo verification). + +diff.addList() // { key, cid }[] +diff.updateList() // { key, cid, prev }[] +diff.deleteList() // { key, cid }[] +diff.newMstBlocks // BlockMap — new MST node blocks produced by this diff +diff.newLeafCids // CidSet — new leaf (record) CIDs referenced but not computed +diff.removedCids // CidSet — MST nodes and leaves removed +``` + +`DataDiff` walks both trees in parallel and emits the minimum set of changes. The `newMstBlocks` is what gets written into the CAR for a delta export; `newLeafCids` tells you which record blocks still need to be included. + +Source: `packages/repo/src/data-diff.ts`. + +## Serialization + +```typescript +const { cid, bytes } = await tree.serialize() +// Equivalent: await cidForLex(data), where data = { l, e: [...] } +``` + +`serialize()` calls `getEntries()`, refreshes any outdated subtree pointers, builds a `NodeData` via `util.serializeNodeData`, encodes through `@atproto/lex-cbor`, and hashes. + +`getUnstoredBlocks()` recursively collects every MST block (self + subtrees) that isn't already in `storage`. Use this to materialize the MST into a `BlockMap` for CAR writing: + +```typescript +const { root, blocks } = await tree.getUnstoredBlocks() +// blocks contains every new MST node. Merge with record blocks before writing CAR. +``` + +## Partial trees + +A "partial tree" is an `MST` whose some subtrees reference CIDs that aren't in `storage`. Firehose delta CARs produce this state: unchanged subtrees are referenced by CID but their blocks aren't included. + +**TS semantics**: any operation that tries to load a missing subtree throws a `MissingBlockError` (from `packages/repo/src/error.ts`). There's no `ErrPartialTree` sentinel like Go, and no `is_partial()` method like Rust. + +To operate safely against a possibly-partial tree, you either: + +- Use `DataDiff.of(newerTree, olderTree)` where `olderTree` has the missing blocks — `DataDiff` only loads subtrees that actually differ, so untouched subtrees are never resolved. +- Use `verifyDiff` with a `SyncStorage` that falls through to the prior repo's store (see `car.md`). +- Catch `MissingBlockError` and treat it as "this operation can't complete without more blocks". + +## Covering proofs + +```typescript +const proofBlocks = await tree.getCoveringProof(key) +// BlockMap containing every node on the path to `key`, plus the leaf block. +// Suitable for producing an inclusion proof CAR. +``` + +Used by `com.atproto.sync.getRecord` to return a minimal CAR that proves a specific record exists in the committed MST. + +## Structural validation + +Unlike Go's `Tree.Verify` (which checks ascending keys, correct heights, no-sibling-children), **TypeScript does not ship a top-level structural verifier**. Trees built through the public API are always valid by construction: `add`/`update`/`delete` maintain invariants, `getEntries` reconstructs keys from prefix data and rejects invalid keys via `ensureValidMstKey`. + +For adversarial input, the invariants you care about are usually enforced at CBOR decode (via zod) or at traversal time (missing blocks throw, invalid keys throw). There's no standalone `verify()` call. + +## File pointers + +| Concern | File | +| ------------------------------ | ---------------------------------------- | +| `MST` class | `packages/repo/src/mst/mst.ts` | +| `Leaf` class | `packages/repo/src/mst/mst.ts` | +| `NodeData`, `TreeEntry` schema | `packages/repo/src/mst/mst.ts:50-56` | +| `leadingZerosOnHash` | `packages/repo/src/mst/util.ts:24` | +| `ensureValidMstKey` | `packages/repo/src/mst/util.ts` | +| `serializeNodeData`, `deserializeNodeData` | `packages/repo/src/mst/util.ts` | +| `DataDiff` | `packages/repo/src/data-diff.ts` | +| `MissingBlockError` | `packages/repo/src/error.ts` | +| Walker | `packages/repo/src/mst/walker.ts` | +| Diff algorithm | `packages/repo/src/mst/diff.ts` | + +## Common errors + +| Error | Cause | +| --------------------------------- | ---------------------------------------------------------------------- | +| `InvalidMstKeyError: ` | Key isn't `/`, or contains invalid chars, or > 1024. | +| `There is already a value at key: ` | `add` called on an existing key. Use `update` instead. | +| `Could not find a record with key: ` | `update` / `delete` called on a non-existent key. | +| `MissingBlockError` | Tried to load an MST node whose block isn't in `storage` (partial tree). | +| `Not a valid node: two subtrees next to each other` | Serialization invariant violated — bug or corrupt in-memory tree. | + +## See also + +- `../shared/mst.md` — language-neutral MST algorithm and node format. +- `drisl.md` — canonical encoding of `NodeData`. +- `car.md` — writing MST blocks to a CAR; partial-tree handling in firehose events. +- `commit.md` — `commit.data` is `tree.getPointer()`. +- `../shared/divergence-matrix.md` §mst — immutable (TS) vs mutable (Rust/Go). -- 2.51.2