diff --git a/devlog/inductive-proof-sdk-affordances.md b/devlog/inductive-proof-sdk-affordances.md new file mode 100644 index 0000000..0af8c44 --- /dev/null +++ b/devlog/inductive-proof-sdk-affordances.md @@ -0,0 +1,114 @@ +> adopted from the notes repo during the 2026-08-23 curation pass; project design/retro content lives with its project. + +# SDK affordances + +general patterns surfaced by studying sync 1.1 integration in [zlay](https://tangled.org/zat.dev/zlay) (an atproto relay). the question: what could [zat](https://tangled.org/zat.dev/zat) (the atproto building-blocks library underneath it) provide that's broadly useful without being experimental or over-specific? + +## pattern 1: firehose frame parsing + +**the problem**: zlay manually extracts fields from decoded CBOR `Value` maps — `getString("repo")`, `get("prevData")`, `getString("since")`, etc. this is repeated across frame_worker.zig and subscriber.zig with subtly different field names for `#commit` vs `#sync` vs `#account` events. + +**what exists**: zat's `FirehoseClient` already decodes frames into typed `FirehoseEvent` structs. but zlay doesn't use it — zlay has its own subscriber that connects to individual PDSes, not upstream relays. it receives the same wire format but doesn't get the typed decode. + +**the affordance**: separate the frame *parsing* from the frame *receiving*. a function like: + +``` +parseFirehoseFrame(header_bytes, payload_bytes) → FirehoseEvent | error +``` + +this is already mostly what the firehose client does internally. exposing it independently means any consumer with raw websocket frames can get typed events without adopting the full client lifecycle. + +**generality**: high. any custom subscriber (relay, indexer, archiver) that handles its own websocket connection would benefit. the wire format is stable — it's the spec. + +**status**: not yet justified — zlay is the only consumer (4 days old). wait for a second consumer to hit this need; declining now is reversible, a public API is forever. + +## pattern 2: operation type bridging + +**the problem**: the firehose wire format and the MST internal format use different representations for the same concept: + +| field | firehose (`repoOp`) | MST (`Operation`) | +|---|---|---| +| path | `path` (single string) | `path` | +| new value | `cid` (nullable CID) | `value` (nullable raw bytes) | +| old value | `prev` (nullable CID) | `prev` (nullable raw bytes) | +| action | `action` string | inferred from value/prev nullability | + +zlay's `extractOps` manually converts between these. it got the field names wrong for months. + +**the affordance**: a conversion function: + +``` +repoOpToMstOperation(repo_op) → MstOperation +``` + +or better: have `verifyCommitDiff` accept `RepoOp[]` directly, doing the conversion internally. + +**generality**: high. this is a spec-level mapping, not an implementation detail. + +## pattern 3: chain state tracking + +**the problem**: every consumer that verifies commits needs to: +1. store (rev, data_cid) per DID +2. compare incoming (since, prevData) against stored state +3. update state after successful verification +4. handle chain breaks (log, mark desynchronized, queue resync) + +this is ~50 lines of boilerplate that's easy to get wrong (race conditions on concurrent updates, multibase encoding of CIDs for comparison, rev ordering). + +**what the SDK could offer**: not a full state machine (that's too opinionated), but: +- a `ChainState` struct: `{ rev: []const u8, data_cid: Cid }` +- a `checkContinuity(stored: ChainState, incoming_since: ?[]const u8, incoming_prev_data: ?Cid) → .ok | .break_since | .break_prev_data | .first_commit` +- let the caller decide what to do on break + +**generality**: medium-high. the continuity check is pure logic — no I/O, no storage assumptions. but it's also simple enough that it's borderline "just write it." the value is in being *correct* — the field comparison has encoding subtleties. + +**hesitation**: this might be too thin to justify SDK surface area. maybe it belongs in documentation/examples rather than code. + +## pattern 4: verify result → storage-ready output + +**the problem**: both `verifyCommitCar` and `verifyCommitDiff` return result structs with raw CID bytes. consumers invariably need to: +1. multibase-encode the data CID for storage +2. extract the rev string +3. store both as the new chain state + +zlay does this manually after every verify call. + +**the affordance**: a method on the verify result: + +``` +result.chainState(allocator) → ChainState { .rev = "3jui...", .data_cid_encoded = "bafyrei..." } +``` + +or just make the verify results include the encoded form. + +**generality**: medium. convenient, but it's two lines of code. might not clear the bar. + +## pattern 5: commit parsing without verification + +**the problem**: sometimes you want to extract commit metadata (DID, rev, data CID) from a CAR *without* running the full verification pipeline. zlay does this in the chain continuity check path — it needs the fields for comparison before deciding whether to verify. + +**what exists**: `loadCommitFromCAR` does exactly this, but it's currently internal to the verify module. + +**the affordance**: make `loadCommitFromCAR` public, or provide a lightweight `parseCommit(car_bytes) → CommitMetadata` that stops after extracting fields. + +**generality**: high. indexers, archivers, and debugging tools all want commit metadata without paying for verification. + +## what NOT to add + +things that seem useful but are too specific or experimental: + +- **stateful chain verifier with storage backend** — too opinionated about persistence. every consumer has different storage (postgres, rocksdb, sqlite, in-memory). +- **automatic resync orchestration** — involves HTTP fetching, backoff, thundering herd mitigation. this is operational, not SDK-level. +- **DID resolution caching** — zlay's LRU cache is highly tuned to its concurrency model. a generic cache would either be too simple or too complex. +- **rate limiting / admission control** — purely operational. + +## summary: what seems worth pursuing + +ranked by (generality × value / complexity): + +1. **expose frame parsing independently** — high value, low complexity, clearly general +2. **operation type bridging** — prevents a real class of bugs, spec-level mapping +3. **make `loadCommitFromCAR` public** — zero-cost, already implemented, broadly useful +4. **chain continuity check helper** — useful but borderline; maybe docs/examples instead + +the theme: **make the SDK's internals composable**. zlay doesn't use the firehose client but needs the frame parser. it doesn't need the full verify pipeline but needs commit metadata. the value is in unbundling, not in adding new features.