stream #

a zig implementation of jetstream — the full-network archive and streaming service for atproto. built on zat.

Stream targets semantic parity with pinned Jetstream V2 commit f29815c. docs/semantic-parity.md is the authority on what that currently rests on, row by row, and no claim here overrides it.

what it does #

• live feeds at /subscribe and /subscribe-v2, with collection and DID filters, cursor replay, runtime option updates, RFC 7692 permessage-deflate on v1, frozen v1 zstd, and fetchable dictionary-negotiated v2 zstd
• a disk-backed historical archive through listSegments, getSegment, getBlock, and planBackfill. Sealed jss v1 segments are byte-compatible with upstream. Sealed headers, block envelopes, DID blooms, and collection summaries are resident and refresh atomically after seal, compaction, and timestamp rewrite; planning opens no segment files, and cold replay reads only selected compressed blocks
• whole-network bootstrap with concurrent live capture and deterministic merge. Repository completion is coupled to ordinary archive-writer durability independently of the dispatch/checkpoint batch, so a restart does not replay work already committed. Startup resumes the existing active segment, truncating and fsyncing only a framing-torn suffix while failing closed on complete-frame corruption and sealed high-water checksum failure
• Sync 1.1 commit verification — PLC key rotation, MST inversion, op-CID checks, replay protection, durable repository-chain state, and transparent authenticated whole-repo repair with ordered pending replay. The live scheduler implements Atmos-shaped 32-worker per-DID FIFO, bounded drop-oldest, and completion-order batching
• RocksDB-backed metadata and archive publication coupled to segment fsync boundaries
• tombstone-aware compaction and continuing repair of repositories that were unavailable or changed during bootstrap
• crash-resumable timestamp imports with a durable RocksDB rule map, topology-preserving sealed-segment repair, and bearer-gated operator XRPC
• operations views at /status — a summary field report plus hosts, accounts, collections, and segments tabs, with durable PDS diagnostics, per-repository reconstruction, and authoritative root verification against the PDS commit root. Prometheus metrics are on the optional debug listener at /metrics

develop #

zig build test && zig build   # unit tests + exe
just simulator                # local fake atproto network on :7777
just run-sim                  # stream against it

The full suite list, the oracles, and the power-loss tier are in docs/testing.md. The Zig build pins its Zig and native dependencies, including otel-zig, RocksDB, zstd, xxHash, and websocket.zig's zlib 1.3.2; it does not depend on a system zlib.

docs #

Start here depending on what you need:

The rest of docs/ holds the port record: the jss format byte spec and seal spec, upstream bootstrap/compaction/subscribe condensations where every deliberate divergence is written down, the live scheduler and repair notes, lessons carried over from zlay, and durable product/copy context for public surfaces.