stream #

a zig implementation of jetstream — the full-network archive and streaming service for atproto. built on zat.

A live instance runs at stream.waow.tech — Jetstream V2-compatible feeds with the whole network archived behind them (~23B events, ~16.3M repos, sealed segments back to seq 1), live-current with the relay. Consumer docs, including endpoints and cursor semantics, are at stream.waow.tech/llms.txt.

what it does #

• live feeds: /subscribe (Jetstream v1-compatible JSON) and /subscribe-v2 (sequence cursors + record CBOR), with collection/DID filters, cursor replay, and zstd
• the archive: listSegments, getSegment, getBlock, planBackfill — gap-free recovery over sealed jss segments byte-compatible with upstream. A working consumer (plan → fetch → decode → sqlite) is examples/backfill-collection-sqlite.py; best suited to dense collections and bulk drains (block-granular retrieval)
• whole-network bootstrap with concurrent live capture, deterministic merge, and crash-safe resume
• Sync 1.1 commit verification and continuing repair of repositories that were unavailable or changed during bootstrap
• operations views at /status, Prometheus /metrics on the debug listener

The full implemented surface is in docs/feature-surface.md. Stream targets semantic parity with pinned Jetstream V2 commit f29815c; docs/semantic-parity.md is the authority on what is proved, row by row, and no claim here overrides it.

develop #

zig build test && zig build   # unit tests + exe
just simulator                # local fake atproto network on :7777
just run-sim                  # stream against it

Suites, oracles, and the power-loss tier: docs/testing.md. The build pins Zig and all native deps (RocksDB, zstd, xxHash, zlib) — no system libraries required.

docs #

The rest of docs/ holds the port record: the jss format byte spec and seal spec, upstream bootstrap/compaction/subscribe condensations where every deliberate divergence is written down, the live scheduler and repair notes, lessons carried over from zlay, and durable product/copy context for public surfaces.