Architecture overview #

A fast orientation map: what the big pieces are, how they fit, and where to read more. Deliberately high-level so it doesn't rot when a function moves. Each module's //! header is authoritative for its own internals; when this file disagrees with one, the module is right and this file should be fixed.

The grouping below is deliberately the same one upstream Jetstream V2 uses in its specs/architecture.md — ingest, storage, serve, plus a testing rig. Sharing the vocabulary makes the parity audit legible: a row in semantic-parity.md names a surface, and that surface lands in exactly one subsystem here.

Stream is ~38,800 lines of Zig across 72 files. It is one static binary on one machine: it ingests every record from every repo on a relay, stores them in the JSS columnar format, and lets clients replay that history and cut over to the live firehose.

Reading the paths below. Every foo.zig / dir/foo.zig in this file is relative to src/internal/, which is grouped one directory per subsystem:

src/internal/{ingest,storage,serve,compact,runtime,timestamp}/

So ingest/backfill/merge.zig is src/internal/ingest/backfill/merge.zig, and a bare pipeline.zig lives under one of those six. If a name here does not resolve, the tree moved and this file is stale — find src -name pipeline.zig settles it in one command.

The three subsystems #

Everything is ingest (data in), storage (data on disk), or serve (data out).

Ingest — getting data onto disk #

A lifecycle state machine with three phases, owned by ingest/backfill/lifecycle.zig, with durable commit points between them; a crash mid-cutover re-enters the machine at the right spot.

• Bootstrap — ingest/backfill/engine.zig paginates listRepos and downloads every repo via getRepo (ingest/backfill/repos.zig, ingest/backfill/fetched_car.zig, ingest/backfill/prepared_fetch.zig; worker count is the only concurrency bound, as it is upstream), while ingest/backfill/capture.zig captures the firehose into a throwaway backfill/live archive so nothing is missed.
• Merge — ingest/backfill/merge.zig drains captured live segments into the permanent archive, dropping events already covered by each repo's backfilled head, then a compaction pass runs before serving ungates.
• Steady state — ingest.zig pumps the firehose through pipeline.zig (32 workers, FIFO per DID) into the archive. ingest/backfill/retry.zig re-downloads repos that failed; the merge's pending pass is the same runner with eligible_status flipped, which is how it inherits backoff and host parking.

convert.zig turns firehose events into wire frames and is the validation gate: invalid upstream data is dropped and counted with a labeled metric. verify.zig does Sync 1.1 signature/MST verification and repair.zig performs authenticated whole-repo repair. cursor.zig holds the durable relay cursor.

lifecycle / cutover	ingest/backfill/lifecycle.zig, ingest/backfill/phase.zig
initial backfill	ingest/backfill/engine.zig, ingest/backfill/repos.zig
per-repo state	ingest/backfill/repo_store.zig, ingest/backfill/host_store.zig
merge	ingest/backfill/merge.zig
retry / resync	ingest/backfill/retry.zig, ingest/backfill/resync.zig
live consumer	ingest.zig, pipeline.zig
conversion + validation	convert.zig, wire.zig
verification + repair	verify.zig, repair.zig

Four worker pools, four different counts #

"32 workers" and "100 workers" describe different pools:

pool	workers	bounded queue = workers * 2	source
initial backfill crawl	100	200	ingest/backfill/engine.zig default_workers
live scheduler (steady state)	32	64	ingest/pipeline.zig worker_count
live repair	32	64	ingest/repair.zig worker_count
failed-repo retry	16 (4 per host)	—	ingest/backfill/retry.zig default_workers

Only the first is operator-tunable in practice (--backfill-workers); the live counts match upstream Atmos defaults.

workers * 2 is one expression serving three pools. It bounds backfill dispatch, the live scheduler's per-DID pending capacity, and the repair job queue — so stream_backfill_queue_depth reads 200 while live-scheduler.md and live-repair.md both describe a "64", and all three are the same rule. It is also the bound in the batch submit/consume ordering rule (invariants.md, "a bounded producer must not outrun its consumer"), so raising --backfill-workers moves a deadlock-relevant bound, not just concurrency.

Storage — the segment format and the metadata store #

Two places hold state, and the durability ordering between them is the invariant that keeps a crash safe: segment fsync first, metadata commit second.

• Segment files — the columnar, zstd-compressed, append-only JSS logs. An active segment is a file state machine (append → flush → fsync → seal). segment_writer.zig writes blocks, segment_footer.zig finalizes the footer and header, segment.zig reads sealed files, and archive.zig owns rotation, sealing, and startup recovery. manifest.zig holds resident sealed-segment metadata (a directory scan plus self-describing headers, not a database).
• The metadata store — meta_store.zig, RocksDB, holds everything not cheaply re-derivable from segments: relay cursor, lifecycle phase, per-DID backfill status, host aggregates, compaction watermark.

compact/ performs delete/update compaction (pass.zig chunked passes, steady.zig the long-running compactor, watermark.zig the durable watermark), with tombstone.zig deciding which rows may be dropped and segment_rewrite.zig doing the physical rewrite. gloom.zig (blocked bloom), zstd.zig, and xxhash128.zig are the format's primitives.

Serve — getting data out #

• /subscribe websocket (server.zig) — pull-based fan-out. Every subscriber is served from wherever its cursor points: tail.zig (the in-memory hot tail) for recent events, or cold.zig + cold_cache.zig (a bounded disk walk through a shared block cache) for older cursors. There is no per-client outbound queue, so a slow reader cannot blow up memory.
• Archive download (xrpcapi.zig) — planBackfill → getSegment / getBlock, the paginated path clients use to pull sealed history.
• Operator surfaces — status_page.zig (hosts, accounts, segments, collections), homepage.zig, repo_export.zig.

filter.zig implements wantedCollections/wantedDids.

Testing rig #

Unusually central, and worth knowing even if you are not touching tests.

• crashpoint.zig and segment_io.zig are deterministic fault seams: named crash points at durable commit boundaries, and injected segment I/O faults.
• tests/oracle.py boots the real binary against the pinned upstream simulator and aborts it at every lifecycle crashpoint, verifying restart convergence.
• tests/powerloss_oracle.py runs a real Linux binary on ext4 over kernel NBD and cuts power at named write/fsync/rename boundaries.
• tests/differential_oracle.py compares Stream against the pinned upstream Go implementation, reader, and client.
• scripts/admit binds a passing run of all of the above to one image digest.

Cross-cutting #

metrics.zig and observability.zig (OpenTelemetry), logging.zig, process_metrics.zig, disk_space.zig, environment.zig, operational.zig / inspect_all.zig (offline operator commands), and timestamp/ (the operator timestamp-import pipeline).

Where to look #

I want to understand…	Start here
What is proven and what is not	docs/semantic-parity.md
The on-disk segment format	docs/jss-format-v1.md, segment.zig
Sealing rules	docs/jss-seal-spec.md, segment_footer.zig
The ingest lifecycle / cutover	docs/upstream-bootstrap-spec.md, ingest/backfill/lifecycle.zig
Compaction	docs/upstream-compaction-spec.md, compact/
The live scheduler	docs/live-scheduler.md, pipeline.zig
Repair	docs/live-repair.md, repair.zig
The subscribe protocol	docs/upstream-subscribe-protocol.md, server.zig
Timestamp import	docs/timestamp-import.md, timestamp/
Running it — flags and env vars	docs/configuration.md, runtime/cli.zig
The offline inspection subcommands	docs/cli.md
Deploying	docs/deployment-runbook.md, deploy/README.md, scripts/admit
Performance	docs/benchmarks.md, just bench
Accepted limitations and traps	docs/gotchas.md
The rules that must not break	docs/invariants.md
Why a thing is the way it is	docs/lessons-from-zlay.md