lessons from zlay (devlogs 006–010 + incident docs) #

What zlay is, and why Stream inherits from it. zlay is a separate project — an atproto relay written in Zig (tangled.org/zat.dev/zlay), a sibling of this one, sharing the zat atproto library and the same runtime choices (Io.Threaded, glibc, ReleaseSafe). It has been in production longer and at higher thread counts, so it hit the Zig-runtime and glibc failure modes first. None of the rules below are about relay behaviour; they are about surviving this language and runtime at scale, which is why they transfer even though Stream is an archive rather than a relay.

Sources, all in sibling checkouts rather than this repository — ~/{forge}/{org}/{repo}, so alongside this one under zat.dev/:

• zat/devlog/006–010
• zlay/docs/gotchas.md, allocation-audit.md, incident-*.md

read those for the full stories; these are the rules stream inherits. Each parenthesised (006), (008), (incident-2026-05-31) is a pointer into them.

Spot-checked 2026-07-30 — still true in Stream: MALLOC_ARENA_MAX=4 is set (deploy/README.md), and the build_info{git_sha,optimize} canary exists as stream_build_info (runtime/metrics.zig).

1. Io.Threaded + ReleaseSafe + c_allocator + 8 MB thread stacks — each bought with an incident. ReleaseFast hid heap corruption for days (008) and has a known double-free; GPA never returns memory to the OS; 2–4 MB stacks overflow in ReleaseSafe TLS/CBOR chains (134 KiB for tls.Client.init alone, 007); default 16 MB stacks map 44 GB VM at ~2.7k threads (006).
2. never mix Io backends — Io.Mutex/Condition/io.sleep crash at runtime (NULL Thread.current) when called from a different backend than they were created under. cross-context comms = pure atomics (DbRequestQueue pattern) (008).
3. durability before emission — never broadcast a seq a subscriber could later replay from before it is durable. atomic committed_seq watermark, NOT broadcast-inside-flush (that serialized 2,670 producers and OOM'd). recovery scans validate every record and truncate torn tails (incident-2026-05-31). applies the day the segment archive lands.
4. socket close ownership — only the thread that owns a consumer's write loop closes the socket; everyone else flips an alive atomic (EBADF race, 009). stream's Handler.close joins the subscriber thread before conn teardown — keep it that way.
5. serialize ws client writes; background loops must be cancellation-cooperative — never catch {} an io.sleep (swallowed error.Canceled = use-after-free, 008); throttle mass reconnects in batches.
6. assume TCP splits everything — handshake/CRLF/HTTP parsing must tolerate byte-at-a-time delivery; one CRLF off-by-one was misblamed on the fiber scheduler for days (006, 008).
7. thin reader threads — websocket read + queue only; decode/verify/fan-out on a shared pool. 0.45 vs 3.9 MiB per-reader RSS is run-vs-OOM (007). stream's single ingest thread currently decodes+verifies inline: fine at one upstream, revisit if per-PDS crawling ever lands. That condition has since been met, so the revisit is due. Backfill now runs 100 workers (--backfill-workers=100) fetching getRepo through the relay's 302 to the origin PDS, with per-host accounting in ingest/backfill/host_store.zig and per-host parking in retry.zig. The live-ingest side is still one upstream, so the rule has not been broken — but "if per-PDS crawling ever lands" now reads as a condition already satisfied, not a hypothetical.
8. memory observability from day one — smaps-based RSS (mallinfo overflows at 2 GiB), malloc_info all-arena gauges, MALLOC_ARENA_MAX=4 in deploy env, build_info{git_sha,optimize} canary metric. balanced allocs + growing RSS = glibc arena fragmentation, not a leak (allocation-audit, incident-2026-05-30).
9. build/deploy — -Dtarget=x86_64-linux-gnu (musl SIGILLs C++ deps), native build on the server, immutable SHA tags, probes on the concurrent port, probe-path+image changes atomic (006, incident-2026-03-04).
10. the network is input — DID/handle resolution is SSRF surface: reject private hosts, DNS-preflight, no redirects, dial the checked address (010). XRPC errors are data: parse the error envelope, retry only transient with capped jittered backoff, honor retry-after.
11. observe → measure → enforce for any new verification, behind a flag; beware checks that trivially pass on empty input (007's extractOps bug).
12. incident discipline — one variable per experiment, verify for hours, keep a last-known-good SHA, roll back when hypothesis #3 fails (incident-2026-03-04, 009).