diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..2de4e54 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,71 @@ +# zlay + +AT Protocol relay in zig 0.15.2. subscribes to every PDS, validates commit +signatures, serves merged firehose to downstream consumers. + +## architecture + +- **reader thread per PDS** (~2,750) — lightweight: TLS read, header decode, + cursor tracking, rate limiting. submits raw frames to pool. +- **frame pool** (16 workers) — CBOR decode, ECDSA verify, DB persist, broadcast. +- **validator** (4-8 resolver threads) — background DID resolution, signing key cache. +- dependencies: zat (AT proto primitives), websocket.zig, pg.zig, rocksdb-zig. + +## build + +```bash +zig build -Doptimize=ReleaseSafe -Dtarget=x86_64-linux-gnu # production +zig build test # run tests +zig fmt --check . # CI checks this +``` + +always run `zig fmt --check .` and `zig build test` before pushing. + +## deploy + +deploy configs live at `../@zzstoatzz.io/relay/` (justfile, helm, terraform). + +```bash +cd ../relay +just zlay-publish-remote ReleaseSafe # build on server, deploy +just zlay-publish-remote # debug build (fallback) +``` + +**critical rules:** +- MUST use `-Dtarget=x86_64-linux-gnu` — musl produces illegal instructions in RocksDB +- MUST set `KUBECONFIG="$(pwd)/zlay-kubeconfig.yaml"` — default context is docker-desktop +- never change probe paths and image in separate operations +- immutable SHA tags, never `:latest` +- ReleaseFast has a known double-free — do not use + +## key files + +| file | purpose | +|---|---| +| `src/main.zig` | entry point, thread stacks (8 MiB), signal handling | +| `src/subscriber.zig` | per-PDS reader thread, FrameHandler | +| `src/frame_worker.zig` | pool worker: decode, validate, persist, broadcast | +| `src/thread_pool.zig` | generic ring-buffer thread pool | +| `src/validator.zig` | signature verify, DID resolver, LRU key cache | +| `src/broadcaster.zig` | fan-out to consumers, prometheus metrics | +| `src/event_log.zig` | disk persist, postgres, cursor replay | +| `src/collection_index.zig` | RocksDB (collection, did) index | +| `src/slurper.zig` | multi-host crawl manager | +| `src/api.zig` | HTTP API endpoints (served via httpFallback on WS port) | + +## current production state + +- running at zlay.waow.tech, ~2,750 PDS hosts +- ReleaseSafe, 8 MiB thread stacks, ~1.1 GiB RSS +- ports: 3000 (WS + HTTP), 3001 (metrics only) +- probes: `/_healthz` (liveness), `/_readyz` (readiness, DB check) +- `relay_build_info{git_sha,optimize}` metric confirms what's running + +## zig gotchas relevant here + +- `&.{...}` in loops creates stack-local arrays that alias — heap-allocate instead +- operator precedence: `(byte & 0xf0) == 0` not `byte & 0xf0 == 0` +- `ArrayList` is unmanaged in 0.15 — pass allocator to each method +- rocksdb-zig iterator Data: do NOT call `.deinit()` on entries (SIGABRT) +- rocksdb-zig DB.open: path must be null-terminated +- pg.zig `QueryRow.deinit()` returns `!void` — use `defer row.deinit() catch {}` diff --git a/README.md b/README.md index a9805f9..efd12b7 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ an [AT Protocol](https://atproto.com/) relay in zig. subscribes to every PDS on - **inline collection index** — indexes `(DID, collection)` pairs in the event processing pipeline using RocksDB. serves `listReposByCollection` from the relay process — no sidecar. the index design draws on [fig](https://tangled.org/microcosm.blue)'s work on [lightrail](https://tangled.org/microcosm.blue/lightrail). -- **reader thread per PDS + frame processing pool** — each PDS gets a lightweight reader thread (cursor tracking, rate limiting, header decode). heavy work (full CBOR decode, validation, DB persist, broadcast) runs on a shared pool of frame workers (configurable, default 16). thread stacks are 4 MB (zig's default is 16 MB). +- **reader thread per PDS + frame processing pool** — each PDS gets a lightweight reader thread (cursor tracking, rate limiting, header decode). heavy work (full CBOR decode, validation, DB persist, broadcast) runs on a shared pool of frame workers (configurable, default 16). ## spec compliance @@ -32,9 +32,9 @@ implements the [AT Protocol sync spec](https://atproto.com/specs/sync) — `subs requires zig 0.15 and a C/C++ toolchain (for RocksDB). ```bash -zig build # build (debug — current production default) +zig build # build (debug) zig build test # run tests -zig build -Doptimize=ReleaseSafe # release build (not used in prod yet, see docs/incident-2026-03-04.md) +zig build -Doptimize=ReleaseSafe # release build (production default) ``` ## configuration diff --git a/docs/deployment.md b/docs/deployment.md index be0fda1..92fec42 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -13,8 +13,8 @@ just zlay-publish-remote this SSHs into the server and: 1. `git pull --ff-only` in `/opt/zlay` -2. `zig build -Dtarget=x86_64-linux-gnu` — debug build (no `-Doptimize`). see [incident-2026-03-04.md](incident-2026-03-04.md) for why ReleaseSafe is not used. -3. `buildah bud -t atcr.io/zzstoatzz.io/zlay:latest -f Dockerfile.runtime .` — thin runtime image +2. `zig build -Doptimize=ReleaseSafe -Dtarget=x86_64-linux-gnu` +3. `buildah bud -f Dockerfile.runtime .` — thin runtime image with SHA tag 4. pushes to k3s containerd via `buildah push` → `ctr images import` 5. `kubectl set image deployment/zlay -n zlay main=` + `kubectl rollout status` @@ -28,7 +28,7 @@ the full `Dockerfile` exists for CI/standalone builds but is slow on Mac (cross- - `-Dtarget=x86_64-linux-gnu` — **must use glibc**, not musl. zig 0.15's C++ codegen for musl produces illegal instructions in RocksDB's LRU cache. - `-Dcpu=baseline` — required when building inside Docker/QEMU (not needed for `zlay-publish-remote` since it builds natively). -- `-Doptimize=ReleaseSafe` — safety checks on, optimizations on. **currently not used in production** — ReleaseSafe inflates per-thread RSS ~10x (see [incident-2026-03-04.md](incident-2026-03-04.md)). the frame pool reduces per-reader-thread work, opening a path back to ReleaseSafe. +- `-Doptimize=ReleaseSafe` — safety checks on, optimizations on. production default since 2026-03-05. previously caused OOM (see [incident-2026-03-04.md](incident-2026-03-04.md)) — resolved by the frame pool moving heavy work off reader threads. ## initial setup @@ -73,19 +73,21 @@ just zlay-ssh # ssh into server ## memory tuning -three changes brought steady-state memory from ~6.6 GiB down to ~1.2 GiB at ~2,750 connected hosts: +four changes brought steady-state memory from ~6.6 GiB down to ~1.1 GiB at ~2,250 connected hosts (ReleaseSafe): **shared TLS CA bundle.** the biggest single win. websocket.zig's TLS client calls `Bundle.rescan()` per connection, loading the system CA certificates into a per-connection arena. with ~2,750 PDS connections, that's ~2,750 copies of the CA bundle in memory (~800 KB each = ~2.2 GiB). fix: load the bundle once in the slurper, pass it to all subscribers via `config.ca_bundle`. memory dropped from ~3.3 GiB to ~1.2 GiB (~65% reduction). -**thread stack sizes.** zig's default thread stack is 16 MB. with ~2,750 subscriber threads that maps 44 GB of virtual memory. most reader threads just read websockets and submit frames to the processing pool — 4 MB is generous. all `Thread.spawn` calls now pass `.{ .stack_size = 4 * 1024 * 1024 }`. the constant is defined in `main.zig` as `default_stack_size` for the threads spawned there; other modules use the literal directly. +**thread stack sizes.** zig's default thread stack is 16 MB. with ~2,750 subscriber threads that maps 44 GB of virtual memory. all `Thread.spawn` calls use `main.default_stack_size` (8 MB). this is virtual memory — only touched pages count as RSS. 8 MB supports ReleaseSafe's TLS handshake path (~134 KiB peak stack). **c_allocator instead of GeneralPurposeAllocator.** GPA is a debug allocator — it tracks per-allocation metadata and never returns freed small allocations to the OS. since zlay links glibc (`build.zig:42`), `std.heap.c_allocator` gives us glibc malloc with per-thread arenas, madvise-based page return, and production-grade fragmentation mitigation. +**frame processing pool.** reader threads (one per PDS) now only do TLS read, header decode, cursor tracking, and rate limiting — then queue raw frames to a shared pool of 16 workers. this dramatically reduced per-thread RSS in ReleaseSafe (from ~3.9 MiB to ~0.45 MiB) by keeping crypto, DB, and broadcast off reader thread stacks. + ## resource usage | metric | value | |--------|-------| -| memory | ~2.7 GiB steady state (~2,750 hosts, with frame pool) | +| memory | ~1.1 GiB at ~2,250 hosts (ReleaseSafe), projected ~1.3 GiB steady state | | CPU | ~1.5 cores peak | | requests | 1 GiB memory, 1000m CPU | | limits | 8 GiB memory | diff --git a/docs/design.md b/docs/design.md index 40533b3..612db8f 100644 --- a/docs/design.md +++ b/docs/design.md @@ -47,14 +47,14 @@ additionally: | thread type | count at ~2,750 PDS | stack size | responsibility | |---|---|---|---| -| subscriber readers | ~2,750 | 4 MB | one per host — lightweight WebSocket read loop, header decode, cursor tracking, rate limiting, submit to pool | -| frame pool workers | 16 (env: `FRAME_WORKERS`) | 4 MB | CBOR decode, validation, DB persist, broadcast | -| resolver threads | 4–8 (env: `RESOLVER_THREADS`) | 4 MB | DID document resolution, signing key extraction, cache population | -| consumer write threads | 1 per downstream consumer | 4 MB | drain ring buffer → WebSocket write, ping/pong keepalive | -| flush thread | 1 | 4 MB | batched fsync of event log (100ms or 400 events) | -| GC thread | 1 | 4 MB | event log file cleanup every 10 minutes | -| crawl queue thread | 1 | 4 MB | process `requestCrawl` — validate hostname, describeServer, spawn worker | -| metrics server | 1 | 4 MB | HTTP on internal port, prometheus scrape | +| subscriber readers | ~2,750 | 8 MB | one per host — lightweight WebSocket read loop, header decode, cursor tracking, rate limiting, submit to pool | +| frame pool workers | 16 (env: `FRAME_WORKERS`) | 8 MB | CBOR decode, validation, DB persist, broadcast | +| resolver threads | 4–8 (env: `RESOLVER_THREADS`) | 8 MB | DID document resolution, signing key extraction, cache population | +| consumer write threads | 1 per downstream consumer | 8 MB | drain ring buffer → WebSocket write, ping/pong keepalive | +| flush thread | 1 | 8 MB | batched fsync of event log (100ms or 400 events) | +| GC thread | 1 | 8 MB | event log file cleanup every 10 minutes | +| crawl queue thread | 1 | 8 MB | process `requestCrawl` — validate hostname, describeServer, spawn worker | +| metrics server | 1 | 8 MB | HTTP on internal port, prometheus scrape | | main thread | 1 | default | signal handling, shutdown coordination | total: ~2,770 + consumers. subscriber reader threads run blocking WebSocket @@ -66,9 +66,10 @@ is intentionally decoded twice — once by the reader for routing, once by the pool worker for full processing. this double decode costs ~1–2μs per frame, far cheaper than serializing parsed state across threads. -the 4 MB stack size (vs zig's 16 MB default) is the key to fitting ~3K threads -in memory. reader thread actual stack usage is well below 4 MB since the -deepest paths (crypto, CBOR, DB) now run on pool workers. +the 8 MB stack size (vs zig's 16 MB default) supports ReleaseSafe's TLS +handshake path (~134 KiB peak stack from `tls.Client.init` + `KeyShare.init`). +only touched pages count as RSS — reader threads use ~0.45 MiB RSS each since +the deepest paths (crypto, CBOR, DB) now run on pool workers. ## memory model @@ -144,12 +145,12 @@ populated live from firehose commits. backfill from source relay's ## scaling limits current deployment: ~2,780 PDS hosts, running on a 32 GB / 16 CPU node. -steady-state memory: ~2.7 GiB (with frame pool). postgres alongside +steady-state memory: ~1.1 GiB (ReleaseSafe, with frame pool). postgres alongside at ~240 MiB. resource limits: 8 GiB memory, 1 GiB request, 1000m CPU. | component | current (~2,750 PDS) | at 10x (~27,500 PDS) | status | |---|---|---|---| -| thread stacks | ~11 GB virtual (2,750 × 4 MB) | ~110 GB virtual | **breaks** — exceeds 32 GB node. reader RSS is lower since heavy work moved to pool workers | +| thread stacks | ~22 GB virtual (2,750 × 8 MB), ~1.2 GiB RSS | ~220 GB virtual | **breaks** — virtual exceeds 32 GB node, but RSS scales sublinearly (~0.45 MiB/thread) | | pg pool | 5 connections (hardcoded) | 5 connections | **breaks** — saturates under concurrent UID lookups | | resolver queue | `ArrayList` + dedupe set | same | **ok** — dedupe prevents unbounded growth from duplicate DIDs | | validator cache | 250K entries, ~19 MB | same (capped) | **degrades** — miss rate climbs with more unique DIDs | @@ -157,7 +158,7 @@ at ~240 MiB. resource limits: 8 GiB memory, 1 GiB request, 1000m CPU. | RocksDB | manageable write rate | ~1.4M writes/sec projected | **needs** compaction tuning | | event log | buffered, 100ms flush | fine — sequential I/O | ok | | kernel threads | ~2,800 (below 30K default) | ~28,000 (near default max) | **breaks** without `sysctl` tuning | -| RSS | ~1.2 GiB | ~5–8 GiB projected (shared CA bundle, malloc overhead scales sublinearly) | ok — fits 32 GB node | +| RSS | ~1.1 GiB (ReleaseSafe) | ~4–6 GiB projected | ok — fits 32 GB node | ### what breaks first diff --git a/docs/incident-2026-03-04.md b/docs/incident-2026-03-04.md index 19e9820..1036443 100644 --- a/docs/incident-2026-03-04.md +++ b/docs/incident-2026-03-04.md @@ -80,8 +80,8 @@ ## current state (2026-03-05) -- **optimize**: Debug (no optimization) — preparing ReleaseSafe deploy -- **thread stacks**: 8 MiB (bumped from 4 MiB to support ReleaseSafe handshake path) +- **optimize**: ReleaseSafe — deployed successfully, ~1.1 GiB RSS at ~2,255 hosts +- **thread stacks**: 8 MiB (supports ReleaseSafe handshake path; only touched pages count as RSS) - **probes**: liveness `/_healthz:3000`, readiness `/_readyz:3000` (helm synced) - **helm**: values applied via `helm upgrade`, no longer drifted from kubectl patches - **reconnect cronjob**: deployed — reconciles PDS host list every 4 hours