diff --git a/AGENTS.md b/AGENTS.md index 7bd0e48..a4e8539 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,18 +1,16 @@ # zlay -AT Protocol relay in zig 0.16. reader thread per PDS + shared frame -processing pool. ReleaseSafe in production. production runs the -[zio](https://tangled.org/zzstoatzz.io/zio) std.Io backend -(`-Dbackend=zio`, cooperative fibers on one scheduler thread, ~40% RSS -cut vs Io.Threaded); the default build is Io.Threaded and remains the -rollback path. a fiber that never yields owns the scheduler — no busy -loops, no sub-ms sleep spins (see docs/handoffs/HANDOFF-2026-08-05-zio-canary.md). +AT Protocol relay in zig 0.16. reader per PDS + shared frame processing +pool, written against `std.Io` — backend selected at build time +(`-Dbackend=zio` for production, `Io.Threaded` default). ReleaseSafe in +production. see docs/design.md for the threading and backend model. ## before pushing - `zig fmt --check .` and `zig build test` -- production is `-Dtarget=x86_64-linux-gnu` (glibc malloc's per-thread arenas + page-return are load-bearing for RSS at ~2,800 threads under the Threaded fallback). the "musl breaks RocksDB" belief is **retired** — musl builds and runs SIGILL-free on 0.16 — but mimalloc (the only static-musl allocator we found viable) leaks under our thread model, so prod stays glibc. see [docs/musl-investigation.md](docs/musl-investigation.md) +- production is `-Dtarget=x86_64-linux-gnu` (glibc malloc's per-thread arenas + page-return are load-bearing for RSS under the Threaded fallback's ~2,800 threads). the "musl breaks RocksDB" belief is **retired** — musl builds and runs SIGILL-free on 0.16 — but mimalloc (the only static-musl allocator we found viable) leaks under our thread model, so prod stays glibc. see [docs/musl-investigation.md](docs/musl-investigation.md) - ReleaseFast has a known double-free — do not use +- under zio a fiber that never yields owns the scheduler — no busy loops, no sub-ms sleep spins ## deploy @@ -20,8 +18,8 @@ configs at `../@zzstoatzz.io/relay/` — `just zlay publish-remote ReleaseSafe < KUBECONFIG is set automatically by the zlay module (`zlay/kubeconfig.yaml`). -if a zio build misbehaves: forensics before rollback (`scripts/zlay-forensics.sh` -in-pod + a /metrics dump) — rolling back deletes the pod and the evidence. +see [docs/deployment.md](docs/deployment.md) for build flags, rollback, and +zio triage/forensics discipline. ## docs @@ -30,5 +28,4 @@ in-pod + a /metrics dump) — rolling back deletes the pod and the evidence. - [docs/gotchas.md](docs/gotchas.md) — zig/pg.zig/rocksdb-zig/deploy traps - [docs/incident-2026-03-04.md](docs/incident-2026-03-04.md) — ReleaseSafe RSS analysis - [docs/evented-attempt.md](docs/evented-attempt.md) — Evented backend attempt and why we reverted -- [docs/handoffs/HANDOFF-2026-08-04-zio-backend.md](docs/handoffs/HANDOFF-2026-08-04-zio-backend.md) — getting zio to fleet scale (eight fixes; the stack-aliasing diagnosis was retracted as a detector artifact) -- [docs/handoffs/HANDOFF-2026-08-05-zio-canary.md](docs/handoffs/HANDOFF-2026-08-05-zio-canary.md) — the sub-ms sleep root cause and canary #2 verdict +- [docs/handoffs/](docs/handoffs/) — dated engineering handoffs (zio adoption arc: 2026-08-03/04/05) diff --git a/CLAUDE.md b/CLAUDE.md index 7bd0e48..a4e8539 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,18 +1,16 @@ # zlay -AT Protocol relay in zig 0.16. reader thread per PDS + shared frame -processing pool. ReleaseSafe in production. production runs the -[zio](https://tangled.org/zzstoatzz.io/zio) std.Io backend -(`-Dbackend=zio`, cooperative fibers on one scheduler thread, ~40% RSS -cut vs Io.Threaded); the default build is Io.Threaded and remains the -rollback path. a fiber that never yields owns the scheduler — no busy -loops, no sub-ms sleep spins (see docs/handoffs/HANDOFF-2026-08-05-zio-canary.md). +AT Protocol relay in zig 0.16. reader per PDS + shared frame processing +pool, written against `std.Io` — backend selected at build time +(`-Dbackend=zio` for production, `Io.Threaded` default). ReleaseSafe in +production. see docs/design.md for the threading and backend model. ## before pushing - `zig fmt --check .` and `zig build test` -- production is `-Dtarget=x86_64-linux-gnu` (glibc malloc's per-thread arenas + page-return are load-bearing for RSS at ~2,800 threads under the Threaded fallback). the "musl breaks RocksDB" belief is **retired** — musl builds and runs SIGILL-free on 0.16 — but mimalloc (the only static-musl allocator we found viable) leaks under our thread model, so prod stays glibc. see [docs/musl-investigation.md](docs/musl-investigation.md) +- production is `-Dtarget=x86_64-linux-gnu` (glibc malloc's per-thread arenas + page-return are load-bearing for RSS under the Threaded fallback's ~2,800 threads). the "musl breaks RocksDB" belief is **retired** — musl builds and runs SIGILL-free on 0.16 — but mimalloc (the only static-musl allocator we found viable) leaks under our thread model, so prod stays glibc. see [docs/musl-investigation.md](docs/musl-investigation.md) - ReleaseFast has a known double-free — do not use +- under zio a fiber that never yields owns the scheduler — no busy loops, no sub-ms sleep spins ## deploy @@ -20,8 +18,8 @@ configs at `../@zzstoatzz.io/relay/` — `just zlay publish-remote ReleaseSafe < KUBECONFIG is set automatically by the zlay module (`zlay/kubeconfig.yaml`). -if a zio build misbehaves: forensics before rollback (`scripts/zlay-forensics.sh` -in-pod + a /metrics dump) — rolling back deletes the pod and the evidence. +see [docs/deployment.md](docs/deployment.md) for build flags, rollback, and +zio triage/forensics discipline. ## docs @@ -30,5 +28,4 @@ in-pod + a /metrics dump) — rolling back deletes the pod and the evidence. - [docs/gotchas.md](docs/gotchas.md) — zig/pg.zig/rocksdb-zig/deploy traps - [docs/incident-2026-03-04.md](docs/incident-2026-03-04.md) — ReleaseSafe RSS analysis - [docs/evented-attempt.md](docs/evented-attempt.md) — Evented backend attempt and why we reverted -- [docs/handoffs/HANDOFF-2026-08-04-zio-backend.md](docs/handoffs/HANDOFF-2026-08-04-zio-backend.md) — getting zio to fleet scale (eight fixes; the stack-aliasing diagnosis was retracted as a detector artifact) -- [docs/handoffs/HANDOFF-2026-08-05-zio-canary.md](docs/handoffs/HANDOFF-2026-08-05-zio-canary.md) — the sub-ms sleep root cause and canary #2 verdict +- [docs/handoffs/](docs/handoffs/) — dated engineering handoffs (zio adoption arc: 2026-08-03/04/05) diff --git a/docs/deployment.md b/docs/deployment.md index a898b18..7d65c5b 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -26,7 +26,18 @@ the full `Dockerfile` exists for CI/standalone builds but is slow on Mac (cross- ### build flags -- `-Dbackend=zio` — production backend since 2026-08-05: [zio](https://tangled.org/zzstoatzz.io/zio) cooperative fibers on one scheduler thread instead of ~2,800 OS threads (~40% RSS cut). omit the flag for the `Io.Threaded` fallback, which is the rollback path (old images stay pinned in containerd; rollback is one `kubectl set image`). deploy as `just zlay publish-remote ReleaseSafe zio`. see [handoffs/HANDOFF-2026-08-05-zio-canary.md](handoffs/HANDOFF-2026-08-05-zio-canary.md). +- `-Dbackend=zio` — production backend since 2026-08-05: [zio](https://tangled.org/zzstoatzz.io/zio) cooperative fibers on one scheduler thread instead of ~2,800 OS threads (~40% RSS cut). omit the flag for the `Io.Threaded` fallback, which is the rollback path (old images stay pinned in containerd; rollback is one `kubectl set image`). deploy as `just zlay publish-remote ReleaseSafe zio`. see [handoffs/HANDOFF-2026-08-05-zio-canary.md](handoffs/HANDOFF-2026-08-05-zio-canary.md). **NB:** the zon currently pins zio `beb13c8`, which lives on zio's `canary/obs-2026-08-04` branch, not zio main — zio main lacks the poller-starvation (`d6d77b7`) and arm-failure (`63baa64`) fixes. reconcile before repinning to a main ref. + +### zio triage and forensics + +if a zio build misbehaves: **forensics before rollback** — run +`scripts/zlay-forensics.sh` in-pod and dump `/metrics`; rolling back deletes +the pod, its logs, and the evidence (how canary #1's forensics were lost). + +triage order: `relay_seq` first (flat ingest reads exactly like broken +serving if you only probe the serving side), then `zio_sched_switches_total` +(flat = wedged loop), `zio_poller_registrations` vs connection count +(collapse = erosion), `zio_poller_arm_failures_total` (any nonzero is bad). - `-Dtarget=x86_64-linux-gnu` — production target, glibc. glibc malloc (per-thread arenas + `madvise` page-return + `malloc_trim`) is load-bearing for RSS at ~2,800 threads. **NB:** the old "musl breaks RocksDB / illegal instructions" claim was a zig 0.15 artifact and is **retired** — on 0.16 musl builds, links, and runs SIGILL-free at thread scale (canary, 2026-06). musl stays off prod only because the one static-musl allocator we validated as buildable — mimalloc — leaks RSS under our thread model (v2/v3/purge-forced/even under glibc). full matrix in [musl-investigation.md](musl-investigation.md). - `-Dcpu=baseline` — required when building inside Docker/QEMU (not needed for `zlay-publish-remote` since it builds natively). - `-Doptimize=ReleaseSafe` — safety checks on, optimizations on. production default since 2026-03-05. previously caused OOM (see [incident-2026-03-04.md](incident-2026-03-04.md)) — resolved by the frame pool moving heavy work off reader threads.