diff --git a/docs/handoffs/HANDOFF-2026-08-03-zio-backend.md b/docs/handoffs/HANDOFF-2026-08-03-zio-backend.md new file mode 100644 index 0000000..3162b3b --- /dev/null +++ b/docs/handoffs/HANDOFF-2026-08-03-zio-backend.md @@ -0,0 +1,168 @@ +# handoff 2026-08-03 — zlay on the zio backend + +zlay compiles and runs on zio under ReleaseSafe on x86_64, holds ~47 upstream +connections, and validates real firehose events. It does **not** sustain: the +broadcaster stops draining, the broadcast queue fills, and the relay stops +making progress. Everything below is what I know, what I got wrong, and where +I'd go next. + +Nothing here is on `main`. `main` is `9eb026e` (the pg.zig bump, already +deployed and healthy). + +## branches + +**zio** — `tangled.org/zzstoatzz.io/zio`, branch `fiber/mmap-guard-stacks`: + +``` +2a84a09 backend: handle the broadcast sentinel before the pointer cast +0f5562a backend: wake fibers when a spilled DNS lookup completes +24341fe fiber: one definition of the initial fiber frame +866298c fiber: switch stacks in a naked function on x86_64 <- the big one +6e843b3 fiber: declare rbp clobbered in the x86_64 context switch +be1d931 fiber: map stacks with a guard page instead of malloc +``` + +**zlay** — `tangled.org/zat.dev/zlay`, branch `fix/subscriber-connect-domain`: + +``` +7d0c229 broadcaster: saturate the queue-full spin counter +3391dba subscriber: keep DNS + connect on pool_io (revert 7593655) <- WRONG, see below +1a186b1 deps: bump zio to the naked x86_64 context switch +3e3109e build: -Dbackend=zio builds the relay on zio's std.Io +7593655 subscriber: connect on the io that reads the socket +``` + +## what was actually fixed + +**1. zio fibers crashed under ReleaseSafe on x86_64 (`866298c`).** This is the +one that mattered. zio's ReleaseSafe suite was 13/16 with 3 crashes on real +x86_64 hardware; it is now 16/16. Debug always passed, which is what hid it — +zio's published matrix is linux/macos **aarch64**, with x86_64 only "under +emulation", and zlay's production target is `x86_64-linux-gnu`. + +Cause: inline asm inside a normal Zig function leaves LLVM owning a frame +around the switch, and it emits that frame's teardown at the label the asm +resumes into: + +``` +jmp *0x10(%rcx) # switch away +add $0x8,%rsp # <- resume lands here +pop %rbp +mov -0x68(%rbp),%rax +``` + +The switch restores rsp/rbp to their pre-jump values, so that teardown is +unbalanced and every later spill read through rbp is garbage. Fix: a +`callconv(.naked)` switch on x86_64 (no prologue/epilogue to misplace), which +also has to carry callee-saved registers on the fiber's own stack. +`fiber.initialContext()` now defines the initial frame in one place — the +gauntlet in `spike.zig` used to build it by hand and broke when the contract +changed. + +**2. DNS-bound fibers parked forever (`0f5562a`).** zio spills `getaddrinfo` +to an inner-Threaded thread so it cannot block the loop. That thread finishes +by feeding an `Io.Queue`, whose wake is a **kernel futex** — which readies +threads and no fibers, because a waiting fiber is parked in the sched's futex +list, not on that word. The wake was simply lost. Added +`Sched.futexWakeAllFibers()`; over-waking is safe because every +`futexWaitFiber` caller re-checks in a loop. + +**3. broadcast queue overflow (`7d0c229`, zlay).** `full_spins: u32` in the +push spin loop overflows and panics under ReleaseSafe when the drain stalls, +turning "consumers are slow" into a crashed relay. Now a saturating add. This +is a real zlay bug independent of zio and is the one commit here I would take +on its own merits. + +## two things I got wrong — do not repeat + +**`3391dba` is wrong and should be reverted.** I moved connect back to +`pool_io` on the strength of `docs/stdlib-patches.md` workaround 2. That split +is correct for `Io.Uring` (no `netLookup`) but **harmful for zio**: a Threaded +call made from a fiber blocks the *loop thread*, so the first subscriber +freezes every other fiber. Measured directly: + +| connect io | result | +|---|---| +| `pool_io` (Threaded) | ramp stalls at batch 1, 1 host connecting, loop blocked | +| `self.io` (zio) | ramp advances, 150 connecting, 47 established | + +zio delegates `netLookup` itself, so the workaround is unnecessary there. The +machine currently has `self.io` patched in locally; the committed state does +not. **Re-revert `3391dba`.** + +**`ss` is blind inside `fly ssh console`.** It reported zero sockets while +`/proc/PID/net/tcp` showed 47 established — it does not even see postgres or +zlay's own listeners. I built two rounds of wrong reasoning ("no socket ever +reaches the kernel", "DNS is the bottleneck") on that. **Use +`grep -c " 01 " /proc/PID/net/tcp`.** + +Also disproven, so nobody re-tries them: missing `rbp` clobber (`6e843b3` is +kept because declaring it is correct, but it fixes nothing), +`.omit_frame_pointer = false`, `noinline contextSwitch` (much worse — 4/16; +`inline` is load-bearing), and stack protector. + +## current symptom and the leading hypothesis + +Running on zio, ReleaseSafe, x86_64: ~88 threads, ~100 MB RSS, 47 established +upstream connections, startup ramp advancing a batch at a time, real events +validating. Then the broadcaster stops draining and the queue fills. + +`Broadcaster.runBroadcastLoop` only emits frames whose `seq <= committed_seq` +(the durability gate from `incident-2026-05-31-seq-rewind.md`). `flushLoop` +runs on `pool_io` (Threaded), so it is not fiber-starved and `committed_seq` +most likely *is* advancing — which points at the broadcaster fiber not being +scheduled rather than the gate being closed. + +**Hypothesis: the same lost-wake bug as DNS, generalized.** Subscriber fibers +do DB work through `DbRequestQueue`, whose responses are signalled from +`pool_io` threads — a kernel futex wake that readies threads and no fibers. +Fibers pile up unwoken, the loop stops cycling, the broadcaster never runs. +If true, the fix is the same shape as `0f5562a` but applied to every +threaded-side completion, not just DNS. I did not get to verifying this. + +## the machine + +`zlay-zio-0801` (fly, personal org, ord, 8 cpu / 16 GB). Built and warm: +`/root/zlay` with `-Dbackend=zio -Doptimize=ReleaseSafe`, postgres seeded with +~1,824 hosts. + +```console +DATABASE_URL="postgres://postgres:postgres@127.0.0.1:5432/relay" \ +RELAY_DATA_DIR=/data/events ./zig-out/bin/zlay +``` + +`/root/setup.sh` reprovisions from scratch. **A fly machine from a plain image +has no volume — a stop/start gives you a fresh rootfs and you lose the build.** +That cost me a full rebuild cycle. + +Other traps, all of which cost me time: + +- `fly ssh console -C` does not parse shell metacharacters. Wrap everything: + `-C "sh -c '...'"`. +- A tangled archive URL built from a branch name containing `/` returns HTML, + not a tarball. Use the commit sha. +- After `pkill zlay`, sleep before restarting or rocksdb fails on a stale + `data/collection-index/LOCK`. +- zlay's startup ramp only spawns hosts already in the DB, so the **first** run + discovers hosts and connects to nothing. Run once to seed, then restart. +- `STARTUP_BATCH_SIZE=0` disables the ramp yields — useful for separating "the + ramp is stalling" from "the workers are stalling". +- Debug builds prove nothing here; production is ReleaseSafe and the whole bug + class is optimizer-dependent. + +## other state + +- Notes: `~/tangled.org/zzstoatzz.io/notes/languages/ziglang/fibers-and-inline-asm.md` + — clobbers that compile but are silently ignored, mmap vs malloc stacks, the + park/resume isolation table, the fly recipe. +- **macOS caveat:** with the guard page, zio's suite hangs on macos-aarch64 and + passes on linux. Looks like `DebugAllocator`'s `captureStackTrace` walking + past the fiber stack base — a testing-allocator interaction, not a production + path, but it makes `zig build test` on a mac useless for guarded fibers. Test + on linux. +- The RSS case for zio on zlay is **~3x, not the 200x on the scoreboard**. That + number is raw TCP with no TLS and no per-conn state. zlay carries 66 KiB/conn + of `std.crypto.tls` buffers (~181 MiB at 2,800) and 256 KiB/fiber of stack + (~700 MiB), for a floor near 880 MiB against prod's flat 2.6 GiB. The + dominant term is fiber stacks, which is what `be1d931` targets and what has + not yet been measured under a working run.