From 16f77f526ad5dba143597703e92da4efe68bc3aa Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Wed, 05 Aug 2026 15:14:55 +0000 Subject: [PATCH] docs: overnight addendum — starvation fix is not the cause (negative result) Co-Authored-By: Claude Fable 5 --- docs/handoffs/HANDOFF-2026-08-04-zio-backend.md | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file(s) changed, 46 insertion(s)(+), 0 deletion(s)(-) diff --git a/docs/handoffs/HANDOFF-2026-08-04-zio-backend.md b/docs/handoffs/HANDOFF-2026-08-04-zio-backend.md --- a/docs/handoffs/HANDOFF-2026-08-04-zio-backend.md +++ b/docs/handoffs/HANDOFF-2026-08-04-zio-backend.md @@ -398,3 +398,49 @@ not. The instrumentation, the forensics script, and the reproduction harness (`/root/repro.py` pattern: hold N consumers, probe a NEW connection every 20s) are all in place to finish this quickly with a rested head. + +## overnight addendum — a NEGATIVE result that matters + +**The ready-queue/poller-starvation fix (`d6d77b7`) is NOT the cause of the +field failure.** A dedicated harness (`zig build starve` in zio) puts 2,000 +fibers on hot socketpairs with an accept loop alongside — no relay, no +external network — and measures how long a new connection waits to be +accepted: + +| scheduler | worst accept latency (2,000 hot fibers) | +|---|---| +| with the fix | 16 ms | +| pre-fix (drain-to-empty, skip poll) | 8 ms | + +Neither starves, and `ready_now` is 0 on every iteration: in this workload +the ready queue always empties, so the poller is never skipped. The fix +addresses a real hazard and should probably stay, but it does **not** +explain production, and my earlier write-up overstated it. + +Also invalid: the "the fix slowed the ramp (575 vs 1,700 conns)" comparison. +Those two runs happened on the box in different states — the later one after +fly's shared-CPU burst quota was exhausted. Not a like-for-like measurement. + +**What this narrows.** A large parked-fiber population by itself does not +starve accept. So the cause lies in what the relay adds on top: TLS +processing on the loop thread, frame-pool cross-domain handoffs, reconnect +timers, and real peers with real latency. The next harness should model +those, not just fiber count. + +**Immediate blocker.** The test box is CPU-throttled (fly shared-cpu burst +exhausted after a day of builds): a cold zlay build is progressing at +~1.4 MB of cache per minute and would need hours. Options for tomorrow, in +order of preference: +1. build zlay locally (cross-compile works, ~25s incremental) and ship the + binary — avoids building on fly entirely; needs a transfer method that + handles ~145 MB (base64-over-ssh fails above ~100 KB) +2. a short-lived `performance-*` machine (cents/hour) purely to get loop + stats from a real relay under the failure +3. wait for the shared-CPU quota to replenish + +**The measurement still wanted**, unchanged: run the diagnostic build +(`zio_debug_loop_stats = true`, prints `[loop] iters/switches/polls/events/ +batched/live/ready_now` to stderr every 5s) on a relay while accept is +starved. `ready_now` and `polls` discriminate between poller starvation, +readiness never arriving, and accept losing a fairness race — and stderr +works when the HTTP metrics path does not. -- tangled.sh