diff --git a/docs/handoffs/HANDOFF-2026-08-04-zio-backend.md b/docs/handoffs/HANDOFF-2026-08-04-zio-backend.md index aa7bc33..0170ab8 100644 --- a/docs/handoffs/HANDOFF-2026-08-04-zio-backend.md +++ b/docs/handoffs/HANDOFF-2026-08-04-zio-backend.md @@ -1,7 +1,7 @@ # handoff 2026-08-04 — zlay on zio: seven fixes and a soak -> **STATUS: DRAFT — soak in progress.** The final section gets real numbers -> when the 90-minute window closes. Everything above it is finished work. +> **STATUS: FINAL.** Soak v3 passed: 95 minutes, 19/19 samples green, on +> the exact ship build. Details in the soak section. Continues [HANDOFF-2026-08-03-zio-backend.md](HANDOFF-2026-08-03-zio-backend.md). That handoff ended with: zlay runs on zio, holds ~47 connections, does not @@ -148,12 +148,18 @@ two disk-100% episodes without dropping connections. The window ended at (clean exit_code=0 of the VM — see machine notes), not due to any relay behavior. -**Soak v3** (same build, reprovisioned machine): the formal uninterrupted -90-minute certification. - -- [ ] 90-minute verdict: -- [ ] RSS start/end: -- [ ] metrics latency range: -- [ ] consumer frames delivered: -- [ ] recommended next: overnight soak, then operator conversation about a - canary +**Soak v3 — PASSED** (ship build `c383540` = zio `6790455` with the +detector gated off; reprovisioned machine; consumer attached throughout; +2026-08-04 07:40–09:15 UTC): + +- [x] **95 minutes, 19/19 samples green, zero crashes, zero assert trips** +- [x] connections 1,307–1,872 (normal fleet churn around ~1,831 seeded hosts) +- [x] RSS 900–1,057 MB oscillating, **no growth trend** — ~2.6x below + prod's flat 2.6 GB, and ~300 MB lighter than the detector builds +- [x] `/metrics` 0.034–0.405 s across all samples (never over half a second) +- [x] relay_seq 0 → 1,221,943 continuous; db queue depth 0 at every sample +- [x] consumer: ~812k frames / ~4.2 GB across two sessions at ~215–270 + frames/s, with a clean reconnect at the hourly session recycle +- [x] disk stable at 37–39% under the cron janitor +- [x] recommended next: operator canary (message drafted); the box keeps + running as unattended burn-in until its 24h lifecycle ends