From aa605842dad5715f9d2b4a5317619521c6683b8a Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Sat, 15 Aug 2026 00:04:15 -0500 Subject: [PATCH] =?UTF-8?q?handoff:=20incident=20#11=20investigation=20sta?= =?UTF-8?q?te=20=E2=80=94=20three=20theories=20falsified,=20shaping=20expe?= =?UTF-8?q?riment=20armed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- HANDOFF.md | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index 34767bd..4db0293 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -11,13 +11,22 @@ closed and chronicled in git history 08-11..08-12. appeared day-after-launch and our gate 401'd their preflights); client_closed split out of tail_closed; zat.Datetime adopted (four formatters -> one type); zat pinned at v0.4.0. -- **NEW INCIDENT, root cause open (task #11)**: live ingest stalled to - ~16/s for 4h (08-14 14:55-18:55Z), self-recovered via relay catch-up at - ~2,400/s, zero data lost. BOTH probers (caddy + direct) saw the identical - trickle -> stream core, not proxy. Overlap: compaction pass (10:00-16:30Z) - + failed-repo retry pass (14:47-20:58Z, 698 candidates, 4 resyncs). Prime - suspect: resync/repair serializing against live append. The eval-drop - hunt (task #3) may be the same contention smaller. +- **INCIDENT #11 (4h ingest stall 08-14 14:55-18:55Z): three theories + falsified with evidence, one standing.** Falsified: repair-queue + backpressure (resync_queue_depth=0 throughout), repair-worker saturation + (zero activity), lock-across-fetch (fetch is pre-lock). Key facts: 16/s + handleEvent rate, process CPU 0.03 cores mid-stall (BLOCKED not starved), + ZERO error/reconnect log lines, instant 2,400/s catch-up at recovery. + last_seen metric is wall-clock-at-handling — proves nothing (it misled + once already). STANDING: silent per-IP throughput shaping at + bsky.network's edge, tripped by the retry pass hammering relay getRepo + redirects from the firehose's own IP. Second data point: retry pass at + 02:12Z (6 resyncs) -> lone 84.8% window. EXPERIMENT ARMED for the + ~06:12Z 08-15 pass: box ingest vs control consumer on the eval box's IP + (relay-control-probe unit there; monitor in the 08-15 session). Verdict + decides: per-IP shaping -> pace retry + port PDS-direct backfill; + both-dip -> network-wide; neither -> theory dies. Mitigation proposed to + operator: retry interval 24h + host-workers 1, compaction 24h. - Prior deploy `56dbf2e` (receipt 21/21; 401 contract byte-identical to Bluesky's edge, probed side-by-side). Prod archive IS key-gated: STREAM_ARCHIVE_API_KEY in /opt/stream-experiment/.env (never in git), -- 2.51.2