From 952857e1dc8613fd6158e5c94248666701b23616 Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Sun, 16 Aug 2026 15:51:08 -0500 Subject: [PATCH] handoff: cutover to relay1.us-east during bsky.network front outage Co-Authored-By: Claude Fable 5 --- HANDOFF.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/HANDOFF.md b/HANDOFF.md index 7253c49..0d0f768 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -6,6 +6,21 @@ closed and chronicled in git history 08-11..08-12. ## where things stand +- **UPSTREAM CUTOVER 20:46Z 08-16: --relay-url now relay1.us-east.bsky.network** + (compose.yaml, backup kept). The bsky.network front hostname wedged from + ~14:00Z (flapping) to fully dead by ~18:10Z — TCP accepts but nothing + above the socket completes; relay backends (relay1.us-east/us-west) stayed + healthy on adjacent IPs the whole time, and every consumer of the front + (including Bluesky's own jetstream fleet) went to zero on relay-eval. + Cutover verified GAPLESS before switching: relay1 shares the front's seq + space (probe with our cursor 32793755551 returned that exact seq first). + Resume confirmed 20:49:33Z, catch-up at ~4.6k/s. Switching back (or not) + when the front recovers is zero-risk — same seq space. Side effect: the + restart killed the 12:17Z compaction pass, but per-chunk persistence had + already advanced the watermark to 23425579908 and drained tombstones + 2.6M -> 243K (first drain since Aug 14). Timers reset: next compaction + ~00:49Z 08-17, retry pass ~20:49Z 08-17. + - **prod runs `5f6de76`** (2026-08-16 07:12Z, receipt 21/21): task #12, listener-first startup (upstream parity — routes registered before storage, 503-gated until ready). Verified live: public+debug listeners -- 2.51.2