diff --git a/docs/experiment-6-serving-receipt.md b/docs/experiment-6-serving-receipt.md new file mode 100644 index 0000000..26342ce --- /dev/null +++ b/docs/experiment-6-serving-receipt.md @@ -0,0 +1,58 @@ +# experiment 6 — serving receipt + +Captured 2026-08-04 ~20:30Z, minutes after the phase 3 commit. Every claim below +was verified against the running instance at capture time. + +## the phase commit + +``` +20:10:47Z merge-tail compaction: watermark 22976771240, 1 chunks, + 937/6479 segments rewritten, 90170067 rows dropped +20:11:43Z merge discovery: skipped by configuration +20:11:44Z phase -> steady_state +20:11:44Z resuming from persisted cursor 32408688748 +``` + +Bootstrap start to steady_state: **2026-07-28 05:01:47Z → 2026-08-04 20:11:44Z +(~183h ≈ 7.6 days)**, including one deliberate cutover (unfetchable tail), one +disk-full crash (reflink snapshot divergence), and a 6.6h merge-tail compaction. + +## proof of serving + +1. **Gate open.** `GET /xrpc/network.bsky.jetstream.listSegments` → **200** with + full segment metadata and cursor pagination (6,480 segments). Was 503 + throughout bootstrap/merging. Note the route: `network.bsky.jetstream.*` — + the `jetstream.listSegments` path used in earlier notes 404s. +2. **Read-back + decode.** `GET .../getSegment?name=seg_0000000000.jss` + (param is `name`, not `segment` — that one belongs to `getBlock`) returned + 200 with all 277,192,887 bytes; `stream inspect-segment` on the download: + magic `jss0`, state sealed, **checksum 0x20f0665e7822a74c valid**, 3,504,089 + events / 858 blocks / seq [1, 3506746], 32 collection NSIDs enumerated. + Size, checksum, counts all match the listSegments row exactly. +3. **Cleanup complete.** `/data/stream/backfill` gone, 0 stale `.tmp`. +4. **`/status`**: `state live`, `phase steady_state`, durable cursor + 22,976,771,240 (~23.0B events durable), 16,300,003 repos total / + 16,285,748 complete. + +## what was still pending at capture + +- **Live ingest.** After the commit, main enters `compact.steady.Compactor.rebuild` + — a synchronous full-archive fold rebuilding steady-compaction state *before* + the consumer dials upstream (verified by stack: `foldSegment` ← + `Compactor.rebuild` ← `main`, walking segments at ~25/min ⇒ ~4h). Until it + finishes: `upstream seq 0`, `events this run 0`, no outbound connection. + This is sequencing, not failure. A watcher fires when `stream_upstream_seq > 0`. +- **Replay.** Cursor 32408688748 confirmed still replayable from bsky.network at + capture (fresh wsprobe: 101 + binary frame). Window closes 2026-08-06 + 07:04:58Z; rebuild + connect fits with >24h margin. +- **Subscribe surface under live load** — verify after replay starts. + +## honest footnotes + +- `merge discovery: skipped by configuration` — the compose sets + `skip-merge-discovery`, so DIDs that first appeared after the bootstrap + cursor were **not** enqueued. Coverage is the bootstrap's 16.29M; new-account + pickup begins with live ingest. +- The kept/dropped ratio question from the handoff (§11) stands, softened by + compaction's 90.17M dropped rows (85.2M sync, 3.2M record, 1.7M account). +- One restart total during merging (disk full, self-healed); oom-kills all run: 1.