From d77fcede066f08560ebbc75f189b76d4057ec4be Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Mon, 10 Aug 2026 03:15:33 -0500 Subject: [PATCH] docs: why delete compaction is slow, measured Co-Authored-By: Claude Opus 5 (1M context) --- docs/compaction-cost.md | 105 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 docs/compaction-cost.md diff --git a/docs/compaction-cost.md b/docs/compaction-cost.md new file mode 100644 index 0000000..d0264a7 --- /dev/null +++ b/docs/compaction-cost.md @@ -0,0 +1,105 @@ +# why delete compaction is slow + +Measured on production (hel1, cx43, 8 vCPU) on 2026-08-10, against build +`22bf62c`, while the archive held 6,584 sealed segments / ~23.2B events. + +The short version: **a pass rewrites the whole archive to apply the deletes of a +much smaller window**, and nothing in the design bounds that. Everything else +here is a multiplier on top of it. + +## the measurements + +| quantity | value | how | +| --- | --- | --- | +| sealed segments | 6,584 | `/status` | +| events per segment | 3.33M median (1.8M–3.6M) | `listSegments`, 100-segment sample | +| segment size | ~277 MB | `getSegment` | +| uncompacted window | 100.9M events ≈ **30 segments** | tip 23,236,818,914 − watermark 23,135,908,535 | +| segments rewritten per pass | **6,584** | `pass.zig` passes all of `metas` to the rewrite | +| live tombstones | 3.69M entries / 472 MB | `jetstream_compaction_tombstone_set_entries` | +| watermark lag | 2.3 days | `jetstream_compaction_watermark_lag_seconds` | +| observed rate | **3.4–3.5 segments/min** | `segments_examined_total` sampled over hours | +| box state while compacting | 56% idle, iowait 1.1%, stream at 245% CPU | `top`, 8 cores | + +So: **~220x amplification** (6,584 rewritten to apply the deletes found in ~30), +CPU-bound on zstd, not disk. + +## why it is O(archive) and not O(new data) + +A delete is retroactive. A tombstone written today can target a record written +any time in the past, so "what is new" does not bound "what must be rewritten." +The fold is over the new window; the rewrite is over everything. + +The mechanism meant to cut that down is the candidate-DID bloom prefilter, and +at this scale it does nothing: + +- `Snapshot.candidateDids` returns `null` once distinct DIDs exceed + `bloom_narrow_max_dids` (100,000). With 3.69M tombstones it is never close. +- Raising the cap would not help and would cost memory. The test is *"does this + segment contain any tombstoned DID"*, but a segment holds 3.3M events from a + large slice of the network, so with millions of candidate DIDs essentially + every segment matches. Expected tombstoned rows per segment ≈ 3.69M / 6,584 ≈ + **560** — no segment is clean. +- The real mismatch: the prefilter is **DID-level** while the unit of deletion + is `(did, collection, rkey)`. A precise-enough skip test is the only thing + that makes this sublinear. + +**Upstream is identical.** Verified against `bluesky-social/jetstream` at the +pinned `f29815c`: `applyCompactionChunk` fans out over all `sealed` segments, +and `compact_deletes.go:342` does the same `candidateDIDs = nil` past the same +100k default. This amplification is inherited, not a porting mistake. + +## the feedback loop that made it permanent + +This is the part that matters for prevention. Until 2026-08-10 the watermark was +saved **once per chunk**, and a chunk was the whole archive (bounded only by a +32M tombstone cap against a 3.7M live set). At 3.4 segments/min a pass needs far +longer than the interval between process restarts, so: + +restart → refold → rewrite for hours → killed by the next deploy → watermark +never moves → backlog grows → more tombstones → slower pass → repeat. + +`jetstream_compaction_passes_total` sat at **0 across two binaries and 4+ days** +while `segments_examined_total` climbed the whole time. The pass was never +stuck; it was never finishing. A counter that only increments on completion +cannot distinguish those two, which is why this went unexplained twice. + +## what was fixed, and what remains + +Fixed 2026-08-10: + +- `7136ebb` — commit the watermark every 64 segments instead of once per chunk. + Each segment is still rewritten once; only commit granularity changes, so a + restart costs at most a batch instead of everything. Breaks the loop above. +- `b3ea690` — reuse a block's compressed frame when it drops nothing. The port + recompressed every block unconditionally and discarded the result when the + segment turned out clean; upstream's `segment/rewrite.go` keeps the source + frame. ~810 blocks per segment, ~50% of them untouched, and zstd compression + is the expensive direction. + +Still open, in the order I would do them: + +1. **Rewrite workers.** Production runs 2. Upstream's default and ours is + `min(NumCPU, 8)`; the 2 is deployment config, on a box measured 56% idle. + Memory is the real constraint — each worker holds a segment's output buffer. +2. **A precise skip test.** Per-segment or per-block filters keyed on the record + identity rather than the DID, so old segments can be rejected without being + decompressed. This is the only fix that changes the complexity class. +3. **Bound the backlog, not just the pass.** See below. + +## how to not get here again + +The failure was not slowness; it was slowness that compounded unobserved. Watch +for the loop, not the symptom: + +- **Alert on `jetstream_compaction_watermark_lag_seconds`.** It is the honest + measure of exposure: everything witnessed inside that window still serves + deleted rows, because nothing in the read path consults the tombstone set — + rewriting the segment is what removes a row. +- **Alert on pass duration approaching the interval.** Once a pass cannot finish + within the interval, the backlog only grows, and every restart is a full + reset of the work. +- **Treat `passes_total == 0` with a climbing `segments_examined_total` as + in-progress, not idle.** Opposite responses. +- The tombstone set is also a memory cost: 3.69M entries held 472 MB of a + 15.6 GB box, and it only shrinks when the watermark advances. -- 2.51.2