diff --git a/docs/upstream-compaction-spec.md b/docs/upstream-compaction-spec.md index d44b6d6..4ce7cc4 100644 --- a/docs/upstream-compaction-spec.md +++ b/docs/upstream-compaction-spec.md @@ -1,9 +1,29 @@ # upstream compaction + tombstone spec (condensed from agent study 2026-07-09) how jetstream physically removes deleted/updated data from sealed segments. -sources: `internal/ingest/orchestrator/compact_deletes.go`, `compaction_watermark.go`, -`internal/tombstone/`, `segment/rewrite.go`, docs/README.md §3.3, the -2026-06-12 cache-invalidation design note. + +**Everything below describes upstream, and every path in this section is an +upstream Go path — none of them exist in this repository.** That matters most +for `internal/tombstone/`, which looks exactly like Stream's own +`src/internal/…` convention and is not ours. + +- upstream sources: `internal/ingest/orchestrator/compact_deletes.go`, + `compaction_watermark.go`, `internal/tombstone/`, `segment/rewrite.go`, and + §3.3 of upstream's `docs/README.md` (upstream's, not this `docs/` directory — + there is no `docs/README.md` here). +- Stream's counterparts, for reading the two side by side: + `storage/tombstone.zig` (which rows may be dropped), + `storage/segment_rewrite.zig` (the physical rewrite), `compact/watermark.zig` + (the durable watermark), and `compact/pass.zig` / `compact/steady.zig`. + +Two references in the original source list are **not committed anywhere here**: +the 2026-07-09 agent study named in the title, and the "2026-06-12 +cache-invalidation design note". Do not go looking for them — this +condensation plus upstream at the pin is the authority that actually exists. +`upstream-bootstrap-spec.md` carries the same caveat. + +`pebble` below is upstream's embedded key-value store (Go); Stream uses RocksDB +in the same role. ## tombstones