From 5f557b44b68c43820d45d3f3c63b57ab68d77c9a Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Thu, 30 Jul 2026 05:22:12 -0500 Subject: [PATCH] docs: compaction spec's source paths are upstream Go, and it never said so All four paths in the sources list -- compact_deletes.go, compaction_watermark.go, internal/tombstone/, segment/rewrite.go -- are upstream's, and none exist in this repo. internal/tombstone/ is the trap: it is indistinguishable from Stream's own src/internal/ convention, so a reader goes looking for it here. Marked the whole section as upstream and listed Stream's actual counterparts (storage/tombstone.zig, storage/segment_rewrite.zig, compact/watermark.zig, compact/pass.zig, compact/steady.zig) so the two can be read side by side. Also names the two references that are not committed anywhere: the 2026-07-09 agent study in the title, and the "2026-06-12 cache-invalidation design note". Same dangling-reference problem fixed in upstream-bootstrap-spec.md last pass; this closes the second of the two. "docs/README.md 3.3" is upstream's docs/README.md -- there is no docs/README.md here, so the citation silently pointed at our own docs/. Defines pebble, consistent with the other upstream spec. Co-Authored-By: Claude Opus 5 (1M context) --- docs/upstream-compaction-spec.md | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/docs/upstream-compaction-spec.md b/docs/upstream-compaction-spec.md index d44b6d6..4ce7cc4 100644 --- a/docs/upstream-compaction-spec.md +++ b/docs/upstream-compaction-spec.md @@ -1,9 +1,29 @@ # upstream compaction + tombstone spec (condensed from agent study 2026-07-09) how jetstream physically removes deleted/updated data from sealed segments. -sources: `internal/ingest/orchestrator/compact_deletes.go`, `compaction_watermark.go`, -`internal/tombstone/`, `segment/rewrite.go`, docs/README.md §3.3, the -2026-06-12 cache-invalidation design note. + +**Everything below describes upstream, and every path in this section is an +upstream Go path — none of them exist in this repository.** That matters most +for `internal/tombstone/`, which looks exactly like Stream's own +`src/internal/…` convention and is not ours. + +- upstream sources: `internal/ingest/orchestrator/compact_deletes.go`, + `compaction_watermark.go`, `internal/tombstone/`, `segment/rewrite.go`, and + §3.3 of upstream's `docs/README.md` (upstream's, not this `docs/` directory — + there is no `docs/README.md` here). +- Stream's counterparts, for reading the two side by side: + `storage/tombstone.zig` (which rows may be dropped), + `storage/segment_rewrite.zig` (the physical rewrite), `compact/watermark.zig` + (the durable watermark), and `compact/pass.zig` / `compact/steady.zig`. + +Two references in the original source list are **not committed anywhere here**: +the 2026-07-09 agent study named in the title, and the "2026-06-12 +cache-invalidation design note". Do not go looking for them — this +condensation plus upstream at the pin is the authority that actually exists. +`upstream-bootstrap-spec.md` carries the same caveat. + +`pebble` below is upstream's embedded key-value store (Go); Stream uses RocksDB +in the same role. ## tombstones -- 2.51.2