From cea018107153f44a2bc5972cd42fe1ed721ad2c6 Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Thu, 30 Jul 2026 05:13:51 -0500 Subject: [PATCH] docs: the bootstrap spec defers to a study that is not in the repo upstream-bootstrap-spec.md and upstream-compaction-spec.md both say "full detail in the study", referring to a 2026-07-09 agent study that is not committed anywhere here. A newcomer hitting a gap goes looking for it and finds nothing. Says so plainly and redirects to upstream at the pin. Also, for a file whose every line is a rule: states up front that it describes UPSTREAM rather than Stream, since a reader can easily take these as Stream's implemented behaviour -- the audit note at the bottom warns about exactly that, but only after 90 lines. Defines pebble (upstream's embedded KV; RocksDB fills the role in Stream), which appears across four docs and is explained in none, and expands Atmos on first use. Records that the 100k batch shuffle is WITHIN a batch, so batch N still draws from listRepos position Nx100k. That is the detail needed to read throughput correctly: repos/s is flat inside a batch and shifts between them, because batch position decides repository sizes. Verified and NOT changed: --max-backfill-repos, which looked like a transposition of --backfill-max-repos. Stream accepts both spellings. Co-Authored-By: Claude Opus 5 (1M context) --- docs/upstream-bootstrap-spec.md | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/docs/upstream-bootstrap-spec.md b/docs/upstream-bootstrap-spec.md index c353589..9164e34 100644 --- a/docs/upstream-bootstrap-spec.md +++ b/docs/upstream-bootstrap-spec.md @@ -1,11 +1,26 @@ # upstream bootstrap + lifecycle spec (condensed from agent study 2026-07-09) Current review pin: `bluesky-social/jetstream` -`f29815c391fc2644f8a3dd36b899fb3697dd1ea6`, Atmos v0.2.14. The pin is an -audit receipt, not permission to drift: refresh this comparison immediately -before provisioning any full-network experiment. +`f29815c391fc2644f8a3dd36b899fb3697dd1ea6`, Atmos v0.2.14 (the Go atproto +library upstream runs). The pin is an audit receipt, not permission to drift: +refresh this comparison immediately before provisioning any full-network +experiment. -full detail in the study; keystones for the zig port: +This describes **upstream's** behaviour, not Stream's. A rule here is what +Stream is measured *against*; see the audit note at the end, and +`bootstrap-semantic-parity.md` for what Stream actually proves. + +> **"the study" is not in this repository.** This file and +> `upstream-compaction-spec.md` both defer to a 2026-07-09 agent study for full +> detail, and no such document is committed here. Treat this condensation as the +> authority we actually have, and go to upstream at the pin above for anything it +> does not cover — not to a study you will not find. + +`pebble` below is upstream's embedded key-value store (Go). Stream uses RocksDB +in the same role, so "pebble `phase`" means the durable phase record, whatever +the engine. + +keystones for the zig port: ## lifecycle phases: bootstrap → merging → steady_state, durable (upstream: pebble `phase`). @@ -18,6 +33,11 @@ fsync data before pebble commit, always; fsync dir before deleting durable state ## bootstrap - listRepos pages of 1000 → batches of 100k, shuffled, dispatched to 100 workers. + The shuffle is *within* a batch, so batch N still draws from listRepos position + N×100k. That distinction matters when reading throughput: repos/s stays roughly + flat inside one batch and shifts between batches, because it is the batch's + place in listRepos order — not the shuffle — that decides the repository sizes + in it. - cursors: relay/list_repos_cursor (checkpointed only after durability drain; drains to "" on completion) + bootstrap/last_listrepos_cursor (for merge discovery), saved atomically. -- 2.51.2