diff --git a/docs/architecture.md b/docs/architecture.md index a3b6f23..2d56351 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -71,6 +71,30 @@ durable relay cursor. | conversion + validation | `convert.zig`, `wire.zig` | | verification + repair | `verify.zig`, `repair.zig` | +#### Four worker pools, four different counts + +"32 workers" in one document and "100 workers" in another are both correct and +describe different pools. Nothing here is stale: + +| pool | workers | bounded queue = `workers * 2` | source | +|---|---|---|---| +| initial backfill crawl | **100** | **200** | `ingest/backfill/engine.zig` `default_workers` | +| live scheduler (steady state) | **32** | **64** | `ingest/pipeline.zig` `worker_count` | +| live repair | **32** | **64** | `ingest/repair.zig` `worker_count` | +| failed-repo retry | **16** (4 per host) | — | `ingest/backfill/retry.zig` `default_workers` | + +Only the first is operator-tunable in practice (`--backfill-workers`); the live +counts are matched to upstream Atmos defaults rather than chosen here. + +**`workers * 2` is one expression serving three pools.** It bounds backfill +dispatch, the live scheduler's per-DID pending capacity, and the repair job +queue — so `stream_backfill_queue_depth` reads 200 while `live-scheduler.md` and +`live-repair.md` both describe a "64", and all three are the same rule. It is +also the bound whose interaction with a 100,000-repo batch caused the experiment +4 deadlock (`invariants.md`, "a bounded producer must not outrun its consumer"), +so raising `--backfill-workers` moves a deadlock-relevant bound, not just +concurrency. + ### Storage — the segment format and the metadata store Two places hold state, and the durability ordering between them is the