From 3dec08dad9a5641de5b7922a64f345dcd28db777 Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Thu, 30 Jul 2026 07:57:53 -0500 Subject: [PATCH] docs: disambiguate the four worker pools, and finish the workers*2 pattern Deliberate cross-doc sweep for contradictions, since reading one file at a time structurally cannot find them. Worker counts looked inconsistent across docs -- 32, 100, 16 -- and are not: they are four distinct pools. Verified each against code and tabled them in architecture.md so "32 workers" here and "100 workers" there no longer reads as one being rotted. initial backfill crawl 100 workers (engine.zig default_workers) live scheduler 32 (pipeline.zig worker_count) live repair 32 (repair.zig worker_count) failed-repo retry 16, 4/host (retry.zig default_workers) This also completes the workers*2 finding from the live-scheduler and live-repair passes. The same expression bounds three of those pools, so backfill's bound is 200 while the two live docs each describe a literal "64" -- one rule, three values. That explains stream_backfill_queue_depth reading exactly 200 on the running experiment. It is also the bound whose interaction with a 100,000-repo batch caused the experiment 4 deadlock, so raising --backfill-workers moves a deadlock-relevant bound rather than just concurrency. No actual cross-doc contradictions found in this sweep -- the gap was a missing disambiguation, not disagreement. Co-Authored-By: Claude Opus 5 (1M context) --- docs/architecture.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/architecture.md b/docs/architecture.md index a3b6f23..2d56351 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -71,6 +71,30 @@ durable relay cursor. | conversion + validation | `convert.zig`, `wire.zig` | | verification + repair | `verify.zig`, `repair.zig` | +#### Four worker pools, four different counts + +"32 workers" in one document and "100 workers" in another are both correct and +describe different pools. Nothing here is stale: + +| pool | workers | bounded queue = `workers * 2` | source | +|---|---|---|---| +| initial backfill crawl | **100** | **200** | `ingest/backfill/engine.zig` `default_workers` | +| live scheduler (steady state) | **32** | **64** | `ingest/pipeline.zig` `worker_count` | +| live repair | **32** | **64** | `ingest/repair.zig` `worker_count` | +| failed-repo retry | **16** (4 per host) | — | `ingest/backfill/retry.zig` `default_workers` | + +Only the first is operator-tunable in practice (`--backfill-workers`); the live +counts are matched to upstream Atmos defaults rather than chosen here. + +**`workers * 2` is one expression serving three pools.** It bounds backfill +dispatch, the live scheduler's per-DID pending capacity, and the repair job +queue — so `stream_backfill_queue_depth` reads 200 while `live-scheduler.md` and +`live-repair.md` both describe a "64", and all three are the same rule. It is +also the bound whose interaction with a 100,000-repo batch caused the experiment +4 deadlock (`invariants.md`, "a bounded producer must not outrun its consumer"), +so raising `--backfill-workers` moves a deadlock-relevant bound, not just +concurrency. + ### Storage — the segment format and the metadata store Two places hold state, and the durability ordering between them is the -- 2.51.2