diff --git a/docs/live-repair.md b/docs/live-repair.md index a17d7f0..cdc14db 100644 --- a/docs/live-repair.md +++ b/docs/live-repair.md @@ -1,7 +1,45 @@ # Live Sync 1.1 repair -Stream follows the pinned Atmos/Jetstream V2 repair contract rather than -inventing a second recovery model. +Stream follows the pinned Atmos/Jetstream V2 repair contract — Atmos being the +Go atproto library upstream Jetstream V2 runs — rather than inventing a second +recovery model. + +Verified against `ingest/repair.zig` on 2026-07-30: `worker_count = 32`, +`pending_capacity = 2048`, `limiter_capacity = 16_384`, and a token bucket of +five with a 12-second refill (five per minute, burst five) all exist as +described. + +**The "64-job bounded queue" is `worker_count * 2`, not an independent 64.** +That expression is now the bound in three separate places — the backfill +dispatch queue, the live scheduler's per-DID pending capacity, and this repair +pool — and each is written in prose as a bare "64" as though it were chosen +there. It is the same expression whose interaction with a 100,000-repo batch +caused the experiment 4 deadlock (`invariants.md`, "a bounded producer must not +outrun its consumer"). Anyone raising `worker_count` moves all three bounds at +once. + +## Expected log noise, so you do not chase it + +Repair logs **one `ERROR` line per failed attempt** (`repair.zig:297`), and the +limiter permits five attempts per DID per minute, so a small permanently-broken +set of accounts produces a steady error stream indefinitely. Measured during +experiment 6: + +``` +141 "repair worker failed" lines in 10 minutes + -- but only 19 distinct DIDs + 94 GetRepoFailed + 39 RepairAuthenticationFailed + 6 RepairRateLimited + 2 AttemptTimeout +``` + +~7 attempts per DID per 10 minutes against a handful of unreachable or +misconfigured PDSes. **This is the system working as specified, not an +incident.** The number to watch is the count of *distinct* DIDs, not the line +rate — line rate scales with the retry allowance, distinct DIDs with the actual +problem. `RepairAuthenticationFailed` in particular is a property of the remote +repository, not of Stream. ## Contract