This repository has no description

atproto: dissolve the sweep's global barriers into per-host ladders master

A sweep had four global barriers: shallow -> deepen, then one after each deepening round. Every one of them ended in a straggler tail with most of the 32 lanes idle. Measured on a 20,747-repo overnight sweep (4h50m): the shallow phase spent 15.2 of its 28.3 minutes past its 90% mark, and the deepen rounds spent 126 of their 262 minutes in their last 10% of windows. Roughly half the wall clock was most-lanes-idle, waiting on the slowest host, four times over. The phases are gone. Each host lane now runs its own complete program -- laneProgram: a shallow queue, then a ladder of repos bucketed by how many windows they have had -- and a lane that finishes hands its slot to a host that has not started yet. Nothing waits for anything on another host. The guarantees that mattered survive, rescoped to the host they were always really about: - shallow strictly before deepening, per repo; - breadth-first within a host: taking work from the lowest non-empty ladder bucket means no repo gets its (n+1)th window while another repo on that host waits for its nth. Bucketing rather than a round-robin queue so a repo that joins late is absorbed into the order instead of being left a lap behind; - one worker per host, ever, and the global cap from --sweep-concurrency. Slots are handed out in the order lanes were added -- own DIDs first -- which now needs an explicit FIFO queue: a buffered channel would have handed them out in whatever order the worker goroutines happened to wake up; - a repo resolved late still preempts its lane's ladder; - failures never abort a lane or the sweep, cancellation stops between steps, and an all-shallow-failed sweep still errors. What is deliberately given up: the global round barrier, "everyone reaches 7d before anyone starts 30d" across hosts. That barrier was exactly the mechanism that made every lane wait on the slowest host. runLanes, hostLanes, the two-phase sweepShallow/sweepDeepen split and the per-round deepenPending scan are deleted; sweepPlan reads every row once up front and Sweep drives one scheduler. The status line loses phase= and round=, which no longer mean anything, and reports both halves at once: backfill sweep shallow=19000/20747 deepened=4300/20013 windows=41022 horizon=1753142400 horizon keeps its meaning -- the instant after which everything this node serves is indexed -- but is now maintained incrementally from the watermark each window reports (DeepenRepo returns it) rather than by rescanning every row at a round boundary, of which there are none. Also: reposync gives a host that is not there at all -- refused connection, or NXDOMAIN -- two attempts instead of five. Timeouts, 429s and 5xx keep the full ladder. The measured straggler tails are mostly switched-off PDSes, and every repo on one was costing a full backoff ladder to rediscover what the first connection attempt already said. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>


+932 -460
5 changed files