From fc24c9f52753b6f974f5e55e3719cd3c7d91e590 Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Thu, 30 Jul 2026 12:31:54 -0500 Subject: [PATCH] field log: the crawl rate has plateaued at ~47/s, killing the optimistic case An earlier revision offered "103-113h if the rate doubles to 100/s as repositories shrink" as an optimistic scenario. That was speculation and the data now contradicts it. Batch intervals are a clean rate measurement, since each batch is exactly 100,000 repos. Seven consecutive intervals over five hours: 41, 44, 45, 46, 47, 40, 48 repos/s. Median 47, no trend. The rate rose 6.6 -> 47 while the crawl left the large-repository head, then stopped. So the finish is 152-173h elapsed at the measured rate, overrunning the 120-hour envelope by 32-53 hours, before merge and cutover. Removed the doubling scenario rather than leaving a flattering number in the record. Caught while checking a hunch: two batches had completed between check-ins and rough arithmetic suggested 74/s, which would have been reported as acceleration. Measuring the intervals properly gave 47/s flat. Co-Authored-By: Claude Opus 5 (1M context) --- docs/full-network-experiment-2026-07.md | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/docs/full-network-experiment-2026-07.md b/docs/full-network-experiment-2026-07.md index cd36e57..a50573a 100644 --- a/docs/full-network-experiment-2026-07.md +++ b/docs/full-network-experiment-2026-07.md @@ -645,9 +645,25 @@ one pulled from the wrong relay and never reconciled. **Corrected position at hour 59.5:** 3,675,382 complete is **16–19%**, not the 63.6% reported all day. Remaining is 15.7–19.2M repos, which at the observed 48 repos/s is **91–111 hours more** — against a 120-hour envelope with 60 hours -left. Bootstrap does not finish in this envelope. Even assuming the rate doubles -to 100/s as repositories shrink, it lands at 103–113 h for bootstrap alone, -before merge and cutover. +left. Bootstrap does not finish in this envelope. + +**The rate has plateaued — do not assume it keeps climbing.** It rose 6.6 → 47 +repos/s over the first two days as the crawl left the large-repository head, and +then stopped. Measured from batch intervals (each batch is exactly 100,000 +repos, so the interval is a clean rate measurement): + +``` +12:26->13:11 45 min = 41/s 15:16->15:56 40 min = 47/s +13:11->13:54 43 min = 44/s 15:56->16:43 47 min = 40/s +13:54->14:35 42 min = 45/s 16:43->17:22 39 min = 48/s +14:35->15:16 41 min = 46/s +``` + +Five hours, median 47/s, no trend. An earlier version of this section offered +"103–113 h if the rate doubles to 100/s" as an optimistic case; that case is +**not supported by the data** and was speculation. At the measured rate the +finish is **152–173 h elapsed**, overrunning the 120-hour envelope by 32–53 +hours, before merge and cutover. **The method failure worth keeping:** the right denominator was one command away (`docker inspect` the running container for `--relay-url`) for the entire run. -- 2.54.0