From de65b561e5e2fccc2e943ab2806c9b260308af11 Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Thu, 30 Jul 2026 12:37:14 -0500 Subject: [PATCH] field log: confirm the denominator against a PDS's own API, and record the near-miss accountCount from listHosts could overcount, which is the correction the feasibility work applied to get 19.4M from 22.7M. Checked directly against a PDS: enoki.us-east reports accountCount 214,128 and its own listRepos returns at least 201,000. So accountCount is close to real repo count, and if anything 19.4M was conservative -- the target is near 22.8M. The same check disproved a reading that nearly reversed the correction. Summing Stream's /status hosts tab gives 3.80M repos across hosts whose accountCount totals 17.3M, a ratio of 0.220. Taken at face value that implies a ~5.0M crawlable network and would have appeared to vindicate the wrong indigo denominator. It is an artifact: Stream's per-host counts are what the global listRepos walk has reached so far, not host totals. Stream has enumerated 94,044 of enoki's ~214,000 -- 44% of one host. Recorded that any per-host figure on /status is walk-progress and must never be read as a host total. Co-Authored-By: Claude Opus 5 (1M context) --- docs/full-network-experiment-2026-07.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/full-network-experiment-2026-07.md b/docs/full-network-experiment-2026-07.md index a50573a..5bb2d0d 100644 --- a/docs/full-network-experiment-2026-07.md +++ b/docs/full-network-experiment-2026-07.md @@ -665,6 +665,28 @@ Five hours, median 47/s, no trend. An earlier version of this section offered finish is **152–173 h elapsed**, overrunning the 120-hour envelope by 32–53 hours, before merge and cutover. +**Independently confirmed against a PDS's own API.** `accountCount` from +`listHosts` could in principle overcount — accounts without crawlable repos — +which is the correction the feasibility work applied to reach 19.4M. Checked +directly: + +| | | +|---|---| +| `enoki.us-east.host.bsky.network` `accountCount` | 214,128 | +| that PDS's own `listRepos` | **≥201,000** (capped at 200 pages) | +| Stream's `/status?tab=hosts` for the same host | **94,044** | + +So `accountCount` ≈ real repos (ratio near 1.0), and **Stream's per-host counts +are partial** — what the *global* `listRepos` walk has reached, not the host's +total. Stream has enumerated 44% of that single host. + +**That trap nearly reversed a correct finding.** Summing Stream's hosts tab +gives 3.80M repos across hosts whose `accountCount` totals 17.3M — a ratio of +0.220, which if taken at face value implies a ~5.0M crawlable network and would +have "vindicated" the wrong indigo denominator. It is an artifact of partial +enumeration. Any per-host progress figure from `/status` must be read as +walk-progress, never as host totals. + **The method failure worth keeping:** the right denominator was one command away (`docker inspect` the running container for `--relay-url`) for the entire run. The wrong one was never checked against the project's own prior work, which -- 2.51.2