From 405a43a8df307b3ec8a63de68deb6644e4db6bab Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Tue, 14 Jul 2026 01:25:50 -0500 Subject: [PATCH] record full-network experiment baseline --- docs/full-network-experiment-2026-07.md | 87 +++++++++++++++++++++++++ 1 file changed, 87 insertions(+) create mode 100644 docs/full-network-experiment-2026-07.md diff --git a/docs/full-network-experiment-2026-07.md b/docs/full-network-experiment-2026-07.md new file mode 100644 index 0000000..c2ea2e4 --- /dev/null +++ b/docs/full-network-experiment-2026-07.md @@ -0,0 +1,87 @@ +# Full-network experiment — July 2026 + +This is the durable field log for the first public whole-network bootstrap +experiment. It records commissioning failures as well as the clean measurement +baseline so a future operator does not have to reconstruct them from chat or +pod logs. + +## Experiment envelope + +- Started: 14 July 2026 +- Decision deadline: 15 July 2026 at 05:47:25 UTC +- Public page: +- Hetzner HEL1 CPX62: 16 shared x86 vCPUs, 32 GiB RAM, 640 GB local disk +- Data volume: 4 TB, XFS, mounted at /data +- Kubernetes: single-node k3s, one Stream replica, Recreate rollout strategy +- Container limit: 16 CPUs / 24 GiB; request: 8 CPUs / 12 GiB +- Approximate experiment cost: USD 18 for the authorized 24-hour window + +The existing Indigo node and stream.waow.tech canary were not repurposed. + +## Commissioning findings + +### Worker memory was bounded but undersized + +Build 94e50a0 requested sixteen workers under an 8 GiB aggregate allocation +budget. Its 1.25 GiB per-worker ceiling was too small for a real sub-512 MiB CAR +while the response body, CID index, MST conversion arena, and arena growth +coexisted. The bounded allocator returned OutOfMemory at about 1.8 GiB aggregate +use and the process treated that as lifecycle-fatal. + +Build ed91c32 raised the conversion envelope to 2 GiB, yielding four effective +workers under the same 8 GiB aggregate budget. Exhausting one bounded worker is +now a failed-repository transition for durable retry; archive append failures +remain lifecycle-fatal. + +### Pooled redirected connections produced false failures + +getRepo requests go through the relay and redirect to independently operated +PDSes. The original long-lived clients emitted dozens of +HttpConnectionClosing failures per minute. Direct curl probes for the same DIDs +returned HTTP 200 and complete CAR bodies. + +Build ade3d5d disables keep-alive for bootstrap downloads. In its first +three-minute public run it produced zero connection-closing errors, compared +with dozens in the same interval before the change. This trades connection +setup work for correctness and should be revisited if the Zig HTTP pool learns +to discard these closing redirected sockets reliably. + +### Restart recovery worked + +Both spontaneous and intentional restarts: + +- sealed the interrupted archive segment; +- reopened the same RocksDB metadata store; +- recovered the live-capture segment; +- resumed the firehose from the persisted upstream cursor; and +- returned the health and public pages while subscription serving remained + gated. + +An interrupted 1,000-repository durability page is intentionally converted from +not_started to pending on reopen. That is correct service recovery, but repeated +commissioning rollouts would make a historical throughput measurement +unrepresentative by deferring those repositories to current-state resync. + +## Clean baseline + +Commissioning produced 7.4 GB of disposable data across several intentional +rollouts. After preserving the findings above, the volume was cleared once and +build 0bfeafb started from a fresh archive at approximately 06:24:39 UTC on +14 July 2026. The billing/decision deadline was not extended. + +At 41 seconds: + +- archive next sequence started at 1; durable cursor reached 2,846,721; +- upstream sequence was 853,375,749; +- 1,728 live events and 1,753 frames were received; +- verification reported 1,724 valid and zero invalid; +- four backfill workers were active with eight queued; +- bounded transient allocation was 21.3 MB, 614.2 MB peak; +- pod RSS was 407 MiB, with zero restarts; +- /data contained 212.9 MB; and +- all drop counters were zero. + +The 24-hour monitor should measure from this clean baseline while retaining the +original cost deadline. Any later rollout must be called out because it changes +the interpretation of repository completion and throughput. + -- 2.51.2