diff --git a/docs/ingest-pacing-evidence-2026-08-05.md b/docs/ingest-pacing-evidence-2026-08-05.md new file mode 100644 index 0000000..6d3d805 --- /dev/null +++ b/docs/ingest-pacing-evidence-2026-08-05.md @@ -0,0 +1,57 @@ +# live ingest is self-paced at ~360 events/s — evidence bundle + +Measured 2026-08-05 ~22:30Z on the experiment 6 box (ccx53, 32 cores, 128 GB), +image `6a511fa`, `--no-verify` active. For whoever owns the consumer/pipeline. + +## the number + +`rate(stream_upstream_seq[1h])` ≈ **360-366 seqs/s**, flat across 10m/30m/2h +windows, zero firehose disconnects. At ~360/s against the relay's ~300-400/s +production, catch-up slope is ~1.1-1.2 — closing a 54h lag takes weeks. + +## what it is NOT + +- **Not the relay.** A bare Python websocket client on the same host, same + relay, nearby backlog cursor (`32417500000`) drained **190.9 MB / ~3,700 + frames/s over 10s**. The pipe offers ≥10x what stream consumes. +- **Not disk.** iostat during ingest: ~0.2 writes/s, 0.04% util. (Merge wrote + ~70k events/s through this same archive on this volume.) +- **Not CPU or memory.** 32 cores mostly idle; consumer threads parked in + `hrtimer_nanosleep`; RSS flat at 53 GB of 122.8. +- **Not verification.** `--no-verify` is on; the plc.directory throttle + (~1.5/s, ~50% key-cache miss) was the previous bottleneck and is gone. + +## where to look + +`src/internal/ingest/pipeline.zig:17-21`: + +```zig +const worker_count = 32; +const key_queue_capacity = worker_count * 2; +const result_capacity = 4096; +const batch_size = 50; +const batch_timeout ... fromMilliseconds(500); +``` + +360/s with all resources idle and threads sleeping is the signature of a +pacing/batching structure, not saturation. Suggestive arithmetic: the observed +rate ≈ 9 events per 25 ms tick. Prior art: the bootstrap-era measurement that +4x workers bought 1.23x and "production is slower than a plain fetcher on half +the cores" — the serialization was already visible then. + +## reproduction + +```bash +# consumption rate +rate(stream_upstream_seq[30m]) # via prometheus on the box +# offered rate (same box, bare client) +# see the 12-line python in this doc's git history / session notes: raw ws +# handshake to bsky.network with a backlog cursor, count frames for 10s +``` + +## why it matters + +The archive serves (receipt: `experiment-6-serving-receipt.md`) but sits ~54h +behind live and closes the gap at ~0.1-0.2h per hour. Every day of glide costs +~€33 (ccx53 pricing). A consumer that merely kept pace with the bare client's +drain rate would catch up in under 8 hours.