jetstream v2 in zig stream.waow.tech
README.md

deploying stream #

This file is the deploy procedure. It deliberately does not track which experiment is running — that state belongs in ../docs/full-network-experiment-2026-07.md, which is maintained as runs happen. An earlier revision asserted that "no replacement experiment should be provisioned" while experiment 6 was in fact running on an admitted artifact, which is exactly the rot that got deployment state removed from the top-level README.

The retired live-only canary no longer runs on the Indigo relay node.

The long-term production target remains a separate host with redundant local storage; passing a cloud-volume experiment is evidence for sizing and throughput, not approval to colocate Stream with Indigo.

The previously deployed e1926f3 image is historical evidence only. It was not the artifact covered by the earlier admission receipt and it failed during the experiment. Do not reuse its digest as a promotion candidate. That gap — an image no receipt covered, with nothing to refuse it — is what scripts/admit now closes.

just package        # ReleaseSafe x86_64-linux-gnu binary in zig-out/bin/
just admit run      # clean tree -> every suite -> exact image -> one receipt
just admit publish  # push that image, pin the receipt to its registry digest
just admit verify sha256:...   # deployment gate; non-zero unless admitted

Nothing may be deployed whose digest just admit verify does not accept. A receipt covers exactly one image digest, never a tag, and a suite recorded as skipped blocks admission just as a failure does — a partial run must never read as a full one. just admission-contract proves those refusals offline.

required environment (docs/lessons-from-zlay.md #8, #9) #

  • MALLOC_ARENA_MAX=4 — the highest-leverage glibc RSS knob for thread-heavy services (per-thread arena fragmentation)
  • public traffic must target --addr; probes and Prometheus must target the private --debug-addr (/healthz, /readyz, and /metrics)
  • immutable image tags (git SHA); confirm what's running via the stream_build_info{git_sha,optimize} metric

flags for production #

serve
--addr=:8080
--debug-addr=127.0.0.1:6060
--relay-url=https://relay.waow.tech
--plc-url=https://plc.directory
--data-dir=/data
--backfill
--backfill-workers=100
--backfill-async-flush-workers=4
--subscribe-read-log-retention-bytes=268435456
--subscribe-block-cache-bytes=67108864
--subscribe-read-batch=1024
--subscribe-slow-window=60s
--subscribe-slow-min-rate=5

Omitting or emptying --debug-addr disables the debug socket. Do not expose it through the public ingress. /readyz proves both configured accept loops are running; use /status and metrics for ingest/bootstrap health. Run just listener-contract before packaging to prove split routing and the disabled-debug case offline.

The production host target is 16 cores, 32–64 GB RAM, and at least 3.84 TB usable mirrored SSD/NVMe. Keep RocksDB and the archive on the same RAID1 XFS filesystem: segment fsync precedes the RocksDB batch that acknowledges those bytes, and the unified filesystem is reflink-snapshotted for backup.

The defaults reserve at most 8 GiB for transient backfill allocations while admitting 100 concurrent disk-backed downloads. CAR preparation is serialized, copies each complete CAR into that budget, and releases the scratch mmap before concurrent emission. This avoids partial-index deadlock while prepared repositories still emit concurrently. Four bootstrap-only workers compress detached JSS blocks and commit them in order. The pod still needs headroom for RocksDB, archive/compaction buffers, live capture, thread stacks, zstd frames, and libc; do not set the container limit equal to the backfill budget.

promotion gate #

The complete gate is ../docs/semantic-parity.md. In particular, a promotion must test individual repository durability inside a large dispatch batch, metadata read corruption, unreadable archive files, persistent live encoding failures, and transient firehose disconnects. The receipt must bind those results to the exact deployed image digest. Throughput, backup restore, client interoperability, and alerts remain necessary, but they cannot waive a blocked semantic row.

dashboard gate #

grafana/jetstream-upstream.json is the byte-exact dashboard from the pinned Jetstream V2 reference. grafana/stream.json is the deployable artifact, generated deterministically from that source. The Go runtime row is adapted to honest process metrics, and one explicit Stream-only row exposes scheduler tickets, unsettled stage gaps, and terminal failures during bootstrap and steady state; every upstream protocol/lifecycle panel remains unchanged. Run just dashboard-test, then just process-metrics-contract, then just dashboard-contract <metrics-url-or-saved-scrape>. These checks are network-independent when given a saved scrape or the loopback harness and fail when a queried family is absent. They verify dashboard structure and metric presence only; they are not a promotion decision.