deploying stream #

This file is the deploy procedure. It deliberately does not track which experiment is running — that state belongs in ../docs/full-network-experiment-2026-07.md, which is maintained as runs happen. An earlier revision asserted that "no replacement experiment should be provisioned" while experiment 6 was in fact running on an admitted artifact, which is exactly the rot that got deployment state removed from the top-level README.

The retired live-only canary no longer runs on the Indigo relay node.

The long-term production target remains a separate host with redundant local storage; passing a cloud-volume experiment is evidence for sizing and throughput, not approval to colocate Stream with Indigo.

The previously deployed e1926f3 image is historical evidence only. It was not the artifact covered by the earlier admission receipt and it failed during the experiment. Do not reuse its digest as a promotion candidate. That gap — an image no receipt covered, with nothing to refuse it — is what scripts/admit now closes.

just package        # ReleaseSafe x86_64-linux-gnu binary in zig-out/bin/
just admit run      # clean tree -> every suite -> exact image -> one receipt
just admit publish  # push that image, pin the receipt to its registry digest
just admit verify sha256:...   # deployment gate; non-zero unless admitted

Nothing may be deployed whose digest just admit verify does not accept. A receipt covers exactly one image digest, never a tag, and a suite recorded as skipped blocks admission just as a failure does — a partial run must never read as a full one. just admission-contract proves those refusals offline.

Do not run just admit run as the deploy gate. It is fine for debugging one suite, but it is invisible to the concurrency limit that keeps two gates off the same machine — which we learned by running two at once and throttling both (see ../docs/deploying.md). The sanctioned entry point is the stream-admission Prefect deployment in the sibling repo zzstoatzz.io/my-prefect-server, which runs the identical scripts/admit run on bare metal (heavypad) with concurrency_limit: 1. .tangled/workflows/admission.yml only triggers it.

Concretely, this repo's deploy path depends on that repo for:

If that deployment is gone or its worker is offline, pushes will not produce receipts and nothing will be deployable. That is deliberate: no receipt, no deploy.

required environment (docs/lessons-from-zlay.md #8, #9) #

• MALLOC_ARENA_MAX=4 — the highest-leverage glibc RSS knob for thread-heavy services (per-thread arena fragmentation)
• public traffic must target --addr; probes and Prometheus must target the private --debug-addr (/healthz, /readyz, and /metrics)
• immutable image tags (git SHA); confirm what's running via the stream_build_info{git_sha,optimize} metric

flags for production #

serve
--addr=:8080
--debug-addr=127.0.0.1:6060
--relay-url=https://relay1.us-east.bsky.network
--upstream-slow-min-rate=50
--plc-url=https://plc.directory
--data-dir=/data
--backfill
--backfill-workers=100
--backfill-async-flush-workers=4
--subscribe-read-log-retention-bytes=268435456
--subscribe-block-cache-bytes=67108864
--subscribe-read-batch=1024
--subscribe-slow-window=60s
--subscribe-slow-min-rate=5

Omitting or emptying --debug-addr disables the debug socket. Do not expose it through the public ingress. /readyz proves both configured accept loops are running; use /status and metrics for ingest/bootstrap health. Run just listener-contract before packaging to prove split routing and the disabled-debug case offline.

The production host target is 16 cores, 32–64 GB RAM, and at least 3.84 TB usable mirrored SSD/NVMe. Keep RocksDB and the archive on the same RAID1 XFS filesystem: segment fsync precedes the RocksDB batch that acknowledges those bytes, and the unified filesystem is reflink-snapshotted for backup.

The defaults reserve at most 8 GiB for transient backfill allocations while admitting 100 concurrent disk-backed downloads. CAR preparation is serialized, copies each complete CAR into that budget, and releases the scratch mmap before concurrent emission. This avoids partial-index deadlock while prepared repositories still emit concurrently. Four bootstrap-only workers compress detached JSS blocks and commit them in order. The pod still needs headroom for RocksDB, archive/compaction buffers, live capture, thread stacks, zstd frames, and libc; do not set the container limit equal to the backfill budget.

promotion gate #

The complete gate is ../docs/semantic-parity.md. In particular, a promotion must test individual repository durability inside a large dispatch batch, metadata read corruption, unreadable archive files, persistent live encoding failures, and transient firehose disconnects. The receipt must bind those results to the exact deployed image digest. Throughput, backup restore, client interoperability, and alerts remain necessary, but they cannot waive a blocked semantic row.

alert rules #

prometheus/rules.yml is the in-repo source for the Prometheus alert rules; the box's Prometheus reads its own copy at /opt/stream-experiment/rules.yml (mounted to /etc/prometheus/rules.yml, listed under rule_files in prometheus.yml), so a rule change here is not live until that copy is replaced and Prometheus restarted (docker compose restart prometheus; the container has no --web.enable-lifecycle, so there is no reload endpoint, and up -d does not restart a container whose compose definition is unchanged). Write the box copy in place (cat new > rules.yml): the file is a single-file bind mount. Wired up 2026-08-29; before that the box had no rules loaded at all. Validate with promtool check rules deploy/prometheus/rules.yml (e.g. via docker run --rm -v "$PWD/deploy/prometheus:/rules:ro" --entrypoint promtool prom/prometheus check rules /rules/rules.yml).

paging #

Since 2026-10-02 an alertmanager service in the box's compose stack (prom/alertmanager:v0.33.0, pinned by digest) posts critical and warning alerts to Discord; info alerts are dropped. prometheus/alertmanager.yml and prometheus/prometheus.yml are the in-repo sources for the box copies at /opt/stream-experiment/. The compose file itself lives only on the box.

The webhook URL is the one zlay's Alertmanager uses (the zlay-discord-webhook secret in the zlay cluster's monitoring namespace). It is on the box as /opt/stream-experiment/discord-webhook, owner 65534, mode 400, and is read through webhook_url_file. Rotating the webhook means replacing both copies. To prove the route end to end:

docker compose exec -T alertmanager amtool --alertmanager.url=http://127.0.0.1:9093 \
  alert add StreamPagingTest severity=warning job=stream
docker compose exec -T alertmanager wget -qO- http://127.0.0.1:9093/metrics \
  | grep 'alertmanager_notifications.*discord'

dashboard gate #

grafana/jetstream-upstream.json is the byte-exact dashboard from the pinned Jetstream V2 reference. grafana/stream.json is the deployable artifact, generated deterministically from that source. The Go runtime row is adapted to honest process metrics, and one explicit Stream-only row exposes scheduler tickets, unsettled stage gaps, and terminal failures during bootstrap and steady state; every upstream protocol/lifecycle panel remains unchanged. Run just dashboard-test, then just process-metrics-contract, then just dashboard-contract <metrics-url-or-saved-scrape>. These checks are network-independent when given a saved scrape or the loopback harness and fail when a queried family is absent. They verify dashboard structure and metric presence only; they are not a promotion decision.