diff --git a/deploy/README.md b/deploy/README.md index cbbe37a..d5956d2 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -32,6 +32,28 @@ receipt covers exactly one image digest, never a tag, and a suite recorded as `skipped` blocks admission just as a failure does — a partial run must never read as a full one. `just admission-contract` proves those refusals offline. +**Do not run `just admit run` as the deploy gate.** It is fine for debugging +one suite, but it is invisible to the concurrency limit that keeps two gates +off the same machine — which we learned by running two at once and throttling +both (see `../docs/deploying.md`). The sanctioned entry point is the +`stream-admission` Prefect deployment in the sibling repo +[`zzstoatzz.io/my-prefect-server`](https://tangled.org/zzstoatzz.io/my-prefect-server), +which runs the identical `scripts/admit run` on bare metal (heavypad) with +`concurrency_limit: 1`. `.tangled/workflows/admission.yml` only triggers it. + +Concretely, this repo's deploy path depends on that repo for: + +| what | where | +|---|---| +| the gate runner (flow + deployment) | `my-prefect-server: flows/stream_admission.py`, `prefect.yaml` | +| the machine it runs on | heavypad, via Prefect's `home-pool` (a *process* pool) | +| serialisation | the deployment's `concurrency_limit: 1` (ENQUEUE) | +| toolchain on that box | zig, go, just, uv, docker — go/just live in the shared nix profile | + +If that deployment is gone or its worker is offline, pushes will not produce +receipts and nothing will be deployable. That is deliberate: no receipt, no +deploy. + ## required environment (docs/lessons-from-zlay.md #8, #9) - `MALLOC_ARENA_MAX=4` — the highest-leverage glibc RSS knob for thread-heavy services (per-thread arena fragmentation) diff --git a/docs/deploying.md b/docs/deploying.md index 72945d8..2d49149 100644 --- a/docs/deploying.md +++ b/docs/deploying.md @@ -9,31 +9,52 @@ The instance is one Hetzner box (stream-cx43, 89.167.122.160, hel1) running docker compose at `/opt/stream-experiment/`. The stream image is pinned in `/opt/stream-experiment/.env` as `STREAM_IMAGE=atcr.io/zat.dev/stream:`. +## where the gate runs (this repo is not self-contained) + +Admission runs on **heavypad**, as the `stream-admission` Prefect deployment +defined in the sibling repo +[`zzstoatzz.io/my-prefect-server`](https://tangled.org/zzstoatzz.io/my-prefect-server) +(`flows/stream_admission.py`, `prefect.yaml`). Prefect's `home-pool` is a +*process* pool, so a flow run there is a bare-metal run on a 32-core box — no +VM tax, no per-run guest setup. + +**That deployment is the only sanctioned entry point**, because it is the only +one carrying `concurrency_limit: 1`. On 2026-08-09 two gates ran on the same +box at once (a spindle microVM guest and a bare-metal run) and crossed the +worker cgroup's `memory.high` 534,646 times, silently throttling both. Running +`./scripts/admit run` by hand still works and is fine for debugging a single +suite, but it is invisible to that limit — do not use it as the deploy gate. + +`.tangled/workflows/admission.yml` therefore only *triggers* the deployment +(`POST /deployments//create_flow_run` — the endpoint that sets +`Scheduled`; `POST /flow_runs/` would create an inert `Pending` run). It needs +the `PREFECT_API_URL` and `PREFECT_API_AUTH_STRING` repo secrets. + ## the pipeline, in order -| stage | command | measured cost | +| stage | how | notes | |---|---|---| -| 1. land the change | commit + push to main (CI: fmt, debug+releasesafe tests) | suite floor ~32 min in CI, but admission below is the deploy gate | -| 2. admission | `STREAM_BUILD_HOST=ssh://root@89.167.122.160 ./scripts/admit run` | ~40 min total: ~5–8 min native amd64 build on the box + ~32 min gate suite | -| 3. publish | `./scripts/admit publish [sha]` | ~1–2 min (push + pin registry digest into `receipts/.json`) | -| 4. cut over | on the box: edit `STREAM_IMAGE` in `/opt/stream-experiment/.env`, then `docker compose up -d stream` | seconds to swap | +| 1. land the change | commit + push to main | `ci.yml` runs fmt + debug/releasesafe tests; admission below is the deploy gate | +| 2. admission | push triggers the `stream-admission` deployment, or trigger it by hand with `sha` + `build_image=true` | bare metal on heavypad; serialised by concurrency_limit=1 | +| 3. publish (optional) | `./scripts/admit publish [sha]` | pins the registry digest into the receipt. Skippable: the registry has been over quota, and deployment verifies by image id instead | +| 4. cut over | flip `STREAM_IMAGE` in `/opt/stream-experiment/.env`, `docker compose up -d stream` | seconds to swap | | 5. restart recovery | (automatic) | ~4–5 min to serving + dialing upstream; see below | -Total wall clock from "fix committed" to "new build serving": **~45 minutes.** - Notes on the stages: - **Admission is the gate, not CI.** A receipt (`receipts/.json`) binds one full-suite run to one image digest; `scripts/admit verify ` refuses anything without a fully-passing receipt. Never point `.env` at an image that `admit verify` rejects. -- **Always build with `STREAM_BUILD_HOST`** (native amd64 on the box). - Local emulated builds on an arm64 workstation take 40+ minutes and produce - a cross-emulated approximation. The build ships the context over ssh, builds - remotely, and streams the image back so registry credentials never leave - the workstation. +- **The registry is not on the critical path.** `22bf62c` shipped while atcr + was over quota by verifying the receipt's recorded `local_id` against the + image id on the box. `docker save | docker load` preserves image ids, so a + gate that built on heavypad can still be deployed by transferring it. - **Uncommitted trees are refused** (`require_clean_tree`) — the image tag is the commit, by construction. +- **Partial runs cannot deploy.** `only`/`skip` record every unrun suite as + `skipped`, and `verify` treats `skipped` exactly like a failure; + `build_image=false` writes no receipt at all. ## what to expect at restart @@ -51,14 +72,12 @@ Notes on the stages: fallback host. - The live tail replays the full gap on reconnect; the archive serves throughout (the ~30s of docker swap excepted). -- **Bounce cursor-consumers that reconnected during the dial window.** Until - the cold→hot seam handoff is fixed (docs/upstream-subscribe-protocol.md, - known-open), any subscriber that connects with a cursor while the hot tail - is still empty (between serving-start and ingest dial, ~4 min) wedges on - the cold path indefinitely — delivery in read-batch bursts at seal cadence. - It looks like "jerky, 0↔40/s". `jetstream_subscribe_cold_reads_total` - ticking steadily after catch-up confirms it; restarting the consumer heals - it (2026-08-08: coral + bufo-bot, twice in one day). +- **No consumer bounce is needed any more.** The cold→hot seam wedge that + made cursor-subscribers deliver in bursts ("jerky, 0↔40/s") was fixed in + `fca6cc7`; the `22bf62c` cutover confirmed it — coral ran 12 clean samples + at 20–42/s with `jetstream_subscribe_cold_reads_total` frozen, and no + consumer was restarted. If burstiness returns, that counter climbing after + catch-up is the tell, and it is a regression, not an expected cost. ## post-deploy verification (do all four)