diff --git a/docs/deployment-runbook.md b/docs/deployment-runbook.md index 35c23f5..30c2e11 100644 --- a/docs/deployment-runbook.md +++ b/docs/deployment-runbook.md @@ -6,6 +6,21 @@ modes are the thing this plan is shaped around rather than a list of hopes. `docs/semantic-parity.md` is the gate; this file is what to actually do, in order, and what to do when a stage goes wrong. +Three terms used throughout, since they are load-bearing here and opaque cold: + +- **the gate** — `semantic-parity.md`, the row-by-row record of which + behaviours are *proven* equivalent to upstream Jetstream V2. Nothing deploys + on the strength of a passing test alone; the gate is what says a surface is + actually settled. +- **a receipt** — a JSON file under `receipts/` binding one run of the full + suite to one **image digest** (`receipts/.json`). It records each suite's + verdict, and a `skipped` verdict blocks admission exactly like a failure — + the point is that nothing untested rides along. An **admitted artifact** is an + image whose digest has such a receipt with everything passing. +- **cutover** — the moment the lifecycle leaves merge for steady state and + serving stops returning 503. Before it, the archive is still being assembled; + after it, clients are being answered. + The rule underneath every stage: **a stage that cannot be measured has not succeeded.** All three previous runs ended because something was true that the running system could not tell us. @@ -95,11 +110,22 @@ than trusting: | Scoped to a target nobody had measured | Stage 0b feasibility check, with the head-of-list trap named | | Experiment 3's tuning deployed unnoticed | `assert_deploy_args.py` refuses backfill flags no receipt covers | -**Admitted artifact for this run:** -`sha256:4300104135c6fd35bd13a264b744c5f842c41ee9a51e0cf1c57c9a59640551a2` -(`receipts/eb95c14.json`, 20/20 suites, native linux/amd64). - -It carries one fix beyond the previous artifact, and it is the kind this +**Admitted artifact actually deployed for experiment 6:** +`sha256:0f45468c2fdba928fb3a9830f79d4693960a9c0ae2174ae042ecb01dfd33e0cd` +(`receipts/29705cb.json`, 20/20 suites all `pass`, receipt cut 2026-07-28 +04:36 UTC against an 05:01 UTC start, tag `atcr.io/zat.dev/stream:29705cb`). + +This superseded `eb95c14` +(`sha256:4300104135c6fd35bd13a264b744c5f842c41ee9a51e0cf1c57c9a59640551a2`), +which earlier revisions of this file named as the artifact for the run. Both +were properly admitted; the runbook simply was not updated when the image moved. +**Check the receipt against the running digest rather than trusting this +paragraph** — `docker inspect` the workload container and compare to +`receipts/.json`'s `image.registry_digest`. This block is the thing an +operator reads mid-incident to learn what is deployed, so a stale digest here is +worse than no digest. + +The `eb95c14` fix below is carried forward into `29705cb`. It is the kind this runbook exists for: three sites reported `OutOfMemory` as a *structural* verdict about a repository's CAR, and the engine retires a repository permanently on that verdict. Under whole-network memory pressure — experiment 3's 24.3 GB —