From 1b84e4710233cdc7f41d3079f37fb92f7d19995f Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Thu, 30 Jul 2026 04:52:25 -0500 Subject: [PATCH] docs: runbook named the wrong deployed artifact, and never defined the gate The "admitted artifact for this run" block named eb95c14 / sha256:4300104..., but experiment 6 is running 29705cb / sha256:0f45468c.... Verified before changing it: 29705cb's receipt is a clean 20/20 with every suite 'pass', its registry_digest matches the running container exactly, and it was cut 04:36 UTC against an 05:01 UTC start. So admission held -- the runbook just was not updated when the image moved. Now says how to check rather than asking the reader to trust the paragraph, since this is what an operator reads mid-incident. Also defines the three terms the file leans on from line 6 onward without ever explaining: "the gate" (semantic-parity.md, proven-equivalence record, not just passing tests), "a receipt" / "admitted artifact" (JSON binding a suite run to an image digest, where skipped blocks admission like a failure), and "cutover" (lifecycle leaves merge, serving stops 503ing). Co-Authored-By: Claude Opus 5 (1M context) --- docs/deployment-runbook.md | 36 +++++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-) diff --git a/docs/deployment-runbook.md b/docs/deployment-runbook.md index 35c23f5..30c2e11 100644 --- a/docs/deployment-runbook.md +++ b/docs/deployment-runbook.md @@ -6,6 +6,21 @@ modes are the thing this plan is shaped around rather than a list of hopes. `docs/semantic-parity.md` is the gate; this file is what to actually do, in order, and what to do when a stage goes wrong. +Three terms used throughout, since they are load-bearing here and opaque cold: + +- **the gate** — `semantic-parity.md`, the row-by-row record of which + behaviours are *proven* equivalent to upstream Jetstream V2. Nothing deploys + on the strength of a passing test alone; the gate is what says a surface is + actually settled. +- **a receipt** — a JSON file under `receipts/` binding one run of the full + suite to one **image digest** (`receipts/.json`). It records each suite's + verdict, and a `skipped` verdict blocks admission exactly like a failure — + the point is that nothing untested rides along. An **admitted artifact** is an + image whose digest has such a receipt with everything passing. +- **cutover** — the moment the lifecycle leaves merge for steady state and + serving stops returning 503. Before it, the archive is still being assembled; + after it, clients are being answered. + The rule underneath every stage: **a stage that cannot be measured has not succeeded.** All three previous runs ended because something was true that the running system could not tell us. @@ -95,11 +110,22 @@ than trusting: | Scoped to a target nobody had measured | Stage 0b feasibility check, with the head-of-list trap named | | Experiment 3's tuning deployed unnoticed | `assert_deploy_args.py` refuses backfill flags no receipt covers | -**Admitted artifact for this run:** -`sha256:4300104135c6fd35bd13a264b744c5f842c41ee9a51e0cf1c57c9a59640551a2` -(`receipts/eb95c14.json`, 20/20 suites, native linux/amd64). - -It carries one fix beyond the previous artifact, and it is the kind this +**Admitted artifact actually deployed for experiment 6:** +`sha256:0f45468c2fdba928fb3a9830f79d4693960a9c0ae2174ae042ecb01dfd33e0cd` +(`receipts/29705cb.json`, 20/20 suites all `pass`, receipt cut 2026-07-28 +04:36 UTC against an 05:01 UTC start, tag `atcr.io/zat.dev/stream:29705cb`). + +This superseded `eb95c14` +(`sha256:4300104135c6fd35bd13a264b744c5f842c41ee9a51e0cf1c57c9a59640551a2`), +which earlier revisions of this file named as the artifact for the run. Both +were properly admitted; the runbook simply was not updated when the image moved. +**Check the receipt against the running digest rather than trusting this +paragraph** — `docker inspect` the workload container and compare to +`receipts/.json`'s `image.registry_digest`. This block is the thing an +operator reads mid-incident to learn what is deployed, so a stale digest here is +worse than no digest. + +The `eb95c14` fix below is carried forward into `29705cb`. It is the kind this runbook exists for: three sites reported `OutOfMemory` as a *structural* verdict about a repository's CAR, and the engine retires a repository permanently on that verdict. Under whole-network memory pressure — experiment 3's 24.3 GB — -- 2.51.2