diff --git a/docs/deployment-runbook.md b/docs/deployment-runbook.md index 48e5db4..35c23f5 100644 --- a/docs/deployment-runbook.md +++ b/docs/deployment-runbook.md @@ -230,6 +230,40 @@ Do not scale workers to chase throughput: experiment 3 went 100 → 200 and got slower. If throughput disappoints, record it and analyse afterwards — the experiment's job is evidence, not a record. +### Stage 4b — if the run succeeds and should keep running + +The watchdog is a cost guard, not a judge: at the deadline it deletes every +server, volume, firewall, network and IP in the project **regardless of whether +the experiment worked**. A successful whole-network archive is deleted on +schedule unless someone intervenes. Decide before the deadline, not after. + +**Protect the archive volume as soon as the run looks like succeeding.** + +``` +hcloud volume enable-protection delete +``` + +Hetzner Cloud has no volume snapshots — protection is the mechanism. The +watchdog's `DELETE /volumes/` then fails, the servers are still destroyed +so compute billing stops, and the archive survives to be reattached to a fresh +box. Two consequences to expect: + +- the watchdog's `drain_kind volumes` loop retries until it gives up, so its + systemd unit exits non-zero. That is the protection working, not a fault. +- **`destroy.sh` will also fail on a protected volume.** Disabling protection + is a deliberate step in tearing the archive down; that is the point. + +Retention is not free: a 2.5 TB volume bills whether attached or not. Keeping +the archive is a standing cost, so it is a decision, not a default. + +**Promoting the experiment to a standing instance** is a separate piece of work +from keeping its data. The experiment shape is deliberately temporary — sslip.io +hostnames derived from the workload IP, a public Grafana share, a watchdog +holding a hard deadline. A standing instance wants real DNS, a deadline that is +not hours away, and a decision about whether the operator surfaces stay public. +None of that is required to preserve the archive, and none of it should be done +in a hurry at hour 71. + ### Stage 5 — teardown Server, volume, primary IPs, firewall, observer and watchdog all deleted, then