the deploy path is production code #

application code accumulates tests, staged rollouts, and review. the code that ships it — Dockerfiles, deploy scripts, machine launchers, credential plumbing — usually has none of that, and several of its branches have never executed at all. latent defects concentrate exactly where execution never goes, so a change gated by a full test suite can still fail five times in a row without a single failure being in the change.

first execution is the test #

any branch of the deploy path that has never run is a bug reservoir: the remote-build path, the credential fallback, the restore flow, the rollback. run each one deliberately, off the critical path, before you need it during an incident. "the code has been there for weeks" is not evidence it works; it is the opposite, because weeks of non-execution means weeks of unnoticed drift around it.

the failures found this way are mundane and independent of each other, which is why they arrive in sequence rather than all at once:

a build context with no .dockerignore, shipping a build cache that had grown for weeks
a cross-architecture intermediate layer poisoning the cache for the target architecture
a build host whose container network has no DNS resolution
a deploy target that was never given registry credentials, discovered only because a "restart" silently kept running the old image

verify the launcher ran your command #

a launcher returning success verifies the launcher, not the job. two ways this goes wrong:

argument position matters and is silently forgiven. a machine launcher that takes the command positionally after the image will accept an invented --command flag, swallow it, and run the image's default entrypoint — often exiting 0 in about a second. the tell was in the boot log the whole time: Preparing to run: python3. read the launcher's own record of what it executed before reporting a job as running.
a pipe eats the exit code. piping a launcher through tail or head reports the pipeline's last command's status. use set -o pipefail, or do not pipe anything whose exit status you act on.

use the repository's recipe #

a justfile or Makefile deploy target encodes trap knowledge: cross-compile targets, flag order, guard checks. invoking the underlying tool directly bypasses precisely the guard someone paid to write down.

the concrete trap: a Dockerfile that copies a prebuilt binary from the build context will happily ship a native arm64 binary to an x86 host when built on an ARM laptop. the repository's recipe guarded against this. running the raw tool did not.

corollary for config: pushing configuration is a restart. fly secrets set and its equivalents restart machines to apply environment changes, which destroys any in-memory state the process was carrying. know what a restart costs before pushing config.

cycle-time variance is the health signal #

the same pipeline completing in seventeen minutes or six hours, depending on invisible cache state, is the symptom. the mean says nothing; the spread says the path has uncontrolled inputs. treat a widening distribution as a defect report against the deploy path itself.

reconcilers-own-config — the deploy script as the owner of a value you tried to change by hand
policy-enforced-by-accident
observability-of-absence

sources #

one night, 2026-08-06/07, in which a two-line fix proven twice by the suite took five attempts to reach production, every failure in the deploy path
follow-on launcher and secrets traps from the same week (2026-08-08)