running and shipping software: what breaks after deploy, how to find out, how to size what you rent, and why the deploy path itself is the least-tested code you own.

design patterns live in systems/. the split is roughly "how to build it" against "what it does to you at 2am".

a recurring theme across these notes: the failures that stay open longest do not produce errors. they produce silence, a green dashboard, or a confidently wrong belief.

topics #

observability and failure #

masked-failure — a reliability feature moves failure from loud and small to silent and large; watchdogs, heartbeats, staleness
health-checks — 200 means "replacing me would not help"; heartbeat of work, verdict at read time, supervisor decides what a failure does (fly checks route, they do not restart), kill switch first
observability-of-absence — absent is not zero, a disabled invariant is an open outage, and the signals you must add deliberately because nothing emits them

diagnosis #

throughput-bottlenecks — bound from outside, profile the real workload, read upstream's source; one variable per deploy
bounded-scans — a bound limits work performed, not work read; push the skip check to the cheapest tier, and guarantee the cursor moves
ci-wall-clock — elapsed time to the last check is not summed job time; optimize the job that finishes last, per attempt

configuration and policy #

tangled — route repository metadata, Git objects, CI, and collaboration records to the component that owns them; Bobbin, knots, spindles, PDS records, and the interactive log-viewer trap
reconcilers-own-config — if a reconciler owns a value, a hand edit is temporary; change the source, not the reconciled object
policy-enforced-by-accident — a rule enforced by a dependency's implementation detail has no owner; enforce at a layer you own, and test the policy rather than the mechanism

what you rent #

deploy-path — untested deploy branches are bug reservoirs; use the repository's recipe, verify the launcher ran your command
instance-sizing — measure the working set at a fresh start; start small because disk rescale is one-way; read the whole bill
fly-custom-domains — one app per hostname, cloudflare universal SSL fights ACME, wildcards are one label deep, the resolver remembers a miss
habitat-pear — self-hosting habitat's organizational data server: keys on the volume, hive domain, DNS per level, org bring-up, running a fork
dotenv-autoload — dagster and vite load env files on their own; load yours with override, keep the default file production's
home-infra/ — the home-box cluster: cost optimization, public cost declaration, tailscale