# Tag-triggered releases `git tag v1.2.3 && git push --tags` builds every image at version `1.2.3`, rolls the fleet onto them, and verifies it actually moved. If anything fails the CI pipeline goes red. ## How it flows ``` tag v1.2.3 pushed └─ .tangled/workflows/release.yml (spindle, microvm) ├─ reject anything that is not vMAJOR.MINOR.PATCH ├─ reject if the commit has not reached the build mirror └─ POST /execute/RunAction → Komodo └─ Action `wisp-release` (deploy/komodo/wisp-release.ts) ├─ find every build that clones this repo ├─ pin each to 1.2.3 @ , build them one at a time ├─ find every stack running one of those images ├─ set WISP_TAG on each stack, deploy └─ verify each service by tag, liveness and uptime ``` Spindle does not build. It gets a fresh overlay disk per run, so building there means a cold build of everything on every tag, plus emulation for any architecture the runner does not share. The build host already has the warm cache and the emulation set up; CI just drives it. ## Observability configuration `observability/` records the restored monitoring stack, an application logging recovery overlay, and focused operational alerts. See its [runbook](observability/README.md). Application topology remains in Komodo. Existing host Compose log settings persist through releases; `verify-logging.py` checks a private rendered config for drift. ## No application topology in this repo, on purpose Neither Action names a host, a directory, an env file or a service. They ask Komodo: - **builds** — every build whose configured source repo is `SOURCE_REPO` - **images** — each build's registry, organization and image name - **stacks** — every stack whose deployed services run one of those images - **paths** — each stack's own run directory, compose files and env files So adding a node, moving a service or renaming a stack needs no change here. The only thing the pipeline is told is which repo it is releasing, set once as `SOURCE_REPO` in the workflow. The consequence worth knowing: **membership is defined by what is running.** A stack joins the release the moment it runs one of our images, and leaves when it stops. That is usually what you want, and it is worth a glance at the Action's log — it prints every stack it selected — the first few times. ## Setting it up 1. **Repo secrets** on tangled (Settings → Secrets, *not* the workflow's `environment:` block, which is public to anyone who can read the repo): `KOMODO_ADDRESS`, `KOMODO_API_KEY`, `KOMODO_API_SECRET`. `KOMODO_ADDRESS` takes no trailing slash — the workflow appends `/execute/RunAction` and a double slash 404s. 2. **Create two Komodo Actions**, pasting the file contents from here: | Action | File | |---|---| | `wisp-release` | `deploy/komodo/wisp-release.ts` | | `wisp-prepare-tag` | `deploy/komodo/wisp-prepare-tag.ts` | 3. **Run `wisp-prepare-tag` once**, by hand, with `{"repo": ""}`. It changes nothing without `apply` — it renders the edit beside the real compose files, diffs it, and discards the previews. Read that diff, then re-run with `{"apply": true}`. It rewrites the `image:` lines that point at our own registry so they read `${WISP_TAG}`, and seeds that variable at each stack's current tag so the first deploy is a no-op. 4. **Check the workflow timeout** on the spindle exceeds a full release. The stock default is 5 minutes, far too short — though the existing flake-check and test workflows already prove it is raised here. 5. **Cut a throwaway tag** and watch it. If nothing fires at all, the spindle may predate the `tag:` trigger field. ## Registry Images live in the registry on valefar, `valefar.mesh.wisp.place:5000` (`registry:2`, plain HTTP, no auth, deliberately), reached over the WireGuard mesh. It is a plain `docker run` container on valefar (not in its Nix config), with data in `/var/lib/registry`, published only on the LAN (`10.0.0.30`), tailnet (`100.64.0.11`) and mesh (`10.88.0.10`) addresses. valefar has `net.ipv4.ip_nonlocal_bind=1` and docker waits for `wg-quick-wg0`, so those binds survive a reboot. This replaced atcr.io on 2026-10-01, after atcr served just-pushed tags before their blobs were readable and a release failed halfway through the fleet. Because it is plain HTTP, every host that pulls or pushes has to opt in: - each node's `/etc/docker/daemon.json` lists it in `insecure-registries`. That setting is reloadable, so apply it with `systemctl reload docker`, which restarts no containers. - the `wisp-multiarch` buildx builder on stolas is created with `--buildkitd-config /etc/buildkit/buildkitd.toml`, which marks the registry `http = true`. Recreate it with `docker buildx rm --keep-state` so the build cache survives. Komodo will not push to a registry without an account, so Core holds a placeholder registry account (`wisp`, dummy token). `docker login` against an unauthenticated `registry:2` succeeds with any credentials. Komodo puts the account in the image path, so images are `valefar.mesh.wisp.place:5000/wisp/`, and compose `image:` lines name `valefar.mesh.wisp.place:5000/wisp/:${WISP_TAG}`. ## Traps this is shaped around **Registry tokens here are short lived and expire as 403, not 401.** This applied to atcr.io, the previous registry, and may matter again if an authenticated registry comes back. Docker only re-authenticates on a 401, so an expired token is fatal rather than transparent. Builds run **sequentially** for this reason: parallel builds contending for one builder stretch each login→push window past the token lifetime. A push failure gets one retry against the now-warm cache, which is the fix — not a config change. Builds configured to push many tags at once (latest, commit, each semver level) spend longer in that window; fewer tags is the cheapest way to shrink the risk. **A successful deploy call is not evidence of a deploy.** Komodo has been seen reporting success with only some of a stack's images actually pulled. So the Action trusts none of its own return values and asks the containers: right tag, running, and *recreated within the last 15 minutes*. A container on the correct tag that was never restarted is precisely the failure mode. **Image ids and digests are never compared across hosts.** A mixed-arch fleet resolves the same multi-arch tag to a different per-arch manifest on each host, so digest equality would fail on a perfectly healthy fleet. **Releases never rewrite compose.** Compose files are hand-maintained and carry per-host detail the release knows nothing about — including which credential a given service reads through. The release moves one variable, through the stack environment Komodo already owns. `wisp-prepare-tag` is the only thing that ever edits compose, it runs once, and it gates on a render diff. **That render diff never prints values.** A rendered compose file contains credentials. Comparing the render is what catches a service being wired to the wrong one; printing it is what leaks one. Differences outside `image:` lines are reported by key only, and abort. **Komodo's execute calls do not wait, and their `success` field lies.** An execute returns the Update in its initial state — `success: true, status: InProgress` — before the work has begun. Gating on it makes a whole release "succeed" in ten seconds having built and deployed nothing. Inside an Action use the client's own `execute_and_poll`; from CI, capture the update id and poll `read/GetUpdate` until `status: Complete`, which is what the workflow does. **The Action pulls before it deploys.** Compose would now do this itself — the `pull_policy: never` lines that made it skip the fetch and then die on a bare "No such image" were removed on 2026-09-02. The explicit pull stays because it is the better failure mode: a registry or credential problem surfaces as a registry error, on the host that has it, before anything is torn down. **Komodo's reported service image is rendered, not observed.** When it renders without the stack's env file the tag interpolates to nothing and comes back as `:0`, so a healthy service reads as a failed deploy. Verification asks the container (`InspectStackContainer`) instead. **Giving up on waiting is not cancelling.** If the poll loop hits its deadline the release continues server-side. Treat it as *unknown* and read the Update before re-running, because a blind retry redeploys. **`TANGLED_SHA` on a tag push is the ref's sha, not the commit's.** For an annotated tag (`git tag -a`) the ref is a tag object, so pinning a build to it finds nothing and the mirror check 422s. The workflow resolves `HEAD^{commit}` from the clone instead, which is right for annotated and lightweight tags alike. **Tag pushes do not run `test.yml`** — it triggers on `branch: [main]`, and a tag push has no branch. Spindle has no cross-workflow `needs:`, so a tag is assumed to point at a commit that already went green on main. ## Rollback Not a new tag. Set `WISP_TAG` back on the affected stacks and redeploy — the Action is idempotent, so re-running a previous version is safe. Every compose file `wisp-prepare-tag` touches is backed up alongside it as `.bak-wisptag-`. ## Type-checking the Actions `./deploy/typecheck.sh` checks both Actions against Komodo's real client types. An Action is not a module — Komodo pastes its contents inside `async function main()` and supplies the globals — so the script wraps each file the same way before checking it, and `komodo-globals.d.ts` declares what Komodo injects. Subtract 2 from a reported line to map it back. The types under `komodo/vendor/` come from a running Komodo and are not committed. `typecheck.sh` runs `./deploy/fetch-komodo-types.sh` when they are missing, which needs `KOMODO_ADDRESS` (the base url, no trailing slash): ```sh KOMODO_ADDRESS=https://komodo.example ./deploy/typecheck.sh ``` Delete `komodo/vendor/` after a Komodo upgrade to re-fetch. This is worth keeping honest: typing these against the real client is what surfaced `execute_and_poll` (which replaced a hand-rolled poller), an Update id read as `.id` when it is `_id.$oid` (so every failure message printed `undefined`), and several optional fields being read as if they were guaranteed. ## Open - Sync the Actions via ResourceSync TOML so changes to these files land as PRs instead of a UI paste. - Verification has no error-log gate. A bad release that starts cleanly and only then floods the logs still passes; a post-deploy scan belongs at the end of `wisp-release`.