# Autopush: how a push to main reaches production

A push to `main` runs `.tangled/workflows/deploy-microvm.yml`, which runs the
test suite, builds the amd64 image inside the microVM, and pushes it to
`ghcr.io/nmokkenstorm/crate` as both `:<short-sha>` and `:latest`. On the
droplet, `crate-pull.timer` polls once a minute; when the digest behind
`:latest` changes it pulls and recreates the `app` container. Tests gate the
push, so a red build never becomes an image.

Nothing SSHes into production to deploy, and production holds no CI
credentials. The pull unit's `ExecStartPre` runs `deploy/sync-caddy.sh`
(installed on the droplet at `/opt/crate/deploy/sync-caddy.sh`), which
extracts the versioned Caddyfile from the pulled app image and recreates
Caddy only when that file changed, before replacing the app container. It's
a real script file rather than an inline `sh -c '...'` on the unit's
`Exec` line because systemd expands `$VAR` on those lines itself before the
shell sees them, which would silently break the script's own `$(...)` and
`"$image"` use. `deploy/publish.sh` does the same two steps by hand for
break-glass: build and push from your machine, then start the timer's unit
immediately instead of waiting for the next poll.

## Repo secrets (Settings → Secrets)

| Secret           | Holds                                                                                                                                                                                                    |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `REGISTRY_USER`  | GitHub username/org for ghcr.io, e.g. `nmokkenstorm`.                                                                                                                                                    |
| `REGISTRY_TOKEN` | GitHub PAT with `write:packages`, plus `read:packages` if the package stays private. Scope a fine-grained token to this one package. The hosted spindle is shared, so scope it as narrowly as it allows. |

## One-time droplet setup

```sh
deploy/setup-autopull.sh
```

Installs `crate-pull.{service,timer}` into `/etc/systemd/system` and
`sync-caddy.sh` into `/opt/crate/deploy/`, enables the timer, installs the
initial Caddyfile, points `.env` at the `latest` tag, and prints the
schedule. Subsequent image pulls extract the versioned Caddyfile, validate
it in the Caddy image, and force-recreate Caddy only when it changes.
Invalid configuration leaves the running proxy untouched. Idempotent.

If the ghcr package is private (GitHub creates new packages private, and the
first CI push is what creates this one), the droplet needs read credentials
first or every pull 403s:

```sh
ssh root@<droplet> 'docker login ghcr.io -u nmokkenstorm'
```

Making the package public removes that step and the stored credential. The
image holds compiled application code; the secrets it runs on come from `.env`
at runtime and are not baked in.

## Rolling back

`CRATE_TAG` in the droplet's `.env` is what the timer re-resolves on each
poll, so it must be `latest` for autopush to move at all. Setting it to a sha
and running `docker compose up -d` does double duty: it pins that image _and_
freezes the timer on it, since an immutable tag never resolves to anything new.
Set it back to `latest` to resume.

## The emulation constraint, and why it no longer shapes any of this

The droplet is x86_64 and runs a `linux/amd64` image, so that is what has to be
produced. Until 2026-08-03 the belief was that only an amd64 machine could
produce it, which is why deploys used to ship a source tarball to the droplet
and build there.

The real constraint is narrower. BeamAsm dual-maps its JIT code, one executable
and one writable alias of the same pages, and user-mode emulators cannot track
writes through the alias, so the BEAM crashes before it boots
([QEMU#1034](https://gitlab.com/qemu-project/qemu/-/issues/1034)).
[erlang/otp#10355](https://github.com/erlang/otp/issues/10355) is that same
failure reported against OTP 28 and closed as an unsupported configuration, the
BeamAsm author supplying `+JMsingle true` as the answer. QEMU 8.1 fixed the
aarch64-guest direction only; amd64-on-arm64 is the unfixed one and the
maintainers have said it stays that way. Rosetta has the same hole, so Docker
Desktop and OrbStack do not rescue it either.

The Dockerfile's build stage therefore sets `ERL_FLAGS="+JMsingle true"`, and
cross-building amd64 from an arm Mac works. Verified 2026-08-03: without it,
`docker run --platform linux/amd64 erlang:29-slim erl` dies with
`failed_to_start_child,user,nouser`; with it, it boots.

Two consequences worth keeping in mind:

- CI is expected to be amd64 anyway (Tangled's hosted Nixery serves
  `architecture: amd64`, and the microVM engine uses KVM so guest arch must
  match host), in which case nothing emulates and the flag costs nothing.
  That is inference, not documentation, which is why the workflow's first step
  prints `uname -m`, `nproc` and whether `/dev/kvm` exists.
- If CI ever cannot produce an amd64 image, the fallback is `publish.sh` from
  any machine, not a rebuild on the droplet. The old SSH-triggered path
  (a `deploy` service account, a forced-command key, and a source tarball
  streamed into production) existed only because local cross-builds were
  believed impossible, and was deleted once they weren't.