zat.dev / zds
atproto pds in zig pds.zat.dev
pds atproto
Star
Fork 1
Watch
zds docs deployment.md
6.3 kB
Markdown
at fix/default-appview-proxy

production deployment #

This runbook describes the instance used to exercise ZDS with real resident accounts and interoperability clients.

production shape #

setting value
public origin https://pds.zat.dev
Fly app zds-pds
primary region ord
machine one shared CPU, 1 GiB RAM
persistent data Fly volume zds_data mounted at /data
database /data/zds.sqlite3
blobstore /data/blobs
hosted handles *.pds.zat.dev
account policy invite required
permissioned data enabled, experimental

The committed non-secret configuration is in fly.toml. Secrets are Fly secrets and must not be copied into source, logs, or issue reports. The repository .env is ignored local operator state; commands that need one of its values should source it without printing it.

SQLite and blob bytes live on the mounted volume. A new application image must reuse that volume. Never create a replacement machine with an empty /data and send production traffic to it as a deployment shortcut.

normal deployment #

Every push to main runs .tangled/workflows/ci.yml, which gates deployment on formatting, unit tests, both HTTP smoke suites, and the ARMv6 ReleaseSafe build. After those checks pass, it executes:

flyctl deploy --remote-only

The workflow owns FLY_API_TOKEN. A normal release is therefore:

  1. Confirm the worktree is the canonical main checkout and is based on origin/main.
  2. Run the required checks from the repository root.
  3. Commit and push main.
  4. Watch the Tangled workflow and verify the live service after Fly replaces the machine.

Required checks:

just test
just smoke-all
zig fmt --check build.zig build.zig.zon src bench tools
git diff --check
zig zen

Also run just smoke-permissioned for permissioned-data storage, OAuth space scope, com.atproto.space.*, com.atproto.simplespace.*, or /account/spaces changes. Run the relevant just bench ... lane for performance-sensitive storage, repo, blob, proxy, or permissioned-data work.

manual deployment #

The manual path is for an operator already authenticated to the Fly app. Push the release first. If no pipeline starts, verify that absence on the spindle, run the same checks locally (including the ARMv6 command below), and deploy a clean archive of the pushed revision so local .env and workspace files are not uploaded to the remote builder:

flyctl auth whoami
release_src=$(mktemp -d)
git archive HEAD | tar -x -C "$release_src"
(cd "$release_src" && flyctl deploy --remote-only --app zds-pds)

Do not create a second configuration path in shell history. fly.toml remains the source of non-secret deployment configuration, and Fly secrets remain the source of secret configuration.

October 1 deployment observation #

The v0.4.0 push (8d0deeb2cf5c) reached the knot, but no matching spindle pipeline appeared. tg pipeline status zat.dev/zds failed resolving the owner repo record; the spindle still reported f68da7c as its latest run. The release used the manual path after all local CI-equivalent checks passed. Restoring the push trigger remains an operations follow-up; a successful Git push alone must not be treated as a successful deployment.

For this repo, query the spindle directly if the CLI lookup fails:

curl -fsSG https://spindle.tangled.sh/xrpc/sh.tangled.ci.queryPipelines \
  --data-urlencode 'repo=did:plc:6atzzmsifkfmfqmeqrl6jwzo' \
  --data-urlencode "commits=$(git rev-parse HEAD)" \
  --data-urlencode 'limit=5'

verification #

Capture the current health response before deploying so there is a baseline:

curl -fsS https://pds.zat.dev/xrpc/_health | jq .
curl -fsS https://pds.zat.dev/xrpc/com.atproto.server.describeServer |
  jq '{did,availableUserDomains,inviteCodeRequired}'

After deployment:

flyctl status --app zds-pds
flyctl logs --app zds-pds --no-tail
curl -fsS https://pds.zat.dev/xrpc/_health | jq .
curl -fsS https://pds.zat.dev/xrpc/com.atproto.server.describeServer |
  jq '{did,availableUserDomains,inviteCodeRequired}'
curl -fsS https://pds.zat.dev/api/openapi.json >/dev/null
curl -fsS https://pds.zat.dev/account >/dev/null

Do not stop at an HTTP 200 from the health route when the change touched a resident workflow. Exercise the affected path with a throwaway account or the appropriate smoke client, then inspect recent logs for panics, repeated 5xx responses, migration failures, proxy timeouts, or memory pressure. Never print passwords, admin tokens, OAuth tokens, PLC keys, or mail credentials while probing.

rollback and recovery #

Application rollback and data rollback are different operations. Re-deploying an older image does not undo a SQLite migration, and restoring SQLite without its corresponding blobstore can produce an incoherent account.

Before a risky storage migration, confirm that a current volume snapshot exists and document the exact application revision and schema transition. If a release fails without changing persistent state, deploy the last known-good source revision through the same Fly path. If persistent state changed, stop and assess the migration and volume snapshot together before rolling either side back.

For ordinary account operations after a healthy deployment, use the operator guide, invite-code guide, and account takedown runbook.

32-bit and small boards #

zds builds and runs on 32-bit ARM without native 64-bit atomics (ARMv6, e.g. a Raspberry Pi 1 with 512 MB). Cross-compile; do not build on the board — zig 0.16's build runner does not compile for 32-bit hosts, independent of RAM:

zig build -Dtarget=arm-linux-musleabihf -Dcpu=arm1176jzf_s -Doptimize=ReleaseSafe

CI builds this target on every push. A complete Pi 1 recipe — Cloudflare Tunnel behind CGNAT, systemd units, Wi-Fi watchdog — is Chad Miller's zds-pi-setup.

Set ZDS_SERVER_DID explicitly or leave it unset and let zds derive did:web:<host> from ZDS_PUBLIC_URL; the one thing not to do is set only a non-localhost public URL on a build older than 0.3.0, which advertised did:web:localhost.