# deployment

zlay runs on a Hetzner CPX41 in Hillsboro OR, managed via k3s. all deployment is orchestrated from the [relay repo](https://tangled.org/zzstoatzz.io/relay) using `just` recipes.

## build and deploy

the preferred method builds natively on the server (fast, no cross-compilation):

```bash
just zlay-publish-remote
```

this SSHs into the server and:

1. `git pull --ff-only` in `/opt/zlay`
2. `zig build -Doptimize=ReleaseSafe -Dtarget=x86_64-linux-gnu`
3. `buildah bud -f Dockerfile.runtime .` — thin runtime image with SHA tag
4. pushes to k3s containerd via `buildah push` → `ctr images import`
5. `kubectl set image deployment/zlay -n zlay main=<sha-tagged-image>` + `kubectl rollout status`

the runtime image (`Dockerfile.runtime`) is minimal: debian bookworm-slim + ca-certificates + the binary.

### why not Docker build?

the full `Dockerfile` exists for CI/standalone builds but is slow on Mac (cross-compilation + QEMU). `zlay-publish-remote` skips all of that by building on the target architecture.

### build flags

- `-Dbackend=zio` — production backend since 2026-08-05: [zio](https://tangled.org/zzstoatzz.io/zio) cooperative fibers on one scheduler thread instead of ~2,800 OS threads (~40% RSS cut). omit the flag for the `Io.Threaded` fallback, which is the rollback path (old images stay pinned in containerd; rollback is one `kubectl set image`). deploy as `just zlay publish-remote ReleaseSafe <ref> zio`. see [handoffs/HANDOFF-2026-08-05-zio-canary.md](handoffs/HANDOFF-2026-08-05-zio-canary.md). **NB:** the zon currently pins zio `beb13c8`, which lives on zio's `canary/obs-2026-08-04` branch, not zio main — zio main lacks the poller-starvation (`d6d77b7`) and arm-failure (`63baa64`) fixes. reconcile before repinning to a main ref.

### zio triage and forensics

if a zio build misbehaves: **forensics before rollback** — run
`scripts/zlay-forensics.sh` in-pod and dump `/metrics`; rolling back deletes
the pod, its logs, and the evidence (how canary #1's forensics were lost).

triage order: `relay_seq` first (flat ingest reads exactly like broken
serving if you only probe the serving side), then `zio_sched_switches_total`
(flat = wedged loop), `zio_poller_registrations` vs connection count
(collapse = erosion), `zio_poller_arm_failures_total` (any nonzero is bad).

**the liveness probe restarts a stalled pod.** since 2026-08-22 `/_healthz`
returns 503 once the *process-wide* ingest rate (`relay_frames_received_total`
sampled every 10s) has been below `RELAY_INGEST_STALL_THRESHOLD_FPS` (50) for
`RELAY_INGEST_STALL_WINDOW_SEC` (900s), after a startup grace of
`RELAY_INGEST_STALL_STARTUP_GRACE_SEC` (900s). the threshold matches the
`ZlayIngestStalled` rule and the window is longer than that rule's lead time,
so the alert pages first; with the probe's `failureThreshold 20 × 30s`,
kubelet kills the pod ~10 minutes after the 503 begins, i.e. ~25 minutes
into a stall. the verdict is never per-worker — silence from a single host is
normal, and judging hosts individually is what caused the 2026-08-19 outage.

this means a wedged pod no longer waits for you, so **throw the kill switch
before forensics** — capture-before-restart is still the standing play:

```bash
# hold the pod up: /_healthz goes back to 200 immediately
curl -X POST -H "Authorization: Bearer $RELAY_ADMIN_PASSWORD" \
  https://zlay.waow.tech/admin/ingest-stall -d '{"enabled":false}'
# inspect (also shows rate, how long it has been below threshold, and the knobs)
curl -H "Authorization: Bearer $RELAY_ADMIN_PASSWORD" https://zlay.waow.tech/admin/ingest-stall
# widen instead of disabling: {"window_sec":3600} / {"threshold_fps":10}
# re-arm when done
curl -X POST -H "Authorization: Bearer $RELAY_ADMIN_PASSWORD" \
  https://zlay.waow.tech/admin/ingest-stall -d '{"enabled":true}'
```

settings made this way are **not persisted**: a restart returns to the env
defaults, so a disabled detector cannot outlive the incident by accident. to
change the defaults, set the `RELAY_INGEST_STALL_*` env vars in
`deploy/zlay-values.yaml`. `relay_ingest_stalled`, `relay_ingest_rate_fps`,
`relay_ingest_below_threshold_seconds` and `relay_ingest_stall_enabled` are
on `/metrics`; when `stalled` is 1 the 503 body carries the rate and window.

**the keepalive pinger detects half-open connections.** the 2026-08-18 and
2026-08-27 wedges were mass half-open TCP: the network path died without
delivering FIN/RST and nothing in the stack could tell a quiet peer from a
dead one. each subscriber connection now pings its upstream after
`RELAY_WS_PING_INTERVAL_SEC` (30s) of silence and reconnects after
`RELAY_WS_PING_MAX_FAILURES` (4) unanswered pings. a ping counts as
answered iff any frame (data or pong) arrives after its send time — never
by comparing silence against the interval, which the first canary showed
wrongly reaps healthy quiet hosts whenever scheduler oversleep exceeds the
pong's rtt. quiet-but-alive hosts are never touched.
under zio this runs as a per-connection pinger with the read loop as a
cancelable task (SO_RCVTIMEO, which drives the Threaded path's heartbeat,
never fires on an epoll-parked fiber). knobs are runtime-settable, not
persisted, same contract as the stall detector:

```bash
curl -H "Authorization: Bearer $RELAY_ADMIN_PASSWORD" https://zlay.waow.tech/admin/ws-ping
# kill switch (reverts to pre-fix no-keepalive behavior):
curl -X POST -H "Authorization: Bearer $RELAY_ADMIN_PASSWORD" \
  https://zlay.waow.tech/admin/ws-ping -d '{"enabled":false}'
# retune: any subset of {"interval_sec":N,"max_failures":N}
```

on `/metrics`: `relay_ws_pings_sent_total` (baseline traffic — quiet hosts
make this tick steadily), `relay_ws_ping_timeout_closes_total` (each is a
connection declared half-open; a mass spike is the 08-18 signature being
caught in the act) and `relay_ws_ping_write_failures_total`.
- `-Dtarget=x86_64-linux-gnu` — production target, glibc. glibc malloc (per-thread arenas + `madvise` page-return + `malloc_trim`) is load-bearing for RSS at ~2,800 threads. **NB:** the old "musl breaks RocksDB / illegal instructions" claim was a zig 0.15 artifact and is **retired** — on 0.16 musl builds, links, and runs SIGILL-free at thread scale (canary, 2026-06). musl stays off prod only because the one static-musl allocator we validated as buildable — mimalloc — leaks RSS under our thread model (v2/v3/purge-forced/even under glibc). full matrix in [musl-investigation.md](musl-investigation.md).
- `-Dcpu=baseline` — required when building inside Docker/QEMU (not needed for `zlay-publish-remote` since it builds natively).
- `-Doptimize=ReleaseSafe` — safety checks on, optimizations on. production default since 2026-03-05. previously caused OOM (see [incident-2026-03-04.md](incident-2026-03-04.md)) — resolved by the frame pool moving heavy work off reader threads.

## initial setup

```bash
just zlay-init          # terraform init
just zlay-infra         # create Hetzner server with k3s
just zlay-kubeconfig    # pull kubeconfig (~2 min after creation)
just zlay-deploy        # full deploy: cert-manager, postgres, relay, monitoring
```

point DNS A record for `ZLAY_DOMAIN` at the server IP (`just zlay-server-ip`) before deploying.

## environment variables

set in `.env` in the relay repo:

| variable | required | description |
|----------|----------|-------------|
| `HCLOUD_TOKEN` | yes | Hetzner Cloud API token |
| `ZLAY_DOMAIN` | yes | public domain (e.g. `zlay.waow.tech`) |
| `ZLAY_ADMIN_PASSWORD` | yes | bearer token for admin endpoints |
| `ZLAY_POSTGRES_PASSWORD` | yes | postgres password |
| `LETSENCRYPT_EMAIL` | yes | email for TLS certificates |

## operations

```bash
just zlay-status        # nodes, pods, health
just zlay-logs          # tail relay logs
just zlay-health        # curl public health endpoint
just zlay-ssh           # ssh into server
```

## infrastructure

- **server**: Hetzner CPX41 — 16 vCPU (AMD), 32 GB RAM, 240 GB NVMe
- **k3s**: single-node kubernetes with traefik ingress
- **cert-manager**: automatic TLS via Let's Encrypt
- **postgres**: bitnami/postgresql helm chart (relay state, backfill progress)
- **monitoring**: prometheus + grafana via kube-prometheus-stack
- **terraform**: `infra/zlay/` in the relay repo

## memory tuning

four changes brought steady-state memory from ~6.6 GiB down to ~1.1 GiB at ~2,250 connected hosts (ReleaseSafe):

**shared TLS CA bundle.** the biggest single win. websocket.zig's TLS client calls `Bundle.rescan()` per connection, loading the system CA certificates into a per-connection arena. with ~2,750 PDS connections, that's ~2,750 copies of the CA bundle in memory (~800 KB each = ~2.2 GiB). fix: load the bundle once in the slurper, pass it to all subscribers via `config.ca_bundle`. memory dropped from ~3.3 GiB to ~1.2 GiB (~65% reduction).

**thread stack sizes.** zig's default thread stack is 16 MB. with ~2,750 subscriber threads that maps 44 GB of virtual memory. all `Thread.spawn` calls use `main.default_stack_size` (8 MB). this is virtual memory — only touched pages count as RSS. 8 MB supports ReleaseSafe's TLS handshake path (~134 KiB peak stack).

**c_allocator instead of GeneralPurposeAllocator.** GPA is a debug allocator — it tracks per-allocation metadata and never returns freed small allocations to the OS. since zlay links glibc (`build.zig:42`), `std.heap.c_allocator` gives us glibc malloc with per-thread arenas, madvise-based page return, and production-grade fragmentation mitigation.

**frame processing pool.** reader threads (one per PDS) now only do TLS read, header decode, cursor tracking, and rate limiting — then queue raw frames to a shared pool of 16 workers. this dramatically reduced per-thread RSS in ReleaseSafe (from ~3.9 MiB to ~0.45 MiB) by keeping crypto, DB, and broadcast off reader thread stacks.

## resource usage

| metric | value |
|--------|-------|
| memory | ~1.1 GiB at ~2,250 hosts (ReleaseSafe), projected ~1.3 GiB steady state |
| CPU | ~1.5 cores peak |
| requests | 1 GiB memory, 1000m CPU |
| limits | 8 GiB memory |
| PVC | 20 GiB (events + RocksDB collection index) |
| postgres | ~238 MiB |

## git push

the zlay repo is hosted on tangled. pushing requires the tangled SSH key:

```bash
GIT_SSH_COMMAND="ssh -i ~/.ssh/tangled_ed25519 -o IdentitiesOnly=yes" git push
```