zat.dev / zlay
atproto relay in zig zlay.waow.tech
relay zig atproto
Star
Fork 0
Watch
zlay docs deployment.md
10 kB
Markdown
at codex/ingest-failure-backport

deployment #

zlay runs on a Hetzner CPX41 in Hillsboro OR, managed via k3s. all deployment is orchestrated from the relay repo using just recipes.

build and deploy #

the preferred method builds natively on the server (fast, no cross-compilation):

just zlay-publish-remote

this SSHs into the server and:

  1. git pull --ff-only in /opt/zlay
  2. zig build -Doptimize=ReleaseSafe -Dtarget=x86_64-linux-gnu
  3. buildah bud -f Dockerfile.runtime . — thin runtime image with SHA tag
  4. pushes to k3s containerd via buildah push → ctr images import
  5. kubectl set image deployment/zlay -n zlay main=<sha-tagged-image> + kubectl rollout status

the runtime image (Dockerfile.runtime) is minimal: debian bookworm-slim + ca-certificates + the binary.

why not Docker build? #

the full Dockerfile exists for CI/standalone builds but is slow on Mac (cross-compilation + QEMU). zlay-publish-remote skips all of that by building on the target architecture.

build flags #

  • -Dbackend=zio — production backend since 2026-08-05: zio cooperative fibers on one scheduler thread instead of ~2,800 OS threads (~40% RSS cut). omit the flag for the Io.Threaded fallback, which is the rollback path (old images stay pinned in containerd; rollback is one kubectl set image). deploy as just zlay publish-remote ReleaseSafe <ref> zio. see handoffs/HANDOFF-2026-08-05-zio-canary.md. NB: the zon currently pins zio beb13c8, which lives on zio's canary/obs-2026-08-04 branch, not zio main — zio main lacks the poller-starvation (d6d77b7) and arm-failure (63baa64) fixes. reconcile before repinning to a main ref.

zio triage and forensics #

if a zio build misbehaves: forensics before rollback — run scripts/zlay-forensics.sh in-pod and dump /metrics; rolling back deletes the pod, its logs, and the evidence (how canary #1's forensics were lost).

triage order: relay_seq first (flat ingest reads exactly like broken serving if you only probe the serving side), then zio_sched_switches_total (flat = wedged loop), zio_poller_registrations vs connection count (collapse = erosion), zio_poller_arm_failures_total (any nonzero is bad).

the liveness probe restarts a stalled pod. since 2026-08-22 /_healthz returns 503 once the process-wide ingest rate (relay_frames_received_total sampled every 10s) has been below RELAY_INGEST_STALL_THRESHOLD_FPS (50) for RELAY_INGEST_STALL_WINDOW_SEC (900s), after a startup grace of RELAY_INGEST_STALL_STARTUP_GRACE_SEC (900s). the threshold matches the ZlayIngestStalled rule and the window is longer than that rule's lead time, so the alert pages first; with the probe's failureThreshold 20 × 30s, kubelet kills the pod ~10 minutes after the 503 begins, i.e. ~25 minutes into a stall. the verdict is never per-worker — silence from a single host is normal, and judging hosts individually is what caused the 2026-08-19 outage.

this means a wedged pod no longer waits for you, so throw the kill switch before forensics — capture-before-restart is still the standing play:

# hold the pod up: /_healthz goes back to 200 immediately
curl -X POST -H "Authorization: Bearer $RELAY_ADMIN_PASSWORD" \
  https://zlay.waow.tech/admin/ingest-stall -d '{"enabled":false}'
# inspect (also shows rate, how long it has been below threshold, and the knobs)
curl -H "Authorization: Bearer $RELAY_ADMIN_PASSWORD" https://zlay.waow.tech/admin/ingest-stall
# widen instead of disabling: {"window_sec":3600} / {"threshold_fps":10}
# re-arm when done
curl -X POST -H "Authorization: Bearer $RELAY_ADMIN_PASSWORD" \
  https://zlay.waow.tech/admin/ingest-stall -d '{"enabled":true}'

settings made this way are not persisted: a restart returns to the env defaults, so a disabled detector cannot outlive the incident by accident. to change the defaults, set the RELAY_INGEST_STALL_* env vars in deploy/zlay-values.yaml. relay_ingest_stalled, relay_ingest_rate_fps, relay_ingest_below_threshold_seconds and relay_ingest_stall_enabled are on /metrics; when stalled is 1 the 503 body carries the rate and window.

the keepalive pinger detects half-open connections. the 2026-08-18 and 2026-08-27 wedges were mass half-open TCP: the network path died without delivering FIN/RST and nothing in the stack could tell a quiet peer from a dead one. each subscriber connection now pings its upstream after RELAY_WS_PING_INTERVAL_SEC (30s) of silence and reconnects after RELAY_WS_PING_MAX_FAILURES (4) unanswered pings. a ping counts as answered iff any frame (data or pong) arrives after its send time — never by comparing silence against the interval, which the first canary showed wrongly reaps healthy quiet hosts whenever scheduler oversleep exceeds the pong's rtt. quiet-but-alive hosts are never touched. under zio this runs as a per-connection pinger with the read loop as a cancelable task (SO_RCVTIMEO, which drives the Threaded path's heartbeat, never fires on an epoll-parked fiber). knobs are runtime-settable, not persisted, same contract as the stall detector:

curl -H "Authorization: Bearer $RELAY_ADMIN_PASSWORD" https://zlay.waow.tech/admin/ws-ping
# kill switch (reverts to pre-fix no-keepalive behavior):
curl -X POST -H "Authorization: Bearer $RELAY_ADMIN_PASSWORD" \
  https://zlay.waow.tech/admin/ws-ping -d '{"enabled":false}'
# retune: any subset of {"interval_sec":N,"max_failures":N}

on /metrics: relay_ws_pings_sent_total (baseline traffic — quiet hosts make this tick steadily), relay_ws_ping_timeout_closes_total (each is a connection declared half-open; a mass spike is the 08-18 signature being caught in the act) and relay_ws_ping_write_failures_total.

  • -Dtarget=x86_64-linux-gnu — production target, glibc. glibc malloc (per-thread arenas + madvise page-return + malloc_trim) is load-bearing for RSS at ~2,800 threads. NB: the old "musl breaks RocksDB / illegal instructions" claim was a zig 0.15 artifact and is retired — on 0.16 musl builds, links, and runs SIGILL-free at thread scale (canary, 2026-06). musl stays off prod only because the one static-musl allocator we validated as buildable — mimalloc — leaks RSS under our thread model (v2/v3/purge-forced/even under glibc). full matrix in musl-investigation.md.
  • -Dcpu=baseline — required when building inside Docker/QEMU (not needed for zlay-publish-remote since it builds natively).
  • -Doptimize=ReleaseSafe — safety checks on, optimizations on. production default since 2026-03-05. previously caused OOM (see incident-2026-03-04.md) — resolved by the frame pool moving heavy work off reader threads.

initial setup #

just zlay-init          # terraform init
just zlay-infra         # create Hetzner server with k3s
just zlay-kubeconfig    # pull kubeconfig (~2 min after creation)
just zlay-deploy        # full deploy: cert-manager, postgres, relay, monitoring

point DNS A record for ZLAY_DOMAIN at the server IP (just zlay-server-ip) before deploying.

environment variables #

set in .env in the relay repo:

variable required description
HCLOUD_TOKEN yes Hetzner Cloud API token
ZLAY_DOMAIN yes public domain (e.g. zlay.waow.tech)
ZLAY_ADMIN_PASSWORD yes bearer token for admin endpoints
ZLAY_POSTGRES_PASSWORD yes postgres password
LETSENCRYPT_EMAIL yes email for TLS certificates

operations #

just zlay-status        # nodes, pods, health
just zlay-logs          # tail relay logs
just zlay-health        # curl public health endpoint
just zlay-ssh           # ssh into server

infrastructure #

  • server: Hetzner CPX41 — 16 vCPU (AMD), 32 GB RAM, 240 GB NVMe
  • k3s: single-node kubernetes with traefik ingress
  • cert-manager: automatic TLS via Let's Encrypt
  • postgres: bitnami/postgresql helm chart (relay state, backfill progress)
  • monitoring: prometheus + grafana via kube-prometheus-stack
  • terraform: infra/zlay/ in the relay repo

memory tuning #

four changes brought steady-state memory from ~6.6 GiB down to ~1.1 GiB at ~2,250 connected hosts (ReleaseSafe):

shared TLS CA bundle. the biggest single win. websocket.zig's TLS client calls Bundle.rescan() per connection, loading the system CA certificates into a per-connection arena. with ~2,750 PDS connections, that's ~2,750 copies of the CA bundle in memory (~800 KB each = ~2.2 GiB). fix: load the bundle once in the slurper, pass it to all subscribers via config.ca_bundle. memory dropped from ~3.3 GiB to ~1.2 GiB (~65% reduction).

thread stack sizes. zig's default thread stack is 16 MB. with ~2,750 subscriber threads that maps 44 GB of virtual memory. all Thread.spawn calls use main.default_stack_size (8 MB). this is virtual memory — only touched pages count as RSS. 8 MB supports ReleaseSafe's TLS handshake path (~134 KiB peak stack).

c_allocator instead of GeneralPurposeAllocator. GPA is a debug allocator — it tracks per-allocation metadata and never returns freed small allocations to the OS. since zlay links glibc (build.zig:42), std.heap.c_allocator gives us glibc malloc with per-thread arenas, madvise-based page return, and production-grade fragmentation mitigation.

frame processing pool. reader threads (one per PDS) now only do TLS read, header decode, cursor tracking, and rate limiting — then queue raw frames to a shared pool of 16 workers. this dramatically reduced per-thread RSS in ReleaseSafe (from ~3.9 MiB to ~0.45 MiB) by keeping crypto, DB, and broadcast off reader thread stacks.

resource usage #

metric value
memory ~1.1 GiB at ~2,250 hosts (ReleaseSafe), projected ~1.3 GiB steady state
CPU ~1.5 cores peak
requests 1 GiB memory, 1000m CPU
limits 8 GiB memory
PVC 20 GiB (events + RocksDB collection index)
postgres ~238 MiB

git push #

the zlay repo is hosted on tangled. pushing requires the tangled SSH key:

GIT_SSH_COMMAND="ssh -i ~/.ssh/tangled_ed25519 -o IdentitiesOnly=yes" git push