From e09f9fa9ad13cf149491d473befaa73334c6b140 Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Sun, 1 Mar 2026 23:47:50 -0600 Subject: [PATCH] docs: add backfill and deployment documentation Co-Authored-By: Claude Opus 4.6 --- docs/backfill.md | 105 +++++++++++++++++++++++++++++++++++++++++++++ docs/deployment.md | 90 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 195 insertions(+) create mode 100644 docs/backfill.md create mode 100644 docs/deployment.md diff --git a/docs/backfill.md b/docs/backfill.md new file mode 100644 index 0000000..28291f2 --- /dev/null +++ b/docs/backfill.md @@ -0,0 +1,105 @@ +# collection index backfill + +the collection index (RocksDB) only has entries for accounts that have created records since live indexing was deployed. the backfill imports historical data from a source relay that already has a complete `listReposByCollection` endpoint. + +## how it works + +the backfiller runs as a background thread, triggered via admin API. it: + +1. **discovers collections** from two sources (unioned): + - [lexicon garden](https://lexicon.garden/llms.txt) `llms.txt` — ~700 known NSIDs, parsed from markdown links + - RocksDB scan — collections already observed from the live firehose (RBC column family prefix scan) + +2. **inserts progress rows** into postgres (`backfill_progress` table) for each collection. existing rows are skipped (`ON CONFLICT DO NOTHING`), so re-triggering is safe. + +3. **pages through each collection** sequentially, calling `com.atproto.sync.listReposByCollection` on the source relay (default: `bsky.network`) with `limit=1000`. each DID in the response is added to the collection index via `addCollection`. cursor and imported count are persisted after each page for resumability. + +4. **marks complete** when a page returns no cursor (no more results). + +## progress tracking + +```sql +CREATE TABLE backfill_progress ( + collection TEXT PRIMARY KEY, + source TEXT NOT NULL, + cursor TEXT NOT NULL DEFAULT '', + imported_count BIGINT NOT NULL DEFAULT 0, + completed_at TIMESTAMPTZ, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +``` + +- cursor = last pagination cursor from source relay +- imported_count = total DIDs added for this collection +- completed_at = null while in progress, set when done +- if the process crashes or restarts, it resumes from the saved cursor + +## admin API + +requires bearer token auth (`RELAY_ADMIN_PASSWORD`). + +### trigger backfill + +``` +POST /admin/backfill-collections?source=bsky.network +``` + +returns 200 with collection count if started, 409 if already running. only one backfill can run at a time. + +### check status + +``` +GET /admin/backfill-collections +``` + +returns JSON: + +```json +{ + "running": true, + "total": 1269, + "completed": 621, + "in_progress": 648, + "total_imported": 13628818, + "collections": [ + {"collection": "app.bsky.feed.post", "source": "bsky.network", "imported": 28000000, "completed": true}, + {"collection": "app.bsky.feed.like", "source": "bsky.network", "imported": 6732000, "completed": false, "cursor": "did:plc:..."} + ] +} +``` + +## using the script + +the relay repo has a convenience script at `scripts/backfill-status`: + +```bash +# check progress (summary with recent incomplete) +./scripts/backfill-status status + +# trigger a new backfill +./scripts/backfill-status start [source] + +# full JSON output +./scripts/backfill-status full +``` + +requires `ZLAY_ADMIN_PASSWORD` and `ZLAY_DOMAIN` in `.env`. + +## performance characteristics + +- collections are processed sequentially (one at a time) +- 100ms pause between pages to avoid hammering the source relay +- one HTTP client is reused across all pages for a given collection +- large collections like `app.bsky.feed.like` (~30M+ DIDs) take 1-2 hours each +- small/niche collections complete in seconds +- full backfill of ~1269 collections takes several hours, dominated by the 5-6 largest `app.bsky.*` collections + +## re-running + +safe to trigger again after completion — existing progress rows are preserved, completed collections are skipped. useful if new collections appear (e.g. new lexicons published to the network). + +## source code + +- `src/backfill.zig` — Backfiller struct with all backfill logic +- `src/event_log.zig` — backfill_progress table creation (in `init()`) +- `src/main.zig` — admin route handlers diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..e5a9c13 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,90 @@ +# deployment + +zlay runs on a Hetzner CPX41 in Hillsboro OR, managed via k3s. all deployment is orchestrated from the [relay repo](https://tangled.org/zzstoatzz.io/relay) using `just` recipes. + +## build and deploy + +the preferred method builds natively on the server (fast, no cross-compilation): + +```bash +just zlay-publish-remote +``` + +this SSHs into the server and: + +1. `git pull --ff-only` in `/opt/zlay` +2. `zig build -Doptimize=ReleaseSafe -Dtarget=x86_64-linux-gnu` — native x86_64 build +3. `buildah bud -t atcr.io/zzstoatzz.io/zlay:latest -f Dockerfile.runtime .` — thin runtime image +4. pushes to k3s containerd via `buildah push` → `ctr images import` +5. `kubectl rollout restart deployment/zlay -n zlay` + +the runtime image (`Dockerfile.runtime`) is minimal: debian bookworm-slim + ca-certificates + the binary. + +### why not Docker build? + +the full `Dockerfile` exists for CI/standalone builds but is slow on Mac (cross-compilation + QEMU). `zlay-publish-remote` skips all of that by building on the target architecture. + +### build flags + +- `-Dtarget=x86_64-linux-gnu` — **must use glibc**, not musl. zig 0.15's C++ codegen for musl produces illegal instructions in RocksDB's LRU cache. +- `-Dcpu=baseline` — required when building inside Docker/QEMU (not needed for `zlay-publish-remote` since it builds natively). +- `-Doptimize=ReleaseSafe` — safety checks on, optimizations on. + +## initial setup + +```bash +just zlay-init # terraform init +just zlay-infra # create Hetzner server with k3s +just zlay-kubeconfig # pull kubeconfig (~2 min after creation) +just zlay-deploy # full deploy: cert-manager, postgres, relay, monitoring +``` + +point DNS A record for `ZLAY_DOMAIN` at the server IP (`just zlay-server-ip`) before deploying. + +## environment variables + +set in `.env` in the relay repo: + +| variable | required | description | +|----------|----------|-------------| +| `HCLOUD_TOKEN` | yes | Hetzner Cloud API token | +| `ZLAY_DOMAIN` | yes | public domain (e.g. `zlay.waow.tech`) | +| `ZLAY_ADMIN_PASSWORD` | yes | bearer token for admin endpoints | +| `ZLAY_POSTGRES_PASSWORD` | yes | postgres password | +| `LETSENCRYPT_EMAIL` | yes | email for TLS certificates | + +## operations + +```bash +just zlay-status # nodes, pods, health +just zlay-logs # tail relay logs +just zlay-health # curl public health endpoint +just zlay-ssh # ssh into server +``` + +## infrastructure + +- **server**: Hetzner CPX41 — 16 vCPU (AMD), 32 GB RAM, 240 GB NVMe +- **k3s**: single-node kubernetes with traefik ingress +- **cert-manager**: automatic TLS via Let's Encrypt +- **postgres**: bitnami/postgresql helm chart (relay state, backfill progress) +- **monitoring**: prometheus + grafana via kube-prometheus-stack +- **terraform**: `infra/zlay/` in the relay repo + +## resource usage + +| metric | value | +|--------|-------| +| memory | ~1.8 GiB steady state (1486 subscribers) | +| CPU | ~1.5 cores peak | +| limits | 8 GiB memory, 250m CPU request | +| PVC | 20 GiB (events + RocksDB collection index) | +| postgres | ~131 MiB | + +## git push + +the zlay repo is hosted on tangled. pushing requires the tangled SSH key: + +```bash +GIT_SSH_COMMAND="ssh -i ~/.ssh/tangled_ed25519 -o IdentitiesOnly=yes" git push +``` -- 2.51.2