From fc934fd3de4f13d81bdb5f05a949862452cad6b4 Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Thu, 5 Mar 2026 13:29:18 -0600 Subject: [PATCH] docs: slim CLAUDE.md, move details to docs/gotchas.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CLAUDE.md loads into every conversation — keep it to guardrails only (23 lines, down from 100). moved pg.zig API patterns, rocksdb-zig traps, websocket.zig fork details, metrics gotchas, and deploy operational tips to docs/gotchas.md. Co-Authored-By: Claude Opus 4.6 --- CLAUDE.md | 103 ++++++------------------------------------------ docs/gotchas.md | 42 ++++++++++++++++++++ 2 files changed, 55 insertions(+), 90 deletions(-) create mode 100644 docs/gotchas.md diff --git a/CLAUDE.md b/CLAUDE.md index 06f563d..1b1fe2c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,100 +1,23 @@ # zlay -AT Protocol relay in zig 0.15.2. subscribes to every PDS, validates commit -signatures, serves merged firehose to downstream consumers. +AT Protocol relay in zig 0.15.2. reader thread per PDS + shared frame +processing pool. ReleaseSafe in production. -## architecture +## before pushing -- **reader thread per PDS** (~2,750) — lightweight: TLS read, header decode, - cursor tracking, rate limiting. submits raw frames to pool. -- **frame pool** (16 workers) — CBOR decode, ECDSA verify, DB persist, broadcast. -- **validator** (4-8 resolver threads) — background DID resolution, signing key cache. -- dependencies: zat (AT proto primitives), websocket.zig, pg.zig, rocksdb-zig. - -## build - -```bash -zig build -Doptimize=ReleaseSafe -Dtarget=x86_64-linux-gnu # production -zig build test # run tests -zig fmt --check . # CI checks this -``` - -always run `zig fmt --check .` and `zig build test` before pushing. - -## deploy - -deploy configs live at `../@zzstoatzz.io/relay/` (justfile, helm, terraform). - -```bash -cd ../relay -just zlay-publish-remote ReleaseSafe # build on server, deploy -just zlay-publish-remote # debug build (fallback) -``` - -**critical rules:** -- MUST use `-Dtarget=x86_64-linux-gnu` — musl produces illegal instructions in RocksDB -- MUST set `KUBECONFIG="$(pwd)/zlay-kubeconfig.yaml"` — default context is docker-desktop -- never change probe paths and image in separate operations -- immutable SHA tags, never `:latest` +- `zig fmt --check .` and `zig build test` +- MUST use `-Dtarget=x86_64-linux-gnu` for production (musl breaks RocksDB) - ReleaseFast has a known double-free — do not use -## key files - -| file | purpose | -|---|---| -| `src/main.zig` | entry point, thread stacks (8 MiB), signal handling | -| `src/subscriber.zig` | per-PDS reader thread, FrameHandler | -| `src/frame_worker.zig` | pool worker: decode, validate, persist, broadcast | -| `src/thread_pool.zig` | generic ring-buffer thread pool | -| `src/validator.zig` | signature verify, DID resolver, LRU key cache | -| `src/broadcaster.zig` | fan-out to consumers, prometheus metrics | -| `src/event_log.zig` | disk persist, postgres, cursor replay | -| `src/collection_index.zig` | RocksDB (collection, did) index | -| `src/slurper.zig` | multi-host crawl manager | -| `src/api.zig` | HTTP API endpoints (served via httpFallback on WS port) | - -## current production state - -- running at zlay.waow.tech, ~2,750 PDS hosts -- ReleaseSafe, 8 MiB thread stacks, ~1.1 GiB RSS -- ports: 3000 (WS + HTTP), 3001 (metrics only) -- probes: `/_healthz` (liveness), `/_readyz` (readiness, DB check) -- `relay_build_info{git_sha,optimize}` metric confirms what's running - -## postgres - -schema: `account`, `account_repo` (rev/cid), `log_file_refs`, `host`, `domain_ban`, `backfill_progress`. - -pg.zig patterns: -- `pool.rowUnsafe()` for single-row queries — `.get(T, col)` returns `T` -- `pool.query()` + `result.nextUnsafe()` for multi-row -- `QueryRow.deinit()` returns `!void` — use `defer row.deinit() catch {}` -- DB-dependent tests skip via `requireDatabaseUrl()` → `error.SkipZigTest` - -## websocket.zig fork - -fork of karlseguin/websocket.zig with patches: -- `httpFallback`: `_handleHandshake` catches parse errors, routes to `H.httpFallback` - if present — this is how all HTTP API endpoints are served on the WS port -- lenient handshake parser: non-WS headers skip instead of error -- all HTTP + WS on port 3000, metrics-only on 3001 +## deploy -## operational notes +configs at `../@zzstoatzz.io/relay/` — `just zlay-publish-remote ReleaseSafe` -- containers are minimal debian — no curl, wget, or shell utilities. - use busybox pod or `kubectl port-forward` to check metrics -- two separate kubeconfigs: Go relay (`kubeconfig.yaml`) vs zlay (`zlay-kubeconfig.yaml`) -- `$ZLAY_KUBECONFIG` is only set inside `just` — raw shell needs the absolute path -- after `helm upgrade`, must `kubectl set image` back to SHA tag (helm uses `latest` from values) -- `f.readAll()` not `f.reader().readAll()` — `File.reader()` takes a buffer arg in zig 0.15 -- mallinfo() overflows at 2 GiB (c_int fields) — smaps metrics are the reliable RSS source -- rocksdb-zig `ColumnFamilyOptions` is a stub — use `extern fn rocksdb_set_options_cf` post-open +MUST set `KUBECONFIG="$(pwd)/zlay-kubeconfig.yaml"` — default context is docker-desktop. -## zig gotchas relevant here +## docs -- `&.{...}` in loops creates stack-local arrays that alias — heap-allocate instead -- operator precedence: `(byte & 0xf0) == 0` not `byte & 0xf0 == 0` -- `ArrayList` is unmanaged in 0.15 — pass allocator to each method -- rocksdb-zig iterator Data: do NOT call `.deinit()` on entries (SIGABRT) -- rocksdb-zig DB.open: path must be null-terminated -- pg.zig `QueryRow.deinit()` returns `!void` — use `defer row.deinit() catch {}` +- [docs/design.md](docs/design.md) — architecture, threading, memory model +- [docs/deployment.md](docs/deployment.md) — build flags, infra, resource usage +- [docs/gotchas.md](docs/gotchas.md) — zig/pg.zig/rocksdb-zig/deploy traps +- [docs/incident-2026-03-04.md](docs/incident-2026-03-04.md) — ReleaseSafe RSS analysis diff --git a/docs/gotchas.md b/docs/gotchas.md new file mode 100644 index 0000000..2941e38 --- /dev/null +++ b/docs/gotchas.md @@ -0,0 +1,42 @@ +# gotchas + +hard-won knowledge from debugging zlay. check here before guessing. + +## zig 0.15 + +- `&.{...}` in loops creates stack-local arrays that alias across iterations — all references point to same memory. heap-allocate instead. +- operator precedence: `(byte & 0xf0) == 0` not `byte & 0xf0 == 0` +- `ArrayList` is unmanaged — pass allocator to each method +- `f.readAll()` not `f.reader().readAll()` — `File.reader()` takes a buffer arg in zig 0.15 + +## pg.zig + +- `pool.rowUnsafe()` for single-row queries — `.get(T, col)` returns `T` +- `pool.query()` + `result.nextUnsafe()` for multi-row +- `QueryRow.deinit()` returns `!void` — use `defer row.deinit() catch {}` +- DB-dependent tests skip via `requireDatabaseUrl()` → `error.SkipZigTest` + +## rocksdb-zig + +- iterator `Data` has `.free = rocksdb_free` but points to internal buffers — do NOT call `.deinit()` on iterator entries (SIGABRT). only deinit `Data` from `db.get()` +- `DB.open` passes `dir.ptr` raw to C API — path must be null-terminated (use `realpathAlloc` or zero-init buffer) +- `ColumnFamilyOptions` is a stub — use `extern fn rocksdb_set_options_cf` to set options post-open + +## websocket.zig fork + +fork of karlseguin/websocket.zig with patches: +- `httpFallback`: `_handleHandshake` catches parse errors, routes to `H.httpFallback` if present — this is how all HTTP API endpoints are served on the WS port +- lenient handshake parser: non-WS headers skip instead of error + +## metrics + +- mallinfo() overflows at 2 GiB (c_int fields) — smaps metrics are the reliable RSS source +- `relay_build_info{git_sha,optimize}` confirms what binary is running + +## deploy + +- containers are minimal debian — no curl, wget, or shell utilities. use busybox pod or `kubectl port-forward` +- two separate kubeconfigs: Go relay (`kubeconfig.yaml`) vs zlay (`zlay-kubeconfig.yaml`) +- `$ZLAY_KUBECONFIG` is only set inside `just` — raw shell needs the absolute path +- after `helm upgrade`, must `kubectl set image` back to SHA tag (helm uses `latest` from values) +- never change probe paths and image in separate operations (see [incident-2026-03-04.md](incident-2026-03-04.md)) -- 2.51.2