diff --git a/AGENTS.md b/AGENTS.md index 5628ffb..5942e10 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,21 +1,29 @@ # zlay AT Protocol relay in zig 0.16. reader thread per PDS + shared frame -processing pool. ReleaseSafe in production. Io.Threaded backend (Evented -attempt shelved — see docs/evented-attempt.md). +processing pool. ReleaseSafe in production. production runs the +[zio](https://tangled.org/zzstoatzz.io/zio) std.Io backend +(`-Dbackend=zio`, cooperative fibers on one scheduler thread, ~40% RSS +cut vs Io.Threaded); the default build is Io.Threaded and remains the +rollback path. a fiber that never yields owns the scheduler — no busy +loops, no sub-ms sleep spins (see docs/handoffs/HANDOFF-2026-08-05-zio-canary.md). ## before pushing - `zig fmt --check .` and `zig build test` -- production is `-Dtarget=x86_64-linux-gnu` (glibc malloc's per-thread arenas + page-return are load-bearing for RSS at ~2,800 threads). the "musl breaks RocksDB" belief is **retired** — musl builds and runs SIGILL-free on 0.16 — but mimalloc (the only static-musl allocator we found viable) leaks under our thread model, so prod stays glibc. see [docs/musl-investigation.md](docs/musl-investigation.md) +- production is `-Dtarget=x86_64-linux-gnu` (glibc malloc's per-thread arenas + page-return are load-bearing for RSS at ~2,800 threads under the Threaded fallback). the "musl breaks RocksDB" belief is **retired** — musl builds and runs SIGILL-free on 0.16 — but mimalloc (the only static-musl allocator we found viable) leaks under our thread model, so prod stays glibc. see [docs/musl-investigation.md](docs/musl-investigation.md) - ReleaseFast has a known double-free — do not use +- don't reintroduce Io.Group usage on hot paths at fleet scale — open zio stack-aliasing bug under group churn (see docs/handoffs/HANDOFF-2026-08-04-zio-backend.md) ## deploy -configs at `../@zzstoatzz.io/relay/` — `just zlay publish-remote ReleaseSafe` +configs at `../@zzstoatzz.io/relay/` — `just zlay publish-remote ReleaseSafe zio` KUBECONFIG is set automatically by the zlay module (`zlay/kubeconfig.yaml`). +if a zio build misbehaves: forensics before rollback (`scripts/zlay-forensics.sh` +in-pod + a /metrics dump) — rolling back deletes the pod and the evidence. + ## docs - [docs/design.md](docs/design.md) — architecture, threading, memory model @@ -23,3 +31,5 @@ KUBECONFIG is set automatically by the zlay module (`zlay/kubeconfig.yaml`). - [docs/gotchas.md](docs/gotchas.md) — zig/pg.zig/rocksdb-zig/deploy traps - [docs/incident-2026-03-04.md](docs/incident-2026-03-04.md) — ReleaseSafe RSS analysis - [docs/evented-attempt.md](docs/evented-attempt.md) — Evented backend attempt and why we reverted +- [docs/handoffs/HANDOFF-2026-08-04-zio-backend.md](docs/handoffs/HANDOFF-2026-08-04-zio-backend.md) — getting zio to fleet scale (eight fixes + open aliasing bug) +- [docs/handoffs/HANDOFF-2026-08-05-zio-canary.md](docs/handoffs/HANDOFF-2026-08-05-zio-canary.md) — the sub-ms sleep root cause and canary #2 verdict diff --git a/CLAUDE.md b/CLAUDE.md index 5628ffb..5942e10 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,21 +1,29 @@ # zlay AT Protocol relay in zig 0.16. reader thread per PDS + shared frame -processing pool. ReleaseSafe in production. Io.Threaded backend (Evented -attempt shelved — see docs/evented-attempt.md). +processing pool. ReleaseSafe in production. production runs the +[zio](https://tangled.org/zzstoatzz.io/zio) std.Io backend +(`-Dbackend=zio`, cooperative fibers on one scheduler thread, ~40% RSS +cut vs Io.Threaded); the default build is Io.Threaded and remains the +rollback path. a fiber that never yields owns the scheduler — no busy +loops, no sub-ms sleep spins (see docs/handoffs/HANDOFF-2026-08-05-zio-canary.md). ## before pushing - `zig fmt --check .` and `zig build test` -- production is `-Dtarget=x86_64-linux-gnu` (glibc malloc's per-thread arenas + page-return are load-bearing for RSS at ~2,800 threads). the "musl breaks RocksDB" belief is **retired** — musl builds and runs SIGILL-free on 0.16 — but mimalloc (the only static-musl allocator we found viable) leaks under our thread model, so prod stays glibc. see [docs/musl-investigation.md](docs/musl-investigation.md) +- production is `-Dtarget=x86_64-linux-gnu` (glibc malloc's per-thread arenas + page-return are load-bearing for RSS at ~2,800 threads under the Threaded fallback). the "musl breaks RocksDB" belief is **retired** — musl builds and runs SIGILL-free on 0.16 — but mimalloc (the only static-musl allocator we found viable) leaks under our thread model, so prod stays glibc. see [docs/musl-investigation.md](docs/musl-investigation.md) - ReleaseFast has a known double-free — do not use +- don't reintroduce Io.Group usage on hot paths at fleet scale — open zio stack-aliasing bug under group churn (see docs/handoffs/HANDOFF-2026-08-04-zio-backend.md) ## deploy -configs at `../@zzstoatzz.io/relay/` — `just zlay publish-remote ReleaseSafe` +configs at `../@zzstoatzz.io/relay/` — `just zlay publish-remote ReleaseSafe zio` KUBECONFIG is set automatically by the zlay module (`zlay/kubeconfig.yaml`). +if a zio build misbehaves: forensics before rollback (`scripts/zlay-forensics.sh` +in-pod + a /metrics dump) — rolling back deletes the pod and the evidence. + ## docs - [docs/design.md](docs/design.md) — architecture, threading, memory model @@ -23,3 +31,5 @@ KUBECONFIG is set automatically by the zlay module (`zlay/kubeconfig.yaml`). - [docs/gotchas.md](docs/gotchas.md) — zig/pg.zig/rocksdb-zig/deploy traps - [docs/incident-2026-03-04.md](docs/incident-2026-03-04.md) — ReleaseSafe RSS analysis - [docs/evented-attempt.md](docs/evented-attempt.md) — Evented backend attempt and why we reverted +- [docs/handoffs/HANDOFF-2026-08-04-zio-backend.md](docs/handoffs/HANDOFF-2026-08-04-zio-backend.md) — getting zio to fleet scale (eight fixes + open aliasing bug) +- [docs/handoffs/HANDOFF-2026-08-05-zio-canary.md](docs/handoffs/HANDOFF-2026-08-05-zio-canary.md) — the sub-ms sleep root cause and canary #2 verdict diff --git a/README.md b/README.md index 396dd04..d92027f 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,9 @@ an [AT Protocol](https://atproto.com/) relay in zig. subscribes to every PDS on - **inline collection index** — indexes `(DID, collection)` pairs in the event processing pipeline using RocksDB. serves `listReposByCollection` from the relay process — no sidecar. the index design draws on [fig](https://tangled.org/microcosm.blue)'s work on [lightrail](https://tangled.org/microcosm.blue/lightrail). -- **reader thread per PDS + frame processing pool** — each PDS gets a lightweight reader thread (cursor tracking, rate limiting, header decode). heavy work (full CBOR decode, validation, DB persist, broadcast) runs on a shared pool of frame workers (configurable, default 16). +- **reader per PDS + frame processing pool** — each PDS gets a lightweight reader (cursor tracking, rate limiting, header decode). heavy work (full CBOR decode, validation, DB persist, broadcast) runs on a shared pool of frame workers (configurable, default 16). + +- **pluggable std.Io backend** — the relay is written against zig's `std.Io` interface. the default backend is `Io.Threaded` (one OS thread per blocking call site); production runs [zio](https://tangled.org/zzstoatzz.io/zio) (`-Dbackend=zio`), which schedules the same code as cooperative fibers on a single epoll-driven scheduler thread — ~90 threads instead of ~2,800 at fleet scale, for a ~40% RSS reduction. ## spec compliance @@ -37,6 +39,7 @@ requires zig 0.16 and a C/C++ toolchain (for RocksDB). zig build # build (debug) zig build test # run tests zig build -Doptimize=ReleaseSafe # release build (production default) +zig build -Dbackend=zio # zio std.Io backend (production runs ReleaseSafe + zio) just docker-build # local container, native Linux architecture just docker-build linux/amd64 # production-compatible container ``` diff --git a/docs/deployment.md b/docs/deployment.md index 6c5f441..a898b18 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -26,6 +26,7 @@ the full `Dockerfile` exists for CI/standalone builds but is slow on Mac (cross- ### build flags +- `-Dbackend=zio` — production backend since 2026-08-05: [zio](https://tangled.org/zzstoatzz.io/zio) cooperative fibers on one scheduler thread instead of ~2,800 OS threads (~40% RSS cut). omit the flag for the `Io.Threaded` fallback, which is the rollback path (old images stay pinned in containerd; rollback is one `kubectl set image`). deploy as `just zlay publish-remote ReleaseSafe zio`. see [handoffs/HANDOFF-2026-08-05-zio-canary.md](handoffs/HANDOFF-2026-08-05-zio-canary.md). - `-Dtarget=x86_64-linux-gnu` — production target, glibc. glibc malloc (per-thread arenas + `madvise` page-return + `malloc_trim`) is load-bearing for RSS at ~2,800 threads. **NB:** the old "musl breaks RocksDB / illegal instructions" claim was a zig 0.15 artifact and is **retired** — on 0.16 musl builds, links, and runs SIGILL-free at thread scale (canary, 2026-06). musl stays off prod only because the one static-musl allocator we validated as buildable — mimalloc — leaks RSS under our thread model (v2/v3/purge-forced/even under glibc). full matrix in [musl-investigation.md](musl-investigation.md). - `-Dcpu=baseline` — required when building inside Docker/QEMU (not needed for `zlay-publish-remote` since it builds natively). - `-Doptimize=ReleaseSafe` — safety checks on, optimizations on. production default since 2026-03-05. previously caused OOM (see [incident-2026-03-04.md](incident-2026-03-04.md)) — resolved by the frame pool moving heavy work off reader threads. diff --git a/docs/design.md b/docs/design.md index 9050b07..bc4a16e 100644 --- a/docs/design.md +++ b/docs/design.md @@ -239,7 +239,22 @@ at ~240 MiB. resource limits: 8 GiB memory, 1 GiB request, 1000m CPU. - thread count is still O(hosts) for readers, but reader threads are lightweight — no crypto, no DB, no broadcast contention. - remaining: IO multiplexing to reduce reader thread count from O(hosts) to O(cores) -### long-term (async I/O) +### long-term (async I/O) — **done**: zio in production (2026-08) + +> the section below is the historical assessment that led here. outcome: +> zlay adopted [zio](https://tangled.org/zzstoatzz.io/zio) via `-Dbackend=zio` +> — a drop-in `std.Io` vtable running cooperative fibers on one epoll scheduler +> thread. after a first canary failed at minute 35 (root cause: zio's ms-truncating +> timeout conversion made sub-ms sleeps return without parking, so `while (!done) +> io.sleep(100us)` spinners owned the scheduler), the fix (round non-zero sleeps +> up to ≥1ms) shipped and canary #2 certified: ~90 threads vs ~2,800, RSS band +> 1.33–1.47 GiB vs 2.27 GiB flat, at identical live load. `Io.Threaded` remains +> the default build and rollback path. full record: +> [handoffs/HANDOFF-2026-08-04-zio-backend.md](handoffs/HANDOFF-2026-08-04-zio-backend.md) +> and [handoffs/HANDOFF-2026-08-05-zio-canary.md](handoffs/HANDOFF-2026-08-05-zio-canary.md). +> known debt: an open zio stack-aliasing bug under `Io.Group` churn — groups are +> off the hot path (sequential per-address connect); don't reintroduce them at +> scale until it's closed. - zig 0.16's `Io` (io_uring/kqueue) would replace OS threads with fibers (~2,800 → ~35), scaling toward 100K+ hosts per process. - **attempted and shelved twice** — `Io.Evented` ran in production, was reverted @@ -295,13 +310,14 @@ appropriate for zlay's architecture. ### per-PDS concurrency model -indigo uses goroutines (M:N scheduling on a small thread pool). zlay uses one -OS thread per host — simple, no async runtime, no event loop. each thread -spends most time blocked in `recv()` with minimal per-frame CPU work. +indigo uses goroutines (M:N scheduling on a small thread pool). zlay writes +one blocking reader per host against `std.Io`; the backend decides what that +costs. under `Io.Threaded` (default build) each reader is an OS thread mostly +blocked in `recv()`. under zio (production, `-Dbackend=zio`) the same readers +run as cooperative fibers on a single scheduler thread. -observability: prometheus metrics expose thread count, RSS, per-host memory. -the 0.16 `Io` migration (io_uring/kqueue) is the planned optimization path, -replacing OS threads with coroutines. +observability: prometheus metrics expose thread count, RSS, per-host memory, +and zio runtime internals (`zio_sched_*`, `zio_poller_*`, `zio_dns_*`). ### skip-on-miss validation diff --git a/docs/handoffs/CRASHLOOP-HANDOFF-2026-06-07.md b/docs/handoffs/CRASHLOOP-HANDOFF-2026-06-07.md deleted file mode 100644 index 9960e23..0000000 --- a/docs/handoffs/CRASHLOOP-HANDOFF-2026-06-07.md +++ /dev/null @@ -1,177 +0,0 @@ -# crashloop handoff — integer overflow in worker teardown (2026-06-07) - -> **Temporary doc — DELETE once the follow-ups below are done.** Written by the -> operator for the zlay engineer. The bug is fixed in prod; what remains is -> merging the fix to main. Fold durable findings into the ops changelog, then -> remove this file. - -## STATUS 2026-06-08: RESOLVED in prod (fix running, not yet on main) - -- **Fixed.** Root cause was the websocket handshake `over_read` underflow - (`client.zig:771`, split terminating CRLF), NOT the mutex (that first theory - was wrong — see below). Fix chain: websocket **v0.1.5** → zat - **hotfix/websocket-crashloop** (`c8970ef`) → zlay **fix/websocket-crashloop-via-zat** - (`b49a95a`, drops the direct websocket dep, consumes via zat). -- **Verified in prod:** `ReleaseSafe-b49a95a` deployed 2026-06-08 05:08Z; - **0 restarts in 11h**, zero cores in `/var/crash`, ~2,600 hosts, ~1.8 GiB RSS, - delivering. Old build crashed every ~15–30 min. Regression test in websocket - panics without the fix, passes with it. - -## REMAINING FOLLOW-UPS (the not-good-state part) - -1. **Prod is on a feature branch, not main.** Open + merge the PR chain: - websocket v0.1.5 (tagged/pushed) → zat `hotfix/websocket-crashloop` → - zlay `fix/websocket-crashloop-via-zat`. Base the zat merge so it doesn't drag - in main's in-flight karlseguin migration or the uncommitted oauth WIP. -2. **Redeploy off main** once merged, so prod isn't pinned to a feature commit: - `just zlay publish-remote ReleaseSafe ` (the recipe now pins a ref - and aborts on a dirty `/opt/zlay` tree). -3. **Delete this file** after 1–2 are done. - ---- - -## tl;dr (REVISED after re-symbolization — supersedes the mutex theory below) - -`ReleaseSafe-9bc99fa` crashloops with `thread N panic: integer overflow`. The -`slurper.zig:611` attribution is **debug-info misattribution** — `0x14a8156` is a -shared cold panic stub parked on the `defer unlock(` line. `runWorker` has all of -`sub.run()` inlined (~13 KB), and 11 of the 17 integer-overflow check sites that -target that stub resolve to the **websocket handshake parser** -(`websocket-0.1.2/src/client/client.zig:691–786`, via `subscriber.zig:356` -`client.handshake(...)`). - -**Leading root cause: `client.zig:771`** -```zig -const over_read = pos - (line_start + 2); // usize — underflows -``` -in the end-of-headers branch (`relative_end == 0`, `\r` found at `line_start`). -It assumes the `\n` after that `\r` is already read but never checks -`line_start + 2 <= pos`. If TCP delivers the terminating `\r` as the last byte of -a read and the `\n` arrives later, `pos == line_start + 1` → -`pos - (line_start + 2)` underflows `usize` → integer-overflow panic. The sibling -CRLF-split case 15 lines down **is** guarded (`if (line_start > pos) break;` at -:787); this `over_read` line is not. - -**Trigger:** the reconnect storm (chain-break-driven worker churn) drives many -handshakes; a fraction split the terminating CRLF across reads and hit :771. This -matches the sharp 20:33Z onset and the survives-restart property (churn continues -across restarts). - -It is **not** a dirty build, **not** the mutex, and **not** seq/account_count -(all ruled out). A rebuild/redeploy is not a fix. - -> **Caveat — not yet proven which site.** The Zig panic collapsed the faulting -> address onto the shared stub, so static analysis can't prove :771 fired vs -> another site in the cluster. The definitive step is a core dump capturing the -> real faulting RIP on the next crash (operator can enable). Secondary candidate: -> `host_ops.zig:109` `free_top += 1` (CursorMap double-unregister) — single site, -> weaker, would more likely throw out-of-bounds first. - ---- - -## ORIGINAL mutex theory (WRONG — retained for the record) - -The first pass claimed the overflow was inside `Io.Mutex.unlock` via a -`waiters: u16` underflow. That is incorrect: `Io.Mutex` (Io.zig:1587) is a -3-state `enum(u32)` swap with no arithmetic; `waiters/signals` belong to -`Condition`, a different struct. `lockUncancelable` loops until acquired and -can't return un-held. The engineer's rebuttal was correct; the re-symbolization -above replaces this section. - -## timeline - -| time (UTC) | event | -|---|---| -| 2026-05-31 21:14Z | `ReleaseSafe-9bc99fa` deployed | -| 06-01 → 06-07 20:30Z | **stable, zero restarts** (~6.5 days) | -| **2026-06-07 20:33Z** | **first crash** — onset, mid `prevData mismatch` chain-break storm | -| 20:33Z → ongoing | crashloop, exit 139, every ~15–30 min | -| 2026-06-07 ~03:55Z (06-08) | operator redeployed pinned `9bc99fa` + forced clean pod (mitigation/diagnostic only) | - -Onset is sharp and correlates with a network-wide `prevData mismatch` chain-break -storm (logs are saturated with `host X: chain break uid=… prevData mismatch`, -mostly `*.host.bsky.network`). That storm drives mass worker connect/disconnect. - -## the panic - -``` -thread 2708 panic: integer overflow -/opt/zlay/src/slurper.zig:611:40: 0x14a8156 in runWorker (zlay) -.../std/Io/Threaded.zig:741:20 in start -.../std/Io/Threaded.zig:1794:29 in worker -``` - -`runWorker` (slurper.zig ~599–619) cleanup block: - -```zig -self.workers_mutex.lockUncancelable(self.io); // 610 -defer self.workers_mutex.unlock(self.io); // 611 <-- panics here, col 40 = unlock( -_ = self.workers.remove(host_id); // 612 -_ = self.bc.stats.connected_inbound.fetchSub(1, .monotonic); // 613 (atomic, wraps — not it) -``` - -`workers_mutex: Io.Mutex` (slurper.zig:246), also locked/unlocked at 465/589/714. -`Io.Mutex` (zig std 0.16-dev.3059) is a futex mutex with a `State` carrying -`waiters: u16` / `signals: u16`. The overflow is in that arithmetic during -`unlock` — strongly suggests `unlock` is being called on a mutex not held (or a -double-unlock / cancellation path leaving `lockUncancelable` un-acquired while the -`defer unlock` still fires). - -## what was ruled out (with evidence) - -- **dirty build** — `/opt/zlay` is clean at `9bc99fa` (`git status --porcelain` - empty); binary symbolizes against committed source via - `addr2line -f -i -e zig-out/bin/zlay 0x14a8156`. Same source. -- **load / throughput** — frames/s (~428) and `connected_inbound` (~2630) flat - across the 20:33Z onset. -- **seq boundary overflow** — `relay_seq` = `0x6363a382…` (~7.16e18), smooth - +~308/s, ~2.06e18 of headroom to 2^63. Not near any 2^n. -- **`computeLimits` `account_count * 10` (subscriber.zig)** — max real - `COUNT(*)` is 804k (jellybaby) → 8.0M, nowhere near u64 overflow; no negative - or extreme `account_limit` in the `host` table (max 100000); and the large - hosts are trusted `.host.bsky.network`, which take the fixed-limit branch and - skip the `* 10` path entirely. - -## why restart/rebuild won't fix it - -The crashloop already restarts the process every ~20 min and keeps crashing, so -the trigger is **not** uptime accumulation. It persists across process restarts — -either ongoing upstream data (the chain-break storm) or persistent on-disk state -(event_log / postgres / cursor). The fix has to be in the worker-lifecycle mutex -handling (make lock/unlock balanced across all `runWorker` exit and cancellation -paths), not a redeploy. - -## how this was re-symbolized (method) - -The static backtrace names only the shared panic stub. To find the real sites: -``` -# on build server (5.78.179.88), binary at /opt/zlay/zig-out/bin/zlay -objdump -d --start-address=0x14a4b00 --stop-address=0x14a8170 zlay \ - | grep 14a8152 # find all jo/jb that target the integerOverflow stub -addr2line -i -e zlay 0x # innermost inlined frame = real source -``` -17 sites target the stub; 11 land in `client.zig:691–786` (handshake) via -`subscriber.zig:356`; the lone `jo` (signed) is `client.zig:747` -(`milliTimestamp + timeout_ms`, unlikely); 2 land in `host_ops.zig` (CursorMap). - -## suggested fix + confirmation - -1. **Fix `client.zig:771`** in websocket-0.1.2: guard the `over_read` computation - for the split-CRLF case, e.g. `if (line_start + 2 > pos) break;` before - computing `over_read` (mirror the existing :787 guard), or compute with a - checked/saturating subtraction. This is in the vendored dep — patch upstream - (ptr.pet/websocket) or carry a local pin. -2. **Confirm before fixing blind:** enable core dumps and capture the next crash's - faulting RIP to prove :771 (vs another handshake site or CursorMap). Operator - can do this on the running pod. -3. Reproduce by fragmenting a handshake response so the final `\r\n` splits across - two reads with `\r` last. - -## operator-side context - -- Build provenance hardened: `just zlay publish-remote [optimize] [ref]` now pins - an exact ref and **aborts on a dirty `/opt/zlay`** tree, so a deployed image tag - maps 1:1 to a commit. -- Source of truth for the running binary is `/opt/zlay` on the build server - (5.78.179.88), incl. `zig-out/bin/zlay` with symbols for `addr2line`. -- Current running pod: freshly restarted clean `9bc99fa`; expected to recur. diff --git a/docs/handoffs/MERGE-TO-MAIN-TODO.md b/docs/handoffs/MERGE-TO-MAIN-TODO.md deleted file mode 100644 index cbc0c47..0000000 --- a/docs/handoffs/MERGE-TO-MAIN-TODO.md +++ /dev/null @@ -1,32 +0,0 @@ -# ACTION: prod is running off a feature branch — get the fix onto main - -> **Delete this file once done.** Left by the operator 2026-06-08 for the zlay -> engineer. Prod is healthy but pinned to an unmerged branch; that's the problem -> to close. Forensic detail in `CRASHLOOP-HANDOFF-2026-06-07.md` (also delete -> when done). - -## the situation - -- **Prod runs `zlay@b49a95a`**, which exists ONLY on branch - `fix/websocket-crashloop-via-zat` (pushed to origin, never merged to main). -- That commit: drops zlay's direct `websocket` dep and consumes it via `zat`, - with `zat` pinned to the crashloop hotfix. -- This fixed a prod crashloop (websocket handshake CRLF underflow). Stable 11h+, - 0 restarts. **Do not revert.** The goal is to land it on main, not to undo it. - -## what to do (zlay side) - -1. Open a PR: `fix/websocket-crashloop-via-zat` → `main`. -2. This depends on the **zat** side landing first (see `zat` repo's - `MERGE-TO-MAIN-TODO.md`) — zlay points at a `zat` commit that must be on a - stable/main `zat` ref before zlay's main can. -3. Once zat is sorted and this merges, redeploy prod off the main sha: - `just zlay publish-remote ReleaseSafe ` - (the recipe pins a ref and aborts if `/opt/zlay` is dirty). -4. Delete this file + `CRASHLOOP-HANDOFF-2026-06-07.md`. - -## upstream chain (FYI) - -websocket fix `github.com/zzstoatzz/websocket.zig` **v0.1.5** (tagged/pushed) -→ zat `hotfix/websocket-crashloop` (`c8970ef`) -→ zlay `fix/websocket-crashloop-via-zat` (`b49a95a`, prod).