# zlay 0.16 migration — status and known issues

last updated: 2026-07-30 (migration content below is from 2026-04-04)

**the 0.16 migration is done and deployed.** zlay runs 0.16.0 release on
`Io.Threaded`; current state is zlay 0.0.5 at `863a6ef`. this document is
retained for the eight crash write-ups and the cross-Io rule, which remain the
best record of why the architecture looks the way it does — but read the
per-section notes: anything describing `Io.Evented` as live is history, shelved
at `e6cdf84` (2026-04-09). see [evented-attempt.md](evented-attempt.md).

historical context, at time of writing:
- stable production build: `a931853` (zig 0.15)
- latest 0.16 build: `b433403` (not yet deployed — all fixes through crash 8)

## what happened

zlay is an AT Protocol relay (~8,400 LOC, 15 source files). it crawls ~2,750
PDS instances, validates frames, persists to disk, and fans out to downstream
consumers over WebSocket. the zig 0.16 migration rewrote all I/O to use
`std.Io` primitives.

eight separate crashes/bugs were found and fixed during deployment attempts:

### crash 1: SIGSEGV on startup (fixed in `f996812`)

**symptom**: exit code 139 immediately after printing "io backend: Evented"

**cause**: the build auto-selected `Io.Evented` (io_uring on linux). evented
futex wait/wake access fiber-local `Thread.current()` state. but zlay's frame
pool (`thread_pool.zig`) spawns plain `std.Thread` workers — they don't have
fiber state. any `Io.Mutex` operation from a plain thread segfaulted.

**fix**: force `Backend = Io.Threaded` in `src/main.zig:62`. threaded futex
uses direct kernel syscalls that work from any execution context. all 0.16 API
benefits are preserved; `io.concurrent` under Threaded spawns real OS threads.

**future path**: either migrate frame workers to `io.concurrent`, or give
cross-boundary modules a dedicated `Threaded` sync_io for mutex operations.
then Evented can be re-enabled for the network I/O layer.

> **2026-07 update**: this "future path" was tried and abandoned. Evented ran in
> production twice and was shelved for good at `e6cdf84` (2026-04-09) — not for
> this crash, which was fixed, but for an untraced ~10-15% coverage regression.
> the durable fix is making `pg.Pool` Io-agnostic, not re-enabling Evented.
> see [evented-attempt.md](evented-attempt.md).

### crash 2: pool acquire panic under load (fixed in `e5ed0d1`)

**symptom**: `unreachable` panic in `Io.Event.waitTimeout` after 30-60s of
processing. stack: `event_log.zig uidForDid → pg pool.zig acquire → Io.zig`

**cause**: `pg.zig` pool used `Io.Event` + `reset()` as a connection-available
signal. `Io.Event.reset()` has a stdlib invariant (Io.zig:1857): **assumes no
pending call to wait**. with 16 frame workers contending for 5 connections:
`set()` wakes all waiters, one calls `reset()`, others hit `unreachable`.

**fix (pg.zig `5ce2355` on dev branch)**: replaced `Io.Event` with a monotonic
`u32` futex counter. `release()` increments counter + `futexWake(1)` (wake one).
`acquire()` snapshots counter under mutex + `futexWaitTimeout()` with snapshot.
no `reset()`, no single-waiter constraint. also fixed a pre-existing double-lock
race in acquire and made zlay's pool size configurable via `DB_POOL_SIZE` env
(default 20, was hardcoded 5).

### crash 3: GPF in websocket client write (fixed in `0f11cfc`)

**symptom**: general protection exception in `memcpy → Writer.zig → writeAll →
websocket client.zig writeFrame`. happens after 30-60s of normal processing.

**cause**: the websocket `Client` struct had no write serialization. three
concurrent writers on the same client:

1. `pingLoop` — `client.writeFrame(.ping, ...)` every 30s (`subscriber.zig:378`)
2. `readLoop` auto-pong — `self.writeFrame(.pong, ...)` on upstream ping
   (inside the library, `client.zig:272`)
3. close path — `client.writeFrame(.close, ...)` on failure (`subscriber.zig:383`)

interleaved frame headers/payloads corrupt the shared stream/TLS writer state.
the server-side `Conn` already had `lock: Io.Mutex` around all writes; the
client was missing it.

**fix (websocket.zig `0261b7d`)**: added `_write_lock: Io.Mutex` to `Client`,
acquired in `writeFrame()` around the two `writeAll` calls. header construction
and payload masking happen outside the lock.

since zat also depends on websocket.zig, a new zat alpha (`v0.3.0-alpha.9`) was
cut to resolve the diamond dependency.

### crash 4: GPF in pingLoop after connection teardown (fixed in `6d6c832`)

**symptom**: GPF in `memcpy → Writer.zig → writeAll → client.zig writeFrame`
every ~60-90s of processing. same stack trace as crash 3 but different root
cause — the write lock from crash 3 was necessary but not sufficient.

**cause**: `pingLoop` runs as an `io.concurrent` task and sleeps in 1s
increments. when `readLoop` returns (connection dies), the defer chain runs:
1. `ping_future.cancel(self.io)` — requests cancellation
2. `client.deinit()` — frees stream/TLS buffers

but `pingLoop` did `self.io.sleep(...) catch {}` — swallowing all errors
including `error.Canceled`. so `cancel()` could not stop the task. `deinit()`
then freed the client while `pingLoop` was still running. next iteration's
`writeFrame(.ping, ...)` hit freed memory → GPF.

**fix (zlay `6d6c832` + websocket.zig `104608b`)**:
- `io.sleep(...) catch {}` → `catch return` — makes pingLoop
  cancellation-cooperative. cancel() can now stop the task before deinit() runs.
- added `Client.isClosed()` check before `writeFrame` — defense-in-depth
  against writes to dead connections.

### fix 5: httpFallback dispatch for health probes (fixed in websocket.zig `4222f98`)

**symptom**: health probes on port 3000 return 400 instead of reaching the
application handler. k8s rollout fails.

**cause**: the websocket server's `_handleHandshake` sent 400 on any
non-upgrade HTTP request. there was no code path to detect "valid HTTP but
not a websocket upgrade" and dispatch to `Handler.httpFallback()`.

**fix (websocket.zig `4222f98`)**: intercepts `MissingHeaders`,
`InvalidConnection`, and `InvalidUpgrade` errors from `Handshake.parse()`.
for these (and only these), a standalone `parseHttpRequest()` re-parses the
raw buffer and dispatches to `H.httpFallback()` if it exists (comptime
`hasFn` check). other handshake errors (e.g. `InvalidVersion`) still get 400.
10 tests cover all request patterns.

### crash 6: SIGSEGV — plain threads calling Evented Io.Mutex (fixed in `6674812`)

**symptom**: SIGSEGV at startup after switching to `Io.Evented` backend. crash in
`Uring.zig` at `Thread` struct field offset from NULL.

**cause**: the resyncer was spawned via `io.concurrent()` which creates an Evented
fiber. but resync work calls `DiskPersist` methods that take
`self.mutex.lockUncancelable(self.io)` where `self.io` is `pool_io` (Threaded).
Threaded futex from an Evented fiber dereferences `Thread.current()` which is a
threadlocal only set on Uring-managed threads — NULL on Evented fibers.

**fix**: run resyncer on a plain `std.Thread` with `pool_io`, not as an Evented
fiber. the thread checks `shutdown_flag` to exit.

### fix 7: startup connection ramp throttling (fixed in `f9bf515`)

**symptom**: event loop starvation during initial PDS connection burst (~2,750
simultaneous connects).

**fix**: throttled startup to connect in batches, preventing io_uring submission
queue overflow.

### crash 8: cross-Io heap corruption in GC + health checks (fixed in `2156d08` + `b433403`)

**symptom**: steady-state heap corruption / SIGSEGV with zero downstream consumers.
dmesg shows crash addresses in `Uring.zig` at `Thread` struct field offsets from
NULL — same signature as crash 6 but in the GC and health check paths.

**cause**: two cross-Io violations active during steady-state operation:

1. **GC loop** ran as an Evented fiber (`io.concurrent(gcLoop, ...)`) but called
   `dp.gc()` which takes `self.mutex.lockUncancelable(self.io)` where `self.io`
   is `pool_io` (Threaded), and queries `pg.Pool` (also Threaded). Threaded futex
   from Evented fiber → NULL `Thread.current()` → heap corruption.

2. **health check endpoints** (`/_readyz`, `/_health`, `/xrpc/_health`) on both the
   metrics server and API router executed `db.exec("SELECT 1")` through `pg.Pool`
   (Threaded) from Evented HTTP handler context. same cross-Io violation.

**fix**:
- GC loop moved from `io.concurrent()` to `std.Thread.spawn(.{}, gcLoop, .{&dp, pool_io})`.
  the thread is joined during shutdown before `dp.deinit()` runs (dp is stack-owned).
- health checks replaced with `isDbHealthy()` — reads an atomic `last_db_success`
  timestamp set by Threaded workers after successful DB queries. safe from any Io.
- `markDbSuccess()` called from `uidForDid()` (every incoming event) and `gc()`
  (every 10 minutes).

### fix 8: broadcaster double-destroy (fixed in `72ba680`)

**symptom**: use-after-free in broadcaster. `broadcast()` was destroying consumers
that `Handler.close()` still referenced.

**cause**: `broadcast()` detected dead consumers (via `alive` atomic) and called
`consumer.shutdown()` + `self.allocator.destroy(consumer)` inline. but the
consumer's `Handler.close()` callback was still running and would later call
`removeConsumer()` on the already-freed pointer.

**fix**: `broadcast()` now only does `swapRemove` + count decrement for dead
consumers. `removeConsumer()` (called from `Handler.close()`) is the sole owner
of `shutdown()` + `destroy()`.

## the cross-Io rule

the single most important lesson from this migration:

**`Io.Mutex`, `Io.Condition`, `io.sleep()`, and any `pg.Pool` operation must be
called from the same Io type they were initialized with.**

- Threaded futex on Evented fiber → dereferences NULL `Thread.current()` threadlocal
  → SIGSEGV or heap corruption (crashes 1, 6, 8)
- Evented futex on plain thread → same NULL deref in the other direction

**safe cross-Io patterns**: raw atomics (`Value`, `fetchAdd`, CAS), `tryLock`
(non-blocking CAS, no futex), MPSC ring buffers with atomic spinlocks.

**unsafe cross-Io patterns**: `Io.Mutex.lockUncancelable(wrong_io)`,
`Io.Condition`, `io.sleep(wrong_io, ...)`, `pg.Pool` queries from wrong context.

**fix pattern**: components that use Threaded resources (mutexes, pg.Pool) must
run on plain `std.Thread`, not Evented `io.concurrent()`. examples: resyncer
(`439c678`), GC loop (`2156d08`).

**known remaining**: XRPC and admin API handlers run on Evented fibers and access
`pg.Pool` (Threaded) on external HTTP requests. not steady-state, but real.
fixing requires either running API handlers on pool_io or making pg.Pool Io-agnostic.

## stdlib patches

six stdlib issues required patches or workarounds to run on the Evented
(io_uring) backend. see **[docs/stdlib-patches.md](docs/stdlib-patches.md)**
for the full writeup: what's patched, what's worked around, and what upstream
changes would let us drop each one.

summary:

| # | issue | fix type |
|---|---|---|
| 1 | Uring networking stubs (`error.NetworkDown`) | patch (`patches/uring-networking.patch`) |
| 2 | DNS resolution (`netLookup`) missing | Threaded fallback in subscriber |
| 3 | ReleaseSafe GPF in fiber context | build with `ReleaseFast` |
| 4 | `debug_io` single-threaded default | `std_options_debug_threaded_io` override |
| 5 | `Io.Event.reset()` single-waiter panic | futex counter in pg.zig fork |
| 6 | cross-Io Mutex/futex incompatibility | Io segregation (plain threads for Threaded work) |

## where things live

### repos and their roles

| repo | location | what |
|---|---|---|
| **zlay** | `tangled.org:zzstoatzz.io/zlay` | the relay. this repo. |
| **zat** | `tangled.org:zat.dev/zat` | AT Protocol primitives (CBOR, CAR, firehose, DID, etc.) |
| **websocket.zig** | `tangled.org:zzstoatzz.io/websocket.zig` (canonical, pinned `v0.1.8`) | fork of karlseguin/websocket.zig |
| **pg.zig** | `github.com/zzstoatzz/pg.zig` (dev branch) | fork of karlseguin/pg.zig |
| **rocksdb-zig** | `github.com/zzstoatzz/rocksdb-zig` | RocksDB bindings |
| **relay ops** | `tangled.org:zzstoatzz.io/relay` | helm chart, terraform, justfile |

### zlay source map

| file | LOC | role |
|---|---|---|
| `main.zig` | 404 | Io bootstrap, signal handlers, GC loop, MetricsServer |
| `subscriber.zig` | ~1021 | per-host PDS connection: connect, readLoop, pingLoop, backoff |
| `slurper.zig` | ~663 | multi-host crawl manager: host discovery, worker lifecycle |
| `broadcaster.zig` | ~1319 | downstream consumer management, broadcast, history, stats |
| `frame_worker.zig` | ~311 | per-frame processing: CBOR decode, validate, persist, broadcast |
| `thread_pool.zig` | ~357 | key-partitioned worker pool (plain `std.Thread`, **not** io.concurrent) |
| `event_log.zig` | ~1446 | disk persistence (indigo-compatible) + Postgres index |
| `validator.zig` | ~1002 | DID resolution, signature verification, validation cache |
| `lru.zig` | ~289 | generic LRU cache (used by validator + event_log) |
| `ring_buffer.zig` | ~256 | per-consumer ring buffer for broadcast fanout |
| `collection_index.zig` | ~200 | RocksDB collection index (listReposByCollection) |
| `backfill.zig` | ~396 | collection index backfill from source relay |
| `resync.zig` | ~315 | collection index resync on #sync events |
| `cleaner.zig` | ~114 | stale entry cleanup from collection index |
| `api/router.zig` | 142 | HTTP request dispatch (health, XRPC, admin) |
| `api/xrpc.zig` | — | XRPC endpoint handlers |
| `api/admin.zig` | — | admin endpoint handlers |
| `api/http.zig` | — | HTTP response helpers |

### concurrency architecture

> **as of `e6cdf84` (2026-04-09) `Backend = Io.Threaded` (`src/main.zig:62`),
> so there are no fibers.** `io.concurrent()` under Threaded spawns real OS
> threads, and the cross-Io hazard below is dormant — every runtime in the
> process is Threaded. the layering is kept because it still describes what runs
> where, and because it is the shape any future fiber attempt has to reproduce.

three layers, originally shaped by the cross-Io constraint:

**`io.concurrent` on the primary backend** (was Evented fibers, now Threaded
OS threads)
- upstream PDS subscribers (read loops, ping loops)
- downstream consumer write loops
- DID resolver loops (validator)
- broadcast loop (drains queue, fans out to consumers)
- slurper coordination
- metrics server, backfill, cleaner

**plain `std.Thread` with `pool_io` (`Io.Threaded`)**
- GC loop — uses `DiskPersist.mutex` + `pg.Pool` (crash 8)
- resyncer — uses `DiskPersist` + HTTP client (crash 6)
- these MUST NOT run as fibers (see "the cross-Io rule") — still true if fibers
  ever return, even though `pool_io` is redundant today

**CPU-bound ordered processing → explicit `std.Thread` workers**
- `thread_pool.zig`: `workers[host_id % N]` ensures per-key FIFO ordering
- `frame_worker.zig`: CBOR decode, DID resolution, signature verify, DB persist
- bounded backpressure: blocking submit when queue full → TCP backpressure
- uses `Io.Mutex` / `Io.Condition` with `pool_io` (Threaded futex)

### dependency versions

current, from `build.zig.zon` at `863a6ef` (zlay 0.0.5):

```
zat           v0.3.20         (tangled.org)
websocket.zig v0.1.11         (tangled.org)
pg.zig        5ce2355         (github, dev branch)
rocksdb-zig   cdef67b         (github)
zig           0.16.0          (release; no Uring networking patch applied)
```

historical, at `b433403` during the Evented attempt:

```
zat           v0.3.0-alpha.16 (tangled.org)
websocket.zig 80c6434         (github, master)
zig           0.16.0-dev.3059+42e33db9d (patched Uring networking)
```

### key env vars

| var | default | what |
|---|---|---|
| `RELAY_PORT` | 3000 | WebSocket firehose + HTTP API |
| `RELAY_METRICS_PORT` | 3001 | internal metrics + health probes |
| `FRAME_WORKERS` | 16 | CPU worker threads for frame processing |
| `FRAME_QUEUE_CAPACITY` | 4096 | per-worker queue depth |
| `DB_POOL_SIZE` | 20 | Postgres connection pool size (new) |
| `RELAY_UPSTREAM` | bsky.network | seed host for PDS discovery |
| `RELAY_DATA_DIR` | data/events | disk persistence directory |
| `RELAY_RETENTION_HOURS` | 72 | event log retention |
| `RELAY_MAX_EVENTS_GB` | 100 | max disk usage |
| `DATABASE_URL` | postgres://relay:relay@localhost:5432/relay | Postgres connection |

## what needed to happen next (2026-04, resolved)

> items 1 and 2 are done — `b433403` shipped and 0.16 has been in production
> since. item 3 was **superseded**: XRPC/admin no longer cross Io boundaries,
> because there is only one Io type now (Threaded). the underlying cause —
> `pg.Pool` not being Io-agnostic — is untouched and is now the main
> prerequisite for any future fiber work. item 4's "Evented viability" line is
> closed; see [evented-attempt.md](evented-attempt.md).

1. **deploy `b433403`** — all eight crashes/fixes are in. the steady-state
   heap corruption (cross-Io GC + health checks) is fixed. broadcaster
   double-destroy is fixed. health probes use atomic flag, not cross-Io DB query.

2. **monitor after deploy** — compare against 0.15 baseline:
   - thread count (2,903 on 0.15 — should be similar under Threaded)
   - memory (24.9 GiB VmSize, 1.44 GiB RSS on 0.15)
   - throughput, reconnect behavior, ConsumerTooSlow rate
   - verify zero crashes after extended run (hours). the GC cross-Io bug
     fired every 10 minutes, so 30+ minutes clean = strong signal.
   - watch `/_health` — should report healthy once `uidForDid` or `gc()` succeeds
     (within 10 minutes of startup at latest)

3. **known issue — XRPC/admin cross-Io** (not blocking deploy):
   - API handlers run on Evented fibers but some query `pg.Pool` (Threaded)
   - only triggered by external HTTP requests, not steady-state
   - fix: either run API handlers on pool_io, or make pg.Pool Io-agnostic

4. **follow-up work**:
   - ~~investigate Evented backend viability (frame workers → io.concurrent?)~~
     — **closed 2026-04-09** (`e6cdf84`), see [evented-attempt.md](evented-attempt.md)
   - consider upstreaming the client write lock to karlseguin/websocket.zig
   - consider upstreaming Uring networking patch (zig#31723) — still relevant;
     upstream `Uring.zig` networking is still stubbed as of 0.16.0 release
   - **evaluate whether pg.Pool can be made Io-agnostic** (would eliminate
     cross-Io issues) — still open, and the highest-leverage item here: it is
     the prerequisite for any fiber-based backend, stdlib or third-party