zlay 0.16 migration — status and known issues #

last updated: 2026-07-30 (migration content below is from 2026-04-04)

the 0.16 migration is done and deployed. zlay runs 0.16.0 release on Io.Threaded; current state is zlay 0.0.5 at 863a6ef. this document is retained for the eight crash write-ups and the cross-Io rule, which remain the best record of why the architecture looks the way it does — but read the per-section notes: anything describing Io.Evented as live is history, shelved at e6cdf84 (2026-04-09). see evented-attempt.md.

historical context, at time of writing:

• stable production build: a931853 (zig 0.15)
• latest 0.16 build: b433403 (not yet deployed — all fixes through crash 8)

what happened #

zlay is an AT Protocol relay (~8,400 LOC, 15 source files). it crawls ~2,750 PDS instances, validates frames, persists to disk, and fans out to downstream consumers over WebSocket. the zig 0.16 migration rewrote all I/O to use std.Io primitives.

eight separate crashes/bugs were found and fixed during deployment attempts:

crash 1: SIGSEGV on startup (fixed in f996812) #

symptom: exit code 139 immediately after printing "io backend: Evented"

cause: the build auto-selected Io.Evented (io_uring on linux). evented futex wait/wake access fiber-local Thread.current() state. but zlay's frame pool (thread_pool.zig) spawns plain std.Thread workers — they don't have fiber state. any Io.Mutex operation from a plain thread segfaulted.

fix: force Backend = Io.Threaded in src/main.zig:62. threaded futex uses direct kernel syscalls that work from any execution context. all 0.16 API benefits are preserved; io.concurrent under Threaded spawns real OS threads.

future path: either migrate frame workers to io.concurrent, or give cross-boundary modules a dedicated Threaded sync_io for mutex operations. then Evented can be re-enabled for the network I/O layer.

2026-07 update: this "future path" was tried and abandoned. Evented ran in production twice and was shelved for good at e6cdf84 (2026-04-09) — not for this crash, which was fixed, but for an untraced ~10-15% coverage regression. the durable fix is making pg.Pool Io-agnostic, not re-enabling Evented. see evented-attempt.md.

crash 2: pool acquire panic under load (fixed in e5ed0d1) #

symptom: unreachable panic in Io.Event.waitTimeout after 30-60s of processing. stack: event_log.zig uidForDid → pg pool.zig acquire → Io.zig

cause: pg.zig pool used Io.Event + reset() as a connection-available signal. Io.Event.reset() has a stdlib invariant (Io.zig:1857): assumes no pending call to wait. with 16 frame workers contending for 5 connections: set() wakes all waiters, one calls reset(), others hit unreachable.

fix (pg.zig 5ce2355 on dev branch): replaced Io.Event with a monotonic u32 futex counter. release() increments counter + futexWake(1) (wake one). acquire() snapshots counter under mutex + futexWaitTimeout() with snapshot. no reset(), no single-waiter constraint. also fixed a pre-existing double-lock race in acquire and made zlay's pool size configurable via DB_POOL_SIZE env (default 20, was hardcoded 5).

crash 3: GPF in websocket client write (fixed in 0f11cfc) #

symptom: general protection exception in memcpy → Writer.zig → writeAll → websocket client.zig writeFrame. happens after 30-60s of normal processing.

cause: the websocket Client struct had no write serialization. three concurrent writers on the same client:

1. pingLoop — client.writeFrame(.ping, ...) every 30s (subscriber.zig:378)
2. readLoop auto-pong — self.writeFrame(.pong, ...) on upstream ping (inside the library, client.zig:272)
3. close path — client.writeFrame(.close, ...) on failure (subscriber.zig:383)

interleaved frame headers/payloads corrupt the shared stream/TLS writer state. the server-side Conn already had lock: Io.Mutex around all writes; the client was missing it.

fix (websocket.zig 0261b7d): added _write_lock: Io.Mutex to Client, acquired in writeFrame() around the two writeAll calls. header construction and payload masking happen outside the lock.

since zat also depends on websocket.zig, a new zat alpha (v0.3.0-alpha.9) was cut to resolve the diamond dependency.

crash 4: GPF in pingLoop after connection teardown (fixed in 6d6c832) #

symptom: GPF in memcpy → Writer.zig → writeAll → client.zig writeFrame every ~60-90s of processing. same stack trace as crash 3 but different root cause — the write lock from crash 3 was necessary but not sufficient.

cause: pingLoop runs as an io.concurrent task and sleeps in 1s increments. when readLoop returns (connection dies), the defer chain runs:

1. ping_future.cancel(self.io) — requests cancellation
2. client.deinit() — frees stream/TLS buffers

but pingLoop did self.io.sleep(...) catch {} — swallowing all errors including error.Canceled. so cancel() could not stop the task. deinit() then freed the client while pingLoop was still running. next iteration's writeFrame(.ping, ...) hit freed memory → GPF.

fix (zlay 6d6c832 + websocket.zig 104608b):

• io.sleep(...) catch {} → catch return — makes pingLoop cancellation-cooperative. cancel() can now stop the task before deinit() runs.
• added Client.isClosed() check before writeFrame — defense-in-depth against writes to dead connections.

fix 5: httpFallback dispatch for health probes (fixed in websocket.zig 4222f98) #

symptom: health probes on port 3000 return 400 instead of reaching the application handler. k8s rollout fails.

cause: the websocket server's _handleHandshake sent 400 on any non-upgrade HTTP request. there was no code path to detect "valid HTTP but not a websocket upgrade" and dispatch to Handler.httpFallback().

fix (websocket.zig 4222f98): intercepts MissingHeaders, InvalidConnection, and InvalidUpgrade errors from Handshake.parse(). for these (and only these), a standalone parseHttpRequest() re-parses the raw buffer and dispatches to H.httpFallback() if it exists (comptime hasFn check). other handshake errors (e.g. InvalidVersion) still get 400. 10 tests cover all request patterns.

crash 6: SIGSEGV — plain threads calling Evented Io.Mutex (fixed in 6674812) #

symptom: SIGSEGV at startup after switching to Io.Evented backend. crash in Uring.zig at Thread struct field offset from NULL.

cause: the resyncer was spawned via io.concurrent() which creates an Evented fiber. but resync work calls DiskPersist methods that take self.mutex.lockUncancelable(self.io) where self.io is pool_io (Threaded). Threaded futex from an Evented fiber dereferences Thread.current() which is a threadlocal only set on Uring-managed threads — NULL on Evented fibers.

fix: run resyncer on a plain std.Thread with pool_io, not as an Evented fiber. the thread checks shutdown_flag to exit.

fix 7: startup connection ramp throttling (fixed in f9bf515) #

symptom: event loop starvation during initial PDS connection burst (~2,750 simultaneous connects).

fix: throttled startup to connect in batches, preventing io_uring submission queue overflow.

crash 8: cross-Io heap corruption in GC + health checks (fixed in 2156d08 + b433403) #

symptom: steady-state heap corruption / SIGSEGV with zero downstream consumers. dmesg shows crash addresses in Uring.zig at Thread struct field offsets from NULL — same signature as crash 6 but in the GC and health check paths.

cause: two cross-Io violations active during steady-state operation:

1. GC loop ran as an Evented fiber (io.concurrent(gcLoop, ...)) but called dp.gc() which takes self.mutex.lockUncancelable(self.io) where self.io is pool_io (Threaded), and queries pg.Pool (also Threaded). Threaded futex from Evented fiber → NULL Thread.current() → heap corruption.

2. health check endpoints (/_readyz, /_health, /xrpc/_health) on both the metrics server and API router executed db.exec("SELECT 1") through pg.Pool (Threaded) from Evented HTTP handler context. same cross-Io violation.

fix:

• GC loop moved from io.concurrent() to std.Thread.spawn(.{}, gcLoop, .{&dp, pool_io}). the thread is joined during shutdown before dp.deinit() runs (dp is stack-owned).
• health checks replaced with isDbHealthy() — reads an atomic last_db_success timestamp set by Threaded workers after successful DB queries. safe from any Io.
• markDbSuccess() called from uidForDid() (every incoming event) and gc() (every 10 minutes).

fix 8: broadcaster double-destroy (fixed in 72ba680) #

symptom: use-after-free in broadcaster. broadcast() was destroying consumers that Handler.close() still referenced.

cause: broadcast() detected dead consumers (via alive atomic) and called consumer.shutdown() + self.allocator.destroy(consumer) inline. but the consumer's Handler.close() callback was still running and would later call removeConsumer() on the already-freed pointer.

fix: broadcast() now only does swapRemove + count decrement for dead consumers. removeConsumer() (called from Handler.close()) is the sole owner of shutdown() + destroy().

the cross-Io rule #

the single most important lesson from this migration:

Io.Mutex, Io.Condition, io.sleep(), and any pg.Pool operation must be called from the same Io type they were initialized with.

• Threaded futex on Evented fiber → dereferences NULL Thread.current() threadlocal → SIGSEGV or heap corruption (crashes 1, 6, 8)
• Evented futex on plain thread → same NULL deref in the other direction

safe cross-Io patterns: raw atomics (Value, fetchAdd, CAS), tryLock (non-blocking CAS, no futex), MPSC ring buffers with atomic spinlocks.

unsafe cross-Io patterns: Io.Mutex.lockUncancelable(wrong_io), Io.Condition, io.sleep(wrong_io, ...), pg.Pool queries from wrong context.

fix pattern: components that use Threaded resources (mutexes, pg.Pool) must run on plain std.Thread, not Evented io.concurrent(). examples: resyncer (439c678), GC loop (2156d08).

known remaining: XRPC and admin API handlers run on Evented fibers and access pg.Pool (Threaded) on external HTTP requests. not steady-state, but real. fixing requires either running API handlers on pool_io or making pg.Pool Io-agnostic.

stdlib patches #

six stdlib issues required patches or workarounds to run on the Evented (io_uring) backend. see docs/stdlib-patches.md for the full writeup: what's patched, what's worked around, and what upstream changes would let us drop each one.

summary:

#	issue	fix type
1	Uring networking stubs (error.NetworkDown)	patch (patches/uring-networking.patch)
2	DNS resolution (netLookup) missing	Threaded fallback in subscriber
3	ReleaseSafe GPF in fiber context	build with ReleaseFast
4	debug_io single-threaded default	std_options_debug_threaded_io override
5	Io.Event.reset() single-waiter panic	futex counter in pg.zig fork
6	cross-Io Mutex/futex incompatibility	Io segregation (plain threads for Threaded work)

where things live #

repos and their roles #

repo	location	what
zlay	tangled.org:zzstoatzz.io/zlay	the relay. this repo.
zat	tangled.org:zat.dev/zat	AT Protocol primitives (CBOR, CAR, firehose, DID, etc.)
websocket.zig	tangled.org:zzstoatzz.io/websocket.zig (canonical, pinned v0.1.8)	fork of karlseguin/websocket.zig
pg.zig	github.com/zzstoatzz/pg.zig (dev branch)	fork of karlseguin/pg.zig
rocksdb-zig	github.com/zzstoatzz/rocksdb-zig	RocksDB bindings
relay ops	tangled.org:zzstoatzz.io/relay	helm chart, terraform, justfile

zlay source map #

file	LOC	role
main.zig	404	Io bootstrap, signal handlers, GC loop, MetricsServer
subscriber.zig	~1021	per-host PDS connection: connect, readLoop, pingLoop, backoff
slurper.zig	~663	multi-host crawl manager: host discovery, worker lifecycle
broadcaster.zig	~1319	downstream consumer management, broadcast, history, stats
frame_worker.zig	~311	per-frame processing: CBOR decode, validate, persist, broadcast
thread_pool.zig	~357	key-partitioned worker pool (plain std.Thread, not io.concurrent)
event_log.zig	~1446	disk persistence (indigo-compatible) + Postgres index
validator.zig	~1002	DID resolution, signature verification, validation cache
lru.zig	~289	generic LRU cache (used by validator + event_log)
ring_buffer.zig	~256	per-consumer ring buffer for broadcast fanout
collection_index.zig	~200	RocksDB collection index (listReposByCollection)
backfill.zig	~396	collection index backfill from source relay
resync.zig	~315	collection index resync on #sync events
cleaner.zig	~114	stale entry cleanup from collection index
api/router.zig	142	HTTP request dispatch (health, XRPC, admin)
api/xrpc.zig	—	XRPC endpoint handlers
api/admin.zig	—	admin endpoint handlers
api/http.zig	—	HTTP response helpers

concurrency architecture #

as of e6cdf84 (2026-04-09) Backend = Io.Threaded (src/main.zig:62), so there are no fibers. io.concurrent() under Threaded spawns real OS threads, and the cross-Io hazard below is dormant — every runtime in the process is Threaded. the layering is kept because it still describes what runs where, and because it is the shape any future fiber attempt has to reproduce.

three layers, originally shaped by the cross-Io constraint:

io.concurrent on the primary backend (was Evented fibers, now Threaded OS threads)

• upstream PDS subscribers (read loops, ping loops)
• downstream consumer write loops
• DID resolver loops (validator)
• broadcast loop (drains queue, fans out to consumers)
• slurper coordination
• metrics server, backfill, cleaner

plain std.Thread with pool_io (Io.Threaded)

• GC loop — uses DiskPersist.mutex + pg.Pool (crash 8)
• resyncer — uses DiskPersist + HTTP client (crash 6)
• these MUST NOT run as fibers (see "the cross-Io rule") — still true if fibers ever return, even though pool_io is redundant today

CPU-bound ordered processing → explicit std.Thread workers

• thread_pool.zig: workers[host_id % N] ensures per-key FIFO ordering
• frame_worker.zig: CBOR decode, DID resolution, signature verify, DB persist
• bounded backpressure: blocking submit when queue full → TCP backpressure
• uses Io.Mutex / Io.Condition with pool_io (Threaded futex)

dependency versions #

current, from build.zig.zon at 863a6ef (zlay 0.0.5):

zat           v0.3.20         (tangled.org)
websocket.zig v0.1.11         (tangled.org)
pg.zig        5ce2355         (github, dev branch)
rocksdb-zig   cdef67b         (github)
zig           0.16.0          (release; no Uring networking patch applied)

historical, at b433403 during the Evented attempt:

zat           v0.3.0-alpha.16 (tangled.org)
websocket.zig 80c6434         (github, master)
zig           0.16.0-dev.3059+42e33db9d (patched Uring networking)

key env vars #

var	default	what
RELAY_PORT	3000	WebSocket firehose + HTTP API
RELAY_METRICS_PORT	3001	internal metrics + health probes
FRAME_WORKERS	16	CPU worker threads for frame processing
FRAME_QUEUE_CAPACITY	4096	per-worker queue depth
DB_POOL_SIZE	20	Postgres connection pool size (new)
RELAY_UPSTREAM	bsky.network	seed host for PDS discovery
RELAY_DATA_DIR	data/events	disk persistence directory
RELAY_RETENTION_HOURS	72	event log retention
RELAY_MAX_EVENTS_GB	100	max disk usage
DATABASE_URL	postgres://relay:relay@localhost:5432/relay	Postgres connection

what needed to happen next (2026-04, resolved) #

items 1 and 2 are done — b433403 shipped and 0.16 has been in production since. item 3 was superseded: XRPC/admin no longer cross Io boundaries, because there is only one Io type now (Threaded). the underlying cause — pg.Pool not being Io-agnostic — is untouched and is now the main prerequisite for any future fiber work. item 4's "Evented viability" line is closed; see evented-attempt.md.

1. deploy b433403 — all eight crashes/fixes are in. the steady-state heap corruption (cross-Io GC + health checks) is fixed. broadcaster double-destroy is fixed. health probes use atomic flag, not cross-Io DB query.

2. monitor after deploy — compare against 0.15 baseline:

   • thread count (2,903 on 0.15 — should be similar under Threaded)
   • memory (24.9 GiB VmSize, 1.44 GiB RSS on 0.15)
   • throughput, reconnect behavior, ConsumerTooSlow rate
   • verify zero crashes after extended run (hours). the GC cross-Io bug fired every 10 minutes, so 30+ minutes clean = strong signal.
   • watch /_health — should report healthy once uidForDid or gc() succeeds (within 10 minutes of startup at latest)

3. known issue — XRPC/admin cross-Io (not blocking deploy):

   • API handlers run on Evented fibers but some query pg.Pool (Threaded)
   • only triggered by external HTTP requests, not steady-state
   • fix: either run API handlers on pool_io, or make pg.Pool Io-agnostic

4. follow-up work:

   • investigate Evented backend viability (frame workers → io.concurrent?) — closed 2026-04-09 (e6cdf84), see evented-attempt.md
   • consider upstreaming the client write lock to karlseguin/websocket.zig
   • consider upstreaming Uring networking patch (zig#31723) — still relevant; upstream Uring.zig networking is still stubbed as of 0.16.0 release
   • evaluate whether pg.Pool can be made Io-agnostic (would eliminate cross-Io issues) — still open, and the highest-leverage item here: it is the prerequisite for any fiber-based backend, stdlib or third-party