From 3f6c85c3322d75f36ce60ac8ede4655877285202 Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Thu, 13 Aug 2026 22:48:21 -0500 Subject: [PATCH] devlog 015: rewrite around the arc and the DVR MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit the previous draft read like release notes. what actually happened between 014 (june 30, 'zat lives there' as thesis) and now: stream went from first commit to carrying the network, and the archive turned jetstream from radio into DVR — seeking too cheap to meter, which deletes the always-on obligation for every consumer. that's the story; the releases and the split serve it. Co-Authored-By: Claude Fable 5 --- devlog/015-the-service-line.md | 192 ++++++++++++++++++--------------- 1 file changed, 104 insertions(+), 88 deletions(-) diff --git a/devlog/015-the-service-line.md b/devlog/015-the-service-line.md index 1c1638a..68ef4b2 100644 --- a/devlog/015-the-service-line.md +++ b/devlog/015-the-service-line.md @@ -1,97 +1,113 @@ # the service line -on august 13th bluesky launched [Protocol Services](https://bsky.network) — -jetstream v2, official, with network replay. tens of billions of records -behind `planSnapshot`, live tails on `subscribeEvents`, TS and Go SDKs with -the announcement. the design comes from -[@calabro.io](https://bsky.app/profile/calabro.io) & team at bluesky, and -the launch is the moment it became a public contract. - -zat's stake in this is [stream.waow.tech](https://stream.waow.tech) — a -zig port of that design, built on zat, held to the upstream repo by a -differential oracle that drives the upstream team's own client against it -on every admission run. port discipline means you re-pin deliberately when upstream -moves, and launch day was the move: the waypoint went to v0.2.0 the same -evening, the one server-side delta (end-to-end kinds filtering) ported and -gated before midnight. keeping pace with a release, same-day, with the -released client as referee — that's what the discipline is for. - -so, was this the biggest test of zat yet? yes — but not in the way a -benchmark is a test. the test was a week of operating a public firehose -under an instrument that refuses to be polite. - -## what the week actually tested - -[relay-eval](https://relay-eval.waow.tech) samples every sync source on the -network for five minutes out of every thirty and publishes the coverage. -it does not care that your unit tests pass. four zat releases in six days -were each bought by something it (or the operating of stream) surfaced: - -- **0.3.27** — reconnect backoff compounded on single-host consumers; a - relay that dropped every ~113s left a tail down a third of the time. -- **0.3.29** — a relay held the socket open and went silent for 2.5 - minutes; both streaming clients grew an idle watchdog, and the first - implementation was rewritten because `SO_RCVTIMEO` panics under - `std.Io.Threaded` in debug builds. -- **0.3.30** — a peer accepted a connection and never answered; - `receiveHead` waited forever. every phase of a fetch is now bounded. - -none of these are the kind of bug a library finds in itself. they are the -kind a library finds when it is the load-bearing wall of a service that -someone graphs in public. +devlog 014 ended with "zat lives there" — there being the layer below +product taste, the relays and indexers and verifiers that move bytes and +decide what deserves trust. that was june 30th, and it was a thesis. + +eight days later, [stream](https://tangled.org/zat.dev/stream) got its +first commit. between that entry and this one, the thesis got a body: a +zig port of jetstream v2 — the archive-bearing design from +[@calabro.io](https://bsky.app/profile/calabro.io) & team at bluesky — +built on zat, bootstrapped across the whole network, and now serving +tens of billions of events at [stream.waow.tech](https://stream.waow.tech) +with sealed segments back to its first sequence. it is held to the +upstream repo by a differential oracle that drives the upstream team's own +client against it on every admission run. on august 13th bluesky launched +[Protocol Services](https://bsky.network) and jetstream v2 became a public +contract; the waypoint moved to the release tag the same evening, the one +server-side delta ported and gated before midnight. that same-day re-pin, +refereed by the released client, is what port discipline is for. + +so this entry covers six weeks, four zat releases, and one architectural +split. but the thing actually worth writing down is what the archive does +to the *consumers*. + +## DVR for the firehose + +jetstream has always been radio. tune in and you hear the network; miss +an hour and that hour is gone — cursor replay reaches back a day or so, +and past that, nothing. every consumer built against radio inherits the +same obligation: **be always-on, or be wrong.** a crashed listener is a +gap in your state forever. a new consumer starts from nothing. history is +someone else's problem. + +the archive turns radio into DVR, and the part that changed this month is +that seeking got too cheap to meter. `planSnapshot` takes collections, +DIDs, kinds, and a sequence window, and prunes the plan to just the +blocks that can match — a two-hour window of one sparse collection plans +~25 blocks, a few megabytes. replaying a specific streamplace broadcast's +chat from three days ago: 39 seconds end-to-end, archive pass plus live +cutover, in the [SDK example](https://tangled.org/zat.dev/jetstream). +rewind, replay, catch up to live, keep listening — one seam, at-least-once +across it, dedup by seq. + +that quietly deletes the always-on obligation, and with it a whole class +of consumer architecture: + +- a tally-keeper like pollz doesn't have to *trust* a count it + accumulated while hoping it never missed a vote — it can recompute from + the poll's creation seq, any time, and the live tail becomes an + optimization instead of a correctness requirement. state becomes a pure + function of the archive. +- an index like typeahead's can cold-start by draining its collections + instead of aging into usefulness. +- a crashed consumer is no longer bleeding data — it's paused. resume is + a bounded plan from your last seq, then cutover. +- and things that were never possible on radio at all: replaying a chat + from before your service existed, backfilling a feature you hadn't + thought to build yet, auditing what a repo did last tuesday. + +bluesky's hosted instances gate their archive behind an API key; +stream.waow.tech grew the same gate, byte-compatible. the SDKs — their TS +and Go, and now [zat.dev/jetstream](https://tangled.org/zat.dev/jetstream) +in zig — all speak the same recipe: plan, drain, resume at +`plannedThroughSeq`. + +## what six weeks of production bought zat + +[relay-eval](https://relay-eval.waow.tech) samples every sync source on +the network for five minutes out of every thirty and publishes the +coverage. it does not care that your unit tests pass. four releases were +each bought by something it, or operating stream, surfaced: compounding +reconnect backoff (0.3.27), silent-socket stalls needing an idle watchdog +(0.3.29, rewritten once because `SO_RCVTIMEO` panics under +`std.Io.Threaded`), unbounded `receiveHead` waits (0.3.30). none are bugs +a library finds in itself. they are what a library learns as the +load-bearing wall of a service someone graphs in public. ## the split -the launch also settled an architecture question by example. bluesky -shipped `@bsky/sdk` split out of `@atproto/api`, "giving the protocol the -space it deserves as a universal technology." their line: protocol on one -side, bluesky services on the other. - -zat was blurring that exact line. `ArchiveBackfill` speaks -`network.bsky.jetstream.*` — a service API — and decodes jss v1, which is -jetstream's storage format. it vendored libzstd to do it. none of that is -atproto. - -so v0.4.0 draws the line the same way they did: `ArchiveBackfill`, the -jss decode, and the vendored zstd moved to -[zat.dev/jetstream](https://tangled.org/zat.dev/jetstream), the zig -jetstream service SDK, which depends on zat for everything protocol. -`JetstreamClient` (the v1 wire) stays for now — its consumers are real -and unbroken — but it's deprecated-in-home, and the `subscribeEvents` v2 -client will be born on the right side of the line. zat links no -compression library at all anymore. the protocol side got smaller, which -is how you know the cut was in the right place. - -## one more thing the week kept hand-rolling - -somewhere in all this, the same forty lines of RFC 3339 parsing appeared -for the fourth time — two byte-identical formatters in stream, a third -hiding in its server, a 45-line parser in the SDK example. the atproto -datetime string is not "time utils"; it's a pinned syntax profile, the -same family as `Tid` and `Nsid`, and zat already carried the interop -fixtures that define it — as test-only validation. - -v0.4.0 promotes that validator to `zat.Datetime`: parse to unix-epoch -microseconds (offsets normalized, the spec's year-zero floor enforced on -the *normalized* instant, so an offset can't smuggle one in), format to -the canonical `.ffffffZ`. the civil math is hinnant's days_from_civil, -because inventing calendar arithmetic in 2026 is how you get bugs with -anniversaries. the profile's divergences from general RFC 3339 — reject -`:60`, reject `-00:00`, upper-case `T`/`Z` only — were cross-checked -against rust's jiff and chrono before being committed to, so they're -decisions with receipts, not accidents. the interop fixtures now drive -the real parser, and four hand-rolled copies are one type. +the launch settled an architecture question by example: bluesky shipped +`@bsky/sdk` split out of `@atproto/api`, protocol on one side, services +on the other. zat was blurring that exact line — `ArchiveBackfill` speaks +`network.bsky.jetstream.*` and decodes jetstream's storage format, none +of which is atproto. v0.4.0 draws the same line they did: the archive +client, the jss decode, and the vendored zstd moved to +[zat.dev/jetstream](https://tangled.org/zat.dev/jetstream); the v1 live +client stays until its consumers migrate; the v2 client will be born on +the right side. zat links no compression library at all now. the protocol +side got smaller, which is how you know the cut was in the right place. + +(v0.4.0 also picked up `zat.Datetime` — the atproto datetime profile +promoted from test-only fixture validation to a real parse/format type, +after the same forty lines of RFC 3339 handling turned up hand-rolled for +the fourth time across these repos. edge-case decisions cross-checked +against rust's jiff and chrono; civil math is hinnant's, because +inventing calendar arithmetic in 2026 is how you get bugs with +anniversaries.) ## the honest ending the eval is still finding things. connections through stream's front door -die at a low background rate that survived a day of instrumentation — -server exonerated, kernel exonerated, the proxy hop now under logging. -that hunt continues, and it will probably buy 0.4.1 or a stream fix or -both. that's the deal with instruments that refuse to be polite: they -don't stop when you'd like a quiet week. - -biggest test yet. passed where it counts, still running, and the design -being tested against was a gift — thanks to -[@calabro.io](https://bsky.app/profile/calabro.io) & the protocol services -team. +die at a low background rate that has so far survived a day of +instrumentation — server exonerated, kernel exonerated, the proxy hop now +under logging. the hunt continues, and it will probably buy 0.4.1 or a +stream fix or both. that's the deal with instruments that refuse to be +polite: they don't stop when you'd like a quiet week. + +fourteen entries ago this devlog was parsing identifiers. the last one +claimed zat belongs to the layer that moves bytes and decides what to +trust. this one gets to say: the network has a DVR now, zat is +load-bearing in it, and the design being ported was a gift — thanks to +[@calabro.io](https://bsky.app/profile/calabro.io) & the protocol +services team. -- 2.51.2