upstream test harness + ingest notes #

condensed study of bluesky-social/jetstream, written 2026-07-08.

Upstream pin: 289b0328c2e1a0ccf8c870cb45de0b2397de19fb (v0.2.0, the 2026-08-13 official release — Bluesky Protocol Services / Jetstream v2 with network replay). Server-side delta from the prior pin d4dd2f0 is PR #336 (end-to-end kinds filtering); the rest is client/SDK/docs. The suites below verify against that commit — see lines citing it further down. This is the pin most Stream docs share; note jss-format-v1.md and jss-seal-spec.md are instead on 0c45f29 (2026-07-08), 56 commits earlier.

You need an upstream checkout, and the paths below are upstream's, not ours. Everything in the simulator and oracle sections — cmd/simulator, segment/*.go, internal/subscribe/, specs/ — lives in upstream's tree. Clone it wherever you keep source and check out the pin:

git clone https://github.com/bluesky-social/jetstream
git -C jetstream checkout d4dd2f0127e8934d125b2a26a90fe88239ca98a2

On the maintainer's machine this sits at ~/github.com/bluesky-social/jetstream (the ~/{forge}/{org}/{repo} layout); nothing in this repository depends on that path.

The checkout answers questions not decidable from Stream alone: for example, git log 0c45f29..f29815c -- segment/ shows the per-block bloom right-sizing described in jss-seal-spec.md.

simulator — our ISP-friendly test network #

self-contained fake atproto network: PLC + one PDS + one relay under a single HTTP listener, default :7777. plain go, no nix: go run ./cmd/simulator serve (builds on macOS arm64, go 1.26).

endpoints (production XRPC paths, no auth, no TLS):

• GET /xrpc/com.atproto.sync.subscribeRepos — websocket firehose, ?cursor= replay from pebble ring (--firehose-history, default 10k events); too-old cursor → #info frame
• GET /xrpc/com.atproto.sync.listRepos (paginated), GET /xrpc/com.atproto.sync.getRepo?did= — backfill path, real signed CAR
• GET /did:... — minimal DID doc pointing back at --public-url

flags: --addr :7777, --data-dir ./data/simulator (pebble), --reset, --seed 42, --accounts 10000, --initial-records-per-account 5, --commits-per-sec 10, --traffic-rate-multiplier.

emits REAL bytes: signed commits, real MST, CAR diffs (Sync 1.1-verifiable), #commit/#identity/#account/#sync. no tooBig. deterministic-ish (seeded logical clock; bootstrap seeded separately from live traffic). adversarial/fault modes are programmatic (oracle-only), not CLI.

zig usage: point stream at ws://localhost:7777/xrpc/com.atproto.sync.subscribeRepos.

oracle (what an equivalent zig harness would check) #

driver boots real server against simulator, walks lifecycle gated on durable-append acks (never sleeps). the simulator world doubles as independent ground truth. contracts: seq ordering, final-state vs world, event-log equivalence (catches lost intermediate events), compaction watermark rules, replay==archive, fold-convergence, anti-vacuity on injected faults (every scheduled fault must prove it fired). tiers incl. crash/restart (SIGKILL real children at crashpoints), deterministic store faults (a selected real RocksDB mutation fails, then the same disk state is reopened), segment faults (real JSS write/fsync/rename calls fail at a selected process-wide ordinal, with exhaustive Patch/Rewrite sweeps and fault-free reopen), and power loss. Upstream models power loss with its strict in-memory VFS; Stream exercises the production Linux filesystem path with ext4 on a strict NBD write-cache backend. detection power measured by mutation campaign.

ingest / live consumer (upstream reference) #

• subscribe url: ws(s)://host/xrpc/com.atproto.sync.subscribeRepos?cursor=N; cursor persisted (pebble) is the verify-watermark (≤ any in-flight seq), inclusive, at-least-once
• decode: CBOR two-part frame; jetstream reads only ops + CAR blocks, never walks MST. verifier (Sync 1.1) mandatory upstream; zat has verifyCommitDiff.
• conversion (pure function): #commit → one internal event per op {witnessed_at µs shared per upstream event, kind create/update/delete, did, collection, rkey, rev, payload=raw DAG-CBOR block}. #identity/#account → single event, payload = envelope CBOR. #sync → tombstone row + resync rows (not on v1 wire).
• validation gate (drop, count, never crash): non-TID rev → whole event (no log line — hostile input must not drive log volume); invalid NSID/rkey → that op; missing CAR block for create/update → that op (partial CARs are spec-legal upstream omissions — no-op updates and non-canonical PDSs — not local corruption; survivors still archived), and ONLY this reason logs: one warn per affected event with seq/count/did, mirroring upstream's processBatch. metrics: dropped_events_total{reason=invalid_rev|invalid_collection|invalid_rkey|field_too_long|missing_block}
• field limits (upstream columnar format): did ≤65535, collection/rkey/rev ≤255, payload ≤u32
• invariants: fsync data before committing cursor; seq assigned at append under writer mutex, starts at 1 (0 = sentinel); per-DID order preserved; never crash on upstream data / crash loud on own corruption
• readable log (hot tail): writer-owned deque, deep-copied entries at seq allocation, 256MiB budget, evicts only below durable watermark (pinned above), notify channel per append, encode-once wire memo per entry shared across fan-out; cursor below floor → cold reader over sealed segments. Stream's tail enforces the same pin: an undurable suffix is retained even when it overruns the byte budget, publishDurable releases only the newly durable byte suffix before eviction, and scrape-time readable/pinned/overrun gauges are O(1) accumulators rather than a scan under the hot lock. Its physical deque retires prefix entries in O(1) and compacts only after a large amortized prefix; the 20,000-row burst regression catches the former orderedRemove(0) implementation that made an 8,192-row repair quadratic and stalled global ingest.
• subscribe retention and slow-client controls: the configured read-log byte budget is the Tail.max_bytes used by that physical eviction path. Each real subscriber receives the configured sustained window and fractional minimum log-scan rate; caught-up or sufficiently progressing clients reset the streak exactly as upstream does. just subscribe-config-contract runs explicit and non-positive/default values through the ReleaseSafe daemon; Zig receipts separately force physical eviction and both sides of the configured rate boundary without sleeps or sockets.
• cold block sharing: production replay uses a 64 MiB byte-bounded LRU by default. A cache item owns one real decompressed JSS block and memoizes v1, v2, and both dictionary-zstd bodies exactly once; memo growth is charged to the same budget. Same-key misses single-flight while unrelated keys decode outside the cache lock. Handles pin evicted items, and manifest refreshes advance a per-segment generation so an in-flight old decode cannot reinsert. The offline receipt opens real sealed files from eight subscriber threads, observes one decode and pointer-identical JSON/zstd payloads, forces memo-growth eviction, then proves a verified manifest refresh re-decodes.
• subscriber read batches: hot and cached-cold pulls scan at most 1,024 raw entries by default (--subscribe-read-batch). The boundary precedes v1 skips and client filters, cursor progress includes suppressed rows, and the cold reader stops at the captured hot floor before resolving a hot index. Unit receipts exercise those rules against the real tail and sealed JSS; just subscribe-read-batch-contract drives a rejecting production WebSocket through 5,000 sealed rows and observes exactly 295 cold pulls at batch 17 plus a stable empty-hot handoff. just subscribe-config-contract covers explicit and non-positive CLI values. A separate real-archive receipt leaves four rows in the unsealed active segment, confirms only its fsynced prefix is cold-readable, and crosses that active-to-hot boundary in two bounded pulls. The archive snapshot captures sealed metadata, active index, and durable byte boundary under one writer lock, then releases it before any decode or subscriber write.
• resync memory lifetime: upstream appends each converted replacement and immediately forwards it to the bounded readable log; it does not retain a second whole-repository wire representation. Stream batches physical archive writes in groups of 1,024 but releases the complete v1/v2 encoder arena after each row is copied into the tail. The offline 4,096-record MST receipt is capped at 1 MiB of repair scratch and mutation-checks the former repair-lifetime JSON ownership by reproducing its exact ArchiveAppendFailed.
• resync capacity envelope: a pinned-simulator ReleaseSafe receipt seeds 96 signed repositories, establishes real verifier chain state, creates 96 genuine chain breaks, and holds only the real getRepo requests at the handler boundary. It observes exactly 32 active workers plus 64 queued jobs, releases 96 complete CAR/MST repairs, and requires the physical JSS archive to converge to simulator ground truth. With 128-record repos the receipt measured 54,886,400 bytes RSS at saturation and a 72,695,808-byte peak; repository size, not hidden scheduler work, determines the production multiplier.
• archive health: free space is collected from the filesystem containing the open archive directory on every scrape; segment-seal latency starts before the pending-block flush and is observed only after footer write/fsync plus finalized-header pwrite/fsync succeeds, using upstream's 0.01 × 2^n bucket boundaries.
• block integrity: the JSS xxh3 authenticates only header/footer metadata, while upstream's zstd decoder independently verifies each frame's content checksum. Stream uses its pinned vendored libzstd for the same check; flipping only the final checksum byte of an upstream-produced block is the mutation receipt.

zig oracle v1 (2026-07-12) #

just oracle (tests/oracle.py): crash-matrix restart harness against the simulator. arms --crashpoint=<seam> (src/internal/crashpoint.zig — abort() at durable commit boundaries incl. mid-download), lets the real binary die, restarts clean, asserts convergence: serving ungates, listSegments non-empty, /subscribe-v2 replay seq-monotonic, serving proves the RocksDB phase reached steady_state, backfill tree cleaned, no legacy metadata stores reappear, and interrupted repos are repaired by the pending pass (anti-vacuity via log markers). Every restart also scrapes the production metrics endpoint: phase 3, the real merging→steady transition, merge/write-state histograms, and all six registered merge counters are mandatory; bootstrap-resume cases additionally prove bootstrap→merging plus drain/seal/close/write timings, and the interrupted download case must perform nonzero source, DID-lookup, and event work. A separate injected metadata-commit failure test proves the archive does not publish or strand a segment durability boundary. Event-log and final state equivalence are covered by the pinned differential oracle below; strict lifecycle, compaction, and timestamp-import power loss are covered by the Linux tier below. Seed-derived restart determinism is covered by the five-world predicate-kill tier below.

The harness is runnable without internet after dependencies are cached. Start the pinned simulator with GOPROXY=off go run ./cmd/simulator serve ..., then run UV_OFFLINE=1 just oracle; the recipe builds and crashes the ReleaseSafe candidate rather than a debug-only binary. Replay checks consume the entire sealed archive when it contains fewer than 200 rows and a 200-row prefix otherwise; they never wait for an arbitrary cardinality the simulator did not produce. Every case also rejects OutOfMemory and fatal event-handler log evidence. The harness caught a live-repair regression that recreated backfill/repo-scratch after cutover; bootstrap, live repair, and failed-repo healing have disjoint scratch namespaces, and every case proves the lifecycle-owned backfill/ tree is absent at steady state.

strict Linux power-loss oracle (2026-07-20) #

just powerloss-oracle runs the real aarch64-linux-gnu ReleaseSafe binary, RocksDB, and JSS on ext4 mounted over a kernel NBD device. The Python nbdkit backend has separate live and durable images: ordinary writes affect only the live image, while NBD FLUSH/FUA copies dirty extents to the durable image and fsyncs it. At the selected boundary the harness kills nbdkit first, SIGKILLs Stream second, recreates the live device exclusively from the durable image, runs e2fsck, remounts it, and starts an uninstrumented recovery process.

The calibration is fail-closed: a raw fsynced OLD block must survive while a later direct, readable, unflushed NEW block must disappear, and the flush receipt must remain unchanged. Every lifecycle case requires a unique cutpoint receipt, at least one pre-cut storage flush, real RocksDB and segment trees, nonempty archive XRPC results, healthy serving, and 100 ordered public /subscribe-v2 rows (or the complete archive when smaller). The passing candidate recovered all eight schedules: after-repo-complete ordinals 1 and 2, destination-flush-before-source-commit, destination-seal-before-discovery, discovery-before-cleanup, cleanup-complete, bootstrap-live-close-before-seal, and steady-phase-before-steady-run.

The same production-binary campaign covers upstream's mutation boundaries. Six delete-compaction schedules cut at each rewrite write/fsync/rename/dir-fsync boundary and on both sides of the durable watermark; every recovery converged to watermark 2500 and exactly 1,507 retained events after dropping 993 rows. Four timestamp-import schedules cut at each sealed-segment patch boundary; the target rows recovered with the imported timestamp while untargeted rows kept their exact witness-derived time. All ten cases reject stale temporary files, require complete duplicate-free public V2 replay, and use fixtures created by the real Stream JSS writer rather than synthetic segment bytes. This campaign found a Linux-only O_PATH parent-directory handle whose fsync failed with EBADF; rewrite directories are opened as readable directory handles and the full campaign passes on Linux.

An additional two-crash schedule covers upstream's restart-after-cleanup guard: the first process is killed after removing backfill/ but before the directory fsync; the second observes the cached absence, must emit another durability flush before deleting merge cursors, and is then power-cut at cleanup-complete. Recovery must still find the source tree absent and every survivor exactly once.

This campaign found production defects that process-local crash tests could not: Linux directory handles opened with O_PATH made the required cleanup fsync fail; cleanup cursor deletion—including its restart guard—was not coupled to durable directory removal; a 256 KiB ping-thread stack failed on Linux; the listener briefly served before bootstrap gating; and leading sequence-zero hot-tail rows could misclassify an archive cursor as resident. The Linux process regression observes repeated 503 responses throughout bootstrap and then replays sequences 1 through 10 from the fresh archive without restart.

The one-time just powerloss-image step builds the digest-pinned Ubuntu tool image containing nbdkit, nbd-client, and ext4 utilities. Actual oracle runs use --pull=never, uv --offline, and the pinned local simulator. They require privileged Linux container access and /dev/nbd3; failure evidence is retained under the printed stream-powerloss-* temporary directory instead of being deleted.

archive client contract #

just archive-contract creates a temporary three-segment, 5,000-row JSS archive and serves it from a real ReleaseSafe Stream process. It directly checks listSegments, getSegment, getBlock, and planBackfill, then runs the upstream Go client through whole-segment, DID-filtered block, sentinel-only block, bounded-range, and archive-to-live-cutover plans. The client harness requires upstream HEAD d4dd2f0127e8934d125b2a26a90fe88239ca98a2, selects a cached Go 1.26.5 toolchain when necessary, and forces GOTOOLCHAIN=local, GOPROXY=off, and GOSUMDB=off for the actual Go build and run. uv is also forced offline; Zig's checksum-pinned dependencies must already be in its global cache. A missing pin, Go toolchain/module, Python environment, row, sentinel, or cursor transition fails the command. The same official-client run checks the resident manifest after its archive-to-live cutover: the sealed gauge equals listSegments, the load histogram has one observation per startup segment, replay has produced a real resident block-index hit, and no unknown-index lookup occurred. Unit adversity holds an old refcounted index across a verified rewrite and proves cold cursor translation combines its historical envelope with offsets read from the newly opened file generation, matching upstream's topology-preserving race contract. The direct planner regression goes further: after startup it renames every JSS file out of the segment namespace, requires an unfiltered plan to remain exact, then restores the files for download tests. That fails if planBackfill opens segment data instead of using its refcounted resident envelopes, DID blooms, and collection summaries.

just http-metrics-contract launches the real candidate with --segment-cache-max-age=1500ms and requires getBlock to return Cache-Control: public, max-age=2. This records upstream's positive-duration ceil behavior independently of the default public, no-cache archive receipt.

just plan-config-contract builds a real three-segment JSS archive whose selected-block density is exactly 3/4, then starts three production Stream processes. It proves the distinct DID/collection caps (including zero disabling filters), one-entry work-unit pagination, zero-entry-limit unbounded paging, and the 1.0-versus-0.75 whole-segment threshold boundary through public planBackfill responses.

just cursor-lookback-contract seeds three real sealed segments and starts three production Stream processes. With the upstream 36-hour default it proves that v2 rejects an old sequence cursor before upgrade while v1 clamps to the oldest eligible segment. A widened window replays from the requested sequence; --cursor-lookback=0 upgrades without replay and records the disabled cursor mode. Timestamp cursors use the same conservative segment floor and clamp on both protocols. The receipt speaks raw WebSocket framing with Python's standard library and needs no network access.

just compaction-config-contract seeds two physical create/delete segments twice. A production process with tombstone cap 1 and one rewrite worker must commit two chunks and report a one-worker rewrite group. A second process with cap 0 (upstream's unlimited sentinel) and two rewrite workers must commit one chunk and report a two-worker group. A third production process repeats that physical pass on a 250 ms Go-duration interval. All public archives must contain only the two surviving marker rows at durable watermark 4. A real archive unit receipt separately proves the first cap trigger wakes and completes in under 500 ms despite a one-hour periodic interval. The receipt checks the timer, trigger, fold, and rewrite paths.

just retry-config-contract seeds durable failed-repository rows, serves real getRepo failures from a threaded loopback HTTP server, and runs four ReleaseSafe Stream processes. Held requests prove explicit global concurrency 3, explicit per-host concurrency 2, and upstream's zero-value defaults of 16 global and 4 per host. Each process is stopped after exactly one completed pass; the receipt reopens the real RocksDB store and requires every row to have one additional attempt plus a next-attempt delay inside the configured 1–1.5-second exponential-jitter window. This covers scheduling, admission, failure persistence, Go-duration parsing, zero defaults, and max-delay clamping without network access.

The pinned upstream simulator drives four complete bootstrap/merge lifecycles. The normal two-page crawl makes three real listRepos requests: two bootstrap pages and the final-page merge-discovery replay. With --skip-merge-discovery, it makes exactly the two bootstrap requests. A max-repo selection proves the same automatic two-request short circuit, while an explicit-DID selection bypasses listRepos entirely. Both automatic cases also traverse the configured-skip lifecycle branch, matching upstream. Three additional runs use the same real request ledger. With batch size 2, the first getRepo precedes the second listRepos page; with batch size 3, both two-entry pages precede every getRepo, proving the page-aligned batch may exceed its target. A zero-sized setting follows Jetstream's 100,000-entry production default and has the same cross-page ordering. Relay resume and merge-discovery cursors are committed together only after the corresponding dispatch batch reaches durable terminal state.

A separate 104-account simulator world holds real getRepo handlers before serving their CARs. Three ReleaseSafe Stream processes reach measured request peaks of 7 for an explicit worker setting, 100 for zero, and 100 when the flag is omitted. Releasing the handlers lets every process prepare the real CARs, converge its physical archive, enter steady state, and drain cleanly. This detects the former behavior where the advertised count was silently reduced to four by a 2 GiB-per-worker formula.

The bootstrap archive has a separate physical compression receipt. Four deterministic 8 MiB columnar blocks overlap in four real zstd workers (no sleep or compression mock). A second archive run compares synchronous and async JSS bytes exactly, then checks all eight committed blocks and the final durability watermark. This verifies that --backfill-async-flush-workers is exercised by the write path.

pinned differential oracle (2026-07-20) #

just differential-oracle refuses to run unless the upstream checkout is at d4dd2f0127e8934d125b2a26a90fe88239ca98a2. A Go overlay places the committed test source inside upstream's internal/oracle package, so the receipt directly reuses its simulator world, physical JSS reader, event-log comparator, independent MST reconstruction, invariant checker, and public archive client. The candidate is the real ReleaseSafe Stream process using real sockets and a temporary on-disk archive; GOPROXY=off, GOSUMDB=off, and a local Go toolchain keep the run offline and fail closed when prerequisites are absent.

The current tier proves bootstrap convergence and controlled live equivalence for create, update, delete, identity, account, sync, and create_resync rows. It also removes an intermediate update to demonstrate comparator detection power, requires clean SIGTERM drain, restarts the same data directory, and reconstructs the complete archive through planBackfill/getSegment/getBlock/subscribe-v2. The production disconnect check additionally terminates and restarts the pinned local simulator while the ReleaseSafe binary is ingesting: Stream must retain its cursor, reconnect, resume delivery, and then exit cleanly on SIGTERM. The archive fallback close is deliberately registered above consumer teardown, so an unexpected consumer error cannot invoke the durability hook through freed cursor state.

just differential-oracle-multiseed adds upstream's restart-chain baseline and exact seeded predicate-kill selector. Five distinct simulator worlds drive real create/update/delete/account/sync intermediate chains through selected (crashpoint, ordinal) process aborts, same-disk recovery, upstream event-log coverage, compaction filtering, final-state reconstruction, recreated-record visibility, and the red-first missing-delete check. The campaign exposed two gaps: bootstrap could begin before its live websocket handshake, and the close-before-seal seam had drained pipeline work whose final partial JSS block was not yet durable. Stream gates backfill on the firehose connection and flushes that block before the seam; all five worlds pass offline. The real-process wire tier injects upstream's duplicate and regressed relay windows on the same connection and requires the exact once-per-frame archive, including account delete/reactivate/recreate. The complete upstream frame-fault campaign is covered: malformed CBOR and FutureCursor advisory frames; two successive 3 MiB read-limit failures followed by cursor-safe reconnect; a swallowed frame reported as an exact one-sequence gap and recovered by authenticated repair; a schema-ordered unknown future frame that occupies the missing sequence without becoming a decode error; and a partial CAR whose missing record block drops exactly one op while its siblings and verifier chain survive. The receipts caught three production divergences: a cursor fallback that lost the global relay high-water, Zat rejecting an omitted optional prevData field, and strict repository DAG-CBOR ordering leaking into the forward-compatible wire inspector. All three are regression-covered.

The adversarial tier now reuses upstream's exact lie tables and ledger. Real signed commits prove that invalid collection, rkey, and column-width ops drop individually while benign siblings survive. A divergent #sync with a non-TID envelope proves the Atmos ordering boundary: authenticated getRepo repair advances durable verifier state before the ingest gate drops the entire directive as live/invalid_rev. A separately signed non-TID commit is rejected by the verifier, then a valid follow-up chain-break repairs both records. A DID-scoped PLC resolution failure cannot wedge a sibling DID. Finally, real bootstrap CARs carry invalid UTF-8, no-slash, dot-dot, and 300-byte keys; only valid records reach physical JSS, filtered MST truth converges, and every canonical (source, reason) counter meets the upstream ledger floor.

The differential tier alone does not claim filesystem durability. The strict Linux tier closes the pinned storage, lifecycle-power-loss, compaction, and timestamp-import mutation schedules. On 2026-07-21 the five-world predicate- kill tier passed again against candidate 8b04c42; every selected real-process abort fired, recovered on the same disk, and matched the upstream oracle.

HTTP and getBlock metrics contract #

just http-metrics-contract seeds the same real sealed archive and launches a ReleaseSafe Stream process with loopback-only dependencies. Standard-library HTTP requests exercise getBlock 200, conditional 304, failed-precondition 412, Range 206, invalid Range 416, malformed 400, and missing-block 404 responses. The archive contract additionally drives both getSegment and getBlock through real multipart ranges, mixed overlapping/non-overlapping members, the aggregate-size fallback, and the different headers and bodies for malformed versus non-overlapping 416s. It verifies Last-Modified against the physical segment mtime and exercises If-Match, If-Unmodified-Since, If-None-Match, If-Modified-Since, and ETag/date If-Range with Go's precondition precedence. Unknown NSIDs traverse the same real route and must return 501 MethodNotImplemented; deleting a manifest-resident segment proves both byte endpoints use the canonical 500 InternalServerError envelope. A separate real-server test holds the serving gate closed and requires the JSON 503 ServiceUnavailable envelope instead of Stream's former plain-text 503. The combined receipts prove upstream's ok, bad_request, not_found, and error partition; duration cardinality; and that served bytes increase by exactly one completely written 200 frame, never by partial or conditional responses. It also sends a real masked WebSocket close frame and waits for the inline subscription handler to finish before checking its HTTP lifetime metric. Handler labels reproduce the pinned upstream public mux: one xrpc/ subtree, individual root/status/ subscribe routes, and no observations for debug health/metrics or unmatched requests.

status diagnostics contract #

just status-contract seeds repository transitions through the real RocksDB repo store, closes the seed process, and reopens the same data directory with the production ReleaseSafe Stream binary. It checks the public text and HTML host views over loopback: exact upstream largest/failing ordering, durable current counts and bounded error history, safe escaping, and the browser filter contract. The seed includes a healthy larger host and a smaller host with repeated failures, so either sort or history silently disappearing makes the receipt fail. Browser checks additionally exercise the filter and the horizontal table layout at desktop and phone-sized viewports. The same restarted binary is queried through the Account tab and legacy did alias, including durable fields, unknown identity presentation, automatic real verification failure presentation, no-store headers, and side-effect-free HEAD. A separate production-route test uses a real JSS archive and signed commit data from loopback PLC/PDS endpoints to prove successful verification and the port-free per-source-IP limiter boundary.