From 18e4ced6a0b75fdce4c0caa7eac2768ac93cd808 Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Mon, 27 Jul 2026 12:10:14 -0500 Subject: [PATCH] docs: make the README an entry point, not a manual The README had grown into a 410-line flag reference whose status section was false -- it claimed no artifact was admitted and described as blocked several rows the gate has since closed. A reader could not tell what Stream is, and an operator could not trust what it said. Reference material moves to docs/{configuration,cli,testing}.md; the README keeps what it is, what is actually proved (23 verified, 4 partial, none blocked), and a map of where to look. The configuration audit had drifted the same way: eleven rows still cited blockers the gate records as closed -- cold-replay holes, planBackfill false-negatives, retry-supervisor health, the reconnect race, persisted host parking. Five of those rows had no remaining limitation at all, so they were promoted rather than left understating what holds. The deployment gate itself is unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 460 ++++++----------------------------- docs/cli.md | 30 +++ docs/configuration-parity.md | 30 +-- docs/configuration.md | 280 +++++++++++++++++++++ docs/deployment-runbook.md | 14 +- docs/semantic-parity.md | 15 +- docs/testing.md | 87 +++++++ 7 files changed, 505 insertions(+), 411 deletions(-) create mode 100644 docs/cli.md create mode 100644 docs/configuration.md create mode 100644 docs/testing.md diff --git a/README.md b/README.md index 5ec30b9..2452f23 100644 --- a/README.md +++ b/README.md @@ -2,59 +2,59 @@ a zig implementation of [jetstream](https://github.com/bluesky-social/jetstream) — the full-network archive and streaming service for atproto. built on [zat](https://tangled.org/zat.dev/zat). -The former live-only `stream.waow.tech` canary and the July 2026 isolated -experiment have been retired. No public Stream deployment is currently -promoted, and no current artifact is admitted for another whole-network -experiment. +## status -## capabilities +No public Stream deployment is currently promoted. The former live-only +`stream.waow.tech` canary and the July 2026 isolated experiments have been +retired. -- live Jetstream feeds at `/subscribe` and `/subscribe-v2`, with collection and - DID filters, cursor replay, runtime option updates, RFC 7692 - permessage-deflate on v1, frozen v1 zstd, and fetchable - dictionary-negotiated v2 zstd; cold replay continuity and remote-data - encoding isolation are not yet admitted -- a disk-backed historical archive exposed through `listSegments`, `getSegment`, - `getBlock`, and `planBackfill`; Stream's sealed `jss` v1 segments are - byte-compatible with upstream Jetstream. Sealed headers, block envelopes, - DID blooms, and collection summaries are resident and refresh atomically - after seal, compaction, and timestamp rewrite; planning opens no segment - files, while cold replay reads only selected compressed blocks. Internal - planner errors and unreadable replay segments still have fail-open paths -- a whole-network bootstrap implementation with concurrent live capture and - deterministic merge. Repository completion is coupled to ordinary archive - writer durability independently of the 100,000-entry dispatch/checkpoint - batch; cold replay, pending repair, and live-ingest paths remain blocked by - the deployment-gate audit. Startup resumes the existing active segment, +The implementation is **not** at semantic parity with pinned Jetstream V2 +commit `f29815c`. [`docs/semantic-parity.md`](docs/semantic-parity.md) is the +deployment gate and the only authority on what is proved: 23 verified rows, +4 partial, none blocked. Four experiments' worth of history — including what +each failure bought — is in +[`docs/full-network-experiment-2026-07.md`](docs/full-network-experiment-2026-07.md), +and [`docs/deployment-runbook.md`](docs/deployment-runbook.md) is what to +actually do, in order. + +An artifact is admitted for the next whole-network experiment: +`sha256:43001041…` (`receipts/eb95c14.json`, 20/20 suites, built natively on +linux/amd64). An admitted artifact means those suites passed against *that +digest* — it is not by itself a claim of parity. + +## what it does + +- **live feeds** at `/subscribe` and `/subscribe-v2`, with collection and DID + filters, cursor replay, runtime option updates, RFC 7692 permessage-deflate + on v1, frozen v1 zstd, and fetchable dictionary-negotiated v2 zstd +- **a disk-backed historical archive** through `listSegments`, `getSegment`, + `getBlock`, and `planBackfill`. Sealed `jss` v1 segments are byte-compatible + with upstream. Sealed headers, block envelopes, DID blooms, and collection + summaries are resident and refresh atomically after seal, compaction, and + timestamp rewrite; planning opens no segment files, and cold replay reads + only selected compressed blocks +- **whole-network bootstrap** with concurrent live capture and deterministic + merge. Repository completion is coupled to ordinary archive-writer durability + independently of the dispatch/checkpoint batch, so a restart does not replay + work already committed. Startup resumes the existing active segment, truncating and fsyncing only a framing-torn suffix while failing closed on complete-frame corruption and sealed high-water checksum failure -- Sync 1.1 commit verification, including PLC key rotation, MST inversion, - op-CID checks, replay protection, durable repository-chain state, and - transparent authenticated whole-repo repair with ordered pending replay; - the live scheduler implements Atmos-shaped 32-worker per-DID FIFO, bounded - drop-oldest, and completion-order batching. Reconnect and cursor-recovery - behavior remain deployment blockers -- RocksDB-backed metadata and archive publication coupled to segment fsync - boundaries; several correctness-metadata reads still fail open and are - called out in the deployment gate -- tombstone-aware compaction and continuing repair of repositories that were - unavailable or changed during bootstrap; the versioned compaction watermark - fails closed on corruption/read failure, while retry-supervisor failure - handling is not yet equivalent -- crash-resumable timestamp imports with a durable RocksDB rule map, +- **Sync 1.1 commit verification** — PLC key rotation, MST inversion, op-CID + checks, replay protection, durable repository-chain state, and transparent + authenticated whole-repo repair with ordered pending replay. The live + scheduler implements Atmos-shaped 32-worker per-DID FIFO, bounded + drop-oldest, and completion-order batching +- **RocksDB-backed metadata** and archive publication coupled to segment fsync + boundaries +- **tombstone-aware compaction** and continuing repair of repositories that + were unavailable or changed during bootstrap +- **crash-resumable timestamp imports** with a durable RocksDB rule map, topology-preserving sealed-segment repair, and bearer-gated operator XRPC -- browser and terminal operations views at `/status`, including durable PDS - diagnostics at `/status?tab=hosts` and per-repository reconstruction and - authoritative root verification at `/status?tab=accounts`, plus Prometheus - metrics at `/metrics` on the optional debug listener. The upstream - collections/segments status views and several lifecycle fields are absent - -The implementation is **not** at semantic parity with pinned Jetstream V2 -commit `f29815c`. The current evidence, blockers, and definition of a valid -admission are maintained in -[`docs/semantic-parity.md`](docs/semantic-parity.md). The July experiment is -historical evidence, not an admission receipt; see -[`docs/full-network-experiment-2026-07.md`](docs/full-network-experiment-2026-07.md). +- **operations views** at `/status`, including durable PDS diagnostics at + `/status?tab=hosts` and per-repository reconstruction and authoritative root + verification at `/status?tab=accounts`, plus Prometheus metrics at `/metrics` + on the optional debug listener. Upstream's collections/segments status views + are absent ## develop @@ -62,349 +62,31 @@ historical evidence, not an admission receipt; see zig build test && zig build # unit tests + exe just simulator # local fake atproto network on :7777 just run-sim # stream against it -just e2e # python wire checks -just archive-contract # archive XRPCs + resident manifest + pinned Go client -just plan-config-contract # non-default planner limits/threshold, real process -just cursor-lookback-contract # v1/v2 replay-window behavior, real processes -just compaction-config-contract # tombstone cap/rewrite workers, physical JSS -just retry-config-contract # global/per-host retry admission + backoff -just subscribe-config-contract # subscriber retention/cache/batching/slow controls -just subscribe-read-batch-contract # 5k-row filtered replay, exact cold pulls -just listener-contract # public/debug isolation + disabled/legacy modes -just status-contract # durable host rows + public HTTP view, offline -just oracle # lifecycle crash + RocksDB fault recovery matrix -just powerloss-image # one-time Linux NBD/ext4 tool-image bootstrap -just powerloss-oracle # real power-cut recovery, offline after bootstrap -just differential-oracle # exact pinned upstream semantics, fully offline -just differential-oracle-multiseed # five upstream predicate-kill worlds, offline -``` - -The binary also exposes upstream's offline operator commands. They do not -start the server or open RocksDB: - -``` -stream version -stream inspect-segment --blocks=table --blocks-truncate=100 -stream inspect-all --data-dir=./data --collections-truncate=100 -``` - -`inspect-segment` accepts `summary`, `table`, and `full` detail, handles both -sealed and active files, and reports a readable checksum-corrupt sealed file -as invalid without weakening the checksum enforcement used by archive reads. -The committed upstream-produced JSS fixture pins its rendered output byte for -byte (apart from the file path). -`inspect-all` walks both the steady archive and Stream's bootstrap live-capture -tree, aggregates network/tree/collection totals, tolerates a racing tail file, -and supports upstream's `--skip-unsealed` and collection-table truncation -controls. - -runtime flags are listed in `src/main.zig`; deploy notes in `deploy/README.md`. -The Zig build pins its Zig and native dependencies, including otel-zig, -RocksDB, zstd, xxHash, and websocket.zig's zlib 1.3.2; it does not depend on a -system zlib installation. - -Production configuration uses Jetstream V2's canonical `--relay-url`, -`--plc-url`, and `--max-backfill-repos` names. `--relay-url` drives both the -live WebSocket and HTTP repository clients; Stream's older split flags remain -accepted for existing local harnesses. - -Every canonical flag also reads the same `JETSTREAM_*` environment variable as -the pinned upstream command. Explicit CLI values take precedence without even -parsing an overridden environment value. Unknown `JETSTREAM_*` names fail the -process before command execution, catching misspelled production settings; -Kubernetes's `JETSTREAM_APP_*` service variables and upstream's -`JETSTREAM_ORACLE_*` and `JETSTREAM_SIM_*` test namespaces remain intentionally exempt. `inspect-all` inherits -`JETSTREAM_DATA_DIR`, and root logging variables remain persistent across every -command. `just environment-contract` proves the hand-maintained Stream map, -sorted unknown rejection, typed validation, CLI precedence, inspection, -listener, storage, logging, and timestamp-import authentication through real -ReleaseSafe processes without external network access. It does not prove that -the map contains every flag in the pinned Go source; the deployment gate now -requires a generated inventory comparison. - -The canonical server command also matches Jetstream V2's listener topology: - -``` -stream serve --addr=:8080 --debug-addr=127.0.0.1:6060 -``` - -The public listener owns `/`, `/status`, `/subscribe`, `/subscribe-v2`, and -the archive XRPC methods. The optional debug listener owns `/healthz`, -`/readyz`, and `/metrics`; an empty or omitted `--debug-addr` binds no debug -socket. `/readyz` means the configured listener accept loops are running, not -that bootstrap has reached steady state. Historical flag-only `--port=6008` -invocations retain Stream's combined listener solely for existing deployments -and offline harnesses. New deployments should use `serve --addr` and keep the -debug listener private. `just listener-contract` proves all three modes using -real ReleaseSafe processes without external network access. - -Intentional shutdown uses Jetstream V2's canonical `--shutdown-timeout` -(default `5s`) and `--client-drain-timeout` (default `10s`). On SIGINT or -SIGTERM, Stream stops listener admission, sends WebSocket close code 1001, -fans those notifications out concurrently so a backpressured client cannot -delay the others, drains cooperative subscribers until the client budget, and -shuts down public and debug HTTP traffic from the same instant while the -accepted ingest pipeline reaches its durable boundary. `just shutdown-contract` -exercises the real ReleaseSafe process offline with both cooperative and silent -peers. It also terminates bootstrap while live capture is connected and -`listRepos` is hung, then reopens the same data directory and repeats; active -backfill downloads are canceled while the archive flushes its recoverable -prefix. The transport suite also saturates one peer's real socket send buffer -and proves another peer still receives code 1001 before the shared deadline. -As upstream does, a non-positive HTTP shutdown timeout selects 30 seconds, -whereas a non-positive client-drain timeout is already expired: teardown is -immediate and delivery of the best-effort close frame is not guaranteed. - -Process logs follow Jetstream V2's canonical controls: `--log-level` accepts -debug/info/warn/error and their upstream aliases, while `--log-format` selects -`json` (the default) or `text`. Filtering remains runtime-configurable in -ReleaseSafe builds and applies to Stream plus its Zig dependencies. Root flags -remain persistent, so both `stream --log-level=debug serve` and -`stream serve --log-level=debug` work. `just logging-contract` proves the real -process formats, filtering, placement, and invalid-configuration failures -without external network access. - -Stream installs the same process-wide OpenTelemetry tracing provider as pinned -Jetstream V2. Tracing remains a real no-op unless -`OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` is set. -When configured, completed spans are batched and exported as OTLP/HTTP -protobuf; the generic endpoint receives the `/v1/traces` suffix and the -trace-specific endpoint is used verbatim. `--otel-service-name` and -`OTEL_SERVICE_NAME` set the `service.name` resource attribute, while the build -version supplies `service.version`. Standard endpoint, traces-endpoint, -header, gzip, insecure, timeout, custom-CA, client-certificate, batch, and -sampler environment controls are documented in -[`docs/opentelemetry.md`](docs/opentelemetry.md). Production -orchestrator, repository, live-batch, block-flush, and rotation work carries -the matching upstream span tree; an offline real-collector receipt exercises -both synchronous and asynchronous archive boundaries. - -The container includes the libcurl 4 runtime used only when an OTLP client -certificate and key are configured. Ordinary HTTPS and custom-CA-only exports -remain on Zig's native TLS path and do not load libcurl. - -The subscriber's writer-owned hot log retains 256 MiB by default and never -evicts rows above the durable archive watermark. Operators can tune the real -byte budget with `--subscribe-read-log-retention-bytes`. The upstream slow -client policy is also configurable with `--subscribe-slow-window` (default -`60s`) and `--subscribe-slow-min-rate` (default `5` log events/second); both -must remain continuously violated while a client is more than 100,000 events -behind before Stream disconnects it. - -Cold subscribers share a 64 MiB decoded-block cache by default, configurable -with `--subscribe-block-cache-bytes`. Its budget includes decoded JSS events, -memoized v1/v2 JSON, and the corresponding dictionary-zstd frames; compaction -and timestamp rewrites invalidate the affected segment generation. - -Subscriber pulls scan at most 1,024 raw log entries at a time by default, -configurable with `--subscribe-read-batch`. The boundary is applied before -filters and v1 skip rules in both hot and durable-disk reads, so selective -subscriptions still yield regularly and slow-client accounting follows raw -network progress. - -Sealed archive downloads use upstream's cache policy. The default is -`Cache-Control: public, no-cache`; operators may set a positive Go-style -duration such as `--segment-cache-max-age=15m`. As upstream does, Stream rounds -positive durations up to whole seconds in `max-age`. This is most useful when -compaction is disabled or the cache lifetime is comfortably shorter than the -configured compaction interval, because compaction can replace a segment while -retaining its name. - -`planBackfill` exposes the same CLI controls and defaults as upstream: - -``` ---plan-max-dids=1000 ---plan-max-collections=25 ---plan-max-entries=100000 ---plan-whole-segment-threshold=0.75 -``` - -Zero disables non-empty DID or collection filters for the corresponding -filter cap. For `--plan-max-entries`, zero instead means one unbounded page; -the threshold must be greater than zero and at most one. - -Subscriber replay implements upstream's 36-hour default lookback controls. A v1 sequence -cursor older than the retained window is conservatively clamped to the first -eligible sealed segment; v2 rejects the same cursor before WebSocket upgrade -with a `cursor too old` response. Timestamp cursors are clamped for both -protocols. Operators may change the window with a Go-style duration such as -`--cursor-lookback=72h`; zero makes both endpoints pure-live. The offline -`just cursor-lookback-contract` receipt exercises all three modes against real -sealed JSS through production Stream processes. Cold replay completeness is a -separate blocked row because an unreadable manifest-listed segment is currently -skipped. - -Delete compaction exposes upstream's resource controls: - -``` ---compaction-interval=4h ---compaction-tombstone-cap=32000000 ---compaction-rewrite-workers=0 -``` - -The interval accepts a non-negative Go-style duration and zero disables all -compaction. The tombstone cap bounds each durable fold/rewrite chunk; zero is -upstream's unlimited sentinel. Rewrite workers bound concurrent sealed-segment -rewrites; zero selects `min(CPU count, 8)`. Cap triggers wake the compactor -immediately, subject to upstream's 30-second between-pass floor. -`just compaction-config-contract` proves the controls reach the physical -compactor rather than stopping at argument parsing, including a 250 ms pass. It -does not waive the remaining lifecycle and serving admission gates. - -Failed-repository healing exposes upstream's complete control set: - -``` ---failed-repo-retry-interval=4h ---failed-repo-retry-workers=16 ---failed-repo-retry-host-workers=4 ---failed-repo-retry-max-delay=168h -``` - -The interval and maximum delay accept non-negative Go-style durations; zero -interval disables the background loop. Zero worker values select upstream's -16-global/4-per-host defaults, and zero maximum delay selects seven days. -Transient failures use upstream's capped exponential delay plus uniform -`[0, delay/2)` jitter. `just retry-config-contract` holds real `getRepo` -requests to measure both admission limits and then reopens RocksDB to verify -the durable retry schedule. The retry supervisor and candidate-scan design -remain blocked independently of those controls. - -The strict power-loss tier is deliberately separate from ordinary unit and -process tests. It cross-builds the production ReleaseSafe Linux binary, runs -RocksDB and JSS on ext4 over a kernel NBD device, kills the block backend before -the Stream process, and reconstructs the device only from writes acknowledged -by FLUSH/FUA. Its campaign covers eight lifecycle cuts, the two-crash cleanup -guard, six delete-compaction cuts, and four timestamp-import rewrite cuts. The -mutation cases use archives written by the production JSS writer and verify -recovery through the production daemon and public V2 replay. It therefore -requires privileged Linux containers and an -available `/dev/nbd3`. The pinned Ubuntu tool image needs network access once -for `just powerloss-image`; `just powerloss-oracle` itself refuses image pulls -and runs from cached Zig, Python, simulator, and container inputs. The current -recipe targets `aarch64-linux-gnu`, matching the development host used for its -recorded receipt. - -Production bootstrap controls: - -``` ---backfill ---backfill-workers=100 ---backfill-batch-size=100000 ---backfill-async-flush-workers=4 ---backfill-max-inflight-bytes=8589934592 ---skip-merge-discovery -``` - -Worker count controls concurrent network downloads. Batch size counts every -`listRepos` entry, including inactive and already-complete repositories; -Stream accumulates whole 1,000-entry protocol pages until it reaches the target, -then shuffles eligible repositories before dispatch. Requests still go through -the relay and follow its redirect, so this does not claim precise pre-request -PDS scheduling. Zero selects the 100,000-entry production default. The byte -flag bounds the owned -complete CAR plus preparation/index state. Preparation is serialized, copies -the downloaded CAR into that budget, and releases the scratch mmap before -emission. Already prepared repositories emit concurrently from the owned CAR -and loaded MST through fixed 1,024-row stack batches; this lets one repository -use the entire budget without allowing partial indexes to deadlock one another. -The zero worker value selects the 100-worker production default. HTTP bodies -remain concurrent and disk-backed. Four bootstrap-only workers detach complete -JSS blocks, perform real zstd compression concurrently, and commit/fsync them -in preparation order; zero explicitly selects synchronous compression. -Downloads need temporary -free space on `--data-dir` equal to the concurrently in-flight CARs; -scratch files are deleted after each attempt and cleared on restart. Bootstrap -owns `/backfill/repo-scratch`; steady live and failed-repo healing use -`/repair-scratch/{live,failed-repo}`, so repair cannot recreate -lifecycle state after cutover. Resource exhaustion is lifecycle-fatal, never -persisted as a fake "large repo skipped" success/failure. Backfill progress, -active workers, queue depth, current -transient bytes, and peak transient bytes are exported on `/metrics`. -Successful CARs record their final emitted archive sequence; ordinary writer -durability commits every covered repository independently. The dispatch batch -still controls listRepos cursor-checkpoint cadence, but a slow sibling no -longer withholds already-durable repository completion. The authoritative -lifecycle analysis is -[`docs/bootstrap-semantic-parity.md`](docs/bootstrap-semantic-parity.md). - -The failed-repo loop uses 16 global workers and four workers per final -post-redirect PDS host; attempts, error class/message, final host, retry count, -and next attempt are durable in RocksDB. Every repo transition also updates a -persistent `host/` diagnostic aggregate in the same synced batch: -current lifecycle/active counts, cumulative failure classes, and the five most -recent bounded error samples therefore survive restart without scanning the -whole-network repo keyspace. The browser and text host views read those rows -directly, preserve upstream's largest/failing sort order, and filter host and -recent-error content in the browser. This implementation is not admission to -another full-network run by itself: `docs/semantic-parity.md` remains -authoritative. - -The debug-only discovery flag omits the post-merge `listRepos` rescan. Matching -upstream, Stream enables it automatically for `--backfill-max-repos` and -`--backfill-repos`; a normal full crawl still replays the last page so accounts -created during bootstrap enter durable retry state. The pinned-simulator -process receipt observes three real `listRepos` requests for a normal two-page -crawl and exactly two when discovery is skipped. The same receipt proves a -page-sized batch dispatches before page two, while a cross-page batch and the -zero/default setting enumerate both pages before the first real `getRepo`. -Initial and post-bootstrap enumeration retain arbitrary-length relay cursors; -the real-process receipt follows a 4 KiB cursor and verifies that discovery -durably records both active and inactive accounts. - -The same durable account row retains the distinct initial-backfill and latest -known revisions, update time, declared handle, PDS endpoint, and reserved -record/byte counters. Matching upstream, the debug-only explicit -`--backfill-repos=did:plc:example[,did:web:example.com]` path processes the -requested DIDs serially, resolves their real DID documents, and removes stale -full-network merge-discovery state. Ordinary whole-network `listRepos` -bootstrap does not add millions of extra identity requests. -Normal post-bootstrap discovery is not yet admitted: inactive unknown -repositories are skipped and an oversized cursor can silently terminate the -scan. - -`handle/` is maintained atomically with every repo transition, and -each merge-source commit advances its cursor together with latest-revision -refreshes. Account lookup resolves handles through the configured identity -directory, reconstructs the repository from sealed JSS segments, the active -writer, pending rows, and bootstrap live segments at a rotation-safe boundary, -then compares the canonical local MST root with the PDS commit root obtained -through Sync 1.1 `getLatestCommit` and `getBlocks`. Verification runs -automatically for a found account. Matching upstream, expensive repo actions -are limited to four per source IP per minute with a bounded 4,096-source map; -trusted operator deployments can explicitly pass -`--disable-repo-action-rate-limits`. - -Timestamp import is disabled to remote callers unless a non-empty bearer token -is configured. Stage plain CSVs under the confined import directory (default -`/imports`), submit a relative path, then poll the returned job id: - -``` ---timestamp-import-token= ---timestamp-import-dir=/srv/stream/imports - -POST /xrpc/network.bsky.jetstream.importTimestamps -GET /xrpc/network.bsky.jetstream.getImportStatus?job= ``` -Both routes require `Authorization: Bearer `. Put them behind TLS at the -reverse proxy; Stream intentionally serves plain HTTP internally. Empty tokens, -missing credentials, and bad credentials all produce the same 401 response. -Submitted paths are canonicalized through symlinks and must resolve to regular -files inside the configured directory. +The full suite list, the oracles, and the power-loss tier are in +[`docs/testing.md`](docs/testing.md). The Zig build pins its Zig and native +dependencies, including otel-zig, RocksDB, zstd, xxHash, and websocket.zig's +zlib 1.3.2; it does not depend on a system zlib. ## docs -The docs folder holds the port record — jss format byte spec, seal spec, -upstream bootstrap/compaction/subscribe condensations (every deliberate -divergence is written down there), lessons carried over from zlay, and durable -product/copy context for public surfaces. `docs/semantic-parity.md` is the -top-level deployment gate. `docs/grafana-dashboard.md` records the -checksum-pinned upstream source, deterministic Stream adaptation, and its -fail-closed metric contract. The dashboard and process receipts are fully -offline: run `just dashboard-test`, `just process-metrics-contract`, -`just http-metrics-contract`, then -`just dashboard-contract http://127.0.0.1:6008/metrics` against a local -candidate. `just process-metrics-contract` also runs that dashboard -family-presence check against the ReleaseSafe binary it launches. Family -presence does not establish producer semantics, restart stability, or parity; -the dashboard audit in `docs/semantic-parity.md` is the deployment gate. +Start here depending on what you need: + +| | | +|---|---| +| [`semantic-parity.md`](docs/semantic-parity.md) | **the deployment gate** — what is proved, row by row | +| [`deployment-runbook.md`](docs/deployment-runbook.md) | staged plan and failure responses for a whole-network run | +| [`architecture.md`](docs/architecture.md) | how the subsystems fit together | +| [`invariants.md`](docs/invariants.md) | the rules that must never break, and what each cost | +| [`gotchas.md`](docs/gotchas.md) | deliberate behavior that looks wrong; traps in the harness | +| [`configuration.md`](docs/configuration.md) | every runtime control and its default | +| [`cli.md`](docs/cli.md) | offline operator commands | +| [`testing.md`](docs/testing.md) | suites, oracles, receipts | +| [`benchmarks.md`](docs/benchmarks.md) | subsystem benchmarks | + +The rest of `docs/` holds the port record: the jss format byte spec and seal +spec, upstream bootstrap/compaction/subscribe condensations where every +deliberate divergence is written down, the live scheduler and repair notes, +lessons carried over from zlay, and durable product/copy context for public +surfaces. diff --git a/docs/cli.md b/docs/cli.md new file mode 100644 index 0000000..265608d --- /dev/null +++ b/docs/cli.md @@ -0,0 +1,30 @@ +# Offline operator commands + +The binary exposes upstream's offline inspection commands. They do not start +the server or open RocksDB, so they are safe against a data directory a live +process owns. + +``` +stream version +stream inspect-segment --blocks=table --blocks-truncate=100 +stream inspect-all --data-dir=./data --collections-truncate=100 +``` + +`version` emits the upstream build-information field shape; values necessarily +identify the Stream artifact rather than a Go build. + +`inspect-segment` accepts `summary`, `table`, and `full` detail and handles +both sealed and active files. It reports a readable checksum-corrupt sealed +file as invalid *without* weakening the checksum enforcement used by archive +reads — inspection must never be a way to read bytes the server would refuse. +A committed upstream-produced JSS fixture pins its rendered output byte for +byte, apart from the file path. + +`inspect-all` walks both the steady archive and Stream's bootstrap live-capture +tree, aggregates network/tree/collection totals, tolerates a racing tail file, +and supports upstream's `--skip-unsealed` and collection-table truncation +controls. It inherits `JETSTREAM_DATA_DIR`, and root logging variables persist +across every command. + +Note that `inspect-all` does not replace the online collections and segments +status tabs, which remain absent — see [`semantic-parity.md`](semantic-parity.md). diff --git a/docs/configuration-parity.md b/docs/configuration-parity.md index fa5face..b61f8bb 100644 --- a/docs/configuration-parity.md +++ b/docs/configuration-parity.md @@ -4,8 +4,8 @@ Authority: - upstream `cmd/jetstream/main.go` and `internal/jetstreamd/options.go` at `f29815c391fc2644f8a3dd36b899fb3697dd1ea6` -- Stream `a345f4054df568e364aff08a9b1726275fc48e29` plus the discovery - changes and audit updates in this commit +- Stream `eb95c14310ef18c7ab9edbcff991a04a560a7d86` + (admitted artifact `sha256:43001041...`, `receipts/eb95c14.json`) This document audits public configuration and command wiring. A **verified** configuration row proves only that the input is parsed/defaulted and reaches @@ -18,37 +18,37 @@ semantically correct. Status vocabulary is defined in | Surface | Status | What source/tests establish | Limitation or remaining proof | |---|---|---|---| | Complete upstream flag inventory | **unverified** | The environment contract contains a hand-maintained pinned source map and rejects unknown `JETSTREAM_*` names. | There is no generated comparison proving every current upstream flag is represented here. Before another admission, generate the inventory from the pinned Go command/options definitions and fail on an unclassified addition/removal. | -| Relay URL | **partial** | `--relay-url` reaches the WebSocket consumer and relay-fronted HTTP listRepos/getRepo/repair clients; CLI and environment precedence are tested. | Correct wiring still feeds a blocked Zat reconnect loop. | +| Relay URL | **verified** | `--relay-url` reaches the WebSocket consumer and relay-fronted HTTP listRepos/getRepo/repair clients; CLI and environment precedence are tested. | The reconnect loop this feeds is no longer blocked: the process-killing race was a `setsockopt` `unreachable` in the shared websocket.zig fork, fixed there and shipped through zat v0.3.18. | | PLC URL | **verified** | `--plc-url` configures the real DID resolver used by live verification and selected-repo metadata; loopback resolution tests observe requests. | This row does not prove every verification/repair outcome. | -| Data directory | **partial** | `--data-dir` owns JSS, scratch, metadata, bootstrap-live, import, and temporary trees in production-process tests. Active-tail startup recovery now resumes in place and fails closed on corruption. | Cold replay, timestamp-import visibility, and retry metadata still have independent blocked paths; path ownership alone does not close them. | -| `--max-backfill-repos` | **partial** | Canonical spelling is parsed, limits dispatch, avoids committing a whole-network resume cursor, and automatically skips merge discovery. | The lifecycle and completion paths it invokes are blocked. It is a debug mode, not evidence for whole-network parity. | +| Data directory | **partial** | `--data-dir` owns JSS, scratch, metadata, bootstrap-live, import, and temporary trees in production-process tests. Active-tail startup recovery now resumes in place and fails closed on corruption. | Path ownership alone does not establish the behavior of what lives there. The cold-replay and retry-metadata blockers it used to cite are closed; the remaining caveat is that timestamp-import visibility is proved only for the cases in that row. | +| `--max-backfill-repos` | **partial** | Canonical spelling is parsed, limits dispatch, avoids committing a whole-network resume cursor, and automatically skips merge discovery. | It is a debug mode, not evidence for whole-network parity -- a capped run dispatches a fresh slice rather than exercising a full crawl. The lifecycle and completion paths it invokes are no longer blocked. | | `--backfill-repos` | **partial** | Canonical spelling and compatibility alias parse; it is mutually exclusive with max-repos, preserves selected order, bypasses listRepos, and enables selected identity resolution. | The exact current artifact still needs the complete selected-repository admission receipt; configuration wiring itself is established. | | Backfill workers | **verified** | Explicit positive values set physical getRepo concurrency; zero/omitted default to 100. Held real requests observe 7/100/100. | The 200-worker production experiment showed that accepting a value does not mean it is efficient. | | Backfill batch size | **verified** | Explicit and zero/default values control only page-aligned listRepos accumulation and cursor-checkpoint cadence; zero/omitted select 100,000. Repository completion now follows independent writer durability boundaries. | This control does not define whole-network size or completion percentage. | | Backfill async-flush workers | **partial** | Omitted defaults to four, positive values start that many bootstrap compression workers, and zero selects synchronous compression. Focused tests observe overlap and byte-equivalent output. | Full cancellation/OOM state coverage was not re-established in this audit. | | Skip merge discovery | **verified** | Explicit configuration and automatic max/selected enablement prevent the discovery request in focused tests. Normal discovery independently preserves active/inactive unknown repositories and follows arbitrary-length cursors. | This is a debug/partial-selection control; enabling it for a whole-network crawl intentionally omits accounts born during bootstrap. | -| Failed-repo retry interval | **partial** | Duration parsing, zero disablement, and timer-driven pass execution have real-process tests. | Pass failures are hidden from health and candidate scans are unbounded in RAM. | -| Failed-repo global/per-host workers | **partial** | Explicit and zero/default values reach real 16-global/4-per-host gates in held-request tests. | Pending-merge retry uses a different path, and the steady candidate set is materialized eagerly. | -| Failed-repo maximum delay | **partial** | Parsed values affect persisted exponential-jitter deadlines in focused RocksDB tests. | Host-park read corruption fails open; whole retry supervision is blocked. | +| Failed-repo retry interval | **verified** | Duration parsing, zero disablement, and timer-driven pass execution have real-process tests. | Closed: a terminal pass failure now stops the service rather than sleeping until the next interval, and candidates stream through a bounded queue instead of being materialized in RAM. | +| Failed-repo global/per-host workers | **partial** | Explicit and zero/default values reach real 16-global/4-per-host gates in held-request tests. | Pending-merge retry uses a different path, though it now runs through the same runner. The steady candidate set is no longer materialized eagerly. | +| Failed-repo maximum delay | **partial** | Parsed values affect persisted exponential-jitter deadlines in focused RocksDB tests. | Retry supervision is no longer blocked, and host parking is no longer persisted at all -- it moved into the runner's memory to match upstream, which removes the fail-open read this row described. | | Expensive account verification limiter | **verified** | Enabled/default and disabled modes cross the real `/status?tab=accounts` HTTP path. Four requests per source IP succeed, the fifth is rejected, and HEAD does not consume a slot. | This verifies the control, not the completeness of the status surface. | -| Cursor lookback | **partial** | Default 36h, explicit duration, and zero/pure-live change v1/v2 cursor admission in production-process tests. | Accepted cold cursors still depend on a replay walker that can skip unreadable segments. | +| Cursor lookback | **verified** | Default 36h, explicit duration, and zero/pure-live change v1/v2 cursor admission in production-process tests. | The replay walker no longer skips an unreadable manifest-listed segment; it propagates, because skipping would advance a cursor past durable events nothing revisits. | | Segment cache max age | **verified** | Go-duration parsing and ceil-to-seconds response headers are observed on real getSegment/getBlock responses; zero/default no-cache behavior is covered. | None known for this control. | -| planBackfill DID/collection caps | **partial** | Positive and zero sentinel values affect request validation in real HTTP tests. | A valid request can still receive a false-negative plan when `matchBlocks` encounters an internal error. | -| planBackfill entry limit | **partial** | Positive values paginate by work unit; zero disables the cap; continuation fixtures match normal upstream responses. | Internal planner errors are silently converted into non-matches. | -| planBackfill density threshold | **partial** | Configured thresholds change whole-segment versus block mode at tested boundaries. | Same planner error blocker. | +| planBackfill DID/collection caps | **verified** | Positive and zero sentinel values affect request validation in real HTTP tests. | Closed: a `matchBlocks` internal error propagates and the request answers 5xx rather than returning a truthful-looking empty plan. | +| planBackfill entry limit | **verified** | Positive values paginate by work unit; zero disables the cap; continuation fixtures match normal upstream responses. | Closed: planner errors answer rather than degrading into non-matches; an allocation-failure sweep asserts the planner never returns 200 with a short plan. | +| planBackfill density threshold | **verified** | Configured thresholds change whole-segment versus block mode at tested boundaries. | The planner error blocker this referenced is closed. | | Subscribe hot-log retention | **verified** | The byte value drives physical eviction in the production tail; tests cover eviction while respecting the durable floor. | Does not verify cold recovery after eviction. | | Subscribe slow-consumer window/rate | **verified** | Parsed duration and fractional rate reach the per-subscriber scan detector; explicit and default/non-positive behavior have focused tests. | None known for the detector control. | -| Subscribe cold-block cache | **partial** | The byte bound controls the shared decoded-block/wire memo LRU; concurrency, eviction, and manifest-generation invalidation have tests. | The enclosing cold walker skips an unreadable segment, so cache correctness cannot establish replay completeness. | +| Subscribe cold-block cache | **partial** | The byte bound controls the shared decoded-block/wire memo LRU; concurrency, eviction, and manifest-generation invalidation have tests. | Cache correctness alone does not establish replay completeness, but the enclosing cold walker no longer skips an unreadable segment. | | Subscribe read batch | **partial** | Explicit positive values and non-positive default 1024 bound raw scanned entries before filters in hot and normal cold fixtures. | The 5,000-row receipt tests batching, not missing-file continuity. | | Deprecated cursor block-index cache flag | **intentional divergence** | Stream accepts signed values as a compatibility no-op because its sealed indexes are always manifest-resident. Parser behavior is tested. | This is not the same runtime mechanism as an implementation that uses the cache. Keep it documented as compatibility, not parity. | | Compaction interval | **verified** | Go-duration parsing, zero disablement, periodic execution, and cap-trigger wakeup are exercised. The enclosing pass now propagates malformed/unreadable watermark errors. | This verifies the control and its watermark precondition, not every lifecycle cutover. | | Compaction tombstone cap | **verified** | Explicit and zero/default values affect real chunk count and physical survivors; watermark corruption cannot silently reset the fold range. | None known for this control. | | Compaction rewrite workers | **verified** | The configured value controls real concurrent rewrite groups in focused physical-file tests, with strict watermark admission. | None known for this control. | | Timestamp import token | **verified** | Missing/wrong/correct bearer behavior crosses the production import/status XRPC handlers, including constant-width comparison. | None known for the token control. | -| Timestamp import directory | **partial** | Configured roots are canonicalized/confined and real jobs survive restart in focused tests. | Corrupt import metadata reads were not re-audited; public results depend on blocked cold replay. | +| Timestamp import directory | **partial** | Configured roots are canonicalized/confined and real jobs survive restart in focused tests. | Corrupt import metadata reads were not re-audited. The cold-replay dependency this row cited is closed. | | Public bind address | **verified** | `serve --addr` binds the protocol listener; public route and WebSocket behavior are exercised by real processes. | Route availability does not imply archive readiness/completeness. | | Debug bind address | **verified** | Empty disables binding; configured debug address exclusively serves health/readiness/metrics and rejects protocol upgrade. Historical combined mode is separately tested. | None known for listener wiring. | -| Shutdown timeout | **partial** | Signed duration parsing/defaults reach listener and lifecycle shutdown; real tests cover positive, non-positive, hung listRepos, and cooperative exit. | Exact-artifact admission was violated, and reconnect failures remain independently process-fatal. | +| Shutdown timeout | **verified** | Signed duration parsing/defaults reach listener and lifecycle shutdown; real tests cover positive, non-positive, hung listRepos, and cooperative exit. | Both former limitations are closed: exact-artifact admission is now enforced by `scripts/admit verify` and `deploy.sh`, and the process-fatal reconnect path was a `setsockopt` `unreachable` in websocket.zig, fixed and shipped through zat v0.3.18. Consistent with the **verified** graceful-shutdown row in [semantic-parity.md](semantic-parity.md). | | Client drain timeout | **verified** | Configured/default/non-positive budgets reach concurrent client close hooks; real blocked-socket tests observe 1001 and shared-deadline interruption. | Limited to client-drain behavior. | | Log level | **verified** | Case-insensitive aliases/default and invalid values affect all `std.log` producers in real ReleaseSafe processes. | Not byte-identical Go logger internals. | | Log format | **verified** | JSON default and text selection produce the documented canonical fields in real processes. | Not byte-identical Go formatting beyond the asserted fields. | diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..cbc76b8 --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,280 @@ +# Configuration reference + +What each runtime control does and what its defaults are. This is the operator +reference; [`configuration-parity.md`](configuration-parity.md) is the audit of +how well each one is *proved*, and [`semantic-parity.md`](semantic-parity.md) +is the deployment gate. Runtime flags are declared in `src/main.zig`; deploy +notes live in `deploy/README.md`. + +Stream uses Jetstream V2's canonical names. Every canonical flag also reads the +same `JETSTREAM_*` environment variable as the pinned upstream command. +Explicit CLI values take precedence without even parsing an overridden +environment value. Unknown `JETSTREAM_*` names fail the process before command +execution, which catches misspelled production settings; Kubernetes's +`JETSTREAM_APP_*` service variables and upstream's `JETSTREAM_ORACLE_*` and +`JETSTREAM_SIM_*` test namespaces are intentionally exempt. + +`--relay-url` drives both the live WebSocket and the HTTP repository clients. +`--plc-url` configures the DID resolver. Stream's older split flags remain +accepted for existing local harnesses. + +## Intentional divergences + +Two controls deliberately differ from upstream. Neither should be described as +parity — the configuration audit requires them recorded here explicitly. + +**The deprecated cursor block-index cache flag is a compatibility no-op.** +Stream accepts signed values and ignores them, because its sealed indexes are +always manifest-resident. Accepting the flag keeps existing command lines +working; it is not the same runtime mechanism as an implementation that uses +the cache. + +**There are no `/debug/pprof/*` routes.** Stream is not Go, and a route-shaped +stub returning something pprof cannot use would be worse than an honest +absence. Zig and process metrics on the debug listener serve that purpose +instead. + +## Listeners + +``` +stream serve --addr=:8080 --debug-addr=127.0.0.1:6060 +``` + +The public listener owns `/`, `/status`, `/subscribe`, `/subscribe-v2`, and the +archive XRPC methods. The optional debug listener owns `/healthz`, `/readyz`, +and `/metrics`; an empty or omitted `--debug-addr` binds no debug socket. + +`/readyz` means the configured accept loops are running — **not** that +bootstrap has reached steady state. Historical flag-only `--port=6008` +invocations retain Stream's combined listener solely for existing deployments +and offline harnesses; new deployments should use `serve --addr` and keep the +debug listener private. + +## Shutdown + +``` +--shutdown-timeout=5s +--client-drain-timeout=10s +``` + +On SIGINT or SIGTERM, Stream stops listener admission, sends WebSocket close +code 1001, fans those notifications out concurrently so a backpressured client +cannot delay the others, drains cooperative subscribers until the client +budget, and shuts down public and debug HTTP from the same instant while the +accepted ingest pipeline reaches its durable boundary. + +As upstream does, a non-positive HTTP shutdown timeout selects 30 seconds, +whereas a non-positive client-drain timeout is *already expired*: teardown is +immediate and delivery of the best-effort close frame is not guaranteed. + +## Logging + +`--log-level` accepts debug/info/warn/error and their upstream aliases; +`--log-format` selects `json` (default) or `text`. Filtering is +runtime-configurable in ReleaseSafe builds and applies to Stream plus its Zig +dependencies. Root flags are persistent, so both `stream --log-level=debug +serve` and `stream serve --log-level=debug` work. + +## Subscribers + +The writer-owned hot log retains 256 MiB by default and never evicts rows above +the durable archive watermark; tune with +`--subscribe-read-log-retention-bytes`. + +Upstream's slow-client policy is configurable with `--subscribe-slow-window` +(default `60s`) and `--subscribe-slow-min-rate` (default `5` log +events/second). Both must remain *continuously* violated while a client is more +than 100,000 events behind before Stream disconnects it. + +Cold subscribers share a 64 MiB decoded-block cache +(`--subscribe-block-cache-bytes`) covering decoded JSS events, memoized v1/v2 +JSON, and the corresponding dictionary-zstd frames; compaction and timestamp +rewrites invalidate the affected segment generation. + +Pulls scan at most 1,024 raw log entries at a time (`--subscribe-read-batch`). +The boundary applies before filters and v1 skip rules in both hot and durable +reads, so selective subscriptions still yield regularly and slow-client +accounting follows raw network progress. + +### Replay window + +Subscriber replay implements upstream's 36-hour default lookback. A v1 sequence +cursor older than the retained window is conservatively clamped to the first +eligible sealed segment; v2 rejects the same cursor before the WebSocket +upgrade with a `cursor too old` response. Timestamp cursors are clamped for +both protocols. `--cursor-lookback=72h` changes the window; zero makes both +endpoints pure-live. + +## Archive serving + +Sealed downloads use upstream's cache policy. The default is `Cache-Control: +public, no-cache`; a positive Go-style duration such as +`--segment-cache-max-age=15m` is rounded up to whole seconds in `max-age`. This +is most useful when compaction is disabled or the cache lifetime is comfortably +shorter than the compaction interval, because compaction can replace a segment +while retaining its name. + +`planBackfill` exposes upstream's controls and defaults: + +``` +--plan-max-dids=1000 +--plan-max-collections=25 +--plan-max-entries=100000 +--plan-whole-segment-threshold=0.75 +``` + +Zero disables non-empty DID or collection filters for the corresponding cap. +For `--plan-max-entries`, zero instead means one unbounded page. The threshold +must be greater than zero and at most one. + +## Compaction + +``` +--compaction-interval=4h +--compaction-tombstone-cap=32000000 +--compaction-rewrite-workers=0 +``` + +The interval accepts a non-negative Go-style duration; zero disables all +compaction. The tombstone cap bounds each durable fold/rewrite chunk; zero is +upstream's unlimited sentinel. Rewrite workers bound concurrent sealed-segment +rewrites; zero selects `min(CPU count, 8)`. Cap triggers wake the compactor +immediately, subject to upstream's 30-second between-pass floor. + +## Failed-repository healing + +``` +--failed-repo-retry-interval=4h +--failed-repo-retry-workers=16 +--failed-repo-retry-host-workers=4 +--failed-repo-retry-max-delay=168h +``` + +Interval and maximum delay accept non-negative Go-style durations; a zero +interval disables the background loop. Zero worker values select upstream's +16-global/4-per-host defaults, and zero maximum delay selects seven days. +Transient failures use upstream's capped exponential delay plus uniform +`[0, delay/2)` jitter. + +Attempts, error class/message, final host, retry count, and next attempt are +durable in RocksDB. Every repo transition also updates a persistent +`host/` diagnostic aggregate in the same synced batch, so current +lifecycle/active counts, cumulative failure classes, and the five most recent +bounded error samples survive restart without scanning the whole-network repo +keyspace. + +## Bootstrap + +``` +--backfill +--backfill-workers=100 +--backfill-batch-size=100000 +--backfill-async-flush-workers=4 +--backfill-max-inflight-bytes=8589934592 +--skip-merge-discovery +``` + +Worker count controls concurrent network downloads; zero selects the 100-worker +default. Batch size counts every `listRepos` entry, including inactive and +already-complete repositories: Stream accumulates whole 1,000-entry protocol +pages until it reaches the target, then shuffles eligible repositories before +dispatch. Requests still go through the relay and follow its redirect, so this +does not claim precise pre-request PDS scheduling. Zero selects the +100,000-entry default. + +The byte flag bounds the owned complete CAR plus preparation/index state. +Preparation is serialized, copies the downloaded CAR into that budget, and +releases the scratch mmap before emission. Already-prepared repositories emit +concurrently from the owned CAR and loaded MST through fixed 1,024-row stack +batches, which lets one repository use the entire budget without letting +partial indexes deadlock one another. Four bootstrap-only workers detach +complete JSS blocks, compress concurrently, and commit/fsync in preparation +order; zero selects synchronous compression. + +Downloads need temporary free space on `--data-dir` equal to the concurrently +in-flight CARs. Scratch files are deleted after each attempt and cleared on +restart. Bootstrap owns `/backfill/repo-scratch`; steady live and +failed-repo healing use `/repair-scratch/{live,failed-repo}`, so +repair cannot recreate lifecycle state after cutover. + +Resource exhaustion is lifecycle-fatal, never persisted as a fake "large repo +skipped" success or failure — and it must never be misfiled as malformed remote +data either, which is the invariant [`invariants.md`](invariants.md) records +and an allocation-failure sweep now pins. + +Backfill progress, active workers, queue depth, and current/peak transient +bytes are exported on `/metrics`. Successful CARs record their final emitted +archive sequence, and ordinary writer durability commits every covered +repository independently: the dispatch batch controls listRepos +cursor-checkpoint cadence, but a slow sibling no longer withholds +already-durable repository completion. The authoritative lifecycle analysis is +[`bootstrap-semantic-parity.md`](bootstrap-semantic-parity.md). + +### Discovery and selected repositories + +`--skip-merge-discovery` omits the post-merge `listRepos` rescan. Matching +upstream, Stream enables it automatically for `--max-backfill-repos` and +`--backfill-repos`; a normal full crawl still replays the last page so accounts +created during bootstrap enter durable retry state. Enabling it for a +whole-network crawl intentionally omits accounts born during bootstrap. + +The durable account row retains the distinct initial-backfill and latest known +revisions, update time, declared handle, PDS endpoint, and reserved record/byte +counters. The debug-only explicit +`--backfill-repos=did:plc:example[,did:web:example.com]` path processes the +requested DIDs serially, resolves their real DID documents, and removes stale +full-network merge-discovery state. Ordinary whole-network bootstrap does not +add millions of extra identity requests. + +`handle/` is maintained atomically with every repo transition, and +each merge-source commit advances its cursor together with latest-revision +refreshes. + +## Account verification + +Account lookup resolves handles through the configured identity directory, +reconstructs the repository from sealed JSS segments, the active writer, +pending rows, and bootstrap live segments at a rotation-safe boundary, then +compares the canonical local MST root with the PDS commit root obtained through +Sync 1.1 `getLatestCommit` and `getBlocks`. Verification runs automatically for +a found account. + +Matching upstream, expensive repo actions are limited to four per source IP per +minute with a bounded 4,096-source map. Trusted operator deployments can pass +`--disable-repo-action-rate-limits`. + +## Timestamp import + +Disabled to remote callers unless a non-empty bearer token is configured. Stage +plain CSVs under the confined import directory, submit a relative path, then +poll the returned job id: + +``` +--timestamp-import-token= +--timestamp-import-dir=/srv/stream/imports + +POST /xrpc/network.bsky.jetstream.importTimestamps +GET /xrpc/network.bsky.jetstream.getImportStatus?job= +``` + +Both routes require `Authorization: Bearer `. Put them behind TLS at the +reverse proxy; Stream intentionally serves plain HTTP internally. Empty, +missing, and bad credentials all produce the same 401. Submitted paths are +canonicalized through symlinks and must resolve to regular files inside the +configured directory. + +## OpenTelemetry + +Stream installs the same process-wide tracing provider as pinned Jetstream V2. +Tracing is a real no-op unless `OTEL_EXPORTER_OTLP_ENDPOINT` or +`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` is set. When configured, completed spans +are batched and exported as OTLP/HTTP protobuf; the generic endpoint receives +the `/v1/traces` suffix and the trace-specific endpoint is used verbatim. +`--otel-service-name` and `OTEL_SERVICE_NAME` set `service.name`; the build +version supplies `service.version`. Full endpoint, header, gzip, insecure, +timeout, custom-CA, client-certificate, batch, and sampler controls are in +[`opentelemetry.md`](opentelemetry.md). + +The container includes the libcurl 4 runtime used *only* when an OTLP client +certificate and key are configured. Ordinary HTTPS and custom-CA-only exports +stay on Zig's native TLS path and do not load libcurl. diff --git a/docs/deployment-runbook.md b/docs/deployment-runbook.md index b1dd65d..d0c13b4 100644 --- a/docs/deployment-runbook.md +++ b/docs/deployment-runbook.md @@ -87,10 +87,20 @@ than trusting: | Unencodable record killed ingest | isolated per row, per wire | | Retry failures hidden behind green health | terminal failure stops the service | | Unadmitted image deployed | `admit verify` gates `deploy.sh`; it refuses `sha256:adc276…` | +| OOM retired a good repo as a malformed CAR | exhaustion stays `OutOfMemory`, pinned by an allocation-failure sweep | +| Stale operator CIDR could lock you out | `provision.sh` checks it against the live egress IP and refuses | **Admitted artifact for this run:** -`sha256:bb3e72377fa3459363155f934cf16b34eebe7f3bfffdce0dcce32a8d9eb158fd` -(`receipts/fde0966.json`, 20/20 suites, native linux/amd64). +`sha256:4300104135c6fd35bd13a264b744c5f842c41ee9a51e0cf1c57c9a59640551a2` +(`receipts/eb95c14.json`, 20/20 suites, native linux/amd64). + +It carries one fix beyond the previous artifact, and it is the kind this +runbook exists for: three sites reported `OutOfMemory` as a *structural* verdict +about a repository's CAR, and the engine retires a repository permanently on +that verdict. Under whole-network memory pressure — experiment 3's 24.3 GB — +Stream would have silently discarded good repositories and recorded their PDS +as the cause. See the whole-network bootstrap row in +[`semantic-parity.md`](semantic-parity.md). ## The plan diff --git a/docs/semantic-parity.md b/docs/semantic-parity.md index 1c3e844..fe5daf7 100644 --- a/docs/semantic-parity.md +++ b/docs/semantic-parity.md @@ -7,13 +7,18 @@ shape, or because a test written around Stream's implementation passes. ## Audit basis -- Stream: `fde096668e33494e7b64ddb60ddfadd85ed75617` -- admitted artifact: `sha256:bb3e72377fa3459363155f934cf16b34eebe7f3bfffdce0dcce32a8d9eb158fd` - (`receipts/fde0966.json`, 20/20 suites, built natively on linux/amd64) +- Stream: `eb95c14310ef18c7ab9edbcff991a04a560a7d86` +- admitted artifact: `sha256:4300104135c6fd35bd13a264b744c5f842c41ee9a51e0cf1c57c9a59640551a2` + (`receipts/eb95c14.json`, 20/20 suites, built natively on linux/amd64) - upstream Jetstream: `f29815c391fc2644f8a3dd36b899fb3697dd1ea6` - Atmos: `v0.2.14` -- Zat: `8db0560c2cb9357a16c9ee65205e3d247b0f7f63` (v0.3.18) -- audit date: 2026-07-26 +- Zat: `955e9ca9afb41d66d41e998803f6fcfc8fc4330f` (v0.3.22) +- audit date: 2026-07-28 + +Earlier receipts remain valid for the digests they cover — a receipt proves +those suites passed against *that* image, and that stays true. "Admitted" means +"earned a receipt", not "current", so confirm an artifact matches the commit +you intend before deploying it. The upstream checkout is exactly at the recorded pin. The untracked `simulator` path in that checkout is not treated as upstream source. diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..18b4244 --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,87 @@ +# Testing and receipts + +Stream's suites are the evidence behind [`semantic-parity.md`](semantic-parity.md). +`scripts/admit run` executes all of them against one image and writes a receipt +binding the results to that image's digest; a `skipped` suite blocks admission +exactly like a failure. See [`gotchas.md`](gotchas.md) for the traps that live +in the harness rather than in Stream. + +## Everyday + +``` +zig build test && zig build # unit tests + exe +just simulator # local fake atproto network on :7777 +just run-sim # stream against it +just e2e # python wire checks +just bench # subsystem benchmarks +``` + +## Contract suites + +Each runs real ReleaseSafe processes, offline. + +``` +just archive-contract # archive XRPCs + resident manifest + pinned Go client +just plan-config-contract # non-default planner limits/threshold +just cursor-lookback-contract # v1/v2 replay-window behavior +just compaction-config-contract # tombstone cap/rewrite workers, physical JSS +just retry-config-contract # global/per-host retry admission + backoff +just subscribe-config-contract # subscriber retention/cache/batching/slow controls +just subscribe-read-batch-contract # 5k-row filtered replay, exact cold pulls +just listener-contract # public/debug isolation + disabled/legacy modes +just status-contract # durable host rows + public HTTP view +just shutdown-contract # drain, close 1001, hung listRepos, restart +just logging-contract # formats, filtering, invalid configuration +just environment-contract # JETSTREAM_* map, precedence, unknown rejection +``` + +`environment-contract` proves the hand-maintained Stream map, sorted unknown +rejection, typed validation, CLI precedence, inspection, listener, storage, +logging, and import authentication. It does **not** prove the map contains +every flag in the pinned Go source; the gate requires a generated inventory +comparison for that. + +## Oracles + +``` +just oracle # lifecycle crash + RocksDB fault recovery matrix +just differential-oracle # exact pinned upstream semantics, fully offline +just differential-oracle-multiseed # five upstream predicate-kill worlds +just powerloss-image # one-time Linux NBD/ext4 tool-image bootstrap +just powerloss-oracle # real power-cut recovery, offline after bootstrap +``` + +The power-loss tier is deliberately separate from ordinary unit and process +tests. It cross-builds the production ReleaseSafe Linux binary, runs RocksDB +and JSS on ext4 over a kernel NBD device, kills the block backend *before* the +Stream process, and reconstructs the device only from writes acknowledged by +FLUSH/FUA. The campaign covers eight lifecycle cuts, the two-crash cleanup +guard, six delete-compaction cuts, and four timestamp-import rewrite cuts. +Mutation cases use archives written by the production JSS writer and verify +recovery through the production daemon and public V2 replay. + +It requires privileged Linux containers and an available `/dev/nbd3`. The +pinned Ubuntu tool image needs network access once for `just powerloss-image`; +`just powerloss-oracle` itself refuses image pulls and runs from cached Zig, +Python, simulator, and container inputs. The current recipe targets +`aarch64-linux-gnu`, matching the development host used for its recorded +receipt. + +## Dashboard and metrics + +Fully offline: + +``` +just dashboard-test +just process-metrics-contract +just http-metrics-contract +just dashboard-contract http://127.0.0.1:6008/metrics # against a local candidate +``` + +`process-metrics-contract` also runs the dashboard family-presence check +against the ReleaseSafe binary it launches. **Family presence does not +establish producer semantics, restart stability, or parity** — that distinction +is why the dashboard rows in the gate are still partial. +[`grafana-dashboard.md`](grafana-dashboard.md) records the checksum-pinned +upstream source, the deterministic Stream adaptation, and the fail-closed +metric contract. -- 2.51.2