diff --git a/README.md b/README.md index 5ec30b9..2452f23 100644 --- a/README.md +++ b/README.md @@ -2,59 +2,59 @@ a zig implementation of [jetstream](https://github.com/bluesky-social/jetstream) — the full-network archive and streaming service for atproto. built on [zat](https://tangled.org/zat.dev/zat). -The former live-only `stream.waow.tech` canary and the July 2026 isolated -experiment have been retired. No public Stream deployment is currently -promoted, and no current artifact is admitted for another whole-network -experiment. +## status -## capabilities +No public Stream deployment is currently promoted. The former live-only +`stream.waow.tech` canary and the July 2026 isolated experiments have been +retired. -- live Jetstream feeds at `/subscribe` and `/subscribe-v2`, with collection and - DID filters, cursor replay, runtime option updates, RFC 7692 - permessage-deflate on v1, frozen v1 zstd, and fetchable - dictionary-negotiated v2 zstd; cold replay continuity and remote-data - encoding isolation are not yet admitted -- a disk-backed historical archive exposed through `listSegments`, `getSegment`, - `getBlock`, and `planBackfill`; Stream's sealed `jss` v1 segments are - byte-compatible with upstream Jetstream. Sealed headers, block envelopes, - DID blooms, and collection summaries are resident and refresh atomically - after seal, compaction, and timestamp rewrite; planning opens no segment - files, while cold replay reads only selected compressed blocks. Internal - planner errors and unreadable replay segments still have fail-open paths -- a whole-network bootstrap implementation with concurrent live capture and - deterministic merge. Repository completion is coupled to ordinary archive - writer durability independently of the 100,000-entry dispatch/checkpoint - batch; cold replay, pending repair, and live-ingest paths remain blocked by - the deployment-gate audit. Startup resumes the existing active segment, +The implementation is **not** at semantic parity with pinned Jetstream V2 +commit `f29815c`. [`docs/semantic-parity.md`](docs/semantic-parity.md) is the +deployment gate and the only authority on what is proved: 23 verified rows, +4 partial, none blocked. Four experiments' worth of history — including what +each failure bought — is in +[`docs/full-network-experiment-2026-07.md`](docs/full-network-experiment-2026-07.md), +and [`docs/deployment-runbook.md`](docs/deployment-runbook.md) is what to +actually do, in order. + +An artifact is admitted for the next whole-network experiment: +`sha256:43001041…` (`receipts/eb95c14.json`, 20/20 suites, built natively on +linux/amd64). An admitted artifact means those suites passed against *that +digest* — it is not by itself a claim of parity. + +## what it does + +- **live feeds** at `/subscribe` and `/subscribe-v2`, with collection and DID + filters, cursor replay, runtime option updates, RFC 7692 permessage-deflate + on v1, frozen v1 zstd, and fetchable dictionary-negotiated v2 zstd +- **a disk-backed historical archive** through `listSegments`, `getSegment`, + `getBlock`, and `planBackfill`. Sealed `jss` v1 segments are byte-compatible + with upstream. Sealed headers, block envelopes, DID blooms, and collection + summaries are resident and refresh atomically after seal, compaction, and + timestamp rewrite; planning opens no segment files, and cold replay reads + only selected compressed blocks +- **whole-network bootstrap** with concurrent live capture and deterministic + merge. Repository completion is coupled to ordinary archive-writer durability + independently of the dispatch/checkpoint batch, so a restart does not replay + work already committed. Startup resumes the existing active segment, truncating and fsyncing only a framing-torn suffix while failing closed on complete-frame corruption and sealed high-water checksum failure -- Sync 1.1 commit verification, including PLC key rotation, MST inversion, - op-CID checks, replay protection, durable repository-chain state, and - transparent authenticated whole-repo repair with ordered pending replay; - the live scheduler implements Atmos-shaped 32-worker per-DID FIFO, bounded - drop-oldest, and completion-order batching. Reconnect and cursor-recovery - behavior remain deployment blockers -- RocksDB-backed metadata and archive publication coupled to segment fsync - boundaries; several correctness-metadata reads still fail open and are - called out in the deployment gate -- tombstone-aware compaction and continuing repair of repositories that were - unavailable or changed during bootstrap; the versioned compaction watermark - fails closed on corruption/read failure, while retry-supervisor failure - handling is not yet equivalent -- crash-resumable timestamp imports with a durable RocksDB rule map, +- **Sync 1.1 commit verification** — PLC key rotation, MST inversion, op-CID + checks, replay protection, durable repository-chain state, and transparent + authenticated whole-repo repair with ordered pending replay. The live + scheduler implements Atmos-shaped 32-worker per-DID FIFO, bounded + drop-oldest, and completion-order batching +- **RocksDB-backed metadata** and archive publication coupled to segment fsync + boundaries +- **tombstone-aware compaction** and continuing repair of repositories that + were unavailable or changed during bootstrap +- **crash-resumable timestamp imports** with a durable RocksDB rule map, topology-preserving sealed-segment repair, and bearer-gated operator XRPC -- browser and terminal operations views at `/status`, including durable PDS - diagnostics at `/status?tab=hosts` and per-repository reconstruction and - authoritative root verification at `/status?tab=accounts`, plus Prometheus - metrics at `/metrics` on the optional debug listener. The upstream - collections/segments status views and several lifecycle fields are absent - -The implementation is **not** at semantic parity with pinned Jetstream V2 -commit `f29815c`. The current evidence, blockers, and definition of a valid -admission are maintained in -[`docs/semantic-parity.md`](docs/semantic-parity.md). The July experiment is -historical evidence, not an admission receipt; see -[`docs/full-network-experiment-2026-07.md`](docs/full-network-experiment-2026-07.md). +- **operations views** at `/status`, including durable PDS diagnostics at + `/status?tab=hosts` and per-repository reconstruction and authoritative root + verification at `/status?tab=accounts`, plus Prometheus metrics at `/metrics` + on the optional debug listener. Upstream's collections/segments status views + are absent ## develop @@ -62,349 +62,31 @@ historical evidence, not an admission receipt; see zig build test && zig build # unit tests + exe just simulator # local fake atproto network on :7777 just run-sim # stream against it -just e2e # python wire checks -just archive-contract # archive XRPCs + resident manifest + pinned Go client -just plan-config-contract # non-default planner limits/threshold, real process -just cursor-lookback-contract # v1/v2 replay-window behavior, real processes -just compaction-config-contract # tombstone cap/rewrite workers, physical JSS -just retry-config-contract # global/per-host retry admission + backoff -just subscribe-config-contract # subscriber retention/cache/batching/slow controls -just subscribe-read-batch-contract # 5k-row filtered replay, exact cold pulls -just listener-contract # public/debug isolation + disabled/legacy modes -just status-contract # durable host rows + public HTTP view, offline -just oracle # lifecycle crash + RocksDB fault recovery matrix -just powerloss-image # one-time Linux NBD/ext4 tool-image bootstrap -just powerloss-oracle # real power-cut recovery, offline after bootstrap -just differential-oracle # exact pinned upstream semantics, fully offline -just differential-oracle-multiseed # five upstream predicate-kill worlds, offline -``` - -The binary also exposes upstream's offline operator commands. They do not -start the server or open RocksDB: - -``` -stream version -stream inspect-segment --blocks=table --blocks-truncate=100 -stream inspect-all --data-dir=./data --collections-truncate=100 -``` - -`inspect-segment` accepts `summary`, `table`, and `full` detail, handles both -sealed and active files, and reports a readable checksum-corrupt sealed file -as invalid without weakening the checksum enforcement used by archive reads. -The committed upstream-produced JSS fixture pins its rendered output byte for -byte (apart from the file path). -`inspect-all` walks both the steady archive and Stream's bootstrap live-capture -tree, aggregates network/tree/collection totals, tolerates a racing tail file, -and supports upstream's `--skip-unsealed` and collection-table truncation -controls. - -runtime flags are listed in `src/main.zig`; deploy notes in `deploy/README.md`. -The Zig build pins its Zig and native dependencies, including otel-zig, -RocksDB, zstd, xxHash, and websocket.zig's zlib 1.3.2; it does not depend on a -system zlib installation. - -Production configuration uses Jetstream V2's canonical `--relay-url`, -`--plc-url`, and `--max-backfill-repos` names. `--relay-url` drives both the -live WebSocket and HTTP repository clients; Stream's older split flags remain -accepted for existing local harnesses. - -Every canonical flag also reads the same `JETSTREAM_*` environment variable as -the pinned upstream command. Explicit CLI values take precedence without even -parsing an overridden environment value. Unknown `JETSTREAM_*` names fail the -process before command execution, catching misspelled production settings; -Kubernetes's `JETSTREAM_APP_*` service variables and upstream's -`JETSTREAM_ORACLE_*` and `JETSTREAM_SIM_*` test namespaces remain intentionally exempt. `inspect-all` inherits -`JETSTREAM_DATA_DIR`, and root logging variables remain persistent across every -command. `just environment-contract` proves the hand-maintained Stream map, -sorted unknown rejection, typed validation, CLI precedence, inspection, -listener, storage, logging, and timestamp-import authentication through real -ReleaseSafe processes without external network access. It does not prove that -the map contains every flag in the pinned Go source; the deployment gate now -requires a generated inventory comparison. - -The canonical server command also matches Jetstream V2's listener topology: - -``` -stream serve --addr=:8080 --debug-addr=127.0.0.1:6060 -``` - -The public listener owns `/`, `/status`, `/subscribe`, `/subscribe-v2`, and -the archive XRPC methods. The optional debug listener owns `/healthz`, -`/readyz`, and `/metrics`; an empty or omitted `--debug-addr` binds no debug -socket. `/readyz` means the configured listener accept loops are running, not -that bootstrap has reached steady state. Historical flag-only `--port=6008` -invocations retain Stream's combined listener solely for existing deployments -and offline harnesses. New deployments should use `serve --addr` and keep the -debug listener private. `just listener-contract` proves all three modes using -real ReleaseSafe processes without external network access. - -Intentional shutdown uses Jetstream V2's canonical `--shutdown-timeout` -(default `5s`) and `--client-drain-timeout` (default `10s`). On SIGINT or -SIGTERM, Stream stops listener admission, sends WebSocket close code 1001, -fans those notifications out concurrently so a backpressured client cannot -delay the others, drains cooperative subscribers until the client budget, and -shuts down public and debug HTTP traffic from the same instant while the -accepted ingest pipeline reaches its durable boundary. `just shutdown-contract` -exercises the real ReleaseSafe process offline with both cooperative and silent -peers. It also terminates bootstrap while live capture is connected and -`listRepos` is hung, then reopens the same data directory and repeats; active -backfill downloads are canceled while the archive flushes its recoverable -prefix. The transport suite also saturates one peer's real socket send buffer -and proves another peer still receives code 1001 before the shared deadline. -As upstream does, a non-positive HTTP shutdown timeout selects 30 seconds, -whereas a non-positive client-drain timeout is already expired: teardown is -immediate and delivery of the best-effort close frame is not guaranteed. - -Process logs follow Jetstream V2's canonical controls: `--log-level` accepts -debug/info/warn/error and their upstream aliases, while `--log-format` selects -`json` (the default) or `text`. Filtering remains runtime-configurable in -ReleaseSafe builds and applies to Stream plus its Zig dependencies. Root flags -remain persistent, so both `stream --log-level=debug serve` and -`stream serve --log-level=debug` work. `just logging-contract` proves the real -process formats, filtering, placement, and invalid-configuration failures -without external network access. - -Stream installs the same process-wide OpenTelemetry tracing provider as pinned -Jetstream V2. Tracing remains a real no-op unless -`OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` is set. -When configured, completed spans are batched and exported as OTLP/HTTP -protobuf; the generic endpoint receives the `/v1/traces` suffix and the -trace-specific endpoint is used verbatim. `--otel-service-name` and -`OTEL_SERVICE_NAME` set the `service.name` resource attribute, while the build -version supplies `service.version`. Standard endpoint, traces-endpoint, -header, gzip, insecure, timeout, custom-CA, client-certificate, batch, and -sampler environment controls are documented in -[`docs/opentelemetry.md`](docs/opentelemetry.md). Production -orchestrator, repository, live-batch, block-flush, and rotation work carries -the matching upstream span tree; an offline real-collector receipt exercises -both synchronous and asynchronous archive boundaries. - -The container includes the libcurl 4 runtime used only when an OTLP client -certificate and key are configured. Ordinary HTTPS and custom-CA-only exports -remain on Zig's native TLS path and do not load libcurl. - -The subscriber's writer-owned hot log retains 256 MiB by default and never -evicts rows above the durable archive watermark. Operators can tune the real -byte budget with `--subscribe-read-log-retention-bytes`. The upstream slow -client policy is also configurable with `--subscribe-slow-window` (default -`60s`) and `--subscribe-slow-min-rate` (default `5` log events/second); both -must remain continuously violated while a client is more than 100,000 events -behind before Stream disconnects it. - -Cold subscribers share a 64 MiB decoded-block cache by default, configurable -with `--subscribe-block-cache-bytes`. Its budget includes decoded JSS events, -memoized v1/v2 JSON, and the corresponding dictionary-zstd frames; compaction -and timestamp rewrites invalidate the affected segment generation. - -Subscriber pulls scan at most 1,024 raw log entries at a time by default, -configurable with `--subscribe-read-batch`. The boundary is applied before -filters and v1 skip rules in both hot and durable-disk reads, so selective -subscriptions still yield regularly and slow-client accounting follows raw -network progress. - -Sealed archive downloads use upstream's cache policy. The default is -`Cache-Control: public, no-cache`; operators may set a positive Go-style -duration such as `--segment-cache-max-age=15m`. As upstream does, Stream rounds -positive durations up to whole seconds in `max-age`. This is most useful when -compaction is disabled or the cache lifetime is comfortably shorter than the -configured compaction interval, because compaction can replace a segment while -retaining its name. - -`planBackfill` exposes the same CLI controls and defaults as upstream: - -``` ---plan-max-dids=1000 ---plan-max-collections=25 ---plan-max-entries=100000 ---plan-whole-segment-threshold=0.75 -``` - -Zero disables non-empty DID or collection filters for the corresponding -filter cap. For `--plan-max-entries`, zero instead means one unbounded page; -the threshold must be greater than zero and at most one. - -Subscriber replay implements upstream's 36-hour default lookback controls. A v1 sequence -cursor older than the retained window is conservatively clamped to the first -eligible sealed segment; v2 rejects the same cursor before WebSocket upgrade -with a `cursor too old` response. Timestamp cursors are clamped for both -protocols. Operators may change the window with a Go-style duration such as -`--cursor-lookback=72h`; zero makes both endpoints pure-live. The offline -`just cursor-lookback-contract` receipt exercises all three modes against real -sealed JSS through production Stream processes. Cold replay completeness is a -separate blocked row because an unreadable manifest-listed segment is currently -skipped. - -Delete compaction exposes upstream's resource controls: - -``` ---compaction-interval=4h ---compaction-tombstone-cap=32000000 ---compaction-rewrite-workers=0 -``` - -The interval accepts a non-negative Go-style duration and zero disables all -compaction. The tombstone cap bounds each durable fold/rewrite chunk; zero is -upstream's unlimited sentinel. Rewrite workers bound concurrent sealed-segment -rewrites; zero selects `min(CPU count, 8)`. Cap triggers wake the compactor -immediately, subject to upstream's 30-second between-pass floor. -`just compaction-config-contract` proves the controls reach the physical -compactor rather than stopping at argument parsing, including a 250 ms pass. It -does not waive the remaining lifecycle and serving admission gates. - -Failed-repository healing exposes upstream's complete control set: - -``` ---failed-repo-retry-interval=4h ---failed-repo-retry-workers=16 ---failed-repo-retry-host-workers=4 ---failed-repo-retry-max-delay=168h -``` - -The interval and maximum delay accept non-negative Go-style durations; zero -interval disables the background loop. Zero worker values select upstream's -16-global/4-per-host defaults, and zero maximum delay selects seven days. -Transient failures use upstream's capped exponential delay plus uniform -`[0, delay/2)` jitter. `just retry-config-contract` holds real `getRepo` -requests to measure both admission limits and then reopens RocksDB to verify -the durable retry schedule. The retry supervisor and candidate-scan design -remain blocked independently of those controls. - -The strict power-loss tier is deliberately separate from ordinary unit and -process tests. It cross-builds the production ReleaseSafe Linux binary, runs -RocksDB and JSS on ext4 over a kernel NBD device, kills the block backend before -the Stream process, and reconstructs the device only from writes acknowledged -by FLUSH/FUA. Its campaign covers eight lifecycle cuts, the two-crash cleanup -guard, six delete-compaction cuts, and four timestamp-import rewrite cuts. The -mutation cases use archives written by the production JSS writer and verify -recovery through the production daemon and public V2 replay. It therefore -requires privileged Linux containers and an -available `/dev/nbd3`. The pinned Ubuntu tool image needs network access once -for `just powerloss-image`; `just powerloss-oracle` itself refuses image pulls -and runs from cached Zig, Python, simulator, and container inputs. The current -recipe targets `aarch64-linux-gnu`, matching the development host used for its -recorded receipt. - -Production bootstrap controls: - -``` ---backfill ---backfill-workers=100 ---backfill-batch-size=100000 ---backfill-async-flush-workers=4 ---backfill-max-inflight-bytes=8589934592 ---skip-merge-discovery -``` - -Worker count controls concurrent network downloads. Batch size counts every -`listRepos` entry, including inactive and already-complete repositories; -Stream accumulates whole 1,000-entry protocol pages until it reaches the target, -then shuffles eligible repositories before dispatch. Requests still go through -the relay and follow its redirect, so this does not claim precise pre-request -PDS scheduling. Zero selects the 100,000-entry production default. The byte -flag bounds the owned -complete CAR plus preparation/index state. Preparation is serialized, copies -the downloaded CAR into that budget, and releases the scratch mmap before -emission. Already prepared repositories emit concurrently from the owned CAR -and loaded MST through fixed 1,024-row stack batches; this lets one repository -use the entire budget without allowing partial indexes to deadlock one another. -The zero worker value selects the 100-worker production default. HTTP bodies -remain concurrent and disk-backed. Four bootstrap-only workers detach complete -JSS blocks, perform real zstd compression concurrently, and commit/fsync them -in preparation order; zero explicitly selects synchronous compression. -Downloads need temporary -free space on `--data-dir` equal to the concurrently in-flight CARs; -scratch files are deleted after each attempt and cleared on restart. Bootstrap -owns `/backfill/repo-scratch`; steady live and failed-repo healing use -`/repair-scratch/{live,failed-repo}`, so repair cannot recreate -lifecycle state after cutover. Resource exhaustion is lifecycle-fatal, never -persisted as a fake "large repo skipped" success/failure. Backfill progress, -active workers, queue depth, current -transient bytes, and peak transient bytes are exported on `/metrics`. -Successful CARs record their final emitted archive sequence; ordinary writer -durability commits every covered repository independently. The dispatch batch -still controls listRepos cursor-checkpoint cadence, but a slow sibling no -longer withholds already-durable repository completion. The authoritative -lifecycle analysis is -[`docs/bootstrap-semantic-parity.md`](docs/bootstrap-semantic-parity.md). - -The failed-repo loop uses 16 global workers and four workers per final -post-redirect PDS host; attempts, error class/message, final host, retry count, -and next attempt are durable in RocksDB. Every repo transition also updates a -persistent `host/` diagnostic aggregate in the same synced batch: -current lifecycle/active counts, cumulative failure classes, and the five most -recent bounded error samples therefore survive restart without scanning the -whole-network repo keyspace. The browser and text host views read those rows -directly, preserve upstream's largest/failing sort order, and filter host and -recent-error content in the browser. This implementation is not admission to -another full-network run by itself: `docs/semantic-parity.md` remains -authoritative. - -The debug-only discovery flag omits the post-merge `listRepos` rescan. Matching -upstream, Stream enables it automatically for `--backfill-max-repos` and -`--backfill-repos`; a normal full crawl still replays the last page so accounts -created during bootstrap enter durable retry state. The pinned-simulator -process receipt observes three real `listRepos` requests for a normal two-page -crawl and exactly two when discovery is skipped. The same receipt proves a -page-sized batch dispatches before page two, while a cross-page batch and the -zero/default setting enumerate both pages before the first real `getRepo`. -Initial and post-bootstrap enumeration retain arbitrary-length relay cursors; -the real-process receipt follows a 4 KiB cursor and verifies that discovery -durably records both active and inactive accounts. - -The same durable account row retains the distinct initial-backfill and latest -known revisions, update time, declared handle, PDS endpoint, and reserved -record/byte counters. Matching upstream, the debug-only explicit -`--backfill-repos=did:plc:example[,did:web:example.com]` path processes the -requested DIDs serially, resolves their real DID documents, and removes stale -full-network merge-discovery state. Ordinary whole-network `listRepos` -bootstrap does not add millions of extra identity requests. -Normal post-bootstrap discovery is not yet admitted: inactive unknown -repositories are skipped and an oversized cursor can silently terminate the -scan. - -`handle/` is maintained atomically with every repo transition, and -each merge-source commit advances its cursor together with latest-revision -refreshes. Account lookup resolves handles through the configured identity -directory, reconstructs the repository from sealed JSS segments, the active -writer, pending rows, and bootstrap live segments at a rotation-safe boundary, -then compares the canonical local MST root with the PDS commit root obtained -through Sync 1.1 `getLatestCommit` and `getBlocks`. Verification runs -automatically for a found account. Matching upstream, expensive repo actions -are limited to four per source IP per minute with a bounded 4,096-source map; -trusted operator deployments can explicitly pass -`--disable-repo-action-rate-limits`. - -Timestamp import is disabled to remote callers unless a non-empty bearer token -is configured. Stage plain CSVs under the confined import directory (default -`/imports`), submit a relative path, then poll the returned job id: - -``` ---timestamp-import-token= ---timestamp-import-dir=/srv/stream/imports - -POST /xrpc/network.bsky.jetstream.importTimestamps -GET /xrpc/network.bsky.jetstream.getImportStatus?job= ``` -Both routes require `Authorization: Bearer `. Put them behind TLS at the -reverse proxy; Stream intentionally serves plain HTTP internally. Empty tokens, -missing credentials, and bad credentials all produce the same 401 response. -Submitted paths are canonicalized through symlinks and must resolve to regular -files inside the configured directory. +The full suite list, the oracles, and the power-loss tier are in +[`docs/testing.md`](docs/testing.md). The Zig build pins its Zig and native +dependencies, including otel-zig, RocksDB, zstd, xxHash, and websocket.zig's +zlib 1.3.2; it does not depend on a system zlib. ## docs -The docs folder holds the port record — jss format byte spec, seal spec, -upstream bootstrap/compaction/subscribe condensations (every deliberate -divergence is written down there), lessons carried over from zlay, and durable -product/copy context for public surfaces. `docs/semantic-parity.md` is the -top-level deployment gate. `docs/grafana-dashboard.md` records the -checksum-pinned upstream source, deterministic Stream adaptation, and its -fail-closed metric contract. The dashboard and process receipts are fully -offline: run `just dashboard-test`, `just process-metrics-contract`, -`just http-metrics-contract`, then -`just dashboard-contract http://127.0.0.1:6008/metrics` against a local -candidate. `just process-metrics-contract` also runs that dashboard -family-presence check against the ReleaseSafe binary it launches. Family -presence does not establish producer semantics, restart stability, or parity; -the dashboard audit in `docs/semantic-parity.md` is the deployment gate. +Start here depending on what you need: + +| | | +|---|---| +| [`semantic-parity.md`](docs/semantic-parity.md) | **the deployment gate** — what is proved, row by row | +| [`deployment-runbook.md`](docs/deployment-runbook.md) | staged plan and failure responses for a whole-network run | +| [`architecture.md`](docs/architecture.md) | how the subsystems fit together | +| [`invariants.md`](docs/invariants.md) | the rules that must never break, and what each cost | +| [`gotchas.md`](docs/gotchas.md) | deliberate behavior that looks wrong; traps in the harness | +| [`configuration.md`](docs/configuration.md) | every runtime control and its default | +| [`cli.md`](docs/cli.md) | offline operator commands | +| [`testing.md`](docs/testing.md) | suites, oracles, receipts | +| [`benchmarks.md`](docs/benchmarks.md) | subsystem benchmarks | + +The rest of `docs/` holds the port record: the jss format byte spec and seal +spec, upstream bootstrap/compaction/subscribe condensations where every +deliberate divergence is written down, the live scheduler and repair notes, +lessons carried over from zlay, and durable product/copy context for public +surfaces. diff --git a/docs/cli.md b/docs/cli.md new file mode 100644 index 0000000..265608d --- /dev/null +++ b/docs/cli.md @@ -0,0 +1,30 @@ +# Offline operator commands + +The binary exposes upstream's offline inspection commands. They do not start +the server or open RocksDB, so they are safe against a data directory a live +process owns. + +``` +stream version +stream inspect-segment --blocks=table --blocks-truncate=100 +stream inspect-all --data-dir=./data --collections-truncate=100 +``` + +`version` emits the upstream build-information field shape; values necessarily +identify the Stream artifact rather than a Go build. + +`inspect-segment` accepts `summary`, `table`, and `full` detail and handles +both sealed and active files. It reports a readable checksum-corrupt sealed +file as invalid *without* weakening the checksum enforcement used by archive +reads — inspection must never be a way to read bytes the server would refuse. +A committed upstream-produced JSS fixture pins its rendered output byte for +byte, apart from the file path. + +`inspect-all` walks both the steady archive and Stream's bootstrap live-capture +tree, aggregates network/tree/collection totals, tolerates a racing tail file, +and supports upstream's `--skip-unsealed` and collection-table truncation +controls. It inherits `JETSTREAM_DATA_DIR`, and root logging variables persist +across every command. + +Note that `inspect-all` does not replace the online collections and segments +status tabs, which remain absent — see [`semantic-parity.md`](semantic-parity.md). diff --git a/docs/configuration-parity.md b/docs/configuration-parity.md index fa5face..b61f8bb 100644 --- a/docs/configuration-parity.md +++ b/docs/configuration-parity.md @@ -4,8 +4,8 @@ Authority: - upstream `cmd/jetstream/main.go` and `internal/jetstreamd/options.go` at `f29815c391fc2644f8a3dd36b899fb3697dd1ea6` -- Stream `a345f4054df568e364aff08a9b1726275fc48e29` plus the discovery - changes and audit updates in this commit +- Stream `eb95c14310ef18c7ab9edbcff991a04a560a7d86` + (admitted artifact `sha256:43001041...`, `receipts/eb95c14.json`) This document audits public configuration and command wiring. A **verified** configuration row proves only that the input is parsed/defaulted and reaches @@ -18,37 +18,37 @@ semantically correct. Status vocabulary is defined in | Surface | Status | What source/tests establish | Limitation or remaining proof | |---|---|---|---| | Complete upstream flag inventory | **unverified** | The environment contract contains a hand-maintained pinned source map and rejects unknown `JETSTREAM_*` names. | There is no generated comparison proving every current upstream flag is represented here. Before another admission, generate the inventory from the pinned Go command/options definitions and fail on an unclassified addition/removal. | -| Relay URL | **partial** | `--relay-url` reaches the WebSocket consumer and relay-fronted HTTP listRepos/getRepo/repair clients; CLI and environment precedence are tested. | Correct wiring still feeds a blocked Zat reconnect loop. | +| Relay URL | **verified** | `--relay-url` reaches the WebSocket consumer and relay-fronted HTTP listRepos/getRepo/repair clients; CLI and environment precedence are tested. | The reconnect loop this feeds is no longer blocked: the process-killing race was a `setsockopt` `unreachable` in the shared websocket.zig fork, fixed there and shipped through zat v0.3.18. | | PLC URL | **verified** | `--plc-url` configures the real DID resolver used by live verification and selected-repo metadata; loopback resolution tests observe requests. | This row does not prove every verification/repair outcome. | -| Data directory | **partial** | `--data-dir` owns JSS, scratch, metadata, bootstrap-live, import, and temporary trees in production-process tests. Active-tail startup recovery now resumes in place and fails closed on corruption. | Cold replay, timestamp-import visibility, and retry metadata still have independent blocked paths; path ownership alone does not close them. | -| `--max-backfill-repos` | **partial** | Canonical spelling is parsed, limits dispatch, avoids committing a whole-network resume cursor, and automatically skips merge discovery. | The lifecycle and completion paths it invokes are blocked. It is a debug mode, not evidence for whole-network parity. | +| Data directory | **partial** | `--data-dir` owns JSS, scratch, metadata, bootstrap-live, import, and temporary trees in production-process tests. Active-tail startup recovery now resumes in place and fails closed on corruption. | Path ownership alone does not establish the behavior of what lives there. The cold-replay and retry-metadata blockers it used to cite are closed; the remaining caveat is that timestamp-import visibility is proved only for the cases in that row. | +| `--max-backfill-repos` | **partial** | Canonical spelling is parsed, limits dispatch, avoids committing a whole-network resume cursor, and automatically skips merge discovery. | It is a debug mode, not evidence for whole-network parity -- a capped run dispatches a fresh slice rather than exercising a full crawl. The lifecycle and completion paths it invokes are no longer blocked. | | `--backfill-repos` | **partial** | Canonical spelling and compatibility alias parse; it is mutually exclusive with max-repos, preserves selected order, bypasses listRepos, and enables selected identity resolution. | The exact current artifact still needs the complete selected-repository admission receipt; configuration wiring itself is established. | | Backfill workers | **verified** | Explicit positive values set physical getRepo concurrency; zero/omitted default to 100. Held real requests observe 7/100/100. | The 200-worker production experiment showed that accepting a value does not mean it is efficient. | | Backfill batch size | **verified** | Explicit and zero/default values control only page-aligned listRepos accumulation and cursor-checkpoint cadence; zero/omitted select 100,000. Repository completion now follows independent writer durability boundaries. | This control does not define whole-network size or completion percentage. | | Backfill async-flush workers | **partial** | Omitted defaults to four, positive values start that many bootstrap compression workers, and zero selects synchronous compression. Focused tests observe overlap and byte-equivalent output. | Full cancellation/OOM state coverage was not re-established in this audit. | | Skip merge discovery | **verified** | Explicit configuration and automatic max/selected enablement prevent the discovery request in focused tests. Normal discovery independently preserves active/inactive unknown repositories and follows arbitrary-length cursors. | This is a debug/partial-selection control; enabling it for a whole-network crawl intentionally omits accounts born during bootstrap. | -| Failed-repo retry interval | **partial** | Duration parsing, zero disablement, and timer-driven pass execution have real-process tests. | Pass failures are hidden from health and candidate scans are unbounded in RAM. | -| Failed-repo global/per-host workers | **partial** | Explicit and zero/default values reach real 16-global/4-per-host gates in held-request tests. | Pending-merge retry uses a different path, and the steady candidate set is materialized eagerly. | -| Failed-repo maximum delay | **partial** | Parsed values affect persisted exponential-jitter deadlines in focused RocksDB tests. | Host-park read corruption fails open; whole retry supervision is blocked. | +| Failed-repo retry interval | **verified** | Duration parsing, zero disablement, and timer-driven pass execution have real-process tests. | Closed: a terminal pass failure now stops the service rather than sleeping until the next interval, and candidates stream through a bounded queue instead of being materialized in RAM. | +| Failed-repo global/per-host workers | **partial** | Explicit and zero/default values reach real 16-global/4-per-host gates in held-request tests. | Pending-merge retry uses a different path, though it now runs through the same runner. The steady candidate set is no longer materialized eagerly. | +| Failed-repo maximum delay | **partial** | Parsed values affect persisted exponential-jitter deadlines in focused RocksDB tests. | Retry supervision is no longer blocked, and host parking is no longer persisted at all -- it moved into the runner's memory to match upstream, which removes the fail-open read this row described. | | Expensive account verification limiter | **verified** | Enabled/default and disabled modes cross the real `/status?tab=accounts` HTTP path. Four requests per source IP succeed, the fifth is rejected, and HEAD does not consume a slot. | This verifies the control, not the completeness of the status surface. | -| Cursor lookback | **partial** | Default 36h, explicit duration, and zero/pure-live change v1/v2 cursor admission in production-process tests. | Accepted cold cursors still depend on a replay walker that can skip unreadable segments. | +| Cursor lookback | **verified** | Default 36h, explicit duration, and zero/pure-live change v1/v2 cursor admission in production-process tests. | The replay walker no longer skips an unreadable manifest-listed segment; it propagates, because skipping would advance a cursor past durable events nothing revisits. | | Segment cache max age | **verified** | Go-duration parsing and ceil-to-seconds response headers are observed on real getSegment/getBlock responses; zero/default no-cache behavior is covered. | None known for this control. | -| planBackfill DID/collection caps | **partial** | Positive and zero sentinel values affect request validation in real HTTP tests. | A valid request can still receive a false-negative plan when `matchBlocks` encounters an internal error. | -| planBackfill entry limit | **partial** | Positive values paginate by work unit; zero disables the cap; continuation fixtures match normal upstream responses. | Internal planner errors are silently converted into non-matches. | -| planBackfill density threshold | **partial** | Configured thresholds change whole-segment versus block mode at tested boundaries. | Same planner error blocker. | +| planBackfill DID/collection caps | **verified** | Positive and zero sentinel values affect request validation in real HTTP tests. | Closed: a `matchBlocks` internal error propagates and the request answers 5xx rather than returning a truthful-looking empty plan. | +| planBackfill entry limit | **verified** | Positive values paginate by work unit; zero disables the cap; continuation fixtures match normal upstream responses. | Closed: planner errors answer rather than degrading into non-matches; an allocation-failure sweep asserts the planner never returns 200 with a short plan. | +| planBackfill density threshold | **verified** | Configured thresholds change whole-segment versus block mode at tested boundaries. | The planner error blocker this referenced is closed. | | Subscribe hot-log retention | **verified** | The byte value drives physical eviction in the production tail; tests cover eviction while respecting the durable floor. | Does not verify cold recovery after eviction. | | Subscribe slow-consumer window/rate | **verified** | Parsed duration and fractional rate reach the per-subscriber scan detector; explicit and default/non-positive behavior have focused tests. | None known for the detector control. | -| Subscribe cold-block cache | **partial** | The byte bound controls the shared decoded-block/wire memo LRU; concurrency, eviction, and manifest-generation invalidation have tests. | The enclosing cold walker skips an unreadable segment, so cache correctness cannot establish replay completeness. | +| Subscribe cold-block cache | **partial** | The byte bound controls the shared decoded-block/wire memo LRU; concurrency, eviction, and manifest-generation invalidation have tests. | Cache correctness alone does not establish replay completeness, but the enclosing cold walker no longer skips an unreadable segment. | | Subscribe read batch | **partial** | Explicit positive values and non-positive default 1024 bound raw scanned entries before filters in hot and normal cold fixtures. | The 5,000-row receipt tests batching, not missing-file continuity. | | Deprecated cursor block-index cache flag | **intentional divergence** | Stream accepts signed values as a compatibility no-op because its sealed indexes are always manifest-resident. Parser behavior is tested. | This is not the same runtime mechanism as an implementation that uses the cache. Keep it documented as compatibility, not parity. | | Compaction interval | **verified** | Go-duration parsing, zero disablement, periodic execution, and cap-trigger wakeup are exercised. The enclosing pass now propagates malformed/unreadable watermark errors. | This verifies the control and its watermark precondition, not every lifecycle cutover. | | Compaction tombstone cap | **verified** | Explicit and zero/default values affect real chunk count and physical survivors; watermark corruption cannot silently reset the fold range. | None known for this control. | | Compaction rewrite workers | **verified** | The configured value controls real concurrent rewrite groups in focused physical-file tests, with strict watermark admission. | None known for this control. | | Timestamp import token | **verified** | Missing/wrong/correct bearer behavior crosses the production import/status XRPC handlers, including constant-width comparison. | None known for the token control. | -| Timestamp import directory | **partial** | Configured roots are canonicalized/confined and real jobs survive restart in focused tests. | Corrupt import metadata reads were not re-audited; public results depend on blocked cold replay. | +| Timestamp import directory | **partial** | Configured roots are canonicalized/confined and real jobs survive restart in focused tests. | Corrupt import metadata reads were not re-audited. The cold-replay dependency this row cited is closed. | | Public bind address | **verified** | `serve --addr` binds the protocol listener; public route and WebSocket behavior are exercised by real processes. | Route availability does not imply archive readiness/completeness. | | Debug bind address | **verified** | Empty disables binding; configured debug address exclusively serves health/readiness/metrics and rejects protocol upgrade. Historical combined mode is separately tested. | None known for listener wiring. | -| Shutdown timeout | **partial** | Signed duration parsing/defaults reach listener and lifecycle shutdown; real tests cover positive, non-positive, hung listRepos, and cooperative exit. | Exact-artifact admission was violated, and reconnect failures remain independently process-fatal. | +| Shutdown timeout | **verified** | Signed duration parsing/defaults reach listener and lifecycle shutdown; real tests cover positive, non-positive, hung listRepos, and cooperative exit. | Both former limitations are closed: exact-artifact admission is now enforced by `scripts/admit verify` and `deploy.sh`, and the process-fatal reconnect path was a `setsockopt` `unreachable` in websocket.zig, fixed and shipped through zat v0.3.18. Consistent with the **verified** graceful-shutdown row in [semantic-parity.md](semantic-parity.md). | | Client drain timeout | **verified** | Configured/default/non-positive budgets reach concurrent client close hooks; real blocked-socket tests observe 1001 and shared-deadline interruption. | Limited to client-drain behavior. | | Log level | **verified** | Case-insensitive aliases/default and invalid values affect all `std.log` producers in real ReleaseSafe processes. | Not byte-identical Go logger internals. | | Log format | **verified** | JSON default and text selection produce the documented canonical fields in real processes. | Not byte-identical Go formatting beyond the asserted fields. | diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..cbc76b8 --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,280 @@ +# Configuration reference + +What each runtime control does and what its defaults are. This is the operator +reference; [`configuration-parity.md`](configuration-parity.md) is the audit of +how well each one is *proved*, and [`semantic-parity.md`](semantic-parity.md) +is the deployment gate. Runtime flags are declared in `src/main.zig`; deploy +notes live in `deploy/README.md`. + +Stream uses Jetstream V2's canonical names. Every canonical flag also reads the +same `JETSTREAM_*` environment variable as the pinned upstream command. +Explicit CLI values take precedence without even parsing an overridden +environment value. Unknown `JETSTREAM_*` names fail the process before command +execution, which catches misspelled production settings; Kubernetes's +`JETSTREAM_APP_*` service variables and upstream's `JETSTREAM_ORACLE_*` and +`JETSTREAM_SIM_*` test namespaces are intentionally exempt. + +`--relay-url` drives both the live WebSocket and the HTTP repository clients. +`--plc-url` configures the DID resolver. Stream's older split flags remain +accepted for existing local harnesses. + +## Intentional divergences + +Two controls deliberately differ from upstream. Neither should be described as +parity — the configuration audit requires them recorded here explicitly. + +**The deprecated cursor block-index cache flag is a compatibility no-op.** +Stream accepts signed values and ignores them, because its sealed indexes are +always manifest-resident. Accepting the flag keeps existing command lines +working; it is not the same runtime mechanism as an implementation that uses +the cache. + +**There are no `/debug/pprof/*` routes.** Stream is not Go, and a route-shaped +stub returning something pprof cannot use would be worse than an honest +absence. Zig and process metrics on the debug listener serve that purpose +instead. + +## Listeners + +``` +stream serve --addr=:8080 --debug-addr=127.0.0.1:6060 +``` + +The public listener owns `/`, `/status`, `/subscribe`, `/subscribe-v2`, and the +archive XRPC methods. The optional debug listener owns `/healthz`, `/readyz`, +and `/metrics`; an empty or omitted `--debug-addr` binds no debug socket. + +`/readyz` means the configured accept loops are running — **not** that +bootstrap has reached steady state. Historical flag-only `--port=6008` +invocations retain Stream's combined listener solely for existing deployments +and offline harnesses; new deployments should use `serve --addr` and keep the +debug listener private. + +## Shutdown + +``` +--shutdown-timeout=5s +--client-drain-timeout=10s +``` + +On SIGINT or SIGTERM, Stream stops listener admission, sends WebSocket close +code 1001, fans those notifications out concurrently so a backpressured client +cannot delay the others, drains cooperative subscribers until the client +budget, and shuts down public and debug HTTP from the same instant while the +accepted ingest pipeline reaches its durable boundary. + +As upstream does, a non-positive HTTP shutdown timeout selects 30 seconds, +whereas a non-positive client-drain timeout is *already expired*: teardown is +immediate and delivery of the best-effort close frame is not guaranteed. + +## Logging + +`--log-level` accepts debug/info/warn/error and their upstream aliases; +`--log-format` selects `json` (default) or `text`. Filtering is +runtime-configurable in ReleaseSafe builds and applies to Stream plus its Zig +dependencies. Root flags are persistent, so both `stream --log-level=debug +serve` and `stream serve --log-level=debug` work. + +## Subscribers + +The writer-owned hot log retains 256 MiB by default and never evicts rows above +the durable archive watermark; tune with +`--subscribe-read-log-retention-bytes`. + +Upstream's slow-client policy is configurable with `--subscribe-slow-window` +(default `60s`) and `--subscribe-slow-min-rate` (default `5` log +events/second). Both must remain *continuously* violated while a client is more +than 100,000 events behind before Stream disconnects it. + +Cold subscribers share a 64 MiB decoded-block cache +(`--subscribe-block-cache-bytes`) covering decoded JSS events, memoized v1/v2 +JSON, and the corresponding dictionary-zstd frames; compaction and timestamp +rewrites invalidate the affected segment generation. + +Pulls scan at most 1,024 raw log entries at a time (`--subscribe-read-batch`). +The boundary applies before filters and v1 skip rules in both hot and durable +reads, so selective subscriptions still yield regularly and slow-client +accounting follows raw network progress. + +### Replay window + +Subscriber replay implements upstream's 36-hour default lookback. A v1 sequence +cursor older than the retained window is conservatively clamped to the first +eligible sealed segment; v2 rejects the same cursor before the WebSocket +upgrade with a `cursor too old` response. Timestamp cursors are clamped for +both protocols. `--cursor-lookback=72h` changes the window; zero makes both +endpoints pure-live. + +## Archive serving + +Sealed downloads use upstream's cache policy. The default is `Cache-Control: +public, no-cache`; a positive Go-style duration such as +`--segment-cache-max-age=15m` is rounded up to whole seconds in `max-age`. This +is most useful when compaction is disabled or the cache lifetime is comfortably +shorter than the compaction interval, because compaction can replace a segment +while retaining its name. + +`planBackfill` exposes upstream's controls and defaults: + +``` +--plan-max-dids=1000 +--plan-max-collections=25 +--plan-max-entries=100000 +--plan-whole-segment-threshold=0.75 +``` + +Zero disables non-empty DID or collection filters for the corresponding cap. +For `--plan-max-entries`, zero instead means one unbounded page. The threshold +must be greater than zero and at most one. + +## Compaction + +``` +--compaction-interval=4h +--compaction-tombstone-cap=32000000 +--compaction-rewrite-workers=0 +``` + +The interval accepts a non-negative Go-style duration; zero disables all +compaction. The tombstone cap bounds each durable fold/rewrite chunk; zero is +upstream's unlimited sentinel. Rewrite workers bound concurrent sealed-segment +rewrites; zero selects `min(CPU count, 8)`. Cap triggers wake the compactor +immediately, subject to upstream's 30-second between-pass floor. + +## Failed-repository healing + +``` +--failed-repo-retry-interval=4h +--failed-repo-retry-workers=16 +--failed-repo-retry-host-workers=4 +--failed-repo-retry-max-delay=168h +``` + +Interval and maximum delay accept non-negative Go-style durations; a zero +interval disables the background loop. Zero worker values select upstream's +16-global/4-per-host defaults, and zero maximum delay selects seven days. +Transient failures use upstream's capped exponential delay plus uniform +`[0, delay/2)` jitter. + +Attempts, error class/message, final host, retry count, and next attempt are +durable in RocksDB. Every repo transition also updates a persistent +`host/` diagnostic aggregate in the same synced batch, so current +lifecycle/active counts, cumulative failure classes, and the five most recent +bounded error samples survive restart without scanning the whole-network repo +keyspace. + +## Bootstrap + +``` +--backfill +--backfill-workers=100 +--backfill-batch-size=100000 +--backfill-async-flush-workers=4 +--backfill-max-inflight-bytes=8589934592 +--skip-merge-discovery +``` + +Worker count controls concurrent network downloads; zero selects the 100-worker +default. Batch size counts every `listRepos` entry, including inactive and +already-complete repositories: Stream accumulates whole 1,000-entry protocol +pages until it reaches the target, then shuffles eligible repositories before +dispatch. Requests still go through the relay and follow its redirect, so this +does not claim precise pre-request PDS scheduling. Zero selects the +100,000-entry default. + +The byte flag bounds the owned complete CAR plus preparation/index state. +Preparation is serialized, copies the downloaded CAR into that budget, and +releases the scratch mmap before emission. Already-prepared repositories emit +concurrently from the owned CAR and loaded MST through fixed 1,024-row stack +batches, which lets one repository use the entire budget without letting +partial indexes deadlock one another. Four bootstrap-only workers detach +complete JSS blocks, compress concurrently, and commit/fsync in preparation +order; zero selects synchronous compression. + +Downloads need temporary free space on `--data-dir` equal to the concurrently +in-flight CARs. Scratch files are deleted after each attempt and cleared on +restart. Bootstrap owns `/backfill/repo-scratch`; steady live and +failed-repo healing use `/repair-scratch/{live,failed-repo}`, so +repair cannot recreate lifecycle state after cutover. + +Resource exhaustion is lifecycle-fatal, never persisted as a fake "large repo +skipped" success or failure — and it must never be misfiled as malformed remote +data either, which is the invariant [`invariants.md`](invariants.md) records +and an allocation-failure sweep now pins. + +Backfill progress, active workers, queue depth, and current/peak transient +bytes are exported on `/metrics`. Successful CARs record their final emitted +archive sequence, and ordinary writer durability commits every covered +repository independently: the dispatch batch controls listRepos +cursor-checkpoint cadence, but a slow sibling no longer withholds +already-durable repository completion. The authoritative lifecycle analysis is +[`bootstrap-semantic-parity.md`](bootstrap-semantic-parity.md). + +### Discovery and selected repositories + +`--skip-merge-discovery` omits the post-merge `listRepos` rescan. Matching +upstream, Stream enables it automatically for `--max-backfill-repos` and +`--backfill-repos`; a normal full crawl still replays the last page so accounts +created during bootstrap enter durable retry state. Enabling it for a +whole-network crawl intentionally omits accounts born during bootstrap. + +The durable account row retains the distinct initial-backfill and latest known +revisions, update time, declared handle, PDS endpoint, and reserved record/byte +counters. The debug-only explicit +`--backfill-repos=did:plc:example[,did:web:example.com]` path processes the +requested DIDs serially, resolves their real DID documents, and removes stale +full-network merge-discovery state. Ordinary whole-network bootstrap does not +add millions of extra identity requests. + +`handle/` is maintained atomically with every repo transition, and +each merge-source commit advances its cursor together with latest-revision +refreshes. + +## Account verification + +Account lookup resolves handles through the configured identity directory, +reconstructs the repository from sealed JSS segments, the active writer, +pending rows, and bootstrap live segments at a rotation-safe boundary, then +compares the canonical local MST root with the PDS commit root obtained through +Sync 1.1 `getLatestCommit` and `getBlocks`. Verification runs automatically for +a found account. + +Matching upstream, expensive repo actions are limited to four per source IP per +minute with a bounded 4,096-source map. Trusted operator deployments can pass +`--disable-repo-action-rate-limits`. + +## Timestamp import + +Disabled to remote callers unless a non-empty bearer token is configured. Stage +plain CSVs under the confined import directory, submit a relative path, then +poll the returned job id: + +``` +--timestamp-import-token= +--timestamp-import-dir=/srv/stream/imports + +POST /xrpc/network.bsky.jetstream.importTimestamps +GET /xrpc/network.bsky.jetstream.getImportStatus?job= +``` + +Both routes require `Authorization: Bearer `. Put them behind TLS at the +reverse proxy; Stream intentionally serves plain HTTP internally. Empty, +missing, and bad credentials all produce the same 401. Submitted paths are +canonicalized through symlinks and must resolve to regular files inside the +configured directory. + +## OpenTelemetry + +Stream installs the same process-wide tracing provider as pinned Jetstream V2. +Tracing is a real no-op unless `OTEL_EXPORTER_OTLP_ENDPOINT` or +`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` is set. When configured, completed spans +are batched and exported as OTLP/HTTP protobuf; the generic endpoint receives +the `/v1/traces` suffix and the trace-specific endpoint is used verbatim. +`--otel-service-name` and `OTEL_SERVICE_NAME` set `service.name`; the build +version supplies `service.version`. Full endpoint, header, gzip, insecure, +timeout, custom-CA, client-certificate, batch, and sampler controls are in +[`opentelemetry.md`](opentelemetry.md). + +The container includes the libcurl 4 runtime used *only* when an OTLP client +certificate and key are configured. Ordinary HTTPS and custom-CA-only exports +stay on Zig's native TLS path and do not load libcurl. diff --git a/docs/deployment-runbook.md b/docs/deployment-runbook.md index b1dd65d..d0c13b4 100644 --- a/docs/deployment-runbook.md +++ b/docs/deployment-runbook.md @@ -87,10 +87,20 @@ than trusting: | Unencodable record killed ingest | isolated per row, per wire | | Retry failures hidden behind green health | terminal failure stops the service | | Unadmitted image deployed | `admit verify` gates `deploy.sh`; it refuses `sha256:adc276…` | +| OOM retired a good repo as a malformed CAR | exhaustion stays `OutOfMemory`, pinned by an allocation-failure sweep | +| Stale operator CIDR could lock you out | `provision.sh` checks it against the live egress IP and refuses | **Admitted artifact for this run:** -`sha256:bb3e72377fa3459363155f934cf16b34eebe7f3bfffdce0dcce32a8d9eb158fd` -(`receipts/fde0966.json`, 20/20 suites, native linux/amd64). +`sha256:4300104135c6fd35bd13a264b744c5f842c41ee9a51e0cf1c57c9a59640551a2` +(`receipts/eb95c14.json`, 20/20 suites, native linux/amd64). + +It carries one fix beyond the previous artifact, and it is the kind this +runbook exists for: three sites reported `OutOfMemory` as a *structural* verdict +about a repository's CAR, and the engine retires a repository permanently on +that verdict. Under whole-network memory pressure — experiment 3's 24.3 GB — +Stream would have silently discarded good repositories and recorded their PDS +as the cause. See the whole-network bootstrap row in +[`semantic-parity.md`](semantic-parity.md). ## The plan diff --git a/docs/semantic-parity.md b/docs/semantic-parity.md index 1c3e844..fe5daf7 100644 --- a/docs/semantic-parity.md +++ b/docs/semantic-parity.md @@ -7,13 +7,18 @@ shape, or because a test written around Stream's implementation passes. ## Audit basis -- Stream: `fde096668e33494e7b64ddb60ddfadd85ed75617` -- admitted artifact: `sha256:bb3e72377fa3459363155f934cf16b34eebe7f3bfffdce0dcce32a8d9eb158fd` - (`receipts/fde0966.json`, 20/20 suites, built natively on linux/amd64) +- Stream: `eb95c14310ef18c7ab9edbcff991a04a560a7d86` +- admitted artifact: `sha256:4300104135c6fd35bd13a264b744c5f842c41ee9a51e0cf1c57c9a59640551a2` + (`receipts/eb95c14.json`, 20/20 suites, built natively on linux/amd64) - upstream Jetstream: `f29815c391fc2644f8a3dd36b899fb3697dd1ea6` - Atmos: `v0.2.14` -- Zat: `8db0560c2cb9357a16c9ee65205e3d247b0f7f63` (v0.3.18) -- audit date: 2026-07-26 +- Zat: `955e9ca9afb41d66d41e998803f6fcfc8fc4330f` (v0.3.22) +- audit date: 2026-07-28 + +Earlier receipts remain valid for the digests they cover — a receipt proves +those suites passed against *that* image, and that stays true. "Admitted" means +"earned a receipt", not "current", so confirm an artifact matches the commit +you intend before deploying it. The upstream checkout is exactly at the recorded pin. The untracked `simulator` path in that checkout is not treated as upstream source. diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..18b4244 --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,87 @@ +# Testing and receipts + +Stream's suites are the evidence behind [`semantic-parity.md`](semantic-parity.md). +`scripts/admit run` executes all of them against one image and writes a receipt +binding the results to that image's digest; a `skipped` suite blocks admission +exactly like a failure. See [`gotchas.md`](gotchas.md) for the traps that live +in the harness rather than in Stream. + +## Everyday + +``` +zig build test && zig build # unit tests + exe +just simulator # local fake atproto network on :7777 +just run-sim # stream against it +just e2e # python wire checks +just bench # subsystem benchmarks +``` + +## Contract suites + +Each runs real ReleaseSafe processes, offline. + +``` +just archive-contract # archive XRPCs + resident manifest + pinned Go client +just plan-config-contract # non-default planner limits/threshold +just cursor-lookback-contract # v1/v2 replay-window behavior +just compaction-config-contract # tombstone cap/rewrite workers, physical JSS +just retry-config-contract # global/per-host retry admission + backoff +just subscribe-config-contract # subscriber retention/cache/batching/slow controls +just subscribe-read-batch-contract # 5k-row filtered replay, exact cold pulls +just listener-contract # public/debug isolation + disabled/legacy modes +just status-contract # durable host rows + public HTTP view +just shutdown-contract # drain, close 1001, hung listRepos, restart +just logging-contract # formats, filtering, invalid configuration +just environment-contract # JETSTREAM_* map, precedence, unknown rejection +``` + +`environment-contract` proves the hand-maintained Stream map, sorted unknown +rejection, typed validation, CLI precedence, inspection, listener, storage, +logging, and import authentication. It does **not** prove the map contains +every flag in the pinned Go source; the gate requires a generated inventory +comparison for that. + +## Oracles + +``` +just oracle # lifecycle crash + RocksDB fault recovery matrix +just differential-oracle # exact pinned upstream semantics, fully offline +just differential-oracle-multiseed # five upstream predicate-kill worlds +just powerloss-image # one-time Linux NBD/ext4 tool-image bootstrap +just powerloss-oracle # real power-cut recovery, offline after bootstrap +``` + +The power-loss tier is deliberately separate from ordinary unit and process +tests. It cross-builds the production ReleaseSafe Linux binary, runs RocksDB +and JSS on ext4 over a kernel NBD device, kills the block backend *before* the +Stream process, and reconstructs the device only from writes acknowledged by +FLUSH/FUA. The campaign covers eight lifecycle cuts, the two-crash cleanup +guard, six delete-compaction cuts, and four timestamp-import rewrite cuts. +Mutation cases use archives written by the production JSS writer and verify +recovery through the production daemon and public V2 replay. + +It requires privileged Linux containers and an available `/dev/nbd3`. The +pinned Ubuntu tool image needs network access once for `just powerloss-image`; +`just powerloss-oracle` itself refuses image pulls and runs from cached Zig, +Python, simulator, and container inputs. The current recipe targets +`aarch64-linux-gnu`, matching the development host used for its recorded +receipt. + +## Dashboard and metrics + +Fully offline: + +``` +just dashboard-test +just process-metrics-contract +just http-metrics-contract +just dashboard-contract http://127.0.0.1:6008/metrics # against a local candidate +``` + +`process-metrics-contract` also runs the dashboard family-presence check +against the ReleaseSafe binary it launches. **Family presence does not +establish producer semantics, restart stability, or parity** — that distinction +is why the dashboard rows in the gate are still partial. +[`grafana-dashboard.md`](grafana-dashboard.md) records the checksum-pinned +upstream source, the deterministic Stream adaptation, and the fail-closed +metric contract.