jetstream v2 in zig stream.waow.tech
Something went wrong. Try again.
Jetstream V2 configuration audit #
Authority:
- upstream
cmd/jetstream/main.goandinternal/jetstreamd/options.goatf29815c391fc2644f8a3dd36b899fb3697dd1ea6 - Stream
eb95c14310ef18c7ab9edbcff991a04a560a7d86(admitted artifactsha256:43001041...,receipts/eb95c14.json) — this hash resolves, but note experiment 6 deployed29705cb(receipts/29705cb.json), so the basis is behind the artifact that ran. Same gap recorded insemantic-parity.md.
Flag and environment names are derived. Since ac09654, a field on the
options struct in runtime/cli.zig generates both spellings at comptime —
backfill_workers becomes --backfill-workers and
JETSTREAM_BACKFILL_WORKERS via flagName / envName — so a canonical name
cannot drift from its environment variable. Each row therefore audits what
derivation does not supply: the default value, the parse and validation rules,
precedence, and whether the parsed value reaches the runtime mechanism named
in the row.
This document audits public configuration and command wiring. A verified configuration row proves only that the input is parsed/defaulted and reaches the named runtime mechanism. It does not make a blocked runtime mechanism semantically correct. Status vocabulary is defined in semantic-parity.md.
Inventory #
| Surface | Status | What source/tests establish | Limitation or remaining proof |
|---|---|---|---|
| Complete upstream flag inventory | unverified | The environment contract contains a hand-maintained pinned source map and rejects unknown JETSTREAM_* names. |
There is no generated comparison proving every current upstream flag is represented here. Before another admission, generate the inventory from the pinned Go command/options definitions and fail on an unclassified addition/removal. |
| Relay URL | verified | --relay-url reaches the WebSocket consumer and relay-fronted HTTP listRepos/getRepo/repair clients; CLI and environment precedence are tested. |
The reconnect loop this feeds is no longer blocked: the process-killing race was a setsockopt unreachable in the shared websocket.zig fork, fixed there and shipped through zat v0.3.18. |
| PLC URL | verified | --plc-url configures the real DID resolver used by live verification and selected-repo metadata; loopback resolution tests observe requests. |
This row does not prove every verification/repair outcome. |
| Data directory | partial | --data-dir owns JSS, scratch, metadata, bootstrap-live, import, and temporary trees in production-process tests. Active-tail startup recovery now resumes in place and fails closed on corruption. |
Path ownership alone does not establish the behavior of what lives there. The cold-replay and retry-metadata blockers it used to cite are closed; the remaining caveat is that timestamp-import visibility is proved only for the cases in that row. |
--max-backfill-repos |
partial | Canonical spelling is parsed, limits dispatch, avoids committing a whole-network resume cursor, and automatically skips merge discovery. | It is a debug mode, not evidence for whole-network parity -- a capped run dispatches a fresh slice rather than exercising a full crawl. The lifecycle and completion paths it invokes are no longer blocked. |
--backfill-repos |
partial | Canonical spelling and compatibility alias parse; it is mutually exclusive with max-repos, preserves selected order, bypasses listRepos, and enables selected identity resolution. | The exact current artifact still needs the complete selected-repository admission receipt; configuration wiring itself is established. |
| Backfill workers | verified | Explicit positive values set physical getRepo concurrency; zero/omitted default to 100. Held real requests observe 7/100/100. | A 200-worker production run was slower than 100 workers. |
| Backfill batch size | verified | Explicit and zero/default values control only page-aligned listRepos accumulation and cursor-checkpoint cadence; zero/omitted select 100,000. Repository completion now follows independent writer durability boundaries. | This control does not define whole-network size or completion percentage. |
| Backfill async-flush workers | partial | Omitted defaults to four, positive values start that many bootstrap compression workers, and zero selects synchronous compression. Focused tests observe overlap and byte-equivalent output. | Full cancellation/OOM state coverage was not re-established in this audit. |
| Skip merge discovery | verified | Explicit configuration and automatic max/selected enablement prevent the discovery request in focused tests. Normal discovery independently preserves active/inactive unknown repositories and follows arbitrary-length cursors. | This is a debug/partial-selection control; enabling it for a whole-network crawl intentionally omits accounts born during bootstrap. |
| Failed-repo retry interval | verified | Duration parsing, zero disablement, and timer-driven pass execution have real-process tests. | Closed: a terminal pass failure now stops the service rather than sleeping until the next interval, and candidates stream through a bounded queue instead of being materialized in RAM. |
| Failed-repo global/per-host workers | partial | Explicit and zero/default values reach real 16-global/4-per-host gates in held-request tests. | Pending-merge retry uses a different path, though it now runs through the same runner. The steady candidate set is no longer materialized eagerly. |
| Failed-repo maximum delay | partial | Parsed values affect persisted exponential-jitter deadlines in focused RocksDB tests. | Retry supervision is no longer blocked, and host parking is no longer persisted at all -- it moved into the runner's memory to match upstream, which removes the fail-open read this row described. |
| Expensive account verification limiter | verified | Enabled/default and disabled modes cross the real /status?tab=accounts HTTP path. Four requests per source IP succeed, the fifth is rejected, and HEAD does not consume a slot. |
This verifies the control, not the completeness of the status surface. |
| Cursor lookback | verified | Default 36h, explicit duration, and zero/pure-live change v1/v2 cursor admission in production-process tests. | The replay walker no longer skips an unreadable manifest-listed segment; it propagates, because skipping would advance a cursor past durable events nothing revisits. |
| Segment cache max age | verified | Go-duration parsing and ceil-to-seconds response headers are observed on real getSegment/getBlock responses; zero/default no-cache behavior is covered. | None known for this control. |
| planBackfill DID/collection caps | verified | Positive and zero sentinel values affect request validation in real HTTP tests. | Closed: a matchBlocks internal error propagates and the request answers 5xx instead of returning an empty plan. |
| planBackfill entry limit | verified | Positive values paginate by work unit; zero disables the cap; continuation fixtures match normal upstream responses. | Closed: planner errors answer rather than degrading into non-matches; an allocation-failure sweep asserts the planner never returns 200 with a short plan. |
| planBackfill density threshold | verified | Configured thresholds change whole-segment versus block mode at tested boundaries. | The planner error blocker this referenced is closed. |
| Subscribe hot-log retention | verified | The byte value drives physical eviction in the production tail; tests cover eviction while respecting the durable floor. | Does not verify cold recovery after eviction. |
| Subscribe slow-consumer window/rate | verified | Parsed duration and fractional rate reach the per-subscriber scan detector; explicit and default/non-positive behavior have focused tests. | None known for the detector control. |
| Subscribe cold-block cache | partial | The byte bound controls the shared decoded-block/wire memo LRU; concurrency, eviction, and manifest-generation invalidation have tests. | Cache correctness alone does not establish replay completeness, but the enclosing cold walker no longer skips an unreadable segment. |
| Subscribe read batch | partial | Explicit positive values and non-positive default 1024 bound raw scanned entries before filters in hot and normal cold fixtures. | The 5,000-row receipt tests batching, not missing-file continuity. |
| Deprecated cursor block-index cache flag | intentional divergence | Stream accepts signed values as a compatibility no-op because its sealed indexes are always manifest-resident. Parser behavior is tested. | This is not the same runtime mechanism as an implementation that uses the cache. Keep it documented as compatibility, not parity. |
| Compaction interval | verified | Go-duration parsing, zero disablement, periodic execution, and cap-trigger wakeup are exercised. The enclosing pass now propagates malformed/unreadable watermark errors. | This verifies the control and its watermark precondition, not every lifecycle cutover. |
| Compaction tombstone cap | verified | Explicit and zero/default values affect real chunk count and physical survivors; watermark corruption cannot silently reset the fold range. | None known for this control. |
| Compaction rewrite workers | verified | The configured value controls real concurrent rewrite groups in focused physical-file tests, with strict watermark admission. | None known for this control. |
| Timestamp import token | verified | Missing/wrong/correct bearer behavior crosses the production import/status XRPC handlers, including constant-width comparison. | None known for the token control. |
| Timestamp import directory | partial | Configured roots are canonicalized/confined and real jobs survive restart in focused tests. | Corrupt import metadata reads were not re-audited. The cold-replay dependency this row cited is closed. |
| Public bind address | verified | serve --addr binds the protocol listener; public route and WebSocket behavior are exercised by real processes. |
Route availability does not imply archive readiness/completeness. |
| Debug bind address | verified | Empty disables binding; configured debug address exclusively serves health/readiness/metrics and rejects protocol upgrade. Historical combined mode is separately tested. | None known for listener wiring. |
| Shutdown timeout | verified | Signed duration parsing/defaults reach listener and lifecycle shutdown; real tests cover positive, non-positive, hung listRepos, and cooperative exit. | Both former limitations are closed: exact-artifact admission is now enforced by scripts/admit verify and deploy.sh, and the process-fatal reconnect path was a setsockopt unreachable in websocket.zig, fixed and shipped through zat v0.3.18. Consistent with the verified graceful-shutdown row in semantic-parity.md. |
| Client drain timeout | verified | Configured/default/non-positive budgets reach concurrent client close hooks; real blocked-socket tests observe 1001 and shared-deadline interruption. | Limited to client-drain behavior. |
| Log level | verified | Case-insensitive aliases/default and invalid values affect all std.log producers in real ReleaseSafe processes. |
Not byte-identical Go logger internals. |
| Log format | verified | JSON default and text selection produce the documented canonical fields in real processes. | Not byte-identical Go formatting beyond the asserted fields. |
| OpenTelemetry endpoint/protocol | verified | Activation, precedence, HTTP/protobuf path construction, insecure mode, headers, gzip, and timeout reach real collectors. | Limited to tested HTTP/protobuf transport; no claim about an unimplemented protocol. |
| OpenTelemetry batching/retry/sampling | partial | Queue/delay/batch/export timeout, upstream retry/backoff responses, six samplers, propagation, and bounded shutdown have focused tests. | This audit did not independently enumerate every upstream span/attribute producer. |
| OpenTelemetry TLS/mTLS | verified | Custom CA, missing client certificate rejection, complete certificate/key selection, and verified mTLS requests use the production transport/image dependencies. | None known for the tested TLS controls. |
JETSTREAM_* environment mapping |
partial | The hand-maintained map covers all currently listed Stream root/serve/inspect/import controls; CLI precedence and typed parsing run through real processes. | Map completeness must be established by generating the inventory from upstream. |
Unknown JETSTREAM_* rejection |
verified | Unknown names are sorted/deduplicated and rejected before command execution; documented Kubernetes/test namespaces are exempt. | The exemption list must be rechecked when the upstream inventory changes. |
serve command |
verified | Explicit serve and the historical flag-only compatibility invocation both enter the same runtime; persistent root flags are exercised. |
Historical invocation is a Stream extension, not an upstream requirement. |
version command |
verified | It emits the upstream build-information field shape in a focused process test. | Values necessarily identify the Stream artifact. |
inspect-segment |
partial | A sealed upstream fixture matches the pinned renderer for asserted output; active/partial/checksum cases have Stream tests. | The complete current golden set was not rerun during this audit. |
inspect-all |
partial | Representative steady/bootstrap trees, missing roots, active skip, aggregation, sorting, and truncation have fixtures, including a copied upstream report. | It does not replace missing online collections/segments status tabs; complete golden coverage was not independently re-audited. |
| Go pprof routes | intentional divergence | Stream does not expose /debug/pprof/* handlers. Zig/process metrics are used instead. |
Documented as an explicit intentional divergence. |
--rebloom-sweep / JETSTREAM_REBLOOM_SWEEP |
intentional divergence | Stream-only operator control: one-shot sweep that right-sizes legacy per-block bloom regions (upstream f02919c changed seal-time sizing only and ships no legacy-migration tool). Allowlisted in tests/environment_contract.py; behavior in docs/configuration.md. |
Remove once the legacy archive is fully right-sized, or keep as an inert no-op. |
--archive-api-key / JETSTREAM_ARCHIVE_API_KEY |
intentional divergence | Stream-only: bearer gate for the archive endpoints (planSnapshot, listSegments, getSegment, getBlock), mirroring the hosted Bluesky instances' edge behavior (Jetstream v2 launch, 2026-08-13: live tail + dictionary open, archive behind a key). Upstream's OSS server carries no archive auth. Empty (default) leaves the archive open. Allowlisted in tests/environment_contract.py. |
Drop if upstream grows a first-party archive auth story; adopt theirs then. |
--upstream-slow-min-rate / JETSTREAM_UPSTREAM_SLOW_MIN_RATE |
intentional divergence | Stream-only: reconnect the live consumer when the relay delivers below this many frames/s over a window. Upstream's atmos client reconnects only on a closed or errored socket, so a trickling relay holds it indefinitely. Unset/non-positive (default) is off and matches upstream. Allowlisted in tests/environment_contract.py; behavior in docs/configuration.md. |
Drop if atmos grows its own low-rate or stall policy; adopt theirs then. |
Required configuration admission #
Before a deployment can cite this checklist:
- generate and diff the public flag/environment inventory from the exact pinned upstream source;
- require every inventory difference to have one row above;
- run each verified row's production-boundary receipt against the exact artifact digest;
- forbid a configuration row from overriding a blocked behavioral row;
- record intentional divergences explicitly in README/operator docs; and
- invalidate the receipt after any source, dependency, dashboard, container, or default-value change.