# Testing and receipts

Stream's suites are the evidence behind [`semantic-parity.md`](semantic-parity.md).
`scripts/admit run` executes all of them against one image and writes a receipt
binding the results to that image's digest; a `skipped` suite blocks admission
exactly like a failure. See [`gotchas.md`](gotchas.md) for the traps that live
in the harness rather than in Stream.

## Everyday

```
zig build test && zig build   # unit tests + exe
just simulator                # local fake atproto network on :7777
just run-sim                  # stream against it
just e2e                      # python wire checks
just bench                    # subsystem benchmarks
```

## Contract suites

Each runs real ReleaseSafe processes, offline.

```
just archive-contract              # archive XRPCs + resident manifest + pinned Go client
just plan-config-contract          # non-default planner limits/threshold
just cursor-lookback-contract      # v1/v2 replay-window behavior
just compaction-config-contract    # tombstone cap/rewrite workers, physical JSS
just retry-config-contract         # global/per-host retry admission + backoff
just subscribe-config-contract     # subscriber retention/cache/batching/slow controls
just subscribe-read-batch-contract # 5k-row filtered replay, exact cold pulls
just listener-contract             # public/debug isolation + disabled/legacy modes
just status-contract               # durable host rows + public HTTP view
just shutdown-contract             # drain, close 1001, hung listRepos, restart
just logging-contract              # formats, filtering, invalid configuration
just environment-contract          # JETSTREAM_* map, precedence, unknown rejection
```

`environment-contract` proves the hand-maintained Stream map, sorted unknown
rejection, typed validation, CLI precedence, inspection, listener, storage,
logging, and import authentication. It does **not** prove the map contains
every flag in the pinned Go source; the gate requires a generated inventory
comparison for that.

## Oracles

```
just oracle                        # lifecycle crash + RocksDB fault recovery matrix
just differential-oracle           # exact pinned upstream semantics, fully offline
just differential-oracle-multiseed # five upstream predicate-kill worlds
just powerloss-image               # one-time Linux NBD/ext4 tool-image bootstrap
just powerloss-oracle              # real power-cut recovery, offline after bootstrap
```

The power-loss tier is deliberately separate from ordinary unit and process
tests. It cross-builds the production ReleaseSafe Linux binary, runs RocksDB
and JSS on ext4 over a kernel NBD device, kills the block backend *before* the
Stream process, and reconstructs the device only from writes acknowledged by
FLUSH/FUA. The campaign covers eight lifecycle cuts, the two-crash cleanup
guard, six delete-compaction cuts, and four timestamp-import rewrite cuts.
Mutation cases use archives written by the production JSS writer and verify
recovery through the production daemon and public V2 replay.

It requires privileged Linux containers and an available `/dev/nbd3`. The
pinned Ubuntu tool image needs network access once for `just powerloss-image`;
`just powerloss-oracle` itself refuses image pulls and runs from cached Zig,
Python, simulator, and container inputs. The current recipe targets
`aarch64-linux-gnu`, matching the development host used for its recorded
receipt.

## When offline suites cannot reach a failure

Some failure modes are structurally unreachable in the fixture, and a green
suite says nothing about them. The clearest example: the pinned simulator's
repositories are ~4 KiB, so concurrent workers can never exhaust an in-flight
budget that a single repository fits in. The batch submit/consume deadlock that
ended experiment 4 survived 20 passing suites for exactly that reason, and four
attempts to force it out of the simulator failed.

A real-network reproduction costs about €0.03. Provision a small cloud box,
stream the image to it, and run against `bsky.network` with the pressure turned
up — for that bug, 50 workers and `--backfill-max-inflight-bytes=268435456`
showed it in 90 seconds, and the before/after was unambiguous. Reach for this
whenever a hypothesis involves memory pressure, worker contention, or repository
sizes the simulator cannot produce.

`just backfill-batch-contract` pins the weaker property that a dispatch batch
far larger than the job queue converges. Its docstring is explicit that it
passes on the broken code; read it before treating it as coverage.

## Dashboard and metrics

Fully offline:

```
just dashboard-test
just process-metrics-contract
just http-metrics-contract
just dashboard-contract http://127.0.0.1:6008/metrics   # against a local candidate
```

`process-metrics-contract` also runs the dashboard family-presence check
against the ReleaseSafe binary it launches. **Family presence does not
establish producer semantics, restart stability, or parity** — that distinction
is why the dashboard rows in the gate are still partial.
[`grafana-dashboard.md`](grafana-dashboard.md) records the checksum-pinned
upstream source, the deterministic Stream adaptation, and the fail-closed
metric contract.