# Testing and receipts

Stream's suites are the evidence behind [`semantic-parity.md`](semantic-parity.md).
`scripts/admit run` executes all of them against one image and writes a receipt
binding the results to that image's digest; a `skipped` suite blocks admission
exactly like a failure. See [`gotchas.md`](gotchas.md) for the traps that live
in the harness rather than in Stream.

## Three receipt names that are not the command name

A receipt records twenty suites. Seventeen are named exactly like the `just`
target that runs them, so a red entry tells you what to re-run. Three do not,
and `scripts/admit` is the authority on the mapping:

| suite in the receipt | how to run it |
|---|---|
| `lifecycle-oracle` | `just oracle` |
| `unit-debug` | `zig build test` |
| `unit-releasesafe` | `zig build test -Doptimize=ReleaseSafe` |

So the everyday `zig build test` below is two of the twenty admission suites,
and the unit tests are graded in both build modes — `ReleaseSafe` differs from
`Debug` in more than speed, since release-mode safety checks and different
undefined-behaviour handling can hide or expose a bug that the other mode does
not. There is no `just lifecycle-oracle` target.

## Everyday

```
zig build test && zig build   # unit tests + exe
just simulator                # local fake atproto network on :7777
just run-sim                  # stream against it
just e2e                      # python wire checks
just bench                    # subsystem benchmarks
```

## Contract suites

Each runs real ReleaseSafe processes, offline.

```
just archive-contract              # archive XRPCs + resident manifest + pinned Go client
just plan-config-contract          # non-default planner limits/threshold
just cursor-lookback-contract      # v1/v2 replay-window behavior
just compaction-config-contract    # tombstone cap/rewrite workers, physical JSS
just retry-config-contract         # global/per-host retry admission + backoff
just subscribe-config-contract     # subscriber retention/cache/batching/slow controls
just subscribe-read-batch-contract # 5k-row filtered replay, exact cold pulls
just listener-contract             # public/debug isolation + disabled/legacy modes
just status-contract               # durable host rows + public HTTP view
just shutdown-contract             # drain, close 1001, hung listRepos, restart
just logging-contract              # formats, filtering, invalid configuration
just environment-contract          # JETSTREAM_* map, precedence, unknown rejection
```

`environment-contract` proves the hand-maintained Stream map, sorted unknown
rejection, typed validation, CLI precedence, inspection, listener, storage,
logging, and import authentication. It does **not** prove the map contains
every flag in the pinned Go source; the gate requires a generated inventory
comparison for that.

## Oracles

```
just oracle                        # lifecycle crash + RocksDB fault recovery matrix
just differential-oracle           # exact pinned upstream semantics, fully offline
just differential-oracle-multiseed # five upstream predicate-kill worlds
just powerloss-image               # one-time Linux NBD/ext4 tool-image bootstrap
just powerloss-oracle              # real power-cut recovery, offline after bootstrap
```

The power-loss tier is deliberately separate from ordinary unit and process
tests. It cross-builds the production ReleaseSafe Linux binary, runs RocksDB
and JSS on ext4 over a kernel NBD device, kills the block backend *before* the
Stream process, and reconstructs the device only from writes acknowledged by
FLUSH/FUA. The campaign covers eight lifecycle cuts, the two-crash cleanup
guard, six delete-compaction cuts, and four timestamp-import rewrite cuts.
Mutation cases use archives written by the production JSS writer and verify
recovery through the production daemon and public V2 replay.

It requires privileged Linux containers and an available `/dev/nbd3`. The
pinned Ubuntu tool image needs network access once for `just powerloss-image`;
`just powerloss-oracle` itself refuses image pulls and runs from cached Zig,
Python, simulator, and container inputs. The current recipe targets
`aarch64-linux-gnu`, matching the development host used for its recorded
receipt.

## When offline suites cannot reach a failure

Some failure modes are structurally unreachable in the fixture, and a green
suite says nothing about them. The pinned simulator's repositories are ~4 KiB;
real ones average ~10 MB. The batch submit/consume deadlock survived 20 passing
suites for this reason and could not be reproduced in the simulator.

A real-network reproduction costs about €0.03. Provision a small cloud box,
stream the image to it, and run against `bsky.network` — that bug reproduced in
90 seconds. This is also the only way to measure realistic throughput: backfill
is bandwidth-bound at ~10 MB/repo, which no local fixture reproduces. Reach for
this whenever a hypothesis involves
memory pressure, worker contention, or repository sizes.

`just backfill-batch-contract` pins the weaker property that a dispatch batch
far larger than the job queue converges. Its docstring notes that it passes on
the broken code; it does not cover the deadlock itself.

## Dashboard and metrics

Fully offline:

```
just dashboard-test
just process-metrics-contract
just http-metrics-contract
just dashboard-contract http://127.0.0.1:6008/metrics   # against a local candidate
```

`process-metrics-contract` also runs the dashboard family-presence check
against the ReleaseSafe binary it launches. **Family presence does not
establish producer semantics, restart stability, or parity** — that distinction
is why the dashboard rows in the gate are still partial.
[`grafana-dashboard.md`](grafana-dashboard.md) records the checksum-pinned
upstream source, the deterministic Stream adaptation, and the fail-closed
metric contract.