From eafd356522835d8830363b8d4bda0020e352cdb7 Mon Sep 17 00:00:00 2001 From: iacore Date: Thu, 6 Aug 2026 11:05:58 +0800 Subject: [PATCH] docs: log S024 - the scenario suite tests itself (gate rule + checks) Slice log for the self-testing scenario harness; roadmap slice line; AGENTS gate rule now requires zig build scenario-self before the suite is trusted to gate zds upgrades; checks list gains the self + isolation steps. --- AGENTS.md | 5 +++- log/s024.dj | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++++ roadmap.dj | 8 ++++++ 3 files changed, 83 insertions(+), 1 deletion(-) create mode 100644 log/s024.dj diff --git a/AGENTS.md b/AGENTS.md index 952b0b3..9905727 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -41,7 +41,10 @@ without making the change explicit. file is re-read; a log line citing an artifact (note, file, link) is written only after the artifact is verified to exist. 2. **No zds upgrade reaches the class before the scenario suite passes - on it** (`app0/tests/scenario_*.zig`). + on it** (`app0/tests/scenario_*.zig`) **and the suite's own gate is + green** (`zig build scenario-self` — the harness tested against + scripted fakes, no zds build needed; a broken suite must not be + trusted to gate anything). 3. **Never edit zat or the zds upstream checkout** unless explicitly asked — contribute via the forks in this workspace. 4. **Build target:** this machine's glibc `crt1.o` breaks zig's linker — diff --git a/log/s024.dj b/log/s024.dj new file mode 100644 index 0000000..54ae1f9 --- /dev/null +++ b/log/s024.dj @@ -0,0 +1,71 @@ +# S024 - the gate's gate: the scenario suite tests itself + +Date: 2026-08-06 +Slice: the scenario harness is load-bearing (`docs/AGENTS.md` rule 2 — +no zds upgrade reaches the class before `scenario_*.zig` passes), yet the +harness itself was untested and unverifiable. When the suite failed on a +zds upgrade, harness-bug was indistinguishable from product-bug. This +slice makes the gate self-testing and isolated from the rig it gates. + +## What changed + +- **L0 — shared harness** (`app0/tests/harness.zig`): the assertion + helpers, rig lifecycle, and subprocess drivers duplicated across + `scenario_space.zig` / `scenario_serve.zig` (~150 lines) are now one + module both scenarios import. Two isolation fixes landed with it: + ports resolve from `LINJI_ZDS_PORT` / `LINJI_API_PORT` (defaults keep + the historical 2591/2592/8791; overrides let concurrent runs avoid + collision) and every linji subprocess call has a bounded wait + (`LINJI_CMD_TIMEOUT_MS`, default 30s) — a hung CLI is killed, not + wedged. `LINJI_SERVE_BIN` lets the serve scenario swap its server too. +- **L1 — the gate's unit tests**: `zig build test` now also runs the + harness module's own tests — every assertion helper in both polarities + (accept + reject), the JSON/URI parsers, and the curl `api()` driver + against an in-process httpz stub. No rig, no network, milliseconds. +- **L2 — `zig build scenario-self`**: the *real, unmodified* scenario + executables run against scripted fakes via the existing env seams — + `tests/fake_zds.zig` (creates the sqlite db + serves `/xrpc/_health`), + `tests/fake_linji.zig` (replays a fixture, one row per CLI call, cursor + in `$LINJI_HOME`), `tests/fake_serve.zig` (replays HTTP rows incl. + SSE hold/trigger and push-subscription side effects). Fixtures live in + `tests/fixtures/*.tsv` — hand-verified oracles, never products of more + tests (the regress terminates there). Three polarities per scenario: + known-good fixture must PASS; a mutated fixture must FAIL at the exact + checkpoint it breaks (exit 1); a `hang` row must be reaped by the + harness timeout. The step depends on nothing but app0 — it runs even + when the rig is what's broken, which is exactly when it's needed. +- **L3 — `zig build scenario-isolation`**: two concurrent scenario-space + runs on distinct ports must both pass, use distinct work dirs, and tear + those dirs down on exit. +- Scenario output moved to stdout (fd 1 via libc): `std.debug.print` + (stderr) was captured by the build runner and echoed as a bogus + "failed command" on green runs; in test binaries the output is + suppressed entirely because fd 1 is the `--listen=-` protocol channel. + +## Verification + +- `zig build test` — core unit suite + 8/8 harness self-tests, clean + output, green across repeated runs. +- `zig build scenario-self` — 16/16 steps: space good PASS (41 + checkpoints), space bad FAIL at "second thread header" (exit 1), space + hang FAIL "timed out after 2000ms" (exit 1), serve good PASS (all + sections incl. push prune + SSE), serve bad FAIL at "alice delete bob's + post denied" (exit 1). +- `zig build scenario-isolation` — PASS: concurrent runs on 2611/2612, + distinct work dirs, teardown verified. +- L0 regression against the real rig (fresh zds from `../zds`): + `zig build scenario-space` PASS (38s), `zig build scenario-serve` + PASS (48s). The refactor changed no scenario behavior. + +## Notes / follow-ups + +- Mutation testing of the real scenarios (mutate an expected string, + rebuild, must fail) is the optional next rung; L2's bad-polarity + fixtures already prove fail-detection on the same code paths. +- The serve scenario's fake-PDS surface is fully scripted (replay + + side-effect rows); a stateful fake that models membership is not + needed — the fixtures are the oracle. +- The L1 `api()` stub test binds fixed port 2599; fine in practice, an + env override would make it immune to exotic local collisions. +- The harness runs' ports (2651-2655, 2611/2612) avoid the default set + so self-runs never collide with real scenario runs or each other. diff --git a/roadmap.dj b/roadmap.dj index c8889a1..b267e8f 100644 --- a/roadmap.dj +++ b/roadmap.dj @@ -124,6 +124,14 @@ Note: refresh scenarios use expireTokens, not setClock — rationale in failures get a distinct 502); oauth.refresh serialized against the concurrent-401 replay race that split token families and locked the app out of zds. [log](log/s023.dj) +- **S024** ✓ — the gate's gate: shared scenario harness + (`app0/tests/harness.zig`) with L1 self-tests; scripted-fake rig + (`fake-zds`/`fake-linji`/`fake-serve` + `tests/fixtures/*.tsv`) + proving `zig build scenario-self` in both polarities (good fixtures + PASS, broken fixtures FAIL at the checkpoint they break, a stuck CLI + is reaped by the harness timeout); port + timeout isolation; + `zig build scenario-isolation` for concurrent runs. The gate is now + testable without the rig it gates. [log](log/s024.dj) ## Open questions -- 2.51.2