diff --git a/openspec/changes/add-integrity-gates/.openspec.yaml b/openspec/changes/add-integrity-gates/.openspec.yaml new file mode 100644 index 0000000..4f63482 --- /dev/null +++ b/openspec/changes/add-integrity-gates/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-07-15 diff --git a/openspec/changes/add-integrity-gates/design.md b/openspec/changes/add-integrity-gates/design.md new file mode 100644 index 0000000..0a45785 --- /dev/null +++ b/openspec/changes/add-integrity-gates/design.md @@ -0,0 +1,112 @@ +# Design: add-integrity-gates + +## Context + +Trawler's data-safety story rests on two invariants that are currently +asserted only narrowly: (1) an edit acknowledged by `persist_update` survives +any crash (tested with simulated drops in `tests/crash_safety.rs`, never with +real process death); (2) all derived state is rebuildable from the Loro doc +and the incremental maintenance never drifts from what a rebuild would +produce (asserted for the search index existing at all, never as an +equivalence property). Both remoras distillation passes independently ranked +executable versions of these two guarantees as the highest-value transfer. +The building blocks exist: the `fixtures` feature (deterministic seeded +graphs), `GraphIndex::rebuild`, a disposable tantivy index, and an expensive +`#[ignore]`d test tier. + +## Goals / Non-Goals + +**Goals:** + +- Make "Crash safety" and "Derived indexes are disposable" executable, with + real process kills and a real equivalence property, reproducible by seed. +- Keep the fast variants cheap enough for every `cargo test --workspace` run. + +**Non-Goals:** + +- Performance regression gates (2×-baseline build-failure policy) — a later, + separate concern; this change is correctness only. +- A full-app (devtools/GPUI) kill test. The durability seam lives entirely in + `trawler-core` (storage + indexes); the app layer above it is already + exercised by the headless UI tests, and putting a windowed app kill-loop in + CI buys little for its cost. Revisit only if an app-layer persistence bug + ever escapes these gates. +- Sync-era convergence fuzzing (the remoras multi-replica harness shape) — + shelved until sync exists. + +## Decisions + +### D1 — Equality is observable-state equality, not struct equality + +The comparator captures what the graph *answers*, not how it stores it: for +every node, its backlink set; for every tag, its member set; every block's +properties; parent/child/order relations for a full traversal; and search +results for a deterministic sample of terms drawn from block content. Two +states are equal iff all captured answers are equal. This survives internal +representation changes (interning, map types, index layout) that struct +equality would false-positive on, and it doubles as a readable diff on +failure. + +### D2 — One seeded op generator, shared by both gates + +A `rand`-seeded generator (seed from env or entropy, always printed on +failure) produces a mixed op stream against a fixture graph: create block, +split, merge, move/indent/outdent/reorder, edit text (including references +and tags typed into content), set/remove properties, delete. Weights favor +text edits (the real-world mix). It lives beside the `fixtures` module under +the same feature gate so UI tests and future gates can reuse it. Determinism +rule: same seed → same op stream → same final state. + +### D3 — Rebuild equivalence as a property over the generator + +Test shape: seed fixture graph → apply N generated ops, maintaining indexes +incrementally exactly as the app does → capture observable state → rebuild +`GraphIndex` and the search index from the doc alone → capture again → +assert equal. Fast tier: a few seeds × moderate N per `cargo test`. Deep +tier: `#[ignore]`d large-N/many-seed run alongside the existing expensive +tests. When the SWR/traced-invalidation query cache lands (see +`add-supertags` design D6), its *traced-revalidation ≡ full-re-evaluation* +gate is this same test shape applied to the query cache — the comparator and +generator are built to be reused for it. + +### D4 — Crash consistency uses real process death, self-spawned + +The test re-invokes its own test binary as a child in "victim mode" +(env-var-gated), following the standard Rust pattern. The victim opens a +graph dir, then loops: apply generated op → `persist_update` → print an +acknowledgment line (op index + content digest) to stdout, flushed *after* +persist returns. The parent kills the child (hard `TerminateProcess`-level +kill, not a polite signal) after a random delay, collects the acknowledged +prefix from the pipe, reopens the graph, and asserts: every acknowledged op's +effect is present; the doc loads cleanly; derived state (rebuilt on open) +matches the comparator run on a reference doc replaying the same acknowledged +prefix. Unacknowledged trailing ops may be present or absent (both are +correct — the guarantee is one-directional). Fast tier: a handful of +kill iterations per `cargo test`; deep tier: `#[ignore]`d loop with many +kills at varied delays. + +### D5 — Gate failures must be reproducible from the log alone + +Every failure prints: the seed, N, the kill delay (crash gate), and the first +divergent comparator entry. CI logs are then sufficient to reproduce locally +with an env var — no flaky-test archaeology. Flakiness policy: these gates +are deterministic by construction (seeded); any nondeterministic failure is +itself a finding (nondeterminism in the code under test), not a retry +candidate. + +## Risks / Trade-offs + +- [Process-kill timing may make the fast tier slow or noisy on CI machines] → + Bounded iterations with fixed short delays for the fast tier; the + wide-delay sweep lives in the `#[ignore]`d tier. +- [Observable-state comparator can silently under-cover (a derived structure + nobody sampled)] → The comparator is a single function with a documented + checklist; adding a derived structure to trawler-core without extending it + is caught in review — and the comparator doubles as documentation of what + "derived state" means. +- [Windows kill semantics differ from POSIX] → The gate targets trawler's + actual platform (Windows first); `std::process::Child::kill` is + TerminateProcess there, which is exactly the no-cleanup death we want. +- [Search-term sampling may miss index divergence] → Sample terms are drawn + deterministically from actual block content (not random dictionary words), + so every sampled term has a known expected hit set. diff --git a/openspec/changes/add-integrity-gates/proposal.md b/openspec/changes/add-integrity-gates/proposal.md new file mode 100644 index 0000000..a784e1a --- /dev/null +++ b/openspec/changes/add-integrity-gates/proposal.md @@ -0,0 +1,52 @@ +# Proposal: add-integrity-gates + +## Why + +The `block-graph` spec promises crash safety and disposable derived indexes, +but nothing enforces those promises end-to-end: the existing crash tests +simulate a crash by dropping the storage handle (no real process death), and +no test proves that incrementally maintained derived state (`GraphIndex`, +tantivy) stays equal to a from-scratch rebuild as ops accumulate. These are +the two seams where data-safety bugs hide silently; making the guarantees +executable (the remoras testing-strategy insight: "these tests running in CI +are what 'no data loss' means in practice") is cheap now that fixtures and a +crash-test file already exist. + +## What Changes + +- A **rebuild-equivalence property test**: after a seeded random sequence of + graph mutations, incrementally maintained derived state must be observably + identical to state rebuilt from scratch from the Loro document. +- A **crash-consistency test with real process death**: a child process + applies and persists edits, is killed at a random moment, and on reopen + every acknowledged edit must be present with derived state consistent. +- A **seeded op generator** (fixtures-adjacent) and an **observable-state + comparator** shared by both gates; failures print their seed for exact + reproduction. +- Fast bounded variants run in `cargo test --workspace`; heavier long-loop + variants are `#[ignore]`d alongside the existing expensive tests. + +## Capabilities + +### New Capabilities + +- `integrity-gates`: the executable data-safety guarantees — rebuild + equivalence and kill-based crash consistency — as permanent, seeded, + CI-run test gates. (Same capability pattern as `ui-test-harness` and + `fixture-graphs`: test infrastructure with spec-level requirements.) + +### Modified Capabilities + +_None — these gates enforce existing `block-graph` requirements ("Crash +safety", "Derived indexes are disposable") without changing them._ + +## Impact + +- `crates/trawler-core`: seeded op generator + observable-state comparator + (test-support/fixtures module), rebuild-equivalence test, child-process + crash helper and kill test (extends `tests/crash_safety.rs`). +- No production code paths change; if a gate fails, the fix lands in the code + under test, not the gate. +- Interacts with `add-auto-compaction`: once that lands, the kill window + naturally covers mid-compaction death too — no coordination needed beyond + running the gates after it merges. diff --git a/openspec/changes/add-integrity-gates/specs/integrity-gates/spec.md b/openspec/changes/add-integrity-gates/specs/integrity-gates/spec.md new file mode 100644 index 0000000..eb62ffa --- /dev/null +++ b/openspec/changes/add-integrity-gates/specs/integrity-gates/spec.md @@ -0,0 +1,45 @@ +# integrity-gates Specification + +## ADDED Requirements + +### Requirement: Rebuild equivalence is continuously verified +The test suite SHALL include a seeded property test asserting that after any +generated sequence of graph mutations, incrementally maintained derived state +(graph index and search index) is observably identical to derived state +rebuilt from scratch from the Loro document. Observable identity SHALL be +defined over query answers (backlinks, tag membership, properties, tree +structure, and search results for content-derived terms), not internal +representation. A fast bounded variant MUST run in the default +`cargo test --workspace`; a deeper variant MAY be `#[ignore]`d. + +#### Scenario: Incremental maintenance equals rebuild +- **WHEN** a seeded random sequence of creates, splits, merges, moves, text + edits, property changes, and deletes is applied with indexes maintained + incrementally +- **THEN** rebuilding all derived state from the Loro document alone yields + observably identical query answers + +#### Scenario: Divergence is reproducible +- **WHEN** the equivalence property fails +- **THEN** the failure output contains the seed and parameters sufficient to + reproduce the identical op sequence and the first divergent answer + +### Requirement: Acknowledged edits survive real process death +The test suite SHALL include a crash-consistency test in which a separate +process applies and persists generated edits and is forcibly killed (no +cleanup, no signal handling) at an arbitrary moment. On reopening the graph +directory, every edit acknowledged before the kill MUST be present, the +document MUST load without error, and derived state MUST match a reference +replay of the acknowledged edits. Edits applied but not yet acknowledged MAY +be present or absent. + +#### Scenario: Kill mid-run loses no acknowledged edit +- **WHEN** the victim process is killed after acknowledging K edits +- **THEN** reopening the graph yields a document containing the effects of + all K acknowledged edits, and rebuilt derived state matches a reference + document produced by replaying those K edits + +#### Scenario: Kill leaves the graph loadable +- **WHEN** the victim process is killed at any point, including mid-write +- **THEN** the next open succeeds without manual intervention and reports no + corruption diff --git a/openspec/changes/add-integrity-gates/tasks.md b/openspec/changes/add-integrity-gates/tasks.md new file mode 100644 index 0000000..512e57d --- /dev/null +++ b/openspec/changes/add-integrity-gates/tasks.md @@ -0,0 +1,22 @@ +## 1. Shared infrastructure (trawler-core, fixtures feature) + +- [ ] 1.1 Seeded op generator: mixed weighted stream (create/split/merge/move/indent/outdent/reorder/text-edit-with-refs-and-tags/set-remove-prop/delete) over a fixture graph; seed from env or entropy, always printed; same seed → same stream +- [ ] 1.2 Observable-state comparator: capture backlinks per node, tag membership, block properties, full tree traversal (parent/children/order), and search results for a deterministic content-derived term sample; equality with a readable first-divergence diff + +## 2. Rebuild-equivalence gate + +- [ ] 2.1 Property test: apply N generated ops with incremental index maintenance → comparator snapshot → rebuild GraphIndex + search index from the doc → comparator snapshot → assert equal; fast tier (few seeds, moderate N) in default `cargo test` +- [ ] 2.2 `#[ignore]`d deep tier: large N, many seeds, alongside the existing expensive tests +- [ ] 2.3 Failure output includes seed, N, and first divergent comparator entry (design D5) + +## 3. Crash-consistency gate + +- [ ] 3.1 Victim mode: env-var-gated child path in the test binary that opens a graph dir, loops apply-op → `persist_update` → flush acknowledgment line (op index + digest) to stdout +- [ ] 3.2 Kill test: parent spawns victim, hard-kills after a random delay, collects acknowledged prefix, reopens, asserts all acknowledged effects present + doc loads + derived state matches a reference replay of the acknowledged prefix; fast tier with a handful of iterations +- [ ] 3.3 `#[ignore]`d deep tier: many kill iterations across varied delays +- [ ] 3.4 Confirm kill semantics on Windows are cleanup-free (`Child::kill` = TerminateProcess) and document in the test header + +## 4. Verification and docs + +- [ ] 4.1 `cargo clippy --workspace --all-targets -- -D warnings` and `cargo test --workspace` pass; fast tiers add acceptable time to the default run +- [ ] 4.2 README development-commands section: document the gates and how to run the deep tiers; note the comparator checklist as the definition of "derived state"