diff --git a/openspec/changes/add-format-versioning/.openspec.yaml b/openspec/changes/add-format-versioning/.openspec.yaml new file mode 100644 index 0000000..4f63482 --- /dev/null +++ b/openspec/changes/add-format-versioning/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-07-15 diff --git a/openspec/changes/add-format-versioning/design.md b/openspec/changes/add-format-versioning/design.md new file mode 100644 index 0000000..f0bc07d --- /dev/null +++ b/openspec/changes/add-format-versioning/design.md @@ -0,0 +1,109 @@ +# Design: add-format-versioning + +## Context + +A graph directory is `snapshot.loro` + `updates.log` (+ disposable +`search-index/`), soon also `snapshot.loro.prev` (add-auto-compaction D7). +Nothing in it says which format it is. Loro has its own internal encoding +version, but that is not trawler's semantic version: trawler can change its +*use* of the format (container names, metadata conventions, file layout, +length-prefix framing) without Loro changing at all, and vice versa. The +remoras migration spec (16) supplies the skeleton worth adopting: monotonic +integer version, sequential compiled migrations, CI-enforced contiguity, +hard refusal of newer formats — minus their mirror-dependent safety story, +which trawler replaces with snapshot retention + a golden-graph tripwire. + +## Goals / Non-Goals + +**Goals:** + +- Every graph directory carries a semantic `format_version`, checked before + any load work. +- Newer-format graphs are refused loudly and harmlessly. +- The migration mechanism exists, tested, before the first migration does. +- Loro upgrades get a tripwire (golden graph) and a written policy. + +**Non-Goals:** + +- Any actual migration (none is needed; the registry ships empty). +- Supertag schema versioning — per-schema, handled in `add-supertags` + (open question 2); this change versions the *container format* only. +- Version negotiation for sync peers — sync-era concern; the stamp is a + prerequisite for it, not an implementation. +- Versioning the devtools wire protocol (already versioned separately). + +## Decisions + +### D1 — The stamp lives in a sidecar file, outside the Loro doc + +`meta.json` in the graph directory: `{"format_version": 1}` (extensible +object, not a bare integer file). Rationale: the version must be readable +*without* decoding the document — a future format change may alter how the +doc itself is stored, so anything inside the doc (or inside Loro's encoding) +cannot gate the decode step. Written via the existing temp-file + fsync + +atomic-rename pattern. + +### D2 — Missing stamp = version 1, stamped on next open + +Every graph that exists today has no `meta.json`; they are all, by +definition, format 1. `open()` treats absence as 1 and writes the stamp — +the only case where open writes metadata, and it is idempotent. This avoids +a flag-day: no graph ever becomes unreadable by gaining the versioning +feature. + +### D3 — Newer is refused before anything is touched + +`CURRENT_FORMAT_VERSION: u32` in `trawler-core`. If the stamp exceeds it, +`open()` returns an error naming both versions ("this graph was created by a +newer trawler (format 3); this build reads up to format 2") without reading +or writing any other file. Refusal-before-touch is the entire safety +property: an old binary can never partially interpret a new layout. + +### D4 — Migrations are sequential compiled steps over the directory + +A static registry: `MIGRATIONS: &[fn(&Path) -> io::Result<()>]` where entry +`i` migrates format `i+1` → `i+2`. `open()` on an older graph runs the steps +in order, re-stamping after each (so an interruption resumes cleanly at the +failed step), then proceeds with the normal load. Migrations operate on +files in the directory, never on a live doc. A unit test asserts +`MIGRATIONS.len() == CURRENT_FORMAT_VERSION - 1` — the contiguity gate, +trivially green today at `0 == 0`, and the compile-visible reminder that a +version bump requires a migration (or an explicit reasoned exemption). +Before the first migration step runs, the pre-migration `snapshot.loro` is +copied aside (`snapshot.loro.v{N}.bak`) so a botched migration is always +recoverable — the no-mirror answer to remoras' mirror-backed safety net. + +### D5 — Golden-graph compatibility test + +A committed test fixture: a small graph directory (built once from the +deterministic fixtures, a few KB — snapshot, a couple of update blobs, and a +`meta.json`) under `crates/trawler-core/tests/golden/`. Every test run opens +it and asserts a comparator snapshot of its content (block texts, structure, +references). This is the tripwire that catches *accidental* format breakage +— a Loro version bump that changes decoding, a framing change, a container +rename — as a test failure instead of a user's unreadable vault. The golden +directory is regenerated only deliberately, in the same commit as a format +version bump and its migration. + +### D6 — Loro upgrade policy (documented, not automated) + +Loro stays pinned to an exact version. An upgrade PR must show: the golden +test passing unmodified, and a round-trip check (open golden → export +snapshot with new Loro → reopen → comparator-equal). If an upgrade cannot +read old snapshots, that is by definition a format bump: write the migration +(re-export via the old pinned version or Loro's own upgrade path) and +regenerate the golden. The policy lives in the README development section — +one paragraph, but written *before* the first tempting `cargo update`. + +## Risks / Trade-offs + +- [Committed binary fixture in the repo] → A few KB, regenerated only on + deliberate format bumps; the alternative (generating it fresh each run) + cannot catch decoding regressions, which is the entire point. +- [Stamping legacy graphs on open is a write during open] → Idempotent, + atomic, and tiny; failure to stamp degrades to the current behavior + (unversioned) rather than blocking the open. +- [`meta.json` could drift into a grab-bag] → Convention noted in the module + docs: format-gating fields only; anything else belongs in the doc itself. +- [Interrupted migration] → Re-stamp-per-step (D4) makes resumption safe; + the `.v{N}.bak` snapshot makes even a wrong migration recoverable. diff --git a/openspec/changes/add-format-versioning/proposal.md b/openspec/changes/add-format-versioning/proposal.md new file mode 100644 index 0000000..c3ac0e9 --- /dev/null +++ b/openspec/changes/add-format-versioning/proposal.md @@ -0,0 +1,53 @@ +# Proposal: add-format-versioning + +## Why + +The graph directory has no semantic format version: nothing records which +trawler wrote it, an older build opening a newer graph would fail confusingly +(or worse, subtly), and there is no skeleton to hang a migration on when the +format eventually changes. Because trawler deliberately has no export mirror, +Loro format stability plus the snapshot files are the *only* escape hatch — +which makes version discipline load-bearing. This is the classic +cheap-now/impossible-retroactively change: the stamp must exist in every +graph *before* the first real format change needs to detect its absence. + +## What Changes + +- A `format_version` stamp (small `meta.json` sidecar) written at graph + creation and checked at open, readable without decoding the Loro document. +- Graphs with a **newer** version than the running build are refused with a + clear error, files untouched. +- Graphs with an **older** version are migrated by sequential compiled + migration steps, then re-stamped. A missing stamp means version 1 (every + existing graph) and is stamped on next open. +- A migration registry skeleton with a test enforcing a contiguous chain + (v1→v2→…→current, no gaps) — trivially satisfied today with zero + migrations, but the discipline exists before it's needed. +- A **golden-graph compatibility test**: a small committed graph directory + that every test run must open and read identically — the tripwire for + accidental format breakage from Loro upgrades or storage changes. +- A documented Loro upgrade policy: pinned exact version; bumps require the + golden test plus a snapshot round-trip before merging. + +## Capabilities + +### New Capabilities + +_None._ + +### Modified Capabilities + +- `block-graph`: adds a requirement that the graph directory format is + versioned — stamped at creation, checked before load, newer-format refusal, + sequential migration of older formats, and cross-release open + compatibility. + +## Impact + +- `crates/trawler-core/src/storage.rs` (or sibling module): `meta.json` + read/write, `CURRENT_FORMAT_VERSION`, open-time check, migration registry. +- `crates/trawler-core/tests/`: golden-graph fixture directory (few KB of + committed binary + log) and its compatibility test; chain-contiguity test. +- No behavior change for existing graphs beyond acquiring a stamp on next + open; no interaction with `add-auto-compaction` beyond both touching the + graph directory (compaction preserves `meta.json` untouched). diff --git a/openspec/changes/add-format-versioning/specs/block-graph/spec.md b/openspec/changes/add-format-versioning/specs/block-graph/spec.md new file mode 100644 index 0000000..da58735 --- /dev/null +++ b/openspec/changes/add-format-versioning/specs/block-graph/spec.md @@ -0,0 +1,53 @@ +# block-graph Delta: add-format-versioning + +## ADDED Requirements + +### Requirement: Graph directory format is versioned +The graph directory SHALL carry a semantic format version, written at graph +creation and readable without decoding the Loro document. Opening a graph +whose format version exceeds the running build's supported version MUST fail +with an error identifying both versions, without reading or modifying any +other file in the directory. A graph directory without a version stamp SHALL +be treated as format version 1 and stamped on next open. + +#### Scenario: Newer format is refused untouched +- **WHEN** a build supporting format 2 opens a graph stamped format 3 +- **THEN** the open fails with an error naming both versions and every file + in the graph directory is byte-identical to its pre-open state + +#### Scenario: Legacy graph acquires a stamp +- **WHEN** a graph directory created before versioning existed is opened +- **THEN** it is treated as format 1, opens normally, and carries a format + stamp afterwards + +### Requirement: Older formats migrate through a contiguous chain +Opening a graph with an older format version SHALL migrate it through +sequential compiled migration steps (each step raising the version by exactly +one) before normal loading, re-stamping after each step so an interrupted +migration resumes at the failed step. The registry of migration steps MUST be +verified contiguous from version 1 to the current version by a test. Before +the first migration step modifies the directory, the pre-migration snapshot +MUST be preserved as a backup. + +#### Scenario: Migration chain has no gaps +- **WHEN** the current format version is N +- **THEN** a test asserts exactly N−1 registered migration steps, so a + version bump without its migration fails the build + +#### Scenario: Interrupted migration resumes safely +- **WHEN** the process dies between migration steps +- **THEN** the next open resumes migration from the recorded intermediate + version and completes normally, and the pre-migration snapshot backup + still exists + +### Requirement: Cross-release open compatibility is continuously verified +The test suite SHALL include a committed golden graph directory that every +test run opens and verifies content-identical (block text, structure, +references) against a recorded expectation. The golden directory SHALL be +regenerated only together with a deliberate format version bump. + +#### Scenario: Dependency upgrade cannot silently break decoding +- **WHEN** a change (including a Loro version bump) alters how existing graph + files are decoded +- **THEN** the golden-graph test fails, forcing either a fix or an explicit + format version bump with a migration diff --git a/openspec/changes/add-format-versioning/tasks.md b/openspec/changes/add-format-versioning/tasks.md new file mode 100644 index 0000000..1e13b94 --- /dev/null +++ b/openspec/changes/add-format-versioning/tasks.md @@ -0,0 +1,22 @@ +## 1. Version stamp (trawler-core) + +- [ ] 1.1 `CURRENT_FORMAT_VERSION: u32 = 1` and `meta.json` read/write (`{"format_version": N}`, temp-file + fsync + atomic rename), written by `GraphStorage::create` +- [ ] 1.2 `open()` reads the stamp before any other load work: absent → treat as 1 and stamp (idempotent, non-fatal on failure); newer than current → refuse with an error naming both versions, touching nothing +- [ ] 1.3 Unit tests: create stamps; legacy dir (no meta.json) opens and gains stamp; artificially newer stamp refused with directory byte-identical afterward + +## 2. Migration skeleton (trawler-core) + +- [ ] 2.1 `MIGRATIONS: &[fn(&Path) -> io::Result<()>]` registry (empty today); `open()` runs steps sequentially for older stamps, re-stamping after each; pre-migration `snapshot.loro` copied to `snapshot.loro.v{N}.bak` before the first step +- [ ] 2.2 Contiguity test: `MIGRATIONS.len() == CURRENT_FORMAT_VERSION - 1` +- [ ] 2.3 Test the machinery with a synthetic migration in test code only (register a step under `#[cfg(test)]`, stamp a dir older, open, assert step ran, version re-stamped, backup exists; simulate interruption between steps and assert resumption) + +## 3. Golden-graph compatibility (trawler-core) + +- [ ] 3.1 Generate the golden fixture once from the deterministic fixture graph: small directory (snapshot.loro, a few update blobs, meta.json) committed under `tests/golden/` +- [ ] 3.2 Compatibility test: open the golden dir, assert a recorded content snapshot (block texts, structure, references, tags); document the regenerate-only-with-format-bump rule in the test header + +## 4. Policy and docs + +- [ ] 4.1 README development section: Loro upgrade policy paragraph (exact pin; upgrade PR must pass the golden test unmodified plus an open→export→reopen round-trip; inability to read old snapshots = format bump + migration + golden regeneration) +- [ ] 4.2 README graph-directory-format section: document `meta.json` and newer-format refusal behavior +- [ ] 4.3 `cargo clippy --workspace --all-targets -- -D warnings` and `cargo test --workspace` pass