diff --git a/README.md b/README.md index 7922340..7483816 100644 --- a/README.md +++ b/README.md @@ -36,8 +36,9 @@ experiment. boundaries; several correctness-metadata reads still fail open and are called out in the deployment gate - tombstone-aware compaction and continuing repair of repositories that were - unavailable or changed during bootstrap; compaction watermark corruption - and retry-supervisor failure handling are not yet equivalent + unavailable or changed during bootstrap; the versioned compaction watermark + fails closed on corruption/read failure, while retry-supervisor failure + handling is not yet equivalent - crash-resumable timestamp imports with a durable RocksDB rule map, topology-preserving sealed-segment repair, and bearer-gated operator XRPC - browser and terminal operations views at `/status`, including durable PDS @@ -246,7 +247,7 @@ rewrites; zero selects `min(CPU count, 8)`. Cap triggers wake the compactor immediately, subject to upstream's 30-second between-pass floor. `just compaction-config-contract` proves the controls reach the physical compactor rather than stopping at argument parsing, including a 250 ms pass. It -does not waive the fail-open compaction-watermark read. +does not waive the remaining lifecycle and serving admission gates. Failed-repository healing exposes upstream's complete control set: diff --git a/docs/bootstrap-semantic-parity.md b/docs/bootstrap-semantic-parity.md index c1ae0fc..cef0273 100644 --- a/docs/bootstrap-semantic-parity.md +++ b/docs/bootstrap-semantic-parity.md @@ -74,7 +74,7 @@ and retry behavior; those still prevent another deployment. | Merge row filtering | **verified** | Rows are dropped only when the repository is complete and the source revision is at/below its backfill watermark; missing lookups are cached. Kept/dropped fixtures exercise this. | The surrounding cursor recovery remains blocked. | | Latest revision refresh | **partial** | Kept rev-bearing rows update latest revision in the same batch as the source cursor while preserving the backfill revision. | Missing repo rows are skipped; this matches the intended defensive behavior, but corrupt cursor state can replay updates and rows. | | Pending-repository pass | **blocked** | Pending repositories are repaired after captured live rows are merged, preserving intended row order. | Stream materializes pending DIDs and processes them through a separate path rather than upstream's bounded retry runner with the same global/per-host gates and retry semantics. | -| Merge-tail compaction and manifest reconcile | **partial** | Destination sealing precedes delete/update compaction; manifest reconciliation happens before serving is enabled. Physical rewrite tests cover named write failures. | Compaction watermark read corruption fails open. | +| Merge-tail compaction and manifest reconcile | **partial** | Destination sealing precedes delete/update compaction; manifest reconciliation happens before serving is enabled. Physical rewrite tests cover named write failures, and the versioned compaction watermark now propagates Store errors and rejects malformed/unknown encodings without reinitializing the key. | The fail-open merge source-existence guard and other cutover rows remain independent blockers. | | Post-bootstrap discovery | **blocked** | It resumes from the last non-empty bootstrap cursor and creates failed rows for unknown active repositories on the normal path. | Unknown inactive repositories are skipped instead of recorded. A next cursor longer than 512 bytes silently exits the loop and cleanup proceeds, omitting the remaining network. | | Cleanup durability | **verified** | On the successful path, the backfill tree is removed, the data directory is synced, and merge/discovery cursors are deleted in a synced RocksDB batch afterward. Restart-after-cleanup repeats the directory sync. | This row assumes the source was correctly classified as absent; that guard is separately blocked. | diff --git a/docs/configuration-parity.md b/docs/configuration-parity.md index 0e3686c..3e2ee0a 100644 --- a/docs/configuration-parity.md +++ b/docs/configuration-parity.md @@ -40,9 +40,9 @@ semantically correct. Status vocabulary is defined in | Subscribe cold-block cache | **partial** | The byte bound controls the shared decoded-block/wire memo LRU; concurrency, eviction, and manifest-generation invalidation have tests. | The enclosing cold walker skips an unreadable segment, so cache correctness cannot establish replay completeness. | | Subscribe read batch | **partial** | Explicit positive values and non-positive default 1024 bound raw scanned entries before filters in hot and normal cold fixtures. | The 5,000-row receipt tests batching, not missing-file continuity. | | Deprecated cursor block-index cache flag | **intentional divergence** | Stream accepts signed values as a compatibility no-op because its sealed indexes are always manifest-resident. Parser behavior is tested. | This is not the same runtime mechanism as an implementation that uses the cache. Keep it documented as compatibility, not parity. | -| Compaction interval | **partial** | Go-duration parsing, zero disablement, periodic execution, and cap-trigger wakeup are exercised. | A malformed/unreadable durable watermark is treated as missing and reinitialized. | -| Compaction tombstone cap | **partial** | Explicit and zero/default values affect real chunk count and physical survivors. | Correctness still depends on the blocked watermark read path. | -| Compaction rewrite workers | **partial** | The configured value controls real concurrent rewrite groups in focused physical-file tests. | Full pass admission remains blocked by watermark corruption behavior. | +| Compaction interval | **verified** | Go-duration parsing, zero disablement, periodic execution, and cap-trigger wakeup are exercised. The enclosing pass now propagates malformed/unreadable watermark errors. | This verifies the control and its watermark precondition, not every lifecycle cutover. | +| Compaction tombstone cap | **verified** | Explicit and zero/default values affect real chunk count and physical survivors; watermark corruption cannot silently reset the fold range. | None known for this control. | +| Compaction rewrite workers | **verified** | The configured value controls real concurrent rewrite groups in focused physical-file tests, with strict watermark admission. | None known for this control. | | Timestamp import token | **verified** | Missing/wrong/correct bearer behavior crosses the production import/status XRPC handlers, including constant-width comparison. | None known for the token control. | | Timestamp import directory | **partial** | Configured roots are canonicalized/confined and real jobs survive restart in focused tests. | Corrupt import metadata reads were not re-audited; public results depend on blocked cold replay. | | Public bind address | **verified** | `serve --addr` binds the protocol listener; public route and WebSocket behavior are exercised by real processes. | Route availability does not imply archive readiness/completeness. | diff --git a/docs/semantic-parity.md b/docs/semantic-parity.md index dbc8555..ba0672b 100644 --- a/docs/semantic-parity.md +++ b/docs/semantic-parity.md @@ -60,7 +60,7 @@ is in [bootstrap-semantic-parity.md](bootstrap-semantic-parity.md). | Live Sync 1.1 verification | **partial** | DID/signature checks, MST inversion, op-CID checks, replay/future-rev gates, durable chain/hosting updates, and several repair-trigger cases use real RocksDB, CARs, and sockets. Relay cursor corruption and read failures now fail closed. | The surface cannot be called closed while an accepted live row can persist and then kill the process during eager wire encoding. | | Live scheduling and cursor durability | **partial** | Per-DID ordering, bounded pending work, worker admission, completion-order emission, archive-before-cursor write ordering, strict versioned cursor decoding, and cursor-read failure propagation are represented in production code and focused tests. | The Zat reconnect sleep can propagate `HttpConnectionClosing`; production observed two process exits. | | Whole-repository repair/resync | **partial** | Fetched repositories are authenticated before replacement emission; valid repair emits sync then bounded replacement batches; repair-tail encoding errors are isolated per row. Capacity tests exercise 32 active plus 64 queued repairs. | The isolation applies only to repair-tail publication. Ordinary live publication still eagerly encodes and can terminate the process. Retry-subsystem terminal failures are not tied to health. | -| Delete/update compaction | **partial** | Rewrite selection, survivor correctness, manifest-before-watermark ordering, fsync/rename cutpoints, bounded workers, and several reason counters have physical-file tests. | Compaction watermark read/corruption is treated as missing and reinitialized. A read-error fault contract against the production metadata store is required. | +| Delete/update compaction | **verified** | Rewrite selection, survivor correctness, manifest-before-watermark ordering, fsync/rename cutpoints, bounded workers, and reason counters have physical-file tests. The versioned watermark now distinguishes absence from Store failure, rejects wrong width/version, and refuses to initialize over corrupt bytes; focused tests inject the production Store read fault. | This row does not close merge orchestration or cold-serving gaps. | | Timestamp import | **partial** | CSV parsing, rule precedence, patch topology preservation, job persistence, restart, HTTP authentication, and write/fsync/rename power-loss cuts have concrete tests. | Public visibility after import still depends on the blocked cold-replay path. No audit has yet proved behavior when timestamp metadata reads are corrupt rather than absent. | | Status: hosts and accounts | **partial** | Durable host aggregates and account lookup/verification have focused production HTTP tests, including rate limiting and restart. | This is not the complete upstream status surface. | | Status: summary, collections, segments | **blocked** | Stream has a custom summary. | Upstream exposes phase entry time, backfill timing/cursor/counts, live cursors, storage/manifest data, imports, and collections/segments views. Stream implements no collections or segments tabs and lacks several underlying persisted fields. | @@ -72,7 +72,7 @@ is in [bootstrap-semantic-parity.md](bootstrap-semantic-parity.md). | OpenTelemetry | **partial** | Provider configuration, propagation, sampling, batching, TLS/mTLS, and representative production spans have real collector tests. | “Every pinned production span” has not been independently re-audited in this pass. Treat the enumerated tested spans as evidence, not a blanket closure. | | Inspect/version command surfaces | **partial** | `version`, sealed `inspect-segment`, active inspection, and representative `inspect-all` reports have fixtures and golden comparisons. | Inspection does not compensate for missing online status tabs, and the current audit did not rerun every golden against the current commit. | | Differential oracle | **blocked as an admission proof** | It provides useful event-log, final-state, restart, repair, and public-client coverage against a pinned upstream simulator. Focused production-boundary tests now prove per-repository writer-gated completion and same-disk reopen with a sibling held incomplete. | The oracle still needs an invariant-based process-kill world that scales this beyond a two-repository fixture; its old four-account `after_repo_complete` schedule could not detect a dispatch-scale replay defect. | -| Strict power-loss oracle | **partial** | It exercises acknowledged-write reconstruction at named write/fsync/rename boundaries. Focused tests separately inject Store read failures for lifecycle phase and relay cursor, and the completion reopen test proves completed-versus-interrupted repository classification around an unchanged listRepos cursor. | It does not yet inject compaction/merge metadata read failures, missing manifest files during replay, long listRepos cursors, or storage loss specifically between archive fsync and the completion metadata commit. | +| Strict power-loss oracle | **partial** | It exercises acknowledged-write reconstruction at named write/fsync/rename boundaries. Focused tests separately inject Store read failures for lifecycle phase, relay cursor, and compaction watermark; the completion reopen test proves completed-versus-interrupted repository classification around an unchanged listRepos cursor. | It does not yet inject merge-cursor read failures, missing manifest files during replay, long listRepos cursors, or storage loss specifically between archive fsync and the completion metadata commit. | | Exact artifact admission | **blocked** | Commit `4d163e8` had a recorded suite and immutable OCI digest. | The experiment deployed `e1926f3`, which included source and dependency changes after that receipt. No mechanism prevented deployment of a non-admitted digest. | ## Evidence that must not be overread diff --git a/docs/upstream-bootstrap-spec.md b/docs/upstream-bootstrap-spec.md index 61b8f8b..cc968cf 100644 --- a/docs/upstream-bootstrap-spec.md +++ b/docs/upstream-bootstrap-spec.md @@ -88,8 +88,8 @@ forced-drain rules. Focused tests hold one sibling incomplete, cross the writer boundary, and fully reopen the store without replaying the completed sibling. The same audit found fail-open phase, relay-cursor, and compaction-watermark -reads. Phase and relay-cursor decoding now fail closed and have injected read -fault tests; the watermark gap remains. The audit also found a fail-open merge +reads. All three now fail closed on Store errors and malformed or unknown +encodings, with injected read-fault tests. The audit also found a fail-open merge source guard, incomplete post-bootstrap discovery, an unbounded retry candidate scan, and retry-supervisor failures hidden from health. Current Stream status and the evidence needed to close each remaining gap are maintained in diff --git a/src/internal/compact/pass.zig b/src/internal/compact/pass.zig index 65975d9..315242a 100644 --- a/src/internal/compact/pass.zig +++ b/src/internal/compact/pass.zig @@ -126,7 +126,7 @@ fn runInner( }; stats.target_watermark = target; - var current = wm.load(io) orelse 0; + var current = (try wm.load(io)) orelse 0; if (opts.instrumentation) |s| { s.compaction_watermark.store(current, .monotonic); s.compaction_watermark_lag_us.store(watermarkLagUs(metas.items, current), .monotonic); @@ -451,7 +451,7 @@ test "pass: drops superseded rows, keeps markers, commits watermark, idempotent" try testing.expectEqual(@as(u64, 1), stats.rows_dropped_by_reason.record); try testing.expectEqual(@as(u64, 1), stats.rows_dropped_by_reason.account); try testing.expect(stats.bytes_rewritten > 0); - try testing.expectEqual(@as(?u64, 6), wm.load(io)); + try testing.expectEqual(@as(?u64, 6), try wm.load(io)); try testing.expectEqual(@as(u64, 1), instrumentation.compaction_passes.getPtrConst(.ok).load(.monotonic)); try testing.expectEqual(@as(u64, 1), instrumentation.compaction_tombstones_collected.getPtrConst(.record).load(.monotonic)); try testing.expectEqual(@as(u64, 1), instrumentation.compaction_tombstones_collected.getPtrConst(.account).load(.monotonic)); @@ -526,7 +526,7 @@ test "pass: tombstone cap splits the pass into chunks across segments" { try testing.expectEqual(@as(u64, 4), stats.segments_examined); try testing.expectEqual(@as(u64, 2), stats.segments_skipped_clean); try testing.expectEqual(@as(u64, 2), stats.max_rewrite_workers); - try testing.expectEqual(@as(?u64, 4), wm.load(io)); + try testing.expectEqual(@as(?u64, 4), try wm.load(io)); } test "pass: commit rename fault fails loud, preserves source, and retry converges" { @@ -562,7 +562,7 @@ test "pass: commit rename fault fails loud, preserves source, and retry converge try testing.expect(segment_io.armSpec("rename:1:eio")); try testing.expectError(error.InputOutput, run(testing.allocator, io, seg_dir, &wm, null, .{ .rewrite_workers = 1 })); - try testing.expectEqual(@as(?u64, null), wm.load(io)); + try testing.expectEqual(@as(?u64, null), try wm.load(io)); try testing.expectError(error.FileNotFound, seg_dir.openFile(io, "seg_0000000000.jss.tmp", .{})); const after_fault = try seg_dir.readFileAlloc(io, "seg_0000000000.jss", testing.allocator, .limited(1 << 20)); defer testing.allocator.free(after_fault); @@ -572,7 +572,7 @@ test "pass: commit rename fault fails loud, preserves source, and retry converge const recovered = try run(testing.allocator, io, seg_dir, &wm, null, .{ .rewrite_workers = 1 }); try testing.expectEqual(@as(u64, 1), recovered.segments_rewritten); try testing.expectEqual(@as(u64, 1), recovered.rows_dropped); - try testing.expectEqual(@as(?u64, 2), wm.load(io)); + try testing.expectEqual(@as(?u64, 2), try wm.load(io)); } test "pass: malformed account marker records failure without advancing watermark" { @@ -604,7 +604,7 @@ test "pass: malformed account marker records failure without advancing watermark try testing.expectError(error.UnexpectedEof, run(testing.allocator, io, seg_dir, &wm, null, .{ .instrumentation = &instrumentation, })); - try testing.expectEqual(@as(?u64, null), wm.load(io)); + try testing.expectEqual(@as(?u64, null), try wm.load(io)); try testing.expectEqual(@as(u64, 1), instrumentation.compaction_passes.getPtrConst(.error_result).load(.monotonic)); try testing.expectEqual(@as(u64, 0), instrumentation.compaction_passes.getPtrConst(.ok).load(.monotonic)); } diff --git a/src/internal/compact/steady.zig b/src/internal/compact/steady.zig index 19ecb65..190e8ee 100644 --- a/src/internal/compact/steady.zig +++ b/src/internal/compact/steady.zig @@ -94,7 +94,7 @@ pub const Compactor = struct { pub fn rebuild(self: *Compactor) !void { var wm = watermark.Store.init(self.meta); defer wm.deinit(self.io); - const w = wm.load(self.io) orelse 0; + const w = (try wm.load(self.io)) orelse 0; var seg_dir = try self.openSegDir(); defer seg_dir.close(self.io); @@ -525,7 +525,7 @@ test "steady: timestamp rewrite lock excludes the real compaction pass without b try testing.expect(!task.finished.load(.acquire)); var wm = watermark.Store.init(&meta); defer wm.deinit(io); - try testing.expectEqual(@as(?u64, null), wm.load(io)); + try testing.expectEqual(@as(?u64, null), try wm.load(io)); locked = false; archive.unlockRewrites(); @@ -533,5 +533,5 @@ test "steady: timestamp rewrite lock excludes the real compaction pass without b if (task.failure) |err| return err; try testing.expect(task.finished.load(.acquire)); try testing.expectEqual(@as(u64, 2), task.result.?.target_watermark); - try testing.expectEqual(@as(?u64, 2), wm.load(io)); + try testing.expectEqual(@as(?u64, 2), try wm.load(io)); } diff --git a/src/internal/compact/watermark.zig b/src/internal/compact/watermark.zig index 08a4010..191aeff 100644 --- a/src/internal/compact/watermark.zig +++ b/src/internal/compact/watermark.zig @@ -15,11 +15,12 @@ pub const Store = struct { } pub fn deinit(_: *Store, _: std.Io) void {} - pub fn load(self: *Store, _: std.Io) ?u64 { - const maybe_bytes = self.meta.getAlloc(self.meta.allocator, key) catch return null; + pub fn load(self: *Store, _: std.Io) !?u64 { + const maybe_bytes = try self.meta.getAlloc(self.meta.allocator, key); const bytes = maybe_bytes orelse return null; defer self.meta.allocator.free(bytes); - if (bytes.len != 9 or bytes[0] != version) return null; + if (bytes.len != 9) return error.InvalidCompactionWatermarkLength; + if (bytes[0] != version) return error.UnsupportedCompactionWatermarkVersion; return std.mem.readInt(u64, bytes[1..9], .little); } @@ -31,7 +32,7 @@ pub const Store = struct { } pub fn initFloor(self: *Store, io: std.Io, next_seq: u64) !u64 { - if (self.load(io)) |w| return w; + if (try self.load(io)) |w| return w; const w = next_seq -| 1; try self.save(io, w); log.info("compaction watermark floored at {d}", .{w}); @@ -48,8 +49,31 @@ test "watermark floor and updates" { var meta = try meta_store.Store.open(testing.allocator, path_buf[0..path_len]); defer meta.deinit(); var store = Store.init(&meta); - try testing.expectEqual(@as(?u64, null), store.load(testing.io)); + try testing.expectEqual(@as(?u64, null), try store.load(testing.io)); try testing.expectEqual(@as(u64, 0), try store.initFloor(testing.io, 0)); try store.save(testing.io, 99); - try testing.expectEqual(@as(?u64, 99), store.load(testing.io)); + try testing.expectEqual(@as(?u64, 99), try store.load(testing.io)); +} + +test "watermark corruption and storage errors fail closed without initialization" { + const testing = std.testing; + var tmp = testing.tmpDir(.{}); + defer tmp.cleanup(); + var path_buf: [std.fs.max_path_bytes]u8 = undefined; + const path_len = try tmp.dir.realPath(testing.io, &path_buf); + var meta = try meta_store.Store.open(testing.allocator, path_buf[0..path_len]); + defer meta.deinit(); + var store = Store.init(&meta); + + try meta.putDurable(key, &.{ 1, 2, 3 }); + try testing.expectError(error.InvalidCompactionWatermarkLength, store.initFloor(testing.io, 100)); + const malformed = (try meta.getAlloc(testing.allocator, key)).?; + defer testing.allocator.free(malformed); + try testing.expectEqualSlices(u8, &.{ 1, 2, 3 }, malformed); + + try meta.putDurable(key, &.{ 2, 0, 0, 0, 0, 0, 0, 0, 0 }); + try testing.expectError(error.UnsupportedCompactionWatermarkVersion, store.load(testing.io)); + try testing.expect(meta_store.armReadFault(key, 1)); + defer meta_store.disarmReadFault(); + try testing.expectError(error.InjectedStoreFault, store.initFloor(testing.io, 100)); }