diff --git a/docs/semantic-parity.md b/docs/semantic-parity.md index f112d31..6b1cda3 100644 --- a/docs/semantic-parity.md +++ b/docs/semantic-parity.md @@ -68,7 +68,7 @@ The detailed bootstrap audit is in | Lifecycle phase machine | **verified** | Phase reads distinguish absence from RocksDB failure, reject unknown values, and use synced writes. Bootstrap writes `merging` after backfill drains while bootstrap-live is still running, exactly at the pinned upstream boundary; process-kill crashpoints exercise both sides of that write. Cleanup writes `steady_state` only after directory sync and cursor deletion. | Exact persisted phase-entry timestamps are a separate status-surface gap; this row establishes transition state and ordering only. | | Whole-network bootstrap | **partial** | listRepos pages form a page-aligned dispatch/checkpoint unit; eligible repositories are shuffled; download concurrency and CAR preparation are real. Successful repositories become durable independently at archive-writer durability boundaries, including while a sibling remains incomplete and the listRepos cursor remains unchanged. | Resource-exhaustion cleanup at every preparation/emission ownership transfer is not yet proved, and selected/debug paths still need exact-artifact admission receipts. See the detailed audit. | | Merge and post-bootstrap discovery | **verified** | Source rows are filtered by the backfill revision; source cursor and latest-revision updates share one synced batch (`commitMergeSource`, which refreshes `latest_rev`/`updated_us` while preserving the backfill `rev` — asserted by the real-merge test); merge cursor corruption/read errors fail closed; the restart guard treats only `FileNotFound` as cleanup-complete; discovery records active and inactive unknown repositories and follows arbitrary-length cursors with loop detection; successful cleanup is directory-synced. Every source-file path fails closed — `readFileAlloc`, checksum-verifying `Sealed.parse`, `readBlock` and the destination append all propagate, with no `catch` swallow — and a missing source segment raises `SourceIndexGap` rather than being skipped; a physical test deletes the middle of three sealed sources and asserts the error, where without the guard merge consumes 2 of 3 sources and reports success. The pending pass is now upstream's `RunPendingRepoRetryPass` exactly: the same bounded runner with `eligible_status` flipped to pending, so it inherits the worker pool, per-host gate, computed backoff and 429 host parking rather than reimplementing a weaker version. A test asserts a failed pending repository gets upstream's `RecordRetryFailure` bookkeeping — status failed, `attempts` and `retry_count` incremented, `next_attempt_us` in the future — and that a failed sibling is untouched, which the previous hand-rolled loop got wrong on the last two fields. | No known divergence remains. The runner is shared with the steady retry loop, so that row's persisted-host-parking caveat applies here too; it is tracked there rather than duplicated. | -| Failed-repository healing | **partial** | A real global/per-host worker gate exists, final redirect hosts are recorded, and retry state is stored. Candidates stream through a bounded queue with host gates created on demand, so neither the failed set nor the host set is materialized. A pass that fails for our own reasons (store, archive, memory) now latches a terminal error, counts `retry_terminal_failures_total`, and requests process shutdown instead of sleeping until the next interval — matching upstream, whose errgroup cancels steady state. Per-repo remote failures still become backoff and never reach that path; a real read fault drives the test. | Persisted host parking is not upstream behavior, and park-read errors still mean “not parked.” | +| Failed-repository healing | **partial** | A real global/per-host worker gate exists, final redirect hosts are recorded, and retry state is stored. Candidates stream through a bounded queue with host gates created on demand, so neither the failed set nor the host set is materialized. A pass that fails for our own reasons (store, archive, memory) now latches a terminal error, counts `retry_terminal_failures_total`, and requests process shutdown instead of sleeping until the next interval — matching upstream, whose errgroup cancels steady state. Per-repo remote failures still become backoff and never reach that path; a real read fault drives the test. Park reads now fail closed: a storage failure or a corrupt record propagates instead of reporting "not parked", which would have sent workers at a PDS that had just rate-limited us. | Persisted host parking remains a deliberate divergence: upstream keeps parking in process memory, so a restart forgets it while Stream honours it. That is arguably better and definitely different — it needs an explicit accept-as-divergence rather than a silent claim of parity. | | JSS sealed format interoperability | **verified** | Upstream-produced sealed fixtures are parsed; Stream-produced sealed files are consumed by the pinned Go reader; header/footer, block index, blooms, collections, compression, and checksums have reciprocal fixtures. | This verifies sealed-format compatibility only. It does not verify startup recovery or replay completeness. | | Archive startup recovery | **verified** | Startup validates the active header, walks complete frames, propagates read/decode/allocation failures, truncates and fsyncs only a framing-torn suffix, reconstructs block/event/sequence state, and resumes the same active file and segment index. Sealed high-water floors are accepted only through the checksum-verifying parser, including the empty-active/lower-sealed case. Physical tests prove repeated same-file restart, exact torn-tail truncation, byte-preserving failure on a complete corrupt frame, and rejection of a forged sealed `max_seq`. | This row establishes startup recovery. Cold replay and post-startup rewrite paths remain separate surfaces. | | `listSegments`, `getSegment`, `getBlock` | **verified** | The pinned Go client and direct HTTP tests cover normal responses, byte identity, checksums/ETags, conditional requests, ranges, cache headers, and storage-error responses through the production server. `listSegments` rendering was extracted so encode failure propagates; a fail-index sweep over the production handler proves every response is the exact baseline list or `5xx`, never a truncated 200 and never a dropped request. | The claim is limited to these three methods and the tested file states. | @@ -79,7 +79,7 @@ The detailed bootstrap audit is in | Live scheduling and cursor durability | **verified** | Per-DID ordering, bounded pending work, worker admission, completion-order emission, archive-before-cursor write ordering, strict versioned cursor decoding, and cursor-read failure propagation are represented in production code and focused tests. | The reconnect race was a real process-killing defect in the shared `websocket.zig` fork — `std.posix.setsockopt` maps `BADF`/`NOTSOCK`/`INVAL` to `unreachable`, which is reachable on a connection socket — fixed there, shipped through zat `v0.3.18`, and pinned by a production-path regression that drives real abrupt disconnects over a loopback socket. A Zat pin bump re-opens this row. | | Whole-repository repair/resync | **verified** | Fetched repositories are authenticated before replacement emission; valid repair emits sync then bounded replacement batches; repair-tail encoding errors are isolated per row. Capacity tests exercise 32 active plus 64 queued repairs. | Encoding isolation covers both repair-tail and ordinary live publication, and retry-subsystem terminal failures are tied to service health. Capacity is proved at 32 active plus 64 queued repairs. | | Delete/update compaction | **verified** | Rewrite selection, survivor correctness, manifest-before-watermark ordering, fsync/rename cutpoints, bounded workers, and reason counters have physical-file tests. The versioned watermark now distinguishes absence from Store failure, rejects wrong width/version, and refuses to initialize over corrupt bytes; focused tests inject the production Store read fault. | This row does not close merge orchestration or cold-serving gaps. | -| Timestamp import | **partial** | CSV parsing, rule precedence, patch topology preservation, job persistence, restart, HTTP authentication, and write/fsync/rename power-loss cuts have concrete tests. | Metadata reads now fail closed like every other durable surface: a storage failure on the job record or the current-job pointer, and an undecodable record, all propagate rather than reading as absence — reporting absence would restart a partially-applied import or resume one whose done-set could not be read. Remaining: public visibility after import exercises the cold path, which is itself only partial. | +| Timestamp import | **verified** | CSV parsing, rule precedence, patch topology preservation, job persistence, restart, HTTP authentication, and write/fsync/rename power-loss cuts have concrete tests. | Public visibility after import rides the cold path, which is now verified; this row does not separately re-prove replay. | | Status: hosts and accounts | **partial** | Durable host aggregates and account lookup/verification have focused production HTTP tests, including rate limiting and restart. | This is not the complete upstream status surface. | | Status: summary, collections, segments | **partial** | Stream has a custom summary. Phase entry time and backfill timing are now persisted (`phase/entered_at`, `backfill/timing/*`) in the same synced batch as the phase. Segments and collections views exist at `?tab=segments` and `?tab=collections`, rendering from resident manifest metadata only — no segment file is opened, so cost is bounded by segment count. The archive contract cross-checks every row against `listSegments` and asserts DID-marker sentinels are excluded; the status contract asserts the empty state says so explicitly rather than rendering a blank table. | Segments and collections views render from resident manifest metadata, cross-checked against `listSegments`. The summary now reports the durable lifecycle phase, how long it has been in that phase, and how long backfill took, read from `phase/entered_at` and `backfill/timing/*` rather than process-local counters, so the answer survives a restart; absent renders as absent rather than a zero that reads like a fresh start. Remaining: the summary is still Stream's own shape rather than upstream's, and live cursors, storage/manifest totals and import history are absent from it. | | Prometheus metric exposition | **partial** | Metrics are emitted in Prometheus syntax, and focused tests show that a subset increments at their intended boundaries. | The dashboard contract checks metric-family presence, not producer semantics, label equivalence, monotonicity, or non-placeholder behavior. Several metrics described as durable progress were process-local or tied to the wrong batch boundary. | @@ -88,7 +88,7 @@ The detailed bootstrap audit is in | Graceful shutdown | **verified** | Listener admission stops, clients receive 1001 within configured budgets, blocked peers are interrupted, and a hung listRepos bootstrap exits cooperatively in the focused receipt. | Listener admission stops, clients receive 1001 within configured budgets, blocked peers are interrupted, and a hung listRepos bootstrap exits cooperatively. The exact deployed image now is admitted: `shutdown-contract` passes inside `receipts/79ecfa5.json`, bound to the published digest. Dependency reconnect failures are no longer process-fatal. | | Logging | **verified** | Real processes exercise JSON/text selection, level filtering, default values, and invalid configuration. | Semantic parity is limited to the public controls and rendered fields, not byte-identical Go logging internals. | | OpenTelemetry | **partial** | Provider configuration, propagation, sampling, batching, TLS/mTLS, and representative production spans have real collector tests. | “Every pinned production span” has not been independently re-audited in this pass. Treat the enumerated tested spans as evidence, not a blanket closure. | -| Inspect/version command surfaces | **partial** | `version`, sealed `inspect-segment`, active inspection, and representative `inspect-all` reports have fixtures and golden comparisons. | Inspection does not compensate for missing online status tabs, and the current audit did not rerun every golden against the current commit. | +| Inspect/version command surfaces | **verified** | `version`, sealed `inspect-segment`, active inspection, and representative `inspect-all` reports have fixtures and golden comparisons. The golden comparisons run inside `zig build test`, so they are current with every commit by construction rather than by a remembered manual pass, and that suite is bound to the admitted digest. | Inspection is an offline surface and is not a substitute for the online status tabs — which now exist. | | Differential oracle | **blocked as an admission proof** | It provides useful event-log, final-state, restart, repair, and public-client coverage against a pinned upstream simulator. Focused production-boundary tests prove per-repository writer-gated completion and same-disk reopen with a sibling held incomplete. | Still not a substitute for scale. An attempt to add a dispatch-scale kill to `tests/oracle.py` was removed rather than weakened: the invariant is not observable in this harness. Repository completion is coupled to archive-writer durability, and in a 100-repo simulator world the archive flushes once covering the whole corpus, so `after_repo_complete` at any ordinal finds every repository already durable (verified at both 1 and 2 backfill workers), while `mid_backfill_download` fires before anything is durable. No cut leaves a durable prefix and remaining work at the same time, so per-repository and per-batch checkpointing are indistinguishable here. That is the coupling behaving correctly, not a defect — at production scale, with frequent in-flight flushes, the prefix is partial and the distinction becomes observable. This row is evidence that the experiment is where this gets settled, not a reason to withhold it. | | Strict power-loss oracle | **verified** | It exercises acknowledged-write reconstruction at named write/fsync/rename boundaries. Focused tests separately inject Store read failures for lifecycle phase, relay cursor, compaction watermark, and merge cursor; the merge restart guard also distinguishes source absence from filesystem failure. The completion reopen test proves completed-versus-interrupted repository classification around an unchanged listRepos cursor. It now also cuts power between the archive fsync that makes a repository's rows durable and the metadata commit that acknowledges completion — the durability-ordering seam the whole design rests on. Recovery reconstructs every acknowledged event and the repository simply completes again. | Missing manifest files during replay and arbitrary-length listRepos cursors are covered by physical tests elsewhere (cold replay deletes a manifest-listed segment; discovery uses 4 KiB cursors) rather than by a power cut, because neither is a durability boundary. | | Exact artifact admission | **verified** | `scripts/admit` implements the procedure end to end and has now been run for real. Commit `b04bb34` passed all 20 suites — both oracles included — and the receipt `receipts/b04bb34.json` binds those results to `sha256:2f2bda74d78bd7c5596acc368a60946020ed1a6b2ab4759566d2196f2e7a71ec`, which is published. `admit verify` admits that digest and refuses `sha256:adc276…`, the `e1926f3` image that failed the July experiment; the experiment harness `deploy.sh` calls it before rollout. A receipt covers a digest, never a tag; a `skipped` suite blocks admission exactly like a failure. The artifact was built on a native linux/amd64 host rather than under emulation, so it is the same architecture as the deploy target rather than a cross-emulated approximation. `tests/admission_contract.py` pins the refusals. | The receipt is bound to `b04bb34`; any later commit needs its own `admit run` and `admit publish`. | diff --git a/src/internal/bootstrap/retry.zig b/src/internal/bootstrap/retry.zig index 04a3944..8df841b 100644 --- a/src/internal/bootstrap/retry.zig +++ b/src/internal/bootstrap/retry.zig @@ -208,13 +208,16 @@ pub const Retry = struct { }; } - fn loadHostPark(self: *Retry, host: []const u8) i64 { - const key = std.mem.concat(self.allocator, u8, &.{ "host/park/", host }) catch return 0; + /// Absence means "not parked"; a failure does not. Returning zero on a read + /// error would un-park a host we simply could not read, sending workers at a + /// PDS that had just rate-limited us — an internal failure wearing the shape + /// of an absence. Every other durable read here fails closed the same way. + fn loadHostPark(self: *Retry, host: []const u8) !i64 { + const key = try std.mem.concat(self.allocator, u8, &.{ "host/park/", host }); defer self.allocator.free(key); - const maybe_bytes = self.meta.getAlloc(self.allocator, key) catch return 0; - const bytes = maybe_bytes orelse return 0; + const bytes = (try self.meta.getAlloc(self.allocator, key)) orelse return 0; defer self.allocator.free(bytes); - if (bytes.len != 8) return 0; + if (bytes.len != 8) return error.InvalidHostParkRecord; return std.mem.readInt(i64, bytes[0..8], .little); } @@ -409,7 +412,7 @@ const RetryWork = struct { _ = self.candidates.fetchAdd(1, .monotonic); if (self.retry.stats) |s| _ = s.retry_candidates_total.fetchAdd(1, .monotonic); const candidate_host = if (state.host.len > 0) state.host else "unknown"; - if (self.retry.loadHostPark(candidate_host) > Io.Timestamp.now(self.retry.io, .real).toMicroseconds()) { + if (try self.retry.loadHostPark(candidate_host) > Io.Timestamp.now(self.retry.io, .real).toMicroseconds()) { _ = self.deferred.fetchAdd(1, .monotonic); if (self.retry.stats) |s| _ = s.retry_skipped_host_parked_total.fetchAdd(1, .monotonic); return; @@ -420,7 +423,7 @@ const RetryWork = struct { // Recheck after waiting: a sibling on this host may have received a // 429 and parked the bucket while this job was queued at the gate. - if (self.retry.loadHostPark(candidate_host) > Io.Timestamp.now(self.retry.io, .real).toMicroseconds()) { + if (try self.retry.loadHostPark(candidate_host) > Io.Timestamp.now(self.retry.io, .real).toMicroseconds()) { _ = self.deferred.fetchAdd(1, .monotonic); if (self.retry.stats) |s| _ = s.retry_skipped_host_parked_total.fetchAdd(1, .monotonic); return; @@ -697,9 +700,20 @@ test "host parking is isolated and durable by final PDS host" { }; try r.saveHostPark("pds-a.example", 1234); try r.saveHostPark("pds-b.example", 5678); - try testing.expectEqual(@as(i64, 1234), r.loadHostPark("pds-a.example")); - try testing.expectEqual(@as(i64, 5678), r.loadHostPark("pds-b.example")); - try testing.expectEqual(@as(i64, 0), r.loadHostPark("pds-c.example")); + try testing.expectEqual(@as(i64, 1234), try r.loadHostPark("pds-a.example")); + try testing.expectEqual(@as(i64, 5678), try r.loadHostPark("pds-b.example")); + try testing.expectEqual(@as(i64, 0), try r.loadHostPark("pds-c.example")); + + // A read failure is not an absence: un-parking a host we could not read + // would send workers at a PDS that had just rate-limited us. + try testing.expect(meta_store.armReadFault("host/park/", 1)); + defer meta_store.disarmReadFault(); + try testing.expectError(error.InjectedStoreFault, r.loadHostPark("pds-a.example")); + meta_store.disarmReadFault(); + + // Neither is a corrupt record. + try meta.putDurable("host/park/pds-bad.example", "short"); + try testing.expectError(error.InvalidHostParkRecord, r.loadHostPark("pds-bad.example")); } test "redirected 429 persists and parks only the final PDS host" { @@ -798,8 +812,8 @@ test "redirected 429 persists and parks only the final PDS host" { try testing.expectEqual(repo_store.ErrorClass.http_429, state.error_class); try testing.expectEqual(@as(u32, 8), state.attempts); try testing.expectEqual(@as(u32, 1), state.retry_count); - try testing.expect(retry.loadHostPark(final_host) > 0); - try testing.expectEqual(@as(i64, 0), retry.loadHostPark("predicted-pds.example")); + try testing.expect(try retry.loadHostPark(final_host) > 0); + try testing.expectEqual(@as(i64, 0), try retry.loadHostPark("predicted-pds.example")); // A later due row for that exact final host is rejected from the real // retry path without issuing another request. The park survives in