Task 12: perf & scale — MST tree cache, streaming reachable-set getRepo, blocks GC, ClaimNext skip-scan master
Removes the four known scaling cliffs before any big community hits them (and the storage prerequisites for the votes-as-records design revisit). Behavior-preserving throughout: golden TID/at-uri tests untouched, firehose CAR slices identical with/without the cache (test-pinned, ordered comparison, create/update/delete ops), seq order == visibility order for both event kinds, TxSideEffect hook semantics unchanged. Benchmarks (Apple M4 Pro, local test postgres, 2k-record repo fixture; "before" = identical bench file against a throwaway HEAD worktree plus the two-line testutil *testing.T -> testing.TB backport): - PutRecord (per-DID MST tree cache, internal/repo/treecache.go): 140.6ms -> 3.30ms/op (42.6x), 2.34MB -> 115KB/op, 38,482 -> 750 allocs. Was one SELECT per MST node, full tree, per commit. LRU keyed by DID, head-CID-validated, detach-on-take, re-cached only after durable commit. MST_CACHE_SIZE env (default 512, n<=0 disables). Cold first commit per DID still pays one full-tree load. - getRepo / ExportCAR (streaming reachable-set export, repo.ExportCARTo): 38.6ms/4.77MB CAR -> 27.7ms/438KB CAR (10.9x smaller, historical blocks no longer exported); the sync handler streams with residency bounded by one 256-block batch + a CID seen-set, never the full CAR. Batched ANY() block fetches — the naive per-block walk measured 698ms/op (25x worse; recorded so nobody simplifies the batching away). Mid-stream failure now panics http.ErrAbortHandler so a truncated CAR is a transport-level failure, not a clean 200; client disconnects log at Debug; vanished-repo race maps to 404. - blocks GC (internal/repo/gc.go, invariant-first design per the task spec): delete only if unreachable from the current head (one REPEATABLE READ snapshot) AND older than BLOCKS_GC_RETENTION (72h default), created_at re-checked inside the DELETE. The commit path's ON CONFLICT now refreshes blocks.created_at ("last written"), so a re-introduced block survives a concurrent sweep — the retention window is the race guard (must exceed app<->DB clock skew + sweep gap). GetRecord/GetRecordProof/ExportCARTo moved to REPEATABLE READ. Wired via internal/prune.Run, 6h sweeps, batched deletes. - ClaimNext (internal/store): recursive-CTE loose index scan over the existing partial index; per-key head rows only, heads materialized via ARRAY(...) — a plain IN regressed the planner to the O(N) scan. 50k-deep backlogged key: 162.6ms/200,855 buffers -> 0.050ms/22. Same semantics: per-key serial ordering, claimed_until fencing, FOR UPDATE SKIP LOCKED. No schema change. Resumed from a killed predecessor agent's partial working tree: its MST cache and streaming-export skeleton were kept (verified against pinned indigo source: WriteDiffBlocks dirty-clearing makes cached-tree diffs identical; no-op inserts never dirty the tree); its unbatched export walk would have been a getRepo latency regression and was rewritten. 7-reviewer second-opinion pass (5 Claude specialty agents + codex gpt-5.6-sol + gemini 3.1 pro; glm watchdog-killed): no high-severity code findings. Applied: client-visible truncation abort (4/7 flagged); false parenthetical in the gc.go invariant header corrected (only NEWLY referenced blocks are in newBlocks — believing the original would have justified deleting the reachability rule); the DELETE-time created_at re-check, the RR isolation choice, and the 256-block batch boundary are now all load-bearing in tests with fail-then-pass proofs (previously the race-guard clause could be deleted with the suite staying green); ExportCARTo pre-first-byte contract doc corrected; WithTreeCacheSize doc de-contradicted; treecache ABA reasoning documented; clock-skew assumption documented (codex unique catch). Full unit suite green; make e2e green (17/17). `since` diff export deliberately deferred and documented as GC-constrained: a real diff export would read historical blocks, which the GC invariant explicitly does not guarantee. v1.1 loop (tasks 09-12) complete. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>