diff --git a/GETREPO_NOTES.md b/GETREPO_NOTES.md new file mode 100644 index 0000000..61d9230 --- /dev/null +++ b/GETREPO_NOTES.md @@ -0,0 +1,27 @@ +# getRepo notes + +Context: on 2026-06-29, operators called out `com.atproto.sync.getRepo` as an expensive route for PDS hosts. The main concerns were compression cost, large repo export reads, avoiding disclosure of stale/deleted blocks, and isolating backup/backfill traffic from user-facing work. + +ZDS is directly relevant because it serves `com.atproto.sync.getRepo` from `src/atproto/sync.zig`. + +Current shape: + +- ZDS serves raw CAR bytes and does not appear to add application-layer gzip, so it avoids the Node compression issue described in the operator thread. +- Full repo export is built from SQLite `repo_blocks`, ordered by CID. +- Incremental export with `since` filters by `repo_rev`. +- Repo writes already use lazy MST loading backed by `repo_blocks`, so ZDS is closer to a Hubble-style block/CAR-serving shape than an eager full-repo rebuild path. +- `subscribeRepos` has a connection cap, but `getRepo` does not appear to have route-specific backpressure. + +Open questions: + +- Should full `getRepo` walk the current commit/MST and include only reachable blocks, instead of exporting every retained block for the DID? +- Do retained old MST or record blocks make the current full export disclose deleted or stale record data? +- Should `getRepo` have route-specific concurrency or rate limits separate from normal public reads? +- Should the benchmark suite compare the current range-scan export against a reachability-based export for small, medium, and large repos? + +Suggested next pass: + +1. Add a regression test for delete/update followed by full `getRepo`, checking whether stale record blocks are exported. +2. If stale blocks are visible, change full export to current-root reachability rather than whole-DID block scan. +3. Add lightweight route backpressure for full `getRepo`. +4. Re-run `bench repo-size` and keep raw range-scan numbers separate from reachable-export numbers.