Something went wrong. Try again.
A public mirror for the whole atmosphere hubble.microcosm.blue
Something went wrong. Try again.
hubble mini
1 folder · 2 files
readme.md
hubble-mini #
rolling snapshots of all repos on non-bluesky PDSes
this is a quick sketch to protect against more repos disappearing while the main hubble project is still under development. scoping it to non-bluesky makes it a quick project (hopefully bluesky is doing a good job backing up repositories).
since the purpose is simple temporary archiving, we make some specific choices:
-
no identity resolution
- which means no signature verification -- any PDS can lie about repos it hosts
- but we keep all repos from every PDS, even if they have cross-PDS DID-conflicts
-
full re-fetch on repo update
- we retain the latest repo
revper (pds,repo) to avoid downloading exact archives we already have - don't run too often, to avoid excessive egress from indie hosts
- offer
Accept-Encodingheaders
- we retain the latest repo
since this is part of hubble, we also make some choices that mostly only make sense for where we're headed:
-
record-level storage
- repositories are spread out over a rocksdb keyspace, with individual records keyed by pds, did, and record path
- repo deletes use rocks' range-delete functionality
- lexicographic key order means we can still recover a full STAR-lite archive efficiently, or a stream-ordered CAR with minimal overhead using the STAR-lite MST-recovery algorithm.
-
repo statistics info
- wire size, uncompressed car size, backlinks counts, blob ref counts by mime type, etc.