A public mirror for the whole atmosphere hubble.microcosm.blue
readme.md

hubble-mini #

rolling snapshots of all repos on non-bluesky PDSes

this is a quick sketch to protect against more repos disappearing while the main hubble project is still under development. scoping it to non-bluesky makes it a quick project (hopefully bluesky is doing a good job backing up repositories).

since the purpose is simple temporary archiving, we make some specific choices:

  • no identity resolution

    • which means no signature verification -- any PDS can lie about repos it hosts
    • but we keep all repos from every PDS, even if they have cross-PDS DID-conflicts
  • full re-fetch on repo update

    • we retain the latest repo rev per (pds, repo) to avoid downloading exact archives we already have
    • don't run too often, to avoid excessive egress from indie hosts
    • offer Accept-Encoding headers

since this is part of hubble, we also make some choices that mostly only make sense for where we're headed:

  • record-level storage

    • repositories are spread out over a rocksdb keyspace, with individual records keyed by pds, did, and record path
    • repo deletes use rocks' range-delete functionality
    • lexicographic key order means we can still recover a full STAR-lite archive efficiently, or a stream-ordered CAR with minimal overhead using the STAR-lite MST-recovery algorithm.
  • repo statistics info

    • wire size, uncompressed car size, backlinks counts, blob ref counts by mime type, etc.