hubble-commit-corpus: real commits for hubble-commit #

captures #commit events off a relay, keeps the ones that actually invert, and times operation inversion against them. the generated and exhaustive test suites cover correctness; this covers the shapes producers really emit and how long they take.

it is a nested crate rather than an example or a feature-gated binary so that its dependencies — tokio, jacquard, a websocket stack — stay out of hubble-commit entirely. examples and dev-dependencies get built by cargo test, and keeping that fast is most of the point of the library.

cargo run --release -p hubble-commit-corpus -- capture local/commits.jsonl --commits 5000
cargo run --release -p hubble-commit-corpus -- bench   local/commits.jsonl
cargo run --release -p hubble-commit-corpus -- sample  local/commits.jsonl \
    hubble-commit/tests/fixtures/firehose/commits.jsonl --commits 100

what gets kept #

relays run lenient — they pass through commits that fail inductive proof, and commits from producers that never spoke sync 1.1 — so every commit is inverted before it is kept, and again after stripping, and the reasons for dropping are counted rather than silently swallowed. a capture run prints them:

seen=5002 kept=5000 no_prev_data=2 bad_car=0 bad_ops=0
  failed_inversion=0 failed_after_strip=0

what gets written #

one json object per commit, holding only what inversion consumes: the MST root the commit claims, the root it claims to have replaced, the operations, and the MST node blocks.

deliberately absent: record blocks (inversion never decodes them, only links to them by CID), the commit object, and the repo's DID. the walk that collects the tree follows l and t links and never v, so records are excluded by construction rather than by filtering. what is left identifies nobody — a record path is a collection name and a TID.

that is what makes sample output safe to commit. captures themselves belong in local/; they run to ~5.5 kB per commit.

sampling #

real commits are overwhelmingly one operation over eight or so blocks, so an even sample would be almost all the same shape. sample keeps every multi-operation commit it can, then spreads the rest over block count so the fixture spans small and large trees instead of clustering at the median.