very fast at protocol indexer with flexible filtering, xrpc queries, cursor-backed event stream, and more, built on fjall
rust fjall at-protocol atproto indexer

[db] add collection id varint encoding master

foundation for interning collection NSIDs in composite keys: a record key spends 18 of its 45 bytes on app.bsky.feed.like, and the top five collections carry 98.7% of all records. LEB128 rather than a fixed width, which decouples id headroom from key size: one byte covers the hot collections, five reach 2^35, and only rare collections pay the wide encoding. a fixed u32 would have forced a choice between headroom and size, and cost 4 more bytes on the dominant path. three invariants the key format will lean on, each property-tested: - prefix-free, so the segment is self-delimiting and composite keys need no separator after it while prefix scans still isolate one collection - the terminating byte is below 0x80, so incrementing a prefix's last byte for an exclusive upper bound cannot overflow - no allocatable id encodes to a leading ASCII letter, and an NSID always starts with [a-zA-Z], so interned and legacy segments never overlap. allocation skips A-Z and a-z, leaving 76 single-byte ids. the chunked migration needs this: a multi-byte id sorts after the text it replaces, so a rewrite does re-scan its own output and must recognize it. not order-preserving, which costs nothing here — id order was never NSID order, and a (did, collection) group stays contiguous as an exact key prefix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>


+312 -1
3 changed files