very fast at protocol indexer with flexible filtering, xrpc queries, cursor-backed event stream, and more, built on fjall
rust fjall at-protocol atproto indexer

[backfill] stream a whole getRepo body like a filtered one master

an unfiltered backfill buffered the whole getRepo body before parsing any of it, so a forged block or a pds that stalls partway only showed after the download ended, which can be up to max_car_body_bytes. the body then stayed pinned by every record until the batch committed, mst nodes included. the whole car now streams through CarStream like a filtered one, every block copied out and hashed as it arrives (WholeCar), and both readers share stream_car. the read loop also stops as soon as the reader fails rather than waiting for the next chunk, which a stalled pds may never send, so this covers the filtered path too. this doesn't bound the unfiltered peak, since that path keeps every block by design: the perf-backfill probe goes from 394.8/394.6 MB above baseline to 309.6/368.8 for pfrazee.com and 96.6/97.4 to 91.0/92.6 for jay.bsky.team, wall time unchanged. the rest of the peak is the records from the walk and the single repo batch. full_get_repo_checks_blocks_as_they_arrive fails (times out) with the whole car buffered and parsed at the end, and with the reader's error not ending the read loop. full_backfill_stores_every_record_a_whole_parse_finds fails at its record comparison when WholeCar drops some record blocks.


+252 -93
5 changed files