From d4e4e7361497b1185baed0e3489c5d8eeb07f52f Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Mon, 28 Sep 2026 11:44:39 -0500 Subject: [PATCH] Explain upstream filtering and bounded state reconciliation --- protocols/atproto/firehose.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/protocols/atproto/firehose.md b/protocols/atproto/firehose.md index d5f99fa..d46bbfd 100644 --- a/protocols/atproto/firehose.md +++ b/protocols/atproto/firehose.md @@ -62,6 +62,32 @@ gotcha found on the way: two canonical CBOR map-key orderings coexist — produc from [music-atmosphere-feed](https://github.com/zzstoatzz/music-atmosphere-feed) - uses jetstream to filter posts containing music links. +## filtering before the network boundary + +A consumer-side collection filter saves processing, not ingress bandwidth. +If an embedded ingestion service receives the full relay stream and then +discards unrelated records, the full stream has already crossed the network +link. To reduce home-network traffic, request filtering from an upstream +service before transmission. Check every ingestion path: adding a filtered +WebSocket does not help if an embedded full-firehose consumer still runs. + +For an application that needs a few collections, a filtered live subscription +can be paired with bounded `listRecords` reconciliation from the relevant +accounts' PDSes. These solve different problems: the stream gives timely +notifications; reconciliation repairs current state after missed events. +Neither reconstructs historical actions beyond the stream's replay retention. + +Only infer deletion after fetching and validating every page successfully. +A timeout on page two is not evidence that records absent from page one were +deleted. For permission-bearing records, reconcile current authoritative state +rather than blindly applying an old replayed grant after a newer revocation. +Persist the event cursor only after successful processing so failure remains +replayable. This design trusts the resolved PDS and filtered stream; it does +not supply the raw firehose's cryptographic proofs. + +Measure socket or interface bytes to verify the bandwidth change. Event counts +and a quiet application log cannot establish how much data crossed the link. + ## event structure firehose events contain: @@ -217,6 +243,7 @@ the For You algorithm on bluesky runs on someone's gaming PC, consuming the same ## sources +- [Spindle filtered-stream fork](https://tangled.org/zzstoatzz.io/spindle) — `SPINDLE_FORK.md`, `spindle/recovery.go`, `spindle/recovery_test.go`, and `jetstream/replay_test.go`; HTTP pagination, permission deletion, replay, and deployment traffic checked 2026-09-28 - [official Go client](https://github.com/bluesky-social/jetstream/blob/58c4d7f/client_core.go) — `runLiveOnly` cursor/dedup coupling; Zig SDK `src/live.zig` and `src/loopback_test.zig`, checked 2026-09-20 - [Jetstream resuming](https://bsky.network/docs/jetstream/#resuming-where-you-left-off) — timestamp cursors on the v2 live endpoint, checked 2026-09-20; Stream `src/internal/serve/server.zig` (`resolveCursor`, `OutdatedCursor`) for retention behavior -- 2.51.2