diff --git a/docs/tap-backfill.md b/docs/tap-backfill.md index daac10c..faec18f 100644 --- a/docs/tap-backfill.md +++ b/docs/tap-backfill.md @@ -11,9 +11,10 @@ The command: - installs `tap` with `go install github.com/bluesky-social/indigo/cmd/tap@latest` when `tap` is not already on `PATH` -- starts TAP with `TAP_FULL_NETWORK=true` -- filters TAP record delivery to `TAP_COLLECTION_FILTERS=fm.teal.*`, so only Teal - records reach Cadet after TAP enumerates repos +- starts TAP with `TAP_SIGNAL_COLLECTION=fm.teal.alpha.feed.play` +- keeps full-network repo enumeration off by default with `TAP_FULL_NETWORK=false` +- filters TAP record delivery to `TAP_COLLECTION_FILTERS=fm.teal.*`, so all Teal + records from discovered Teal repos reach Cadet - stores TAP state and logs under `.teal-tap/` - runs `cargo run -p cadet --bin tap-backfill` @@ -37,18 +38,18 @@ DATABASE_URL=postgres://teal:teal@127.0.0.1:5432/teal TAP_HOST=127.0.0.1 TAP_PORT=2480 TAP_CHANNEL_URL=ws://127.0.0.1:2480/channel -TAP_FULL_NETWORK=true +TAP_SIGNAL_COLLECTION=fm.teal.alpha.feed.play TAP_COLLECTION_FILTERS=fm.teal.* TEAL_TAP_DIR=.teal-tap ``` -Full-network mode is intentionally the default so Teal can discover records -across every `fm.teal.*` lexicon, including repos that may not have a -`fm.teal.alpha.feed.play` signal record. For a narrower run that only tracks -repos with Teal play records, use TAP's signal-collection mode: +TAP's signal collection controls which repos are backfilled. Teal defaults that +boundary to repos with `fm.teal.alpha.feed.play` records, then applies +`fm.teal.*` as the record-delivery filter. To enumerate every repo on the +network instead, run: ```bash -TAP_FULL_NETWORK=false TAP_SIGNAL_COLLECTION=fm.teal.alpha.feed.play pnpm backfill +TAP_FULL_NETWORK=true TAP_SIGNAL_COLLECTION= pnpm backfill ``` Keep Postgres running and migrated before starting the backfill. The command is diff --git a/scripts/backfill-tap.sh b/scripts/backfill-tap.sh index d3ffa3f..2f80d03 100755 --- a/scripts/backfill-tap.sh +++ b/scripts/backfill-tap.sh @@ -9,10 +9,11 @@ TAP_BASE_URL="${TAP_BASE_URL:-http://$TAP_HOST:$TAP_PORT}" TAP_CHANNEL_URL="${TAP_CHANNEL_URL:-ws://$TAP_HOST:$TAP_PORT/channel}" TAP_LOG="${TAP_LOG:-$TAP_DIR/tap.log}" -# Full-network mode discovers every findable repo, then record delivery is -# filtered separately so only fm.teal.* records flow through Cadet. -export TAP_FULL_NETWORK="${TAP_FULL_NETWORK:-true}" -export TAP_SIGNAL_COLLECTION="${TAP_SIGNAL_COLLECTION:-}" +# TAP uses this collection to discover repos worth backfilling. Record delivery +# is still filtered separately so every fm.teal.* record from those repos flows +# through Cadet. +export TAP_FULL_NETWORK="${TAP_FULL_NETWORK:-false}" +export TAP_SIGNAL_COLLECTION="${TAP_SIGNAL_COLLECTION:-fm.teal.alpha.feed.play}" export TAP_COLLECTION_FILTERS="${TAP_COLLECTION_FILTERS:-fm.teal.*}" export TAP_DISABLE_ACKS="${TAP_DISABLE_ACKS:-true}" export TAP_CHANNEL_URL diff --git a/todo.md b/todo.md index 617bd32..6042149 100644 --- a/todo.md +++ b/todo.md @@ -13,7 +13,7 @@ This file is the working handoff for the Teal-native Teal clone. Keep it updated - Amethyst social posts and recent listens merge indexed Teal actor data with Bluesky fallback handle/display-name/avatar data when the appview only has partial profile rows. - Amethyst profile images use the Bluesky CDN avatar/banner transforms for indexed Teal blob CIDs, and signed-in users can edit their Teal display name, bio, avatar, and banner from their own profile page. - Amethyst music track pretty URLs resolve from artist/release/track slugs when no play URI query string is present, instead of falling back to the latest global play. -- Cadet has a TAP backfill consumer and `pnpm backfill` command. The command runs TAP full-network backfill by default, filters delivered records with `TAP_COLLECTION_FILTERS=fm.teal.*`, and consumes TAP record events through the existing Teal ingestors. Signal-collection TAP backfill remains an explicit env override for narrower runs. +- Cadet has a TAP backfill consumer and `pnpm backfill` command. The command discovers Teal repos with `TAP_SIGNAL_COLLECTION=fm.teal.alpha.feed.play`, filters delivered records with `TAP_COLLECTION_FILTERS=fm.teal.*`, and consumes TAP record events through the existing Teal ingestors. Full-network TAP backfill remains an explicit env override. - Live Jetstream ingestion has been verified end-to-end through Cadet, Postgres, Aqua, and the public preview URL. - Missing Teal profiles fall back to public Bluesky profile data with an in-app disclaimer, and signed-in listeners can publish a Teal profile through the onboarding wizard. - Development and production Compose files include Amethyst, Aqua, Cadet, Satellite, Postgres, and Garnet. @@ -64,7 +64,7 @@ This file is the working handoff for the Teal-native Teal clone. Keep it updated - [x] Verify delete handling removes the play URI from `plays`, `play_to_artists`, and `play_to_artists_extended`. - [x] Add a `subscribeRepos` CBOR adapter only if relay-level firehose sync becomes necessary. - [x] Keep CAR import as a backfill path and add regression tests for it. -- [x] Add TAP backfill setup for full-network `fm.teal.*` sync and a repeatable `pnpm backfill` command. +- [x] Add TAP backfill setup for signal-collection `fm.teal.*` sync and a repeatable `pnpm backfill` command. ## Next: Aqua And Lexicons