diff --git a/skills/software-development/atproto-ecosystem-map/SKILL.md b/skills/software-development/atproto-ecosystem-map/SKILL.md new file mode 100644 index 0000000..f5f41c4 --- /dev/null +++ b/skills/software-development/atproto-ecosystem-map/SKILL.md @@ -0,0 +1,154 @@ +--- +name: atproto-ecosystem-map +description: "Use to map the ATProto network: PDS, relays, jetstreams." +version: 1.0.0 +author: Hermes Agent +license: MIT +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [ATProto, Bluesky, Network, Relay, Jetstream, PDS, AppView, Infrastructure] +--- + +# ATProto Ecosystem Map + +One-page orientation to the AT Protocol network: what each service type is, who runs +the notable public instances, and which skill to load for deeper work. Distilled from +the AT://links and DEEP AT://magic Semble collections (897 URLs, digested 2026-08-17). +Source data: `~/dev/_shared/docs/atproto-digest/` (catalog.md, classification.json, +notes/). + +## When to Use + +- You need to know which service does what (PDS vs relay vs jetstream vs appview vs + labeler) before writing any ATProto code or picking a provider. +- You need a registry of public ATProto infrastructure: relays, jetstreams, PDS + hosting, identity lookups, lexicon registries. +- You need to route to the right deeper skill for a specific task. + +## The network in one page + +``` + PDS (Personal Data Server) hosts accounts: repo, blobs, records + | sync (subscribeRepos) | + v v + RELAY <-- aggregates repos --- other PDSes; serves firehose of change events + | + | firehose (binary DAG-CBOR) + v + APPVIEW -- indexes app data, serves queries (feeds, profiles, likes...) + JETSTREAM -- consumes relay firehose, re-emits lightweight JSON (filtered) + LABELER -- signs moderation labels over records (e.g. mod.bsky.app) +``` + +| Service | Role | Owns/stores | Query API | +|---|---|---|---| +| **PDS** | Account host: repo + blobs + records | the user's repo (their data) | `com.atproto.repo.*`, `com.atproto.sync.*` (direct to PDS) | +| **Relay** | Aggregates repos from many PDSes; produces the firehose | copy of repos; no accounts | `com.atproto.sync.subscribeRepos` (websocket) | +| **AppView** | Indexes the network into app-level data | derived index (feeds, profiles) | `app.bsky.*` queries (e.g. `getProfile`, `getTimeline`) | +| **Jetstream** | JSON re-stream of relay firehose, server-side filters | nothing (transient) | `wss:///tap` (JSON) | +| **Labeler** | Moderation labels | labels | `com.atproto.label.*` | +| **PLC directory** | Identity registry (DID -> DID document) | DID documents | `plc.directory`, read replica `didplc.directory` | + +Key mental model: +- **Handles/DIDs resolve to a PDS** (via PLC directory). "Where is the account hosted?" + is answered by the DID document's `#atproto_pds` service endpoint. +- **Relays do not host accounts.** They mirror repos for indexing and streaming. +- **Bluesky's appview** (bsky.social public API) is NOT a PDS — it serves derived app + data. Record writes go to the account's PDS; reads of feeds/profiles go to an appview. + +## Public infrastructure registry (as of 2026-08, from the digest) + +### Relays (firehose) +- **firehose.network** (vayumandala, community): northamerica / europe / asia + regional relays. Replay window 72h. `wss://.firehose.network/xrpc/com.atproto.sync.subscribeRepos`. + Per-relay pages offer: check PDS status, request PDS crawl, DID/handle status. +- **bsky.network** (+ relay1.us-west/east.bsky.network): Bluesky's own relay, + largest (5,942 PDSes, ~23M accounts). +- **fire.hose.cam**: relay.fire.hose.cam, relay3.fr.hose.cam. +- **atproto.africa**: runs **rsky-relay** codebase (github.com/blacksky-algorithms/rsky) + — a community relay implementation, not the official one. +- Others: relay.feeds.blue, relay1.eurosky.network, zlay.waow.tech, relay.waow.tech, + relay.upcloud.world. (relay.xero.systems was unreachable at digest time.) +- **Index/status:** firehose.directory — relay index (16 relays, ~6,175 PDSes). + +### Jetstreams (JSON streaming) +- **firehose.stream** (vayumandala): sfo (relay: northamerica, replay 72h), + jet (Bluesky mirror, replay 24h), chennai (asia, 24h), nyc (72h), london (europe). + Endpoint `wss:///tap`. +- **jetstream.fire.hose.cam**, **jetstream.waow.tech**, jetstream1/2 regional hosts. +- Edge cache: **slingshot.firehose.stream** — caches records from Jetstream for fast + lookups. Notifications firehose: **spacedust.firehose.stream** — watch any + handle/DID/AT-URI's notifications. +- Full provider detail: load `atproto-jetstream-providers` (planned, Phase 3). + +### PDS hosting / ops +- **pds.directory** — index of all public PDSes (6,175; hostname, version, users, + open registration, relay status). +- **protobase.at** — managed multi-PDS hosting (provision in minutes, quota views). +- **pds.tokyonight.city** (Tranquil PDS, small private), **altq.net** (self-hosted + PDS, code = bluesky-social/pds), **pds.club** (unreachable at digest time). +- **atpairport.com** (Airport) — PDS migration + PLC key recovery assistance. +- Deeper ops knowledge: load `atproto-pds-ops` (planned, Phase 3). + +### Identity / DID +- **plc.directory** — canonical DID:PLC registry. **didplc.directory** — community + read replica (fast lookups, last-op, audit log). +- did:web creation guides: whtwnd.com "Creating a did:web atproto account using goat" + (bnewbold), blog.smokesignal.events did-method-web identity post. +- Passkeys as PLC rotation keys: plc-passkey.wisp.place. +- Deeper: load `atproto-identity-deep` (planned, Phase 4); for DID:WEB serving + patterns see the atproto-development skill. + +### Lexicon registries +- **lexicon.garden** (browser), **lexicon.store** (schema registry), **lexicons.bio**, + **rite.mino.mobi/lexicon** (word-level archive analysis), **at-store** lexicons. +- Deeper: load `atproto-lexicon-registry` (planned, Phase 4). + +### Content/community (misc) +- Firehose games (firehose.games, tic-tac-toe on ATProto), wisp.place static hosting, + tangled.org git hosting, Semble (semble.so) knowledge network, ATmosphereConf VODs. + +## Skill routing (load these for depth) + +| Task | Skill | +|---|---| +| Writing Python against any of this | `atproto-python` (Client, Firehose, Jetstream, IdResolver) | +| JS/TS apps, OAuth, wisp/Tangled deploy, DID:WEB | `atproto-development` | +| Blob tooling (orphans, backup, cleanup) | `atproto-blob-lifecycle` | +| Static site deploy (wispctl, Tangled Sites) | `atproto-site-deployment` | +| Small-biz domain/DID/PDS migration | `small-business-atproto-migration` | +| Bluesky post embeds in web apps | `bluesky-post-embeds` | +| Semble (Cosmik) API | `semble-cosmik` | +| Relays + firehose in depth | `atproto-relays-firehose` | +| Jetstream providers, edge caches, notifications firehose | `atproto-jetstream-providers` (planned) | +| PDS provisioning/ops/migration | `atproto-pds-ops` (planned) | +| Lexicon authoring/discovery | `atproto-lexicon-registry` (planned) | +| did:web via goat, PLC replica, passkey rotation | `atproto-identity-deep` (planned) | +| Feed generators, engagement graphs | `atproto-feeds-algorithms` (planned) | +| Ozone, labelers, moderation | `atproto-moderation-tools` (planned) | +| Scam/phish/security landscape | `atproto-security-landscape` (planned) | +| Client/viewer/utility catalog | `atproto-tools-catalog` (planned) | + +## Pitfalls + +- **Relay != AppView.** A relay gives you raw repo events (subscribeRepos); it will + not answer `getTimeline` or `getProfile`. Use an appview (bsky.social or similar) + for app-level queries. +- **Records live on the PDS, engagement lives in the appview.** Deleting a record + at its PDS removes it from appviews eventually; don't expect instant propagation. +- **Jetstream data is not cryptographically verifiable** (no repo signatures). + Use the firehose when verification matters. +- **Handles are not storage locations.** Always resolve handle -> DID -> DID doc -> + PDS endpoint before assuming where data lives. +- Public instances come and go (relay.xero.systems unreachable, pds.club private-IP). + Treat this registry as a snapshot; re-check firehose.directory / pds.directory for + current status. + +## Verify + +- `curl -s https://firehose.directory` shows live relay statuses. +- `curl -s https://pds.directory` shows current PDS count (6,175 at digest time). +- `curl -s https://plc.directory/` resolves any DID to its document. +- A websocket client connecting to `wss://sfo.firehose.stream/tap` receives JSON + frames within seconds (no auth needed for public streams). diff --git a/skills/software-development/atproto-relays-firehose/SKILL.md b/skills/software-development/atproto-relays-firehose/SKILL.md new file mode 100644 index 0000000..4643bec --- /dev/null +++ b/skills/software-development/atproto-relays-firehose/SKILL.md @@ -0,0 +1,122 @@ +--- +name: atproto-relays-firehose +description: "Use when working with ATProto relays and firehose streams." +version: 1.0.0 +author: Hermes Agent +license: MIT +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [ATProto, Relay, Firehose, subscribeRepos, Streaming, Infrastructure] +--- + +# ATProto Relays & Firehose + +How the relay network works, which public relays exist, and how to subscribe to the +firehose. Distilled from the AT://links and DEEP AT://magic Semble collections +(digested 2026-08-17); re-verify live status via firehose.directory. + +## When to Use + +- You need real-time ATProto data: every post, like, follow, delete, label event + across the network (or a subset). +- You are picking a relay to subscribe to, or wondering why your relay isn't showing + certain accounts. +- You need to know the difference between relay firehose and jetstream JSON streams. + +## Core concepts + +- **Relay** = an aggregator that syncs repos from PDSes and produces a firehose of + change events. AppViews consume a relay to build their indexes. +- **subscribeRepos** = the websocket subscription (`com.atproto.sync.subscribeRepos`) + that delivers commits as binary DAG-CBOR frames (MST/CAR blocks) — NOT JSON. +- **Replay window** = how far back a relay can replay history to a fresh cursor + (typically 72h; Bluesky-mirror jetstreams often 24h). Events older than the window + are unrecoverable from that relay. +- **Crawl requests**: relays can be asked to crawl a specific PDS (per-relay status + pages offer this) — useful when a newly-hosted account isn't appearing. + +## Public relays (status as of 2026-08) + +| Relay | Region | Notes | +|---|---|---| +| northamerica.firehose.network | NA | vayumandala community relay, replay 72h | +| europe.firehose.network | EU | vayumandala, replay 72h | +| asia.firehose.network | ASIA | vayumandala, replay 72h | +| bsky.network | global | Bluesky's own relay, largest (~5,942 PDSes, ~23M accounts) | +| relay1.us-west / relay1.us-east.bsky.network | US | Bluesky, same scale | +| relay.fire.hose.cam / relay3.fr.hose.cam | EU | community | +| atproto.africa | AF | runs **rsky-relay** codebase (github.com/blacksky-algorithms/rsky) | +| relay.feeds.blue, relay1.eurosky.network, zlay.waow.tech, relay.waow.tech, relay.upcloud.world | — | smaller community relays | +| relay.xero.systems | — | unreachable at digest time (deprecated) | + +Live index: **firehose.directory** (relay count, PDSes per relay, account counts). +PDS index: **pds.directory**. + +## Subscribing + +Websocket endpoint pattern: + +``` +wss:///xrpc/com.atproto.sync.subscribeRepos +``` + +Python (see atproto-python skill for full details): + +```python +from atproto import FirehoseSubscribeReposClient, parse_subscribe_repos_message + +client = FirehoseSubscribeReposClient() # defaults to wss://bsky.network/xrpc +def on_message(message): + commit = parse_subscribe_repos_message(message) + # commits are DAG-CBOR; decode blocks via CAR.from_bytes(commit.blocks) + ... +client.start(on_message) +``` + +- Filter cheaply on `message.header` (op/type) before decoding payloads. +- Pass a cursor in params to resume; respect the relay's replay window (see + FirehoseSubscribeReposClient docs). +- To subscribe to a specific region's relay: `FirehoseSubscribeReposClient(base_uri='wss://europe.firehose.network/xrpc')`. +- **Verifiability**: relay firehose commits carry repo signatures/MST proofs — + this is the stream to use when you must verify data. Jetstream (plain JSON) does + NOT carry signatures; use it only for convenience/filtering (see + `atproto-jetstream-providers`). + +## rsky-relay (community relay codebase) + +- atproto.africa runs `rsky-relay` (github.com/blacksky-algorithms/rsky). +- Same subscribeRepos wire path (`/xrpc/com.atproto.sync.subscribeRepos`). +- Use when you want to run your own relay without the official Go implementation + (bluesky-social/indigo). + +## Pitfalls + +- **subscribeRepos is binary, not JSON.** Don't parse frames as text; use an SDK + client or CAR decoding. +- **Replay windows are short.** If your consumer is down longer than the window, + you must re-crawl or start fresh — cursors into the past are rejected/empty. +- **Not every PDS is on every relay.** Regional relays prioritize their region; + request a crawl from the relay's status page if an account is missing. +- **Relays die or go stale** (relay.xero.systems). Check firehose.directory before + wiring a relay into production; prefer the big stable ones (bsky.network, + firehose.network) unless you need region-local data. +- **Don't confuse relay events with appview state.** A commit event means the repo + changed; it does not mean Bluesky's appview has indexed it yet. +- atproto.africa's relay shows 0 accounts for its own PDS crawl — smaller relays may + serve limited account sets. Confirm coverage before relying on one for full-network + analysis. + +## Verify + +- `curl -s https://firehose.directory` — relay statuses are live. +- Connect to `wss://northamerica.firehose.network/xrpc/com.atproto.sync.subscribeRepos` + and confirm frames arrive within seconds (no auth). +- SDK check: `FirehoseSubscribeReposClient().start(print)` shows commits immediately. + +## Related skills + +- `atproto-ecosystem-map` — network roles + full provider registry (load first if + you're not sure which service you need). +- `atproto-python` — FirehoseSubscribeReposClient / Labels client in depth. +- `atproto-jetstream-providers` (planned) — JSON re-streams, edge caches, filters.