diff --git a/skills/software-development/atproto-jetstream-providers/SKILL.md b/skills/software-development/atproto-jetstream-providers/SKILL.md new file mode 100644 index 0000000..fac375a --- /dev/null +++ b/skills/software-development/atproto-jetstream-providers/SKILL.md @@ -0,0 +1,125 @@ +--- +name: atproto-jetstream-providers +description: "Use for ATProto Jetstream JSON streams and edge caches." +version: 1.0.0 +author: Hermes Agent +license: MIT +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [ATProto, Jetstream, Streaming, JSON, EdgeCache, Infrastructure] +--- + +# ATProto Jetstream Providers + +Jetstream is the JSON layer over the ATProto firehose: it consumes a relay's binary +stream and re-emits lightweight JSON, filtered server-side. This skill covers the +public provider network, the v2 wire protocol, and the supporting services (edge +caches, notifications firehose). Distilled from the AT://links and DEEP AT://magic +Semble collections (2026-08-17); re-verify live status before relying on any host. + +## When to Use + +- You want real-time ATProto events as plain JSON without decoding DAG-CBOR/CAR. +- You need server-side filtering (collections, DIDs, kinds) to cut bandwidth. +- You need fast record/identity lookups (edge cache) or a notification watcher + without building one. +- You are deciding between firehose (verifiable binary) and jetstream (convenient + JSON) — see atproto-relays-firehose for the firehose side. + +## Core concepts + +- **Jetstream v2 wire** is the supported protocol. Legacy v1 hosts + (`jetstream1.*`, `jetstream2.*`) speak a frozen, different protocol — do not use + them with modern clients (the Python SDK rejects them). +- **Filters** (AND-combined, server-side): `kinds` = commit | identity | account | + sync; `dids` = repo list; `collections` = NSIDs or `prefix.*` patterns. + **`collections` constrains commit events only** — identity/account/sync events + always arrive (they signal deactivation/deletion). +- **Cursor**: track the last event's seq; pass it to resume across restarts. + Reconnects dedupe server replays (no gaps/dupes in a live session). Cursors are + instance-local, not portable between providers. +- **Compression**: dict-zstd by default (~60% bandwidth cut), auto-negotiated; + best-effort, falls back to uncompressed if the dictionary can't be fetched. +- **Verifiability**: jetstream carries NO repo signatures/MST proofs — data cannot + be cryptographically verified. Use the firehose when verification matters. + +## Public providers (status as of 2026-08) + +### vāyumaṇḍala network — firehose.stream (endpoint: `wss:///tap`) + +| Instance | Relay fed by | Replay window | +|---|---|---| +| sfo.firehose.stream | northamerica.firehose.network | 72h | +| nyc.firehose.stream | northamerica.firehose.network | 72h | +| london.firehose.stream | europe.firehose.network | (72h class) | +| frankfurt.firehose.stream | europe.firehose.network | (72h class) | +| chennai.firehose.stream | asia.firehose.network | 24h | +| jet.firehose.stream | bsky.network (Bluesky mirror) | 24h | + +### Other hosts +- **jetstream.fire.hose.cam**, **jetstream.waow.tech** — v2-capable community hosts + ("Welcome to Jetstream" landing pages). +- **jetstream1/2.us-east/west.bsky.network** — Bluesky's v1-era hosts; legacy + protocol, avoid with modern clients. +- **slingshot.firehose.stream** — ATProto **edge cache**: caches records from + Jetstream for fast record/identity lookups (one lookup page, no auth). +- **spacedust.firehose.stream** — **notifications firehose**: syncs with Jetstream; + watch any handle/DID/AT-URI's notifications (instant mode skips the ~21s buffer). + +## Using it (Python — full details in the atproto-python skill) + +```python +from atproto import JetstreamClient, models + +client = JetstreamClient(params={ + 'collections': [models.ids.AppBskyFeedPost], + 'kinds': ['commit'], +}) +def on_message(event): + if event.operation == 'create': + print(event.seq, event.record.text) # already a model; DotDict fallback +client.start(on_message) +``` + +- Read `client.cursor` after each event and persist it (`params={'cursor': saved}`) + to resume across restarts. +- `client.compressed` tells you what the current connection negotiated; + `JetstreamClient(compress=False)` disables dict-zstd. +- To hit a specific provider: pass its `base_uri` (e.g. `wss://sfo.firehose.stream`). + +## Archive replay (Jetstream keeps network history) + +- `JetstreamClient(api_key=...)` enables `snapshot(after_seq=0)` (sealed archive, + then stop) and `replay(after_seq=0)` (sweep archive then continue live tail). +- The `api_key` comes from bsky.network/account — it is NOT an ATProto credential + (PDS session tokens and getServiceAuth tokens are rejected). +- **Metered in bytes downloaded, not requests.** The whole network is ~1.85 TB. + `dids` filters prune hard (per-DID bloom filters); a popular `collections` + filter prunes almost nothing — resume from a stored cursor rather than sweeping + from seq 0 to follow a busy collection. Watch `client.bytes_downloaded`. + +## Pitfalls + +- v1 hosts are dead for modern clients — you'll get protocol errors, not data. +- Jetstream events are unverifiable; never trust them for security-critical + decisions without cross-checking the firehose or PDS. +- Cursors don't transfer across providers or versions — store with the provider + context. +- The archive is a bytes-metered money pit if you filter poorly; plan sweeps. +- Community hosts (fire.hose.cam, waow.tech) may have limited replay windows or + uptime — prefer the vāyumaṇḍala network or bsky mirror for production. + +## Verify + +- `wscat -c wss://sfo.firehose.stream/tap` (or any ws client) receives JSON frames + within seconds, no auth. +- `curl -s https://slingshot.firehose.stream` — edge cache lookup works. +- `curl -s https://spacedust.firehose.stream` — notification watcher page loads. +- Python: `JetstreamClient().start(print)` prints parsed events immediately. + +## Related skills + +- `atproto-ecosystem-map` — where jetstream sits in the network; provider registry. +- `atproto-relays-firehose` — the verifiable binary stream jetstream is built on. +- `atproto-python` — JetstreamClient/AsyncJetstreamClient API in depth. diff --git a/skills/software-development/atproto-pds-ops/SKILL.md b/skills/software-development/atproto-pds-ops/SKILL.md new file mode 100644 index 0000000..44338d4 --- /dev/null +++ b/skills/software-development/atproto-pds-ops/SKILL.md @@ -0,0 +1,134 @@ +--- +name: atproto-pds-ops +description: "Use for ATProto PDS provisioning, migration, debugging." +version: 1.0.0 +author: Hermes Agent +license: MIT +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [ATProto, PDS, Hosting, Migration, Debugging, Operations] +--- + +# ATProto PDS Operations + +Operating, migrating, and debugging AT Protocol Personal Data Servers. Distilled from +the AT://links and DEEP AT://magic Semble collections (2026-08-17). PDS = the server +that hosts accounts: their repo, records, and blobs. + +## When to Use + +- You are provisioning or choosing a PDS (self-host vs managed). +- You need to migrate an account between PDSes — especially against a hostile or + offline provider. +- You need to check why an account/PDS looks broken (debug, health, relay crawl). + +## Core concepts + +- A PDS hosts: the user's **repo** (MST of records), **blobs** (binary media, + content-addressed by CID), and serves `com.atproto.*` XRPC endpoints. +- Identity is separate from hosting: handle -> DID (PLC/Web) -> DID document -> + `#atproto_pds` service endpoint. Migrating PDSes = updating that endpoint (PLC + ops or did:web file), then moving data. +- **Relays crawl PDSes**; a PDS missing from a relay just needs a crawl request + (relay status pages and debuggers offer this). +- **Credible exit**: the protocol is designed so you can move PDS providers at any + time, even against a hostile provider — but you need the right keys and backups. + +## Public services (status as of 2026-08) + +### Index / discovery +- **pds.directory** — index of ~6,175 public PDSes: hostname, PDS version, user + count, open registration, relay status, "banned somewhere". +- **firehose.directory** — relay index (which relays crawl which PDSes). + +### Health & debugging +- **check.cirrus.earth** (PDS Check) — anonymous read-only checks (~60): identity + resolution, repo reads, sync, blobs, firehose framing, OAuth discovery; can + download the full repo CAR. Optional sign-in write tests (createRecord, + applyWrites, uploadBlob, deleteRecord roundtrips with disposable records) and an + **OAuth conformance** walk (PAR + DPoP + PKCE + token exchange + refresh + + revoke) that surfaces spec deviations and security edges. +- **debug.hose.cam** — PDS & account debugger: server online/version/open + registration, accounts, relay host status + request crawl, DID repo rev/size, + relay repo status (active/current/behind/ahead), labels, PLC PDS history. + +### Hosting +- **protobase.at** — managed multi-PDS hosting: provision PDSes in minutes + (containers, TLS, identity), per-PDS quota views, private beta waitlist. Runs + many isolated PDSes from one pane. +- **pds.tokyonight.city** (Tranquil PDS) — small private instance pattern + (~26 users, friends-and-family). +- **altq.net** — self-hosted PDS running the official code + (github.com/bluesky-social/pds; admin @fry69.dev). +- **atcr.io** — distributed container registry on ATProto (docker push/pull with + your handle/DID as the registry path) — infrastructure flavor, not account data. + +### Migration & recovery +- **atpairport.com** (Airport) — PDS migration assistance ("terminal for ATProto + account actions"); provides a **PLC key for account recovery** when the current + PDS is hostile or offline; data backups (baggage claim) planned. +- **da.vidbuchanan.co.uk — "Adversarial ATProto PDS Migration"** (July 2025) — the + definitive write-up: non-adversarial migration is documented; adversarial + migration (provider hostile/offline, e.g. seagull-carried Pi) needs the PLC + rotation key, DID document control, and repo/blob recovery paths. + +### Supporting services +- **Porxie** (porxie project, blooym.dev) — "a correct, fast ATProto blob proxy"; + the reference for serving blobs without breaking CID addressing/range semantics. + Naive proxies get blob semantics wrong — use/test a real implementation. +- **comail / atmospheremail** — cooperative email: send via shared SMTP relay + (`smtp.atmos.email:587`, STARTTLS), identity = your DID, a labeling service + verifies DNS and publishes signed `verified-mail-operator` labels; dual DKIM + (your domain + pool). +- **wisp.place** — static hosting on your PDS (see atproto-site-deployment). +- **airglow.run** — automations: listen to ATProto events by lexicon, fire + webhooks / create records / post on condition. + +## Migration checklist (from the adversarial-migration write-up + Airport) + +1. Confirm you control the **PLC rotation key** (did:plc) or the **did:web + signing key** — without it, a hostile provider can block your move. +2. Grab the repo: `com.atproto.sync.getRepo` -> `{handle}.car` (can be hundreds + of MB — debug/check tools warn about size). +3. Back up **blobs**: enumerate `com.atproto.sync.listBlobs`, fetch each via + `com.atproto.sync.getBlob`, keep the CID manifest. +4. Provision the new PDS; import repo; update the DID document's PDS endpoint + (PLC op or did.json). +5. Request a crawl from your relay(s) and verify via debug.hose.cam relay repo + status (active/current). +6. Decommission old PDS only after the new one is live and crawled. + +## Pitfalls + +- **PDS version drift**: pds.directory shows versions in the wild (0.4.x, 0.5.x); + features and bug fixes differ — pin and upgrade deliberately. +- **Open registration**: a misconfigured PDS that's open to registration gets + spammed accounts and can be flagged/banned — check "banned somewhere" on + pds.directory. +- **Blob proxies are subtle**: CIDs, range requests, and content negotiation break + in naive implementations — use Porxie-grade tooling. +- **Migration without keys = account loss**: if the provider is hostile and you + lack the PLC rotation key, the account may be unrecoverable (this is what + atpairport's PLC-key service exists for). +- **Relay crawl lag**: after migration or provisioning, accounts may be missing + from appviews until the relay re-crawls — request it explicitly. +- Private hosts (pds.club etc.) may resolve oddly or vanish; re-check + pds.directory for current state. + +## Verify + +- `curl -s https://pds.directory` — live PDS index. +- Run a target PDS/handle through check.cirrus.earth read-only checks. +- `curl -s https://debug.hose.cam` — debugger loads; paste a handle to see + relay repo status and PLC PDS history. +- Migration drill: `com.atproto.sync.getRepo` on your own account downloads a + valid .car. + +## Related skills + +- `atproto-ecosystem-map` — PDS role in the network + hosting registry. +- `atproto-relays-firehose` / `atproto-jetstream-providers` — the relay/crawl side. +- `atproto-site-deployment` — wisp.place static hosting on PDS. +- `atproto-blob-lifecycle` — blob backup/orphan tooling. +- `small-business-atproto-migration` — domain/DID/PDS migration for businesses.