diff --git a/README.md b/README.md index 0f711a7..916ee53 100644 --- a/README.md +++ b/README.md @@ -1,51 +1,57 @@ # Contrail -> **Work in progress!** Pre-alpha, expect breaking changes. +> **Pre-alpha.** Expect breaking changes. -Contrail is a library (and a small set of sibling packages) for building AT Protocol appviews. Define collections, get automatic Jetstream ingestion, PDS backfill, typed XRPC endpoints, permissioned spaces for private records, group-controlled communities, and a client-side reactive sync layer. +A library for building AT Protocol appviews. Declare your collections, get automatic Jetstream ingestion, typed XRPC endpoints, optional permissioned spaces for private records, group-controlled communities, and a client-side reactive sync layer. Runs on Cloudflare Workers + D1, Node.js + PostgreSQL, or SvelteKit. -## Packages - -| Package | Description | -|---|---| -| [`@atmo-dev/contrail`](./packages/contrail) | Core library — indexing, XRPC server, spaces, communities, realtime publishing. | -| [`@atmo-dev/contrail-sync`](./packages/sync) | Client-side reactive watch-store over `watchRecords`. SSE + WebSocket, IndexedDB cache. | -| [`@atmo-dev/contrail-lexicons`](./packages/lexicons) | Lexicon codegen from a Contrail config + CLI (`contrail-lex`) wrapping `@atcute/lex-cli`. | +## Install -## Apps (reference deployments — `workspace:*`-linked) +```bash +pnpm add @atmo-dev/contrail +``` -| App | Description | -|---|---| -| [`rsvp-atmo`](./apps/rsvp-atmo) | Cloudflare Workers + D1 indexer for `community.lexicon.calendar.*`. | -| [`group-chat`](./apps/group-chat) | Full-featured SvelteKit + Workers group chat using spaces, communities, and realtime. | -| [`postgres`](./apps/postgres) | Node + PostgreSQL minimal indexer. | -| [`cloudflare-workers`](./apps/cloudflare-workers) | Minimal Worker example. | -| [`sveltekit-cloudflare-workers`](./apps/sveltekit-cloudflare-workers) | SvelteKit Statusphere-style example. | +## Minimal example + +```ts +import { Contrail } from "@atmo-dev/contrail"; + +const contrail = new Contrail({ + namespace: "com.example", + db, // D1, node:sqlite, or @atmo-dev/contrail/postgres + collections: { + event: { + collection: "community.lexicon.calendar.event", + queryable: { startsAt: { type: "range" } }, + searchable: ["name", "description"], + }, + }, +}); + +await contrail.init(); +await contrail.ingest(); // pulls from Jetstream +``` -## Setup +Mount the XRPC handler in any fetch-style framework: -```bash -pnpm install -pnpm build -pnpm test +```ts +import { createHandler } from "@atmo-dev/contrail/server"; +export default { fetch: createHandler(contrail) }; ``` -Per-package commands run through Turbo: +## Docs -```bash -pnpm build # turbo run build -pnpm typecheck # turbo run typecheck -pnpm test # turbo run test -pnpm --filter @atmo-dev/contrail build -pnpm --filter rsvp-atmo dev -``` +- [Indexing](./docs/indexing.md) — the core: collections, queries, ingestion, adapters +- [Spaces](./docs/spaces.md) — permissioned records stored by the appview +- [Communities](./docs/communities.md) — group-controlled atproto DIDs +- [Sync](./docs/sync.md) — reactive client-side store over `watchRecords` +- [Lexicons](./docs/lexicons.md) — `contrail-lex` CLI and codegen -## Releasing +## Packages -Changesets drive versioning. `@atmo-dev/contrail` and `@atmo-dev/contrail-sync` are `linked` so their versions stay aligned (they share the realtime wire protocol). +| Package | | +|---|---| +| `@atmo-dev/contrail` | Core library — indexing, XRPC server, spaces, communities, realtime | +| `@atmo-dev/contrail-sync` | Client-side reactive watch-store with optional IndexedDB cache | +| `@atmo-dev/contrail-lexicons` | Codegen + `contrail-lex` CLI | -```bash -pnpm changeset # add a changeset -pnpm changeset version # bump versions -pnpm release # build + publish -``` +Working in this repo? See [development.md](https://github.com/flo-bit/contrail/blob/main/development.md) for the monorepo layout and commands. diff --git a/development.md b/development.md new file mode 100644 index 0000000..0f711a7 --- /dev/null +++ b/development.md @@ -0,0 +1,51 @@ +# Contrail + +> **Work in progress!** Pre-alpha, expect breaking changes. + +Contrail is a library (and a small set of sibling packages) for building AT Protocol appviews. Define collections, get automatic Jetstream ingestion, PDS backfill, typed XRPC endpoints, permissioned spaces for private records, group-controlled communities, and a client-side reactive sync layer. + +## Packages + +| Package | Description | +|---|---| +| [`@atmo-dev/contrail`](./packages/contrail) | Core library — indexing, XRPC server, spaces, communities, realtime publishing. | +| [`@atmo-dev/contrail-sync`](./packages/sync) | Client-side reactive watch-store over `watchRecords`. SSE + WebSocket, IndexedDB cache. | +| [`@atmo-dev/contrail-lexicons`](./packages/lexicons) | Lexicon codegen from a Contrail config + CLI (`contrail-lex`) wrapping `@atcute/lex-cli`. | + +## Apps (reference deployments — `workspace:*`-linked) + +| App | Description | +|---|---| +| [`rsvp-atmo`](./apps/rsvp-atmo) | Cloudflare Workers + D1 indexer for `community.lexicon.calendar.*`. | +| [`group-chat`](./apps/group-chat) | Full-featured SvelteKit + Workers group chat using spaces, communities, and realtime. | +| [`postgres`](./apps/postgres) | Node + PostgreSQL minimal indexer. | +| [`cloudflare-workers`](./apps/cloudflare-workers) | Minimal Worker example. | +| [`sveltekit-cloudflare-workers`](./apps/sveltekit-cloudflare-workers) | SvelteKit Statusphere-style example. | + +## Setup + +```bash +pnpm install +pnpm build +pnpm test +``` + +Per-package commands run through Turbo: + +```bash +pnpm build # turbo run build +pnpm typecheck # turbo run typecheck +pnpm test # turbo run test +pnpm --filter @atmo-dev/contrail build +pnpm --filter rsvp-atmo dev +``` + +## Releasing + +Changesets drive versioning. `@atmo-dev/contrail` and `@atmo-dev/contrail-sync` are `linked` so their versions stay aligned (they share the realtime wire protocol). + +```bash +pnpm changeset # add a changeset +pnpm changeset version # bump versions +pnpm release # build + publish +``` diff --git a/docs/communities.md b/docs/communities.md new file mode 100644 index 0000000..af2c1db --- /dev/null +++ b/docs/communities.md @@ -0,0 +1,54 @@ +# Communities + +Group-controlled atproto DIDs. A community is a DID whose signing/rotation keys are held by the appview on behalf of multiple members, with tiered access levels. Built on top of [spaces](./spaces.md). + +## When to use this + +When you want atproto records published under a *shared* identity — a team, a project, a channel — not a single user. Think: a group's published calendar events, a community's published posts. + +## Two modes + +- **Minted** — contrail creates a fresh `did:plc` for the community, holds the signing + rotation keys, publishes from it. +- **Adopted** — contrail takes over an existing DID whose rotation keys were handed over by the owner. + +Either way, the result is the same: a DID that multiple members can act through, gated by access levels. + +## Access levels + +Each member has a level (ranked). Levels map to write permissions. Owners can grant/revoke levels. Two reserved levels exist: `owner` and `member`. Your deployment defines the rest: + +```ts +community: { + masterKey: ENV.COMMUNITY_MASTER_KEY, // 32-byte encryption key for stored credentials + serviceDid: "did:web:example.com", + levels: ["admin", "moderator"], // ranked, highest-first +} +``` + +Stored credentials (app passwords for adopted communities, signing keys for minted) are envelope-encrypted with `masterKey`. Never ship the placeholder. + +## How it composes with spaces + +A community *owns* spaces. Members of the community get access to community-owned spaces based on their level. Grant access per-space per-level: + +``` +community.space.grant { spaceUri, level: "admin", perms: "write" } +``` + +The spaces layer stays ignorant of access levels — it just sees "this DID is a member with these perms." The community layer projects member × level → space perms. + +## XRPCs + +- `com.example.community.mint | adopt | list | delete` +- `com.example.community.invite.create | redeem | revoke | list` +- `com.example.community.setAccessLevel | revoke | listMembers` +- `com.example.community.space.create | grant | revoke | ...` — community-owned spaces +- `com.example.community.putRecord | deleteRecord` — publish records as the community DID + +## What's not here + +- No per-record per-level ACLs. Model as spaces. +- No auto-rotation on key compromise yet. +- Adoption is irreversible without manual key surrender back to the owner. + +The design follows zicklag's [Arbiter design sketch](https://zicklag.leaflet.pub/3mjrvb5pul224) for group management on atproto. The post is an early design note; our implementation will track it as the spec firms up. diff --git a/docs/indexing.md b/docs/indexing.md new file mode 100644 index 0000000..244899a --- /dev/null +++ b/docs/indexing.md @@ -0,0 +1,96 @@ +# Indexing + +Contrail's core job: mirror atproto records into your DB and expose them via XRPC. You describe what to index with a config object; everything else is automatic. + +## Collection shape + +```ts +collections: { + event: { + collection: "community.lexicon.calendar.event", // full NSID + queryable: { + mode: {}, // ?mode=online + startsAt: { type: "range" }, // ?startsAtMin=...&startsAtMax=... + }, + searchable: ["name", "description"], // FTS5 / tsvector + relations: { + rsvps: { + collection: "rsvp", + groupBy: "status", + groups: { going: "community.lexicon.calendar.rsvp#going" }, + }, + }, + references: { + event: { collection: "event", field: "subject.uri" }, + }, + }, +} +``` + +- **queryable** — string equality or range, exposed as query params. +- **searchable** — FTS5 on D1/Postgres. Not available on `node:sqlite`. +- **relations** — materialized many-to-one counts (`rsvpsGoingCount`). +- **references** — forward lookups. Hydrate inline with `?hydrateEvent=true`. + +## Ingestion + +Three ways records land in the DB: + +```ts +await contrail.ingest(); // one Jetstream cycle, then stops +await contrail.runPersistent(); // long-lived connection, auto-reconnect +await contrail.notify(uri); // immediate PDS fetch for one record +``` + +Call `notify()` after your app writes to a PDS and needs the change reflected now. Jetstream catches the same event later; the duplicate is detected by CID. + +## Discovery + backfill + +```ts +await contrail.discover(); // find users from relays +await contrail.backfill({ concurrency: 100 }); // pull their history +await contrail.sync(); // both +``` + +## Querying + +```ts +const { records, cursor } = await contrail.query("event", { + filters: { mode: "online" }, + sort: { field: "startsAt", direction: "asc" }, + limit: 20, +}); +``` + +Or via HTTP once the handler is mounted: + +``` +/xrpc/com.example.event.listRecords?mode=online&sort=startsAt&limit=20 +/xrpc/com.example.event.getRecord?uri=at://did:plc:.../... +``` + +## Adapters + +| Adapter | Use when | FTS | +|---|---|---| +| Cloudflare D1 | Workers | ✅ | +| `@atmo-dev/contrail/sqlite` | Node 22+ local dev | ❌ | +| `@atmo-dev/contrail/postgres` | Node server | ✅ | + +```ts +import { createPostgresDatabase } from "@atmo-dev/contrail/postgres"; +const db = createPostgresDatabase(pool); +``` + +## Top-level config + +| Key | Default | | +|---|---|---| +| `namespace` | — | Reverse-domain for XRPC paths | +| `profiles` | `["app.bsky.actor.profile"]` | Profile NSIDs, auto-hydrated via `?profiles=true` | +| `jetstreams` | Bluesky | Jetstream URLs | +| `relays` | Bluesky | Relay URLs for discovery | +| `notify` | off | `true` opens `notifyOfUpdate`; a string requires `Bearer` | +| `spaces` | — | See [Spaces](./spaces.md) | +| `community` | — | See [Communities](./communities.md) | +| `realtime` | — | See [Sync](./sync.md) | diff --git a/docs/lexicons.md b/docs/lexicons.md new file mode 100644 index 0000000..32fba96 --- /dev/null +++ b/docs/lexicons.md @@ -0,0 +1,91 @@ +# Lexicons + +Contrail emits lexicon JSON for every XRPC method it exposes. Your app needs those JSON files for two things: publishing to your PDS (so other apps can discover them) and generating TypeScript types for typed clients. The `contrail-lex` CLI handles both. + +```bash +pnpm add -D @atmo-dev/contrail-lexicons @atcute/lex-cli +``` + +`@atcute/lex-cli` is a peer dep — you pin the version. + +## CLI + +```bash +contrail-lex generate # emit lexicon JSON from your Contrail config +contrail-lex pull # wraps `lex-cli pull` (fetches external lexicons) +contrail-lex types # wraps `lex-cli generate` (JSON → TS types) +contrail-lex all # generate → pull → generate → pull → types +contrail-lex all --no-types # skip the type step +``` + +The CLI auto-detects your config at `contrail.config.ts`, `app/config.ts`, or `src/lib/contrail/config.ts`. Override with `--config `. + +## What lands where + +Running `contrail-lex all` against a config with `namespace: "com.example"` writes: + +``` +lexicons-generated/ # JSON lexicons emitted from your config +lexicons-pulled/ # external NSIDs fetched by lex-cli pull +lex.config.js # regenerated each run (add to .gitignore) +src/lexicon-types/ # TS types emitted by lex-cli generate +``` + +`lexicons-generated/` and `lexicons-pulled/` should be **committed** — that way CI doesn't need network access and consumers of your repo can pull the JSON as a lexicon source. `lex.config.js` and `src/lexicon-types/` are generated on demand and safe to gitignore. + +## Publishing to a PDS + +Once committed, publish the lexicons to an atproto account so other apps can resolve them: + +```ts +import { publishLexicons } from "@atmo-dev/contrail-lexicons"; + +await publishLexicons({ + generatedDir: "lexicons-generated", + identifier: process.env.LEXICON_ACCOUNT_IDENTIFIER, + password: process.env.LEXICON_ACCOUNT_PASSWORD, +}); +``` + +Writes each lexicon as a `com.atproto.lexicon.schema` record under `at:///com.atproto.lexicon.schema/`. You'll also need DNS TXT records — publishing prints them for you. + +## Programmatic API + +```ts +import { generateLexicons, extractXrpcMethods } from "@atmo-dev/contrail-lexicons"; + +const generated = generateLexicons({ + config, + rootDir: process.cwd(), + outputDir: "lexicons-generated", +}); + +const methods = extractXrpcMethods(generated); // NSIDs of every query + procedure +``` + +Handy for emitting the method list for your OAuth permission set. + +## Consuming a third-party contrail instance + +Someone else runs a contrail deployment; you want typed client access to their XRPCs. Point `lex-cli pull` at their git repo: + +```js +// lex.config.js +import { defineLexiconConfig } from "@atcute/lex-cli"; + +export default defineLexiconConfig({ + outdir: "src/lexicon-types/", + imports: ["@atcute/atproto"], + files: ["lexicons/**/*.json"], + pull: { + outdir: "lexicons/", + sources: [{ + type: "git", + remote: "https://github.com/them/their-contrail.git", + pattern: ["lexicons-generated/**/*.json"], + }], + }, +}); +``` + +Then `npx lex-cli pull && npx lex-cli generate`, and their endpoints are typed in your client. diff --git a/docs/spaces.md b/docs/spaces.md new file mode 100644 index 0000000..8bcb97f --- /dev/null +++ b/docs/spaces.md @@ -0,0 +1,70 @@ +# Spaces + +Auth-gated store for records that can't live on public PDSes — private events, invite-only groups, members-only chat. Opt-in; zero cost if you don't enable it. + +## Mental model + +> A **space** is a bag of records with one lock. The **member list** says who has the key. + +- One owner (DID), one type (NSID), one key. Identified by `at:////`. +- Members have `read` or `write`. Owner is implicit `write`. +- Optional **app policy** gates which OAuth clients can act in the space. + +Every permission boundary is its own space. No nested ACLs. Richer roles = more spaces or app-layer checks. + +## Enable + +```ts +import type { ContrailConfig } from "@atmo-dev/contrail"; + +const config: ContrailConfig = { + namespace: "com.example", + collections: { /* ... */ }, + spaces: { + type: "com.example.event.space", + serviceDid: "did:web:example.com", + }, +}; +``` + +Each collection gets a parallel `spaces_records_` table. Opt out per-collection: + +```ts +public_only: { collection: "com.example.public", allowInSpaces: false } +``` + +## Auth + +atproto service-auth JWTs via `@atcute/xrpc-server`. Middleware validates signature, `aud`, and `lxm` before the handler runs. Apps acting in a space mint a JWT against the user's PDS with `Atproto-Proxy: did:web:example.com#com_example_space`, forward to your service, it verifies and executes. + +**Note:** Use the plain DID (no `#fragment`) as `serviceDid` — the fragment form only belongs in your DID doc's service entry. + +## Unified `listRecords` + +| Call | Returns | +|---|---| +| no auth, no `spaceUri` | public only | +| `?spaceUri=…` + JWT | one space (ACL-gated) | +| JWT, no `spaceUri` | public **unioned** with every space the caller is a member of | + +Filters, sorts, hydration, and references work across all three. Records from a space carry a `space: ` field. + +## Invites + +Stored as hashed tokens. Redemption is a single atomic UPDATE that checks `!revoked && !expired && !exhausted`. Generate, hand out the plaintext once, verify later. + +## XRPCs + +- `com.example.space.create | get | list | delete` +- `com.example.space.putRecord | deleteRecord | listRecords | getRecord` +- `com.example.space.invite.create | redeem | revoke | list` +- `com.example.space.listMembers | removeMember` + +## What's not here + +- No E2EE (data is operator-readable). +- No FTS on `?spaceUri=…` yet. +- No per-space sharding — one DB, one operator. +- Not a long-term replacement for real atproto permissioned repos. + +The design follows Daniel Holmgren's [permissioned data rough spec](https://dholms.leaflet.pub/3mhj6bcqats2o). The goal is that when real atproto permissioned repos ship, migration is mostly data movement — the API your app speaks doesn't change. diff --git a/docs/sync.md b/docs/sync.md new file mode 100644 index 0000000..e2e29e1 --- /dev/null +++ b/docs/sync.md @@ -0,0 +1,93 @@ +# Sync + +Client-side reactive store over contrail's `watchRecords` endpoints. Subscribes once, reconciles forever, ships with optimistic updates and an optional IndexedDB cache. + +Lives in its own package: + +```bash +pnpm add @atmo-dev/contrail-sync +``` + +## Basic use + +```ts +import { createWatchStore } from "@atmo-dev/contrail-sync"; + +const store = createWatchStore({ + url: "/xrpc/com.example.message.watchRecords?roomUri=at://...", + transport: "sse", // or "ws" +}); + +store.subscribe(({ records, status }) => { /* re-render */ }); +store.start(); +``` + +Framework-agnostic — wrap it in Svelte `$state`, React `useSyncExternalStore`, Vue `ref`, whatever. + +## Transports + +- **SSE** (default) — one HTTP request, simplest. Works everywhere. +- **WS** — a two-step handshake: HTTP GET returns a snapshot + watch-scoped ticket, then you upgrade to WS. On Cloudflare, the WS terminates on a Durable Object that hibernates idle connections. Same event stream either way. + +## Authenticated watches + +Pass `mintTicket` for any non-public endpoint. One-shot string for SSR-minted tickets, function for fresh tickets per reconnect: + +```ts +mintTicket: async () => (await fetch("/api/ticket")).then((r) => r.text()), +``` + +Tickets are minted server-side via `com.example.realtime.ticket` (or any app-specific route). + +## Optimistic updates + +```ts +store.addOptimistic({ rkey, did, record: { text: "hi" } }); +// later, on mutation failure: +store.markFailed(rkey, err); +// or explicit rollback: +store.removeOptimistic(rkey); +``` + +When a real record with the same `rkey` arrives via the stream, the optimistic entry is dropped automatically. + +## IndexedDB cache + +Instant first paint from last session's records: + +```ts +import { createIndexedDBCache } from "@atmo-dev/contrail-sync/cache-idb"; + +createWatchStore({ + url, + cache: createIndexedDBCache(), + cacheMaxRecords: 200, +}); +``` + +Cached records show immediately; the live snapshot reconciles when the connection opens. + +## Server-side config + +Enable `watchRecords` emission in your Contrail config: + +```ts +realtime: { + ticketSecret: ENV.REALTIME_TICKET_SECRET, // 32 bytes + pubsub: new DurableObjectPubSub(env.PUBSUB), // or in-memory for dev +} +``` + +See [indexing.md](./indexing.md) for the full config surface. + +## Lifecycle + +``` +idle → connecting → snapshot → live + ↓ (disconnect) + reconnecting → snapshot → live + ↓ (stop) + closed +``` + +Stale records stay visible across reconnects until the fresh snapshot arrives, at which point anything the server didn't re-send is evicted. Survives offline periods cleanly.