From 8513b3e47849613c9344c49ea2238eddfde0e6c6 Mon Sep 17 00:00:00 2001 From: Florian <45694132+flo-bit@users.noreply.github.com> Date: Thu, 30 Apr 2026 15:39:49 +0200 Subject: [PATCH] small fixes, update starter --- .changeset/cyan-geckos-act.md | 5 + README.md | 13 +- apps/atproto-starter/CONTRAIL.md | 221 +++++++++++++++++- apps/atproto-starter/README.md | 47 +++- apps/atproto-starter/eslint.config.js | 3 +- apps/atproto-starter/package.json | 1 + apps/atproto-starter/src/app.css | 8 +- apps/atproto-starter/src/lib/atproto/index.ts | 2 +- .../src/lib/atproto/methods.ts | 8 +- .../src/lib/atproto/server/profile.ts | 2 +- .../src/lib/atproto/ui/LoginModal.svelte | 144 +++++++++++- .../oauth-client-metadata.json/+server.ts | 16 +- .../atproto-starter/src/routes/+layout.svelte | 20 +- apps/atproto-starter/src/routes/+page.svelte | 17 +- apps/atproto-starter/tsconfig.json | 5 +- docs/01-indexing.md | 9 +- docs/04-feeds.md | 109 +++++++++ docs/{04-auth.md => 05-auth.md} | 2 +- docs/{05-spaces.md => 06-spaces.md} | 4 +- docs/{06-communities.md => 07-communities.md} | 2 +- docs/{07-sync.md => 08-sync.md} | 0 docs/{08-labels.md => 09-labels.md} | 0 docs/frameworks/sveltekit-cloudflare.md | 7 +- 23 files changed, 576 insertions(+), 69 deletions(-) create mode 100644 .changeset/cyan-geckos-act.md create mode 100644 docs/04-feeds.md rename docs/{04-auth.md => 05-auth.md} (99%) rename docs/{05-spaces.md => 06-spaces.md} (96%) rename docs/{06-communities.md => 07-communities.md} (97%) rename docs/{07-sync.md => 08-sync.md} (100%) rename docs/{08-labels.md => 09-labels.md} (100%) diff --git a/.changeset/cyan-geckos-act.md b/.changeset/cyan-geckos-act.md new file mode 100644 index 0000000..ac46110 --- /dev/null +++ b/.changeset/cyan-geckos-act.md @@ -0,0 +1,5 @@ +--- +"@atmo-dev/contrail": patch +--- + +small fixes diff --git a/README.md b/README.md index efd97d0..1f88ec9 100644 --- a/README.md +++ b/README.md @@ -74,7 +74,7 @@ the worker keeps itself fresh from now on via the cron. hit: GET https://.workers.dev/xrpc/com.example.event.listRecords?startsAtMin=2026-01-01&limit=10 ``` -returns every `community.lexicon.calendar.event` record published anywhere on atproto that matches, as JSON. that's it — no PDS setup, no lexicon publishing, no relay configuration. everything scales from there: add filters, add full-text search, add more collections, turn on [spaces](https://github.com/flo-bit/contrail/blob/main/docs/05-spaces.md) for private records, mount the handler in sveltekit instead, swap the adapter for postgres. +returns every `community.lexicon.calendar.event` record published anywhere on atproto that matches, as JSON. that's it — no PDS setup, no lexicon publishing, no relay configuration. everything scales from there: add filters, add full-text search, add more collections, turn on [spaces](https://github.com/flo-bit/contrail/blob/main/docs/06-spaces.md) for private records, mount the handler in sveltekit instead, swap the adapter for postgres. **not using workers?** same library, different `db`. see [adapters](https://github.com/flo-bit/contrail/blob/main/docs/01-indexing.md#adapters) for node:sqlite and postgres. @@ -83,11 +83,12 @@ returns every `community.lexicon.calendar.event` record published anywhere on at - [Indexing](https://github.com/flo-bit/contrail/blob/main/docs/01-indexing.md) — the core: collections, ingestion, adapters - [Querying](https://github.com/flo-bit/contrail/blob/main/docs/02-querying.md) — filters, sorts, hydration, search, pagination - [Lexicons](https://github.com/flo-bit/contrail/blob/main/docs/03-lexicons.md) — `contrail-lex` CLI, codegen, publishing -- [Auth](https://github.com/flo-bit/contrail/blob/main/docs/04-auth.md) — service-auth JWTs, invite tokens, watch tickets, OAuth permission sets -- [Spaces](https://github.com/flo-bit/contrail/blob/main/docs/05-spaces.md) — permissioned records stored by the appview -- [Communities](https://github.com/flo-bit/contrail/blob/main/docs/06-communities.md) — group-controlled atproto DIDs -- [Sync](https://github.com/flo-bit/contrail/blob/main/docs/07-sync.md) — reactive client-side store over `watchRecords` -- [Labels](https://github.com/flo-bit/contrail/blob/main/docs/08-labels.md) — atproto-native moderation hydration from external labelers +- [Feeds](https://github.com/flo-bit/contrail/blob/main/docs/04-feeds.md) — personalized timelines via follow + target collections +- [Auth](https://github.com/flo-bit/contrail/blob/main/docs/05-auth.md) — service-auth JWTs, invite tokens, watch tickets, OAuth permission sets +- [Spaces](https://github.com/flo-bit/contrail/blob/main/docs/06-spaces.md) — permissioned records stored by the appview +- [Communities](https://github.com/flo-bit/contrail/blob/main/docs/07-communities.md) — group-controlled atproto DIDs +- [Sync](https://github.com/flo-bit/contrail/blob/main/docs/08-sync.md) — reactive client-side store over `watchRecords` +- [Labels](https://github.com/flo-bit/contrail/blob/main/docs/09-labels.md) — atproto-native moderation hydration from external labelers - Frameworks: [SvelteKit + Cloudflare](https://github.com/flo-bit/contrail/blob/main/docs/frameworks/sveltekit-cloudflare.md) ## Packages diff --git a/apps/atproto-starter/CONTRAIL.md b/apps/atproto-starter/CONTRAIL.md index 833e705..e8810c0 100644 --- a/apps/atproto-starter/CONTRAIL.md +++ b/apps/atproto-starter/CONTRAIL.md @@ -30,9 +30,9 @@ Where: If the user gives a vague spec ("show me X"), the answer is almost always a `listRecords` call with a filter. Reach for `getRecord` only when you genuinely need one record by URI — listing with a filter that narrows to one is usually fine and keeps the codepath uniform. -## Filtering and hydration come from `contrail.config.ts` +## Filtering — declared in `contrail.config.ts` -Both the available filter params *and* the hydration flags are declared in `contrail.config.ts`. If a filter doesn't exist, add it there and re-run `pnpm generate:pull` — don't filter in JS after the fact. +Available filter params are declared per-collection in `queryable`. If a filter doesn't exist, add it there and re-run `pnpm generate:pull` — don't filter in JS after the fact. ```ts // src/lib/contrail.config.ts @@ -43,12 +43,49 @@ collections: { tag: {}, // → ?tag=design createdAt: { type: 'range' } // → ?createdAtMin=...&createdAtMax=... } - // hydration: profiles, etc. — see llms-full.txt } } ``` -`profiles: true` joins each record's author profile into `res.data.profiles` — use this instead of fetching profiles in a loop. Same shape for any other hydration flags contrail exposes for the collection. +### "Filter by author" is built-in — don't add it to `queryable` + +`?actor=` (and its alias `?did=`) is a built-in filter on every `listRecords` endpoint. It resolves the actor to a DID, triggers a one-shot PDS backfill for fresh data, and filters records to that author. **Do not** add `did: {}` or `actor: {}` to `queryable` — it's redundant and may fight the built-in handler. + +```ts +// ✅ "this user's bookmarks" — works with no queryable changes +client.get('.bookmark.listRecords', { params: { actor: did } }); +``` + +`queryable` is for *record-content* filters (fields inside the record JSON), not for author lookup. + +There is **no `hydration` field** on a collection config — don't add one. Hydration is controlled by *query parameters at request time* (next section). + +## Hydration — query-time, not config-time + +To pull related data alongside records, pass it as a `params` flag on the `listRecords` call: + +```ts +const res = await client.get('..listRecords', { + params: { limit: 50, profiles: true } // ← query param +}); + +res.data.records // the records +res.data.profiles // the hydrated author profiles, keyed by did +``` + +`profiles: true` joins each record's author profile (defaults to `app.bsky.actor.profile`) into `res.data.profiles`. Use this instead of fetching profiles in a loop. + +If you need *additional* profile NSIDs beyond `app.bsky.actor.profile` (e.g. an app-specific profile type), declare them at the **top level** of `ContrailConfig`: + +```ts +export const config: ContrailConfig = { + namespace: 'app.example', + profiles: ['app.bsky.actor.profile', 'app.example.actor.profile'], + collections: { /* ... */ } +}; +``` + +That's a config-time concern (which collections count as profiles); the `profiles: true` flag at query time is what triggers hydration on a given request. ## Where to call from @@ -56,6 +93,39 @@ collections: { |---|---|---| | `+page.server.ts`, `+layout.server.ts`, `+server.ts` | `getServerClient(platform!.env.DB)` | In-process, zero HTTP, full types | | `.svelte` (client-side, on user action) | `getClient()` from `$lib/contrail/client` | Hits `/xrpc/` over fetch — same typed surface | + +### Two `getClient` exports — they're different + +There are two `getClient`s in the codebase. They do different things: + +| Import | Returns | Use when | +|---|---|---| +| `getClient()` from `$lib/contrail/client` | typed contrail client (calls local `/xrpc/`) | Any in-app data fetch from the browser. **Default to this.** | +| `getPDSClient({ did })` from `$lib/atproto/methods` | `Promise` for a specific user's PDS | Direct PDS calls (rare in app code — usually only the `methods.ts` helpers need this) | + +If autocomplete gives you `Promise` and `.get` doesn't exist on the result, you imported the wrong one. Switch to `$lib/contrail/client`. + +### "Get *this user's* records" → `actor` param, not the `did` arg of `getServerClient` + +The `did` second argument to `getServerClient(db, did)` is **only** for acting as that user when reading permissioned data (spaces, communities). Public `listRecords` calls do *not* need it — and passing it does nothing useful for filtering. + +To fetch records created by a specific user, pass `actor` as a **query param**: + +```ts +// ✅ Correct — filter records to those whose author is this DID +const client = getServerClient(platform!.env.DB); +const res = await client.get('..listRecords', { + params: { actor: did, limit: 1 } +}); + +// ❌ Wrong — `did` here means "auth as this user", not "filter by author" +const client = getServerClient(platform!.env.DB, did); +const res = await client.get('..listRecords', { + params: { limit: 1 } +}); +``` + +`actor` accepts a DID or a handle and is resolved to a DID server-side. `did` is also accepted as a synonym. Use `actor` for both "this is the user whose records I want" and "this is the actor of a feed read". | Inside `routes/api/cron/+server.ts` | Don't call client APIs — use `contrail.ingest()` directly | That's the indexer, not a reader | Default to server-side loading via `+page.server.ts` for initial page data. Use client-side only for live updates, infinite scroll, or post-action refetches. @@ -86,6 +156,50 @@ After every successful `putRecord`, the remote function calls `contrail.notify(u Even with auto-notify, `putRecord` takes ~100–300ms round-trip. For interactive flows (post-as-you-type, emoji reactions, like buttons), render the local copy immediately and let the network call settle in the background — don't await before updating state. +Concrete pattern with rollback on failure: + +```ts +import { SvelteSet } from 'svelte/reactivity'; +import { putRecord, deleteRecord, createTID } from '$lib/atproto/methods'; + +let local = $state>([]); +let pending = new SvelteSet(); // rkeys mid-flight + +async function post(text: string) { + const rkey = createTID(); + const createdAt = new Date().toISOString(); + // 1. Render immediately — UI updates this tick. + local = [{ rkey, text, createdAt }, ...local]; + pending.add(rkey); + + try { + // 2. Write in the background. + await putRecord({ + collection: '.note', + rkey, + record: { text, createdAt } + }); + pending.delete(rkey); + // Auto-notify already re-indexed; the next listRecords will see it. + } catch (e) { + // 3. Roll back on failure. + local = local.filter((r) => r.rkey !== rkey); + pending.delete(rkey); + throw e; + } +} + +// In the template, dedupe local + server records (see "List rendering" above) +// so the server copy doesn't re-render once it shows up via listRecords. +$: allRecords = Array.from( + new Map( + [...local, ...serverRecords].map((r) => [r.rkey ?? r.uri, r]) + ).values() +); +``` + +Key points: generate the `rkey` *client-side* (via `createTID()`) so the optimistic copy and the server copy share the same identity for dedup. Track in-flight rkeys in a `SvelteSet` if you want to disable retries / show a spinner. Roll back the local insert if the write throws. + ### Blobs `uploadBlob` handles the bytes-over-remote-function dance and auto-detects image dimensions for `aspectRatio`. Embed the returned blob in a record: @@ -101,6 +215,70 @@ await putRecord({ To enable blob uploads, add `scope.blob({ accept: ['image/*'] })` to `scopes` in `settings.ts` (and adjust `accept` for the mime types you allow). Without that scope, the OAuth flow won't grant blob-write permission and uploads 401. +## Following feeds — opt-in + +If the app has a "follow other users → see their stuff in a feed" pattern, contrail's `feeds` config does the fan-out for you. **Skip this section if the app doesn't have a social graph** — public `listRecords` is the right primitive for most apps. + +### Mental model + +A feed is a (follow-collection, [target-collections]) pair, named by you. Every time someone the *actor* follows posts to a target collection, contrail inserts one row into `feed_items` for that actor. Reading a feed reads back through that table, joined with the standard pipeline (filters, hydration, profiles). + +``` +feed: timeline + follow: app.bsky.graph.follow // contains { subject: did, createdAt } + targets: [app.bsky.feed.post] +``` + +### Config + +```ts +// src/lib/contrail.config.ts +export const config: ContrailConfig = { + namespace: 'app.example', + collections: { + follow: { collection: 'app.bsky.graph.follow' }, + post: { collection: 'app.bsky.feed.post', queryable: { /* ... */ } } + }, + feeds: { + timeline: { + follow: 'follow', // short name from collections, NOT the NSID + targets: ['post'], + maxItems: 500 // optional, default 200 + } + } +}; +``` + +Both the follow collection and every target collection must be declared in `collections`. Names in `feeds` are the *short names* (the keys of `collections`), not NSIDs. Config validation will throw if you reference an unknown short name. + +### Follow-record shape requirement + +The follow collection's record must have a `subject` field at the top level whose value is the followed DID. `app.bsky.graph.follow` matches this naturally (`{ subject: 'did:plc:...', createdAt }`). For a custom follow lexicon, keep the `subject` field — contrail extracts it via JSON path `$.subject` to determine "who is being followed." + +### Read + +``` +GET /xrpc/.getFeed?feed=timeline&actor=&limit=50 +``` + +Optional `&collection=` to filter to one target (defaults to the first in `targets`). Filters and hydration from the target collection's `queryable` config also work — same params as `listRecords`. + +The `actor` parameter is **whose feed** you're reading, not a filter on record creator. Feeds are always per-user; there's no anonymous feed read. + +### How fan-out works + +- **On a write to a target collection** (e.g. someone you follow posts): contrail inserts a `feed_items` row for every follower whose follow record has `subject = `. One write → N inserts. There's no max-followers cap; if a celebrity has 1M followers and posts, that's 1M inserts. +- **On a write to the follow collection** (someone follows a new user): contrail backfills the most recent 100 target records from that user into the new follower's feed. The 100 is hardcoded — separate from the per-feed `maxItems` cap. +- **On first read for a (actor, feed) pair**: contrail backfills the actor's follow records from their PDS, then populates `feed_items` from existing target records. Marked complete in `feed_backfills` so it only happens once. +- **Pruning**: the cron run trims `feed_items` per actor down to `maxItems`, keeping newest by `time_us`. + +### Gotchas + +- The 100-record-per-new-follow backfill is hardcoded in `core/router/feed.ts` — not currently tunable per feed. +- No tests exist specifically for feeds yet; the write/read paths are live but treat the integration as load-bearing-but-untested in your sanity checks. +- Following many users with a viral target collection is expensive on writes (one row per follower). For an app expecting that scale, partition feeds or rate-limit writes upstream — contrail will not back off on its own. +- Feeds live in the main DB regardless of the spaces split — no `feeds_db` binding. + ## Spaces and communities — opt-in, default to *not* using them The template ships with public records only. **Don't enable spaces or communities unless the user's data model genuinely needs them** — they add config, secrets, and a parallel set of XRPC methods (`.space.*`, `.community.*`). For 80% of atproto apps (public posts, public lists, public anything), skip this whole section. @@ -217,6 +395,41 @@ Everything is typed: When something looks off, **read the generated types in `src/lexicon-types/`** — they're the ground truth for what the API actually returns. Don't guess from the lexicon JSON. +## List rendering — always dedupe before `{#each}` + +Contrail's `listRecords` can return the same record twice during a tight indexing window (especially right after `notify`). With `{#each records as r (r.uri)}`, that throws `each_key_duplicate` at runtime. Dedupe before rendering: + +```svelte +{#each Array.from(new Map(records.map((r) => [r.uri, r])).values()) as r (r.uri)} + ... +{/each} +``` + +Same pattern for any union of sources (server data + optimistic local data + jetstream live updates) — collapse all of them through one `Map` keyed by `uri` before passing to `{#each}`. Cheaper than rendering, and saves you a runtime crash the first time the index races itself. + +## Upsert pattern — look up before write + +User clicks "subscribe" twice while the first request is in flight → two records created → duplicate sidebar entries → `each_key_duplicate`. The fix is to look up by a queryable field first, reuse the existing rkey: + +```ts +const client = getClient(); +const existing = await client.get('.subscription.listRecords', { + params: { actor: user.did, feedUrl, limit: 1 } // requires `did` + `feedUrl` queryable +}); + +if (existing.ok && existing.data.records.length > 0) { + return; // already subscribed — no-op (or update the existing record) +} + +await putRecord({ + collection: '.subscription', + rkey: createTID(), + record: { feedUrl, createdAt: new Date().toISOString() } +}); +``` + +For things that are unique-per-user-per-target (subscriptions, follows, likes), this is the right shape. Add a queryable on the "uniqueness" field so the lookup is one query, not a scan. + ## Profile data, specifically Most apps want to show "who posted this" alongside records. Two paths: diff --git a/apps/atproto-starter/README.md b/apps/atproto-starter/README.md index 24997a7..f911c10 100644 --- a/apps/atproto-starter/README.md +++ b/apps/atproto-starter/README.md @@ -45,6 +45,20 @@ pnpm dev Dev mode uses a loopback OAuth client — no public URL, no Cloudflare resources, no secrets in CF. Verify in the browser that login works against a real PDS before continuing. +### Custom env vars at runtime — use `.dev.vars`, not `.env` + +`.env` is read by Vite for *app code* at build/dev time. `platform.env.MY_VAR` (Cloudflare bindings, used by anything in `+server.ts`, `+page.server.ts`, `repo.remote.ts`, etc.) reads from **wrangler bindings**, not `.env`. In `pnpm dev` (vite + miniflare), wrangler bindings come from a gitignored `.dev.vars` file at the project root: + +``` +# .dev.vars +PODCAST_INDEX_KEY=xxxxx +PODCAST_INDEX_SECRET=yyyyy +``` + +Then `platform.env.PODCAST_INDEX_KEY` works in dev. For production, set the same names via `npx wrangler secret put PODCAST_INDEX_KEY`. + +`.dev.vars` is already covered by the starter's `.gitignore` (matches `.env.*`), but double-check before committing. + --- ## Phase 2 — Apply the user's answers @@ -108,13 +122,15 @@ Three secrets needed in production. **Never put these in `wrangler.jsonc` `vars` ```sh # 1. CLIENT_ASSERTION_KEY — OAuth client signing key (JWK JSON) -pnpm env:generate-key | npx wrangler secret put CLIENT_ASSERTION_KEY +# `--silent` is critical: without it pnpm 10 prepends its `> name@version: …` header +# to stdout and the secret value gets corrupted. Production then 500s on JSON.parse. +pnpm --silent env:generate-key | npx wrangler secret put CLIENT_ASSERTION_KEY # 2. COOKIE_SECRET — HMAC for signed session cookies -pnpm env:generate-secret | npx wrangler secret put COOKIE_SECRET +pnpm --silent env:generate-secret | npx wrangler secret put COOKIE_SECRET # 3. CRON_SECRET — bearer the scheduled handler uses to call /api/cron -pnpm env:generate-secret | npx wrangler secret put CRON_SECRET +pnpm --silent env:generate-secret | npx wrangler secret put CRON_SECRET ``` `OAUTH_PUBLIC_URL` is a `var` (already set in Phase 2), not a secret — it's the public URL. @@ -133,6 +149,8 @@ For a new lexicon under the user's namespace: 4. Add the collection to `src/lib/contrail.config.ts` (and `src/lib/atproto/settings.ts` if it's writable from the app). 5. `pnpm generate:pull` — regenerates `lex.config.js` from the contrail config, pulls referenced NSIDs, emits types into `src/lexicon-types/`. +> **Expected warnings:** `pnpm generate:pull` prints `lexicon authority not found` for any NSID under your namespace that isn't published yet. The local JSON is still used and types are still emitted — these warnings are harmless until you run `contrail-lex publish`. + --- ## Phase 6 — Deploy + backfill @@ -143,6 +161,10 @@ npx wrangler deploy pnpm backfill:remote # one-shot historical backfill from Jetstream ``` +> **Expected warnings during backfill:** wrangler may print `--env=production not configured`. Harmless — the script still runs against the deployed worker. Ignore. + +> **After deploying, hard-refresh the browser** (Cmd/Ctrl+Shift+R). Cloudflare caches `_app/immutable/*.js` chunks aggressively; if the user reports "the fix didn't take", it's almost always a stale chunk. + After deploy, the cron trigger (`*/1 * * * *` in `wrangler.jsonc`) keeps the index fresh. Sanity check: hit `https:///xrpc/..listRecords?limit=10` and confirm JSON comes back. Then load the site root and try logging in. @@ -153,11 +175,24 @@ Sanity check: hit `https:///xrpc/..listReco **Before composing any UI, invoke `Skill(skill: 'impeccable', args: 'craft')`.** This produces components shape-first using the design guidelines created by `impeccable teach` in Phase 0. It's what stops the output from defaulting to generic AI aesthetics. Run it once per significant UI surface (home page, settings page, feature flow) — not per component. -**Default to [foxui](https://flo-bit.dev/ui-kit/docs/llms.txt).** It's already a dependency (`@foxui/core`, `@foxui/social`, `@foxui/time`) and ships components specifically built for atproto apps — `AtprotoLoginModal`, `GithubCorner`, `RelativeTime`, profile/handle helpers, etc. Using foxui means the OAuth login flow, profile rendering, and social bits all "just work" without you reinventing them. Fetch the llms.txt above before composing UI. +### The starter is foxui-free by default + +`src/routes/+page.svelte`, `+layout.svelte`, and `src/lib/atproto/ui/LoginModal.svelte` are all hand-rolled with plain Tailwind v4. **Custom styling is the path of least resistance** — there's no foxui component to fight or override; just edit the markup. Tailwind v4 is set up, and `bits-ui` is also pre-installed if you need accessible headless primitives. + +### Optional: opt back into [foxui](https://flo-bit.dev/ui-kit/docs/llms.txt) + +`@foxui/core`, `@foxui/social`, `@foxui/time` are still in `package.json` deps (so no `pnpm add` needed) — foxui ships polished atproto-specific components: `AtprotoLoginModal`, `GithubCorner`, `RelativeTime`, profile/handle helpers, theme tokens. Worth using *if* the design direction from Phase 0 fits foxui's look, since they handle OAuth subtleties (loopback vs. confidential client, redirect URIs, scope handling) correctly. -If foxui's look doesn't fit the design guidelines from Phase 0, **build custom components but don't reinvent the structure.** First read the foxui source for the component you're replacing (e.g. how `AtprotoLoginModal` handles the handle-input → PDS-resolution → redirect flow). Re-create with your own styling, but keep the data flow and prop shapes the same — atproto OAuth has subtle requirements (loopback vs. confidential client, redirect URIs, scope handling) that foxui already gets right. +**To use foxui:** +1. Add these two lines to `src/app.css` (a comment in the file shows you exactly where): + ```css + @import '@foxui/core/theme.css'; + @source "../node_modules/@foxui"; + ``` + Without **both** lines, foxui components mount but render invisibly — Tailwind doesn't scan their classes. This is a frequent silent-failure mode. +2. Import + use components as normal. See [foxui's repo](https://github.com/flo-bit/foxui) for prop shapes. -Tailwind v4 is set up, so utility classes work out of the box. The `bits-ui` primitives (also already a dependency) are a good base for accessible, headless components if you're going custom. +Mixing modes (some foxui, some custom) is fine — but the moment any `@foxui/*` component is in use, both `app.css` lines must be present. **After building, optional finishing skills** — invoke when the work calls for them, not as a checklist: diff --git a/apps/atproto-starter/eslint.config.js b/apps/atproto-starter/eslint.config.js index aa5987f..97d98fe 100644 --- a/apps/atproto-starter/eslint.config.js +++ b/apps/atproto-starter/eslint.config.js @@ -27,7 +27,8 @@ export default ts.config( languageOptions: { parserOptions: { - parser: ts.parser + parser: ts.parser, + tsconfigRootDir: import.meta.dirname } } } diff --git a/apps/atproto-starter/package.json b/apps/atproto-starter/package.json index ac36343..fb280b4 100644 --- a/apps/atproto-starter/package.json +++ b/apps/atproto-starter/package.json @@ -35,6 +35,7 @@ "@atcute/tid": "^1.1.2", "@cloudflare/workers-types": "^4.20260317.1", "@eslint/compat": "^2.0.3", + "@types/node": "^22.0.0", "@eslint/js": "^10.0.1", "@sveltejs/adapter-cloudflare": "^7.2.8", "@sveltejs/kit": "^2.55.0", diff --git a/apps/atproto-starter/src/app.css b/apps/atproto-starter/src/app.css index 6289b7f..d947f65 100644 --- a/apps/atproto-starter/src/app.css +++ b/apps/atproto-starter/src/app.css @@ -4,6 +4,8 @@ @custom-variant dark (&:where(.dark, .dark *)); -@import '@foxui/core/theme.css'; - -@source "../node_modules/@foxui"; +/* If you import any @foxui/* component (Button, AtprotoLoginModal, etc.), add: + * @import '@foxui/core/theme.css'; + * @source "../node_modules/@foxui"; + * Without both lines foxui components mount but render invisibly because Tailwind + * doesn't scan their classes. The default starter is foxui-free, so they're omitted. */ diff --git a/apps/atproto-starter/src/lib/atproto/index.ts b/apps/atproto-starter/src/lib/atproto/index.ts index 8d3232a..c0cf642 100644 --- a/apps/atproto-starter/src/lib/atproto/index.ts +++ b/apps/atproto-starter/src/lib/atproto/index.ts @@ -6,7 +6,7 @@ export { actorToDid, getPDS, getDetailedProfile, - getClient, + getPDSClient, listRecords, getRecord, putRecord, diff --git a/apps/atproto-starter/src/lib/atproto/methods.ts b/apps/atproto-starter/src/lib/atproto/methods.ts index 4a84fd5..c768ab5 100644 --- a/apps/atproto-starter/src/lib/atproto/methods.ts +++ b/apps/atproto-starter/src/lib/atproto/methods.ts @@ -93,7 +93,7 @@ export async function getDetailedProfile(data?: { did?: Did; client?: Client }) /** * Creates an AT Protocol client for a user's PDS. */ -export async function getClient({ did }: { did: Did }) { +export async function getPDSClient({ did }: { did: Did }) { const pds = await getPDS(did); if (!pds) throw new Error('PDS not found'); @@ -128,7 +128,7 @@ export async function listRecords({ throw new Error('Missing did for listRecords'); } - client ??= await getClient({ did }); + client ??= await getPDSClient({ did }); const allRecords = []; @@ -177,7 +177,7 @@ export async function getRecord({ throw new Error('Missing did for getRecord'); } - client ??= await getClient({ did }); + client ??= await getPDSClient({ did }); const record = await client.get('com.atproto.repo.getRecord', { params: { @@ -288,7 +288,7 @@ export async function describeRepo({ client, did }: { client?: Client; did?: Did if (!did) { throw new Error('Error describeRepo: No did'); } - client ??= await getClient({ did }); + client ??= await getPDSClient({ did }); const repo = await client.get('com.atproto.repo.describeRepo', { params: { diff --git a/apps/atproto-starter/src/lib/atproto/server/profile.ts b/apps/atproto-starter/src/lib/atproto/server/profile.ts index 20521fc..6e51975 100644 --- a/apps/atproto-starter/src/lib/atproto/server/profile.ts +++ b/apps/atproto-starter/src/lib/atproto/server/profile.ts @@ -8,7 +8,7 @@ const PROFILE_CACHE_TTL = 60 * 60; // 1 hour * Falls back to a fresh fetch if the cache KV doesn't exist or on cache miss. * Returns undefined if the profile can't be loaded. */ -export async function loadProfile(did: Did, profileCache?: KVNamespace) { +export async function getProfile(did: Did, profileCache?: KVNamespace) { // Try cache first if (profileCache) { try { diff --git a/apps/atproto-starter/src/lib/atproto/ui/LoginModal.svelte b/apps/atproto-starter/src/lib/atproto/ui/LoginModal.svelte index b3bad3a..db13102 100644 --- a/apps/atproto-starter/src/lib/atproto/ui/LoginModal.svelte +++ b/apps/atproto-starter/src/lib/atproto/ui/LoginModal.svelte @@ -5,18 +5,138 @@ - { - await login(handle); - return true; - }} - signup={async () => { - signup(); - return true; - }} -/> +{#if atProtoLoginModalState.open} + +{/if} diff --git a/apps/atproto-starter/src/routes/(oauth)/oauth-client-metadata.json/+server.ts b/apps/atproto-starter/src/routes/(oauth)/oauth-client-metadata.json/+server.ts index 44794e4..c09dfb0 100644 --- a/apps/atproto-starter/src/routes/(oauth)/oauth-client-metadata.json/+server.ts +++ b/apps/atproto-starter/src/routes/(oauth)/oauth-client-metadata.json/+server.ts @@ -1,8 +1,20 @@ import { json } from '@sveltejs/kit'; +import { dev } from '$app/environment'; import { createOAuthClient } from '$lib/atproto/server/oauth'; import type { RequestHandler } from './$types'; export const GET: RequestHandler = async ({ platform }) => { - const oauth = createOAuthClient(platform?.env); - return json(oauth.metadata); + try { + const oauth = createOAuthClient(platform?.env); + return json(oauth.metadata); + } catch (e) { + const message = e instanceof Error ? e.message : String(e); + console.error('[oauth-client-metadata]', message, e); + // Surface the real cause in dev so misconfig isn't a silent 500. + // In production we still hide details, but they land in `wrangler tail`. + return json( + { error: 'oauth_client_misconfigured', message: dev ? message : 'See server logs' }, + { status: 500 } + ); + } }; diff --git a/apps/atproto-starter/src/routes/+layout.svelte b/apps/atproto-starter/src/routes/+layout.svelte index 3ea070f..9053fd5 100644 --- a/apps/atproto-starter/src/routes/+layout.svelte +++ b/apps/atproto-starter/src/routes/+layout.svelte @@ -1,21 +1,19 @@ + + atproto-starter + + + + {@render children()} - - - - diff --git a/apps/atproto-starter/src/routes/+page.svelte b/apps/atproto-starter/src/routes/+page.svelte index bdfe12d..9c54bdd 100644 --- a/apps/atproto-starter/src/routes/+page.svelte +++ b/apps/atproto-starter/src/routes/+page.svelte @@ -1,6 +1,5 @@ @@ -8,15 +7,23 @@

atproto-starter

{#if !user.isLoggedIn} - + {:else}
- + Signed in as {user.profile?.handle ?? user.did} - +
{/if} diff --git a/apps/atproto-starter/tsconfig.json b/apps/atproto-starter/tsconfig.json index d1af521..46af250 100644 --- a/apps/atproto-starter/tsconfig.json +++ b/apps/atproto-starter/tsconfig.json @@ -10,8 +10,9 @@ "sourceMap": true, "strict": true, "moduleResolution": "bundler", - "types": ["@cloudflare/workers-types"] - } + "types": ["@cloudflare/workers-types", "node"] + }, + "exclude": ["src/lib/atproto/scripts/**"] // Path aliases are handled by https://svelte.dev/docs/kit/configuration#alias // except $lib which is handled by https://svelte.dev/docs/kit/configuration#files // diff --git a/docs/01-indexing.md b/docs/01-indexing.md index 81e44f1..54659fd 100644 --- a/docs/01-indexing.md +++ b/docs/01-indexing.md @@ -200,7 +200,8 @@ const db = createPostgresDatabase(pool); | `jetstreams` | Bluesky | Jetstream URLs | | `relays` | Bluesky | Relay URLs for discovery | | `notify` | off | `true` opens `notifyOfUpdate`; a string requires `Bearer` | -| `spaces` | — | See [Spaces](./05-spaces.md) | -| `community` | — | See [Communities](./06-communities.md) | -| `realtime` | — | See [Sync](./07-sync.md) | -| `labels` | — | See [Labels](./08-labels.md) | +| `feeds` | — | See [Feeds](./04-feeds.md) | +| `spaces` | — | See [Spaces](./06-spaces.md) | +| `community` | — | See [Communities](./07-communities.md) | +| `realtime` | — | See [Sync](./08-sync.md) | +| `labels` | — | See [Labels](./09-labels.md) | diff --git a/docs/04-feeds.md b/docs/04-feeds.md new file mode 100644 index 0000000..16e384a --- /dev/null +++ b/docs/04-feeds.md @@ -0,0 +1,109 @@ +# Feeds + +Personalized "what the people I follow are doing" timelines, fanned out at write time. Opt-in; no cost if you don't enable it. + +## Mental model + +> A feed is a (follow-collection, [target-collections]) pair, named by you. Every time someone an *actor* follows posts to a target collection, contrail inserts one row into `feed_items` for that actor. + +Reading a feed is a join through `feed_items` plus the standard pipeline (filters, sorts, hydration, references). The actor parameter on a read is *whose feed* you want — there is no anonymous feed read. + +## Enable + +```ts +import type { ContrailConfig } from "@atmo-dev/contrail"; + +const config: ContrailConfig = { + namespace: "com.example", + collections: { + follow: { collection: "app.bsky.graph.follow" }, + post: { collection: "app.bsky.feed.post", queryable: { /* ... */ } }, + }, + feeds: { + timeline: { + follow: "follow", // short name (key in `collections`), NOT the NSID + targets: ["post"], + maxItems: 500, // optional, default 200 + }, + }, +}; +``` + +Both the follow collection and every target collection must be declared in `collections`. Names in `feeds` are the **short names** (the keys of `collections`), not NSIDs. Config validation throws if you reference an unknown short name. + +## Follow-record shape + +The follow collection's record must have a `subject` field at the top level whose value is the followed DID. `app.bsky.graph.follow` matches this naturally: + +```json +{ "subject": "did:plc:abc...", "createdAt": "2026-01-01T00:00:00Z" } +``` + +Custom follow lexicons work as long as `subject` is the followed DID at JSON path `$.subject`. Contrail extracts via that path during ingest fan-out and during follow-event backfill. + +## Schema + +Two tables, one shared across all feeds: + +| Table | Purpose | +|---|---| +| `feed_items (actor, uri, collection, time_us)` | One row per (viewer, target record). Primary key `(actor, uri)` so a single target record can appear in many feeds. | +| `feed_backfills (actor, feed, completed)` | Marker so first-read backfill only runs once per (actor, feed). | + +Indexes: `(actor, collection, time_us DESC)` and `(actor, time_us DESC)` on `feed_items`, plus a JSON `subject` index on each follow collection's records table for the fan-out join. + +## Read + +``` +GET /xrpc/{namespace}.getFeed?feed=timeline&actor=&limit=50 +``` + +| Param | Meaning | +|---|---| +| `feed` | Feed name from `config.feeds` (required) | +| `actor` | Whose feed — DID or handle (required) | +| `collection` | Restrict to one target collection's short name (default: first in `targets`) | +| `limit`, `cursor`, filters from the target's `queryable`, hydration flags, sort/order | Same as `listRecords` on the target collection | + +The `actor` parameter is **whose feed** you're reading, not a filter on record creator. Feeds are always per-user. + +```ts +const feed = await fetch( + `/xrpc/com.example.getFeed?feed=timeline&actor=${did}&limit=50&profiles=true` +).then((r) => r.json()); +// feed.records — target records by users `actor` follows, newest first +// feed.profiles — hydrated profile records for record authors +``` + +## How fan-out works + +Three moments: + +1. **A target write.** Someone followed by N actors posts to a target collection. Contrail inserts N `feed_items` rows in one statement (`INSERT … SELECT … FROM WHERE subject = ?`). Cost is linear in N — there is no max-followers cap; a viral author with 1M followers is 1M inserts. + +2. **A follow write.** An actor follows a new user. Contrail backfills the most recent **100** target records from that user into the new follower's feed. The 100 is hardcoded in `core/router/feed.ts` — separate from the per-feed `maxItems` cap, and not tunable per feed today. + +3. **First read for an (actor, feed) pair.** Contrail backfills the actor's follow records from their PDS (so their `feed_items` rows can be computed), then populates `feed_items` from existing target records by users they already follow. Marked complete in `feed_backfills` so it runs once per pair. + +## Pruning + +The persistent worker (or cron run) trims `feed_items` per actor down to the largest configured `maxItems` across feeds, keeping newest by `time_us`. There is no per-feed prune — one global cap. + +## Deletes + +Deleting a target record removes its `feed_items` rows across all actors. Deleting a follow record currently does not retroactively prune the feed_items inserted during the original follow backfill — they age out via the global pruner instead. + +## XRPCs + +- `{namespace}.getFeed` — read + +That's it. Feeds are read-only over XRPC; writes to follow / target collections happen through `com.atproto.repo.putRecord` on the user's PDS as normal, and Jetstream ingestion drives the fan-out. + +## What's not here + +- No per-feed prune cap; the global pruner uses the largest `maxItems` across all feeds. +- The 100-record backfill on a new follow is hardcoded — not tunable per feed. +- No dedicated test coverage for feeds yet (the production paths work, but treat the integration as load-bearing-but-untested). +- No max-followers cap on target writes — a target record by a user with 1M followers means 1M `feed_items` inserts. For apps expecting that scale, partition feeds or rate-limit upstream. +- Feeds live in the **main DB** even when [spaces](./06-spaces.md) are split onto a separate DB; there is no `feeds_db` binding. +- Feeds do not currently union with [spaces](./06-spaces.md) records — `getFeed` reads only public target records. diff --git a/docs/04-auth.md b/docs/05-auth.md similarity index 99% rename from docs/04-auth.md rename to docs/05-auth.md index c628661..ad8f659 100644 --- a/docs/04-auth.md +++ b/docs/05-auth.md @@ -93,7 +93,7 @@ createWatchStore({ }); ``` -Each reconnect mints a fresh ticket, so expiry doesn't matter for long-lived subscriptions. See [Sync](./07-sync.md) for the full flow. +Each reconnect mints a fresh ticket, so expiry doesn't matter for long-lived subscriptions. See [Sync](./08-sync.md) for the full flow. ## OAuth permission sets diff --git a/docs/05-spaces.md b/docs/06-spaces.md similarity index 96% rename from docs/05-spaces.md rename to docs/06-spaces.md index bad1bd1..8bb9fb8 100644 --- a/docs/05-spaces.md +++ b/docs/06-spaces.md @@ -35,7 +35,7 @@ public_only: { collection: "com.example.public", allowInSpaces: false } ## Auth -Spaces use the standard contrail auth surface — service-auth JWTs for third-party apps, in-process server clients for your own loaders, invite tokens for anonymous read-grant links. See [Auth](./04-auth.md) for the full picture. +Spaces use the standard contrail auth surface — service-auth JWTs for third-party apps, in-process server clients for your own loaders, invite tokens for anonymous read-grant links. See [Auth](./05-auth.md) for the full picture. Space-specific wiring: @@ -55,7 +55,7 @@ Filters, sorts, hydration, and references work across all three. Records from a ## Invites -First-class primitive — see [Auth § Invite tokens](./04-auth.md#invite-tokens) for the mechanism. Space-specific: create via `com.example.space.invite.create`, redeem via `.redeem` (membership grant) or `?inviteToken=...` query param (read-only bearer grant). +First-class primitive — see [Auth § Invite tokens](./05-auth.md#invite-tokens) for the mechanism. Space-specific: create via `com.example.space.invite.create`, redeem via `.redeem` (membership grant) or `?inviteToken=...` query param (read-only bearer grant). ## XRPCs diff --git a/docs/06-communities.md b/docs/07-communities.md similarity index 97% rename from docs/06-communities.md rename to docs/07-communities.md index 59eab45..a7f2e26 100644 --- a/docs/06-communities.md +++ b/docs/07-communities.md @@ -1,6 +1,6 @@ # Communities -Group-controlled atproto DIDs. A community is a DID whose signing/rotation keys are held by the appview on behalf of multiple members, with tiered access levels. Built on top of [spaces](./05-spaces.md). +Group-controlled atproto DIDs. A community is a DID whose signing/rotation keys are held by the appview on behalf of multiple members, with tiered access levels. Built on top of [spaces](./06-spaces.md). ## When to use this diff --git a/docs/07-sync.md b/docs/08-sync.md similarity index 100% rename from docs/07-sync.md rename to docs/08-sync.md diff --git a/docs/08-labels.md b/docs/09-labels.md similarity index 100% rename from docs/08-labels.md rename to docs/09-labels.md diff --git a/docs/frameworks/sveltekit-cloudflare.md b/docs/frameworks/sveltekit-cloudflare.md index d03a488..6d01872 100644 --- a/docs/frameworks/sveltekit-cloudflare.md +++ b/docs/frameworks/sveltekit-cloudflare.md @@ -203,9 +203,10 @@ From now on: - [Indexing](../01-indexing.md) — config options, adapter choices - [Querying](../02-querying.md) — filters, sorts, hydration, search - [Lexicons](../03-lexicons.md) — generate TS types for your XRPC surface -- [Auth](../04-auth.md) — service-auth JWTs, invite tokens, watch tickets, OAuth permission sets -- [Spaces](../05-spaces.md) / [Communities](../06-communities.md) — private records + group-controlled DIDs, which both slot into the same handler you just mounted -- [Sync](../07-sync.md) — reactive client-side subscriptions (`createWatchStore`) wrapped in Svelte `$state` +- [Feeds](../04-feeds.md) — personalized timelines via follow + target collections +- [Auth](../05-auth.md) — service-auth JWTs, invite tokens, watch tickets, OAuth permission sets +- [Spaces](../06-spaces.md) / [Communities](../07-communities.md) — private records + group-controlled DIDs, which both slot into the same handler you just mounted +- [Sync](../08-sync.md) — reactive client-side subscriptions (`createWatchStore`) wrapped in Svelte `$state` ## Common gotchas -- 2.51.2