From 1f33d192bfba77043510896b489b2a5b4a33079b Mon Sep 17 00:00:00 2001 From: Niels Mokkenstorm Date: Fri, 10 Jul 2026 00:45:57 +0200 Subject: [PATCH] docs: product roadmap + spec accuracy pass; catalog-entities, catalog-browse, and bff-architecture specs --- README.md | 135 +++++++++--------------- ROADMAP.md | 212 ++++++++++++++++++++++++++++++++------ docs/bff-architecture.md | 170 ++++++++++++++++++++++++++++++ docs/catalog-authority.md | 45 +++++--- docs/catalog-browse.md | 147 ++++++++++++++++++++++++++ docs/catalog-entities.md | 79 ++++++++++++++ docs/discogs-import.md | 137 +++++++++--------------- docs/discogs-sync.md | 18 ++-- docs/shelf-events.md | 54 ++++------ 9 files changed, 744 insertions(+), 253 deletions(-) create mode 100644 docs/bff-architecture.md create mode 100644 docs/catalog-browse.md create mode 100644 docs/catalog-entities.md diff --git a/README.md b/README.md index 402c8a6..709d9ed 100644 --- a/README.md +++ b/README.md @@ -1,91 +1,66 @@ # at-record A Discogs-like, atproto-native record collection app, written in Gleam. Your -crate (the vinyl kind) is stored as records (the atproto kind) in your own PDS. - -This is the Gleam reincarnation of the `crate` design: same lexicons +crate (the vinyl kind) is stored as records (the atproto kind) in your own +PDS. Gleam reincarnation of the `crate` design: same lexicons (`dev.mokkenstorm.crate.*`), same PDS-as-storage idea, different stack. ## What it does today -- **Log in with atproto OAuth** (PAR + PKCE + DPoP, granular scopes). Tokens - and DPoP keys live server-side in an encrypted-at-rest store; the browser - only ever holds an opaque signed cookie. -- **Your crate is an append-only event log.** Every mutation is an immutable - `shelf.entry` event; the first event for a copy (the genesis) has no - `subject`, its record TID is the entry's identity, and later events - reference it via a `com.atproto.repo.strongRef`. The current crate, - wishlist, and history are folds over the log ([spec](./docs/shelf-events.md)). -- **Discogs seed search** on the add form, and **Discogs account connection + - collection import** (OAuth 1.0a): capped, resumable runs that dedup by - release id and upsert cover art onto existing entries +- atproto OAuth login (PAR + PKCE + DPoP); tokens live server-side, encrypted + at rest. +- Your crate is an append-only event log, folded into current crate, + wishlist, and history ([spec](./docs/shelf-events.md)). +- Discogs seed search, account connection, and collection import ([spec](./docs/discogs-import.md)). -- **Cover art is stored as blobs in your repo** (content-addressed, served via - `getBlob`), not hotlinked, so entries stay self-contained. +- Barcode scan-to-import (Chromium's `BarcodeDetector` only, for now). +- Cover art stored as blobs in your repo, not hotlinked. +- `via @handle` attribution chip on entries adopted from someone else's + catalog release. ## Layout -A Gleam multi-target monorepo. `server` and `web` depend on `shared` by path; -the generic atproto plumbing comes from Hex. - -| Package | Target | What it is | -| -------- | ---------- | ---------------------------------------------------------------------------------- | -| `shared` | erlang+js | Generated lexicon codecs (`gen/`) + a hand-written `StoredItem` envelope. | -| `server` | erlang | Wisp BFF: routes, OAuth, sessions, Discogs client. Uses `atproto/*` for PDS calls. | -| `web` | javascript | Lustre SPA (modem-routed pages), talks only to the BFF. | - -Lexicons live under `lexicons/` (our `dev.mokkenstorm.crate.*` plus vendored -`com.atproto.*` refs like `strongRef`). - -## The gleam-atproto packages - -The atproto client and the lexicon codegen were built here and extracted to -[gleam-atproto](https://tangled.org/@mokkenstorm.dev/gleam-atproto), publishing -independently to Hex: - -- [`atproto_client`](https://hex.pm/packages/atproto_client): transport-agnostic - client (XRPC, identity via Slingshot, OAuth machinery, blobs, repo CRUD). - Dual-target; modules under the `atproto/` namespace. -- [`atproto_codegen`](https://hex.pm/packages/atproto_codegen): lexicon-to-Gleam - types + JSON codecs. +Gleam multi-target monorepo. `server`/`web` depend on `shared` by path; the +generic atproto plumbing comes from Hex +([gleam-atproto](https://tangled.org/@mokkenstorm.dev/gleam-atproto): +[`atproto_client`](https://hex.pm/packages/atproto_client) + +[`atproto_codegen`](https://hex.pm/packages/atproto_codegen)). -## Lexicon codegen +| Package | Target | What it is | +| -------- | ---------- | --------------------------------------------------------------------------------------- | +| `shared` | erlang+js | Generated lexicon codecs (`gen/`) + `StoredItem`. | +| `server` | erlang | Wisp BFF: routes, OAuth, sessions, Discogs client. | +| `web` | javascript | Lustre SPA, talks to the BFF (+ the public Bluesky AppView directly for handle search). | -`shared/src/at_record/gen/` is **generated** and **gitignored**. `make gen` -runs `atproto_codegen` (a dev dependency of `shared`) over `lexicons/`, -emitting one module per lexicon: type + encoder + decoder, plus a `_fields` -helper per def. Supported: record/object defs, scalars, arrays, refs -(cross-lexicon), `cid-link`, `bytes`, `blob`, `unknown`, and **open unions** -(a variant per ref plus an `Other(Dynamic)` that round-trips unknown -`$type`s). Because the output is gitignored, **run `make gen` (or anything -that depends on it, like `make test`) before building a fresh checkout.** +Lexicons live under `lexicons/`. `shared/src/at_record/gen/` is +**generated and gitignored**: `make gen` runs `atproto_codegen` over +`lexicons/`. Run `make gen` (or `make test`, which depends on it) before +building a fresh checkout. ## Run it locally -Needs `gleam`, `erlang`, `node`, `docker` (for Postgres), and `make` -(`make help` lists every target). +Needs `gleam`, `erlang`, `node`, `docker`, `make`. ```sh make dev # watch loop: Postgres in docker, rebuilds server/web on change ``` -Browse **http://127.0.0.1:8080** (not `localhost`: cookies are host-specific -and every OAuth callback lands on the loopback IP; the server bounces -`localhost` over for you). Sign in with your handle and authorize on your PDS. +Browse **http://127.0.0.1:8080**, not `localhost` (cookies are host-specific +and OAuth callbacks land on the loopback IP). -Configuration comes from the environment; `scripts/dev.sh` and docker-compose -provide dev defaults and read `.env` (gitignored) for secrets: +Config comes from the environment (`scripts/dev.sh` / docker-compose provide +dev defaults, read `.env` for secrets): -| Variable | Required | Purpose | -| -------------------------------------------------- | -------- | ----------------------------------------------------------------------- | -| `SECRET_KEY_BASE` | yes | cookie signing secret | -| `STORE_KEY` | yes | at-rest encryption key for sessions + Discogs tokens (base64, 32 bytes) | -| `DATABASE_URL` | no | Postgres for persistent stores (in-memory fallback) | -| `DISCOGS_CONSUMER_KEY` / `DISCOGS_CONSUMER_SECRET` | no | Discogs app credential: search covers + account connect/import | -| `BASE_URL`, `OAUTH_CLIENT_JWK` | deploys | https origin switches OAuth to the confidential client | -| `PORT`, `SLINGSHOT_URL` | no | port (8080) and identity resolver overrides | +| Variable | Required | Purpose | +| -------------------------------------------------- | -------- | ----------------------------------------- | +| `SECRET_KEY_BASE` | yes | cookie signing secret | +| `STORE_KEY` | yes | at-rest encryption key (base64, 32 bytes) | +| `DATABASE_URL` | no | Postgres (in-memory fallback) | +| `DISCOGS_CONSUMER_KEY` / `DISCOGS_CONSUMER_SECRET` | no | Discogs search + import | +| `BASE_URL`, `OAUTH_CLIENT_JWK` | deploys | switches OAuth to the confidential client | +| `PORT`, `SLINGSHOT_URL` | no | port/identity-resolver overrides | -Confirm independently that your data is really on your PDS: +Confirm your data lives on your PDS: ```sh curl "https:///xrpc/com.atproto.repo.listRecords?repo=&collection=dev.mokkenstorm.crate.shelf.entry" @@ -99,29 +74,21 @@ make up # server + Postgres on http://localhost:8080 make down ``` -A multi-stage build runs codegen + the Lustre bundle, then ships an Erlang -release on `erlang:29-slim`. Crate data lives on your PDS; Postgres only holds -the encrypted session and Discogs-token stores. - ## Tests ```sh make test # codegen, then shared + server + web suites ``` -## Roadmap (from the `crate` design) - -Discogs account connection + collection import -([spec](./docs/discogs-import.md)) -> scan-to-import flow (barcode shipped, -photo mode design-ahead; see -[ROADMAP.md](./ROADMAP.md#scan-to-import-flow-barcode-shipped-2026-07-zxing-fallback-outstanding)) --> catalog authority + lazy promotion -([spec](./docs/catalog-authority.md)) -> two-way Discogs sync -([spec](./docs/discogs-sync.md)) -> Jetstream indexer + cross-user -search/appview -> social feed -> hybrid catalog edits. Mobile, when wanted, is -the same Lustre app wrapped in Capacitor (see -[ROADMAP.md](./ROADMAP.md#native-wrap-capacitor-over-tauri-deferred-until-scan-justifies-it)); -the camera-driven scan flow is the natural trigger. - -Engineering and architecture decisions (with rationale and triggers to revisit) -live in [ROADMAP.md](./ROADMAP.md). +Real OAuth and cross-account flows are covered by a separate +[e2e suite](./e2e/README.md) against a pinned local atproto devnet: + +```sh +make e2e-setup # once +make e2e +``` + +## Roadmap + +Product direction and engineering decisions live in +[ROADMAP.md](./ROADMAP.md). diff --git a/ROADMAP.md b/ROADMAP.md index c1b4b5c..550723e 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,23 +1,151 @@ # Roadmap & decisions -Engineering and architecture decisions for at-record, with the reasoning and the -**trigger to revisit** each. Product/feature direction lives in the -[README "Roadmap"](./README.md#roadmap-from-the-crate-design) section. +The product roadmap, then engineering and architecture decisions with the +reasoning and the **trigger to revisit** each. + +## Product roadmap + +Shipped baseline: atproto OAuth login (with handle autocomplete), the +append-only event-log crate, Discogs search/connect/import, barcode +scan-to-import, adopt-or-mint catalog promotion with `via @handle` +attribution, and the amend flow (own records superseded, foreign ones get +`basedOn` + a `catalog.edit` proposal). + +Three axes from here. Cross-axis order: design catch-up and the UX-hole +audit first, then data-model completeness, then catalog depth. + +### Clean design + +The direction is chosen and prototyped in Figma ("Zine × Greenhouse": +parchment, bark ink, hard offset shadows; a design-system page with color, +spacing, and stroke tokens plus components). The app's CSS mirrors the +tokens but not yet the full system. + +- **Figma catch-up (Figma-side sync done 2026-07):** the prototype page now + matches shipped reality: a Login screen (handle autocomplete open, with + suggestion rows), a "Crate — account menu" popout state, the + `via @handle` chip on Record detail, a "Scan — scanned rows" states card + (looking up / matched / no match / adding / added), a "Buttons — busy + states" strip (SAVING…/PUBLISHING…), and the fake phone status bar + removed from all nine screens (the `Structure/Status bar` component on + the design-system page is now unused; retire it there too). The other + direction still stands: the app is missing states the prototype designed + ("did you mean" suggestions, manual-search fallback, drafts), which are + the UX-holes list below. +- **UX flow holes** (filed 2026-07, tracked here): + - [x] Saving a record gives no feedback (done 2026-07: success notice + naming the saved title). + - [x] `dropped` (REMOVE) fires from a single tap (done 2026-07: two-tap + confirm, button arms to "CONFIRM REMOVE", disarms on navigation). + - [x] Scan no-match dead end, cheap fix (done 2026-07: SEARCH ghost + button linking to Add). Full fix stays open (needs the designed + "did you mean" flow + fuzzy matching, per the Figma no-match + screen). + - [ ] Login/OAuth failures surface only as a generic banner. Needs a + design pass first (see State design below). +- **State design:** empty crate, loading, and error states are unstyled + afterthoughts today (loading labels only just landed). Design them as + first-class screens. +- **Login screen:** skipped during the prototype round; it has since grown + handle autocomplete and deserves a real design pass. +- **Component parity:** finish turning the prototype's ad-hoc frames into + real Figma components and rebuild the screens from them, so design changes + propagate instead of drifting. +- **List view** as an alternative to the cover grid (long crates). +- **Dark mode:** a second variable mode in the Figma collection, then a CSS + custom-property theme flip; the token indirection is already in place on + both sides. + +### Complete data model + +The lexicons are ahead of the app: several fields exist on `shelf.entry` and +`catalog.release` that nothing writes or shows. Concretely, as of today: +`sleeveGrade` is absent even from the web `Entry` type; `notes` and `folder` +render when present but nothing can set them (the server's `annotated` and +`moved` reducer actions have no UI that dispatches them); `price`, +`counterparty`, and `acquiredAt` are wired end to end through lexicon and +fold but appear nowhere. + +- **Widen the add/edit surface to the lexicon** (tracked in "API surface + gaps" below): `sleeveGrade`, `folder`, `rating`, `acquiredAt`, and `notes` + are not settable at genesis; only media grade and rating are editable + after. This pairs with the Figma catch-up: each field needs a home in a + designed screen, not just a form input bolted on. +- **Money and provenance UI:** `price` and `counterparty` exist on + `shelf.entry` for acquired/sold events; no UI captures them. The event log + was designed for exactly this history ("bought 2019 for X, sold 2022 for + Y"). +- **Wantlist import:** `wanted` entries are modeled and viewable, but the + Discogs wantlist endpoint is not imported (cheap: same import machinery, + `action: "wanted"`). +- **`artistDisplay` on `catalog.release`:** a plain display-string field + (additive), so releases render "Title, Artist" without artist entities + existing (see [catalog-browse](./docs/catalog-browse.md)). +- **MusicBrainz ids as the cross-app bridge (before any shared lexicon):** + the atproto music ecosystem already converges on MBIDs as the neutral + key. teal.fm's `fm.teal.alpha.feed.play` carries `artistMbIds`, + `releaseMbId`, and `recordingMbId` (verified 2026-07; it has no + standalone artist records and no genre property), and our repeatable + `defs#externalId` was designed for exactly this (`provider: +"musicbrainz"`). Capturing MBIDs on releases (MusicBrainz supports + barcode lookup, and maps Discogs URLs to releases) makes at-record + entries and teal.fm plays correlatable today via shared natural keys and + Constellation URL backlinks, with zero lexicon coordination. +- **Artist + genre entities: PoC built and Discogs-fed 2026-07** (see + [catalog-entities](./docs/catalog-entities.md)): `catalog.artist` and + `catalog.genre` lexicons exist, and every mint path now fills them from + the (single, existing) per-release Discogs fetch: artists minted with + own-repo dedup and wired into `creditedArtists`; genres minted as + vocabulary from genre/style strings. Still open: cross-user artist + adoption via Constellation, MBIDs on artists, any read path, and + graduation past PoC (which means coordinating with teal.fm and similar + first; "required is forever" cuts both ways). Master and label entities + remain unbuilt (reserved in `catalog.edit`). + +### More complete catalog + +- **Enrich minted releases (mostly done 2026-07).** Minted releases now + carry `artistDisplay`, `country`, `released`, `genres`, `styles`, + barcode `identifiers`, and `creditedArtists`, all from the single + per-release Discogs fetch that was already made for cover art. Still + stubs: `tracklist`, `labels`, `master`. +- **Barcodes as identifiers (capture done 2026-07):** scanned/imported + barcodes land in `catalog.release.identifiers`. The payoff half is + open: making the scan flow match the network catalog first and Discogs + second, which compounds as the catalog grows. +- **Browsable catalog** ([spec sketch](./docs/catalog-browse.md)): v1 over + at-record's own known users, v2 a Jetstream-filtered indexer; want-it / + have-it straight from the browse grid. +- **Adoption counts** via Constellation backlinks: "N people have this", + browse sorting, and the answer to the open canonical-selection decision. +- **`catalog.edit` inbox:** amending a foreign record files a proposal + today, but the original author has no way to see or apply one. A + provider-side review/apply flow closes the loop (apply = mint + + `supersedes`). +- **Dedup in browse:** group near-duplicate releases by shared + `externalIds` before real canonical convergence exists. + +Longer arc, unchanged: two-way Discogs sync +([spec](./docs/discogs-sync.md)) -> cross-user discovery and social feed -> +native wrap when scan justifies it (sections below). The +[BFF-thinning work](./docs/bff-architecture.md) runs alongside as an +architecture track, not a product one. ## Decisions made -| Decision | Why | -| --------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Extract a generic `atproto` package** (XRPC, identity, auth, repo CRUD) | The generic plumbing is ~80% of a Gleam atproto SDK and none existed; isolating it keeps the app thin and the lib publishable. | -| **Transport-agnostic client** (`xrpc.Client` wraps a `send` fn; server injects `httpc`) | No HTTP dependency in the core, so it compiles on both targets and a JS caller could supply `fetch`. | -| **Generic `repo` CRUD** (collection NSID + row decoder, no lexicon knowledge) | Keeps `atproto` free of `crate` types; the shelf-specific decoder lives in the server. | -| **Slingshot for identity resolution** (`com.bad-example.identity.resolveMiniDoc`) | One call returns did+handle+pds, replacing the appview + `plc.directory` two-hop. | -| **MVU module split of the web app** | A 426-line single file became `model`/`msg`/`update`/`effects`/`view` + thin `main`. | -| **Docker** (multi-stage, `gleam export erlang-shipment`, `erlang:29-slim`, binds `0.0.0.0`) + **Makefile** | Single-service deploy (storage is the PDS, no DB). `0.0.0.0` bind is required or the container is unreachable. | -| **codegen scalar sweep** (`cid-link`->`String`, `bytes`->`BitArray`, `unknown`->`Dynamic` via `internal` helpers) | Removed the last skip (`catalog.edit`). `unknown` round-trips through a `dynamic_to_json` passthrough. | -| **External IDs as a repeatable `externalId` ref** (`provider`+`id`+`url`), replacing scalar `discogsReleaseId`/`mbid` | Many providers per item (Discogs, MusicBrainz, ...) without a `V2` to add one; relaxes `catalog.release.required` to `[title, createdAt]`. | -| **Finalised `required[]` + string limits** while pre-adoption (single-user) | "Required is forever"; every free-text string now carries `maxGraphemes`/`maxLength` (~10:1). The window to do this closes once others adopt the lexicons. | -| **Entry identity = genesis TID + `subject` strongRef** (replaced the client-minted `entryId`; BC break pre-adoption) | Stable linkable at-uris per entry (Constellation backlinks for free), no minted id to thread through, appends validate their genesis via `getRecord`. | +| Decision | Why | +| --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Extract a generic `atproto` package** (XRPC, identity, auth, repo CRUD) | The generic plumbing is ~80% of a Gleam atproto SDK and none existed; isolating it keeps the app thin and the lib publishable. | +| **Transport-agnostic client** (`xrpc.Client` wraps a `send` fn; server injects `httpc`) | No HTTP dependency in the core, so it compiles on both targets and a JS caller could supply `fetch`. | +| **Generic `repo` CRUD** (collection NSID + row decoder, no lexicon knowledge) | Keeps `atproto` free of `crate` types; the shelf-specific decoder lives in the server. | +| **Slingshot for identity resolution** (`com.bad-example.identity.resolveMiniDoc`) | One call returns did+handle+pds, replacing the appview + `plc.directory` two-hop. | +| **MVU module split of the web app** | A 426-line single file became `model`/`msg`/`update`/`effects`/`view` + thin `main`. | +| **Docker** (multi-stage, `gleam export erlang-shipment`, `erlang:29-slim`, binds `0.0.0.0`) + **Makefile** | Single-service deploy (storage is the PDS, no DB). `0.0.0.0` bind is required or the container is unreachable. | +| **codegen scalar sweep** (`cid-link`->`String`, `bytes`->`BitArray`, `unknown`->`Dynamic` via `internal` helpers) | Removed the last skip (`catalog.edit`). `unknown` round-trips through a `dynamic_to_json` passthrough. | +| **External IDs as a repeatable `externalId` ref** (`provider`+`id`+`url`), replacing scalar `discogsReleaseId`/`mbid` | Many providers per item (Discogs, MusicBrainz, ...) without a `V2` to add one; relaxes `catalog.release.required` to `[title, createdAt]`. | +| **Finalised `required[]` + string limits** while pre-adoption (single-user) | "Required is forever"; every free-text string now carries `maxGraphemes`/`maxLength` (~10:1). The window to do this closes once others adopt the lexicons. | +| **Entry identity = genesis TID + `subject` strongRef** (replaced the client-minted `entryId`; BC break pre-adoption) | Stable linkable at-uris per entry (Constellation backlinks for free), no minted id to thread through, appends validate their genesis via `getRecord`. | +| **Failure posture: best-effort for reads, accounted for writes** (decided 2026-07) | Display reads (handle chip, avatar) may silently degrade. Writes that create immutable records must surface failures (counts in run results, user notices) or fail the item; 429s always propagate to run-level rate-limit handling, never swallowed. | ## Deferred / planned @@ -85,17 +213,16 @@ in `server/src/at_record_server/oauth/`; discovery lives in the dual-target - **DPoP-authed PDS requests + token refresh** (a client wrapper, so the generic `repo` CRUD is reused unchanged). -**Granular scopes (least privilege):** `atproto repo:dev.mokkenstorm.crate.shelf.item` +**Granular scopes (least privilege):** `atproto repo:dev.mokkenstorm.crate.shelf.entry`, +`repo:dev.mokkenstorm.crate.catalog.release`, `repo:dev.mokkenstorm.crate.catalog.edit` (defined as the `client_metadata.scopes` list). `repo:` is write-only; repo reads are public. This replaced the discouraged `transition:generic`. -Scope follow-ups: +Scope follow-up: -- **When catalog writes land**, add `repo:dev.mokkenstorm.crate.catalog.release` - and `repo:dev.mokkenstorm.crate.catalog.edit` to `scopes`. - **Permission-set lexicon** (`include:dev.mokkenstorm.crate.`) is the next step up: it bundles permissions under one branded entry in the consent UI. - Overkill for a single collection; worth it once the scope list grows. + Overkill for three collections; worth it once the scope list grows further. Session store is a swappable `sessions.Store` interface with **Postgres** (`sessions_postgres`, used when `DATABASE_URL` is set) and **in-memory** @@ -109,6 +236,12 @@ Remaining: - Confidential deploys need a stable `OAUTH_CLIENT_JWK` and `BASE_URL` (`SECRET_KEY_BASE` and `STORE_KEY` are now required everywhere). +- **Whether to stay confidential at all is now an open question** — see + [bff-architecture.md](./docs/bff-architecture.md) (sketch): most of the + server's logic doesn't need a server secret, and a public-client, browser-custodied + session is the more atproto-native shape. Blocked today on DPoP crypto having + no WebCrypto (browser) backing anywhere in the dependency chain, only + Node's `crypto`. ### API surface gaps @@ -133,7 +266,7 @@ changelogs by hand. The atproto OAuth client machinery (PKCE, PAR, DPoP, tokens) was ported into `atproto_client`; migrating the server's `oauth/` modules onto it is the natural follow-up. -### Scan-to-import flow (barcode shipped 2026-07; ZXing fallback outstanding) +### Scan-to-import flow (barcode shipped 2026-07; universal scanner planned) The full flow is prototyped in [Figma](https://www.figma.com/design/xoAzpB0FCgpv4zzIZqA4TX/AT-Record?node-id=33-2): @@ -149,13 +282,28 @@ review / duplicate rows, unresolved items saved as drafts) -> import done. only — the API is Chromium-only, never implemented by Firefox (desktop or Android) or Safari. Unsupported browsers get a graceful message pointing at manual add instead of a silent broken camera. -- **ZXing (or similar) fallback (open).** Decode frames pulled off the - `