From dc2c0b3cf571dd183d9da686d76804b0c2eb28fa Mon Sep 17 00:00:00 2001 From: Owais Jamil Date: Sun, 31 May 2026 13:12:04 -0500 Subject: [PATCH] docs: adjust priorities and reorganize --- ARCHITECTURE.md | 108 ------------- docs/reference/README.md | 25 ++++ docs/reference/architecture.md | 134 +++++++++++++++++ docs/reference/blobs.md | 60 ++++++++ docs/reference/lexicon-schemas.md | 58 +++++++ docs/reference/record-apis.md | 60 ++++++++ docs/reference/repo-core.md | 56 +++++++ docs/reference/storage-sqlite.md | 66 ++++++++ docs/reference/sync-firehose.md | 72 +++++++++ docs/reference/xrpc.md | 61 ++++++++ docs/specs/README.md | 17 +-- docs/specs/architecture.md | 109 +------------- docs/specs/blobs.md | 115 +------------- docs/specs/identity-handles.md | 27 +++- docs/specs/lexicon-schemas.md | 183 +---------------------- docs/specs/record-apis.md | 104 +------------ docs/specs/repo-core.md | 125 +--------------- docs/specs/security-oauth.md | 46 ++++-- docs/specs/storage-sqlite.md | 118 +-------------- docs/specs/sync-firehose.md | 5 +- docs/specs/target-profile.md | 36 +++++ docs/specs/xrpc.md | 105 +------------ docs/tasks/00-foundation.md | 8 +- docs/tasks/01-xrpc-shell.md | 8 +- docs/tasks/02-accounts-sessions.md | 14 +- docs/tasks/03-identity-handles.md | 17 ++- docs/tasks/04-repo-core.md | 8 +- docs/tasks/05-record-apis.md | 8 +- docs/tasks/06-car-sync-reads.md | 8 +- docs/tasks/07-firehose.md | 38 +---- docs/tasks/08-blobs.md | 37 +---- docs/tasks/09-lexicon-schemas.md | 48 +----- docs/tasks/10-compatibility-hardening.md | 7 +- docs/tasks/11-security-oauth.md | 7 + docs/tasks/13-admin-deployment.md | 15 +- docs/tasks/14-operator-account-ux.md | 10 +- docs/tasks/README.md | 25 +++- 37 files changed, 864 insertions(+), 1084 deletions(-) delete mode 100644 ARCHITECTURE.md create mode 100644 docs/reference/README.md create mode 100644 docs/reference/architecture.md create mode 100644 docs/reference/blobs.md create mode 100644 docs/reference/lexicon-schemas.md create mode 100644 docs/reference/record-apis.md create mode 100644 docs/reference/repo-core.md create mode 100644 docs/reference/storage-sqlite.md create mode 100644 docs/reference/sync-firehose.md create mode 100644 docs/reference/xrpc.md create mode 100644 docs/specs/target-profile.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md deleted file mode 100644 index 96090f8..0000000 --- a/ARCHITECTURE.md +++ /dev/null @@ -1,108 +0,0 @@ -# Tempest's Architecture - -## AT Proto - -ATProto (Bluesky's protocol) is a federated social network protocol where users own their data. -The key concepts Tempest implements: - -- **DID** - decentralized identity (e.g. `did:plc:abc123`) - your permanent identifier -- **Handle** - human-readable username (e.g. `alice.example.com`), resolved to a DID -- **Repo** - a personal cryptographically-signed data store per user -- **XRPC** - the HTTP RPC layer (`GET|POST /xrpc/com.atproto.repo.createRecord`) -- **Firehose** - a WebSocket stream (`subscribeRepos`) that broadcasts every write in real-time -- **MST** - Merkle Search Tree, the core data structure for repos (think: a content-addressed B-tree) -- **CAR** - Content Addressable aRchive, the binary export format for repos -- **CID** - Content Identifiers (SHA-256 hashes used as block addresses) -- **TID** - Timestamp IDs, used as record keys with monotonic guarantees - -## High-Level - -```text -Bluesky App / ATProto Client - │ - ▼ - TempestWeb.Endpoint (Bandit HTTP) - │ - ┌─────┴──────────────────────────┐ - │ Router │ - │ GET /xrpc/:method ──────────►│XrpcController - │ WebSocket /xrpc/...subscribeR │FirehoseSocket ──► Phoenix PubSub - │ GET /.well-known/atproto-did │WellKnownController - │ GET / │PageController (landing page) - └────────────────────────────────┘ - │ - ┌─────┴───────────┐ - │ Contexts │ - │ Accounts │──► account.sqlite (Ecto) - │ Identity │──► DNS / HTTPS resolution - │ Records │──► per-DID repo SQLite (raw) - │ Sync │──► CAR exports, block reads - │ Sequencer │──► sequencer.sqlite + PubSub broadcast - └─────────────────┘ -``` - -## Dual SQLite Strategy - -Two SQLite systems running in parallel: - -| System | Driver | Databases | Contents | -| --------------------- | ------------- | ----------------------------- | -------------------------------- | -| Ecto + `ecto_sqlite3` | Standard Ecto | `account.sqlite` | accounts, sessions, signing keys | -| Raw `Exqlite.Sqlite3` | Manual SQL | `repos/{did}.sqlite` per user | blocks, records, commits, MST | -| Raw `Exqlite.Sqlite3` | Manual SQL | `sequencer.sqlite` | global event log (firehose) | - -Raw SQL is used for repos because each account's repo is created on demand and has a -different lifecycle than a typical Ecto repo. Fitting dynamic, per-DID databases into -Ecto's single-repo model would be awkward. - -## Cryptographic Write Cycle - -Every record write (`createRecord`, `putRecord`, `deleteRecord`) triggers a full commit cycle inside a `BEGIN IMMEDIATE` SQLite transaction: - -1. Load current MST entries from the block store -2. Insert/delete the new entry -3. Rebuild the entire MST (produces new CBOR blocks) -4. Sign a new commit block with the account's secp256k1 key -5. Insert all new CBOR blocks into `blocks` table -6. Update `records` index -7. Update `repo_metadata` (current rev/CID) -8. Write event to `sequencer.sqlite`, broadcast via PubSub - -The MST is **rebuilt from scratch** on every write (correctness-first; incremental updates are a future optimization). - -## Key Modules - -| Module | Role | -| ------------------- | --------------------------------------------------- | -| `RepoCore.Mst` | Merkle Search Tree - pure Elixir, content-addressed | -| `RepoCore.Drisl` | DAG-CBOR encoder/decoder (strict atproto subset) | -| `RepoCore.Car` | CAR v1 archive encoder | -| `RepoCore.Commit` | Signed v3 commit blocks (secp256k1) | -| `RepoStorage` | Per-DID SQLite management - 1100+ lines | -| `Sequencer` | Durable event log + PubSub fan-out | -| `Xrpc.Registry` | Compile-time XRPC method table (O(1) dispatch) | -| `Identity.KeyStore` | secp256k1 key gen + AES-GCM encryption at rest | -| `Lexicon.Validator` | Runtime ATProto schema validation | - -## Authentication - -Two-token system: - -- **Access tokens** - 15-minute `Phoenix.Token`-signed tokens (HMAC, not standard JWT) -- **Refresh tokens** - opaque 48-byte random tokens, stored as SHA-256 hashes, 30-day TTL - -Token rotation with **reuse detection**: if a previously-rotated refresh token is presented again (replay attack), the entire token family is immediately revoked across all sessions. - -## OTP Supervision Tree - -```text -Tempest.Supervisor -├── TempestWeb.Telemetry -├── Tempest.Repo (Ecto/SQLite) -├── DNSCluster (multi-node ready) -├── Phoenix.PubSub (firehose fan-out only) -└── TempestWeb.Endpoint (Bandit) -``` - -The firehose WebSocket clients (`FirehoseSocket`) are spawned by Bandit per connection using the `WebSock` behavior. -They subscribe to PubSub directly and receive live events as messages. diff --git a/docs/reference/README.md b/docs/reference/README.md new file mode 100644 index 0000000..2fd75dd --- /dev/null +++ b/docs/reference/README.md @@ -0,0 +1,25 @@ +--- +title: Reference Documentation +updated: 2026-05-31 +--- + +These docs describe current, verified behavior of Tempest. They explain the +implementation at a high level and introduce the AT Protocol/PDS concepts needed +to understand that implementation. + +Graduated reference docs: + +- [Architecture](architecture.md) +- [SQLite Storage](storage-sqlite.md) +- [XRPC HTTP Surface](xrpc.md) +- [Repository Core](repo-core.md) +- [Record APIs](record-apis.md) +- [Blobs](blobs.md) +- [Lexicon Schemas](lexicon-schemas.md) + +Additional reference notes: + +- [Sync and Firehose](sync-firehose.md) + +When a spec graduates, `docs/specs/.md` becomes a short pointer to the +reference doc and its verification commands. diff --git a/docs/reference/architecture.md b/docs/reference/architecture.md new file mode 100644 index 0000000..526e080 --- /dev/null +++ b/docs/reference/architecture.md @@ -0,0 +1,134 @@ +--- +title: Architecture +updated: 2026-05-31 +--- + +Tempest is a Phoenix application that implements AT Protocol PDS behavior through +small contexts: XRPC routing, accounts/auth, identity, repository storage, blobs, +sync, and operator tooling. + +## Concepts + +A PDS hosts user repositories. Each account has a DID, a mutable handle, a signed +repository, and blob data referenced by records. Other services discover where a +repo lives through the account's DID document and then read repo data through +XRPC sync endpoints or the firehose. + +XRPC is the HTTP RPC layer. Repository data is content-addressed and exported as +CAR. Firehose events announce changes after they are durably sequenced. + +Key protocol objects: + +- DID: stable decentralized identifier for an account. +- Handle: mutable human-readable name that resolves to a DID. +- Repo: signed personal data store for one DID. +- MST: Merkle Search Tree used to represent current repo records. +- CID: content identifier for blocks. +- TID: timestamp identifier used for sortable record keys and revisions. +- CAR: content-addressed archive used for repo/block export. + +## Request flow + +```text +AT Protocol client + │ + ▼ +TempestWeb.Endpoint (Bandit) + │ + ▼ +TempestWeb.Router + ├─ /xrpc/:method -> XrpcController + ├─ /xrpc/...subscribeRepos -> FirehoseController / WebSocket + ├─ /.well-known/atproto-did -> WellKnownController + └─ / -> PageController + │ + ▼ +Contexts + ├─ Accounts -> account.sqlite via Ecto + ├─ Identity -> DID docs, handles, signing keys + ├─ Records -> per-DID repo SQLite files + ├─ Blobs -> account.sqlite metadata + local bytes + ├─ Sync -> CAR/block/blob reads + └─ Sequencer -> sequencer.sqlite + PubSub fanout +``` + +Controllers should not build commits directly. They validate/dispatch, then +contexts handle storage, signing, sequencing, and response data. + +## Storage shape + +Tempest uses SQLite in three roles: + +| System | Storage | Contents | +| --- | --- | --- | +| Ecto + `ecto_sqlite3` | `account.sqlite` | accounts, sessions, keys, auth, blob metadata | +| Raw `Exqlite.Sqlite3` | `repos/.sqlite` | blocks, records, commits, MST metadata | +| Raw `Exqlite.Sqlite3` | `sequencer.sqlite` | global event log / firehose cursor | + +Repo databases are per-DID because accounts are created dynamically and each +repo has a lifecycle independent of ordinary account metadata. + +## Write cycle + +A record write (`createRecord`, `putRecord`, `deleteRecord`, or `applyWrites`) +performs a complete repo commit cycle: + +1. Load current MST entries from repo storage. +2. Insert, replace, or delete the record entry. +3. Rebuild the MST for correctness. +4. Sign a new commit block with the account signing key. +5. Store new CBOR blocks. +6. Update the record index and repo metadata. +7. Write a durable sequencer event. +8. Fan out the event to live firehose subscribers. + +Tempest currently rebuilds the MST from scratch on writes. That is a deliberate +correctness-first tradeoff. + +## Key modules + +| Module | Role | +| --- | --- | +| `Tempest.Xrpc.Registry` | XRPC method table and dispatch metadata | +| `Tempest.Accounts` | accounts, sessions, auth contexts, app passwords | +| `Tempest.Identity.KeyStore` | account signing key generation and storage | +| `Tempest.Records` | record write/read boundary | +| `Tempest.RepoStorage` | per-DID SQLite repository storage | +| `Tempest.RepoCore.Mst` | Merkle Search Tree implementation | +| `Tempest.RepoCore.Drisl` | deterministic DAG-CBOR subset | +| `Tempest.RepoCore.Car` | CAR v1 archive encoding/decoding | +| `Tempest.RepoCore.Commit` | signed repo commit blocks | +| `Tempest.Blobs` | blob metadata, reference lifecycle, GC | +| `Tempest.Sequencer` | durable event log and PubSub fanout | +| `Tempest.Lexicon.Validator` | runtime Lexicon validation | + +## Auth shape + +Legacy sessions use two credentials: + +- access token: short-lived `Phoenix.Token` token +- refresh token: opaque random token stored only as a SHA-256 hash + +Refresh tokens rotate. Reuse of an old rotated refresh token revokes the token +family. OAuth and app passwords share the centralized permission boundary added +for modern atproto clients. + +## Supervision + +```text +Tempest.Supervisor +├── TempestWeb.Telemetry +├── Tempest.Repo +├── DNSCluster +├── Phoenix.PubSub +└── TempestWeb.Endpoint +``` + +Firehose WebSocket connections are spawned by the web server and subscribe to +PubSub for live events after durable sequencer insertion. + +## Verification + +```bash +hurl --test --variable base_url=http://localhost:4000 test/smoke/health.hurl +``` diff --git a/docs/reference/blobs.md b/docs/reference/blobs.md new file mode 100644 index 0000000..3793deb --- /dev/null +++ b/docs/reference/blobs.md @@ -0,0 +1,60 @@ +--- +title: Blobs +updated: 2026-05-31 +--- + +Blobs are binary data uploaded outside repository records. Records refer to blobs +by CID. Tempest only exposes a blob publicly after a committed record references +it. + +## Concepts + +A blob is content-addressed data, commonly image or video bytes. Uploading a blob +is not enough to publish it. Publication happens when a repo record includes a +valid blob reference and that record is committed. + +This split lets clients upload bytes first, then create or update records that +reference them atomically from the repo's point of view. + +## Implemented endpoints + +- `com.atproto.repo.uploadBlob` +- `com.atproto.sync.listBlobs` +- `com.atproto.sync.getBlob` + +`uploadBlob` requires auth. Sync blob reads are public for active accounts when +the blob is public. + +## Implementation + +Blob metadata lives in `account.sqlite`. Local bytes live under: + +```text +/blobs// +``` + +Tempest computes raw CIDv1 SHA-256 CIDs for uploaded bytes, validates size and +MIME boundaries, stores new uploads as temporary metadata, scans record writes +for blob references, rejects references to missing blobs, and promotes referenced +blobs after the record commit succeeds. + +Deletes and updates recompute references so unreferenced blobs can be garbage +collected after their temporary grace period. + +## Safety behavior + +Temp uploads are not listed as public. Inactive accounts suppress blob serving. +Blob responses include defensive headers such as `X-Content-Type-Options: +nosniff` and a restrictive Content Security Policy. + +Local storage is the default adapter. S3/CDN behavior is tracked separately as +operator/deployment work. + +## Verification + +```bash +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/blobs.hurl +``` + +The smoke test covers upload, reference promotion, listing, download, missing +reference rejection, and response headers. diff --git a/docs/reference/lexicon-schemas.md b/docs/reference/lexicon-schemas.md new file mode 100644 index 0000000..c7d05b7 --- /dev/null +++ b/docs/reference/lexicon-schemas.md @@ -0,0 +1,58 @@ +--- +title: Lexicon Schemas +updated: 2026-05-31 +--- + +Lexicons describe AT Protocol schemas: XRPC methods, record shapes, object types, +refs, unions, and primitive constraints. Tempest uses Lexicons to validate record +writes without hardcoding application schemas in validator code. + +## Concepts + +A collection such as `app.bsky.actor.profile` is also a Lexicon NSID. A record's +`$type` should match its collection. If the PDS knows the schema, it can validate +fields and rkey rules before committing the record. + +Unknown schemas are possible in an open protocol. Clients can ask for strict +validation; otherwise a PDS may accept unknown records while marking validation +status as unknown. + +## Implementation + +Tempest separates three concerns: + +- schema engine: validates Lexicon documents and record values generically +- registry: answers whether a schema is known and trusted +- sources: bundled generated schemas, local configured schemas, and explicitly + enabled external resolution + +Bundled schemas are generated from a pinned atproto Lexicon checkout and include +source metadata. Operators can configure local schema files for custom apps. +External resolution is behind policy controls and uses cache, timeout, redirect, +response-size, recursion, and private-address protections. + +## Record validation behavior + +For record writes: + +- known schema + validation enabled: validate and return `valid` +- unknown schema + `validate: true`: reject with `InvalidRequest` +- unknown schema + validation unset: accept and return `unknown` +- `validate: false`: skip schema validation after generic safety checks + +Generic safety checks still apply: `$type`, collection NSID, rkey syntax, record +size, CBOR limits, and repo write authorization. + +## Operator workflow + +Regenerate bundled schemas from a pinned source when updating the known atproto +Lexicon set. Local schemas should fail startup/generation if they contain invalid +or duplicate IDs. + +## Verification + +```bash +mix tempest.lexicon.generate --source ../atproto/lexicons --commit +mix test test/tempest/lexicon +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/lexicon-schemas.hurl +``` diff --git a/docs/reference/record-apis.md b/docs/reference/record-apis.md new file mode 100644 index 0000000..5a8e79b --- /dev/null +++ b/docs/reference/record-apis.md @@ -0,0 +1,60 @@ +--- +title: Record APIs +updated: 2026-05-31 +--- + +Record APIs expose repository reads and writes through XRPC. They are the main +way clients create profile records, posts, follows, custom app records, and other +atproto data. + +## Concepts + +A record belongs to a collection NSID and has a record key (`rkey`). Together +with the account DID they form an AT URI: + +```text +at://// +``` + +Writes create signed repository commits. Reads return the current record and CID. +Deleted records disappear from the current repo view; the repo history is carried +by commits and blocks rather than application-level tombstone rows. + +## Implemented endpoints + +- `com.atproto.repo.createRecord` +- `com.atproto.repo.putRecord` +- `com.atproto.repo.deleteRecord` +- `com.atproto.repo.applyWrites` +- `com.atproto.repo.getRecord` +- `com.atproto.repo.listRecords` +- `com.atproto.repo.describeRepo` + +Write endpoints require auth. Read endpoints are public for hosted active repos. + +## Write path + +A write validates input, checks authorization, validates collection/rkey syntax, +validates the record against known Lexicons when possible, applies swap checks, +updates repo storage, signs a new commit, and writes a sequencer event. + +`createRecord` can generate an rkey when absent. `putRecord` replaces or creates +a specific rkey. `deleteRecord` removes the current record from the MST and record +index. `applyWrites` batches multiple create/update/delete operations. + +Compare-and-swap fields protect callers from overwriting unexpected repo or +record state. Conflicts return protocol-shaped errors. + +## Read path + +`getRecord` fetches one current record. `listRecords` scans a collection with +pagination. `describeRepo` reports repo metadata and available collections. + +## Verification + +```bash +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/records.hurl +``` + +The smoke test covers create, put, delete, get, list, compare-and-swap behavior, +and record validation boundaries. diff --git a/docs/reference/repo-core.md b/docs/reference/repo-core.md new file mode 100644 index 0000000..53052be --- /dev/null +++ b/docs/reference/repo-core.md @@ -0,0 +1,56 @@ +--- +title: Repository Core +updated: 2026-05-31 +--- + +Repo-core is Tempest's content-addressed repository engine. It is implemented in +pure Elixir and is the byte-level compatibility boundary for records, commits, +CAR exports, and firehose commit slices. + +## Concepts + +An AT Protocol repository is a signed data store for one DID. Records live at +paths shaped like: + +```text +/ +``` + +The current set of records is represented by a Merkle Search Tree (MST). Record +blocks, MST nodes, and commit blocks are content-addressed by CIDs. A commit signs +the current root and gives the repo a new revision (`rev`). + +A CAR file is a portable bundle of content-addressed blocks. Sync endpoints use +CAR to export repo data or selected blocks. + +## Implementation + +Important modules under `Tempest.RepoCore` include: + +- `AtUri`, `Did`, `Handle`, `Nsid`, `RecordKey`, `Tid`: syntax and identifiers +- `Cid`: CID encoding/decoding +- `Drisl`: deterministic DAG-CBOR subset used by atproto data +- `Car`: CAR v1 encode/decode +- `Mst`: deterministic Merkle Search Tree operations +- `Commit`: signed repository commit creation and verification + +`Tempest.RepoStorage` persists blocks, records, commits, and metadata in a +per-DID SQLite database. Record writes rebuild the MST for correctness, sign a +new commit, store new blocks, update indexes, and emit a sequencer event. + +## Compatibility notes + +The repo engine enforces canonical encoding, CID digest checks, syntax checks, +and decode limits. Tests cover parser behavior, CBOR/CAR round trips, MST +operations, commit signatures, and storage persistence. + +Correctness matters more than incremental MST performance. Rebuilding the MST on +write is acceptable until profiling proves otherwise. + +## Verification + +```bash +mix test test/tempest/repo_core test/tempest/repo_storage_test.exs +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/records.hurl +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/car-sync.hurl +``` diff --git a/docs/reference/storage-sqlite.md b/docs/reference/storage-sqlite.md new file mode 100644 index 0000000..d4d7dc5 --- /dev/null +++ b/docs/reference/storage-sqlite.md @@ -0,0 +1,66 @@ +--- +title: SQLite Storage +updated: 2026-05-31 +--- + +Tempest uses SQLite as the default storage backend for a small self-hosted PDS. +The design favors inspectable local files over a required external database. + +## Concepts + +A PDS stores several kinds of state: + +- account/auth state: who can act as whom +- repository state: signed record blocks and commits +- sequencer state: global event order for firehose consumers +- blob state: uploaded bytes and whether records currently reference them + +Repository data is public and content-addressed, but account/session keys and +operator secrets are private operational state. + +## Data directory + +`TEMPEST_DATA_DIR` owns all local persistent state. Development defaults to +`priv/tempest_dev`. + +Typical layout: + +```text +/ + account.sqlite + sequencer.sqlite + repos/ + .sqlite + blobs/ + / + +``` + +## Databases + +`account.sqlite` is managed through Ecto and stores account-oriented state: +accounts, sessions, signing keys, app passwords, OAuth data, and blob metadata. + +`sequencer.sqlite` is a raw SQLite database for the monotonic PDS-wide event +sequence. It is the durable source for firehose cursors. + +Each hosted DID gets a separate repo database under `repos/`. Repo databases +store blocks, current record indexes, commits, and repo metadata. + +## Operational notes + +SQLite runs with WAL where configured. Repo writes are serialized per DID and use +explicit transaction boundaries so a record write cannot become visible without +its corresponding commit metadata. + +Backups must include `account.sqlite`, `sequencer.sqlite`, `repos/`, `blobs/`, +and any WAL files or checkpointed equivalents. + +## Verification + +```bash +hurl --test --variable base_url=http://localhost:4000 test/smoke/health.hurl +``` + +The health response includes the active data directory, account DB, sequencer DB, +and writability status. diff --git a/docs/reference/sync-firehose.md b/docs/reference/sync-firehose.md new file mode 100644 index 0000000..8a5efcc --- /dev/null +++ b/docs/reference/sync-firehose.md @@ -0,0 +1,72 @@ +--- +title: Sync and Firehose (Reference) +updated: 2026-05-31 +--- + +Tempest exposes read-only sync exports and a live event stream ("firehose"). + +The firehose cursor is backed by `sequencer.sqlite` (`repo_seq` table). Sequence +numbers are global and monotonic within a Tempest instance. + +## Endpoints + +Sync reads: + +- `com.atproto.sync.getRepo` (CAR) +- `com.atproto.sync.getBlocks` (CAR) +- `com.atproto.sync.getRecord` (CAR) +- `com.atproto.sync.getLatestCommit` (JSON) +- `com.atproto.sync.getRepoStatus` (JSON) +- `com.atproto.sync.listRepos` (JSON) +- `com.atproto.sync.listBlobs` (JSON) +- `com.atproto.sync.getBlob` (bytes) + +Firehose: + +- `com.atproto.sync.subscribeRepos` (WebSocket, CBOR frames) + +Relay hint: + +- `com.atproto.sync.requestCrawl` (JSON) + +## Firehose usage + +Connect: + +- `ws(s):///xrpc/com.atproto.sync.subscribeRepos?cursor=` + +Rules: + +- If `cursor` is provided, Tempest backfills events with `seq > cursor` before + switching to live fanout. +- Events are emitted only after a durable insert into `sequencer.sqlite`. + +### Event kinds + +Tempest emits the atproto eventstream envelope types: + +- `#commit` +- `#identity` +- `#account` +- `#sync` + +## Verification + +Smoke test: + +```bash +hurl --test --jobs 1 \ + --variable base_url=http://localhost:4000 \ + test/smoke/firehose.hurl +``` + +This test exercises: subscribe, write, receive a commit event with increasing +`seq`. + +## Current limitations + +- This page documents Tempest's firehose behavior *within one instance*. + Federation-wide delivery (external relays/AppViews) depends on deployment and + interoperability work tracked elsewhere. +- `requestCrawl` behavior is compatibility-sensitive; treat it as an interop + surface and keep `test/smoke/tempest_compat.hurl` green. diff --git a/docs/reference/xrpc.md b/docs/reference/xrpc.md new file mode 100644 index 0000000..10aafc2 --- /dev/null +++ b/docs/reference/xrpc.md @@ -0,0 +1,61 @@ +--- +title: XRPC HTTP Surface +updated: 2026-05-31 +--- + +XRPC is the public HTTP RPC layer used by AT Protocol clients and services. +Tempest exposes XRPC methods under `/xrpc/`. + +## Concepts + +A Lexicon method has an NSID such as `com.atproto.repo.createRecord`. Methods are +queries, procedures, or subscriptions: + +- queries usually use `GET` and URL query parameters +- procedures use `POST` and a JSON or byte body +- subscriptions upgrade to a streaming protocol such as WebSocket + +Errors are protocol-shaped JSON with an `error` string and optional `message`. + +## Implementation + +`TempestWeb.Router` sends `/xrpc/*` requests through the XRPC pipeline. +`Tempest.Xrpc.Registry` is the method table. Each entry declares: + +- NSID +- kind (`query`, `procedure`, `subscription`) +- auth requirement +- input/output content type +- handler module/function +- known errors + +`TempestWeb.XrpcController` performs registry lookup, method checks, content +handling, auth assignment, and error rendering before calling the handler. + +Handlers live under `Tempest.Xrpc.*` and should remain thin. Long-running or +stateful work belongs in contexts such as `Tempest.Accounts`, `Tempest.Records`, +`Tempest.Sync`, or `Tempest.Blobs`. + +## Error behavior + +Unknown methods return JSON, not Phoenix HTML errors. Verb mismatches and content +mismatches fail before handler execution. + +Common status mappings: + +- `400`: invalid params/body +- `401`: missing or invalid auth +- `403`: authenticated but forbidden +- `404`: unknown method or missing resource +- `409`: compare-and-swap conflict +- `500`: unexpected internal error +- `502`: upstream failure + +## Verification + +```bash +hurl --test --variable base_url=http://localhost:4000 test/smoke/xrpc.hurl +``` + +The smoke test covers health, method dispatch, unknown methods, and verb/error +shape behavior. diff --git a/docs/specs/README.md b/docs/specs/README.md index 9dea382..9ceaafb 100644 --- a/docs/specs/README.md +++ b/docs/specs/README.md @@ -3,14 +3,17 @@ title: Tempest AT Protocol PDS Specification Index updated: 2026-05-08 --- -# Tempest PDS Specs +Tempest is a Phoenix application that will become a self-hostable AT Protocol +Personal Data Server. -Tempest is a Phoenix application that will become a self-hostable AT Protocol Personal Data Server. The first target is a single-node server for one user or a small community. Hosted-provider scale can come later. +Near-term focus: [Target Profile - Solo Login + Archive](target-profile.md). +Hosted-provider scale can come later. -Subsystem specifications live in this directory. Milestone task plans live in `../tasks/`. +Subsystem specifications live in this directory. ## Reading Order +0. [Target Profile - Solo Login + Archive](target-profile.md) 1. [Architecture](architecture.md) 2. [XRPC HTTP Surface](xrpc.md) 3. [SQLite Storage](storage-sqlite.md) @@ -28,12 +31,6 @@ Subsystem specifications live in this directory. Milestone task plans live in `. 15. [Interop and Integration Testing](interop-testing.md) 16. [Hurl Smoke Tests](hurl-smoke-tests.md) -## Milestones - -Milestone tasks live in `docs/tasks/`. Each task file is intended to be small enough to implement and review without guessing at scope. Each milestone must end with a black-box Hurl smoke test. - -Start with [Milestone 00](../tasks/00-foundation.md), then continue in numeric order. - ## Source Baseline Research was checked on 2026-05-07 against: @@ -57,7 +54,7 @@ Research was checked on 2026-05-07 against: - Prefer plain technical prose. - Keep protocol facts separate from project choices. - Every subsystem spec must include HTTP verification. -- Every milestone must have integration tests before it is considered done. +- Every completed subsystem must have integration tests before it is considered done. - Smoke tests must be Hurl files under `test/smoke/*.hurl` and must run with `hurl --test`. - Use `Req` for outbound HTTP in application code. - Run `mix precommit` before marking implementation work complete. diff --git a/docs/specs/architecture.md b/docs/specs/architecture.md index 021470b..47e8ef7 100644 --- a/docs/specs/architecture.md +++ b/docs/specs/architecture.md @@ -1,110 +1,11 @@ --- title: Architecture -updated: 2026-05-08 +updated: 2026-05-31 +status: done --- -# Architecture +Reference documentation: ../reference/architecture.md -Tempest should stay a Phoenix application while the PDS shape is still forming. Use internal contexts first, then split into an umbrella only if compile time, release packaging, or ownership boundaries justify it. +Verification: -## Target Runtime - -```text -TempestWeb - /xrpc HTTP routes - WebSocket firehose endpoint - health and admin routes - -Tempest.Xrpc - method registry - request/response validation - error mapping - auth plugs - -Tempest.Accounts - accounts, sessions, app passwords, account status, recovery - -Tempest.Identity - DID documents, handle verification, PLC calls, signing keys - -Tempest.Authz - OAuth provider, permission engine, delegated access, MFA - -Tempest.RepoCore - AT URIs, NSIDs, TIDs, CIDs, DRISL CBOR, MST, commits, CAR - -Tempest.Records - createRecord, putRecord, deleteRecord, getRecord, listRecords - -Tempest.Blobs - upload, metadata, local/S3 storage, references, GC - -Tempest.Sync - sequencer, repo events, backfill, firehose fanout - -Tempest.Admin - repair, backup, restore, verification tasks - -TempestWeb.AccountUI - account management, OAuth consent, repo browser, security settings -``` - -## Protocol Facts - -- XRPC endpoints are top-level paths under `/xrpc/`. -- A PDS is the authoritative host for the repositories it serves; the account DID document declares that location. -- Repository data is public, content-addressed, signed, and exported as CAR. -- Blobs are content-addressed but managed in the context of a DID. -- Firehose streams include repository, identity, account, and sync events for accounts hosted by this server. - -## Project Choices - -- SQLite is the default storage target. -- The local filesystem is the default blob store. -- Repo-core correctness has priority over broad endpoint coverage. -- OAuth is a later milestone, but its data model must be anticipated by sessions, app passwords, and delegated access. -- A complete self-hosted PDS needs account migration, lifecycle, recovery, and operator UX before production-ready claims. -- Rustler is allowed for repo-core if Elixir libraries cannot meet byte-level requirements. - -## Write Path - -```text -HTTP request - -> XRPC method lookup - -> Lexicon input validation - -> auth - -> context operation - -> SQLite transaction - -> repo commit - -> sequencer event - -> response rendering -``` - -## Adversarial Checks - -- A controller must not build commits directly. -- A repo mutation must not be visible without a matching sequencer event. -- DID, handle, NSID, AT URI, and rkey parsers must reject invalid syntax before storage. -- Every externally fetched URL must pass SSRF checks. -- Every feature must have a black-box HTTP test before it is called done. -- OAuth, app passwords, delegated access, and admin tokens must share one auditable permission boundary. -- Import, restore, and migration flows must fail closed and must never activate partially verified account state. - -## HTTP Verification - -```bash -curl -fsS http://localhost:4000/xrpc/_health -http GET :4000/xrpc/com.atproto.server.describeServer -``` - -Expected once Milestone 01 is complete: - -- `_health` returns JSON. -- `describeServer` returns JSON with the Tempest PDS service metadata. - -## Sources - -- -- -- -- +- `test/smoke/health.hurl` diff --git a/docs/specs/blobs.md b/docs/specs/blobs.md index 660a0d8..760536b 100644 --- a/docs/specs/blobs.md +++ b/docs/specs/blobs.md @@ -1,116 +1,11 @@ --- title: Blobs -updated: 2026-05-08 +updated: 2026-05-31 +status: done --- -# Blobs +Reference documentation: ../reference/blobs.md -Blobs are uploaded separately from records and become public only when a committed record references them. +Verification: -## Endpoints - -Implement: - -```text -com.atproto.repo.uploadBlob -com.atproto.sync.getBlob -com.atproto.sync.listBlobs -``` - -Optional public route for direct serving can follow after `getBlob`. - -## Blob CID - -Generated blob CIDs use: - -```text -CIDv1 -raw codec -sha-256, 256-bit hash -base32 string encoding outside CBOR -``` - -## Lifecycle - -```text -upload - -> validate declared size and MIME - -> sniff MIME where possible - -> compute CID - -> store temp bytes - -> insert metadata - -record write - -> scan record for blob references - -> verify each blob exists for the DID - -> mark referenced blobs public after commit - -record delete - -> recompute references - -> delete blob if no current record references it - -garbage collection - -> delete unreferenced temp blobs after TTL -``` - -## Storage - -Default path: - -```text -/blobs// -``` - -The storage adapter must support: - -- put temp blob. -- promote referenced blob. -- get blob with content length and MIME. -- delete blob. -- list by DID and optional cursor. - -S3-compatible storage is a planned production adapter, not the default: - -- Store blobs under `blobs//`. -- Support private bucket proxying through `getBlob`. -- Optionally support CDN redirects when explicitly configured. -- Keep local metadata authoritative even when bytes live in object storage. - -## Adversarial Checks - -- Reject uploads whose actual size differs from `Content-Length`. -- Reject files larger than the configured limit. -- Do not make temp uploads public. -- Do not delete a blob still referenced by any current record. -- Account deactivation, takedown, suspension, and deletion must suppress blob serving. -- `getBlob` must set a restrictive Content Security Policy and `X-Content-Type-Options: nosniff`. -- Temp blob GC must allow a generous grace period; one hour is the lower bound, several hours is preferred. -- S3/CDN redirects must not expose blobs for inactive, deleted, suspended, or takendown accounts. - -## HTTP Verification - -```bash -printf 'hello blob' > /tmp/tempest-blob.txt - -http --form POST :4000/xrpc/com.atproto.repo.uploadBlob \ - "Authorization:Bearer $TOKEN" \ - Content-Type:text/plain < /tmp/tempest-blob.txt - -http GET :4000/xrpc/com.atproto.sync.listBlobs \ - did==did:plc:example - -curl -fsS -H "Authorization: Bearer $TOKEN" \ - "http://localhost:4000/xrpc/com.atproto.sync.getBlob?did=did:plc:example&cid=bafk..." -``` - -Expected: - -- Upload returns a Lexicon blob object. -- Temp blobs do not appear in `listBlobs`. -- Referenced blobs are downloadable with correct `Content-Type` and `Content-Length`. - -## Sources - -- -- -- +- `test/smoke/blobs.hurl` diff --git a/docs/specs/identity-handles.md b/docs/specs/identity-handles.md index 58938f6..0b31b15 100644 --- a/docs/specs/identity-handles.md +++ b/docs/specs/identity-handles.md @@ -3,16 +3,29 @@ title: Identity and Handles updated: 2026-05-08 --- -Identity is rooted in DIDs. Handles are mutable DNS names that must resolve back to the DID, while the DID document must claim the handle. +Identity is rooted in DIDs. Handles are mutable DNS names that must resolve back to the +DID, while the DID document must claim the handle. ## DID Support Supported methods: -- `did:plc` for normal hosted accounts. -- `did:web` for development and explicit import/migration cases. +- `did:web` for self-hosted identities where the operator controls the domain. +- `did:plc` for identities published to the PLC directory. -Tempest must distinguish invalid DID syntax, unsupported DID method, and supported method resolution failure. +Tempest must distinguish invalid DID syntax, unsupported DID method, and +supported method resolution failure. + +### Hosted identity modes + +Tempest must not mint identifiers that are not externally resolvable. + +- In `did:web` mode, Tempest serves the DID document at `/.well-known/did.json` + and expects handle verification via DNS TXT and/or HTTPS well-known. +- In `did:plc` mode, Tempest publishes PLC operations to `https://plc.directory` + for identity creation and handle/service updates. + +OAuth and relay/AppView interop depend on this being correct. ## DID Document Requirements @@ -93,3 +106,9 @@ Expected: - - - + +## References + +- Cocoon PDS: +- Tranquil PDS: +- Cirrus PDS: diff --git a/docs/specs/lexicon-schemas.md b/docs/specs/lexicon-schemas.md index 056cda8..7f6444f 100644 --- a/docs/specs/lexicon-schemas.md +++ b/docs/specs/lexicon-schemas.md @@ -1,183 +1,12 @@ --- title: Lexicon Schema Loading and Generation -updated: 2026-05-16 +updated: 2026-05-31 +status: done --- -Tempest needs generic Lexicon validation without baking application schemas into -validator code. Record APIs should validate known schemas, tolerate unknown schemas -when the request permits it, and give operators a reproducible way to update the known -schema set. The registry should support bundled generated schemas, operator-configured -local schemas, and policy-controlled network resolution. +Reference documentation: ../reference/lexicon-schemas.md -## Reference Implementation Baseline +Verification: -The TypeScript reference PDS does not dynamically resolve arbitrary record Lexicons today. - -Observed behavior: - -- Canonical `com.atproto.*`, `app.bsky.*`, `chat.bsky.*`, and related Lexicons live in the atproto repository under `lexicons/`. -- Generated TypeScript modules expose those Lexicons to PDS code. -- Repo write preparation keeps a `knownSchemas` map of selected record schemas. -- `validateRecord` uses that map: - - `validate: false` skips schema validation. - - known `$type` validates the rkey and record body against the schema. - - unknown `$type` with `validate: true` fails. - - unknown `$type` with `validate` unset returns `validationStatus: "unknown"`. -- The reference implementation has an explicit TODO to replace the static known schema - map with automatically fetched and built schemas. - -Other references: - -- Cocoon currently does not appear to implement a real Lexicon schema registry for - record writes. It validates basic request fields, writes records, and marks write - responses `valid` with an inline TODO noting that status may not be true. -- Tranquil PDS separates registry, validator, and optional resolver concerns. Its - resolver path supports DNS/DID/PDS lookup, positive and negative caching, - single-flight resolution, response-size limits, stale-cache behavior, and an offline - mode. - -Sources checked: - -- `packages/pds/src/repo/prepare.ts` -- `packages/pds/src/api/com/atproto/repo/createRecord.ts` -- `packages/lexicon/src/lexicons.ts` -- `packages/lexicon/src/validators/complex.ts` -- `packages/lexicon/src/validators/primitives.ts` -- `haileyok/cocoon/server/repo.go` -- `haileyok/cocoon/server/handle_repo_create_record.go` -- `tranquil-pds/crates/tranquil-lexicon/src/registry.rs` -- `tranquil-pds/crates/tranquil-lexicon/src/validate.rs` -- `tranquil-pds/crates/tranquil-lexicon/src/resolve.rs` -- `tranquil-pds/crates/tranquil-lexicon/src/dynamic.rs` - -## Tempest Design - -Tempest should keep three concerns separate: - -1. Lexicon schema engine: generic validation of documents, definitions, refs, unions, - objects, arrays, and primitive constraints. -2. Lexicon registry: lookup boundary for known Lexicon documents. -3. Lexicon sources: bundled, generated, configured, or externally resolved schema - documents. - -The schema engine must not contain product- or app-specific schemas. Bluesky Lexicons -can be used in tests and can be bundled as generated artifacts, but they should remain -data, not branches in validator code. - -The registry exists to answer one runtime question for record writes: given a collection -NSID, can Tempest find a trusted Lexicon document that validates the write? - -The registry should be a deterministic lookup/index boundary over trusted local sources -plus explicitly enabled external sources. It should not silently swallow invalid -documents or duplicate ids at startup/generation time, because operators need failures -that explain why a schema set is unsafe or ambiguous. Externally resolved schemas should -be cached with metadata and should never override bundled or configured local schemas -without an explicit policy. - -## Known Schema Sources - -Tempest should support these sources in order: - -- Bundled generated schemas from an explicit atproto source commit. -- Operator-configured local schema files for custom applications. -- Policy-controlled external Lexicon resolution and caching. - -Bundled and configured local schemas are the deterministic baseline. External resolution -is part of schema completion, but it must be explicitly configured, bounded, cached, and -safe. If external resolution is disabled or unavailable in optimistic validation mode, -unknown schemas still follow reference fail-open behavior. - -Bundled schemas should include metadata: - -```text -source_repo -source_commit -generated_at -document_count -document_ids -``` - -## Generation Pipeline - -The generator should: - -1. Read Lexicon JSON files from a pinned checkout or vendored source directory. -2. Validate each Lexicon document shape. -3. Normalize and index definition refs. -4. Emit deterministic Elixir data or an efficient serialized manifest. -5. Record source metadata. -6. Fail on duplicate document ids or duplicate definition refs. -7. Exclude unrelated files. - -Generated output must be reproducible from the same source commit. - -## External Resolution - -External resolution is part of milestone 09 and must be policy-driven. The -implementation should be behind explicit configuration, use conservative network limits, -and preserve reference validation semantics. - -Policy decisions to settle before enabling it by default: - -- Which NSID authorities can be resolved. -- Whether resolution uses DNS `_lexicon`, DNS `_atproto` fallback, DID documents, - PLC metadata, or another mechanism. -- How to verify schema authorship. -- Cache TTL and invalidation. -- Whether a PDS should allow writes using externally resolved schemas by default. -- How to protect against SSRF, oversized schemas, deep refs, and dependency cycles. - -Resolution should follow the published Lexicon model: derive the authority from the -collection NSID, resolve the authority to a DID, resolve that DID to a PDS service -endpoint, then fetch the `com.atproto.lexicon.schema` record whose rkey is the schema -NSID. Fetched documents must match the requested NSID, use supported Lexicon language -version `1`, pass generic document validation, and respect response-size, redirect, -timeout, address-range, cache, and recursion limits. - -If external resolution is disabled, fails, or cannot find a schema, unknown schemas -should follow the reference behavior: accepted as `unknown` unless `validate: true`. - -## Validation Semantics - -For record writes: - -- `$type` must match `collection`. -- If a known record schema exists, validate rkey against the record key schema. -- If a known record schema exists and validation is not disabled, validate the record body. -- If no schema exists and `validate: true`, reject with `InvalidRequest`. -- If no schema exists and validation is unset, return `validationStatus: "unknown"`. -- If `validate: false`, skip schema validation and omit or return unknown validation - status according to endpoint compatibility needs. - -## Adversarial Checks - -- A schema cannot define refs that escape loader limits or create unbounded recursion. -- External schema fetches must not reach private or local addresses. -- A generated schema bundle must be tied to a source commit so compatibility regressions - are explainable. -- Unknown Lexicon records cannot bypass generic record safety checks such as `$type`, - collection NSID, record key syntax, record size, and CBOR limits. -- A malicious configured Lexicon must not crash validation or exhaust CPU/memory. - -## HTTP Verification - -Record API verification remains the black-box check: - -```bash -http POST :4000/xrpc/com.atproto.repo.createRecord \ - "Authorization:Bearer $TOKEN" \ - repo=alice.test collection=app.bsky.actor.profile rkey=self \ - record:='{"$type":"app.bsky.actor.profile","displayName":"Alice"}' -``` - -Expected with bundled/generated profile schema: - -- Response includes `validationStatus: "valid"`. - -Expected for unknown schema with validation unset: - -- Record can be created and response includes `validationStatus: "unknown"`. - -Expected for unknown schema with `validate: true`: - -- Request fails with `InvalidRequest`. +- `test/tempest/lexicon/` +- `test/smoke/lexicon-schemas.hurl` diff --git a/docs/specs/record-apis.md b/docs/specs/record-apis.md index e3e5dea..f1009df 100644 --- a/docs/specs/record-apis.md +++ b/docs/specs/record-apis.md @@ -1,105 +1,11 @@ --- title: Record APIs -updated: 2026-05-07 +updated: 2026-05-31 +status: done --- -# Record APIs +Reference documentation: ../reference/record-apis.md -Record APIs expose repository reads and writes through XRPC. Handlers should stay thin and delegate validation, storage, commits, and sequencing to contexts. +Verification: -## Endpoints - -Implement: - -```text -com.atproto.repo.createRecord -com.atproto.repo.putRecord -com.atproto.repo.deleteRecord -com.atproto.repo.getRecord -com.atproto.repo.listRecords -com.atproto.repo.describeRepo -``` - -Later: - -```text -com.atproto.repo.applyWrites -com.atproto.repo.importRepo -``` - -## Write Semantics - -`createRecord`: - -1. Authenticate. -2. Validate input against Lexicon. -3. Resolve `repo` to the authenticated DID. -4. Validate collection NSID. -5. Generate rkey when absent. -6. Validate record `$type` and schema when known. -7. Apply compare-and-swap if `swapCommit` is present. -8. Create a repo commit. -9. Store blocks, current record row, and commit row. -10. Insert sequencer event. -11. Return `uri`, `cid`, and commit metadata. - -`putRecord` must support `swapRecord` and `swapCommit`. - -`deleteRecord` must remove the current record from the MST and record index without creating tombstones in the repo. - -## Read Semantics - -- `getRecord` resolves handle or DID and returns the current record. -- `listRecords` scans by collection and supports pagination. -- Deleted records are absent. -- Responses should include CIDs where Lexicon requires them. - -## Atomicity - -A write is complete only when these are durable: - -```text -record block(s) -MST block(s) -signed commit block -record index update -commit metadata -sequencer event -``` - -If this cannot be one SQLite transaction because data spans files, use per-DID write serialization and recovery checks. - -## Adversarial Checks - -- The authenticated actor cannot write to another DID unless a future delegation feature permits it. -- Unknown Lexicon records may be accepted only when validation mode permits it. -- Compare-and-swap conflicts must return `409`. -- Duplicate `createRecord` with the same rkey must fail. -- Pagination cursors must not allow collection escape. - -## HTTP Verification - -```bash -http POST :4000/xrpc/com.atproto.repo.createRecord \ - "Authorization:Bearer $TOKEN" \ - repo=alice.test collection=app.bsky.actor.profile rkey=self \ - record:='{"$type":"app.bsky.actor.profile","displayName":"Alice"}' - -http GET :4000/xrpc/com.atproto.repo.getRecord \ - repo==alice.test collection==app.bsky.actor.profile rkey==self - -http GET :4000/xrpc/com.atproto.repo.listRecords \ - repo==alice.test collection==app.bsky.actor.profile -``` - -Expected: - -- Create returns `at:///app.bsky.actor.profile/self`. -- Get returns the same record and CID. -- List includes the record and stable pagination shape. - -## Sources - -- -- -- +- `test/smoke/records.hurl` diff --git a/docs/specs/repo-core.md b/docs/specs/repo-core.md index 502f600..637cd37 100644 --- a/docs/specs/repo-core.md +++ b/docs/specs/repo-core.md @@ -1,123 +1,14 @@ --- title: Repository Core -updated: 2026-05-08 +updated: 2026-05-31 +status: done --- -Repo-core is the highest-risk subsystem. Treat it as a small storage engine with strict binary compatibility requirements. +Reference documentation: ../reference/repo-core.md -## Required Primitives +Verification: -```text -AT URI parser -NSID parser -Record key parser -TID generator -CID encoding and decoding -DRISL CBOR encoding and decoding -CAR v1 reader and writer -MST insert/get/delete/range -commit creation -commit signing and verification -repo diff generation -block graph traversal -``` - -## Repository Facts - -- Repo paths are `/`. -- Collections are valid normalized NSIDs. -- TIDs are 13-character base32-sortable strings built from a 53-bit UNIX microsecond timestamp and 10-bit clock id. -- Repository format is v3. -- Commits include `did`, `version`, `data`, `rev`, `prev`, and `sig`. -- In v3, `prev` must exist and is usually `null`. -- The commit `rev` is a TID logical clock and must increase per repo. -- Commit signatures are compact 64-byte low-S secp256k1 signatures over SHA-256 of the unsigned DRISL commit bytes. -- CIDs are CIDv1 using DRISL-CBOR (`dag-cbor`, `0x71`) or raw (`0x55`) codecs, sha-256 multihashes, and lowercase `b` base32 strings outside CBOR. -- DRISL-CBOR is deterministic shortest-form CBOR with string-only map keys, bytewise encoded-key map ordering, no indefinite-length items, no floats for atproto data, and only tag 42 for CID links. -- CBOR decoders must enforce input-size, nesting-depth, item-count, collection-length, string-size, and byte-string-size limits. -- CAR v1 files use a DRISL metadata header with `version: 1` and `roots`, followed by varint-length-prefixed `{CID, block bytes}` sections. -- CAR readers and writers must verify that section CIDs match the SHA-256 digest of their block bytes, tolerate arbitrary block order and duplicate blocks, and enforce size/count limits. -- MST shape must be deterministic from current key/value contents. -- MST key depth uses SHA-256 and counts leading zero bits in two-bit chunks. -- MST nodes serialize as DRISL objects with `l` and `e`; entries use `p`, `k`, `v`, and `t`, with byte-string key suffix compression scoped to each node. -- Empty repositories serialize as a single empty MST node. Empty intermediate MST nodes are allowed only to preserve depth levels and must not be pruned between populated nodes. -- Repository export uses CAR v1 with MIME type `application/vnd.ipld.car`. - -## Implementation Boundary - -Repo-core is implemented in pure Elixir. Do not use Rustler for repository primitives unless this spec is explicitly revised. - -Golden tests and official atproto vectors are the compatibility boundary for byte-level work. - -Candidate API: - -```elixir -defmodule Tempest.RepoCore.Engine do - @callback apply_writes(binary(), list(map()), map()) :: - {:ok, - %{ - rev: String.t(), - root_cid: String.t(), - commit_cid: String.t(), - commit_bytes: binary(), - blocks: list({String.t(), binary()}), - ops: list(map()) - }} - | {:error, term()} -end -``` - -## Golden Tests - -Required before record write endpoints are considered complete: - -- TID official syntax examples and monotonic generation. -- CID known bytes to known string. -- DRISL CBOR known object to known bytes. -- CBOR decoder limit and non-canonical encoding rejection cases. -- CAR v1 known bytes round trip and malformed archive rejection. -- MST depth examples from the official spec. -- MST insert/get/delete/range. -- MST deterministic root regardless of insertion order. -- Commit signing and verification. -- Official-shape repo CAR fixture import rooted at a signed commit. -- CAR export/import round trip. -- Repo diff operation inversion against fixtures. - -## Adversarial Checks - -- Decode limits must protect against deep CBOR nesting and oversized objects. -- CAR import must ignore unreferenced blocks. -- CAR import must reject incomplete repo graphs. -- Record key mining must not create unbounded MST node size or depth. -- Commit verification must use the DID document signing key. - -## HTTP Verification - -Repo-core itself is internal, so its black-box check is the first endpoint that depends on it: - -```bash -http POST :4000/xrpc/com.atproto.repo.createRecord \ - "Authorization:Bearer $TOKEN" \ - repo=alice.test collection=app.bsky.actor.profile \ - record:='{"$type":"app.bsky.actor.profile","displayName":"Alice"}' - -http GET :4000/xrpc/com.atproto.sync.getLatestCommit did==did:plc:example -``` - -Expected once repo-core and record writes are implemented: - -- Record creation returns `uri`, `cid`, and commit metadata. -- Latest commit returns a monotonic `rev` and commit CID. - -## Sources - -- -- -- -- -- -- -- -- -- +- `test/tempest/repo_core/` +- `test/tempest/repo_storage_test.exs` +- `test/smoke/records.hurl` +- `test/smoke/car-sync.hurl` diff --git a/docs/specs/security-oauth.md b/docs/specs/security-oauth.md index b112b32..55a0030 100644 --- a/docs/specs/security-oauth.md +++ b/docs/specs/security-oauth.md @@ -5,7 +5,9 @@ updated: 2026-05-08 # Security, OAuth, and Delegated Access -Tempest should not stop at legacy password sessions. OAuth is the primary long-term authz surface for user-facing atproto clients, and the same permission engine should constrain OAuth tokens, app passwords, and delegated account access. +Tempest should not stop at legacy password sessions. OAuth is the primary long-term +authz surface for user-facing atproto clients, and the same permission engine should +constrain OAuth tokens, app passwords, and delegated account access. ## Required OAuth Surface @@ -28,8 +30,13 @@ Required profile behavior: - DPoP is required for token-bound requests. - Authorization responses must include the approved scopes. - Token responses must include and validate the account DID as `sub`. -- Client metadata fetches must use hardened HTTP with SSRF protection, body limits, timeouts, and no unsafe redirects. -- Unknown client metadata must be displayed conservatively; only trusted clients may show rich names/logos. +- The subject DID's document must be externally resolvable and must contain an + `#atproto_pds` service endpoint matching the public URL of this Tempest + instance. +- Client metadata fetches must use hardened HTTP with SSRF protection, body limits, + timeouts, and no unsafe redirects. +- Unknown client metadata must be displayed conservatively; only trusted clients may + show rich names/logos. ## Permission Model @@ -63,7 +70,8 @@ Initial compatibility scopes: - `blob:*/*` - `rpc:?aud=` -Granular permission support should be added behind the same data model, even if the first UI only exposes coarse choices. +Granular permission support should be added behind the same data model, even if the first +UI only exposes coarse choices. ## Account Security @@ -80,7 +88,8 @@ Implement after stable sessions, before calling the PDS production-ready: ## App Passwords -Tempest should support app passwords because they remain part of current atproto usage for bots and CLI tools. +Tempest should support app passwords because they remain part of current atproto usage +for bots and CLI tools. Rules: @@ -89,11 +98,13 @@ Rules: - Names are labels, not credentials. - Scopes must be at least as restrictive as OAuth. - Revocation must be immediate for future requests. -- App passwords must not authorize account deletion, email changes, handle changes, PLC operations, or MFA changes. +- App passwords must not authorize account deletion, email changes, handle changes, + PLC operations, or MFA changes. ## Delegated Access -Delegation allows one account to manage another with explicit permission boundaries. Treat it as a separate credential class, not as shared passwords. +Delegation allows one account to manage another with explicit permission boundaries. +Treat it as a separate credential class, not as shared passwords. Minimum model: @@ -105,15 +116,20 @@ Minimum model: - optional expiry - audit log -Delegates must never gain identity recovery or deletion powers unless a later spec explicitly designs those flows with strong reauth. +Delegates must never gain identity recovery or deletion powers unless a later spec +explicitly designs those flows with strong reauth. ## Adversarial Checks - DPoP nonce failures must not silently fall back to bearer semantics. -- A token must be rejected if the `sub`, authorized PDS, and current DID document do not agree. -- SSRF protections apply to OAuth client metadata, logos, JWKS URIs, DID documents, and handle well-known fetches. -- Partial-scope approval must be represented exactly; do not assume all requested scopes were granted. -- Permission checks must be centralized enough that new XRPC methods cannot accidentally bypass them. +- A token must be rejected if the `sub`, authorized PDS, and current DID document do not + agree. +- SSRF protections apply to OAuth client metadata, logos, JWKS URIs, DID documents, and + handle well-known fetches. +- Partial-scope approval must be represented exactly; do not assume all requested scopes + were granted. +- Permission checks must be centralized enough that new XRPC methods cannot accidentally + bypass them. - Auth and recovery endpoints need independent rate limits. ## HTTP Verification @@ -138,3 +154,9 @@ Expected: - - - + +## References + +- Cocoon PDS: +- Tranquil PDS: +- Cirrus PDS: diff --git a/docs/specs/storage-sqlite.md b/docs/specs/storage-sqlite.md index fc1bf77..44b2521 100644 --- a/docs/specs/storage-sqlite.md +++ b/docs/specs/storage-sqlite.md @@ -1,118 +1,12 @@ --- title: SQLite Storage -updated: 2026-05-07 +updated: 2026-05-31 +status: done --- -# SQLite Storage +Reference documentation: ../reference/storage-sqlite.md -SQLite is the default storage design for a self-hosted Tempest PDS. The official PDS distribution is operationally SQLite-oriented, and the self-host target is one to twenty users. +Verification: -## Data Directory - -Default layout: - -```text -/var/lib/tempest/ - account.sqlite - sequencer.sqlite - repos/ - did_plc_.sqlite - blobs/ - did_plc_/ - - tmp/ - backups/ -``` - -Development and test must use an isolated `TEMPEST_DATA_DIR`. - -## Databases - -### `account.sqlite` - -Owns account, identity, auth, blob metadata, and admin tables. - -Core tables: - -```text -accounts -sessions -app_passwords -signing_keys -handle_verifications -blobs -admin_audit_events -``` - -### `sequencer.sqlite` - -Owns the monotonic PDS-wide event sequence. - -Core table: - -```text -repo_seq -``` - -Fields: - -```text -seq integer primary key autoincrement -did text not null -event_type text not null -rev text -commit_cid text -event_cbor blob not null -created_at text not null -``` - -### Per-DID Repo Databases - -Own records and repo blocks for one hosted account. - -Core tables: - -```text -blocks -records -commits -repo_metadata -``` - -## SQLite Settings - -- Enable WAL mode. -- Set a busy timeout. -- Keep write transactions short. -- Serialize repo mutations per DID. -- Use explicit transaction boundaries for repo writes. -- Use SQLite backup APIs or a maintenance mode for backups. - -## Ecto Boundary - -Use Ecto for `account.sqlite` and simple admin metadata. Repo-core storage may use `ecto_sqlite3`, `Exqlite`, or a small storage adapter if block operations need tighter control. Do not hide byte-level repo invariants behind generic schemas. - -## Adversarial Checks - -- A crashed process must not leave a partially created account with no repo metadata. -- A repo write must not commit without the corresponding sequence event. -- Backup docs must include WAL checkpoint behavior. -- Test databases must never point at the developer's real PDS data directory. - -## HTTP Verification - -```bash -curl -fsS http://localhost:4000/xrpc/_health -http GET :4000/xrpc/_health -``` - -Expected once storage bootstrapping is implemented: - -- Health response includes `dataDir`, `accountDb`, `sequencerDb`, and `writable`. -- The endpoint must not expose secrets or full private paths in production mode. - -## Sources - -- -- -- +- `test/smoke/health.hurl` +- `test/smoke/accounts.hurl` diff --git a/docs/specs/sync-firehose.md b/docs/specs/sync-firehose.md index 97d03b8..ca4f28d 100644 --- a/docs/specs/sync-firehose.md +++ b/docs/specs/sync-firehose.md @@ -1,10 +1,13 @@ --- title: Sync and Firehose -updated: 2026-05-08 +updated: 2026-05-31 +status: implemented --- # Sync and Firehose +Reference documentation: ../reference/sync-firehose.md + Sync exposes repository exports and live event streams. `sequencer.sqlite` is the source of truth for the PDS-wide event cursor. ## Endpoints diff --git a/docs/specs/target-profile.md b/docs/specs/target-profile.md new file mode 100644 index 0000000..72ebeec --- /dev/null +++ b/docs/specs/target-profile.md @@ -0,0 +1,36 @@ +--- +title: Target Profile - Solo Login + Archive +updated: 2026-05-31 +--- + +# Target Profile: Solo Login + Archive + +Tempest's near-term target user experience sits between: + +- a single-user PDS usable by real atproto clients (login, write, read), and +- a personal data appliance (durable custody, export, verify, backup/restore). + +This profile is a single node, one primary operator, and one primary identity. + +## Blocking Capabilities + +Tempest is considered *usable* for this profile when both tracks are true: + +### Track A: Client Login Compatibility + +- Identity is externally verifiable for the chosen hosted DID mode. +- Modern client auth works end-to-end (OAuth/app passwords as required). +- Records, blobs, sync reads, and firehose interoperate with known SDKs/fixtures. + +### Track B: Data Custody + +- Repo export is correct and repeatable. +- Repo verification can detect corruption/missing blocks. +- Backup and restore procedures are documented and exercised. + +## Planned Expanded Capabilities + +- multi-account hosting and community administration +- account migration between PDS instances (service auth, activation sequencing) +- MFA and advanced account recovery UX +- S3/CDN production adapters (beyond local default) diff --git a/docs/specs/xrpc.md b/docs/specs/xrpc.md index a7c2bb0..6188432 100644 --- a/docs/specs/xrpc.md +++ b/docs/specs/xrpc.md @@ -1,106 +1,11 @@ --- title: XRPC HTTP Surface -updated: 2026-05-07 +updated: 2026-05-31 +status: done --- -# XRPC HTTP Surface +Reference documentation: ../reference/xrpc.md -XRPC is the public HTTP contract for the PDS. Tempest should implement the routing and error behavior before deeper PDS features. +Verification: -## Requirements - -- Mount all methods under `/xrpc/`. -- Use `GET` for Lexicon queries and `POST` for Lexicon procedures. -- Parse URL query parameters for query methods. -- Parse JSON bodies for JSON procedures. -- Accept raw byte bodies for blob upload. -- Return JSON errors with `error` and optional `message`. -- Include CORS support where needed for clients. -- Keep XRPC error rendering in one module. - -## Method Registry - -Each method entry should define: - -```text -nsid -kind: query | procedure | subscription -auth: none | bearer | admin -input content type -output content type -handler module/function -known errors -``` - -Initial methods: - -```text -com.atproto.server.describeServer -com.atproto.server.createAccount -com.atproto.server.createSession -com.atproto.server.refreshSession -com.atproto.server.deleteSession -com.atproto.identity.resolveHandle -com.atproto.repo.createRecord -com.atproto.repo.putRecord -com.atproto.repo.deleteRecord -com.atproto.repo.getRecord -com.atproto.repo.listRecords -com.atproto.repo.uploadBlob -com.atproto.sync.getRepo -com.atproto.sync.getLatestCommit -com.atproto.sync.getRepoStatus -com.atproto.sync.subscribeRepos -``` - -## Error Shape - -```json -{ - "error": "InvalidRequest", - "message": "collection is required" -} -``` - -Use the correct HTTP status: - -- `400` for invalid params or body. -- `401` for missing or invalid auth. -- `403` for authenticated but forbidden. -- `404` for unknown XRPC method or resource. -- `409` for compare-and-swap conflicts. -- `500` for unexpected internal errors. -- `502` for upstream service failures. - -## Phoenix Notes - -- The router scope is already aliased with `TempestWeb`; avoid duplicate module prefixes. -- Add a separate `:xrpc` pipeline instead of overloading the browser or generic API pipeline. -- Do not put inline JavaScript in templates. -- Do not use `Phoenix.View`. - -## Adversarial Checks - -- Unknown XRPC methods must not fall through to Phoenix HTML errors. -- `POST` to a query method should fail. -- `GET` to a procedure method should fail. -- Content-type mismatches should fail before handler execution. -- JSON errors should not leak stack traces. - -## HTTP Verification - -```bash -curl -i http://localhost:4000/xrpc/com.atproto.server.describeServer -http POST :4000/xrpc/com.atproto.server.describeServer -curl -i http://localhost:4000/xrpc/com.atproto.unknown.method -``` - -Expected once this spec is implemented: - -- Valid `GET describeServer` returns `200` JSON. -- Invalid method/verb cases return JSON errors. - -## Sources - -- -- +- `test/smoke/xrpc.hurl` diff --git a/docs/tasks/00-foundation.md b/docs/tasks/00-foundation.md index 3bb2671..954bb39 100644 --- a/docs/tasks/00-foundation.md +++ b/docs/tasks/00-foundation.md @@ -5,4 +5,10 @@ specs: - ../specs/storage-sqlite.md --- -Completed [May 7th](../../CHANGELOG.md#2026-05-07) +Completed [May 7, 2026](../../CHANGELOG.md#2026-05-07). + +## Verification + +```bash +hurl --test --variable base_url=http://localhost:4000 test/smoke/health.hurl +``` diff --git a/docs/tasks/01-xrpc-shell.md b/docs/tasks/01-xrpc-shell.md index b0d3976..dc03d4c 100644 --- a/docs/tasks/01-xrpc-shell.md +++ b/docs/tasks/01-xrpc-shell.md @@ -4,4 +4,10 @@ specs: - ../specs/xrpc.md --- -Completed [May 7th](../../CHANGELOG.md#2026-05-07) +Completed [May 7, 2026](../../CHANGELOG.md#2026-05-07). + +## Verification + +```bash +hurl --test --variable base_url=http://localhost:4000 test/smoke/xrpc.hurl +``` diff --git a/docs/tasks/02-accounts-sessions.md b/docs/tasks/02-accounts-sessions.md index e911784..3344d1a 100644 --- a/docs/tasks/02-accounts-sessions.md +++ b/docs/tasks/02-accounts-sessions.md @@ -5,4 +5,16 @@ specs: - ../specs/accounts-auth.md --- -Completed [May 7th](../../CHANGELOG.md#2026-05-07) +Completed [May 7, 2026](../../CHANGELOG.md#2026-05-07). + +## Verification + +```bash +suffix="$(date +%s)" +hurl --test --jobs 1 \ + --variable base_url=http://localhost:4000 \ + --variable account_handle="smoke-${suffix}.test" \ + --variable account_email="smoke-${suffix}@example.com" \ + --variable account_password="correct horse battery staple" \ + test/smoke/accounts.hurl +``` diff --git a/docs/tasks/03-identity-handles.md b/docs/tasks/03-identity-handles.md index ef969bd..ae8e6d8 100644 --- a/docs/tasks/03-identity-handles.md +++ b/docs/tasks/03-identity-handles.md @@ -4,4 +4,19 @@ specs: - ../specs/identity-handles.md --- -Completed [May 7th](../../CHANGELOG.md#2026-05-07) +Completed [May 7, 2026](../../CHANGELOG.md#2026-05-07). + +## Verification + +```bash +suffix="$(date +%s)" +hurl --test --jobs 1 \ + --variable base_url=http://localhost:4000 \ + --variable account_handle="identity-${suffix}.test" \ + --variable account_email="identity-${suffix}@example.com" \ + --variable account_password="correct horse battery staple" \ + test/smoke/identity.hurl +``` + +This milestone covers local identity and handle behavior. Network identity +correctness for hosted DIDs is tracked in Milestone 11. diff --git a/docs/tasks/04-repo-core.md b/docs/tasks/04-repo-core.md index 77a2422..728419f 100644 --- a/docs/tasks/04-repo-core.md +++ b/docs/tasks/04-repo-core.md @@ -4,4 +4,10 @@ specs: - ../specs/repo-core.md --- -Completed [May 7, 2026](../../CHANGELOG.md#2026-05-07) +Completed [May 7, 2026](../../CHANGELOG.md#2026-05-07). + +## Verification + +```bash +mix test test/tempest/repo_core test/tempest/repo_storage_test.exs +``` diff --git a/docs/tasks/05-record-apis.md b/docs/tasks/05-record-apis.md index fdb85d1..5a248a5 100644 --- a/docs/tasks/05-record-apis.md +++ b/docs/tasks/05-record-apis.md @@ -4,6 +4,10 @@ specs: - ../specs/record-apis.md --- -Goal: persist records in per-account repositories and expose repository XRPC reads and writes. +Completed [May 8, 2026](../../CHANGELOG.md#2026-05-08). -Completed [May 8, 2026](../../CHANGELOG.md#2026-05-08) +## Verification + +```bash +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/records.hurl +``` diff --git a/docs/tasks/06-car-sync-reads.md b/docs/tasks/06-car-sync-reads.md index 6ee1735..fdd0a59 100644 --- a/docs/tasks/06-car-sync-reads.md +++ b/docs/tasks/06-car-sync-reads.md @@ -5,6 +5,10 @@ specs: - ../specs/sync-firehose.md --- -Goal: expose repository state through sync read endpoints. +Completed [May 8, 2026](../../CHANGELOG.md#2026-05-08). -Completed [May 8, 2026](../../CHANGELOG.md#2026-05-08) +## Verification + +```bash +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/car-sync.hurl +``` diff --git a/docs/tasks/07-firehose.md b/docs/tasks/07-firehose.md index eaebffa..f4bca6c 100644 --- a/docs/tasks/07-firehose.md +++ b/docs/tasks/07-firehose.md @@ -4,43 +4,13 @@ specs: - ../specs/sync-firehose.md --- -Goal: persist and stream repository, identity, and account events. +Completed [May 8, 2026](../../CHANGELOG.md#2026-05-08). -## Tasks - -- [x] T07-01: Add `sequencer.sqlite` bootstrap. -- [x] T07-02: Add `repo_seq` table and indexes. -- [x] T07-03: Add sequencer insert API. -- [x] T07-04: Insert account and identity events during account creation/update. -- [x] T07-05: Insert commit events during record writes. -- [x] T07-06: Generate CAR slices for commit events. -- [x] T07-07: Add Phoenix.PubSub fanout after durable insert. -- [x] T07-08: Add WebSocket route for `subscribeRepos`. -- [x] T07-09: Encode event stream frames as required by current Lexicon/event stream specs. -- [x] T07-10: Add cursor backfill. -- [x] T07-11: Add event size checks and `tooBig` behavior. -- [x] T07-12: Add MST inversion verification for emitted commit events. -- [x] T07-13: Add `requestCrawl` support for configured relays with rate limiting. -- [x] T07-14: Add integration test for subscribe, write, receive event. -- [x] T07-15: Add restart test proving sequence continuity. -- [x] T07-16: Add durable-tail recovery test for crash between event persistence and fanout. -- [x] T07-17: Add torn-write detection test for sequencer storage. - -## Integration Tests - -- New account emits identity/account/commit events. -- Record write emits commit event. -- Cursor backfill returns missed events. -- Sequence number survives restart. -- Event stream catches invalid commit/event mismatches before fanout. -- Relay crawl requests are bounded and observable. - -## HTTP Verification +## Verification ```bash hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/firehose.hurl ``` -Expected: - -- Subscriber receives a commit event with increasing `seq`. +The smoke test verifies subscription, a repository write, and receipt of a +commit event with increasing `seq`. diff --git a/docs/tasks/08-blobs.md b/docs/tasks/08-blobs.md index f94b58c..7a4bcca 100644 --- a/docs/tasks/08-blobs.md +++ b/docs/tasks/08-blobs.md @@ -4,40 +4,13 @@ specs: - ../specs/blobs.md --- -Goal: upload, reference, serve, and garbage collect blobs. +Completed [May 8, 2026](../../CHANGELOG.md#2026-05-08). -## Tasks - -- [x] T08-01: Add blob metadata table to `account.sqlite`. -- [x] T08-02: Add local blob storage adapter. -- [x] T08-03: Add blob CID calculation. -- [x] T08-04: Add upload size validation. -- [x] T08-05: Add MIME validation/sniffing boundary. -- [x] T08-06: Implement `uploadBlob`. -- [x] T08-07: Scan records for blob references during writes. -- [x] T08-08: Reject new records that reference missing blobs. -- [x] T08-09: Promote temp blobs after successful record commit. -- [x] T08-10: Implement `listBlobs`. -- [x] T08-11: Implement `getBlob`. -- [x] T08-12: Suppress blob serving for inactive accounts. -- [x] T08-13: Add blob garbage collector. -- [x] T08-14: Add integration tests for upload, reference, get, delete. -- [x] T08-15: Add CSP and nosniff headers to `getBlob`. -- [x] T08-16: Add S3-compatible storage behavior and local adapter contract tests. -- [x] T08-17: Add optional CDN redirect behavior with inactive-account suppression. - -## Integration Tests - -- Upload returns Lexicon blob metadata. -- Temp blob is not listed. -- Referenced blob is listed and downloadable. -- Missing blob reference fails. -- Blob remains after restart. -- Blob downloads use defensive content headers. -- S3/CDN mode preserves account-status enforcement. - -## HTTP Verification +## Verification ```bash hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/blobs.hurl ``` + +The smoke test verifies upload, temporary/private state, record reference, +listing, serving, and blob response headers. diff --git a/docs/tasks/09-lexicon-schemas.md b/docs/tasks/09-lexicon-schemas.md index 6bdcb2e..1a96f51 100644 --- a/docs/tasks/09-lexicon-schemas.md +++ b/docs/tasks/09-lexicon-schemas.md @@ -4,55 +4,15 @@ specs: - ../specs/lexicon-schemas.md --- -Goal: load, generate, resolve, and manage known Lexicon schemas without hardcoding application schemas in validation logic. +Completed [May 8, 2026](../../CHANGELOG.md#2026-05-08). -## Tasks - -- [x] T09-01: Define Lexicon registry behaviour and document provider boundary. -- [x] T09-02: Add generic Lexicon document validation, including duplicate ids, duplicate refs, and loader limits. -- [x] T09-03: Add deterministic Lexicon manifest format with source repo, commit, generated timestamp, counts, and document ids. -- [x] T09-04: Add generator task for pinned atproto `lexicons/` input. -- [x] T09-05: Generate bundled known-schema data for selected record schemas and wire it into the runtime registry. -- [x] T09-06: Add operator-configured local Lexicon directory support with startup validation. -- [x] T09-07: Preserve record-write validation modes: known `valid`, optimistic unknown, strict unknown failure, and explicit validation skip. -- [x] T09-08: Add tests for refs, unions, ref cycles, deep refs, oversized schemas, duplicate ids, and duplicate refs. -- [x] T09-09: Document external Lexicon resolution policy, default configuration, and source precedence. -- [x] T09-10: Add external resolver interface behind an explicit config flag. -- [x] T09-11: Implement NSID authority resolution through DNS/DID/PDS `com.atproto.lexicon.schema` records. -- [x] T09-12: Add SSRF, redirect, timeout, response-size, address-range, and recursion protections for the external resolver. -- [x] T09-13: Add positive, negative, stale, and single-flight cache behavior for externally resolved schemas. -- [x] T09-14: Ensure externally resolved schemas cannot override bundled or configured local schemas unless explicitly allowed. -- [x] T09-15: Add compatibility tests against official `com.atproto.*` Lexicons relevant to Tempest, excluding official app.bsky profile/post/follow record schemas. -- [x] T09-16: Add resolver tests for disabled, unknown, success, cache hit, negative cache, oversized response, and private-address rejection paths. -- [x] T09-17: Document operator schema update workflow and add Hurl smoke tests for generated and unknown schema behavior. - -## Integration Tests - -- Generated profile schema validates a profile record. -- Unknown schema with validation unset stores successfully as `unknown`. -- Unknown schema with `validate: true` fails. -- Configured local custom schema validates a custom record. -- Duplicate schema ids fail startup or generation. -- Externally resolved custom schema validates when resolver config permits it. -- External resolver failures return `unknown` in optimistic mode and fail when `validate: true`. -- External resolver refuses private/local network targets and oversized schema responses. - -## CLI Verification +## Verification ```bash mix tempest.lexicon.generate --source ../atproto/lexicons --commit mix test test/tempest/lexicon -``` - -## HTTP Verification - -```bash hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/lexicon-schemas.hurl ``` -Expected: - -- Known generated schema returns `validationStatus: "valid"`. -- Configured external schema resolver returns `validationStatus: "valid"` for a resolvable schema. -- Unknown schema with validation unset returns `validationStatus: "unknown"`. -- Unknown schema with `validate: true` returns `InvalidRequest`. +The smoke test verifies known-schema validation, configured resolver behavior, +optimistic unknown-schema writes, and strict validation failure. diff --git a/docs/tasks/10-compatibility-hardening.md b/docs/tasks/10-compatibility-hardening.md index 49f3572..96444fa 100644 --- a/docs/tasks/10-compatibility-hardening.md +++ b/docs/tasks/10-compatibility-hardening.md @@ -17,7 +17,7 @@ Goal: close protocol gaps and verify behavior against external clients, fixtures - [x] T10-07: Add `app.bsky.actor.getPreferences` and `app.bsky.actor.putPreferences` for private preference migration compatibility. - [x] T10-08: Add XRPC proxy fallback rules for service endpoints that should be proxied, not locally implemented. - [x] T10-09: Add interop fixture test suite. -- [x] T10-10: Add Hurl compatibility smoke tests. +- [/] T10-10: Add Hurl compatibility smoke tests and keep them passing. - [x] T10-11: Add migration/import tests that hand off to Milestone 12. - [x] T10-12: Add abuse cases for oversized records, deep CBOR, invalid CIDs, malformed CAR, and invalid firehose frames. - [x] T10-13: Add external relay/AppView verification notes. @@ -40,3 +40,8 @@ Expected: - Both Hurl smoke tests exit successfully. - Output includes account DID, latest commit, exported CAR size, blob CID, and observed firehose seq. + +Known gap: + +- `test/smoke/tempest_compat.hurl` currently needs behavior alignment for + `com.atproto.sync.requestCrawl`. diff --git a/docs/tasks/11-security-oauth.md b/docs/tasks/11-security-oauth.md index 62357b6..954ac5e 100644 --- a/docs/tasks/11-security-oauth.md +++ b/docs/tasks/11-security-oauth.md @@ -3,6 +3,7 @@ title: Milestone 11 - Security, OAuth, and Delegated Access specs: - ../specs/security-oauth.md - ../specs/accounts-auth.md + - ../specs/identity-handles.md --- Goal: make Tempest usable by modern atproto clients without weakening account security. @@ -26,9 +27,15 @@ Goal: make Tempest usable by modern atproto clients without weakening account se - [ ] T11-15: Add delegated-access schema and revoke flow. - [ ] T11-16: Add auth, reset, OAuth, and app-password rate limits. - [ ] T11-17: Add Hurl smoke tests for OAuth metadata, PAR nonce, token issue, scope enforcement, and revoke. +- [ ] T11-18: Add hosted DID mode configuration (`did:plc` vs `did:web`) and refuse unsupported combinations. +- [ ] T11-19: Implement PLC operation publish flow for `did:plc` hosted accounts (create/update). +- [ ] T11-20: Add identity correctness checks used by OAuth flows (DID resolution + `#atproto_pds` service endpoint match). +- [ ] T11-21: Add Hurl smoke tests for identity correctness (handle -> DID, DID doc -> handle, service endpoint). +- [ ] T11-22: Document operator workflows for identity troubleshooting (what to check, what to rotate, what to republish). ## Integration Tests +- Identity correctness checks detect mismatched DID documents and service endpoints. - OAuth client can complete an authorization-code flow. - DPoP-bound tokens fail without valid DPoP proof. - Partial scopes are enforced exactly. diff --git a/docs/tasks/13-admin-deployment.md b/docs/tasks/13-admin-deployment.md index dce5f61..ff9a42f 100644 --- a/docs/tasks/13-admin-deployment.md +++ b/docs/tasks/13-admin-deployment.md @@ -7,26 +7,27 @@ specs: Goal: make Tempest deployable and maintainable as a single-node self-hosted PDS. -## Tasks - - [ ] T13-01: Add release configuration. - [ ] T13-02: Add Dockerfile. - [ ] T13-03: Add docker-compose example. - [ ] T13-04: Add Caddy reverse proxy example. - [ ] T13-05: Add production env template. -- [ ] T13-06: Add admin token hashing and verification. -- [ ] T13-07: Add admin status endpoint. - [ ] T13-08: Add `mix pds.repo.verify`. - [ ] T13-09: Add `mix pds.repo.export`. -- [ ] T13-10: Add `mix pds.repo.import`. -- [ ] T13-11: Add `mix pds.sequencer.status`. - [ ] T13-12: Add `mix pds.blob.gc`. - [ ] T13-13: Add backup create/restore docs. +- [ ] T13-18: Add restore drill that validates accounts, sequencer, repos, blobs, and keys together. - [ ] T13-14: Add Hurl smoke test for deployed HTTPS target. + +--- + +- [ ] T13-06: Add admin token hashing and verification. +- [ ] T13-07: Add admin status endpoint. +- [ ] T13-10: Add `mix pds.repo.import`. +- [ ] T13-11: Add `mix pds.sequencer.status`. - [ ] T13-15: Add telemetry events for XRPC, repo writes, blobs, and firehose. - [ ] T13-16: Add SMTP env and verification for account email flows. - [ ] T13-17: Add S3 backup configuration docs. -- [ ] T13-18: Add restore drill that validates accounts, sequencer, repos, blobs, and keys together. ## Integration Tests diff --git a/docs/tasks/14-operator-account-ux.md b/docs/tasks/14-operator-account-ux.md index b0ec79b..01ff237 100644 --- a/docs/tasks/14-operator-account-ux.md +++ b/docs/tasks/14-operator-account-ux.md @@ -8,12 +8,18 @@ specs: Goal: give self-hosters enough UI and operational tooling to run the PDS safely. -## Tasks +This slice exists to increase development observability: - [ ] T14-01: Add minimal authenticated account dashboard. +- [ ] T14-04: Add repo browser for the authenticated account. +- [ ] T14-14: Add blob browser (temp/public state + download links + headers summary). +- [ ] T14-15: Add sequencer viewer (tail `repo_seq` + filters by DID/type). +- [ ] T14-16: Add firehose viewer (subscribe + decode frames for display). + +--- + - [ ] T14-02: Add UI for sessions, OAuth grants, app passwords, and delegated access. - [ ] T14-03: Add UI for email, password, MFA, backup codes, and trusted devices. -- [ ] T14-04: Add repo browser for the authenticated account. - [ ] T14-05: Add admin dashboard for account status, sequencer, storage, and relay crawl status. - [ ] T14-06: Add invite-code management UI. - [ ] T14-07: Add one-click repo verify from admin UI. diff --git a/docs/tasks/README.md b/docs/tasks/README.md index 3e7f503..c13c1cf 100644 --- a/docs/tasks/README.md +++ b/docs/tasks/README.md @@ -20,7 +20,30 @@ title: Milestone Tasks 14. [13 Admin and Deployment](./13-admin-deployment.md) 15. [14 Operator and Account Management UX](./14-operator-account-ux.md) -Each file in this directory is a milestone. Each task is intended to be the smallest useful unit of work: one focused implementation change, test, or integration check. +Each file in this directory is a milestone. Each task is intended to be the +smallest useful unit of work: one focused implementation change, test, or +integration check. + +## Target Profile Priority Map (Solo Login + Archive) + +Blocking work for the current target profile (see `docs/specs/target-profile.md`): + +- Compatibility stays green: `test/smoke/tempest_basic.hurl` and + `test/smoke/tempest_compat.hurl` must pass. +- Identity is externally verifiable for the chosen hosted DID mode. +- Modern client auth works end-to-end (OAuth/app passwords + rate limits) with + black-box Hurl coverage. +- Data custody exists as operator workflows: repo verify/export, backup/restore, + and a restore drill. +- Minimal operator UI exists for inspection and "play around" testing (account + dashboard + repo/blob/sequencer visibility). + +Deprioritized behind the above for this profile: + +- migration and lifecycle between PDS instances (Milestone 12) +- MFA and advanced account/security UX (parts of Milestone 11/14) +- hosted-provider scale features + ## Status Labels -- 2.54.0