diff --git a/README.md b/README.md index 1b82478..4f36ecd 100644 --- a/README.md +++ b/README.md @@ -10,42 +10,24 @@ protocol primitives: syntax, tids, did resolution, jwt helpers, dag-cbor, car, mst, repo verification, and key encoding. WebAuthn support comes from [`webauthn`](https://tangled.org/zzstoatzz.io/webauthn). -## dependency posture - -ZDS is an application. Dependencies are chosen around that boundary: - -- `zat` is the sibling library we maintain for reusable AT Protocol primitives. - If ZDS grows a generally useful atproto parser, codec, verifier, resolver, or - client helper, expect it to move upstream into `zat`. -- `httpz` is the HTTP/1.1 server boundary. ZDS should consume it directly from - Tangled, not vendor it. HTTP server behavior should stay local to ZDS unless - it is a focused compatibility fix or improvement for `httpz` itself. -- `zat` and `httpz` both rely on the canonical first-party - [`websocket.zig`](https://tangled.org/zzstoatzz.io/websocket.zig). Keep the - graph on one websocket implementation; fix the upstream package we control - rather than copying it into this repo. -- Vendoring is a last resort for short-lived debugging only. Do not leave - `vendor/` as the dependency strategy. +## docs -See [development](docs/development.md#dependency-boundary) for the longer -decision rules. +Start with the document that matches the work: -## docs +- Operators: [operator guide](docs/operations.md), + [production deployment](docs/deployment.md), [invite codes](docs/invite-codes.md), + [Comail](docs/comail.md), and the + [account takedown runbook](docs/account-takedown-runbook.md) +- Account work: [create-account flow](docs/create-account-flow.md), + [account security](docs/account-security.md), and [passkeys](docs/passkeys.md) +- Protocol work: [architecture](docs/architecture.md), + [permissioned data](docs/permissioned-data.md), [getRepo notes](docs/getrepo-notes.md), + and [ecosystem references](docs/references.md) +- Engineering: [development](docs/development.md), [benchmarks](bench/README.md), + and [comparison notes](docs/benchmarking.md) -- [architecture](docs/architecture.md) -- [account management](docs/account-security.md) -- [comail](docs/comail.md) -- [development](docs/development.md) -- [getRepo notes](docs/getrepo-notes.md) -- [operator guide](docs/operations.md) -- [account takedown runbook](docs/account-takedown-runbook.md) -- [invite codes](docs/invite-codes.md) -- [passkeys](docs/passkeys.md) -- [permissioned data](docs/permissioned-data.md) -- [attested payments and write delegation](docs/attested-payments-and-write-delegation.md) -- [references](docs/references.md) -- [benchmarks](bench/README.md) -- [benchmark comparison notes](docs/benchmarking.md) +Exploratory protocol notes live in `docs/` beside the durable guide they inform; +they are labeled as such and are not implementation contracts.
run locally @@ -61,6 +43,8 @@ before committing: ```sh just test just smoke +just smoke-permissioned # when permissioned-data behavior changes +git diff --check zig zen ``` @@ -95,8 +79,8 @@ zig build run -- \ --server-did did:web:pds.example.com ``` -See the [operator guide](docs/operations.md) for deployment, configuration, -invite-code, migration, and release notes. +See the [operator guide](docs/operations.md) for configuration and the +[deployment runbook](docs/deployment.md) for the production instance.
@@ -153,6 +137,15 @@ just docker-publish-release v0.1.1 `com.atproto.simplespace.*`, but the upstream proposal is still moving and this surface is not a stable compatibility contract. +## dependency posture + +ZDS is an application. It consumes [`zat`](../zat) for reusable AT Protocol +primitives and `httpz` for its HTTP server boundary. Both use the canonical +first-party [`websocket.zig`](https://tangled.org/zzstoatzz.io/websocket.zig); +keep the build graph on one websocket implementation. Do not vendor or patch +dependency internals in this repository. See +[development](docs/development.md#dependency-boundary) for the full boundary. + ## references - [Tranquil PDS](https://tangled.org/tranquil.farm/tranquil-pds) diff --git a/docs/account-takedown-runbook.md b/docs/account-takedown-runbook.md index c917657..f2e0b39 100644 --- a/docs/account-takedown-runbook.md +++ b/docs/account-takedown-runbook.md @@ -258,43 +258,15 @@ For non-test accounts, prefer a two-step process: This keeps the emergency action reversible while still stopping local hosting. -## current known test account - -As of 2026-06-24, the likely disposable private-media test account on -`pds.zat.dev` is: - -- handle: `plyr-priv-test.pds.zat.dev` -- DID: `did:plc:dm3jejwohil763nmotc6cu2m` -- email: `plyr-priv-test@example.com` - -Recent read-only footprint: - -```text -records: 0 -repo_blocks: 0 -commits: 0 -blobs: 3 -spaces_owned: 1 -space_actor_state: 1 -space_records: 1 -space_repos: 1 -session_tokens: 6 -oauth_tokens: 0 -``` - -Treat this as a candidate for testing the runbook. Do not use it as precedent -for deleting a real user's account without a first-class admin action and audit -trail. - -## implementation backlog - -ZDS should eventually add: - -- explicit admin account-status endpoint or CLI -- distinct statuses: `deactivated`, `suspended`, `takendown`, `deleted` -- admin audit log entries for status changes -- token revocation tied to operator actions -- dry-run account footprint command -- reviewed purge tool with table-by-table output -- tests for `#account` status events and `getRepoStatus` behavior for each - status +## remaining manual boundary + +ZDS has first-class user deactivation and operator takedown, including token +revocation, repo-status responses, and `#account` events. It does not expose a +one-step physical purge command. Purge remains a separately reviewed retention +operation because public repo state, permissioned repos, blobs, credentials, +audit state, and downstream copies have different ownership and recovery +semantics. + +Do not keep live account identifiers or database footprints in this runbook. +Capture incident-specific evidence in the operator's private record, then use +the investigation and verification steps above. diff --git a/docs/create-account-flow.md b/docs/create-account-flow.md new file mode 100644 index 0000000..db65bd4 --- /dev/null +++ b/docs/create-account-flow.md @@ -0,0 +1,82 @@ +# create-account flow + +Status: implementation handoff. This document describes the current boundary +for someone building a resident-facing account creation flow. It separates +facts about ZDS from product choices the next engineer still needs to make. + +## current server surface + +ZDS already implements the protocol account-creation backend: + +- `com.atproto.server.describeServer` advertises hosted handle suffixes and + whether an invite is required. +- `com.atproto.server.reserveSigningKey` reserves an account signing key. +- `com.atproto.server.createAccount` accepts `handle`, `email`, `password`, + `inviteCode`, and optional `signingKey`. +- A normal new account gets a signing key, a PLC genesis operation, durable + account/repo state, and an initial access/refresh session. +- Migration creation for an existing DID is a distinct, service-authenticated + path used by tools such as PDS Moover. +- Invite validation and consumption occur in the same SQLite transaction as + account creation. + +The production instance advertises `.pds.zat.dev`, requires an invite, sends +mail through Comail, and has wildcard DNS/TLS for hosted account handles. + +There is no resident-facing create-account page today. The root page, OAuth +login, and `/account` management hub are server-rendered in +`src/internal/account.zig`; XRPC dispatch is in `src/atproto/server.zig`, route +metadata is in `src/http/router.zig`, and account/invite persistence is in +`src/storage/store.zig`. + +## invariants for a UI + +A UI should be a client of the existing XRPC behavior, not a second account +creation implementation. In particular: + +- Fetch `describeServer`; do not hard-code invite or handle-domain policy. +- Submit the complete handle selected by the user and enforce one advertised + suffix. Do not infer that the bare PDS hostname is an account suffix. +- Never log or retain the account password, invite code, session tokens, or PLC + key material in browser-visible diagnostics. +- Preserve the server's XRPC error code and message so invalid invites, handles, + existing accounts, PLC failures, and transient server failures are not all + rendered as the same error. +- Treat a successful response as a newly authenticated session and hand it to + the resident account experience deliberately. +- Keep migration out of the normal sign-up form. It has different identity and + authorization semantics. + +Account creation is security-sensitive and mutates PLC plus local persistent +state. Add an end-to-end smoke case for the browser flow rather than relying +only on DOM tests. Use a disposable local database for destructive tests and a +fresh invite for each attempted creation. + +## product questions + +These are intentionally not answered by the server implementation: + +- whether the public page asks for a full handle or a short name plus an + operator-selected suffix +- how an invite is delivered and explained +- whether email verification is requested immediately after creation +- whether passkey registration is offered after the password-backed account is + established +- what recovery and migration language is appropriate for an experimental PDS + +The existing `/account` pages establish the visual and accessibility baseline. +Account creation should feel like the beginning of that resident workflow, not +an unrelated marketing page. + +## checks and references + +Before changing account semantics, compare the current AT Protocol account +lexicons and both mature references listed in [references](references.md). +Relevant local guides are [invite codes](invite-codes.md), +[account security](account-security.md), [passkeys](passkeys.md), +[Comail](comail.md), and [production deployment](deployment.md). + +At minimum, retain the existing `just smoke` coverage and add browser-flow +coverage for validation errors and successful creation. Browser-reported +failures require the actual request, response, and server log evidence before +the server behavior is changed. diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..f6295bb --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,118 @@ +# production deployment + +This runbook describes the instance used to exercise ZDS with real resident +accounts and interoperability clients. + +## production shape + +| setting | value | +| --- | --- | +| public origin | `https://pds.zat.dev` | +| Fly app | `zds-pds` | +| primary region | `ord` | +| machine | one shared CPU, 1 GiB RAM | +| persistent data | Fly volume `zds_data` mounted at `/data` | +| database | `/data/zds.sqlite3` | +| blobstore | `/data/blobs` | +| hosted handles | `*.pds.zat.dev` | +| account policy | invite required | +| permissioned data | enabled, experimental | + +The committed non-secret configuration is in [`fly.toml`](../fly.toml). +Secrets are Fly secrets and must not be copied into source, logs, or issue +reports. The repository `.env` is ignored local operator state; commands that +need one of its values should source it without printing it. + +SQLite and blob bytes live on the mounted volume. A new application image must +reuse that volume. Never create a replacement machine with an empty `/data` and +send production traffic to it as a deployment shortcut. + +## normal deployment + +Every push to `main` runs [`.tangled/workflows/deploy.yml`](../.tangled/workflows/deploy.yml), +which executes: + +```sh +flyctl deploy --remote-only +``` + +The workflow owns `FLY_API_TOKEN`. A normal release is therefore: + +1. Confirm the worktree is the canonical `main` checkout and is based on + `origin/main`. +2. Run the required checks from the repository root. +3. Commit and push `main`. +4. Watch the Tangled workflow and verify the live service after Fly replaces + the machine. + +Required checks: + +```sh +just test +just smoke +git diff --check +zig zen +``` + +Also run `just smoke-permissioned` for permissioned-data storage, OAuth space +scope, `com.atproto.space.*`, `com.atproto.simplespace.*`, or `/account/spaces` +changes. Run the relevant `just bench ...` lane for performance-sensitive +storage, repo, blob, proxy, or permissioned-data work. + +## manual deployment + +The manual path is for an operator already authenticated to the Fly app: + +```sh +flyctl auth whoami +flyctl deploy --remote-only --app zds-pds +``` + +Do not create a second configuration path in shell history. `fly.toml` remains +the source of non-secret deployment configuration, and Fly secrets remain the +source of secret configuration. + +## verification + +Capture the current health response before deploying so there is a baseline: + +```sh +curl -fsS https://pds.zat.dev/xrpc/_health | jq . +curl -fsS https://pds.zat.dev/xrpc/com.atproto.server.describeServer | + jq '{did,availableUserDomains,inviteCodeRequired}' +``` + +After deployment: + +```sh +flyctl status --app zds-pds +flyctl logs --app zds-pds --no-tail +curl -fsS https://pds.zat.dev/xrpc/_health | jq . +curl -fsS https://pds.zat.dev/xrpc/com.atproto.server.describeServer | + jq '{did,availableUserDomains,inviteCodeRequired}' +curl -fsS https://pds.zat.dev/api/openapi.json >/dev/null +curl -fsS https://pds.zat.dev/account >/dev/null +``` + +Do not stop at an HTTP 200 from the health route when the change touched a +resident workflow. Exercise the affected path with a throwaway account or the +appropriate smoke client, then inspect recent logs for panics, repeated 5xx +responses, migration failures, proxy timeouts, or memory pressure. Never print +passwords, admin tokens, OAuth tokens, PLC keys, or mail credentials while +probing. + +## rollback and recovery + +Application rollback and data rollback are different operations. Re-deploying +an older image does not undo a SQLite migration, and restoring SQLite without +its corresponding blobstore can produce an incoherent account. + +Before a risky storage migration, confirm that a current volume snapshot exists +and document the exact application revision and schema transition. If a release +fails without changing persistent state, deploy the last known-good source +revision through the same Fly path. If persistent state changed, stop and assess +the migration and volume snapshot together before rolling either side back. + +For ordinary account operations after a healthy deployment, use the +[operator guide](operations.md), [invite-code guide](invite-codes.md), and +[account takedown runbook](account-takedown-runbook.md). diff --git a/docs/development.md b/docs/development.md index 490356c..16bd948 100644 --- a/docs/development.md +++ b/docs/development.md @@ -101,10 +101,14 @@ The root `justfile` is the stable task surface: ```sh just test just smoke +just smoke-permissioned just invite https://pds.zat.dev just plc-repair did:plc:... just bench all just docker-publish-current ``` -Run `zig zen` before committing. +Run the permissioned smoke only when that surface changes. Always run +`git diff --check` and `zig zen` before committing. See the +[production deployment runbook](deployment.md) before pushing changes that +will deploy to `pds.zat.dev`. diff --git a/docs/operations.md b/docs/operations.md index be61445..79c24ae 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -1,8 +1,9 @@ # operator guide -This is the operator-facing guide for running ZDS: local startup, deployment -configuration, containers, invite-code administration, account migration notes, -and release procedure. +This is the operator-facing guide for running ZDS: local startup, +configuration, containers, invite administration, account migration, and +release policy. The concrete `pds.zat.dev` Fly workflow is documented in the +[production deployment runbook](deployment.md). For account suspension, takedown, and test-account cleanup, see the [account takedown runbook](account-takedown-runbook.md). ZDS supports temporary @@ -16,6 +17,7 @@ Run these before committing: ```sh just test just smoke +git diff --check zig zen ``` @@ -23,6 +25,9 @@ zig zen checks blob upload, verifies repo/sync endpoints, and asserts that known app records use valid key shapes. +Run `just smoke-permissioned` as well when changing permissioned-data routes, +storage, OAuth space scopes, or the resident space browser. + ## benchmarks Local store benchmarks live under `bench/`: diff --git a/docs/permissioned-data.md b/docs/permissioned-data.md index ab395fc..abbd05b 100644 --- a/docs/permissioned-data.md +++ b/docs/permissioned-data.md @@ -34,6 +34,8 @@ even when an operator has not enabled it. - Local alignment notes: [permissioned-data proposal 94](permissioned-data-proposal-94.md) +- Migration and CAR conformance experiments: + [space-host migration](space-host-migration.md) The upstream design is still moving. The May branch included protocol-level member lists. The later discussion moved toward making space credentials the diff --git a/docs/session-handoff-2026-07-16.md b/docs/session-handoff-2026-07-16.md deleted file mode 100644 index abd7194..0000000 --- a/docs/session-handoff-2026-07-16.md +++ /dev/null @@ -1,379 +0,0 @@ -# session handoff: zds exploration through 2026-07-16 - -This handoff summarizes the long ZDS work session that moved the project from a -small experimental PDS into a more serious, operator-run sandcastle with OAuth, -account management, permissioned-data experiments, better sync behavior, and -benchmark coverage. - -It is intentionally descriptive. Treat it as context for the next engineer, not -as a command queue. Re-read the current protocol docs, ZDS source, Tranquil, -reference PDS, and `zat` before changing semantics. - -## current state - -- Branch: `main` -- Remote status at handoff time: clean against `origin/main` -- Current head: `fd0c1af Align space listRecords value shape` -- Latest deployed target during the session: `pds.zat.dev` / Fly app `zds-pds` -- Latest known deployed Fly machine after `fd0c1af`: version `226`, region `ord` -- Permissioned data is operator gated with `ZDS_PERMISSIONED_DATA=true`. -- Invite-code admin assumes `ZDS_ADMIN_TOKEN` is in the repo `.env`; use: - -```sh -set -a; . ./.env; set +a; just invite -``` - -Before committing code changes, run: - -```sh -just test -just smoke -git diff --check -zig zen -``` - -Also run `just smoke-permissioned` when touching `com.atproto.space.*`, -permissioned-data storage, or `/account/spaces`. - -## posture learned the hard way - -- Start from protocol text, then compare against reference PDS, Tranquil, - Pegasus, and local notes. Do not invent PDS behavior from vibes. -- ZDS owns ZDS. Do not edit sibling `zat` from this repo without explicit - approval. If a primitive clearly belongs in `zat`, write down the desired API - and let the `zat` owner decide. -- Do not vendor or patch dependency internals as a convenience. Install pinned - dependencies normally. -- Avoid compatibility hedging for old ZDS-only experimental shapes. There are - no broad external users to preserve accidental legacy for. If an experimental - protocol shape moves, remove the old shape rather than keeping aliases unless - the user explicitly asks for a transition window. -- Browser-visible failures need request/response evidence, not intuition. -- Benchmarks should compare equivalent work only. Keep adjacent probes adjacent, - not mixed into the same table. - -## major shipped areas - -### Operator configuration and docs - -- Added/cleaned operator docs under `docs/operations.md`. -- Documented invite-code behavior in `docs/invite-codes.md`. -- Added environment-variable support for port and DB path. -- Adopted SemVer tags; patch releases were cut for operator-visible fixes. -- README references now include `haileyok/cocoon`. - -### Email delivery - -- Added pluggable email-provider structure. -- Implemented Comail as the default provider. -- Documented Comail in `docs/comail.md`. -- Important deployment details: - - `ZDS_MAIL_PROVIDER=comail` - - `ZDS_COMAIL_API_KEY` - - `ZDS_COMAIL_DID` - - `ZDS_EMAIL_FROM` must be a bare email address. -- Verified account email flow against Bluesky UI after configuring Fly secrets. - -### Account, sessions, and resident UX - -- Added `/account` as the resident-facing hub. -- Folded security/session/app-password/passkey/account status surfaces into the - account hub. -- Improved the resident sessions UI after several UX passes. -- Added operator/admin account-session visibility. -- Added account status/takedown support and runbook docs: - `docs/account-takedown-runbook.md`. -- Important semantic point: ZDS now models more than active/deactivated, but do - not claim full support for protocol statuses unless storage, eventing, and - API responses all represent the distinction. - -### OAuth and auth hardening - -- Implemented and hardened ATProto OAuth/DPoP behavior. -- Added stateless DPoP nonce behavior using a shared secret - (`ZDS_DPOP_SECRET`, falling back to `ZDS_JWT_SECRET`). -- Hardened token families and refresh behavior. -- Added discriminating OAuth logging for invalid grant/code paths. -- Fixed several OAuth migration-order and token-row bugs. -- Verified behavior against atproto.com expectations and compared with - reference PDS/Tranquil while doing the DPoP work. - -### Repo and sync correctness - -- Fixed large `applyWrites` request bodies by moving off the small fixed buffer - path and mapping oversize bodies correctly. -- Fixed firehose `#commit.since` to use previous rev semantics rather than the - previous commit CID. Mia reported this; it was vetted against the sync spec - and Daniel Holmgren's synchronization draft before fixing. -- Fixed sync event migrations and bounded sync event rebuild memory. -- Fixed repo block migration order and import/export reachability. -- Added stricter full-CAR import behavior; incomplete repo CARs are rejected. -- Fixed `getRepo` behavior: - - full repo export reachability - - `HEAD` response path - - notes moved to `docs/getrepo-notes.md` -- Adopted `zat.signCommit` for repo commit signing after `zat` released the - helper. - -### Dependency graph and release posture - -- Moved ZDS onto the first-party canonical `http.zig` / `websocket.zig` / - `zat` dependency graph. -- Avoid resurrecting old Karl/httpz/websocket forks. -- The relevant dependency lesson: if `zat`, `httpz`, and ZDS each see different - websocket packages, Zig module identity breaks. Keep the graph on one - canonical websocket release. -- The local rule is documented in README and development docs: ZDS is an - application and should consume `zat`/`httpz`; broadly useful protocol - primitives can move upstream only after being made explicit. - -### Performance and benchmarks - -- Added benchmark coverage in `bench/`, with docs in `bench/README.md`. -- Added focused `getRepo` export benchmark. -- Added public repo read/write, sync, blob, HTTP route, and permissioned-data - benchmark slices. -- Fixed obvious repo-write performance problems by adopting lazy MST loading. -- Added apples-to-apples notes comparing ZDS, Tranquil, and official PDS where - equivalent measurements exist. -- Do not put unlike operations in the same comparison table. Example: - Tranquil index-only list rows are not the same unit as ZDS full - materialization. - -### Stats and health UI - -- Added a `/stats` health/latency page. -- Iterated on it after user feedback: - - do not count the stats page's own refreshes as user-facing API traffic - - avoid noisy not-found rows - - separate appview/proxy latency from local PDS behavior - - make “needs attention” more legible -- Open desire: persistent/historical stats that survive deploys. Current stats - remain in-process. - -## permissioned data work - -This was the biggest exploratory arc. Current docs: - -- `docs/permissioned-data.md` -- `docs/permissioned-data-proposal-94.md` -- `bench/README.md#permissioned-data` - -### Overall direction - -- Permissioned data is explicitly experimental and behind - `ZDS_PERMISSIONED_DATA`. -- The project intentionally tracks the current proposal direction instead of - preserving old local shapes. -- Protocol routes live under `com.atproto.space.*`. -- Baseline PDS-managed space management lives under - `com.atproto.simplespace.*`. -- Old protocol member-list routes were removed. The later proposal/discussion - direction pushes rich reader/group semantics into applications or space-host - policy instead of making a universal protocol member list. - -### Current implemented surface - -Protocol/data surface: - -- `com.atproto.space.getSpace` -- `com.atproto.space.listSpaces` -- `com.atproto.space.listRepos` -- `com.atproto.space.getDelegationToken` -- `com.atproto.space.getSpaceCredential` -- `com.atproto.space.createRecord` -- `com.atproto.space.putRecord` -- `com.atproto.space.deleteRecord` -- `com.atproto.space.applyWrites` -- `com.atproto.space.getRecord` -- `com.atproto.space.listRecords` -- `com.atproto.space.getBlob` -- `com.atproto.space.getLatestCommit` -- `com.atproto.space.listRepoOps` -- `com.atproto.space.notifyWrite` -- `com.atproto.space.notifySpaceDeleted` - -Baseline management surface: - -- `com.atproto.simplespace.createSpace` -- `com.atproto.simplespace.updateSpace` -- `com.atproto.simplespace.deleteSpace` -- `com.atproto.simplespace.addMember` -- `com.atproto.simplespace.removeMember` -- `com.atproto.simplespace.listMembers` - -### Access model - -- ZDS treats permissioned data as private writer repos plus space credentials. -- Application-level access semantics stay above the PDS. Examples: supporter - access, label rosters, private subscriptions, follower-only spaces, group - roles. -- `simplespace` membership is only baseline policy state for PDS-managed - spaces, not a protocol sync surface. -- `managing-app` policy calls the configured `checkUserAccess` service with - authority service auth and fails closed on resolution or response errors. -- `appAccess` supports `open` and `allowList`; allow-list decisions use a - separately verified client attestation rather than a delegation-token claim. -- Space roots now use the canonical - `at://{authorityDid}/space/{spaceType}/{skey}` syntax. The old `ats://` - experiment is migrated in storage and rejected at the HTTP boundary. -- OAuth grants use proposal-shaped `authority`, `action`, and `manage` - parameters; `authority=self` is resolved to the resident DID at issuance. -- `listRepos` reads a distinct authority-owned writer registry populated from - `notifyWrite` revision and commit-digest summaries. It does not expose the - repo host's raw LtHash state. -- Deniable commits now use the proposal context and AT JSON byte encoding. -- Delegation tokens and cryptographically verified client attestations are - short-lived, one-use inputs consumed atomically during credential exchange. - -### Blob handling - -- Permissioned record blob refs are tracked separately from public record blob - refs. -- Blob bytes are still stored once in the author's PDS blobstore. -- A blob referenced only by a permissioned record must not become public via - `com.atproto.sync.getBlob`. -- Permissioned blob reads go through `com.atproto.space.getBlob(space, repo, - cid)` because the space is the auth context. -- This matches Daniel's explanation: author uploads blob to author PDS, creates - a permissioned record referencing it, and authorized syncers fetch the blob - from the author's PDS through the space-authenticated path. - -### `listRecords` value shape - -- Latest commit `fd0c1af` aligned `com.atproto.space.listRecords` with the - proposal: records include `value` by default. -- `excludeValues=true` returns metadata-only rows with collection/rkey/CID. -- This was motivated by the sibling Racine project - (`~/tangled.org/zzstoatzz.io/racine`), which was forced into an N+1 - hydration pattern: `listRecords` then many `getRecord` calls. -- Live production probe after deploy confirmed: - - default `listRecords` includes `value` - - `excludeValues=true` omits `value` - - cursor shape is unchanged - -### Permissioned-data follow-up audit - -These are not necessarily bugs; they are places where the evolving proposal may -expect more than current ZDS provides. - -- `listRepoOps` now inlines current values by default and supports - `excludeValues=true`, matching the proposal's sync shape. -- ZDS implements the proposal's full-state permissioned `getRepo` CAR: deniable - signed-commit root, DAG-CBOR index root, then lexicographically ordered record - blocks. Daniel's branch exposes the lexicon but its handler is still a stub. -- `registerNotify` supports expiring whole-space and repo-scoped subscriptions, - matching the current lexicon and branch's 24-hour lifetime. -- Space credential/delegation-token details should keep tracking proposal - changes, especially `typ`, `aud`, `sub`, and dedicated space DID material. -- The generic PDS `checkUserAccess` endpoint deliberately denies; a configured - managing app supplies the application-specific authorization decision. - -## sibling projects touched or used as forcing functions - -### plyr.fm - -Plyr.fm was the first permissioned-media adopter and drove much of the -permissioned-data work: - -- private media should upload blobs to the user's PDS with normal - `com.atproto.repo.uploadBlob` -- records should live in a permissioned space -- playback should fetch through `com.atproto.space.getBlob` -- OAuth scope expansion and permission-set handling exposed several ZDS auth - bugs -- token lifetime/refresh behavior was hardened because plyr sessions died after - roughly an hour - -There is an issue in `zzstoatzz/plyr.fm` tracking the minimum viable private -media shape. Its body/comments were updated during the session. Re-read it -before changing ZDS behavior that plyr depends on. - -### racine - -Racine is a small family-tree app at -`~/tangled.org/zzstoatzz.io/racine`. It reads private records from a -permissioned space on `waow.tech`: - -- space type: `tech.waow.tree` -- skey: `self` -- collections: `tech.waow.tree.person`, `tech.waow.tree.union`, - `tech.waow.tree.edge` - -Racine made the `listRecords` value-shape problem obvious. It should be able to -drop the N+1 `getRecord` hydration path now that ZDS returns values by default. - -### zlay, atproto-bench, Tranquil, reference PDS - -- `zlay` and `atproto-bench` were used as context for commit signing, - dependency graph, and benchmarking. -- Tranquil and reference PDS are semantic references, not authorities to copy - blindly. Use them to understand mature behavior and then reason against the - protocol. - -## issue/bug arcs handled - -- Invalid email tokens: ZDS generated tokens containing digits the Bluesky app - rejected. Fixed and released as a patch. -- Missing env vars: added env support/docs for port and DB path. -- Email not sending: implemented Comail provider and deployed secrets. -- Migration/account deactivated in Bluesky: root cause involved relay/appview - state, not a ZDS code patch. Important diagnostic lesson: Blacksky working - means the repo/PDS can be fine while Bluesky appview state is stale. -- Slow feeds/appview proxy: added stats and diagnosed slow proxied XRPCs rather - than assuming the user was mistaken. -- PDS debugger relay statuses: noted surprising offline statuses; still worth - following up, likely via crawler/request-crawl/vsky behavior. -- ApplyWrites large body: fixed request-size limit and error mapping. -- Firehose `#commit.since`: fixed previous-rev semantics and responded to Mia. -- Dependency graph breakage: resolved by moving onto canonical first-party - websocket/httpz/zat graph. - -## deployment and verification habits - -Do not deploy because a build completed. Deploy after: - -1. reading the relevant protocol/reference behavior -2. implementing narrowly -3. running the required checks -4. running the relevant smoke/bench lane -5. probing production behavior when possible - -Useful production probes: - -- `fly status` -- `fly logs` -- `/xrpc/_health` -- `/api/openapi.json` -- authenticated `com.atproto.space.listRecords` for a known resident account -- PDS debugger and pdsls firehose when sync behavior changed - -When probing with local `.env` files, do not print secrets or access tokens. - -## immediate next useful work - -Prioritize these only after checking current source and protocol state: - -1. Align `com.atproto.space.listRepoOps` with the proposal's values-by-default - plus `excludeValues` shape. -2. Let Racine remove its N+1 hydration fallback and verify cold-load latency - drops from seconds to one paged call per collection. -3. Re-check HappyView and proposal PR #94 for any further permissioned-data - drift, especially `getRepo`, `getLatestCommit`, notification registration, - and credential JWT details. -4. Add persistent historical stats, but design it as operator/resident - diagnostics rather than process-internals telemetry. -5. Follow up on relay/PDS debugger statuses and whether `vsky`/crawler config - should be represented better. -6. Keep benchmark tables honest: rerun comparable ZDS/Tranquil/reference PDS - probes before claiming performance wins or regressions. -7. Continue reviewing account-status behavior against protocol docs before - exposing stronger operator workflows. - -## final caution - -This session included several false starts caused by treating local guesses as -protocol knowledge. The durable lesson is simple: read the protocol, read the -reference implementations, then make the smallest semantically correct ZDS -change. If something feels like it should live in `zat`, stop and write the -case down instead of editing `zat` from this repo.