# operator guide

This is the operator-facing guide for running ZDS: local startup,
configuration, containers, invite administration, account migration, and
release policy. The concrete `pds.zat.dev` Fly workflow is documented in the
[production deployment runbook](deployment.md).

For account suspension, takedown, and test-account cleanup, see the
[account takedown runbook](account-takedown-runbook.md). ZDS supports temporary
account deactivation and operator takedown for hosted repo subjects; `suspended`
and `deleted` remain protocol vocabulary without local operator workflows.

## local checks

Run these before committing:

```sh
just test
just smoke
git diff --check
zig zen
```

`just smoke` starts a local server, creates a session, writes records,
checks blob upload, verifies repo/sync endpoints, and asserts that known app
records use valid key shapes.

Run `just smoke-permissioned` as well when changing permissioned-data routes,
storage, OAuth space scopes, or the resident space browser.

## benchmarks

Local store benchmarks live under `bench/`:

```sh
just bench all
just bench write
just bench read
just bench repo
just bench blob
just bench blob-gc 512 65536
just bench http
just bench websocket-churn
just bench metastore 10 1000
just bench get-cid 10 1000
just bench write-profile 10 500
just bench run --records 1000
```

These measure repo writes, indexed reads, full repo CAR materialization, blob
disk IO and collection, and selected HTTP route overhead against temporary SQLite/blobstore
state. Treat the numbers as local trend data; compare runs on the same machine
and optimize against the shape of the curve, not a single absolute value.

## blob lifecycle

Uploaded blobs begin untethered. ZDS gives an untethered upload 24 hours to be
referenced by either a public repo record or a permissioned-space record. Once
a blob is old enough, removing its final record reference makes it collectable.
A reference from either record system keeps the blob.

Collection is bounded and automatic. A background worker starts with the
service and reclaims at most 256 eligible blobs per minute. Upload bodies are
streamed through a fixed 64 KiB buffer into unique temporary files without
holding a repo-write lane; only the
atomic promotion and metadata update are serialized with collection for the
same CID. Collection rechecks both record systems under the account write lane
before deleting metadata and disk bytes. The service logs deleted counts,
bytes, and failures. `just bench blob-gc` measures the deletion path against
temporary state.

Keep experimental permissioned-data probes separate from this baseline. Their
first job is to catch obviously bad access patterns in `com.atproto.space.*`,
not to expand the normal benchmark gate.

See [bench/README.md](../bench/README.md) for the Tranquil comparison target
and the permissioned-data benchmark notes.

## local server

```sh
zig build run -- --port 2583 --db dev/zds.sqlite3
```

Use Caddy for local HTTPS when a client requires TLS:

```sh
caddy run --config dev/Caddyfile
```

## configuration

Common deployment settings:

- `ZDS_PUBLIC_URL`: public PDS origin.
- `ZDS_SERVER_DID`: PDS service DID, usually `did:web:<host>`.
- `ZDS_HOST`: bind address. Default: `127.0.0.1`, or `0.0.0.0` in the
  container image.
- `ZDS_PORT`: bind port. Default: `2583`.
- `ZDS_DB`: SQLite database path. Default: `dev/zds.sqlite3`, or
  `/data/zds.sqlite3` in the container image. `ZDS_DB_PATH` is accepted as a
  compatibility alias when `ZDS_DB` is unset.
- `ZDS_PLC_ROTATION_KEY`: PDS rotation authority private key. Accepts the
  official PDS 32-byte secp256k1 hex form and the private multikey form emitted
  by PDS Moover. New `did:plc` accounts use this as the PDS rotation authority.
- `ZDS_RECOVERY_DID_KEY`: optional recovery `did:key` returned before the PDS
  rotation key in recommended DID credentials.
- `ZDS_JWT_SECRET`: stable secret for access and refresh JWTs.
- `ZDS_DPOP_SECRET`: stable secret for stateless DPoP nonce generation. If
  unset, ZDS uses `ZDS_JWT_SECRET`. A separate value is recommended for
  production. Multi-node deployments must configure every node with the same
  effective DPoP secret so each node accepts the same nonce window.
- `ZDS_HANDLE_DOMAINS`: comma-separated leading-dot hosted-handle suffixes
  advertised by `describeServer`, for example `.pds.example.com`. Each suffix
  needs wildcard DNS and TLS (`*.pds.example.com`) routed to ZDS so hosted
  handles can answer `/.well-known/atproto-did`. ZDS treats configured suffixes
  as authoritative, exposes only active accounts through public handle
  resolution, and limits handle updates to 10 per five minutes and 50 per day.
  Do not include the bare PDS hostname.
- `ZDS_MAIL_PROVIDER`: email delivery provider. Default: `comail`. Supported:
  `comail`, `resend`.
- `ZDS_EMAIL_FROM`: sender address for account and PLC email tokens. The
  domain must be enrolled with the configured provider. Comail requires a bare
  email address such as `noreply@example.com`, not a display-name form such as
  `ZDS <noreply@example.com>`.
- `ZDS_COMAIL_API_KEY` and `ZDS_COMAIL_DID`: Comail HTTP Send API credentials.
  `ZDS_COMAIL_DID` is the atproto DID that enrolled the sending domain; ZDS
  sends it as `X-Atmos-DID`.
- `ZDS_RESEND_API_KEY`: Resend API key, used only when
  `ZDS_MAIL_PROVIDER=resend`. See [comail](comail.md) for the default
  provider's identity model, request shape, and production troubleshooting
  notes.
- `ZDS_BLOB_UPLOAD_LIMIT`: upload body limit. Default: `100000000`.
- `ZDS_BLOBSTORE_PATH`: disk blobstore root.
- `ZDS_CRAWLERS`: comma-separated relay crawl targets.
- `ZDS_OPERATOR_HANDLE`: atproto handle of the person operating this PDS,
  shown on the public signup page as the contact for invite codes.
- `ZDS_PLC_DIRECTORY`: PLC directory origin. Default: `https://plc.directory`.
- `ZDS_PROXY_SERVICE_URL`, `ZDS_PROXY_SERVICE_DID`, `ZDS_PROXY_SERVICE_ID`:
  appview that receives `app.bsky.*` requests sent without an `atproto-proxy`
  header, and the endpoint used when the header names this DID. Defaults:
  `https://api.bsky.app`, `did:web:api.bsky.app`, `bsky_appview`.
- `ZDS_LOG_LEVEL`: `err`, `info`, or `debug`. Default: `info`.
- `ZDS_DEBUG`: set to `true` as a shorthand for `ZDS_LOG_LEVEL=debug`.
- `ZDS_MAX_CONCURRENT_REPO_EXPORTS`: maximum concurrent full
  `com.atproto.sync.getRepo` exports. Default: `4`. Incremental
  `getRepo?since=...` exports are not intended to consume this full-export
  backpressure budget.
- `ZDS_ADMIN_TOKEN`: bearer token for privileged local PDS administration,
  including invite-code minting.
- `ZDS_INVITE_REQUIRED`: set to `true` to require invite codes for account
  creation. `describeServer` reflects this value.
- `ZDS_PERMISSIONED_DATA`: set to `true` to opt into experimental
  permissioned-data routes. Default: disabled. ZDS exposes protocol reads and
  writes under `com.atproto.space.*`, and baseline PDS-managed space management
  under `com.atproto.simplespace.*`, tracking the lexicons published for the
  [spaces alpha](https://atproto.com/blog/atproto-spaces-alpha). Enable it if
  residents want to use alpha apps such as
  [Bulletin](https://bulletin.my). The implementation remains experimental:
  reader/group semantics should stay in application policy, while `simplespace`
  only provides the small required baseline. Use `just smoke-permissioned` for
  the experimental smoke lane.
- Passkeys do not require a separate deployment secret. WebAuthn RP ID is
  derived from the public URL host.

PLC key lifecycle:

- Repo signing keys are for repository commits only.
- PLC rotation keys are for DID PLC operations only.
- `getRecommendedDidCredentials` follows the official PDS shape: the account
  signing key is returned under `verificationMethods.atproto`; rotation keys
  are the optional recovery DID key followed by the configured PDS rotation key.
- ZDS rejects new PLC operations that put the account signing key in
  `rotationKeys`.
- Existing early accounts can be repaired with `just plc-repair <did>` after
  setting `ZDS_DB` or `ZDS_DB_PATH`, `ZDS_PUBLIC_URL`, `ZDS_PLC_ROTATION_KEY`,
  and optional `ZDS_REPAIR_RECOVERY_DID_KEY`, `ZDS_REPAIR_RECOVERY_KEY`, or
  `ZDS_RECOVERY_DID_KEY`. The repair tool is intentionally separate from normal
  XRPC handling; it only uses the account signing key if the current public PLC
  state already authorizes that key as a rotation key, and submits a replacement
  operation containing the intended recovery/PDS rotation keys. The
  `ZDS_REPAIR_RECOVERY_KEY` form accepts a PDS Moover private multikey and
  derives the public `did:key` locally.

When invites are required and the database has no accounts or invite codes, ZDS
creates one bootstrap code and logs it during startup. Invite codes are stored
as metadata rows and uses are recorded separately, matching the official PDS
model.

## container

Images are published to ATCR:

```sh
docker pull atcr.io/zat.dev/zds:latest
docker run -p 2583:2583 \
  -v zds-data:/data \
  -e ZDS_PUBLIC_URL=https://pds.example.com \
  -e ZDS_SERVER_DID=did:web:pds.example.com \
  -e ZDS_JWT_SECRET='at-least-32-random-bytes-here' \
  -e ZDS_DB=/data/zds.sqlite3 \
  atcr.io/zat.dev/zds:latest
```

Set a random JWT secret and serve the configured public origin over HTTPS.

Publish commands:

```sh
just docker-publish
just docker-publish vX.Y.Z
just docker-publish-current
just docker-publish-release vX.Y.Z
```

## releases

ZDS uses SemVer tags (`vMAJOR.MINOR.PATCH`). Before `v1.0.0`, patch releases
are still appropriate for user-visible compatibility fixes and operator-safe
bug fixes. Minor releases are for meaningful protocol surface, storage shape,
or operator workflow changes. Major releases are reserved for explicit
incompatible production contracts once ZDS declares a stable surface.

The `/release` skill (`.agents/skills/release/SKILL.md`) walks this end to end,
including the changelog promotion, the spindle deploy watch, and the live
version check. Release checklist:

```sh
just test
just smoke-all
zig fmt --check build.zig build.zig.zon src bench tools
git diff --check
zig zen
git tag -a vX.Y.Z -m "vX.Y.Z"
git push origin main vX.Y.Z
just docker-publish-release vX.Y.Z
```

For protocol-parity or storage-behavior releases, also complete the
[benchmark stage checklist](../bench/README.md#stage-checklist): revisit the
Zat boundary, run matching Tranquil and official PDS probes where comparable,
and document any remaining mismatch as an explicit decision or follow-up.

## migration

ZDS supports browser-based account migration flows such as PDS Moover:

- service-auth-gated `createAccount` for an existing DID
- repo CAR import/export
- missing blob reporting and blob upload
- recommended DID credentials
- PLC operation email token, signing, and submission
- account activation/deactivation and status checks

Migration requires working public HTTPS, stable JWT secret, email delivery, blob
storage, and writable SQLite storage.

## account security

The resident account surface lives at `/account`. The security section,
`/account/manage`, is for account-owner management of passkeys and app
passwords and requires the account password for management actions.
App-password revocation also revokes active sessions that were created with
that app password. `/security` and `/passkeys` are compatibility redirects.

## conformance

Useful external checks:

- `https://check.cirrus.earth/?target=<handle>` for PDS and OAuth conformance.
- bsky.app and pdsls.dev for browser-visible OAuth, proxy, repo, and blob
  behavior.

Treat browser failures as request/response evidence: inspect the failing XRPC,
status code, response body, and relevant server logs before changing behavior.

## Web drawings

Operators can assign individual Doodl drawings to UI slots through
`config/drawings.json` and `just drawings`, then explicitly select the cached
manifest with `-Dweb-drawings=src/internal/web/icons/generated.zig` at build time.
Unconfigured builds use generic icons. Our Fly deployment opts in using
`ZDS_WEB_DRAWINGS`; its selections live in `config/pds.zat.dev.drawings.json`. See [Doodl drawings](iconography-proposal.md)
for the refresh and deployment workflow. Assets are bundled at build time;
startup and account requests do not contact drawing authors’ PDSes.

### Grouped account access

`GET /xrpc/dev.zat.account.listAccess` requires an account password session,
matching `listSessions`. `kind=oauth|password` selects app authorizations or
password credentials; `state=authorized|past|all` defaults to `authorized`.
`search` matches the app client ID or credential label, case-insensitively.
Without `group`, the response contains grouped summaries (`key`, `name`,
`count`, `newest`). With a returned group key, `connections` contains that
group's existing connection details. OAuth groups use the complete client ID;
password groups use authentication method and credential name.

Both modes default to 20 results and accept `limit` from 1 through 100.
Pass the returned `cursor` with the same filters for the next page; null ends
pagination. Cursors are non-secret offsets, not snapshot guarantees: changing
sessions between requests can shift pages. `authorized` and `total` always
count the entire signed-in account's selected kind, independently of filters
and pagination. The overview uses these totals; search results show per-group
counts. No JWT, refresh token, or OAuth family identifier is returned.

The app-access UI loads summaries first and connection pages only on expansion.
"Authorized" describes unrevoked, unexpired credentials, not online activity.
This endpoint does not revoke grants or change session lifetime policy.