operator guide #

This is the operator-facing guide for running ZDS: local startup, configuration, containers, invite administration, account migration, and release policy. The concrete pds.zat.dev Fly workflow is documented in the production deployment runbook.

For account suspension, takedown, and test-account cleanup, see the account takedown runbook. ZDS supports temporary account deactivation and operator takedown for hosted repo subjects; suspended and deleted remain protocol vocabulary without local operator workflows.

local checks #

Run these before committing:

just test
just smoke
git diff --check
zig zen

just smoke starts a local server, creates a session, writes records, checks blob upload, verifies repo/sync endpoints, and asserts that known app records use valid key shapes.

Run just smoke-permissioned as well when changing permissioned-data routes, storage, OAuth space scopes, or the resident space browser.

benchmarks #

Local store benchmarks live under bench/:

just bench all
just bench write
just bench read
just bench repo
just bench blob
just bench blob-gc 512 65536
just bench http
just bench websocket-churn
just bench metastore 10 1000
just bench get-cid 10 1000
just bench write-profile 10 500
just bench run --records 1000

These measure repo writes, indexed reads, full repo CAR materialization, blob disk IO and collection, and selected HTTP route overhead against temporary SQLite/blobstore state. Treat the numbers as local trend data; compare runs on the same machine and optimize against the shape of the curve, not a single absolute value.

blob lifecycle #

Uploaded blobs begin untethered. ZDS gives an untethered upload 24 hours to be referenced by either a public repo record or a permissioned-space record. Once a blob is old enough, removing its final record reference makes it collectable. A reference from either record system keeps the blob.

Collection is bounded and automatic. A background worker starts with the service and reclaims at most 256 eligible blobs per minute. Upload bodies are streamed through a fixed 64 KiB buffer into unique temporary files without holding a repo-write lane; only the atomic promotion and metadata update are serialized with collection for the same CID. Collection rechecks both record systems under the account write lane before deleting metadata and disk bytes. The service logs deleted counts, bytes, and failures. just bench blob-gc measures the deletion path against temporary state.

Keep experimental permissioned-data probes separate from this baseline. Their first job is to catch obviously bad access patterns in com.atproto.space.*, not to expand the normal benchmark gate.

See bench/README.md for the Tranquil comparison target and the permissioned-data benchmark notes.

local server #

zig build run -- --port 2583 --db dev/zds.sqlite3

Use Caddy for local HTTPS when a client requires TLS:

caddy run --config dev/Caddyfile

configuration #

Common deployment settings:

• ZDS_PUBLIC_URL: public PDS origin.
• ZDS_SERVER_DID: PDS service DID, usually did:web:<host>.
• ZDS_HOST: bind address. Default: 127.0.0.1, or 0.0.0.0 in the container image.
• ZDS_PORT: bind port. Default: 2583.
• ZDS_DB: SQLite database path. Default: dev/zds.sqlite3, or /data/zds.sqlite3 in the container image. ZDS_DB_PATH is accepted as a compatibility alias when ZDS_DB is unset.
• ZDS_PLC_ROTATION_KEY: PDS rotation authority private key. Accepts the official PDS 32-byte secp256k1 hex form and the private multikey form emitted by PDS Moover. New did:plc accounts use this as the PDS rotation authority.
• ZDS_RECOVERY_DID_KEY: optional recovery did:key returned before the PDS rotation key in recommended DID credentials.
• ZDS_JWT_SECRET: stable secret for access and refresh JWTs.
• ZDS_DPOP_SECRET: stable secret for stateless DPoP nonce generation. If unset, ZDS uses ZDS_JWT_SECRET. A separate value is recommended for production. Multi-node deployments must configure every node with the same effective DPoP secret so each node accepts the same nonce window.
• ZDS_HANDLE_DOMAINS: comma-separated leading-dot hosted-handle suffixes advertised by describeServer, for example .pds.example.com. Each suffix needs wildcard DNS and TLS (*.pds.example.com) routed to ZDS so hosted handles can answer /.well-known/atproto-did. ZDS treats configured suffixes as authoritative, exposes only active accounts through public handle resolution, and limits handle updates to 10 per five minutes and 50 per day. Do not include the bare PDS hostname.
• ZDS_MAIL_PROVIDER: email delivery provider. Default: comail. Supported: comail, resend.
• ZDS_EMAIL_FROM: sender address for account and PLC email tokens. The domain must be enrolled with the configured provider. Comail requires a bare email address such as noreply@example.com, not a display-name form such as ZDS <noreply@example.com>.
• ZDS_COMAIL_API_KEY and ZDS_COMAIL_DID: Comail HTTP Send API credentials. ZDS_COMAIL_DID is the atproto DID that enrolled the sending domain; ZDS sends it as X-Atmos-DID.
• ZDS_RESEND_API_KEY: Resend API key, used only when ZDS_MAIL_PROVIDER=resend. See comail for the default provider's identity model, request shape, and production troubleshooting notes.
• ZDS_BLOB_UPLOAD_LIMIT: upload body limit. Default: 100000000.
• ZDS_BLOBSTORE_PATH: disk blobstore root.
• ZDS_CRAWLERS: comma-separated relay crawl targets.
• ZDS_OPERATOR_HANDLE: atproto handle of the person operating this PDS, shown on the public signup page as the contact for invite codes.
• ZDS_PLC_DIRECTORY: PLC directory origin. Default: https://plc.directory.
• ZDS_PROXY_SERVICE_URL, ZDS_PROXY_SERVICE_DID, ZDS_PROXY_SERVICE_ID: appview that receives app.bsky.* requests sent without an atproto-proxy header, and the endpoint used when the header names this DID. Defaults: https://api.bsky.app, did:web:api.bsky.app, bsky_appview.
• ZDS_LOG_LEVEL: err, info, or debug. Default: info.
• ZDS_DEBUG: set to true as a shorthand for ZDS_LOG_LEVEL=debug.
• ZDS_MAX_CONCURRENT_REPO_EXPORTS: maximum concurrent full com.atproto.sync.getRepo exports. Default: 4. Incremental getRepo?since=... exports are not intended to consume this full-export backpressure budget.
• ZDS_ADMIN_TOKEN: bearer token for privileged local PDS administration, including invite-code minting.
• ZDS_INVITE_REQUIRED: set to true to require invite codes for account creation. describeServer reflects this value.
• ZDS_PERMISSIONED_DATA: set to true to opt into experimental permissioned-data routes. Default: disabled. ZDS exposes protocol reads and writes under com.atproto.space.*, and baseline PDS-managed space management under com.atproto.simplespace.*, tracking the lexicons published for the spaces alpha. Enable it if residents want to use alpha apps such as Bulletin. The implementation remains experimental: reader/group semantics should stay in application policy, while simplespace only provides the small required baseline. Use just smoke-permissioned for the experimental smoke lane.
• Passkeys do not require a separate deployment secret. WebAuthn RP ID is derived from the public URL host.

PLC key lifecycle:

• Repo signing keys are for repository commits only.
• PLC rotation keys are for DID PLC operations only.
• getRecommendedDidCredentials follows the official PDS shape: the account signing key is returned under verificationMethods.atproto; rotation keys are the optional recovery DID key followed by the configured PDS rotation key.
• ZDS rejects new PLC operations that put the account signing key in rotationKeys.
• Existing early accounts can be repaired with just plc-repair <did> after setting ZDS_DB or ZDS_DB_PATH, ZDS_PUBLIC_URL, ZDS_PLC_ROTATION_KEY, and optional ZDS_REPAIR_RECOVERY_DID_KEY, ZDS_REPAIR_RECOVERY_KEY, or ZDS_RECOVERY_DID_KEY. The repair tool is intentionally separate from normal XRPC handling; it only uses the account signing key if the current public PLC state already authorizes that key as a rotation key, and submits a replacement operation containing the intended recovery/PDS rotation keys. The ZDS_REPAIR_RECOVERY_KEY form accepts a PDS Moover private multikey and derives the public did:key locally.

When invites are required and the database has no accounts or invite codes, ZDS creates one bootstrap code and logs it during startup. Invite codes are stored as metadata rows and uses are recorded separately, matching the official PDS model.

container #

Images are published to ATCR:

docker pull atcr.io/zat.dev/zds:latest
docker run -p 2583:2583 \
  -v zds-data:/data \
  -e ZDS_PUBLIC_URL=https://pds.example.com \
  -e ZDS_SERVER_DID=did:web:pds.example.com \
  -e ZDS_JWT_SECRET='at-least-32-random-bytes-here' \
  -e ZDS_DB=/data/zds.sqlite3 \
  atcr.io/zat.dev/zds:latest

Set a random JWT secret and serve the configured public origin over HTTPS.

Publish commands:

just docker-publish
just docker-publish vX.Y.Z
just docker-publish-current
just docker-publish-release vX.Y.Z

releases #

ZDS uses SemVer tags (vMAJOR.MINOR.PATCH). Before v1.0.0, patch releases are still appropriate for user-visible compatibility fixes and operator-safe bug fixes. Minor releases are for meaningful protocol surface, storage shape, or operator workflow changes. Major releases are reserved for explicit incompatible production contracts once ZDS declares a stable surface.

The /release skill (.agents/skills/release/SKILL.md) walks this end to end, including the changelog promotion, the spindle deploy watch, and the live version check. Release checklist:

just test
just smoke-all
zig fmt --check build.zig build.zig.zon src bench tools
git diff --check
zig zen
git tag -a vX.Y.Z -m "vX.Y.Z"
git push origin main vX.Y.Z
just docker-publish-release vX.Y.Z

For protocol-parity or storage-behavior releases, also complete the benchmark stage checklist: revisit the Zat boundary, run matching Tranquil and official PDS probes where comparable, and document any remaining mismatch as an explicit decision or follow-up.

migration #

ZDS supports browser-based account migration flows such as PDS Moover:

• service-auth-gated createAccount for an existing DID
• repo CAR import/export
• missing blob reporting and blob upload
• recommended DID credentials
• PLC operation email token, signing, and submission
• account activation/deactivation and status checks

Migration requires working public HTTPS, stable JWT secret, email delivery, blob storage, and writable SQLite storage.

account security #

The resident account surface lives at /account. The security section, /account/manage, is for account-owner management of passkeys and app passwords and requires the account password for management actions. App-password revocation also revokes active sessions that were created with that app password. /security and /passkeys are compatibility redirects.

conformance #

Useful external checks:

• https://check.cirrus.earth/?target=<handle> for PDS and OAuth conformance.
• bsky.app and pdsls.dev for browser-visible OAuth, proxy, repo, and blob behavior.

Treat browser failures as request/response evidence: inspect the failing XRPC, status code, response body, and relevant server logs before changing behavior.

Web drawings #

Operators can assign individual Doodl drawings to UI slots through config/drawings.json and just drawings, then explicitly select the cached manifest with -Dweb-drawings=src/internal/web/icons/generated.zig at build time. Unconfigured builds use generic icons. Our Fly deployment opts in using ZDS_WEB_DRAWINGS; its selections live in config/pds.zat.dev.drawings.json. See Doodl drawings for the refresh and deployment workflow. Assets are bundled at build time; startup and account requests do not contact drawing authors’ PDSes.

Grouped account access #

GET /xrpc/dev.zat.account.listAccess requires an account password session, matching listSessions. kind=oauth|password selects app authorizations or password credentials; state=authorized|past|all defaults to authorized. search matches the app client ID or credential label, case-insensitively. Without group, the response contains grouped summaries (key, name, count, newest). With a returned group key, connections contains that group's existing connection details. OAuth groups use the complete client ID; password groups use authentication method and credential name.

Both modes default to 20 results and accept limit from 1 through 100. Pass the returned cursor with the same filters for the next page; null ends pagination. Cursors are non-secret offsets, not snapshot guarantees: changing sessions between requests can shift pages. authorized and total always count the entire signed-in account's selected kind, independently of filters and pagination. The overview uses these totals; search results show per-group counts. No JWT, refresh token, or OAuth family identifier is returned.

The app-access UI loads summaries first and connection pages only on expansion. "Authorized" describes unrevoked, unexpired credentials, not online activity. This endpoint does not revoke grants or change session lifetime policy.