Deployment #

What running this safely requires. The shape of the system draws the same hosts; this page states the rules, and operations is the procedure an operator runs against them.

What runs #

A container image, started by one systemd unit, on one instance. Not a binary and a unit file; not an orchestrator. scripts/ is a developer's interface and no part of a deployment.

The image is built from the repository's Dockerfile with --features route53, tagged with the release it holds, and pushed to ECR. infra/pds/templates/user_data.sh.tftpl writes /etc/systemd/system/ didbot-pds.service at first boot, substituting the image reference and the deployment's zone, owner and hosted zone id directly into the unit — so systemctl cat didbot-pds shows exactly what is running, with no environment file to also go read.

Three properties decide this, and each of them is a property of the process rather than a preference:

• The unit must be a single-writer service. The store is one write-ahead log behind one lock, and two live writers interleave frames neither can replay. infra/pds/ec2.tf is one instance, not an autoscaling group; the unit runs the container under a fixed --name with an ExecStartPre that removes a previous one, so a restart racing a slow shutdown cannot produce two. Systemd is not the only way to start a process on a host, so the exclusion does not rest on it: didbot_pds::lock takes an exclusive flock on pds.lock inside the data directory before the log is opened, and an operator running the binary by hand against the mounted volume is refused by name. The kernel releases the lock when the process dies, so a crash leaves nothing to clean up.
• The image is a name that can be said. The exit criterion for deploy is that a tagged release runs on a host nobody develops on. A container digest is that name, and a git pull on the instance is not.
• The process binds 443 and holds its own certificate. Nothing terminates TLS in front of it, so the run unit uses --network host rather than a published-port mapping — which is also what lets the container's IMDSv2 request reach the metadata endpoint at http_put_response_hop_limit = 1. Raising that to 2 is required if the run unit ever leaves host networking. The image runs as an unprivileged user, and with the host's network namespace there is no net.ipv4.ip_unprivileged_port_start to relax, so the binary carries cap_net_bind_service=+ep — one capability, bind a low port — rather than the container running as root.

Restarts are Restart=on-failure with RestartSec=5, capped at five failures in two minutes so a crash loop stops rather than burning the instance. A restart is not free: didbot_pds::session::SessionAuth is in-process memory, so every outstanding access and refresh token is invalidated by one — see "say what a restart does" in deploy.

The zone #

One zone, used for everything: the server, the agent handles, the authority serving every did.json, and the zone named in the owner's ownership record. HTTPS throughout.

The server sits at the apex and agents in a subzone beneath it. Agents are off the apex so a wildcard certificate cannot shadow the server's own name. The zone does not have to sit at or below the server's own hostname — which server serves a given agent is answered by resolving the agent's DID document, not by comparing hostnames — but the recommended layout keeps them together.

Two hostnames per account — one for the DID, one for the handle — and both must resolve before the account exists.

One wildcard record makes both of them resolve. The zone holds three names whatever the account count: the apex, *.<zone>, and _acme-challenge.<zone> while a certificate is being issued. Provisioning writes no record of its own — WildcardDns is the provider for a zone in this shape, and it publishes nothing because the wildcard already answers. The names an agent has works one account through the whole path, from what exists in the zone to what a stranger reads.

The cost is that there is no per-name existence guarantee and no DNS-level garbage collection: a name no account holds still resolves, and answers 404 rather than NXDOMAIN. That is the correct trade here — DNS removal was never a revocation mechanism, since a withdrawn name stays cached downstream for its TTL, and an HTTP status is the only thing that can express a decommissioned account still serving its document.

A deployment can still publish per-agent records instead; Route53Dns does. It buys the per-name existence check, and it costs a hosted-zone record per retained account, a network call on the provisioning path, and — because a real record at a name stops the wildcard synthesizing anything beneath it — any prospect of hostnames deeper than one label.

A timer keeps the zone and the account store in agreement. Provisioning publishes an agent's record before it writes the account, and unwinds the record itself on every error it returns through; a process killed between the two runs none of that unwinding. didbot_serve::stale_sweep runs Provisioner::sweep_stale_bounded hourly, deleting the accounts whose retention has run out — a week for one that carries no window of its own — and each deletion withdraws that account's hostname with it. One tick takes a hundred accounts at most, since each is a call to the zone's provider, and the next tick takes what the last left.

What that timer can withdraw is what the startup zone read put in front of it. Route53Dns builds a withdrawal out of its own record of the record set, because a Route53 DELETE names the type, TTL and every value it removes; the two things that put a name there are a publish this process made and the read it takes of the hosted zone at startup. A restored or restarted process published none of the names its accounts resolve at, so until that read lands the deployment serves every route normally, bot.did.deleteAgent answers NotPublished, and the sweep takes nothing. The read is retried every five minutes until it succeeds (didbot_serve::zone_read), and GET /health carries the state under zoneRecords: read is the summary an alarm expression matches on, zones names each zone and whether this process has read it.

The DNS credential #

Control of the zone is control of every name in it. It is as powerful as the signing keys, and it lives beside them: the server writes the _acme-challenge TXT records its certificate depends on and the CAA record below, so the credential has to be reachable from the process. Under a wildcard those are the only records it writes on an ordinary day — agent hostnames cost none, and zone creation is refused unless dns.may_create_zones turns it on. Holding it elsewhere would protect nothing: whoever holds this host holds the signing keys, and with those they can sign as any account it serves.

So the question is not where the credential sits. It is what it reaches, and what a holder of it can be caught doing.

What it can do #

Precisely, for the one hosted zone it is scoped to:

• Point any name in the zone somewhere else. A specific record at a name overrides the wildcard, so a live agent's two hostnames — the DID's and the handle's, whose /.well-known/atproto-did is what binds the two together — can be made to answer from another host, which then serves its own did.json. did:web has no history. A document swapped today changes what every signature that account ever made verifies against, retroactively, and nothing an observer kept proves what the document said yesterday.
• Point the server's own hostname somewhere else, when the zone contains it. That is every route, every handle answer, and the certificate for it.
• Pass DNS-01 for any name in the zone, including names no account holds, and obtain a publicly trusted certificate that validates. An impersonation built on either of the two above survives TLS.
• Rewrite the CAA record, which is the step that makes the one above reachable through an authority other than the one this zone authorises.

Serving a document at a name this server never minted is not on that list, and it is worth saying why: under a wildcard every name in the zone already resolves here without any DNS write at all. What answers at one is this server's business, not DNS's — which is why a name no account holds answers 404 rather than NXDOMAIN, and why a zone that does publish per-agent records gets a per-name existence check the wildcard cannot give.

What it does not reach is anything outside that one hosted zone — not other zones in the account, and not the NS records at the registrar that delegate into this one, which live in the parent zone. That boundary is the IAM policy below and nothing else, which is why the policy is the load-bearing part.

What is detectable #

Detection is what bounds a compromise here, because isolation cannot. Three channels, and they differ in the one way that matters — whether the evidence is somewhere the holder of this host can reach.

Certificate transparency. Every publicly trusted certificate is logged by the authority that issued it, to logs this deployment does not run. A certificate naming any hostname in the zone that this server did not order is visible there, and only there: no record of it exists on this host, and nothing about the issuance touched it. Subscribe an external monitor to the zone. This is the one channel a compromised host cannot suppress, and it is after the fact — the certificate already exists when the log entry appears.

A refused issuance, reported by the authority. The CAA record the server publishes at the zone apex names one authority and, per RFC 8657's accounturi, the one ACME account at it this deployment holds the key for. An attempt to issue for a name in this zone through any other account is refused by any authority honouring the record, and reported to the iodef contact in it — a report that arrives from the authority, not from this host. This does not stop the holder of the zone credential, who can rewrite the record. It makes doing so a second, separate act, and it closes issuance to everyone who does not hold the credential.

A zone holding a record nobody here wrote. ListResourceRecordSets says what the zone actually contains, and this server knows what it put there. A wildcard zone makes that check almost trivial — it holds the apex, *.<zone> and _acme-challenge.<zone>, and a fourth name is a name somebody else added. A zone publishing per-agent records checks the same thing against the ledger, which keeps a name past the account it described. Either way this is observed on the host holding the credential, so it proves nothing against an attacker who holds it; what it catches is a credential copy still live somewhere, a second stack writing the same zone, or a change made by hand in a console. Nothing runs the comparison today — Route53Dns::resync already reads every record set in the zone, and no code diffs what it finds against what this server expects; see plan/agent-accounts.md.

What is not recoverable #

Rotating the credential and rewriting the zone recovers the zone. It does not recover any of the following, and no procedure does:

• Confidence in anything signed before the compromise was noticed. A swapped did:web document leaves no trace of the key it replaced. A record signed by the agent and a record signed with the substituted key are indistinguishable afterwards, in both directions — the compromise does not only forge new records, it makes the real ones unprovable.
• Whatever resolved during the window. A repointed name was cached by every resolver that asked, and served to every client that followed it.

A certificate is the exception: an issued certificate can be revoked, and the authority that issued it is who to ask.

The policy that bounds it #

crates/didbot-dns's Route53Dns is the provider that scopes this: an IAM credential given only

route53:ChangeResourceRecordSets, route53:ListResourceRecordSets
  on arn:aws:route53:::hostedzone/<HOSTED_ZONE_ID>
route53:GetChange
  on arn:aws:route53:::change/*

can write and read records in the one hosted zone this server owns, and nothing else in the account. GetChange has to be unscoped to a zone because Route53 change ids are not namespaced under one; it grants read of a change's propagation status, not write of anything. A cloud DNS service is the right kind of provider for exactly this reason — the credential narrows to a single zone — and across several servers that narrowing is the blast-radius boundary between them. See the module docs on Route53Dns for the full policy JSON and for how it handles eventual consistency, rate limits and hosted-zone record limits, all things this posture note assumes are handled correctly one layer down.

The boundaries this design actually rests on are elsewhere, and neither is inside this host: the agent hosts are separate machines from this one, and the policy that governs it lives in a repository this server holds no credential for. A compromised server can break policy; it cannot change it.

Certificates #

The server terminates TLS itself. Nothing in front of it holds a certificate, so it obtains and renews its own over ACME DNS-01, through the same zone credential above (crates/didbot-tls, on instant-acme).

One <zone> + *.<zone> pair per configured zone. A wildcard in a certificate matches exactly one label: *.pds.did.bot covers claudes.pds.did.bot and does not cover agent.claudes.pds.did.bot. This is not the DNS rule — a wildcard record of the same spelling answers at any depth — so what bounds how deep an agent hostname may sit is the certificate, not the zone. See the names an agent has. A deployment configured with claudes.pds.did.bot and vllm.pds.did.bot therefore holds two certificates, each from one ACME order carrying both of that zone's identifiers — the apex and the wildcard — whose two DNS-01 challenges share one _acme-challenge.<zone> record and must be published together. didbot_tls::fleet::CertificateFleet holds one manager per zone and ZoneResolver picks between their certificates on the SNI name each handshake carries. A name no configured zone covers is refused rather than handed a certificate that does not name it.

A renewal never rebinds the listener. The certificate lives behind an ArcSwap that rustls reads once per handshake, so a renewal is a pointer store: connections already established — the firehose relays and indexers hold open for hours — are untouched, and the next handshake sees the new certificate.

A failed renewal keeps serving what it has, and gets louder. Renewal starts 30 days before expiry and retries hourly; every failure leaves the current certificate in place and is logged with the zone, the failure count and the expiry it is measured against. Three consecutive failures raise that zone to Warning; inside three days of expiry it is Critical whether or not anything has failed, because at that point expiry itself is the risk. CertificateFleet::alert_level reports the worst level any zone is at and zone_alert_levels says which zone it came from. GET /health carries both, under certificates: worst is the summary an alarm expression matches on, zones names each zone and its own level. infra/pds/monitoring.tf is what reads it — a Route53 health check string-matching worst, and a CloudWatch alarm on that check — so a zone above ok and a server that stopped answering raise the same alarm. An expired certificate is a total outage for every hostname in that zone, agents included.

The ACME directory a run is configured for decides what it serves. acme_environment is staging by default (infra/pds/variables.tf), and a staging leaf is signed by an authority no client trusts. So the certificate on disk is served only when meta.json records the directory this run is configured for, and the saved ACME account is restored only when its credentials record that same directory; otherwise the run registers an account there and orders a certificate from it, which is what makes the switch to production take effect on the first boot after it. A restart under an unchanged environment places no order and serves the certificate that is already on disk. The CAA record below is republished from the account actually in hand before that order is placed, so a switch carries the accounturi with it.

The zone's CAA record is written by the server, on every start. It is published before the first order and restated on each start, naming the authority the order is about to go to and the ACME account the order will be placed through — didbot_tls::dns::IssuancePolicy, reading both off the live objects rather than off configuration. That is deliberate: a CAA record naming an account this deployment cannot use does not fail safe, it stops the next renewal, and an expired certificate is a total outage for every hostname in the zone. Reading both halves from the objects the order itself uses makes that disagreement unreachable. A backend whose zone already says exactly this skips the write. There is no issuewild property — RFC 8659 falls back to issue for a wildcard request when none is present, and the wildcard is in the same order — and failing to publish the record is a warning rather than a startup failure, because a zone with no CAA record is the state every zone starts in.

The contact is what the authority warns. --acme-contact <uri> is the address Let's Encrypt registers against this deployment's ACME account and sends an expiry warning to; infra/pds/variables.tf's acme_contact supplies it, and the unit carries the flag only when that variable is non-empty. Empty is a supported deployment and the one this configuration defaults to: an ACME account with no contact registers, orders and renews identically, so the address buys a second warning beside infra/pds/monitoring.tf's alarm rather than the ability to hold a certificate. Demanding one would make a deployment's boot depend on a field the CA does not require, and an instance that does not come up is every agent on it offline. Terraform refuses anything but a mailto: URI or the empty string at plan time, because a contact Let's Encrypt rejects fails at account registration, which is a startup this deployment does not come back from.

Where the address goes. Two places, both by design. IssuancePolicy publishes it as the RFC 8659 iodef value of the zone's CAA record, which is what points a refused issuance at somebody the authority can reach without going through this host — so it is in public DNS, and the address to use is one suited to that. The DNS backend logs the record set it wrote at INFO when the set changes, so it is also in the instance's journal. The two public surfaces publish their own complete lists: wire::StatsResponse is counts and byte totals over this deployment's contents, and GET /health is the tick and per-zone certificate alert levels.

The server is handed the zones it may write. Each zone needs a hosted zone id configured for it (--route53-zone-id <zone>=<id>); a zone without one publishes nothing and says so at startup. Creating a hosted zone, or delegating a new one with NS records in the zone above, is a separate posture — dns.may_create_zones in didbot-config — and is off by default.

Announcing to a relay #

A relay learns a personal data server exists by being told. This deployment names one in the relay_hostname Terraform variable — bsky.network by default, "" to run naming none — and infra/pds/templates/user_data.sh.tftpl puts it on the unit's ExecStart as --relay-hostname. That flag decides which relay com.atproto.sync.requestCrawl and the deprecated notifyOfUpdate are sent to; sending either is an operator action.

Boot does not dial the relay. The server builds a client for the named host and hands it to the e-stop admin socket, and that is all it does with it until somebody asks. A relay that is down while this instance comes up therefore costs the instance nothing — there is no retry loop to wedge in and no startup step to fail. A relay that goes away later, or one that answers HostBanned, is felt by the operator who runs the command: the reply distinguishes accepted from refused (with the relay's status and body) from unreachable (with the transport error), and a call that gets no answer gives up after ten seconds rather than holding the socket open.

The command is reached over the data volume. The unit passes --estop-socket /data/estop.sock, which is /data/pds/estop.sock on the instance, so an operator on a break-glass SSH or SSM session reaches it without entering the container. Opening it is the whole authorization: the server creates the directory it sits in, re-tightens that directory to 0700 on every start and the socket to 0600, so the callers are the server's own uid and root. ANNOUNCE sends requestCrawl; NOTIFY sends notifyOfUpdate, which upstream deprecates in its favor but still names for resuming a relay's subscription after a gap.

$ printf 'ANNOUNCE\n' | sudo socat - UNIX-CONNECT:/data/pds/estop.sock
OK {"result":"accepted"}

Under a .localhost zone the flag is read and the command is left unwired, and the server says so at startup: no relay resolves a loopback name, so the only thing wiring it up there could produce is a failure an operator asked for.

The data directory #

One directory, --data, on a volume separate from the root volume so an AMI upgrade or an instance resize can re-attach it rather than lose it. The instance mounts it at /data and the container sees /data/pds as /data.

Path	Holds	Mode
pds.wal	The log: accounts, records, ledger, commits, name holds — and the account signing keys	0600
pds.wal.compact	A compaction's scratch file, replaced by rename	0600
pds.lock	The single-writer claim on this directory	0600
pds.layout	Which log shape wrote this directory	0600
blobs/	Blob content, addressed by CID	0700
blobs/.incoming/	Uploads in flight, discarded on restart	0700
tls/	The ACME account key	0700
tls/zones/<zone>/	That zone's certificate, its key, and issuance metadata	0700

Everything else a deployment has — configuration, the e-stop file and socket — is elsewhere and is not state.

Permissions are set on every open, not only on creation. The signing keys are in pds.wal, so the directory holding it is only as private as whoever made it. A directory the server creates is 0700 from DirBuilder::mode; a directory it was handed — an operator's mkdir, a mount point, a restored snapshot's filesystem root — carries whatever umask made it. didbot_pds::wal::create_dir therefore chmods an existing directory to 0700 every time it runs rather than trusting it, and didbot_tls::storage::create_dir does the same for tls/. That costs one syscall on the common path and closes the case that actually matters.

What it does not fix is ownership. A directory owned by another user is not made safe by its mode, and the server does not check: it will fail to write and say so, which is correct but is a boot failure rather than a diagnosis. Ownership is the boot script's. The image runs as uid 10001, pinned in the Dockerfile rather than left to useradd, and user_data.sh.tftpl chowns /data/pds to that uid before the unit starts — which also re-homes a restored snapshot whose files were written by a different uid.

The log's budget #

--log-budget is the number of bytes the log may reach; past it an append is refused before a byte is written, the file stands exactly where it was, and the write comes back as 507 StorageFull. infra/pds/variables.tf's log_budget_bytes sets it, and takes a tenth of data_volume_size_gb unless a number is pinned — 2 GiB on the default 20 GiB volume. The remaining nine tenths are the blob store's, which shares this volume and grows in megabytes at a time where the log grows in hundreds of bytes.

What has to fit inside the number is the deployment's state, not its history. Every start rewrites the log as one entry per live thing once the history has run to didbot_pds::COMPACT_RATIO times the state, which at a few hundred bytes an entry puts millions of live records inside 2 GiB. A released name, a deleted account, an overwritten record and a reissued credential are all history a compaction drops, so a log that reached its budget on churn is under it again after a restart — didbot-pds/tests/durability.rs asserts that round trip.

An operator sees this in three places. The startup line the write-ahead log will refuse writes past its budget carries the budget and the bytes held, on every boot. compacted the write-ahead log on a later boot says the rewrite ran and how many entries it kept. And a deployment at its budget answers 507 whose message names the bytes held, the budget, and the bytes the refused entry wanted. A budget a compaction cannot bring the log below is a deployment whose state alone exceeds the number: raise log_budget_bytes, or raise data_volume_size_gb and with it the tenth this derives.

Secrets, and where each lives #

Three, and the process holds all three itself:

Secret	Lives	Reachable as
Account signing keys	pds.wal, inside the data directory	The file, 0600
The ACME account key	tls/acme-account.json and the certificate keys under tls/zones/	The file, 0600
The Route53 credential	Nowhere on disk	The instance role, over IMDSv2

The DNS credential is the only one that is not a file, and deliberately: the instance profile in infra/pds/iam.tf grants ChangeResourceRecordSets/ListResourceRecordSets on the one hosted zone and GetChange unscoped. crates/didbot-dns/src/route53.rs reads that role's short-lived credentials over IMDSv2 itself, and signs its own SigV4 requests with them. Nothing is passed on the unit's command line and nothing is fetched into a file at boot.

The owner's DID is on the command line, not here. It is an identifier, not a credential — this server holds nothing that lets it write the owner's repository, which is the whole reason policy can constrain it.

What one compromised host gets an attacker: all three. They are on one box by design, and there is no ordering that changes the answer — the keys sign as any account the server serves, and the zone credential extends that to names it has never minted. Separating the credential from the keys is not a mitigation here and is not offered as one: it has to be reachable from this process to write DNS-01 challenges, so whoever holds the process holds it. What bounds a compromise instead is scope and detection, and the DNS credential above states both.

Backups and restore #

The commands are in operations; this section states what a restore is and is not.

infra/pds/backup.tf puts the data volume under an AWS Backup plan: daily at 09:00 UTC, snapshots kept thirty days. The volume is the whole of the state — the log and the blobs are both on it — so one snapshot is one consistent deployment, give or take writes in the seconds around it.

A restore recovers accounts, records, blobs, name holds and ledger history, because those are the log. It recovers the signing keys with them, which is what makes the restored accounts the same accounts rather than new ones with the same handles.

A restore does not recover four things, and each fails differently:

• Sessions. Every access and refresh token is in-process memory. A restore is a restart, so every signed-in client is logged out, and the failure is indistinguishable from a revoked session.
• The DNS zone. Snapshots are of a volume, not of Route53. A zone that was also lost has to be recreated and re-delegated at the registrar before any hostname resolves, and the restored server's records are written by the server itself once it is running. A zone that survived is ahead of the volume instead: it holds an address record for every account provisioned after the snapshot, pointed at the instance, and the restored server has no account behind any of them. The reconciler's survey reports those as unattributed and repairs nothing — it has no variant that deletes — and the stale sweep walks the account store, so it never reaches them either. Deleting them is the operator's, in Route53.
• A usable certificate. tls/ is on the volume, so a certificate restored within its validity is served immediately. One restored from a snapshot older than the certificate is expired, and the server will re-issue over ACME DNS-01 — which needs the zone above to be working first, and is subject to the issuer's rate limits on a name that has been issued repeatedly.
• Anything written since the snapshot. Up to twenty-four hours of accounts and records. There is no point-in-time recovery here and no log shipping.

The restore drill. Step 4 is the whole of it: a snapshot that attaches and mounts proves the filesystem, not the log.

1. Create a volume from a recovery point in the didbot-pds vault, in the instance's availability zone.
2. Attach it to a fresh instance — never to the running one; two mounts of one log is the corruption pds.lock exists to refuse.
3. Mount it at /data, where the fresh instance's boot script chowns /data/pds to uid 10001, and run didbot-pds --data /data/pds against a development zone.
4. Confirm the accounts that should be there are there: bot.did.listAgents returns the expected count, one agent's did.json resolves out of the restored keys, and a blob that account referenced fetches back byte for byte.
5. Detach and delete the test volume.

Step 4 runs in this workspace against a copied data directory rather than a volume: crates/didbot-serve/tests/restore_drill.rs takes a snapshot of a live directory, serves the copy, and requires the two servers to answer the same listAgents, getRecord, did.json and bot.did.stats — with the blob's bytes fetched back through getBlob. What steps 1 through 3 add is the volume itself and the uid the instance's boot script sets.

Where to run it #

One server serves many agent hosts. Agents are grouped, and a compromise of one host does not reach the others.

Posture	A compromised agent reaches
Server and agents on one laptop	Everything: keys, zone credential, every account, and the operator's own logins beside them.
Server in the cloud, agents on a laptop	That laptop's session credentials, its node credential, and the ability to mis-stamp identity for later sessions there.
Server in the cloud, agents in cloud sandboxes	One sandbox, for as long as it lives.

The first row is the development posture. The step to the second is the one that matters and the cheapest: it moves the keys and the zone credential off the machine running agent code. The third also makes browser isolation moot — one sandbox per session is one browser profile per agent — and allows instance-identity attestation, which is stronger evidence than a credential file on a shared host.

Agents are capable and sandboxing between them is not guaranteed. Separating agent hosts from the server host is achievable; separating agents from each other on one host is not. Record what cannot be enforced.

Configuration tiers #

Tier	Holds	Reachable on the host
The owner's repository	What an agent may hold: ceilings, admitted apps, collection permissions	No
Process arguments and the configuration file (didbot-config, see config)	How the server runs: bind address, zone, state directory, blob and disclosure settings, which DID's policy to read	Yes
Local state	Caches, refusal records, the emergency stop	Yes

Nothing below the owner's repository may widen what an agent may do. The management surface is not a replacement for configuration: what the process needs in order to start is an argument, and what an agent may do is a record somewhere else.

Clocks #

An attestation claim is valid inside a window. A host with a wrong clock cannot provision, and the failure does not look like a clock problem.