# Deployment

What running this safely requires. [The shape of the system](architecture.md)
draws the same hosts; this page states the rules, and
[operations](operations.md) is the procedure an operator runs against them.

## What runs

**A container image, started by one systemd unit, on one instance.** Not a
binary and a unit file; not an orchestrator. `scripts/` is a developer's
interface and no part of a deployment.

The image is built from the repository's `Dockerfile` with `--features
route53`, tagged with the release it holds, and pushed to ECR.
`infra/pds/templates/user_data.sh.tftpl` writes `/etc/systemd/system/
didbot-pds.service` at first boot, substituting the image reference and the
deployment's zone, owner and hosted zone id directly into the unit — so
`systemctl cat didbot-pds` shows exactly what is running, with no environment
file to also go read.

Three properties decide this, and each of them is a property of the process
rather than a preference:

- **The unit must be a single-writer service.** The store is one write-ahead
  log behind one lock, and two live writers interleave frames neither can
  replay. `infra/pds/ec2.tf` is one instance, not an autoscaling group; the unit
  runs the container under a fixed `--name` with an `ExecStartPre` that
  removes a previous one, so a restart racing a slow shutdown cannot produce
  two. Systemd is not the only way to start a process on a host, so the
  exclusion does not rest on it: `didbot_pds::lock` takes an exclusive
  `flock` on `pds.lock` inside the data directory before the log is opened,
  and an operator running the binary by hand against the mounted volume is
  refused by name. The kernel releases the lock when the process dies, so a
  crash leaves nothing to clean up.
- **The image is a name that can be said.** The exit criterion for
  [deploy](../plan/deploy.md) is that *a tagged release* runs on a host
  nobody develops on. A container digest is that name, and a `git pull` on
  the instance is not.
- **The process binds 443 and holds its own certificate.** Nothing terminates
  TLS in front of it, so the run unit uses `--network host` rather than a
  published-port mapping — which is also what lets the container's IMDSv2
  request reach the metadata endpoint at `http_put_response_hop_limit = 1`.
  Raising that to 2 is required if the run unit ever leaves host networking.
  The image runs as an unprivileged user, and with the host's network
  namespace there is no `net.ipv4.ip_unprivileged_port_start` to relax, so
  the binary carries `cap_net_bind_service=+ep` — one capability, bind a low
  port — rather than the container running as root.

Restarts are `Restart=on-failure` with `RestartSec=5`, capped at five failures
in two minutes so a crash loop stops rather than burning the instance. A
restart is not free: `didbot_pds::session::SessionAuth` is in-process memory,
so every outstanding access and refresh token is invalidated by one — see
"say what a restart does" in [deploy](../plan/deploy.md).

## The zone

One zone, used for everything: the server, the agent handles, the authority
serving every `did.json`, and the zone named in the owner's ownership record.
HTTPS throughout.

The server sits at the apex and agents in a subzone beneath it. Agents are off
the apex so a wildcard certificate cannot shadow the server's own name. The
zone does not have to sit at or below the server's own hostname — which server
serves a given agent is answered by resolving the agent's DID document, not by
comparing hostnames — but the recommended layout keeps them together.

Two hostnames per account — one for the DID, one for the handle — and both must
resolve before the account exists.

**One wildcard record makes both of them resolve.** The zone holds three names
whatever the account count: the apex, `*.<zone>`, and `_acme-challenge.<zone>`
while a certificate is being issued. Provisioning writes no record of its own —
`WildcardDns` is the provider for a zone in this shape, and it publishes
nothing because the wildcard already answers. [The names an agent
has](names.md) works one account through the whole path, from what exists in
the zone to what a stranger reads.

The cost is that there is no per-name existence guarantee and no DNS-level
garbage collection: a name no account holds still resolves, and answers 404
rather than `NXDOMAIN`. That is the correct trade here — DNS removal was never
a revocation mechanism, since a withdrawn name stays cached downstream for its
TTL, and an HTTP status is the only thing that can express a decommissioned
account still serving its document.

A deployment can still publish per-agent records instead; `Route53Dns` does. It
buys the per-name existence check, and it costs a hosted-zone record per
retained account, a network call on the provisioning path, and — because a real
record at a name stops the wildcard synthesizing anything beneath it — any
prospect of hostnames deeper than one label.

**A timer keeps the zone and the account store in agreement.** Provisioning
publishes an agent's record before it writes the account, and unwinds the
record itself on every error it returns through; a process killed between the
two runs none of that unwinding. `didbot_serve::stale_sweep` runs
`Provisioner::sweep_stale_bounded` hourly, deleting the accounts whose
retention has run out — a week for one that carries no window of its own —
and each deletion withdraws that account's hostname with it. One tick takes a
hundred accounts at most, since each is a call to the zone's provider, and the
next tick takes what the last left.

**What that timer can withdraw is what the startup zone read put in front of
it.** `Route53Dns` builds a withdrawal out of its own record of the record
set, because a Route53 `DELETE` names the type, TTL and every value it
removes; the two things that put a name there are a publish this process made
and the read it takes of the hosted zone at startup. A restored or restarted
process published none of the names its accounts resolve at, so until that
read lands the deployment serves every route normally, `bot.did.deleteAgent`
answers `NotPublished`, and the sweep takes nothing. The read is retried every
five minutes until it succeeds (`didbot_serve::zone_read`), and `GET /health`
carries the state under `zoneRecords`: `read` is the summary an alarm
expression matches on, `zones` names each zone and whether this process has
read it.

## The DNS credential

Control of the zone is control of every name in it. It is as powerful as the
signing keys, and it lives beside them: the server writes the
`_acme-challenge` TXT records its certificate depends on and the `CAA` record
below, so the credential has to be reachable from the process. Under a
wildcard those are the only records it writes on an ordinary day — agent
hostnames cost none, and zone creation is refused unless
`dns.may_create_zones` turns it on. Holding it elsewhere would protect
nothing: whoever holds this host holds the signing keys, and with those they
can sign as any account it serves.

So the question is not where the credential sits. It is what it reaches, and
what a holder of it can be caught doing.

### What it can do

Precisely, for the one hosted zone it is scoped to:

- **Point any name in the zone somewhere else.** A specific record at a name
  overrides the wildcard, so a live agent's two hostnames — the DID's and the
  handle's, whose `/.well-known/atproto-did` is what binds the two together —
  can be made to answer from another host, which then serves its own
  `did.json`. `did:web` has no history. A document swapped today changes what
  every signature that account ever made verifies against, retroactively, and
  nothing an observer kept proves what the document said yesterday.
- **Point the server's own hostname somewhere else**, when the zone contains
  it. That is every route, every handle answer, and the certificate for it.
- **Pass DNS-01 for any name in the zone**, including names no account holds,
  and obtain a publicly trusted certificate that validates. An impersonation
  built on either of the two above survives TLS.
- **Rewrite the `CAA` record**, which is the step that makes the one above
  reachable through an authority other than the one this zone authorises.

Serving a document at a name this server never minted is not on that list, and
it is worth saying why: under a wildcard every name in the zone already
resolves here without any DNS write at all. What answers at one is this
server's business, not DNS's — which is why a name no account holds answers
404 rather than `NXDOMAIN`, and why a zone that does publish per-agent records
gets a per-name existence check the wildcard cannot give.

What it does not reach is anything outside that one hosted zone — not other
zones in the account, and not the `NS` records at the registrar that delegate
into this one, which live in the parent zone. That boundary is the IAM policy
below and nothing else, which is why the policy is the load-bearing part.

### What is detectable

Detection is what bounds a compromise here, because isolation cannot. Three
channels, and they differ in the one way that matters — whether the evidence
is somewhere the holder of this host can reach.

**Certificate transparency.** Every publicly trusted certificate is logged by
the authority that issued it, to logs this deployment does not run. A
certificate naming any hostname in the zone that this server did not order is
visible there, and only there: no record of it exists on this host, and
nothing about the issuance touched it. Subscribe an external monitor to the
zone. This is the one channel a compromised host cannot suppress, and it is
after the fact — the certificate already exists when the log entry appears.

**A refused issuance, reported by the authority.** The `CAA` record the
server publishes at the zone apex names one authority and, per RFC 8657's
`accounturi`, the one ACME account at it this deployment holds the key for.
An attempt to issue for a name in this zone through any other account is
refused by any authority honouring the record, and reported to the `iodef`
contact in it — a report that arrives from the authority, not from this host.
This does not stop the holder of the zone credential, who can rewrite the
record. It makes doing so a second, separate act, and it closes issuance to
everyone who does not hold the credential.

**A zone holding a record nobody here wrote.** `ListResourceRecordSets` says
what the zone actually contains, and this server knows what it put there. A
wildcard zone makes that check almost trivial — it holds the apex, `*.<zone>`
and `_acme-challenge.<zone>`, and a fourth name is a name somebody else
added. A zone publishing per-agent records checks the same thing against the
ledger, which keeps a name past the account it described. Either way this is
observed on the host holding the credential, so it proves nothing against an
attacker who holds it; what it catches is a credential copy still live
somewhere, a second stack writing the same zone, or a change made by hand in
a console. Nothing runs the comparison today — `Route53Dns::resync` already
reads every record set in the zone, and no code diffs what it finds against
what this server expects; see `plan/agent-accounts.md`.

### What is not recoverable

Rotating the credential and rewriting the zone recovers the zone. It does not
recover any of the following, and no procedure does:

- **Confidence in anything signed before the compromise was noticed.** A
  swapped `did:web` document leaves no trace of the key it replaced. A record
  signed by the agent and a record signed with the substituted key are
  indistinguishable afterwards, in both directions — the compromise does not
  only forge new records, it makes the real ones unprovable.
- **Whatever resolved during the window.** A repointed name was cached by
  every resolver that asked, and served to every client that followed it.

A certificate is the exception: an issued certificate can be revoked, and the
authority that issued it is who to ask.

### The policy that bounds it

`crates/didbot-dns`'s `Route53Dns` is the provider that scopes this: an IAM
credential given only

```text
route53:ChangeResourceRecordSets, route53:ListResourceRecordSets
  on arn:aws:route53:::hostedzone/<HOSTED_ZONE_ID>
route53:GetChange
  on arn:aws:route53:::change/*
```

can write and read records in the one hosted zone this server owns, and
nothing else in the account. `GetChange` has to be unscoped to a zone because
Route53 change ids are not namespaced under one; it grants read of a change's
propagation status, not write of anything. A cloud DNS service is the right
kind of provider for exactly this reason — the credential narrows to a single
zone — and across several servers that narrowing is the blast-radius boundary
between them. See the module docs on `Route53Dns` for the full policy JSON
and for how it handles eventual consistency, rate limits and hosted-zone
record limits, all things this posture note assumes are handled correctly one
layer down.

The boundaries this design actually rests on are elsewhere, and neither is
inside this host: the agent hosts are separate machines from this one, and the
policy that governs it lives in a repository this server holds no credential
for. A compromised server can break policy; it cannot change it.

## Certificates

The server terminates TLS itself. Nothing in front of it holds a certificate,
so it obtains and renews its own over ACME DNS-01, through the same zone
credential above (`crates/didbot-tls`, on `instant-acme`).

**One `<zone>` + `*.<zone>` pair per configured zone.** A wildcard *in a
certificate* matches exactly one label: `*.pds.did.bot` covers
`claudes.pds.did.bot` and does not cover `agent.claudes.pds.did.bot`. This is
not the DNS rule — a wildcard *record* of the same spelling answers at any
depth — so what bounds how deep an agent hostname may sit is the certificate,
not the zone. See [the names an agent has](names.md). A deployment configured with
`claudes.pds.did.bot` and `vllm.pds.did.bot` therefore holds two
certificates, each from one ACME order carrying both of that zone's
identifiers — the apex and the wildcard — whose two DNS-01 challenges share
one `_acme-challenge.<zone>` record and must be published together.
`didbot_tls::fleet::CertificateFleet` holds one manager per zone and
`ZoneResolver` picks between their certificates on the SNI name each
handshake carries. A name no configured zone covers is refused rather than
handed a certificate that does not name it.

**A renewal never rebinds the listener.** The certificate lives behind an
`ArcSwap` that `rustls` reads once per handshake, so a renewal is a pointer
store: connections already established — the firehose relays and indexers
hold open for hours — are untouched, and the next handshake sees the new
certificate.

**A failed renewal keeps serving what it has, and gets louder.** Renewal
starts 30 days before expiry and retries hourly; every failure leaves the
current certificate in place and is logged with the zone, the failure count
and the expiry it is measured against. Three consecutive failures raise that
zone to `Warning`; inside three days of expiry it is `Critical` whether or
not anything has failed, because at that point expiry itself is the risk.
`CertificateFleet::alert_level` reports the worst level any zone is at and
`zone_alert_levels` says which zone it came from. `GET /health` carries both,
under `certificates`: `worst` is the summary an alarm expression matches on,
`zones` names each zone and its own level. `infra/pds/monitoring.tf` is what
reads it — a Route53 health check string-matching `worst`, and a CloudWatch
alarm on that check — so a zone above `ok` and a server that stopped
answering raise the same alarm. An expired certificate is a total outage for
every hostname in that zone, agents included.

**The ACME directory a run is configured for decides what it serves.**
`acme_environment` is `staging` by default (`infra/pds/variables.tf`), and a
staging leaf is signed by an authority no client trusts. `infra/site/`'s
`pds_serves_a_production_certificate` is the other half of the same decision:
it decides whether did.bot's two-year HSTS header covers the names this stack
serves, and a browser that has seen that header refuses a certificate error
under every covered name with no way to click through. The two move together,
in one change: `acme_environment = "production"` beside
`pds_serves_a_production_certificate = true`, once a real certificate has
been issued and served. So the certificate
on disk is served only when `meta.json` records the directory this run is
configured for, and the saved ACME account is restored only when its
credentials record that same directory; otherwise the run registers an
account there and orders a certificate from it, which is what makes the
switch to `production` take effect on the first boot after it. A restart
under an unchanged environment places no order and serves the certificate
that is already on disk, unless that certificate cannot be served: a file
missing or unparsable, or a key that is not the certificate's, as a crash
mid-save can leave. Boot then orders a new one. The `CAA` record below is republished from the
account actually in hand before that order is placed, so a switch carries the
`accounturi` with it.

**The zone's `CAA` record is written by the server, on every start.** It is
published before the first order and restated on each start, naming the
authority the order is about to go to and the ACME account the order will be
placed through — `didbot_tls::dns::IssuancePolicy`, reading both off the live
objects rather than off configuration. That is deliberate: a `CAA` record
naming an account this deployment cannot use does not fail safe, it stops the
next renewal, and an expired certificate is a total outage for every hostname
in the zone. Reading both halves from the objects the order itself uses makes
that disagreement unreachable. A backend whose zone already says exactly this
skips the write. There is no `issuewild` property — RFC 8659 falls back to
`issue` for a wildcard request when none is present, and the wildcard is in
the same order — and failing to publish the record is a warning rather than a
startup failure, because a zone with no `CAA` record is the state every zone
starts in.

**The contact is what the authority warns.** `--acme-contact <uri>` is the
address Let's Encrypt registers against this deployment's ACME account and
sends an expiry warning to; `infra/pds/variables.tf`'s `acme_contact` supplies
it, and the unit carries the flag only when that variable is non-empty. Empty
is a supported deployment and the one this configuration defaults to: an ACME
account with no contact registers, orders and renews identically, so the
address buys a second warning beside `infra/pds/monitoring.tf`'s alarm rather
than the ability to hold a certificate. Demanding one would make a
deployment's boot depend on a field the CA does not require, and an instance
that does not come up is every agent on it offline. Terraform refuses anything
but a `mailto:` URI or the empty string at plan time, because a contact Let's
Encrypt rejects fails at account registration, which is a startup this
deployment does not come back from.

**Where the address goes.** Two places, both by design. `IssuancePolicy`
publishes it as the RFC 8659 `iodef` value of the zone's `CAA` record, which
is what points a refused issuance at somebody the authority can reach without
going through this host — so it is in public DNS, and the address to use is
one suited to that. The DNS backend logs the record set it wrote at `INFO`
when the set changes, so it is also in the instance's journal. The two public
surfaces publish their own complete lists: `wire::StatsResponse` is counts and
byte totals over this deployment's contents, and `GET /health` is the tick and
per-zone certificate alert levels.

**The server is handed the zones it may write.** Each zone needs a hosted
zone id configured for it (`--route53-zone-id <zone>=<id>`); a zone without
one publishes nothing and says so at startup. Creating a hosted zone, or
delegating a new one with `NS` records in the zone above, is a separate
posture — `dns.may_create_zones` in `didbot-config` — and is off by default.

## Announcing to a relay

A relay learns a personal data server exists by being told. This deployment
names one in the `relay_hostname` Terraform variable — `bsky.network` by
default, `""` to run naming none — and `infra/pds/templates/user_data.sh.tftpl`
puts it on the unit's `ExecStart` as `--relay-hostname`. That flag decides
which relay `com.atproto.sync.requestCrawl` and the deprecated
`notifyOfUpdate` are sent to; sending either is an operator action.

**Boot does not dial the relay.** The server builds a client for the named
host and hands it to the e-stop admin socket, and that is all it does with it
until somebody asks. A relay that is down while this instance comes up
therefore costs the instance nothing — there is no retry loop to wedge in and
no startup step to fail. A relay that goes away later, or one that answers
`HostBanned`, is felt by the operator who runs the command: the reply
distinguishes `accepted` from `refused` (with the relay's status and body)
from `unreachable` (with the transport error), and a call that gets no answer
gives up after ten seconds rather than holding the socket open.

**The command is reached over the data volume.** The unit passes
`--estop-socket /data/estop.sock`, which is `/data/pds/estop.sock` on the
instance, so an operator on a break-glass SSH or SSM session reaches it
without entering the container. Opening it is the whole authorization: the
server creates the directory it sits in, re-tightens that directory to `0700`
on every start and the socket to `0600`, so the callers are the server's own
uid and root. `ANNOUNCE` sends `requestCrawl`; `NOTIFY` sends
`notifyOfUpdate`, which upstream deprecates in its favor but still names for
resuming a relay's subscription after a gap.

```console
$ printf 'ANNOUNCE\n' | sudo socat - UNIX-CONNECT:/data/pds/estop.sock
OK {"result":"accepted"}
```

Under a `.localhost` zone the flag is read and the command is left unwired,
and the server says so at startup: no relay resolves a loopback name, so the
only thing wiring it up there could produce is a failure an operator asked
for.

## The data directory

One directory, `--data`, on a volume separate from the root volume so an AMI
upgrade or an instance resize can re-attach it rather than lose it. The
instance mounts it at `/data` and the container sees `/data/pds` as `/data`.

| Path | Holds | Mode |
|---|---|---|
| `pds.wal.NNNNNN` | The log, in segments: accounts, records, ledger, commits, name holds — and the account signing keys. The highest ordinal is appended to; the ones before the checkpoint's position are deleted once it is durable | `0600` |
| `pds.checkpoint` | The state as the shortest log that reproduces it, and the log position it covers; a restart reads the log from there | `0600` |
| `pds.checkpoint.incoming` | A checkpoint being written, renamed over `pds.checkpoint` once synced | `0600` |
| `records/pds.heap` | Record bodies, addressed by the slots the log carries | `0600` |
| `policy-evaluations.log.NNNNNN` | Denial rows the policy engine wrote down: an evaluation id, which policies fired, the scope atoms a narrowing cut, a hash of the payload — never the payload itself. Segmented the same way the log is, but trimmed by the age of a sealed segment against `[capacity] evaluation_log_retention_days`, not by a checkpoint | `0600` |
| `writes.log.NNNNNN` | One row per record a caller wrote or deleted: the commit's revision, the key, the app whose OAuth token presented it, and the operator policy digest that judged it. Segmented like the evaluation log, and trimmed by the age of a sealed segment against `[capacity] write_log_retention_days` | `0600` |
| `pds.lock` | The single-writer claim on this directory | `0600` |
| `pds.layout` | Which log shape wrote this directory | `0600` |
| `blobs/` | Blob content, addressed by CID | `0700` |
| `blobs/.incoming/` | Uploads in flight, discarded on restart | `0700` |
| `tls/` | The ACME account key | `0700` |
| `tls/zones/<zone>/` | That zone's certificate, its key, and issuance metadata | `0700` |

Everything else a deployment has — configuration, the e-stop file and socket —
is elsewhere and is not state.

**Permissions are set on every open, not only on creation.** The signing keys
are in the log, so the directory holding it is only as private as whoever
made it. A directory the server creates is `0700` from `DirBuilder::mode`; a
directory it was handed — an operator's `mkdir`, a mount point, a restored
snapshot's filesystem root — carries whatever umask made it.
`didbot_pds::wal::create_dir` therefore `chmod`s an existing directory to
`0700` every time it runs rather than trusting it, and
`didbot_tls::storage::create_dir` does the same for `tls/`. That costs one
syscall on the common path and closes the case that actually matters.

What it does not fix is ownership. A directory owned by another user is not
made safe by its mode, and the server does not check: it will fail to write
and say so, which is correct but is a boot failure rather than a diagnosis.
Ownership is the boot script's. The image runs as uid **10001**, pinned in
the `Dockerfile` rather than left to `useradd`, and
`user_data.sh.tftpl` chowns `/data/pds` to that uid before the unit starts —
which also re-homes a restored snapshot whose files were written by a
different uid.

### The log's budget

`--log-budget` is the number of bytes the log may reach; past it an append is
refused before a byte is written, the file stands exactly where it was, and
the write comes back as `507 StorageFull`. `infra/pds/variables.tf`'s
`log_budget_bytes` sets it, and takes a tenth of `data_volume_size_gb` unless
a number is pinned — **2 GiB on the default 20 GiB volume**. The remaining
nine tenths are the blob store's, which shares this volume and grows in
megabytes at a time where the log grows in hundreds of bytes.

What has to fit inside the number is the deployment's *state*, not its
history. Every start rewrites the log as one entry per live thing once the
history has run to `didbot_pds::COMPACT_RATIO` times the state, which at a few
hundred bytes an entry puts millions of live records inside 2 GiB. A released
name, a deleted account, an overwritten record and a reissued credential are
all history a compaction drops, so a log that reached its budget on churn is
under it again after a restart — `didbot-pds/tests/durability.rs` asserts that
round trip.

An operator sees this in three places. The startup line `the write-ahead log
will refuse writes past its budget` carries the `budget` and the bytes `held`,
on every boot. `compacted the write-ahead log` on a later boot says the
rewrite ran and how many entries it kept. And a deployment at its budget
answers `507` whose `message` names the bytes held, the budget, and the bytes
the refused entry wanted. A budget a compaction cannot bring the log below is
a deployment whose state alone exceeds the number: raise `log_budget_bytes`,
or raise `data_volume_size_gb` and with it the tenth this derives.

## Secrets, and where each lives

Three, and the process holds all three itself:

| Secret | Lives | Reachable as |
|---|---|---|
| Account signing keys | The log's segments and `pds.checkpoint`, inside the data directory | The files, `0600` |
| The ACME account key | `tls/acme-account.json` and the certificate keys under `tls/zones/` | The file, `0600` |
| The Route53 credential | Nowhere on disk | The instance role, over IMDSv2 |

The DNS credential is the only one that is not a file, and deliberately: the
instance profile in `infra/pds/iam.tf` grants
`ChangeResourceRecordSets`/`ListResourceRecordSets` on the one hosted zone and
`GetChange` unscoped. `crates/didbot-dns/src/route53.rs` reads that role's
short-lived credentials over IMDSv2 itself, and signs its own SigV4 requests
with them. Nothing is passed on the unit's command line
and nothing is fetched into a file at boot.

The owner's DID is on the command line, not here. It is an identifier, not a
credential — this server holds nothing that lets it write the owner's
repository, which is the whole reason policy can constrain it.

**What one compromised host gets an attacker: all three.** They are on one
box by design, and there is no ordering that changes the answer — the keys
sign as any account the server serves, and the zone credential extends that to
names it has never minted. Separating the credential from the keys is not a
mitigation here and is not offered as one: it has to be reachable from this
process to write DNS-01 challenges, so whoever holds the process holds it.
What bounds a compromise instead is scope and detection, and [the DNS
credential](#the-dns-credential) above states both.

## Backups and restore

The commands are in [operations](operations.md); this section states what a
restore is and is not.

`infra/pds/backup.tf` puts the data volume under an AWS Backup plan: daily at
09:00 UTC, snapshots kept thirty days. The volume is the whole of the state —
the log and the blobs are both on it — so one snapshot is one consistent
deployment, give or take writes in the seconds around it.

**A restore recovers accounts, records, blobs, name holds and ledger
history**, because those are the log. It recovers the signing keys with them,
which is what makes the restored accounts the *same* accounts rather than new
ones with the same handles.

**The policy evaluation log's segments keep their real modified times across
a restore**, so a snapshot older than `[capacity] evaluation_log_retention_days`
has its stale segments trimmed the moment the restored server opens the
directory — trim runs at startup by wall-clock age, not by anything the
snapshot itself records — so an operator restoring history specifically to
investigate old denials must raise the retention before starting the
restored server, not after. The write log's segments are trimmed the same
way, against `[capacity] write_log_retention_days`.

**A restore does not recover five things, and each fails differently:**

- **Sessions.** Every access and refresh token is in-process memory. A restore
  is a restart, so every signed-in client is logged out, and the failure is
  indistinguishable from a revoked session.
- **The DNS zone.** Snapshots are of a volume, not of Route53. A zone that was
  also lost has to be recreated and re-delegated at the registrar before any
  hostname resolves, and the restored server's records are written by the
  server itself once it is running. A zone that *survived* is ahead of the
  volume instead: it holds an address record for every account provisioned
  after the snapshot, pointed at the instance, and the restored server has no
  account behind any of them. The reconciler's survey reports those as
  unattributed and repairs nothing — it has no variant that deletes — and the
  stale sweep walks the account store, so it never reaches them either.
  Deleting them is the operator's, in Route53.
- **A usable certificate.** `tls/` is on the volume, so a certificate restored
  within its validity is served immediately. One restored from a snapshot
  older than the certificate is expired, and the server will re-issue over
  ACME DNS-01 — which needs the zone above to be working first, and is subject
  to the issuer's rate limits on a name that has been issued repeatedly.
- **Anything written since the snapshot.** Up to twenty-four hours of accounts
  and records. There is no point-in-time recovery here and no log shipping.
- **The position of `com.atproto.sync.subscribeRepos`.** The restart floor is
  whatever the restored log carries, so the stream resumes below the numbers
  the lost run handed out and issues those same numbers to different commits.
  A relay holding a cursor from the lost run is not told: from the restored
  server's side the numbers are new, and its repository heads are behind what
  the relay already applied, so the next commit the relay does receive names a
  `prev` it does not hold. **Reset every consumer of that stream after a
  restore rather than letting it resume.** `/firehose` is not affected — its
  cursor names the run that minted it, so a consumer of that one is told
  `OutdatedCursor` and reads the repositories for the gap.

**The restore drill.** Step 4 is the whole of it: a snapshot that attaches and
mounts proves the filesystem, not the log.

1. Create a volume from a recovery point in the `didbot-pds` vault, in the
   instance's availability zone.
2. Attach it to a *fresh* instance — never to the running one; two mounts of
   one log is the corruption `pds.lock` exists to refuse.
3. Mount it at `/data`, where the fresh instance's boot script chowns
   `/data/pds` to uid 10001, and run `didbot-pds --data /data/pds` against a
   development zone.
4. Confirm the accounts that should be there are there: `bot.did.listAgents`
   returns the expected count, one agent's `did.json` resolves out of the
   restored keys, and a blob that account referenced fetches back byte for
   byte.
5. Detach and delete the test volume.

Step 4 runs in this workspace against a copied data directory rather than a
volume: `crates/didbot-serve/tests/restore_drill.rs` takes a snapshot of a
live directory, serves the copy, and requires the two servers to answer the
same `listAgents`, `getRecord`, `did.json` and `bot.did.stats` — with the
blob's bytes fetched back through `getBlob`. What steps 1 through 3 add is
the volume itself and the uid the instance's boot script sets.

## Where to run it

One server serves many agent hosts. Agents are grouped, and a compromise of one
host does not reach the others.

| Posture | A compromised agent reaches |
|---|---|
| Server and agents on one laptop | Everything: keys, zone credential, every account, and the operator's own logins beside them. |
| Server in the cloud, agents on a laptop | That laptop's session credentials, its node credential, and the ability to mis-stamp identity for later sessions there. |
| Server in the cloud, agents in cloud sandboxes | One sandbox, for as long as it lives. |

The first row is the development posture. The step to the second is the one
that matters and the cheapest: it moves the keys and the zone credential off
the machine running agent code. The third also makes browser isolation moot —
one sandbox per session is one browser profile per agent — and allows
instance-identity attestation, which is stronger evidence than a credential
file on a shared host.

Agents are capable and sandboxing between them is not guaranteed. Separating
agent hosts from the server host is achievable; separating agents from each
other on one host is not. Record what cannot be enforced.

## The state bucket

`infra/pds/` and `infra/site/` keep their OpenTofu state in one S3 bucket, one
key each, so neither root can lock or overwrite the other's. Both roots are
partial backends: the key stays in `backend.tf`, and the bucket is read at init
time out of `backend.hcl`.

```sh
cd infra/pds && tofu init -backend-config=backend.hcl
```

The same line applies in `infra/site/`. Deploying to another AWS account means
pointing init at your own file -- `-backend-config=mine.hcl`, or
`-backend-config="bucket=..."` for a CI job that already holds the name in its
own configuration. Any versioned, encrypted, public-access-blocked bucket the
deployer can write works.

## Configuration tiers

| Tier | Holds | Reachable on the host |
|---|---|---|
| The owner's repository | What an agent may hold: ceilings, admitted apps, collection permissions | No |
| Process arguments and the configuration file (`didbot-config`, see [config](../plan/config.md)) | How the server runs: bind address, zone, state directory, blob and disclosure settings, which DID's policy to read | Yes |
| Local state | Caches, refusal records, the emergency stop | Yes |

Nothing below the owner's repository may widen what an agent may do. The
management surface is not a replacement for configuration: what the process
needs in order to start is an argument, and what an agent may do is a record
somewhere else.

## Clocks

An attestation claim is valid inside a window. A host with a wrong clock cannot
provision, and the failure does not look like a clock problem.