---
id: deploy
title: A named version runs somewhere that is not a laptop
status: open
crates: [didbot-serve, didbot-pds]
dependsOn: [agent-accounts]
exitCriterion: >
  A tagged release runs on a host nobody develops on, serving agents over real
  DNS and TLS, and a documented upgrade replaces it without losing a record.
---

# deploy

Everything here runs from a checkout. Nothing says what ships, where it puts
its state, how it is upgraded, or what happens when a certificate expires.

The security shape is settled and is a constraint on this epic rather than a
question for it: the server runs on a host that is not an agent host, the zone
credential is scoped to exactly the one hosted zone this deployment holds
and sits on the same host as the signing keys — the server writes its own
DNS, so it holds its own Route53 credential — and policy is read from the
operator's repository.
[Deployment](../docs/deployment.md) states those; this is what it takes to
satisfy them.

The marketing site is a separate deployment target from the server this epic
ships: [site](site.md) is a static build behind S3 and CloudFront, with its
own bucket and distribution to provision — see that epic's open items.

## What ships

- [ ] **Certificates: issuance and renewal**, and what a failed renewal does.
      An expired wildcard stops every agent resolving at once. `didbot-tls`
      now carries the ACME DNS-01 orchestration, the renewal clock, and
      `0600`-inside-`0700` storage for both the certificate and the ACME
      account key — see its Done entry below for what is and is not verified.
      `Route53Dns` and its multi-value TXT support have landed and
      `didbot-tls` publishes DNS-01 through them, one order per zone.
      Left open here because nothing has issued a real certificate yet: that
      needs a real zone, and network access to a directory, neither of which
      exists in the environment this was built in.

## Staying up

- [ ] **Backups and restore.** `infra/pds/backup.tf` puts the data volume — the
      write-ahead log and the blob store both live on it — under an AWS
      Backup plan, daily, kept thirty days.
      [Deployment](../docs/deployment.md) states what a restore recovers
      (accounts, records, blobs, name holds, ledger history, and the signing
      keys that make them the same accounts rather than new ones with the
      same handles), what it does not (every session, the Route53 zone, a
      certificate older than its own validity, and up to a day of writes),
      and the five-step drill that would prove it. Open because nobody has
      run that drill: there is no AWS account to apply this against, and a
      snapshot that has never been read back is not a backup. Closing this
      item is the drill succeeding, not the plan existing.
- [ ] **Upgrades across a log format change.** Refusing by name has landed:
      a data directory carries a `pds.layout` stamp, it is checked before the
      log is read, and a mismatch stops the server with a message naming both
      layouts — see [local-dev](local-dev.md). What is still open here is the
      other half, which is a deployment's rather than a developer's: reading an
      older log, or converting one, so that an upgrade does not mean deleting
      the accounts.
- [ ] **Say what a restart does** to sessions holding credentials and to a write
      in flight, and whether downtime is acceptable or has to be avoided.
      OAuth grants are in the log and survive one; a sign-in in flight —
      a pushed authorization request, a consent reference, an authorization
      code — is in-process memory and does not, so a client mid-sign-in
      starts over. The restart-persistence issue (`3mviwkxahvt2s`) decides
      what else a restart keeps, and where each thing lives.
- [ ] **Rollback**, including whether an older binary can read a newer log.
      Unaddressed here: the `pds.layout` stamp refuses a mismatched binary by
      name rather than allowing a downgrade, so today's answer is "restore
      the backup taken before the upgrade," not "run the old binary."

## Reloading, per component

- [ ] **Write down what changes without a restart and what does not.** Policy is
      a poll and needs nothing. Hooks need a new session. Prompt text needs a
      rebuild. Configuration is process arguments and needs a restart by
      design. The record host depends on its transport: an HTTP one restarts
      like any service, a stdio one is owned by the harness and comes back when
      the harness spawns it. **The TLS certificate needs no restart either.**
      `didbot-tls` builds one `rustls::ServerConfig` for the life of the
      process, backed by a certificate resolver
      (`didbot_tls::resolver::SwappableCert`) that a renewal swaps under an
      atomic pointer store. A handshake already in progress or a connection
      already established — the firehose holds these for hours to days — is
      unaffected; only the *next* handshake sees the new certificate. Proven
      with a real TCP/TLS loop in `crates/didbot-tls/tests/rotation.rs`: a
      connection held open across a swap keeps working, and a fresh
      connection made after the swap gets the new certificate. Nothing else
      about the listener, the router or the process is rebuilt.
- [ ] **Agent hosts are their own deployment.** The node agent, the hook binary
      and the harness settings all live there and all need updating, and
      [credentials](credentials.md) has the question of how a host knows the
      hook it runs is the one that was built.

## Knowing it is alive

- [ ] **Alert on the things that fail quietly:** certificate expiry, a policy
      poll that has not succeeded, clock skew against the attestation window,
      and disk. [alerts](alerts.md) is the channel — this server writing a
      record with its own key — and carries the reason that channel cannot be
      trusted as an alarm: the failures most worth hearing about are the ones
      that stop the alert being read. Certificate renewal has the escalation
      logic now -- `didbot_tls::renew::RenewalTracker::alert_level` climbs
      from `Ok` to `Warning` after repeated failures and to `Critical` inside
      a fixed window of actual expiry, tested without a network in
      `crates/didbot-tls/src/renew.rs`, and `GET /health`'s `certificates`
      carries every zone's level so `infra/pds/monitoring.tf`'s Route53 health
      check and CloudWatch alarm can read it. The other three failures above,
      and [alerts](alerts.md)'s own channel, are what is left here.
- [ ] **Debugging without a redeploy.** Log level changeable at runtime, and a
      way to ask why one request was refused that does not require
      [ops-dashboard](ops-dashboard.md) to exist yet.
- [ ] **Capacity.** [pds-writes](pds-writes.md) carries the load question;
      deployment is where an answer becomes a number.

## TLS and DNS, mid-revision

The project owner corrected the original design here: ALB-terminated TLS put
plaintext on the wire between the load balancer and the instance, and a
Terraform-managed wildcard DNS record meant the server never wrote DNS at
all — removing a capability [agent-accounts](agent-accounts.md) required
from the start. `infra/pds/` has been reworked to match (the ALB and its ACM
certificates are gone; `infra/pds/route53.tf` now creates only the hosted zone;
`infra/pds/iam.tf` grants the instance write on it; the instance takes an
Elastic IP and is reached on 443 directly). The three items below are what
is left, and all three depend on work sibling agents are doing in parallel
that had not landed as this revision was written.

## Done

- [x] **Two roots, one state bucket.** `infra/pds/` is this deployment and
      `infra/site/` is the did.bot website; each has its own key in the
      did.bot account's state bucket, so applying one cannot lock or roll
      back the other. The wire between them is one value: `infra/pds/`'s
      `name_servers` output goes into `infra/site/`'s
      `pds_zone_name_servers`, which writes the `NS` record at `pds.did.bot`
      that makes this zone answer.
- [x] **The unit is a container image under one systemd unit.** Not binaries
      and unit files, not an orchestrator; `scripts/` is a developer's
      interface and no part of a deployment.
      [Deployment](../docs/deployment.md)'s "what runs" states it and the
      three process properties that decide it. The deciding one is that the
      store is a single-writer log: `infra/pds/ec2.tf` is one instance rather
      than an autoscaling group, and the unit runs the container under a
      fixed `--name` behind an `ExecStartPre` that removes a previous one, so
      a restart racing a slow shutdown cannot produce two. That is the unit
      being careful, and it does not bind an operator with a shell, so the
      exclusion no longer rests on it: `didbot_pds::lock` takes an exclusive
      `flock` on `pds.lock` in the data directory before the layout check and
      holds it for the life of the `Durable`. A second opener — a hand-run
      binary against the mounted volume — is refused by name before it reads
      or writes a byte, and the kernel releases the lock when the holder
      dies, so a crash leaves no stale lock to clear. One existing test was
      opening two `Durable`s on one directory and is now scoped; that it had
      to change is the point.
- [x] **A data directory layout, and permissions on one this server did not
      create.** [Deployment](../docs/deployment.md) tables what is in it:
      `pds.wal` (which holds the account signing keys), `pds.wal.compact`,
      `pds.lock`, `pds.layout`, `blobs/` with its `.incoming/`, and `tls/`
      with `zones/<zone>/` beneath it. Permissions do not depend on who made
      the directory: `didbot_pds::wal::create_dir` `chmod`s an existing
      directory to `0700` on every open rather than trusting the umask that
      made it, and `didbot_tls::storage::create_dir` does the same for
      `tls/`; files inside are created `0600`. Ownership is the one thing
      this cannot fix — a directory owned by another user fails to write and
      says so, which is right but is not a diagnosis — and it is the boot
      script's to get right.
- [x] **Secrets, and where each lives.** Three, and this process holds all
      three: the account signing keys, in `pds.wal`; the ACME account key and
      the per-zone certificate keys, under `tls/`; and the Route53 credential,
      which is not on disk at all — it is the instance role, read over IMDSv2,
      granted `ChangeResourceRecordSets` on the one hosted zone and
      `GetChange` on changes. Nothing is on the unit's command line. The operator DID is an
      identifier rather than a credential, and this server holds nothing that
      can write the operator's repository, which is why policy can constrain it.
      One compromised host gets an attacker all three; separating the
      credential from the keys is not a mitigation on a single host and
      [deployment](../docs/deployment.md) does not offer it as one, because
      the credential has to be reachable from the process to write DNS-01
      challenges. Scope and detection are what bound a compromise.
- [x] **The zone layout, in the server rather than in Terraform.** The
      server at the apex, agents at `*.<zone>` for every zone the deployment
      is configured with — the recommended shape
      [agent-accounts](agent-accounts.md) describes, unenforced but unchanged
      by this move, and no longer Terraform's to enforce. `route53.tf`
      creates the hosted zone, the apex record and the `*.<zone>` wildcard
      every agent hostname resolves through, all at the instance's Elastic
      IP. Which zones exist is the `didbot-identity`/`didbot-pds` zone
      registry's decision: `didbot-pds` takes `--zone` once per zone and
      `--route53-zone-id <zone>=<id>` for each, writes each zone's
      `_acme-challenge` and `CAA` records through one `Route53Dns` per zone,
      and terminates TLS for all of them from one listener — a `<zone>` +
      `*.<zone>` certificate each, since a wildcard matches one label.
      Minting still happens under the first zone only; that half is
      [zone-scale](zone-scale.md)'s.
- [x] **Agent hostnames resolve through the zone's wildcard record.**
      `aws_route53_record.agents` in `route53.tf` is `*.<zone>` at the
      instance's Elastic IP, beside the apex. The server writes no record per
      agent, so the records it holds a credential for are the
      `_acme-challenge` TXT values and the apex `CAA`, and the grant in
      `iam.tf` is `ChangeResourceRecordSets` on the one zone plus
      `GetChange`. This server never creates a hosted zone — it is handed
      the zones it may write, the posture `dns.may_create_zones` describes.
- [x] **Certificates: issuance and renewal, in the instance.** The server
      terminates TLS itself over a certificate it obtains and renews
      in-process, and `infra/pds/templates/user_data.sh.tftpl`'s `ExecStart`
      passes `--tls acme`, `--acme-environment` and the hosted zone id the
      DNS-01 challenge is published into; no credential is on that command
      line, since `Route53Dns` reads the instance role over IMDSv2. What a
      failed renewal does is written down in
      [deployment](../docs/deployment.md): the existing certificate keeps
      being served, the failure is logged with the zone and the expiry it is
      measured against, and the level escalates before expiry rather than at
      it. `GET /health` reports the same levels, and `infra/pds/monitoring.tf`
      alarms on them — see "alert on the things that fail quietly" above for
      the failures that channel still covers.
- [x] **The certificate hot-reload seam, and TLS termination itself.**
      `didbot-tls` obtains a wildcard-covering certificate over ACME DNS-01
      (`instant-acme`, on `ring` — its `hyper-rustls` feature is deliberately
      left off because it pulls `aws-lc-rs` in regardless, checked with
      `cargo tree -i aws-lc-rs`), stores it and the ACME account key `0600`
      inside `0700`, and serves it through `didbot_serve::serve_tls` via a
      certificate resolver a renewal swaps atomically rather than a listener
      anything rebinds. What is genuinely exercised without a network: the
      renewal clock and its escalating alert level, the DNS-01 seam's
      handling of two identifiers sharing one challenge name, storage
      permissions, and — the property [deployment](../docs/deployment.md)'s
      firehose case depends on — that an established connection survives a
      certificate swap while a fresh one sees the new certificate.
      `tests/acme_conformance.rs` now drives the real `instant-acme` client
      through a full order — directory, nonce, account, order, both
      authorizations, `dns-01` challenges, finalize, certificate — against
      an in-process fake ACME directory, covering the apex+wildcard grouping
      landing on a CA-shaped server, a challenge that never validates, a
      renewal that keeps failing, a crash between publishing a challenge and
      withdrawing it (and the next order replacing the record it stranded),
      and two zones ordering concurrently. What is still not
      exercised: a real CA, and DNS-01 propagation over a real network —
      there is no network access to one from this environment.
      `crates/didbot-tls/src/dns.rs`'s seam is wired to `didbot-dns`'s
      multi-value TXT support through `DnsProviderAdapter`.