---
id: deploy
title: A named version runs somewhere that is not a laptop
status: open
crates: [didbot-serve, didbot-pds]
dependsOn: [agent-accounts]
exitCriterion: >
  A tagged release runs on a host nobody develops on, serving agents over real
  DNS and TLS, and a documented upgrade replaces it without losing a record.
---

# deploy

Everything here runs from a checkout. Nothing says what ships, where it puts
its state, how it is upgraded, or what happens when a certificate expires.

The security shape is settled and is a constraint on this epic rather than a
question for it: the server runs on a host that is not an agent host, the zone
credential is scoped to exactly the one hosted zone this deployment holds
and sits on the same host as the signing keys — the server writes its own
DNS, so it holds its own Route53 credential — and policy is read from the
owner's repository.
[Deployment](../docs/deployment.md) states those; this is what it takes to
satisfy them.

The marketing site is a separate deployment target from the server this epic
ships: [site](site.md) is a static build behind S3 and CloudFront, with its
own bucket and distribution to provision — see that epic's open items.

## What ships

- [ ] **Certificates: issuance and renewal**, and what a failed renewal does.
      An expired wildcard stops every agent resolving at once. `didbot-tls`
      now carries the ACME DNS-01 orchestration, the renewal clock, and
      `0600`-inside-`0700` storage for both the certificate and the ACME
      account key — see its Done entry below for what is and is not verified.
      `Route53Dns` and its multi-value TXT support have landed and
      `didbot-tls` publishes DNS-01 through them, one order per zone.
      Left open here because nothing has issued a real certificate yet: that
      needs a real zone, and network access to a directory, neither of which
      exists in the environment this was built in.

## Staying up

- [ ] **Backups and restore.** `infra/backup.tf` puts the data volume — the
      write-ahead log and the blob store both live on it — under an AWS
      Backup plan, daily, kept thirty days.
      [Deployment](../docs/deployment.md) states what a restore recovers
      (accounts, records, blobs, name holds, ledger history, and the signing
      keys that make them the same accounts rather than new ones with the
      same handles), what it does not (every session, the Route53 zone, a
      certificate older than its own validity, and up to a day of writes),
      and the five-step drill that would prove it. Open because nobody has
      run that drill: there is no AWS account to apply this against, and a
      snapshot that has never been read back is not a backup. Closing this
      item is the drill succeeding, not the plan existing.
- [ ] **Upgrades across a log format change.** Refusing by name has landed:
      a data directory carries a `pds.layout` stamp, it is checked before the
      log is read, and a mismatch stops the server with a message naming both
      layouts — see [local-dev](local-dev.md). What is still open here is the
      other half, which is a deployment's rather than a developer's: reading an
      older log, or converting one, so that an upgrade does not mean deleting
      the accounts.
- [ ] **Say what a restart does** to sessions holding credentials and to a write
      in flight, and whether downtime is acceptable or has to be avoided.
      Answered for legacy sessions already, and it is the answer this bullet
      needs to reckon with: `didbot_pds::session::SessionAuth` is in-process
      memory with no durable or shared backing, so **a restart silently
      invalidates every outstanding access and refresh token** — every signed-in
      client is logged out with no error distinguishing it from a revoked
      session, and a deployment running more than one instance has no way to
      share sessions between them at all. `plan/pds-xrpc.md`'s security review
      (finding 08) named this and it is deliberately left open here rather
      than fixed alongside the same review's other two findings: a durable or
      shared session store is a real design question — where it lives, what
      it costs per lookup, how it survives the store this deployment already
      has for records — not a bug fix, and belongs to this epic's "what ships"
      decision rather than to a patch.
- [ ] **Rollback**, including whether an older binary can read a newer log.
      Unaddressed here: the `pds.layout` stamp refuses a mismatched binary by
      name rather than allowing a downgrade, so today's answer is "restore
      the backup taken before the upgrade," not "run the old binary."

## Reloading, per component

- [ ] **Write down what changes without a restart and what does not.** Policy is
      a poll and needs nothing. Hooks need a new session. Prompt text needs a
      rebuild. Configuration is process arguments and needs a restart by
      design. The scrobble host depends on its transport: an HTTP one restarts
      like any service, a stdio one is owned by the harness and comes back when
      the harness spawns it. **The TLS certificate needs no restart either.**
      `didbot-tls` builds one `rustls::ServerConfig` for the life of the
      process, backed by a certificate resolver
      (`didbot_tls::resolver::SwappableCert`) that a renewal swaps under an
      atomic pointer store. A handshake already in progress or a connection
      already established — the firehose holds these for hours to days — is
      unaffected; only the *next* handshake sees the new certificate. Proven
      with a real TCP/TLS loop in `crates/didbot-tls/tests/rotation.rs`: a
      connection held open across a swap keeps working, and a fresh
      connection made after the swap gets the new certificate. Nothing else
      about the listener, the router or the process is rebuilt.
- [ ] **Agent hosts are their own deployment.** The node agent, the hook binary
      and the harness settings all live there and all need updating, and
      [credentials](credentials.md) has the question of how a host knows the
      hook it runs is the one that was built.

## Knowing it is alive

- [ ] **Alert on the things that fail quietly:** certificate expiry, a policy
      poll that has not succeeded, clock skew against the attestation window,
      and disk. [alerts](alerts.md) is the channel — this server writing a
      record with its own key — and carries the reason that channel cannot be
      trusted as an alarm: the failures most worth hearing about are the ones
      that stop the alert being read. Certificate renewal has the escalation
      logic now -- `didbot_tls::renew::RenewalTracker::alert_level` climbs
      from `Ok` to `Warning` after repeated failures and to `Critical` inside
      a fixed window of actual expiry, tested without a network in
      `crates/didbot-tls/src/renew.rs` -- but it only reaches `tracing` today,
      not [alerts](alerts.md)'s channel.
- [ ] **Debugging without a redeploy.** Log level changeable at runtime, and a
      way to ask why one request was refused that does not require
      [ops-dashboard](ops-dashboard.md) to exist yet.
- [ ] **Capacity.** [pds-writes](pds-writes.md) carries the load question;
      deployment is where an answer becomes a number.

## TLS and DNS, mid-revision

The project owner corrected the original design here: ALB-terminated TLS put
plaintext on the wire between the load balancer and the instance, and a
Terraform-managed wildcard DNS record meant the server never wrote DNS at
all — removing a capability [agent-accounts](agent-accounts.md) required
from the start. `infra/` has been reworked to match (the ALB and its ACM
certificates are gone; `infra/route53.tf` now creates only the hosted zone;
`infra/iam.tf` grants the instance write on it; the instance takes an
Elastic IP and is reached on 443 directly). The three items below are what
is left, and all three depend on work sibling agents are doing in parallel
that had not landed as this revision was written.

## Done

- [x] **The unit is a container image under one systemd unit.** Not binaries
      and unit files, not an orchestrator; `scripts/` is a developer's
      interface and no part of a deployment.
      [Deployment](../docs/deployment.md)'s "what runs" states it and the
      three process properties that decide it. The deciding one is that the
      store is a single-writer log: `infra/ec2.tf` is one instance rather
      than an autoscaling group, and the unit runs the container under a
      fixed `--name` behind an `ExecStartPre` that removes a previous one, so
      a restart racing a slow shutdown cannot produce two. That is the unit
      being careful, and it does not bind an operator with a shell, so the
      exclusion no longer rests on it: `didbot_pds::lock` takes an exclusive
      `flock` on `pds.lock` in the data directory before the layout check and
      holds it for the life of the `Durable`. A second opener — a hand-run
      binary against the mounted volume — is refused by name before it reads
      or writes a byte, and the kernel releases the lock when the holder
      dies, so a crash leaves no stale lock to clear. One existing test was
      opening two `Durable`s on one directory and is now scoped; that it had
      to change is the point.
- [x] **A data directory layout, and permissions on one this server did not
      create.** [Deployment](../docs/deployment.md) tables what is in it:
      `pds.wal` (which holds the account signing keys), `pds.wal.compact`,
      `pds.lock`, `pds.layout`, `blobs/` with its `.incoming/`, and `tls/`
      with `zones/<zone>/` beneath it. Permissions do not depend on who made
      the directory: `didbot_pds::wal::create_dir` `chmod`s an existing
      directory to `0700` on every open rather than trusting the umask that
      made it, and `didbot_tls::storage::create_dir` does the same for
      `tls/`; files inside are created `0600`. Ownership is the one thing
      this cannot fix — a directory owned by another user fails to write and
      says so, which is right but is not a diagnosis — and it is the boot
      script's to get right.
- [x] **Secrets, and where each lives.** Three, and this process holds all
      three: the account signing keys, in `pds.wal`; the ACME account key and
      the per-zone certificate keys, under `tls/`; and the Route53 credential,
      which is not on disk at all — it is the instance role, read over IMDSv2,
      granted `ChangeResourceRecordSets`/`ListResourceRecordSets` on the one
      hosted zone. Nothing is on the unit's command line. The owner DID is an
      identifier rather than a credential, and this server holds nothing that
      can write the owner's repository, which is why policy can constrain it.
      One compromised host gets an attacker all three; separating the
      credential from the keys is not a mitigation on a single host and
      [deployment](../docs/deployment.md) does not offer it as one, because
      the credential has to be reachable from the process to write DNS-01
      challenges. Scope and detection are what bound a compromise.
- [x] **The zone layout, in the server rather than in Terraform.** The
      server at the apex, agents at `*.<zone>` for every zone the deployment
      is configured with — the recommended shape
      [agent-accounts](agent-accounts.md) describes, unenforced but unchanged
      by this move, and no longer Terraform's to enforce. An earlier revision of this stack had `acm.tf`/`route53.tf`/
      `alb.tf` create a certificate and a wildcard `ALIAS` record per zone,
      `for_each` over `var.zones`; `acm.tf` and `alb.tf` are gone and
      `route53.tf` now creates only the hosted zone. Which zones exist and
      what resolves under them is entirely the `didbot-identity`/
      `didbot-pds` zone registry's decision now, written through the
      Route53 `DnsProvider` in `crates/didbot-dns`. `didbot-dev` takes
      `--zone` once per zone and `--route53-zone-id <zone>=<id>` for each,
      builds a `MultiZoneDns` over one backend per zone, and terminates TLS
      for all of them from one listener — a `<zone>` + `*.<zone>`
      certificate each, since a wildcard matches one label. Minting still
      happens under the first zone only; that half is
      [zone-scale](zone-scale.md)'s.
- [x] **`WildcardDns` is being replaced.** The production DNS backend
      (`crates/didbot-dns`) that published nothing, on the theory that a
      Terraform-managed wildcard `ALIAS` record already covered every
      hostname under a managed zone, no longer matches this stack: that
      record does not exist any more (see the item above), so a real
      `Route53` `DnsProvider` that actually calls the API is required, not
      optional. `Route53Dns` is that backend, and a zone with a hosted zone
      id configured now gets it: `WildcardDns` is left as the development
      stand-in for a zone with no id, publishing nothing and saying so at
      startup. This server never creates a hosted zone — it is handed the
      zones it may write, the posture `dns.may_create_zones` describes.
- [x] **Certificates: issuance and renewal, in the instance.** The server
      terminates TLS itself over a certificate it obtains and renews
      in-process, and `infra/templates/user_data.sh.tftpl`'s `ExecStart`
      passes `--tls acme`, `--acme-environment` and the hosted zone id the
      DNS-01 challenge is published into; no credential is on that command
      line, since `Route53Dns` reads the instance role over IMDSv2. What a
      failed renewal does is written down in
      [deployment](../docs/deployment.md): the existing certificate keeps
      being served, the failure is logged with the zone and the expiry it is
      measured against, and the level escalates before expiry rather than at
      it. Nothing pages on it — "alert on the things that fail quietly"
      above is where that stays.
- [x] **The certificate hot-reload seam, and TLS termination itself.**
      `didbot-tls` obtains a wildcard-covering certificate over ACME DNS-01
      (`instant-acme`, on `ring` — its `hyper-rustls` feature is deliberately
      left off because it pulls `aws-lc-rs` in regardless, checked with
      `cargo tree -i aws-lc-rs`), stores it and the ACME account key `0600`
      inside `0700`, and serves it through `didbot_serve::serve_tls` via a
      certificate resolver a renewal swaps atomically rather than a listener
      anything rebinds. What is genuinely exercised without a network: the
      renewal clock and its escalating alert level, the DNS-01 seam's
      handling of two identifiers sharing one challenge name, storage
      permissions, and — the property [deployment](../docs/deployment.md)'s
      firehose case depends on — that an established connection survives a
      certificate swap while a fresh one sees the new certificate. What is
      not: the ACME protocol exchange has never run against a real directory,
      because there is neither network access to one from this environment
      because there is no network access to one from this environment.
      `crates/didbot-tls/src/dns.rs`'s seam is wired to `didbot-dns`'s
      multi-value TXT support through `DnsProviderAdapter`.