id: deploy title: A named version runs somewhere that is not a laptop status: open crates: [didbot-serve, didbot-pds] dependsOn: [agent-accounts] exitCriterion: > A tagged release runs on a host nobody develops on, serving agents over real DNS and TLS, and a documented upgrade replaces it without losing a record. #

deploy #

Everything here runs from a checkout. Nothing says what ships, where it puts its state, how it is upgraded, or what happens when a certificate expires.

The security shape is settled and is a constraint on this epic rather than a question for it: the server runs on a host that is not an agent host, the zone credential is scoped to exactly the one hosted zone this deployment holds and sits on the same host as the signing keys — the server writes its own DNS, so it holds its own Route53 credential — and policy is read from the owner's repository. Deployment states those; this is what it takes to satisfy them.

The marketing site is a separate deployment target from the server this epic ships: site is a static build behind S3 and CloudFront, with its own bucket and distribution to provision — see that epic's open items.

What ships #

• Certificates: issuance and renewal, and what a failed renewal does. An expired wildcard stops every agent resolving at once. didbot-tls now carries the ACME DNS-01 orchestration, the renewal clock, and 0600-inside-0700 storage for both the certificate and the ACME account key — see its Done entry below for what is and is not verified. Route53Dns and its multi-value TXT support have landed and didbot-tls publishes DNS-01 through them, one order per zone. Left open here because nothing has issued a real certificate yet: that needs a real zone, and network access to a directory, neither of which exists in the environment this was built in.

Staying up #

• Backups and restore. infra/backup.tf puts the data volume — the write-ahead log and the blob store both live on it — under an AWS Backup plan, daily, kept thirty days. Deployment states what a restore recovers (accounts, records, blobs, name holds, ledger history, and the signing keys that make them the same accounts rather than new ones with the same handles), what it does not (every session, the Route53 zone, a certificate older than its own validity, and up to a day of writes), and the five-step drill that would prove it. Open because nobody has run that drill: there is no AWS account to apply this against, and a snapshot that has never been read back is not a backup. Closing this item is the drill succeeding, not the plan existing.
• Upgrades across a log format change. Refusing by name has landed: a data directory carries a pds.layout stamp, it is checked before the log is read, and a mismatch stops the server with a message naming both layouts — see local-dev. What is still open here is the other half, which is a deployment's rather than a developer's: reading an older log, or converting one, so that an upgrade does not mean deleting the accounts.
• Say what a restart does to sessions holding credentials and to a write in flight, and whether downtime is acceptable or has to be avoided. Answered for legacy sessions already, and it is the answer this bullet needs to reckon with: didbot_pds::session::SessionAuth is in-process memory with no durable or shared backing, so a restart silently invalidates every outstanding access and refresh token — every signed-in client is logged out with no error distinguishing it from a revoked session, and a deployment running more than one instance has no way to share sessions between them at all. plan/pds-xrpc.md's security review (finding 08) named this and it is deliberately left open here rather than fixed alongside the same review's other two findings: a durable or shared session store is a real design question — where it lives, what it costs per lookup, how it survives the store this deployment already has for records — not a bug fix, and belongs to this epic's "what ships" decision rather than to a patch.
• Rollback, including whether an older binary can read a newer log. Unaddressed here: the pds.layout stamp refuses a mismatched binary by name rather than allowing a downgrade, so today's answer is "restore the backup taken before the upgrade," not "run the old binary."

Reloading, per component #

• Write down what changes without a restart and what does not. Policy is a poll and needs nothing. Hooks need a new session. Prompt text needs a rebuild. Configuration is process arguments and needs a restart by design. The scrobble host depends on its transport: an HTTP one restarts like any service, a stdio one is owned by the harness and comes back when the harness spawns it. The TLS certificate needs no restart either. didbot-tls builds one rustls::ServerConfig for the life of the process, backed by a certificate resolver (didbot_tls::resolver::SwappableCert) that a renewal swaps under an atomic pointer store. A handshake already in progress or a connection already established — the firehose holds these for hours to days — is unaffected; only the next handshake sees the new certificate. Proven with a real TCP/TLS loop in crates/didbot-tls/tests/rotation.rs: a connection held open across a swap keeps working, and a fresh connection made after the swap gets the new certificate. Nothing else about the listener, the router or the process is rebuilt.
• Agent hosts are their own deployment. The node agent, the hook binary and the harness settings all live there and all need updating, and credentials has the question of how a host knows the hook it runs is the one that was built.

Knowing it is alive #

• Alert on the things that fail quietly: certificate expiry, a policy poll that has not succeeded, clock skew against the attestation window, and disk. alerts is the channel — this server writing a record with its own key — and carries the reason that channel cannot be trusted as an alarm: the failures most worth hearing about are the ones that stop the alert being read. Certificate renewal has the escalation logic now -- didbot_tls::renew::RenewalTracker::alert_level climbs from Ok to Warning after repeated failures and to Critical inside a fixed window of actual expiry, tested without a network in crates/didbot-tls/src/renew.rs -- but it only reaches tracing today, not alerts's channel.
• Debugging without a redeploy. Log level changeable at runtime, and a way to ask why one request was refused that does not require ops-dashboard to exist yet.
• Capacity. pds-writes carries the load question; deployment is where an answer becomes a number.

TLS and DNS, mid-revision #

The project owner corrected the original design here: ALB-terminated TLS put plaintext on the wire between the load balancer and the instance, and a Terraform-managed wildcard DNS record meant the server never wrote DNS at all — removing a capability agent-accounts required from the start. infra/ has been reworked to match (the ALB and its ACM certificates are gone; infra/route53.tf now creates only the hosted zone; infra/iam.tf grants the instance write on it; the instance takes an Elastic IP and is reached on 443 directly). The three items below are what is left, and all three depend on work sibling agents are doing in parallel that had not landed as this revision was written.

Done #

• The unit is a container image under one systemd unit. Not binaries and unit files, not an orchestrator; scripts/ is a developer's interface and no part of a deployment. Deployment's "what runs" states it and the three process properties that decide it. The deciding one is that the store is a single-writer log: infra/ec2.tf is one instance rather than an autoscaling group, and the unit runs the container under a fixed --name behind an ExecStartPre that removes a previous one, so a restart racing a slow shutdown cannot produce two. That is the unit being careful, and it does not bind an operator with a shell, so the exclusion no longer rests on it: didbot_pds::lock takes an exclusive flock on pds.lock in the data directory before the layout check and holds it for the life of the Durable. A second opener — a hand-run binary against the mounted volume — is refused by name before it reads or writes a byte, and the kernel releases the lock when the holder dies, so a crash leaves no stale lock to clear. One existing test was opening two Durables on one directory and is now scoped; that it had to change is the point.
• A data directory layout, and permissions on one this server did not create. Deployment tables what is in it: pds.wal (which holds the account signing keys), pds.wal.compact, pds.lock, pds.layout, blobs/ with its .incoming/, and tls/ with zones/<zone>/ beneath it. Permissions do not depend on who made the directory: didbot_pds::wal::create_dir chmods an existing directory to 0700 on every open rather than trusting the umask that made it, and didbot_tls::storage::create_dir does the same for tls/; files inside are created 0600. Ownership is the one thing this cannot fix — a directory owned by another user fails to write and says so, which is right but is not a diagnosis — and it is the boot script's to get right.
• Secrets, and where each lives. Three, and this process holds all three: the account signing keys, in pds.wal; the ACME account key and the per-zone certificate keys, under tls/; and the Route53 credential, which is not on disk at all — it is the instance role, read over IMDSv2, granted ChangeResourceRecordSets/ListResourceRecordSets on the one hosted zone. Nothing is on the unit's command line. The owner DID is an identifier rather than a credential, and this server holds nothing that can write the owner's repository, which is why policy can constrain it. One compromised host gets an attacker all three; separating the credential from the keys is not a mitigation on a single host and deployment does not offer it as one, because the credential has to be reachable from the process to write DNS-01 challenges. Scope and detection are what bound a compromise.
• The zone layout, in the server rather than in Terraform. The server at the apex, agents at *.<zone> for every zone the deployment is configured with — the recommended shape agent-accounts describes, unenforced but unchanged by this move, and no longer Terraform's to enforce. An earlier revision of this stack had acm.tf/route53.tf/ alb.tf create a certificate and a wildcard ALIAS record per zone, for_each over var.zones; acm.tf and alb.tf are gone and route53.tf now creates only the hosted zone. Which zones exist and what resolves under them is entirely the didbot-identity/ didbot-pds zone registry's decision now, written through the Route53 DnsProvider in crates/didbot-dns. didbot-dev takes --zone once per zone and --route53-zone-id <zone>=<id> for each, builds a MultiZoneDns over one backend per zone, and terminates TLS for all of them from one listener — a <zone> + *.<zone> certificate each, since a wildcard matches one label. Minting still happens under the first zone only; that half is zone-scale's.
• WildcardDns is being replaced. The production DNS backend (crates/didbot-dns) that published nothing, on the theory that a Terraform-managed wildcard ALIAS record already covered every hostname under a managed zone, no longer matches this stack: that record does not exist any more (see the item above), so a real Route53 DnsProvider that actually calls the API is required, not optional. Route53Dns is that backend, and a zone with a hosted zone id configured now gets it: WildcardDns is left as the development stand-in for a zone with no id, publishing nothing and saying so at startup. This server never creates a hosted zone — it is handed the zones it may write, the posture dns.may_create_zones describes.
• Certificates: issuance and renewal, in the instance. The server terminates TLS itself over a certificate it obtains and renews in-process, and infra/templates/user_data.sh.tftpl's ExecStart passes --tls acme, --acme-environment and the hosted zone id the DNS-01 challenge is published into; no credential is on that command line, since Route53Dns reads the instance role over IMDSv2. What a failed renewal does is written down in deployment: the existing certificate keeps being served, the failure is logged with the zone and the expiry it is measured against, and the level escalates before expiry rather than at it. Nothing pages on it — "alert on the things that fail quietly" above is where that stays.
• The certificate hot-reload seam, and TLS termination itself. didbot-tls obtains a wildcard-covering certificate over ACME DNS-01 (instant-acme, on ring — its hyper-rustls feature is deliberately left off because it pulls aws-lc-rs in regardless, checked with cargo tree -i aws-lc-rs), stores it and the ACME account key 0600 inside 0700, and serves it through didbot_serve::serve_tls via a certificate resolver a renewal swaps atomically rather than a listener anything rebinds. What is genuinely exercised without a network: the renewal clock and its escalating alert level, the DNS-01 seam's handling of two identifiers sharing one challenge name, storage permissions, and — the property deployment's firehose case depends on — that an established connection survives a certificate swap while a fresh one sees the new certificate. What is not: the ACME protocol exchange has never run against a real directory, because there is neither network access to one from this environment because there is no network access to one from this environment. crates/didbot-tls/src/dns.rs's seam is wired to didbot-dns's multi-value TXT support through DnsProviderAdapter.