id: deploy title: A named version runs somewhere that is not a laptop status: open crates: [didbot-serve, didbot-pds] dependsOn: [agent-accounts] exitCriterion: > A tagged release runs on a host nobody develops on, serving agents over real DNS and TLS, and a documented upgrade replaces it without losing a record. #

deploy #

Everything here runs from a checkout. Nothing says what ships, where it puts its state, how it is upgraded, or what happens when a certificate expires.

The security shape is settled and is a constraint on this epic rather than a question for it: the server runs on a host that is not an agent host, the zone credential is scoped to exactly the one hosted zone this deployment holds and sits on the same host as the signing keys — the server writes its own DNS, so it holds its own Route53 credential — and policy is read from the operator's repository. Deployment states those; this is what it takes to satisfy them.

The marketing site is a separate deployment target from the server this epic ships: site is a static build behind S3 and CloudFront, with its own bucket and distribution to provision — see that epic's open items.

What ships #

• Certificates: issuance and renewal, and what a failed renewal does. An expired wildcard stops every agent resolving at once. didbot-tls now carries the ACME DNS-01 orchestration, the renewal clock, and 0600-inside-0700 storage for both the certificate and the ACME account key — see its Done entry below for what is and is not verified. Route53Dns and its multi-value TXT support have landed and didbot-tls publishes DNS-01 through them, one order per zone. Left open here because nothing has issued a real certificate yet: that needs a real zone, and network access to a directory, neither of which exists in the environment this was built in.

Staying up #

• Backups and restore. infra/pds/backup.tf puts the data volume — the write-ahead log and the blob store both live on it — under an AWS Backup plan, daily, kept thirty days. Deployment states what a restore recovers (accounts, records, blobs, name holds, ledger history, and the signing keys that make them the same accounts rather than new ones with the same handles), what it does not (every session, the Route53 zone, a certificate older than its own validity, and up to a day of writes), and the five-step drill that would prove it. Open because nobody has run that drill: there is no AWS account to apply this against, and a snapshot that has never been read back is not a backup. Closing this item is the drill succeeding, not the plan existing.
• Upgrades across a log format change. Refusing by name has landed: a data directory carries a pds.layout stamp, it is checked before the log is read, and a mismatch stops the server with a message naming both layouts — see local-dev. What is still open here is the other half, which is a deployment's rather than a developer's: reading an older log, or converting one, so that an upgrade does not mean deleting the accounts.
• Say what a restart does to sessions holding credentials and to a write in flight, and whether downtime is acceptable or has to be avoided. OAuth grants are in the log and survive one; a sign-in in flight — a pushed authorization request, a consent reference, an authorization code — is in-process memory and does not, so a client mid-sign-in starts over. The restart-persistence issue (3mviwkxahvt2s) decides what else a restart keeps, and where each thing lives.
• Rollback, including whether an older binary can read a newer log. Unaddressed here: the pds.layout stamp refuses a mismatched binary by name rather than allowing a downgrade, so today's answer is "restore the backup taken before the upgrade," not "run the old binary."

Reloading, per component #

• Write down what changes without a restart and what does not. Policy is a poll and needs nothing. Hooks need a new session. Prompt text needs a rebuild. Configuration is process arguments and needs a restart by design. The record host depends on its transport: an HTTP one restarts like any service, a stdio one is owned by the harness and comes back when the harness spawns it. The TLS certificate needs no restart either. didbot-tls builds one rustls::ServerConfig for the life of the process, backed by a certificate resolver (didbot_tls::resolver::SwappableCert) that a renewal swaps under an atomic pointer store. A handshake already in progress or a connection already established — the firehose holds these for hours to days — is unaffected; only the next handshake sees the new certificate. Proven with a real TCP/TLS loop in crates/didbot-tls/tests/rotation.rs: a connection held open across a swap keeps working, and a fresh connection made after the swap gets the new certificate. Nothing else about the listener, the router or the process is rebuilt.
• Agent hosts are their own deployment. The node agent, the hook binary and the harness settings all live there and all need updating, and credentials has the question of how a host knows the hook it runs is the one that was built.

Knowing it is alive #

• Alert on the things that fail quietly: certificate expiry, a policy poll that has not succeeded, clock skew against the attestation window, and disk. alerts is the channel — this server writing a record with its own key — and carries the reason that channel cannot be trusted as an alarm: the failures most worth hearing about are the ones that stop the alert being read. Certificate renewal has the escalation logic now -- didbot_tls::renew::RenewalTracker::alert_level climbs from Ok to Warning after repeated failures and to Critical inside a fixed window of actual expiry, tested without a network in crates/didbot-tls/src/renew.rs, and GET /health's certificates carries every zone's level so infra/pds/monitoring.tf's Route53 health check and CloudWatch alarm can read it. The other three failures above, and alerts's own channel, are what is left here.
• Debugging without a redeploy. Log level changeable at runtime, and a way to ask why one request was refused that does not require ops-dashboard to exist yet.
• Capacity. pds-writes carries the load question; deployment is where an answer becomes a number.

TLS and DNS, mid-revision #

The project owner corrected the original design here: ALB-terminated TLS put plaintext on the wire between the load balancer and the instance, and a Terraform-managed wildcard DNS record meant the server never wrote DNS at all — removing a capability agent-accounts required from the start. infra/pds/ has been reworked to match (the ALB and its ACM certificates are gone; infra/pds/route53.tf now creates only the hosted zone; infra/pds/iam.tf grants the instance write on it; the instance takes an Elastic IP and is reached on 443 directly). The three items below are what is left, and all three depend on work sibling agents are doing in parallel that had not landed as this revision was written.

Done #

• Two roots, one state bucket. infra/pds/ is this deployment and infra/site/ is the did.bot website; each has its own key in the did.bot account's state bucket, so applying one cannot lock or roll back the other. The wire between them is one value: infra/pds/'s name_servers output goes into infra/site/'s pds_zone_name_servers, which writes the NS record at pds.did.bot that makes this zone answer.
• The unit is a container image under one systemd unit. Not binaries and unit files, not an orchestrator; scripts/ is a developer's interface and no part of a deployment. Deployment's "what runs" states it and the three process properties that decide it. The deciding one is that the store is a single-writer log: infra/pds/ec2.tf is one instance rather than an autoscaling group, and the unit runs the container under a fixed --name behind an ExecStartPre that removes a previous one, so a restart racing a slow shutdown cannot produce two. That is the unit being careful, and it does not bind an operator with a shell, so the exclusion no longer rests on it: didbot_pds::lock takes an exclusive flock on pds.lock in the data directory before the layout check and holds it for the life of the Durable. A second opener — a hand-run binary against the mounted volume — is refused by name before it reads or writes a byte, and the kernel releases the lock when the holder dies, so a crash leaves no stale lock to clear. One existing test was opening two Durables on one directory and is now scoped; that it had to change is the point.
• A data directory layout, and permissions on one this server did not create. Deployment tables what is in it: pds.wal (which holds the account signing keys), pds.wal.compact, pds.lock, pds.layout, blobs/ with its .incoming/, and tls/ with zones/<zone>/ beneath it. Permissions do not depend on who made the directory: didbot_pds::wal::create_dir chmods an existing directory to 0700 on every open rather than trusting the umask that made it, and didbot_tls::storage::create_dir does the same for tls/; files inside are created 0600. Ownership is the one thing this cannot fix — a directory owned by another user fails to write and says so, which is right but is not a diagnosis — and it is the boot script's to get right.
• Secrets, and where each lives. Three, and this process holds all three: the account signing keys, in pds.wal; the ACME account key and the per-zone certificate keys, under tls/; and the Route53 credential, which is not on disk at all — it is the instance role, read over IMDSv2, granted ChangeResourceRecordSets on the one hosted zone and GetChange on changes. Nothing is on the unit's command line. The operator DID is an identifier rather than a credential, and this server holds nothing that can write the operator's repository, which is why policy can constrain it. One compromised host gets an attacker all three; separating the credential from the keys is not a mitigation on a single host and deployment does not offer it as one, because the credential has to be reachable from the process to write DNS-01 challenges. Scope and detection are what bound a compromise.
• The zone layout, in the server rather than in Terraform. The server at the apex, agents at *.<zone> for every zone the deployment is configured with — the recommended shape agent-accounts describes, unenforced but unchanged by this move, and no longer Terraform's to enforce. route53.tf creates the hosted zone, the apex record and the *.<zone> wildcard every agent hostname resolves through, all at the instance's Elastic IP. Which zones exist is the didbot-identity/didbot-pds zone registry's decision: didbot-pds takes --zone once per zone and --route53-zone-id <zone>=<id> for each, writes each zone's _acme-challenge and CAA records through one Route53Dns per zone, and terminates TLS for all of them from one listener — a <zone> + *.<zone> certificate each, since a wildcard matches one label. Minting still happens under the first zone only; that half is zone-scale's.
• Agent hostnames resolve through the zone's wildcard record. aws_route53_record.agents in route53.tf is *.<zone> at the instance's Elastic IP, beside the apex. The server writes no record per agent, so the records it holds a credential for are the _acme-challenge TXT values and the apex CAA, and the grant in iam.tf is ChangeResourceRecordSets on the one zone plus GetChange. This server never creates a hosted zone — it is handed the zones it may write, the posture dns.may_create_zones describes.
• Certificates: issuance and renewal, in the instance. The server terminates TLS itself over a certificate it obtains and renews in-process, and infra/pds/templates/user_data.sh.tftpl's ExecStart passes --tls acme, --acme-environment and the hosted zone id the DNS-01 challenge is published into; no credential is on that command line, since Route53Dns reads the instance role over IMDSv2. What a failed renewal does is written down in deployment: the existing certificate keeps being served, the failure is logged with the zone and the expiry it is measured against, and the level escalates before expiry rather than at it. GET /health reports the same levels, and infra/pds/monitoring.tf alarms on them — see "alert on the things that fail quietly" above for the failures that channel still covers.
• The certificate hot-reload seam, and TLS termination itself. didbot-tls obtains a wildcard-covering certificate over ACME DNS-01 (instant-acme, on ring — its hyper-rustls feature is deliberately left off because it pulls aws-lc-rs in regardless, checked with cargo tree -i aws-lc-rs), stores it and the ACME account key 0600 inside 0700, and serves it through didbot_serve::serve_tls via a certificate resolver a renewal swaps atomically rather than a listener anything rebinds. What is genuinely exercised without a network: the renewal clock and its escalating alert level, the DNS-01 seam's handling of two identifiers sharing one challenge name, storage permissions, and — the property deployment's firehose case depends on — that an established connection survives a certificate swap while a fresh one sees the new certificate. tests/acme_conformance.rs now drives the real instant-acme client through a full order — directory, nonce, account, order, both authorizations, dns-01 challenges, finalize, certificate — against an in-process fake ACME directory, covering the apex+wildcard grouping landing on a CA-shaped server, a challenge that never validates, a renewal that keeps failing, a crash between publishing a challenge and withdrawing it (and the next order replacing the record it stranded), and two zones ordering concurrently. What is still not exercised: a real CA, and DNS-01 propagation over a real network — there is no network access to one from this environment. crates/didbot-tls/src/dns.rs's seam is wired to didbot-dns's multi-value TXT support through DnsProviderAdapter.