Operations #

Procedures for the deployment infra/pds/templates/user_data.sh.tftpl builds: one instance, one systemd unit didbot-pds, one container of the release image, one data volume at /data with the data directory at /data/pds.

Question	Command
Is the unit up	`systemctl status didbot-pds`
What did it say	`journalctl -u didbot-pds -f`
Is a container running	`docker ps --filter name=didbot-pds`
Is the process live and progressing	`curl -s https://<zone>/health`
Are sign-in pushes being dropped	`curl -s https://<zone>/health \| jq .oauthPushes`
What does it hold	`curl -s https://<zone>/xrpc/bot.did.stats`

Upgrading #

systemctl stop didbot-pds, then confirm with docker ps --filter name=didbot-pds that nothing is running.
sync on the host.
Take an on-demand backup of the data volume into the didbot-pds vault, or an EBS snapshot of it. Wait for it to report completed.
docker pull <image>, then point ExecStart at it in /etc/systemd/system/didbot-pds.service. systemctl cat didbot-pds shows what is actually running.
systemctl daemon-reload, then systemctl start didbot-pds.
Watch the journal through startup. On a refusal see "Boot refusals"; otherwise run the checks under "Verifying a restore".

Boot refusals #

Each stops the process at startup, before it serves anything, and the unit then restarts every five seconds logging the same message.

Journal message	What it means	What to do
`the data directory at … was written as version …`	The stamp disagrees with the binary	"Rolling back"
`the data directory at … carries a stamp written before` this binary's check	Half the stamp has nothing to compare against	Run the build that wrote it, or delete the directory
`blob bytes are present and pds.wal is not`	Blob files with no log to name them	"Restoring"
`entry <n> did not parse`	Intact bytes this binary does not read as an entry	Run the build that wrote it. Do not delete frames by hand
Another process holds the lock	A second writer against one directory	Find and stop it; never mount one data directory twice
A permission error writing to the data directory	Ownership, not mode	`chown` — see "Restoring"

Rolling back #

A data directory carrying no pds.layout stamp is adopted and stamped with the opening binary's own layout, so the first run of a new binary restamps it and the old binary then meets a mismatch on a directory it wrote itself.

The rollback is a restore. Run the old image against a snapshot taken before the upgrade: its stamp is the old layout, or absent, and either is accepted. The cost is everything written since that snapshot — up to twenty-four hours on the daily plan alone, minutes with an on-demand snapshot taken just before the swap.

Backups #

infra/pds/backup.tf puts the one EBS data volume under an AWS Backup plan: vault didbot-pds, rule daily, schedule cron(0 9 * * ? *) — 09:00 UTC — and delete_after = 30, thirty daily recovery points. The RPO is twenty-four hours. aws_backup_selection.pds names the one resource the plan covers: the EBS data volume mounted at /data.

Snapshot the volume; do not rsync the directory.

Restoring #

Create a volume from a recovery point in the didbot-pds vault, in the target instance's availability zone.
Attach it to a fresh instance, never to the running one, and mount it at /data, where that instance's boot script chowns /data/pds to uid 10001. The image runs as that uid and chmods the data directory to 0700 on every open: the mode a snapshot carries is re-tightened, the owner is the chown.
Start the unit, and run the checks below.
Detach and delete any test volume.

On a blob bytes are present and pds.wal is not refusal: finish the restore so the log is present, or move blobs/ aside and start with an empty blob store, after which getBlob answers NotFound for the blobs the accounts reference.

Verifying a restore #

The startup line is restored the deployment from its write-ahead log, and its fields are the count of each thing that came back. It may be followed by compacted the write-ahead log. A the write-ahead log ends in an incomplete write; truncating it warning reports the discarded bytes and the offset it stopped at; on a snapshot restore that is the torn tail of the append in flight when the snapshot was taken.

Check	Confirms
`GET /health` returns `200` with `ticks` advancing between two calls a few seconds apart	The process is live and its ten-second tick is running
`GET /xrpc/bot.did.stats`	Accounts, records and bytes per collection match what the deployment had, and `blobs.held` matches its blob inventory
`blobs.missingAtBoot` in that response reads `{"count": 0, "bytes": 0}`	Every blob the log names had its bytes on the disk this deployment started on. A nonzero `count` is that many `404`s waiting, and `bytes` is what the restore would have to bring back; the CIDs are in the startup `ERROR` line
`blobs.discardedAtBoot` reads `{"count": 0, "bytes": 0}`	The copy carried no blob bytes the log could not account for. A nonzero count is that many files deleted during startup
`GET /xrpc/bot.did.listAgents`	The accounts that should be there are there
One agent's `did.json` resolves	The signing keys came back
Fetch one blob that account references	The bytes are on the disk; `getBlob` re-hashes what it reads and refuses on a mismatch

POST /oauth/par is unauthenticated, and no answer it gives may depend on whether an account exists — so a client flooding one agent's pending sign-ins gets the same 201 an ordinary push gets, and the agent sees only that its own sign-in never arrives. GET /health carries oauthPushes, three counts since this process started:

Count	What moved it
`displaced`	A push past an account's own `pending_per_account` bound. That account's oldest undecided sign-in made way for it
`unresolvedHint`	A push whose `login_hint` named no account here
`serverFull`	A push refused because this server holds `pending_total` undecided records already. This is the one an ordinary caller can see, as a `403`

Read them twice a minute apart: these are totals, so what says anything is how fast they move. displaced climbing steadily is a flood aimed at accounts this deployment holds; unresolvedHint climbing is one probing for accounts it does not. The journal carries a sampled info line for the first, one in a hundred, naming the account — sampled because logging every one would make the log the thing a flood amplifies.

bot.did.stats and bot.did.listAgents are public by default; under --close-disclosure they answer 403 and the figures are in the startup line.