Operations #

Procedures for the deployment infra/pds/templates/user_data.sh.tftpl builds: one instance, one systemd unit didbot-pds, one container of the release image, one data volume at /data with the data directory at /data/pds.

Question	Command
Is the unit up	systemctl status didbot-pds
What did it say	journalctl -u didbot-pds -f
Is a container running	docker ps --filter name=didbot-pds
Is the process live and progressing	curl -s https://<zone>/health
Are sign-in pushes being dropped	curl -s https://<zone>/health | jq .oauthPushes
How full is the data volume	curl -s https://<zone>/health | jq .disk
What does it hold	curl -s https://<zone>/xrpc/bot.did.stats

Upgrading #

• systemctl stop didbot-pds, then confirm with docker ps --filter name=didbot-pds that nothing is running.
• sync on the host.
• Take an on-demand backup of the data volume into the didbot-pds vault, or an EBS snapshot of it. Wait for it to report completed.
• docker pull <image>, then point ExecStart at it in /etc/systemd/system/didbot-pds.service. systemctl cat didbot-pds shows what is actually running.
• systemctl daemon-reload, then systemctl start didbot-pds.
• Watch the journal through startup. On a refusal see "Boot refusals"; otherwise run the checks under "Verifying a restore".

An image that writes a different layout from the one the directory's pds.layout stamp names needs the longer procedure below rather than this one.

Boot refusals #

Each stops the process at startup, before it serves anything, and the unit then restarts every five seconds logging the same message.

Journal message	What it means	What to do
the data directory at … was written as version …	The stamp names a version this image does not read. The message says which versions it does	Going forward, "Upgrading across a layout change": run one of those images first. Going backward, "Rolling back"
the data directory at … carries a stamp written before this binary's check	Half the stamp has nothing to compare against	Run the build that wrote it, or delete the directory
the data directory at … holds state and carries no pds.layout	The directory says nothing about what wrote it	"Restoring" — bring pds.layout back with the rest of the volume
the commit history for … names …, which is not a content identifier	A commit entry the log holds cannot be read	Run the build that wrote it. Do not edit the log by hand
blob bytes are present and pds.wal names none of them	Blob files with no log to name them	"Restoring"
entry <n> did not parse	Intact bytes this binary does not read as an entry	Run the build that wrote it. Do not delete frames by hand
Another process holds the lock	A second writer against one directory	Find and stop it; never mount one data directory twice
A permission error writing to the data directory	Ownership, not mode	chown — see "Restoring"
the data volume is full: … could not be written	A write the boot makes found no room. The message names the file and says how much is free	Grow the volume and resize2fs its ext4 filesystem — see "The data volume"

Upgrading across a layout change #

A data directory carries a pds.layout stamp naming the layout that wrote it: its version number, entry-shape hash and on-disk-names hash. An image reads the layout it writes and at most the one before it. The boot states what the image reads: a refusal names the versions it reads, and a rewrite logs the layout it read.

• Take an on-demand snapshot of the data volume and wait for it to complete. The rollback for this is that snapshot and nothing else.
• systemctl stop didbot-pds, wait for docker ps to show nothing, sync.
• docker pull <image>, point ExecStart at it, systemctl daemon-reload, systemctl start didbot-pds.
• Watch the journal. The boot line reads read a data directory written under layout <n>; rewriting it in this one, and then restored the deployment from its write-ahead log.
• Run the five checks under "Verifying a restore".
• If the stream position moved, announce it to the relay.

That first boot rewrites the directory in the new layout: it takes a checkpoint, trims the log against it, and rewrites the stamp last. From then on the old image refuses the directory, so the rollback is the snapshot.

An upgrade that skips more than one layout is refused by name, and the message says which versions the image does read. Run the intermediate image against the directory first: that boot rewrites it in a layout the new image reads.

The files beside the log #

The write-ahead log holds the accounts. Beside it sits one small file per kind of state that a restart must not drop and a restore does not need. Each can be deleted on its own, and what that costs is below. None of them is covered by the layout stamp, so a change to one does not refuse the directory.

File	Holds	Deleting it costs
pds.grants	OAuth grants: which tokens are live, whose they are, when they expire	Every application signed out. Each has to be authorized again
pds.proofs	bot.did.createAccount and bot.did.createSession proofs already spent	A spent proof can be presented once more inside its window
pds.estop	The halt an operator threw	The halt is released, and the server starts accepting writes
pds.policy	The policy set in force	Every policy is gone. An empty set refuses nothing
pds.sequence	The highest firehose sequence number handed out	The number comes back from the log instead, which is behind on a restored volume

The data volume #

The volume at /data holds the log, the blobs, the checkpoint and the record heap. The rendered config gives the log a tenth of the volume, [capacity] log_budget_bytes, and every account's blobs together half of it, [blobs] total_quota_bytes. Each account's blobs also stop at [blobs] account_quota_bytes. A write past either of the first two is refused with 507 StorageFull, an upload past an account's quota with 403 AccountQuotaExceeded, and the server keeps serving what it holds. A checkpoint is as large as the state while a boot writes it. The record heap, records/pds.heap, grows with every record written.

GET /health carries disk: the volume's freeBytes and totalBytes, read as the request is answered. freeBytes leaves out the blocks ext4 keeps back for root, which the image's uid cannot write. A boot whose own write finds no room stops with the message under "Boot refusals".

Growing the volume takes two steps. Raise data_volume_size_gb and apply: that resizes the EBS volume in place, and it replaces the instance, because the two bounds above are derived from the size and the rendered config changes with them. Then run resize2fs on the volume's device on the host, because ext4 does not grow with the volume under it.

Rolling back #

A data directory carries a pds.layout stamp naming the layout that wrote it, and a binary refuses a stamp it does not read. That runs one way: a new binary restamps the directory, and the old one then meets a mismatch on a directory it wrote itself.

The rollback is a restore. Run the old image against a snapshot taken before the upgrade: its stamp is the old layout, which that image accepts. The cost is everything written since that snapshot — up to twenty-four hours on the daily plan alone, minutes with an on-demand snapshot taken just before the swap.

Backups #

infra/pds/backup.tf puts the one EBS data volume under an AWS Backup plan: vault didbot-pds, rule daily, schedule cron(0 9 * * ? *) — 09:00 UTC — and delete_after = 30, thirty daily recovery points. The RPO is twenty-four hours. aws_backup_selection.pds names the one resource the plan covers: the EBS data volume mounted at /data.

Snapshot the volume; do not rsync the directory.

Restoring #

• Create a volume from a recovery point in the didbot-pds vault, in the target instance's availability zone.
• Copy the whole directory, pds.layout included. A volume that came back without it is refused: the directory then says nothing about what wrote it.
• Attach it to a fresh instance, never to the running one, and mount it at /data, where that instance's boot script chowns /data/pds to uid 10001. The image runs as that uid and chmods the data directory to 0700 on every open: the mode a snapshot carries is re-tightened, the owner is the chown.
• Start the unit, and run the checks below.
• Detach and delete any test volume.

On a blob bytes are present and pds.wal names none of them refusal: finish the restore so the log is present, or move blobs/ aside and start with an empty blob store, after which getBlob answers NotFound for the blobs the accounts reference.

Verifying a restore #

The startup line is restored the deployment from its write-ahead log, and its fields are the count of each thing that came back. It may be followed by wrote the checkpoint the next boot reads the log from. A the write-ahead log ends in an incomplete write; truncating it warning reports the discarded bytes and the offset it stopped at; on a snapshot restore that is the torn tail of the append in flight when the snapshot was taken.

Check	Confirms
GET /health returns 200 with ticks advancing between two calls a few seconds apart	The process is live and its ten-second tick is running
GET /xrpc/bot.did.stats	Accounts, records and bytes per collection match what the deployment had, and blobs.held matches its blob inventory
blobs.missingAtBoot in that response reads {"count": 0, "bytes": 0}	Every blob the log names had its bytes on the disk this deployment started on. A nonzero count is that many 404s waiting, and bytes is what the restore would have to bring back; the CIDs are in the startup ERROR line
blobs.discardedAtBoot reads {"count": 0, "bytes": 0}	The copy carried no blob bytes the log could not account for. A nonzero count is that many files deleted during startup
GET /xrpc/bot.did.listAccounts	The accounts that should be there are there
One agent's did.json resolves	The signing keys came back
Fetch one blob that account references	The bytes are on the disk; getBlob re-hashes what it reads and refuses on a mismatch

Sign-in pushes that leave no decision behind #

POST /oauth/par is unauthenticated, and no answer it gives may depend on whether an account exists — so a client flooding one agent's pending sign-ins gets the same 201 an ordinary push gets, and the agent sees only that its own sign-in never arrives. GET /health carries oauthPushes, three counts since this process started:

Count	What moved it
displaced	A push past an account's own pending_per_account bound. That account's oldest undecided sign-in made way for it
unresolvedHint	A push whose login_hint named no account here
serverFull	A push refused because this server holds pending_total decision records already, counting an answered one until its request expires. This is the one an ordinary caller can see, as a 403

Read them twice a minute apart: these are totals, so what says anything is how fast they move. displaced climbing steadily is a flood aimed at accounts this deployment holds; unresolvedHint climbing is one probing for accounts it does not. The journal carries a sampled info line for the first, one in a hundred, naming the account — sampled because logging every one would make the log the thing a flood amplifies.

bot.did.stats and bot.did.listAccounts are public by default; under --close-disclosure they answer 403 and the figures are in the startup line.

Scopes a sign-in is not granted, and forms outside the spec #

A sign-in that asks for a transition:, identity: or account: scope has that scope dropped and the rest granted. So does one that names a scope this server cannot read, such as the unknown kind space:* or the value account:status. The agent's decision record lists each dropped scope under cut.

An app may write a scope in a form the permission spec does not define. The server reads it as well as it can, and once it has checked the client it writes a warn line naming the client, the atom as the app wrote it, and how it was read. The operator's own [oauth] scope_ceiling is read without one, so its partial wildcards write nothing.

Form	Read as
A partial wildcard, such as repo:app.bsky.*	The NSIDs under that prefix, at any depth
rpc: with no aud	Every audience
rpc:*?aud=*	Every method at every audience, as written
An aud that is not * or a DID with a service fragment	That audience alone
repo:…?action=manage	No record operation, so the scope adds no write
A comma list of actions, action=create,update	Each action in it
A parameter its kind does not take	Ignored
A list parameter both after the : and by name, such as blob:image/png?accept=text/html	Every value
A parameter that takes one value, given twice	The first
A scope this server cannot read	Kept as written, and dropped