# Operations

Procedures for the deployment `infra/pds/templates/user_data.sh.tftpl` builds: one
instance, one systemd unit `didbot-pds`, one container of the release image,
one data volume at `/data` with the data directory at `/data/pds`.

| Question | Command |
|---|---|
| Is the unit up | `systemctl status didbot-pds` |
| What did it say | `journalctl -u didbot-pds -f` |
| Is a container running | `docker ps --filter name=didbot-pds` |
| Is the process live and progressing | `curl -s https://<zone>/health` |
| Are sign-in pushes being dropped | `curl -s https://<zone>/health \| jq .oauthPushes` |
| How full is the data volume | `curl -s https://<zone>/health \| jq .disk` |
| What does it hold | `curl -s https://<zone>/xrpc/bot.did.stats` |

## Upgrading

- `systemctl stop didbot-pds`, then confirm with
  `docker ps --filter name=didbot-pds` that nothing is running.
- `sync` on the host.
- Take an on-demand backup of the data volume into the `didbot-pds` vault, or
  an EBS snapshot of it. Wait for it to report completed.
- `docker pull <image>`, then point `ExecStart` at it in
  `/etc/systemd/system/didbot-pds.service`. `systemctl cat didbot-pds` shows
  what is actually running.
- `systemctl daemon-reload`, then `systemctl start didbot-pds`.
- Watch the journal through startup. On a refusal see "Boot refusals";
  otherwise run the checks under "Verifying a restore".

An image that writes a different layout from the one the directory's
`pds.layout` stamp names needs the longer procedure below rather than this
one.

### Boot refusals

Each stops the process at startup, before it serves anything, and the unit
then restarts every five seconds logging the same message.

| Journal message | What it means | What to do |
|---|---|---|
| `the data directory at … was written as version …` | The stamp names a version this image does not read. The message says which versions it does | Going forward, "Upgrading across a layout change": run one of those images first. Going backward, "Rolling back" |
| `the data directory at … carries a stamp written before` this binary's check | Half the stamp has nothing to compare against | Run the build that wrote it, or delete the directory |
| `the data directory at … holds state and carries no pds.layout` | The directory says nothing about what wrote it | "Restoring" — bring `pds.layout` back with the rest of the volume |
| `the commit history for … names …, which is not a content identifier` | A commit entry the log holds cannot be read | Run the build that wrote it. Do not edit the log by hand |
| `blob bytes are present and pds.wal names none of them` | Blob files with no log to name them | "Restoring" |
| `entry <n> did not parse` | Intact bytes this binary does not read as an entry | Run the build that wrote it. Do not delete frames by hand |
| Another process holds the lock | A second writer against one directory | Find and stop it; never mount one data directory twice |
| A permission error writing to the data directory | Ownership, not mode | `chown` — see "Restoring" |
| `the data volume is full: … could not be written` | A write the boot makes found no room. The message names the file and says how much is free | Grow the volume and `resize2fs` its ext4 filesystem — see "The data volume" |

## Upgrading across a layout change

A data directory carries a `pds.layout` stamp naming the layout that wrote it:
its version number, entry-shape hash and on-disk-names hash. An image reads the
layout it writes and at most the one before it. The boot states what the image
reads: a refusal names the versions it reads, and a rewrite logs the layout it
read.

- Take an on-demand snapshot of the data volume and wait for it to complete.
  The rollback for this is that snapshot and nothing else.
- `systemctl stop didbot-pds`, wait for `docker ps` to show nothing, `sync`.
- `docker pull <image>`, point `ExecStart` at it, `systemctl daemon-reload`,
  `systemctl start didbot-pds`.
- Watch the journal. The boot line reads `read a data directory written under
  layout <n>; rewriting it in this one`, and then `restored the deployment
  from its write-ahead log`.
- Run the five checks under "Verifying a restore".
- If the stream position moved, announce it to the relay.

That first boot rewrites the directory in the new layout: it takes a
checkpoint, trims the log against it, and rewrites the stamp last. From then
on the old image refuses the directory, so the rollback is the snapshot.

An upgrade that skips more than one layout is refused by name, and the
message says which versions the image does read. Run the intermediate image
against the directory first: that boot rewrites it in a layout the new image
reads.

## The files beside the log

The write-ahead log holds the accounts. Beside it sits one small file per kind
of state that a restart must not drop and a restore does not need. Each can be
deleted on its own, and what that costs is below. None of them is covered by
the layout stamp, so a change to one does not refuse the directory.

| File | Holds | Deleting it costs |
|---|---|---|
| `pds.grants` | OAuth grants: which tokens are live, whose they are, when they expire | Every application signed out. Each has to be authorized again |
| `pds.proofs` | `bot.did.createAccount` and `bot.did.createSession` proofs already spent | A spent proof can be presented once more inside its window |
| `pds.estop` | The halt an operator threw | The halt is released, and the server starts accepting writes |
| `pds.policy` | The policy set in force | Every policy is gone. An empty set refuses nothing |
| `pds.sequence` | The highest firehose sequence number handed out | The number comes back from the log instead, which is behind on a restored volume |

## The data volume

The volume at `/data` holds the log, the blobs, the checkpoint and the record
heap. The rendered config gives the log a tenth of the volume, `[capacity]
log_budget_bytes`, and every account's blobs together half of it, `[blobs]
total_quota_bytes`. Each account's blobs also stop at `[blobs]
account_quota_bytes`. A write past either of the first two is refused with
`507 StorageFull`, an upload past an account's quota with `403
AccountQuotaExceeded`, and the server keeps serving what it holds. A
checkpoint is as large as the state while a boot writes it. The record heap,
`records/pds.heap`, grows with every record written.

`GET /health` carries `disk`: the volume's `freeBytes` and `totalBytes`, read
as the request is answered. `freeBytes` leaves out the blocks ext4 keeps back
for root, which the image's uid cannot write. A boot whose own write finds no
room stops with the message under "Boot refusals".

Growing the volume takes two steps. Raise `data_volume_size_gb` and apply:
that resizes the EBS volume in place, and it replaces the instance, because
the two bounds above are derived from the size and the rendered config
changes with them. Then run `resize2fs` on the volume's device on the host,
because ext4 does not grow with the volume under it.

## Rolling back

A data directory carries a `pds.layout` stamp naming the layout that wrote it,
and a binary refuses a stamp it does not read. That runs one way: a new binary
restamps the directory, and the old one then meets a mismatch on a directory it
wrote itself.

**The rollback is a restore.** Run the old image against a snapshot taken
before the upgrade: its stamp is the old layout, which that image accepts. The
cost is everything written since that snapshot — up to twenty-four hours on the
daily plan alone, minutes with an on-demand snapshot taken just before the
swap.

## Backups

`infra/pds/backup.tf` puts the one EBS data volume under an AWS Backup plan: vault
`didbot-pds`, rule `daily`, schedule `cron(0 9 * * ? *)` — 09:00 UTC — and
`delete_after = 30`, thirty daily recovery points. **The RPO is twenty-four
hours.** `aws_backup_selection.pds` names the one resource the plan covers:
the EBS data volume mounted at `/data`.

Snapshot the volume; do not `rsync` the directory.

## Restoring

- Create a volume from a recovery point in the `didbot-pds` vault, in the
  target instance's availability zone.
- Copy the whole directory, `pds.layout` included. A volume that came back
  without it is refused: the directory then says nothing about what wrote it.
- Attach it to a **fresh** instance, never to the running one, and mount it at
  `/data`, where that instance's boot script chowns `/data/pds` to uid 10001.
  The image runs as that uid and chmods the data directory to `0700` on every
  open: the mode a snapshot carries is re-tightened, the owner is the chown.
- Start the unit, and run the checks below.
- Detach and delete any test volume.

On a `blob bytes are present and pds.wal names none of them` refusal: finish
the restore so the log is present, or move `blobs/` aside and start with an
empty blob store, after which `getBlob` answers `NotFound` for the blobs the
accounts reference.

## Verifying a restore

The startup line is `restored the deployment from its write-ahead log`, and its
fields are the count of each thing that came back. It may be followed by
`wrote the checkpoint the next boot reads the log from`. A `the write-ahead
log ends in an incomplete write; truncating it` warning reports the
`discarded` bytes and the
`offset` it stopped at; on a snapshot restore that is the torn tail of the
append in flight when the snapshot was taken.

| Check | Confirms |
|---|---|
| `GET /health` returns `200` with `ticks` advancing between two calls a few seconds apart | The process is live and its ten-second tick is running |
| `GET /xrpc/bot.did.stats` | Accounts, records and bytes per collection match what the deployment had, and `blobs.held` matches its blob inventory |
| `blobs.missingAtBoot` in that response reads `{"count": 0, "bytes": 0}` | Every blob the log names had its bytes on the disk this deployment started on. A nonzero `count` is that many `404`s waiting, and `bytes` is what the restore would have to bring back; the CIDs are in the startup `ERROR` line |
| `blobs.discardedAtBoot` reads `{"count": 0, "bytes": 0}` | The copy carried no blob bytes the log could not account for. A nonzero count is that many files deleted during startup |
| `GET /xrpc/bot.did.listAccounts` | The accounts that should be there are there |
| One agent's `did.json` resolves | The signing keys came back |
| Fetch one blob that account references | The bytes are on the disk; `getBlob` re-hashes what it reads and refuses on a mismatch |

## Sign-in pushes that leave no decision behind

`POST /oauth/par` is unauthenticated, and no answer it gives may depend on
whether an account exists — so a client flooding one agent's pending sign-ins
gets the same `201` an ordinary push gets, and the agent sees only that its
own sign-in never arrives. `GET /health` carries `oauthPushes`, three counts
since this process started:

| Count | What moved it |
|---|---|
| `displaced` | A push past an account's own `pending_per_account` bound. That account's oldest undecided sign-in made way for it |
| `unresolvedHint` | A push whose `login_hint` named no account here |
| `serverFull` | A push refused because this server holds `pending_total` decision records already, counting an answered one until its request expires. This is the one an ordinary caller can see, as a `403` |

Read them twice a minute apart: these are totals, so what says anything is
how fast they move. `displaced` climbing steadily is a flood aimed at
accounts this deployment holds; `unresolvedHint` climbing is one probing for
accounts it does not. The journal carries a sampled `info` line for the
first, one in a hundred, naming the account — sampled because logging every
one would make the log the thing a flood amplifies.

`bot.did.stats` and `bot.did.listAccounts` are public by default; under
`--close-disclosure` they answer 403 and the figures are in the startup line.

## Scopes a sign-in is not granted, and forms outside the spec

A sign-in that asks for a `transition:`, `identity:` or `account:` scope has
that scope dropped and the rest granted. So does one that names a scope this
server cannot read, such as the unknown kind `space:*` or the value
`account:status`. The agent's decision record lists each dropped scope
under `cut`.

An app may write a scope in a form the permission spec does not define.
The server reads it as well as it can, and once it has checked the client
it writes a `warn` line naming the client, the atom as the app wrote it,
and how it was read. The operator's own `[oauth] scope_ceiling` is read
without one, so its partial wildcards write nothing.

| Form | Read as |
|---|---|
| A partial wildcard, such as `repo:app.bsky.*` | The NSIDs under that prefix, at any depth |
| `rpc:` with no `aud` | Every audience |
| `rpc:*?aud=*` | Every method at every audience, as written |
| An `aud` that is not `*` or a DID with a service fragment | That audience alone |
| `repo:…?action=manage` | No record operation, so the scope adds no write |
| A comma list of actions, `action=create,update` | Each action in it |
| A parameter its kind does not take | Ignored |
| A list parameter both after the `:` and by name, such as `blob:image/png?accept=text/html` | Every value |
| A parameter that takes one value, given twice | The first |
| A scope this server cannot read | Kept as written, and dropped |