# The write pipeline

What a write passes through before it reaches the log, in order, and what each
stage costs, checks and answers.

The order is not arbitrary. It runs **broadest and cheapest first**, so a write
that cannot possibly succeed is refused before anything expensive happens, and
every stage's refusal is **distinguishable on the wire** — an agent retries
differently for each, and one that cannot tell them apart retries the wrong
thing forever.

```mermaid
flowchart TD
    R[["write request"]] --> E

    E{{"1 · e-stop<br/>Estop::check_use / check_issue"}}
    E -->|"Pause or Revoke"| EH["503 Halted"]
    E -->|clear| L

    L{{"2 · lifecycle<br/>ServerState policy"}}
    L -->|"not accepting writes"| LH["503 ServerNotReady<br/>Retry-After, blocking gates named"]
    L -->|accepting| F

    F{{"3 · per-repo freeze<br/>Provisioner::require_writable"}}
    F -->|"a lock refuses writes"| FH["403 AccountNotWritable<br/>the lock, and the party that hung it"]
    F -->|writable| P

    P{{"4 · parsing<br/>record + diff"}}
    P -->|malformed| PH["400 InvalidRecord"]
    P -->|"parsed, diff computed"| Q

    Q{{"5 · per-repo queue<br/>admission order"}}
    Q -->|full| QH["429 / 503 backpressure"]
    Q -->|queued| T

    T{{"6 · policy tree<br/>applicability index"}}
    T -->|"no policy matches"| W
    T -->|"candidates, in order"| V

    V{{"7 · evaluation<br/>Evaluator::evaluate"}}
    V -->|"Reject"| VH["403 policy refused<br/>policy-authored reason"]
    V -->|"Freeze"| VF["403 account frozen<br/>drains this repo's queue"]
    V -->|"unavailable / over its deadline"| VU["403 refused<br/>fail closed, scoped to this policy"]
    V -->|Allow| W

    W(["8 · commit<br/>repository's write lock held here only"]) --> S["sequence assigned<br/>policy version + now recorded"]
```

## What each stage checks

### Stage 1 · E-stop — `didbot_pds::Estop`

The cheapest check there is: an atomic latch read, no credential, no lookup.
It runs first because it is both cheapest and most urgent, and because
`Halted` is the wire answer callers already switch on.

**Both settings refuse the same requests.** They differ in what a release
gives back:

| | `Mode::Pause` | `Mode::Revoke` |
| --- | --- | --- |
| record write with a token issued before the stop (`check_use`) | halted | halted |
| provisioning a new account (`check_issue`) | halted | halted |
| that token, once the stop is released | works | ended |

`Pause` refuses a token until the stop is released and ends nothing, which is
why a lapsed operator claim throws `Pause` and not `Revoke`: once the
operator's record is back, the agents already here write with the tokens they
hold.

`Revoke` ends the credentials as well as refusing them. Throwing it ends
every OAuth grant. Releasing it ends every agent token this deployment has
issued — on the way out rather than on the way in, because
`bot.did.freezeAccount` and `bot.did.deactivateAccount` are reached with
that token and are exempt from the stop. Either way, releasing the latch
leaves them ended and each account signs in again for a fresh one.

A service-auth token is the exception to both settings. Either one refuses
every new service-auth token and every proxied call, but a token already
minted works at the service it names until its `exp`, at most an hour, or a
minute for the ones the proxy mints.

E-stop and the lifecycle are **independent facts**. A server can be fully
claimed and halted, and neither is derivable from the other.

### Stage 2 · Lifecycle — `didbot_pds::ServerState`

Whether this *deployment* accepts writes at all. Public: no credential is
spent to learn it, because "this server is not accepting writes right now" is
not a secret and paying for a credential lookup to reach it is work for
nothing.

Answers `503 ServerNotReady` with `Retry-After` and the blocking gates named,
never a `404` — "this method does not exist" and "this method is not ready"
are facts a relay reacts to completely differently.

### Stage 3 · Per-repo freeze — `Provisioner::require_writable`

`AccountState`'s policy for the named repository. Cheap, and **needs no
authentication** for a reason worth stating: the repository identifier is
caller-supplied, but the only thing a caller achieves by supplying one is
getting itself refused. **A check that can only deny is safe on
unauthenticated input.** Trusting caller-supplied input is dangerous when it
grants and harmless when it only denies.

A lock carries *who* hung it — the operator, a policy, the account itself, or
a parent — and that is distinct from a policy refusal. "Account frozen" says nothing
you write will work until an operator acts; "policy refused" says fix this
write. An agent that cannot tell them apart retries the wrong one.

### Stage 4 · Parsing

Necessarily before policy: a diff cannot be computed without parsing, and
policies are evaluated against the diff.

The diff carries **before and after values**, not only changed paths, with
absence treated as a value. The write's action travels beside it. A rule about
what a value is reads only the diff. A rule that a field must not change reads
the action too: it trips on an update, never on a create or a delete. A create
at a key that still holds a deleted record's tombstone is judged as an update
from that record.

### Stage 5 · Per-repo queue

The linearization unit is the **repository, not the server**. Order within one
repository is load-bearing: its commits chain, each naming its predecessor.
Order between two repositories is observable by nobody. So each repository has
its own queue, different repositories evaluate concurrently, and head-of-line
blocking is bounded to the agent that caused it.

The queue is bounded. A full queue is backpressure a caller can act on, not a
memory leak.

### Stage 6 · Policy tree — applicability index

The tree **indexes**; what comes out of it is an ordered list. Candidates are
selected on collection, path, action and subject kind. A write matching
nothing pays only for the walk it already owed, because the diff it needed for
indexing is the diff it needed anyway.

Selecting on kind at the tree is what keeps a policy scoped to agents from
ever seeing a host's write.

### Stage 7 · Evaluation

Policies **only deny**. Any deny from any source is sufficient, so adding a
policy can never widen what is permitted. That monotonicity is what makes the
rest coherent: order is presentational rather than semantic, precedence
between sources needs no rule, and **failing closed is strictly conservative**
— a refusal on an unavailable evaluator is guaranteed to be at least as
restrictive as a complete evaluation would have been.

Failing closed is **scoped by the tree**: if the evaluator serving one policy
is down, only writes that policy would have judged are refused. Everything
else is untouched.

The engines themselves — what one is, the six channels it answers on, and
where its state lives — are in
[policy-evaluators.md](policy-evaluators.md).

`Freeze` is a transition on the repository's queue rather than a message sent
beside it. The write that trips it drains that repository's pending writes
with a rejection naming the freeze — which is also why a metapolicy that
freezes agents for tripping policies never observes the denials its own freeze
caused: those writes are drained without evaluation. That safety property
falls out of the per-repo queue, and parallelising the drain would silently
reintroduce the loop.

### The floor, which is always in the list

Some policies are compiled into the binary. They apply to every account and
to the server's own, and no record or configuration key reaches them. They
are `bot.did.policy` records like any other, in
`crates/didbot-policy-records/floor/`, read and compiled by the same code
an operator's record goes through; `floor/README.md` says what each one is
for and where its numbers come from, and the cases beside each one say what
it refuses and what it admits.

A built-in's id is `builtin/{name}/{statement}/{action}`, which is not an
at-uri, and a `bot.did.policyBinding` names policies by at-uri. So a
binding has nothing to name: an operator's records narrow the floor and
cannot widen it, which is monotonicity again rather than a check.

The counting engine keeps its windows in memory, so a restart is a fresh
window for every account and for the deployment.

### Stage 8 · Commit

The **only** stage that holds a store lock, and the lock it holds is the
repository's own: two accounts' writes do not wait on each other, and the
server-wide lock a write also holds is shared — only the blob collection pass
takes that one whole. Evaluation happens outside both, so a slow policy holds
neither, and a denied write never needs un-committing — which the write-ahead
log cannot do cheaply.

The commit applies the keys the write changed to the repository held in
memory at its head, and signs the result. Its cost grows with the depth of
the tree, not the number of records. A repository with nothing held — its
first write since the process started, or one pushed out of the
held set by writes to others — is built from its records first.

The sequence number is assigned at admission, not arrival. The policy version
and the `now` that judged the write are recorded with it, because decisions
are not reproducible — an evaluator may have a model in the loop, and an agent
may delete data an evaluation read.

Because stages 6 and 7 run outside the lock, a write's subject is built from a
read of its key taken before stage 5, and the write is applied against a
second read taken here. This stage compares the two and refuses the write as
`InvalidSwap` when they disagree, so a verdict reached about one record never
admits a different one. The same comparison is what stops a delete that found
an empty key — and therefore skipped stages 5 through 7 entirely — from
removing a record that landed while it was on its way. A caller re-reads the
key and retries, as it would after any other compare-and-swap refusal.

## Where the records a policy declares are read

A policy may name records in repositories this deployment does not hold —
what an account the write names has declared about itself. Reading one is
network I/O and stage 8 holds a store lock, so the read cannot happen where
the decision does. It happens in the route that accepted the write, between
stages 4 and 5: the applicable policies' `lookups` say which records they
want, `didbot-policy-records`'s account extraction says whose repositories
to ask, and `didbot-lookup` fills its deployment-wide cache within its own
deadlines. Stage 7 then reads that cache synchronously and hands each answer
to the evaluator as a field of the subject. A record the read could not
establish is an unknown rather than a value or an absence, and each statement
says for itself whether an unknown denies or stands aside.

A write naming more accounts than `didbot_lookup::MAX_NAMED_ACCOUNTS`, while a
policy that declares a lookup applies to it, is refused at stage 7 before any
statement is asked. Judging it on the first few accounts would let one extra
name carry any name at all.

## A batch is one commit, judged operation by operation

`com.atproto.repo.applyWrites` carries several writes and produces exactly one
commit, so it carries at most 200: atproto's sync specification allows no more
record operations in one commit, and a relay drops a commit that has them. A
larger batch is refused as soon as its body is read. It runs the same eight
stages, with three things settled by that "one commit":

- **One subject per operation.** Each operation has its own collection, action
  and diff, so each is its own write subject at stages 6 and 7. Folding a
  batch into a single subject would hide every operation but one from the
  tree.
- **One turn in the repository's queue, not one per operation.** The whole
  batch takes stage 5's line once. Judging operation three after another
  write to the same repository slipped in behind operation two would leave the
  batch judged against a state it is not committed against.
- **All or nothing.** Stage 8 runs only if every operation was allowed. The
  record store already applies a batch atomically, a partially applied batch
  would be a commit no caller asked for, and deny-only monotonicity's answer to
  "some of this is refused" is to refuse.

The batch's answer is the **most severe** of its operations' outcomes, not the
first: a freeze tripped by one operation outranks a rejection tripped by
another, and reaches the caller as a freeze with the account frozen. Judgment
stops at the first freeze and no later operation is judged or observed, for the
same reason a freeze drains the writes queued behind it without judging them.
A rejection does not stop judgment, so every operation the caller attempted is
still observed — a batch must not be a cheaper way to hide attempts from a
stateful evaluator than the same writes sent one at a time.

## Where a blob upload is judged

`uploadBlob` is not a record write and has no diff, so it does not pass
through the stages above. It is judged under `blob.write` before its body,
while the body arrives, and once it is in.

The first judgment is on what the request claims — the account, the type the
uploader declared and the `Content-Length`, if it sent one — before a byte of
the body is read, so a blob a policy refuses for its type or its size is
never carried. What it does not know is absent rather than guessed, and
absent facts do not deny.

While the body arrives, the upload is judged again once 64 KiB is in, and
again each time what has arrived doubles. These judgments add the type read
from the first bytes. Their size is the `Content-Length`, or the bytes
received so far when the request sent none, so a statement that refuses
small blobs can refuse a large one sent with no `Content-Length` before the
rest of it arrives. With no `Content-Length`, a body a policy refuses for
its size stops at the first frame that carries it past twice the policy's
limit, or past 64 KiB when that is more. A large upload costs a few
judgments, not one per frame. None of these judgments admits the upload:
each refuses it or lets it go on.

The last judgment is once the bytes are in and before they are stored. It
has the type read from the blob and the size that actually arrived, and its
answer is the upload's. An evaluator observes the upload once, at the
judgment that refuses it or at this one.

A body whose length differs from its `Content-Length` is refused as
`InvalidRequest`, as atproto's blob specification asks, and one that runs
past its length is refused at the frame that does.

A refused upload is discarded. An upload presented by an OAuth token names
its client, and its declared type is checked against that token's `blob:`
scope before any judgment. A `blob:` scope names types and not sizes, so
what bounds a body is these judgments and the server's blob limit. One
presented by the account's own token names no client and is not
scoped.

## Where authentication sits

Not as a stage of its own, and deliberately.

Verifying a token cryptographically is mechanism, and cheap. Deciding whether
a token **may be used** is a policy question and belongs at stage 7: a token
issued before its application was denied is still perfectly valid and still
names a real account, and only a write-tree rule that can see `client_id`
catches it. That is why `client_id` is on the write subject and not only on
the grant subject. It is the client the OAuth token was issued to; a write
made with the account's own token names none.

A token's scope is not policy. It is what the client requested, as the
ceiling in force at the moment of the write grants it — so a ceiling moved
since sign-in moves every live login with it — and it is checked with the
credential: once the
repository is known, before the record is judged, since the collection and
the action are both in the envelope. A write outside it is `403
InsufficientScope`, naming the scope it needed. A collection that is not an
NSID is a `400 InvalidRequest` before the scope is asked. `putRecord` needs both
`create` and `update`, because which one it does is not known until the store
is read. The account's own token has no scope.

So nothing before stage 7 decides whether a token may be used beyond what it
was granted.

## Where each gate actually runs

The order above is the order the routes run these checks in, and two details
of *how far* each one gets ahead of the body are worth stating, because both
are load-bearing and neither is obvious from the diagram.

**Stages 1 and 2 read nothing.** Every `com.atproto.repo.*` write route
consults the e-stop and then the lifecycle as its first two acts — before
authenticating, before deserializing, before resolving a handle. A caller
that sends an unparseable body to a halted server is told `Halted`, not
`InvalidRequest`, because the latch is an atomic read that needs no body and
the former is the fact worth acting on. The same gate covers the `bot.did.*`
routes that destroy an account or widen what it may do: `deleteAccount`,
`unfreezeAccount` and `activateAccount` are refused while the stop stands.
`freezeAccount` and `deactivateAccount` are not, and deliberately — each can
only ever narrow what an account may do, and a stop that stood between a
caller and restraining itself would be worse than the one it replaced.

**Stage 3 needs the envelope and nothing more.** Which repository is meant is
in the request, so the freeze check cannot precede deserialization outright
the way stages 1 and 2 do. It precedes every judgment of the *record*, which
is what stage 4 is: a frozen account sending a record its lexicon refuses is
told `AccountNotWritable`, never `InvalidRecord`.

**Stages 1 and 3 are read twice.** Both latches are read fresh every time, so
each is checked again immediately before the write — the e-stop just before
the store lock is taken, and the account's writability inside it, against a
re-read of the account rather than the one the write entered with. A write
can wait a long time at stage 5 while a slow judgment runs ahead of it, and
an operator who freezes an account in that window must not be told the
account is frozen while a write already past stage 3 goes on to commit. The
second read is what closes that. It does not make the gates atomic with the
commit — nothing before the store lock can be — but every window it leaves is
inside the repository's own write lock rather than around a policy evaluation.

Every path that queues reads it twice, `applyWrites` included, and a batch
that finds the account frozen at the second read is refused whole: a batch is
one commit, so there is no half of it to keep. That refusal is
`AccountNotWritable`, not the freeze a policy verdict raises — an operator's
freeze arriving from outside neither stops judgment of the operations behind
it nor freezes the repository's line, which only stage 7 does.

## Shortcuts

- **Stages 1 and 2 short-circuit before any parsing, and stage 3 before any
  parsing of the record.** A halted server, a server not yet accepting
  writes, and a frozen account are all decided without judging the request
  body.
- **Stage 6 short-circuits on an empty match**, which is the common case.
- **Stage 7 short-circuits the decision channel** — once a `Freeze` is
  reached, evaluation stops. The **observation** channel does not
  short-circuit: every evaluator watching a matching surface sees every
  attempt and its outcome, including attempts an earlier policy already
  denied, because a metapolicy counting policy failures must see denials it
  did not cause.
- **A freeze drains the rest of that repository's queue** without evaluating
  any of it.

## What is recorded

- **An allow** costs a policy version and a hash of the matched set. A row per
  admitted write is where the volume is and buys nothing.
- **A denial** records an evaluation id, the policy version, which policies
  fired, the verdict, a hash of the payload, and a **policy-authored** reason —
  and **no part of the refused payload**. The refused data is by definition
  what a policy decided should not exist here; storing it would put it durably
  on disk in a log with its own retention, written by the mechanism meant to
  prevent it.
- Detail flows to the party that already has it: the rejection returned to the
  caller may be as specific as the policy likes, because the caller sent the
  data.