---
id: relay-sync
title: A boot tells relays only what changed, and a restore tells them what it lost
status: open
crates: [didbot-pds, didbot-serve]
dependsOn: []
exitCriterion: >
  A test restores a data directory from an older copy of itself and boots it,
  and a consumer that applied the lost run's frames and follows the sync spec
  ends holding each repository's restored records at a newer rev. A boot of a
  deployment where no account changed puts no frame on subscribeRepos.
---

# relay-sync

**Everything below is a proposal for the owner to accept, amend or refuse.**
It adds stored state, so nothing here is built until the owner has answered
the questions at the end.

An account's *position* is the `active` and `status` pair its `#account`
frame carries, as `HostedAccount::sync_status` answers it.

## The problem

- [#153](https://tangled.org/did:plc:swttlkbjvcoln67iievud7b3/issues/153): a
  restore moves every repository's head back by up to the snapshot's age.
  Relays and appviews keep the lost records, and nothing on `subscribeRepos`
  tells them.
- [#154](https://tangled.org/did:plc:swttlkbjvcoln67iievud7b3/issues/154):
  every boot sends one `#account` per account, erased ones included
  (`Provisioner::announce_accounts`, called from `assemble` in
  `crates/didbot-serve/src/bin/didbot-pds/run.rs`). Bluesky's relay counts
  each against a new host's 50 events a second, 2,600 an hour and 21,000 a
  day ([rate limits](https://bsky.network/docs/rate-limits/)).

## What the spec and the reference code say

Read on 2026-09-23: the specs at atproto.com, the reference PDS and the
lexicons at atproto `9c76c34`, indigo's relay at `dbcca56`. Where the spec and
the code differ, the owner's rule is to follow the spec.

- **`#sync` resets a repository.** "Sync events are broadcast when the
  account repository state has been reset to a new state"
  ([sync](https://atproto.com/specs/sync)). It carries the signed commit block
  and its `rev`, and a consumer fetches the repository to catch up.
- **A consumer ignores an old rev.** "services should ignore `#commit` and
  `#sync` events with a `rev` lower or equal to the most recent successfully
  processed `rev` for that DID" (sync). After a migration the spec asks for "a
  `#sync` and/or `#commit` event … with a higher commit `rev`"
  ([account](https://atproto.com/specs/account)). indigo stores any `#sync`,
  whatever its rev
  ([`ingest.go:169`](https://github.com/bluesky-social/indigo/blob/dbcca561c7035fbda2c92ddd6a35b303db1f4a0f/cmd/relay/relay/ingest.go#L169-L176)),
  and drops one for an account it holds as inactive
  ([`ingest.go:159`](https://github.com/bluesky-social/indigo/blob/dbcca561c7035fbda2c92ddd6a35b303db1f4a0f/cmd/relay/relay/ingest.go#L159-L162)).
- **A reset stream starts well above its old numbers.** "it is recommended to
  chose a new initial sequence number with a healthy margin above any previous
  sequence number. For example, after persistent storage loss"
  ([event-stream](https://atproto.com/specs/event-stream)).
- **indigo stops dialing a host that answers `FutureCursor`.** It marks the
  host idle and does not reconnect
  ([`slurper.go:442`](https://github.com/bluesky-social/indigo/blob/dbcca561c7035fbda2c92ddd6a35b303db1f4a0f/cmd/relay/relay/slurper.go#L442-L447)).
  A later `requestCrawl` resubscribes it from the same stored cursor, and is a
  no-op for a host it is still reading
  ([`crawl.go`](https://github.com/bluesky-social/indigo/blob/dbcca561c7035fbda2c92ddd6a35b303db1f4a0f/cmd/relay/relay/crawl.go#L11-L55)).
  An `#info` frame is logged and passed over
  ([`slurper.go:453`](https://github.com/bluesky-social/indigo/blob/dbcca561c7035fbda2c92ddd6a35b303db1f4a0f/cmd/relay/relay/slurper.go#L453-L456)).
- **Every frame counts, and a frame over the limit waits.** indigo counts
  frames of every type against a second, an hour and a day
  ([`slurper.go:187`](https://github.com/bluesky-social/indigo/blob/dbcca561c7035fbda2c92ddd6a35b303db1f4a0f/cmd/relay/relay/slurper.go#L187-L189)),
  and holds one over the limit rather than dropping it
  ([`consumer.go:71`](https://github.com/bluesky-social/indigo/blob/dbcca561c7035fbda2c92ddd6a35b303db1f4a0f/cmd/relay/stream/consumer.go#L71-L87)).
  Once every worker is waiting, it stops reading the socket
  ([`parallel.go:122`](https://github.com/bluesky-social/indigo/blob/dbcca561c7035fbda2c92ddd6a35b303db1f4a0f/cmd/relay/stream/schedulers/parallel/parallel.go#L122-L127)),
  and this server closes a subscriber 4,096 frames behind with
  `ConsumerTooSlow`.
- **`#account` states a position, and a repeat still goes out.** "The event
  describes the current hosting status, not what changed" (sync). indigo
  forwards every one
  ([`ingest.go:271`](https://github.com/bluesky-social/indigo/blob/dbcca561c7035fbda2c92ddd6a35b303db1f4a0f/cmd/relay/relay/ingest.go#L271-L287)).
  The reference PDS sends nothing at boot
  ([`index.ts`](https://github.com/bluesky-social/atproto/blob/9c76c3422ed0c5369871633d2c4ed67a2f61ae33/packages/pds/src/index.ts#L216-L224)).
- **A relay can say how far it read.** `com.atproto.sync.getHostStatus`,
  "implemented by relays", returns its `seq` for a host
  ([lexicon](https://github.com/bluesky-social/atproto/blob/9c76c3422ed0c5369871633d2c4ed67a2f61ae33/lexicons/com/atproto/sync/getHostStatus.json)).

## When a relay holds more than this server

| Cause | Heads | Sequence | Positions |
| --- | --- | --- | --- |
| A restore | back by up to the snapshot's age | back to the snapshot's floor, so numbers are handed out again | back |
| A power cut | back by up to 250 ms of commits: a record and its commit are announced before the log syncs them (`DEFAULT_SYNC_INTERVAL` in `crates/didbot-pds/src/wal/mod.rs`) | unchanged: a reservation syncs before its numbers are used | unchanged: an account change syncs before its frame |
| A crash between an account change and its frame | unchanged | unchanged | the relay is behind, not ahead |

An account created after the snapshot has no row, no key and no document on
the restored server, so nothing this server sends can speak for it.

## 1. How the server learns it was restored

The volume cannot tell a restore from a restart: every fact on it came back
with the snapshot.

**A. An operator step leaves a marker.** Before the first start of a restored
volume, the operator creates `pds.restored` in the data directory. The boot
that finds it sends the restore frames (section 2) and removes it.
- Costs: one line in the restore runbook.
- Breaks: nothing when it is run. An operator who skips it gets today's
  behaviour.
- Stores: one empty file, from the step to the end of the next boot.

**B. A consumer's cursor ahead of the stream.** A cursor above the newest
number is evidence the stream went back. Today it gets `FutureCursor` and a
close.
- Costs: nothing.
- Breaks: anyone can send any cursor, so acting on one hands a stranger the
  relay budget. It sees a lost run only once that run reserved a new block of
  1,024 numbers; a smaller loss moves heads back with no future cursor. And
  indigo stops dialing after one `FutureCursor`, so the evidence arrives once,
  after the damage.
- Stores: nothing.

**C. Ask the relay.** `getHostStatus` on the configured relay gives its `seq`
for this host, and its `getRepoStatus` gives its `rev` for one account.
- Costs: a call to the relay at boot, which `docs/deployment.md` rules out
  ("Boot does not dial the relay"), or an operator command instead. One call
  per account to learn which heads moved.
- Breaks: it hears from one relay and from no appview.
- Stores: nothing.

**Proposed: A.** It is the only one that knows rather than guesses. The
warning `crates/didbot-serve/src/subscribe.rs` logs on a `FutureCursor` stays,
and names the step. The Sync 1.1 proposal gives the host the same job:
"It is not the responsibility of a PDS, or a relay, to proactively detect if
downstream consumers have gotten out of sync"
([0006](https://github.com/bluesky-social/proposals/blob/main/0006-sync-iteration/README.md)).

A power cut is the one case no marker covers. Question 5 asks whether to
close it.

## 2. What a restore sends

**A. `#sync` for every live repository, over a fresh commit.** First the
sequence floor rises, so a relay's cursor is behind the stream rather than
ahead of it. Then, for each account whose position is `active`, the server
signs a commit over the tree the account already has: the same `data`, a new
`rev`, no record changed. It sends `#account`, then `#sync` carrying that
commit. Every other account that is not `decommissioned` gets `#account`.
- Costs: one signature, one log append and two frames per live account, once.
  1,000 live accounts is about 2,000 frames, which fits a new host's hour on
  Bluesky's relay only if nothing else is sent.
- Breaks: each live repository gains a commit no client wrote, and `getRepo`
  and `listRepos` answer it.
- Stores: the new heads, in the commit history that exists.

**B. `#sync` only where the head moved back.**
- Costs: it needs 1C to learn which heads moved.
- Breaks: what 1C breaks.
- Stores: nothing.

**C. Nothing: the operator resets every consumer, as `docs/deployment.md`
says today.**
- Costs: nothing to build.
- Breaks: an operator cannot reset Bluesky's relay or anybody's appview. The
  lost records stay up, and the agent cannot delete what the server no longer
  holds. An account that never writes again stays wrong for good.
- Stores: nothing.

**Proposed: A.** An account whose head did not move costs a consumer little:
the fresh commit names the `data` it already holds.

## 3. What a boot sends

**A. Every account, every boot (today).**
- Costs: one frame per account with a position, every boot. At the default cap
  of 1,000 accounts, three boots in an hour spend more than a new host's 2,600.
  Past 4,096 accounts, the boot's own frames push each other out of the replay
  buffer before a relay connects.
- Breaks: the relay budget.
- Stores: nothing.

**B. Only accounts whose position changed since it was last sent.** Each
account row carries `announced`: the position of the last `#account` frame
numbered for it. A boot sends where the row's position and `announced` differ.
- Costs: one log append per `#account` frame. The first boot after the upgrade
  sends every account once, because no row has `announced` yet.
- Breaks: a frame numbered and then lost with the process, before any
  consumer read it, is not sent again. The consumer gets `#info OutdatedCursor`
  for that gap, as it does for a lost commit.
- Stores: one field per account row, and one new log entry.

**C. Nothing.**
- Costs: nothing.
- Breaks: it reopens the window `announce_accounts` closes. A process killed
  between a lift and its frame leaves every relay holding `suspended` for an
  account that serves.
- Stores: nothing.

**Proposed: B.** A boot that changed nothing sends nothing, and the crash
window stays closed. A lock and a lift use the same rule, comparing the row
with `announced` rather than with the row before the change. So a frame lost
to a failed reservation also goes out with the account's next change.

## The proposed machine

### Stored facts

| Fact | Type | Lives in | Written | Read |
| --- | --- | --- | --- | --- |
| `announced` (new) | `Option<SyncAccountStatus>`: the position of the last `#account` frame numbered for the account | a field on `HostedAccount`. A new log entry `AccountAnnounced { did, active, status }`, `Durability::Sync` like every account entry. The checkpoint carries it inside `AccountInserted` | when an `#account` frame for the account is numbered | at every boot, and after every account change |
| `pds.restored` (new) | an empty file, present or absent | the data directory, beside `pds.sequence` | created by the operator's restore step, removed by the boot that finished the restore frames | at every boot, before any frame |
| sequence floor | `u64` | `Entry::StreamReserved` and `pds.sequence`, as now | as now, and raised by 10^12 by a boot that finds `pds.restored` | at every boot, as now |
| repository head | `history::Head` | the commit history, as now | as now, and a fresh commit per live account by a boot that finds `pds.restored` | as now |

The new log entry and the new file name each move the layout stamp, in one
layout change.

One rule covers both kinds of boot: the replay buffer holds every frame a boot
numbers before the listener binds, on top of its usual 4,096.

### Transitions of `announced`

Each reads the row and `announced` under the account's own turn
(`Provisioner::lifecycle_of`), which a lock, a lift and an erasure already
take.

| # | Trigger | Fires when | Frame | Then |
| --- | --- | --- | --- | --- |
| A1 | an account change is written: activation, a lock, a lift, an erasure | the row's position differs from `announced` | `#account` with the row's position | `announced` becomes that position, if the frame was numbered |
| A2 | a boot, before the listener binds | for each account with a position, it differs from `announced` | the same | the same |
| A3 | a frame that could not be numbered, because a reservation did not reach the log | — | none | `announced` is unchanged, so the next A1 or A2 sends it |
| A4 | the row is removed | — | none | the field goes with the row |

### Transitions of `pds.restored`

| # | Trigger | Before → after | What happens, in order |
| --- | --- | --- | --- |
| R1 | the operator's restore step, before the unit's first start | absent → present | nothing else |
| R2 | a boot that finds it, after the log is replayed and before A2 | present → present | 1. The floor rises by 10^12, in the log and in `pds.sequence`, before any frame is numbered. 2. For each account that has a position and is not `decommissioned`, in DID order: `#account` (A1's write follows it), then, if the position is `active`, a fresh commit and `#sync`. 3. A2, for the `decommissioned` accounts. 4. The log syncs. |
| R3 | R2 finished | present → absent | the file is removed, then the listener binds |
| R4 | the boot dies during R2 | present → present | nothing; the next boot runs R2 from the start, and repeats frames a relay may already hold |

**Why 10^12.** It is more numbers than this server can hand out in the thirty
days a recovery point is kept, at 385,000 a second. A relay's cursor from the
lost run is then behind the stream. It gets `#info OutdatedCursor` and the
whole replay buffer, which holds the restore frames. `FutureCursor` would have
made indigo stop dialing.

**Why a fresh commit.** The restored head's `rev` is older than the one a
relay holds, and the spec tells a consumer to ignore it. The minter reads the
stored rev and the clock (`Provisioner::commit_write`), so the fresh `rev` is
newer than any the lost run minted, as long as the restored host's clock is
not behind the lost host's.

**Why R2 is not paced.** indigo holds a frame over its limit rather than
dropping it, and every R2 frame is in the replay buffer before a relay
connects. Pacing cannot fix the budget: past about 1,300 live accounts, R2
alone spends more than a new host's hour, and the deployment wants its relay
limit raised first ([capacity](capacity.md)).

**What R2 leaves.** An account locked at the restore gets `#account` and no
`#sync`, because indigo drops `#sync` for an inactive account. Its first
commit after the lift breaks the chain, and a consumer that follows the spec
resyncs then. An account erased after the snapshot comes back active, and R2
says so. R2 sends no `#identity`: an account's handle and key are fixed when
it is created.

**The runbook.** `docs/operations.md`'s "Restoring" gains the step before
"Start the unit", and `didbot announce` after the checks: it redials a relay
that already saw `FutureCursor`, and is a no-op for one still reading. The
advice in `docs/deployment.md` to reset every consumer, or to raise
`pds.sequence` by hand, goes. `docs/conformance.md`'s rows on `#sync` and
`desynchronized` change with it.

## Tests

In `crates/didbot-pds/tests/account_announce_restart.rs`, against a recording
sink:
- A boot after a clean stop sends no `#account`.
- A boot after a lift that never reached the sink sends that account's
  `#account` and no other.
- A lock whose frame could not be numbered goes out with the account's next
  change, and otherwise with the next boot.
- The first boot over a log written before `announced` sends every account
  once, and the boot after it sends none.

In `crates/didbot-pds/tests/restore.rs`, over a copied data directory:
- With the marker, the first frame's number is above every number the lost
  run handed out.
- With the marker, each live account gets `#account` and then `#sync`. The
  `#sync` commit verifies with the account's key, names the restored `data`,
  and carries a `rev` above the lost run's highest for that account. A
  `decommissioned` account gets nothing.
- A boot stopped part way through R2 leaves the marker, and the next boot
  sends every frame again and removes it.
- A boot without the marker signs no commit.

In `crates/didbot/tests/`, over the wire, with a consumer built on
`support::ws` and `support::mst_invert` that applies the sync spec's rules: it
ignores a rev that is not newer, checks `since` and `prevData`, and fetches
`getRepo` when a `#sync` changes `data`.
- The exit criterion. The consumer applies a live run; the directory is
  copied; the lost run writes more; the copy boots with the marker. The
  consumer reconnects with its cursor, gets `OutdatedCursor` rather than
  `FutureCursor`, and ends holding each repository's restored records. The
  next write chains onto the `#sync` commit.
- A restart where nothing changed puts no frame on the wire before the next
  write.
- A boot that numbers more than 4,096 frames before the listener binds replays
  all of them to a consumer that connects after it.

`crates/didbot-serve/tests/restore_drill.rs` restores with the marker.
`listRepos` then answers a newer head and `rev` for each account, over the
same records.

If the owner picks the first answer to question 5: with the log's sync held
back, a commit's frame is withheld until its append syncs.

## Questions for the owner

1. How the server learns of a restore: an operator step that leaves
   `pds.restored` (proposed) / a consumer's future cursor / the relay's
   `getHostStatus`.
2. What a boot sends: positions that differ from `announced`, a new field per
   account (proposed) / every account, as now / nothing.
3. What a restore's `#sync` carries: a fresh commit no client wrote
   (proposed) / the restored commit, whose older `rev` consumers ignore.
4. How far a restore raises the sequence floor: by 10^12 (proposed) / to the
   current time in microseconds / by hand, as now.
5. The power-cut window: publish a commit only once its log append syncs, up
   to 250 ms later / keep publishing at once.

## Done