---

id: relay-sync title: A boot tells relays only what changed, and a restore tells them what it lost status: open crates: [didbot-pds, didbot-serve] dependsOn: [] exitCriterion: > A test restores a data directory from an older copy of itself and boots it, and a consumer that applied the lost run's frames and follows the sync spec ends holding each repository's restored records at a newer rev. A boot of a deployment where no account changed puts no frame on subscribeRepos. #

relay-sync #

Everything below is a proposal for the owner to accept, amend or refuse. It adds stored state, so nothing here is built until the owner has answered the questions at the end.

An account's position is the active and status pair its #account frame carries, as HostedAccount::sync_status answers it.

The problem #

• #153: a restore moves every repository's head back by up to the snapshot's age. Relays and appviews keep the lost records, and nothing on subscribeRepos tells them.
• #154: every boot sends one #account per account, erased ones included (Provisioner::announce_accounts, called from assemble in crates/didbot-serve/src/bin/didbot-pds/run.rs). Bluesky's relay counts each against a new host's 50 events a second, 2,600 an hour and 21,000 a day (rate limits).

What the spec and the reference code say #

Read on 2026-09-23: the specs at atproto.com, the reference PDS and the lexicons at atproto 9c76c34, indigo's relay at dbcca56. Where the spec and the code differ, the owner's rule is to follow the spec.

• #sync resets a repository. "Sync events are broadcast when the account repository state has been reset to a new state" (sync). It carries the signed commit block and its rev, and a consumer fetches the repository to catch up.
• A consumer ignores an old rev. "services should ignore #commit and #sync events with a rev lower or equal to the most recent successfully processed rev for that DID" (sync). After a migration the spec asks for "a #sync and/or #commit event … with a higher commit rev" (account). indigo stores any #sync, whatever its rev (ingest.go:169), and drops one for an account it holds as inactive (ingest.go:159).
• A reset stream starts well above its old numbers. "it is recommended to chose a new initial sequence number with a healthy margin above any previous sequence number. For example, after persistent storage loss" (event-stream).
• indigo stops dialing a host that answers FutureCursor. It marks the host idle and does not reconnect (slurper.go:442). A later requestCrawl resubscribes it from the same stored cursor, and is a no-op for a host it is still reading (crawl.go). An #info frame is logged and passed over (slurper.go:453).
• Every frame counts, and a frame over the limit waits. indigo counts frames of every type against a second, an hour and a day (slurper.go:187), and holds one over the limit rather than dropping it (consumer.go:71). Once every worker is waiting, it stops reading the socket (parallel.go:122), and this server closes a subscriber 4,096 frames behind with ConsumerTooSlow.
• #account states a position, and a repeat still goes out. "The event describes the current hosting status, not what changed" (sync). indigo forwards every one (ingest.go:271). The reference PDS sends nothing at boot (index.ts).
• A relay can say how far it read. com.atproto.sync.getHostStatus, "implemented by relays", returns its seq for a host (lexicon).

When a relay holds more than this server #

Cause	Heads	Sequence	Positions
A restore	back by up to the snapshot's age	back to the snapshot's floor, so numbers are handed out again	back
A power cut	back by up to 250 ms of commits: a record and its commit are announced before the log syncs them (DEFAULT_SYNC_INTERVAL in crates/didbot-pds/src/wal/mod.rs)	unchanged: a reservation syncs before its numbers are used	unchanged: an account change syncs before its frame
A crash between an account change and its frame	unchanged	unchanged	the relay is behind, not ahead

An account created after the snapshot has no row, no key and no document on the restored server, so nothing this server sends can speak for it.

1. How the server learns it was restored #

The volume cannot tell a restore from a restart: every fact on it came back with the snapshot.

A. An operator step leaves a marker. Before the first start of a restored volume, the operator creates pds.restored in the data directory. The boot that finds it sends the restore frames (section 2) and removes it.

• Costs: one line in the restore runbook.
• Breaks: nothing when it is run. An operator who skips it gets today's behaviour.
• Stores: one empty file, from the step to the end of the next boot.

B. A consumer's cursor ahead of the stream. A cursor above the newest number is evidence the stream went back. Today it gets FutureCursor and a close.

• Costs: nothing.
• Breaks: anyone can send any cursor, so acting on one hands a stranger the relay budget. It sees a lost run only once that run reserved a new block of 1,024 numbers; a smaller loss moves heads back with no future cursor. And indigo stops dialing after one FutureCursor, so the evidence arrives once, after the damage.
• Stores: nothing.

C. Ask the relay. getHostStatus on the configured relay gives its seq for this host, and its getRepoStatus gives its rev for one account.

• Costs: a call to the relay at boot, which docs/deployment.md rules out ("Boot does not dial the relay"), or an operator command instead. One call per account to learn which heads moved.
• Breaks: it hears from one relay and from no appview.
• Stores: nothing.

Proposed: A. It is the only one that knows rather than guesses. The warning crates/didbot-serve/src/subscribe.rs logs on a FutureCursor stays, and names the step. The Sync 1.1 proposal gives the host the same job: "It is not the responsibility of a PDS, or a relay, to proactively detect if downstream consumers have gotten out of sync" (0006).

A power cut is the one case no marker covers. Question 5 asks whether to close it.

2. What a restore sends #

A. #sync for every live repository, over a fresh commit. First the sequence floor rises, so a relay's cursor is behind the stream rather than ahead of it. Then, for each account whose position is active, the server signs a commit over the tree the account already has: the same data, a new rev, no record changed. It sends #account, then #sync carrying that commit. Every other account that is not decommissioned gets #account.

• Costs: one signature, one log append and two frames per live account, once. 1,000 live accounts is about 2,000 frames, which fits a new host's hour on Bluesky's relay only if nothing else is sent.
• Breaks: each live repository gains a commit no client wrote, and getRepo and listRepos answer it.
• Stores: the new heads, in the commit history that exists.

B. #sync only where the head moved back.

• Costs: it needs 1C to learn which heads moved.
• Breaks: what 1C breaks.
• Stores: nothing.

C. Nothing: the operator resets every consumer, as docs/deployment.md says today.

• Costs: nothing to build.
• Breaks: an operator cannot reset Bluesky's relay or anybody's appview. The lost records stay up, and the agent cannot delete what the server no longer holds. An account that never writes again stays wrong for good.
• Stores: nothing.

Proposed: A. An account whose head did not move costs a consumer little: the fresh commit names the data it already holds.

3. What a boot sends #

A. Every account, every boot (today).

• Costs: one frame per account with a position, every boot. At the default cap of 1,000 accounts, three boots in an hour spend more than a new host's 2,600. Past 4,096 accounts, the boot's own frames push each other out of the replay buffer before a relay connects.
• Breaks: the relay budget.
• Stores: nothing.

B. Only accounts whose position changed since it was last sent. Each account row carries announced: the position of the last #account frame numbered for it. A boot sends where the row's position and announced differ.

• Costs: one log append per #account frame. The first boot after the upgrade sends every account once, because no row has announced yet.
• Breaks: a frame numbered and then lost with the process, before any consumer read it, is not sent again. The consumer gets #info OutdatedCursor for that gap, as it does for a lost commit.
• Stores: one field per account row, and one new log entry.

C. Nothing.

• Costs: nothing.
• Breaks: it reopens the window announce_accounts closes. A process killed between a lift and its frame leaves every relay holding suspended for an account that serves.
• Stores: nothing.

Proposed: B. A boot that changed nothing sends nothing, and the crash window stays closed. A lock and a lift use the same rule, comparing the row with announced rather than with the row before the change. So a frame lost to a failed reservation also goes out with the account's next change.

The proposed machine #

Stored facts #

Fact	Type	Lives in	Written	Read
announced (new)	Option<SyncAccountStatus>: the position of the last #account frame numbered for the account	a field on HostedAccount. A new log entry AccountAnnounced { did, active, status }, Durability::Sync like every account entry. The checkpoint carries it inside AccountInserted	when an #account frame for the account is numbered	at every boot, and after every account change
pds.restored (new)	an empty file, present or absent	the data directory, beside pds.sequence	created by the operator's restore step, removed by the boot that finished the restore frames	at every boot, before any frame
sequence floor	u64	Entry::StreamReserved and pds.sequence, as now	as now, and raised by 10^12 by a boot that finds pds.restored	at every boot, as now
repository head	history::Head	the commit history, as now	as now, and a fresh commit per live account by a boot that finds pds.restored	as now

The new log entry and the new file name each move the layout stamp, in one layout change.

One rule covers both kinds of boot: the replay buffer holds every frame a boot numbers before the listener binds, on top of its usual 4,096.

Transitions of announced #

Each reads the row and announced under the account's own turn (Provisioner::lifecycle_of), which a lock, a lift and an erasure already take.

#	Trigger	Fires when	Frame	Then
A1	an account change is written: activation, a lock, a lift, an erasure	the row's position differs from announced	#account with the row's position	announced becomes that position, if the frame was numbered
A2	a boot, before the listener binds	for each account with a position, it differs from announced	the same	the same
A3	a frame that could not be numbered, because a reservation did not reach the log	—	none	announced is unchanged, so the next A1 or A2 sends it
A4	the row is removed	—	none	the field goes with the row

Transitions of pds.restored #

#	Trigger	Before → after	What happens, in order
R1	the operator's restore step, before the unit's first start	absent → present	nothing else
R2	a boot that finds it, after the log is replayed and before A2	present → present	1. The floor rises by 10^12, in the log and in pds.sequence, before any frame is numbered. 2. For each account that has a position and is not decommissioned, in DID order: #account (A1's write follows it), then, if the position is active, a fresh commit and #sync. 3. A2, for the decommissioned accounts. 4. The log syncs.
R3	R2 finished	present → absent	the file is removed, then the listener binds
R4	the boot dies during R2	present → present	nothing; the next boot runs R2 from the start, and repeats frames a relay may already hold

Why 10^12. It is more numbers than this server can hand out in the thirty days a recovery point is kept, at 385,000 a second. A relay's cursor from the lost run is then behind the stream. It gets #info OutdatedCursor and the whole replay buffer, which holds the restore frames. FutureCursor would have made indigo stop dialing.

Why a fresh commit. The restored head's rev is older than the one a relay holds, and the spec tells a consumer to ignore it. The minter reads the stored rev and the clock (Provisioner::commit_write), so the fresh rev is newer than any the lost run minted, as long as the restored host's clock is not behind the lost host's.

Why R2 is not paced. indigo holds a frame over its limit rather than dropping it, and every R2 frame is in the replay buffer before a relay connects. Pacing cannot fix the budget: past about 1,300 live accounts, R2 alone spends more than a new host's hour, and the deployment wants its relay limit raised first (capacity).

What R2 leaves. An account locked at the restore gets #account and no #sync, because indigo drops #sync for an inactive account. Its first commit after the lift breaks the chain, and a consumer that follows the spec resyncs then. An account erased after the snapshot comes back active, and R2 says so. R2 sends no #identity: an account's handle and key are fixed when it is created.

The runbook. docs/operations.md's "Restoring" gains the step before "Start the unit", and didbot announce after the checks: it redials a relay that already saw FutureCursor, and is a no-op for one still reading. The advice in docs/deployment.md to reset every consumer, or to raise pds.sequence by hand, goes. docs/conformance.md's rows on #sync and desynchronized change with it.

Tests #

In crates/didbot-pds/tests/account_announce_restart.rs, against a recording sink:

• A boot after a clean stop sends no #account.
• A boot after a lift that never reached the sink sends that account's #account and no other.
• A lock whose frame could not be numbered goes out with the account's next change, and otherwise with the next boot.
• The first boot over a log written before announced sends every account once, and the boot after it sends none.

In crates/didbot-pds/tests/restore.rs, over a copied data directory:

• With the marker, the first frame's number is above every number the lost run handed out.
• With the marker, each live account gets #account and then #sync. The #sync commit verifies with the account's key, names the restored data, and carries a rev above the lost run's highest for that account. A decommissioned account gets nothing.
• A boot stopped part way through R2 leaves the marker, and the next boot sends every frame again and removes it.
• A boot without the marker signs no commit.

In crates/didbot/tests/, over the wire, with a consumer built on support::ws and support::mst_invert that applies the sync spec's rules: it ignores a rev that is not newer, checks since and prevData, and fetches getRepo when a #sync changes data.

• The exit criterion. The consumer applies a live run; the directory is copied; the lost run writes more; the copy boots with the marker. The consumer reconnects with its cursor, gets OutdatedCursor rather than FutureCursor, and ends holding each repository's restored records. The next write chains onto the #sync commit.
• A restart where nothing changed puts no frame on the wire before the next write.
• A boot that numbers more than 4,096 frames before the listener binds replays all of them to a consumer that connects after it.

crates/didbot-serve/tests/restore_drill.rs restores with the marker. listRepos then answers a newer head and rev for each account, over the same records.

If the owner picks the first answer to question 5: with the log's sync held back, a commit's frame is withheld until its append syncs.

Questions for the owner #

1. How the server learns of a restore: an operator step that leaves pds.restored (proposed) / a consumer's future cursor / the relay's getHostStatus.
2. What a boot sends: positions that differ from announced, a new field per account (proposed) / every account, as now / nothing.
3. What a restore's #sync carries: a fresh commit no client wrote (proposed) / the restored commit, whose older rev consumers ignore.
4. How far a restore raises the sequence floor: by 10^12 (proposed) / to the current time in microseconds / by hand, as now.
5. The power-cut window: publish a commit only once its log append syncs, up to 250 ms later / keep publishing at once.

Done #