id: fleet title: One operator runs ten personal data servers and can see and stop all of them status: open crates: [didbot-serve] dependsOn: [ownership, e-stop, policy-store] exitCriterion: > An operator enrolls a second and tenth server, sees which agents each is running and which policy revision each believes it is enforcing, halts all of them from one place, and is told plainly which ones did not answer. #

fleet #

Ten servers, one operator, and the layer above them. Each deployment is a whole personal data server in its own right — its own zone, its own accounts, its own keys, its own repositories — and this epic is about operating the set rather than any one of them.

It is not high availability, and it is not this project's answer to capacity. Two processes serving one deployment's state is a different question and largely a different problem; the vertical half of it lives in store-scale. Nothing here makes one server survive its own failure. It makes ten servers legible and controllable by one person.

Several servers is the natural shape rather than a scaling workaround. Blast radius is the first reason — a compromise takes one zone and its accounts, not all of them — and separate zones, separate trust domains, separate machines and separate tenants are the rest. That is blast radius between whole deployments; services is the same reasoning applied inside one, where the component that can mint a hostname and pass DNS-01 is still the same process that serves agent data.

Most of the read and configuration path already exists #

This is worth stating before proposing anything, because the temptation is to build a control plane that duplicates three things this project already has.

• Membership is already expressible. index discovers every vouched server and walks the vouch chain to find them. An operator vouching for a server's DID is the roster, and it is a public, checkable statement rather than a list in a console's database.
• Configuration already fans out. policy-store puts policy in the operator's repository and has each server poll for it. Ten servers polling one repository is the same mechanism with no change: the operator writes once and the fleet converges. That the fan-out is a pull is what keeps it from needing a credential per server.
• The aggregate view already exists in shape. The index reads many servers and canvas draws them; what is missing is the operational half, not the observational one.

So the work is what those three do not cover.

The tension this epic is really about #

A console that can operate ten servers is a thing whose compromise operates ten servers. That is exactly the concentration the rest of this design spends effort avoiding — policy is a pull precisely so no credential to write it has to exist anywhere.

The same trick does not obviously work here. e-stop is local and immediate by design, because it has to work when everything else is broken; a halt that a server discovers on its next poll has a propagation delay measured in the poll interval, and the whole argument for the stop being local is that it crosses nothing. So a fleet stop is either a fan-out from something holding credentials to ten servers, or it is slower than the mechanism it fans out to.

• Decide that, and write down which property was traded. A pull-based halt with a stated interval is defensible; so is a credentialed fan-out with the blast radius named. Choosing without noticing is not.
• A fan-out must report per-server outcomes. "Stopped the fleet" when three servers did not answer is worse than useless, because it is believed. Partial success is the normal case in a fan-out and the interface has to make it the obvious one.
• No fleet mechanism may widen anything. A server takes ceilings from the operator's repository; a console sitting above it must be able to halt and to observe, and must not become a second path to grant.

What has to be built #

• Enrollment and decommissioning. How a server joins the roster, what proves it is the operator's, and what removing it means for the agents it holds. ownership's bidirectional check is the mechanism: a server the operator vouches for and which claims that operator back, with a verifier that refuses a pair where only one side says so.
• Divergence, as a table. Each server should publish the policy revision it believes it is enforcing — policy-store's argument is that this turns an audit into a comparison of two strings. Across a fleet that becomes one column, and a server stuck on an old revision is visible without asking it anything.
• Aggregate health that distinguishes silence from health. Ten servers means the interesting states are "one is not answering" and "one is answering and has stopped making progress". deploy asks for that distinction per server; here it also has to survive being summarised, because a green fleet light over a dead server is the failure mode.
• Upgrades across a fleet, which multiplies deploy's unfinished ones: staged rollout, version skew between servers while it is in progress, and the log-layout refusal happening on ten machines rather than one.
• A third surface, and saying why. Policy authoring is a static site the servers do not host; per-server operations is a pane the server serves and which can lie about itself under compromise. A fleet console is neither, and its own trust properties need stating rather than inheriting.

What does not move #

• An agent belongs to one server for its whole life. Its DID is a hostname under that server's zone, and a DID cannot be corrected, so "move an agent between servers" is not a feature this epic can offer. Migration is a total block elsewhere in this design and stays one here; what a fleet can do is place a new agent, not relocate an existing one.
• Each server keeps its own e-stop. Whatever fans out, the local latch stays the thing that actually stops a server, so a fleet mechanism that is down leaves ten working stops behind it rather than none.

Done #

Nothing closed yet.