id: alerts title: The server says when it is failing, somewhere its operator will see it status: open crates: [didbot-serve, didbot-pds, didbot-lexicon] dependsOn: [pds-writes, ownership] exitCriterion: > A certificate renewal that has been failing for a day is a record in this server's own repository naming its operator, a reader can tell it from a transient blip, and the operator can tell a healthy server from one that has stopped talking. #

alerts #

This server holds a DID and a repository, so it can write records like anything else on the network. That makes the notification channel one it already has: when something is wrong, say so as a record, and name the operator.

deploy lists what wants saying — certificate expiry, a policy poll that has not succeeded, clock skew against the attestation window, and disk — under "alert on the things that fail quietly". This epic is where that list gets a mechanism instead of an intention.

The appeal is that it adds no dependency. There is no paging service to hold an account with, no SMTP credential on the box, and nothing new to compromise: the server signs an alert with the key it already signs commits with, and the operator reads it with any client. It is also public, which is the same property ownership leans on — an outsider can see that this deployment reported a problem, and when.

The failure that breaks it, and what to do about it #

The failures most worth hearing about are the ones that stop the alert being delivered. Writing the record is a local operation and will succeed; being read is not. If the wildcard certificate expires, TLS is down, the firehose is unreachable, and the alert about the expired certificate is behind the thing it is about. A full disk stops the write itself. A partition stops both.

So this channel is an early warning and must not be mistaken for an alarm.

• A heartbeat, so silence means something. An alert is a positive signal: its presence says something is wrong, and its absence says nothing at all. A record written on a schedule inverts that — the operator watches for the gap, and the modes that suppress an alert suppress the heartbeat too. Without one, a server that has stopped talking is indistinguishable from a server with nothing to report, which is the same distinction the disclosure routes in auth-types had to make and the opposite way round.
• Say plainly, in the epic and in the operator documentation, what this channel cannot tell you. A mechanism believed to be an alarm when it is a warning is worse than no mechanism, because it is trusted.

What an alert may contain #

• No paths, no tokens, no internal state. A record is public and permanent. The exposure review found filesystem paths reaching anonymous clients through 500 bodies; the same discipline applies harder here, because a leak in an error response is transient and a leak in a repository is not. An alert names what is failing and since when, and leaves the detail to a log nobody else can read.
• Deduplicate, and rate-limit. A renewal retried hourly must not become twenty-four permanent records a day. One record per condition, updated or superseded rather than repeated, and a floor on how often a flapping condition may write at all. Repository growth is the cost of getting this wrong and it is not recoverable.
• Decide what clears an alert. A condition that recovers should say so, or the operator is left reading a permanent record of a problem that ended.

The shape #

• A lexicon for the record. There is no lexicon epic — a bot.did.* schema belongs to whichever epic needs it, and this is that epic. It wants a severity, a condition identifier stable enough to deduplicate on, a first-seen time, and the operator it names.
• Which DID writes it. The service DID, the one ownership already has the operator vouching for, so a reader checking who is complaining lands on a chain that is already checkable.
• Tagging the operator is a mention, and delivery is mentions's question rather than this epic's. Until that lands, an alert is a record the operator has to look for.
• The conditions themselves, each of which lives with the thing it watches rather than here: certificate renewal in agent-accounts, the policy poll in policy-store, clock skew in attestation, and disk in deploy. This epic owns the channel; it does not own the thresholds.

Where it does not belong #

• Not a replacement for ops-dashboard. That pane shows live state and can lie about it under compromise. This channel has the opposite property and the opposite weakness: it cannot be edited after the fact, and it can be silenced. They answer different questions and neither subsumes the other.
• Not the e-stop. The stop path is local and immediate precisely because it must work when everything else does not. An alert crosses a repository and a reader; that is fine for a warning and useless for a halt.

Done #

Nothing closed yet.