Operational incidents #

Operational incidents are a content-dark projection over existing failure and recovery evidence. They make failures queryable, appendable to one private ledger, and eligible for a separate alert policy without turning source bodies, provider detail, or arbitrary error strings into another data sink.

Contract #

stream.thought.runtime.incident@1 is append-only and always sensitive. Its payload contains only bounded values from this allowlist:

• incidentVersion: 1
• incidentId: deterministic identity for one source failure, recovery, run terminal state, scheduler exhaustion, or delivery failure
• state: open or recovered
• category: versioned incident category
• severity: info, warning, error, or critical
• component: bounded runtime component identifier
• code and optional stage: allowlisted machine classifications
• fingerprint: SHA-256 over normalized category, component, source, code, and stage; never over raw error text
• occurredAt: the originating evidence time
• retryable: whether ordinary reconciliation can attempt the underlying work again
• progress: advanced, unchanged, unknown, or not-applicable
• optional bounded source, agentId, agentVersion, attempt, and strong reference ids

The incident payload never contains Error.message, stack traces, provider or Telegram bodies, prompts, source payloads, model output or reasoning, tool arguments/results, routes, credentials, environment values, command lines, or hashes of credential-bearing raw text. The source failure row may predate this contract and contain more detail; the incident projector does not copy it.

Projection inputs #

The first slice normalizes:

• stream.thought.connector.failed as an open connector warning;
• a failed stream.thought.connector.subscription.stopped as an open terminal connector error;
• stream.thought.connector.recovered as connector recovery evidence;
• stream.thought.agent.run.failed, .blocked, and .abandoned using the terminal run, trigger, and consumer progress evidence;
• exhausted ConsumerScheduler operations after their configured retries;
• stream.thought.action.telegram.send.failed as a delivery incident.

Projection is deterministic and idempotent. The first valid append-only incident row for an origin event is authoritative on later projection passes; current consumer progress cannot rewrite its retryable or progress classification. If an earlier failed attempt is first projected only after a later attempt under the same execution key exists, the earlier failure is classified as progress-unchanged and retryable even when the later attempt has since advanced source progress. It does not alter the originating event, retry count, consumer progress, model invocation, or delivery claim. A failed incident projection is an observability failure, not authority to replay an external action.

Private ledger #

When the manifest enables incidents, the incident service writes every normalized incident to one configured JSONL file below the runtime root. The writer:

• rejects absolute, escaping, and symlink paths;
• creates private parent directories and enforces mode 0600 on the ledger;
• emits canonical one-line JSON bounded to 4096 bytes;
• opens in append mode, writes one complete line, and calls fsync before acknowledging it;
• reloads deterministic incident ids on restart so a completed append is not repeated;
• treats a malformed or oversized existing ledger as a startup failure rather than silently replacing it.

The ledger is a private operational document. It is not a source for model context, training export, public content, or ordinary inspector payload rendering.

Telegram incident alerts #

Incident alerts are a separate action policy from normal output notifications. They select normalized incident categories, not trigger-source allowlists. A Jetstream-rooted agent failure can therefore alert without making successful Bluesky observations Telegram-eligible.

The dispatcher does not suppress incidents by agent identity or retry disposition. Canary, test, and retryable failures all generate alerts. Every alert says what subsystem failed, whether the runtime will retry, and a short receipt hash for correlation. Technical component identifiers, source labels, classification codes, and stage names remain in the private ledger and inspector projection — they do not appear as raw field labels in the Telegram message.

Alert format #

Each alert is a short multi-line message:

1. A header line: The Stream · <short title> (with occurrence count when > 1).
2. A plain-language description of the failure.
3. A retry line: Will retry or Won't retry automatically, with attempt number when available.
4. A receipt line: Receipt <short id> for correlation.

The alert does not contain raw component:, source:, classification:, retry:, or progress: field labels, agent ids, event ids, run ids, or tool names. The title and description use plain human language naming the affected subsystem and the nature of the failure.

Dispatch protocol #

For each destination, the incident dispatcher:

1. establishes one durable activation cutoff;
2. reads open incident events after that cutoff;
3. groups by normalized fingerprint;
4. excludes Telegram delivery failures to prevent recursive alert attempts;
5. enforces a durable destination window and per-fingerprint cooldown;
6. appends a deterministic started claim before one Bot API call;
7. appends delivered or content-dark failed evidence.

A claimed Telegram attempt is intentionally at-most-once: after the durable started claim, it is never retried automatically, even when no terminal receipt is observed. Telegram provides no idempotency key, so a blind retry could duplicate a notification whose first Bot API call succeeded but whose response was lost. Repeated incidents inside the cooldown remain in the ledger and may be summarized after the cooldown.

Process boundary #

This slice runs as a dedicated incident projection/ledger/alert process. It can record errors emitted by other running processes, but it cannot reliably report its own startup failure, a Jazz-open failure, or a host-level crash after it has stopped.

The external supervisor seam is explicit: a later systemd ExecStopPost/OnFailure helper should emit the same normalized contract from bounded unit metadata (SERVICE_RESULT, exit code/status, unit name, invocation identity), with explicit restart limits. It must not copy journal messages, environment, command lines, or stacks. This follow-up is required before claiming complete process-crash and malformed-startup coverage.