# Connector contracts

## Shared interface

Every connector implements:

- `id`: stable connector instance id.
- `kind`: source kind.
- `describe()`: nonsecret runtime metadata.
- `poll(cursor, signal)`, `subscribe(cursor, emit, signal)`, or `ingest(delivery)`.
- `normalize(raw)`: zero or more registered event candidates.
- `checkpoint()`: source cursor safe to persist after durable append.
- `health()`: last success, last failure, lag, and reconnect count.

Connectors return event candidates to their producer process. The producer derives deterministic Jazz row ids, assigns its source-local sequence, and commits events with its external cursor.

## Supervisor manifest

The local runtime reads `thoughtstream.yaml` by default. The filename is an internal filesystem identifier; the product and command remain `thought stream`.

- The manifest is strict and versioned. Unknown keys, duplicate source ids, unsafe intervals, non-loopback inspector hosts, and paths escaping the project root where containment is required fail before any source starts.
- Every source has an explicit `enabled` flag. Omission means disabled.
- Relative source paths resolve from the project root, not the invoking shell's current directory.
- Filesystem watchers perform one durable initial scan, then coalesce changes and serialize scans.
- RSS and Telegram spool poll loops wait for one invocation to complete before sleeping; they never overlap.
- Telegram Bot API ingress is a bounded loopback webhook receiver behind an operator-owned HTTPS reverse proxy. It never calls `getUpdates`.
- Jetstream runs as one abortable subscription with durable rewind/replay and bounded reconnects, then restarts only after supervisor backoff.
- Captured Fastmail/JMAP files remain one-shot diagnostics. A separate manifest source owns authenticated read-only JMAP polling; it never shares credentials with consumers or action services.
- Disabled source entries never open files, sockets, or credentials.
- Service credentials are compartmentalized before activation. Webhook ingress receives only the Telegram bot token and webhook secret; the dispatcher receives only the bot token; consumers receive only configured model-provider credentials and resident identifiers; Jetstream receives no Telegram or model credentials. Service-specific systemd drop-ins clear any inherited shared `EnvironmentFile` before loading the compartment file.
- The credential splitter parses assignment names without evaluating shell syntax, copies selected raw assignments into owner-only files without logging values, rejects duplicate or malformed names, and refuses output beneath a Git worktree or configured public-content root. Installing the generated files and drop-ins is a separate explicit operator step.

## Cursor invariant

Never persist a cursor ahead of durable events. A crash may replay a source item; idempotency absorbs duplicates. Losing an item because a cursor advanced early is unacceptable.

## X Activity webhooks

X ingress uses the current X Activity API and v2 Webhooks API, not the deprecated Account Activity API. One ingress service may expose several exact loopback routes, but every route owns an independent source id, source-local sequence, privacy lane, webhook registration, and expected subscription set. Public watchlist posts never share a source id with private personal activity.

The receiver handles provider CRC `GET` requests and signature-verified event `POST` requests, then acknowledges only after durable append. It has no registration, subscription, replay, OAuth-refresh, X-action, model, or channel authority. Management commands run separately with their own credential compartment and never execute on startup. The complete event, privacy, cost, recovery, and acceptance contract is in [`x-webhook.md`](x-webhook.md).

## Fastmail / JMAP

- Discover session and account ids through the JMAP session endpoint.
- Use `Email/queryChanges` for mailbox query membership and `Email/changes` for changed email ids.
- Fetch only required properties by default: ids, thread id, mailbox ids, keywords, sender, recipients, subject, receivedAt, preview, attachment metadata.
- The live source never requests body properties. Full-body transport is not implemented.
- Cursor is the relevant JMAP state string. A `cannotCalculateChanges` response triggers a bounded resnapshot and deduplicated re-ingestion.
- No `Email/set`, submission, mailbox mutation, keyword mutation, or draft creation in ingress code.

The live `fastmail-jmap` source is one serial polling producer for one exact source id. It resolves Fastmail's primary mail account through the pinned `api.fastmail.com` session endpoint and accepts API URLs only on Fastmail's unported `api.fastmail.com`, `*.api.fastmail.com`, or `jmap.fastmail.com` HTTPS hosts. It sends only `Email/get`, `Email/query`, `Email/changes`, and `Email/queryChanges` method calls. Its environment compartment contains only the token named by the manifest. The source cannot invoke a model, send mail, create drafts, mutate keywords or mailboxes, or reach Telegram.

Provider capability advertisement is not credential-scope evidence. JMAP session and account capabilities describe server/account support and may advertise submission even when the token cannot use it. An enabled source therefore requires an explicit `credentialCustody` declaration: `dedicated-mail-ingress` means the operator provisioned a separate token for this producer and verified that it differs from the email-management credential; `shared-operator-accepted` records explicit acceptance of broader shared custody. `unprovisioned` is valid only while the source is disabled. None of these values adds a submission method to the client. Fastmail's mail scope itself includes mailbox management, so the actual read-only behavior boundary is the fixed method allowlist plus the isolated process compartment. Production should use a dedicated token minted without submission wherever Fastmail exposes that choice.

First activation uses `replay: now`: the producer reads current email and query state, appends cursor and lifecycle evidence, and emits no historical email observations. Later polls collect bounded `Email/changes` pages and one bounded `Email/queryChanges` response, fetch metadata for changed messages in bounded chunks, collapse repeated ids to one final operation, and append observations plus both new states in one transaction. A crash before settlement replays the same source state; deterministic event identity absorbs duplicates. Poll cycles never overlap.

If either JMAP changes method returns `cannotCalculateChanges`, or `Email/queryChanges` returns `tooManyChanges`, the producer performs one bounded current-mailbox resnapshot, emits at most the configured number of metadata-only `updated` observations, and advances both states in the same producer transaction. Other JMAP method errors fail the cycle without advancing either cursor. Response bytes, email-change pages, changed ids, resnapshot ids, request time, and poll interval are bounded before activation.

Continuous polling backs off exponentially after consecutive failures, capped at 15 minutes, and returns to the configured interval after a successful cycle. Query membership deltas are validated only to advance the unfiltered query state; `Email/changes` owns operations for this all-mail source. Adding a mailbox or keyword filter requires merging query additions and removals into the event operations.

The captured JMAP-response adapter remains a fixture and diagnostic path. It accepts one synthetic or local response containing `Email/queryChanges`, `Email/changes`, and `Email/get`; emits sensitive created, updated, and destroyed observations; retains only envelope, address, preview, keyword/mailbox, and attachment metadata; and excludes body values even if a capture contains them. It binds cursors to a hash of the configured account id plus the JMAP email/query state strings, absorbs exact replay, and fails closed on state gaps. It never discovers a session, authenticates, contacts Fastmail, mutates mail, or submits a message.

## ATProto Jetstream

- Subscribe to configured collections and optional DIDs.
- Collection filters may be exact NSIDs or Jetstream-supported `.*` prefixes; require at least one and enforce Jetstream's 100-collection / 10,000-DID limits.
- Cursor is Jetstream `time_us`; reconnect from the last durable cursor.
- Keep DID, collection, rkey, revision, operation, CID, and record JSON.
- Deletes remain events even when no record body exists unless an exact source contract deliberately admits create only. The current `network.cosmik.collectionLink` slice is create-only, so update/delete messages advance the Jetstream cursor without becoming source events or resident turns.
- A collection filter is required. The global firehose is not a default.
- ATProto record URI and CID remain strong source references.

The captured-batch path normalizes fixture messages and proves cursor/idempotency behavior without opening a live WebSocket. Each cursor stores a hash of its collection/DID filters and any exact-collection admission rule; changing that contract under the same source id fails closed rather than silently resuming after events the new filter would have admitted.

The repository's disabled tracked source filters exactly `app.bsky.feed.post`, `app.bsky.feed.like`, and `network.cosmik.collectionLink`. Semble cards, collections, and note-child cards are intentionally absent. The collection-link record carries strong references to the saved card and organizing collection, so one save yields one source trigger while the trusted context compiler dereferences its public context later. Only collection-link creates are admitted; updates and deletes are intentionally ignored after advancing the durable Jetstream cursor. The context compiler also refuses stale dereferences defensively if historical delete evidence is supplied directly.

The live subscriber is an explicit CLI operation, never a background default. It:

- requires at least one collection filter and an explicit source id;
- uses the same normalizer, registry, durable append, and filter-bound cursor as captured batches;
- reconnects from the last durable `time_us` minus a small replay window, then relies on event idempotency to absorb overlap;
- processes socket messages serially so cursor advancement cannot outrun durable append;
- bounds runtime and received-message count at the CLI boundary;
- closes and drains the current durable write on `SIGINT`, `SIGTERM`, runtime expiry, or message-limit completion;
- uses bounded exponential reconnect backoff with jitter and records connection/failure/recovery evidence;
- does not request Jetstream compression until the zstd dictionary path is implemented and tested.

In a deployed split-process topology, the Jetstream unit must use `--producer-only`. That mode opens the source subscription and appends durable events/cursor evidence but never loads declarations, starts consumers, or performs a post-subscription backlog pass. A separate consumer process owns every model agent. Running producer and consumer loops together remains available only for bounded local acceptance tests; it is not a valid way to feed a persistent resident agent whose main conversation is already owned by another process.

The same ownership rule applies to a deployed filesystem watcher. `thought stream watch --producer-only` scans and watches one bounded root, writes document versions and source events, and never loads declarations or starts a second consumer runtime. This is the required mode for roots that feed subscribed agent context while the dedicated consumer service owns model execution.

The replay window must affect admission as well as the WebSocket URL. Messages inside the requested overlap are offered to Jazz even when their `time_us` is at or below the prior durable cursor; otherwise the reconnect buffer would be decorative and a crash between equal-timestamp events could lose data. The stored cursor never regresses.

## Telegram

- Do not call `getUpdates`. A dedicated thought stream bot receives Bot API updates through one authenticated HTTPS webhook.
- Existing shared bots use a mirror/spool written by their owning runtime or an explicit webhook fan-out. thought stream never steals update ownership from another runtime.
- Preserve account id, chat id, message id, sender id, media metadata, edit date, reply target, and route.
- Attachments are references by default. Content extraction is a separate event.
- For an admitted photo or PNG/JPEG image document, the trusted webhook process downloads at most one selected image via `getFile` and the Bot API file endpoint, with hard timeout, redirect, normalized-POSIX-path, and size bounds (≤ 7 MiB raw; base64 is bounded consistently by the sandbox frame contract). If a malformed update contains both photo and image-document fields, the photo owns the single image slot. Actual PNG/JPEG start/end magic bytes are validated; SHA-256 is computed; and the image is atomically written as a content-addressed owner-only file beneath a real nonsymlink artifact root and real nonsymlink content-addressing parents. The event carries only stored status, opaque relative path reference, SHA-256, MIME, and byte count — never raw bytes, base64, the bot token, file URL, or absolute host path.
- Image-only messages (empty text with one stored, validated image attachment) are admitted as `stream.thought.source.telegram.message` conversation triggers. Empty messages without a stored image, including rejected images and audio/video/file-only messages, are preserved instead as `stream.thought.source.telegram.nonconversation`; the conversation declaration does not subscribe to that event type. The local reconstructed Pi transcript renders a neutral `[image]` placeholder and never replays prior image bytes. A persistent SDK conversation may retain a successfully sent multimodal turn as private remote history.
- The consumer process (model credentials, no Telegram token) resolves artifact references through the trusted Pi or Letta SDK parent, which validates each file beneath the artifact root before provider dispatch. Malformed stored references and failed resolution fail closed.
- Telegram ingress remains send-dark. Delivery belongs to the separate dispatcher capability and is backed by action receipts.
- Ingress and dispatcher may reference the same bot token through separate owner-only compartment files, but the dispatcher never receives the webhook secret and neither process receives model-provider credentials.
- The receiver binds only to a configured loopback address. Public TLS termination and routing belong to an operator-controlled reverse proxy that exposes only the exact webhook path.
- Every request must carry the configured `X-Telegram-Bot-Api-Secret-Token`. The secret enters only through an environment-variable reference, is compared in constant time, and is never logged or persisted.
- Requests are POST-only, require JSON, and have a strict body-size limit. Unauthorized, wrong-path, wrong-method, oversized, and malformed requests are rejected before Jazz access.
- The receiver serializes admitted requests even if upstream concurrency is misconfigured. Registration sets `max_connections=1` as defense in depth.
- Return `2xx` only after the accepted or intentionally ignored update, connector lifecycle evidence, and diagnostic high-water mark are durable. A durable failure returns `5xx` so Telegram retries. Replayed updates address the same deterministic event rows.
- Webhook registration and deletion are explicit operator commands. The ingress process cannot alter its own webhook registration.
- Telegram's visible bot name, native command list, and per-chat Menu button are installed only by the explicit `telegram-menu-register` operator command. The command sets and reads back the exact bot name `The Stream`, fixed `/help` and `/correct` metadata, and each enabled private chat's `commands` menu button before sending the versioned help text through the existing receipt-backed dispatcher operational-notice path. An operator may pass `--skip-help-notice` for an idempotent identity/menu repair that performs the same readbacks without sending another help message. Webhook ingress cannot mutate bot identity, commands, or menu state.
- `message_reaction` admission requires an enabled private chat and an explicit user id allowlist for that chat.
- Reaction labels require an exact delivered-message receipt with exactly one run. Unknown Telegram message ids and multi-run digest messages remain unlabeled observations.
- The mapping is deliberately narrow: `👍` is accept, `👎` is reject, and every other emoji is decorative. Changes supersede and removals retract through new events; no historical event is mutated.
- The v1 target-bound correction command is an allowlisted private message whose text begins with the exact prefix `/correct ` followed by nonempty replacement text. It must reply to a bot delivery. The adapter resolves the reply target through the same unique delivery → run → output → source-root checks as reactions and appends `stream.thought.source.telegram.correction@1`. Authorized commands with no reply or an unresolved target remain explicit inert sensitive source events. Ineligible users produce no source event. A correction command is never also a `stream.thought.source.telegram.message` conversation turn.
- For `stream.thought.output.observation@1`, deterministic correction projection canonicalizes the original structured output, preserves every non-display field, replaces only `summary` with the exact command text, and appends a quality-eligible, externally ineligible `correct` judgment. It explicitly supersedes the active `telegram-reaction@1` judgment for that run. Unsupported or invalid contracts fail closed without a judgment.
- Reaction and correction projection is deterministic connector/judgment work. Feedback source events are not model-agent triggers, invoke no model, and grant no send capability.

The captured-spool reader accepts newline-delimited JSON with one strict versioned record per completed line. Each record contains normalized account/chat/message/sender identity, source and edit timestamps, text, optional thread/reply metadata, the resolved route, and attachment metadata plus opaque references. Raw bot updates, credentials, attachment bytes, base64 data, and transcriptions are outside the spool contract.

The reader:

- opens one explicitly named regular file read-only and refuses symlinks;
- advances a byte-offset cursor only to completed newline boundaries and only after all selected message events are durable;
- bounds records and bytes per invocation, leaving a partial trailing line for a later invocation;
- binds the cursor to the source id, normalized file path hash, spool schema revision, file identity, and a hash of the complete consumed prefix;
- fails closed on truncation, replacement, or mutation behind its durable cursor;
- uses `(accountId, chatId, messageId, editedAt-or-original)` as source identity, so an edit is a separate immutable observation;
- emits sensitive source events and connector lifecycle/cursor receipts;
- inserts only new events into Jazz and never sends, replies, acknowledges Telegram, downloads media, or mutates the spool. Consumer processes observe those events through Jazz subscriptions.

The dedicated Bot API adapter accepts message, edited-message, and message-reaction webhook deliveries. Ordinary messages require a configured chat id. Reactions and `/correct ` commands additionally require a private chat, a concrete user actor, and that actor's id in the channel feedback allowlist. Command-shaped messages from ineligible users are ignored rather than passed to a conversation consumer. The adapter stores an identity-bound `highestUpdateId` for diagnostics only; it never rejects a lower update solely because a higher id was observed first. Event idempotency, not the high-water mark, absorbs retries. Its manifest contains only credential environment-variable names plus the public URL and loopback receiver settings; token and webhook-secret material remain in the runtime environment. Webhook ingress and delivery use separate commands and capabilities even when they share the dedicated bot identity.

For admitted reaction and correction feedback, the adapter resolves the target Telegram message against `stream.thought.action.telegram.send.delivered`. Classification requires one matching receipt from the exact same-chat dispatcher source/actor, one referenced run, and valid receipt-to-trigger root lineage. Correction additionally requires one exact completed canonical agent output whose parent is the run trigger and whose id is the receipt parent. Source feedback events reference the receipt, exact run, output when present, and source root. Judgment creation revalidates the same exact dispatcher/run/output chain. A deterministic projector appends `stream.thought.judgment.training-example` or `stream.thought.judgment.training-example.retracted`. If judgment projection is interrupted after feedback ingestion, Telegram retries the non-2xx delivery; the next attempt first reconciles durable feedback events and then idempotently reoffers the update.

## RSS/Atom

- Poll with `If-None-Match` and `If-Modified-Since`; persist ETag and Last-Modified with the source cursor.
- Identity preference: GUID/id, canonical URL, then content hash.
- Normalize title, canonical URL, author, published/updated time, categories, summary, and source feed.
- Keep raw item XML only when needed for parser debugging and under bounded retention.
- Backoff honors `Retry-After` and applies per-feed jitter.

## Filesystem

See `filesystem-charter.md`.

## Useful later connectors

- Git: refs, commits, changed paths, diffs, CI receipts, and pull request events.
- Calendar: read-only event changes and free/busy windows, with strict private/sensitive classification.
- Webhooks/timers: generic signed ingress and scheduled ticks for polling or reflection workflows.