cameron.stream / thought-stream
A local-first event pipeline for independent agents, built on Jazz.
Star
Fork 0
Watch
TypeScript 97%
JavaScript 3%
<1%
Shell <1%
Dockerfile <1%
77 5 0
adapters
agents
config
deploy
docker
fixtures
focuses
prompts
public
scripts
spec
src
test
work-packets
.gitignore
AGENTS.md
package.json
pnpm-lock.yaml
README.md
thoughtstream.yaml
tsconfig.json
vitest.config.ts
README.md

thought stream #

thought stream collects events from local and external sources into a shared Jazz database. Independent consumers subscribe to the events they need, run deterministic transforms or models, and write derived events back with provenance.

The project is experimental. The connectors, Jazz persistence layer, and process-owned consumer runtime are functional. Consumers recover from durable per-source progress and can follow narrow Jazz query subscriptions without a central event dispatch service.

Architecture #

  • Producers observe a source and append typed events. Each producer owns one source id, source-local sequence, and external cursor.
  • Jazz stores events, cursors, consumer progress, execution records, and rebuildable projections.
  • Consumers subscribe with a narrow query, process matching events, and append outputs and lifecycle records.
  • Compactors are specialized no-tool consumers that turn frozen canonical conversation prefixes into private recursive boundaries; ordinary agents receive the latest boundary plus exact raw tail.
  • Dispatchers independently accumulate completed candidate activity, render destination-specific batches, apply channel policy and velocity limits, perform external actions, and append delivery receipts.
  • Inspector exposes local views of events, executions, source health, lineage, and blinded Review items. It is read-only unless the separately configured OAuth-only Review capability is active; even then, its only mutation is one append-only decision route.

Producers and consumers communicate through Jazz. There is no central matcher, event dispatch queue, or lease scheduler. External egress uses separate dispatcher processes so source ingestion and consumer throughput never wait on notification policy or destination rate limits.

Requirements #

  • Node.js 22.19 or newer
  • pnpm 10.20
  • Jazz 2.0.0-alpha.55's Linux native binary requires a newer userspace than Ubuntu 22.04. The verified container baseline is node:22-trixie; upgrading Node alone does not upgrade glibc.
  • x86_64 Linux with Bubblewrap and prlimit for Pi-backed consumers (the current launcher binds the x86_64 glibc runtime explicitly)

Quick start #

git clone https://tangled.org/cameron.stream/thought-stream
cd thought-stream
pnpm install
bash scripts/provision-jazz-storage.sh "$PWD"
set -a
. .thoughtstream/state/jazz.sqlite.server.env
set +a
THOUGHTSTREAM_ROOT="$PWD" pnpm exec tsx scripts/serve-jazz-storage.ts

Leave that storage-owner process running. In a second terminal at the repository root:

set -a
. .thoughtstream/jazz-clients.env
set +a
pnpm demo

The owner and client use separate private environment files. Do not commit or print them. Ordinary services run in client mode and never acquire ownership of storage. Alpha.53 SQLite files cannot be opened with alpha.55; use a fresh root or an independently tested migration. See the port for validation scope and deployment limits.

Chat deployment requires private THOUGHTSTREAM_CO_AGENT_ID and THOUGHTSTREAM_CO_COMPUTER_DEVICE_ID settings in addition to the API credential and OAuth configuration. It never accepts these bindings from the browser. The existing encrypted registry must match the configured agent.

The demo scans fixtures/vault, writes filesystem events to a local Jazz database, runs the example document-structure consumer, and builds the activity projection.

To inspect the result:

pnpm serve

Open http://127.0.0.1:4317.

Runtime data is stored under .thoughtstream/. Set THOUGHTSTREAM_ROOT to use a different project root.

Commands #

Command Description
pnpm demo Run the fixture-backed filesystem demo.
pnpm scan -- --root <path> --source filesystem:<name> Scan a filesystem source once.
pnpm watch -- --root <path> --source filesystem:<name> Watch a filesystem source until interrupted; add --producer-only when a separate consumer service owns declarations.
pnpm thought rss --url <url> --source rss:<name> Poll an RSS or Atom feed once.
pnpm thought jetstream --source jetstream:<name> --collections <nsid,...> Run a bounded ATProto Jetstream subscription.
pnpm thought telegram-spool --file <path> --source telegram:<name> Ingest an append-only Telegram NDJSON spool.
pnpm thought telegram-webhook --source telegram:<name> Receive authenticated allowlisted Telegram webhook deliveries into Jazz. This process cannot send or register itself.
pnpm thought telegram-webhook-register --source telegram:<name> Explicitly register the configured HTTPS webhook with Telegram.
pnpm thought telegram-webhook-delete --source telegram:<name> Explicitly remove the configured Telegram webhook.
pnpm thought telegram-dispatcher --source telegram:<name> Run Telegram batching, channel selection, velocity policy, delivery, and receipt writing as a separate process.
pnpm thought x-webhook --source x:<name> Receive signed allowlisted X Activity deliveries into Jazz. This process cannot call X management APIs.
pnpm thought x-webhook-status --source x:<name> Read webhook validity and subscription drift without mutation.
pnpm thought x-webhook-register --source x:<name> --confirm-url-hash <sha256> Register or revalidate the exact configured HTTPS callback.
pnpm thought x-subscriptions-plan --source x:<name> Read and hash the desired-versus-live subscription plan.
pnpm thought x-subscriptions-apply --source x:<name> --confirm-plan-hash <sha256> Apply one unchanged source-owned subscription plan and verify convergence.
pnpm thought x-webhook-replay --source x:<name> --from <UTC-minute> --to <UTC-minute> --confirm-webhook-id <id> Request one explicit replay within X's preceding 24-hour window.
pnpm thought x-webhook-delete --source x:<name> --confirm-webhook-id <id> Delete an exact callback only after no live subscription still references it.
pnpm thought x-user-lookup --source x:<name> --username <handle> Resolve one public handle to its exact X user id without mutation.
pnpm thought incidents --config <manifest> Project content-dark operational incidents, append the private JSONL ledger, and run the independent incident-alert policy.
pnpm thought fastmail-jmap --config <manifest> --source fastmail:<name> Poll one enabled read-only Fastmail/JMAP source; first activation records current state without historical replay.
pnpm thought fastmail-capture --file <path> --source fastmail:<name> --account-id <id> Ingest a captured JMAP response.
pnpm thought consume Run enabled consumers against Jazz subscriptions until interrupted.
pnpm thought consume --once Consume the current durable backlog once, then exit.
pnpm thought status Show database and projection status.
pnpm thought events List recent events.
pnpm thought event <event-id> Show one event.
pnpm thought runs List consumer executions.
pnpm thought run <run-id> Show one execution and its trace.
pnpm thought context <run-id> Verify one run's durable context snapshot and, when declaration plus prompt-hash traces permit, reconstruct the exact Pi model input. Add both --show-content and --acknowledge-sensitive-private to print private content locally; content is withheld when reconstruction is incomplete. Hidden provider reasoning is not available.
pnpm thought agent-send --from co --to stream-agent-conversation --thread <id> --message-id <id> --file <text-file> Append one typed private Co→Stream message. Add --wait-seconds <1-600> for the exact run receipt; printing the response additionally requires both --show-response and --acknowledge-sensitive-private.
pnpm thought review-prompt --file <payload.json> --external-id <id> Append one complete versioned Review prompt. Public export preauthorization is accepted only with --privacy public-source.
pnpm thought review-item --prompt-event <id> --candidate-runs <run-a>,<run-b> Materialize one immutable blinded pair from two exact completed same-trigger candidate runs.
pnpm thought review-queue Inspect the projected Review queue and active append-only decisions.
pnpm thought focus-propose --file <focus.yaml> Validate and append one inert versioned focus-declaration proposal. This does not activate a consumer or grant authority.
pnpm thought focus-list List body-dark focus proposal identities, lineage, fingerprints, schema versions, and event receipts. Semantic ids remain sensitive metadata.
pnpm thought judgment <run-id> --kind <accept|reject|correct|prefer> Append an explicit quality judgment. External use requires --external-export-eligible; sensitive/private material also requires --authorize-sensitive-external-export.
pnpm thought proposal-list List agent proposal ids, kinds, decisions, and materialization receipts without printing proposal content.
pnpm thought proposal-decision <proposal-event-id> --disposition <accept|edit|reject> Append one human proposal decision. Edits read exact text from --replacement-file; accepted memory changes require the configured --context-root.
pnpm thought proposal-project [--context-root <path>] Reconcile accepted correction decisions and accepted memory decisions after an interrupted CLI run.
pnpm thought private-training-export --output <private-file.jsonl> --acknowledge-sensitive-private-training Export active quality-eligible judgments, including sensitive self-corrections from The Stream, through the private exact-provenance path. Output is file-only, owner-only, and rejected inside Git or public-content roots.
pnpm thought training-export --output <file.jsonl> Export only externally eligible, entirely public-source judgments into privacy-minimized JSONL and a content-addressed manifest. Sensitive/private export additionally requires both explicit private-export flags and a non-Git destination.
pnpm thought artifact-request --workspace-root <root> --workspace-lease-id <id> --file <relative-path> --request-id <id> --artifact-id <id> ... Trusted host-side seam that records a private request and promotes one exact lease file into durable content-addressed storage. Never publishes externally; no exe.dev transport is implied.
pnpm thought artifacts List the artifact catalog.
pnpm serve Start the local inspector on port 4317.
pnpm configure:inspector-oauth -- --origin <https-origin> --did <did> --handle <handle> Generate owner-only OAuth client/store configuration without changing the separately configured Basic fallback.
pnpm configure:inspector-review Generate one owner-only proxy-to-inspector Review capability. Generation does not restart or activate either service.
pnpm test Run the test suite.

The inspector is deliberately loopback-only. scripts/serve-inspector-proxy.ts serves a fixed public landing/documentation allowlist and forwards only authenticated GET/HEAD requests below /inspector to the private upstream. Public routes have no Jazz handle, manifest reader, runtime-state reader, directory listing, or arbitrary file fallback. The proxy strips credentials before forwarding.

ATProto OAuth uses the official Node client for PKCE, PAR, DPoP, nonce handling, identity resolution, and refresh. Public client metadata and JWKS live at /oauth/client-metadata.json and /oauth/jwks.json; the callback is /oauth/callback. Only OAUTH_ALLOWED_DID may receive an inspector session. SDK protocol state, browser application state, DPoP keys, access/refresh tokens, and opaque browser-session records are AES-256-GCM encrypted in an owner-only directory outside the runtime root. The two state values are intentionally distinct. Every store has hard entry/byte limits and one process owner; a second proxy cannot open the fixed ~/.local/share/thoughtstream-inspector-auth directory, and custom store paths are rejected because systemd cannot write them. Browser cookies contain random ids only. Login and callback are rate-limited at nginx and in-process, callback query strings are excluded from access logs, and www redirects to the canonical origin before application routing. The authoritative watchdog covers SDK exchange, application-state consumption, generation promotion, browser-session persistence, and cleanup. Timeout synchronously makes the attempt non-promotable before advancing the callback queue. Persistent writes recheck authority before atomic rename. Promoted sessions and browser cookies carry per-DID generations, so application cleanup can remove only its own local generation and does not request remote revocation. The unmodified SDK may still revoke after issuer or session-store failure; provider-side effects are outside the local authority guarantee. Never-settling callback quarantines are capped at eight; exhaustion returns an operator-recycle-required response until attempts settle or the process restarts.

Basic Auth is an optional independent break-glass path. It is disabled when PROXY_BASIC_FALLBACK_ENABLED is absent or false and enabled only by the exact values 1 or true with a valid credential. When enabled it authorizes inspector reads without consulting OAuth and is stripped before forwarding. When disabled the proxy does not require or decode the Basic credential and does not advertise a Basic challenge. The service refuses to start if neither authentication path exists.

Browser writes remain narrower than inspector authentication. Basic is always read-only. An allowlisted OAuth browser can append a Review decision or post-training course question only when both inspector processes load the matching generated capability. Review and course chat use separate keys. The proxy verifies the browser session and CSRF token, signs the exact method, path, and body with a fresh nonce, and forwards no cookie, Authorization header, CSRF value, OAuth token, or DID. The inspector verifies the one-time envelope and accepts only the route's fixed schema. Neither route grants generic Jazz mutation authority.

Run scripts/configure-inspector-credentials.sh to create or rotate the optional inspector-basic.env file without putting the credential in shell history. After the domain and exact DID are known, run:

pnpm configure:inspector-oauth -- --origin https://thought.stream --did did:plc:REPLACE_ME --handle cameron.stream
pnpm configure:inspector-review
pnpm configure:inspector-course-chat

The OAuth command creates owner-only client/store keys and does not enable or disable Basic. It does not install units, reload nginx, restart the proxy, or prove OAuth. Deployment templates live under deploy/systemd/ and deploy/nginx/; activate them only after DNS, TLS, the HTTP-to-HTTPS redirect, external metadata/JWKS fetches, and rollback copies are verified. The two capability commands create separate owner-only proxy/inspector keys. They don't restart either process.

The proxy unit loads inspector-basic.env as an optional credential compartment. After the external OAuth acceptance gate and one bounded rollback exercise pass, retire Basic by removing that file and restarting only the proxy. Verify that an old Basic header receives the same generic 401 as an unauthenticated request, that no WWW-Authenticate header remains, and that OAuth still admits a private read. Deployments created before the credential split must stop loading the old inspector-proxy.env; leaving it in an effective unit keeps the retired secret in the process environment.

Course activation also installs post-training-course-tutor@1 into the live source tree and restarts thoughtstream-consumers.service. The consumer compartment must already contain Tinker authority. Verify the compiled declaration selects the expected public model, then require one natural question → source event → tutor run → output → authenticated status readback before calling the chat path live. Restarting only the inspector and proxy accepts questions but leaves them pending.

Connectors #

Connector Current support
Filesystem One-shot scans and read-only watching with add, change, rename, and delete detection.
RSS/Atom Explicit one-shot polling with ETag and Last-Modified cursor support.
ATProto Jetstream Explicit bounded live subscriptions with collection filters, rewind, and reconnect handling.
Telegram Sensitive authenticated Bot API webhook ingress and receipt-bound reaction judgments with deterministic replay handling, plus a separate config-driven outbound dispatcher with batching, rate limits, and receipts. Append-only spool ingestion remains available.
X Activity API Signature-verified public post.create and post.delete plus direction-bound sensitive outbound like.create normalization. Registration, subscription CRUD, replay, and deletion are explicit management commands. The private-like source remains disabled until its separate user-context grant and refresh custody are provisioned.
Fastmail Read-only authenticated JMAP polling with current-state initialization, bounded changes pagination/resnapshot, metadata-only events, and a separate captured-response fixture path.

Network connectors run only when invoked explicitly. Ingress processes cannot register themselves or perform outbound management actions. Telegram sends exist only in the separately invoked dispatcher and only for enabled channels in the active manifest.

Service credential compartments #

Production services must not share one all-secrets environment file. pnpm split:service-credentials reads one explicitly named private assignment file, selects raw assignments by variable name without evaluating or printing values, and writes owner-only service files for ingress, management, consumers, Telegram dispatch, Fastmail, and Jetstream. Fastmail receives only FASTMAIL_API_KEY; Telegram ingress receives the bot token and webhook secret; X ingress receives only the consumer secret used for CRC and delivery HMACs; public X management and Cameron user-context management have separate token compartments; consumers receive only selected provider credentials and agent identifiers. The systemd drop-in templates live under deploy/systemd/credential-compartments/.

The splitter refuses Git worktrees and configured public-content roots. Generating files does not install drop-ins, reload systemd, restart a service, or prove that a process loaded the new compartment.

Telegram process split #

The Telegram source and its outbound dispatcher share channel configuration but not authority. Keep real destination ids in the ignored thoughtstream.local.yaml; the tracked manifest is an inert example. A minimal source looks like this:

sources:
  - id: telegram:personal
    kind: telegram-webhook
    enabled: true
    tokenEnv: THOUGHTSTREAM_TELEGRAM_BOT_TOKEN
    webhookSecretEnv: THOUGHTSTREAM_TELEGRAM_WEBHOOK_SECRET
    webhookUrl: https://thoughtstream.example/webhooks/telegram
    webhookPath: /webhooks/telegram
    listenHost: 127.0.0.1
    listenPort: 4318
    maxBodyBytes: 1048576
    dispatchIntervalMs: 1000
    channels:
      - id: "123456789"
        enabled: true
        bootMessage:
          enabled: true
          text: thought stream is live.
        reactionFeedback:
          enabled: true
          allowedUserIds: ["123456789"]
        notifications:
          enabled: true
          includeNormal: true
          runStatuses: [completed, failed]
          allowedSources: [jetstream:personal]
          allowedActors: [did:plc:example]
          maxMessagesPerWindow: 3
          windowMs: 60000
          likeDigestDelayMs: 60000
          maxLikesPerDigest: 10

Register the webhook once, then run ingress and egress as separate processes against the same local manifest:

pnpm thought telegram-webhook-register --config thoughtstream.local.yaml --source telegram:personal
pnpm thought telegram-webhook --config thoughtstream.local.yaml --source telegram:personal
pnpm thought telegram-dispatcher --config thoughtstream.local.yaml --source telegram:personal

telegram-webhook binds only to the configured loopback address. An operator-owned HTTPS reverse proxy must expose only webhookPath. Every delivery must carry Telegram's configured secret header; the receiver returns success only after Jazz durability and serializes admitted requests even though registration already uses one upstream connection. It accepts messages from enabled channel ids and reactions only from the user ids named under reactionFeedback. A reaction is eligible for judgment only when its private-chat message id resolves to one delivered dispatcher receipt containing exactly one run. 👍 appends a quality-eligible accept judgment and 👎 appends a quality-eligible reject judgment. Both are externally ineligible: reacting to a private reply is not declassification. Other emoji remain reaction observations without labels. Changes append a superseding judgment; removal appends a retraction. These deterministic projections never enter the model consumer path.

The bot menu exposes /focus. Bare /focus returns deterministic usage without inference. /focus <description> lets the Stream submit one complete private inert focus declaration through the fixed proposal tool; it does not activate a listener, expand source access, start training, promote an adapter, or deliver focus output. See spec/focuses.md.

telegram-dispatcher watches the configured consumer run statuses in Jazz, filters them against each destination's source and actor allowlists, accumulates eligible likes into digest batches, and applies the channel's velocity limit. When failed is enabled, failure notifications contain only classified diagnostics and redacted content counts. Raw prompts, model output, provider thinking, tool arguments, and provider bodies never cross the Telegram boundary. Agent processing remains unthrottled; policy is applied at the last responsible boundary.

incidents is a separate process and policy. It projects connector, scheduler, run, and delivery evidence into stream.thought.runtime.incident@1, appends only that normalized contract to the configured private ledger, and optionally alerts selected categories without consulting normal-output source allowlists. Telegram delivery failures are ledger-only to prevent recursive alert attempts. See spec/incidents.md.

On first activation, a dispatcher writes a durable activation event and ignores older history. On later process starts it resumes unclaimed candidate activity from that activation point. Boot messages are process-start notices, controlled per channel, and use the same receipt and rate-limit path as ordinary sends.

X Activity ingress #

An x-webhook source owns one callback path, one privacy lane, and exact event/user/direction/tag subscriptions. The receiver binds only to loopback, answers CRC with the app consumer secret, verifies x-twitter-webhooks-signature over the raw POST body, and acknowledges only after durable settlement. It serializes writes through a bounded queue, returning retryable failure instead of accumulating unbounded work. It canonicalizes stable post or outbound-like identity while dropping expanded profiles and mutable public metrics. Signed activity outside the source allowlist is durably counted and acknowledged without becoming a source event.

Tracked account lists live in config/x-cameron-public.yaml and config/x-public-watch.yaml. Each account has a readable handle and the immutable numeric user id used by X subscription filters. To add an account, run the read-only x-user-lookup command, verify the returned name and handle, add that pair to the appropriate file, then run x-subscriptions-plan. Editing the file alone neither contacts X nor changes a live subscription. Startup expands every account across the file's exact event types and fails closed on duplicates, path escape, symlinks, source mismatch, or unresolved ids.

Public management uses a separate app-bearer environment. Sensitive outbound likes require the separate x-user-management compartment and user-context OAuth custody; the tracked source and batch stay disabled until that grant and its refresh procedure exist. Status and plan are read-only. Register, apply, replay, and delete require hashes or ids copied from current readback, and none runs during receiver startup.

The existing resident Stream agent receives one mixed-privacy activity window at most every ten minutes. The deterministic batch includes complete per-source prefixes from the configured filesystem, ATProto, non-conversational Telegram activity, email, agent-message, and X inputs, then promotes the batch to its most-private member. Direct Stream Telegram messages are excluded because the same persistent conversation already receives them as direct turns; duplicating a partial copy inside the activity window produces no new evidence. Direct replies remain prose while activity windows require strict JSON. A high-importance notification tuple is only a proposal; the separately configured Telegram dispatcher must still match the exact agent/source route, claim the run, deliver the message, and append the receipt. See spec/agents.md.

Three additional declarations are tracked but inert. cameron-bluesky-listener consumes the existing exact-member ATProto batch, cameron-x-listener consumes one separate deterministic X window, and cameron-social-listener consumes only completed observations from those two source-local listeners. Before social context construction, every observation must join to one exact completed run and matching output lineage. The source listeners use separate persistent output-only Luna conversations and have no Telegram proposal route. The Terra social listener may emit the exact inert notification tuple, but only the independent dispatcher can send it.

Activation requires the corresponding THOUGHTSTREAM_LETTA_*_LISTENER_AGENT_ID plus THOUGHTSTREAM_LETTA_ENABLE_*_LISTENER=1 values in the consumer compartment and explicit enablement of the X and social batch declarations in a private manifest. Enable the source-local paths first and require natural completed-output receipts before enabling social synthesis. These switches do not provision agents, grant X user-context OAuth, mutate X subscriptions, restart services, or prove a live model turn.

Provision each identity separately. The default command is read-only and prints a declaration/prompt-bound confirmation. --apply requires that exact confirmation and LETTA_API_KEY, creates at most one hidden no-tool agent, and writes an owner-only resumable receipt beneath the private runtime root. It does not edit the credential compartment, enable a declaration, restart a service, or send a model turn:

pnpm provision:letta-listener -- --listener bluesky
pnpm provision:letta-listener -- --listener bluesky --apply --confirm <current-confirmation>

Valid listener names are bluesky, x, and social. Re-run the read-only plan after any declaration or prompt change; stale confirmations and drifted local provisioning state fail closed.

Consumer declarations #

Product-level focus declarations live separately under focuses/. They describe a governed policy cell's scope, objective, subscriptions, budgets, requested permissions, DAG lineage, and retirement rule. focus-propose writes an inert sensitive proposal event; it does not compile or enable an agent. focuses/news.yaml and the disabled agents/news-focus.example.yaml show the first proposed broad-news focus and its separate Tinker runtime companion. Full contract: spec/focuses.md.

Consumers are declared in agents/*.yaml. A declaration defines:

  • the event types and sources to subscribe to;
  • context limits and prompt revision;
  • the deterministic or model-backed runner;
  • allowed output event types;
  • tool and external-action policy.

See agents/document-structure.yaml for a working example.

Pi declarations select a trusted provider profile rather than supplying endpoints or credential names. tinker-default uses the fixed Tinker endpoint and TINKER_API_KEY; extra concrete models require THOUGHTSTREAM_TINKER_ALLOWED_MODELS or a trusted tier mapping. openai-compatible-default requires trusted-host THOUGHTSTREAM_MODEL_BASE_URL, THOUGHTSTREAM_MODEL_ALLOWED_MODELS, and THOUGHTSTREAM_MODEL_API_KEY configuration. Model execution runs in a disposable Bubblewrap sandbox with no inherited environment, no network namespace, no host filesystem view, and no tools. A single-use Unix-socket broker in the trusted parent validates the destination, model, headers, request size, token limit, and deadline before adding the provider credential. Sandbox or broker failure never falls back to in-process inference.

Pi declarations may opt into the read-only atproto.fetch-markdown and web.download-image tools. The trusted parent executes those bounded reads before inference and passes only their evidence into the sandbox. The image tool accepts only URLs discovered in the current source record or fetched Markdown, rejects non-public network destinations, bounds response size, and stores content-addressed artifacts under .thoughtstream/artifacts/. Durable tool outcomes contain status, field names, counts, and hashes rather than arguments, source bodies, image bytes, or arbitrary errors. agents/bluesky-enrichment-observer.yaml is disabled by default because it requires a configured Tinker credential and deliberate source/actor policy.

Output contracts are versioned registry entries rather than one universal observation shape. agents/conceptualizer.yaml defines the active OpenAI-backed consumer using stream.thought.output.conceptualization@1. A successful run settles one bounded private stream.thought.derived.concept.graph event with source/run lineage. It does not publish ATProto records or mutate an external graph.

agents/coil-public-knowledge.yaml is the disabled stateful Coil proposed-diff consumer. One explicit producer watches the Coil as filesystem:coil; the trusted consumer resolves each immutable Jazz document version, applies the default-deny Public Knowledge policy, and skips blocked/deferred/deleted inputs before opening an SDK session. Each eligible stable documentId retains one conversation on Co's configured existing agent across renames and declaration upgrades. The local SDK profile is proposal-only: API-backed App Server, strict permission mode, no skills, memory-root filesystem confinement, and exactly one controller-owned submit_public_knowledge_diff tool. That callback validates and captures one complete proposal in memory but performs no filesystem, Jazz, Git, PDS, site, channel, deployment, or publication effect. The model's final acknowledgment is ignored as semantic output; zero, duplicate, malformed, context-invalid, unknown, or failed tool activity fails closed. The parent admits only a bounded set of exact current public target bodies and hashes. Its only accepted effect after trusted runtime settlement is one sensitive stream.thought.agent.public-knowledge-diff.proposed event with publication authority fixed false; no current consumer materializes or publishes it. Full contract: spec/public-knowledge.md.

Activation is environment-bound rather than a tracked enabled: true. The consumer process requires THOUGHTSTREAM_ENABLE_COIL_PUBLIC_KNOWLEDGE=1, THOUGHTSTREAM_LETTA_CO_AGENT_ID, THOUGHTSTREAM_LETTA_CO_MEMORY_DIR, THOUGHTSTREAM_PUBLIC_KNOWLEDGE_POLICY_PATH, and THOUGHTSTREAM_PUBLIC_KNOWLEDGE_CATALOG_ROOT. The filesystem producer remains a separate process:

pnpm thought watch --producer-only --root <coil-root> --source filesystem:coil

deploy/systemd/thoughtstream-coil-filesystem.service is the persistent producer template. It grants read-only home access, write access only to the private thought stream root, loads no credential file, and excludes Obsidian's private .obsidian/ application state.

The declaration serializes one agent operation at a time and uses durable 30-second, hourly, and daily accounting windows. Version 3 starts at the observed source head rather than replaying the existing Coil. Run a bounded private canary before activating the producer.

Observed Agent SDK 0.5.7 behavior: exact summarySearch plus conversation management is sufficient to recover a remote conversation created before Jazz binding evidence settles, and local API-backed sessions successfully enforce the memory-root/no-tools profile. The local management transport keeps its SDK-owned App Server pooled and exposes no top-level client shutdown method, so one-shot probes do not naturally terminate after their result. Long-running production consumers therefore own the App Server in one systemd control group; bounded canaries require an external process-group supervisor. This is the same lifecycle gap tracked in letta-agent-sdk#245.

agents/review-candidate-a.example.yaml and agents/review-candidate-b.example.yaml show the disabled two-candidate path. Both consume the same stream.thought.source.review.prompt, emit strict stream.thought.output.review-response@1 outputs, and have no tools or external actions. review-item freezes the exact completed pair before browser grading. spec/review.md owns judgeability, correction, blinding, supersession, browser authority, and export semantics.

A local public-safe canary starts from the reviewed payload fixture:

pnpm thought review-prompt \
  --file fixtures/review/prompt.example.json \
  --external-id response-quality-canary-001 \
  --privacy public-source

Enable two concrete declarations only after replacing their model or immutable adapter selections, run the ordinary consumer process, then materialize the two completed run ids:

pnpm thought review-item --prompt-event <event-id> --candidate-runs <run-a>,<run-b>
pnpm thought review-queue

Import and model execution remain CLI/runtime operations. The browser can append decisions but cannot create campaigns or trigger inference.

Learned models use an immutable startup catalog: release manifests under adapters/releases/ plus one adapters/deployment.yaml selecting an exact active release for each declaration. Startup resolves checkpoint environment references into process-local private bindings and persists only public release, binding digest, catalog digest, and generation. Activation or retirement requires stopping all adapter-egress services, atomically installing a new catalog generation, restarting, verifying PID/start identity and loaded digest, and running a canary. There is no hot Jazz lifecycle or file-lock authority. The tracked Julia files are inert examples and contain no real checkpoint.

Agent outputs do not become training data merely because a run completed. judgment writes an explicit accept, reject, correction, or pairwise preference event with separate quality and external-export eligibility. Receipt-bound 👍 and 👎 reactions provide quality evidence only; their reaction, delivery, run, output, source root, supersession, and retraction lineage remain private and durable.

Default training-export includes only active, externally eligible judgments whose complete source/output chain is public-source. Legacy judgments remain thoughtstream.training-example.v3. Reviewed preferences and corrections use v4, adding the exact preauthorized public prompt/evidence, bounded criterion metadata, one chosen response, one or two rejected candidates, campaign identity, and exact candidate model/adapter/catalog provenance. The dataset manifest is v4 and records mixed example-format counts and Review campaigns. Adapter privacy/export policy is checked independently on every participating run. Export omits notes, browser submission ids, source actor/route/external/correlation/idempotency identifiers, event/run/delivery ids, source and trace-content hashes, trace timestamps, arbitrary context fields, and private checkpoints. Sensitive/private export requires explicit authority at judgment creation, both --include-sensitive-private and --authorize-sensitive-private-export, a file destination outside every Git worktree and configured public-content root, and owner-only atomic dataset/manifest files. The browser cannot declassify private Review material.

The Stream's native request_memory_change and submit_correction tools create inert, sensitive, snapshot-bound proposals inside the same atomic settlement as the conversational output. Only a local human decision can turn a correction into an agent-self-correction@1 judgment or a memory proposal into a stale-checked owner-only memory.md write with an explicit filesystem receipt. Agent proposals never enter training directly. private-training-export is the separate exact-provenance path for quality-eligible private judgments and never writes examples to stdout. See spec/proposals.md.

Eligible terminal output-validation failures append one deterministic repair request. The separately declared output-repair Pi consumer regenerates the original bounded context, runs through the same Bubblewrap/broker boundary, and may append one contract-valid correction proposal. A proposal is inert until an accept or correct judgment names its repair run; rejection, supersession, or retraction is preserved append-only. See spec/repairs.md for eligibility, privacy, authority, and training rules.

Development #

pnpm test

The test suite uses temporary Jazz databases and local fixtures. It does not contact Telegram, Fastmail, or the public Jetstream service.

The binding design documents are under spec/. The core producer/consumer topology is implemented; the capability-gate sections also identify harder durability, authorization, and multi-process proofs that remain.