diff --git a/.gitignore b/.gitignore index 5886d80..fa029cc 100644 --- a/.gitignore +++ b/.gitignore @@ -4,5 +4,7 @@ dist/ .env .env.* !.env.example +.letta/ +thoughtstream.local.yaml coverage/ *.log diff --git a/README.md b/README.md index 6adf4b6..e853764 100644 --- a/README.md +++ b/README.md @@ -2,16 +2,17 @@ thought stream collects events from local and external sources into a shared Jazz database. Independent consumers subscribe to the events they need, run deterministic transforms or models, and write derived events back with provenance. -The project is experimental. The connectors and local persistence layer are functional; the consumer runtime is being migrated from the original in-process scheduler to the process-owned Jazz subscription model described in [`spec/`](spec/README.md). +The project is experimental. The connectors, Jazz persistence layer, and process-owned consumer runtime are functional. Consumers recover from durable per-source progress and can follow narrow Jazz query subscriptions without a central event dispatch service. ## Architecture - **Producers** observe a source and append typed events. Each producer owns one source id, source-local sequence, and external cursor. - **Jazz** stores events, cursors, consumer progress, execution records, and rebuildable projections. - **Consumers** subscribe with a narrow query, process matching events, and append outputs and lifecycle records. +- **Dispatchers** independently accumulate completed candidate activity, render destination-specific batches, apply channel policy and velocity limits, perform external actions, and append delivery receipts. - **Inspector** exposes local read-only views of events, executions, source health, and lineage. -Producers and consumers communicate through Jazz. The target architecture has no central matcher, dispatch queue, or lease scheduler. +Producers and consumers communicate through Jazz. There is no central matcher, event dispatch queue, or lease scheduler. External egress uses separate dispatcher processes so source ingestion and consumer throughput never wait on notification policy or destination rate limits. ## Requirements @@ -49,12 +50,18 @@ Runtime data is stored under `.thoughtstream/`. Set `THOUGHTSTREAM_ROOT` to use | `pnpm thought rss --url --source rss:` | Poll an RSS or Atom feed once. | | `pnpm thought jetstream --source jetstream: --collections ` | Run a bounded ATProto Jetstream subscription. | | `pnpm thought telegram-spool --file --source telegram:` | Ingest an append-only Telegram NDJSON spool. | +| `pnpm thought telegram-bot --source telegram:` | Long-poll an allowlisted Telegram Bot API source into Jazz. This process cannot send. | +| `pnpm thought telegram-dispatcher --source telegram:` | Run Telegram batching, channel selection, velocity policy, delivery, and receipt writing as a separate process. | | `pnpm thought fastmail-capture --file --source fastmail: --account-id ` | Ingest a captured JMAP response. | +| `pnpm thought consume` | Run enabled consumers against Jazz subscriptions until interrupted. | +| `pnpm thought consume --once` | Consume the current durable backlog once, then exit. | | `pnpm thought status` | Show database and projection status. | | `pnpm thought events` | List recent events. | | `pnpm thought event ` | Show one event. | -| `pnpm thought runs` | List consumer executions in the current prototype. | +| `pnpm thought runs` | List consumer executions. | | `pnpm thought run ` | Show one execution and its trace. | +| `pnpm thought judgment --kind ` | Append an explicit output judgment. Training eligibility requires `--export-eligible`. | +| `pnpm thought training-export --output ` | Project explicitly eligible judgments into provenance-bearing JSONL and a content-addressed dataset manifest. | | `pnpm serve` | Start the local inspector on port 4317. | | `pnpm test` | Run the test suite. | @@ -65,10 +72,52 @@ Runtime data is stored under `.thoughtstream/`. Set `THOUGHTSTREAM_ROOT` to use | Filesystem | One-shot scans and read-only watching with add, change, rename, and delete detection. | | RSS/Atom | Explicit one-shot polling with ETag and Last-Modified cursor support. | | ATProto Jetstream | Explicit bounded live subscriptions with collection filters, rewind, and reconnect handling. | -| Telegram | Strict append-only spool ingestion. The Telegram runtime does not write the spool yet. | +| Telegram | Sensitive allowlisted Bot API ingress with a durable update cursor, plus a separate config-driven outbound dispatcher with batching, rate limits, and receipts. Append-only spool ingestion remains available. | | Fastmail | Captured JMAP response ingestion using synthetic fixtures. Authenticated transport is not implemented. | -Network connectors run only when invoked explicitly. Ingress is read-only: the project does not send messages, publish posts, or mutate source systems. +Network connectors run only when invoked explicitly. Ingress processes are read-only. Telegram sends exist only in the separately invoked dispatcher and only for enabled channels in the active manifest. + +## Telegram process split + +The Telegram source and its outbound dispatcher share channel configuration but not authority. Keep real destination ids in the ignored `thoughtstream.local.yaml`; the tracked manifest is an inert example. A minimal source looks like this: + +```yaml +sources: + - id: telegram:personal + kind: telegram-bot + enabled: true + tokenEnv: THOUGHTSTREAM_TELEGRAM_BOT_TOKEN + pollTimeoutSeconds: 5 + dispatchIntervalMs: 1000 + channels: + - id: "123456789" + enabled: true + bootMessage: + enabled: true + text: ThoughtStream is live. + notifications: + enabled: true + includeNormal: true + runStatuses: [completed, failed] + allowedSources: [jetstream:personal] + allowedActors: [did:plc:example] + maxMessagesPerWindow: 3 + windowMs: 60000 + likeDigestDelayMs: 60000 + maxLikesPerDigest: 10 + failureTranscriptMaxChars: 3000 +``` + +Run both processes against the same local manifest: + +```sh +pnpm thought telegram-bot --config thoughtstream.local.yaml --source telegram:personal +pnpm thought telegram-dispatcher --config thoughtstream.local.yaml --source telegram:personal +``` + +`telegram-bot` only accepts messages from enabled channel ids and advances its durable Bot API cursor. `telegram-dispatcher` watches the configured consumer run statuses in Jazz, filters them against each destination's source and actor allowlists, accumulates eligible likes into digest batches, and applies the channel's velocity limit. Failed runs retain any attempted model transcript and can dispatch that raw transcript with the error when `failed` is enabled. Agent processing remains unthrottled; policy is applied at the last responsible boundary. + +On first activation, a dispatcher writes a durable activation event and ignores older history. On later process starts it resumes unclaimed candidate activity from that activation point. Boot messages are process-start notices, controlled per channel, and use the same receipt and rate-limit path as ordinary sends. ## Consumer declarations @@ -82,6 +131,10 @@ Consumers are declared in `agents/*.yaml`. A declaration defines: See [`agents/document-structure.yaml`](agents/document-structure.yaml) for a working example. +Pi declarations may opt into the read-only `atproto.fetch-markdown` and `web.download-image` tools. The image tool accepts only URLs discovered in the current source record or fetched Markdown, rejects non-public network destinations, bounds response size, and stores content-addressed artifacts under `.thoughtstream/artifacts/`. Tool calls and outcomes are written into the existing run trace. For Tinker runners, ThoughtStream executes these bounded reads before inference and supplies their evidence to the model instead of depending on native tool invocation. [`agents/bluesky-enrichment-observer.yaml`](agents/bluesky-enrichment-observer.yaml) is disabled by default because it requires a configured Tinker credential and deliberate source/actor policy. + +Agent outputs do not become training data merely because a run completed. `judgment` writes an explicit accept, reject, correction, or pairwise preference event. `training-export` includes only judgments marked `--export-eligible`, with the source event, full run trace, chosen and/or rejected output, concrete model and adapter provenance, prompt hash, and judgment criterion. It writes a sibling `.manifest.json` whose stable dataset id is the JSONL SHA-256. + ## Development ```sh @@ -90,4 +143,4 @@ pnpm test The test suite uses temporary Jazz databases and local fixtures. It does not contact Telegram, Fastmail, or the public Jetstream service. -The binding design documents are under [`spec/`](spec/README.md). They describe the target producer/consumer topology and take precedence over older scheduler terminology that remains in the prototype implementation. +The binding design documents are under [`spec/`](spec/README.md). The core producer/consumer topology is implemented; the capability-gate sections also identify harder durability, authorization, and multi-process proofs that remain. diff --git a/agents/bluesky-activity-observer.yaml b/agents/bluesky-activity-observer.yaml new file mode 100644 index 0000000..aa04b20 --- /dev/null +++ b/agents/bluesky-activity-observer.yaml @@ -0,0 +1,25 @@ +id: bluesky-activity-observer +version: 1 +name: Bluesky activity observer +description: Turn scoped public Bluesky post and like commits into legible, inspectable activity. +enabled: true +subscribe: + types: + - stream.thought.source.atproto.commit + sources: + - jetstream:* + privacy: + - public-source +context: + maxEvents: 1 + maxChars: 24000 +runner: + kind: deterministic + maxOutputTokens: 1000 + timeoutMs: 30000 +prompt: prompts/bluesky-activity-observer.md +emit: + - stream.thought.derived.topics +policy: + tools: [] + externalActions: false diff --git a/agents/bluesky-enrichment-observer.yaml b/agents/bluesky-enrichment-observer.yaml new file mode 100644 index 0000000..0ac2a5b --- /dev/null +++ b/agents/bluesky-enrichment-observer.yaml @@ -0,0 +1,29 @@ +id: bluesky-enrichment-observer +version: 1 +name: Bluesky enrichment observer +description: Inspect canonical public ATProto records and referenced media before producing a bounded activity interpretation. +enabled: false +subscribe: + types: + - stream.thought.source.atproto.commit + sources: + - jetstream:* + privacy: + - public-source +context: + maxEvents: 1 + maxChars: 32000 +runner: + kind: pi + provider: tinker + tier: triage-small + maxOutputTokens: 2000 + timeoutMs: 60000 +prompt: prompts/bluesky-enrichment-observer.md +emit: + - stream.thought.derived.topics +policy: + tools: + - atproto.fetch-markdown + - web.download-image + externalActions: false diff --git a/package.json b/package.json index f21c57b..9cd0be1 100644 --- a/package.json +++ b/package.json @@ -18,7 +18,7 @@ "events": "tsx src/cli.ts events", "run": "tsx src/cli.ts run", "serve": "tsx src/cli.ts serve", - "demo": "tsx src/cli.ts scan --root fixtures/vault --source filesystem:fixture" + "demo": "tsx src/cli.ts demo --root fixtures/vault --source filesystem:fixture" }, "dependencies": { "@earendil-works/pi-agent-core": "^0.80.6", @@ -29,6 +29,7 @@ "fast-xml-parser": "^5.10.0", "jazz-napi": "2.0.0-alpha.53", "jazz-tools": "2.0.0-alpha.53", + "typebox": "1.1.38", "yaml": "^2.8.1", "zod": "^4.1.12" }, diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index e3abdcd..a972102 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -32,6 +32,9 @@ importers: jazz-tools: specifier: 2.0.0-alpha.53 version: 2.0.0-alpha.53 + typebox: + specifier: 1.1.38 + version: 1.1.38 yaml: specifier: ^2.8.1 version: 2.9.0 diff --git a/prompts/bluesky-activity-observer.md b/prompts/bluesky-activity-observer.md new file mode 100644 index 0000000..ed86b49 --- /dev/null +++ b/prompts/bluesky-activity-observer.md @@ -0,0 +1,3 @@ +# Bluesky activity observer + +Describe the shape of one scoped public Bluesky post or like commit. Preserve its collection, operation, URI or liked subject, and obvious structural tags. This deterministic rule does not infer intent, evaluate the author, or take external action. diff --git a/prompts/bluesky-enrichment-observer.md b/prompts/bluesky-enrichment-observer.md new file mode 100644 index 0000000..2b27b97 --- /dev/null +++ b/prompts/bluesky-enrichment-observer.md @@ -0,0 +1,9 @@ +# Bluesky enrichment observer + +Inspect one public ATProto commit as source evidence. + +Fetch the canonical record Markdown. For likes or repost-like records, inspect both the activity record and the referenced subject when available. Download every relevant image exposed by the source record or fetched Markdown before interpreting it. + +Do not narrate what you intend to fetch. Either use the available read-only tools immediately or, when ThoughtStream has supplied pre-fetched evidence, inspect that evidence directly. Do not emit planning prose or thinking tags. + +Summarize what the record contains and why it may be relevant to the observed activity stream. Distinguish direct evidence from inference. Preserve canonical AT URIs when they help provenance. Do not infer private intent, contact anyone, publish anything, or take external action. diff --git a/spec/README.md b/spec/README.md index a14cb37..2cb461b 100644 --- a/spec/README.md +++ b/spec/README.md @@ -1,6 +1,8 @@ # thought stream specification -This directory is binding for the next implementation revision. The existing prototype still contains the superseded central matcher/scheduler/lease path; implementation work must migrate to the producer/consumer process topology here rather than preserving that path as an abstraction. +This directory is the binding design contract. The implementation now uses the producer/consumer process topology: source-local event sequences, narrow Jazz queries and subscriptions, durable per-consumer progress, and deterministic execution evidence. It does not contain the superseded central matcher/scheduler/lease path. + +The core local milestone is implemented and exercised in `test/agent-runtime.test.ts` and `test/acceptance.test.ts`. The broader capability gates in `testing.md`, especially edge-durability, permission, fault-injection, and independent-process proofs, remain requirements rather than implied accomplishments. ## Documents @@ -21,6 +23,6 @@ This directory is binding for the next implementation revision. The existing pro The milestone is complete only when an ordinary file edit produces this verified chain in a real local Jazz database: -`producer observes source → event and source cursor settle in Jazz → consumer subscription receives event → execution starts → trace captured → typed result and consumer progress settle in Jazz → root activity projection updates` +`producer observes source → event and source sequence settle in Jazz (with an external cursor where the source has one) → consumer subscription receives event → execution starts → trace captured → typed result and consumer progress settle in Jazz → root activity projection updates` The first runner may use a deterministic test model. The production boundary must already be Pi-shaped and Tinker-configurable so replacing the model does not alter event semantics. diff --git a/spec/agents.md b/spec/agents.md index 7d92824..aea10a6 100644 --- a/spec/agents.md +++ b/spec/agents.md @@ -21,13 +21,10 @@ context: maxEvents: 8 maxChars: 24000 runner: - kind: pi - provider: deterministic - model: structure-fixture-v1 + kind: deterministic prompt: prompts/document-structure.md emit: - stream.thought.derived.document.structure - - stream.thought.derived.charter.reevaluation policy: tools: [] externalActions: false @@ -70,6 +67,8 @@ The packet records omitted/truncated content. Silent truncation is forbidden. The embedded Pi agent core supplies the model/tool loop and event stream. The consumer runtime owns context selection, persistence, output validation, authorization, bounded concurrency, and retry policy. Pi is not allowed to become an alternate event store. +Tools are an explicit declaration capability. The initial read-only set can dereference the current ATProto record through `atproto.md` and download only image URLs discovered in that record or its fetched Markdown. The runtime validates public destinations, follows a bounded redirect chain, caps response bytes, persists content-addressed image artifacts, and records each success or failure in the run trace. A declaration without a tool name cannot call it. + The runner subscribes to Pi events and maps them to trace chunks. Raw chain-of-thought is not required and may not be available; preserve provider-visible thinking blocks only when policy allows, otherwise store counts and a redacted marker. ## Outputs @@ -82,6 +81,8 @@ An agent returns a JSON object with: Every output validates against the registry and the agent's `emit` allowlist. Invalid output produces failed execution evidence and no derived event. +Completed output is not implicit approval. `stream.thought.judgment.training-example` records explicit acceptance, rejection, correction, or pairwise preference with a criterion version and a separate export-eligibility bit. The rebuildable training projection joins eligible judgments back to the source event, run output, model/checkpoint metadata, prompt hash, and context manifest. + ## Lifecycle Model-backed consumer attempts use: diff --git a/spec/architecture.md b/spec/architecture.md index a514f83..eece490 100644 --- a/spec/architecture.md +++ b/spec/architecture.md @@ -12,7 +12,7 @@ Initial adapters: - `rss`: RSS/Atom polling with HTTP validators. - `jetstream`: ATProto Jetstream commits with microsecond cursors. - `fastmail`: JMAP Email/queryChanges and Email/changes. -- `telegram`: mirrored updates or a read-only spool supplied by the existing channel runtime. +- `telegram`: allowlisted Bot API polling or a read-only spool supplied by another runtime. ### 2. Event registry @@ -20,7 +20,7 @@ The registry maps an NSID-style event type and schema version to validation, pri ### 3. Jazz database -Jazz 2 is the shared persistence, query, subscription, durability, synchronization, and permission system. Historical tables hold events, document versions, and trace chunks; operational tables hold sources/cursors, consumer declarations/progress, execution state, and projections. Historical mutation is denied by Jazz policy rather than merely omitted from a TypeScript interface. +Jazz 2 is the shared persistence, query, subscription, durability, synchronization, and permission system. Historical tables hold events, document versions, and trace chunks; operational tables hold sources/cursors, consumer declarations/progress, execution state, and projections. A deployable Jazz policy must deny historical mutation rather than merely omitting mutators from a TypeScript interface. The current local-development policy remains permissive and is not deployment evidence. Domain modules validate event meaning, derive deterministic row identity, and construct transactions, but they remain thin Jazz-facing boundaries. There is no generic backend-portability layer and no application-owned query engine. @@ -36,7 +36,11 @@ For model-backed consumers, the runner builds a bounded context packet, invokes Projectors are consumers with no special delivery path. They follow Jazz subscriptions, recover from per-source progress, and maintain rebuildable Jazz views: root activity, topic index, unresolved recommendations, source health, consumer health, document identity, and trace summaries. Jazz performs filtering, ordering, and bounded pagination. -### 7. Local interface +### 7. Egress dispatchers + +External actions are owned by separate destination-specific dispatcher processes. A dispatcher reads completed candidate activity from Jazz, filters it against channel policy, accumulates and renders batches, applies destination velocity limits, performs the action, and appends started/delivered/failed evidence. Producers and consumers never wait on dispatcher policy or destination throughput. The first implementation supports Telegram delivery only. + +### 8. Local interface The first interface is a local server and dense activity page. It reads projections and can open the full event/execution lineage. It is not runtime authority. @@ -53,6 +57,7 @@ The first interface is a local server and dense activity page. It reads projecti 9. Output validator accepts or rejects each declared derived event. 10. A Jazz transaction writes derived/lifecycle events, terminal execution evidence, and the consumer's per-source progress together. 11. Projector consumers update the root activity view through the same query/subscription mechanism. +12. An independently running egress dispatcher may select eligible completed activity, render a destination batch, claim it durably, perform the external action, and append delivery evidence. ## Authority layers @@ -63,11 +68,12 @@ The first interface is a local server and dense activity page. It reads projecti | Document versions | Exact observed content needed for replay and diffs | | Agent traces | What the runtime and model emitted | | Derived events | Versioned claims or proposals by a named agent/runtime | +| Dispatcher receipts | Durable evidence that a configured external action was claimed, delivered, or failed | | Projections/UI | Rebuildable convenience views only | ## Process shape -Development may run producers, consumers, projectors, and the HTTP server under one local supervisor. Deployment may run them independently on different computers without changing event contracts. In both forms, each source id and consumer id/version has one active owning process. +Development may run producers, consumers, projectors, dispatchers, and the HTTP server under one local supervisor. Deployment may run them independently on different computers without changing event contracts. In both forms, each source id, consumer id/version, and destination dispatcher id has one active owning process. `thought stream` is the human-facing local entrypoint. It loads one strict, versioned manifest from the project root (or an explicitly named path), opens Jazz, registers producer and consumer declarations, recovers nonterminal local consumer attempts, starts the loopback inspector, and starts only entries explicitly enabled for that process. diff --git a/spec/jazz.md b/spec/jazz.md index c78b174..9dc0b22 100644 --- a/spec/jazz.md +++ b/spec/jazz.md @@ -60,7 +60,7 @@ A local probe verified that `insert(..., { id })` accepts a caller-supplied UUID - `documentVersions`: exact observed versions required for replay and diffs. - `traceChunks`: immutable raw or redacted runner/provider stream records. -Historical rows are caller-id inserts. Jazz permission policy denies update, delete, and restore. Corrections append new events that reference prior rows. +Historical rows are caller-id inserts. The deployed Jazz permission policy must deny update, delete, and restore; the current local-development policy does not yet provide that guarantee. Corrections append new events that reference prior rows. ### Operational state diff --git a/spec/recovery.md b/spec/recovery.md index 82d3e7b..b816953 100644 --- a/spec/recovery.md +++ b/spec/recovery.md @@ -28,6 +28,15 @@ A live subscription processes messages serially. If local backpressure crosses i - Accepted outputs, terminal evidence, and progress for all consumed source sequences settle in one Jazz transaction. - One active process owns each consumer id/version. Redundant workers are a later explicit topology, not an implicit lease requirement. +## Dispatcher recovery + +- The first run for a channel writes one durable dispatcher activation event. This establishes the lower bound for notification candidates without flooding a new destination with historical activity. +- Completed consumer runs after activation are the durable accumulator. The dispatcher can stop without blocking producers or consumers; unclaimed candidates remain available when it returns. +- Each rendered batch gets a deterministic identity over dispatcher, destination, and included run ids. +- The dispatcher appends a `started` claim before external delivery. Only the process that inserts that claim may call the Bot API. +- `delivered` records the Telegram message id. `failed` preserves the transport error. A crash after `started` is status-unknown and is not blindly retried because Telegram does not provide an idempotency key for `sendMessage`; replay must be explicit to avoid duplicate human-visible sends. +- Velocity accounting reads recent durable `delivered` receipts, so restarting a dispatcher does not reset the channel's rate limit. + ## Terminal evidence A consumer execution is terminal only when one Jazz transaction has established: diff --git a/spec/security.md b/spec/security.md index c3363bf..d617a66 100644 --- a/spec/security.md +++ b/spec/security.md @@ -10,7 +10,9 @@ thought stream has unusually broad read access. Its first security property is c - Model capabilities: receive a bounded context, produce typed proposals. - Action capabilities: send, publish, edit, delete, transact. -The initial system implements ingress and model capabilities only. Action capabilities require separate code paths, credentials, policies, and explicit authorization events. +The system implements ingress and model capabilities plus one narrow action capability: Telegram delivery. Telegram ingress and egress are separate processes. The ingress connector has no send path. The dispatcher requires an enabled channel, source and actor allowlists, a destination velocity policy, and explicit started/delivered/failed receipt events. + +Action filtering happens at the egress boundary. Producers and consumers continue at source speed; the dispatcher alone decides which completed candidate activity may cross into a channel, how candidates are batched, and when destination capacity is available. ## Credentials @@ -43,6 +45,8 @@ All source content is untrusted data. Context rendering wraps it with source bou Configuration changes, agent activation, connector activation, model tier changes, and any future action capability changes append audit events with actor and revision. +Every Telegram attempt appends a durable `started` claim before calling the Bot API and then appends `delivered` or `failed` evidence. Claims use deterministic identities so concurrent dispatcher processes cannot intentionally claim the same batch twice. A claimed attempt is never inferred as delivered merely because the process exited cleanly. + ## Jazz policy boundary Historical immutability and private-row access are database policies, not conventions in a repository class. The deployed Jazz policy denies update/delete/restore for events, document versions, and trace chunks; producer cursors and consumer progress are mutable only by the process identity that owns their namespace. The permissive local fixture policy may never be reused for a synchronized deployment. diff --git a/spec/tinker.md b/spec/tinker.md index 00a2cf8..0aea128 100644 --- a/spec/tinker.md +++ b/spec/tinker.md @@ -29,6 +29,8 @@ Agent declarations choose capability tiers rather than hard-coding one global mo The tier resolver maps a tier to a current Tinker sampler path and Pi model descriptor. Historical runs retain the concrete mapping. +The current runtime accepts a concrete model id in each declaration and preserves it on the run. Tier resolution remains future work; a checked-in declaration must not imply that a tier resolver already exists. + ## Adapter evolution Training examples come from explicit judgments over run outputs, not from silently treating every accepted-looking output as correct. A judgment event records: @@ -41,6 +43,8 @@ Training examples come from explicit judgments over run outputs, not from silent An adapter release records its dataset manifest, base model, training configuration, eval set, and checkpoint path. Activating a new adapter updates the tier mapping; it never rewrites prior run metadata. +The implemented first projection writes JSONL only from judgment events whose `exportEligible` field is explicitly true. Corrections become chosen/rejected pairs; preferences join two comparable completed runs; accepted outputs become chosen-only examples; rejected outputs become rejected-only examples. This projection is input material for a later Tinker training job, not proof that training has run. + ## Initial limitation Tinker OpenAI-compatible sampling is treated as a beta/testing runtime rather than a highly available production service. Any consumer using it requires timeouts, bounded concurrency, retry classification, and a deterministic local provider for tests and degraded operation. diff --git a/spec/ui.md b/spec/ui.md index b14091a..bca39c0 100644 --- a/spec/ui.md +++ b/spec/ui.md @@ -17,12 +17,17 @@ Each activity row shows: - Derived output count. - Failure or blocked evidence. +The root view has one row per originating observation. Consumer lifecycle and derived events are grouped beneath that root rather than rendered as peer activity rows. The row must summarize the consumer's semantic result in ordinary language; lifecycle NSIDs and record identifiers are execution details. + +Deterministic transforms are labeled **rules**, not agents. The interface states explicitly whether LLM inference occurred. It must not imply model reasoning when a path-based or other hard-coded transform ran. + Filters: source, event family, agent, run status, privacy, time range, and text. The source-health view shows each connector's durable cursor, last success/failure, current error, poll receipt counts, recovery count, and whether a poll has started without terminal evidence. Health is derived from durable cursor and connector events rather than process liveness. ## Detail view +- Consumer lifecycle and derived-output details lead with a plain-language **what processed this** block: what ran, whether it was a rule or model, whether LLM inference occurred, the actual result, recommendation or proposal, derived-record count, and whether external actions were enabled. A reader must not have to traverse raw lifecycle payloads to discover the semantic result. - Canonical envelope and payload. - Source strong reference. - Document version/diff when applicable. diff --git a/src/agents/declarations.ts b/src/agents/declarations.ts index 8d17892..2e42bca 100644 --- a/src/agents/declarations.ts +++ b/src/agents/declarations.ts @@ -3,6 +3,7 @@ import path from "node:path"; import YAML from "yaml"; import { z } from "zod"; import { createDefaultRegistry } from "../events/registry.js"; +import { AGENT_TOOL_NAMES } from "./tools.js"; import type { ThoughtAgentDeclaration } from "./types.js"; const declarationFileSchema = z.object({ @@ -22,6 +23,7 @@ const declarationFileSchema = z.object({ runner: z.object({ kind: z.enum(["deterministic", "pi"]), provider: z.enum(["tinker", "openai-compatible"]).optional(), + tier: z.enum(["triage-small", "reasoning-small", "escalation"]).optional(), model: z.string().min(1).optional(), baseUrl: z.string().url().optional(), apiKeyEnv: z.string().min(1).optional(), @@ -31,13 +33,16 @@ const declarationFileSchema = z.object({ prompt: z.string().min(1), emit: z.array(z.string().min(1)).length(1), policy: z.object({ - tools: z.array(z.string()).max(0).default([]), + tools: z.array(z.enum(AGENT_TOOL_NAMES)).max(8).default([]), externalActions: z.literal(false), }), enabled: z.boolean().default(true), }).superRefine((value, context) => { - if (value.runner.kind === "pi" && !value.runner.model) { - context.addIssue({ code: "custom", path: ["runner", "model"], message: "Pi agents require a model" }); + if (value.runner.kind === "pi" && !value.runner.model && !value.runner.tier) { + context.addIssue({ code: "custom", path: ["runner", "model"], message: "Pi agents require a model or capability tier" }); + } + if (value.runner.kind !== "pi" && value.policy.tools.length > 0) { + context.addIssue({ code: "custom", path: ["policy", "tools"], message: "Only Pi agents may declare tools" }); } }); @@ -61,6 +66,13 @@ export async function loadAgentDeclarations(directory: string): Promise definition.type) + .filter((type) => file.subscribe.types.some((pattern) => matchesPattern(pattern, type))); + if (compiledEventTypes.length === 0) { + throw new Error(`Agent ${file.id} subscription compiles to no registered event types`); + } + const concreteModel = resolveRunnerModel(file.runner); declarations.push({ id: file.id, version: file.version, @@ -68,10 +80,12 @@ export async function loadAgentDeclarations(directory: string): Promise Promise): Promise { + if (input.event.type === "stream.thought.source.atproto.commit") { + return classifyAtprotoCommit(input, onTrace); + } const payload = input.event.payload; const documentPath = typeof payload.path === "string" ? payload.path : "unknown"; const extension = path.extname(documentPath).slice(1) || "document"; @@ -34,6 +37,70 @@ export class DeterministicTriageRunner implements AgentRunner { } } +async function classifyAtprotoCommit( + input: AgentRunInput, + onTrace: (trace: { kind: string; data: unknown }) => Promise, +): Promise { + const payload = input.event.payload; + const operation = stringField(payload.operation, "observed"); + const collection = stringField(payload.collection, "unknown-collection"); + const atUri = stringField(payload.atUri, input.event.externalId); + const record = objectField(payload.record); + const text = record && typeof record.text === "string" ? compactText(record.text) : undefined; + const subject = record ? objectField(record.subject) : undefined; + const subjectUri = subject && typeof subject.uri === "string" ? subject.uri : undefined; + const tags = ["atproto", collection, operation]; + if (collection === "app.bsky.feed.post") tags.push("post"); + if (collection === "app.bsky.feed.like") tags.push("like"); + if (record && typeof record.reply === "object") tags.push("reply"); + if (record && Array.isArray(record.facets) && record.facets.length > 0) tags.push("facets"); + if (record && typeof record.embed === "object") tags.push("embed"); + await onTrace({ + kind: "deterministic.atproto-classification", + data: { operation, collection, atUri, tags, textChars: text?.length ?? 0, ...(subjectUri ? { subjectUri } : {}) }, + }); + return { + summary: atprotoSummary(operation, collection, atUri, text, subjectUri), + tags, + importance: "normal", + confidence: 1, + }; +} + +function atprotoSummary(operation: string, collection: string, atUri: string, text?: string, subjectUri?: string): string { + const noun = collection === "app.bsky.feed.post" ? "post" : collection === "app.bsky.feed.like" ? "like" : collection; + if (text) return `${capitalize(operation)} ${noun}: ${truncate(text, 220)}`; + if (subjectUri) return `${capitalize(operation)} ${noun}: ${atUriToWebUrl(subjectUri)}`; + return `${capitalize(operation)} ${noun} at ${atUri}`; +} + +function atUriToWebUrl(value: string): string { + const match = value.match(/^at:\/\/([^/]+)\/app\.bsky\.feed\.post\/([^/]+)$/); + return match ? `https://bsky.app/profile/${match[1]}/post/${match[2]}` : value; +} + +function compactText(value: string): string { + return value.replaceAll(/\s+/g, " ").trim(); +} + +function truncate(value: string, limit: number): string { + return value.length <= limit ? value : `${value.slice(0, limit - 1).trimEnd()}…`; +} + +function stringField(value: unknown, fallback: string): string { + return typeof value === "string" && value.length > 0 ? value : fallback; +} + +function objectField(value: unknown): Record | undefined { + return value && typeof value === "object" && !Array.isArray(value) + ? value as Record + : undefined; +} + +function capitalize(value: string): string { + return value.charAt(0).toUpperCase() + value.slice(1); +} + function humanizeEvent(type: string): string { const action = type.split(".").at(-1) ?? "observed"; return action.charAt(0).toUpperCase() + action.slice(1); diff --git a/src/agents/pi.ts b/src/agents/pi.ts index cf1f00d..fb75328 100644 --- a/src/agents/pi.ts +++ b/src/agents/pi.ts @@ -1,8 +1,10 @@ import { Agent, type AgentEvent, type StreamFn } from "@earendil-works/pi-agent-core"; -import type { AssistantMessage, Model } from "@earendil-works/pi-ai"; +import type { AssistantMessage, ImageContent, Model } from "@earendil-works/pi-ai"; import { streamSimple as streamSimpleOpenAICompletions } from "@earendil-works/pi-ai/api/openai-completions"; +import { createHash, randomUUID } from "node:crypto"; import { z } from "zod"; -import type { AgentOutput, AgentRunInput, AgentRunner, RunnerTrace, ThoughtAgentDeclaration } from "./types.js"; +import { createRunTools, type AgentToolName, type RunToolOptions } from "./tools.js"; +import { AgentRunFailure, type AgentOutput, type AgentRunInput, type AgentRunner, type RunnerTrace, type ThoughtAgentDeclaration } from "./types.js"; const outputSchema = z.object({ summary: z.string().min(1), @@ -19,17 +21,35 @@ const outputSchema = z.object({ export class PiAgentRunner implements AgentRunner { readonly mode = "pi" as const; + constructor(private readonly options: Pick = {}) {} + async run(input: AgentRunInput, onTrace: (trace: RunnerTrace) => Promise): Promise { const declaration = input.declaration; if (!declaration.model) throw new Error(`Pi agent ${declaration.id} has no model`); const model = buildPiModelDescriptor(declaration); const apiKeyName = resolveApiKeyName(declaration); + const toolSet = createRunTools({ + event: input.event, + names: declaration.tools as AgentToolName[], + onTrace, + ...this.options, + }); + const useTinkerToolFallback = declaration.provider === "tinker" && toolSet.tools.length > 0; + const prefetched = useTinkerToolFallback + ? await prefetchTinkerEvidence(input.event, toolSet.tools, onTrace) + : { text: "", images: [] as ImageContent[] }; + const toolInstructions = useTinkerToolFallback + ? "ThoughtStream has already executed the configured read-only tools and supplied their results below. Do not describe, propose, simulate, or attempt tool calls. Inspect the pre-fetched evidence and images, then make your first and only response the final JSON object. Tool failures are evidence: report uncertainty rather than inventing missing context." + : toolSet.tools.length > 0 + ? "Do not narrate or plan tool calls. Your first action must be fetch_atproto_markdown with target event. For a like or repost, your next action must be fetch_atproto_markdown with target subject. Download every relevant image URL returned by those calls. After the required tool results, return the final JSON object. Tool failures are evidence: continue with available context and report uncertainty." + : "You have no tools."; + let observedRevision: string | undefined; const agent = new Agent({ initialState: { - systemPrompt: `${declaration.systemPrompt}\n\nReturn one JSON object only with: summary, tags, importance (low|normal|high), confidence (0..1), and optional recommendation {target, reason, proposedAction}. You have no tools and no authority to change external state.`, + systemPrompt: `${declaration.systemPrompt}\n\n${toolInstructions}\n\nThe final answer must be one JSON object only with: summary, tags, importance (low|normal|high), confidence (0..1), and optional recommendation {target, reason, proposedAction}. Do not emit analysis, a plan, Markdown, or tags. You have no authority to change external state.`, model, thinkingLevel: "off", - tools: [], + tools: useTinkerToolFallback ? [] : toolSet.tools, messages: [], }, streamFn: streamSimpleOpenAICompletions as StreamFn, @@ -39,6 +59,9 @@ export class PiAgentRunner implements AgentRunner { return undefined; }, onResponse: async (response) => { + observedRevision = response.headers["x-tinker-checkpoint"] + ?? response.headers["x-model-revision"] + ?? response.headers["x-checkpoint-revision"]; await onTrace({ kind: "provider.response", data: redactSensitive({ status: response.status, headers: response.headers }, process.env[apiKeyName]), @@ -47,7 +70,9 @@ export class PiAgentRunner implements AgentRunner { maxRetryDelayMs: 30_000, }); agent.subscribe(async (event) => onTrace({ kind: `pi.${event.type}`, data: compactAgentEvent(event) })); - const prompt = input.context.text; + const prompt = prefetched.text + ? `${input.context.text}\n\n## Pre-fetched read-only evidence\n${prefetched.text}` + : input.context.text; const controller = new AbortController(); const timeout = setTimeout(() => { controller.abort(); @@ -56,9 +81,10 @@ export class PiAgentRunner implements AgentRunner { try { await onTrace({ kind: "prompt", data: { chars: prompt.length, text: prompt } }); if (controller.signal.aborted) throw new Error("Agent run timed out before start"); - await agent.prompt(prompt); + await agent.prompt(prompt, prefetched.images); if (controller.signal.aborted) throw new Error(`Agent run timed out after ${declaration.timeoutMs}ms`); - const rawText = extractFinalText(agent.state.messages); + const finalMessage = extractFinalAssistant(agent.state.messages); + const rawText = extractFinalText(finalMessage); const parsed = outputSchema.parse(parseJsonObject(rawText)); return { summary: parsed.summary, @@ -67,16 +93,80 @@ export class PiAgentRunner implements AgentRunner { confidence: parsed.confidence, ...(parsed.recommendation ? { recommendation: parsed.recommendation } : {}), rawText, - model: { provider: model.provider, id: model.id }, + model: { + provider: model.provider, + id: finalMessage.model || model.id, + ...(observedRevision ? { revision: observedRevision } : {}), + }, + ...(toolSet.outcomes.length > 0 ? { enrichments: toolSet.outcomes } : {}), }; + } catch (error) { + if (error instanceof AgentRunFailure) throw error; + const message = error instanceof Error ? error.message : String(error); + const attemptedTranscript = extractAttemptedTranscript(agent.state.messages); + throw new AgentRunFailure(message, { + cause: error, + ...(attemptedTranscript ? { attemptedTranscript } : {}), + }); } finally { clearTimeout(timeout); } } } +async function prefetchTinkerEvidence( + event: AgentRunInput["event"], + tools: ReturnType["tools"], + onTrace: (trace: RunnerTrace) => Promise, +): Promise<{ text: string; images: ImageContent[] }> { + const markdownTool = tools.find((tool) => tool.name === "fetch_atproto_markdown"); + const imageTool = tools.find((tool) => tool.name === "download_image"); + const evidence: string[] = []; + const images: ImageContent[] = []; + const imageUrls = new Set(); + const collection = typeof event.payload.collection === "string" ? event.payload.collection : ""; + const targets: Array<"event" | "subject"> = ["event"]; + if (collection === "app.bsky.feed.like" || collection === "app.bsky.feed.repost") targets.push("subject"); + + for (const target of targets) { + if (!markdownTool) break; + try { + await onTrace({ kind: "pi.tool_prefetch_start", data: { toolName: markdownTool.name, args: { target } } }); + const result = await markdownTool.execute(randomUUID(), { target }); + evidence.push(...result.content.filter((part) => part.type === "text").map((part) => part.text)); + const details = result.details as { imageUrls?: unknown }; + if (Array.isArray(details.imageUrls)) { + for (const url of details.imageUrls) if (typeof url === "string") imageUrls.add(url); + } + await onTrace({ kind: "pi.tool_prefetch_end", data: { toolName: markdownTool.name, target, status: "succeeded" } }); + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + evidence.push(`[${markdownTool.name} ${target} failed: ${message}]`); + await onTrace({ kind: "pi.tool_prefetch_end", data: { toolName: markdownTool.name, target, status: "failed", error: message } }); + } + } + + for (const url of imageUrls) { + if (!imageTool) break; + try { + await onTrace({ kind: "pi.tool_prefetch_start", data: { toolName: imageTool.name, args: { url } } }); + const result = await imageTool.execute(randomUUID(), { url }); + evidence.push(...result.content.filter((part) => part.type === "text").map((part) => part.text)); + images.push(...result.content.filter((part): part is ImageContent => part.type === "image")); + await onTrace({ kind: "pi.tool_prefetch_end", data: { toolName: imageTool.name, status: "succeeded" } }); + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + evidence.push(`[${imageTool.name} failed: ${message}]`); + await onTrace({ kind: "pi.tool_prefetch_end", data: { toolName: imageTool.name, status: "failed", error: message } }); + } + } + + return { text: evidence.join("\n\n"), images }; +} + export function buildPiModelDescriptor(declaration: ThoughtAgentDeclaration): Model<"openai-completions"> { if (!declaration.model) throw new Error(`Pi agent ${declaration.id} has no model`); + const isTinker = declaration.provider === "tinker"; const baseUrl = declaration.baseUrl ?? (declaration.provider === "tinker" ? "https://tinker.thinkingmachines.dev/services/tinker-prod/oai/api/v1" : undefined); @@ -87,8 +177,15 @@ export function buildPiModelDescriptor(declaration: ThoughtAgentDeclaration): Mo api: "openai-completions", provider: declaration.provider ?? "openai-compatible", baseUrl, - reasoning: false, - input: ["text"], + reasoning: isTinker, + ...(isTinker ? { + compat: { + supportsDeveloperRole: false, + supportsReasoningEffort: false, + thinkingFormat: "qwen-chat-template" as const, + }, + } : {}), + input: declaration.tools.includes("web.download-image") ? ["text", "image"] : ["text"], cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, contextWindow: 131_072, maxTokens: declaration.maxOutputTokens, @@ -99,17 +196,35 @@ export function resolveApiKeyName(declaration: ThoughtAgentDeclaration): string return declaration.apiKeyEnv ?? (declaration.provider === "tinker" ? "TINKER_API_KEY" : "OPENAI_API_KEY"); } -function extractFinalText(messages: unknown[]): string { +function extractFinalAssistant(messages: unknown[]): AssistantMessage { const assistants = messages.filter((message): message is AssistantMessage => Boolean(message && typeof message === "object" && "role" in message && message.role === "assistant")); const last = assistants.at(-1); if (!last) throw new Error("Pi agent returned no assistant message"); - const text = last.content.filter((part) => part.type === "text").map((part) => part.text).join("\n").trim(); - if (!text) throw new Error(last.errorMessage ?? "Pi agent returned no text"); + return last; +} + +function extractFinalText(message: AssistantMessage): string { + const text = message.content.filter((part) => part.type === "text").map((part) => part.text).join("\n").trim(); + if (!text) throw new Error(message.errorMessage ?? "Pi agent returned no text"); return text; } +function extractAttemptedTranscript(messages: unknown[]): string | undefined { + const assistants = messages.filter((message): message is AssistantMessage => Boolean( + message && typeof message === "object" && "role" in message && message.role === "assistant", + )); + const transcript = assistants.flatMap((message) => message.content.map((part) => { + if (part.type === "thinking") return part.thinking ? `[thinking]\n${part.thinking}` : ""; + if (part.type === "text") return part.text ? `[assistant]\n${part.text}` : ""; + if (part.type === "toolCall") return `[tool ${part.name}]\n${JSON.stringify(part.arguments)}`; + return ""; + })).filter(Boolean).join("\n\n").trim(); + return transcript || undefined; +} + function parseJsonObject(text: string): unknown { - const unfenced = text.replace(/^```(?:json)?\s*/i, "").replace(/\s*```$/i, "").trim(); + const afterThinking = text.includes("") ? text.slice(text.lastIndexOf("") + "".length) : text; + const unfenced = afterThinking.replace(/^```(?:json)?\s*/i, "").replace(/\s*```$/i, "").trim(); try { return JSON.parse(unfenced); } catch { @@ -122,17 +237,24 @@ function parseJsonObject(text: string): unknown { function compactAgentEvent(event: AgentEvent): unknown { if (event.type === "message_update") { - return { type: event.type, assistantMessageEvent: event.assistantMessageEvent }; + return compactBinary({ type: event.type, assistantMessageEvent: event.assistantMessageEvent }); } - return event; + return compactBinary(event); } export function redactSensitive(value: unknown, knownSecret?: string): unknown { if (typeof value === "string") { + if (value.startsWith("data:image/") && value.includes(",")) return compactDataUrl(value); return knownSecret && knownSecret.length >= 6 ? value.replaceAll(knownSecret, "[redacted]") : value; } if (Array.isArray(value)) return value.map((item) => redactSensitive(item, knownSecret)); if (!value || typeof value !== "object") return value; + if ("type" in value && value.type === "image" && "data" in value && typeof value.data === "string") { + return { + ...Object.fromEntries(Object.entries(value).filter(([key]) => key !== "data")), + data: compactBase64(value.data), + }; + } const redacted: Record = {}; for (const [key, item] of Object.entries(value)) { const normalized = key.toLowerCase().replaceAll(/[^a-z0-9]/g, ""); @@ -144,3 +266,33 @@ export function redactSensitive(value: unknown, knownSecret?: string): unknown { } return redacted; } + +function compactBinary(value: unknown): unknown { + return redactSensitive(value); +} + +function compactDataUrl(value: string): Record { + const [header = "data:image/unknown;base64", data = ""] = value.split(",", 2); + return { + omitted: "base64-image", + mediaType: header.slice(5).split(";", 1)[0], + base64Chars: data.length, + sha256: hashBase64(data), + }; +} + +function compactBase64(value: string): Record { + return { + omitted: "base64-image", + base64Chars: value.length, + sha256: hashBase64(value), + }; +} + +function hashBase64(value: string): string { + try { + return createHash("sha256").update(Buffer.from(value, "base64")).digest("hex"); + } catch { + return createHash("sha256").update(value).digest("hex"); + } +} diff --git a/src/agents/runtime.ts b/src/agents/runtime.ts index d764044..a8f46ef 100644 --- a/src/agents/runtime.ts +++ b/src/agents/runtime.ts @@ -1,13 +1,17 @@ -import { newId, stableKey } from "../core/ids.js"; import { canonicalJson, sha256, type JsonObject } from "../core/json.js"; -import type { ThoughtEvent } from "../events/types.js"; -import type { AgentRun, ThoughtStore } from "../store/types.js"; +import { stableKey } from "../core/ids.js"; +import type { EventCandidate, ThoughtEvent } from "../events/types.js"; +import type { JazzThoughtStore } from "../jazz/store.js"; +import type { AgentRun, ConsumerEventQuery, ConsumerProgress } from "../store/types.js"; import { buildContextPacket } from "./context.js"; -import { matchesDeclaration } from "./declarations.js"; import { DeterministicTriageRunner } from "./deterministic.js"; import { PiAgentRunner } from "./pi.js"; -import { leaseRun, renewLease } from "./scheduler.js"; -import type { AgentOutput, AgentRunner, ThoughtAgentDeclaration } from "./types.js"; +import { AgentRunFailure, type AgentOutput, type AgentRunner, type ThoughtAgentDeclaration } from "./types.js"; + +export interface ConsumerHandle { + drain(): Promise; + stop(): Promise; +} export interface ProcessEventResult { declaration: ThoughtAgentDeclaration; @@ -20,14 +24,19 @@ export interface ProcessEventResult { export class ThoughtAgentRuntime { private readonly runners = new Map(); - constructor(private readonly store: ThoughtStore, runners: AgentRunner[] = [new DeterministicTriageRunner(), new PiAgentRunner()]) { - for (const runner of runners) this.runners.set(runner.mode, runner); + constructor( + private readonly store: JazzThoughtStore, + runners?: AgentRunner[], + ) { + const selected = runners ?? [ + new DeterministicTriageRunner(), + new PiAgentRunner({ artifactRoot: store.getArtifactRoot() }), + ]; + for (const runner of selected) this.runners.set(runner.mode, runner); } - async processEvent(event: ThoughtEvent, declarations: ThoughtAgentDeclaration[]): Promise { - const results: ProcessEventResult[] = []; - for (const declaration of declarations.filter((candidate) => matchesDeclaration(candidate, event))) { - const now = new Date().toISOString(); + async registerDeclarations(declarations: ThoughtAgentDeclaration[]): Promise { + for (const declaration of declarations) { const declarationJson = asJsonObject(declaration); await this.store.upsertAgent({ id: declaration.id, @@ -35,48 +44,141 @@ export class ThoughtAgentRuntime { enabled: declaration.enabled, spec: declarationJson, specHash: sha256(canonicalJson(declarationJson)), - updatedAt: now, + updatedAt: new Date().toISOString(), }); - results.push(await this.runDeclaration(event, declaration)); } - await this.store.flush(); - return results; } - private async runDeclaration(event: ThoughtEvent, declaration: ThoughtAgentDeclaration): Promise { - const runner = this.runners.get(declaration.mode); - const matchId = stableKey("match", event.id, declaration.id, String(declaration.version)); - const runId = stableKey("run", matchId, "1"); - if (!runner) return { declaration, runId, error: `No runner for mode ${declaration.mode}` }; - const matchInserted = await this.store.insertMatch({ - id: matchId, - eventId: event.id, - agentId: declaration.id, - agentVersion: declaration.version, - matcherRevision: "event-type-prefix-v1", - createdAt: new Date().toISOString(), - }); - const prior = (await this.store.listRuns()).find((run) => run.id === runId); - if (prior?.status === "completed") return { declaration, runId }; - if (prior?.status === "failed") return { declaration, runId, error: prior.errorText ?? "Previous run failed" }; - if (prior?.status === "leased" || prior?.status === "running") { - return { declaration, runId, error: `Run is already ${prior.status}; recover it only after its lease expires` }; + async consumeBacklog(declarations: ThoughtAgentDeclaration[]): Promise { + await this.registerDeclarations(declarations); + const runs: ProcessEventResult[] = []; + for (const declaration of declarations.filter((candidate) => candidate.enabled)) { + for (const source of await this.resolveSources(declaration)) { + for (;;) { + const progress = await this.store.getConsumerProgress(progressId(declaration, source)); + const events = await this.store.queryConsumerEvents(this.consumerQuery(declaration, source, progress?.lastSequence ?? 0)); + for (const event of events) runs.push(await this.executeEvent(event, declaration)); + if (events.length < 1_000) break; + } + } } - const attempt = prior?.attempt ?? 1; - const queuedAt = new Date().toISOString(); - const context = buildContextPacket(declaration, event); - if (matchInserted) { - await this.appendLifecycleEvent("matched", declaration, event, runId, { - matchId, - status: "matched", - }, 1); + return runs; + } + + async startConsumers(declarations: ThoughtAgentDeclaration[]): Promise { + await this.registerDeclarations(declarations); + const unsubscribers: Array<() => void> = []; + const subscriptions = new Set(); + let chain = Promise.resolve(); + let stopped = false; + const consumerErrors: unknown[] = []; + const enqueue = (operation: () => Promise): void => { + chain = chain.then(async () => { + let lastError: unknown; + for (let attempt = 1; attempt <= 3; attempt += 1) { + try { + await operation(); + return; + } catch (error) { + lastError = error; + if (attempt < 3) await new Promise((resolve) => setTimeout(resolve, 25 * attempt)); + } + } + throw lastError; + }).catch((error: unknown) => { + consumerErrors.push(error); + process.stderr.write(`ThoughtStream consumer cycle failed: ${error instanceof Error ? error.message : String(error)}\n`); + }); + }; + const enabled = declarations.filter((candidate) => candidate.enabled); + const install = async (declaration: ThoughtAgentDeclaration, source: string): Promise => { + const key = `${declaration.id}@${declaration.version}:${source}`; + if (stopped || subscriptions.has(key)) return; + const progress = await this.store.getConsumerProgress(progressId(declaration, source)); + const query = this.consumerQuery(declaration, source, progress?.lastSequence ?? 0); + unsubscribers.push(this.store.subscribeConsumerEvents(query, () => { + if (stopped) return; + enqueue(async () => { + for (;;) { + const currentProgress = await this.store.getConsumerProgress(progressId(declaration, source)); + const events = await this.store.queryConsumerEvents(this.consumerQuery( + declaration, + source, + currentProgress?.lastSequence ?? 0, + )); + for (const event of events) { + const current = await this.store.getConsumerProgress(progressId(declaration, source)); + if ((current?.lastSequence ?? 0) >= event.sourceSequence) continue; + await this.executeEvent(event, declaration); + } + if (events.length < 1_000) break; + } + }); + })); + subscriptions.add(key); + }; + for (const declaration of enabled) { + for (const source of await this.resolveSources(declaration)) await install(declaration, source); + } + if (enabled.some((declaration) => declaration.sourcePatterns.some((pattern) => pattern.includes("*")))) { + unsubscribers.push(this.store.subscribeSources((sources) => { + if (stopped) return; + enqueue(async () => { + for (const declaration of enabled) { + for (const source of sources) { + if (declaration.sourcePatterns.some((pattern) => matchesPatternValue(pattern, source.id))) { + await install(declaration, source.id); + } + } + } + }); + })); + } + return { + drain: async () => { + for (;;) { + const pending = chain; + await pending; + if (pending === chain) { + if (consumerErrors.length > 0) throw new AggregateError(consumerErrors.splice(0), "ThoughtStream consumer cycles failed"); + return; + } + } + }, + stop: async () => { + stopped = true; + for (const unsubscribe of unsubscribers) unsubscribe(); + await chain; + if (consumerErrors.length > 0) throw new AggregateError(consumerErrors.splice(0), "ThoughtStream consumer cycles failed"); + }, + }; + } + + private async executeEvent(event: ThoughtEvent, declaration: ThoughtAgentDeclaration): Promise { + const progressKey = progressId(declaration, event.source); + const progress = await this.store.getConsumerProgress(progressKey); + if ((progress?.lastSequence ?? 0) >= event.sourceSequence) { + const prior = await this.store.latestRunForExecution(executionKey(event, declaration)); + if (!prior) throw new Error(`Progress exists without execution evidence for ${event.id}`); + return { declaration, runId: prior.id, ...(prior.errorText ? { error: prior.errorText } : {}) }; } - const queuedRun: AgentRun = { + + const key = executionKey(event, declaration); + const prior = await this.store.latestRunForExecution(key); + if (prior?.status === "running") await this.abandonInterruptedRun(prior, declaration, event); + const attempt = (prior?.attempt ?? 0) + 1; + const runId = stableKey("run", key, String(attempt)); + const context = buildContextPacket(declaration, event); + const runner = this.runners.get(declaration.mode); + if (!runner) return { declaration, runId, error: `No runner for mode ${declaration.mode}` }; + const startedAt = new Date().toISOString(); + let run: AgentRun = { id: runId, - matchId, + executionKey: key, + triggerEventId: event.id, agentId: declaration.id, agentVersion: declaration.version, - status: "queued", + status: "running", inputEventIds: [event.id], outputEventIds: [], attempt, @@ -85,197 +187,265 @@ export class ThoughtAgentRuntime { adapterRevision: declaration.mode === "pi" ? "pi-openai-completions-v1" : "deterministic-triage-v1", promptHash: sha256(declaration.systemPrompt), contextManifest: context.manifest, - createdAt: prior?.createdAt ?? queuedAt, - updatedAt: queuedAt, + createdAt: startedAt, + startedAt, + updatedAt: startedAt, }; - await this.store.upsertRun(queuedRun); - await this.appendLifecycleEvent("queued", declaration, event, runId, { matchId, status: "queued" }, attempt); - const workerId = "thoughtstream-local"; - const leaseMs = declaration.timeoutMs + 30_000; - const leasedRun = await leaseRun(this.store, runId, { workerId, leaseMs }); - await this.appendLifecycleEvent("leased", declaration, event, runId, { - matchId, - status: "leased", - workerId: leasedRun.workerId ?? "thoughtstream-local", - leaseId: leasedRun.leaseId ?? "missing", - leaseExpiresAt: leasedRun.leaseExpiresAt ?? "missing", - }, attempt); - const startedAt = new Date().toISOString(); - const run: AgentRun = { ...leasedRun, status: "running", startedAt, updatedAt: startedAt }; await this.store.upsertRun(run); - await this.appendLifecycleEvent("started", declaration, event, runId, { - matchId, + await this.appendLifecycleEvent("started", declaration, event, run, { status: "running", - mode: declaration.mode, - }, attempt); - let renewalTimer: NodeJS.Timeout | undefined; - let renewalPromise: Promise | undefined; - let renewalError: Error | undefined; - let renewalStopped = false; - const renewalIntervalMs = Math.max(1_000, Math.min(30_000, Math.floor(leaseMs / 3))); - const scheduleRenewal = () => { - if (renewalStopped) return; - renewalTimer = setTimeout(() => { - renewalPromise = renewLease(this.store, runId, { - workerId, - leaseId: leasedRun.leaseId!, - leaseMs, - }).then(() => undefined).catch((error: unknown) => { - renewalError = error instanceof Error ? error : new Error(String(error)); - }).finally(scheduleRenewal); - }, renewalIntervalMs); - renewalTimer.unref(); - }; - const stopRenewal = async () => { - renewalStopped = true; - if (renewalTimer) clearTimeout(renewalTimer); - if (renewalPromise) await renewalPromise; - }; - scheduleRenewal(); - let traceSequence = (await this.store.listTrace(runId)).reduce((next, trace) => Math.max(next, trace.sequence + 1), 0); + executionKey: key, + sourceSequence: event.sourceSequence, + }); + + let traceSequence = 0; + let output: AgentOutput; try { - const output = await runner.run({ runId, declaration, event, context }, async (trace) => { - const sequence = traceSequence++; + output = await runner.run({ runId, declaration, event, context }, async (trace) => { + traceSequence += 1; + const at = new Date().toISOString(); await this.store.appendTrace({ - id: newId("trace"), + id: stableKey("trace", runId, String(traceSequence)), runId, - sequence, + sequence: traceSequence, type: trace.kind, payload: { data: asJsonValue(trace.data) }, - createdAt: trace.at ?? new Date().toISOString(), + createdAt: trace.at ?? at, }); - await this.appendLifecycleEvent("trace", declaration, event, runId, { + await this.appendLifecycleEvent("trace", declaration, event, run, { status: "running", - sequence, + sequence: traceSequence, traceType: trace.kind, data: asJsonValue(trace.data), - }, attempt); - }); - await stopRenewal(); - if (renewalError) throw renewalError; - const derived = await this.store.appendEvent({ - type: declaration.outputEventType, - schemaVersion: 1, - source: `agent:${declaration.id}`, - sourceKind: "agent", - externalId: runId, - idempotencyKey: `${event.id}:${declaration.id}:${declaration.outputEventType}`, - occurredAt: new Date().toISOString(), - actor: declaration.id, - correlationId: runId, - parentEventId: event.id, - rootEventId: event.rootEventId, - privacy: event.privacy, - payload: asJsonObject({ runId, agentId: declaration.id, triggerEventId: event.id, ...output }), + }); }); + } catch (error) { const completedAt = new Date().toISOString(); - await this.store.upsertRun({ + const message = error instanceof Error ? error.message : String(error); + const attemptedTranscript = error instanceof AgentRunFailure ? error.attemptedTranscript : undefined; + run = { ...run, - status: "completed", - outputEventIds: [derived.event.id], - result: asJsonObject(output), + status: "failed", + errorText: message, + ...(attemptedTranscript ? { result: { attemptedTranscript } } : {}), completedAt, updatedAt: completedAt, - }); - await this.store.appendEvent({ - type: "stream.thought.agent.run.completed", - schemaVersion: 1, - source: `agent:${declaration.id}`, - sourceKind: "agent", - externalId: runId, - idempotencyKey: `${runId}:completed`, - occurredAt: completedAt, - actor: declaration.id, - correlationId: runId, - parentEventId: derived.event.id, - rootEventId: event.rootEventId, - privacy: event.privacy, - payload: { - runId, - agentId: declaration.id, - agentVersion: declaration.version, - inputEventIds: [event.id], - attempt, - status: "completed", - outputEventId: derived.event.id, - traceCount: traceSequence, - }, - }); - return { declaration, runId, output, derivedEvent: derived.event }; - } catch (error) { - await stopRenewal(); - const message = error instanceof Error ? error.message : String(error); - const failedAt = new Date().toISOString(); - await this.store.upsertRun({ ...run, status: "failed", errorText: message, completedAt: failedAt, updatedAt: failedAt }); - await this.store.appendEvent({ - type: "stream.thought.agent.run.failed", - schemaVersion: 1, - source: `agent:${declaration.id}`, - sourceKind: "agent", - externalId: runId, - idempotencyKey: `${runId}:failed`, - occurredAt: failedAt, - actor: declaration.id, - correlationId: runId, - parentEventId: event.id, - rootEventId: event.rootEventId, - privacy: event.privacy, - payload: { - runId, - agentId: declaration.id, - agentVersion: declaration.version, - inputEventIds: [event.id], - attempt, + }; + await this.store.settleConsumerFailure({ + run, + inputEvent: event, + terminal: this.lifecycleCandidate("failed", declaration, event, run, completedAt, { status: "failed", - triggerEventId: event.id, error: message, - traceCount: traceSequence, - }, + ...(attemptedTranscript ? { attemptedTranscript } : {}), + }), + progress: progressFor(declaration, event, completedAt), }); return { declaration, runId, error: message }; } + + const completedAt = new Date().toISOString(); + const outputCandidate = this.outputCandidate(declaration, event, run, output, completedAt); + const outputEventId = deterministicEventId(outputCandidate); + run = { + ...run, + status: "completed", + outputEventIds: [outputEventId], + ...(output.model ? { + provider: output.model.provider, + model: output.model.id, + ...(output.model.revision ? { adapterRevision: output.model.revision } : {}), + } : {}), + result: asJsonObject(output), + completedAt, + updatedAt: completedAt, + }; + const completed = this.lifecycleCandidate("completed", declaration, event, run, completedAt, { + status: "completed", + outputEventId, + }); + const settled = await this.store.settleConsumerSuccess({ + run, + inputEvent: event, + output: outputCandidate, + completed, + progress: progressFor(declaration, event, completedAt), + }); + return { + declaration, + runId, + output, + derivedEvent: settled.outputEvent, + }; } - private async appendLifecycleEvent( - phase: "matched" | "queued" | "leased" | "started" | "trace", + private async abandonInterruptedRun( + prior: AgentRun, declaration: ThoughtAgentDeclaration, event: ThoughtEvent, - runId: string, - details: JsonObject, - attempt: number, ): Promise { - const type = phase === "matched" ? "stream.thought.agent.matched" : `stream.thought.agent.run.${phase}`; - const occurredAt = new Date().toISOString(); - await this.store.appendEvent({ - type, + const at = new Date().toISOString(); + const abandoned: AgentRun = { + ...prior, + status: "abandoned", + errorText: "Interrupted before terminal consumer settlement", + completedAt: at, + updatedAt: at, + }; + await this.store.settleConsumerAbandonment(abandoned, this.lifecycleCandidate("abandoned", declaration, event, abandoned, at, { + status: "abandoned", + reason: "restart-recovery", + })); + } + + private consumerQuery(declaration: ThoughtAgentDeclaration, source: string, afterSequence: number): ConsumerEventQuery { + return { + consumerId: declaration.id, + consumerVersion: declaration.version, + source, + eventTypes: declaration.compiledEventTypes, + acceptedPrivacy: declaration.acceptedPrivacy, + afterSequence, + limit: 1_000, + }; + } + + private async resolveSources(declaration: ThoughtAgentDeclaration): Promise { + const exact = declaration.sourcePatterns.filter((pattern) => !pattern.includes("*")); + const discovered = (await this.store.listSources()) + .map((source) => source.id) + .filter((source) => declaration.sourcePatterns.some((pattern) => matchesPatternValue(pattern, source))); + return [...new Set([...exact, ...discovered])].sort(); + } + + private outputCandidate( + declaration: ThoughtAgentDeclaration, + event: ThoughtEvent, + run: AgentRun, + result: AgentOutput, + at: string, + ): EventCandidate { + return { + type: declaration.outputEventType, schemaVersion: 1, source: `agent:${declaration.id}`, sourceKind: "agent", - externalId: runId, - idempotencyKey: `${runId}:${phase}:attempt:${attempt}${phase === "trace" ? `:${String(details.sequence)}` : ""}`, - occurredAt, + externalId: run.id, + idempotencyKey: `${run.id}:output`, + occurredAt: at, actor: declaration.id, - correlationId: runId, + rootEventId: event.rootEventId, parentEventId: event.id, + correlationId: event.correlationId, + privacy: event.privacy, + payload: { + runId: run.id, + executionKey: run.executionKey, + inputEventId: event.id, + inputSourceSequence: event.sourceSequence, + summary: result.summary, + ...(result.tags ? { tags: result.tags } : {}), + ...(result.importance ? { importance: result.importance } : {}), + ...(result.recommendation ? { recommendation: result.recommendation as JsonObject } : {}), + confidence: result.confidence, + ...(result.rawText ? { rawText: result.rawText } : {}), + ...(result.model ? { model: asJsonObject(result.model) } : {}), + }, + traceId: run.id, + }; + } + + private async appendLifecycleEvent( + phase: "started" | "trace", + declaration: ThoughtAgentDeclaration, + event: ThoughtEvent, + run: AgentRun, + payload: JsonObject, + ): Promise { + await this.store.appendEvent(this.lifecycleCandidate(phase, declaration, event, run, new Date().toISOString(), payload)); + } + + private lifecycleCandidate( + phase: "started" | "trace" | "completed" | "failed" | "abandoned", + declaration: ThoughtAgentDeclaration, + event: ThoughtEvent, + run: AgentRun, + at: string, + payload: JsonObject, + ): EventCandidate { + const sequence = phase === "trace" && typeof payload.sequence === "number" ? `:${payload.sequence}` : ""; + return { + type: `stream.thought.agent.run.${phase}`, + schemaVersion: 1, + source: `agent:${declaration.id}`, + sourceKind: "agent", + externalId: run.id, + idempotencyKey: `${run.id}:${phase}${sequence}`, + occurredAt: at, + actor: declaration.id, rootEventId: event.rootEventId, + parentEventId: event.id, + correlationId: event.correlationId, privacy: event.privacy, payload: { - runId, + runId: run.id, + executionKey: run.executionKey, agentId: declaration.id, agentVersion: declaration.version, inputEventIds: [event.id], - attempt, - ...details, + attempt: run.attempt, + ...payload, }, - }); + traceId: run.id, + }; } } +function executionKey(event: ThoughtEvent, declaration: ThoughtAgentDeclaration): string { + return stableKey("execution", declaration.id, String(declaration.version), event.id); +} + +function progressId(declaration: ThoughtAgentDeclaration, source: string): string { + return stableKey("consumer-progress", declaration.id, String(declaration.version), source); +} + +function progressFor(declaration: ThoughtAgentDeclaration, event: ThoughtEvent, at: string): ConsumerProgress { + return { + id: progressId(declaration, event.source), + consumerId: declaration.id, + consumerVersion: declaration.version, + source: event.source, + lastSequence: event.sourceSequence, + lastEventId: event.id, + updatedAt: at, + }; +} + +function deterministicEventId(candidate: EventCandidate): string { + return `evt_${sha256(`${candidate.source}\u0000${candidate.idempotencyKey}`).slice(0, 48)}`; +} + +export function declarationExecutionFingerprint(declaration: ThoughtAgentDeclaration): string { + return sha256(canonicalJson({ + id: declaration.id, + version: declaration.version, + compiledEventTypes: declaration.compiledEventTypes, + sourcePatterns: declaration.sourcePatterns, + acceptedPrivacy: declaration.acceptedPrivacy, + promptHash: sha256(declaration.systemPrompt), + tools: declaration.tools, + })); +} + +function matchesPatternValue(pattern: string, value: string): boolean { + if (pattern === "*") return true; + if (pattern.endsWith("*")) return value.startsWith(pattern.slice(0, -1)); + return pattern === value; +} + function asJsonObject(value: unknown): JsonObject { const normalized = JSON.parse(JSON.stringify(value)) as unknown; - if (!normalized || typeof normalized !== "object" || Array.isArray(normalized)) { - throw new Error("Expected a JSON object"); - } + if (!normalized || typeof normalized !== "object" || Array.isArray(normalized)) throw new Error("Expected a JSON object"); return normalized as JsonObject; } diff --git a/src/agents/scheduler.ts b/src/agents/scheduler.ts deleted file mode 100644 index b84d8c2..0000000 --- a/src/agents/scheduler.ts +++ /dev/null @@ -1,82 +0,0 @@ -import { newId } from "../core/ids.js"; -import type { AgentRun, ThoughtStore } from "../store/types.js"; - -export interface LeaseOptions { - workerId: string; - leaseMs: number; - now?: Date; -} - -export interface RenewLeaseOptions extends LeaseOptions { - leaseId: string; -} - -export async function leaseRun(store: ThoughtStore, runId: string, options: LeaseOptions): Promise { - const run = await store.getRun(runId); - if (!run) throw new Error(`Run not found: ${runId}`); - const now = options.now ?? new Date(); - if (run.status !== "queued" && !hasExpiredLease(run, now)) { - throw new Error(`Run ${runId} is not leaseable from status ${run.status}`); - } - const leased: AgentRun = { - ...withoutLease(run), - status: "leased", - workerId: options.workerId, - leaseId: newId("lease"), - leaseExpiresAt: new Date(now.getTime() + options.leaseMs).toISOString(), - updatedAt: now.toISOString(), - }; - await store.upsertRun(leased); - return leased; -} - -export async function recoverExpiredRuns(store: ThoughtStore, now = new Date()): Promise { - const recovered: AgentRun[] = []; - for (const run of await store.listRuns()) { - if (!hasExpiredLease(run, now)) continue; - const queued: AgentRun = { - ...withoutLease(run), - status: "queued", - attempt: run.attempt + 1, - outputEventIds: [], - updatedAt: now.toISOString(), - }; - delete queued.startedAt; - delete queued.completedAt; - delete queued.errorText; - delete queued.result; - await store.upsertRun(queued); - recovered.push(queued); - } - return recovered; -} - -export async function renewLease(store: ThoughtStore, runId: string, options: RenewLeaseOptions): Promise { - const run = await store.getRun(runId); - if (!run) throw new Error(`Run not found: ${runId}`); - const now = options.now ?? new Date(); - if (run.status !== "leased" && run.status !== "running") { - throw new Error(`Run ${runId} cannot renew a lease from status ${run.status}`); - } - if (run.workerId !== options.workerId || run.leaseId !== options.leaseId) { - throw new Error(`Run ${runId} lease ownership changed`); - } - if (hasExpiredLease(run, now)) throw new Error(`Run ${runId} lease already expired`); - const renewed: AgentRun = { - ...run, - leaseExpiresAt: new Date(now.getTime() + options.leaseMs).toISOString(), - updatedAt: now.toISOString(), - }; - await store.upsertRun(renewed); - return renewed; -} - -export function hasExpiredLease(run: AgentRun, now = new Date()): boolean { - if (run.status !== "leased" && run.status !== "running") return false; - return Boolean(run.leaseExpiresAt && Date.parse(run.leaseExpiresAt) <= now.getTime()); -} - -function withoutLease(run: AgentRun): AgentRun { - const { workerId: _workerId, leaseId: _leaseId, leaseExpiresAt: _leaseExpiresAt, ...rest } = run; - return rest; -} diff --git a/src/agents/tools.ts b/src/agents/tools.ts new file mode 100644 index 0000000..d52da72 --- /dev/null +++ b/src/agents/tools.ts @@ -0,0 +1,409 @@ +import { createHash, randomUUID } from "node:crypto"; +import { BlockList, isIP } from "node:net"; +import { resolve4, resolve6 } from "node:dns/promises"; +import { mkdir, rename, rm, writeFile } from "node:fs/promises"; +import path from "node:path"; +import type { AgentTool } from "@earendil-works/pi-agent-core"; +import { Type } from "typebox"; +import type { ThoughtEvent } from "../events/types.js"; +import type { EnrichmentOutcome, RunnerTrace } from "./types.js"; + +export const AGENT_TOOL_NAMES = ["atproto.fetch-markdown", "web.download-image"] as const; +export type AgentToolName = (typeof AGENT_TOOL_NAMES)[number]; + +const MARKDOWN_MAX_BYTES = 256 * 1024; +const APPVIEW_MAX_BYTES = 512 * 1024; +const IMAGE_MAX_BYTES = 8 * 1024 * 1024; +const MAX_REDIRECTS = 3; +const nonPublicAddresses = createNonPublicBlockList(); + +type FetchLike = (input: string | URL | Request, init?: RequestInit) => Promise; +type ResolveHostname = (hostname: string) => Promise; + +export interface RunToolOptions { + event: ThoughtEvent; + names: AgentToolName[]; + artifactRoot?: string | undefined; + fetchImpl?: FetchLike | undefined; + resolveHostname?: ResolveHostname | undefined; + onTrace: (trace: RunnerTrace) => Promise; +} + +export interface RunToolSet { + tools: AgentTool[]; + outcomes: EnrichmentOutcome[]; +} + +export function createRunTools(options: RunToolOptions): RunToolSet { + const outcomes: EnrichmentOutcome[] = []; + const allowedImages = new Set(extractImageUrls(options.event.payload)); + const fetchImpl = options.fetchImpl ?? fetch; + const resolveHostname = options.resolveHostname ?? resolvePublicAddresses; + const tools = options.names.map((name) => { + if (name === "atproto.fetch-markdown") { + return createAtprotoMarkdownTool(options.event, allowedImages, outcomes, fetchImpl, resolveHostname, options.onTrace); + } + if (name === "web.download-image") { + return createImageDownloadTool( + allowedImages, + outcomes, + fetchImpl, + resolveHostname, + options.artifactRoot, + options.onTrace, + ); + } + return assertNever(name); + }); + return { tools, outcomes }; +} + +const markdownParameters = Type.Object({ + target: Type.Union([Type.Literal("event"), Type.Literal("subject")], { + description: "Fetch the current AT record, or the liked/reposted subject referenced by it.", + }), +}); + +function createAtprotoMarkdownTool( + event: ThoughtEvent, + allowedImages: Set, + outcomes: EnrichmentOutcome[], + fetchImpl: FetchLike, + resolveHostname: ResolveHostname, + onTrace: (trace: RunnerTrace) => Promise, +): AgentTool { + return { + name: "fetch_atproto_markdown", + label: "Fetch ATProto Markdown", + description: "Fetch clean public Markdown for the current AT record or its referenced subject from atproto.md.", + parameters: markdownParameters, + executionMode: "sequential", + execute: async (_toolCallId, params, signal) => { + const request = { target: params.target }; + try { + const atUri = resolveAtUri(event, params.target); + const endpoint = new URL(`https://atproto.md/${atUri}`); + await assertPublicHost(endpoint, resolveHostname, "ATProto Markdown endpoint"); + const response = await fetchImpl(endpoint, { + headers: { accept: "text/markdown" }, + redirect: "error", + ...(signal ? { signal } : {}), + }); + if (!response.ok) throw new Error(`atproto.md returned HTTP ${response.status}`); + const bytes = await readBoundedBody(response, MARKDOWN_MAX_BYTES, "ATProto Markdown"); + const markdown = bytes.toString("utf8"); + for (const url of extractMarkdownImageUrls(markdown)) allowedImages.add(url); + const imageResolution = await resolveBlueskyImageUrls(atUri, fetchImpl, resolveHostname, signal); + for (const url of imageResolution.imageUrls) allowedImages.add(url); + const enrichedMarkdown = imageResolution.imageUrls.length > 0 + ? `${markdown.trimEnd()}\n\n## Resolved image URLs\n${imageResolution.imageUrls.map((url) => `- ${url}`).join("\n")}\n` + : markdown; + const result = { + atUri, + endpoint: endpoint.toString(), + mediaType: response.headers.get("content-type")?.split(";", 1)[0] ?? "text/markdown", + sizeBytes: bytes.byteLength, + sha256: digest(bytes), + imageUrls: [...allowedImages], + imageResolution, + }; + await recordOutcome(outcomes, onTrace, { + tool: "atproto.fetch-markdown", + status: "succeeded", + request, + result, + }); + return { + content: [{ type: "text", text: enrichedMarkdown }], + details: result, + }; + } catch (error) { + await recordFailure(outcomes, onTrace, "atproto.fetch-markdown", request, error); + throw error; + } + }, + }; +} + +async function resolveBlueskyImageUrls( + atUri: string, + fetchImpl: FetchLike, + resolveHostname: ResolveHostname, + signal: AbortSignal | undefined, +): Promise<{ status: "succeeded" | "unavailable"; endpoint?: string; imageUrls: string[]; error?: string }> { + if (!/^at:\/\/[^/]+\/app\.bsky\.feed\.post\/[^/?#]+$/.test(atUri)) { + return { status: "unavailable", imageUrls: [], error: "Record is not a Bluesky feed post" }; + } + const endpoint = new URL("https://public.api.bsky.app/xrpc/app.bsky.feed.getPosts"); + endpoint.searchParams.append("uris", atUri); + try { + await assertPublicHost(endpoint, resolveHostname, "Bluesky AppView endpoint"); + const response = await fetchImpl(endpoint, { + headers: { accept: "application/json" }, + redirect: "error", + ...(signal ? { signal } : {}), + }); + if (!response.ok) return { status: "unavailable", endpoint: endpoint.toString(), imageUrls: [], error: `Bluesky AppView returned HTTP ${response.status}` }; + const bytes = await readBoundedBody(response, APPVIEW_MAX_BYTES, "Bluesky AppView response"); + const payload = JSON.parse(bytes.toString("utf8")) as unknown; + const posts = asRecord(payload)?.posts; + const imageUrls = Array.isArray(posts) + ? posts.flatMap((post) => extractImageUrls(asRecord(post)?.embed)) + : []; + return { status: "succeeded", endpoint: endpoint.toString(), imageUrls: [...new Set(imageUrls)] }; + } catch (error) { + return { + status: "unavailable", + endpoint: endpoint.toString(), + imageUrls: [], + error: error instanceof Error ? error.message : String(error), + }; + } +} + +const imageParameters = Type.Object({ + url: Type.String({ description: "An image URL returned by the ATProto Markdown tool or present in the source record." }), +}); + +function createImageDownloadTool( + allowedImages: Set, + outcomes: EnrichmentOutcome[], + fetchImpl: FetchLike, + resolveHostname: ResolveHostname, + artifactRoot: string | undefined, + onTrace: (trace: RunnerTrace) => Promise, +): AgentTool { + return { + name: "download_image", + label: "Download referenced image", + description: "Download one image already discovered in the current AT record or fetched Markdown and return it for inspection.", + parameters: imageParameters, + executionMode: "sequential", + execute: async (_toolCallId, params, signal) => { + const request = { url: params.url }; + try { + if (!allowedImages.has(params.url)) throw new Error("Image URL was not discovered in the current record or fetched Markdown"); + const { response, finalUrl } = await fetchPublic(params.url, fetchImpl, resolveHostname, signal); + if (!response.ok) throw new Error(`Image origin returned HTTP ${response.status}`); + const mediaType = response.headers.get("content-type")?.split(";", 1)[0]?.trim().toLowerCase() ?? ""; + if (!mediaType.startsWith("image/")) throw new Error(`Expected image content but received ${mediaType || "unknown content type"}`); + const bytes = await readBoundedBody(response, IMAGE_MAX_BYTES, "Image"); + const sha256 = digest(bytes); + const artifactPath = artifactRoot ? await writeArtifact(artifactRoot, sha256, bytes) : undefined; + const result = { + sourceUrl: params.url, + finalUrl, + mediaType, + sizeBytes: bytes.byteLength, + sha256, + ...(artifactPath ? { artifactPath } : {}), + }; + await recordOutcome(outcomes, onTrace, { + tool: "web.download-image", + status: "succeeded", + request, + result, + }); + return { + content: [ + { type: "text", text: JSON.stringify(result) }, + { type: "image", data: bytes.toString("base64"), mimeType: mediaType }, + ], + details: result, + }; + } catch (error) { + await recordFailure(outcomes, onTrace, "web.download-image", request, error); + throw error; + } + }, + }; +} + +function resolveAtUri(event: ThoughtEvent, target: "event" | "subject"): string { + const payload = event.payload as Record; + if (target === "event") { + const atUri = payload.atUri; + if (typeof atUri === "string" && isAtUri(atUri)) return atUri; + throw new Error("Current event has no canonical AT URI"); + } + const record = asRecord(payload.record); + const subject = asRecord(record?.subject); + const uri = subject?.uri; + if (typeof uri === "string" && isAtUri(uri)) return uri; + throw new Error("Current AT record has no referenced subject URI"); +} + +function isAtUri(value: string): boolean { + return /^at:\/\/[^/]+\/[a-zA-Z0-9.-]+\/[^/?#]+$/.test(value); +} + +async function fetchPublic( + initialUrl: string, + fetchImpl: FetchLike, + resolveHostname: ResolveHostname, + signal: AbortSignal | undefined, +): Promise<{ response: Response; finalUrl: string }> { + let current = initialUrl; + for (let redirects = 0; redirects <= MAX_REDIRECTS; redirects += 1) { + const url = new URL(current); + if (url.protocol !== "https:") throw new Error("Only HTTPS image URLs are allowed"); + await assertPublicHost(url, resolveHostname, "Image URL"); + const response = await fetchImpl(url, { + redirect: "manual", + headers: { accept: "image/*" }, + ...(signal ? { signal } : {}), + }); + if (![301, 302, 303, 307, 308].includes(response.status)) return { response, finalUrl: url.toString() }; + const location = response.headers.get("location"); + if (!location) throw new Error("Image redirect omitted its destination"); + current = new URL(location, url).toString(); + } + throw new Error(`Image exceeded ${MAX_REDIRECTS} redirects`); +} + +async function assertPublicHost(url: URL, resolveHostname: ResolveHostname, label: string): Promise { + if (url.protocol !== "https:") throw new Error(`${label} must use HTTPS`); + const addresses = await resolveHostname(url.hostname); + if (addresses.length === 0 || addresses.some(isPrivateAddress)) { + throw new Error(`${label} does not resolve exclusively to public addresses`); + } +} + +async function resolvePublicAddresses(hostname: string): Promise { + if (isIP(hostname)) return [hostname]; + const [ipv4, ipv6] = await Promise.all([ + resolve4(hostname).catch(() => []), + resolve6(hostname).catch(() => []), + ]); + return [...ipv4, ...ipv6]; +} + +function isPrivateAddress(address: string): boolean { + const normalized = address.toLowerCase(); + if (normalized.startsWith("::ffff:")) return isPrivateAddress(normalized.slice(7)); + const family = isIP(address); + if (family === 0) return true; + return nonPublicAddresses.check(address, family === 4 ? "ipv4" : "ipv6"); +} + +function createNonPublicBlockList(): BlockList { + const list = new BlockList(); + for (const [network, prefix] of [ + ["0.0.0.0", 8], ["10.0.0.0", 8], ["100.64.0.0", 10], ["127.0.0.0", 8], + ["169.254.0.0", 16], ["172.16.0.0", 12], ["192.0.0.0", 24], ["192.0.2.0", 24], + ["192.168.0.0", 16], ["198.18.0.0", 15], ["198.51.100.0", 24], ["203.0.113.0", 24], + ["224.0.0.0", 4], ["240.0.0.0", 4], + ] as const) list.addSubnet(network, prefix, "ipv4"); + for (const [network, prefix] of [ + ["::", 128], ["::1", 128], ["64:ff9b::", 96], ["100::", 64], + ["2001:db8::", 32], ["fc00::", 7], ["fe80::", 10], ["ff00::", 8], + ] as const) list.addSubnet(network, prefix, "ipv6"); + return list; +} + +async function readBoundedBody(response: Response, maxBytes: number, label: string): Promise { + const announced = Number(response.headers.get("content-length")); + if (Number.isFinite(announced) && announced > maxBytes) throw new Error(`${label} exceeds ${maxBytes} bytes`); + if (!response.body) return Buffer.alloc(0); + const reader = response.body.getReader(); + const chunks: Buffer[] = []; + let size = 0; + while (true) { + const { value, done } = await reader.read(); + if (done) break; + size += value.byteLength; + if (size > maxBytes) { + await reader.cancel(); + throw new Error(`${label} exceeds ${maxBytes} bytes`); + } + chunks.push(Buffer.from(value)); + } + return Buffer.concat(chunks, size); +} + +async function writeArtifact(root: string, sha256: string, bytes: Buffer): Promise { + const relative = path.join("sha256", sha256.slice(0, 2), sha256); + const destination = path.join(root, relative); + await mkdir(path.dirname(destination), { recursive: true }); + const temporary = `${destination}.${randomUUID()}.tmp`; + await writeFile(temporary, bytes, { flag: "wx" }); + try { + await rename(temporary, destination); + } finally { + await rm(temporary, { force: true }); + } + return relative; +} + +function extractImageUrls(value: unknown): string[] { + const urls = new Set(); + walk(value, undefined, urls); + return [...urls]; +} + +function walk(value: unknown, key: string | undefined, urls: Set): void { + if (typeof value === "string") { + if (key && /^(image|fullsize|thumb|avatar|banner)$/i.test(key) && isHttpsUrl(value)) urls.add(value); + return; + } + if (Array.isArray(value)) { + for (const item of value) walk(item, key, urls); + return; + } + if (!value || typeof value !== "object") return; + for (const [childKey, child] of Object.entries(value)) walk(child, childKey, urls); +} + +function extractMarkdownImageUrls(markdown: string): string[] { + const urls = new Set(); + const pattern = /!\[[^\]]*]\(\s]+)>?(?:\s+"[^"]*")?\)/g; + for (const match of markdown.matchAll(pattern)) { + const url = match[1]; + if (url && isHttpsUrl(url)) urls.add(url); + } + return [...urls]; +} + +function isHttpsUrl(value: string): boolean { + try { + return new URL(value).protocol === "https:"; + } catch { + return false; + } +} + +async function recordOutcome( + outcomes: EnrichmentOutcome[], + onTrace: (trace: RunnerTrace) => Promise, + outcome: EnrichmentOutcome, +): Promise { + outcomes.push(outcome); + await onTrace({ kind: `enrichment.${outcome.status}`, data: outcome }); +} + +async function recordFailure( + outcomes: EnrichmentOutcome[], + onTrace: (trace: RunnerTrace) => Promise, + tool: string, + request: Record, + error: unknown, +): Promise { + await recordOutcome(outcomes, onTrace, { + tool, + status: "failed", + request, + error: error instanceof Error ? error.message : String(error), + }); +} + +function digest(value: Uint8Array): string { + return createHash("sha256").update(value).digest("hex"); +} + +function asRecord(value: unknown): Record | undefined { + return value && typeof value === "object" && !Array.isArray(value) ? value as Record : undefined; +} + +function assertNever(value: never): never { + throw new Error(`Unsupported agent tool: ${String(value)}`); +} diff --git a/src/agents/types.ts b/src/agents/types.ts index 28f36a5..3d3ec1f 100644 --- a/src/agents/types.ts +++ b/src/agents/types.ts @@ -10,10 +10,12 @@ export interface ThoughtAgentDeclaration { description: string; mode: AgentMode; provider?: "tinker" | "openai-compatible" | undefined; + modelTier?: "triage-small" | "reasoning-small" | "escalation" | undefined; model?: string | undefined; baseUrl?: string | undefined; apiKeyEnv?: string | undefined; eventTypes: string[]; + compiledEventTypes: string[]; sourcePatterns: string[]; acceptedPrivacy: Array; outputEventType: string; @@ -50,7 +52,17 @@ export interface AgentOutput { model?: { provider: string; id: string; + revision?: string | undefined; } | undefined; + enrichments?: EnrichmentOutcome[] | undefined; +} + +export interface EnrichmentOutcome { + tool: string; + status: "succeeded" | "failed"; + request: Record; + result?: Record | undefined; + error?: string | undefined; } export interface RunnerTrace { @@ -59,6 +71,16 @@ export interface RunnerTrace { at?: string; } +export class AgentRunFailure extends Error { + readonly attemptedTranscript: string | undefined; + + constructor(message: string, options: { cause?: unknown; attemptedTranscript?: string } = {}) { + super(message, options.cause === undefined ? undefined : { cause: options.cause }); + this.name = "AgentRunFailure"; + this.attemptedTranscript = options.attemptedTranscript; + } +} + export interface AgentRunner { readonly mode: AgentMode; run(input: AgentRunInput, onTrace: (trace: RunnerTrace) => Promise): Promise; diff --git a/src/bridges/telegram-dispatcher.ts b/src/bridges/telegram-dispatcher.ts new file mode 100644 index 0000000..b747384 --- /dev/null +++ b/src/bridges/telegram-dispatcher.ts @@ -0,0 +1,427 @@ +import { stableKey } from "../core/ids.js"; +import { canonicalJson, sha256, type JsonObject } from "../core/json.js"; +import type { EventCandidate, ThoughtEvent } from "../events/types.js"; +import type { JazzThoughtStore } from "../jazz/store.js"; +import type { AgentRun, AgentRunStatus } from "../store/types.js"; +import { TelegramBotClient } from "../connectors/telegram-bot.js"; + +const RECEIPT_TYPES = [ + "stream.thought.action.telegram.send.started", + "stream.thought.action.telegram.send.delivered", + "stream.thought.action.telegram.send.failed", +]; + +export interface TelegramChannelDispatcherOptions { + id: string; + client: TelegramBotClient; + chatId: string; + allowedSources: string[]; + allowedActors?: string[]; + runStatuses?: Array>; + maxMessagesPerWindow?: number; + windowMs?: number; + maxLikesPerDigest?: number; + likeDigestDelayMs?: number; + failureTranscriptMaxChars?: number; +} + +export interface TelegramNotificationCycleOptions { + since?: string; + includeNormal?: boolean; + now?: Date; + signal?: AbortSignal; +} + +export interface TelegramNotificationCycleResult { + pending: number; + eligible: number; + delivered: number; + failed: number; + rateLimited: number; + deferredLikes: number; + availableMessages: number; + deliveries: Array<{ deliveryId: string; runIds: string[]; messageId?: string; error?: string }>; +} + +export interface TelegramOperationalNoticeResult { + status: "delivered" | "failed" | "rate-limited" | "already-claimed"; + deliveryId: string; + messageId?: string; + error?: string; +} + +interface NotificationCandidate { + run: AgentRun; + kind: "observation" | "failure"; + summary: string; + tags: string[]; + importance: string; + trigger: ThoughtEvent; + output?: ThoughtEvent; +} + +export class TelegramChannelDispatcher { + readonly id: string; + private readonly client: TelegramBotClient; + private readonly chatId: string; + private readonly allowedSources: Set; + private readonly allowedActors: Set; + private readonly runStatuses: Set>; + private readonly maxMessagesPerWindow: number; + private readonly windowMs: number; + private readonly maxLikesPerDigest: number; + private readonly likeDigestDelayMs: number; + private readonly failureTranscriptMaxChars: number; + + constructor(options: TelegramChannelDispatcherOptions) { + this.id = required(options.id, "Telegram channel dispatcher id"); + this.client = options.client; + this.chatId = required(options.chatId, "Telegram notifier chat id"); + this.allowedSources = new Set(options.allowedSources.map((source) => required(source, "Allowed notification source"))); + if (this.allowedSources.size === 0) throw new Error("Telegram channel dispatcher requires at least one allowed source"); + this.allowedActors = new Set((options.allowedActors ?? []).map((actor) => required(actor, "Allowed notification actor"))); + this.runStatuses = new Set(options.runStatuses ?? ["completed"]); + if (this.runStatuses.size === 0) throw new Error("Telegram channel dispatcher requires at least one run status"); + this.maxMessagesPerWindow = boundedPositiveInteger(options.maxMessagesPerWindow ?? 3, "maxMessagesPerWindow", 100); + this.windowMs = boundedPositiveInteger(options.windowMs ?? 60_000, "windowMs", 24 * 60 * 60 * 1_000); + this.maxLikesPerDigest = boundedPositiveInteger(options.maxLikesPerDigest ?? 10, "maxLikesPerDigest", 100); + this.likeDigestDelayMs = boundedNonnegativeInteger(options.likeDigestDelayMs ?? 60_000, "likeDigestDelayMs", 24 * 60 * 60 * 1_000); + this.failureTranscriptMaxChars = boundedPositiveInteger(options.failureTranscriptMaxChars ?? 3_000, "failureTranscriptMaxChars", 3_500); + } + + async activate(store: JazzThoughtStore, now = new Date()): Promise { + const activationId = stableKey("dispatcher-activation", this.id); + const activation = await store.appendEvent({ + type: "stream.thought.dispatcher.activated", + schemaVersion: 1, + source: this.id, + sourceKind: "system", + externalId: activationId, + idempotencyKey: "activation", + occurredAt: now.toISOString(), + actor: this.id, + correlationId: activationId, + privacy: "sensitive", + payload: { status: "activated" }, + }); + return activation.event.occurredAt; + } + + async sendPending( + store: JazzThoughtStore, + options: TelegramNotificationCycleOptions = {}, + ): Promise { + const now = options.now ?? new Date(); + const receipts = await store.listEvents({ types: RECEIPT_TYPES, source: this.id }); + const claimedRunIds = new Set(receipts + .filter((event) => event.type === "stream.thought.action.telegram.send.started") + .flatMap((event) => stringArray(event.payload.runIds))); + const recentDelivered = receipts.filter((event) => ( + event.type === "stream.thought.action.telegram.send.delivered" + && Date.parse(event.occurredAt) > now.getTime() - this.windowMs + )).length; + const availableMessages = Math.max(0, this.maxMessagesPerWindow - recentDelivered); + const pendingRuns = (await Promise.all([...this.runStatuses].map((status) => store.listRuns({ + status, + ...(options.since ? { completedAfter: options.since } : {}), + })))).flat() + .filter((run) => !claimedRunIds.has(run.id)) + .sort((left, right) => completedAt(left).localeCompare(completedAt(right)) || left.id.localeCompare(right.id)); + const candidates = (await Promise.all(pendingRuns.map((run) => this.notificationCandidate(store, run)))) + .filter((candidate): candidate is NotificationCandidate => candidate !== undefined) + .filter((candidate) => candidate.kind === "failure" || options.includeNormal || candidate.importance === "high"); + const likes = candidates + .filter((candidate) => candidate.tags.includes("like")) + .sort((left, right) => completedAt(left.run).localeCompare(completedAt(right.run))); + const nonLikes = candidates.filter((candidate) => !candidate.tags.includes("like")); + const likeBatchReady = likes.length > 0 + && now.getTime() - Date.parse(completedAt(likes[0]!.run)) >= this.likeDigestDelayMs; + const accumulatedLikes = likeBatchReady ? likes : []; + const groups: NotificationCandidate[][] = nonLikes.map((candidate) => [candidate]); + for (let index = 0; index < accumulatedLikes.length; index += this.maxLikesPerDigest) { + groups.push(accumulatedLikes.slice(index, index + this.maxLikesPerDigest)); + } + const selected = groups.slice(0, availableMessages); + const deliveries: TelegramNotificationCycleResult["deliveries"] = []; + let delivered = 0; + let failed = 0; + for (const group of selected) { + const runIds = group.map((candidate) => candidate.run.id); + const deliveryId = stableKey("telegram-delivery", this.id, this.chatId, ...runIds); + const at = now.toISOString(); + const messageKind = group[0]!.kind === "failure" + ? "agent-failure" + : group.every((candidate) => candidate.tags.includes("like")) ? "like-digest" : "observation"; + const claim = await store.appendEvent(this.receipt("started", deliveryId, group, at, { + status: "started", + runIds, + messageKind, + rateLimit: { + maximum: this.maxMessagesPerWindow, + windowMs: this.windowMs, + }, + })); + if (!claim.inserted) continue; + try { + const sent = await this.client.sendMessage(this.chatId, formatNotification(group, this.failureTranscriptMaxChars), options.signal); + await store.appendEvent(this.receipt("delivered", deliveryId, group, now.toISOString(), { + status: "delivered", + runIds, + messageKind, + chatId: sent.chatId, + messageId: sent.messageId, + })); + delivered += 1; + deliveries.push({ deliveryId, runIds, messageId: sent.messageId }); + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + await store.appendEvent(this.receipt("failed", deliveryId, group, now.toISOString(), { + status: "failed", + runIds, + messageKind, + error: message, + })); + failed += 1; + deliveries.push({ deliveryId, runIds, error: message }); + } + } + return { + pending: pendingRuns.length, + eligible: candidates.length, + delivered, + failed, + rateLimited: Math.max(0, groups.length - selected.length), + deferredLikes: likeBatchReady ? 0 : likes.length, + availableMessages, + deliveries, + }; + } + + async sendOperationalNotice( + store: JazzThoughtStore, + text: string, + options: { now?: Date; signal?: AbortSignal; scope?: string } = {}, + ): Promise { + const message = required(text, "Telegram operational notice"); + const now = options.now ?? new Date(); + const deliveryId = stableKey( + "telegram-operational-notice", + this.id, + sha256(canonicalJson({ chatId: this.chatId, text: message, scope: options.scope ?? "once" })), + ); + const receipts = await store.listEvents({ types: RECEIPT_TYPES, source: this.id }); + if (receipts.some((event) => ( + event.type === "stream.thought.action.telegram.send.started" + && event.externalId === deliveryId + ))) return { status: "already-claimed", deliveryId }; + const recentDelivered = receipts.filter((event) => ( + event.type === "stream.thought.action.telegram.send.delivered" + && Date.parse(event.occurredAt) > now.getTime() - this.windowMs + )).length; + if (recentDelivered >= this.maxMessagesPerWindow) return { status: "rate-limited", deliveryId }; + const at = now.toISOString(); + const notice = await store.appendEvent({ + type: "stream.thought.runtime.notice", + schemaVersion: 1, + source: this.id, + sourceKind: "system", + externalId: deliveryId, + idempotencyKey: `${deliveryId}:notice`, + occurredAt: at, + actor: this.id, + correlationId: deliveryId, + privacy: "sensitive", + payload: { text: message, destination: "telegram" }, + }); + const action = (phase: "started" | "delivered" | "failed", payload: JsonObject): EventCandidate => ({ + type: `stream.thought.action.telegram.send.${phase}`, + schemaVersion: 1, + source: this.id, + sourceKind: "system", + externalId: deliveryId, + idempotencyKey: `${deliveryId}:${phase}`, + occurredAt: at, + actor: this.id, + rootEventId: notice.event.rootEventId, + parentEventId: notice.event.id, + correlationId: deliveryId, + privacy: "sensitive", + payload, + }); + const claim = await store.appendEvent(action("started", { + status: "started", + runIds: [], + messageKind: "operational-notice", + rateLimit: { maximum: this.maxMessagesPerWindow, windowMs: this.windowMs }, + })); + if (!claim.inserted) return { status: "already-claimed", deliveryId }; + try { + const sent = await this.client.sendMessage(this.chatId, message, options.signal); + await store.appendEvent(action("delivered", { + status: "delivered", + runIds: [], + chatId: sent.chatId, + messageId: sent.messageId, + })); + return { status: "delivered", deliveryId, messageId: sent.messageId }; + } catch (error) { + const failure = error instanceof Error ? error.message : String(error); + await store.appendEvent(action("failed", { status: "failed", runIds: [], error: failure })); + return { status: "failed", deliveryId, error: failure }; + } + } + + private async notificationCandidate(store: JazzThoughtStore, run: AgentRun): Promise { + const trigger = await store.getEvent(run.triggerEventId); + if (!trigger) return undefined; + if (!this.allowedSources.has(trigger.source)) return undefined; + if (this.allowedActors.size > 0 && !this.allowedActors.has(trigger.actor)) return undefined; + if (run.status === "failed") { + const attemptedTranscript = await this.attemptedTranscript(store, run); + return { + run, + kind: "failure", + summary: attemptedTranscript, + tags: ["failure"], + importance: "high", + trigger, + }; + } + const summary = run.result?.summary; + if (typeof summary !== "string" || summary.trim().length === 0) return undefined; + const output = run.outputEventIds[0] ? await store.getEvent(run.outputEventIds[0]) : undefined; + return { + run, + kind: "observation", + summary: summary.trim(), + tags: stringArray(run.result?.tags), + importance: typeof run.result?.importance === "string" ? run.result.importance : "normal", + trigger, + ...(output ? { output } : {}), + }; + } + + private async attemptedTranscript(store: JazzThoughtStore, run: AgentRun): Promise { + if (typeof run.result?.attemptedTranscript === "string") return run.result.attemptedTranscript.trim(); + const traces = await store.listTrace(run.id); + for (const trace of traces.reverse()) { + if (trace.type !== "pi.message_end") continue; + const data = asRecord(trace.payload.data); + const message = asRecord(data?.message); + if (message?.role !== "assistant" || !Array.isArray(message.content)) continue; + const parts = message.content.flatMap((part) => { + const content = asRecord(part); + if (content?.type === "text" && typeof content.text === "string") return [content.text]; + if (content?.type === "thinking" && typeof content.thinking === "string") return [content.thinking]; + if (content?.type === "toolCall" && typeof content.name === "string") { + return [`[tool ${content.name}]\n${JSON.stringify(content.arguments ?? {})}`]; + } + return []; + }).join("\n\n").trim(); + if (parts) return parts; + } + return ""; + } + + private receipt( + phase: "started" | "delivered" | "failed", + deliveryId: string, + group: NotificationCandidate[], + at: string, + payload: JsonObject, + ): EventCandidate { + const first = group[0]!; + return { + type: `stream.thought.action.telegram.send.${phase}`, + schemaVersion: 1, + source: this.id, + sourceKind: "system", + externalId: deliveryId, + idempotencyKey: `${deliveryId}:${phase}`, + occurredAt: at, + actor: this.id, + rootEventId: first.trigger.rootEventId, + parentEventId: first.output?.id ?? first.trigger.id, + correlationId: deliveryId, + privacy: "sensitive", + payload, + }; + } +} + +function formatNotification(group: NotificationCandidate[], failureTranscriptMaxChars: number): string { + if (group[0]!.kind === "failure") { + const candidate = group[0]!; + const transcript = candidate.summary + ? truncate(candidate.summary, failureTranscriptMaxChars) + : "[No model transcript was captured before failure.]"; + return truncate([ + "ThoughtStream · Agent run failed", + "", + transcript, + "", + `Error: ${candidate.run.errorText ?? "Unknown failure"}`, + `${candidate.trigger.source} → ${candidate.run.agentId}`, + `receipt ${shortReceipt(candidate.run.id)}`, + ].join("\n"), 4_096); + } + if (group.every((candidate) => candidate.tags.includes("like"))) { + const lines = group.map((candidate) => `• ${candidate.summary.replace(/^\w+ like:\s*/i, "")}`); + return [ + `ThoughtStream · ${group.length} Bluesky ${group.length === 1 ? "like" : "likes"}`, + "", + ...lines, + "", + `via ${group[0]!.run.agentId} · ${shortReceipt(group[0]!.run.id)}`, + ].join("\n"); + } + const candidate = group[0]!; + const label = candidate.tags.includes("post") ? "Bluesky post" : "Observation"; + return [ + `ThoughtStream · ${label}`, + "", + candidate.summary, + "", + `${candidate.trigger.source} → ${candidate.run.agentId}`, + `receipt ${shortReceipt(candidate.run.id)}`, + ].join("\n"); +} + +function truncate(value: string, maximum: number): string { + if (value.length <= maximum) return value; + return `${value.slice(0, Math.max(0, maximum - 22))}\n[transcript truncated]`; +} + +function shortReceipt(runId: string): string { + return runId.length <= 12 ? runId : runId.slice(-12); +} + +function completedAt(run: AgentRun): string { + return run.completedAt ?? run.updatedAt; +} + +function stringArray(value: unknown): string[] { + return Array.isArray(value) ? value.filter((item): item is string => typeof item === "string") : []; +} + +function asRecord(value: unknown): Record | undefined { + return value && typeof value === "object" && !Array.isArray(value) + ? value as Record + : undefined; +} + +function boundedPositiveInteger(value: number, label: string, maximum: number): number { + if (!Number.isSafeInteger(value) || value <= 0 || value > maximum) throw new Error(`${label} must be a positive integer <= ${maximum}`); + return value; +} + +function boundedNonnegativeInteger(value: number, label: string, maximum: number): number { + if (!Number.isSafeInteger(value) || value < 0 || value > maximum) throw new Error(`${label} must be a nonnegative integer <= ${maximum}`); + return value; +} + +function required(value: string, label: string): string { + const normalized = value.trim(); + if (!normalized) throw new Error(`${label} is required`); + return normalized; +} diff --git a/src/cli.ts b/src/cli.ts index 5f4972d..517b094 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -4,15 +4,23 @@ import path from "node:path"; import chokidar from "chokidar"; import { loadAgentDeclarations } from "./agents/declarations.js"; import { ThoughtAgentRuntime } from "./agents/runtime.js"; +import { TelegramChannelDispatcher } from "./bridges/telegram-dispatcher.js"; import { FilesystemConnector } from "./connectors/filesystem.js"; import { FastmailConnector } from "./connectors/fastmail.js"; import { subscribeJetstream } from "./connectors/jetstream-live.js"; import { JetstreamConnector } from "./connectors/jetstream.js"; import { RssConnector } from "./connectors/rss.js"; +import { TelegramBotClient, TelegramBotConnector } from "./connectors/telegram-bot.js"; import { TelegramSpoolConnector } from "./connectors/telegram-spool.js"; import { JazzThoughtStore } from "./jazz/store.js"; -import { runFilesystemPipeline } from "./pipeline/filesystem.js"; import { rebuildRootActivity } from "./projections/activity.js"; +import { loadThoughtStreamManifest, type TelegramBotSourceManifest } from "./runtime/manifest.js"; +import { + projectTrainingExamples, + recordJudgment, + writeTrainingJsonl, + type JudgmentKind, +} from "./training/judgments.js"; import { startInspectorServer } from "./web/inspector.js"; const projectRoot = process.env.THOUGHTSTREAM_ROOT ?? process.cwd(); @@ -26,11 +34,30 @@ let commandFailed = false; try { if (command === "scan") { + const root = path.resolve(valueAfter("--root") ?? path.join(projectRoot, "fixtures", "vault")); + const source = valueAfter("--source") ?? "filesystem:fixture"; + const connector = new FilesystemConnector({ id: source, root }); + print({ scan: await connector.scan(store) }); + } else if (command === "demo") { const root = path.resolve(valueAfter("--root") ?? path.join(projectRoot, "fixtures", "vault")); const source = valueAfter("--source") ?? "filesystem:fixture"; const connector = new FilesystemConnector({ id: source, root }); const declarations = await loadAgentDeclarations(path.join(projectRoot, "agents")); - print(await runFilesystemPipeline(store, connector, declarations)); + const runtime = new ThoughtAgentRuntime(store); + const consumers = await runtime.startConsumers(declarations); + try { + const scan = await connector.scan(store); + const finalSequence = Math.max(0, ...scan.events.map((event) => event.sourceSequence)); + if (finalSequence > 0) { + await waitUntil(async () => (await store.listConsumerProgress()).some((progress) => ( + progress.source === source && progress.lastSequence >= finalSequence + ))); + } + await consumers.drain(); + print({ scan, runs: await store.listRuns(), activity: await rebuildRootActivity(store) }); + } finally { + await consumers.stop(); + } } else if (command === "watch") { const root = path.resolve(valueAfter("--root") ?? path.join(projectRoot, "fixtures", "vault")); const source = valueAfter("--source") ?? "filesystem:fixture"; @@ -38,6 +65,8 @@ try { if (!Number.isFinite(debounceMs) || debounceMs < 0) throw new Error("--debounce must be a nonnegative number"); const connector = new FilesystemConnector({ id: source, root }); const declarations = await loadAgentDeclarations(path.join(projectRoot, "agents")); + const runtime = new ThoughtAgentRuntime(store); + const consumers = await runtime.startConsumers(declarations); const watcher = chokidar.watch(root, { ignoreInitial: true, followSymlinks: false, @@ -58,8 +87,8 @@ try { running = true; activeCycle = (async () => { try { - const result = await runFilesystemPipeline(store, connector, declarations); - if (result.scan.events.length > 0) print(result); + const scan = await connector.scan(store); + if (scan.events.length > 0) print({ scan }); } catch (error) { process.stderr.write(`thought stream watch cycle failed: ${error instanceof Error ? error.message : String(error)}\n`); } finally { @@ -84,7 +113,7 @@ try { }); running = true; try { - print(await runFilesystemPipeline(store, connector, declarations)); + print({ scan: await connector.scan(store) }); } finally { running = false; if (pending) schedule(); @@ -96,6 +125,8 @@ try { if (timer) clearTimeout(timer); const cycle = activeCycle; if (cycle) await cycle; + await consumers.drain(); + await consumers.stop(); watcher.removeAllListeners(); await watcher.unwatch(root); await watcher.close(); @@ -106,11 +137,7 @@ try { const source = valueAfter("--source") ?? `rss:${new URL(feedUrl).hostname}`; const connector = new RssConnector({ id: source, feedUrl }); const poll = await connector.poll(store); - const declarations = await loadAgentDeclarations(path.join(projectRoot, "agents")); - const runtime = new ThoughtAgentRuntime(store); - const agentResults = []; - for (const event of poll.events) agentResults.push(...await runtime.processEvent(event, declarations)); - print({ poll, agentResults }); + print({ poll }); } else if (command === "jetstream") { const source = valueAfter("--source"); const collections = commaSeparated(valueAfter("--collections")); @@ -125,7 +152,9 @@ try { const connector = new JetstreamConnector({ id: source, collections, dids }); const declarations = await loadAgentDeclarations(path.join(projectRoot, "agents")); const runtime = new ThoughtAgentRuntime(store); - let agentRuns = 0; + const runsBefore = (await store.listRuns()).length; + const consumers = await runtime.startConsumers(declarations); + let consumersStopped = false; const controller = new AbortController(); const abort = () => controller.abort(); const timer = setTimeout(abort, maxRuntimeSeconds * 1_000); @@ -138,12 +167,17 @@ try { rewindUs, maxMessages, maxReconnects, - onEvent: async (event) => { - agentRuns += (await runtime.processEvent(event, declarations)).length; - }, }); - print({ subscription, agentRuns }); + // A bounded subscription can finish before a wildcard consumer's source-discovery + // callback has attached to a source that did not exist at startup. Reconcile the + // durable backlog after stopping subscriptions, avoiding concurrent transactions + // while still preserving any work completed by the live consumer. + await consumers.stop(); + consumersStopped = true; + await runtime.consumeBacklog(declarations); + print({ subscription, consumerRuns: (await store.listRuns()).length - runsBefore }); } finally { + if (!consumersStopped) await consumers.stop(); clearTimeout(timer); process.removeListener("SIGINT", abort); process.removeListener("SIGTERM", abort); @@ -158,11 +192,139 @@ try { const maxReadBytes = positiveInteger(valueAfter("--max-read-bytes") ?? "8388608", "--max-read-bytes", 64 * 1024 * 1024); const connector = new TelegramSpoolConnector({ id: source, file, maxRecords, maxReadBytes }); const ingest = await connector.ingest(store); - const declarations = await loadAgentDeclarations(path.join(projectRoot, "agents")); - const runtime = new ThoughtAgentRuntime(store); - const agentResults = []; - for (const event of ingest.events) agentResults.push(...await runtime.processEvent(event, declarations)); - print({ ingest, agentResults }); + print({ ingest }); + } else if (command === "telegram-bot") { + const sourceConfig = await telegramSourceConfig(); + const channels = sourceConfig.channels.filter((channel) => channel.enabled); + const chatIds = channels.map((channel) => channel.id); + const token = process.env[sourceConfig.tokenEnv]; + if (!token) throw new Error(`Telegram bot credential environment variable is unavailable: ${sourceConfig.tokenEnv}`); + const source = sourceConfig.id; + const maxRuntimeSeconds = positiveInteger(valueAfter("--max-runtime") ?? "300", "--max-runtime", 86_400); + const pollTimeoutSeconds = sourceConfig.pollTimeoutSeconds; + const client = new TelegramBotClient({ + token, + ...(process.env.THOUGHTSTREAM_TELEGRAM_API_BASE_URL ? { baseUrl: process.env.THOUGHTSTREAM_TELEGRAM_API_BASE_URL } : {}), + requestTimeoutMs: sourceConfig.requestTimeoutMs, + }); + const connector = new TelegramBotConnector({ id: source, client, allowedChatIds: chatIds }); + const deadline = Date.now() + maxRuntimeSeconds * 1_000; + let stopping = false; + const stop = () => { stopping = true; }; + process.once("SIGINT", stop); + process.once("SIGTERM", stop); + let polls = 0; + let accepted = 0; + try { + do { + const poll = await connector.poll(store, { timeoutSeconds: pollTimeoutSeconds }); + polls += 1; + accepted += poll.accepted; + if (poll.accepted > 0 || poll.ignored > 0) print({ poll }); + } while (!stopping && !process.argv.includes("--once") && Date.now() < deadline); + print({ + telegramBot: { + source, + polls, + accepted, + stopped: stopping ? "signal" : process.argv.includes("--once") ? "once" : "runtime-limit", + }, + }); + } finally { + process.removeListener("SIGINT", stop); + process.removeListener("SIGTERM", stop); + } + } else if (command === "telegram-dispatcher") { + const sourceConfig = await telegramSourceConfig(); + const token = process.env[sourceConfig.tokenEnv]; + if (!token) throw new Error(`Telegram bot credential environment variable is unavailable: ${sourceConfig.tokenEnv}`); + const client = new TelegramBotClient({ + token, + ...(process.env.THOUGHTSTREAM_TELEGRAM_API_BASE_URL ? { baseUrl: process.env.THOUGHTSTREAM_TELEGRAM_API_BASE_URL } : {}), + requestTimeoutMs: sourceConfig.requestTimeoutMs, + }); + const startedAt = new Date().toISOString(); + const channels = sourceConfig.channels.filter((channel) => channel.enabled); + const channelRuntimes = channels.map((channel) => { + const notification = channel.notifications; + const dispatcher = channel.bootMessage?.enabled || notification?.enabled + ? new TelegramChannelDispatcher({ + id: `telegram-dispatcher:${sourceConfig.id}:${channel.id}`, + client, + chatId: channel.id, + allowedSources: notification?.allowedSources ?? ["system:boot"], + ...(notification?.allowedActors ? { allowedActors: notification.allowedActors } : {}), + runStatuses: notification?.runStatuses ?? ["completed"], + maxMessagesPerWindow: notification?.maxMessagesPerWindow ?? 3, + windowMs: notification?.windowMs ?? 60_000, + maxLikesPerDigest: notification?.maxLikesPerDigest ?? 10, + likeDigestDelayMs: notification?.likeDigestDelayMs ?? 60_000, + failureTranscriptMaxChars: notification?.failureTranscriptMaxChars ?? 3_000, + }) + : undefined; + return { channel, notification, dispatcher }; + }); + const activeSince = new Map(); + for (const runtime of channelRuntimes) { + if (!runtime.dispatcher || !runtime.notification?.enabled) continue; + activeSince.set(runtime.channel.id, await runtime.dispatcher.activate(store)); + } + const maxRuntimeSeconds = positiveInteger(valueAfter("--max-runtime") ?? "300", "--max-runtime", 86_400); + const deadline = Date.now() + maxRuntimeSeconds * 1_000; + let stopping = false; + const stop = () => { stopping = true; }; + process.once("SIGINT", stop); + process.once("SIGTERM", stop); + let cycles = 0; + let delivered = 0; + let failed = 0; + const bootedChannels = new Set(channelRuntimes + .filter((runtime) => !runtime.channel.bootMessage?.enabled) + .map((runtime) => runtime.channel.id)); + try { + do { + cycles += 1; + const bootMessages = []; + for (const runtime of channelRuntimes) { + if (!runtime.dispatcher || !runtime.channel.bootMessage?.enabled || bootedChannels.has(runtime.channel.id)) continue; + const result = await runtime.dispatcher.sendOperationalNotice(store, runtime.channel.bootMessage.text, { + scope: startedAt, + }); + bootMessages.push({ channelId: runtime.channel.id, ...result }); + if (result.status === "delivered") delivered += 1; + if (result.status === "failed") failed += 1; + if (result.status !== "rate-limited") bootedChannels.add(runtime.channel.id); + } + const notifications = []; + for (const runtime of channelRuntimes) { + if (!runtime.dispatcher || !runtime.notification?.enabled || !bootedChannels.has(runtime.channel.id)) continue; + const result = await runtime.dispatcher.sendPending(store, { + ...(process.argv.includes("--notify-existing") ? {} : { since: activeSince.get(runtime.channel.id)! }), + includeNormal: runtime.notification.includeNormal, + }); + notifications.push({ channelId: runtime.channel.id, ...result }); + delivered += result.delivered; + failed += result.failed; + } + if (bootMessages.length > 0 || notifications.some((result) => result.delivered || result.failed)) { + print({ ...(bootMessages.length > 0 ? { bootMessages } : {}), ...(notifications.length > 0 ? { notifications } : {}) }); + } + if (!stopping && !process.argv.includes("--once") && Date.now() < deadline) { + await delay(sourceConfig.dispatchIntervalMs); + } + } while (!stopping && !process.argv.includes("--once") && Date.now() < deadline); + print({ + telegramDispatcher: { + source: sourceConfig.id, + cycles, + notifications: { delivered, failed }, + stopped: stopping ? "signal" : process.argv.includes("--once") ? "once" : "runtime-limit", + }, + }); + } finally { + process.removeListener("SIGINT", stop); + process.removeListener("SIGTERM", stop); + } } else if (command === "fastmail-capture") { const file = valueAfter("--file"); const source = valueAfter("--source"); @@ -173,11 +335,22 @@ try { const raw = JSON.parse(await fs.readFile(path.resolve(file), "utf8")) as unknown; const connector = new FastmailConnector({ id: source, accountId }); const ingest = await connector.ingestCapture(store, raw); + print({ ingest }); + } else if (command === "consume") { const declarations = await loadAgentDeclarations(path.join(projectRoot, "agents")); const runtime = new ThoughtAgentRuntime(store); - const agentResults = []; - for (const event of ingest.events) agentResults.push(...await runtime.processEvent(event, declarations)); - print({ ingest, agentResults }); + if (process.argv.includes("--once")) { + print({ runs: await runtime.consumeBacklog(declarations) }); + } else { + const consumers = await runtime.startConsumers(declarations); + process.stdout.write("thought stream consumers following Jazz subscriptions.\n"); + try { + await waitForSignal(); + } finally { + await consumers.stop(); + } + process.stderr.write("thought stream consumers stopped.\n"); + } } else if (command === "events") { const limit = Number(valueAfter("--limit") ?? "100"); print(await store.listEvents({ limit })); @@ -195,6 +368,36 @@ try { const run = await store.getRun(id); if (!run) throw new Error(`Run not found: ${id}`); print({ run, trace: await store.listTrace(id) }); + } else if (command === "judgment") { + const runId = process.argv[3]; + const kind = valueAfter("--kind") as JudgmentKind | undefined; + if (!runId || !kind || !["accept", "reject", "correct", "prefer"].includes(kind)) { + throw new Error("Usage: thought stream judgment --kind [--criterion output-quality] [--criterion-version 1] [--compared-run ] [--replacement ] [--export-eligible] [--notes ]"); + } + const replacementPath = valueAfter("--replacement"); + const replacementOutput = replacementPath + ? jsonObject(JSON.parse(await fs.readFile(path.resolve(replacementPath), "utf8")), "--replacement") + : undefined; + const judgment = await recordJudgment(store, { + runId, + kind, + criterion: valueAfter("--criterion") ?? "output-quality", + criterionVersion: positiveInteger(valueAfter("--criterion-version") ?? "1", "--criterion-version", 1_000_000), + exportEligible: process.argv.includes("--export-eligible"), + ...(valueAfter("--compared-run") ? { comparedRunId: valueAfter("--compared-run")! } : {}), + ...(replacementOutput ? { replacementOutput } : {}), + ...(valueAfter("--notes") ? { notes: valueAfter("--notes")! } : {}), + }); + print({ judgment }); + } else if (command === "training-export") { + const examples = await projectTrainingExamples(store); + const output = valueAfter("--output"); + if (output) { + const manifest = await writeTrainingJsonl(output, examples); + print({ trainingExport: { output: path.resolve(output), manifest: `${path.resolve(output)}.manifest.json`, ...manifest } }); + } else { + for (const example of examples) process.stdout.write(`${JSON.stringify(example)}\n`); + } } else if (command === "status") { const events = await store.listEvents(); const runs = await store.listRuns(); @@ -236,6 +439,38 @@ function commaSeparated(value: string | undefined): string[] { return value ? value.split(",").map((item) => item.trim()).filter(Boolean) : []; } +function jsonObject(value: unknown, flag: string): Record { + if (!value || typeof value !== "object" || Array.isArray(value)) throw new Error(`${flag} must contain a JSON object`); + return value as Record; +} + +async function telegramSourceConfig(): Promise { + const loaded = await loadThoughtStreamManifest(projectRoot, valueAfter("--config") ?? "thoughtstream.yaml"); + const requestedSource = valueAfter("--source"); + const source = loaded.manifest.sources.find((candidate) => ( + candidate.kind === "telegram-bot" + && (requestedSource ? candidate.id === requestedSource : candidate.enabled) + )); + if (!source || source.kind !== "telegram-bot") { + throw new Error("No matching telegram-bot source in the ThoughtStream manifest"); + } + if (!source.enabled) throw new Error(`Telegram bot source is disabled: ${source.id}`); + return source; +} + +async function delay(milliseconds: number): Promise { + await new Promise((resolve) => setTimeout(resolve, milliseconds)); +} + +async function waitUntil(predicate: () => Promise, timeoutMs = 5_000): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + if (await predicate()) return; + await new Promise((resolve) => setTimeout(resolve, 10)); + } + throw new Error("Timed out waiting for consumer subscription progress"); +} + function positiveInteger(value: string, flag: string, maximum: number): number { const parsed = Number(value); if (!Number.isSafeInteger(parsed) || parsed <= 0 || parsed > maximum) { diff --git a/src/connectors/fastmail.ts b/src/connectors/fastmail.ts index 83dc68a..73807fd 100644 --- a/src/connectors/fastmail.ts +++ b/src/connectors/fastmail.ts @@ -1,7 +1,8 @@ import { canonicalJson, sha256, type JsonObject } from "../core/json.js"; import { newId } from "../core/ids.js"; import type { EventCandidate, ThoughtEvent } from "../events/types.js"; -import type { SourceCursor, ThoughtStore } from "../store/types.js"; +import type { JazzThoughtStore } from "../jazz/store.js"; +import type { SourceCursor } from "../store/types.js"; import { z } from "zod"; const addressSchema = z.object({ @@ -118,7 +119,7 @@ export class FastmailConnector { }; } - async ingestCapture(store: ThoughtStore, raw: unknown): Promise { + async ingestCapture(store: JazzThoughtStore, raw: unknown): Promise { const cursorId = `cursor:${this.id}`; const prior = await store.getSourceCursor(cursorId); const correlationId = newId("fastmail_capture"); @@ -137,15 +138,13 @@ export class FastmailConnector { ...capture.changes.updated.map((id) => ({ id, operation: "updated" as const })), ...capture.changes.destroyed.map((id) => ({ id, operation: "destroyed" as const })), ]; - const events: ThoughtEvent[] = []; - let inserted = 0; - let unchanged = 0; + const candidates: EventCandidate[] = []; for (const item of operations) { const email = item.operation === "destroyed" ? undefined : capture.emails.get(item.id); if (item.operation !== "destroyed" && !email) { throw new Error(`JMAP Email/get response is missing changed email ${item.id}`); } - const appended = await store.appendEvent(this.eventCandidate( + candidates.push(this.eventCandidate( item.id, item.operation, capture.changes.newState, @@ -153,14 +152,7 @@ export class FastmailConnector { correlationId, startedAt, )); - if (appended.inserted) { - inserted += 1; - events.push(appended.event); - } else { - unchanged += 1; - } } - await store.flush(); const completedAt = new Date().toISOString(); const cursor: SourceCursor = { @@ -175,23 +167,27 @@ export class FastmailConnector { ...(prior?.lastFailureAt ? { lastFailureAt: prior.lastFailureAt } : {}), updatedAt: completedAt, }; - await store.upsertSourceCursor(cursor); - await this.appendCursorEvent(store, correlationId, completedAt, cursor); + const sourceCount = candidates.length; + candidates.push(this.cursorEvent(correlationId, completedAt, cursor)); if (prior?.lastFailureAt && (!prior.lastSuccessAt || prior.lastFailureAt > prior.lastSuccessAt)) { - await this.appendConnectorEvent(store, "recovered", correlationId, completedAt, { + candidates.push(this.connectorEvent("recovered", correlationId, completedAt, { status: "recovered", previousFailureAt: prior.lastFailureAt, - }); + })); } - await this.appendConnectorEvent(store, "completed", correlationId, completedAt, { + candidates.push(this.connectorEvent("completed", correlationId, completedAt, { status: "updated", - inserted, - unchanged, created: capture.changes.created.length, updated: capture.changes.updated.length, destroyed: capture.changes.destroyed.length, hasMoreChanges: capture.changes.hasMoreChanges, - }); + })); + const batch = await store.appendProducerBatch(candidates, cursor); + const offeredEvents = batch.events.slice(0, sourceCount); + const insertedIds = new Set(batch.inserted.map((event) => event.id)); + const events = offeredEvents.filter((event) => insertedIds.has(event.id)); + const inserted = events.length; + const unchanged = sourceCount - inserted; return { inserted, unchanged, @@ -205,13 +201,7 @@ export class FastmailConnector { } catch (error) { const failedAt = new Date().toISOString(); const message = error instanceof Error ? error.message : String(error); - await this.appendConnectorEvent(store, "failed", correlationId, failedAt, { - status: "failed", - error: message, - phase: "captured-jmap-ingest", - accountIdHash: sha256(this.accountId), - }); - await store.upsertSourceCursor({ + const failedCursor: SourceCursor = { id: cursorId, source: this.id, cursor: prior?.cursor ?? { accountIdHash: sha256(this.accountId) }, @@ -219,7 +209,13 @@ export class FastmailConnector { lastFailureAt: failedAt, lastError: message, updatedAt: failedAt, - }); + }; + await store.appendProducerBatch([this.connectorEvent("failed", correlationId, failedAt, { + status: "failed", + error: message, + phase: "captured-jmap-ingest", + accountIdHash: sha256(this.accountId), + })], failedCursor); throw error; } } @@ -289,8 +285,8 @@ export class FastmailConnector { }; } - private async appendCursorEvent(store: ThoughtStore, correlationId: string, at: string, cursor: SourceCursor): Promise { - await store.appendEvent({ + private cursorEvent(correlationId: string, at: string, cursor: SourceCursor): EventCandidate { + return { type: "stream.thought.connector.cursor.advanced", schemaVersion: 1, source: this.id, @@ -302,22 +298,21 @@ export class FastmailConnector { correlationId, privacy: "sensitive", payload: { source: this.id, cursorId: cursor.id, cursor: cursor.cursor }, - }); + }; } - private async appendConnectorEvent( - store: ThoughtStore, + private connectorEvent( phase: "started" | "completed" | "failed" | "recovered", correlationId: string, at: string, payload: JsonObject, - ): Promise { + ): EventCandidate { const type = phase === "failed" ? "stream.thought.connector.failed" : phase === "recovered" ? "stream.thought.connector.recovered" : `stream.thought.connector.poll.${phase}`; - await store.appendEvent({ + return { type, schemaVersion: 1, source: this.id, @@ -329,7 +324,17 @@ export class FastmailConnector { correlationId, privacy: "sensitive", payload, - }); + }; + } + + private async appendConnectorEvent( + store: JazzThoughtStore, + phase: "started" | "completed" | "failed" | "recovered", + correlationId: string, + at: string, + payload: JsonObject, + ): Promise { + await store.appendEvent(this.connectorEvent(phase, correlationId, at, payload)); } } diff --git a/src/connectors/filesystem.ts b/src/connectors/filesystem.ts index fd9485b..1afef6d 100644 --- a/src/connectors/filesystem.ts +++ b/src/connectors/filesystem.ts @@ -6,7 +6,7 @@ import YAML from "yaml"; import { newId, stableKey } from "../core/ids.js"; import { sha256 } from "../core/json.js"; import type { PrivacyClass, ThoughtEvent } from "../events/types.js"; -import type { ThoughtStore } from "../store/types.js"; +import type { JazzThoughtStore } from "../jazz/store.js"; export interface FilesystemConnectorConfig { id: string; @@ -78,7 +78,7 @@ export class FilesystemConnector { this.storeContent = config.storeContent ?? true; } - async scan(store: ThoughtStore): Promise { + async scan(store: JazzThoughtStore): Promise { const root = await fs.realpath(this.config.root); const correlationId = newId("scan"); const paths = await fg(this.include, { diff --git a/src/connectors/jetstream-live.ts b/src/connectors/jetstream-live.ts index 22505c9..e4b2b97 100644 --- a/src/connectors/jetstream-live.ts +++ b/src/connectors/jetstream-live.ts @@ -1,7 +1,7 @@ import { newId } from "../core/ids.js"; import type { JsonObject } from "../core/json.js"; -import type { ThoughtEvent } from "../events/types.js"; -import type { SourceCursor, ThoughtStore } from "../store/types.js"; +import type { JazzThoughtStore } from "../jazz/store.js"; +import type { SourceCursor } from "../store/types.js"; import { cursorTimeUs, JetstreamConnector } from "./jetstream.js"; const DEFAULT_ENDPOINT = "wss://jetstream2.us-west.bsky.network/subscribe"; @@ -25,7 +25,6 @@ export interface JetstreamSubscriptionOptions { maxBackoffMs?: number; maxMessageSizeBytes?: number; pendingMessageLimit?: number; - onEvent?: (event: ThoughtEvent) => Promise; webSocketFactory?: JetstreamWebSocketFactory; sleep?: (milliseconds: number, signal?: AbortSignal) => Promise; random?: () => number; @@ -75,7 +74,7 @@ class ConnectionProgressError extends Error { constructor( readonly cause: unknown, readonly progress: ConnectionProgress, - readonly phase: "live-subscribe" | "live-agent-handoff", + readonly phase: "live-subscribe", readonly failureRecorded: boolean, ) { super(cause instanceof Error ? cause.message : String(cause)); @@ -84,7 +83,7 @@ class ConnectionProgressError extends Error { } export async function subscribeJetstream( - store: ThoughtStore, + store: JazzThoughtStore, connector: JetstreamConnector, options: JetstreamSubscriptionOptions = {}, ): Promise { @@ -127,7 +126,6 @@ export async function subscribeJetstream( pendingMessageLimit: config.pendingMessageLimit, maxMessageSizeBytes: config.maxMessageSizeBytes, webSocketFactory: config.webSocketFactory, - ...(config.onEvent ? { onEvent: config.onEvent } : {}), onOpen: async () => { const connectedAt = new Date().toISOString(); await appendLifecycle(store, connector, "stream.thought.connector.subscription.connected", connectionId, connectedAt, { @@ -258,7 +256,7 @@ export function buildJetstreamSubscriptionUrl( } async function consumeConnection( - store: ThoughtStore, + store: JazzThoughtStore, connector: JetstreamConnector, url: string, connectionId: string, @@ -269,7 +267,6 @@ async function consumeConnection( pendingMessageLimit: number; maxMessageSizeBytes: number; webSocketFactory: JetstreamWebSocketFactory; - onEvent?: (event: ThoughtEvent) => Promise; onOpen: () => Promise; }, ): Promise { @@ -349,14 +346,6 @@ async function consumeConnection( totals.unchanged += batch.unchanged; totals.ignored += batch.ignored; totals.stale += batch.stale; - if (options.onEvent) { - try { - for (const event of batch.offeredEvents) await options.onEvent(event); - } catch (error) { - throw new ConnectionProgressError(error, { ...totals }, "live-agent-handoff", false); - } - } - if (totals.messages >= options.remainingMessages) { accepting = false; socket.close(1000, "message limit reached"); @@ -409,7 +398,7 @@ async function waitForOpen(socket: JetstreamWebSocket, signal?: AbortSignal): Pr } async function appendLifecycle( - store: ThoughtStore, + store: JazzThoughtStore, connector: JetstreamConnector, type: "stream.thought.connector.subscription.started" | "stream.thought.connector.subscription.connected" | "stream.thought.connector.subscription.stopped", id: string, @@ -432,7 +421,7 @@ async function appendLifecycle( } async function recordFailure( - store: ThoughtStore, + store: JazzThoughtStore, connector: JetstreamConnector, connectionId: string, phase: string, @@ -466,7 +455,7 @@ async function recordFailure( } async function recordRecovery( - store: ThoughtStore, + store: JazzThoughtStore, connector: JetstreamConnector, connectionId: string, at: string, @@ -520,7 +509,6 @@ function normalizedOptions(options: JetstreamSubscriptionOptions) { maxMessageSizeBytes, pendingMessageLimit, signal: options.signal, - onEvent: options.onEvent, webSocketFactory: options.webSocketFactory ?? defaultWebSocketFactory, sleep: options.sleep ?? sleep, random: options.random ?? Math.random, diff --git a/src/connectors/jetstream.ts b/src/connectors/jetstream.ts index 242e286..273897a 100644 --- a/src/connectors/jetstream.ts +++ b/src/connectors/jetstream.ts @@ -2,7 +2,8 @@ import { z } from "zod"; import { canonicalJson, sha256, type JsonObject, type JsonValue } from "../core/json.js"; import { newId } from "../core/ids.js"; import type { EventCandidate, ThoughtEvent } from "../events/types.js"; -import type { SourceCursor, ThoughtStore } from "../store/types.js"; +import type { JazzThoughtStore } from "../jazz/store.js"; +import type { SourceCursor } from "../store/types.js"; export interface JetstreamConnectorOptions { id: string; @@ -109,7 +110,6 @@ export class JetstreamConnector { privacy: "public-source", payload: compact({ did: message.did, - timeUs: message.timeUs, operation: commit.operation, collection: commit.collection, rkey: commit.rkey, @@ -122,7 +122,7 @@ export class JetstreamConnector { } async ingestBatch( - store: ThoughtStore, + store: JazzThoughtStore, rawMessages: Iterable, options: JetstreamBatchOptions = {}, ): Promise { @@ -140,12 +140,9 @@ export class JetstreamConnector { throw new Error("Jetstream replayFromTimeUs must be a nonnegative safe integer at or before the durable cursor"); } let maxSeenTimeUs = priorTimeUs; - let inserted = 0; - let unchanged = 0; let ignored = 0; let stale = 0; - const events: ThoughtEvent[] = []; - const offeredEvents: ThoughtEvent[] = []; + const candidates: EventCandidate[] = []; for (const message of messages) { if (message.timeUs <= replayFromTimeUs) { @@ -158,27 +155,24 @@ export class JetstreamConnector { ignored += 1; continue; } - const result = await store.appendEvent(candidate); - offeredEvents.push(result.event); - if (result.inserted) { - inserted += 1; - events.push(result.event); - } else { - unchanged += 1; - } + candidates.push(candidate); } const completedAt = new Date().toISOString(); const cursor = this.successCursor(cursorId, prior, maxSeenTimeUs, completedAt); - if (cursor) { - await store.upsertSourceCursor(cursor); - if (maxSeenTimeUs > priorTimeUs) await this.appendCursorEvent(store, correlationId, completedAt, cursor); - } + const sourceCount = candidates.length; + if (cursor && maxSeenTimeUs > priorTimeUs) candidates.push(this.cursorEvent(correlationId, completedAt, cursor)); + const batch = await store.appendProducerBatch(candidates, cursor); + const offeredEvents = batch.events.slice(0, sourceCount); + const insertedIds = new Set(batch.inserted.map((event) => event.id)); + const events = offeredEvents.filter((event) => insertedIds.has(event.id)); + const inserted = events.length; + const unchanged = sourceCount - inserted; return { inserted, unchanged, ignored, stale, events, offeredEvents, ...(cursor ? { cursor } : {}) }; } catch (error) { const failedAt = new Date().toISOString(); const message = error instanceof Error ? error.message : String(error); - await store.appendEvent({ + const failedEvent: EventCandidate = { type: "stream.thought.connector.failed", schemaVersion: 1, source: this.id, @@ -190,8 +184,8 @@ export class JetstreamConnector { correlationId, privacy: "public-source", payload: { status: "failed", error: message, phase: "captured-batch-ingest" }, - }); - await store.upsertSourceCursor({ + }; + const failedCursor: SourceCursor = { id: cursorId, source: this.id, cursor: prior?.cursor ?? { filterRevision: this.filterRevision }, @@ -199,12 +193,13 @@ export class JetstreamConnector { lastFailureAt: failedAt, lastError: message, updatedAt: failedAt, - }); + }; + await store.appendProducerBatch([failedEvent], failedCursor); throw error; } } - async readCursor(store: ThoughtStore): Promise { + async readCursor(store: JazzThoughtStore): Promise { const cursor = await store.getSourceCursor(`cursor:${this.id}`); this.assertCursorCompatible(cursor); return cursor; @@ -229,8 +224,8 @@ export class JetstreamConnector { }; } - private async appendCursorEvent(store: ThoughtStore, correlationId: string, at: string, cursor: SourceCursor): Promise { - await store.appendEvent({ + private cursorEvent(correlationId: string, at: string, cursor: SourceCursor): EventCandidate { + return { type: "stream.thought.connector.cursor.advanced", schemaVersion: 1, source: this.id, @@ -242,7 +237,7 @@ export class JetstreamConnector { correlationId, privacy: "public-source", payload: { source: this.id, cursorId: cursor.id, cursor: cursor.cursor }, - }); + }; } } diff --git a/src/connectors/rss.ts b/src/connectors/rss.ts index 94a746a..ea2252f 100644 --- a/src/connectors/rss.ts +++ b/src/connectors/rss.ts @@ -1,8 +1,9 @@ import { XMLParser } from "fast-xml-parser"; import { sha256, type JsonObject } from "../core/json.js"; import { newId } from "../core/ids.js"; -import type { ThoughtEvent } from "../events/types.js"; -import type { SourceCursor, ThoughtStore } from "../store/types.js"; +import type { EventCandidate, ThoughtEvent } from "../events/types.js"; +import type { JazzThoughtStore } from "../jazz/store.js"; +import type { SourceCursor } from "../store/types.js"; export interface RssConnectorOptions { id: string; @@ -53,15 +54,15 @@ export class RssConnector { return { id: this.id, kind: this.kind, feedUrl: this.feedUrl, privacy: this.privacy, maxItems: this.maxItems }; } - async poll(store: ThoughtStore, fetchImpl: typeof fetch = fetch): Promise { + async poll(store: JazzThoughtStore, fetchImpl: typeof fetch = fetch): Promise { const cursorId = `cursor:${this.id}`; const prior = await store.getSourceCursor(cursorId); const startedAt = new Date().toISOString(); const correlationId = newId("rss_poll"); - await this.appendConnectorEvent(store, "started", correlationId, startedAt, { + await store.appendEvent(this.connectorEvent("started", correlationId, startedAt, { status: "started", feedUrl: this.feedUrl, - }); + })); const controller = new AbortController(); const timer = setTimeout(() => controller.abort(), this.timeoutMs); try { @@ -72,11 +73,12 @@ export class RssConnector { if (response.status === 304) { const completedAt = new Date().toISOString(); const cursor = this.successCursor(cursorId, prior?.cursor ?? {}, response, completedAt); - await store.upsertSourceCursor(cursor); - await this.appendCursorEvent(store, correlationId, completedAt, cursor); - await this.appendConnectorEvent(store, "completed", correlationId, completedAt, { - status: "not-modified", inserted: 0, unchanged: 0, httpStatus: 304, - }); + await store.appendProducerBatch([ + this.cursorEvent(correlationId, completedAt, cursor), + this.connectorEvent("completed", correlationId, completedAt, { + status: "not-modified", inserted: 0, unchanged: 0, httpStatus: 304, + }), + ], cursor); return { status: "not-modified", inserted: 0, unchanged: 0, events: [], cursor }; } if (!response.ok) { @@ -85,12 +87,10 @@ export class RssConnector { } const body = await response.text(); const { title: feedTitle, items } = parseFeed(body, this.feedUrl); - let inserted = 0; - let unchanged = 0; - const events: ThoughtEvent[] = []; + const candidates: EventCandidate[] = []; for (const item of items.slice(0, this.maxItems)) { const occurredAt = item.publishedAt ?? item.updatedAt ?? startedAt; - const appended = await store.appendEvent({ + candidates.push({ type: "stream.thought.source.rss.item", schemaVersion: 1, source: this.id, @@ -114,28 +114,27 @@ export class RssConnector { summary: item.summary, }), }); - if (appended.inserted) { - inserted += 1; - events.push(appended.event); - } else { - unchanged += 1; - } } const completedAt = new Date().toISOString(); const cursor = this.successCursor(cursorId, prior?.cursor ?? {}, response, completedAt); - await store.upsertSourceCursor(cursor); - await this.appendCursorEvent(store, correlationId, completedAt, cursor); - await this.appendConnectorEvent(store, "completed", correlationId, completedAt, { - status: "updated", inserted, unchanged, httpStatus: response.status, - }); + const sourceCount = candidates.length; + const batch = await store.appendProducerBatch([ + ...candidates, + this.cursorEvent(correlationId, completedAt, cursor), + this.connectorEvent("completed", correlationId, completedAt, { + status: "updated", offered: sourceCount, httpStatus: response.status, + }), + ], cursor); + const sourceEvents = batch.events.slice(0, sourceCount); + const insertedIds = new Set(batch.inserted.map((event) => event.id)); + const events = sourceEvents.filter((event) => insertedIds.has(event.id)); + const inserted = events.length; + const unchanged = sourceCount - inserted; return { status: "updated", inserted, unchanged, events, cursor }; } catch (error) { const failedAt = new Date().toISOString(); const message = error instanceof Error ? error.message : String(error); - await this.appendConnectorEvent(store, "failed", correlationId, failedAt, { - status: "failed", error: message, feedUrl: this.feedUrl, - }); - await store.upsertSourceCursor({ + const failedCursor: SourceCursor = { id: cursorId, source: this.id, cursor: prior?.cursor ?? {}, @@ -143,7 +142,10 @@ export class RssConnector { lastFailureAt: failedAt, lastError: message, updatedAt: failedAt, - }); + }; + await store.appendProducerBatch([this.connectorEvent("failed", correlationId, failedAt, { + status: "failed", error: message, feedUrl: this.feedUrl, + })], failedCursor); throw error; } finally { clearTimeout(timer); @@ -166,8 +168,8 @@ export class RssConnector { }; } - private async appendCursorEvent(store: ThoughtStore, correlationId: string, at: string, cursor: SourceCursor): Promise { - await store.appendEvent({ + private cursorEvent(correlationId: string, at: string, cursor: SourceCursor): EventCandidate { + return { type: "stream.thought.connector.cursor.advanced", schemaVersion: 1, source: this.id, @@ -179,17 +181,16 @@ export class RssConnector { correlationId, privacy: this.privacy, payload: { source: this.id, cursorId: cursor.id, cursor: cursor.cursor }, - }); + }; } - private async appendConnectorEvent( - store: ThoughtStore, + private connectorEvent( phase: "started" | "completed" | "failed", correlationId: string, at: string, payload: JsonObject, - ): Promise { - await store.appendEvent({ + ): EventCandidate { + return { type: phase === "failed" ? "stream.thought.connector.failed" : `stream.thought.connector.poll.${phase}`, schemaVersion: 1, source: this.id, @@ -201,7 +202,7 @@ export class RssConnector { correlationId, privacy: this.privacy, payload, - }); + }; } } diff --git a/src/connectors/telegram-bot.ts b/src/connectors/telegram-bot.ts new file mode 100644 index 0000000..37445b9 --- /dev/null +++ b/src/connectors/telegram-bot.ts @@ -0,0 +1,463 @@ +import { z } from "zod"; +import { canonicalJson, sha256, type JsonObject } from "../core/json.js"; +import { newId } from "../core/ids.js"; +import type { EventCandidate, ThoughtEvent } from "../events/types.js"; +import type { JazzThoughtStore } from "../jazz/store.js"; +import type { SourceCursor } from "../store/types.js"; + +const TELEGRAM_BOT_REVISION = "telegram-bot-api-v1"; + +const userSchema = z.object({ + id: z.number().int(), + is_bot: z.boolean(), + first_name: z.string(), + last_name: z.string().optional(), + username: z.string().optional(), +}).passthrough(); + +const chatSchema = z.object({ + id: z.number().int(), + type: z.enum(["private", "group", "supergroup", "channel"]), + title: z.string().optional(), + username: z.string().optional(), + first_name: z.string().optional(), + last_name: z.string().optional(), +}).passthrough(); + +const photoSizeSchema = z.object({ + file_id: z.string(), + file_unique_id: z.string(), + width: z.number().int(), + height: z.number().int(), + file_size: z.number().int().nonnegative().optional(), +}).passthrough(); + +const fileSchema = z.object({ + file_id: z.string(), + file_unique_id: z.string(), + file_name: z.string().optional(), + mime_type: z.string().optional(), + file_size: z.number().int().nonnegative().optional(), +}).passthrough(); + +const messageSchema = z.object({ + message_id: z.number().int(), + message_thread_id: z.number().int().optional(), + from: userSchema.optional(), + date: z.number().int().nonnegative(), + edit_date: z.number().int().nonnegative().optional(), + chat: chatSchema, + text: z.string().optional(), + caption: z.string().optional(), + reply_to_message: z.object({ message_id: z.number().int() }).passthrough().optional(), + photo: z.array(photoSizeSchema).optional(), + document: fileSchema.optional(), + audio: fileSchema.optional(), + voice: fileSchema.optional(), + video: fileSchema.optional(), +}).passthrough(); + +const updateSchema = z.object({ + update_id: z.number().int().nonnegative(), + message: messageSchema.optional(), + edited_message: messageSchema.optional(), +}).passthrough(); + +const envelopeSchema = z.object({ + ok: z.boolean(), + result: z.unknown().optional(), + error_code: z.number().int().optional(), + description: z.string().optional(), + parameters: z.object({ retry_after: z.number().int().positive().optional() }).passthrough().optional(), +}).passthrough(); + +export type TelegramBotUser = z.infer; +export type TelegramBotUpdate = z.infer; + +export interface TelegramBotClientOptions { + token: string; + baseUrl?: string; + requestTimeoutMs?: number; +} + +export interface TelegramSendResult { + messageId: string; + chatId: string; + date: string; +} + +export class TelegramBotApiError extends Error { + constructor( + message: string, + readonly errorCode?: number, + readonly retryAfterSeconds?: number, + ) { + super(message); + this.name = "TelegramBotApiError"; + } +} + +export class TelegramBotClient { + private readonly token: string; + private readonly baseUrl: string; + private readonly requestTimeoutMs: number; + private identityValue?: TelegramBotUser; + + constructor(options: TelegramBotClientOptions) { + this.token = required(options.token, "Telegram bot token"); + this.baseUrl = (options.baseUrl ?? "https://api.telegram.org").replace(/\/$/, ""); + this.requestTimeoutMs = boundedPositiveInteger(options.requestTimeoutMs ?? 35_000, "requestTimeoutMs", 120_000); + } + + async identity(signal?: AbortSignal): Promise { + this.identityValue ??= await this.call("getMe", {}, userSchema, signal); + return this.identityValue; + } + + async getUpdates(options: { + offset?: number; + limit?: number; + timeoutSeconds?: number; + signal?: AbortSignal; + } = {}): Promise { + return this.call("getUpdates", { + ...(options.offset === undefined ? {} : { offset: options.offset }), + limit: options.limit ?? 100, + timeout: options.timeoutSeconds ?? 0, + allowed_updates: ["message", "edited_message"], + }, z.array(updateSchema), options.signal); + } + + async sendMessage(chatId: string, text: string, signal?: AbortSignal): Promise { + const message = await this.call("sendMessage", { + chat_id: required(chatId, "Telegram chat id"), + text: required(text, "Telegram message text"), + link_preview_options: { is_disabled: true }, + }, messageSchema, signal); + return { + messageId: String(message.message_id), + chatId: String(message.chat.id), + date: new Date(message.date * 1_000).toISOString(), + }; + } + + private async call( + method: string, + body: Record, + resultSchema: z.ZodType, + signal?: AbortSignal, + ): Promise { + const timeout = AbortSignal.timeout(this.requestTimeoutMs); + const combinedSignal = signal ? AbortSignal.any([signal, timeout]) : timeout; + let response: Response; + try { + response = await fetch(`${this.baseUrl}/bot${this.token}/${method}`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify(body), + signal: combinedSignal, + }); + } catch (error) { + const reason = error instanceof Error && error.name === "AbortError" ? "timed out or was cancelled" : "failed"; + throw new TelegramBotApiError(`Telegram Bot API ${method} request ${reason}`); + } + let raw: unknown; + try { + raw = await response.json(); + } catch { + throw new TelegramBotApiError(`Telegram Bot API ${method} returned non-JSON HTTP ${response.status}`); + } + const envelope = envelopeSchema.parse(raw); + if (!response.ok || !envelope.ok) { + throw new TelegramBotApiError( + envelope.description ?? `Telegram Bot API ${method} failed with HTTP ${response.status}`, + envelope.error_code ?? response.status, + envelope.parameters?.retry_after, + ); + } + return resultSchema.parse(envelope.result); + } +} + +export interface TelegramBotConnectorOptions { + id: string; + client: TelegramBotClient; + allowedChatIds: string[]; +} + +export interface TelegramBotPollResult { + status: "updated" | "unchanged"; + received: number; + accepted: number; + ignored: number; + inserted: number; + unchanged: number; + events: ThoughtEvent[]; + cursor: SourceCursor; +} + +export class TelegramBotConnector { + readonly kind = "telegram" as const; + readonly id: string; + private readonly client: TelegramBotClient; + private readonly allowedChatIds: Set; + + constructor(options: TelegramBotConnectorOptions) { + this.id = required(options.id, "Telegram bot connector id"); + this.client = options.client; + this.allowedChatIds = new Set(options.allowedChatIds.map((value) => required(value, "Allowed Telegram chat id"))); + if (this.allowedChatIds.size === 0) throw new Error("Telegram bot connector requires at least one allowed chat id"); + } + + describe(): JsonObject { + return { + id: this.id, + kind: this.kind, + transport: "telegram-bot-api-long-poll", + revision: TELEGRAM_BOT_REVISION, + allowedChatIdsHash: sha256([...this.allowedChatIds].sort().join("\n")), + authority: "read-private-chat-updates", + }; + } + + async poll( + store: JazzThoughtStore, + options: { timeoutSeconds?: number; limit?: number; signal?: AbortSignal } = {}, + ): Promise { + const cursorId = `cursor:${this.id}`; + const prior = await store.getSourceCursor(cursorId); + const correlationId = newId("telegram_bot_poll"); + const startedAt = new Date().toISOString(); + await store.appendEvent(this.connectorEvent("started", correlationId, startedAt, { + status: "started", + transport: "telegram-bot-api-long-poll", + })); + try { + const bot = await this.client.identity(options.signal); + this.assertCursorCompatible(prior, String(bot.id)); + const offset = cursorOffset(prior); + const updates = await this.client.getUpdates({ + offset, + limit: options.limit ?? 100, + timeoutSeconds: options.timeoutSeconds ?? 0, + ...(options.signal ? { signal: options.signal } : {}), + }); + const maxUpdateId = Math.max(offset - 1, ...updates.map((update) => update.update_id)); + const acceptedUpdates = updates.filter((update) => { + const message = update.edited_message ?? update.message; + return message && this.allowedChatIds.has(String(message.chat.id)); + }); + const candidates = acceptedUpdates.flatMap((update) => { + const event = this.eventCandidate(update, bot, correlationId); + return event ? [event] : []; + }); + const completedAt = new Date().toISOString(); + const cursor: SourceCursor = { + id: cursorId, + source: this.id, + cursor: { + revision: TELEGRAM_BOT_REVISION, + botId: String(bot.id), + updateOffset: maxUpdateId + 1, + }, + lastSuccessAt: completedAt, + ...(prior?.lastFailureAt ? { lastFailureAt: prior.lastFailureAt } : {}), + updatedAt: completedAt, + }; + const sourceCount = candidates.length; + candidates.push(this.cursorEvent(correlationId, completedAt, cursor)); + if (prior?.lastFailureAt && (!prior.lastSuccessAt || prior.lastFailureAt > prior.lastSuccessAt)) { + candidates.push(this.connectorEvent("recovered", correlationId, completedAt, { + status: "recovered", + previousFailureAt: prior.lastFailureAt, + })); + } + candidates.push(this.connectorEvent("completed", correlationId, completedAt, { + status: acceptedUpdates.length > 0 ? "updated" : "unchanged", + received: updates.length, + accepted: acceptedUpdates.length, + ignored: updates.length - acceptedUpdates.length, + })); + const batch = await store.appendProducerBatch(candidates, cursor); + const offeredEvents = batch.events.slice(0, sourceCount); + const insertedIds = new Set(batch.inserted.map((event) => event.id)); + const events = offeredEvents.filter((event) => insertedIds.has(event.id)); + return { + status: acceptedUpdates.length > 0 ? "updated" : "unchanged", + received: updates.length, + accepted: acceptedUpdates.length, + ignored: updates.length - acceptedUpdates.length, + inserted: events.length, + unchanged: sourceCount - events.length, + events, + cursor, + }; + } catch (error) { + const failedAt = new Date().toISOString(); + const message = error instanceof Error ? error.message : String(error); + const failedCursor: SourceCursor = { + id: cursorId, + source: this.id, + cursor: prior?.cursor ?? { revision: TELEGRAM_BOT_REVISION, updateOffset: 0 }, + ...(prior?.lastSuccessAt ? { lastSuccessAt: prior.lastSuccessAt } : {}), + lastFailureAt: failedAt, + lastError: message, + updatedAt: failedAt, + }; + await store.appendProducerBatch([this.connectorEvent("failed", correlationId, failedAt, { + status: "failed", + error: message, + phase: "telegram-bot-poll", + })], failedCursor); + throw error; + } + } + + private eventCandidate(update: TelegramBotUpdate, bot: TelegramBotUser, correlationId: string): EventCandidate | undefined { + const message = update.edited_message ?? update.message; + if (!message) return undefined; + const edited = update.edited_message !== undefined; + const occurredAt = new Date((message.edit_date ?? message.date) * 1_000).toISOString(); + const senderId = message.from ? String(message.from.id) : String(message.chat.id); + const senderName = message.from + ? message.from.username ?? [message.from.first_name, message.from.last_name].filter(Boolean).join(" ") + : message.chat.username ?? message.chat.title ?? "unknown"; + const payload = compact({ + accountId: String(bot.id), + accountUsername: bot.username, + updateId: String(update.update_id), + chatId: String(message.chat.id), + messageId: String(message.message_id), + senderId, + senderName, + chatType: message.chat.type === "private" ? "direct" : "channel", + occurredAt: new Date(message.date * 1_000).toISOString(), + editedAt: edited ? occurredAt : undefined, + text: message.text ?? message.caption ?? "", + threadId: message.message_thread_id === undefined ? undefined : String(message.message_thread_id), + replyToMessageId: message.reply_to_message?.message_id === undefined ? undefined : String(message.reply_to_message.message_id), + attachments: attachmentsFor(message), + transport: "telegram-bot-api-long-poll", + }); + return { + type: "stream.thought.source.telegram.message", + schemaVersion: 1, + source: this.id, + sourceKind: this.kind, + externalId: `${bot.id}:${message.chat.id}:${message.message_id}`, + idempotencyKey: sha256(canonicalJson({ + botId: bot.id, + chatId: message.chat.id, + messageId: message.message_id, + revision: message.edit_date ?? "original", + })), + occurredAt, + actor: senderId, + correlationId, + privacy: "sensitive", + payload, + }; + } + + private assertCursorCompatible(cursor: SourceCursor | undefined, botId: string): void { + const revision = cursor?.cursor.revision; + if (revision !== undefined && revision !== TELEGRAM_BOT_REVISION) throw new Error("Telegram bot cursor revision does not match connector configuration"); + const priorBotId = cursor?.cursor.botId; + if (priorBotId !== undefined && priorBotId !== botId) throw new Error("Telegram bot cursor belongs to a different bot identity"); + cursorOffset(cursor); + } + + private cursorEvent(correlationId: string, at: string, cursor: SourceCursor): EventCandidate { + return { + type: "stream.thought.connector.cursor.advanced", + schemaVersion: 1, + source: this.id, + sourceKind: this.kind, + externalId: cursor.id, + idempotencyKey: `${correlationId}:cursor`, + occurredAt: at, + actor: this.id, + correlationId, + privacy: "sensitive", + payload: { source: this.id, cursorId: cursor.id, cursor: cursor.cursor }, + }; + } + + private connectorEvent( + phase: "started" | "completed" | "failed" | "recovered", + correlationId: string, + at: string, + payload: JsonObject, + ): EventCandidate { + const type = phase === "failed" + ? "stream.thought.connector.failed" + : phase === "recovered" + ? "stream.thought.connector.recovered" + : `stream.thought.connector.poll.${phase}`; + return { + type, + schemaVersion: 1, + source: this.id, + sourceKind: this.kind, + externalId: correlationId, + idempotencyKey: `${correlationId}:${phase}`, + occurredAt: at, + actor: this.id, + correlationId, + privacy: "sensitive", + payload, + }; + } +} + +function attachmentsFor(message: z.infer): JsonObject[] | undefined { + const attachments: JsonObject[] = []; + const photo = message.photo?.at(-1); + if (photo) attachments.push(compact({ + id: photo.file_unique_id, + kind: "image", + sizeBytes: photo.file_size, + reference: `telegram-file:${photo.file_id}`, + })); + for (const [kind, file] of [ + ["file", message.document], + ["audio", message.audio], + ["audio", message.voice], + ["video", message.video], + ] as const) { + if (!file) continue; + attachments.push(compact({ + id: file.file_unique_id, + name: file.file_name, + mimeType: file.mime_type, + sizeBytes: file.file_size, + kind, + reference: `telegram-file:${file.file_id}`, + })); + } + return attachments.length > 0 ? attachments : undefined; +} + +function cursorOffset(cursor: SourceCursor | undefined): number { + const value = cursor?.cursor.updateOffset; + if (value === undefined) return 0; + if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 0) { + throw new Error("Telegram bot cursor updateOffset must be a nonnegative safe integer"); + } + return value; +} + +function compact(value: Record): JsonObject { + return Object.fromEntries(Object.entries(value).filter(([, child]) => child !== undefined)) as JsonObject; +} + +function boundedPositiveInteger(value: number, label: string, maximum: number): number { + if (!Number.isSafeInteger(value) || value <= 0 || value > maximum) throw new Error(`${label} must be a positive integer <= ${maximum}`); + return value; +} + +function required(value: string, label: string): string { + const normalized = value.trim(); + if (!normalized) throw new Error(`${label} is required`); + return normalized; +} diff --git a/src/connectors/telegram-spool.ts b/src/connectors/telegram-spool.ts index dbaf5c4..7c96b56 100644 --- a/src/connectors/telegram-spool.ts +++ b/src/connectors/telegram-spool.ts @@ -6,8 +6,9 @@ import { TextDecoder } from "node:util"; import { z } from "zod"; import { canonicalJson, sha256, type JsonObject } from "../core/json.js"; import { newId } from "../core/ids.js"; -import type { ThoughtEvent } from "../events/types.js"; -import type { SourceCursor, ThoughtStore } from "../store/types.js"; +import type { EventCandidate, ThoughtEvent } from "../events/types.js"; +import type { JazzThoughtStore } from "../jazz/store.js"; +import type { SourceCursor } from "../store/types.js"; const SPOOL_REVISION = "telegram-spool-ndjson-v1"; const utf8 = new TextDecoder("utf-8", { fatal: true }); @@ -91,7 +92,7 @@ export class TelegramSpoolConnector { }; } - async ingest(store: ThoughtStore): Promise { + async ingest(store: JazzThoughtStore): Promise { const cursorId = `cursor:${this.id}`; const prior = await store.getSourceCursor(cursorId); const startedAt = new Date().toISOString(); @@ -124,19 +125,7 @@ export class TelegramSpoolConnector { throw new Error(`Telegram spool has no complete record within maxReadBytes=${this.maxReadBytes}`); } - const events: ThoughtEvent[] = []; - let inserted = 0; - let unchanged = 0; - for (const record of records) { - const appended = await store.appendEvent(this.eventCandidate(record, correlationId)); - if (appended.inserted) { - inserted += 1; - events.push(appended.event); - } else { - unchanged += 1; - } - } - await store.flush(); + const candidates: EventCandidate[] = records.map((record) => this.eventCandidate(record, correlationId)); const completedAt = new Date().toISOString(); const newOffset = offset + consumedBytes; @@ -156,22 +145,26 @@ export class TelegramSpoolConnector { ...(prior?.lastFailureAt ? { lastFailureAt: prior.lastFailureAt } : {}), updatedAt: completedAt, }; - await store.upsertSourceCursor(cursor); - await this.appendCursorEvent(store, correlationId, completedAt, cursor); + const sourceCount = candidates.length; + candidates.push(this.cursorEvent(correlationId, completedAt, cursor)); if (prior?.lastFailureAt && (!prior.lastSuccessAt || prior.lastFailureAt > prior.lastSuccessAt)) { - await this.appendConnectorEvent(store, "recovered", correlationId, completedAt, { + candidates.push(this.connectorEvent("recovered", correlationId, completedAt, { status: "recovered", previousFailureAt: prior.lastFailureAt, - }); + })); } const status = records.length > 0 ? "updated" : "unchanged"; - await this.appendConnectorEvent(store, "completed", correlationId, completedAt, { + candidates.push(this.connectorEvent("completed", correlationId, completedAt, { status, - inserted, - unchanged, consumedRecords: records.length, partialTrailingBytes, - }); + })); + const batch = await store.appendProducerBatch(candidates, cursor); + const offeredEvents = batch.events.slice(0, sourceCount); + const insertedIds = new Set(batch.inserted.map((event) => event.id)); + const events = offeredEvents.filter((event) => insertedIds.has(event.id)); + const inserted = events.length; + const unchanged = sourceCount - inserted; return { status, inserted, @@ -184,13 +177,7 @@ export class TelegramSpoolConnector { } catch (error) { const failedAt = new Date().toISOString(); const message = error instanceof Error ? error.message : String(error); - await this.appendConnectorEvent(store, "failed", correlationId, failedAt, { - status: "failed", - error: message, - transport: "append-only-ndjson-spool", - filePathHash: this.filePathHash, - }); - await store.upsertSourceCursor({ + const failedCursor: SourceCursor = { id: cursorId, source: this.id, cursor: prior?.cursor ?? { @@ -203,7 +190,13 @@ export class TelegramSpoolConnector { lastFailureAt: failedAt, lastError: message, updatedAt: failedAt, - }); + }; + await store.appendProducerBatch([this.connectorEvent("failed", correlationId, failedAt, { + status: "failed", + error: message, + transport: "append-only-ndjson-spool", + filePathHash: this.filePathHash, + })], failedCursor); throw error; } finally { await handle?.close(); @@ -266,8 +259,8 @@ export class TelegramSpoolConnector { } } - private async appendCursorEvent(store: ThoughtStore, correlationId: string, at: string, cursor: SourceCursor): Promise { - await store.appendEvent({ + private cursorEvent(correlationId: string, at: string, cursor: SourceCursor): EventCandidate { + return { type: "stream.thought.connector.cursor.advanced", schemaVersion: 1, source: this.id, @@ -279,22 +272,21 @@ export class TelegramSpoolConnector { correlationId, privacy: "sensitive", payload: { source: this.id, cursorId: cursor.id, cursor: cursor.cursor }, - }); + }; } - private async appendConnectorEvent( - store: ThoughtStore, + private connectorEvent( phase: "started" | "completed" | "failed" | "recovered", correlationId: string, at: string, payload: JsonObject, - ): Promise { + ): EventCandidate { const type = phase === "failed" ? "stream.thought.connector.failed" : phase === "recovered" ? "stream.thought.connector.recovered" : `stream.thought.connector.poll.${phase}`; - await store.appendEvent({ + return { type, schemaVersion: 1, source: this.id, @@ -306,7 +298,17 @@ export class TelegramSpoolConnector { correlationId, privacy: "sensitive", payload, - }); + }; + } + + private async appendConnectorEvent( + store: JazzThoughtStore, + phase: "started" | "completed" | "failed" | "recovered", + correlationId: string, + at: string, + payload: JsonObject, + ): Promise { + await store.appendEvent(this.connectorEvent(phase, correlationId, at, payload)); } } diff --git a/src/events/registry.ts b/src/events/registry.ts index e263704..882a189 100644 --- a/src/events/registry.ts +++ b/src/events/registry.ts @@ -72,6 +72,26 @@ const runPayload = z.object({ status: z.string(), }).passthrough() as unknown as z.ZodType; +const judgmentPayload = z.object({ + runId: z.string().min(1), + outputEventId: z.string().min(1), + kind: z.enum(["accept", "reject", "correct", "prefer"]), + criterion: z.string().min(1), + criterionVersion: z.number().int().positive(), + exportEligible: z.boolean(), + comparedRunId: z.string().min(1).optional(), + comparedOutputEventId: z.string().min(1).optional(), + replacementOutput: objectPayload.optional(), + notes: z.string().max(10_000).optional(), +}).superRefine((value, context) => { + if (value.kind === "correct" && !value.replacementOutput) { + context.addIssue({ code: "custom", path: ["replacementOutput"], message: "Corrections require a replacement output" }); + } + if (value.kind === "prefer" && (!value.comparedRunId || !value.comparedOutputEventId)) { + context.addIssue({ code: "custom", path: ["comparedRunId"], message: "Preferences require a compared run and output" }); + } +}) as unknown as z.ZodType; + export function createDefaultRegistry(): EventRegistry { const registry = new EventRegistry(); for (const type of [ @@ -84,9 +104,6 @@ export function createDefaultRegistry(): EventRegistry { } for (const type of [ - "stream.thought.agent.matched", - "stream.thought.agent.run.queued", - "stream.thought.agent.run.leased", "stream.thought.agent.run.started", "stream.thought.agent.run.trace", "stream.thought.agent.run.completed", @@ -97,6 +114,13 @@ export function createDefaultRegistry(): EventRegistry { registry.register({ type, schemaVersion: 1, description: "Agent lifecycle evidence", payload: runPayload }); } + registry.register({ + type: "stream.thought.judgment.training-example", + schemaVersion: 1, + description: "Explicit human or operator judgment over an agent output", + payload: judgmentPayload, + }); + for (const type of [ "stream.thought.derived.topics", "stream.thought.derived.entities", @@ -109,6 +133,8 @@ export function createDefaultRegistry(): EventRegistry { "stream.thought.source.atproto.commit", "stream.thought.source.email.observed", "stream.thought.source.telegram.message", + "stream.thought.runtime.notice", + "stream.thought.dispatcher.activated", "stream.thought.connector.poll.started", "stream.thought.connector.poll.completed", "stream.thought.connector.subscription.started", @@ -117,6 +143,9 @@ export function createDefaultRegistry(): EventRegistry { "stream.thought.connector.cursor.advanced", "stream.thought.connector.failed", "stream.thought.connector.recovered", + "stream.thought.action.telegram.send.started", + "stream.thought.action.telegram.send.delivered", + "stream.thought.action.telegram.send.failed", ]) { registry.register({ type, schemaVersion: 1, description: "Registered thought stream payload", payload: objectPayload }); } diff --git a/src/events/types.ts b/src/events/types.ts index 56672d2..307a227 100644 --- a/src/events/types.ts +++ b/src/events/types.ts @@ -33,6 +33,7 @@ export interface EventCandidate { export interface ThoughtEvent { id: string; + sourceSequence: number; type: string; schemaVersion: number; source: string; diff --git a/src/jazz/permissions.ts b/src/jazz/permissions.ts index a42d3b5..666641f 100644 --- a/src/jazz/permissions.ts +++ b/src/jazz/permissions.ts @@ -10,9 +10,9 @@ export const thoughtstreamPermissions = s.definePermissions(thoughtstreamApp, ({ policy.sources, policy.sourceCursors, policy.agents, - policy.matches, policy.runs, policy.traceChunks, + policy.consumerProgress, policy.projections, ]; for (const table of tables) { diff --git a/src/jazz/schema.ts b/src/jazz/schema.ts index 63e59a8..5a7a822 100644 --- a/src/jazz/schema.ts +++ b/src/jazz/schema.ts @@ -3,6 +3,7 @@ import { schema as s } from "jazz-tools"; const schema = { events: s.table({ key: s.string(), + sourceSequence: s.int().optional(), type: s.string(), schemaVersion: s.int(), source: s.string(), @@ -51,6 +52,7 @@ const schema = { kind: s.string(), enabled: s.boolean(), configJson: s.string(), + lastSequence: s.int().optional(), createdAt: s.string(), updatedAt: s.string(), }), @@ -71,26 +73,16 @@ const schema = { specHash: s.string(), updatedAt: s.string(), }), - matches: s.table({ - key: s.string(), - eventKey: s.string(), - agentKey: s.string(), - agentVersion: s.int(), - matcherRevision: s.string(), - createdAt: s.string(), - }), runs: s.table({ key: s.string(), - matchKey: s.string(), + executionKey: s.string().optional(), + triggerEventKey: s.string().optional(), agentKey: s.string(), agentVersion: s.int(), status: s.string(), inputEventIdsJson: s.string(), outputEventIdsJson: s.string(), attempt: s.int(), - workerId: s.string(), - leaseId: s.string(), - leaseExpiresAt: s.string(), provider: s.string(), model: s.string(), adapterRevision: s.string(), @@ -111,6 +103,15 @@ const schema = { payloadJson: s.string(), createdAt: s.string(), }), + consumerProgress: s.table({ + key: s.string(), + consumerKey: s.string(), + consumerVersion: s.int(), + source: s.string(), + lastSequence: s.int(), + lastEventKey: s.string(), + updatedAt: s.string(), + }), projections: s.table({ key: s.string(), payloadJson: s.string(), diff --git a/src/jazz/store.ts b/src/jazz/store.ts index 140e714..0f35f9a 100644 --- a/src/jazz/store.ts +++ b/src/jazz/store.ts @@ -1,9 +1,9 @@ import { mkdirSync } from "node:fs"; +import { createHash } from "node:crypto"; import path from "node:path"; -import type { Db } from "jazz-tools"; +import { RowChangeKind, type Db, type TransactionScope } from "jazz-tools"; import { createJazzContext, type JazzContext } from "jazz-tools/backend"; -import { canonicalJson, hashJson, parseJsonObject } from "../core/json.js"; -import { newId } from "../core/ids.js"; +import { canonicalJson, hashJson, parseJsonObject, sha256 } from "../core/json.js"; import { createDefaultRegistry, type EventRegistry } from "../events/registry.js"; import type { AppendEventResult, @@ -13,12 +13,17 @@ import type { ThoughtEvent, } from "../events/types.js"; import type { - AgentMatch, AgentRun, + AgentRunStatus, + ConsumerEventQuery, + ConsumerFailureSettlement, + ConsumerProgress, + ConsumerSuccessSettlement, + ProducerBatchResult, Projection, + SourceState, SourceCursor, StoredAgent, - ThoughtStore, TraceChunk, } from "../store/types.js"; import thoughtstreamPermissions from "./permissions.js"; @@ -37,7 +42,7 @@ export interface JazzThoughtStoreOptions { type JazzTable = { where(input: Record): { limit(count: number): unknown } }; -export class JazzThoughtStore implements ThoughtStore { +export class JazzThoughtStore { private readonly context: JazzContext; private readonly db: Db; private readonly durabilityTier: "local" | "edge"; @@ -68,38 +73,80 @@ export class JazzThoughtStore implements ThoughtStore { this.db.onMutationError((error) => console.error("thought stream Jazz mutation rejected", error)); } + getArtifactRoot(): string { + return path.join(this.options.projectRoot, ".thoughtstream", "artifacts"); + } + async appendEvent(candidate: EventCandidate): Promise { - const existing = await this.db.one( - thoughtstreamApp.events.where({ source: candidate.source, idempotencyKey: candidate.idempotencyKey }).limit(1), - ); - if (existing) return { event: eventFromJazz(existing), inserted: false }; - - const payload = this.registry.validate(candidate.type, candidate.schemaVersion, candidate.payload); - const observedAt = candidate.observedAt ?? new Date().toISOString(); - const id = newId("evt", new Date(observedAt)); - const event: ThoughtEvent = { - id, - type: candidate.type, - schemaVersion: candidate.schemaVersion, - source: candidate.source, - sourceKind: candidate.sourceKind, - externalId: candidate.externalId, - idempotencyKey: candidate.idempotencyKey, - occurredAt: candidate.occurredAt, - observedAt, - actor: candidate.actor, - rootEventId: candidate.rootEventId ?? id, - ...(candidate.parentEventId ? { parentEventId: candidate.parentEventId } : {}), - correlationId: candidate.correlationId, - privacy: candidate.privacy, - payload, - payloadHash: hashJson(payload), - ...(candidate.traceId ? { traceId: candidate.traceId } : {}), - createdByRuntime: candidate.createdByRuntime ?? this.runtimeRevision, - }; - const write = this.db.insert(thoughtstreamApp.events, eventToJazz(event)); - await this.wait(write); - return { event, inserted: true }; + const result = await this.appendProducerBatch([candidate]); + const event = result.events[0]; + if (!event) throw new Error("Atomic event append returned no event"); + return { event, inserted: result.inserted.length === 1 }; + } + + async appendProducerBatch(candidates: EventCandidate[], cursor?: SourceCursor): Promise { + if (candidates.length === 0 && !cursor) return { events: [], inserted: [], unchanged: [] }; + const source = candidates[0]?.source ?? cursor!.source; + if (candidates.some((candidate) => candidate.source !== source)) { + throw new Error("A producer transaction may settle events for only one source"); + } + if (candidates.some((candidate) => candidate.sourceKind !== candidates[0]?.sourceKind)) { + throw new Error("A producer transaction may not mix source kinds"); + } + if (cursor && cursor.source !== source) throw new Error("Producer cursor source does not match event source"); + const sourceKind = candidates[0]?.sourceKind ?? "system"; + const observedAt = new Date().toISOString(); + const result = await this.db.transaction(async (tx) => { + const sourceRow = await tx.one(thoughtstreamApp.sources.where({ key: source }).limit(1)); + if (sourceRow && candidates.length > 0 && String(sourceRow.kind) !== sourceKind) { + throw new Error(`Source kind changed for ${source}: ${String(sourceRow.kind)} -> ${sourceKind}`); + } + let sequence = Number(sourceRow?.lastSequence ?? 0); + const events: ThoughtEvent[] = []; + const inserted: ThoughtEvent[] = []; + const unchanged: ThoughtEvent[] = []; + const seen = new Map(); + for (const candidate of candidates) { + const id = eventId(candidate); + const repeated = seen.get(id); + if (repeated) { + assertSameEventIdentity(repeated, candidate); + events.push(repeated); + unchanged.push(repeated); + continue; + } + const existing = await tx.one(thoughtstreamApp.events.where({ key: id }).limit(1)); + if (existing) { + const event = eventFromJazz(existing); + assertSameEventIdentity(event, candidate); + events.push(event); + unchanged.push(event); + seen.set(id, event); + continue; + } + sequence += 1; + const event = buildEvent(candidate, id, sequence, observedAt, this.runtimeRevision, this.registry); + tx.insert(thoughtstreamApp.events, eventToJazz(event), { id: jazzRowId("event", id) }); + events.push(event); + inserted.push(event); + seen.set(id, event); + } + const sourceData = { + key: source, + kind: sourceRow ? String(sourceRow.kind) : sourceKind, + enabled: true, + configJson: sourceRow ? String(sourceRow.configJson) : "{}", + lastSequence: sequence, + createdAt: sourceRow ? String(sourceRow.createdAt) : observedAt, + updatedAt: observedAt, + }; + if (sourceRow) tx.update(thoughtstreamApp.sources, String(sourceRow.id), sourceData); + else tx.insert(thoughtstreamApp.sources, sourceData, { id: jazzRowId("source", source) }); + if (cursor) await upsertCursorInTransaction(tx, cursor); + return { events, inserted, unchanged }; + }); + await result.wait({ tier: this.durabilityTier }); + return result.value; } async getEvent(id: string): Promise { @@ -118,6 +165,12 @@ export class JazzThoughtStore implements ThoughtStore { return options.limit ? events.slice(-options.limit) : events; } + async listSources(): Promise { + return (await this.db.all(thoughtstreamApp.sources.where({}))) + .map(sourceStateFromJazz) + .sort((left, right) => left.id.localeCompare(right.id)); + } + async appendDocumentVersion(version: DocumentVersion): Promise { const existing = await this.db.one(thoughtstreamApp.documentVersions.where({ key: version.id }).limit(1)); if (existing) return false; @@ -177,25 +230,6 @@ export class JazzThoughtStore implements ThoughtStore { return (await this.db.all(thoughtstreamApp.agents.where({}))).map(agentFromJazz); } - async insertMatch(match: AgentMatch): Promise { - const existing = await this.db.one(thoughtstreamApp.matches.where({ key: match.id }).limit(1)); - if (existing) return false; - await this.wait(this.db.insert(thoughtstreamApp.matches, { - key: match.id, - eventKey: match.eventId, - agentKey: match.agentId, - agentVersion: match.agentVersion, - matcherRevision: match.matcherRevision, - createdAt: match.createdAt, - })); - return true; - } - - async getMatch(id: string): Promise { - const row = await this.db.one(thoughtstreamApp.matches.where({ key: id }).limit(1)); - return row ? matchFromJazz(row) : undefined; - } - async upsertRun(run: AgentRun): Promise { await this.upsert(thoughtstreamApp.runs, { key: run.id }, runToJazz(run)); } @@ -205,9 +239,18 @@ export class JazzThoughtStore implements ThoughtStore { return row ? runFromJazz(row) : undefined; } - async listRuns(): Promise { - return (await this.db.all(thoughtstreamApp.runs.where({}))) + async latestRunForExecution(executionKey: string): Promise { + const row = await this.db.one(thoughtstreamApp.runs.where({ executionKey }).orderBy("attempt", "desc").limit(1)); + return row ? runFromJazz(row) : undefined; + } + + async listRuns(query: { status?: AgentRunStatus; completedAfter?: string } = {}): Promise { + const rows = query.status + ? await this.db.all(thoughtstreamApp.runs.where({ status: query.status })) + : await this.db.all(thoughtstreamApp.runs.where({})); + return rows .map(runFromJazz) + .filter((run) => !query.completedAfter || (run.completedAt ?? run.updatedAt) > query.completedAfter) .sort((left, right) => left.createdAt.localeCompare(right.createdAt) || left.id.localeCompare(right.id)); } @@ -269,6 +312,82 @@ export class JazzThoughtStore implements ThoughtStore { .sort((left, right) => left.source.localeCompare(right.source) || left.id.localeCompare(right.id)); } + async getConsumerProgress(id: string): Promise { + const row = await this.db.one(thoughtstreamApp.consumerProgress.where({ key: id }).limit(1)); + return row ? consumerProgressFromJazz(row) : undefined; + } + + async listConsumerProgress(): Promise { + return (await this.db.all(thoughtstreamApp.consumerProgress.where({}))) + .map(consumerProgressFromJazz) + .sort((left, right) => left.consumerId.localeCompare(right.consumerId) + || left.consumerVersion - right.consumerVersion + || left.source.localeCompare(right.source)); + } + + subscribeSources(onSources: (sources: SourceState[]) => void): () => void { + return this.db.subscribeAll( + thoughtstreamApp.sources.where({ enabled: true }), + ({ all }) => onSources(all.map(sourceStateFromJazz)), + ); + } + + async queryConsumerEvents(query: ConsumerEventQuery): Promise { + if (query.eventTypes.length === 0 || query.acceptedPrivacy.length === 0) return []; + const rows = await this.db.all(thoughtstreamApp.events.where({ + source: query.source, + sourceSequence: { gt: query.afterSequence }, + type: { in: query.eventTypes }, + privacy: { in: query.acceptedPrivacy }, + }).orderBy("sourceSequence", "asc").limit(query.limit ?? 1_000)); + return rows.map(eventFromJazz); + } + + subscribeConsumerEvents(query: ConsumerEventQuery, onEvents: (events: ThoughtEvent[]) => void): () => void { + if (query.eventTypes.length === 0 || query.acceptedPrivacy.length === 0) return () => undefined; + // The callback receives only rows added by each Jazz query delta. Durable + // consumer progress, not an accumulated in-memory subscription snapshot, + // decides which work remains after restart. + return this.db.subscribeAll(thoughtstreamApp.events.where({ + source: query.source, + sourceSequence: { gt: query.afterSequence }, + type: { in: query.eventTypes }, + privacy: { in: query.acceptedPrivacy }, + }).orderBy("sourceSequence", "asc"), ({ delta }) => onEvents(delta.flatMap((change) => ( + change.kind === RowChangeKind.Added ? [eventFromJazz(change.item)] : [] + )))); + } + + async settleConsumerSuccess( + settlement: ConsumerSuccessSettlement, + ): Promise<{ outputEvent: ThoughtEvent; completedEvent: ThoughtEvent }> { + const [outputEvent, completedEvent] = await this.settleConsumerTerminal( + settlement.run, + settlement.inputEvent, + [settlement.output, settlement.completed], + settlement.progress, + ); + if (!outputEvent || !completedEvent) throw new Error("Consumer success settlement did not produce both receipts"); + return { outputEvent, completedEvent }; + } + + async settleConsumerFailure(settlement: ConsumerFailureSettlement): Promise { + const [terminal] = await this.settleConsumerTerminal( + settlement.run, + settlement.inputEvent, + [settlement.terminal], + settlement.progress, + ); + if (!terminal) throw new Error("Consumer failure settlement did not produce a terminal receipt"); + return terminal; + } + + async settleConsumerAbandonment(run: AgentRun, terminal: EventCandidate): Promise { + const [event] = await this.settleRunAndEvents(run, [terminal]); + if (!event) throw new Error("Consumer abandonment did not produce a terminal receipt"); + return event; + } + async flush(): Promise { this.context.flush(); } @@ -277,6 +396,72 @@ export class JazzThoughtStore implements ThoughtStore { await this.context.shutdown(); } + private async settleConsumerTerminal( + run: AgentRun, + inputEvent: ThoughtEvent, + candidates: EventCandidate[], + progress: ConsumerProgress, + ): Promise { + if (progress.lastSequence !== inputEvent.sourceSequence || progress.lastEventId !== inputEvent.id) { + throw new Error("Consumer progress must settle exactly at the terminal input event"); + } + if (progress.source !== inputEvent.source) throw new Error("Consumer progress source does not match input event source"); + return this.settleRunAndEvents(run, candidates, progress); + } + + private async settleRunAndEvents( + run: AgentRun, + candidates: EventCandidate[], + progress?: ConsumerProgress, + ): Promise { + const outputSource = candidates[0]?.source; + if (!outputSource || candidates.some((candidate) => candidate.source !== outputSource)) { + throw new Error("Consumer settlement receipts must share one output source"); + } + const settledAt = new Date().toISOString(); + const result = await this.db.transaction(async (tx) => { + const sourceRow = await tx.one(thoughtstreamApp.sources.where({ key: outputSource }).limit(1)); + if (sourceRow && String(sourceRow.kind) !== candidates[0]!.sourceKind) { + throw new Error(`Source kind changed for ${outputSource}`); + } + let sequence = Number(sourceRow?.lastSequence ?? 0); + const events: ThoughtEvent[] = []; + for (const candidate of candidates) { + const id = eventId(candidate); + const existing = await tx.one(thoughtstreamApp.events.where({ key: id }).limit(1)); + if (existing) { + const event = eventFromJazz(existing); + assertSameEventIdentity(event, candidate); + events.push(event); + continue; + } + sequence += 1; + const event = buildEvent(candidate, id, sequence, settledAt, this.runtimeRevision, this.registry); + tx.insert(thoughtstreamApp.events, eventToJazz(event), { id: jazzRowId("event", id) }); + events.push(event); + } + const sourceData = { + key: outputSource, + kind: candidates[0]!.sourceKind, + enabled: true, + configJson: sourceRow ? String(sourceRow.configJson) : "{}", + lastSequence: sequence, + createdAt: sourceRow ? String(sourceRow.createdAt) : settledAt, + updatedAt: settledAt, + }; + if (sourceRow) tx.update(thoughtstreamApp.sources, String(sourceRow.id), sourceData); + else tx.insert(thoughtstreamApp.sources, sourceData, { id: jazzRowId("source", outputSource) }); + + const runRow = await tx.one(thoughtstreamApp.runs.where({ key: run.id }).limit(1)); + if (!runRow) throw new Error(`Cannot settle missing run ${run.id}`); + tx.update(thoughtstreamApp.runs, String(runRow.id), runToJazz(run)); + if (progress) await upsertProgressInTransaction(tx, progress); + return events; + }); + await result.wait({ tier: this.durabilityTier }); + return result.value; + } + private async upsert(table: JazzTable, query: Record, data: Record): Promise { const existing = await this.db.one(table.where(query).limit(1) as never); if (existing && typeof existing === "object" && "id" in existing) { @@ -294,6 +479,7 @@ export class JazzThoughtStore implements ThoughtStore { function eventToJazz(event: ThoughtEvent): Record { return { key: event.id, + sourceSequence: event.sourceSequence, type: event.type, schemaVersion: event.schemaVersion, source: event.source, @@ -319,6 +505,7 @@ function eventFromJazz(row: Record): ThoughtEvent { const traceId = String(row.traceKey ?? ""); return { id: String(row.key), + sourceSequence: Number(row.sourceSequence ?? 0), type: String(row.type), schemaVersion: Number(row.schemaVersion), source: String(row.source), @@ -360,15 +547,12 @@ function agentFromJazz(row: Record): StoredAgent { return { id: String(row.key), version: Number(row.version), enabled: Boolean(row.enabled), spec: parseJsonObject(String(row.specJson)), specHash: String(row.specHash), updatedAt: String(row.updatedAt) }; } -function matchFromJazz(row: Record): AgentMatch { - return { id: String(row.key), eventId: String(row.eventKey), agentId: String(row.agentKey), agentVersion: Number(row.agentVersion), matcherRevision: String(row.matcherRevision), createdAt: String(row.createdAt) }; -} - function runToJazz(run: AgentRun): Record { return { - key: run.id, matchKey: run.matchId, agentKey: run.agentId, agentVersion: run.agentVersion, status: run.status, + key: run.id, executionKey: run.executionKey, triggerEventKey: run.triggerEventId, + agentKey: run.agentId, agentVersion: run.agentVersion, status: run.status, inputEventIdsJson: canonicalJson(run.inputEventIds), outputEventIdsJson: canonicalJson(run.outputEventIds), attempt: run.attempt, - workerId: run.workerId ?? "", leaseId: run.leaseId ?? "", leaseExpiresAt: run.leaseExpiresAt ?? "", provider: run.provider, + provider: run.provider, model: run.model, adapterRevision: run.adapterRevision ?? "", promptHash: run.promptHash, contextManifestJson: canonicalJson(run.contextManifest), resultJson: run.result ? canonicalJson(run.result) : "", errorText: run.errorText ?? "", createdAt: run.createdAt, startedAt: run.startedAt ?? "", completedAt: run.completedAt ?? "", updatedAt: run.updatedAt, @@ -380,11 +564,9 @@ function runFromJazz(row: Record): AgentRun { const outputEventIds = JSON.parse(String(row.outputEventIdsJson)) as string[]; const resultText = String(row.resultJson ?? ""); return { - id: String(row.key), matchId: String(row.matchKey), agentId: String(row.agentKey), agentVersion: Number(row.agentVersion), + id: String(row.key), executionKey: String(row.executionKey ?? row.key), triggerEventId: String(row.triggerEventKey ?? ""), + agentId: String(row.agentKey), agentVersion: Number(row.agentVersion), status: String(row.status) as AgentRun["status"], inputEventIds, outputEventIds, attempt: Number(row.attempt), - ...(String(row.workerId ?? "") ? { workerId: String(row.workerId) } : {}), - ...(String(row.leaseId ?? "") ? { leaseId: String(row.leaseId) } : {}), - ...(String(row.leaseExpiresAt ?? "") ? { leaseExpiresAt: String(row.leaseExpiresAt) } : {}), provider: String(row.provider), model: String(row.model), ...(String(row.adapterRevision ?? "") ? { adapterRevision: String(row.adapterRevision) } : {}), promptHash: String(row.promptHash), contextManifest: parseJsonObject(String(row.contextManifestJson)), @@ -419,3 +601,129 @@ function sourceCursorFromJazz(row: Record): SourceCursor { updatedAt: String(row.updatedAt), }; } + +function buildEvent( + candidate: EventCandidate, + id: string, + sourceSequence: number, + defaultObservedAt: string, + runtimeRevision: string, + registry: EventRegistry, +): ThoughtEvent { + const payload = registry.validate(candidate.type, candidate.schemaVersion, candidate.payload); + return { + id, + sourceSequence, + type: candidate.type, + schemaVersion: candidate.schemaVersion, + source: candidate.source, + sourceKind: candidate.sourceKind, + externalId: candidate.externalId, + idempotencyKey: candidate.idempotencyKey, + occurredAt: candidate.occurredAt, + observedAt: candidate.observedAt ?? defaultObservedAt, + actor: candidate.actor, + rootEventId: candidate.rootEventId ?? id, + ...(candidate.parentEventId ? { parentEventId: candidate.parentEventId } : {}), + correlationId: candidate.correlationId, + privacy: candidate.privacy, + payload, + payloadHash: hashJson(payload), + ...(candidate.traceId ? { traceId: candidate.traceId } : {}), + createdByRuntime: candidate.createdByRuntime ?? runtimeRevision, + }; +} + +function eventId(candidate: EventCandidate): string { + return `evt_${sha256(`${candidate.source}\u0000${candidate.idempotencyKey}`).slice(0, 48)}`; +} + +function jazzRowId(namespace: string, key: string): string { + const bytes = createHash("sha256").update(`${namespace}\u0000${key}`).digest().subarray(0, 16); + bytes[6] = (bytes[6]! & 0x0f) | 0x50; + bytes[8] = (bytes[8]! & 0x3f) | 0x80; + const hex = bytes.toString("hex"); + return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20)}`; +} + +function assertSameEventIdentity(event: ThoughtEvent, candidate: EventCandidate): void { + if (event.source !== candidate.source + || event.idempotencyKey !== candidate.idempotencyKey + || event.type !== candidate.type + || event.schemaVersion !== candidate.schemaVersion) { + throw new Error(`Deterministic event identity conflict for ${event.id}`); + } + if (event.payloadHash !== hashJson(candidate.payload)) { + throw new Error(`Idempotency key reused with different payload for ${event.id}`); + } +} + +function sourceStateFromJazz(row: Record): SourceState { + return { + id: String(row.key), + kind: String(row.kind), + enabled: Boolean(row.enabled), + config: parseJsonObject(String(row.configJson)), + lastSequence: Number(row.lastSequence ?? 0), + createdAt: String(row.createdAt), + updatedAt: String(row.updatedAt), + }; +} + +function consumerProgressFromJazz(row: Record): ConsumerProgress { + return { + id: String(row.key), + consumerId: String(row.consumerKey), + consumerVersion: Number(row.consumerVersion), + source: String(row.source), + lastSequence: Number(row.lastSequence), + lastEventId: String(row.lastEventKey), + updatedAt: String(row.updatedAt), + }; +} + +function cursorToJazz(cursor: SourceCursor): Record { + return { + key: cursor.id, + source: cursor.source, + cursorJson: canonicalJson(cursor.cursor), + lastSuccessAt: cursor.lastSuccessAt ?? "", + lastFailureAt: cursor.lastFailureAt ?? "", + lastError: cursor.lastError ?? "", + updatedAt: cursor.updatedAt, + }; +} + +async function upsertCursorInTransaction(tx: TransactionScope, cursor: SourceCursor): Promise { + const row = await tx.one(thoughtstreamApp.sourceCursors.where({ key: cursor.id }).limit(1)); + const data = cursorToJazz(cursor); + if (row) tx.update(thoughtstreamApp.sourceCursors, String(row.id), data); + else tx.insert(thoughtstreamApp.sourceCursors, data, { id: jazzRowId("cursor", cursor.id) }); +} + +async function upsertProgressInTransaction(tx: TransactionScope, progress: ConsumerProgress): Promise { + const row = await tx.one(thoughtstreamApp.consumerProgress.where({ key: progress.id }).limit(1)); + const data = { + key: progress.id, + consumerKey: progress.consumerId, + consumerVersion: progress.consumerVersion, + source: progress.source, + lastSequence: progress.lastSequence, + lastEventKey: progress.lastEventId, + updatedAt: progress.updatedAt, + }; + if (row) { + if (Number(row.lastSequence) > progress.lastSequence) throw new Error(`Consumer progress regression for ${progress.id}`); + if (Number(row.lastSequence) === progress.lastSequence && String(row.lastEventKey) !== progress.lastEventId) { + throw new Error(`Consumer progress sequence conflict for ${progress.id}`); + } + if (String(row.consumerKey) !== progress.consumerId + || Number(row.consumerVersion) !== progress.consumerVersion + || String(row.source) !== progress.source) { + throw new Error(`Consumer progress identity conflict for ${progress.id}`); + } + tx.update(thoughtstreamApp.consumerProgress, String(row.id), data); + } else { + tx.insert(thoughtstreamApp.consumerProgress, data, { id: jazzRowId("progress", progress.id) }); + } +} diff --git a/src/pipeline/filesystem.ts b/src/pipeline/filesystem.ts deleted file mode 100644 index 923848f..0000000 --- a/src/pipeline/filesystem.ts +++ /dev/null @@ -1,24 +0,0 @@ -import type { ThoughtAgentDeclaration } from "../agents/types.js"; -import { ThoughtAgentRuntime, type ProcessEventResult } from "../agents/runtime.js"; -import { FilesystemConnector, type FilesystemScanResult } from "../connectors/filesystem.js"; -import type { RootActivityProjection } from "../projections/activity.js"; -import { rebuildRootActivity } from "../projections/activity.js"; -import type { ThoughtStore } from "../store/types.js"; - -export interface FilesystemPipelineResult { - scan: FilesystemScanResult; - runs: ProcessEventResult[]; - activity: RootActivityProjection; -} - -export async function runFilesystemPipeline( - store: ThoughtStore, - connector: FilesystemConnector, - declarations: ThoughtAgentDeclaration[], -): Promise { - const scan = await connector.scan(store); - const runtime = new ThoughtAgentRuntime(store); - const runs: ProcessEventResult[] = []; - for (const event of scan.events) runs.push(...await runtime.processEvent(event, declarations)); - return { scan, runs, activity: await rebuildRootActivity(store) }; -} diff --git a/src/projections/activity.ts b/src/projections/activity.ts index 6c89a23..6a872ae 100644 --- a/src/projections/activity.ts +++ b/src/projections/activity.ts @@ -1,5 +1,16 @@ import type { JsonObject } from "../core/json.js"; -import type { ThoughtStore } from "../store/types.js"; +import type { JazzThoughtStore } from "../jazz/store.js"; +import type { ThoughtEvent } from "../events/types.js"; +import type { AgentRun } from "../store/types.js"; + +export interface ActivityConsumerRun { + id: string; + agentId: string; + status: AgentRun["status"]; + kind: "rule" | "model"; + description: string; + outputCount: number; +} export interface ActivityItem { id: string; @@ -11,6 +22,8 @@ export interface ActivityItem { parentEventId?: string; privacy: string; summary: string; + descendantEventCount: number; + consumerRuns: ActivityConsumerRun[]; } export interface RootActivityProjection { @@ -19,11 +32,24 @@ export interface RootActivityProjection { items: ActivityItem[]; } -export async function buildRootActivity(store: ThoughtStore, limit = 100): Promise { - const events = await store.listEvents(); +export async function buildRootActivity(store: JazzThoughtStore, limit = 100): Promise { + const [events, runs] = await Promise.all([store.listEvents(), store.listRuns()]); + const eventsById = new Map(events.map((event) => [event.id, event])); + const descendantsByRoot = new Map(); + const runsByRoot = new Map(); const byType: Record = {}; - for (const event of events) byType[event.type] = (byType[event.type] ?? 0) + 1; - const items = events.slice(-limit).reverse().map((event) => ({ + for (const event of events) { + byType[event.type] = (byType[event.type] ?? 0) + 1; + if (event.id !== event.rootEventId) { + descendantsByRoot.set(event.rootEventId, (descendantsByRoot.get(event.rootEventId) ?? 0) + 1); + } + } + for (const run of runs) { + const rootIds = new Set(run.inputEventIds.map((id) => eventsById.get(id)?.rootEventId).filter((id): id is string => Boolean(id))); + for (const rootId of rootIds) runsByRoot.set(rootId, [...(runsByRoot.get(rootId) ?? []), run]); + } + const roots = events.filter((event) => event.id === event.rootEventId); + const items = roots.slice(-limit).reverse().map((event) => ({ id: event.id, type: event.type, occurredAt: event.occurredAt, @@ -32,12 +58,21 @@ export async function buildRootActivity(store: ThoughtStore, limit = 100): Promi rootEventId: event.rootEventId, ...(event.parentEventId ? { parentEventId: event.parentEventId } : {}), privacy: event.privacy, - summary: summarize(event.type, event.payload), + summary: summarizeRoot(event), + descendantEventCount: descendantsByRoot.get(event.id) ?? 0, + consumerRuns: (runsByRoot.get(event.id) ?? []).map((run) => ({ + id: run.id, + agentId: run.agentId, + status: run.status, + kind: runKind(run), + description: describeRunResult(run, run.inputEventIds.map((id) => eventsById.get(id)).filter((input): input is ThoughtEvent => Boolean(input))), + outputCount: run.outputEventIds.length, + })), })); return { totalEvents: events.length, byType, items }; } -export async function rebuildRootActivity(store: ThoughtStore, limit = 100): Promise { +export async function rebuildRootActivity(store: JazzThoughtStore, limit = 100): Promise { const projection = await buildRootActivity(store, limit); const events = await store.listEvents(); const latest = events.at(-1); @@ -51,9 +86,74 @@ export async function rebuildRootActivity(store: ThoughtStore, limit = 100): Pro return projection; } -function summarize(type: string, payload: JsonObject): string { - if (typeof payload.summary === "string") return payload.summary; - if (typeof payload.path === "string") return `${type.split(".").at(-1) ?? "observed"}: ${payload.path}`; - if (typeof payload.agentId === "string") return `${type.split(".").at(-1) ?? "agent"}: ${payload.agentId}`; - return type; +export function describeRunResult(run: AgentRun, inputs: ThoughtEvent[] = []): string { + if (run.status === "failed") return `Failed: ${run.errorText ?? "no error detail recorded"}`; + if (!run.result) return `${capitalize(run.status)}; no result recorded yet.`; + const tags = stringArray(run.result.tags); + const importance = typeof run.result.importance === "string" ? run.result.importance : undefined; + const path = inputs.map((event) => event.payload.path).find((value): value is string => typeof value === "string"); + const classification = tags.includes("specification") + ? "a specification change" + : tags.includes("agent-definition") + ? "an agent-definition change" + : tags.includes("md") + ? "an ordinary Markdown file change" + : undefined; + const resultSummary = typeof run.result.summary === "string" ? run.result.summary : "Completed without a summary"; + const firstSentence = classification + ? `Classified ${path ?? "the event"} as ${classification}${importance ? ` (${importance} importance)` : ""}.` + : `Produced: ${resultSummary}.`; + const recommendation = isObject(run.result.recommendation) && typeof run.result.recommendation.proposedAction === "string" + ? ` Proposed: ${run.result.recommendation.proposedAction} The proposal was not executed.` + : " No recommendation was produced."; + const externalActions = run.contextManifest.externalActions === false + ? " External actions were disabled." + : ""; + return `${firstSentence}${recommendation}${externalActions}`; +} + +function summarizeRoot(event: ThoughtEvent): string { + if (typeof event.payload.path === "string") { + return `${event.payload.path} ${event.type.split(".").at(-1) ?? "observed"}`; + } + if (event.type === "stream.thought.source.atproto.commit") { + const operation = typeof event.payload.operation === "string" ? event.payload.operation : "observed"; + const collection = typeof event.payload.collection === "string" ? event.payload.collection : "ATProto record"; + const record = isObject(event.payload.record) ? event.payload.record : undefined; + const text = record && typeof record.text === "string" ? record.text.replaceAll(/\s+/g, " ").trim() : undefined; + const subject = record && isObject(record.subject) ? record.subject : undefined; + const subjectUri = subject && typeof subject.uri === "string" ? subject.uri : undefined; + const noun = collection === "app.bsky.feed.post" ? "Post" : collection === "app.bsky.feed.like" ? "Like" : collection; + if (text) return `${capitalize(operation)} ${noun.toLowerCase()}: ${truncate(text, 180)}`; + if (subjectUri) return `${capitalize(operation)} ${noun.toLowerCase()}: ${atUriToWebUrl(subjectUri)}`; + return `${capitalize(operation)} ${noun}`; + } + if (typeof event.payload.title === "string") return event.payload.title; + if (typeof event.payload.summary === "string") return event.payload.summary; + return event.type; +} + +function runKind(run: AgentRun): ActivityConsumerRun["kind"] { + return run.provider === "deterministic" && run.model === "deterministic" ? "rule" : "model"; +} + +function stringArray(value: unknown): string[] { + return Array.isArray(value) ? value.filter((item): item is string => typeof item === "string") : []; +} + +function isObject(value: unknown): value is JsonObject { + return Boolean(value && typeof value === "object" && !Array.isArray(value)); +} + +function capitalize(value: string): string { + return value.charAt(0).toUpperCase() + value.slice(1); +} + +function truncate(value: string, limit: number): string { + return value.length <= limit ? value : `${value.slice(0, limit - 1).trimEnd()}…`; +} + +function atUriToWebUrl(value: string): string { + const match = value.match(/^at:\/\/([^/]+)\/app\.bsky\.feed\.post\/([^/]+)$/); + return match ? `https://bsky.app/profile/${match[1]}/post/${match[2]}` : value; } diff --git a/src/projections/source-health.ts b/src/projections/source-health.ts index 2970632..fad5b08 100644 --- a/src/projections/source-health.ts +++ b/src/projections/source-health.ts @@ -1,5 +1,5 @@ import type { JsonObject } from "../core/json.js"; -import type { ThoughtStore } from "../store/types.js"; +import type { JazzThoughtStore } from "../jazz/store.js"; export type SourceHealthStatus = "healthy" | "failing" | "polling" | "unknown"; @@ -26,7 +26,7 @@ const CONNECTOR_EVENT_TYPES = [ "stream.thought.connector.recovered", ]; -export async function buildSourceHealth(store: ThoughtStore): Promise { +export async function buildSourceHealth(store: JazzThoughtStore): Promise { const [cursors, events] = await Promise.all([ store.listSourceCursors(), store.listEvents({ types: CONNECTOR_EVENT_TYPES }), diff --git a/src/runtime/manifest.ts b/src/runtime/manifest.ts index 1858ee8..17fb2ae 100644 --- a/src/runtime/manifest.ts +++ b/src/runtime/manifest.ts @@ -2,7 +2,7 @@ import fs from "node:fs/promises"; import path from "node:path"; import YAML from "yaml"; import { z } from "zod"; -import { canonicalJson, sha256 } from "../core/json.js"; +import { canonicalJson, sha256, type JsonObject } from "../core/json.js"; const privacySchema = z.enum(["public-source", "private", "sensitive"]); const idSchema = z.string().min(1).max(200).regex(/^[a-z0-9][a-z0-9._:-]*$/); @@ -47,6 +47,55 @@ const telegramSpoolSourceSchema = z.object({ pollOnStart: z.boolean().default(true), }).strict(); +const telegramBootMessageSchema = z.object({ + enabled: z.boolean().default(false), + text: z.string().min(1).max(4_096), +}).strict(); + +const telegramNotificationSchema = z.object({ + enabled: z.boolean().default(false), + includeNormal: z.boolean().default(true), + runStatuses: z.array(z.enum(["completed", "failed"])).min(1).max(2).default(["completed"]), + allowedSources: z.array(idSchema).min(1).max(100), + allowedActors: z.array(z.string().min(1).max(500)).max(100).default([]), + maxMessagesPerWindow: z.number().int().positive().max(100).default(3), + windowMs: z.number().int().min(1_000).max(24 * 60 * 60 * 1_000).default(60_000), + likeDigestDelayMs: z.number().int().nonnegative().max(24 * 60 * 60 * 1_000).default(60_000), + maxLikesPerDigest: z.number().int().positive().max(100).default(10), + failureTranscriptMaxChars: z.number().int().positive().max(3_500).default(3_000), +}).strict(); + +const telegramBotChannelSchema = z.object({ + id: z.string().regex(/^-?[0-9]+$/, "Telegram chat id must be an integer string"), + enabled: z.boolean().default(true), + bootMessage: telegramBootMessageSchema.optional(), + notifications: telegramNotificationSchema.optional(), +}).strict(); + +const telegramBotSourceSchema = z.object({ + ...sourceBase, + kind: z.literal("telegram-bot"), + tokenEnv: z.string().regex(/^[A-Z_][A-Z0-9_]*$/, "tokenEnv must be an environment variable name"), + pollTimeoutSeconds: z.number().int().nonnegative().max(30).default(5), + requestTimeoutMs: z.number().int().min(1_000).max(120_000).default(35_000), + dispatchIntervalMs: z.number().int().min(100).max(60_000).default(1_000), + channels: z.array(telegramBotChannelSchema).min(1).max(100), +}).strict().superRefine((value, context) => { + const seen = new Set(); + for (const [index, channel] of value.channels.entries()) { + if (seen.has(channel.id)) { + context.addIssue({ code: "custom", path: ["channels", index, "id"], message: `Duplicate Telegram channel id: ${channel.id}` }); + } + seen.add(channel.id); + if (channel.notifications?.enabled && channel.notifications.allowedSources.length === 0) { + context.addIssue({ code: "custom", path: ["channels", index, "notifications", "allowedSources"], message: "Enabled notifications require an allowed source" }); + } + } + if (!value.channels.some((channel) => channel.enabled)) { + context.addIssue({ code: "custom", path: ["channels"], message: "Telegram bot source requires at least one enabled channel" }); + } +}); + const jetstreamSourceSchema = z.object({ ...sourceBase, kind: z.literal("jetstream"), @@ -64,6 +113,7 @@ const sourceSchema = z.discriminatedUnion("kind", [ filesystemSourceSchema, rssSourceSchema, telegramSpoolSourceSchema, + telegramBotSourceSchema, jetstreamSourceSchema, ]); @@ -95,6 +145,7 @@ export type ThoughtStreamSource = ThoughtStreamManifest["sources"][number]; export type FilesystemSourceManifest = Extract; export type RssSourceManifest = Extract; export type TelegramSpoolSourceManifest = Extract; +export type TelegramBotSourceManifest = Extract; export type JetstreamSourceManifest = Extract; export interface LoadedThoughtStreamManifest { @@ -123,7 +174,7 @@ export async function loadThoughtStreamManifest( if (!isWithin(projectRoot, agentDirectory)) { throw new Error(`Agent directory escapes project root: ${manifest.agents.directory}`); } - return { path: manifestPath, manifest, hash: sha256(canonicalJson(manifest)) }; + return { path: manifestPath, manifest, hash: sha256(canonicalJson(asJsonObject(manifest))) }; } export function resolveProjectPath(projectRoot: string, value: string): string { @@ -134,3 +185,11 @@ function isWithin(root: string, candidate: string): boolean { const relative = path.relative(path.resolve(root), candidate); return relative === "" || (!relative.startsWith("..") && !path.isAbsolute(relative)); } + +function asJsonObject(value: unknown): JsonObject { + const normalized = JSON.parse(JSON.stringify(value)) as unknown; + if (!normalized || typeof normalized !== "object" || Array.isArray(normalized)) { + throw new Error("Expected a JSON object"); + } + return normalized as JsonObject; +} diff --git a/src/store/types.ts b/src/store/types.ts index 476e090..d78cd07 100644 --- a/src/store/types.ts +++ b/src/store/types.ts @@ -1,8 +1,5 @@ import type { JsonObject } from "../core/json.js"; import type { - AppendEventResult, - CurrentDocument, - DocumentVersion, EventCandidate, ThoughtEvent, } from "../events/types.js"; @@ -16,29 +13,18 @@ export interface StoredAgent { updatedAt: string; } -export interface AgentMatch { - id: string; - eventId: string; - agentId: string; - agentVersion: number; - matcherRevision: string; - createdAt: string; -} - -export type AgentRunStatus = "queued" | "leased" | "running" | "completed" | "failed" | "blocked" | "abandoned"; +export type AgentRunStatus = "running" | "completed" | "failed" | "blocked" | "abandoned"; export interface AgentRun { id: string; - matchId: string; + executionKey: string; + triggerEventId: string; agentId: string; agentVersion: number; status: AgentRunStatus; inputEventIds: string[]; outputEventIds: string[]; attempt: number; - workerId?: string; - leaseId?: string; - leaseExpiresAt?: string; provider: string; model: string; adapterRevision?: string; @@ -52,6 +38,57 @@ export interface AgentRun { updatedAt: string; } +export interface SourceState { + id: string; + kind: string; + enabled: boolean; + config: JsonObject; + lastSequence: number; + createdAt: string; + updatedAt: string; +} + +export interface ConsumerProgress { + id: string; + consumerId: string; + consumerVersion: number; + source: string; + lastSequence: number; + lastEventId: string; + updatedAt: string; +} + +export interface ConsumerEventQuery { + consumerId: string; + consumerVersion: number; + source: string; + eventTypes: string[]; + acceptedPrivacy: string[]; + afterSequence: number; + limit?: number | undefined; +} + +export interface ProducerBatchResult { + events: ThoughtEvent[]; + inserted: ThoughtEvent[]; + unchanged: ThoughtEvent[]; +} + +export interface ConsumerSuccessSettlement { + run: AgentRun; + inputEvent: ThoughtEvent; + output: EventCandidate; + completed: EventCandidate; + progress: ConsumerProgress; +} + +export interface ConsumerFailureSettlement { + run: AgentRun; + inputEvent: ThoughtEvent; + terminal: EventCandidate; + progress: ConsumerProgress; +} + export interface TraceChunk { id: string; runId: string; @@ -78,29 +115,3 @@ export interface SourceCursor { lastError?: string; updatedAt: string; } - -export interface ThoughtStore { - appendEvent(candidate: EventCandidate): Promise; - getEvent(id: string): Promise; - listEvents(options?: { types?: string[]; source?: string; limit?: number }): Promise; - appendDocumentVersion(version: DocumentVersion): Promise; - getDocumentVersion(id: string): Promise; - listCurrentDocuments(source: string): Promise; - upsertCurrentDocument(document: CurrentDocument): Promise; - upsertAgent(agent: StoredAgent): Promise; - listAgents(): Promise; - insertMatch(match: AgentMatch): Promise; - getMatch(id: string): Promise; - upsertRun(run: AgentRun): Promise; - getRun(id: string): Promise; - listRuns(): Promise; - appendTrace(chunk: TraceChunk): Promise; - listTrace(runId: string): Promise; - upsertProjection(projection: Projection): Promise; - getProjection(id: string): Promise; - upsertSourceCursor(cursor: SourceCursor): Promise; - getSourceCursor(id: string): Promise; - listSourceCursors(): Promise; - flush(): Promise; - close(): Promise; -} diff --git a/src/training/judgments.ts b/src/training/judgments.ts new file mode 100644 index 0000000..c604b97 --- /dev/null +++ b/src/training/judgments.ts @@ -0,0 +1,247 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import { createHash } from "node:crypto"; +import { canonicalJson, type JsonObject } from "../core/json.js"; +import { stableKey } from "../core/ids.js"; +import type { ThoughtEvent } from "../events/types.js"; +import type { JazzThoughtStore } from "../jazz/store.js"; +import type { AgentRun, TraceChunk } from "../store/types.js"; + +export type JudgmentKind = "accept" | "reject" | "correct" | "prefer"; + +export interface RecordJudgmentInput { + runId: string; + kind: JudgmentKind; + criterion: string; + criterionVersion: number; + exportEligible: boolean; + comparedRunId?: string | undefined; + replacementOutput?: JsonObject | undefined; + notes?: string | undefined; + actor?: string | undefined; +} + +export interface TrainingExample { + format: "thoughtstream.training-example.v1"; + kind: JudgmentKind; + judgment: { + eventId: string; + criterion: string; + criterionVersion: number; + observedAt: string; + }; + input: { + event: ThoughtEvent; + contextManifest: JsonObject; + }; + trajectory: TraceChunk[]; + comparedTrajectory?: TraceChunk[] | undefined; + chosen?: JsonObject | undefined; + rejected?: JsonObject | undefined; + provenance: { + runId: string; + outputEventId: string; + agentId: string; + agentVersion: number; + provider: string; + model: string; + adapterRevision?: string | undefined; + promptHash: string; + comparedRunId?: string | undefined; + comparedOutputEventId?: string | undefined; + }; +} + +export interface TrainingDatasetManifest { + format: "thoughtstream.training-dataset-manifest.v1"; + datasetId: string; + generatedAt: string; + dataFile: string; + sha256: string; + examples: number; + judgmentEventIds: string[]; + kinds: Partial>; + models: string[]; +} + +export async function recordJudgment(store: JazzThoughtStore, input: RecordJudgmentInput): Promise { + validateInput(input); + const run = await requireCompletedRun(store, input.runId); + const outputEventId = requireOutputEventId(run); + const sourceEvent = await requireEvent(store, run.triggerEventId); + let comparedRun: AgentRun | undefined; + let comparedOutputEventId: string | undefined; + if (input.kind === "prefer") { + comparedRun = await requireCompletedRun(store, input.comparedRunId!); + if (comparedRun.triggerEventId !== run.triggerEventId) { + throw new Error("Preference judgments require runs over the same trigger event"); + } + comparedOutputEventId = requireOutputEventId(comparedRun); + } + const payload: JsonObject = { + runId: run.id, + outputEventId, + kind: input.kind, + criterion: input.criterion, + criterionVersion: input.criterionVersion, + exportEligible: input.exportEligible, + ...(comparedRun ? { comparedRunId: comparedRun.id } : {}), + ...(comparedOutputEventId ? { comparedOutputEventId } : {}), + ...(input.replacementOutput ? { replacementOutput: input.replacementOutput } : {}), + ...(input.notes ? { notes: input.notes } : {}), + }; + const idempotencyKey = stableKey( + "judgment", + run.id, + input.kind, + input.criterion, + String(input.criterionVersion), + comparedRun?.id ?? "none", + ); + const result = await store.appendEvent({ + type: "stream.thought.judgment.training-example", + schemaVersion: 1, + source: "judgment:local", + sourceKind: "system", + externalId: idempotencyKey, + idempotencyKey, + occurredAt: new Date().toISOString(), + actor: input.actor ?? "operator:local", + rootEventId: sourceEvent.rootEventId, + parentEventId: outputEventId, + correlationId: run.id, + privacy: sourceEvent.privacy, + payload, + createdByRuntime: "thoughtstream-judgment-v1", + }); + return result.event; +} + +export async function projectTrainingExamples(store: JazzThoughtStore): Promise { + const judgments = await store.listEvents({ types: ["stream.thought.judgment.training-example"] }); + const examples: TrainingExample[] = []; + for (const judgment of judgments) { + if (judgment.payload.exportEligible !== true) continue; + const run = await requireCompletedRun(store, stringField(judgment.payload.runId, "runId")); + const inputEvent = await requireEvent(store, run.triggerEventId); + const outputEventId = requireOutputEventId(run); + const kind = judgment.payload.kind as JudgmentKind; + const replacement = objectField(judgment.payload.replacementOutput); + let chosen: JsonObject | undefined; + let rejected: JsonObject | undefined; + let comparedRun: AgentRun | undefined; + if (kind === "accept") chosen = run.result; + if (kind === "reject") rejected = run.result; + if (kind === "correct") { + chosen = replacement; + rejected = run.result; + } + if (kind === "prefer") { + comparedRun = await requireCompletedRun(store, stringField(judgment.payload.comparedRunId, "comparedRunId")); + chosen = run.result; + rejected = comparedRun.result; + } + examples.push({ + format: "thoughtstream.training-example.v1", + kind, + judgment: { + eventId: judgment.id, + criterion: stringField(judgment.payload.criterion, "criterion"), + criterionVersion: numberField(judgment.payload.criterionVersion, "criterionVersion"), + observedAt: judgment.observedAt, + }, + input: { event: inputEvent, contextManifest: run.contextManifest }, + trajectory: await store.listTrace(run.id), + ...(comparedRun ? { comparedTrajectory: await store.listTrace(comparedRun.id) } : {}), + ...(chosen ? { chosen } : {}), + ...(rejected ? { rejected } : {}), + provenance: { + runId: run.id, + outputEventId, + agentId: run.agentId, + agentVersion: run.agentVersion, + provider: run.provider, + model: run.model, + ...(run.adapterRevision ? { adapterRevision: run.adapterRevision } : {}), + promptHash: run.promptHash, + ...(comparedRun ? { + comparedRunId: comparedRun.id, + comparedOutputEventId: requireOutputEventId(comparedRun), + } : {}), + }, + }); + } + return examples; +} + +export async function writeTrainingJsonl(destination: string, examples: TrainingExample[]): Promise { + const absolute = path.resolve(destination); + await fs.mkdir(path.dirname(absolute), { recursive: true }); + const temporary = `${absolute}.${process.pid}.tmp`; + const content = examples.map((example) => canonicalJson(example as unknown as JsonObject)).join("\n"); + const serialized = content ? `${content}\n` : ""; + await fs.writeFile(temporary, serialized, { flag: "w" }); + await fs.rename(temporary, absolute); + const sha256 = createHash("sha256").update(serialized).digest("hex"); + const manifest: TrainingDatasetManifest = { + format: "thoughtstream.training-dataset-manifest.v1", + datasetId: `sha256:${sha256}`, + generatedAt: new Date().toISOString(), + dataFile: path.basename(absolute), + sha256, + examples: examples.length, + judgmentEventIds: examples.map((example) => example.judgment.eventId), + kinds: examples.reduce>>((counts, example) => { + counts[example.kind] = (counts[example.kind] ?? 0) + 1; + return counts; + }, {}), + models: [...new Set(examples.map((example) => `${example.provenance.provider}:${example.provenance.model}${example.provenance.adapterRevision ? `@${example.provenance.adapterRevision}` : ""}`))].sort(), + }; + const manifestPath = `${absolute}.manifest.json`; + const manifestTemporary = `${manifestPath}.${process.pid}.tmp`; + await fs.writeFile(manifestTemporary, `${canonicalJson(manifest as unknown as JsonObject)}\n`, { flag: "w" }); + await fs.rename(manifestTemporary, manifestPath); + return manifest; +} + +function validateInput(input: RecordJudgmentInput): void { + if (!input.runId || !input.criterion) throw new Error("Judgments require a run id and criterion"); + if (!Number.isSafeInteger(input.criterionVersion) || input.criterionVersion <= 0) { + throw new Error("Judgment criterion version must be a positive integer"); + } + if (input.kind === "correct" && !input.replacementOutput) throw new Error("Correction judgments require replacement output"); + if (input.kind === "prefer" && !input.comparedRunId) throw new Error("Preference judgments require --compared-run"); +} + +async function requireCompletedRun(store: JazzThoughtStore, id: string): Promise { + const run = await store.getRun(id); + if (!run) throw new Error(`Run not found: ${id}`); + if (run.status !== "completed" || !run.result) throw new Error(`Run is not a completed training candidate: ${id}`); + return run; +} + +async function requireEvent(store: JazzThoughtStore, id: string): Promise { + const event = await store.getEvent(id); + if (!event) throw new Error(`Event not found: ${id}`); + return event; +} + +function requireOutputEventId(run: AgentRun): string { + const output = run.outputEventIds[0]; + if (!output) throw new Error(`Completed run has no output event: ${run.id}`); + return output; +} + +function stringField(value: unknown, name: string): string { + if (typeof value !== "string" || value.length === 0) throw new Error(`Judgment has invalid ${name}`); + return value; +} + +function numberField(value: unknown, name: string): number { + if (typeof value !== "number" || !Number.isSafeInteger(value)) throw new Error(`Judgment has invalid ${name}`); + return value; +} + +function objectField(value: unknown): JsonObject | undefined { + return value && typeof value === "object" && !Array.isArray(value) ? value as JsonObject : undefined; +} diff --git a/src/web/inspector.ts b/src/web/inspector.ts index 23e5a13..7f335a2 100644 --- a/src/web/inspector.ts +++ b/src/web/inspector.ts @@ -1,8 +1,10 @@ import http, { type IncomingMessage, type ServerResponse } from "node:http"; -import type { ThoughtStore } from "../store/types.js"; -import { buildRootActivity } from "../projections/activity.js"; +import type { JazzThoughtStore } from "../jazz/store.js"; +import { buildRootActivity, describeRunResult } from "../projections/activity.js"; import { buildSourceHealth } from "../projections/source-health.js"; import { auditRunEvidence } from "../agents/evidence.js"; +import type { AgentRun } from "../store/types.js"; +import type { ThoughtEvent } from "../events/types.js"; export interface InspectorServerOptions { host?: string; @@ -10,7 +12,7 @@ export interface InspectorServerOptions { } export async function startInspectorServer( - store: ThoughtStore, + store: JazzThoughtStore, options: InspectorServerOptions = {}, ): Promise { const host = options.host ?? "127.0.0.1"; @@ -31,7 +33,7 @@ export async function startInspectorServer( return server; } -async function handleRequest(store: ThoughtStore, request: IncomingMessage, response: ServerResponse): Promise { +async function handleRequest(store: JazzThoughtStore, request: IncomingMessage, response: ServerResponse): Promise { if (request.method !== "GET" && request.method !== "HEAD") { sendJson(response, 405, { error: "Inspector is read-only" }); return; @@ -65,12 +67,14 @@ async function handleRequest(store: ThoughtStore, request: IncomingMessage, resp const id = decodeURIComponent(eventMatch[1]!); const event = await store.getEvent(id); if (!event) return sendJson(response, 404, { error: `Event not found: ${id}` }); - const events = await store.listEvents(); + const [events, runs] = await Promise.all([store.listEvents(), store.listRuns()]); + const relevantRuns = runs.filter((run) => eventBelongsToRun(event, run)); sendJson(response, 200, { event, parent: event.parentEventId ? await store.getEvent(event.parentEventId) : undefined, children: events.filter((candidate) => candidate.parentEventId === event.id), root: event.rootEventId === event.id ? event : await store.getEvent(event.rootEventId), + agentActivities: await Promise.all(relevantRuns.map((run) => expandRun(store, run))), }); return; } @@ -80,7 +84,7 @@ async function handleRequest(store: ThoughtStore, request: IncomingMessage, resp const run = await store.getRun(id); if (!run) return sendJson(response, 404, { error: `Run not found: ${id}` }); const evidence = auditRunEvidence([run], await store.listEvents())[0]; - sendJson(response, 200, { run, trace: await store.listTrace(id), evidence }); + sendJson(response, 200, { ...(await expandRun(store, run)), evidence }); return; } sendJson(response, 404, { error: "Not found" }); @@ -102,26 +106,31 @@ export function renderInspectorHtml(): string { section { min-width:0; border-right:1px solid var(--line) } section:last-child { border-right:0 } .section-head { display:flex; justify-content:space-between; padding:10px 14px; border-bottom:1px solid var(--line); color:var(--muted); text-transform:uppercase; letter-spacing:.08em; font-size:11px } .item { width:100%; display:block; border:0; border-bottom:1px solid #1b242b; padding:10px 14px; background:transparent; color:inherit; text-align:left; font:inherit; cursor:pointer } - .item:hover,.item.active { background:#152028 } .type { color:var(--cyan); overflow-wrap:anywhere } .summary { margin:3px 0 } .meta { color:var(--muted); font-size:11px; overflow-wrap:anywhere } + .item:hover,.item.active { background:#152028 } .type { color:var(--cyan); overflow-wrap:anywhere } .summary { margin:3px 0; font-size:14px } .meta { color:var(--muted); font-size:11px; overflow-wrap:anywhere } + .consumer { margin:6px 0 2px; padding-left:9px; border-left:2px solid #35515c; color:#c7d2d9; font-size:12px } .consumer strong { color:var(--cyan) } .badge { display:inline-block; border:1px solid var(--line); padding:1px 5px; margin-left:5px; color:var(--amber) } .failed { color:var(--red) } #detail { padding:14px; position:sticky; top:57px; max-height:calc(100vh - 57px); overflow:auto } .empty { color:var(--muted); padding:30px 8px } h2 { font-size:14px; margin:0 0 12px } h3 { color:var(--muted); text-transform:uppercase; letter-spacing:.08em; font-size:11px; margin:20px 0 7px } pre { margin:0; padding:10px; border:1px solid var(--line); background:#0d1216; white-space:pre-wrap; overflow-wrap:anywhere; font:11px/1.5 inherit } .lineage { display:grid; gap:6px } .lineage button { color:var(--cyan); background:transparent; border:1px solid var(--line); padding:7px; text-align:left; cursor:pointer; overflow-wrap:anywhere } + .work { border:1px solid #35515c; background:#0e181d; padding:12px; margin:8px 0 14px } .work-head { display:flex; align-items:center; gap:8px; margin-bottom:8px } .work-head strong { color:var(--cyan) } + .work-summary { font-size:14px; margin:5px 0 10px } .work-label { color:var(--muted); font-size:10px; text-transform:uppercase; letter-spacing:.08em; margin-top:10px } .proposal { border-left:2px solid var(--amber); padding-left:9px; margin-top:5px } .output-list { display:grid; gap:5px; margin-top:5px } .output-list button { color:var(--cyan); background:transparent; border:1px solid var(--line); padding:7px; text-align:left; cursor:pointer; overflow-wrap:anywhere } + .facts { display:flex; flex-wrap:wrap; gap:5px 12px; color:var(--muted); font-size:11px; margin-top:10px } .facts strong { color:var(--text); font-weight:normal } + details { border-top:1px solid var(--line); margin-top:12px; padding-top:9px } summary { color:var(--muted); cursor:pointer; text-transform:uppercase; letter-spacing:.08em; font-size:11px } details pre { margin-top:7px } nav { display:flex; gap:8px } nav button { background:transparent; color:var(--muted); border:0; border-bottom:1px solid transparent; font:inherit; padding:0 0 3px; cursor:pointer } nav button.active { color:var(--text); border-color:var(--cyan) } @media(max-width:800px){ main{grid-template-columns:1fr} section{border-right:0} #detail{position:static;max-height:none} }

thought stream

loading
-
root activity
Select an event to inspect its envelope and lineage.
+
root activity
Select an observation to see what processed it and what, if anything, happened.