diff --git a/README.md b/README.md index 8dc2951..a3b6122 100644 --- a/README.md +++ b/README.md @@ -69,6 +69,8 @@ Runtime data is stored under `.thoughtstream/`. Set `THOUGHTSTREAM_ROOT` to use | `pnpm thought review-queue` | Inspect the projected Review queue and active append-only decisions. | | `pnpm thought judgment --kind ` | Append an explicit quality judgment. External use requires `--external-export-eligible`; sensitive/private material also requires `--authorize-sensitive-external-export`. | | `pnpm thought training-export --output ` | Export only externally eligible, entirely public-source judgments into privacy-minimized JSONL and a content-addressed manifest. Sensitive/private export additionally requires both explicit private-export flags and a non-Git destination. | +| `pnpm thought artifact-request --workspace-root --workspace-lease-id --file --request-id --artifact-id ...` | Trusted host-side seam that records a private request and promotes one exact lease file into durable content-addressed storage. Never publishes externally; no exe.dev transport is implied. | +| `pnpm thought artifacts` | List the artifact catalog. | | `pnpm serve` | Start the local inspector on port 4317. | | `pnpm configure:inspector-oauth -- --origin --did --handle ` | Generate owner-only OAuth client/store configuration while retaining Basic fallback. | | `pnpm configure:inspector-review` | Generate one owner-only proxy-to-inspector Review capability. Generation does not restart or activate either service. | diff --git a/deploy/systemd/thoughtstream-exe-artifact-transport.service b/deploy/systemd/thoughtstream-exe-artifact-transport.service new file mode 100644 index 0000000..7d86bfb --- /dev/null +++ b/deploy/systemd/thoughtstream-exe-artifact-transport.service @@ -0,0 +1,25 @@ +[Unit] +Description=ThoughtStream private exe.dev artifact transport +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +WorkingDirectory=/home/cameron/code/thought-stream/.letta/worktrees/thoughtstream-artifacts +Environment=PATH=/home/cameron/.nvm/versions/node/v22.19.0/bin:/usr/local/bin:/usr/bin:/bin +Environment=THOUGHTSTREAM_ROOT=/home/cameron/.local/share/thoughtstream/live +ExecStart=/home/cameron/.nvm/versions/node/v22.19.0/bin/node --import tsx src/cli.ts exe-artifact-transport --host cameron.exe.xyz --workspace-root /srv/thoughtstream/workspaces/stream-v1 --workspace-id stream-v1 --lease-id 81aa06aa-fa70-4003-a41f-70147686dc61 --staging-root /home/cameron/.local/state/thoughtstream-exe-transport --poll-seconds 15 +Restart=on-failure +RestartSec=15s +UMask=0077 +NoNewPrivileges=true +PrivateTmp=true +ProtectSystem=strict +ProtectHome=read-only +ReadWritePaths=/home/cameron/.local/share/thoughtstream/live /home/cameron/.local/state/thoughtstream-exe-transport +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX +RestrictSUIDSGID=true +LockPersonality=true + +[Install] +WantedBy=default.target diff --git a/spec/README.md b/spec/README.md index b488bca..41d2a12 100644 --- a/spec/README.md +++ b/spec/README.md @@ -17,6 +17,7 @@ The core local milestone is implemented and exercised in `test/agent-runtime.tes - [`repairs.md`](repairs.md): deterministic repair eligibility, sandboxed correction proposals, judgment authority, effective-output rebuilding, and training boundaries. - [`review.md`](review.md): complete review prompts, blinded candidate pairs, judgeability, append-only human decisions, OAuth-only browser writes, and training-data custody. - [`incidents.md`](incidents.md): content-dark operational incident projection, private ledger, and independent Telegram alert policy. +- [`artifacts.md`](artifacts.md): immutable self-rooted content-addressed thought artifacts, artifact-catalog projection, and private inspector rendering. - [`tinker.md`](tinker.md): Tinker model and adapter boundary. - [`security.md`](security.md): privacy, credentials, authority, and prompt-injection boundaries. - [`recovery.md`](recovery.md): producer cursors, consumer progress, retries, replay, and terminal evidence. diff --git a/spec/artifacts.md b/spec/artifacts.md new file mode 100644 index 0000000..99d94ef --- /dev/null +++ b/spec/artifacts.md @@ -0,0 +1,95 @@ +# Private durable artifacts + +## Purpose and boundary + +An agent workspace (including a future exe.dev workspace) is a persistent working filesystem, not durable ThoughtStream storage. Promotion is a separate trusted capability: one explicitly named regular file under one explicitly named workspace lease root is copied into ThoughtStream's private content-addressed blob root and represented by append-only evidence. + +This release implements the host-side capability seam. It does **not** implement an exe.dev transport or a model-callable tool. A future bridge may stage exactly one remote file into an authorized local lease root and invoke the same materializer without changing this contract. Importing code from an artifact into a host worktree remains a separate capability. + +## Events + +### `stream.thought.artifact.requested@1` + +A request is append-only evidence, not publication authority. It identifies: + +- `requestId`: stable bounded request identity; +- `workspaceLeaseId`: explicit lease identity; +- `workspaceRootLabel`: non-path label for the explicit root supplied to the trusted process; +- `relativeFilePath`: one normalized relative path, never absolute and never containing `.` or `..` components; +- requested artifact metadata: stable `artifactId`, immutable positive `artifactVersion`, `kind`, `title`, `summary`, declared `mediaType`, provenance labels, visibility, optional supersession; +- optional `requestingAgentId`, `requestingRunId`, and `sourceEventId`; +- `publicationEligible: false` and `publicationAuthority: false`; +- `status`: `pending`, `failed`, or `denied`, with a bounded content-dark reason code for inert terminal evidence. + +The accepted request event never contains the workspace's absolute root. The trusted API receives that root out of band. Structurally invalid requests fail before durable acceptance. A valid but non-materializable request may be recorded as failed/denied evidence; the CLI uses a pending request followed by exact materialization and reports failure without broadening the selected path. + +### `stream.thought.artifact@1` + +The artifact event is immutable, private/sensitive, self-rooted, and metadata/reference-only. It contains: + +- `artifactId`, `artifactVersion`, `kind`, `title`, `summary`; +- declared and verified `mediaType`; +- `blob`: `{ algorithm: "sha256", sha256, relativePath, byteCount }`; +- provenance labels and bounded relations; +- a required strong `materialized-from-request` relation to the request event; +- `visibility`, `publicationEligible: false`, and optional append-only supersession. + +It contains no inline body, base64 bytes, workspace path, absolute path, or publication capability. Blob paths are canonical POSIX paths of the exact form `sha256//<64-hex-digest>`. + +## Formats and limits + +Supported pairs are strict: + +| Extensions | Media type | Validation | +| --- | --- | --- | +| `.md`, `.markdown` | `text/markdown` | valid UTF-8; no NUL | +| `.txt` | `text/plain` | valid UTF-8; no NUL | +| `.json` | `application/json` | valid UTF-8 JSON | +| `.yaml`, `.yml` | `application/yaml` or `text/yaml` | valid UTF-8 YAML | +| `.png` | `image/png` | PNG signature | +| `.jpg`, `.jpeg` | `image/jpeg` | JPEG SOI and EOI markers | + +Text/structured artifacts are capped at 262,144 bytes. PNG/JPEG artifacts are capped at 8,388,608 bytes. Empty files are rejected. Reads are bounded and fail as soon as the exact cap is exceeded. + +## Workspace containment + +The broker is supplied one explicit local lease root and one relative file path. It canonicalizes the root, rejects a symlink root, verifies every selected parent component with `lstat`, rejects file and parent symlinks, and opens only the resulting regular file. The canonical file must remain strictly beneath the canonical root. Missing, special, changed-size, or path-swapped files fail closed. No scan, wildcard, fallback root, or path broadening occurs. + +Absolute workspace roots are process-local authority and are never copied into request/artifact payloads, catalog data, errors intended for durable evidence, or provenance labels. + +## Blob storage + +Artifact bytes share `JazzThoughtStore.getArtifactRoot()` with existing downloaded image blobs. The canonical destination is `sha256//`. Directories are mode `0700`; blobs are mode `0600`. Creation uses a private temporary file and an atomic hard-link no-overwrite publication step. An already existing blob is accepted only after its regular-file type, mode-independent bytes, exact size, and SHA-256 are verified. Blob-root and parent symlinks are rejected. + +Every read resolves only the canonical relative path under the canonical blob root, rejects symlinks including parents, performs a bounded read, and re-verifies size, hash, extension-independent media magic, and structured/text validity against event metadata. Tampering fails closed. + +## Identity and supersession + +The artifact idempotency key is `artifact::v`. Same identity and same complete event payload/bytes is a no-op. Same `(artifactId, artifactVersion)` with different bytes or metadata is a hard conflict. Blob deduplication is independent: identical bytes are stored once even when referenced by different artifacts. + +A later event may name `supersedesArtifactEventId`. The old event remains; the metadata-only catalog marks it superseded. + +## Trusted API and CLI + +The trusted API has two explicit operations: append a validated request, then materialize that request using its matching explicit lease root. The CLI seam is: + +```text +thought artifact-request --workspace-root --workspace-lease-id \ + --file --request-id --artifact-id --artifact-version \ + --kind --title --summary <summary> --media-type <type> \ + [--visibility private|sensitive] [--requesting-agent <id>] [--requesting-run <id>] \ + [--source-event <event-id>] [--provenance-source <label>] [--provenance-label <label>] \ + [--supersedes <event-id>] +``` + +The CLI appends request evidence and invokes the exact trusted materializer. It prints identifiers, hashes, byte counts, and insertion status only; never artifact bytes or absolute paths. + +## Catalog and private inspector + +`GET /api/artifacts` is metadata-only. It contains no body or encoded bytes. `GET /api/artifacts/:eventId` resolves and verifies the blob on demand. Text detail returns bounded UTF-8 text. Image detail returns metadata and a private same-origin `contentPath`; raw bytes are available only from `GET /api/artifacts/:eventId/content`, with a verified image content type and `nosniff`. Catalog/list/log output never embeds image bytes. + +The inspector remains loopback-only and read-only. The authenticated private proxy may forward its existing authenticated `GET`/`HEAD` requests. No generic browser mutation route or public artifact route is introduced. + +## Privacy and leakage rules + +Both event families have minimum privacy `private`; `sensitive` visibility maps to sensitive event privacy. `publicationEligible` is always false. Provenance and workspace labels reject absolute filesystem paths. Artifact bytes are excluded from events, catalog projections, ordinary logs, notifications, and training/publication paths. Tests scan serialized events/catalogs and command output for source roots, credentials, and private bodies. diff --git a/spec/events.md b/spec/events.md index 0ec6f92..4fa92fc 100644 --- a/spec/events.md +++ b/spec/events.md @@ -87,6 +87,13 @@ Examples: An operational incident is a deterministic, content-dark projection over connector, scheduler, agent-run, and action failure/recovery evidence. It is always sensitive and contains only versioned classifications, bounded identifiers, retry/progress disposition, and strong references. See `incidents.md`. +### Thought artifacts + +- `stream.thought.artifact.requested@1` +- `stream.thought.artifact@1` + +The first event is append-only private evidence requesting promotion of one relative file from one explicit workspace lease. It grants no publication authority and contains no absolute root. The second is immutable metadata referencing verified bytes in the private content-addressed blob root; it contains hash, byte count, canonical relative blob path, media type, provenance, request lineage, visibility, publication eligibility (always `false`), and optional supersession, but no inline bytes or source path. See `artifacts.md`. + ### Consumer execution lifecycle - `stream.thought.consumer.execution.received` diff --git a/spec/security.md b/spec/security.md index 1ec7c85..25d8e90 100644 --- a/spec/security.md +++ b/spec/security.md @@ -85,6 +85,10 @@ The July 26, 2026 production audit has one unresolved high-severity finding: `sh - Apply file size and extension limits before reading. - Never watch `/home/cameron` or the entire Coil by default. +## Thought artifact containment + +The trusted artifact-request broker reads exactly one relative file beneath one explicitly supplied workspace lease root. It rejects path escape, root/file/parent symlinks, special files, extension/media/magic mismatch, malformed structured text, and oversized reads. It stores verified bytes once under the existing private content-addressed root with private modes and appends metadata-only request/artifact evidence; no workspace root or artifact bytes enter events, catalogs, logs, notifications, training exports, or public projections. See [`artifacts.md`](artifacts.md). + ## Audit Configuration changes, agent activation, connector activation, model tier changes, adapter catalog installation, and any future action capability changes require an operator-visible deployment receipt. An adapter manifest cannot grant a provider endpoint, credential value, host path, or action capability. It may name one non-secret environment reference and selects only an already trusted provider profile. The public base model and private runtime checkpoint are checked independently against the trusted provider allowlist. Startup compilation is the only path that creates the process-local checkpoint binding; compiled identities are deeply frozen, and clone/rehydration/forgery cannot recover that binding. Adapter activation and retirement require stop/install/restart plus PID/start-time, loaded-digest, and canary receipts. There is no runtime lifecycle mutation API. diff --git a/spec/ui.md b/spec/ui.md index bc76547..f941289 100644 --- a/spec/ui.md +++ b/spec/ui.md @@ -25,8 +25,12 @@ Filters: source, event family, agent, run status, privacy, time range, and text. The inspector exposes a read-only adapter inventory with public-safe release metadata, canonical lifecycle status/generation, a separately rendered active deployment binding when one exists, selecting consumers, bound runs, and output event ids. It must distinguish the execution adapter from the learned model adapter, and release status from deployment binding, and must never render checkpoint paths or resolved environment values. +The inspector exposes a read-only metadata-only artifact catalog and per-artifact detail view. The catalog lists artifact id, version, kind, title, summary, media type, byte count, SHA-256 prefix, visibility, provenance label, and supersession status, never bytes. Detail resolves and re-verifies the private blob on demand: text is escaped and images use a separate private content route. Artifact bytes are never exposed through public routes. See [`artifacts.md`](artifacts.md). + The source-health view shows each connector's durable cursor, last success/failure, current error, poll receipt counts, recovery count, and whether a poll has started without terminal evidence. Health is derived from durable cursor and connector events rather than process liveness. +Inspector bootstrap requests that scan shared Jazz state run serially rather than contending in the browser. Root activity renders as soon as its bounded summary is available; slower secondary tabs continue loading independently. Every tab has an explicit loading, empty, and failed state, and the client verifies HTTP status and JSON content type before parsing. A reverse-proxy HTML error page must never appear as a raw `response.json()` exception or leave another pane permanently labeled `Loading`. + ## Detail view - Consumer lifecycle and derived-output details lead with a plain-language **what processed this** block: the declaration display name, actual trigger source, whether it was a rule or model, whether LLM inference occurred, the exact produced summary, any recommendation or proposal that actually exists, derived-record count, and whether external actions were enabled. Technical agent ids and context strategies belong in execution details. Runtime metadata must not be appended to model-authored prose. A reader must not have to traverse raw lifecycle payloads to discover either the semantic result or its provenance. diff --git a/src/agent-proposals/contracts.ts b/src/agent-proposals/contracts.ts new file mode 100644 index 0000000..b97fda2 --- /dev/null +++ b/src/agent-proposals/contracts.ts @@ -0,0 +1,155 @@ +import { z } from "zod"; +import { createOutputContractRegistry } from "../agents/output-contracts.js"; +import { outputContractIdentitySchema, proposalMemoryTargetSchema } from "../agents/proposals.js"; +import { sha256, type JsonObject, type JsonValue } from "../core/json.js"; + +export const MEMORY_PROPOSAL_EVENT_TYPE = "stream.thought.agent.memory-change.proposed"; +export const CORRECTION_PROPOSAL_EVENT_TYPE = "stream.thought.agent.correction.proposed"; +export const PROPOSAL_DECISION_EVENT_TYPE = "stream.thought.agent.proposal.decision"; +export const MEMORY_MATERIALIZED_EVENT_TYPE = "stream.thought.agent.memory-change.materialized"; +export const MEMORY_MATERIALIZATION_FAILED_EVENT_TYPE = "stream.thought.agent.memory-change.materialization.failed"; +export const AGENT_PROPOSAL_SCHEMA_VERSION = 1; + +const idSchema = z.string().min(1).max(500); +const sha256Schema = z.string().regex(/^[a-f0-9]{64}$/); +const jsonValueSchema: z.ZodType<JsonValue> = z.lazy(() => z.union([ + z.string(), z.number(), z.boolean(), z.null(), z.array(jsonValueSchema), z.record(z.string(), jsonValueSchema), +])); +const objectSchema = z.record(z.string(), jsonValueSchema) as z.ZodType<JsonObject>; +const evidenceIdsSchema = z.array(idSchema).max(16).superRefine((ids, context) => { + if (new Set(ids).size !== ids.length) context.addIssue({ code: "custom", message: "Proposal evidence ids must be unique" }); +}); + +const proposerSchema = z.object({ + runId: idSchema, + outputEventId: idSchema, + triggerEventId: idSchema, + agentId: idSchema, + agentVersion: z.number().int().positive(), + declarationFingerprint: sha256Schema, + provider: z.string().min(1).max(200), + model: z.string().min(1).max(500), + contextSnapshotId: idSchema, +}).strict(); + +export const memoryProposalPayloadSchema = z.object({ + proposalState: z.literal("agent-proposed"), + proposer: proposerSchema, + target: proposalMemoryTargetSchema, + operation: z.enum(["append", "replace-document"]), + proposedText: z.string().min(1).max(32_768), + proposedTextChars: z.number().int().positive().max(32_768), + proposedTextSha256: sha256Schema, + reason: z.string().min(1).max(1_000), + evidenceEventIds: evidenceIdsSchema, + publicationEligible: z.literal(false), +}).strict().superRefine((value, context) => { + if (value.proposedText.length !== value.proposedTextChars) { + context.addIssue({ code: "custom", path: ["proposedTextChars"], message: "Proposal text length does not match" }); + } + if (sha256(value.proposedText) !== value.proposedTextSha256) { + context.addIssue({ code: "custom", path: ["proposedTextSha256"], message: "Proposal text hash does not match" }); + } +}); + +const correctionTargetSchema = z.object({ + runId: idSchema, + outputEventId: idSchema, + deliveryReceiptEventId: idSchema, + sourceRootEventId: idSchema, + outputContract: outputContractIdentitySchema, +}).strict(); + +export const correctionProposalPayloadSchema = z.object({ + proposalState: z.literal("agent-proposed"), + proposer: proposerSchema, + target: correctionTargetSchema, + replacementOutput: objectSchema, + replacementText: z.string().min(1).max(4_096), + replacementTextChars: z.number().int().positive().max(4_096), + replacementTextSha256: sha256Schema, + reason: z.string().min(1).max(1_000), + evidenceEventIds: evidenceIdsSchema, + qualityEligible: z.literal(false), + externalExportEligible: z.literal(false), + publicationEligible: z.literal(false), +}).strict().superRefine((value, context) => { + if (value.replacementText.length !== value.replacementTextChars) { + context.addIssue({ code: "custom", path: ["replacementTextChars"], message: "Correction text length does not match" }); + } + if (sha256(value.replacementText) !== value.replacementTextSha256) { + context.addIssue({ code: "custom", path: ["replacementTextSha256"], message: "Correction text hash does not match" }); + } + try { + createOutputContractRegistry().canonicalize(value.target.outputContract, value.replacementOutput); + } catch { + context.addIssue({ code: "custom", path: ["replacementOutput"], message: "Correction replacement does not satisfy the frozen output contract" }); + } +}); + +export const proposalDecisionPayloadSchema = z.object({ + proposalEventId: idSchema, + proposalType: z.enum(["memory-change", "self-correction"]), + disposition: z.enum(["accept", "edit", "reject"]), + submissionId: idSchema, + authority: z.literal("human"), + replacementText: z.string().min(1).max(32_768).optional(), + replacementTextChars: z.number().int().positive().max(32_768).optional(), + replacementTextSha256: sha256Schema.optional(), +}).strict().superRefine((value, context) => { + const hasReplacement = value.replacementText !== undefined; + if ((value.disposition === "edit") !== hasReplacement) { + context.addIssue({ code: "custom", path: ["replacementText"], message: "Only edit decisions require replacement text" }); + } + if (hasReplacement && (value.replacementText!.length !== value.replacementTextChars + || sha256(value.replacementText!) !== value.replacementTextSha256)) { + context.addIssue({ code: "custom", path: ["replacementTextSha256"], message: "Decision replacement metadata does not match" }); + } + if (value.proposalType === "self-correction" && hasReplacement && value.replacementText!.length > 4_096) { + context.addIssue({ code: "custom", path: ["replacementText"], message: "Correction edit exceeds its output bound" }); + } +}); + +export const memoryMaterializedPayloadSchema = z.object({ + proposalEventId: idSchema, + decisionEventId: idSchema, + operation: z.enum(["append", "replace-document"]), + base: proposalMemoryTargetSchema, + result: z.object({ + documentId: idSchema, + versionId: idSchema, + sha256: sha256Schema, + sizeBytes: z.number().int().positive().max(2_000_000), + filesystemEventId: idSchema, + }).strict(), + materializedBy: z.literal("trusted-local-memory-materializer@1"), +}).strict(); + +export const MEMORY_MATERIALIZATION_FAILURE_CODES = [ + "invalid-proposal", + "invalid-decision", + "decision-rejected", + "stale-base", + "base-evidence-invalid", + "root-invalid", + "target-invalid", + "symlink-refused", + "frontmatter-invalid", + "document-identity-changed", + "content-too-large", + "write-failed", + "filesystem-scan-failed", + "receipt-mismatch", + "already-decided", +] as const; +export type MemoryMaterializationFailureCode = typeof MEMORY_MATERIALIZATION_FAILURE_CODES[number]; + +export const memoryMaterializationFailedPayloadSchema = z.object({ + proposalEventId: idSchema, + decisionEventId: idSchema, + baseVersionId: idSchema, + baseSha256: sha256Schema, + reasonCode: z.enum(MEMORY_MATERIALIZATION_FAILURE_CODES), + contentRedacted: z.literal(true), + materializedBy: z.literal("trusted-local-memory-materializer@1"), +}).strict(); diff --git a/src/agent-proposals/memory-materializer.ts b/src/agent-proposals/memory-materializer.ts new file mode 100644 index 0000000..ebd032c --- /dev/null +++ b/src/agent-proposals/memory-materializer.ts @@ -0,0 +1,353 @@ +import { randomUUID } from "node:crypto"; +import fs, { type FileHandle } from "node:fs/promises"; +import path from "node:path"; +import YAML from "yaml"; +import { FilesystemConnector } from "../connectors/filesystem.js"; +import { sha256, type JsonObject } from "../core/json.js"; +import { stableKey } from "../core/ids.js"; +import type { ThoughtEvent } from "../events/types.js"; +import type { JazzThoughtStore } from "../jazz/store.js"; +import { + MEMORY_MATERIALIZATION_FAILED_EVENT_TYPE, + MEMORY_MATERIALIZED_EVENT_TYPE, + MEMORY_PROPOSAL_EVENT_TYPE, + memoryProposalPayloadSchema, + proposalDecisionPayloadSchema, + type MemoryMaterializationFailureCode, +} from "./contracts.js"; +import { decisionsForProposal, requireProposal } from "./review.js"; + +export const STREAM_MEMORY_SOURCE = "filesystem:telegram-agent-context"; +export const STREAM_MEMORY_PATH = "memory.md"; +const MATERIALIZER = "trusted-local-memory-materializer@1"; +const DEFAULT_MAX_BYTES = 1_000_000; + +type FailureCode = MemoryMaterializationFailureCode; + +export interface MemoryMaterializerOptions { + contextRoot: string; + source?: typeof STREAM_MEMORY_SOURCE | undefined; + maxFileBytes?: number | undefined; + beforeRename?: (() => void | Promise<void>) | undefined; +} + +export interface MemoryMaterializationResult { + status: "materialized" | "failed"; + event: ThoughtEvent; +} + +class MaterializationFailure extends Error { + constructor(readonly code: FailureCode) { + super(code); + this.name = "MaterializationFailure"; + } +} + +export async function materializeMemoryDecision( + store: JazzThoughtStore, + decisionEventId: string, + options: MemoryMaterializerOptions, +): Promise<MemoryMaterializationResult> { + const decision = await store.getEvent(decisionEventId); + if (!decision || decision.type !== "stream.thought.agent.proposal.decision") { + throw new Error("Memory materialization requires a proposal decision event"); + } + const decisionPayload = proposalDecisionPayloadSchema.parse(decision.payload); + if (decisionPayload.proposalType !== "memory-change") throw new Error("Decision does not target a memory proposal"); + const proposal = await requireProposal(store, decisionPayload.proposalEventId); + if (proposal.type !== MEMORY_PROPOSAL_EVENT_TYPE) throw new Error("Memory decision targets the wrong proposal kind"); + const proposalPayload = memoryProposalPayloadSchema.parse(proposal.payload); + const decisions = await decisionsForProposal(store, proposal.id); + if (decisions.length !== 1 || decisions[0]!.id !== decision.id) { + const failed = await appendFailure(store, proposal, decision, proposalPayload.target.versionId, proposalPayload.target.sha256, "already-decided"); + return { status: "failed", event: failed }; + } + if (decision.parentEventId !== proposal.id || decision.rootEventId !== proposal.rootEventId) { + throw new Error("Memory decision lineage is inconsistent"); + } + const existing = await materializationEvents(store, decision.id); + const completed = existing.find((event) => event.type === MEMORY_MATERIALIZED_EVENT_TYPE); + if (completed) return { status: "materialized", event: completed }; + if (decisionPayload.disposition === "reject") { + const failed = await appendFailure(store, proposal, decision, proposalPayload.target.versionId, proposalPayload.target.sha256, "decision-rejected"); + return { status: "failed", event: failed }; + } + + const source = options.source ?? STREAM_MEMORY_SOURCE; + if (source !== STREAM_MEMORY_SOURCE + || proposalPayload.target.source !== STREAM_MEMORY_SOURCE + || proposalPayload.target.path !== STREAM_MEMORY_PATH) { + const failed = await appendFailure(store, proposal, decision, proposalPayload.target.versionId, proposalPayload.target.sha256, "invalid-proposal"); + return { status: "failed", event: failed }; + } + + try { + const newContent = await materializeContent(store, proposalPayload, decisionPayload, options); + const connector = new FilesystemConnector({ + id: STREAM_MEMORY_SOURCE, + root: options.contextRoot, + privacy: "sensitive", + storeContent: true, + maxFileBytes: options.maxFileBytes ?? DEFAULT_MAX_BYTES, + }); + let scan; + try { + scan = await connector.scan(store); + } catch { + throw new MaterializationFailure("filesystem-scan-failed"); + } + const expectedSha256 = sha256(newContent); + const current = (await store.listCurrentDocuments(STREAM_MEMORY_SOURCE)).find((item) => ( + !item.deleted && item.path === STREAM_MEMORY_PATH + )); + if (!current + || current.documentId !== proposalPayload.target.documentId + || current.sha256 !== expectedSha256 + || current.versionId !== stableKey("version", STREAM_MEMORY_SOURCE, current.documentId, expectedSha256)) { + throw new MaterializationFailure("receipt-mismatch"); + } + const version = await store.getDocumentVersion(current.versionId); + if (!version + || version.source !== STREAM_MEMORY_SOURCE + || version.documentId !== current.documentId + || version.path !== STREAM_MEMORY_PATH + || version.sha256 !== expectedSha256 + || version.content !== newContent + || version.sizeBytes !== Buffer.byteLength(newContent)) { + throw new MaterializationFailure("receipt-mismatch"); + } + const filesystemEvent = scan.events.find((event) => event.payload.versionId === current.versionId) + ?? (await store.listEvents({ source: STREAM_MEMORY_SOURCE })).find((event) => event.payload.versionId === current.versionId); + if (!filesystemEvent) throw new MaterializationFailure("receipt-mismatch"); + const materialized = (await store.appendEvent({ + type: MEMORY_MATERIALIZED_EVENT_TYPE, + schemaVersion: 1, + source: "materializer:agent-context", + sourceKind: "system", + externalId: decision.id, + idempotencyKey: stableKey("memory-proposal-materialized", decision.id, current.versionId), + occurredAt: new Date().toISOString(), + actor: "operator:local", + rootEventId: proposal.rootEventId, + parentEventId: decision.id, + correlationId: proposal.id, + privacy: "sensitive", + payload: { + proposalEventId: proposal.id, + decisionEventId: decision.id, + operation: proposalPayload.operation, + base: proposalPayload.target, + result: { + documentId: current.documentId, + versionId: current.versionId, + sha256: current.sha256, + sizeBytes: current.sizeBytes, + filesystemEventId: filesystemEvent.id, + }, + materializedBy: MATERIALIZER, + }, + createdByRuntime: "thoughtstream-memory-materializer-v1", + })).event; + return { status: "materialized", event: materialized }; + } catch (error) { + const code = error instanceof MaterializationFailure ? error.code : "write-failed"; + const failed = await appendFailure(store, proposal, decision, proposalPayload.target.versionId, proposalPayload.target.sha256, code); + return { status: "failed", event: failed }; + } +} + +async function materializeContent( + store: JazzThoughtStore, + proposal: ReturnType<typeof memoryProposalPayloadSchema.parse>, + decision: ReturnType<typeof proposalDecisionPayloadSchema.parse>, + options: MemoryMaterializerOptions, +): Promise<string> { + const current = (await store.listCurrentDocuments(STREAM_MEMORY_SOURCE)).find((item) => ( + !item.deleted && item.path === STREAM_MEMORY_PATH + )); + if (!current + || current.documentId !== proposal.target.documentId + || current.versionId !== proposal.target.versionId + || current.sha256 !== proposal.target.sha256 + || current.contentType !== "text/markdown") { + throw new MaterializationFailure("stale-base"); + } + const base = await store.getDocumentVersion(proposal.target.versionId); + if (!base + || base.source !== STREAM_MEMORY_SOURCE + || base.documentId !== proposal.target.documentId + || base.path !== STREAM_MEMORY_PATH + || base.contentType !== "text/markdown" + || base.sha256 !== proposal.target.sha256 + || sha256(base.content) !== base.sha256 + || Buffer.byteLength(base.content) !== base.sizeBytes) { + throw new MaterializationFailure("base-evidence-invalid"); + } + const root = path.resolve(options.contextRoot); + const rootStat = await fs.lstat(root).catch(() => undefined); + if (!rootStat || !rootStat.isDirectory() || rootStat.isSymbolicLink()) throw new MaterializationFailure("root-invalid"); + const realRoot = await fs.realpath(root).catch(() => undefined); + if (!realRoot || realRoot !== root) throw new MaterializationFailure("root-invalid"); + const target = path.join(root, STREAM_MEMORY_PATH); + if (path.dirname(target) !== root) throw new MaterializationFailure("target-invalid"); + const targetStat = await fs.lstat(target).catch(() => undefined); + if (!targetStat) throw new MaterializationFailure("target-invalid"); + if (targetStat.isSymbolicLink()) throw new MaterializationFailure("symlink-refused"); + if (!targetStat.isFile()) throw new MaterializationFailure("target-invalid"); + const realTarget = await fs.realpath(target).catch(() => undefined); + if (realTarget !== target) throw new MaterializationFailure("symlink-refused"); + const diskContent = normalize(await fs.readFile(target, "utf8")); + const baseId = frontmatterId(base.content); + if (!baseId) throw new MaterializationFailure("frontmatter-invalid"); + const approvedText = decision.disposition === "edit" ? decision.replacementText! : proposal.proposedText; + const newContent = proposal.operation === "append" + ? appendText(base.content, approvedText) + : normalize(approvedText); + const newId = frontmatterId(newContent); + if (!newId) throw new MaterializationFailure("frontmatter-invalid"); + if (newId !== baseId) throw new MaterializationFailure("document-identity-changed"); + if (Buffer.byteLength(newContent) > (options.maxFileBytes ?? DEFAULT_MAX_BYTES)) { + throw new MaterializationFailure("content-too-large"); + } + const diskSha256 = sha256(diskContent); + const newSha256 = sha256(newContent); + if (diskSha256 !== base.sha256 && diskSha256 !== newSha256) throw new MaterializationFailure("stale-base"); + if (diskSha256 === newSha256) { + if ((targetStat.mode & 0o777) !== 0o600) throw new MaterializationFailure("write-failed"); + return newContent; + } + await atomicReplaceMemory(root, target, base.sha256, newContent, { + rootDev: rootStat.dev, + rootIno: rootStat.ino, + targetDev: targetStat.dev, + targetIno: targetStat.ino, + }, options.beforeRename); + const written = await fs.lstat(target).catch(() => undefined); + if (!written || !written.isFile() || written.isSymbolicLink() || (written.mode & 0o777) !== 0o600) { + throw new MaterializationFailure("write-failed"); + } + if (sha256(normalize(await fs.readFile(target, "utf8"))) !== newSha256) { + throw new MaterializationFailure("write-failed"); + } + return newContent; +} + +async function atomicReplaceMemory( + root: string, + target: string, + expectedBaseSha256: string, + content: string, + expectedIdentity: { rootDev: number | bigint; rootIno: number | bigint; targetDev: number | bigint; targetIno: number | bigint }, + beforeRename: MemoryMaterializerOptions["beforeRename"], +): Promise<void> { + const lockPath = path.join(root, ".thoughtstream-memory-materializer.lock"); + let lock: FileHandle | undefined; + let temporary: string | undefined; + try { + try { + lock = await fs.open(lockPath, "wx", 0o600); + } catch { + throw new MaterializationFailure("write-failed"); + } + temporary = path.join(root, `.memory.md.thoughtstream.${process.pid}.${randomUUID()}.tmp`); + const temporaryHandle = await fs.open(temporary, "wx", 0o600); + try { + await temporaryHandle.writeFile(content, "utf8"); + await temporaryHandle.sync(); + await temporaryHandle.chmod(0o600); + } finally { + await temporaryHandle.close(); + } + await beforeRename?.(); + const rootStat = await fs.lstat(root); + const targetStat = await fs.lstat(target); + if (!rootStat.isDirectory() || rootStat.isSymbolicLink() || targetStat.isSymbolicLink()) { + throw new MaterializationFailure("symlink-refused"); + } + if (!targetStat.isFile()) throw new MaterializationFailure("target-invalid"); + if ( + rootStat.dev !== expectedIdentity.rootDev + || rootStat.ino !== expectedIdentity.rootIno + || targetStat.dev !== expectedIdentity.targetDev + || targetStat.ino !== expectedIdentity.targetIno + ) { + throw new MaterializationFailure("stale-base"); + } + if (sha256(normalize(await fs.readFile(target, "utf8"))) !== expectedBaseSha256) { + throw new MaterializationFailure("stale-base"); + } + await fs.rename(temporary, target); + temporary = undefined; + const rootHandle = await fs.open(root, "r"); + try { await rootHandle.sync(); } finally { await rootHandle.close(); } + } finally { + if (temporary) await fs.rm(temporary, { force: true }).catch(() => undefined); + if (lock) { + await lock.close().catch(() => undefined); + await fs.rm(lockPath, { force: true }).catch(() => undefined); + } + } +} + +async function appendFailure( + store: JazzThoughtStore, + proposal: ThoughtEvent, + decision: ThoughtEvent, + baseVersionId: string, + baseSha256: string, + reasonCode: FailureCode, +): Promise<ThoughtEvent> { + return (await store.appendEvent({ + type: MEMORY_MATERIALIZATION_FAILED_EVENT_TYPE, + schemaVersion: 1, + source: "materializer:agent-context", + sourceKind: "system", + externalId: decision.id, + idempotencyKey: stableKey("memory-proposal-materialization-failed", decision.id, reasonCode), + occurredAt: new Date().toISOString(), + actor: "operator:local", + rootEventId: proposal.rootEventId, + parentEventId: decision.id, + correlationId: proposal.id, + privacy: "sensitive", + payload: { + proposalEventId: proposal.id, + decisionEventId: decision.id, + baseVersionId, + baseSha256, + reasonCode, + contentRedacted: true, + materializedBy: MATERIALIZER, + }, + createdByRuntime: "thoughtstream-memory-materializer-v1", + })).event; +} + +async function materializationEvents(store: JazzThoughtStore, decisionEventId: string): Promise<ThoughtEvent[]> { + return (await store.listEvents({ + types: [MEMORY_MATERIALIZED_EVENT_TYPE, MEMORY_MATERIALIZATION_FAILED_EVENT_TYPE], + })).filter((event) => event.payload.decisionEventId === decisionEventId); +} + +function normalize(value: string): string { + return value.replace(/\r\n/g, "\n"); +} + +function appendText(base: string, addition: string): string { + return `${normalize(base).trimEnd()}\n\n${normalize(addition).trim()}\n`; +} + +function frontmatterId(content: string): string | undefined { + const normalized = normalize(content); + if (!normalized.startsWith("---\n")) return undefined; + const end = normalized.indexOf("\n---\n", 4); + if (end < 0) return undefined; + try { + const parsed = YAML.parse(normalized.slice(4, end)); + if (!parsed || typeof parsed !== "object" || Array.isArray(parsed) || !("id" in parsed)) return undefined; + const id = String((parsed as { id: unknown }).id).trim(); + return id || undefined; + } catch { + return undefined; + } +} diff --git a/src/agent-proposals/review.ts b/src/agent-proposals/review.ts new file mode 100644 index 0000000..0c20d27 --- /dev/null +++ b/src/agent-proposals/review.ts @@ -0,0 +1,191 @@ +import { randomUUID } from "node:crypto"; +import { + createOutputContractRegistry, + parseOutputContractIdentity, +} from "../agents/output-contracts.js"; +import { canonicalJson, sha256, type JsonObject } from "../core/json.js"; +import { stableKey } from "../core/ids.js"; +import type { ThoughtEvent } from "../events/types.js"; +import type { JazzThoughtStore } from "../jazz/store.js"; +import { recordJudgment } from "../training/judgments.js"; +import { + CORRECTION_PROPOSAL_EVENT_TYPE, + MEMORY_PROPOSAL_EVENT_TYPE, + PROPOSAL_DECISION_EVENT_TYPE, + correctionProposalPayloadSchema, + memoryProposalPayloadSchema, + proposalDecisionPayloadSchema, +} from "./contracts.js"; + +export type ProposalDecisionDisposition = "accept" | "edit" | "reject"; + +export interface RecordProposalDecisionInput { + proposalEventId: string; + disposition: ProposalDecisionDisposition; + replacementText?: string | undefined; + submissionId?: string | undefined; + actor?: string | undefined; +} + +export interface ProposalDecisionResult { + decision: ThoughtEvent; + proposal: ThoughtEvent; + projectedJudgment?: ThoughtEvent | undefined; +} + +export async function recordProposalDecision( + store: JazzThoughtStore, + input: RecordProposalDecisionInput, +): Promise<ProposalDecisionResult> { + const proposal = await requireProposal(store, input.proposalEventId); + const proposalType = proposal.type === MEMORY_PROPOSAL_EVENT_TYPE ? "memory-change" : "self-correction"; + const submissionId = input.submissionId ?? randomUUID(); + const payload = proposalDecisionPayloadSchema.parse({ + proposalEventId: proposal.id, + proposalType, + disposition: input.disposition, + submissionId, + authority: "human", + ...(input.replacementText !== undefined ? { + replacementText: input.replacementText, + replacementTextChars: input.replacementText.length, + replacementTextSha256: sha256(input.replacementText), + } : {}), + }); + const existing = await decisionsForProposal(store, proposal.id); + const sameSubmission = existing.find((decision) => decision.payload.submissionId === submissionId); + if (sameSubmission) { + if (canonicalJson(sameSubmission.payload) !== canonicalJson(payload as unknown as JsonObject)) { + throw new Error("Proposal decision submission id conflicts with existing content"); + } + const projectedJudgment = proposalType === "self-correction" + ? await projectCorrectionDecision(store, sameSubmission.id) + : undefined; + return { decision: sameSubmission, proposal, ...(projectedJudgment ? { projectedJudgment } : {}) }; + } + if (existing.length > 0) throw new Error("Proposal already has a human decision"); + const decision = (await store.appendEvent({ + type: PROPOSAL_DECISION_EVENT_TYPE, + schemaVersion: 1, + source: "proposal-review:local", + sourceKind: "system", + externalId: submissionId, + idempotencyKey: stableKey("agent-proposal-decision", proposal.id, submissionId), + occurredAt: new Date().toISOString(), + actor: input.actor ?? "operator:local", + rootEventId: proposal.rootEventId, + parentEventId: proposal.id, + correlationId: proposal.id, + privacy: "sensitive", + payload: payload as unknown as JsonObject, + createdByRuntime: "thoughtstream-agent-proposal-review-v1", + })).event; + const projectedJudgment = proposalType === "self-correction" + ? await projectCorrectionDecision(store, decision.id) + : undefined; + return { decision, proposal, ...(projectedJudgment ? { projectedJudgment } : {}) }; +} + +export async function projectAcceptedCorrectionDecisions(store: JazzThoughtStore): Promise<{ + examined: number; + projected: number; + rejected: number; +}> { + const decisions = await store.listEvents({ types: [PROPOSAL_DECISION_EVENT_TYPE] }); + let projected = 0; + let rejected = 0; + for (const decision of decisions) { + if (decision.payload.proposalType !== "self-correction") continue; + if (decision.payload.disposition === "reject") { + rejected += 1; + continue; + } + await projectCorrectionDecision(store, decision.id); + projected += 1; + } + return { examined: decisions.length, projected, rejected }; +} + +export async function projectCorrectionDecision( + store: JazzThoughtStore, + decisionEventId: string, +): Promise<ThoughtEvent | undefined> { + const decision = await store.getEvent(decisionEventId); + if (!decision || decision.type !== PROPOSAL_DECISION_EVENT_TYPE) throw new Error("Correction projection requires a proposal decision event"); + const decisionPayload = proposalDecisionPayloadSchema.parse(decision.payload); + if (decisionPayload.proposalType !== "self-correction") throw new Error("Proposal decision does not target a self-correction"); + const proposal = await requireProposal(store, decisionPayload.proposalEventId); + if (proposal.type !== CORRECTION_PROPOSAL_EVENT_TYPE) throw new Error("Correction decision targets the wrong proposal kind"); + const proposalPayload = correctionProposalPayloadSchema.parse(proposal.payload); + const decisions = await decisionsForProposal(store, proposal.id); + if (decisions.length !== 1 || decisions[0]!.id !== decision.id) throw new Error("Correction proposal has conflicting decisions"); + if (decisionPayload.disposition === "reject") return undefined; + if (decision.parentEventId !== proposal.id || decision.rootEventId !== proposal.rootEventId) { + throw new Error("Correction decision lineage is inconsistent"); + } + const targetRun = await store.getRun(proposalPayload.target.runId); + const targetOutput = await store.getEvent(proposalPayload.target.outputEventId); + const delivery = await store.getEvent(proposalPayload.target.deliveryReceiptEventId); + if ( + !targetRun + || targetRun.status !== "completed" + || targetRun.outputEventIds.length !== 1 + || targetRun.outputEventIds[0] !== proposalPayload.target.outputEventId + || !targetOutput + || targetOutput.type !== "stream.thought.derived.message.observation" + || targetOutput.source !== `agent:${targetRun.agentId}` + || targetOutput.sourceKind !== "agent" + || targetOutput.rootEventId !== proposalPayload.target.sourceRootEventId + || targetOutput.payload.runId !== targetRun.id + || !delivery + || delivery.type !== "stream.thought.action.telegram.send.delivered" + || delivery.sourceKind !== "system" + || delivery.parentEventId !== targetOutput.id + || delivery.rootEventId !== proposalPayload.target.sourceRootEventId + || !Array.isArray(delivery.payload.runIds) + || delivery.payload.runIds.length !== 1 + || delivery.payload.runIds[0] !== targetRun.id + ) { + throw new Error("Correction decision target evidence is incomplete or inconsistent"); + } + const contract = parseOutputContractIdentity(proposalPayload.target.outputContract); + const registry = createOutputContractRegistry(); + const proposedOutput = registry.canonicalize(contract, proposalPayload.replacementOutput); + const replacementOutput = decisionPayload.disposition === "accept" + ? proposedOutput + : registry.canonicalize(contract, { ...proposedOutput, summary: decisionPayload.replacementText! }); + return recordJudgment(store, { + runId: targetRun.id, + kind: "correct", + criterion: "agent-self-correction", + criterionVersion: 1, + qualityEligible: true, + externalExportEligible: false, + replacementOutput, + notes: decisionPayload.disposition === "accept" + ? "Human accepted agent-proposed self-correction" + : "Human edited and accepted agent-proposed self-correction", + actor: decision.actor, + source: "judgment:agent-self-correction", + feedbackSourceEventId: decision.id, + deliveryReceiptEventId: delivery.id, + }); +} + +export async function decisionsForProposal(store: JazzThoughtStore, proposalEventId: string): Promise<ThoughtEvent[]> { + return (await store.listEvents({ types: [PROPOSAL_DECISION_EVENT_TYPE] })) + .filter((event) => event.payload.proposalEventId === proposalEventId) + .sort((left, right) => left.observedAt.localeCompare(right.observedAt) || left.id.localeCompare(right.id)); +} + +export async function requireProposal(store: JazzThoughtStore, proposalEventId: string): Promise<ThoughtEvent> { + const proposal = await store.getEvent(proposalEventId); + if (!proposal) throw new Error("Agent proposal event was not found"); + if (proposal.type === MEMORY_PROPOSAL_EVENT_TYPE) memoryProposalPayloadSchema.parse(proposal.payload); + else if (proposal.type === CORRECTION_PROPOSAL_EVENT_TYPE) correctionProposalPayloadSchema.parse(proposal.payload); + else throw new Error("Event is not an agent proposal"); + if (proposal.privacy !== "sensitive" || proposal.sourceKind !== "agent") { + throw new Error("Agent proposal privacy or source authority is invalid"); + } + return proposal; +} diff --git a/src/agents/proposals.ts b/src/agents/proposals.ts new file mode 100644 index 0000000..9498839 --- /dev/null +++ b/src/agents/proposals.ts @@ -0,0 +1,134 @@ +import { z } from "zod"; +import type { JsonObject } from "../core/json.js"; + +export const PROPOSAL_DECLARATION_NAMES = ["memory-change", "self-correction"] as const; +export type ProposalDeclarationName = typeof PROPOSAL_DECLARATION_NAMES[number]; + +export const PROPOSAL_TOOL_NAMES = ["request_memory_change", "submit_correction"] as const; +export type ProposalToolName = typeof PROPOSAL_TOOL_NAMES[number]; + +const idSchema = z.string().min(1).max(500); +const sha256Schema = z.string().regex(/^[a-f0-9]{64}$/); +const evidenceIdsSchema = z.array(idSchema).max(16).superRefine((ids, context) => { + const seen = new Set<string>(); + for (let index = 0; index < ids.length; index += 1) { + if (seen.has(ids[index]!)) context.addIssue({ code: "custom", path: [index], message: "Evidence ids must be unique" }); + seen.add(ids[index]!); + } +}); + +export const memoryChangeArgumentsSchema = z.object({ + operation: z.enum(["append", "replace-document"]), + proposed_text: z.string().min(1).max(32_768), + reason: z.string().min(1).max(1_000), + evidence_event_ids: evidenceIdsSchema, +}).strict(); + +export const correctionArgumentsSchema = z.object({ + target_output: idSchema, + replacement: z.string().min(1).max(4_096), + reason: z.string().min(1).max(1_000), + evidence_event_ids: evidenceIdsSchema, +}).strict(); + +export const proposalMemoryTargetSchema = z.object({ + source: z.literal("filesystem:telegram-agent-context"), + documentId: idSchema, + path: z.literal("memory.md"), + versionId: idSchema, + sha256: sha256Schema, + contentType: z.literal("text/markdown"), +}).strict(); + +export const outputContractIdentitySchema = z.object({ + id: idSchema, + version: z.number().int().positive(), + sha256: sha256Schema, +}).strict(); + +export const proposalCorrectionTargetSchema = z.object({ + runId: idSchema, + outputEventId: idSchema, + deliveryReceiptEventId: idSchema, + sourceRootEventId: idSchema, + outputContract: outputContractIdentitySchema, +}).strict(); + +export const proposalCapabilitiesSchema = z.object({ + enabled: z.array(z.enum(PROPOSAL_DECLARATION_NAMES)).max(2).superRefine((values, context) => { + if (new Set(values).size !== values.length) context.addIssue({ code: "custom", message: "Proposal capability names must be unique" }); + }), + evidenceEventIds: z.array(idSchema).max(300).superRefine((values, context) => { + if (new Set(values).size !== values.length) context.addIssue({ code: "custom", message: "Proposal evidence ids must be unique" }); + }), + memoryTarget: proposalMemoryTargetSchema.optional(), + correctionTargets: z.array(proposalCorrectionTargetSchema).max(100).superRefine((targets, context) => { + const ids = targets.map((target) => target.outputEventId); + if (new Set(ids).size !== ids.length) context.addIssue({ code: "custom", message: "Correction target output ids must be unique" }); + }), +}).strict().superRefine((value, context) => { + if (value.enabled.includes("memory-change") !== Boolean(value.memoryTarget)) { + context.addIssue({ code: "custom", path: ["memoryTarget"], message: "Memory proposal capability requires exactly one memory target" }); + } + if (!value.enabled.includes("self-correction") && value.correctionTargets.length > 0) { + context.addIssue({ code: "custom", path: ["correctionTargets"], message: "Correction targets require self-correction capability" }); + } +}); + +export const capturedMemoryProposalSchema = z.object({ + toolCallId: idSchema, + kind: z.literal("memory-change"), + arguments: memoryChangeArgumentsSchema, +}).strict(); + +export const capturedCorrectionProposalSchema = z.object({ + toolCallId: idSchema, + kind: z.literal("self-correction"), + arguments: correctionArgumentsSchema, +}).strict(); + +export const capturedProposalSchema = z.discriminatedUnion("kind", [ + capturedMemoryProposalSchema, + capturedCorrectionProposalSchema, +]); + +export const capturedProposalsSchema = z.array(capturedProposalSchema).max(2).superRefine((proposals, context) => { + const calls = proposals.map((proposal) => proposal.toolCallId); + if (new Set(calls).size !== calls.length) context.addIssue({ code: "custom", message: "Proposal tool-call ids must be unique" }); + const kinds = proposals.map((proposal) => proposal.kind); + if (new Set(kinds).size !== kinds.length) context.addIssue({ code: "custom", message: "Only one proposal of each kind is allowed" }); +}); + +export type MemoryChangeArguments = z.infer<typeof memoryChangeArgumentsSchema>; +export type CorrectionArguments = z.infer<typeof correctionArgumentsSchema>; +export type ProposalMemoryTarget = z.infer<typeof proposalMemoryTargetSchema>; +export type ProposalCorrectionTarget = z.infer<typeof proposalCorrectionTargetSchema>; +export type ProposalCapabilities = z.infer<typeof proposalCapabilitiesSchema>; +export type CapturedProposal = z.infer<typeof capturedProposalSchema>; + +export function validateCapturedProposalAgainstCapabilities( + proposal: CapturedProposal, + capabilities: ProposalCapabilities, +): CapturedProposal { + const admittedEvidence = new Set(capabilities.evidenceEventIds); + for (const id of proposal.arguments.evidence_event_ids) { + if (!admittedEvidence.has(id)) throw new Error("Proposal evidence id is outside the context snapshot"); + } + if (proposal.kind === "memory-change") { + if (!capabilities.enabled.includes("memory-change") || !capabilities.memoryTarget) { + throw new Error("Memory proposal capability is unavailable"); + } + return proposal; + } + if (!capabilities.enabled.includes("self-correction")) { + throw new Error("Correction proposal capability is unavailable"); + } + if (!capabilities.correctionTargets.some((target) => target.outputEventId === proposal.arguments.target_output)) { + throw new Error("Correction target is outside the context snapshot"); + } + return proposal; +} + +export function proposalCapabilitiesJson(value: ProposalCapabilities): JsonObject { + return proposalCapabilitiesSchema.parse(value) as unknown as JsonObject; +} diff --git a/src/agents/tools.ts b/src/agents/tools.ts index 60ee204..b29dc95 100644 --- a/src/agents/tools.ts +++ b/src/agents/tools.ts @@ -1,13 +1,12 @@ -import { createHash, randomUUID } from "node:crypto"; +import { createHash } from "node:crypto"; import { BlockList, isIP } from "node:net"; import { resolve4, resolve6 } from "node:dns/promises"; -import { mkdir, rename, rm, writeFile } from "node:fs/promises"; -import path from "node:path"; import type { AgentTool } from "@earendil-works/pi-agent-core"; import { Type } from "typebox"; import type { JsonObject } from "../core/json.js"; import type { ThoughtEvent } from "../events/types.js"; import type { EnrichmentOutcome, RunnerTrace } from "./types.js"; +import { storeBlob } from "../artifacts/blob.js"; export const AGENT_TOOL_NAMES = ["atproto.fetch-markdown", "web.download-image"] as const; export type AgentToolName = (typeof AGENT_TOOL_NAMES)[number]; @@ -463,17 +462,9 @@ async function readBoundedBody(response: Response, maxBytes: number, label: stri } async function writeArtifact(root: string, sha256: string, bytes: Buffer): Promise<string> { - const relative = path.join("sha256", sha256.slice(0, 2), sha256); - const destination = path.join(root, relative); - await mkdir(path.dirname(destination), { recursive: true }); - const temporary = `${destination}.${randomUUID()}.tmp`; - await writeFile(temporary, bytes, { flag: "wx" }); - try { - await rename(temporary, destination); - } finally { - await rm(temporary, { force: true }); - } - return relative; + const blob = await storeBlob(root, bytes); + if (blob.sha256 !== sha256) throw new Error("Downloaded image digest changed before blob storage"); + return blob.relativePath; } function extractImageUrls(value: unknown): string[] { diff --git a/src/artifacts/append.ts b/src/artifacts/append.ts new file mode 100644 index 0000000..da61d9f --- /dev/null +++ b/src/artifacts/append.ts @@ -0,0 +1,115 @@ +import { createHash } from "node:crypto"; +import fs from "node:fs/promises"; +import path from "node:path"; +import type { JsonObject } from "../core/json.js"; +import type { EventCandidate, PrivacyClass, ThoughtEvent } from "../events/types.js"; +import type { JazzThoughtStore } from "../jazz/store.js"; +import { readWorkspaceArtifact, storeBlob } from "./blob.js"; +import { rebuildArtifactCatalog } from "./catalog.js"; +import type { ArtifactKind, ArtifactMediaType, ArtifactPayload, ArtifactProvenance, ArtifactRelation, ArtifactRequestPayload, ArtifactRequestStatus, ArtifactVisibility } from "./types.js"; +import { ARTIFACT_EVENT_TYPE, ARTIFACT_REQUEST_EVENT_TYPE, ARTIFACT_REQUEST_SCHEMA_VERSION, ARTIFACT_SCHEMA_VERSION } from "./types.js"; + +export interface ArtifactRequestInput { + requestId: string; + workspaceLeaseId: string; + workspaceRootLabel: string; + relativeFilePath: string; + artifactId: string; + artifactVersion: number; + kind: ArtifactKind; + title: string; + summary: string; + mediaType: ArtifactMediaType; + provenance: ArtifactProvenance; + visibility: ArtifactVisibility; + requestingAgentId?: string; + requestingRunId?: string; + sourceEventId?: string; + supersedesArtifactEventId?: string; + status?: ArtifactRequestStatus; + reasonCode?: string; +} +export interface AppendArtifactRequestOptions extends ArtifactRequestInput { store: JazzThoughtStore; occurredAt?: string } +export interface MaterializeArtifactOptions { store: JazzThoughtStore; requestEventId: string; workspaceRoot: string; workspaceLeaseId: string; occurredAt?: string } +export interface AppendArtifactResult { requestEvent?: ThoughtEvent; event: ThoughtEvent; inserted: boolean; blobInserted: boolean; bodySha256: string; byteCount: number } +export interface RequestAndMaterializeOptions extends ArtifactRequestInput { store: JazzThoughtStore; workspaceRoot: string; occurredAt?: string } + +export async function appendArtifactRequest(options: AppendArtifactRequestOptions): Promise<{ event: ThoughtEvent; inserted: boolean }> { + const status = options.status ?? "pending"; + const payload: ArtifactRequestPayload = { + requestId: options.requestId, workspaceLeaseId: options.workspaceLeaseId, workspaceRootLabel: options.workspaceRootLabel, + relativeFilePath: options.relativeFilePath, artifactId: options.artifactId, artifactVersion: options.artifactVersion, + kind: options.kind, title: options.title, summary: options.summary, mediaType: options.mediaType, + provenance: options.provenance, visibility: options.visibility, publicationEligible: false, publicationAuthority: false, + status, ...(options.reasonCode ? { reasonCode: options.reasonCode } : {}), + ...(options.requestingAgentId ? { requestingAgentId: options.requestingAgentId } : {}), + ...(options.requestingRunId ? { requestingRunId: options.requestingRunId } : {}), + ...(options.sourceEventId ? { sourceEventId: options.sourceEventId } : {}), + ...(options.supersedesArtifactEventId ? { supersedesArtifactEventId: options.supersedesArtifactEventId } : {}), + }; + const privacy = privacyFor(options.visibility); + const result = await options.store.appendEvent({ + type: ARTIFACT_REQUEST_EVENT_TYPE, schemaVersion: ARTIFACT_REQUEST_SCHEMA_VERSION, + source: "artifact-request:local-broker", sourceKind: "system", externalId: `artifact-request:${options.requestId}:${status}`, + idempotencyKey: `artifact-request:${options.requestId}:${status}`, occurredAt: options.occurredAt ?? new Date().toISOString(), + actor: options.requestingAgentId ?? "trusted-artifact-broker", correlationId: `artifact-request:${options.requestId}`, + privacy, payload: payload as unknown as JsonObject, + }); + return result; +} + +export async function materializeArtifactRequest(options: MaterializeArtifactOptions): Promise<AppendArtifactResult> { + const request = await options.store.getEvent(options.requestEventId); + if (!request || request.type !== ARTIFACT_REQUEST_EVENT_TYPE) throw new Error("Artifact request event not found"); + const payload = request.payload as unknown as ArtifactRequestPayload; + if (payload.status !== "pending") throw new Error("Only pending artifact requests can be materialized"); + if (payload.workspaceLeaseId !== options.workspaceLeaseId) throw new Error("Workspace lease identity does not match artifact request"); + const bytes = await readWorkspaceArtifact(options.workspaceRoot, payload.relativeFilePath, payload.mediaType); + const hash = hashBytes(bytes); + const blobPath = `sha256/${hash.slice(0, 2)}/${hash}`; + let blobInserted = false; + try { await fs.access(path.join(options.store.getArtifactRoot(), ...blobPath.split("/"))); } + catch { blobInserted = true; } + const blob = await storeBlob(options.store.getArtifactRoot(), bytes); + const relations: ArtifactRelation[] = [{ type: "materialized-from-request", eventId: request.id }]; + if (payload.sourceEventId) relations.push({ type: "source-event", eventId: payload.sourceEventId }); + if (payload.requestingRunId) relations.push({ type: "requesting-run", eventId: payload.requestingRunId }); + const artifactPayload: ArtifactPayload = { + artifactId: payload.artifactId, artifactVersion: payload.artifactVersion, kind: payload.kind, title: payload.title, + summary: payload.summary, mediaType: payload.mediaType, blob, provenance: payload.provenance, relations, + visibility: payload.visibility, publicationEligible: false, + ...(payload.supersedesArtifactEventId ? { supersedesArtifactEventId: payload.supersedesArtifactEventId } : {}), + }; + const identity = `artifact:${payload.artifactId}:v${payload.artifactVersion}`; + const candidate: EventCandidate = { + type: ARTIFACT_EVENT_TYPE, schemaVersion: ARTIFACT_SCHEMA_VERSION, source: "artifact-materializer:local-broker", sourceKind: "system", + externalId: identity, idempotencyKey: identity, occurredAt: options.occurredAt ?? request.occurredAt, + actor: "trusted-artifact-materializer", correlationId: request.id, privacy: privacyFor(payload.visibility), + payload: artifactPayload as unknown as JsonObject, + }; + const result = await options.store.appendEvent(candidate); + await rebuildArtifactCatalog(options.store); + return { requestEvent: request, event: result.event, inserted: result.inserted, blobInserted, bodySha256: blob.sha256, byteCount: blob.byteCount }; +} + +export async function requestArtifactStorage(options: RequestAndMaterializeOptions): Promise<AppendArtifactResult> { + const request = await appendArtifactRequest(options); + const result = await materializeArtifactRequest({ store: options.store, requestEventId: request.event.id, workspaceRoot: options.workspaceRoot, workspaceLeaseId: options.workspaceLeaseId, ...(options.occurredAt ? { occurredAt: options.occurredAt } : {}) }); + return { ...result, requestEvent: request.event }; +} + +/** Compatibility wrapper retained only for callers migrating to the explicit request seam. */ +export async function appendArtifactFromFile(options: { + store: JazzThoughtStore; rootPath: string; filePath: string; artifactId: string; artifactVersion: number; kind: ArtifactKind; + title: string; summary: string; mediaType: ArtifactMediaType; provenance: ArtifactProvenance; visibility: ArtifactVisibility; + source: string; actor: string; relations?: ArtifactRelation[]; supersedesArtifactEventId?: string; occurredAt?: string; +}): Promise<AppendArtifactResult> { + const relativeFilePath = path.relative(options.rootPath, options.filePath).split(path.sep).join("/"); + return requestArtifactStorage({ store: options.store, workspaceRoot: options.rootPath, workspaceLeaseId: "legacy-explicit-root", workspaceRootLabel: "legacy explicit root", relativeFilePath, + requestId: `${options.artifactId}-v${options.artifactVersion}`, artifactId: options.artifactId, artifactVersion: options.artifactVersion, kind: options.kind, + title: options.title, summary: options.summary, mediaType: options.mediaType, provenance: options.provenance, visibility: options.visibility, + ...(options.supersedesArtifactEventId ? { supersedesArtifactEventId: options.supersedesArtifactEventId } : {}), ...(options.occurredAt ? { occurredAt: options.occurredAt } : {}) }); +} + +function privacyFor(visibility: ArtifactVisibility): PrivacyClass { return visibility === "sensitive" ? "sensitive" : "private" } +function hashBytes(bytes: Buffer): string { return createHash("sha256").update(bytes).digest("hex") } diff --git a/src/artifacts/blob.ts b/src/artifacts/blob.ts new file mode 100644 index 0000000..c823418 --- /dev/null +++ b/src/artifacts/blob.ts @@ -0,0 +1,150 @@ +import { createHash, randomUUID } from "node:crypto"; +import { constants } from "node:fs"; +import fs from "node:fs/promises"; +import path from "node:path"; +import YAML from "yaml"; +import type { ArtifactBlobReference, ArtifactMediaType } from "./types.js"; +import { ARTIFACT_MAX_IMAGE_BYTES, ARTIFACT_MAX_TEXT_BYTES } from "./types.js"; + +const PAIRS: Record<string, readonly ArtifactMediaType[]> = { + ".md": ["text/markdown"], ".markdown": ["text/markdown"], ".txt": ["text/plain"], + ".json": ["application/json"], ".yaml": ["application/yaml", "text/yaml"], ".yml": ["application/yaml", "text/yaml"], + ".png": ["image/png"], ".jpg": ["image/jpeg"], ".jpeg": ["image/jpeg"], +}; + +export function canonicalBlobPath(hash: string): string { return `sha256/${hash.slice(0, 2)}/${hash}` } + +export async function readWorkspaceArtifact(rootInput: string, relativeFilePath: string, mediaType: ArtifactMediaType): Promise<Buffer> { + if (!path.isAbsolute(rootInput)) throw new Error("Workspace lease root must be absolute"); + validateRelative(relativeFilePath); + const root = path.resolve(rootInput); + const rootStat = await fs.lstat(root); + if (!rootStat.isDirectory() || rootStat.isSymbolicLink()) throw new Error("Workspace lease root must be a non-symlink directory"); + const canonicalRoot = await fs.realpath(root); + if (canonicalRoot !== root) throw new Error("Workspace lease root must be canonical and contain no symlinks"); + const parts = relativeFilePath.split("/"); + let current = root; + for (const part of parts) { + current = path.join(current, part); + const stat = await fs.lstat(current); + if (stat.isSymbolicLink()) throw new Error("Artifact file and parent components must not be symbolic links"); + } + const candidate = path.resolve(root, ...parts); + const relative = path.relative(root, candidate); + if (!relative || relative.startsWith("..") || path.isAbsolute(relative)) throw new Error("Artifact file must be contained by the explicit workspace lease root"); + const canonical = await fs.realpath(candidate); + if (canonical !== candidate) throw new Error("Artifact file and parent components must not be symbolic links"); + const stat = await fs.lstat(candidate); + if (!stat.isFile()) throw new Error("Artifact path is not a regular file"); + const allowed = PAIRS[path.extname(candidate).toLowerCase()]; + if (!allowed?.includes(mediaType)) throw new Error("Artifact extension and media type do not match"); + const max = maxBytes(mediaType); + if (stat.size === 0) throw new Error("Artifact file is empty"); + if (stat.size > max) throw new Error(`Artifact file exceeds ${max} bytes`); + const handle = await fs.open(candidate, constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const opened = await handle.stat(); + if (!opened.isFile() || opened.size !== stat.size) throw new Error("Artifact file changed during validation"); + const bytes = await readHandleBounded(handle, max); + if (bytes.byteLength !== opened.size) throw new Error("Artifact file changed during read"); + validateBytes(bytes, mediaType); + return bytes; + } finally { await handle.close(); } +} + +export async function storeBlob(rootInput: string, bytes: Buffer): Promise<ArtifactBlobReference> { + const hash = createHash("sha256").update(bytes).digest("hex"); + const relativePath = canonicalBlobPath(hash); + const root = path.resolve(rootInput); + await ensurePrivateDirectory(root); + await rejectSymlinkComponents(root, relativePath.split("/").slice(0, -1), true); + const destination = path.join(root, ...relativePath.split("/")); + const temporary = path.join(path.dirname(destination), `.${hash}.${randomUUID()}.tmp`); + await fs.writeFile(temporary, bytes, { flag: "wx", mode: 0o600 }); + try { + try { await fs.link(temporary, destination); } + catch (error) { + if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error; + await verifyExisting(destination, bytes, hash); + } + } finally { await fs.rm(temporary, { force: true }); } + return { algorithm: "sha256", sha256: hash, relativePath, byteCount: bytes.byteLength }; +} + +export async function readVerifiedBlob(rootInput: string, blob: ArtifactBlobReference, mediaType: ArtifactMediaType): Promise<Buffer> { + if (blob.relativePath !== canonicalBlobPath(blob.sha256)) throw new Error("Artifact blob path is not canonical"); + if (blob.byteCount <= 0 || blob.byteCount > maxBytes(mediaType)) throw new Error("Artifact blob size metadata is invalid"); + const root = path.resolve(rootInput); + const rootStat = await fs.lstat(root); + if (!rootStat.isDirectory() || rootStat.isSymbolicLink() || await fs.realpath(root) !== root) throw new Error("Artifact blob root is not canonical"); + await rejectSymlinkComponents(root, blob.relativePath.split("/"), false); + const destination = path.join(root, ...blob.relativePath.split("/")); + const handle = await fs.open(destination, constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const stat = await handle.stat(); + if (!stat.isFile() || stat.size !== blob.byteCount) throw new Error("Artifact blob size integrity check failed"); + const bytes = await readHandleBounded(handle, maxBytes(mediaType)); + const hash = createHash("sha256").update(bytes).digest("hex"); + if (hash !== blob.sha256) throw new Error("Artifact blob hash integrity check failed"); + validateBytes(bytes, mediaType); + return bytes; + } finally { await handle.close(); } +} + +function validateRelative(value: string): void { + if (!value || value.includes("\\") || path.isAbsolute(value) || value.endsWith("/") || value.split("/").some((part) => !part || part === "." || part === "..")) { + throw new Error("Artifact file path must be one normalized relative path"); + } +} +function maxBytes(mediaType: ArtifactMediaType): number { return mediaType.startsWith("image/") ? ARTIFACT_MAX_IMAGE_BYTES : ARTIFACT_MAX_TEXT_BYTES } +async function readHandleBounded(handle: fs.FileHandle, max: number): Promise<Buffer> { + const chunks: Buffer[] = []; let total = 0; let position = 0; + while (true) { + const chunk = Buffer.alloc(Math.min(64 * 1024, max + 1 - total)); + const { bytesRead } = await handle.read(chunk, 0, chunk.length, position); + if (bytesRead === 0) break; + total += bytesRead; position += bytesRead; + if (total > max) throw new Error(`Artifact exceeds ${max} bytes`); + chunks.push(chunk.subarray(0, bytesRead)); + } + return Buffer.concat(chunks, total); +} +function validateBytes(bytes: Buffer, mediaType: ArtifactMediaType): void { + if (mediaType === "image/png") { + if (bytes.length < 8 || !bytes.subarray(0, 8).equals(Buffer.from([137,80,78,71,13,10,26,10]))) throw new Error("Artifact PNG magic does not match media type"); + return; + } + if (mediaType === "image/jpeg") { + if (bytes.length < 4 || bytes[0] !== 0xff || bytes[1] !== 0xd8 || bytes.at(-2) !== 0xff || bytes.at(-1) !== 0xd9) throw new Error("Artifact JPEG magic does not match media type"); + return; + } + const text = new TextDecoder("utf-8", { fatal: true }).decode(bytes); + if (text.includes("\0")) throw new Error("Text artifact contains a NUL byte"); + if (mediaType === "application/json") JSON.parse(text); + if (mediaType === "application/yaml" || mediaType === "text/yaml") YAML.parse(text); +} +async function ensurePrivateDirectory(root: string): Promise<void> { + await fs.mkdir(root, { recursive: true, mode: 0o700 }); + const stat = await fs.lstat(root); + if (!stat.isDirectory() || stat.isSymbolicLink() || await fs.realpath(root) !== root) throw new Error("Artifact blob root must be a canonical non-symlink directory"); + await fs.chmod(root, 0o700); +} +async function rejectSymlinkComponents(root: string, parts: string[], createDirectories: boolean): Promise<void> { + let current = root; + for (const part of parts) { + current = path.join(current, part); + if (createDirectories) await fs.mkdir(current, { mode: 0o700 }).catch((error: NodeJS.ErrnoException) => { if (error.code !== "EEXIST") throw error; }); + const stat = await fs.lstat(current); + if (stat.isSymbolicLink()) throw new Error("Artifact blob path contains a symbolic link"); + if (part !== parts.at(-1) || createDirectories) { + if (!stat.isDirectory()) throw new Error("Artifact blob parent is not a directory"); + if (createDirectories) await fs.chmod(current, 0o700); + } + } +} +async function verifyExisting(destination: string, expected: Buffer, hash: string): Promise<void> { + const stat = await fs.lstat(destination); + if (!stat.isFile() || stat.isSymbolicLink() || stat.size !== expected.byteLength) throw new Error("Existing artifact blob conflicts with requested bytes"); + const actual = await fs.readFile(destination); + if (createHash("sha256").update(actual).digest("hex") !== hash || !actual.equals(expected)) throw new Error("Existing artifact blob conflicts with requested bytes"); +} diff --git a/src/artifacts/catalog.ts b/src/artifacts/catalog.ts new file mode 100644 index 0000000..8c1c868 --- /dev/null +++ b/src/artifacts/catalog.ts @@ -0,0 +1,62 @@ +import type { JsonObject } from "../core/json.js"; +import type { ThoughtEvent } from "../events/types.js"; +import type { JazzThoughtStore } from "../jazz/store.js"; +import { readVerifiedBlob } from "./blob.js"; +import type { ArtifactBlobReference, ArtifactMediaType, ArtifactPayload } from "./types.js"; +import { ARTIFACT_EVENT_TYPE } from "./types.js"; + +export interface ArtifactCatalogEntry { + eventId: string; artifactId: string; artifactVersion: number; kind: string; title: string; summary: string; + mediaType: string; bodySha256: string; bodySizeBytes: number; blobPath: string; + provenance: { source: string; label: string }; relations: Array<{ type: string; eventId: string }>; + visibility: string; publicationEligible: boolean; superseded: boolean; supersedesArtifactEventId?: string; + occurredAt: string; source: string; actor: string; privacy: string; +} +export interface ArtifactCatalogProjection { total: number; active: number; superseded: number; entries: ArtifactCatalogEntry[] } +export type ArtifactDetail = { event: ThoughtEvent; bodySha256: string; byteCount: number; mediaType: ArtifactMediaType; text?: string; contentPath?: string }; + +export async function buildArtifactCatalog(store: JazzThoughtStore): Promise<ArtifactCatalogProjection> { + const events = await store.listEvents({ types: [ARTIFACT_EVENT_TYPE] }); + const supersededIds = new Set(events.map((event) => optionalString(event.payload.supersedesArtifactEventId)).filter(Boolean) as string[]); + const entries = events.map((event) => entryFromEvent(event, supersededIds.has(event.id))).sort((a, b) => a.artifactId.localeCompare(b.artifactId) || a.artifactVersion - b.artifactVersion || a.eventId.localeCompare(b.eventId)); + return { total: entries.length, active: entries.filter((entry) => !entry.superseded).length, superseded: entries.filter((entry) => entry.superseded).length, entries }; +} +export async function rebuildArtifactCatalog(store: JazzThoughtStore): Promise<ArtifactCatalogProjection> { + const catalog = await buildArtifactCatalog(store); + await store.upsertProjection({ id: "artifact-catalog", payload: JSON.parse(JSON.stringify(catalog)) as JsonObject, lastEventId: catalog.entries.at(-1)?.eventId ?? "none", projectionVersion: 1, updatedAt: new Date().toISOString() }); + return catalog; +} +export async function getArtifactCatalog(store: JazzThoughtStore): Promise<ArtifactCatalogProjection> { + const projection = await store.getProjection("artifact-catalog"); + const payload = projection?.payload; + if ( + projection?.projectionVersion === 1 + && payload + && Number.isSafeInteger(payload.total) + && Number.isSafeInteger(payload.active) + && Number.isSafeInteger(payload.superseded) + && Array.isArray(payload.entries) + ) return payload as unknown as ArtifactCatalogProjection; + return rebuildArtifactCatalog(store); +} +export async function getArtifactBody(store: JazzThoughtStore, eventId: string): Promise<ArtifactDetail | undefined> { + const resolved = await getArtifactContent(store, eventId); if (!resolved) return undefined; + const { event, bytes, payload } = resolved; + return payload.mediaType.startsWith("image/") + ? { event, bodySha256: payload.blob.sha256, byteCount: bytes.byteLength, mediaType: payload.mediaType, contentPath: `/api/artifacts/${encodeURIComponent(event.id)}/content` } + : (() => { const text = new TextDecoder("utf-8", { fatal: true }).decode(bytes); return { event, bodySha256: payload.blob.sha256, byteCount: bytes.byteLength, mediaType: payload.mediaType, text }; })(); +} +export async function getArtifactContent(store: JazzThoughtStore, eventId: string): Promise<{ event: ThoughtEvent; payload: ArtifactPayload; bytes: Buffer } | undefined> { + const event = await store.getEvent(eventId); if (!event || event.type !== ARTIFACT_EVENT_TYPE) return undefined; + const payload = event.payload as unknown as ArtifactPayload; + const bytes = await readVerifiedBlob(store.getArtifactRoot(), payload.blob as ArtifactBlobReference, payload.mediaType as ArtifactMediaType); + return { event, payload, bytes }; +} +function entryFromEvent(event: ThoughtEvent, superseded: boolean): ArtifactCatalogEntry { + const payload = event.payload as unknown as ArtifactPayload; + return { eventId: event.id, artifactId: payload.artifactId, artifactVersion: payload.artifactVersion, kind: payload.kind, title: payload.title, + summary: payload.summary, mediaType: payload.mediaType, bodySha256: payload.blob.sha256, bodySizeBytes: payload.blob.byteCount, blobPath: payload.blob.relativePath, + provenance: payload.provenance, relations: payload.relations, visibility: payload.visibility, publicationEligible: payload.publicationEligible, superseded, + ...(payload.supersedesArtifactEventId ? { supersedesArtifactEventId: payload.supersedesArtifactEventId } : {}), occurredAt: event.occurredAt, source: event.source, actor: event.actor, privacy: event.privacy }; +} +function optionalString(value: unknown): string | undefined { return typeof value === "string" && value.length > 0 ? value : undefined } diff --git a/src/artifacts/exe-transport.ts b/src/artifacts/exe-transport.ts new file mode 100644 index 0000000..7e924a5 --- /dev/null +++ b/src/artifacts/exe-transport.ts @@ -0,0 +1,169 @@ +import { createHash } from "node:crypto"; +import { constants, createWriteStream } from "node:fs"; +import fs from "node:fs/promises"; +import path from "node:path"; +import { spawn } from "node:child_process"; +import { z } from "zod"; +import type { JazzThoughtStore } from "../jazz/store.js"; +import { readVerifiedBlob } from "./blob.js"; +import { requestArtifactStorage } from "./append.js"; +import { ARTIFACT_KINDS, ARTIFACT_MEDIA_TYPES, ARTIFACT_VISIBILITY } from "./types.js"; + +const id = z.string().min(1).max(200).regex(/^[A-Za-z0-9][A-Za-z0-9._:-]*$/); +const text = z.string().min(1).max(2_000); +const MAX_SSH_OUTPUT_BYTES = 12 * 1024 * 1024; +export const exeArtifactRequestSchema = z.object({ + requestId: id, leaseId: id, + relativeOutputFilePath: z.string().min(1).max(500).refine(normalizedRelative, "not-normalized-relative"), + artifactId: id, artifactVersion: z.number().int().positive().max(1_000_000), + kind: z.enum(ARTIFACT_KINDS), title: text, summary: text, mediaType: z.enum(ARTIFACT_MEDIA_TYPES), + provenanceLabel: text, visibility: z.enum(ARTIFACT_VISIBILITY).default("private"), + requestingAgentId: id.optional(), runId: id.optional(), sourceEventId: id.optional(), supersedes: id.optional(), +}).strict(); +export type ExeArtifactRequest = z.infer<typeof exeArtifactRequestSchema>; + +export interface ExeTransportConfig { + host: "cameron.exe.xyz"; + workspaceRoot: "/srv/thoughtstream/workspaces/stream-v1"; + workspaceId: "stream-v1"; + leaseId: string; + stagingRoot: string; +} +export interface ExeTransportReceipt { requestId: string; requestEventId: string; artifactEventId: string; artifactId: string; sha256: string; bytes: number; status: "stored"; timestamp: string } + +const REMOTE_HELPER = String.raw`import base64,hashlib,json,os,stat,sys +cfg=json.loads(base64.urlsafe_b64decode(sys.argv[1]+"==")) +root=cfg["root"]; rel=cfg["relative"] +def fail(code): + print(json.dumps({"ok":False,"reasonCode":code},separators=(",",":")));sys.exit(0) +try: + mpath=os.path.join(root,".thoughtstream-workspace.json") + rst=os.lstat(root) + if stat.S_ISLNK(rst.st_mode) or not stat.S_ISDIR(rst.st_mode) or os.path.realpath(root)!=root: fail("workspace-invalid") + with open(mpath,"rb") as f: m=json.load(f) + if m.get("workspaceId")!=cfg["workspaceId"] or m.get("leaseId")!=cfg["leaseId"] or m.get("vmIdentity")!=cfg["host"]: fail("workspace-mismatch") + parts=rel.split("/") + if not rel or rel.startswith("/") or "\\" in rel or any(x in ("",".","..") for x in parts): fail("path-invalid") + cur=os.path.join(root,"output") + for part in parts: + cur=os.path.join(cur,part); s=os.lstat(cur) + if stat.S_ISLNK(s.st_mode): fail("symlink") + if os.path.realpath(cur)!=cur or not stat.S_ISREG(os.lstat(cur).st_mode): fail("not-regular") + ext=os.path.splitext(cur)[1].lower(); pairs={".md":["text/markdown"],".markdown":["text/markdown"],".txt":["text/plain"],".json":["application/json"],".yaml":["application/yaml","text/yaml"],".yml":["application/yaml","text/yaml"],".png":["image/png"],".jpg":["image/jpeg"],".jpeg":["image/jpeg"]} + if cfg["mediaType"] not in pairs.get(ext,[]): fail("media-mismatch") + maximum=10485760 if cfg["mediaType"].startswith("image/") else 1048576 + before=os.lstat(cur) + if before.st_size<=0 or before.st_size>maximum: fail("size-invalid") + fd=os.open(cur,os.O_RDONLY|os.O_NOFOLLOW) + try: + opened=os.fstat(fd); data=b"" + while len(data)<=maximum: + chunk=os.read(fd,min(65536,maximum+1-len(data))) + if not chunk: break + data+=chunk + after=os.fstat(fd) + finally: os.close(fd) + ident=lambda s:(s.st_dev,s.st_ino,s.st_size,s.st_mtime_ns,s.st_ctime_ns) + if ident(before)!=ident(opened) or ident(opened)!=ident(after) or len(data)!=after.st_size: fail("unstable-snapshot") + mt=cfg["mediaType"] + if mt=="image/png" and not data.startswith(b"\x89PNG\r\n\x1a\n"): fail("magic-invalid") + if mt=="image/jpeg" and not (data.startswith(b"\xff\xd8") and data.endswith(b"\xff\xd9")): fail("magic-invalid") + if not mt.startswith("image/"): + try: value=data.decode("utf-8") + except UnicodeDecodeError: fail("encoding-invalid") + if "\x00" in value: fail("encoding-invalid") + if mt=="application/json": json.loads(value) + header=json.dumps({"ok":True,"bytes":len(data),"sha256":hashlib.sha256(data).hexdigest(),"mediaType":mt,"relativePath":rel},separators=(",",":" )).encode() + sys.stdout.buffer.write(str(len(header)).encode()+b"\n"+header+data) +except (OSError,ValueError,json.JSONDecodeError): fail("remote-validation-failed")`; + +const REMOTE_RECEIPT_HELPER = String.raw`import base64,json,os,sys +p=sys.argv[1] +r=json.loads(base64.urlsafe_b64decode(sys.argv[2]+"==")) +n=r["requestId"]+".json" +assert all(c.isalnum() or c in "._:-" for c in r["requestId"]) +d=(json.dumps(r,separators=(",",":"),sort_keys=True)+"\n").encode() +q=os.path.join(p,n) +try: + f=os.open(q,os.O_WRONLY|os.O_CREAT|os.O_EXCL,0o600) + os.write(f,d) + os.close(f) +except FileExistsError: + assert open(q,"rb").read()==d`; + +const REMOTE_PROCESSED_HELPER = String.raw`import os,sys +s=os.path.join(sys.argv[1],sys.argv[3]) +d=os.path.join(sys.argv[2],sys.argv[3]) +if os.path.exists(d): + assert open(s,"rb").read()==open(d,"rb").read() + os.unlink(s) +else: + os.rename(s,d)`; + +export async function processExeArtifactRequest(store: JazzThoughtStore, config: ExeTransportConfig, raw: unknown): Promise<ExeTransportReceipt> { + const request = exeArtifactRequestSchema.parse(raw); + if (request.leaseId !== config.leaseId) throw reason("lease-mismatch"); + await ensurePrivateStaging(config.stagingRoot); + const framed = await remoteFetch(config, request); + const hash = createHash("sha256").update(framed.bytes).digest("hex"); + if (hash !== framed.header.sha256 || framed.bytes.byteLength !== framed.header.bytes || framed.header.mediaType !== request.mediaType || framed.header.relativePath !== request.relativeOutputFilePath) throw reason("content-integrity"); + const requestStage = await fs.mkdtemp(path.join(config.stagingRoot, "request-")); + await fs.chmod(requestStage, 0o700); + const staged = path.join(requestStage, "artifact" + path.extname(request.relativeOutputFilePath)); + try { + await fs.writeFile(staged, framed.bytes, { mode: 0o600, flag: "wx" }); + const result = await requestArtifactStorage({ + store, workspaceRoot: requestStage, workspaceLeaseId: config.leaseId, workspaceRootLabel: `exe-workspace:${config.workspaceId}`, + requestId: request.requestId, relativeFilePath: path.basename(staged), artifactId: request.artifactId, artifactVersion: request.artifactVersion, + kind: request.kind, title: request.title, summary: request.summary, mediaType: request.mediaType, + provenance: { source: "exe-dev-private-workspace", label: request.provenanceLabel }, visibility: request.visibility, + ...(request.requestingAgentId ? { requestingAgentId: request.requestingAgentId } : {}), ...(request.runId ? { requestingRunId: request.runId } : {}), + ...(request.sourceEventId ? { sourceEventId: request.sourceEventId } : {}), ...(request.supersedes ? { supersedesArtifactEventId: request.supersedes } : {}), + }); + const persisted = await store.getEvent(result.event.id); + if (!persisted || persisted.id !== result.event.id) throw reason("durable-readback-failed"); + const blob = persisted.payload.blob as { algorithm: "sha256"; sha256: string; relativePath: string; byteCount: number }; + const readback = await readVerifiedBlob(store.getArtifactRoot(), blob, request.mediaType); + if (!readback.equals(framed.bytes)) throw reason("durable-readback-failed"); + const receipt: ExeTransportReceipt = { requestId: request.requestId, requestEventId: result.requestEvent!.id, artifactEventId: result.event.id, artifactId: request.artifactId, sha256: hash, bytes: framed.bytes.byteLength, status: "stored", timestamp: persisted.observedAt }; + await remoteReceipt(config, receipt); + return receipt; + } finally { await fs.rm(requestStage, { recursive: true, force: true }); } +} + +export async function runExeArtifactTransportOnce(store: JazzThoughtStore, config: ExeTransportConfig): Promise<{ receipts: ExeTransportReceipt[]; failures: Array<{ requestId: string; reasonCode: string }> }> { + const lock = await acquireLock(path.join(config.stagingRoot, "transport.lock")); + try { + const names = await remoteList(config); const receipts: ExeTransportReceipt[] = []; const failures: Array<{ requestId: string; reasonCode: string }> = []; + for (const name of names) { + let raw: unknown; let requestId = name.replace(/\.json$/, ""); + try { raw = JSON.parse(await remoteReadRequest(config, name)); requestId = typeof (raw as { requestId?: unknown }).requestId === "string" ? (raw as { requestId: string }).requestId : requestId; receipts.push(await processExeArtifactRequest(store, config, raw)); await remoteProcessed(config, name); } + catch (error) { failures.push({ requestId: safeId(requestId), reasonCode: reasonCode(error) }); } + } + return { receipts, failures }; + } finally { await lock.close(); await fs.rm(path.join(config.stagingRoot, "transport.lock"), { force: true }); } +} + +async function remoteFetch(config: ExeTransportConfig, request: ExeArtifactRequest): Promise<{ header: { bytes: number; sha256: string; mediaType: string; relativePath: string }; bytes: Buffer }> { + const payload = b64({ root: config.workspaceRoot, workspaceId: config.workspaceId, leaseId: config.leaseId, host: config.host, relative: request.relativeOutputFilePath, mediaType: request.mediaType }); + const output = await ssh(config.host, ["python3", "-c", b64(REMOTE_HELPER), payload]); + const newline = output.indexOf(10); if (newline < 1) throw reason("remote-frame-invalid"); + const length = Number(output.subarray(0, newline).toString("ascii")); + if (!Number.isSafeInteger(length) || length < 2 || length > 2048) { const failure = parseFailure(output); throw reason(failure); } + const header = JSON.parse(output.subarray(newline + 1, newline + 1 + length).toString("utf8")); const bytes = output.subarray(newline + 1 + length); + if (!header.ok) throw reason(header.reasonCode ?? "remote-validation-failed"); + return { header, bytes }; +} +async function remoteList(config: ExeTransportConfig): Promise<string[]> { const output = await ssh(config.host, ["python3", "-c", b64("import json,os,sys;p=sys.argv[1];v=sorted(x for x in os.listdir(p) if x.endswith('.json') and '/' not in x);assert len(v)<=256;print(json.dumps(v))"), `${config.workspaceRoot}/requests`]); return JSON.parse(output.toString("utf8")); } +async function remoteReadRequest(config: ExeTransportConfig, name: string): Promise<string> { if (!/^[A-Za-z0-9][A-Za-z0-9._-]{0,199}\.json$/.test(name)) throw reason("request-name-invalid"); return (await ssh(config.host, ["python3", "-c", b64("import os,sys; p=os.path.join(sys.argv[1],sys.argv[2]); s=os.lstat(p); assert not os.path.islink(p) and os.path.isfile(p) and s.st_size<=16384; sys.stdout.buffer.write(open(p,'rb').read())"), `${config.workspaceRoot}/requests`, name])).toString("utf8"); } +async function remoteReceipt(config: ExeTransportConfig, receipt: ExeTransportReceipt): Promise<void> { await ssh(config.host, ["python3", "-c", b64(REMOTE_RECEIPT_HELPER), `${config.workspaceRoot}/receipts`, b64(receipt)]); } +async function remoteProcessed(config: ExeTransportConfig, name: string): Promise<void> { await ssh(config.host, ["python3", "-c", b64(REMOTE_PROCESSED_HELPER), `${config.workspaceRoot}/requests`, `${config.workspaceRoot}/requests/processed`, name]); } +function ssh(host: string, args: string[], input?: Buffer): Promise<Buffer> { const quote=(arg:string)=>`'${arg.replaceAll("'", "'\\''")}'`; return new Promise((resolve, reject) => { const remote = args[1] === "-c" && args.length >= 3 ? `python3 -c 'import base64,sys;exec(base64.urlsafe_b64decode(\"${args[2]}==\"))' ${quote(args[3] ?? "")}${args.slice(4).map((arg)=>` ${quote(arg)}`).join("")}` : args.map(quote).join(" "); const child = spawn("ssh", ["-o", "BatchMode=yes", "-o", "ConnectTimeout=10", "-o", "StrictHostKeyChecking=yes", "-o", "ClearAllForwardings=yes", "--", host, remote], { stdio: ["pipe", "pipe", "pipe"] }); const out: Buffer[]=[]; const err: Buffer[]=[]; let outputBytes=0; let outputOverflow=false; child.stdout.on("data", x=>{ outputBytes+=x.length; if(outputBytes>MAX_SSH_OUTPUT_BYTES){ outputOverflow=true; child.kill("SIGKILL"); return; } out.push(x); }); child.stderr.on("data", x=>{ if(Buffer.concat(err).byteLength<4096) err.push(x); }); child.on("error", reject); child.on("close", code=>outputOverflow?reject(reason("ssh-output-too-large")):code===0?resolve(Buffer.concat(out)):reject(Object.assign(reason("ssh-failed"), { detail: Buffer.concat(err).toString("utf8").slice(0, 500) }))); if (input) child.stdin.end(input); else child.stdin.end(); }); } +async function ensurePrivateStaging(root: string): Promise<void> { if (!path.isAbsolute(root)) throw reason("staging-invalid"); await fs.mkdir(root,{recursive:true,mode:0o700}); await fs.chmod(root,0o700); const s=await fs.lstat(root); if (!s.isDirectory()||s.isSymbolicLink()||await fs.realpath(root)!==root) throw reason("staging-invalid"); } +async function acquireLock(file: string): Promise<fs.FileHandle> { await ensurePrivateStaging(path.dirname(file)); try { return await fs.open(file, constants.O_CREAT|constants.O_EXCL|constants.O_WRONLY,0o600); } catch { throw reason("already-running"); } } +function b64(value: unknown): string { return Buffer.from(typeof value === "string" ? value : JSON.stringify(value)).toString("base64url"); } +function normalizedRelative(value: string): boolean { return !path.isAbsolute(value) && !value.includes("\\") && !value.endsWith("/") && value.split("/").every(part=>part!==""&&part!=="."&&part!==".."); } +function reason(code: string): Error { return Object.assign(new Error(code), { reasonCode: code }); } +function reasonCode(error: unknown): string { const code=(error as {reasonCode?:unknown})?.reasonCode; return typeof code==="string"&&/^[a-z0-9-]{1,64}$/.test(code)?code:"request-rejected"; } +function safeId(value: string): string { return /^[A-Za-z0-9][A-Za-z0-9._:-]{0,199}$/.test(value)?value:"invalid-request"; } +function parseFailure(output: Buffer): string { try { const value=JSON.parse(output.toString("utf8")); return typeof value.reasonCode==="string"?value.reasonCode:"remote-frame-invalid"; } catch { return "remote-frame-invalid"; } } diff --git a/src/artifacts/schema.ts b/src/artifacts/schema.ts new file mode 100644 index 0000000..36cd878 --- /dev/null +++ b/src/artifacts/schema.ts @@ -0,0 +1,70 @@ +import { z } from "zod"; +import type { JsonObject } from "../core/json.js"; +import { + ARTIFACT_KINDS, ARTIFACT_MAX_IMAGE_BYTES, ARTIFACT_MAX_SUMMARY_LENGTH, ARTIFACT_MAX_TEXT_BYTES, + ARTIFACT_MAX_TITLE_LENGTH, ARTIFACT_MEDIA_TYPES, ARTIFACT_REQUEST_STATUSES, ARTIFACT_VISIBILITY, +} from "./types.js"; + +const sha256Schema = z.string().regex(/^[a-f0-9]{64}$/); +const bounded = z.string().min(1).max(200); +const notAbsolute = (value: string) => !/^(?:\/|[A-Za-z]:[\\/])/.test(value); +const labelSchema = bounded.refine(notAbsolute, "Label must not be an absolute filesystem path"); +const relativePathSchema = z.string().min(1).max(500).refine((value) => { + if (value.includes("\\") || value.startsWith("/") || value.endsWith("/")) return false; + const parts = value.split("/"); + return parts.every((part) => part.length > 0 && part !== "." && part !== ".."); +}, "Artifact file path must be one normalized relative path"); + +const provenanceSchema = z.object({ source: labelSchema, label: labelSchema }).strict(); +const relationSchema = z.object({ type: bounded, eventId: bounded }).strict(); +const commonMetadata = { + artifactId: bounded, + artifactVersion: z.number().int().positive().max(1_000_000), + kind: z.enum(ARTIFACT_KINDS), + title: z.string().min(1).max(ARTIFACT_MAX_TITLE_LENGTH), + summary: z.string().min(1).max(ARTIFACT_MAX_SUMMARY_LENGTH), + mediaType: z.enum(ARTIFACT_MEDIA_TYPES), + provenance: provenanceSchema, + visibility: z.enum(ARTIFACT_VISIBILITY), + publicationEligible: z.literal(false), +}; + +export const artifactPayloadSchema = z.object({ + ...commonMetadata, + blob: z.object({ + algorithm: z.literal("sha256"), + sha256: sha256Schema, + relativePath: z.string().regex(/^sha256\/[a-f0-9]{2}\/[a-f0-9]{64}$/), + byteCount: z.number().int().positive().max(ARTIFACT_MAX_IMAGE_BYTES), + }).strict().superRefine((blob, context) => { + if (blob.relativePath !== `sha256/${blob.sha256.slice(0, 2)}/${blob.sha256}`) { + context.addIssue({ code: "custom", path: ["relativePath"], message: "Blob path is not canonical for its SHA-256" }); + } + }), + relations: z.array(relationSchema).min(1).max(50).refine( + (relations) => relations.some((relation) => relation.type === "materialized-from-request"), + "Artifact must strongly relate to its materialization request", + ), + supersedesArtifactEventId: bounded.optional(), +}).strict().superRefine((value, context) => { + const max = value.mediaType.startsWith("image/") ? ARTIFACT_MAX_IMAGE_BYTES : ARTIFACT_MAX_TEXT_BYTES; + if (value.blob.byteCount > max) context.addIssue({ code: "custom", path: ["blob", "byteCount"], message: `Artifact exceeds ${max} bytes` }); +}) as unknown as z.ZodType<JsonObject>; + +export const artifactRequestPayloadSchema = z.object({ + requestId: bounded, + workspaceLeaseId: bounded, + workspaceRootLabel: labelSchema, + relativeFilePath: relativePathSchema, + ...commonMetadata, + publicationAuthority: z.literal(false), + status: z.enum(ARTIFACT_REQUEST_STATUSES), + reasonCode: z.string().min(1).max(100).regex(/^[a-z0-9-]+$/).optional(), + requestingAgentId: bounded.optional(), + requestingRunId: bounded.optional(), + sourceEventId: bounded.optional(), + supersedesArtifactEventId: bounded.optional(), +}).strict().superRefine((value, context) => { + if (value.status === "pending" && value.reasonCode) context.addIssue({ code: "custom", path: ["reasonCode"], message: "Pending request cannot have a reason code" }); + if (value.status !== "pending" && !value.reasonCode) context.addIssue({ code: "custom", path: ["reasonCode"], message: "Terminal request requires a reason code" }); +}) as unknown as z.ZodType<JsonObject>; diff --git a/src/artifacts/types.ts b/src/artifacts/types.ts new file mode 100644 index 0000000..4e2d8da --- /dev/null +++ b/src/artifacts/types.ts @@ -0,0 +1,76 @@ +export const ARTIFACT_EVENT_TYPE = "stream.thought.artifact"; +export const ARTIFACT_REQUEST_EVENT_TYPE = "stream.thought.artifact.requested"; +export const ARTIFACT_SCHEMA_VERSION = 1; +export const ARTIFACT_REQUEST_SCHEMA_VERSION = 1; + +export const ARTIFACT_KINDS = ["skill", "spec", "report", "design", "code", "document", "image"] as const; +export type ArtifactKind = (typeof ARTIFACT_KINDS)[number]; + +export const ARTIFACT_MEDIA_TYPES = [ + "text/markdown", + "text/plain", + "application/json", + "application/yaml", + "text/yaml", + "image/png", + "image/jpeg", +] as const; +export type ArtifactMediaType = (typeof ARTIFACT_MEDIA_TYPES)[number]; + +export const ARTIFACT_VISIBILITY = ["private", "sensitive"] as const; +export type ArtifactVisibility = (typeof ARTIFACT_VISIBILITY)[number]; +export const ARTIFACT_REQUEST_STATUSES = ["pending", "failed", "denied"] as const; +export type ArtifactRequestStatus = (typeof ARTIFACT_REQUEST_STATUSES)[number]; + +export const ARTIFACT_MAX_TEXT_BYTES = 256 * 1024; +export const ARTIFACT_MAX_IMAGE_BYTES = 8 * 1024 * 1024; +export const ARTIFACT_MAX_BODY_BYTES = ARTIFACT_MAX_TEXT_BYTES; +export const ARTIFACT_MAX_TITLE_LENGTH = 500; +export const ARTIFACT_MAX_SUMMARY_LENGTH = 2_000; + +export interface ArtifactProvenance { source: string; label: string } +export interface ArtifactRelation { type: string; eventId: string } +export interface ArtifactBlobReference { + algorithm: "sha256"; + sha256: string; + relativePath: string; + byteCount: number; +} + +export interface ArtifactPayload { + artifactId: string; + artifactVersion: number; + kind: ArtifactKind; + title: string; + summary: string; + mediaType: ArtifactMediaType; + blob: ArtifactBlobReference; + provenance: ArtifactProvenance; + relations: ArtifactRelation[]; + visibility: ArtifactVisibility; + publicationEligible: false; + supersedesArtifactEventId?: string; +} + +export interface ArtifactRequestPayload { + requestId: string; + workspaceLeaseId: string; + workspaceRootLabel: string; + relativeFilePath: string; + artifactId: string; + artifactVersion: number; + kind: ArtifactKind; + title: string; + summary: string; + mediaType: ArtifactMediaType; + provenance: ArtifactProvenance; + visibility: ArtifactVisibility; + publicationEligible: false; + publicationAuthority: false; + status: ArtifactRequestStatus; + reasonCode?: string; + requestingAgentId?: string; + requestingRunId?: string; + sourceEventId?: string; + supersedesArtifactEventId?: string; +} diff --git a/src/cli.ts b/src/cli.ts index a279112..e3402bd 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -598,11 +598,87 @@ try { } else { for (const example of examples) process.stdout.write(`${JSON.stringify(example)}\n`); } + } else if (command === "exe-artifact-transport") { + const host = valueAfter("--host"); + const workspaceRoot = valueAfter("--workspace-root"); + const workspaceId = valueAfter("--workspace-id"); + const leaseId = valueAfter("--lease-id"); + const stagingRoot = valueAfter("--staging-root"); + if (host !== "cameron.exe.xyz" || workspaceRoot !== "/srv/thoughtstream/workspaces/stream-v1" || workspaceId !== "stream-v1" || !leaseId || !stagingRoot) { + throw new Error("Usage: thought exe-artifact-transport --host cameron.exe.xyz --workspace-root /srv/thoughtstream/workspaces/stream-v1 --workspace-id stream-v1 --lease-id <id> --staging-root <absolute-private-path> [--poll-seconds <5-300>]"); + } + const { runExeArtifactTransportOnce } = await import("./artifacts/exe-transport.js"); + const config = { host, workspaceRoot, workspaceId, leaseId, stagingRoot } as const; + const poll = valueAfter("--poll-seconds"); + if (!poll) print(await runExeArtifactTransportOnce(store, config)); + else { + const seconds = positiveInteger(poll, "--poll-seconds", 300); + if (seconds < 5) throw new Error("--poll-seconds must be at least 5"); + while (true) { + const result = await runExeArtifactTransportOnce(store, config); + if (result.receipts.length || result.failures.length) print(result); + await new Promise((resolve) => setTimeout(resolve, seconds * 1_000)); + } + } + } else if (command === "artifact-request" || command === "request-artifact-storage") { + const file = valueAfter("--file"); + const root = valueAfter("--workspace-root"); + const workspaceLeaseId = valueAfter("--workspace-lease-id"); + const requestId = valueAfter("--request-id"); + const artifactId = valueAfter("--artifact-id"); + const kind = valueAfter("--kind"); + const title = valueAfter("--title"); + const summary = valueAfter("--summary"); + const mediaType = valueAfter("--media-type") ?? "text/markdown"; + const visibility = valueAfter("--visibility") ?? "private"; + const provenanceSource = valueAfter("--provenance-source") ?? "agent-memory"; + const provenanceLabel = valueAfter("--provenance-label") ?? "agent-memory skill source"; + const supersedes = valueAfter("--supersedes"); + if (!root || !workspaceLeaseId || !requestId || !file || !artifactId || !kind || !title || !summary) { + throw new Error("Usage: thought artifact-request --workspace-root <absolute-local-lease-root> --workspace-lease-id <id> --file <relative-path> --request-id <id> --artifact-id <id> --kind <skill|spec|report|design|code|document|image> --title <title> --summary <summary> [--media-type <type>] [--visibility <private|sensitive>]"); + } + const { requestArtifactStorage } = await import("./artifacts/append.js"); + const { ARTIFACT_KINDS, ARTIFACT_MEDIA_TYPES, ARTIFACT_VISIBILITY } = await import("./artifacts/types.js"); + if (!(ARTIFACT_KINDS as readonly string[]).includes(kind)) { + throw new Error(`--kind must be one of: ${ARTIFACT_KINDS.join(", ")}`); + } + if (!(ARTIFACT_MEDIA_TYPES as readonly string[]).includes(mediaType)) { + throw new Error(`--media-type must be one of: ${ARTIFACT_MEDIA_TYPES.join(", ")}`); + } + if (!(ARTIFACT_VISIBILITY as readonly string[]).includes(visibility)) { + throw new Error(`--visibility must be one of: ${ARTIFACT_VISIBILITY.join(", ")}`); + } + const result = await requestArtifactStorage({ + store, + workspaceRoot: root, + workspaceLeaseId, + workspaceRootLabel: valueAfter("--workspace-root-label") ?? "explicit workspace lease root", + requestId, + relativeFilePath: file, + artifactId, + artifactVersion: positiveInteger(valueAfter("--artifact-version") ?? "1", "--artifact-version", 1_000_000), + kind: kind as import("./artifacts/types.js").ArtifactKind, + title, + summary, + mediaType: mediaType as import("./artifacts/types.js").ArtifactMediaType, + provenance: { source: provenanceSource, label: provenanceLabel }, + visibility: visibility as import("./artifacts/types.js").ArtifactVisibility, + ...(valueAfter("--requesting-agent") ? { requestingAgentId: valueAfter("--requesting-agent")! } : {}), + ...(valueAfter("--requesting-run") ? { requestingRunId: valueAfter("--requesting-run")! } : {}), + ...(valueAfter("--source-event") ? { sourceEventId: valueAfter("--source-event")! } : {}), + ...(supersedes ? { supersedesArtifactEventId: supersedes } : {}), + }); + print({ artifactRequest: { eventId: result.requestEvent?.id, requestId }, artifact: { eventId: result.event.id, inserted: result.inserted, blobInserted: result.blobInserted, bodySha256: result.bodySha256, byteCount: result.byteCount, artifactId: result.event.payload.artifactId, artifactVersion: result.event.payload.artifactVersion, blobPath: (result.event.payload.blob as { relativePath: string }).relativePath } }); + } else if (command === "artifacts") { + const { buildArtifactCatalog } = await import("./artifacts/catalog.js"); + print(await buildArtifactCatalog(store)); } else if (command === "status") { const events = await store.listEvents(); const runs = await store.listRuns(); const projection = await rebuildRootActivity(store); const review = await projectReviewQueue(store); + const { buildArtifactCatalog } = await import("./artifacts/catalog.js"); + const artifacts = await buildArtifactCatalog(store); print({ database: path.join(projectRoot, ".thoughtstream", "state", "jazz.sqlite"), events: events.length, @@ -610,13 +686,21 @@ try { completedRuns: runs.filter((run) => run.status === "completed").length, failedRuns: runs.filter((run) => run.status === "failed").length, review: review.counts, + artifacts: { total: artifacts.total, active: artifacts.active, superseded: artifacts.superseded }, projection, }); } else if (command === "serve") { const host = valueAfter("--host") ?? "127.0.0.1"; const port = Number(valueAfter("--port") ?? "4317"); const reviewCapability = decodeReviewCapability(process.env.THOUGHTSTREAM_REVIEW_CAPABILITY_B64); - const server = await startInspectorServer(store, { host, port, ...(reviewCapability ? { reviewCapability } : {}) }); + const agentContextRoot = valueAfter("--agent-context-root"); + if (agentContextRoot && !path.isAbsolute(agentContextRoot)) throw new Error("--agent-context-root must be absolute"); + const server = await startInspectorServer(store, { + host, + port, + ...(reviewCapability ? { reviewCapability } : {}), + ...(agentContextRoot ? { agentContextRoot: path.resolve(agentContextRoot) } : {}), + }); process.stdout.write(`thought stream inspector: http://${host}:${port}\n`); await waitForShutdown(server); } else { diff --git a/src/events/registry.ts b/src/events/registry.ts index d94f5b7..52d7426 100644 --- a/src/events/registry.ts +++ b/src/events/registry.ts @@ -20,6 +20,21 @@ import { reviewPromptPayloadSchema, reviewResponseEventPayloadSchema, } from "../review/types.js"; +import { artifactPayloadSchema, artifactRequestPayloadSchema } from "../artifacts/schema.js"; +import { ARTIFACT_EVENT_TYPE, ARTIFACT_REQUEST_EVENT_TYPE, ARTIFACT_REQUEST_SCHEMA_VERSION, ARTIFACT_SCHEMA_VERSION } from "../artifacts/types.js"; +import { + AGENT_PROPOSAL_SCHEMA_VERSION, + CORRECTION_PROPOSAL_EVENT_TYPE, + MEMORY_MATERIALIZATION_FAILED_EVENT_TYPE, + MEMORY_MATERIALIZED_EVENT_TYPE, + MEMORY_PROPOSAL_EVENT_TYPE, + PROPOSAL_DECISION_EVENT_TYPE, + correctionProposalPayloadSchema as agentCorrectionProposalPayloadSchema, + memoryMaterializationFailedPayloadSchema, + memoryMaterializedPayloadSchema, + memoryProposalPayloadSchema, + proposalDecisionPayloadSchema, +} from "../agent-proposals/contracts.js"; import type { ThoughtEvent } from "./types.js"; export interface RegisteredEventType { @@ -372,6 +387,41 @@ export function createDefaultRegistry(): EventRegistry { description: "Inert contract-valid output correction proposal", payload: correctionProposalPayload, }); + registry.register({ + type: MEMORY_PROPOSAL_EVENT_TYPE, + schemaVersion: AGENT_PROPOSAL_SCHEMA_VERSION, + description: "Inert snapshot-bound agent memory-change proposal", + payload: memoryProposalPayloadSchema, + minimumPrivacy: "sensitive", + }); + registry.register({ + type: CORRECTION_PROPOSAL_EVENT_TYPE, + schemaVersion: AGENT_PROPOSAL_SCHEMA_VERSION, + description: "Inert target-bound agent self-correction proposal", + payload: agentCorrectionProposalPayloadSchema, + minimumPrivacy: "sensitive", + }); + registry.register({ + type: PROPOSAL_DECISION_EVENT_TYPE, + schemaVersion: AGENT_PROPOSAL_SCHEMA_VERSION, + description: "Append-only human decision over one exact agent proposal", + payload: proposalDecisionPayloadSchema as unknown as z.ZodType<JsonObject>, + minimumPrivacy: "sensitive", + }); + registry.register({ + type: MEMORY_MATERIALIZED_EVENT_TYPE, + schemaVersion: AGENT_PROPOSAL_SCHEMA_VERSION, + description: "Verified stale-checked materialization of an accepted Stream memory proposal", + payload: memoryMaterializedPayloadSchema, + minimumPrivacy: "sensitive", + }); + registry.register({ + type: MEMORY_MATERIALIZATION_FAILED_EVENT_TYPE, + schemaVersion: AGENT_PROPOSAL_SCHEMA_VERSION, + description: "Content-dark failed materialization of an accepted Stream memory proposal", + payload: memoryMaterializationFailedPayloadSchema, + minimumPrivacy: "sensitive", + }); registry.register({ type: "stream.thought.judgment.training-example", @@ -439,6 +489,22 @@ export function createDefaultRegistry(): EventRegistry { payload: operationalIncidentPayloadSchema as z.ZodType<JsonObject>, }); + registry.register({ + type: ARTIFACT_REQUEST_EVENT_TYPE, + schemaVersion: ARTIFACT_REQUEST_SCHEMA_VERSION, + description: "Append-only private request to materialize one workspace file", + payload: artifactRequestPayloadSchema, + minimumPrivacy: "private", + }); + + registry.register({ + type: ARTIFACT_EVENT_TYPE, + schemaVersion: ARTIFACT_SCHEMA_VERSION, + description: "Immutable private metadata reference to a content-addressed durable artifact", + payload: artifactPayloadSchema, + minimumPrivacy: "private", + }); + for (const type of [ "stream.thought.derived.topics", "stream.thought.derived.entities", diff --git a/src/jazz/store.ts b/src/jazz/store.ts index fddd15c..5d30204 100644 --- a/src/jazz/store.ts +++ b/src/jazz/store.ts @@ -324,22 +324,74 @@ export class JazzThoughtStore { } async listEvents(options: { types?: string[]; source?: string; limit?: number } = {}): Promise<ThoughtEvent[]> { - const where = options.source ? { source: options.source } : {}; - const rows = await this.db.all(thoughtstreamApp.events.where(where)); - const types = options.types ? new Set(options.types) : undefined; + const sourceWhere = options.source ? { source: options.source } : {}; + const types = options.types ? [...new Set(options.types)] : undefined; + const readRows = async (type?: string) => { + return this.db.all(thoughtstreamApp.events.where({ + ...sourceWhere, + ...(type ? { type } : {}), + })); + }; + const rows = types + ? (await Promise.all(types.map((type) => readRows(type)))).flat() + : await readRows(); const events = rows .map(eventFromJazz) - .filter((event) => !types || types.has(event.type)) .sort((left, right) => left.observedAt.localeCompare(right.observedAt) || left.id.localeCompare(right.id)); return options.limit ? events.slice(-options.limit) : events; } + async getEvents(ids: string[]): Promise<ThoughtEvent[]> { + const events = await Promise.all([...new Set(ids)].map((id) => this.getEvent(id))); + return events.filter((event): event is ThoughtEvent => Boolean(event)); + } + + async listChildEvents(parentEventId: string): Promise<ThoughtEvent[]> { + return (await this.db.all(thoughtstreamApp.events.where({ parentEventKey: parentEventId }))) + .map(eventFromJazz) + .sort((left, right) => left.observedAt.localeCompare(right.observedAt) || left.id.localeCompare(right.id)); + } + async listSources(): Promise<SourceState[]> { return (await this.db.all(thoughtstreamApp.sources.where({}))) .map(sourceStateFromJazz) .sort((left, right) => left.id.localeCompare(right.id)); } + async registerSource(input: { + id: string; + kind: string; + enabled: boolean; + config: JsonObject; + updatedAt?: string; + }): Promise<SourceState> { + const existingRow = await this.db.one(thoughtstreamApp.sources.where({ key: input.id }).limit(1)); + const existing = existingRow ? sourceStateFromJazz(existingRow) : undefined; + if (existing && existing.kind !== input.kind) { + throw new Error(`Source kind changed for ${input.id}: ${existing.kind} -> ${input.kind}`); + } + const updatedAt = input.updatedAt ?? new Date().toISOString(); + const source: SourceState = { + id: input.id, + kind: input.kind, + enabled: input.enabled, + config: input.config, + lastSequence: existing?.lastSequence ?? 0, + createdAt: existing?.createdAt ?? updatedAt, + updatedAt, + }; + await this.upsert(thoughtstreamApp.sources, { key: input.id }, { + key: source.id, + kind: source.kind, + enabled: source.enabled, + configJson: canonicalJson(source.config), + lastSequence: source.lastSequence, + createdAt: source.createdAt, + updatedAt: source.updatedAt, + }); + return source; + } + async appendDocumentVersion(version: DocumentVersion): Promise<boolean> { const existing = await this.db.one(thoughtstreamApp.documentVersions.where({ key: version.id }).limit(1)); if (existing) return false; diff --git a/src/projections/activity.ts b/src/projections/activity.ts index 7923169..ce9414d 100644 --- a/src/projections/activity.ts +++ b/src/projections/activity.ts @@ -2,6 +2,7 @@ import type { JsonObject } from "../core/json.js"; import type { JazzThoughtStore } from "../jazz/store.js"; import type { ThoughtEvent } from "../events/types.js"; import type { AgentRun } from "../store/types.js"; +import { createDefaultRegistry } from "../events/registry.js"; export interface ActivityConsumerRun { id: string; @@ -33,6 +34,17 @@ export interface RootActivityProjection { items: ActivityItem[]; } +const ROOT_SOURCE_EVENT_TYPES = [...new Set([ + ...createDefaultRegistry().list().map((entry) => entry.type).filter((type) => type.startsWith("stream.thought.source.")), + "stream.thought.source.x.activity", +])]; +const TERMINAL_RUN_EVENT_TYPES = [ + "stream.thought.agent.run.completed", + "stream.thought.agent.run.failed", + "stream.thought.agent.run.blocked", + "stream.thought.agent.run.abandoned", +]; + export async function buildRootActivity(store: JazzThoughtStore, limit = 100): Promise<RootActivityProjection> { const [events, runs] = await Promise.all([store.listEvents(), store.listRuns()]); const eventsById = new Map(events.map((event) => [event.id, event])); @@ -77,6 +89,73 @@ export async function buildRootActivity(store: JazzThoughtStore, limit = 100): P return { totalEvents: events.length, byType, items }; } +export async function buildRecentRootActivity( + store: JazzThoughtStore, + runs: AgentRun[], + limit = 100, +): Promise<RootActivityProjection> { + const inputIds = [...new Set(runs.flatMap((run) => run.inputEventIds))]; + const [sourceEvents, inputEvents, terminalEvents, sources] = await Promise.all([ + store.listEvents({ types: ROOT_SOURCE_EVENT_TYPES }), + store.getEvents(inputIds), + store.listEvents({ types: TERMINAL_RUN_EVENT_TYPES }), + store.listSources(), + ]); + const roots = sourceEvents.filter((event) => event.id === event.rootEventId); + const inputsById = new Map(inputEvents.map((event) => [event.id, event])); + const terminalByRun = new Map<string, ThoughtEvent[]>(); + for (const event of terminalEvents) { + const runId = typeof event.payload.runId === "string" ? event.payload.runId : undefined; + if (!runId) continue; + terminalByRun.set(runId, [...(terminalByRun.get(runId) ?? []), event]); + } + const descendantsByRoot = new Map<string, Set<string>>(); + const runsByRoot = new Map<string, AgentRun[]>(); + for (const run of runs) { + const inputs = run.inputEventIds.map((id) => inputsById.get(id)).filter((event): event is ThoughtEvent => Boolean(event)); + const rootIds = new Set(inputs.map((event) => event.rootEventId)); + for (const rootId of rootIds) { + runsByRoot.set(rootId, [...(runsByRoot.get(rootId) ?? []), run]); + const descendants = descendantsByRoot.get(rootId) ?? new Set<string>(); + for (const input of inputs) if (input.id !== rootId) descendants.add(input.id); + for (const outputId of run.outputEventIds) descendants.add(outputId); + for (const terminal of terminalByRun.get(run.id) ?? []) descendants.add(terminal.id); + descendantsByRoot.set(rootId, descendants); + } + } + const byType: Record<string, number> = {}; + for (const root of roots) byType[root.type] = (byType[root.type] ?? 0) + 1; + const items = roots.slice(-limit).reverse().map((event) => ({ + id: event.id, + type: event.type, + occurredAt: event.occurredAt, + source: event.source, + actor: event.actor, + rootEventId: event.rootEventId, + ...(event.parentEventId ? { parentEventId: event.parentEventId } : {}), + privacy: event.privacy, + summary: summarizeRoot(event), + descendantEventCount: descendantsByRoot.get(event.id)?.size ?? 0, + consumerRuns: (runsByRoot.get(event.id) ?? []).map((run) => ({ + id: run.id, + agentId: run.agentId, + agentVersion: run.agentVersion, + status: run.status, + kind: runKind(run), + description: describeRunResult( + run, + run.inputEventIds.map((id) => inputsById.get(id)).filter((input): input is ThoughtEvent => Boolean(input)), + ), + outputCount: run.outputEventIds.length, + })), + })); + return { + totalEvents: sources.reduce((total, source) => total + source.lastSequence, 0), + byType, + items, + }; +} + export async function rebuildRootActivity(store: JazzThoughtStore, limit = 100): Promise<RootActivityProjection> { const projection = await buildRootActivity(store, limit); const events = await store.listEvents(); diff --git a/src/projections/effective-output.ts b/src/projections/effective-output.ts index f0c5956..bde0ec3 100644 --- a/src/projections/effective-output.ts +++ b/src/projections/effective-output.ts @@ -13,7 +13,7 @@ import type { JazzThoughtStore } from "../jazz/store.js"; import type { AgentRun, Projection } from "../store/types.js"; import { joinPrivacy, runPrivacy } from "../security/privacy.js"; -export const EFFECTIVE_OUTPUT_PROJECTION_VERSION = 1; +export const EFFECTIVE_OUTPUT_PROJECTION_VERSION = 2; export interface ActiveJudgmentSet { active: ThoughtEvent[]; @@ -76,6 +76,28 @@ export async function rebuildEffectiveOutput( const registry = createOutputContractRegistry(); registry.resolve(contract); const judgmentSet = await activeJudgments(store); + const originalOutput = await validatedOriginalOutput(store, originalRun, contract); + const directCorrections: Array<{ judgment: ThoughtEvent; output: JsonObject }> = []; + if (originalOutput) { + for (const judgment of judgmentSet.active.filter((event) => ( + event.payload.runId === originalRun.id + && event.payload.outputEventId === originalOutput.event.id + && event.payload.kind === "correct" + ))) { + const candidate = objectField(judgment.payload.replacementOutput); + if (!candidate) continue; + try { + directCorrections.push({ + judgment, + output: canonicalStructuredOutput(registry, contract, candidate), + }); + } catch { + // Corrupt historical direct corrections remain inert. + } + } + } + directCorrections.sort((left, right) => compareEvents(left.judgment, right.judgment)); + const selectedDirectCorrection = directCorrections.at(-1); const proposals = (await store.listEvents({ types: [CORRECTION_PROPOSAL_EVENT_TYPE] })) .filter((event) => event.payload.originalRunId === originalRunId); const acceptedRepairs: Array<{ @@ -117,7 +139,29 @@ export async function rebuildEffectiveOutput( let payload: JsonObject; let lastEventId: string; - if (selectedRepair) { + if (selectedDirectCorrection && originalOutput) { + payload = { + originalRunId, + status: "corrected", + sourceRootEventId: originalSource.rootEventId, + originalModel: executionProvenance(originalRun), + privacy: joinPrivacy( + originalSource.privacy, + runPrivacy(originalRun), + originalOutput.event.privacy, + selectedDirectCorrection.judgment.privacy, + ), + outputContract: outputContractIdentityJson(contract), + structuredOutput: selectedDirectCorrection.output, + outputEventId: originalOutput.event.id, + judgmentEventId: selectedDirectCorrection.judgment.id, + ...(optionalString(selectedDirectCorrection.judgment.payload.feedbackSourceEventId) ? { + feedbackSourceEventId: optionalString(selectedDirectCorrection.judgment.payload.feedbackSourceEventId)!, + } : {}), + authority: "correct", + }; + lastEventId = selectedDirectCorrection.judgment.id; + } else if (selectedRepair) { payload = { originalRunId, status: "repair", @@ -141,7 +185,6 @@ export async function rebuildEffectiveOutput( }; lastEventId = selectedRepair.judgment.id; } else { - const originalOutput = await validatedOriginalOutput(store, originalRun, contract); if (originalOutput) { payload = { originalRunId, diff --git a/src/projections/source-health.ts b/src/projections/source-health.ts index f649520..d466635 100644 --- a/src/projections/source-health.ts +++ b/src/projections/source-health.ts @@ -1,3 +1,4 @@ +import { z } from "zod"; import type { JsonObject } from "../core/json.js"; import type { JazzThoughtStore } from "../jazz/store.js"; @@ -5,7 +6,30 @@ export type SourceHealthStatus = "healthy" | "failing" | "processing" | "unknown export interface SourceHealth { source: string; + kind?: string; + configured: boolean; + enabled?: boolean; + control?: { + lane: string; + runtime?: { + state: "ready"; + readyAt: string; + revision: string; + current: boolean; + }; + upstream?: { + registered: boolean; + valid: boolean; + webhookId?: string; + desiredSubscriptionCount: number; + liveSubscriptionCount: number; + subscriptionsConverged: boolean; + checkedAt: string; + current: boolean; + }; + }; status: SourceHealthStatus; + hasActivityEvidence: boolean; cursorId?: string; cursor: JsonObject; lastSuccessAt?: string; @@ -28,16 +52,45 @@ const CONNECTOR_EVENT_TYPES = [ "stream.thought.connector.recovered", ]; -export async function buildSourceHealth(store: JazzThoughtStore): Promise<SourceHealth[]> { - const [cursors, events] = await Promise.all([ +const CONTROL_LEASE_MS = 5 * 60 * 1_000; +const sourceControlSchema = z.object({ + kind: z.string().min(1), + enabled: z.boolean(), + lane: z.string().min(1), + runtime: z.object({ + state: z.literal("ready"), + readyAt: z.iso.datetime(), + revision: z.string().min(1), + }).strict().optional(), + upstream: z.object({ + registered: z.boolean(), + valid: z.boolean(), + webhookId: z.string().optional(), + desiredSubscriptionCount: z.number().int().min(0), + liveSubscriptionCount: z.number().int().min(0), + subscriptionsConverged: z.boolean(), + checkedAt: z.iso.datetime(), + }).strict().optional(), +}).passthrough(); + +export async function buildSourceHealth(store: JazzThoughtStore, nowMs = Date.now()): Promise<SourceHealth[]> { + const [registrations, cursors, events] = await Promise.all([ + store.listSources(), store.listSourceCursors(), store.listEvents({ types: CONNECTOR_EVENT_TYPES }), ]); + const registrationsBySource = new Map(registrations.map((source) => [source.id, source])); const cursorsBySource = new Map(cursors.map((cursor) => [cursor.source, cursor])); - const sources = new Set(cursors.map((cursor) => cursor.source)); + const sources = new Set(registrations.map((source) => source.id)); + for (const cursor of cursors) sources.add(cursor.source); for (const event of events) sources.add(event.source); return [...sources].sort().map((source) => { + const registration = registrationsBySource.get(source); + const parsedControl = registration?.config.control + ? sourceControlSchema.safeParse(registration.config.control) + : undefined; + const control = parsedControl?.success ? parsedControl.data : undefined; const cursor = cursorsBySource.get(source); const sourceEvents = events.filter((event) => event.source === source); const operationsStarted = count(sourceEvents, "stream.thought.connector.poll.started") @@ -47,9 +100,36 @@ export async function buildSourceHealth(store: JazzThoughtStore): Promise<Source const operationFailures = count(sourceEvents, "stream.thought.connector.failed"); const recoveries = count(sourceEvents, "stream.thought.connector.recovered"); const inFlight = Math.max(0, operationsStarted - operationsCompleted - operationFailures); + const hasActivityEvidence = Boolean(cursor || sourceEvents.length > 0 || (registration?.lastSequence ?? 0) > 0); return { source, + ...(registration ? { kind: registration.kind, enabled: registration.enabled } : {}), + configured: Boolean(control), + ...(control ? { + control: { + lane: control.lane, + ...(control.runtime ? { + runtime: { + ...control.runtime, + current: withinLease(control.runtime.readyAt, nowMs), + }, + } : {}), + ...(control.upstream ? { + upstream: { + registered: control.upstream.registered, + valid: control.upstream.valid, + ...(control.upstream.webhookId ? { webhookId: control.upstream.webhookId } : {}), + desiredSubscriptionCount: control.upstream.desiredSubscriptionCount, + liveSubscriptionCount: control.upstream.liveSubscriptionCount, + subscriptionsConverged: control.upstream.subscriptionsConverged, + checkedAt: control.upstream.checkedAt, + current: withinLease(control.upstream.checkedAt, nowMs), + }, + } : {}), + }, + } : {}), status: statusFor(cursor?.lastSuccessAt, cursor?.lastFailureAt, inFlight), + hasActivityEvidence, ...(cursor ? { cursorId: cursor.id, cursor: cursor.cursor, @@ -67,6 +147,11 @@ export async function buildSourceHealth(store: JazzThoughtStore): Promise<Source }); } +function withinLease(timestamp: string, nowMs: number): boolean { + const observedAt = Date.parse(timestamp); + return Number.isFinite(observedAt) && observedAt <= nowMs && nowMs - observedAt <= CONTROL_LEASE_MS; +} + function count(events: Array<{ type: string }>, type: string): number { return events.filter((event) => event.type === type).length; } diff --git a/src/web/authenticated-proxy.ts b/src/web/authenticated-proxy.ts index b580d92..6f74525 100644 --- a/src/web/authenticated-proxy.ts +++ b/src/web/authenticated-proxy.ts @@ -266,7 +266,7 @@ async function handleRequest(options: { return notFound(options.response); } - const reviewWriteRoute = url.search === "" && /^\/inspector\/api\/reviews\/[^/]+\/decisions$/.test(url.pathname); + const reviewWriteRoute = url.search === "" && /^\/inspector\/api\/(?:reviews|proposals)\/[^/]+\/decisions$/.test(url.pathname); const basicAuthorized = options.basicFallbackEnabled && isAuthorized(options.request.headers.authorization, options.expectedAuthorizationDigest); const oauthAuthorized = (reviewWriteRoute || !basicAuthorized) && options.oauth diff --git a/src/web/inspector.ts b/src/web/inspector.ts index 08eef8e..95c019c 100644 --- a/src/web/inspector.ts +++ b/src/web/inspector.ts @@ -1,6 +1,6 @@ import http, { type IncomingMessage, type ServerResponse } from "node:http"; import type { JazzThoughtStore } from "../jazz/store.js"; -import { buildRootActivity, describeRunResult } from "../projections/activity.js"; +import { buildRecentRootActivity, describeRunResult } from "../projections/activity.js"; import { buildSourceHealth } from "../projections/source-health.js"; import { auditRunEvidence } from "../agents/evidence.js"; import type { AgentRun } from "../store/types.js"; @@ -11,12 +11,37 @@ import { recordBrowserReviewDecision, } from "../review/review.js"; import { ReviewCapabilityVerifier } from "../review/web-capability.js"; +import { getArtifactBody, getArtifactCatalog, getArtifactContent } from "../artifacts/catalog.js"; +import { + CORRECTION_PROPOSAL_EVENT_TYPE, + MEMORY_MATERIALIZATION_FAILED_EVENT_TYPE, + MEMORY_MATERIALIZED_EVENT_TYPE, + MEMORY_PROPOSAL_EVENT_TYPE, + PROPOSAL_DECISION_EVENT_TYPE, + correctionProposalPayloadSchema, + memoryMaterializationFailedPayloadSchema, + memoryMaterializedPayloadSchema, + memoryProposalPayloadSchema, + proposalDecisionPayloadSchema, +} from "../agent-proposals/contracts.js"; +import { materializeMemoryDecision } from "../agent-proposals/memory-materializer.js"; +import { recordProposalDecision } from "../agent-proposals/review.js"; +import { z } from "zod"; + +const RUN_TERMINAL_EVENT_TYPES = [ + "stream.thought.agent.run.completed", + "stream.thought.agent.run.failed", + "stream.thought.agent.run.blocked", + "stream.thought.agent.run.abandoned", +]; export interface InspectorServerOptions { host?: string; port?: number; reviewCapability?: Buffer | undefined; reviewVerifier?: ReviewCapabilityVerifier | undefined; + agentContextRoot?: string | undefined; + proposalActor?: string | undefined; } export async function startInspectorServer( @@ -28,8 +53,9 @@ export async function startInspectorServer( const port = options.port ?? 4317; const reviewVerifier = options.reviewVerifier ?? (options.reviewCapability ? new ReviewCapabilityVerifier(options.reviewCapability) : undefined); + const proposalWritesEnabled = Boolean(reviewVerifier && options.agentContextRoot); const server = http.createServer((request, response) => { - void handleRequest(store, request, response, reviewVerifier).catch((error) => { + void handleRequest(store, request, response, reviewVerifier, proposalWritesEnabled ? options.agentContextRoot : undefined, options.proposalActor ?? "operator:cameron").catch((error) => { sendJson(response, 500, { error: error instanceof Error ? error.message : String(error) }); }); }); @@ -48,8 +74,52 @@ async function handleRequest( request: IncomingMessage, response: ServerResponse, reviewVerifier?: ReviewCapabilityVerifier, + agentContextRoot?: string, + proposalActor = "operator:cameron", ): Promise<void> { const url = new URL(request.url ?? "/", "http://127.0.0.1"); + const proposalDecisionMatch = url.pathname.match(/^\/api\/proposals\/([^/]+)\/decisions$/); + if (request.method === "POST" && proposalDecisionMatch) { + if (!reviewVerifier || !agentContextRoot) { + request.resume(); + sendJson(response, 405, { error: "Proposal review is read-only" }); + return; + } + let body: Buffer; + try { + body = await readBody(request, 34_816); + } catch { + request.resume(); + sendJson(response, 400, { error: "Proposal decision is invalid" }); + return; + } + if (!reviewVerifier.verify(request.headers, "POST", url.pathname, body)) { + sendJson(response, 403, { error: "Proposal decision could not be verified" }); + return; + } + try { + const eventId = decodeURIComponent(proposalDecisionMatch[1]!); + const input = proposalDecisionRequestSchema.parse(JSON.parse(body.toString("utf8"))); + const result = await recordProposalDecision(store, { proposalEventId: eventId, ...input, actor: proposalActor }); + const materialization = result.proposal.type === MEMORY_PROPOSAL_EVENT_TYPE && input.disposition !== "reject" + ? await materializeMemoryDecision(store, result.decision.id, { contextRoot: agentContextRoot }) + : undefined; + sendJson(response, 201, { + decisionEventId: result.decision.id, + disposition: input.disposition, + ...(result.projectedJudgment ? { judgmentEventId: result.projectedJudgment.id, receipt: "Correction accepted as a private quality judgment" } : {}), + ...(materialization ? { materializationEventId: materialization.event.id, materializationStatus: materialization.status, receipt: materialization.status === "materialized" ? "Stream memory updated" : "Stream memory was not changed" } : {}), + }); + } catch (error) { + const message = error instanceof Error ? error.message : "Proposal decision is invalid"; + if (message.includes("already has a human decision") || message.includes("submission id conflicts")) { + sendJson(response, 409, { error: "Proposal decision changed; reload before submitting" }); + } else { + sendJson(response, 400, { error: "Proposal decision is invalid" }); + } + } + return; + } const reviewDecisionMatch = url.pathname.match(/^\/api\/reviews\/([^/]+)\/decisions$/); if (request.method === "POST" && reviewDecisionMatch) { if (!reviewVerifier) { @@ -91,14 +161,16 @@ async function handleRequest( return; } if (url.pathname === "/api/snapshot") { - const [activity, runs, agents, events, sources] = await Promise.all([ - buildRootActivity(store), + const [runs, agents, sources] = await Promise.all([ store.listRuns(), store.listAgents(), - store.listEvents(), buildSourceHealth(store), ]); - const runEvidence = auditRunEvidence(runs, events); + const [activity, evidenceEvents] = await Promise.all([ + buildRecentRootActivity(store, runs), + loadRunEvidenceEvents(store, runs), + ]); + const runEvidence = auditRunEvidence(runs, evidenceEvents); const adapterSelections = agents .map((agent) => ({ id: agent.id, @@ -113,7 +185,7 @@ async function handleRequest( ])).values()]; sendJson(response, 200, { activity, - runs: runs.slice().reverse(), + runs: runs.slice().reverse().map(inspectorRunSummary), agents, sources, adapterInventory: { catalogs: adapterCatalogs, selections: adapterSelections }, @@ -122,6 +194,10 @@ async function handleRequest( }); return; } + if (url.pathname === "/api/proposals") { + sendJson(response, 200, await projectProposalQueue(store)); + return; + } if (url.pathname === "/api/reviews") { sendJson(response, 200, await projectReviewQueue(store)); return; @@ -130,18 +206,56 @@ async function handleRequest( sendJson(response, 200, { reviewWriteEnabled: false }); return; } + if (url.pathname === "/api/artifacts") { + sendJson(response, 200, await getArtifactCatalog(store)); + return; + } + const artifactContentMatch = url.pathname.match(/^\/api\/artifacts\/([^/]+)\/content$/); + if (artifactContentMatch) { + const id = decodeURIComponent(artifactContentMatch[1]!); + const detail = await getArtifactContent(store, id); + if (!detail) return sendJson(response, 404, { error: `Artifact not found: ${id}` }); + if (!detail.payload.mediaType.startsWith("image/")) return sendJson(response, 404, { error: "Artifact has no binary inspection route" }); + response.writeHead(200, { "content-type": detail.payload.mediaType, "content-length": detail.bytes.byteLength, "x-content-type-options": "nosniff", "cache-control": "private, no-store" }); + response.end(detail.bytes); + return; + } + const artifactMatch = url.pathname.match(/^\/api\/artifacts\/([^/]+)$/); + if (artifactMatch) { + const id = decodeURIComponent(artifactMatch[1]!); + const detail = await getArtifactBody(store, id); + if (!detail) return sendJson(response, 404, { error: `Artifact not found: ${id}` }); + const catalog = await getArtifactCatalog(store); + const entry = catalog.entries.find((candidate) => candidate.eventId === id); + sendJson(response, 200, { + event: detail.event, + ...(detail.text !== undefined ? { text: detail.text } : {}), + ...(detail.text !== undefined && detail.mediaType === "text/markdown" ? { renderedHtml: renderArtifactMarkdown(detail.text) } : {}), + ...(detail.contentPath ? { contentPath: detail.contentPath } : {}), + bodySha256: detail.bodySha256, + byteCount: detail.byteCount, + mediaType: detail.mediaType, + catalogEntry: entry, + }); + return; + } const eventMatch = url.pathname.match(/^\/api\/events\/([^/]+)$/); if (eventMatch) { const id = decodeURIComponent(eventMatch[1]!); const event = await store.getEvent(id); if (!event) return sendJson(response, 404, { error: `Event not found: ${id}` }); - const [events, runs] = await Promise.all([store.listEvents(), store.listRuns()]); + const [children, runs, parent, root] = await Promise.all([ + store.listChildEvents(event.id), + store.listRuns(), + event.parentEventId ? store.getEvent(event.parentEventId) : undefined, + event.rootEventId === event.id ? event : store.getEvent(event.rootEventId), + ]); const relevantRuns = runs.filter((run) => eventBelongsToRun(event, run)); sendJson(response, 200, { event, - parent: event.parentEventId ? await store.getEvent(event.parentEventId) : undefined, - children: events.filter((candidate) => candidate.parentEventId === event.id), - root: event.rootEventId === event.id ? event : await store.getEvent(event.rootEventId), + parent, + children, + root, agentActivities: await Promise.all(relevantRuns.map((run) => expandRun(store, run))), }); return; @@ -151,13 +265,133 @@ async function handleRequest( const id = decodeURIComponent(runMatch[1]!); const run = await store.getRun(id); if (!run) return sendJson(response, 404, { error: `Run not found: ${id}` }); - const evidence = auditRunEvidence([run], await store.listEvents())[0]; - sendJson(response, 200, { ...(await expandRun(store, run)), evidence }); + const [activity, evidenceEvents] = await Promise.all([ + expandRun(store, run), + loadRunEvidenceEvents(store, [run]), + ]); + const evidence = auditRunEvidence([run], evidenceEvents)[0]; + sendJson(response, 200, { ...activity, evidence }); return; } sendJson(response, 404, { error: "Not found" }); } +const proposalDecisionRequestSchema = z.object({ + disposition: z.enum(["accept", "edit", "reject"]), + replacementText: z.string().min(1).max(32_768).optional(), + submissionId: z.string().min(1).max(500), +}).strict().superRefine((value, context) => { + if ((value.disposition === "edit") !== (value.replacementText !== undefined)) { + context.addIssue({ code: "custom", path: ["replacementText"], message: "Only edit decisions require replacement text" }); + } +}); + +export interface ProposalQueueItem { + eventId: string; + kind: "Memory suggestion" | "Proposed correction"; + proposedText: string; + reason: string; + operation: string; + originalText?: string | undefined; + observedAt: string; + status: "Awaiting review" | "Accepted" | "Edited and accepted" | "Rejected" | "Materialized" | "Failed"; + decisionEventId?: string | undefined; + receiptEventId?: string | undefined; + failureReason?: string | undefined; + technical: Record<string, string | number>; +} + +export async function projectProposalQueue(store: JazzThoughtStore): Promise<{ count: number; items: ProposalQueueItem[] }> { + const [proposals, decisions, receipts] = await Promise.all([ + store.listEvents({ types: [MEMORY_PROPOSAL_EVENT_TYPE, CORRECTION_PROPOSAL_EVENT_TYPE] }), + store.listEvents({ types: [PROPOSAL_DECISION_EVENT_TYPE] }), + store.listEvents({ types: [MEMORY_MATERIALIZED_EVENT_TYPE, MEMORY_MATERIALIZATION_FAILED_EVENT_TYPE] }), + ]); + const items: ProposalQueueItem[] = []; + for (const proposal of proposals) { + const decision = decisions.find((event) => event.payload.proposalEventId === proposal.id); + const receipt = decision && receipts.find((event) => event.payload.decisionEventId === decision.id); + const decisionPayload = decision ? proposalDecisionPayloadSchema.parse(decision.payload) : undefined; + let status: ProposalQueueItem["status"] = "Awaiting review"; + if (receipt?.type === MEMORY_MATERIALIZED_EVENT_TYPE) status = "Materialized"; + else if (receipt?.type === MEMORY_MATERIALIZATION_FAILED_EVENT_TYPE) status = "Failed"; + else if (decisionPayload?.disposition === "reject") status = "Rejected"; + else if (decisionPayload?.disposition === "edit") status = "Edited and accepted"; + else if (decisionPayload?.disposition === "accept") status = "Accepted"; + if (proposal.type === MEMORY_PROPOSAL_EVENT_TYPE) { + const payload = memoryProposalPayloadSchema.parse(proposal.payload); + const failed = receipt?.type === MEMORY_MATERIALIZATION_FAILED_EVENT_TYPE ? memoryMaterializationFailedPayloadSchema.parse(receipt.payload) : undefined; + if (receipt?.type === MEMORY_MATERIALIZED_EVENT_TYPE) memoryMaterializedPayloadSchema.parse(receipt.payload); + items.push({ + eventId: proposal.id, + kind: "Memory suggestion", + proposedText: decisionPayload?.replacementText ?? payload.proposedText, + reason: payload.reason, + operation: payload.operation === "append" ? "Append to Stream memory" : "Replace Stream memory", + observedAt: proposal.observedAt, + status, + ...(decision ? { decisionEventId: decision.id } : {}), + ...(receipt ? { receiptEventId: receipt.id } : {}), + ...(failed ? { failureReason: humanMaterializationFailure(failed.reasonCode) } : {}), + technical: { + proposalEventId: proposal.id, + agentId: payload.proposer.agentId, + agentVersion: payload.proposer.agentVersion, + runId: payload.proposer.runId, + contextSnapshotId: payload.proposer.contextSnapshotId, + targetDocumentId: payload.target.documentId, + targetVersionId: payload.target.versionId, + targetSha256: payload.target.sha256, + proposedTextSha256: payload.proposedTextSha256, + }, + }); + continue; + } + const payload = correctionProposalPayloadSchema.parse(proposal.payload); + const target = await store.getEvent(payload.target.outputEventId); + const originalText = typeof target?.payload.structuredOutput === "object" && target.payload.structuredOutput && !Array.isArray(target.payload.structuredOutput) + && typeof (target.payload.structuredOutput as Record<string, unknown>).summary === "string" + ? String((target.payload.structuredOutput as Record<string, unknown>).summary) + : typeof target?.payload.summary === "string" ? target.payload.summary : "Original delivered reply unavailable"; + items.push({ + eventId: proposal.id, + kind: "Proposed correction", + proposedText: decisionPayload?.replacementText ?? payload.replacementText, + reason: payload.reason, + operation: "Correct the delivered reply", + originalText, + observedAt: proposal.observedAt, + status, + ...(decision ? { decisionEventId: decision.id } : {}), + technical: { + proposalEventId: proposal.id, + agentId: payload.proposer.agentId, + agentVersion: payload.proposer.agentVersion, + runId: payload.proposer.runId, + targetRunId: payload.target.runId, + targetOutputEventId: payload.target.outputEventId, + deliveryReceiptEventId: payload.target.deliveryReceiptEventId, + outputContractId: payload.target.outputContract.id, + outputContractVersion: payload.target.outputContract.version, + outputContractSha256: payload.target.outputContract.sha256, + replacementTextSha256: payload.replacementTextSha256, + }, + }); + } + items.sort((left, right) => right.observedAt.localeCompare(left.observedAt) || right.eventId.localeCompare(left.eventId)); + return { count: items.length, items }; +} + +function humanMaterializationFailure(code: string): string { + return ({ + "stale-base": "Stream memory changed after this suggestion was created", + "symlink-refused": "The memory target did not pass the safe-file check", + "frontmatter-invalid": "The approved text did not preserve Stream memory identity", + "document-identity-changed": "The approved text changed Stream memory identity", + "write-failed": "The trusted memory update could not be completed", + } as Record<string, string>)[code] ?? "The trusted memory update failed its safety checks"; +} + function adapterSelectionFromSpec(spec: Record<string, unknown>): { digest: string; generation: number; release: Record<string, unknown> } | undefined { const digest = spec.adapterCatalogDigest; const generation = spec.adapterCatalogGeneration; @@ -167,6 +401,100 @@ function adapterSelectionFromSpec(spec: Record<string, unknown>): { digest: stri return { digest, generation: Number(generation), release: release as Record<string, unknown> }; } +export function renderArtifactMarkdown(markdown: string): string { + let text = markdown.replace(/\r\n?/g, "\n"); + if (text.startsWith("---\n")) { + const frontmatterEnd = text.indexOf("\n---\n", 4); + if (frontmatterEnd >= 0) text = text.slice(frontmatterEnd + 5); + } + const lines = text.split("\n"); + let html = ""; + let paragraph: string[] = []; + let list: "ul" | "ol" | undefined; + let code = false; + let codeLines: string[] = []; + const flushParagraph = (): void => { + if (paragraph.length === 0) return; + html += `<p>${renderInlineMarkdown(paragraph.join(" "))}</p>`; + paragraph = []; + }; + const closeList = (): void => { + if (!list) return; + html += `</${list}>`; + list = undefined; + }; + const flushCode = (): void => { + html += `<pre><code>${escapeArtifactHtml(codeLines.join("\n"))}</code></pre>`; + codeLines = []; + }; + for (const line of lines) { + if (line.startsWith("```")) { + flushParagraph(); + closeList(); + if (code) flushCode(); + code = !code; + continue; + } + if (code) { + codeLines.push(line); + continue; + } + const heading = line.match(/^(#{1,6})\s+(.+)$/); + const unordered = line.match(/^\s*[-*+]\s+(.+)$/); + const ordered = line.match(/^\s*\d+[.)]\s+(.+)$/); + const quote = line.match(/^>\s?(.*)$/); + if (heading) { + flushParagraph(); + closeList(); + const level = heading[1]!.length; + html += `<h${level}>${renderInlineMarkdown(heading[2]!)}</h${level}>`; + continue; + } + if (unordered || ordered) { + flushParagraph(); + const next = unordered ? "ul" : "ol"; + if (list !== next) { + closeList(); + list = next; + html += `<${list}>`; + } + html += `<li>${renderInlineMarkdown((unordered ?? ordered)![1]!)}</li>`; + continue; + } + if (quote) { + flushParagraph(); + closeList(); + html += `<blockquote>${renderInlineMarkdown(quote[1]!)}</blockquote>`; + continue; + } + if (!line.trim()) { + flushParagraph(); + closeList(); + continue; + } + paragraph.push(line.trim()); + } + flushParagraph(); + closeList(); + if (code) flushCode(); + return html; +} + +function renderInlineMarkdown(value: string): string { + return escapeArtifactHtml(value) + .replace(/`([^`]+)`/g, "<code>$1</code>") + .replace(/\*\*\*([^*]+)\*\*\*/g, "<strong><em>$1</em></strong>") + .replace(/\*\*([^*]+)\*\*/g, "<strong>$1</strong>") + .replace(/\*([^*]+)\*/g, "<em>$1</em>") + .replace(/\[([^\]]+)\]\((https?:\/\/[^\s)]+)\)/g, '<a href="$2" rel="noreferrer">$1</a>'); +} + +function escapeArtifactHtml(value: string): string { + return value.replace(/[&<>"']/g, (character) => ({ + "&": "&", "<": "<", ">": ">", '"': """, "'": "'", + })[character]!); +} + export function renderInspectorHtml(): string { return `<!doctype html> <html lang="en"> @@ -200,46 +528,102 @@ export function renderInspectorHtml(): string { select,textarea,input[type=text] { width:100%; background:#0d1216; color:var(--text); border:1px solid var(--line); padding:8px; font:12px/1.45 inherit } textarea { min-height:86px; resize:vertical } .choice-row { display:flex; flex-wrap:wrap; gap:8px 14px } .choice-row label { color:var(--text); text-transform:none; letter-spacing:0 } .submit-row { display:flex; align-items:center; gap:10px } .submit-row button { border:1px solid var(--cyan); background:#102226; color:var(--text); padding:8px 12px; font:inherit; cursor:pointer } .submit-row button:disabled { opacity:.5; cursor:not-allowed } .provenance { margin-top:10px; color:var(--muted); font-size:10px; overflow-wrap:anywhere } + .human-grid { display:grid; grid-template-columns:repeat(2,minmax(0,1fr)); gap:8px; margin:8px 0 14px } + .human-card { min-width:0; border:1px solid var(--line); background:#0d1216; padding:10px } + .human-card.wide { grid-column:1/-1 } + .field-label { color:var(--muted); font-size:10px; text-transform:uppercase; letter-spacing:.08em; margin-bottom:4px } + .field-value { color:var(--text); overflow-wrap:anywhere } + .mono { font:11px/1.45 inherit; color:var(--cyan); overflow-wrap:anywhere } + .relation-list { display:grid; gap:8px; margin:8px 0 14px } + .relation-card { border-left:2px solid var(--cyan); background:#0d1216; padding:10px } + .artifact-content { max-height:500px; overflow:auto } + .artifact-back { display:none; border:0; background:transparent; color:var(--cyan); padding:0 0 14px; font:inherit; cursor:pointer } + .artifact-document { color:#edf2f4; font:15px/1.65 system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif } + .artifact-document h1 { font-size:24px; line-height:1.2; margin:0 0 18px; letter-spacing:0 } + .artifact-document h2 { font-size:19px; line-height:1.25; margin:28px 0 10px } + .artifact-document h3 { color:var(--text); text-transform:none; letter-spacing:0; font-size:16px; margin:24px 0 8px } + .artifact-document h4,.artifact-document h5,.artifact-document h6 { margin:20px 0 7px } + .artifact-document p { margin:0 0 13px } + .artifact-document ul,.artifact-document ol { margin:0 0 15px; padding-left:24px } + .artifact-document li { margin:4px 0 } + .artifact-document blockquote { margin:14px 0; padding:2px 0 2px 14px; border-left:2px solid var(--cyan); color:#bac6cc } + .artifact-document code { padding:1px 4px; background:#121a1f; color:#9ae5e8; font:12px/1.5 ui-monospace,SFMono-Regular,Menlo,monospace } + .artifact-document pre { margin:14px 0; max-height:none; font:12px/1.55 ui-monospace,SFMono-Regular,Menlo,monospace } + .artifact-document pre code { padding:0; background:transparent; color:inherit } + .artifact-document a { color:var(--cyan); text-underline-offset:2px } + .artifact-metadata { margin-top:30px } + .artifact-metadata > summary { padding:10px 0 } details { border-top:1px solid var(--line); margin-top:12px; padding-top:9px } summary { color:var(--muted); cursor:pointer; text-transform:uppercase; letter-spacing:.08em; font-size:11px } details pre { margin-top:7px } nav { display:flex; gap:8px } nav button { background:transparent; color:var(--muted); border:0; border-bottom:1px solid transparent; font:inherit; padding:0 0 3px; cursor:pointer } nav button.active { color:var(--text); border-color:var(--cyan) } - @media(max-width:800px){ main{grid-template-columns:1fr} section{border-right:0} #detail{position:static;max-height:none} .candidate-grid{grid-template-columns:1fr} } + .suggestion-content { border:1px solid var(--line); background:#0d1216; padding:14px; margin:8px 0 14px; white-space:pre-wrap; overflow-wrap:anywhere; font:14px/1.6 system-ui,sans-serif } + .suggestion-compare { display:grid; grid-template-columns:1fr 1fr; gap:10px; margin:8px 0 14px } .suggestion-compare > div { min-width:0 } + .suggestion-actions { display:grid; gap:10px; border-top:1px solid var(--line); padding-top:14px; margin-top:16px } .suggestion-buttons { display:flex; flex-wrap:wrap; gap:8px } .suggestion-buttons button { border:1px solid var(--cyan); background:#102226; color:var(--text); padding:9px 12px; font:inherit; cursor:pointer } .suggestion-buttons button.reject { border-color:var(--red) } + .spinner { display:inline-block; width:10px; height:10px; margin-right:7px; border:1px solid var(--line); border-top-color:var(--cyan); border-radius:50%; animation:spin .8s linear infinite; vertical-align:-1px } + .retry { margin-top:12px; border:1px solid var(--red); background:transparent; color:var(--text); padding:7px 10px; font:inherit; cursor:pointer } + @keyframes spin { to { transform:rotate(360deg) } } + @media (prefers-reduced-motion:reduce) { .spinner { animation:none; border-color:var(--cyan) } } + @media(max-width:800px){ header{align-items:flex-start;flex-wrap:wrap;gap:10px} nav{width:100%;overflow-x:auto} #status{margin-left:0} main{display:block;min-height:0} section{border-right:0} #detail{position:static;max-height:none} .candidate-grid,.human-grid,.suggestion-compare{grid-template-columns:1fr} .human-card.wide{grid-column:auto} main.artifact-selected #list-pane{display:none} main.artifact-selected #detail-pane{display:block} main.artifact-selected .artifact-back{display:block} } </style> </head> <body> -<header><h1>thought stream</h1><nav><button id="events-tab" class="active">events</button><button id="runs-tab">runs</button><button id="sources-tab">sources</button><button id="reviews-tab">review</button></nav><span id="status" class="muted">loading</span></header> -<main><section><div class="section-head"><span id="list-title">root activity</span><span id="count"></span></div><div id="list"></div></section><section><div id="detail"><div class="empty">Select an observation to see what processed it and what, if anything, happened.</div></div></section></main> +<header><h1>thought stream</h1><nav><button id="events-tab" class="active">events</button><button id="runs-tab">runs</button><button id="sources-tab">sources</button><button id="suggestions-tab">suggestions</button><button id="reviews-tab">review</button><button id="artifacts-tab">artifacts</button></nav><span id="status" class="muted">loading</span></header> +<main><section id="list-pane"><div class="section-head"><span id="list-title">root activity</span><span id="count">…</span></div><div id="list"><div class="empty"><span class="spinner" aria-hidden="true"></span>Loading root activity…</div></div></section><section id="detail-pane"><div id="detail"><div class="empty">Select an observation to see what processed it and what, if anything, happened.</div></div></section></main> <script> -const state={snapshot:null,reviews:{items:[],counts:{}},session:{reviewWriteEnabled:false},tab:'events',selected:null}; +const state={snapshot:null,proposals:null,reviews:null,artifacts:null,session:null,loading:{snapshot:true,proposals:true,reviews:true,artifacts:true,session:true},errors:{},tab:'events',selected:null,proposalReceipt:''}; +const resources={snapshot:{path:'api/snapshot',label:'root activity'},proposals:{path:'api/proposals',label:'suggestions'},reviews:{path:'api/reviews',label:'review queue'},artifacts:{path:'api/artifacts',label:'artifacts'},session:{path:'api/session',label:'session'}}; const esc=s=>String(s??'').replace(/[&<>"']/g,c=>({'&':'&','<':'<','>':'>','"':'"',"'":'''}[c])); const json=value=>esc(JSON.stringify(value,null,2)); const agentName=id=>state.snapshot?.agents?.find(agent=>agent.id===id)?.spec?.name??id; +async function fetchJson(path,label,options){const response=await fetch(path,options);const contentType=(response.headers.get('content-type')??'').split(';',1)[0].trim().toLowerCase();if(!response.ok)throw new Error(label+' failed (HTTP '+response.status+')');if(contentType!=='application/json')throw new Error(label+' returned '+(contentType||'an unknown content type')+' instead of JSON');return response.json()} +async function loadResource(key){const resource=resources[key];state[key]=null;state.loading[key]=true;delete state.errors[key];updateStatus();if(tabResource()===key)renderList();try{state[key]=await fetchJson(resource.path,resource.label)}catch(error){state.errors[key]=error instanceof Error?error.message:String(error)}finally{state.loading[key]=false;updateStatus();if(tabResource()===key)renderList()}} async function load(){ - const [snapshot,reviews,session]=await Promise.all([fetch('api/snapshot'),fetch('api/reviews'),fetch('api/session')]); - state.snapshot=await snapshot.json(); state.reviews=await reviews.json(); state.session=await session.json(); - document.querySelector('#status').textContent=state.snapshot.activity.items.length+' root events · '+state.snapshot.runs.length+' runs · '+state.reviews.counts.unresolved+' to review · '+(state.session.reviewWriteEnabled?'OAuth review writes enabled':'read-only'); renderList(); + renderList(); + await loadResource('snapshot'); + await loadResource('session'); + await loadResource('artifacts'); + await loadResource('proposals'); + await loadResource('reviews'); } +function tabResource(){return state.tab==='suggestions'?'proposals':state.tab==='reviews'?'reviews':state.tab==='artifacts'?'artifacts':'snapshot'} +function updateStatus(){const parts=[];if(state.snapshot)parts.push(state.snapshot.activity.items.length+' root events',state.snapshot.runs.length+' runs');if(state.proposals)parts.push(state.proposals.items.filter(item=>item.status==='Awaiting review').length+' suggestions');if(state.reviews)parts.push(state.reviews.counts.unresolved+' to review');if(state.artifacts)parts.push(state.artifacts.active+' artifacts');if(state.session)parts.push(state.session.reviewWriteEnabled?'OAuth review writes enabled':'read-only');const pending=Object.values(state.loading).filter(Boolean).length;const failed=Object.keys(state.errors).length;if(pending)parts.push('loading '+pending);if(failed)parts.push(failed+' failed');document.querySelector('#status').textContent=parts.join(' · ')||'ready'} +function emptyMessage(){return state.tab==='events'?'No root activity has been recorded.':state.tab==='runs'?'No agent runs have been recorded.':state.tab==='sources'?'No configured or observed sources are available.':state.tab==='suggestions'?'No suggestions are waiting.':state.tab==='reviews'?'No reviews are waiting.':'No artifacts are available.'} function renderList(){ - const list=document.querySelector('#list'); const isEvents=state.tab==='events'; const isRuns=state.tab==='runs'; const isReviews=state.tab==='reviews'; - document.querySelector('#list-title').textContent=isEvents?'root activity':isRuns?'agent runs':isReviews?'review queue':'source health'; - const values=isEvents?state.snapshot.activity.items:isRuns?state.snapshot.runs:isReviews?state.reviews.items:state.snapshot.sources; - document.querySelector('#count').textContent=values.length; - list.innerHTML=values.map(value=>isEvents?eventItem(value):isRuns?runItem(value):isReviews?reviewItem(value):sourceItem(value)).join('')||'<div class="empty">Nothing recorded.</div>'; - list.querySelectorAll('button').forEach(button=>button.addEventListener('click',()=>select(button.dataset.kind,button.dataset.id))); + const list=document.querySelector('#list'); const isEvents=state.tab==='events'; const isRuns=state.tab==='runs'; const isSuggestions=state.tab==='suggestions'; const isReviews=state.tab==='reviews'; const isArtifacts=state.tab==='artifacts'; + document.querySelector('#list-title').textContent=isEvents?'root activity':isRuns?'agent runs':isSuggestions?'suggestions':isReviews?'review queue':isArtifacts?'artifact catalog':'source registry'; + const resourceKey=tabResource();const resource=resources[resourceKey];const error=state.errors[resourceKey]; + const ready=isSuggestions?state.proposals:isReviews?state.reviews:isArtifacts?state.artifacts:state.snapshot; + const values=!ready?[]:isEvents?state.snapshot.activity.items:isRuns?state.snapshot.runs:isSuggestions?state.proposals.items:isReviews?state.reviews.items:isArtifacts?state.artifacts.entries:state.snapshot.sources; + document.querySelector('#count').textContent=error?'error':ready?values.length:'…'; + list.innerHTML=error?'<div class="empty failed">'+esc(error)+'<div><button class="retry" data-resource="'+esc(resourceKey)+'">Retry</button></div></div>':!ready?'<div class="empty"><span class="spinner" aria-hidden="true"></span>Loading '+esc(resource.label)+'…</div>':values.map(value=>isEvents?eventItem(value):isRuns?runItem(value):isSuggestions?suggestionItem(value):isReviews?reviewItem(value):isArtifacts?artifactItem(value):sourceItem(value)).join('')||'<div class="empty">'+esc(emptyMessage())+'</div>'; + list.querySelector('.retry')?.addEventListener('click',event=>loadResource(event.currentTarget.dataset.resource)); + list.querySelectorAll('button.item').forEach(button=>button.addEventListener('click',()=>select(button.dataset.kind,button.dataset.id))); } function eventItem(event){ const consumers=(event.consumerRuns??[]).map(run=>'<div class="consumer"><strong>'+esc(agentName(run.agentId))+'</strong> <span class="badge">'+esc(run.kind)+'</span> <span class="badge '+(run.status==='failed'?'failed':'')+'">'+esc(run.status)+'</span><br>'+esc(run.description)+'</div>').join(''); - const descendants=event.descendantEventCount?' · '+event.descendantEventCount+' execution record'+(event.descendantEventCount===1?'':'s'):''; + const descendants=event.descendantEventCount?' · '+event.descendantEventCount+' known execution record'+(event.descendantEventCount===1?'':'s'):''; return '<button class="item" data-kind="event" data-id="'+esc(event.id)+'"><div class="summary">'+esc(event.summary)+'</div>'+consumers+'<div class="meta">'+esc(event.occurredAt)+' · '+esc(event.source)+' · '+esc(event.type)+descendants+' <span class="badge">'+esc(event.privacy)+'</span></div></button>'; } -function runItem(run){const evidence=state.snapshot.runEvidence.find(report=>report.runId===run.id);const kind=isModelRun(run)?'model':'rule';return '<button class="item" data-kind="run" data-id="'+esc(run.id)+'"><div class="type">'+esc(agentName(run.agentId))+' <span class="badge">'+kind+'</span> <span class="badge '+(run.status==='failed'?'failed':'')+'">'+esc(run.status)+'</span>'+(evidence&&!evidence.consistent?'<span class="badge failed">evidence contradiction</span>':'')+'</div><div class="summary">'+esc(run.result?.summary??run.errorText??run.id)+'</div><div class="meta">LLM inference: '+(kind==='model'?'yes':'no')+' · '+esc(run.startedAt??run.createdAt)+' · '+esc(run.provider)+'/'+esc(run.model)+'</div></button>'} -function sourceItem(source){return '<button class="item" data-kind="source" data-id="'+esc(source.source)+'"><div class="type">'+esc(source.source)+' <span class="badge '+(source.status==='failing'?'failed':'')+'">'+esc(source.status)+'</span></div><div class="summary">'+esc(source.lastError??source.cursorId??'no durable cursor')+'</div><div class="meta">'+source.operationsCompleted+' completed · '+source.operationFailures+' failed · '+source.inFlight+' in flight</div></button>'} +function runItem(run){const evidence=state.snapshot.runEvidence.find(report=>report.runId===run.id);const kind=isModelRun(run)?'model':'rule';return '<button class="item" data-kind="run" data-id="'+esc(run.id)+'"><div class="type">'+esc(agentName(run.agentId))+' <span class="badge">'+kind+'</span> <span class="badge '+(run.status==='failed'?'failed':'')+'">'+esc(run.status)+'</span>'+(evidence&&!evidence.consistent?'<span class="badge failed">evidence contradiction</span>':'')+'</div><div class="summary">'+esc(run.summary??run.errorText??run.id)+'</div><div class="meta">LLM inference: '+(kind==='model'?'yes':'no')+' · '+esc(run.startedAt??run.createdAt)+' · '+esc(run.provider)+'/'+esc(run.model)+'</div></button>'} +function sourceItem(source){ + const runtime=source.control?.runtime;const upstream=source.control?.upstream;const badges=[]; + badges.push('<span class="badge '+(source.configured&&source.enabled?'':'failed')+'">'+(source.configured?(source.enabled?'configured':'disabled'):'observed only')+'</span>'); + if(runtime)badges.push('<span class="badge '+(runtime.current?'':'failed')+'">runtime '+(runtime.current?'ready':'stale')+'</span>'); + if(upstream)badges.push('<span class="badge '+(upstream.current&&upstream.registered&&upstream.valid?'':'failed')+'">provider '+(upstream.registered&&upstream.valid?'registered':'unavailable')+'</span>'); + badges.push('<span class="badge '+(source.status==='failing'?'failed':'')+'">'+(source.hasActivityEvidence?'activity '+source.status:'awaiting first activity')+'</span>'); + const summary=upstream?(upstream.liveSubscriptionCount+'/'+upstream.desiredSubscriptionCount+' subscriptions '+(upstream.subscriptionsConverged?'converged':'drifting')):(source.lastError??source.cursorId??'No upstream control plane recorded.'); + const meta='control plane '+(source.control?'present':'not recorded')+' · data plane '+(source.hasActivityEvidence?(source.operationsCompleted+' completed · '+source.operationFailures+' failed'):'no durable activity yet'); + return '<button class="item" data-kind="source" data-id="'+esc(source.source)+'"><div class="type">'+esc(source.source)+' '+badges.join(' ')+'</div><div class="summary">'+esc(summary)+'</div><div class="meta">'+esc(meta)+'</div></button>'; +} +function suggestionItem(item){return '<button class="item" data-kind="suggestion" data-id="'+esc(item.eventId)+'"><div class="type">'+esc(item.kind)+' <span class="badge '+(item.status==='Failed'?'failed':'')+'">'+esc(item.status)+'</span></div><div class="summary">'+esc(item.proposedText.slice(0,240))+'</div><div class="meta">'+esc(item.reason)+' · '+esc(new Date(item.observedAt).toLocaleString())+'</div></button>'} function reviewItem(item){const status=item.decision?item.decision.disposition:'unreviewed';return '<button class="item" data-kind="review" data-id="'+esc(item.id)+'"><div class="type">'+esc(item.campaign.label)+' <span class="badge">'+esc(status)+'</span></div><div class="summary">'+esc(item.prompt.slice(0,240))+'</div><div class="meta">'+esc(item.criterion.label)+' · '+esc(item.privacy)+(item.decision?.externalExportEligible?' · export eligible':'')+'</div></button>'} async function select(kind,id){ if(kind==='source'){const source=state.snapshot.sources.find(value=>value.source===id);document.querySelector('#detail').innerHTML=renderSource(source);return} + if(kind==='suggestion'){const item=state.proposals.items.find(value=>value.eventId===id);document.querySelector('#detail').innerHTML=renderSuggestion(item);bindSuggestionForm(item);return} if(kind==='review'){const item=state.reviews.items.find(value=>value.id===id);document.querySelector('#detail').innerHTML=renderReview(item);bindReviewForm(item);return} - const response=await fetch('api/'+(kind==='event'?'events/':'runs/')+encodeURIComponent(id)); const data=await response.json(); - document.querySelector('#detail').innerHTML=kind==='event'?renderEvent(data):renderRun(data); - document.querySelectorAll('#detail [data-event-id]').forEach(button=>button.addEventListener('click',()=>select('event',button.dataset.eventId))); + if(kind==='artifact'){state.selected={kind,id};document.querySelector('main').classList.add('artifact-selected');document.querySelector('#detail').innerHTML='<button class="artifact-back">← Artifacts</button><div class="empty">Loading artifact…</div>';bindArtifactBack();const response=await fetch('api/artifacts/'+encodeURIComponent(id));const data=await response.json();document.querySelector('#detail').innerHTML=renderArtifact(data);bindArtifactBack();return} + state.selected={kind,id};const selection=kind+':'+id;const label=kind==='event'?'observation':'run';const detail=document.querySelector('#detail'); + detail.innerHTML='<div class="empty"><span class="spinner" aria-hidden="true"></span>Loading '+label+'…</div>'; + try{const data=await fetchJson('api/'+(kind==='event'?'events/':'runs/')+encodeURIComponent(id),label);if(state.selected?.kind+':'+state.selected?.id!==selection)return;detail.innerHTML=kind==='event'?renderEvent(data):renderRun(data);document.querySelectorAll('#detail [data-event-id]').forEach(button=>button.addEventListener('click',()=>select('event',button.dataset.eventId)))}catch(error){if(state.selected?.kind+':'+state.selected?.id!==selection)return;detail.innerHTML='<div class="empty failed">'+esc(error instanceof Error?error.message:String(error))+'<div><button class="retry" id="detail-retry">Retry</button></div></div>';document.querySelector('#detail-retry')?.addEventListener('click',()=>select(kind,id))} } function eventLink(label,event){return event?'<button data-event-id="'+esc(event.id)+'">'+esc(label)+': '+esc(event.type)+' · '+esc(event.id)+'</button>':''} function eventSummary(event){if(event?.payload?.path)return event.payload.path+' '+(event.type.split('.').at(-1)??'observed');if(event?.payload?.title)return event.payload.title;if(event?.payload?.summary)return event.payload.summary;return event?.type??'event'} @@ -265,8 +649,25 @@ function renderAgentWork(activity){ } function isModelRun(run){return !(run.provider==='deterministic'&&run.model==='deterministic')} function renderSource(source){ - if(!source)return '<div class="failed">Source not found.</div>'; return '<h2>'+esc(source.source)+'</h2><div class="meta">'+esc(source.status)+'</div><h3>operation receipts</h3><pre>'+json({operationsStarted:source.operationsStarted,operationsCompleted:source.operationsCompleted,operationFailures:source.operationFailures,recoveries:source.recoveries,inFlight:source.inFlight,lastSuccessAt:source.lastSuccessAt,lastFailureAt:source.lastFailureAt,lastError:source.lastError})+'</pre><h3>durable cursor</h3><pre>'+json({cursorId:source.cursorId,cursor:source.cursor,updatedAt:source.updatedAt})+'</pre>'; + if(!source)return '<div class="failed">Source not found.</div>'; + const runtime=source.control?.runtime;const upstream=source.control?.upstream; + const facts='<div class="facts"><span>Configuration: <strong>'+(source.configured?(source.enabled?'enabled':'disabled'):'not registered')+'</strong></span><span>Runtime: <strong>'+(runtime?(runtime.current?'ready':'stale'):'not recorded')+'</strong></span><span>Provider: <strong>'+(upstream?(upstream.registered&&upstream.valid?'registered':'unavailable'):'not applicable / unrecorded')+'</strong></span><span>Subscriptions: <strong>'+(upstream?(upstream.liveSubscriptionCount+'/'+upstream.desiredSubscriptionCount+(upstream.subscriptionsConverged?' converged':' drifting')):'not recorded')+'</strong></span><span>Activity: <strong>'+(source.hasActivityEvidence?source.status:'none yet')+'</strong></span></div>'; + const control={configured:source.configured,enabled:source.enabled,kind:source.kind,lane:source.control?.lane,runtime:runtime?{state:runtime.state,current:runtime.current,readyAt:runtime.readyAt,revision:runtime.revision}:undefined,upstream:upstream?{registered:upstream.registered,valid:upstream.valid,current:upstream.current,webhookId:upstream.webhookId,subscriptions:upstream.liveSubscriptionCount+'/'+upstream.desiredSubscriptionCount,subscriptionsConverged:upstream.subscriptionsConverged,checkedAt:upstream.checkedAt}:undefined}; + const evidence={status:source.status,hasActivityEvidence:source.hasActivityEvidence,operationsStarted:source.operationsStarted,operationsCompleted:source.operationsCompleted,operationFailures:source.operationFailures,recoveries:source.recoveries,inFlight:source.inFlight,lastSuccessAt:source.lastSuccessAt,lastFailureAt:source.lastFailureAt,lastError:source.lastError,cursorId:source.cursorId,cursor:source.cursor,updatedAt:source.updatedAt}; + return '<h2>'+esc(source.source)+'</h2><div class="meta">Control-plane registration and data-plane evidence are independent receipts.</div>'+facts+'<h3>control plane</h3><pre>'+json(control)+'</pre><h3>data-plane evidence</h3><pre>'+json(evidence)+'</pre>'; +} +function renderSuggestion(item){ + if(!item)return '<div class="failed">Suggestion not found.</div>'; + const content=item.kind==='Memory suggestion'?'<article class="artifact-document suggestion-content">'+renderMarkdownClient(item.proposedText)+'</article>':'<div class="suggestion-compare"><div><h3>Original delivered reply</h3><div class="suggestion-content">'+esc(item.originalText)+'</div></div><div><h3>Proposed replacement</h3><div class="suggestion-content">'+esc(item.proposedText)+'</div></div></div>'; + const details=Object.entries(item.technical).map(([key,value])=>'<div class="human-card"><div class="field-label">'+esc(humanLabel(key))+'</div><div class="mono">'+esc(value)+'</div></div>').join(''); + const receipt=state.proposalReceipt?'<div class="decision"><strong>'+esc(state.proposalReceipt)+'</strong></div>':''; + const failure=item.failureReason?'<div class="decision failed"><strong>Stream memory was not changed.</strong><div>'+esc(item.failureReason)+'</div></div>':''; + const controls=item.status==='Awaiting review'?(state.session.reviewWriteEnabled?'<div class="suggestion-actions"><div class="field-label">Review this suggestion</div><div class="form-row"><label for="suggestion-edit">Edit before accepting</label><textarea id="suggestion-edit">'+esc(item.proposedText)+'</textarea></div><div class="suggestion-buttons"><button data-disposition="accept">Accept as written</button><button data-disposition="edit">Accept edited text</button><button class="reject" data-disposition="reject">Reject suggestion</button></div><div id="suggestion-result" class="muted"></div></div>':'<div class="empty">Sign in with Cameron’s OAuth operator session to review suggestions. Basic access is read-only.</div>'):''; + return '<h2>'+esc(item.kind)+'</h2><div class="facts"><span>Status: <strong>'+esc(item.status)+'</strong></span><span>Action: <strong>'+esc(item.operation)+'</strong></span><span>Observed: <strong>'+esc(new Date(item.observedAt).toLocaleString())+'</strong></span></div><h3>Reason</h3><div class="work-summary">'+esc(item.reason)+'</div><h3>Suggested content</h3>'+content+receipt+failure+controls+'<details><summary>Technical details</summary><div class="human-grid">'+details+'</div></details>'; } +function renderMarkdownClient(markdown){return esc(markdown).split(/\\n\\n+/).map(block=>'<p>'+block.replace(/\\n/g,'<br>')+'</p>').join('')} +function bindSuggestionForm(item){if(!item||item.status!=='Awaiting review'||!state.session.reviewWriteEnabled)return;document.querySelectorAll('.suggestion-buttons button').forEach(button=>button.addEventListener('click',()=>submitSuggestion(item,button.dataset.disposition)))} +async function submitSuggestion(item,disposition){const result=document.querySelector('#suggestion-result');document.querySelectorAll('.suggestion-buttons button').forEach(button=>button.disabled=true);result.textContent='Saving decision…';const body={disposition,submissionId:crypto.randomUUID()};if(disposition==='edit')body.replacementText=document.querySelector('#suggestion-edit').value;try{const response=await fetch('api/proposals/'+encodeURIComponent(item.eventId)+'/decisions',{method:'POST',headers:{'content-type':'application/json','x-thoughtstream-csrf':state.session.csrfToken},body:JSON.stringify(body)});const receipt=await response.json();if(!response.ok)throw new Error(receipt.error??'Suggestion decision failed');state.proposalReceipt=receipt.receipt??(disposition==='reject'?'Suggestion rejected':'Suggestion accepted');state.proposals=await (await fetch('api/proposals')).json();updateStatus();renderList();select('suggestion',item.eventId)}catch(error){result.textContent=error.message;document.querySelectorAll('.suggestion-buttons button').forEach(button=>button.disabled=false)}} function renderReview(item){ if(!item)return '<div class="failed">Review item not found.</div>'; const evidence=item.evidence?'<h3>evidence</h3><div class="review-prompt">'+esc(item.evidence)+'</div>':''; @@ -294,7 +695,22 @@ async function submitReview(item){ if(body.reasonCodes.length>2||body.responseTags.length>2){result.textContent='choose at most two codes and two tags';button.disabled=false;return} try{const response=await fetch('api/reviews/'+encodeURIComponent(item.id)+'/decisions',{method:'POST',headers:{'content-type':'application/json','x-thoughtstream-csrf':state.session.csrfToken},body:JSON.stringify(body)});if(!response.ok)throw new Error((await response.json()).error??'write failed');const reviews=await fetch('api/reviews');state.reviews=await reviews.json();renderList();select('review',item.id)}catch(error){result.textContent=error.message;button.disabled=false} } -for(const tab of ['events','runs','sources','reviews'])document.querySelector('#'+tab+'-tab').addEventListener('click',()=>{state.tab=tab;document.querySelectorAll('nav button').forEach(b=>b.classList.toggle('active',b.id===tab+'-tab'));renderList()}); +function artifactItem(entry){const status=entry.superseded?' · Superseded':'';return '<button class="item" data-kind="artifact" data-id="'+esc(entry.eventId)+'"><div class="summary">'+esc(entry.title)+'</div><div class="meta">'+esc(humanLabel(entry.kind))+' · '+esc(humanMediaType(entry.mediaType))+' · '+esc(humanLabel(entry.visibility))+' · '+esc(formatBytes(entry.bodySizeBytes))+status+'</div></button>'} +function renderArtifact(data){ + if(data.error)return '<div class="failed">'+esc(data.error)+'</div>';const e=data.event;const entry=data.catalogEntry||{}; + const superseded=entry.superseded?'<span class="badge failed">superseded</span>':'<span class="badge" style="color:var(--cyan)">active</span>'; + const content=data.renderedHtml!==undefined?'<article class="artifact-document">'+data.renderedHtml+'</article>':data.text!==undefined?'<pre class="artifact-content">'+esc(data.text)+'</pre>':data.contentPath?'<img src="'+esc(data.contentPath)+'" alt="Private artifact: '+esc(e.payload.title)+'" style="max-width:100%;max-height:600px">':'<div class="failed">Content unavailable</div>'; + const provenance='<div class="human-grid"><div class="human-card"><div class="field-label">Source</div><div class="field-value">'+esc(humanLabel(e.payload.provenance.source))+'</div></div><div class="human-card"><div class="field-label">Source label</div><div class="field-value">'+esc(humanLabel(e.payload.provenance.label))+'</div></div></div>'; + const relations=(e.payload.relations??[]).map(relation=>'<div class="relation-card"><div class="field-label">'+esc(humanLabel(relation.type))+'</div><div class="field-value">'+esc(relation.type==='materialized-from-request'?'Artifact storage request':'Related event')+'</div><div class="mono">'+esc(relation.eventId)+'</div></div>').join(''); + const record='<div class="human-grid"><div class="human-card"><div class="field-label">Artifact</div><div class="field-value">'+esc(e.payload.artifactId)+' · version '+esc(e.payload.artifactVersion)+'</div></div><div class="human-card"><div class="field-label">Stored</div><div class="field-value">'+esc(new Date(e.occurredAt).toLocaleString())+'</div></div><div class="human-card wide"><div class="field-label">Artifact event</div><div class="mono">'+esc(e.id)+'</div></div>'+(entry.supersedesArtifactEventId?'<div class="human-card wide"><div class="field-label">Supersedes artifact event</div><div class="mono">'+esc(entry.supersedesArtifactEventId)+'</div></div>':'')+'</div>'; + const metadata='<details class="artifact-metadata"><summary>Artifact details</summary><div class="facts"><span>Status: <strong>'+superseded+'</strong></span><span>Format: <strong>'+esc(humanMediaType(e.payload.mediaType))+'</strong></span><span>Access: <strong>'+esc(humanLabel(e.payload.visibility))+'</strong></span><span>Public release: <strong>'+(e.payload.publicationEligible?'Eligible':'Not allowed')+'</strong></span><span>Size: <strong>'+esc(formatBytes(entry.bodySizeBytes))+'</strong></span></div><h3>Summary</h3><div class="work-summary">'+esc(e.payload.summary)+'</div><h3>Provenance</h3>'+provenance+(relations?'<h3>Relations</h3><div class="relation-list">'+relations+'</div>':'')+'<h3>Record</h3>'+record+'<div class="human-card wide" style="margin-top:8px"><div class="field-label">SHA-256 fingerprint</div><div class="mono">'+esc(data.bodySha256)+'</div></div></details>'; + return '<button class="artifact-back">← Artifacts</button><h2>'+esc(e.payload.title)+'</h2>'+content+metadata; +} +function bindArtifactBack(){document.querySelector('.artifact-back')?.addEventListener('click',()=>{state.selected=null;document.querySelector('main').classList.remove('artifact-selected');document.querySelector('#detail').innerHTML='<div class="empty">Select an artifact to read it.</div>';window.scrollTo({top:0,behavior:'smooth'})})} +function humanLabel(value){return String(value??'').replace(/[-_.:]+/g,' ').replace(/\b\w/g,char=>char.toUpperCase())} +function humanMediaType(value){return ({'text/markdown':'Markdown','text/plain':'Plain text','application/json':'JSON document','application/yaml':'YAML document','text/yaml':'YAML document','image/png':'PNG image','image/jpeg':'JPEG image'})[value]??humanLabel(value)} +function formatBytes(value){const n=Number(value);if(!Number.isFinite(n))return 'Unknown';if(n<1024)return n+' bytes';if(n<1024*1024)return (n/1024).toFixed(n<10240?1:0)+' KB';return (n/(1024*1024)).toFixed(1)+' MB'} +for(const tab of ['events','runs','sources','suggestions','reviews','artifacts'])document.querySelector('#'+tab+'-tab').addEventListener('click',()=>{state.tab=tab;state.selected=null;document.querySelector('main').classList.remove('artifact-selected');document.querySelectorAll('nav button').forEach(b=>b.classList.toggle('active',b.id===tab+'-tab'));renderList()}); load().catch(error=>{document.querySelector('#status').textContent='error';document.querySelector('#detail').innerHTML='<div class="failed">'+esc(error.message)+'</div>'}); </script></body></html>`; } @@ -335,6 +751,22 @@ function isLoopback(host: string): boolean { return host === "127.0.0.1" || host === "::1" || host === "localhost"; } +function inspectorRunSummary(run: AgentRun) { + return { + id: run.id, + agentId: run.agentId, + agentVersion: run.agentVersion, + status: run.status, + provider: run.provider, + model: run.model, + createdAt: run.createdAt, + ...(run.startedAt ? { startedAt: run.startedAt } : {}), + ...(run.completedAt ? { completedAt: run.completedAt } : {}), + ...(run.errorText ? { errorText: run.errorText } : {}), + ...(typeof run.result?.summary === "string" ? { summary: run.result.summary } : {}), + }; +} + function eventBelongsToRun(event: ThoughtEvent, run: AgentRun): boolean { const payloadRunId = typeof event.payload.runId === "string" ? event.payload.runId : undefined; return payloadRunId === run.id @@ -342,6 +774,16 @@ function eventBelongsToRun(event: ThoughtEvent, run: AgentRun): boolean { || run.outputEventIds.includes(event.id); } +async function loadRunEvidenceEvents(store: JazzThoughtStore, runs: AgentRun[]): Promise<ThoughtEvent[]> { + const terminalEvents = await store.listEvents({ types: RUN_TERMINAL_EVENT_TYPES }); + const referencedOutputIds = [ + ...runs.flatMap((run) => run.outputEventIds), + ...terminalEvents.map((event) => event.payload.outputEventId).filter((id): id is string => typeof id === "string"), + ]; + const outputEvents = await store.getEvents(referencedOutputIds); + return [...new Map([...terminalEvents, ...outputEvents].map((event) => [event.id, event])).values()]; +} + async function expandRun(store: JazzThoughtStore, run: AgentRun): Promise<{ run: AgentRun; kind: "rule" | "model"; diff --git a/test/artifacts.test.ts b/test/artifacts.test.ts new file mode 100644 index 0000000..d0e1eb1 --- /dev/null +++ b/test/artifacts.test.ts @@ -0,0 +1,43 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import { afterEach, describe, expect, test } from "vitest"; +import { appendArtifactRequest, materializeArtifactRequest, requestArtifactStorage } from "../src/artifacts/append.js"; +import { buildArtifactCatalog, getArtifactBody, getArtifactCatalog } from "../src/artifacts/catalog.js"; +import { ARTIFACT_EVENT_TYPE, ARTIFACT_MAX_IMAGE_BYTES, ARTIFACT_MAX_TEXT_BYTES, ARTIFACT_REQUEST_EVENT_TYPE } from "../src/artifacts/types.js"; +import { createDefaultRegistry } from "../src/events/registry.js"; +import { JazzThoughtStore } from "../src/jazz/store.js"; +import { renderArtifactMarkdown, startInspectorServer } from "../src/web/inspector.js"; +import { temporaryProject, testStore } from "./helpers.js"; + +const stores: JazzThoughtStore[] = []; const roots: string[] = []; const servers: import("node:http").Server[] = []; +afterEach(async () => { await Promise.all(servers.splice(0).map((server) => new Promise<void>((resolve) => { server.closeAllConnections(); server.close(() => resolve()); }))); await Promise.all(stores.splice(0).map((store) => store.close())); await Promise.all(roots.splice(0).map((root) => fs.rm(root, { recursive: true, force: true }))); }); +async function fixture(name = "artifact.md", bytes: string | Buffer = "# Artifact\n") { const project = await temporaryProject("artifact-store-"); const workspace = await temporaryProject("artifact-workspace-"); roots.push(project, workspace); const store = testStore(project); stores.push(store); await fs.writeFile(path.join(workspace, name), bytes); return { project, workspace, store }; } +const input = { requestId: "req-1", workspaceLeaseId: "lease-1", workspaceRootLabel: "exe workspace lease", relativeFilePath: "artifact.md", artifactId: "skill:test", artifactVersion: 1, kind: "skill" as const, title: "Test", summary: "Private test artifact", mediaType: "text/markdown" as const, provenance: { source: "agent-workspace", label: "selected skill file" }, visibility: "private" as const, requestingAgentId: "agent-test", requestingRunId: "run-test", sourceEventId: "evt-source" }; +const png = Buffer.from([137,80,78,71,13,10,26,10,0,0,0,0]); + +describe("durable private artifact storage", () => { + test("request evidence is private, bounded, path-dark, and has no authority", async () => { const { workspace, store } = await fixture(); const request = await appendArtifactRequest({ store, ...input }); expect(request.event.type).toBe(ARTIFACT_REQUEST_EVENT_TYPE); expect(request.event.privacy).toBe("private"); expect(request.event.payload.publicationEligible).toBe(false); expect(request.event.payload.publicationAuthority).toBe(false); expect(JSON.stringify(request.event)).not.toContain(workspace); }); + test("schema rejects absolute request paths and provenance labels", () => { const registry = createDefaultRegistry(); const payload = { ...input, publicationEligible: false, publicationAuthority: false, status: "pending" }; expect(() => registry.validateEvent(ARTIFACT_REQUEST_EVENT_TYPE, 1, "private", { ...payload, relativeFilePath: "/etc/passwd" } as never)).toThrow(); expect(() => registry.validateEvent(ARTIFACT_REQUEST_EVENT_TYPE, 1, "private", { ...payload, provenance: { source: "x", label: "/secret" } } as never)).toThrow(); }); + test("materializes exact bytes once with strong request lineage and no inline bytes", async () => { const { workspace, store } = await fixture(); const result = await requestArtifactStorage({ store, workspaceRoot: workspace, ...input }); expect(result.event.type).toBe(ARTIFACT_EVENT_TYPE); expect(result.event.payload.publicationEligible).toBe(false); expect(result.event.payload.relations).toContainEqual({ type: "materialized-from-request", eventId: result.requestEvent!.id }); expect(result.event.payload.blob).toMatchObject({ algorithm: "sha256", byteCount: 11 }); expect(result.event.payload.body).toBeUndefined(); expect(result.event.payload.bodySha256).toBeUndefined(); const detail = await getArtifactBody(store, result.event.id); expect(detail?.text).toBe("# Artifact\n"); }); + test("same identity and bytes is idempotent; changed bytes conflict", async () => { const { workspace, store } = await fixture(); const first = await requestArtifactStorage({ store, workspaceRoot: workspace, ...input }); const replay = await requestArtifactStorage({ store, workspaceRoot: workspace, ...input }); expect(first.inserted).toBe(true); expect(replay.inserted).toBe(false); await fs.writeFile(path.join(workspace, "artifact.md"), "changed"); await expect(requestArtifactStorage({ store, workspaceRoot: workspace, ...input })).rejects.toThrow("Idempotency key reused with different payload"); }); + test("request lease identity must match materializer authority", async () => { const { workspace, store } = await fixture(); const request = await appendArtifactRequest({ store, ...input }); await expect(materializeArtifactRequest({ store, requestEventId: request.event.id, workspaceRoot: workspace, workspaceLeaseId: "other" })).rejects.toThrow("lease identity"); }); + test("rejects workspace escape", async () => { const { workspace, store } = await fixture(); await expect(requestArtifactStorage({ store, workspaceRoot: workspace, ...input, relativeFilePath: "../outside.md" })).rejects.toThrow("normalized relative path"); }); + test("rejects file symlink", async () => { const { workspace, store } = await fixture(); await fs.symlink("artifact.md", path.join(workspace, "alias.md")); await expect(requestArtifactStorage({ store, workspaceRoot: workspace, ...input, relativeFilePath: "alias.md" })).rejects.toThrow("symbolic links"); }); + test("rejects parent symlink", async () => { const { workspace, store } = await fixture(); await fs.mkdir(path.join(workspace, "real")); await fs.writeFile(path.join(workspace, "real", "x.md"), "x"); await fs.symlink("real", path.join(workspace, "alias")); await expect(requestArtifactStorage({ store, workspaceRoot: workspace, ...input, relativeFilePath: "alias/x.md" })).rejects.toThrow("symbolic links"); }); + test("rejects extension and MIME mismatch", async () => { const { workspace, store } = await fixture("artifact.txt", "x"); await expect(requestArtifactStorage({ store, workspaceRoot: workspace, ...input, relativeFilePath: "artifact.txt" })).rejects.toThrow("extension and media type"); }); + test("rejects PNG and JPEG magic mismatch", async () => { const a = await fixture("artifact.png", "not png"); await expect(requestArtifactStorage({ store: a.store, workspaceRoot: a.workspace, ...input, relativeFilePath: "artifact.png", mediaType: "image/png", kind: "image" })).rejects.toThrow("PNG magic"); const b = await fixture("artifact.jpg", "not jpeg"); await expect(requestArtifactStorage({ store: b.store, workspaceRoot: b.workspace, ...input, relativeFilePath: "artifact.jpg", mediaType: "image/jpeg", kind: "image" })).rejects.toThrow("JPEG magic"); }); + test("validates JSON and YAML structure", async () => { const a = await fixture("artifact.json", "{"); await expect(requestArtifactStorage({ store: a.store, workspaceRoot: a.workspace, ...input, relativeFilePath: "artifact.json", mediaType: "application/json" })).rejects.toThrow(); const b = await fixture("artifact.yaml", "a: ["); await expect(requestArtifactStorage({ store: b.store, workspaceRoot: b.workspace, ...input, relativeFilePath: "artifact.yaml", mediaType: "application/yaml" })).rejects.toThrow(); }); + test("enforces exact text and image caps", async () => { const a = await fixture("artifact.txt", Buffer.alloc(ARTIFACT_MAX_TEXT_BYTES + 1, 1)); await expect(requestArtifactStorage({ store: a.store, workspaceRoot: a.workspace, ...input, relativeFilePath: "artifact.txt", mediaType: "text/plain" })).rejects.toThrow("exceeds"); const b = await fixture("artifact.png", Buffer.alloc(ARTIFACT_MAX_IMAGE_BYTES + 1, 1)); await expect(requestArtifactStorage({ store: b.store, workspaceRoot: b.workspace, ...input, relativeFilePath: "artifact.png", mediaType: "image/png", kind: "image" })).rejects.toThrow("exceeds"); }); + test("stores blobs with canonical path and private modes", async () => { const { workspace, store } = await fixture(); const result = await requestArtifactStorage({ store, workspaceRoot: workspace, ...input }); const blob = result.event.payload.blob as { relativePath: string }; const rootStat = await fs.stat(store.getArtifactRoot()); const fileStat = await fs.stat(path.join(store.getArtifactRoot(), ...blob.relativePath.split("/"))); expect(rootStat.mode & 0o777).toBe(0o700); expect(fileStat.mode & 0o777).toBe(0o600); expect(blob.relativePath).toMatch(/^sha256\/[a-f0-9]{2}\/[a-f0-9]{64}$/); }); + test("blob tamper fails on read", async () => { const { workspace, store } = await fixture(); const result = await requestArtifactStorage({ store, workspaceRoot: workspace, ...input }); const blob = result.event.payload.blob as { relativePath: string }; await fs.writeFile(path.join(store.getArtifactRoot(), ...blob.relativePath.split("/")), "tamper"); await expect(getArtifactBody(store, result.event.id)).rejects.toThrow("integrity check failed"); }); + test("catalog is projected, metadata-only, and path/credential/body dark", async () => { const { workspace, store } = await fixture("artifact.md", "PRIVATE-BODY credential=secret-token"); const result = await requestArtifactStorage({ store, workspaceRoot: workspace, ...input }); const catalog = await getArtifactCatalog(store); const projection = await store.getProjection("artifact-catalog"); const serialized = JSON.stringify(catalog); expect(catalog.entries[0]?.eventId).toBe(result.event.id); expect(projection?.projectionVersion).toBe(1); expect(projection?.payload).toEqual(JSON.parse(serialized)); expect(serialized).not.toContain("PRIVATE-BODY"); expect(serialized).not.toContain("secret-token"); expect(serialized).not.toContain(workspace); }); + test("private inspector returns safe text detail and rejects mutations", async () => { const { workspace, store } = await fixture(); const result = await requestArtifactStorage({ store, workspaceRoot: workspace, ...input }); const { base } = await inspector(store); const catalog = await (await fetch(`${base}/api/artifacts`)).text(); expect(catalog).not.toContain("# Artifact"); const detail = await (await fetch(`${base}/api/artifacts/${result.event.id}`)).json() as { text: string; body?: string }; expect(detail.text).toBe("# Artifact\n"); expect(detail.body).toBeUndefined(); expect((await fetch(`${base}/api/artifacts`, { method: "POST" })).status).toBe(405); }); + test("artifact UI renders content first, collapses metadata, and hides the list on mobile selection", async () => { const { store } = await fixture(); const { base } = await inspector(store); const page = await (await fetch(base)).text(); const renderer = page.slice(page.indexOf("function renderArtifact"), page.indexOf("function bindArtifactBack")); expect(page).toContain("main.artifact-selected #list-pane{display:none}"); expect(page).toContain("data.renderedHtml"); expect(page).toContain("<summary>Artifact details</summary>"); expect(renderer).not.toContain("JSON.stringify(item"); expect(renderer).not.toContain("canonical envelope"); expect(renderer.indexOf("+content+metadata")).toBeGreaterThan(-1); }); + test("Markdown rendering strips frontmatter, preserves document structure, and escapes active content", () => { const rendered = renderArtifactMarkdown("---\nid: secret\n---\n# Heading\n\n- one\n- **two**\n\n***both*** `code` [site](https://example.com) <script>alert(1)</script> [bad](javascript:alert(2))\n"); expect(rendered).toContain("<h1>Heading</h1>"); expect(rendered).toContain("<ul><li>one</li><li><strong>two</strong></li></ul>"); expect(rendered).toContain("<strong><em>both</em></strong>"); expect(rendered).toContain("<code>code</code>"); expect(rendered).toContain('<a href="https://example.com" rel="noreferrer">site</a>'); expect(rendered).toContain("<script>alert(1)</script>"); expect(rendered).toContain("[bad](javascript:alert(2))"); expect(rendered).not.toContain("id: secret"); }); + test("suggestions UI is human-first, precedes review, and never renders proposal JSON", async () => { const { store } = await fixture(); const { base } = await inspector(store); const queue = await (await fetch(`${base}/api/proposals`)).json() as { count: number; items: unknown[] }; expect(queue).toEqual({ count: 0, items: [] }); const page = await (await fetch(base)).text(); expect(page.indexOf('id="suggestions-tab"')).toBeLessThan(page.indexOf('id="reviews-tab"')); const renderer = page.slice(page.indexOf("function renderSuggestion"), page.indexOf("function renderReview")); expect(renderer).toContain("Memory suggestion"); expect(renderer).toContain("Original delivered reply"); expect(renderer).toContain("Technical details"); expect(renderer).not.toContain("JSON.stringify(item"); expect(renderer).not.toContain("raw payload"); }); + test("generated inspector client JavaScript parses", async () => { const { store } = await fixture(); const { base } = await inspector(store); const page = await (await fetch(base)).text(); const script = page.match(/<script>([\s\S]*?)<\/script>/)?.[1]; expect(script).toBeDefined(); expect(() => new Function(script!)).not.toThrow(); }); + test("private inspector image detail is metadata-only and content is on-demand", async () => { const { workspace, store } = await fixture("artifact.png", png); const result = await requestArtifactStorage({ store, workspaceRoot: workspace, ...input, relativeFilePath: "artifact.png", mediaType: "image/png", kind: "image" }); const { base } = await inspector(store); const detail = await (await fetch(`${base}/api/artifacts/${result.event.id}`)).json() as { contentPath: string; text?: string }; expect(detail.text).toBeUndefined(); expect(JSON.stringify(detail)).not.toContain(png.toString("base64")); const content = await fetch(`${base}${detail.contentPath}`); expect(content.headers.get("content-type")).toBe("image/png"); expect(Buffer.from(await content.arrayBuffer())).toEqual(png); }); + test("sensitive request and artifact remain sensitive and non-publishable", async () => { const { workspace, store } = await fixture(); const result = await requestArtifactStorage({ store, workspaceRoot: workspace, ...input, visibility: "sensitive" }); expect(result.requestEvent?.privacy).toBe("sensitive"); expect(result.event.privacy).toBe("sensitive"); expect(result.event.payload.publicationEligible).toBe(false); }); + test("supersession remains append-only", async () => { const { workspace, store } = await fixture(); const v1 = await requestArtifactStorage({ store, workspaceRoot: workspace, ...input }); await fs.writeFile(path.join(workspace, "artifact.md"), "v2"); const v2 = await requestArtifactStorage({ store, workspaceRoot: workspace, ...input, requestId: "req-2", artifactVersion: 2, supersedesArtifactEventId: v1.event.id }); const catalog = await buildArtifactCatalog(store); expect(catalog.entries.find((e) => e.eventId === v1.event.id)?.superseded).toBe(true); expect(v2.event.payload.supersedesArtifactEventId).toBe(v1.event.id); }); +}); +async function inspector(store: JazzThoughtStore) { const server = await startInspectorServer(store, { port: 0 }); servers.push(server); const address = server.address(); if (!address || typeof address === "string") throw new Error("missing address"); return { base: `http://127.0.0.1:${address.port}` }; } diff --git a/test/exe-artifact-transport.test.ts b/test/exe-artifact-transport.test.ts new file mode 100644 index 0000000..db0e3e9 --- /dev/null +++ b/test/exe-artifact-transport.test.ts @@ -0,0 +1,37 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import { exeArtifactRequestSchema } from "../src/artifacts/exe-transport.js"; + +const valid = { + requestId: "request-1", leaseId: "lease-1", relativeOutputFilePath: "reports/result.md", + artifactId: "artifact-1", artifactVersion: 1, kind: "report", title: "Result", summary: "Private result", + mediaType: "text/markdown", provenanceLabel: "exe workspace", visibility: "private", +}; + +describe("exe artifact transport request contract", () => { + it("accepts only the strict bounded request schema", () => { + expect(exeArtifactRequestSchema.parse(valid)).toMatchObject(valid); + for (const value of [ + { ...valid, command: "cat /etc/passwd" }, { ...valid, destinationUrl: "https://example.com" }, + { ...valid, publicationAuthority: true }, { ...valid, relativeOutputFilePath: "../secret" }, + { ...valid, relativeOutputFilePath: "/etc/passwd" }, { ...valid, relativeOutputFilePath: "a\\b.md" }, + { ...valid, artifactVersion: 0 }, { ...valid, visibility: "public" }, + ]) expect(exeArtifactRequestSchema.safeParse(value).success).toBe(false); + }); + it("treats shell metacharacters as invalid inert data", () => { + for (const field of ["requestId", "artifactId", "leaseId"] as const) expect(exeArtifactRequestSchema.safeParse({ ...valid, [field]: "x;touch /tmp/pwned" }).success).toBe(false); + expect(exeArtifactRequestSchema.parse({ ...valid, title: "$(not executed); still text" }).title).toContain("not executed"); + }); + it("defaults visibility to private", () => { + const { visibility: _, ...without } = valid; + expect(exeArtifactRequestSchema.parse(without).visibility).toBe("private"); + }); + it("can create owner-only staging fixtures", async () => { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "exe-transport-")); + await fs.chmod(root, 0o700); const file = path.join(root, "artifact.md"); await fs.writeFile(file, "# private\n", { mode: 0o600 }); + expect((await fs.stat(root)).mode & 0o777).toBe(0o700); expect((await fs.stat(file)).mode & 0o777).toBe(0o600); + await fs.rm(root, { recursive: true }); + }); +}); diff --git a/test/inspector.test.ts b/test/inspector.test.ts index 737f11d..e35ef02 100644 --- a/test/inspector.test.ts +++ b/test/inspector.test.ts @@ -99,8 +99,15 @@ describe("thought stream inspector", () => { expect(page.status).toBe(200); const pageHtml = await page.text(); expect(pageHtml).toContain("thought stream inspector"); - expect(pageHtml).toContain("fetch('api/snapshot')"); + expect(pageHtml).toContain("await loadResource('snapshot')"); + expect(pageHtml).not.toContain("Promise.all([fetch('api/snapshot')"); expect(pageHtml).not.toContain("fetch('/api/snapshot')"); + expect(pageHtml).toContain("response.headers.get('content-type')"); + expect(pageHtml).toContain("Loading root activity…"); + expect(pageHtml).toContain("Loading '+label+'…"); + expect(pageHtml).toContain("detail-retry"); + expect(pageHtml).toContain("No root activity has been recorded."); + expect(pageHtml).toContain("No artifacts are available."); expect(pageHtml).toContain("cursor:pointer; overflow-wrap:anywhere"); expect(pageHtml).toContain("font-size:14px; overflow-wrap:anywhere"); expect(page.headers.get("content-security-policy")).toContain("frame-ancestors 'none'"); @@ -108,11 +115,14 @@ describe("thought stream inspector", () => { const snapshot = await (await fetch(`${base}/api/snapshot`)).json() as { activity: { totalEvents: number }; evidenceContradictions: number; + runs: Array<{ id: string; summary?: string; contextManifest?: unknown }>; runEvidence: Array<{ runId: string; consistent: boolean; issues: Array<{ code: string }> }>; sources: Array<{ source: string; status: string; operationsStarted: number; operationFailures: number; inFlight: number; cursor: { etag?: string } }>; }; expect(snapshot.activity.totalEvents).toBe(3); expect(snapshot.evidenceContradictions).toBe(1); + expect(snapshot.runs[0]).toMatchObject({ id: "run_contradictory" }); + expect(snapshot.runs[0]?.contextManifest).toBeUndefined(); expect(snapshot.runEvidence[0]).toMatchObject({ runId: "run_contradictory", consistent: false, diff --git a/test/letta-agent-sdk.test.ts b/test/letta-agent-sdk.test.ts index b0694b6..4a83d09 100644 --- a/test/letta-agent-sdk.test.ts +++ b/test/letta-agent-sdk.test.ts @@ -611,7 +611,7 @@ class FakeSession { async listMessages(_options?: ListMessagesOptions): Promise<ListMessagesResult> { if (this.listMessagesError) throw this.listMessagesError; return { - messages: this.history, + messages: this.history as ListMessagesResult["messages"], nextBefore: this.historyNextBefore, hasMore: this.historyHasMore, }; diff --git a/test/source-health.test.ts b/test/source-health.test.ts index 21ee98f..b49ed9a 100644 --- a/test/source-health.test.ts +++ b/test/source-health.test.ts @@ -62,6 +62,68 @@ describe("source health projection", () => { }), ]); }); + + test("shows configured runtime and upstream state before the first activity event", async () => { + const project = await temporaryProject(); + roots.push(project); + const store = testStore(project); + stores.push(store); + const now = "2026-08-10T23:12:00.000Z"; + + await store.registerSource({ + id: "x:public-watch", + kind: "x-webhook", + enabled: true, + updatedAt: now, + config: { + lane: "public-watchlist", + control: { + version: 1, + source: "x:public-watch", + kind: "x-webhook", + enabled: true, + lane: "public-watchlist", + expectedSubscriptionCount: 12, + eventTypes: ["post.create", "post.delete"], + configuredAt: now, + updatedAt: now, + runtime: { state: "ready", readyAt: now, revision: "x-activity-v2-webhook-v1" }, + upstream: { + registered: true, + valid: true, + webhookId: "2086948647707852801", + desiredSubscriptionCount: 12, + liveSubscriptionCount: 12, + subscriptionsConverged: true, + checkedAt: now, + }, + }, + }, + }); + + expect(await buildSourceHealth(store, Date.parse(now))).toEqual([ + expect.objectContaining({ + source: "x:public-watch", + kind: "x-webhook", + configured: true, + enabled: true, + status: "unknown", + hasActivityEvidence: false, + control: { + lane: "public-watchlist", + runtime: expect.objectContaining({ state: "ready", current: true }), + upstream: expect.objectContaining({ + registered: true, + valid: true, + liveSubscriptionCount: 12, + desiredSubscriptionCount: 12, + subscriptionsConverged: true, + current: true, + }), + }, + }), + ]); + }); }); async function appendConnectorEvent(