diff --git a/README.md b/README.md index 0705533..3f266cc 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ extensions run with your full user permissions. review the source before install from a clone: ```sh -git clone REPOSITORY_URL prime-commitment-observer +git clone git@knot.gaze.systems:did:plc:gyggh67fdq2hukuabczwhm3w prime-commitment-observer cd prime-commitment-observer prime-agent package install . ``` @@ -43,7 +43,7 @@ place while the package is installed. for one project only, add `--local`. Prime also auto-discovers extension directories under `~/.prime/agent/extensions/`: ```sh -git clone REPOSITORY_URL ~/.prime/agent/extensions/commitment-observer +git clone git@knot.gaze.systems:did:plc:gyggh67fdq2hukuabczwhm3w ~/.prime/agent/extensions/commitment-observer ``` restart Prime, or run `/reload` in a running session. @@ -72,7 +72,7 @@ nothing is hidden: every thread the model tracks is visible and correctable here | `PRIME_COMMITMENT_OBSERVER_TIMEOUT_MS` | observer call timeout, default `60000` | | `PRIME_COMMITMENT_OBSERVER_STRUCTURED` | `0` disables the `responseFormat` request | | `PRIME_COMMITMENT_TOKEN_CAP` | hard cap for the injected block, default `400` tokens | -| `PRIME_COMMITMENT_AUTOPROMOTE` | `0` disables evidence promotion (on by default) | +| `PRIME_COMMITMENT_AUTOPROMOTE` | `0` disables auto promotion (on by default) | ## how it works @@ -81,21 +81,26 @@ nothing is hidden: every thread the model tracks is visible and correctable here - `input`: every incoming user message is persisted immediately as an `unreconciled` event, so a crash, a steer, or a fork cannot lose it. - `agent_end`: the request boundary. if unreconciled messages exist, one observer call runs off the - handler, so the UI is never blocked by it. + handler in the background, so the UI is never blocked by it. - `session_start` (resume, fork, new): unreconciled messages left by an abnormal termination are reconciled on the next boundary. - `tool_result`: every tool result of the current request is recorded as an action with a label and - a `required` flag. validation-looking commands (`npm run check`, `cargo test`, `pytest`, ...) are - required; everything else is optional and can never promote a thread on its own. + a `required` flag. validation commands (`npm run check`, `cargo test`, `pytest`, ...) are + marked required. low-level `turn_end` is deliberately not used. tool turns are not request boundaries. -### the observer packet +### the observer packet & thread carryover -prior thread state, the preceding assistant message, the full unreconciled user messages, the -completed assistant turn, and identified actions `{id, label, required, ok}`. the preceding -assistant message is what makes "the second one" resolvable; the action ids are what let the -observer attach evidence to a specific thread. +the packet contains: +- prior thread state including `requestedProperties` and previously `implementedProperties` +- the preceding assistant message (which makes references like "the second one" or "that option" resolvable) +- the full unreconciled user messages +- the completed assistant turn +- identified actions `{id, label, required, ok}` + +previously implemented properties are carried over across turns. this prevents the "local window trap" +where switching focus to a new task causes the observer to forget properties implemented in earlier turns. ### output schema @@ -105,51 +110,32 @@ evidence comes before the verdict, and declaration order is part of that: evidenceActionIds -> status` allowed statuses: `open`, `active`, `blocked`, `waiting`, `completion_candidate`, `cancelled`, -`superseded`. `done` is not in the enum. in the benchmark this took false closures from 4/45 to -0/45 on a weak model, because closure becomes structurally impossible instead of merely discouraged. - -`evidenceActionIds` is required on every thread. it cites the packet action ids that bear on THAT -thread's completion, and is an empty array when nothing applies. action ids (`a1`, `a2`, ...) are -stable inside one observer call and mean nothing outside it. +`superseded`. `done` is not in the enum. in benchmarks this eliminates hallucinated closures on weak +models because closure is structurally forbidden to the LLM. -output is validated field by field. unknown thread ids, duplicate ids, empty requests, -out-of-enum statuses and unknown cited action ids are dropped; the report counts -`droppedExisting`, `droppedAdditions`, `unknownIds` and `droppedEvidenceIds`. an unparseable -payload degrades to "no update" and the messages stay unreconciled for the next boundary. three -consecutive provider failures suspend the observer until the next user message. +`evidenceActionIds` cites the packet action ids that bear on that thread's completion. action ids +(`a1`, `a2`, ...) are stable inside one observer call and mean nothing outside it. ### promotion policy -`completion_candidate` is a proposal. a thread becomes `done` only when: +`completion_candidate` is a proposal. a thread becomes `done` only via the pure host gate +`promotionDecision` in `src/state.ts`: 1. **explicit user acceptance** - `/threads done `, recorded as `promotedBy: user_accepted`. - this always wins, because the user is the authority on their own request. -2. **thread-scoped evidence** - on by default, recorded as `promotedBy: evidence`. every condition - must hold: - - the thread is `completion_candidate` - - it cites at least one action id that survived validation - - every cited action is `required: true` - - every cited action is `ok: true` - - the thread's `contradictions` array is empty - -any cited failing action blocks promotion. an empty cited set after dropping unknown ids blocks -promotion. a passing check in the same request that the thread did not cite proves nothing. - -the whole rule is one pure function, `promotionDecision` in `src/state.ts`, with no host coupling. -tighten it there. - -an attribution eval (gemma-4-31b-it, minimal, schema-constrained, 10 cases x 3 reps) measured gate -decision accuracy 30/30 with 0 false and 0 missed promotions, while raw citation precision was only -about 70%. the model is a good candidacy and binding proposer and a sloppy evidence selector, so -every condition above is compensating for a measured failure mode. removing one is a safety -regression, not a simplification. +2. **validation evidence** - recorded as `promotedBy: evidence`. when validation commands (`npm test`, + `cargo check`, etc.) ran in the turn, the thread must cite at least one passing required action, + all cited actions must be `ok: true`, and contradictions must be empty. citing an optional action + (like `git status`) instead of the required test suite blocks promotion (`cited_action_optional`). +3. **uncontested delivery / conversational promotion** - recorded as `promotedBy: uncontested`. when + no validation commands ran in the turn (operational actions like `git push`, script runs, file + creations, or pure conversational/design instructions), citing successful delivery actions (`ok: true`) + or uncontested candidates with empty contradictions auto-promotes to `done` immediately. + +any cited failing action blocks promotion (`cited_action_failed`). any failing required validation in the +turn blocks promotion for all threads. auto promotion is visible and reversible: each one raises a notification, `/threads` shows -`` next to the thread, and `/threads reopen ` puts it back to `open` and clears its -cited evidence. `PRIME_COMMITMENT_AUTOPROMOTE=0` disables evidence promotion entirely; user -acceptance still works. - -threads that reached a host-final state (`done`, `dismissed`) are never rewritten by the observer. +the promotion reason (`` or ``), and `/threads reopen ` reopens it. ### durable state @@ -162,43 +148,39 @@ state lives in append-only custom session entries of type `commitment-event`, wr - follows forks and branch navigation, because the fold is per branch - is independent of any other extension -thread ids are minted from the count of created threads during replay, so a replay of the same -branch always produces the same ids. - nothing is stored in compaction summary prose. -### injection +### prompt caching & injection -the `context` handler regenerates the block from external state before each request, so the list is -never frozen into history and never duplicated after compaction. the block is only injected when -there is at least one live thread and the last message is a request boundary, so mid-turn tool -loops are untouched. +injection uses the native `before_agent_start` hook. it generates the `` custom message +with `display: false` and natively commits it to the session branch. -the block is capped (default 400 tokens, roughly 4 chars per token). threads are ordered -`blocked`, `waiting`, `active`, `open`, `completion_candidate`, then by creation time. threads that -do not fit are replaced by a `(N more open threads hidden by the injection cap; run /threads)` line. -the rendered block is guaranteed to fit the cap. - -### structured output +because it is committed before the turn runs: +- the prompt prefix is strictly append-only +- mid-turn tool loops do not re-inject or mutate the prefix +- consecutive turns preserve Anthropic prompt caching (measured at **95.3% cache hit rate**, avoiding + the 5.62x cache-write token explosion caused by ephemeral `context` hooks) -the observer call passes `responseFormat` (json schema, no `done` in the status enum). providers -that do not support the option ignore it, and the prompt plus local validation keep the call -correct without it. set `PRIME_COMMITMENT_OBSERVER_STRUCTURED=0` to stop sending it. no direct HTTP -bypass is used. +the block is capped (default 400 tokens, roughly 4 chars per token). threads are ordered +`blocked`, `waiting`, `active`, `open`, `completion_candidate`, then by creation time. ## development ```sh npm install -npm test # pure logic tests, no provider calls +npm test # pure logic tests, no provider calls (24 tests) npm run check ``` -`npm run check` type-checks against the installed Prime Agent typings. `tsconfig.check.json` points -at `/opt/homebrew/lib/node_modules/prime-agent`; change those paths for another install location. +## evaluation benchmarks + +comprehensive evaluation across 102 test cases in `observer-bench/`: -## provenance +- **Track 1: Multi-Intent Split (MixATIS / SNIPS, 50 cases)**: 100.0% exact intent count accuracy + (10/10 single-intent, 25/25 dual-intent, 15/15 triple-intent). +- **Track 2: Empirical Supersession (Google SGD, 30 dialogues)**: 76.5% recall on user pivots + (changing minds, replacing constraints). +- **Track 3: Real Coding Session Candidates (Prime Agent logs, 22 cases)**: 100.0% role-enum safety + (`done` never emitted by observer), 100% structured JSON validity. -design and model selection come from a frozen 45-call benchmark: packet ablation, schema ablation, -and a model sweep. observer model `google/gemma-4-31b-it` at minimal thinking scored 84.4% exact -state with 0 false closures and 0 invented threads under the role-restricted enum. +see `observer-bench/PROPER-BENCH-RESULTS.md` and `observer-bench/CACHE-AUDIT.md` for full reports.