From 906f3a0fe6d80e4e0c9980aba9e7fb06065ba282 Mon Sep 17 00:00:00 2001 From: onevcat Date: Sat, 29 Aug 2026 11:03:54 +0900 Subject: [PATCH] Record the T0 version attestation slice in docs-ai 064 Document the attestation schema, where each attested version comes from (all eight from the S3c live acceptance, with the date and Claude ambiguities spelled out), the decisions behind one generic parser and the interactive-shell PATH fallback, and the actual `make agent-versions` output from this Mac; amend the 064 plan accordingly. T1 (headless contract tests) stays in R2b. Claude-Session: https://claude.ai/code/session_01YSXSCVNoycSbCmwPMvcCnw --- .../064-agent-completion-signals/000-plan.md | 6 + .../015-t0-version-attestation.md | 144 ++++++++++++++++++ 2 files changed, 150 insertions(+) create mode 100644 docs-ai/064-agent-completion-signals/015-t0-version-attestation.md diff --git a/docs-ai/064-agent-completion-signals/000-plan.md b/docs-ai/064-agent-completion-signals/000-plan.md index dc488be2..55f61b9e 100644 --- a/docs-ai/064-agent-completion-signals/000-plan.md +++ b/docs-ai/064-agent-completion-signals/000-plan.md @@ -246,6 +246,12 @@ opencode; partial for qodercli/qwen/amp; docs/bundle for the rest). Key conclusi ## Amendments +- Updated 2026-08-29 (#726 T0): [agent-attestation.json](agent-attestation.json) now records the + version each tier-A runtime last passed a live sweep against (all eight from + [011-s3c-action.md](011-s3c-action.md)); `make agent-versions` compares the installed binaries + against it and warns on newer builds, and the research matrix's tier-A line is generated from + the record with `make test-scripts` guarding drift. See + [015-t0-version-attestation.md](015-t0-version-attestation.md). T1 stays in R2b. - Updated 2026-08-29 (063 B1 kickoff): 063's `expect` activations are records in this entry's dispatch store (`launch` via the S2 prompted-launch path, `message` via #733's re-dispatch), and S5's watchdog part ships with 063-B2 instead of D2. #733 therefore lands before 063-B3. diff --git a/docs-ai/064-agent-completion-signals/015-t0-version-attestation.md b/docs-ai/064-agent-completion-signals/015-t0-version-attestation.md new file mode 100644 index 00000000..1ff3bb00 --- /dev/null +++ b/docs-ai/064-agent-completion-signals/015-t0-version-attestation.md @@ -0,0 +1,144 @@ +# 064.015 — T0 Version Attestation: Plan and Action + +## Status + +Implemented from `feat/agent-version-attestation` as the T0 half of +[#726](https://github.com/onevcat/Prowl/issues/726) (R2a in the shared +[release plan](../063-agent-workflows/release-plan.md)). T1 — headless contract tests against +the real binaries, `make test-agent-contracts` — is a later slice and is not started here. + +## Scope + +The managed hooks shipped by S3 wave 1 ([007](007-s3a-action.md), [009](009-s3b-action.md), +[011](011-s3c-action.md)) are a contract with eight external binaries, and three drifts found +while closing S3b (Droid's `droid exec` engine child, Codex 0.149.1's app-server EOF behavior, +Qoder's folder-trust gate on flag hooks) were invisible to the unit suite. T0 makes the +*version* side of that contract explicit and cheap to check: + +- an attestation record that says, per tier-A runtime, which version the contract last passed a + live sweep against, when, and where that is recorded; +- `make agent-versions`, which compares the binaries installed on this Mac with the record; +- the research matrix's tier-A baseline derived from the record instead of hand-edited. + +Non-goals: running any agent (T1), the optional scheduled npm/brew "latest" check from the issue +(nothing in the repo runs on a schedule yet; revisit with T1), and touching the interactive E2E +sweep. + +## The record + +`agent-attestation.json` (next to the research matrix), schema 1: + +| Key | Meaning | +| --- | --- | +| `schema` | `1`; the loader rejects anything else | +| `description` | free text for readers of the file | +| `runtimes[]` | one object per tier-A runtime, in the S3 order claude, codex, copilot, droid, qodercli, pi, omp, opencode | +| `runtimes[].runtime` | `AgentProfileRuntime` raw value (`qodercli`, not `qoder`) | +| `runtimes[].name` | display name | +| `runtimes[].binary` | executable looked up on PATH | +| `runtimes[].version_command` | argv that prints the version, e.g. `["claude", "--version"]`; the first element must be the binary | +| `runtimes[].attested_version` | the version the live sweep passed against; must parse as `MAJOR.MINOR.PATCH[-pre]` | +| `runtimes[].attested_on` | ISO date of that sweep | +| `runtimes[].record` | the `docs-ai/064` file that documents the sweep; must exist next to the JSON | + +Every key is required and no other key is allowed, so a typo fails `make test-scripts` rather +than silently going unread. Update rule until T1 exists: after a live sweep that passes for a +runtime, set its `attested_version` / `attested_on` / `record` by hand and run +`scripts/agent_versions.py --write-matrix`. T1's `make test-agent-contracts` is meant to do the +same on a passing run. + +### Attested versions and their provenance + +All eight entries point at the S3c live acceptance in [011-s3c-action.md](011-s3c-action.md), +the last sweep that exercised every tier-A runtime through a Prowl-launched Profile: Pi 0.84.3, +Oh My Pi 18.0.6, OpenCode 1.18.23, plus the "regression on the same build" row for Claude +2.1.245, Codex 0.149.1, Copilot 1.0.80, Droid 0.203.0, and Qoder 1.1.29 (PASS ×5). Two +ambiguities, resolved as follows: + +- 011 carries no explicit date. `attested_on` is 2026-08-26, the day PR #728 was opened with that + record (merged 2026-08-27); the same five versions had already passed 009's upgrade + re-verification on 2026-08-25, so the date is bounded either way. +- [013](013-idle-evidence-fallback.md) exercised Claude live again on 2026-08-28/29 but never + states the binary version, so Claude stays at the last explicitly verified 2.1.245 rather than + the 2.1.251 installed when this record was written. + +## Decisions + +| Decision | Rejected alternatives | +| --- | --- | +| One generic semantic-version parser (first `MAJOR.MINOR.PATCH[-pre]` token in stdout, else stderr, ANSI stripped) pinned by tests to the verbatim banner each of the eight CLIs printed on 2026-08-29. | Per-CLI parsers keyed by runtime: more code to maintain for no measured gain — every banner (`2.1.251 (Claude Code)`, `codex-cli 0.149.1`, `GitHub Copilot CLI 1.0.80.` + an update hint, `omp/18.0.6`, bare versions) yields the right token with one pattern, and the tests fail loudly if a banner changes. | +| Binary lookup on the process PATH first, then once on the PATH of `$SHELL -lic` for anything still missing; a version command runs with the PATH it was found on. | Process PATH only: `make` run from an editor or agent harness misses Homebrew and `mise` tools (measured on this Mac: a bare PATH plus `zsh -lc` sees 9 entries and no `/opt/homebrew/bin`; `zsh -lic` sees 34 with Homebrew and mise). `mise which` per binary: only covers mise, and none of the eight is mise-managed here. Login-only (`-lc`): skips `.zshrc`, which is where Homebrew's `shellenv` and `mise activate` live on a stock macOS setup. | +| A missing binary is a warning; nothing fails without `--strict`. `--strict` fails on any status but `attested`. | Failing on missing by default: a machine without every agent installed is normal, and the pre-release use is "look at the table", not a gate. | +| The matrix keeps its dated re-attestation paragraphs as history and gains one generated `**Tier-A attestation**` line; `--check-matrix` compares that line with the record and prints a per-runtime diff, `--write-matrix` regenerates it, and `make test-scripts` runs the check. | Rewriting the research paragraphs from the record: they are evidence about specific dates and would lose meaning. Deleting the line and pointing at the JSON only: readers of the matrix would have to open a second file for the one number they ask most often. | +| No `docs/` change: `make agent-versions` is a maintainer tool, so it is listed in `CLAUDE.md`'s build commands and here, not in the agent-facing manual. | — | + +## Delivered behavior + +- `docs-ai/064-agent-completion-signals/agent-attestation.json` — the record above. +- `scripts/agent_versions.py` + `make agent-versions` (`AGENT_VERSIONS_ARGS="--json"`, + `"--strict"`, `"--check-matrix"`, `"--write-matrix"`, `"--timeout N"`, `"--no-login-shell"`): + prints `runtime / attested / installed / status` with status one of `attested`, `newer`, + `older`, `missing`, `unparseable`; `newer` warns with a hint to run `make test-agent-contracts` + (T1) and update the record on a pass; `missing` and `unparseable` warn with the resolved path + or the first output line; the exit code is 0 unless `--strict`. `--json` emits, per runtime, + the attestation fields plus `path`, `resolution` (`path` / `login-shell`), `installed_version`, + `raw_output`, `status`, `detail`, and a `summary` count per status. +- `research-agent-completion-signals.md` — intro sentence naming the record as the source of + the tier-A versions, and the generated line. +- `scripts/test_agent_versions.py` (32 tests, run by `make test-scripts` and therefore `make + check`): parsers against the captured banners, comparison including prerelease ordering, the + record's shape and tier-A coverage, every status through a fake resolver/runner, the shell + PATH fallback through a stub shell, JSON and table shapes, strict exit codes, the matrix check + against the committed files and against tampered copies, and the script end to end with stub + binaries on a private PATH. + +## Verification + +- `make test-scripts`: 76 tests, OK (44 before this slice plus the 32 in `test_agent_versions.py`). +- `make agent-versions` on this Mac, 2026-08-29: + + ``` + runtime attested installed status + claude 2.1.245 2.1.251 newer + codex 0.149.1 0.149.1 attested + copilot 1.0.80 1.0.80 attested + droid 0.203.0 0.204.0 newer + qodercli 1.1.29 1.1.31 newer + pi 0.84.3 0.84.3 attested + omp 18.0.6 18.0.6 attested + opencode 1.18.23 1.18.23 attested + warning: claude 2.1.251 is newer than the attested 2.1.245 (011-s3c-action.md, 2026-08-26); the managed-hook contract is unverified against it — run `make test-agent-contracts` (#726 T1) and update docs-ai/064-agent-completion-signals/agent-attestation.json on a pass + warning: droid 0.204.0 is newer than the attested 0.203.0 (…) + warning: qodercli 1.1.31 is newer than the attested 1.1.29 (…) + ``` + + Three runtimes have moved past their attestation since the S3c sweep; that is exactly the + signal T0 exists to surface, and T1 is what clears it. +- `make agent-versions AGENT_VERSIONS_ARGS=--json`: summary `attested 5, newer 3, older 0, + missing 0, unparseable 0`; every binary resolved on the process PATH. +- `make agent-versions AGENT_VERSIONS_ARGS=--check-matrix`: `research-agent-completion-signals.md + matches agent-attestation.json`, exit 0. `--strict` exits 1 from the script (`make` reports 2). +- Shell fallback, with PATH reduced to `/usr/bin:/bin`: all eight resolved as `login-shell` + (`~/.local/bin` and `/opt/homebrew/bin`) with the same statuses; with `--no-login-shell` all + eight are `missing`. A login-only shell had found just the three under `~/.local/bin`, which is + what settled `-lic`. +- `make check`: format-changed (no Swift changes), format-lint, lint, test-scripts all pass. + +## What T1 builds on + +- The record is the place a passing `make test-agent-contracts` writes to: per runtime, set + `attested_version` to the version it just passed against, `attested_on` to today, and `record` + to the T1 record, then regenerate the matrix line. +- `scripts/agent_versions.py` exposes `load_attestation`, `parse_version`, `compare_versions`, + and `BinaryResolver`, so T1 can reuse the same binary lookup and version parsing to name the + exact build each contract ran against, and `--strict` gives a release gate once every runtime is + attested. +- The `newer` warning text already names T1's target so nothing has to change when it lands. + +## Observed but not changed + +- `/opt/homebrew/bin/droid` is a stale cask symlink to 0.134.0, shadowed by `~/.local/bin/droid` + 0.204.0 from Factory's installer. First-on-PATH wins here, as it does in a shell, so the report + shows 0.204.0; the cask is left alone. +- The release runbook has no pre-release checklist section to hang `make agent-versions` on; the + release skill and runbook are unchanged. -- 2.51.2