diff --git a/.claude/skills/run-benchmark/SKILL.md b/.claude/skills/run-benchmark/SKILL.md new file mode 100644 index 00000000..fd7c1dc2 --- /dev/null +++ b/.claude/skills/run-benchmark/SKILL.md @@ -0,0 +1,179 @@ +--- +name: run-benchmark +description: Run Prowl's performance benchmark layers for the current branch or a release, compare against recorded baselines, and write a persistent local report. Use only when the user explicitly invokes run-benchmark or explicitly asks to run performance benchmarks / verify performance impact. Do not invoke automatically after implementation work. Classify the user's intent first (hot-path change, new optimization, or release-level sweep), confirm it with the user, then run the matching layers. +--- + +# Run Benchmark + +## Invocation contract + +Run this skill only after an explicit user request. Implementing, fixing, or reviewing +performance-adjacent code does not by itself authorize a benchmark run. + +All timing in this workflow assumes **one dedicated, quiet local machine** accumulating a +series over time. Numbers from CI runners, other machines, or a loaded host are not +comparable and must not enter the baseline series. Never run other builds, tests, or +heavy agents concurrently with a timed run. + +Background and design decisions live in `docs-ai/057-performance-benchmark-suite/`; the +measurement methodology (why workload context matters, why load kills comparability) in +`docs-ai/032-performance-hardening/004-agent-detection-steady-state.md`. + +## Step 0 — Preflight + +1. `git status --short` — an A/B run switches the checkout to the baseline commit, so the + tree must be clean and HEAD committed. If dirty, tell the user what is blocked and ask + whether to commit/stash or fall back to a branch-only run. +2. Quiet-host check: compare `uptime` load1 against `sysctl -n hw.ncpu`. Above roughly + half the cores, warn that starvation inflates numbers unevenly and ask whether to wait + or proceed; above one runnable process per core, refuse the timed run and say why. +3. Note what is available for the live layer: is a Prowl instance running + (`prowl list --json`), and are any agents working (animating titles)? + +## Step 1 — Classify intent, then always confirm + +Gather evidence before guessing: + +- The user's own words in the invocation — these override everything below. +- `git branch --show-current`; `BASE=$(git merge-base HEAD origin/main)`; + `git diff --name-only $BASE` and `git log --oneline $BASE..HEAD`. +- Map changed files through the table in **Reference** to see which benchmark suites the + branch can plausibly affect. + +Classify into one of three modes: + +| Mode | Signal | What runs | +| --- | --- | --- | +| **hot-path** | Branch touches files mapped to existing suites; user wants to know whether the change regressed them | A/B `make bench`: merge-base baseline vs branch | +| **new-optimization** | Branch introduces a performance improvement no suite pins yet | Add a benchmark first (057 pattern), then the hot-path A/B | +| **release** | Pre-release or "how is performance overall" with no specific change in question | `make test` + `make bench` vs trailing series + live layer | + +Then **confirm with the user before running anything**, even when the evidence is clear: +present the guessed mode with its one-line evidence and the alternatives (AskUserQuestion +with the recommended option first). If the evidence is contradictory or absent, do not +guess — ask. Benchmark runs take minutes and switch the checkout; a wrong mode wastes +both. + +## Step 2 — Run the confirmed mode + +Every `make bench` appends records to `~/Library/Logs/Prowl/measurements/bench/bench.jsonl` +keyed by `git rev-parse --short HEAD`. Record the exact SHA before each run so the +comparison step can find its rows. + +### hot-path — A/B against the merge-base + +1. Check whether the baseline already exists on this machine: + `jq -c "select(.gitSHA == \"\")" bench.jsonl`. Reuse it if present. +2. If absent: `git switch --detach $BASE` → `make bench` → `git switch -`. The first + optimized build after a switch takes minutes; later ones are incremental. +3. `make bench` on the branch HEAD. +4. Compare per Step 3. + +### new-optimization — pin the claim before measuring it + +1. Check the **Reference** inventory: does any suite already cover the optimized path? +2. If not, the primary deliverable is a new benchmark following the 057 pattern, in the + same PR as the optimization: + - dig the replaced implementation out of git history and keep it **verbatim** as the + reference; + - assert output equivalence before timing; + - assert a floor ratio 2–4x below the measured Release ratio; + - gate Swift-vs-Swift ratios on optimized builds + (`BenchmarkMeasurement.isOptimizedBuild`); C-call/filesystem ratios assert everywhere. +3. Then run the hot-path A/B: the new suite quantifies the win, the existing suites catch + collateral damage. +4. The measured ratio goes into the report as the claim's durable evidence — this is + exactly what the #644–#665 wave lacked. + +### release — full sweep plus series + +1. `make test` — the behavioral + ratio layer (Debug). +2. `make bench` — compare against the trailing series (last ~5 records per case across + recent SHAs), not a single baseline. +3. Live layer, best-effort, report each as run/inconclusive/skipped: + - `make measure-titles` — needs a running app with at least one animating agent tab; + exit 3 means nothing was animating (inconclusive, not a pass). + - `make measure-cpu` — needs a running Prowl Debug instance; set `PROWL_PID` when + several run. Records agent mix and load next to the attribution, so quote them + together. +4. `make capture-spike` is a standing watch for long-running instances, not a step here; + mention it if the user is chasing an intermittent spike. + +## Step 3 — Compare and judge + +Extract both sides: + +```bash +jq -r 'select(.gitSHA == "SHA") | [.suite, .name, .referenceMilliseconds, .shippedMilliseconds, .ratio] | @tsv' \ + ~/Library/Logs/Prowl/measurements/bench/bench.jsonl +``` + +Judgment rubric (single machine, warm cache, medians of 15): + +- A ratio below its asserted floor already failed the test — confirmed regression, no + further interpretation needed. +- `shippedMilliseconds` against baseline: within ±20% is noise; 1.2–1.5x growth is a + **watch** — rerun `make bench` once to confirm before claiming anything; above 1.5x + sustained is a **regression**; above 2x is **major**. Name suspect commits via the + file→suite map. +- Use `referenceMilliseconds` as the built-in host canary: the reference workload never + changes, so if it moved by a similar factor the *host* moved, not the code; if shipped + moved alone, the code moved. +- Never average across load regimes. If load was elevated during a run, discard the run + and redo it — do not "correct" the numbers. + +## Step 4 — Persistent report + +Write a markdown report to +`~/Library/Logs/Prowl/measurements/reports/$(date +%Y%m%d-%H%M)-.md` +(`mkdir -p` the directory; it lives with the rest of the measurement home, survives +reboots, and is never cleaned by the system). Reports are append-only history — never +overwrite or delete earlier ones. + +Required sections: + +- **Intent** — the mode the user confirmed, in one line. +- **Context** — date, branch, baseline and branch SHAs, load average and core count at + run time, app/agent state if the live layer ran. +- **Results** — one table per layer: case, baseline shipped ms, branch shipped ms, delta, + ratio vs floor. +- **Verdict** — `no-regression` / `watch` / `regression` / `major`, with suspect commits + for anything above noise, and the host-canary observation. +- **Not run** — layers skipped or inconclusive, and why. +- **Raw refs** — the bench.jsonl SHAs and any script run directories. + +Reply to the user with the verdict and the report path. The report, not the chat, is the +durable record. + +## Reference + +Benchmark inventory (all under `supacodeTests/`, suite root `PerformanceBenchmarks`): + +| Suite / check | Pins | Floor | Asserts in | +| --- | --- | --- | --- | +| `LineCountScanBenchmarks` sparse | #644/#652 memchr scan vs `Data.reduce` | 3x | all builds | +| `LineCountScanBenchmarks` all-newline | #652 hybrid dense fallback | 2x | optimized only | +| `FingerprintNormalizeBenchmarks` | #650/#657 escape-absence guard | 1.3x | optimized only | +| `WorktreeDirectoryIndexBenchmarks` | #648/#655 index vs per-row scan | 2x | all builds | +| `SessionScoringBenchmarks` | #650/#657 warm fragment cache | 3x | all builds | +| `make measure-titles` (live) | #649/#656 title coalescing ≤ ~1/s | 1.25/s | running app | + +File → suite map for intent classification and suspect naming: + +| Changed path | Affected check | +| --- | --- | +| `supacode/Clients/Git/GitClient.swift`, `UntrackedLineCountCache.swift` | LineCountScan | +| `supacode/Infrastructure/AgentDetection/AgentSessionResolver.swift` | FingerprintNormalize, SessionScoring | +| `supacode/Domain/WorktreeDirectoryIndex.swift`, `supacode/Support/PathPolicy.swift` | WorktreeDirectoryIndex | +| `supacode/Features/Terminal/Models/TerminalTabManager.swift`, `WorktreeTerminalState+AgentDetection.swift` | measure-titles, coalescing/dedup unit tests | +| Sidebar / Active Agents views and reducers | counting tests only (no timing suite) | + +Caveats: + +- `referenceMilliseconds` values measured inside the Release test bundle are not + comparable to standalone `-O` binaries (~10x apart for `Data.reduce`); ratios are + unaffected. Compare bench.jsonl rows only with other bench.jsonl rows. +- The suite is `.serialized` and `make bench` disables parallel testing and pins the + destination arch deliberately — do not "speed it up" by removing either. +- `bench.jsonl` is append-only; the same SHA can have several rows (reruns). Prefer the + newest row per (suite, name, SHA) and note reruns in the report. diff --git a/docs-ai/057-performance-benchmark-suite/000-plan.md b/docs-ai/057-performance-benchmark-suite/000-plan.md index d3648daf..bba94544 100644 --- a/docs-ai/057-performance-benchmark-suite/000-plan.md +++ b/docs-ai/057-performance-benchmark-suite/000-plan.md @@ -103,4 +103,6 @@ optimized settings.) ## Amendments -(append `- Updated 2026-MM-DD: ... — see [00N-topic.md](00N-topic.md)` lines here) +- Updated 2026-08-02: added the explicit-invocation `run-benchmark` skill so agents can + run the layers with the right comparison and a persistent report — see + [002-benchmark-runbook-skill.md](002-benchmark-runbook-skill.md) diff --git a/docs-ai/057-performance-benchmark-suite/002-benchmark-runbook-skill.md b/docs-ai/057-performance-benchmark-suite/002-benchmark-runbook-skill.md new file mode 100644 index 00000000..4c61febe --- /dev/null +++ b/docs-ai/057-performance-benchmark-suite/002-benchmark-runbook-skill.md @@ -0,0 +1,28 @@ +# 057.002 — Benchmark Runbook Skill + +## Context + +The suite's value depends on being run at the right moments with the right comparison, +and that judgment was living only in one conversation. Benchmark monitoring is planned to +be delegated to agents, so the workflow needed a durable, explicitly-invoked home. + +## Change + +Added `.claude/skills/run-benchmark/SKILL.md`: an explicit-invocation-only runbook that + +- classifies the user's intent from the branch diff and their words into hot-path A/B, + new-optimization pinning, or a release-level sweep — and always confirms the guessed + mode with the user before running, since a run takes minutes and can switch the + checkout; +- runs the matching layers (`make bench` A/B against the merge-base, the 057 benchmark + pattern for new optimizations, `make test` + trailing-series + live checks for + releases); +- judges results with fixed noise bands (±20% noise, 1.5x regression, 2x major) and uses + the never-changing reference implementation as a host-noise canary; +- writes an append-only markdown report to + `~/Library/Logs/Prowl/measurements/reports/` — local to the one quiet machine the + series assumes, but persistent across sessions. + +## Refs + +- PR #668