Prime Agent capability evals #

These harnesses measure end-to-end agent capability with a real model. They complement the repo's unit tests (which never call a model) and the PR performance benchmarks (which measure startup, not capability). They never run in CI and never need credentials to validate: every harness is exercised by model-free self-tests, and a real-model run is a manual step.

swarm-fanout #

Measures multi-agent orchestration under one prompt: decompose a job into per-shard child tasks, fan out one RLM subagent per shard, fan the answers back in, and account for child failures. Everything is scored from on-disk artifacts - the RLM spawn ledger, sub-* child session dirs, the parent session transcript, and the combined answer artifact - never from the agent's own claims.

1. Fixtures (swarm_fanout/fixtures/) are two self-contained sets (JSONL event logs, CSV host metrics) of eight stdlib-only data shards each, with a machine-checkable question and expected answer per shard. task.txt is the orchestration prompt; fixture.json is the scoring manifest and stays out of the agent's repo copy.
2. Runner (swarm_fanout/runner.py) copies a fixture to a temp git repo, runs the parent agent headless (--mode json, --daemon-socket, --session-dir, isolated agent home), then records the post-state. The ledger path is computed exactly as the product does: <agentHome>/rlm-ledger/<sha256(canonical sessions dir)[:16]>.jsonl.
3. Scorer (swarm_fanout/scorer.py) applies the rubric:
   • coverage: every shard's answer in combined-index.md matches the machine-computed expected value;
   • delegation evidence: a live depth-1 ledger edge for every shard's worker name, each edge backed by its real child session dir (the edge's recorded child file must exist in the collected sub-* dirs) - helper-named spawns do not substitute, and a parent that answers everything itself fails here;
   • dedup: no shard's worker name is spawned twice while another child for that shard is live, and total depth-1 spawns stay within one retry per shard (2x the shard count);
   • receipts: every depth-1 child has a parent-visible reply or an explicit failure/terminal notice in the parent transcript;
   • efficiency: tokens, turns, and wall time (informational until a single-agent baseline datapoint exists).

A run counts as resolved only when coverage, delegation, dedup, and receipts all pass.

Running a real-model eval #

The eval needs a model selector and provider credentials in the environment (the agent's normal auth):

cd scripts/evals/swarm_fanout
uv run --locked python runner.py --fixture fixtures/json-events --model anthropic/claude-sonnet-4-5
uv run --locked python runner.py --fixture fixtures/csv-metrics --model anthropic/claude-sonnet-4-5

The runner prints the scored outcome as JSON and exits 0 only when the run is resolved. --timeout (default 1800s) bounds the agent run.

Real-model runs should be planned against one constraint: a one-shot --mode json session is client-owned, and when the CLI process exits the daemon stops its worker (after a 30s grace), taking still-running children with it. The task prompt therefore requires the parent to keep its run active until every child has replied. If a real-model datapoint shows models reliably ending their turn before the fan-in completes, drive the parent as a resident session over the daemon protocol (create + prompt + wait_for_headless_completion with waitForRlmQuiescence: true) instead and re-check; the scorer needs no changes either way, since it reads only the artifacts.

Self-tests #

Validate the harness with the model-free self-tests (fixture integrity with independently recomputed answers, every scorer rule plus the no-spawn, double-spawn, fabricated-answer, and silent-drop cheats, ledger-path computation, and a stub-agent end-to-end run) - never a model call:

uv run --locked ruff check .
uv run --locked ruff format --check .
uv run --locked python -m unittest discover -s tests -v

Adding a fixture #

Copy an existing fixture directory, keep the shape: shards/ (data files with machine-checkable answers), fixture.json (name, artifact, shards with file/worker/question/expected), and task.txt (orchestration prompt naming every shard and its question). Extend the fixture integrity tests with an independent recompute for the new questions.