notplants.net / agent-orchestrator-benchmark
wip: benchmarks for agent-orchestrator
Star
Fork 0
Watch

Commits on main

artifacts: add calculators/ — the 30 built calculators (5/variant) + machine-docs + git logs main
mfowler committed
chore: gitignore the runner's regenerated .data.hdr artifact
mfowler committed
docs: rewrite README — full study, how it works, points to FINDINGS (2026-06-16)
mfowler committed
docs: finalize deferred at N=5 (median 12.89M, ~tied with orig)
mfowler committed
docs: FINDINGS.md — benchmark synthesis; track raw results data
mfowler committed
feat: benchmark builder-adversary-deferred (4-phase incl review); limit-detect across all roots
mfowler committed
results: full 5-variant campaign complete (incl. builder-solo control)
mfowler committed
results: 4-variant campaign complete (5/5 each); analysis with ratios
mfowler committed
fix: veto check matches all-caps '## VETO <reason>', not '## Veto log' header
mfowler committed
chore: append-mode data file; engine -> c6c7ce8 (orig-based stateless/lean)
mfowler committed
feat: builder-solo control runner (run after campaign) + limit-detect for it
mfowler committed
fix: flag usage-limit-affected runs; correct tok/sec
mfowler committed
feat: analyze.py — efficiency ratios (tokens/LOC, tokens/sec, tokens/commit)
mfowler committed
feat: keep run repos + record commits/LOC per run
mfowler committed
feat: campaign mode — repeat each variant N times, aggregate distributions
mfowler committed
results: 4-way + the variance finding (N=1 is not enough)
mfowler committed
feat: add builder-adversary-lean variant; runner takes variant args
mfowler committed
results: full-harness 3-way (orig/min/stateless) on the calculator
mfowler committed
feat: add stateless variant, pre-trust work dirs, loop over 3 variants
mfowler committed
chore: gitignore the runner's transient .tmp file
mfowler committed
feat: multi-phase calculator problem + full-harness benchmark runner
mfowler committed
feat: agent-orchestrator-benchmark — prompt token comparison harness
mfowler committed