Running an experiment #

One idea, one branch, one comparison in the pull request.

git checkout -b claude/flank-weighting origin/main
# change the bot
./scripts/build.sh
./sds.sh bench --games 60 --against main

The last command prints a comparison and writes comparison.md into the run directory. That file is the PR body.

The baseline #

baselines/*.json are committed. They hold an aggregate — games, decided, wins, BV left, the commit they were measured at — and never the match files, which are large, derived, and regenerated by any run.

Record one when main moves:

./sds.sh bench --games 60 \
    --save-baseline main --notes "after the terrain-cost fix"

Then commit it. A baseline that lives on one machine cannot be compared against in a review.

What the comparison refuses to do #

Compare across different rules. Scenario, victory condition, round limit and opponent travel with the baseline, and a mismatch is an error rather than a warning. A baseline recorded at a 50% BV threshold and a run at 70% produce numbers of the same shape that mean different things, and nothing in a diff would show it.

Read a difference that is not there. Every comparison carries a 95% interval on the difference in win rate — Newcombe's method, which stays sane at rates near 0 and 1 where the normal approximation does not — and states the verdict in a sentence:

The interval spans zero, so this run does not show a difference. An effect this size would need roughly 380 decided games per side to separate from noise.

That sentence is the point of the whole apparatus. 12–8 against 10–10 looks like progress and is nothing, and a table of numbers invites reading the one you hoped for.

How many games #

Rough, for a mirror match:

A match is 1–4 minutes, and --jobs matches run at once. Sixty games is about half an hour at --jobs 3 and is enough to catch a large change; a subtle one needs an overnight run and there is no way around that.

Watch the undecided count. Games that hit the round limit are excluded from the rate, so 60 games with 20 undecided is a 40-game experiment.

What the units actually did #

python3 -m sds.cli moves runs/<run>

A win rate says which side won. It cannot say that our machines spent 44% of their unit-rounds in the hex they started in, which was the state of the bot when this was first measured. "Did not change hex" is not one thing, and the report splits it: moved, held position, stood up, moved while prone, stayed prone, went prone, shut down, crew out, immobile. Princess is the column beside it, over the same suite.

The last three are not decisions. A shut-down machine, one whose crew is dead or unconscious, and one MegaMek calls immobile all took no turn at all, so they are counted apart and a second table repeats the choices over the decisions alone. Two more states cannot be seen in a position log and are printed from the seat's own movement tally: illegal and failed are orders the host refused, defaulted is the bot failing to answer, and all three stand the unit still in a hex it did not choose to stay in.

A run recorded before immobile reached the round snapshot cannot be split fully. The report says so and marks the affected rows as ceilings rather than printing a number that is quietly too high.

Before you believe any of it #

./sds.sh control --games 40

Princess against itself must come out near 50/50. If it does not, the harness is biased and every number it has printed is suspect. Run it after touching the harness, not after being surprised by a result.

What belongs in the PR #

• comparison.md, as the body.
• What was changed and why you expected it to help — before the numbers, so the reader can tell a prediction from a rationalisation.
• The answered / decisions split if the bot's own counters moved. A change that improves the win rate by making the bot answer more of its decisions is a bug fix wearing a strategy's clothes, and worth naming as one.
• Any run that was thrown away, and why. A benchmark you can re-roll is not a benchmark.