# Running an experiment

One idea, one branch, one comparison in the pull request.

    git checkout -b claude/flank-weighting origin/main
    # change the bot
    ./scripts/build.sh
    ./sds.sh bench --games 60 --against main

The last command prints a comparison and writes `comparison.md` into the run
directory. That file is the PR body.

## The baseline

`baselines/*.json` are committed. They hold an aggregate — games, decided, wins,
BV left, the commit they were measured at — and never the match files, which are
large, derived, and regenerated by any run.

Record one when `main` moves:

    ./sds.sh bench --games 60 \
        --save-baseline main --notes "after the terrain-cost fix"

Then commit it. A baseline that lives on one machine cannot be compared against
in a review.

## What the comparison refuses to do

**Compare across different rules.** Scenario, victory condition, round limit and
opponent travel with the baseline, and a mismatch is an error rather than a
warning. A baseline recorded at a 50% BV threshold and a run at 70% produce
numbers of the same shape that mean different things, and nothing in a diff
would show it.

**Read a difference that is not there.** Every comparison carries a 95% interval
on the *difference* in win rate — Newcombe's method, which stays sane at rates
near 0 and 1 where the normal approximation does not — and states the verdict in
a sentence:

> The interval spans zero, so this run does not show a difference. An effect this
> size would need roughly 380 decided games per side to separate from noise.

That sentence is the point of the whole apparatus. 12–8 against 10–10 looks like
progress and is nothing, and a table of numbers invites reading the one you hoped
for.

## How many games

Rough, for a mirror match:

| difference you want to see | decided games per side |
|---|---|
| 20 points | ~25 |
| 10 points | ~100 |
| 5 points | ~385 |

A match is 1–4 minutes, and `--jobs` matches run at once. Sixty games is about
half an hour at `--jobs 3` and is enough to catch a large change; a subtle one
needs an overnight run and there is no way around that.

Watch the undecided count. Games that hit the round limit are excluded from the
rate, so 60 games with 20 undecided is a 40-game experiment.

## What the units actually did

    python3 -m sds.cli moves runs/<run>

A win rate says which side won. It cannot say that our machines spent 44% of
their unit-rounds in the hex they started in, which was the state of the bot
when this was first measured. "Did not change hex" is not one thing, and the
report splits it: moved, held position, stood up, moved while prone, stayed
prone, went prone, shut down, crew out, immobile. Princess is the column beside
it, over the same suite.

The last three are not decisions. A shut-down machine, one whose crew is dead
or unconscious, and one MegaMek calls immobile all took no turn at all, so they
are counted apart and a second table repeats the choices over the decisions
alone. Two more states cannot be seen in a position log and are printed from
the seat's own movement tally: `illegal` and `failed` are orders the host
refused, `defaulted` is the bot failing to answer, and all three stand the unit
still in a hex it did not choose to stay in.

A run recorded before `immobile` reached the round snapshot cannot be split
fully. The report says so and marks the affected rows as ceilings rather than
printing a number that is quietly too high.

## Before you believe any of it

    ./sds.sh control --games 40

Princess against itself must come out near 50/50. If it does not, the harness is
biased and every number it has printed is suspect. Run it after touching the
harness, not after being surprised by a result.

## What belongs in the PR

- `comparison.md`, as the body.
- What was changed and why you expected it to help — before the numbers, so the
  reader can tell a prediction from a rationalisation.
- The `answered` / `decisions` split if the bot's own counters moved. A change
  that improves the win rate by making the bot answer more of its decisions is a
  bug fix wearing a strategy's clothes, and worth naming as one.
- Any run that was thrown away, and why. A benchmark you can re-roll is not a
  benchmark.