From 928e857fc2e65f3d62d221e43f63ea77cd6b05ca Mon Sep 17 00:00:00 2001 From: "@permadeath.com" Date: Thu, 20 Aug 2026 13:49:12 -0400 Subject: [PATCH] docs(candidates): scenario benchmarks that need no MegaMek Hand-built boards and two units, asserting on counts rather than on which state wins: state count, survivors per prune, distinct cache keys are deterministic, while the winner depends on weights and would break on every tune. The already-moved scenario is a discriminating test - damage-max and mission-kill-max must pick different states, which is the guard against the duplicate column that shipped today. --- plan/candidates.md | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/plan/candidates.md b/plan/candidates.md index be09785..1f6bded 100644 --- a/plan/candidates.md +++ b/plan/candidates.md @@ -166,6 +166,43 @@ good offence and bad defence, and a single operator over `N` cannot say so. micro-benchmark, deliberately, because a 24-game bench costs twenty minutes and carries +/- 25 points of noise +## Scenario benchmarks, without MegaMek + +Hand-built boards and two units, in code. No jar, no container, no match. The +point is that **the counts are deterministic** - given a board and two units, +the number of reachable states, the number of pairs surviving each prune, and +the number of distinct cache keys are fixed. Those are what a regression moves. +What is *not* fixed is which state wins, because that depends on weights and on +the minimax exponent, so asserting on it would break every time a weight is +tuned and would be testing the wrong thing. + +Assert on counts and on ordinal relationships. Time separately, as a benchmark +rather than a test, because wall-clock assertions are flaky under load. + +- [ ] **`they-have-not-moved`**: one enemy behind us, cover on the board, their + movement still to come. Asserts the minimax path end to end - `M` is a + state set rather than a point, both aggregations run, and a state that is + merely good on average loses to one that is good against most of `M`. + Assert: state count, pairs surviving each prune, and that lowering the + power-mean exponent moves the answer toward cover +- [ ] **`they-have-already-moved`**: the enemy is placed, so `M` collapses to a + point. Two questions from one board: which state maximises expected + damage, and which maximises the chance of a mission kill. **They must be + different states.** If they are not, either the board does not + discriminate or the two measures are the same column wearing two names - + which shipped today, with `p_kill` and `p_mission_kill` identical on 92.8% + of a real corpus. This test is the guard against a repeat +- [ ] **`nothing-bears`**: an enemy directly behind, weapons forward only. + Asserts the arc prune - almost every `L` facing away should drop before + any statistics are computed, and one twist should recover exactly one + hexside of them +- [ ] **`walled-off`**: an enemy behind a ridge with no line at any range. + Asserts the LOS prune drops the whole `M` set, and that the volley + estimator is never entered +- [ ] Each scenario records its counts in a committed snapshot, so a change in + the number of states or survivors fails loudly and has to be explained + rather than absorbed + ## Deferred **Jump** is a second pathfinding pass, not a variant of the first: a jump cannot -- 2.51.2