diff --git a/plan/training.md b/plan/training.md index 5c1e32d..b9ecac1 100644 --- a/plan/training.md +++ b/plan/training.md @@ -547,6 +547,48 @@ that row order does not move the answer at all. `sds corpus save` scans the same way, for the same reason. +## The displaced rows move the fit more than the damage model does + +`--drop-displaced` leaves out decisions where the weights' first choice was +taken by another unit before the force reconciled, so `chosen` is what was left +rather than what was preferred. This epic already records that dropping them +moved the offence-to-defence balance sharply toward defence on the 300-match +corpus. **It reproduces on an independent corpus, recorded three days later +against a corrected damage model.** + +| fit | corpus | offence:defence | +|---|---|---| +| `fitted-r11` | 300 matches, old damage model | 2.72:1 | +| `fitted-r12-nodisp` | same corpus, displaced dropped | **0.59:1** | +| `fitted-r13` | 240 matches, corrected grouping | 1.84:1 | +| `fitted-r13-nodisp` | same corpus, displaced dropped | **0.82:1** | + +Ratios on `compare-weights.py`'s definition - only terms still pointing their +family's way, wrong-way terms named and excluded - which is not the computation +that produced the 1.69:1 and 0.40:1 recorded earlier in this epic. Read the +columns against each other, not across. + +**The comparison worth making is which change moves more.** The corrected damage +model took the balance from 2.72:1 to 1.84:1. Dropping the displaced rows took +it from 2.72:1 to 0.59:1 on one corpus and 1.84:1 to 0.82:1 on the other. **A +bookkeeping decision about which rows are evidence outweighs the damage model +correction, on both corpora, by roughly a factor of two.** + +One concrete instance of the same thing: `incoming_damage` comes out **+0.0087** +in `fitted-r13`, which is the wrong sign and is warned about - and **-0.0336** +in `fitted-r13-nodisp`, which is the right one and is not. The rows where the +bot did not get the hex it asked for are, on their own, enough to make defence +price backwards. + +- [ ] This is now two corpora agreeing. `--drop-displaced` should stop being a + flag somebody remembers and become either the default or a documented + refusal to fit without a decision. A 20% subset of rows that inverts the + central balance of the vector is not a nuisance correction +- [ ] The two arms still are not rankable - `r_squared` 0.2577 against 0.2408 + and agreement 0.4830 against 0.4426 are measured over 3.85M rows against + 2.79M. The direction of the weight change is the finding; the fit + statistics cannot say which vector is better + ## For jmm: which features should declare an expected sign `EXPECTED_SIGNS` covers the **firing family only** - `expected_damage`,