--- id: observability title: Read back what the bot believed and chose status: open dependsOn: [beliefs] exitCriterion: > A finished match can be stepped through turn by turn, showing what the bot believed and what it rejected. --- # observability The `rationale` field already carries each force's stance, what changed its mind, and every proposal it rejected. Nothing reads it. "The lance chose badly" and "no unit offered anything better" look identical in a record that keeps only the winner. `SdsClient` now appends every decision - the question, the answer, what the host did with it, and the rationale - to `-.decisions.jsonl` in the run directory. `sds view` renders one as a self-contained HTML page. - [x] Capture the rationale somewhere a tool can read it, rather than only in the container's stdout mixed with MegaMek's output - [x] Count the movement decisions that had nothing to choose. `prone` and `pronePinned` are per phase in a seat's stats and printed by `sds stats`: a prone unit the observation says cannot stand gets a menu one item long, so `decisions - pronePinned` is how much of a corpus carries a preference. Measured at 5.0% of our movement decisions over the 418-match run, and about half that once `canStand` became MegaMek's own answer. Split by reason - `pinnedGyro`, `pinnedNoMp`, `pinnedLimb` - because a rate that only halves reads as a fix that did nothing until you can see that what is left is a gyro, spent points, or the one limb state MegaMek really refuses. See [candidates](candidates.md) - [ ] Render a match turn by turn as HTML: threat map, rounds-remaining posterior, opponent doctrine posterior, per-target damage distributions, proposals kept and dropped - [x] proposals kept and dropped, with each one's value under the stance the force was actually holding, so a bad call is visibly different from a bad menu - [x] the force's stance as a bar, its dominant intent, and how many rounds it has held - [ ] threat map, rounds-remaining posterior, opponent doctrine posterior, per-target damage distributions - none of these exist yet; they arrive with `beliefs`, which is what this epic depends on - [x] Publish it as an artifact so a tab can be left open while playing - [x] `--watch` regenerates the file and the page reloads itself, which is the offline half of this. A `file://` page cannot fetch its own siblings, so polling a data file was not available - [x] `sds watch` is the online half: a loopback HTTP server that tails the log by byte offset and hands the page only the decisions it has not seen. Nothing was added to the bot - `SdsClient` already flushes per line and the run directory is inside the mount - so the same command replays a finished run - [x] A hex map of the board the scenario named, read from `$MM_HOME` at run time, with every proposal's end hex coloured by rank rather than by absolute value, terrain as a top overlay, and unit footprints outlined along real hex edges. What is ranked is switchable: the proposal's value, the damage it deals or takes, its risk, or any single feature - [x] The enemy on the same map. The decision log carried no enemy at all: `positions` is now every unit's hex and facing at the moment the question was asked, and each force decision carries `threats` - per enemy, where it stood and every hex of the `may_be` set the whole menu was scored against. `sds watch` draws the set as a dotted outline on real hex edges, the worst member of it for the order we gave, and a solid arrow to where the enemy actually went, read out of a later decision rather than predicted. There is deliberately no predicted move: `Presence` carries no probability and the collapse is at the mean, so the bot has no opinion about which hex an enemy will pick, and a dashed "we think they will go here" arrow would be a belief invented by the viewer - [x] The firing decision read as a decision. A `FIRING` frame accounts for every weapon on the shooter under exactly one reason it did or did not fire - declared, legal and left below the rung the volley stopped at, out of the arc the torso arrived in, out of range, or on the floor - beside the hull facing, the torso facing and the twist that was declared. The reasons are `sds firing`'s, handed to the page per decision, so the report and the panel cannot word one status two ways. Nothing of it is drawn on the map: an arc is not a place - [x] Name the units on the page. `sds view` reads `Turkina C (#4)` rather than `unit 4`, and each seat lists what it fielded. The names come from the `positions` array the log already carries, so nothing was added to the bot; a log recorded before `positions` named units still renders, with the bare id - [x] Watch a match from MegaMek's own view. `sds one --replay` writes an animated GIF next to the result, one frame per round, drawn by `MinimapPanel` and encoded by `GifWriter`. It needs a frame buffer: MegaMek's minimap factories return null with no graphics configuration, so a replay match runs under `xvfb-run`. Off by default; a benchmark never starts an X server - [x] Highlight latch transitions - "round 7, honour broken against Jade Falcon" is the story beat - [x] Per-phase `answered`/`defaulted`/`illegal` already exist; surface them here - [x] Decompose one decision: `sds explain` prints every candidate on a menu with its score broken into terms - feature, raw value, weight, product - including the default the bot could have taken instead. The decision log already carried each candidate's `phi` and `value`; what it lacked was the weight vector, which `sds-bot` now writes beside the log as `-.weights.json` - [x] Separate "the bot declined a shot" from "the bot had no shot": every `FIRING` row now carries the shooter's whole arsenal, every shot `WeaponAttackAction.toHit` allowed, and the ladder rung each one sat on. `sds firing` reports it per weapon. The totals in `sds weapons` count what was fired and cannot say what was available, which is the difference between a menu problem and a position problem - [x] Separate "a shot the unit had" from "a shot only a torso twist reaches": an offer carries the facings it was priced at, one entry per weapon and target however many twists reach it, and `sds firing` counts the twist-only ones and how often a twist was declared. Pooling the facings would have claimed the bot was offered three times what it was, and left the one number that moves unreadable - [x] Separate "the unit held position" from the four other ways a unit ends a round in the hex it started in: prone and staying down, prone and standing, shut down by heat, and immobile - a destroyed gyro or a crew dead or unconscious, which is no decision at all. The round snapshot now carries `immobile`, `crewDead`, `crewUnconscious` and `crewHits`, and `sds moves` classifies every unit-round against Princess's. Refusals come from the seat's movement tally, because a refused order and a chosen hold are the same two hexes in any position log ## sds labels Correlates every label against every other over a corpus. Every label decision this project has made came out of this measurement, and until now it lived in a scratch file in `/tmp` - which meant the method was reproducible by whoever had that file and nobody else. Reads result documents only and never opens a decision log, so it is seconds rather than the half hour a decision-log pass costs. It prints three things: labels that are never usable on this corpus, pairs at |r| >= 0.90, and every label ranked by the largest correlation it has against any other, with a verdict - under 0.4 "a new direction", under 0.8 "partly new", above that "mostly a rename". The bands are the ones the write-ups already used; having the tool say them keeps a reader and the numbers agreeing. **One corpus does not settle a correlation.** `bv_lost ~ damage_taken` measured 0.50 on one 96-match corpus and 0.95 on the next. The command's own help says so, because that lesson cost a wrong conclusion in a report. Takes a stored corpus name or a run directory: sds labels maximalist-r4 sds labels runs/20260822T214253Z-bench ## `corpus save` was dropping the only copy of the refusal reasons `CORPUS_GLOBS` left the per-match `.log` behind, with the reason written next to it: it "is the bulk of a run directory and nothing reads it twice". Measured on the 120-match 4v4 run in flight, the decision logs are **7.3G** and every `.log` in the run together is **4.0M** - 0.05% of what was already being kept, not the bulk. The second half is wrong too: those files were read three times this week, for MegaMek's refusal reasons, for the victory condition, and to count how often the force explored. `.log` and `.html` are now saved. A corpus exists so the run directory can be deleted, so anything a later question might need has to be inside it - and a saved corpus that cannot answer "why did it refuse the shot" is missing the question most often asked of it. Not retroactive: the corpora already saved have no `.log`, and the runs they came from are gone. ## `sds progress` - the standings of a run that has not finished `bench` prints its table once, at the end, and a suite run is hours. The question asked long before then is "is this vector even in the fight", and it was being answered by one-off scripts reading the per-match JSON by hand - three times in one night, once against the wrong denominator, and never picking up the `answered`/`defaulted` columns that say whether the bot played the match it is credited with. `sds progress ` renders the same `Summary` the final table is built from, over whatever matches have finished. It skips the files that are not match results and a result caught mid-write. The interval it prints is the honest width for the games played. A partial read is not a reason to stop a run early: stopping one because the number looked good is how a benchmark turns into a story about when somebody stopped looking. The command's own help says so. ## Every baseline from a two-seat run recorded zero wins `from_run` asked `summary.sides.get("sds")` for the bot's side. A benchmark alternates which seat the bot plays, so `Summary` is keyed by role and the sides are called `sds North` and `sds South`; there is no side named `sds`. The lookup returned `None`, the `wins` field took its `else 0` branch, and so did both BV columns. Every baseline recorded from a two-seat run has `wins: 0`. `sds baselines` has been printing `overnight-1v1`, `overnight-2v2`, `overnight-4v4` and `overnight-8v8` at **0.0%** for as long as they have existed, and it was read past several times tonight as if it were a property of those old runs. It matters more than a wrong number in a listing: `baseline.py` is the machinery that is supposed to refuse misleading comparisons, and `compare` against one of these would have measured a candidate against zero and called any result a huge improvement. Fixed by summing over sides by `kind` rather than looking one up by name. `fitted-r7-suite` is rebuilt from its run directory - the run writes `baseline.json` whether or not `--save-baseline` was passed, which is what made the repair possible without replaying 120 matches - and now reads 54 of 110 decided, 49.1%. The four `overnight-*` baselines cannot be repaired: their run directories are gone. They should be treated as unrecorded rather than as zero. ### And a comparison against one is refused rather than reported Fixing `from_run` stops new baselines being zeroed; it does nothing about the four already stored, whose run directories are gone. `compare` now refuses them. The signature is exact rather than a guess about strength: `wins`, `bv_left` and `bv_left_opponent` all zero over a run with decided games. A bot that genuinely lost every game still leaves BV on the board in some match, so it is a result and stays comparable; an opponent that finished 150 games with none of its BV remaining in any of them did not happen. Refusing is the conservative direction here. The failure it prevents is a candidate measured against zero, where every vector is a large improvement and the report says so in confident language with an interval attached. ## `sds correlate` - the win rate broken down by what was in the match A win rate says the bot is better or worse. It cannot say *at what*, and at-what is the question that turns a benchmark into a direction to work in. `sds correlate ` joins each result to the `.meta.json` the tiered generator writes beside every scenario, and reports the win rate within groups: battle value tier, era, mean gunnery, mean piloting, map, heaviest machine, tonnage spread, Clan machines present, machines that can jump, mean weapon count, role, weight class, and each individual design. Three things it is careful about: - **A match whose scenario it cannot find is reported, not dropped.** Silently shrinking a group is how a join lies. - **A group under a dozen matches gets no interval.** Those are summed into one line instead. A three-match group at 100% is a story about noise, and printing a confidence interval next to it invites the story. - **It says "mirrored" in the output.** Both sides field the same machines with the same crews, so a group the bot wins more than half of is one it plays better than Princess does - not one that is stronger. Read the other way the whole report is backwards, so the report says which way round it is. ### Breadth and per-machine resolution are different suites The 418-scenario tiered suite fields 1029 distinct designs across 1672 slots. That is the right shape for "does the bot generalise, or is it good at our scenarios" - and it is the wrong shape for "which machines does it fly badly", because every design appears once or twice and `sds correlate` suppresses every per-design group as too thin to read. The table exists and is empty, which is the honest outcome and not a useful one. `--designs-per-era N` draws from a fixed sample instead of the whole pool. At 25 designs a 20-scenario suite already repeats each machine a median of 3 times; scaled to a few hundred scenarios that is tens of matches per design, which is readable. The two are not better and worse. A narrow suite that says the bot cannot fly a Hunchback has learned something a broad suite cannot, and has learned nothing about the nine hundred machines it never fielded. Pick the question per run, and say which one the run asked. ## Three questions in one night that the record could not answer Each was answerable only by inference, a second run, or reading the source; each is a field the record already had the inputs for. - **Why did this unit propose nothing?** A pure-vehicle bench defaulted **21.3%** of its movement decisions, against 0.23%-1.06% on the Mek corpora, and it was raised as a defect. It is not one: joined against the match document's per-round snapshot, **all 64 were a unit MegaMek had already marked immobile**. Reaching that took an inference first - 17 units defaulted, all 17 never answered again, and all 64 decisions had an empty menu - because nothing in `decisions.jsonl` carried `immobile`, `elevationValid` or `stuck`. `SdsClient.record` writes all three per unit now, and `Unit::immobility` names which one so the turn label says it outright. - **What was the to-hit, and what arc was the shot in?** The vehicles agent hit the same wall twice on the firing side. The shape is one shape: **the observation carries the field, the bot reads it, and the record drops it.** Everything in `Observation.unit` is available at `record` time; what gets written is a chosen summary, and the choice was made before anyone had the question. So the test is not "is this field interesting today" but **"if this decision looked wrong, could I say why from this file alone?"** The cost of being wrong is asymmetric and that is the argument. A field nobody reads is a few bytes a decision against a rationale already measured in hundreds of kilobytes. A field nobody wrote is a re-run - and a re-run on a suite that has since changed cannot be compared with the run that raised the question anyway, which is the trap `plan/harness.md` is about. **A summary is a proxy, and the proxy rule applies to it.** `positions` looked complete - id, name, side, hex, facing, done, destroyed - which is exactly why nobody checked it for the field they needed. ### The sweep, and what it cost Rather than wait for a fourth wall, `Observation.unit` was walked against what `record` writes: **45 fields emitted, 11 kept, 34 dropped.** The cost of keeping them settles most of the argument before it starts - measured on a real decision log, `positions` is **283 bytes a decision against `rationale`'s 1,531,561**, a factor of **5,400**. The whole ground-truth record was 0.002% of the file. Nobody traded anything away; the summary was written before anyone had the questions. The 34 sort into four groups, and only one of them was a real gap: - **Recoverable per round from the match `.json`** (`armor`, `internal`, `bv`, `heat`, `prone`, `shutDown`, `crippled`). The `entities` keys are entity ids and join to `positions[].id`, so this is a join rather than a loss - at a coarser grain and across two files. **Left dropped, join written down here** - and writing it down was not bookkeeping: that join is what proved all 64 vehicle defaults were immobile units, a question that had otherwise been settled only by inference. Recoverable is not the same as available; the difference is whether anyone knows the join is there. - **Static for the match** (`weight`, `maxElevationChange`, `heatCapacity`, the `*Max` pair, `ownerId`, `team`, `twists`, `role`, `explodableAmmo`, `bvOriginal`, `canUnstickByJumping`). Wrong grain to repeat per decision and not worth a header for. **Left dropped**, except the two below. - **Per decision and recoverable nowhere.** The real gap, and now kept: `walkMp`, `runMp`, `jumpMp`, `jumpMpTerrain`, `mpUsed`, `canStand`. - **Large structures** (`locations`, `weapons`). A real cost rather than a rounding error, so they get their own decision when a question needs them rather than being swept in now. `movementMode` and `gunnery` are kept too, each for a named question rather than for completeness: **`plan/harness.md` asks for a rate split by the mover's own type, and this file could not do it** - the only alternative was matching on the unit's name - and `gunnery` is the direct input to every to-hit figure. **Written into `positions` rather than a per-match header**, which is a deliberate departure: `record` already holds that *"a corpus is easier to read and harder to get wrong when one decision is one flat document"*, and the only thing a header buys is bytes that the 5,400x figure says are not worth buying. A widened entry is 313 bytes against 128, and the whole block is 0.08% of the rationale beside it. ### The failure mode this had to be guarded against `JsonNode.path` does not fail on a name that is not there. It returns a missing node, and `asInt()` on that is **0**, `asBoolean()` **false**, `asText()` the empty string. A typo in a field name would not throw and would not log - it would write a plausible figure into the only copy of the ground truth, which is the `0.00`-for-"no answer" defect this repository has already paid for. `tests/test_decision_record.py` reads both files and asserts every name `record` takes off the observation is one `Observation.unit` puts on it. A one-letter typo fails it.