# Performance

The budget the design is built against, and what actually costs anything.
Measured with `cargo run --release -p sds-core --example perf`, single-threaded,
on an i7-1270P under load — deployed runs get one to two CPUs, so a number that
needs sixteen cores is not a number.

## Measured

Four runs, `uptime` load average 8-18: several agents building and a 40-game
benchmark, which is this machine's normal state. Each figure is the example's
own answer, which is the fastest of five batches of CPU time — see [How these
are measured](#how-these-are-measured) for why it is neither a mean nor a wall
clock. The range across the three runs is what is left after both.

| operation | cost |
|---|---|
| threat map, 8 enemies x 544 hexes | 5.2-11.9 us |
| 160 LOS queries, no cache | 175-458 us (1.1-2.9 us each) |
| 1280 LOS queries, per-turn cache | 525-1288 us (0.41-1.0 us each) |
| stance blend `w = S^T M` | 0.1 us |
| score 160 candidates x 30 features | 2.0-2.4 us |
| fire allocation, 48 shots x 8 targets, greedy PMF convolution | 159-403 us |
| one shot's PMF, built cold | 0.2 us |
| parse one 28KB observation | 96-123 us |
| per-location damage, one volley, cold cache | 44-101 us |
| per-location damage, one volley, warm cache | 1.4-2.7 us |
| 20 candidates, per-location | 66-178 us |
| **`volley::gather`, one (stand, position) pair** | **615-1083 ns** |
| **`volley::best_volley`, warm memo** | **768-1736 ns** |
| **`stands::score_stands`, one 8v8 unit-decision** | **2.0-4.0 s** |
| **— per exchange** | **1451-2870 ns** |
| **`stands::rank` over that sweep** | **84-215 ms (197-503 ns/entry)** |

Everything above the bold rows is a hot *step*. The bold rows are the hot
*loop*, and they are where the runtime is: see [The sweep](#the-sweep).

The 2-3x spread between runs is contention, not variance in the work. Take the
high numbers.

Per-location damage costs 33 us for a whole unit's 20 candidates, against 66 us
for the fire allocation beside it. That is affordable, and the reason is the
share. Damage arriving at a location is memoised on the shots and on that
location's chances in 36, and a pristine front table has only four distinct
shares - 7, 5, 4 and 1 - so eight locations cost four convolutions rather than
eight, and a second candidate with the same volley costs 0.6 us instead of 19.
A whittled-down target has fewer distinct shares still, because the destroyed
locations' rolls fold into the ones behind them.

Nothing here enumerates critical slots or recomputes battle value per location.
That would be roughly 20 candidates x 8 locations x a battle value pass, and it
is what the capped-candidate invariant exists to prevent; what each location
holds is one scalar off the observation instead.

Fire allocation measures `sds_core::ev`, not a stand-in. It is 48 shots against
8 targets but only ~10 distinct `(rack, packet damage, to-hit)` triples, so the
memo answers 4838 of 4848 asks and the convolution is sparse-into-dense: 12
outcomes into 128 buckets, not 128 x 128. Without the memo it is the same work
485 times over.

## The sweep

**`stands::score_stands` is 93% of the runtime, and nothing measured it until
now.** Counters over 229 unit-decisions and 1805 s of thinking put 96.9% of it
inside the sweep and 93.2% inside the volley estimate the sweep calls twice a
pair — 76% of that in `volley::gather` and 24% in the memo lookup it feeds.
Paths were 0.0%, envelopes 0.0%, `rank` 2.8%, `surface` 0.3%.

The benchmark's fixture is two real MegaMek sheets side by side — `Map Set
5/16x17 Open Terrain 1` and `Map Set 4/16x17 Heavy Forest 1`, out of
`tests/corpus/pathfind.jsonl` — with eight 5/8 Meks a side carrying six guns
across five mounts, both forces at their run allowance because nobody has moved.
That is 327 stands against 8 enemies over 2133 positions: **697,491 pairs and
1,394,982 exchanges in one unit-decision**, of which 38.8% store nothing and the
rest occupy 25 MiB. The volley memo answers 99.5% of its asks.

At 2.0-4.0 s a sweep it lands close to the 7.6 s a real unit-decision took in
the match logs, which is the check that says the fixture is the right shape and
not a toy.

**The per-item figures are the ones a change has to move.** The sweep is not
expensive because any step in it is slow — one exchange is under three
microseconds and one line of sight ask is 265 ns. It is expensive because there
are 1.4 million of them. A fixture half this size should report the same
`ns/exchange`, and a change that improves the model should be read there rather
than in the total, which moves whenever the fixture does.

Three things inside it are worth naming, because they are where the
`ns/exchange` goes:

- **Line of sight is ~18% of the sweep**, at 265 ns an ask and ~4.6 million asks
  a unit-decision.
- **`gather` runs before the memo**, because it is what builds the key. A 99%
  hit rate does not make key construction free — a hit pays for it in full,
  which is why the `gather` row and the `best_volley` row are close together.
- **`LocationProfile::of` is ~7.5%**, built once an exchange, and its cost is
  `Unit::location` doing a linear scan of `String` comparisons eight times a
  profile.

## How these are measured

`examples/perf.rs` reports **the minimum of five batches of CPU time**, and both
halves of that are load-bearing on a shared machine.

**Not a mean.** Several agents build here and a benchmark of 40 games runs
beside them. The same binary has been timed at 59.5 ms and 129.7 ms an hour
apart, and an experiment that deliberately doubled the work once came back
*faster*. A mean over that spread is a measurement of the other agents' work and
it moves when they finish; the minimum is the closest available reading of what
the code costs with nothing in the way. It is a lower bound and it is meant to
be — nothing here is a service-level figure, and every number exists to compare
one build against another.

**Not wall clock.** A run descheduled for 40 ms because somebody else's build
got the core has a wall time 40 ms longer and has done identical work. The
kernel already counts what we were given: `/proc/self/schedstat`'s first field
is nanoseconds on CPU. That leaves the contention that is real — cache and
memory bandwidth are shared whether or not we are running — and removes the
waiting, which is the part that moves by a factor of two.

It ticks once a millisecond, so the harness grows each batch until it has run
for at least 100 ms and divides by the count that actually ran. Without that the
cheap rows quantise: they came out as 5.0, 15.0 and 130.0 us, which are the
clock's numbers and not the code's.

Where `/proc/self/schedstat` is not readable everything falls back to wall
clock, and the header line the example prints says which of the two produced the
table.

## Per round, 8v8, 16 decisions

Bot: **~6-10 ms typical, ~30 ms pessimistic**. Per unit decision, 1-3 ms.

**That was before the sweep existed and it is now wrong by three orders of
magnitude** — a unit-decision is 2-4 seconds, and [The sweep](#the-sweep) is why.
The rest of this section is still true of everything outside it.

Dominated today by 16 observation parses (~4.7 ms). That is a statement about how
little thinking exists yet, not about JSON being slow: scoring 160 candidates
takes 3 us because the scoring is one dot product.

The JVM side is now measured rather than estimated. `sds los-dump` times
MegaMek's own `LosEffects.calculateLOS` over 2383 calls on four boards and gets
**95-107 us mean** across runs. That is a cold JVM with no warm-up pass, so
treat it as an upper bound - but it is the right order, and it lands at the top of the 20-200
us range this file used to guess. `WeaponAttackAction.toHit` runs 8 shooters x 6
weapons x 8 targets = **384 of those per firing phase**, so ~40 ms/round on the
host: still several times the bot's whole budget.

## What will dominate once the thinking lands

**Line of sight**, and it turned out cheaper than feared. `exposure`,
`cover_quality`, `los_in`, `los_out`, `rear_arc_gain` and a terrain-aware threat
map all need it, and it is O(hexes along the line) per pair: ~1280 queries a
round at 8v8. The nine positional features that shipped ask through one
`Posture::survey` per candidate, so the count is two traces per enemy per
candidate however many features read the result.
This file predicted 5-20 us each and 6-25 ms/round. Measured, one query is
**0.7-1.5 us** and a whole round is **0.3-0.7 ms** — better than a fortieth of
the guess even taking the high numbers, and about a tenth of what the host
spends answering the same question.

The **per-turn LOS cache shared across the side** still earns its place: at 8v8
it answers seven queries in eight, which is where 1.5 us falls to 0.6 us.
MegaMek does the same internally — the server passes a `losCache` into
`filterEntities`.

## What blows up

1. **Candidate explosion.** Everything here assumes ~20 curated candidates per
   unit. Enumerating paths the way Princess does is 100-1000x, and puts a
   "thinking..." message back in front of the player. The cap *is* the design.
2. **EV over subsets.** Greedy with an incremental PMF is O(shots x targets). Any
   search over allocation subsets is exponential.
3. **`toHit` scaling** as shooters x weapons x targets. A 12v12 of missile boats
   is ~4x worse — still only ~300 ms/round.
4. **Observation per decision** — 28KB x 16 = ~450KB a round encoded and decoded.
   Sending it once per phase would remove most of that. Tidiness, not
   performance, until the thinking lands.

Board size is linear and cheap: four sheets is ~34 us for the threat map. Memory
is a non-issue — a PMF is 128 floats, the threat map 544, `M` is 8x30, all
L2-resident.

## Headroom

On one CPU with a 100 ms/unit budget we use ~3 ms. **Roughly 30x headroom.** That
is the answer to whether the EV calculator, per-location armour, opponent
modelling and multi-turn projection are affordable: yes, provided candidates stay
capped.

## Concurrency

The tokio fan-out buys nothing on one core and is not there for speed — a force
must have heard from every unit before it decides. It degrades to sequential
cleanly and must not be removed as a pointless optimisation.

Where concurrency does pay, even on one CPU, is a compute-once many-waiters LOS
cache: eight units will ask overlapping questions. `sds_core::los::LosCache` is
that cache - a `Mutex` around a memo, with a `Condvar` per entry so the second
asker waits rather than computing the same line again. Nothing iterates the map,
so no result depends on its order.

**What a hit costs is its own question.** At 98-99% hits the memo is working and
the bill is the asking: a warm `LosCache::get` measured at **213ns** minimum over
a 2,176-key working set, of which SipHash over the 48-byte `AttackInfo` was
~110ns and the rest was the `Arc<Slot>` clone and the second lock a hit took to
read the answer out of the slot. Two changes, neither of which touches what is
computed: a resolved line lives in the map by value, so a hit copies it out
under the lock it already holds, and the map hashes with a seedless multiply-
rotate rather than SipHash. **76ns** minimum after, over eight alternating runs
of each build.

End to end, `examples/stands` spends **5-7% less user CPU** - min ratio 0.960,
median 0.927, mean 0.932 over twenty-five alternating runs of each build - and
prints a byte-identical board. That figure is worth stating carefully, because a
first measurement of it came out at 14% and did not survive being taken again:
the arithmetic says 7%. Line of sight is ~18% of the sweep, the sweep is ~60% of
this binary and a get lost 64% of its cost, so 0.18 x 0.60 x 0.64 is where the
win has to land. A whole-binary number taken in a lucky window on a shared
machine will beat its own mechanism, and when it does, the mechanism is right.

**It lives for the match, and deployment shares it.** Measured over eight rounds
of `scenarios/mirror-lance.mms`, one seat, with `scripts/los-growth.sh`:

| round | phase | lines held | new | MB | asked | hit% |
|------:|:------|-----------:|----:|---:|------:|-----:|
| 0 | deployment | 37,656 | 37,656 | 4.74 | 10,054,462 | 99.63 |
| 1 | movement | 38,674 | 1,018 | 4.87 | 20,563,140 | 99.995 |
| 2 | movement | 58,278 | 19,604 | 7.34 | 21,108,220 | 99.91 |
| 4 | movement | 61,486 | 710 | 7.74 | 15,626,016 | 99.995 |
| 6 | movement | 61,900 | 0 | 7.79 | 3,213,544 | 100.00 |
| 8 | movement | 64,020 | 1,600 | 8.06 | 3,416,936 | 99.95 |

It flattens: the first two rounds put down 94% of what eight rounds hold, and
rounds five to eight add 0.05 MB each. 90.4M lines were asked for and 64,020
computed. There is deliberately no bound on it - a cap chosen before the growth
was measured would have been a guess, and 8 MB does not need one.

The same eight rounds with the old per-round clear recompute 20-45k lines every
round instead. At the 0.7-1.5 us a line costs that is tens of milliseconds
against a round that thinks for ten seconds, so **the clear was costing almost
nothing**: the reason to drop it is that its stated reason - "units move" - was
never true of a cache keyed on `AttackInfo`.

Results must not depend on completion order. Reduce in fixed order, budget in
work units rather than wall-clock, give each unit its own seeded RNG stream, and
never read a clock inside decision logic.

## Comparing against Princess

The turn clock measures wall time between turn changes, which is the same
quantity for both sides and says nothing about how much of the machine each side
used to fill it. That gap is not small here, and it runs one way:

| | threads doing the work |
|---|---|
| Princess | one decision thread, plus one `Precognition` thread |
| sds | one tokio runtime, units fanned out across its workers |

MegaMek contains no `parallelStream`, no `ForkJoinPool` and no `ExecutorService`
outside two non-bot classes; `megamek/client/bot` has exactly one thread of its
own, `Princess.precognitionThread`. `BasicPathRanker.rankPath` over every legal
path is strictly serial. So an unpinned wall-clock comparison between these two
is a comparison of two different amounts of hardware, and no sample size fixes
it.

**So a run whose timings are going to be quoted uses `--pin`.** Each seat gets
one physical core and nothing else:

- Seats are separate processes - see `bridge/sds/SdsSeat.java` - so one
  `taskset` per seat covers its JVM, the per-turn threads MegaMek creates inside
  it, `Precognition`, JIT, GC, and the bot subprocess with every tokio worker in
  it. A mask is inherited across `clone` and `exec`, which is what makes one call
  enough.
- The host JVM, and so the server and the clock, is started with a mask that
  excludes the seats' cores, so nothing of MegaMek's own drifts onto one.
- `SDS_WORKER_THREADS=1`, because four workers time-slicing one core is context
  switching and nothing else.
- Cores are chosen by `sds/cores.py`, never by CPU number. Two traps it exists
  for: on an SMT part `-c 0` and `-c 1` are usually the same physical core, and
  this is a hybrid part where `-c 0` is a 4.8GHz P core and `-c 8` a 3.5GHz E
  one. Seats only ever get distinct physical cores of one speed class, and the
  SMT siblings of those cores are kept out of the container's cpuset so a seat's
  core is a whole core.

A pinned figure and an unpinned one are two different measurements. The result
document records which it was in `pinning`, and `sds stats` prints it beside the
thinking-time table.

**What the split costs.** A match is three JVMs rather than one. Measured over a
`mirror-lance` match: a seat peaks at 668MB resident, the host at 865MB, so
~2.2GB a match against roughly a third of that before. Seats are capped at
`-Xmx2g` because the default maximum heap is a quarter of the machine's RAM and
three JVMs each claiming 7.5GB is how `--jobs 2` ran a 30GB machine down to 3GB
free. Memory is now what bounds `--jobs`, not CPU.

## Match throughput

~200 s/match at `--jobs 3`, so **~54 matches/hour**; solo is ~88 s, so three at
once buys only ~20% — one match already saturates ~2 cores. At ~0.1 core-hours a
match, 20,000 matches is ~$24 of Graviton spot. Compute is not the constraint;
see `plan/training.md`.