State: a match launched by headquarters played through to victory on Fargate
(2026-08-03), which exercised the upload path for the first time. The latency
work that followed is measured and shipped — PERFORMANCE.md
is the record, and what is still open there is not repeated here. What is
weak is everything around the game: end-of-match handling, and the artifacts
the finish is supposed to leave behind.
Shipped both sides and seen working in production on 2026-08-08 —
`uploaded 2 client log(s)` with no minting warning above it. That line is
now unambiguous: with the direct route gone it can only be printed by a
presigned PUT that S3 accepted.
Two consequences. Arena requires a headquarters that mints: an upload key
without `signUrl` or `signTemplate` is refused by `10-manifest.sh` at
container start rather than discovered at finalize. And arena had to ship
before headquarters dropped `url`/`urlTemplate`, because an un-updated
arena against a manifest with no signing targets has no route at all. Both
are deployed as of 2026-08-08, arena first.
There is no event to use instead. Suramadu forwards a window title when
it creates the window and not when it changes, so MegaMek relabelling
its frame "Victory phase" on every phase change never reaches the
browser: a probe on `base.processMessage` saw exactly one title across
six rounds of a live match. If a Suramadu bump ever starts sending
them, that is the zero-latency signal and this poll can go.
What is undecided is where the deriving lives. In the container it would
have to run at victory, on two cores, over a file that grows with the
match - a long one is megabytes of TSV and the parse is not free at the
moment a player is waiting to be handed on. In headquarters it can run
off S3, be re-run when the logic improves, and never delay anybody; but
that is a parser in a second language against a format upstream owns.
The field and the derivation are also not exclusive: the field is
evidence and the derivation is a claim, and a record could carry both.
Deliberately parked until matches are on-lexicon, because what the
numbers are *for* decides where they should be computed.
What is left: headquarters' proxy matching caller DID to path DID
(a string compare on its side), the container-side identity check
above, and actually exercising two browsers in one container —
reconnection is where the spike said it breaks. Until a second browser
has been through a match, treat this as plumbing, not a feature.
Rendering was sluggish; the shipped fixes and what each one measured are in
PERFORMANCE.md - median downpress-to-pixels roughly halved
and the p90 tail fell from ~470ms to ~330ms. The proxy was ruled out early
(the api host stayed under 5% CPU during a match). What is still open:
*Verified 2026-08-07.* The agent loads under the jlink-trimmed runtime on
both JVMs, and a SIGTERM produced 12447 client stacks and 1483 host
stacks, collected by finalize along with the stats table. The first
profile confirms by measurement what the DirectDraw item below argues
from source. Of the 8407 EDT samples that were running rather than
waiting, 85% are under `BoardView` painting, 76% under DirectDraw and
**72% under PNG encode and pixel hashing** - and `libzpng` alone is 43%
of the EDT's whole working time. The hottest working stack is
`BoardView.drawHexes -> drawHex -> AbstractVectorGraphics.drawImage ->
WebGraphics.writeImage -> DrawInstructionFactory.drawImage ->
ImageConst.<init> -> ImageServiceImpl.getPngImage -> ZPNG.encode`.
That is the encode-in-the-constructor path, on the EDT, inside paint.
`tests/profile-summary.py` is what produces those numbers.
(An earlier reading of the same profile said 79% and 53%. It was wrong:
the split between waiting and working excluded parked stacks but the
bucket totals did not, so the denominators disagreed.)
Two caveats on that run: the match never left the lobby, so these are
ambient costs rather than interaction costs, and the host was busy with
an unrelated build, so `latencyClientRendering` from the same run is not
usable. A run that plays is what settles the top two items below.
A match that plays itself now exists for exactly this - `arena.sh up
--spectate`, Princess on every faction and the browser watching as
`ArenaSpectator`. *Written, not verified:* the MatchHost side has never
been compiled or run, because no build has happened since. The next
profile should be taken from a spectated match, not a lobby.
Still open: `scripts/build.sh` does not pass the four async-profiler
build args - the Dockerfile carries them as defaults and the test fails
if those drift from versions.env, but they belong in build.sh. A short
run can also finish before the client JVM has a session to flush, and
then only the host profile is collected, with nothing said about the
missing one.
It is also a second PNG encoder: the summary goes through the JDK's
`com.sun.imageio.plugins.png.PNGImageWriter`, not Suramadu's native ZPNG,
so the two compete for the same EDT.
Caveat before acting: a bot-vs-bot match changes phase far more often per
minute than a human one, so 18% is the fast end. The fix is to decide
whether these images are wanted at all - `container/exit/finalize.sh`
collects them - and if they are, whether they can be written off the EDT.
Written and wired, but **off by default** behind `ARENA_LAZY_PNG=1`, so
this stays open. Two things are missing and they are the same thing: the
72% figure above came from a profile of the *lobby*, and no client
wall-clock profile has ever been collected during play - two attempts
produced none - so nothing says what the transform is worth, or that the
encode is still where the EDT goes once the frame path is warm. Until
that A/B exists an unmeasured classload rewrite of the path that encodes
every frame does not belong on every player's client, and while
"intermittent frame tearing" is open it would confound that
investigation as well.
`instrument/lazy-png-agent/test/HarnessMain.java` is the correctness
check - it runs the transform against the real `ImageConst` out of
`suramadu-directdraw-swing-*.jar` and asserts the constructor stashed the
source image and did *not* fill the byte field - and it is not in the
build gate. Wiring it in wants the war's directdraw jars in the
`lazy-png-agent` stage; do that with the A/B, when the transform has to
be trusted rather than merely available.
*Interaction widened 2026-08-31.* The routine was a pan and two clicks -
eight mousemoves, both clicks on the same hex - which is not what a
player does and not what feels slow. It now pans, sweeps the cursor
across the board at roughly hand speed, and clicks a grid: about six
times the recorded input per run, and enough of it to put a percentile
on. Still short of the real thing in one way worth knowing - on a
16-unit board only three of fifty-two clicks drew a reply over 4KB, so
most landed on bare terrain and the expensive case, selecting a unit and
redrawing the unit display, is under-sampled. The grid is the cheap fix;
knowing where the units are would need the board read out of the client.
A production cold start measured 1m42s from "click deploy" to a visible mech:
18.0s Fargate provisioning, 15.7s image pull, 6.1s container start, 16.9s init
to host-ready, then 31.7s from the client JVM starting to the board being on
screen. PERFORMANCE.md has the anatomy of that last number and the two things it
is not - it is not CPU-bound, so a larger task buys nothing, and the client
JVM cannot use CDS at all. scripts/perf/startup.py measures it. None of the
below is worth pursuing yet; they are written down so the next person starts
from the measurement rather than from the guess.
Two options differ from stock: `ejected_pilots_flee`, so a dead mek's
MechWarrior does not stay on the board as a unit its owner moves every
round, and `skip_ineligible_firing`, so a shut-down unit gets no firing
turn it can only press Done on.
Verified by mounting the file into the shipped image and loading with
`ScenarioReport`: MegaMek logs both as set, and all 51 scenarios load
exactly as they did without it. What has not been seen is the effect in
play — a mek ejecting in a hosted match and the MechWarrior not appearing.
Considered and left alone: `skip_ineligible_movement`, because the
movement phase is where a shut-down mek starts up and where TacOps:AR
abandonment happens; the nag dialogs, which are protective rather than
noise; and turn timers, which are a feature rather than a default worth
correcting.
Crits are applied by `applyDamage()` on the server, not by `createGame()`,
so `ScenarioReport` cannot see the difference and nothing in `tests/`
covers it. Confirming it needs a hosted match: load The Mistake and check
the Redfield's Renegades unit 4 starts with its head slot 2 already out.
One unknown unit data key is left over after this run,
`Unit_PlayerA_1_Camo` in `Examples/Example.mms`. `PARAM_CAMO` is `camo`
and `PARAM_ALTITUDE` is `altitude`, but the per-unit key regex requires an
initial capital, so neither case can ever match. Not fixed here.
With the fix in, a match that hits it now says so by name. If nothing ever
logs, the `ImageIcon`/`PixelGrabber` load was the whole story and this can
close. `finalize.sh` now collects the client's logs under the `diagnostics`
upload key, so from the next match on there is somewhere to read the answer
— as long as the manifest carries that key, which is its own item above.
Two things to look at while a camo is on screen: that the shading is back
on every facing, and that the unit-overview icon matches the board sprite.
Verified 2026-08-09 in a container: real values (7-14% during hover
interaction) and no initialiser failure in the log. That number is the
one worth having on a 2-vCPU task - it is how you answer "is this match
CPU-bound" without guessing.
The prefix was `-ws` until the Suramadu rename. Tags already in ECR keep
it and simply age out; nothing resolves an image by parsing the tag, so
the two forms coexist. infra's `ecr` and `match-cluster` modules and
headquarters' `config.rs` describe the format and were updated with it.
SOCI is now argued against rather than merely unbuilt. What a match does
not read before it is listening is almost entirely MegaMek's render
assets, and those are read on the client's first board paint - so lazy
loading moves the delay into the frame path, which is the one number
PERFORMANCE.md exists to protect, and adds a mid-match failure mode when
a chunk fetch fails. It is also subsumed by pre-warming: a task already
running when the match is created has nobody waiting on its pull. Revisit
only if pre-warming is abandoned.
Unmeasured on Fargate: the overlap is real but the two JVMs now contend,
and only a live task says what the net is. The phase marks above are how
to read it - compare `host-ready` against the old serial sum.
It parses the commit out of the tag, which the entry above says nothing
does. Reporting is all it does with it: `resolve` will hand back a tag whose
commit this repo has never seen, because a local image nobody can rebuild is
still an image you may want to run. `deploy.sh` remains the place that
refuses one.
Not started, and written down because the measurement points at it and the code
turns out to be most of the way there already.
What the numbers say. On the 16-unit bench at the production 2 vCPU, the
match host takes ~155% of the two cores once Princess is thinking, and 0% while
a human is deciding. Rendering wants ~1.35 cores on its own. They do not fit, the
renderer gets throttled, and the throttled repaints coalesce into larger frames -
so the contention costs bandwidth as well as latency. Doubling the task to 4 vCPU
halves p90 input latency (146ms -> 77ms).
Why the split is cheap.MatchHost already constructs each bot as
new Princess(name, LOCALHOST, port) and calls connect(). Princess is a
network client that happens to live in the server's JVM. Moving it out is a new
main that parses a slot name, host and port, and the same three lines - not a
re-architecture.
What it buys, stated correctly. A container's cpu in a task definition is a
reservation, not a cap: it sets a relative weight, and a container is free to
burst into whatever the other containers are not using. It is not CPU pinning,
which is exclusive in both directions. So the split does not make it impossible
for the bots to slow the renderer down - it gives the renderer a floor under
contention while leaving it free to use the whole task when the bots are idle.
For this workload that soft behaviour is the right one, and it is worth being
explicit about why, because the intuitive fix is the wrong one. Load here
alternates hard: Princess is at 0% while a human is deciding and ~155% once it
is thinking, while rendering wants ~1.35 cores during interaction. A static
partition - N cores dedicated to bots, M shared - is therefore wrong half the
time by construction. Reserve enough for Princess's peak and the renderer is
capped during the human's turn, when Princess is using nothing; reserve less and
there is no floor when it matters. A work-conserving share beats any fixed split
on a workload whose two consumers are rarely busy at once, and it is the reason
the "N dedicated + M burst" shape can measure worse than doing nothing if the
weights are picked badly.
Within a task the containers share a network namespace, so Princess still reaches
the server over loopback and the connection stays as cheap as the uncompressed-
loopback change made it.
What it costs. A second JVM's memory and startup; container ordering, since
the bots cannot connect before the server listens; and a new failure mode where
the bot container dies and the game stalls with nobody to take a turn. The
entrypoint's supervision would have to grow a case for it.
Why the scheduling unit is the actual mechanism. Inside one container the
kernel schedules threads, and the two consumers are wildly asymmetric in thread
count: Princess's search is parallel across many runnable threads, while
rendering is bottlenecked on a single hot EDT. Fair-share scheduling between
threads therefore hands Princess the larger portion by construction, however
modest its priority. Container-level shares are applied to the cgroup instead, so
the arena container keeps its weight no matter how many threads the bots run.
That - not isolation - is what the split would actually fix.
Do not pin. An earlier draft of this entry suggested trying taskset or
-XX:ActiveProcessorCount first as the cheap version. On this workload that is
likely to be actively worse: pinning is exclusive in both directions, so a
renderer pinned to one core is capped at one core during the human's turn, which
is exactly the phase when Princess is idle and the whole task is free. The
measurement says rendering wants ~1.35 cores when it can have them.
Re-measure before building any of this. The 155% figure was taken before
loopback packet compression was removed, which cut the host's CPU per game round
by 44%. The contention this entry exists to solve is materially smaller than when
it was written, and it may no longer clear the bar for a second container. Run
the 16-unit bench again first; scripts/perf/stats.py and the CPU sampling
method are already in the repo.
