david.boston / ultimate-tool
Hardware-in-the-loop development platform for the Commodore 64.
Star
Fork 0
Watch
ultimate-tool tools release-docs CHECKPOINT_COMPARISON.md
12 kB
Markdown
at main

Comparing frozen memory checkpoints #

Distribution copy. This is the sanitized public copy shipped in the docs/ directory of the release archives. It is derived from the repository document docs/CHECKPOINT_COMPARISON.md, which is withheld from every archive in its original form. Transformations applied for distribution: this provenance notice; the emulator acceptance log was reduced to the observations it proves, with per-run record identifiers, dates, and pinned-device identity removed. Public artifact and ledger digests are retained, and the Strict physical gate heading is unchanged because other shipped documents link to its anchor. Every relative link below resolves inside both archives.

Observed VICE-only acceptance #

The scenario runs end to end on VICE 3.8 with the preview workspace and the accepted fixture build wrapper. Corrected runs retain the corrected 8bd464d7…2d9a PRG, verify every registered file of each capture, and match the independently computed golden ledger digest 5860b43de7095543399a9f99028c91c8e37e49e9ce82819ff0db00cdfac96a00. A deliberately buggy run fails as intended and resolves the same unique $0EF1 source site in fixtures/checkpoint-demo/src/main.rs, and a following VICE smoke run passes with the accepted dac6abb2…2560 smoke PRG. This is emulator execution and regression evidence, not a physical strict-gate run or approval; the physical acceptance records are withheld from the distribution and require fresh separate authorization.

This guide covers the frozen memory-checkpoint capture and the comparison operations built on top of it. It documents the CLI commands, the HTTP API and dashboard surfaces, the typed evidence those operations read, and the limits of what a comparison proves.

The capture and comparison path is demonstrated end to end by tools/vice_debugging_and_checkpoint_acceptance.py against fixtures/checkpoint-demo/. Run that scenario for a worked example.

What a frozen checkpoint capture is #

A test may declare memory_checkpoint_capture under [tests] in a schema-11 manifest. When it does, the runner reads a declared safe-state marker and a small set of declared RAM ranges after the program reports its terminal state, reads each range twice, and requires every read to be byte-for-byte identical. The result is stored as memory-checkpoints.json (schema 1) beside the run, and every raw read is written as its own registered evidence file. See CONTRACTS.md for the full record.

The capture is typed: the document records the marker address and expected bytes, each row's name, address, and length, the per-read SHA-256, the file that holds each read, a stable verdict, and a valid verdict. A torn or failed capture is preserved with valid = false and an invalid_reason; it is never treated as a comparison input and never reported as a mismatch.

Two properties keep the capture meaningful:

  • The marker is frozen, not live. The fixture publishes the safe-state marker before its terminal event and then stops writing. A program that keeps writing to the captured ranges after its terminal event invalidates the frozen-ledger claim; this is cooperative evidence for a reviewed fixture, not a live-debugging probe.
  • No DMA or timing-sensitive path is involved. The reads happen after the program has stopped mutating memory, so they do not race a UCI or REU DMA path.

CLI commands #

The CLI reports a checkpoint_comparison object only when a test run records a capture; the comparison subcommands themselves never build or run anything.

# Run the corrected manifest twice from one build: one valid capture per run.
ultimate-tool --format json test checkpoint-pass \
  --manifest fixtures/checkpoint-demo/c64dev.toml --target vice --repeat 2

# Compare the two captures. Read-only: it stores nothing and touches no target.
# Exits 0 when the state is `matched`, otherwise 4.
ultimate-tool --format json runs compare-checkpoints RUN_A RUN_B \
  --manifest fixtures/checkpoint-demo/c64dev.toml

# List separately persisted pair records, newest first.
ultimate-tool --format json runs comparisons --manifest c64dev.toml

# Inspect one checksummed pair record.
ultimate-tool --format json runs comparison COMPARISON_UUID --manifest c64dev.toml

runs compare-checkpoints runs the core compare_run_checkpoints method and prints a schema-1 comparison. state is one of matched, diverged, incompatible, or unavailable. The report and report-items commands expose the same capture through the diagnostic report when the evidence loader is available.

The chained matrix form is a different operation: it builds once, runs VICE first, and only starts the Ultimate candidate after the VICE baseline passed with a valid capture. It always stores exactly one pair record.

# Requires exactly the two targets `vice` then `ultimate` with --repeat 1,
# and a test that declares a memory_checkpoint_capture.
ultimate-tool test <name> --target vice --target ultimate \
  --compare-checkpoints --repeat 1 --manifest <manifest>

An unsupported target order or repeat count is rejected before any build or target access, so it never reaches a target.

HTTP API and dashboard #

The checkpoint routes are documented in full in HTTP_API.md; the schema versions they carry are the outer API schema_version = 13, the memory-checkpoint capture schema 1, and the comparison schema 1.

  • GET /api/v1/runs/checkpoint-comparison?baseline=RUN&candidate=RUN — provisional, read-only, stores nothing. Repeatable.
  • GET /api/v1/checkpoint-comparisons?limit=50 — bounded summaries of stored pair records.
  • GET /api/v1/checkpoint-comparisons/{comparison_id} — one stored record.

In the dashboard, the Compare checkpoints control becomes available only when the selected test declares memory_checkpoint_capture, exactly two targets vice then ultimate are selected, and the repeat count is 1. Checking it adds a read-only comparison record to an ordinary matrix job; the job snapshot keeps the stored checkpoint_comparison_id.

A VICE-versus-Ultimate gate #

A checkpoint-matrix job on a mutating target goes through the ordinary typed authorization gate: a fresh expiring probe, and confirmation of the exact reset count (and any REU-write count) before mutation. The comparison itself never mutates anything.

The deterministic fake target must not be used as execution proof for this path. It manufactures observations and does not execute the PRG, and its crash site is fixed at $10C4. A comparison of fake-target runs demonstrates orchestration and the record format only; it is not evidence that the program ran.

What a comparison does and does not say #

A comparison matches runs on identity before comparing bytes: project, test, target kind, the selected test's semantic fingerprint (expected screen, terminal, timeouts, and startup input), the primary artifact digest, and every supporting-artifact digest. A fixed or explicitly absent startup input is part of that identity, so two runs are never matched if one took a startup command and the other did not.

When two captures diverge, the result names the earliest recorded logical checkpoint that differs — the first declared row, and the first byte within it. That is the first checkpoint whose value changed, not the instruction, timestamp, or cause of the difference.

Strict physical gate #

An optional strict_gate restriction can be requested on a test run and on a read-only probe. It defaults to false, and when it is absent the ordinary, unchanged behavior is selected. Strict mode is a fail-stop restriction, never an added permission: it does not relax the pinned-identity check, the reset budget, the probe expiry, the mutation token, the origin check, or any lease. It grants no extra hardware authorization and makes no physical claim of its own.

The bounded policy supports exactly two shapes, each with repeat = 1:

  • One mailbox-driven Ultimate test.
  • The specialized VICE-then-Ultimate checkpoint pair (see above).

Screen-observation tests, REU uploads, and wider matrices are rejected before the build runs and before any target is touched, so an unsupported shape never reaches a device. Strict mode is orthogonal to compare_checkpoints.

On the first transport, status, body-length, or parse error the strict run stops immediately: it does not issue another polling iteration, retry a read, replay a redirect, or silently swallow an optional-capability probe failure. Bounded readiness polling and the protocol's deliberate two-read stability check are observations, not retries, and continue to run within their existing deadline. After a strict failure the run suppresses additional best-effort target evidence acquisition, including the outer screen fallback and post-crash capture, while retaining the raw reads and events already obtained.

A strict job must consume a probe that was itself requested with strict_gate; an ordinary probe that may already have retried is rejected (422 probe_strictness_mismatch) before the permit is consumed and before any mutation.

Mutation envelope #

These are upper bounds on the failure paths and expected counts on a successful path. A dispatched ambiguous request consumes its slot; there is no retry, reset, redeploy, or substitute mutation after a failure.

  • Checkpoint hardware candidate: one run_prg; then one screen pause and one screen resume — 3 mutation requests on success.
  • Smoke health: one run_prg; one command pause, one writemem to the declared command slot, and one command resume; then one screen pause and one screen resume — 6 mutation requests on success.

The two operations are separately invoked and separately authorized, so the reviewed total is 3 + 6 = 9 mutation requests. The strict flag does not grant the conditional cleanup described next.

Pause and resume ownership #

Each pause episode gets at most one resume dispatch in total. On success that is the normal resume; after a pause/read/write failure it is the conditional cleanup resume, not a second retry and not a new authorization. The adapter marks resume attempted before sending, so finish never sends another resume once that one fails or is ambiguous.

A dispatched request whose response is a send error, unreadable, or malformed is ambiguous, not success: HTTP status alone is not proof the device did nothing. An ambiguous pause keeps its cleanup obligation; an ambiguous resume leaves the CPU-resumed state unknown and is never reported as resumed. Primary and cleanup errors are preserved separately. When the outcome is ambiguous the sequence must stop before the separate health operation; a strict failure does not automatically launch health or recovery. If the reviewed physical gate plan does not permit the conditional cleanup resume, the caller must return that authorization gap rather than inventing a reset or recovery path.

No physical-device run of this restriction has been recorded. The strict shapes are exercised only on the deterministic fake and in local unit tests; a physical claim requires the separate, freshly authorized gate in the section below.

Limitations #

  • The matched/diverged verdict is a statement about the declared rows and marker of one reviewed fixture. It is not a general-purpose memory-capture format and says nothing about memory outside the declared ranges.
  • A comparison reports the earliest differing recorded checkpoint, not the instruction that produced the difference.
  • The capture is cooperative post-terminal evidence. A program still mutating the declared ranges after its terminal event invalidates the frozen claim.
  • A physical VICE-versus-Ultimate comparison is an emulator-and-hardware claim that must be recorded on the intended, pinned device with fresh authorization. Until such a run is recorded, the matrix path is exercised on VICE and on the deterministic fake only.
  • The tools/vice_debugging_and_checkpoint_acceptance.py scenario is emulator-only. It performs no C64 Ultimate access or reset and claims no physical-device gate.