david.boston / ultimate-tool
Hardware-in-the-loop development platform for the Commodore 64.
Star
Fork 0
Watch
ultimate-tool docs CHECKPOINT_COMPARISON.md
13 kB
Markdown
at main

Comparing frozen memory checkpoints #

Observed VICE-only acceptance #

The scenario was rerun successfully with the preview workspace and accepted fixture build wrapper in record 136298cb-7d72-46a5-9cbc-77ec1ce32dc9 on 2026-10-11. Corrected runs 864c3ea9-05fb-4578-a61b-1993540bd043 and 1c43361f-79cf-4eed-9e9a-01a075655c19 retained the corrected 8bd464d7…2d9a PRG, verified the golden ledger and compared matched. Buggy run fc489523-4ca5-4fd3-b84a-3d9262568079 resolved the same unique $0EF1 source site. Following VICE health run 0860998e-3930-449e-a0c3-b32c001f6654 passed with the accepted dac6abb2…2560 smoke PRG. This is emulator execution and regression evidence, not a physical strict-gate run or approval.

Earlier development record #

On 2026-10-11, local scenario record cac7c830-866b-4329-a0ee-97d0f46d252f verified VICE 3.8 source execution: corrected runs 2f166a6a-eda0-4a61-93ba-0bff8dc73d79 and 0a85f21e-36f7-4bf4-baee-50504a0da8bc passed with PRG SHA-256 8bd464d72c8cd3c1db1a49f946cfd1de186023c5b42820d63b23e28086452d9a, all 26 registered files per capture verified, and both frozen ledgers matched the independently computed golden digest 5860b43de7095543399a9f99028c91c8e37e49e9ce82819ff0db00cdfac96a00. The read-only comparison reported matched.

Buggy run 3e451b88-c6bc-4b81-bfb0-2b9849d089da failed as intended and resolved $0EF1 to main at fixtures/checkpoint-demo/src/main.rs:107:17. The following known-good VICE smoke run 90a9b368-2d7c-4aff-9472-0c903eed937c passed with the previously accepted dac6abb2ea01752e94c4318463efa6c54942713ffb021b67f1a4e1d66fa02560 PRG. This record used the built CLI reporting 0.2.0-preview.1 while the C64 SDK source/package inputs retained their accepted 0.1.0 identity. It predates later strict-gate changes and is not their verification. No Ultimate access or physical comparison occurred; those require fresh separate authorization.

This guide covers the frozen memory-checkpoint capture and the comparison operations built on top of it. It documents the CLI commands, the HTTP API and dashboard surfaces, the typed evidence those operations read, and the limits of what a comparison proves.

The capture and comparison path is demonstrated end to end by tools/vice_debugging_and_checkpoint_acceptance.py against fixtures/checkpoint-demo/. Run that scenario for a worked example.

What a frozen checkpoint capture is #

A test may declare memory_checkpoint_capture under [tests] in a schema-11 manifest. When it does, the runner reads a declared safe-state marker and a small set of declared RAM ranges after the program reports its terminal state, reads each range twice, and requires every read to be byte-for-byte identical. The result is stored as memory-checkpoints.json (schema 1) beside the run, and every raw read is written as its own registered evidence file. See CONTRACTS.md for the full record.

The capture is typed: the document records the marker address and expected bytes, each row's name, address, and length, the per-read SHA-256, the file that holds each read, a stable verdict, and a valid verdict. A torn or failed capture is preserved with valid = false and an invalid_reason; it is never treated as a comparison input and never reported as a mismatch.

Two properties keep the capture meaningful:

  • The marker is frozen, not live. The fixture publishes the safe-state marker before its terminal event and then stops writing. A program that keeps writing to the captured ranges after its terminal event invalidates the frozen-ledger claim; this is cooperative evidence for a reviewed fixture, not a live-debugging probe.
  • No DMA or timing-sensitive path is involved. The reads happen after the program has stopped mutating memory, so they do not race a UCI or REU DMA path.

CLI commands #

The CLI reports a checkpoint_comparison object only when a test run records a capture; the comparison subcommands themselves never build or run anything.

# Run the corrected manifest twice from one build: one valid capture per run.
ultimate-tool --format json test checkpoint-pass \
  --manifest fixtures/checkpoint-demo/c64dev.toml --target vice --repeat 2

# Compare the two captures. Read-only: it stores nothing and touches no target.
# Exits 0 when the state is `matched`, otherwise 4.
ultimate-tool --format json runs compare-checkpoints RUN_A RUN_B \
  --manifest fixtures/checkpoint-demo/c64dev.toml

# List separately persisted pair records, newest first.
ultimate-tool --format json runs comparisons --manifest c64dev.toml

# Inspect one checksummed pair record.
ultimate-tool --format json runs comparison COMPARISON_UUID --manifest c64dev.toml

runs compare-checkpoints runs the core compare_run_checkpoints method and prints a schema-1 comparison. state is one of matched, diverged, incompatible, or unavailable. The report and report-items commands expose the same capture through the diagnostic report when the evidence loader is available.

The chained matrix form is a different operation: it builds once, runs VICE first, and only starts the Ultimate candidate after the VICE baseline passed with a valid capture. It always stores exactly one pair record.

# Requires exactly the two targets `vice` then `ultimate` with --repeat 1,
# and a test that declares a memory_checkpoint_capture.
ultimate-tool test <name> --target vice --target ultimate \
  --compare-checkpoints --repeat 1 --manifest <manifest>

An unsupported target order or repeat count is rejected before any build or target access, so it never reaches a target.

HTTP API and dashboard #

The checkpoint routes are documented in full in HTTP_API.md; the schema versions they carry are the outer API schema_version = 13, the memory-checkpoint capture schema 1, and the comparison schema 1.

  • GET /api/v1/runs/checkpoint-comparison?baseline=RUN&candidate=RUN — provisional, read-only, stores nothing. Repeatable.
  • GET /api/v1/checkpoint-comparisons?limit=50 — bounded summaries of stored pair records.
  • GET /api/v1/checkpoint-comparisons/{comparison_id} — one stored record.

In the dashboard, the Compare checkpoints control becomes available only when the selected test declares memory_checkpoint_capture, exactly two targets vice then ultimate are selected, and the repeat count is 1. Checking it adds a read-only comparison record to an ordinary matrix job; the job snapshot keeps the stored checkpoint_comparison_id.

A VICE-versus-Ultimate gate #

A checkpoint-matrix job on a mutating target goes through the ordinary typed authorization gate: a fresh expiring probe, and confirmation of the exact reset count (and any REU-write count) before mutation. The comparison itself never mutates anything.

The deterministic fake target must not be used as execution proof for this path. It manufactures observations and does not execute the PRG, and its crash site is fixed at $10C4. A comparison of fake-target runs demonstrates orchestration and the record format only; it is not evidence that the program ran.

What a comparison does and does not say #

A comparison matches runs on identity before comparing bytes: project, test, target kind, the selected test's semantic fingerprint (expected screen, terminal, timeouts, and startup input), the primary artifact digest, and every supporting-artifact digest. A fixed or explicitly absent startup input is part of that identity, so two runs are never matched if one took a startup command and the other did not.

When two captures diverge, the result names the earliest recorded logical checkpoint that differs — the first declared row, and the first byte within it. That is the first checkpoint whose value changed, not the instruction, timestamp, or cause of the difference.

Strict physical gate #

An optional strict_gate restriction can be requested on a test run and on a read-only probe. It defaults to false, and when it is absent the ordinary, unchanged behavior is selected. Strict mode is a fail-stop restriction, never an added permission: it does not relax the pinned-identity check, the reset budget, the probe expiry, the mutation token, the origin check, or any lease. It grants no extra hardware authorization and makes no physical claim of its own.

The bounded policy supports exactly two shapes, each with repeat = 1:

  • One mailbox-driven Ultimate test.
  • The specialized VICE-then-Ultimate checkpoint pair (see above).

Screen-observation tests, REU uploads, and wider matrices are rejected before the build runs and before any target is touched, so an unsupported shape never reaches a device. Strict mode is orthogonal to compare_checkpoints.

On the first transport, status, body-length, or parse error the strict run stops immediately: it does not issue another polling iteration, retry a read, replay a redirect, or silently swallow an optional-capability probe failure. Bounded readiness polling and the protocol's deliberate two-read stability check are observations, not retries, and continue to run within their existing deadline. After a strict failure the run suppresses additional best-effort target evidence acquisition, including the outer screen fallback and post-crash capture, while retaining the raw reads and events already obtained.

A strict job must consume a probe that was itself requested with strict_gate; an ordinary probe that may already have retried is rejected (422 probe_strictness_mismatch) before the permit is consumed and before any mutation.

Mutation envelope #

These are upper bounds on the failure paths and expected counts on a successful path. A dispatched ambiguous request consumes its slot; there is no retry, reset, redeploy, or substitute mutation after a failure.

  • Checkpoint hardware candidate: one run_prg; then one screen pause and one screen resume — 3 mutation requests on success.
  • Smoke health: one run_prg; one command pause, one writemem to the declared command slot, and one command resume; then one screen pause and one screen resume — 6 mutation requests on success.

The two operations are separately invoked and separately authorized, so the reviewed total is 3 + 6 = 9 mutation requests. The strict flag does not grant the conditional cleanup described next.

Pause and resume ownership #

Each pause episode gets at most one resume dispatch in total. On success that is the normal resume; after a pause/read/write failure it is the conditional cleanup resume, not a second retry and not a new authorization. The adapter marks resume attempted before sending, so finish never sends another resume once that one fails or is ambiguous.

A dispatched request whose response is a send error, unreadable, or malformed is ambiguous, not success: HTTP status alone is not proof the device did nothing. An ambiguous pause keeps its cleanup obligation; an ambiguous resume leaves the CPU-resumed state unknown and is never reported as resumed. Primary and cleanup errors are preserved separately. When the outcome is ambiguous the sequence must stop before the separate health operation; a strict failure does not automatically launch health or recovery. If the reviewed physical gate plan does not permit the conditional cleanup resume, the caller must return that authorization gap rather than inventing a reset or recovery path.

No physical-device run of this restriction has been recorded. The strict shapes are exercised only on the deterministic fake and in local unit tests; a physical claim requires the separate, freshly authorized gate in the section below.

Limitations #

  • The matched/diverged verdict is a statement about the declared rows and marker of one reviewed fixture. It is not a general-purpose memory-capture format and says nothing about memory outside the declared ranges.
  • A comparison reports the earliest differing recorded checkpoint, not the instruction that produced the difference.
  • The capture is cooperative post-terminal evidence. A program still mutating the declared ranges after its terminal event invalidates the frozen claim.
  • A physical VICE-versus-Ultimate comparison is an emulator-and-hardware claim that must be recorded on the intended, pinned device with fresh authorization. Until such a run is recorded, the matrix path is exercised on VICE and on the deterministic fake only.
  • The tools/vice_debugging_and_checkpoint_acceptance.py scenario is emulator-only. It performs no C64 Ultimate access or reset and claims no physical-device gate.