# Testing Ultimate Tool

Use the cheapest trustworthy layer first. Fake/scripted targets prove
orchestration and contracts, not execution of C64 instructions. VICE proves
emulator behavior; physical claims require separately authorized acceptance.
A test being present is not a claim that the current tree passes it.

## Commands and prerequisites

| Command | What it runs / prerequisites |
| --- | --- |
| `make check` | Locked Rustfmt, strict Clippy, workspace tests, all Node helper tests, whitespace check. Rust 1.93 with fmt/Clippy, Make, Node 22. No VICE or hardware. |
| `make check-python` | Python 3 standard-library test suite, run by `tools/run_python_checks.py` so each failure traceback is written as it happens, and the distribution-document link/anchor check. |
| `make check-browser` | Builds the actual CLI and runs Playwright against real embedded assets/routes. Explicitly run `npm ci` and `npm run setup:browser` first; Linux browser libraries may also be needed. |
| `make check-all` | Rust, Node, Python and real-browser checks above. No hardware. |
| `make demo` | Builds the host CLI and runs the prebuilt first-run fake sequence. Python 3; no Docker/VICE/device. |
| `make release-env` | Explicit Docker image/tool setup: pinned Ubuntu 24.04, Rust 1.93.0, Node 22.23.3, locked Playwright 1.64.0 and matching Chromium. Downloads occur here, not silently in Make check targets. |
| `make dist DIST_DIR=NEW_PATH` | Freezes corresponding source, builds its fresh offline extraction in the prepared image, and packages both executables and examples. New distribution and build directories required. |
| `make check-release DIST_DIR=PATH` | Checks development archive integrity, packaged documentation links/anchors, and pending policies. |
| `make check-release-publication DIST_DIR=PATH` | Strict local readiness check; pending policies are failures. Does not publish. |
| `make test-fake` | Root smoke **build requires Docker** and the pinned Rust-MOS image; fake observations do not execute the PRG. |
| `make test-vice` | Same source build plus prepared VICE 3.8 (`x64sc`). Separate from ordinary host checks. |

The Cargo package for the CLI is `ultimate-tool`, not its directory name
`ultimate-tool-cli`. The Rust-MOS image is
`ghcr.io/koalabuttz/rust-mos:ac2fb2277-4537158-4aaa40e16`.

## Python check output

`make check-python` runs `tools/run_python_checks.py` instead of
`python3 -m unittest discover -s tests/python -v`. The helper selects the same
tests with the same loader and verbosity 2, keeps the `Ran N tests` summary and
the exit status, and writes each failure, error and failing subtest with its
formatted traceback to stderr as soon as `unittest` produces it. A hosted run
that caps or expires its captured output therefore keeps the tracebacks it needs
rather than only the summary line. Tests at or above one second also get a
`[slow N.NNs]` line; `--slow-seconds` moves that threshold. The helper adds no
fail-fast, no retries and no skip handling, and it suppresses the end-of-run
reprint of tracebacks it has already written once.

CI uses explicit setup stages and the same local entry points. It never needs a
device or publication credential. See [CI](CI.md) for the still-gated Spindle
pilot, and [local release evidence](LOCAL_RELEASE_EVIDENCE.md) for isolated
runtime/source-rebuild commands.

## Acceptance coverage

This maps the accepted criteria to actual test identifiers. Umbrella names from
the plan may be split across cases; a different spelling is not a coverage gap.
Commands above execute the suites. Pending observations are listed separately.

| Criterion | Automated behavior and actual tests/scenarios |
| --- | --- |
| **AC.1 Compatible contracts** | Core config `manifest_v11_feature_gates`, `future_manifest_schema_is_rejected`, `legacy_manifest_reads_without_v11_fields`; evidence `run_v13_expectations_roundtrip`, `failed_build_run_records_expectations`, `legacy_runs_remain_readable`, `future_run_schema_is_rejected`. |
| **AC.2 Honest no-Docker first result** | CLI `first_run.rs::packaged_first_run_relocates_and_reuses_inputs_with_source_provenance`: relocated eight-run sequence, distinct immutable records, accepted PRG/source/inline provenance and ordinary-panic uncertainty. Python snapshot cases cover reuse, conflicting destinations, tampering, traversal/symlinks/FIFOs. Core `noncontainer_build_never_spawns_docker`. Full checkout-inaccessible/compiler-free qualification requires the separate release runtime gate, not relocation alone. |
| **AC.3 Safe onboarding** | Core `doctor_is_filesystem_only`, `doctor_reports_missing_build_executable`, `doctor_not_checked_ultimate_does_not_block_readiness` and injected-inspector cases; CLI `external_project_guide_scenario_and_doctor_do_not_build` validates an external temporary project and runs fake smoke without SDK adoption. |
| **AC.4 Evidence-backed diagnostics** | Core report expectation, legacy/unavailable, manifest-change and corrupt-evidence tests. CLI `cli_project_root_history_without_manifest_and_bounded_detail`; service `diagnostic_report_survives_manifest_deletion`. Typed capture report cases distinguish verified, torn/unavailable and corrupt evidence. |
| **AC.5 Bounded client parity** | Core `diagnostic_pages_are_bounded_and_bound`; CLI `report_parity.rs` compares complete finalized/provisional reports and pages through actual CLI, loopback HTTP and MCP. MCP wrapper/subprocess tests preserve the same core judgments. |
| **AC.6 Safe dashboard** | Playwright `dashboard.spec.cjs`: question-driven report states, hostile text, read-only history/manual paging, no timeline for non-capture runs, advanced-panel ordering, origin/token checks and stale-response protection. Node diagnostic/timeline/comparison renderer suites. |
| **AC.7 Real test infrastructure** | Ignored real-service `browser_fixture_server`; Playwright deadline/interruption cases; Node browser-lifecycle cleanup cases; service `production_has_no_fixture_seed_route`, ephemeral bound-origin tests; production restart history and old-token rejection. |
| **AC.8 Usable/source-complete Linux release** | Python package/checker suites cover archive member bytes/hashes, corresponding-source selection, sentinels, tag/version/worktree rules, bounded inputs and rebuilt-output evidence. `verify_release_local.py` provides real extracted-source and compiler-free runtime gates. Passing synthetic harness tests alone does not qualify a real candidate or replace human license/privacy review. |
| **AC.9 Hosted Tangled CI** | Workflow/helper structural tests in `test_ci_workflow.py`. Actual `spindle_link_smoke` and `spindle_full_check` remain operator-assisted observations; local tests cannot satisfy them. |
| **AC.10 Safe publishing/showcase preparation** | Publisher dry-run/no-network, preflight-before-secret, mocked XRPC, owner/session DID, TLS/redirect, response/timeout, partial-outcome and receipt tests. Showcase verified export/fresh hashes, withheld private fields, source immutability, static/no-live-controls and unsafe-output tests, including symlinked evidence storage. Real publication remains separately approved. |
| **AC.11 Bounded frozen capture** | Core `checkpoint_capture_persists_both_reads_and_metadata`, declared read order, torn/missing marker, short/zero/late/partial-read preservation, digest/identity/semantic proof and no-mutation tests. Runner visible-state-before-ledger, nonpass/screen-failure skip, capability-before-deploy and no-extra-screen-on-capture-failure cases. |
| **AC.12 Valid comparison/durable matrix** | Core matched/first-difference/incompatible/partial/store-load-restart cases. Runner build-once pair, VICE failure skip, invalid order/repeat before build, ambiguous deployment, cleanup/evidence failures and full-eligibility baseline gate. Pair records are separate from finalized runs. |
| **AC.13 Comparison parity/safety** | CLI `checkpoint_parity.rs`: all four states, paired byte differences, durable get/list parity and manifest deletion across actual CLI/HTTP/MCP. Service `checkpoint_job_validation_precedes_probe_consumption`; Playwright read-only comparison/uncertainty and exact supported matrix selection. Strict-gate flags additionally preserve default behavior and require a strict probe. |
| **AC.14 Emulator execution** | `vice_debugging_and_checkpoint_acceptance.py`: exact VICE 3.8, actual corrected/buggy source builds, unique source diagnosis, independent golden ledger, repeat stability and accepted smoke recovery. The latest recorded VICE-only pass is documented in [checkpoint comparison](CHECKPOINT_COMPARISON.md). It preserves the accepted smoke PRG with the preview fixture build wrapper; it does not verify the separately gated physical policy. |
| **AC.15 Physical acceptance** | Explicitly pending `ultimate_checkpoint_physical_acceptance`; no fake/VICE substitution. |
| **AC.16 Public/user readiness** | Explicitly pending new participant session, reviewed truthful/private video, and approved published-download/install verification. Automated tutorial scenarios cannot replace human comprehension. |

## Real-service browser harness

The frontend stays dependency-free and embedded. Playwright is test tooling
only; it loads the actual Axum app, not a production fixture-seeding endpoint.

The Node harness owns a temporary root and starts:

```sh
cargo test --locked -p ultimate-tool-server --test browser_fixture \
  browser_fixture_server -- --ignored --exact --nocapture
```

It supplies `ULTIMATE_TOOL_BROWSER_ROOT`, binds a kernel-selected loopback port,
waits for a bounded structured readiness announcement and health check, and
owns shutdown/forced process-group cleanup. The fixture has a deadline even if
stdin stays open. Normally use `make check-browser`; do not launch it against a
real project. Production binary cases use `serve --port 0`, not a reserved fixed
port, and verify history plus old-token rejection after restart.

The matrix-control browser case overrides only the target-availability response
for presentation testing. It never submits a hardware job or grants permission.
Comparison/history/report cases use real core evidence and routes. Screenshots,
traces and browser output are ignored software-test artifacts, not physical
evidence; review privacy before sharing.

## Remaining qualification observations

- Run the complete checks on the final stable candidate; an earlier green run
  does not cover later changes.
- Run real packager-produced source extraction/rebuild and Ubuntu runtime
  qualification. A synthetic package self-test proves the harness works, not
  release source completeness or the compiler baseline.
- Regenerate/reconfirm fixture execution after any build-input or artifact
  change; retain earlier failures and never relabel changed PRGs as accepted.
- Obtain independent review and fresh permissions before the Spindle setup,
  physical comparison/health, tag/push, publication or visibility gates.
- Record human license/privacy review, the first-use session and video approval;
  then verify anonymous published binary/source/checksum downloads if approved.

See [preview readiness](RELEASE_READINESS.md). No hardware access, credentials
or publication is part of an ordinary check command.