Testing Ultimate Tool #
Use the cheapest trustworthy layer first. Fake/scripted targets prove orchestration and contracts, not execution of C64 instructions. VICE proves emulator behavior; physical claims require separately authorized acceptance. A test being present is not a claim that the current tree passes it.
Commands and prerequisites #
| Command | What it runs / prerequisites |
|---|---|
make check |
Locked Rustfmt, strict Clippy, workspace tests, all Node helper tests, whitespace check. Rust 1.93 with fmt/Clippy, Make, Node 22. No VICE or hardware. |
make check-python |
Python 3 standard-library test suite, run by tools/run_python_checks.py so each failure traceback is written as it happens, and the distribution-document link/anchor check. |
make check-browser |
Builds the actual CLI and runs Playwright against real embedded assets/routes. Explicitly run npm ci and npm run setup:browser first; Linux browser libraries may also be needed. |
make check-all |
Rust, Node, Python and real-browser checks above. No hardware. |
make demo |
Builds the host CLI and runs the prebuilt first-run fake sequence. Python 3; no Docker/VICE/device. |
make release-env |
Explicit Docker image/tool setup: pinned Ubuntu 24.04, Rust 1.93.0, Node 22.23.3, locked Playwright 1.64.0 and matching Chromium. Downloads occur here, not silently in Make check targets. |
make dist DIST_DIR=NEW_PATH |
Freezes corresponding source, builds its fresh offline extraction in the prepared image, and packages both executables and examples. New distribution and build directories required. |
make check-release DIST_DIR=PATH |
Checks development archive integrity, packaged documentation links/anchors, and pending policies. |
make check-release-publication DIST_DIR=PATH |
Strict local readiness check; pending policies are failures. Does not publish. |
make test-fake |
Root smoke build requires Docker and the pinned Rust-MOS image; fake observations do not execute the PRG. |
make test-vice |
Same source build plus prepared VICE 3.8 (x64sc). Separate from ordinary host checks. |
The Cargo package for the CLI is ultimate-tool, not its directory name
ultimate-tool-cli. The Rust-MOS image is
ghcr.io/koalabuttz/rust-mos:ac2fb2277-4537158-4aaa40e16.
Python check output #
make check-python runs tools/run_python_checks.py instead of
python3 -m unittest discover -s tests/python -v. The helper selects the same
tests with the same loader and verbosity 2, keeps the Ran N tests summary and
the exit status, and writes each failure, error and failing subtest with its
formatted traceback to stderr as soon as unittest produces it. A hosted run
that caps or expires its captured output therefore keeps the tracebacks it needs
rather than only the summary line. Tests at or above one second also get a
[slow N.NNs] line; --slow-seconds moves that threshold. The helper adds no
fail-fast, no retries and no skip handling, and it suppresses the end-of-run
reprint of tracebacks it has already written once.
CI uses explicit setup stages and the same local entry points. It never needs a device or publication credential. See CI for the still-gated Spindle pilot, and local release evidence for isolated runtime/source-rebuild commands.
Acceptance coverage #
This maps the accepted criteria to actual test identifiers. Umbrella names from the plan may be split across cases; a different spelling is not a coverage gap. Commands above execute the suites. Pending observations are listed separately.
| Criterion | Automated behavior and actual tests/scenarios |
|---|---|
| AC.1 Compatible contracts | Core config manifest_v11_feature_gates, future_manifest_schema_is_rejected, legacy_manifest_reads_without_v11_fields; evidence run_v13_expectations_roundtrip, failed_build_run_records_expectations, legacy_runs_remain_readable, future_run_schema_is_rejected. |
| AC.2 Honest no-Docker first result | CLI first_run.rs::packaged_first_run_relocates_and_reuses_inputs_with_source_provenance: relocated eight-run sequence, distinct immutable records, accepted PRG/source/inline provenance and ordinary-panic uncertainty. Python snapshot cases cover reuse, conflicting destinations, tampering, traversal/symlinks/FIFOs. Core noncontainer_build_never_spawns_docker. Full checkout-inaccessible/compiler-free qualification requires the separate release runtime gate, not relocation alone. |
| AC.3 Safe onboarding | Core doctor_is_filesystem_only, doctor_reports_missing_build_executable, doctor_not_checked_ultimate_does_not_block_readiness and injected-inspector cases; CLI external_project_guide_scenario_and_doctor_do_not_build validates an external temporary project and runs fake smoke without SDK adoption. |
| AC.4 Evidence-backed diagnostics | Core report expectation, legacy/unavailable, manifest-change and corrupt-evidence tests. CLI cli_project_root_history_without_manifest_and_bounded_detail; service diagnostic_report_survives_manifest_deletion. Typed capture report cases distinguish verified, torn/unavailable and corrupt evidence. |
| AC.5 Bounded client parity | Core diagnostic_pages_are_bounded_and_bound; CLI report_parity.rs compares complete finalized/provisional reports and pages through actual CLI, loopback HTTP and MCP. MCP wrapper/subprocess tests preserve the same core judgments. |
| AC.6 Safe dashboard | Playwright dashboard.spec.cjs: question-driven report states, hostile text, read-only history/manual paging, no timeline for non-capture runs, advanced-panel ordering, origin/token checks and stale-response protection. Node diagnostic/timeline/comparison renderer suites. |
| AC.7 Real test infrastructure | Ignored real-service browser_fixture_server; Playwright deadline/interruption cases; Node browser-lifecycle cleanup cases; service production_has_no_fixture_seed_route, ephemeral bound-origin tests; production restart history and old-token rejection. |
| AC.8 Usable/source-complete Linux release | Python package/checker suites cover archive member bytes/hashes, corresponding-source selection, sentinels, tag/version/worktree rules, bounded inputs and rebuilt-output evidence. verify_release_local.py provides real extracted-source and compiler-free runtime gates. Passing synthetic harness tests alone does not qualify a real candidate or replace human license/privacy review. |
| AC.9 Hosted Tangled CI | Workflow/helper structural tests in test_ci_workflow.py. Actual spindle_link_smoke and spindle_full_check remain operator-assisted observations; local tests cannot satisfy them. |
| AC.10 Safe publishing/showcase preparation | Publisher dry-run/no-network, preflight-before-secret, mocked XRPC, owner/session DID, TLS/redirect, response/timeout, partial-outcome and receipt tests. Showcase verified export/fresh hashes, withheld private fields, source immutability, static/no-live-controls and unsafe-output tests, including symlinked evidence storage. Real publication remains separately approved. |
| AC.11 Bounded frozen capture | Core checkpoint_capture_persists_both_reads_and_metadata, declared read order, torn/missing marker, short/zero/late/partial-read preservation, digest/identity/semantic proof and no-mutation tests. Runner visible-state-before-ledger, nonpass/screen-failure skip, capability-before-deploy and no-extra-screen-on-capture-failure cases. |
| AC.12 Valid comparison/durable matrix | Core matched/first-difference/incompatible/partial/store-load-restart cases. Runner build-once pair, VICE failure skip, invalid order/repeat before build, ambiguous deployment, cleanup/evidence failures and full-eligibility baseline gate. Pair records are separate from finalized runs. |
| AC.13 Comparison parity/safety | CLI checkpoint_parity.rs: all four states, paired byte differences, durable get/list parity and manifest deletion across actual CLI/HTTP/MCP. Service checkpoint_job_validation_precedes_probe_consumption; Playwright read-only comparison/uncertainty and exact supported matrix selection. Strict-gate flags additionally preserve default behavior and require a strict probe. |
| AC.14 Emulator execution | vice_debugging_and_checkpoint_acceptance.py: exact VICE 3.8, actual corrected/buggy source builds, unique source diagnosis, independent golden ledger, repeat stability and accepted smoke recovery. The latest recorded VICE-only pass is documented in checkpoint comparison. It preserves the accepted smoke PRG with the preview fixture build wrapper; it does not verify the separately gated physical policy. |
| AC.15 Physical acceptance | Explicitly pending ultimate_checkpoint_physical_acceptance; no fake/VICE substitution. |
| AC.16 Public/user readiness | Explicitly pending new participant session, reviewed truthful/private video, and approved published-download/install verification. Automated tutorial scenarios cannot replace human comprehension. |
Real-service browser harness #
The frontend stays dependency-free and embedded. Playwright is test tooling only; it loads the actual Axum app, not a production fixture-seeding endpoint.
The Node harness owns a temporary root and starts:
cargo test --locked -p ultimate-tool-server --test browser_fixture \
browser_fixture_server -- --ignored --exact --nocapture
It supplies ULTIMATE_TOOL_BROWSER_ROOT, binds a kernel-selected loopback port,
waits for a bounded structured readiness announcement and health check, and
owns shutdown/forced process-group cleanup. The fixture has a deadline even if
stdin stays open. Normally use make check-browser; do not launch it against a
real project. Production binary cases use serve --port 0, not a reserved fixed
port, and verify history plus old-token rejection after restart.
The matrix-control browser case overrides only the target-availability response for presentation testing. It never submits a hardware job or grants permission. Comparison/history/report cases use real core evidence and routes. Screenshots, traces and browser output are ignored software-test artifacts, not physical evidence; review privacy before sharing.
Remaining qualification observations #
- Run the complete checks on the final stable candidate; an earlier green run does not cover later changes.
- Run real packager-produced source extraction/rebuild and Ubuntu runtime qualification. A synthetic package self-test proves the harness works, not release source completeness or the compiler baseline.
- Regenerate/reconfirm fixture execution after any build-input or artifact change; retain earlier failures and never relabel changed PRGs as accepted.
- Obtain independent review and fresh permissions before the Spindle setup, physical comparison/health, tag/push, publication or visibility gates.
- Record human license/privacy review, the first-use session and video approval; then verify anonymous published binary/source/checksum downloads if approved.
See preview readiness. No hardware access, credentials or publication is part of an ordinary check command.