Continuous integration (local Tangled scaffold) #

This document describes the local Tangled CI scaffold: three workflow files under .tangled/workflows and the helper they call, tools/run_ci.py.

Hosted smoke passed; full check pending. Pipeline 3mxwqmomeci22 passed smoke.yml for commit 2db36210f904ec4ddfbc1715daeb9d123117cf42 on 2026-10-15 in 10m51s: image setup, CLI build, manifest validation and the fake pass/crash/panic/recovery sequence. The installed nixos image allocates 4 GiB RAM, two vCPUs and a 24 GiB writable volume; admission includes a 96 MiB runner allowance and the server permits 30 minutes. These are allocations, not measured peak usage. check.yml now uses the same observed image and guest dependencies, but has not passed hosted execution. The helper spends at most 1500 of those seconds (section 6), so a cold container build plus the whole gate may end in a bounded budget failure instead of a verdict. Emulator and physical acceptance remain separate. Local success is not hosted-CI evidence.

1. What exists #

Path	Purpose
.tangled/workflows/smoke.yml	Build, validate, and drive the first-run fake.
.tangled/workflows/check.yml	Full gate plus packaging, verification, and the runtime harness.
.tangled/workflows/emulator.yml	VICE debugging and checkpoint acceptance.
tools/run_ci.py	The single entry point every workflow invokes.
tests/python/test_ci_workflow.py	Structural checks over the workflows and the helper.

2. Workflow shape #

Each workflow uses the microVM engine. Smoke and full check explicitly select the observed nixos image and declare python3, git and gnumake dependencies. The emulator workflow remains separately gated. The manual trigger shape is:

when:
  - event: ["manual"]

engine: "microvm"

virtualisation:
  docker: true

steps:
  - name: "Assert guest preconditions"
    command: "python3 tools/run_ci.py guest-check"
  - name: "..."
    command: "python3 tools/run_ci.py <action>"

• The pilot uses the observed installed name. smoke.yml selects nixos; inspection found that name points to a NixOS x86-64 image with a spec.json. This is an installed-image observation, not a reproducible image build claim. The emulator workflow still uses the configured default pending its separate gate. No image is provisioned or server setting changed by these files.
• virtualisation.docker: true brings up Docker inside the NixOS guest.
• The approved target is a microVM with a NixOS guest and Docker in the VM. Because the image is not pinned, the first step asserts the guest actually is that shape.

3. Triggers are manual only #

All three workflows trigger on manual only. Push and pull-request triggers are deliberately absent.

A repository with Tangled CI activated runs its workflow files automatically on push. The reviewed pilot commit and normal push are authorized, but automatic push/PR activation is not. A workflow that only accepts manual cannot be started by that push. Adding push (to main) and pull_request triggers is a deliberate later step that belongs with the spindle pilot authorization, not with this scaffold.

4. Guest preconditions (fail-closed) #

tools/run_ci.py guest-check reads the running guest and fails unless all of these hold:

• /etc/os-release reports ID=nixos;
• platform.machine() is x86_64;
• docker is on PATH;
• a read-only docker info against unix:///var/run/docker.sock succeeds for the workflow user within 15 seconds. Root is not required; the hosted runner was observed to execute steps as UID 970.

If any precondition is unmet the helper exits non-zero with the specific reason and no step runs. It never falls back to a different distribution, a different architecture, or a host Docker socket.

5. The prepared image, not the bare guest #

The NixOS microVM guest is unprovisioned for Rust, so no action compiles or tests on the guest directly. smoke, check and package build the local release-environment image from tools/release-env/Dockerfile and run the tool inside it. The container then holds no Docker daemon and no host socket: the packaging path is a direct cargo/python3 argv sequence, so nothing inside the container needs Docker. Building and launching that container is the pilot proof-of-concept the guest alone cannot provide.

6. Actions #

Run any action by hand on the guest:

python3 tools/run_ci.py guest-check
python3 tools/run_ci.py smoke
python3 tools/run_ci.py check
python3 tools/run_ci.py emulator --require-vice-version 3.8
python3 tools/run_ci.py package

Every action also accepts --budget-seconds N for a directly invoked run. It can only shorten the 1500s budget, never extend it; the budget is derived from the workflow step ceiling below.

smoke #

Builds the prepared image, then runs inside it:

1. cargo build --locked -p ultimate-tool — build the actual tool;
2. cargo run -q -p ultimate-tool -- validate — a read-only manifest check;
3. python3 tools/run_demo.py --manifest examples/first-run/c64dev.toml --binary target/debug/ultimate-tool — the prebuilt first-run manifest driven against the deterministic fake target.

check #

Builds the prepared image, then runs inside it (--rm --network default, only the source tree mounted at /work, using the invoking guest user's numeric UID/GID so the checkout remains writable):

1. npm ci — an explicit browser-tooling setup stage, so dependencies come from the lockfile rather than a hidden download inside a make target;
2. make check-all;
3. make demo;
4. the release path (see below).

Afterwards, on the guest (where the Docker daemon lives), check runs the runtime harness:

python3 tools/verify_release_local.py --mode runtime \
  --release-dir dist --out /tmp/ultimate-tool-runtime-verification.json --json

The check-browser half of make check-all needs a Playwright browser (npm run setup:browser, Playwright 1.64.0). Explicit image setup installs the matching Chromium and operating-system dependencies. The container action sets PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright; missing browser inputs fail rather than silently skipping the browser tests.

The release path (no Docker invokes) #

Inside the container the packaging path is a direct argv sequence — it never calls make dist, because that target would reach for Docker, which the container does not have:

Step	Command
release build	cargo build --locked --release --target-dir target/release-ubuntu-2404 -p ultimate-tool -p ultimate-tool-mcp
source archive	python3 tools/package_release.py --prepare-source --output dist --development ci-local
build-environment descriptor	/usr/local/bin/ultimate-tool-build-env /tmp/ultimate-tool-build-env.json
package binaries	python3 tools/package_release.py --pack --binary-dir target/release-ubuntu-2404/release --build-env /tmp/ultimate-tool-build-env.json --output dist --development ci-local
verify	python3 tools/check_release.py --release-dir dist --mode development

This is a development packaging smoke, not the source-complete release gate. The container builds from its checkout; make dist instead freezes the source archive first and builds from its isolated, vendored extraction. Dependency source paths can change binary bytes, so those outputs are not assumed identical. A candidate must still pass the independent extracted-source rebuild and runtime checks described in local release evidence. The packager requires a fresh dist and rejects conflicting files.

The build-environment descriptor is written to a temp path outside dist, because the packager requires its output directory to be new or empty; writing the descriptor into dist first would make that impossible. The descriptor is the observed environment written by the writer shipped in the image, not the release source manifest's expectations.

The workflow calls the tools directly inside the container. It does not mount or invoke the host Docker socket, and it cannot perform publication.

emulator #

Runs the debugging and checkpoint acceptance script (tools/vice_debugging_and_checkpoint_acceptance.py) on the guest. It requires a prepared VICE 3.8 executable and never installs or substitutes one:

• supply the executable with --vice-executable PATH or the VICE_EXECUTABLE environment variable, and the version with --require-vice-version (default 3.8);
• the helper queries the executable's own --version, compares the first MAJOR.MINOR it finds, and fails closed on any mismatch;
• the helper then runs the acceptance script passing the executable through its own --vice-executable flag — the script does not read the environment — and the script owns its source-build bug / corrected / recovery / repeat checks against exact VICE 3.8.

package #

Builds the prepared image and runs only the release path inside it, then the runtime harness on the guest. It never runs the release preflight mode and never invokes the publisher (tools/publish_tangled_artifacts.py). Publication remains a separate, separately-approved step.

Deadlines, logs, and what a bounded failure means #

One monotonic budget covers a whole action. It is created once, spent as the action runs, never reset by a stage, and every child timeout is the smaller of that stage's own ceiling and the time still left.

Limit	Seconds	Meaning
workflow step ceiling	1800	the server's admission limit for one step; unchanged by this scaffold
reserve inside that ceiling	300	guest setup, teardown and image removal
CI_BUDGET_SECONDS	1500	the helper's budget for one action
container handoff margin	30	the inner helper's budget is the docker run timeout minus this
minimum usable remainder	5	below this no further stage is started
cleanup reserve	60	the most any one command may spend stopping and removing what it owns

Per-command ceilings, all at or below the 1500s budget: validate 120, guest docker probe 15, VICE version query 120, build-environment descriptor 120, first-run demo 300, make demo 900, npm ci 900, debug build 900, release check 600, package 1200, runtime harness 1200, and 1500 for make check-all, the release build, the VICE acceptance run, docker build and docker run.

So the deadlines nest: the innermost stage ceiling first, then what is left of the helper budget, then the outer docker run, then the workflow step. The container is given a budget 30s shorter than the outer docker run timeout, so the helper that knows which stage it was prints the expiry before the outer process group is terminated.

When a command exceeds its bound, the helper prints that it is about to clean up before it starts the risky part, so a run killed during cleanup still leaves the reason in the log. It then creates one monotonic cleanup deadline of 60s and spends it on everything that follows: the cooperative SIGTERM, the forced SIGKILL, the reap, the output drain, and the container listing, removal and confirmation. Every wait and every Docker call inside that window is capped by the time still left on it, and every poll is truncated to it, so no step can run past the deadline. A step that no longer fits is skipped and reported instead. A timed-out stage is never retried.

The stop is only claimed when the exit was confirmed. If the group could not be reaped inside the deadline — a process stuck in an uninterruptible sleep, for instance — the helper reports cleanup unconfirmed and says that anything the stage started in the Docker daemon may still exist. A reaped process is also not treated as proof that its containers are gone: resource cleanup is reported on its own evidence.

The Docker calls the cleanup makes run through the same runner, and they are handed the same deadline rather than a fresh allowance, so a listing or removal that itself times out spends the time that is left — including its own stop and drain — and cannot extend the cleanup past its end.

What this cannot promise: if the helper itself is killed with SIGKILL, no cleanup code runs at all. Nothing here claims otherwise; a container left that way is identifiable by its label (see below) and is not swept by a later run.

When the budget itself runs out the helper prints the stage it was about to run and how little time was left, and that the stage and every later one did not run. That is a bounded failure, not a stage verdict: a cold container build plus the whole gate may not fit 1500s, and the honest result is that the remaining stages were not attempted. Raising the workflow duration to fit a cold run is a separate change that needs its own approval; this helper never raises its own limit, and nothing here changes the workflow files.

Command output is streamed live as the child writes it, so a step that is still running is visible and a workflow that kills the helper still leaves the log behind. Volume is bounded twice:

• at most 256 KiB is echoed live per command;
• at most a 64 KiB head and a 64 KiB tail are retained in memory for the failure diagnostic, and the number of bytes withheld is always reported.

Nothing is spooled to an unbounded temp file, and nothing is printed twice: the diagnostic prints only the retained tail that the live echo did not already show, which is also what keeps a nested log (the inner _container helper inside the outer docker run) from being replayed. Output is decoded incrementally, so a multi-byte character split across two reads is echoed whole; the returned head and tail are cut at byte offsets and can still split a character at the truncation point, which is why the withheld byte count is reported. The helper never prints the environment.

Per-stage timing and read-only resource snapshots #

Every stage prints one line when it returns:

[run_ci] elapsed 512.3s status=0 'make check-all'

elapsed is wall time on one monotonic clock, from just before the child was started to the moment the helper finished with it. For a stage that had to be stopped it therefore includes the bounded cleanup, and it is the stage's own wall time — not the child's CPU time, and not a per-command figure for the targets a stage runs inside itself. A stopped stage adds (timed out) and reports status=124.

The stage this helper marks as measured — make check-all, named in MEASURED_ARGV next to the command definitions — prints one more line at each edge of its window. [run_ci] resources before 'make check-all': … appears before the child starts, and the same allowlist is appended to the elapsed line as resources: with a delta: of the counters that moved:

[run_ci] elapsed 512.3s status=0 'make check-all' resources: psi.cpu.some.avg60=0.02 … delta: psi.cpu.some.total=+4200 cgroup.memory.events.oom_kill=+0

Only these fixed paths are read, and only these keys of them:

Path	Keys reported
/proc/pressure/cpu, io, memory	some and full: avg10, avg60, avg300, total
/sys/fs/cgroup/{cpu,io,memory}.pressure	the same shape, for the visible cgroup mount root
/sys/fs/cgroup/memory.events	low, high, max, oom, oom_kill
/sys/fs/cgroup/cpu.stat	usage_usec, nr_periods, nr_throttled, throttled_usec
/sys/fs/cgroup/memory.current, memory.max	the value, where max means no limit

Cgroup fields describe the visible mount root, which is container-local only when the runner provides a private cgroup namespace. Otherwise they may describe an ancestor or host root; these measurements do not establish that namespace.

Nothing about the read is discoverable or operator-chosen: no path is globbed, listed, or taken from an argument or the environment, so no input can redirect it. Each read is a single synchronous os.open with O_NOFOLLOW — a symlink is refused rather than followed — capped at 4096 bytes, and a capped read that ends mid-value drops that line instead of reporting a number the kernel never wrote. The rendered block is capped at 4096 characters and states how many fields were withheld; a file that cannot be read contributes no fields, and when none can be read the line says resources: unavailable. Two synchronous snapshots at a stage's edges are the whole cost: there is no sampler, no background thread, no network call, and no telemetry.

Only a value that is a plain number is reported, so free text from a file — a name, a path, a host, an address — cannot reach the log. What the numbers are is what the kernel reported at those two instants: they describe the load present around the stage, and the delta: is the movement of monotonic counters across that window. Neither is evidence that the stage caused that load, and no verdict is drawn from them. They exist so a bounded timeout can be read next to the pressure that accompanied it.

This reporting adds nothing to RunOutcome or to any record: an injected runner still unpacks as (code, output), and only the log gained lines. The budgets above are unchanged — a measured stage costs the same time it did before.

Container ownership and teardown #

Killing the local docker client does not stop the container the daemon started, and --rm only removes a container that exits, so neither one is enough on its own. Every container this helper starts is created with both ownership marks:

• a unique name, ultimate-tool-ci-<action>-<pid>-<12 hex>;
• the labels ultimate-tool-ci.run=<pid>-<12 hex> and ultimate-tool-ci.stage=<action>.

The token is generated per invocation, so no two runs can address each other's containers. It is a namespace, not a credential: it selects containers, and it authorises nothing.

Removal happens in a finally block, independently of whether the client exited cleanly, failed, or was killed, and inside the single cleanup deadline:

1. list the containers carrying this invocation's label (docker ps --all --quiet --no-trunc --filter label=ultimate-tool-ci.run=<token>);
2. remove exactly the identifiers that listing printed (docker rm --force <id>...), accepting only values that match an identifier pattern and never a guess;
3. re-probe until nothing carrying the token remains, then report removed.

The outcome is removed, absent, or unconfirmed, and only removed and absent are treated as confirmed. unconfirmed is an error (exit 3), not a warning, and it is what you get when:

• the daemon cannot be asked, or its listing is truncated or contains something that is not an identifier (fail closed: nothing is removed on a guess);
• a docker rm returns an ambiguous status — it is never retried, because a second attempt could remove a container that appeared after the first;
• the process that owned the containers could not be confirmed stopped;
• the command timed out at all. A confirmed, unforced stop is a clean process fact, not a clean resource fact: the run may have been killed with a container half-created;
• an observation window closes without ever seeing a container. A container killed mid-create can appear later, so "nothing listed yet" is a bounded observation, not proof of absence.

There is no docker system prune, no filter wildcard, and no cross-invocation cleanup: the only identifiers this can touch are ones the daemon printed for this invocation's own random label. A label prefix a teardown may filter on must be on the helper's allowlist, so a typo cannot turn into a broad sweep.

The runtime harness's own containers #

tools/verify_release_local.py labels the containers it starts with the key-only label ultimate-tool-verify-local.<id>. Its id is normally minted inside its own process, which the caller cannot know — so the CI action supplies one and makes the division of ownership explicit:

• --invocation-id <token>: the caller-chosen id. It is validated (1-64 characters of a-z, 0-9 and -) before any journal, scratch directory, image or container exists, so an unacceptable id cannot leave a resource behind under a name it chose. Omitted, the harness generates its own id exactly as before.
• --external-container-cleanup: requires --mode runtime and an explicit --invocation-id. In this mode the supervisor is the sole remover: the harness skips its three container-removal paths (pre-clean, the registered cleanup action, and the post-run sweep) and keeps its read-only checks and its scratch-directory cleanup. Two removers would make every docker rm result ambiguous, which is exactly what this switch prevents. Standalone runs are unchanged.

The CI helper therefore passes both flags, keeps the harness's scratch HOME alive until the teardown has finished, and removes exactly ultimate-tool-verify-local.<token> inside the same cleanup deadline that stopped the harness. A harness that had to be forced, or whose stop could not be confirmed, leaves its resource cleanup explicitly unconfirmed even when the listing is empty.

7. Source manifest gate #

The release source manifest denies the maintainer development manifest at the extracted source: tools/release_source_manifest.json lists c64dev.toml under source_paths.deny (the packaged first-run example manifests replace it). The structural tests assert that deny entry still exists, so the gate cannot be silently dropped.

8. Secrets and environment #

• The workflows declare no environment block and no secrets. Nothing credential-shaped appears in any workflow file, and the tests enforce that.
• Child processes receive only an allowlist of environment variables (PATH, HOME, LANG, LC_ALL, LC_CTYPE, TERM, TMPDIR, CI, plus TANGLED_*). Anything whose name looks like a password, token, secret, credential, or API key is dropped, and the whole C64U_* namespace is dropped. C64U_PASSWORD therefore never reaches a child process.
• Commands are argv arrays. There is no shell=True, no eval, and no command assembled by string concatenation.
• The helper refuses any step whose --target is ultimate; CI runs only the fake and VICE targets.

9. Discovery and pilot gate #

Read-only inspection established the nixos image name and its resource spec. The image requests 4,096 MiB plus the engine's 96 MiB runner allowance. Both server memory caps were separately approved and raised to 4,192 MiB; CPU, disk and timeout were unchanged. The immediate service-only runtime override is volatile across reboot, while the Nix configuration source records the two settings for a later separately approved normal rebuild.

Repository linkage and HTTPS cloning were verified after a bounded approved DHCP recovery. The successful manual smoke is recorded above; those results justify preparing the full check, not claiming that it has passed. Automatic trigger activation remains separately gated.

10. What this scaffold does not do #

• It does not configure a spindle, register a workflow, or trigger a run.
• The pilot selects an observed installed image name, not a reproducible image digest; it does not certify hosted execution.
• It does not publish a release or a release artifact.
• It does not touch a C64 Ultimate device.
• It does not claim that any check has passed on a hosted runner. The structural tests in tests/python/test_ci_workflow.py check the workflow and helper specifications; they never execute Docker, a workflow, or any infrastructure.