# Continuous integration (local Tangled scaffold)

This document describes the **local** Tangled CI scaffold: three workflow
files under `.tangled/workflows` and the helper they call, `tools/run_ci.py`.

> **Hosted smoke passed; full check pending.** Pipeline `3mxwqmomeci22` passed
> `smoke.yml` for commit `2db36210f904ec4ddfbc1715daeb9d123117cf42` on
> 2026-10-15 in 10m51s: image setup, CLI build, manifest validation and the fake
> pass/crash/panic/recovery sequence. The installed `nixos` image allocates
> 4 GiB RAM, two vCPUs and a 24 GiB writable volume; admission includes a 96 MiB
> runner allowance. `check.yml` now uses the same observed image and
> guest dependencies, but has not passed hosted execution. The helper spends at
> most 4800 of the 5400 seconds the temporary qualification allowance grants
> (section 6), to allow a complete cold-run baseline; completion within that
> window is not guaranteed. Emulator
> and physical acceptance remain separate. Local success is not hosted-CI
> evidence.
>
> **The 5400s/4800s ceilings are a temporary qualification budget, not the
> normal duration.** They apply only while the full check is being qualified on
> the hosted runner and require a volatile server-side override. The server
> returns to its 1800s ceiling after the run; the helper's final budget remains a
> separate reviewed decision. Section 6.1 states the rollback and what was deliberately
> not changed.

## 1. What exists

| Path | Purpose |
| --- | --- |
| `.tangled/workflows/smoke.yml` | Build, validate, and drive the first-run fake. |
| `.tangled/workflows/check.yml` | Full gate plus packaging, verification, and the runtime harness. |
| `.tangled/workflows/emulator.yml` | VICE debugging and checkpoint acceptance. |
| `tools/run_ci.py` | The single entry point every workflow invokes. |
| `tests/python/test_ci_workflow.py` | Structural checks over the workflows and the helper. |

## 2. Workflow shape

Each workflow uses the microVM engine. Smoke and full check explicitly select
the observed `nixos` image and declare `python3`, `git` and `gnumake`
dependencies. The emulator workflow remains separately gated. The manual
trigger shape is:

```yaml
when:
  - event: ["manual"]

engine: "microvm"

virtualisation:
  docker: true

steps:
  - name: "Assert guest preconditions"
    command: "python3 tools/run_ci.py guest-check"
  - name: "..."
    command: "python3 tools/run_ci.py <action>"
```

- **The pilot uses the observed installed name.** `smoke.yml` selects `nixos`;
  inspection found that name points to a NixOS x86-64 image with a `spec.json`.
  This is an installed-image observation, not a reproducible image build claim.
  The emulator workflow still uses the configured default pending its separate gate.
  No image is provisioned or server setting changed by these files.
- **`virtualisation.docker: true`** brings up Docker inside the NixOS guest.
- The approved target is a microVM with a **NixOS guest** and Docker in the VM.
  Because the image is not pinned, the first step asserts the guest actually is
  that shape.

## 3. Triggers are manual only

All three workflows trigger on `manual` only. Push and pull-request triggers
are deliberately absent.

A repository with Tangled CI activated runs its workflow files automatically on
push. The reviewed pilot commit and normal push are authorized, but automatic
push/PR activation is not. A workflow that only accepts `manual` cannot be
started by that push. Adding `push` (to `main`) and `pull_request` triggers is a deliberate
later step that belongs with the spindle pilot authorization, not with this
scaffold.

## 4. Guest preconditions (fail-closed)

`tools/run_ci.py guest-check` reads the running guest and fails unless all of
these hold:

- `/etc/os-release` reports `ID=nixos`;
- `platform.machine()` is `x86_64`;
- `docker` is on `PATH`;
- a read-only `docker info` against `unix:///var/run/docker.sock` succeeds for
  the workflow user within 15 seconds. Root is not required; the hosted runner
  was observed to execute steps as UID 970.

If any precondition is unmet the helper exits non-zero with the specific reason
and no step runs. It never falls back to a different distribution, a different
architecture, or a host Docker socket.

## 5. The prepared image, not the bare guest

The NixOS microVM guest is unprovisioned for Rust, so **no action compiles or
tests on the guest directly**. `smoke`, `check` and `package` build the local
release-environment image from `tools/release-env/Dockerfile` and run the tool
inside it. The container then holds no Docker daemon and no host socket: the
packaging path is a direct `cargo`/`python3` argv sequence, so nothing inside
the container needs Docker. Building and launching that container is the pilot
proof-of-concept the guest alone cannot provide.

## 6. Actions

Run any action by hand on the guest:

```sh
python3 tools/run_ci.py guest-check
python3 tools/run_ci.py smoke
python3 tools/run_ci.py check
python3 tools/run_ci.py emulator --require-vice-version 3.8
python3 tools/run_ci.py package
```

Every action also accepts `--budget-seconds N` for a directly invoked run. It
can only shorten the 4800s budget, never extend it; the budget is derived from
the workflow step ceiling below.

### `smoke`

Builds the prepared image, then runs inside it:

1. `cargo build --locked -p ultimate-tool` — build the actual tool;
2. `cargo run -q -p ultimate-tool -- validate` — a read-only manifest check;
3. `python3 tools/run_demo.py --manifest examples/first-run/c64dev.toml
   --binary target/debug/ultimate-tool` — the **prebuilt first-run manifest**
   driven against the deterministic fake target.

### `check`

Builds the prepared image, then runs inside it (`--rm --network default`,
only the source tree mounted at `/work`, using the invoking guest user's
numeric UID/GID so the checkout remains writable):

1. `npm ci` — an **explicit** browser-tooling setup stage, so dependencies come
   from the lockfile rather than a hidden download inside a make target;
2. `make check-all`;
3. `make demo`;
4. the release path (see below).

Afterwards, on the guest (where the Docker daemon lives), `check` runs the
runtime harness:

```sh
python3 tools/verify_release_local.py --mode runtime \
  --release-dir dist --out /tmp/ultimate-tool-runtime-verification.json --json
```

The `check-browser` half of `make check-all` needs a Playwright browser
(`npm run setup:browser`, Playwright 1.64.0). Explicit image setup installs the
matching Chromium and operating-system dependencies. The container action sets
`PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright`; missing browser inputs fail rather
than silently skipping the browser tests.

### The release path (no Docker invokes)

Inside the container the packaging path is a direct argv sequence — it never
calls `make dist`, because that target would reach for Docker, which the
container does not have:

| Step | Command |
| --- | --- |
| release build | `cargo build --locked --release --target-dir target/release-ubuntu-2404 -p ultimate-tool -p ultimate-tool-mcp` |
| source archive | `python3 tools/package_release.py --prepare-source --output dist --development ci-local` |
| build-environment descriptor | `/usr/local/bin/ultimate-tool-build-env /tmp/ultimate-tool-build-env.json` |
| package binaries | `python3 tools/package_release.py --pack --binary-dir target/release-ubuntu-2404/release --build-env /tmp/ultimate-tool-build-env.json --output dist --development ci-local` |
| verify | `python3 tools/check_release.py --release-dir dist --mode development` |

This is a development packaging smoke, not the source-complete release gate.
The container builds from its checkout; `make dist` instead freezes the source
archive first and builds from its isolated, vendored extraction. Dependency
source paths can change binary bytes, so those outputs are not assumed identical.
A candidate must still pass the independent extracted-source rebuild and runtime
checks described in [local release evidence](LOCAL_RELEASE_EVIDENCE.md).
The packager requires a fresh `dist` and rejects conflicting files.

The **build-environment descriptor is written to a temp path outside `dist`**,
because the packager requires its output directory to be new or empty; writing
the descriptor into `dist` first would make that impossible. The descriptor is
the *observed* environment written by the writer shipped in the image, not the
release source manifest's expectations.

The workflow calls the tools directly inside the container. It does not mount
or invoke the host Docker socket, and it cannot perform publication.

### `emulator`

Runs the debugging and checkpoint acceptance script
(`tools/vice_debugging_and_checkpoint_acceptance.py`) on the guest. It requires
a **prepared** VICE 3.8 executable and never installs or substitutes one:

- supply the executable with `--vice-executable PATH` or the `VICE_EXECUTABLE`
  environment variable, and the version with `--require-vice-version`
  (default `3.8`);
- the helper queries the executable's own `--version`, compares the first
  `MAJOR.MINOR` it finds, and fails closed on any mismatch;
- the helper then runs the acceptance script passing the executable through its
  own `--vice-executable` flag — **the script does not read the environment** —
  and the script owns its source-build bug / corrected / recovery / repeat
  checks against exact VICE 3.8.

### `package`

Builds the prepared image and runs only the release path inside it, then the
runtime harness on the guest. It never runs the `release` preflight mode and
never invokes the publisher (`tools/publish_tangled_artifacts.py`). Publication
remains a separate, separately-approved step.

### Deadlines, logs, and what a bounded failure means

One monotonic budget covers a whole action. It is created once, spent as the
action runs, never reset by a stage, and every child timeout is the smaller of
that stage's own ceiling and the time still left.

| Limit | Seconds | Meaning |
| --- | --- | --- |
| workflow step ceiling | 5400 | the *temporary qualification* per-step admission allowance; requires a server-side override (section 6.1) |
| reserve inside that ceiling | 600 | guest setup, teardown and image removal |
| `CI_BUDGET_SECONDS` | 4800 | the helper's budget for one action (80 minutes) |
| container handoff margin | 30 | the inner helper's budget is the `docker run` timeout minus this |
| minimum usable remainder | 5 | below this no further stage is started |
| cleanup reserve | 60 | the most any one command may spend stopping and removing what it owns |

Per-command ceilings, all at or below the 4800s budget: validate 120, guest
docker probe 15, VICE version query 120, build-environment descriptor 120,
first-run demo 300, `make demo` 900, `npm ci` 900, debug build 900, release
check 600, package 1200, runtime harness 1200, and 4800 for `make check-all`,
the release build, the VICE acceptance run, `docker build` and `docker run`.

Those last five are written as `CI_BUDGET_SECONDS`, so they follow the budget:
they were 1500s under the previous ceiling and are 4800s under the qualification
budget without an edit of their own. The fixed stage ceilings — package 1200,
runtime harness 1200, and the smaller ones — were **deliberately left alone**.
No observed run has shown them too short, and a stage still gets the smaller of
its own ceiling and the remaining budget, so a longer action does not extend any
single stage by itself. The cleanup reserve (60s), the container handoff margin
(30s), and the minimum remainder (5s) are unchanged too: they bound how a stop
happens, not how much work fits, so raising them with the duration would only
make every timeout less predictable.

So the deadlines nest: the innermost stage ceiling first, then what is left of
the helper budget, then the outer `docker run`, then the workflow step. The
container is given a budget 30s shorter than the outer `docker run` timeout, so
the helper that knows which stage it was prints the expiry before the outer
process group is terminated.

### 6.1 The qualification budget is temporary

The 5400s step ceiling and the 4800s helper budget exist for one purpose: to let
a cold container build plus the whole gate finish inside a single hosted
admission window while `check.yml` is being qualified. They are a **temporary
qualification allowance**, not the functional normal duration.

- **Server override.** The 5400s ceiling is not the server's default admission
  limit. It requires a per-step timeout override applied on the server, which is
  volatile: it does not survive a reboot. The persistent Nix timeout configuration
  is not changed for this qualification. Nothing
  in `.tangled/workflows` or in this helper requests or sets it — the helper only
  derives its own budget from the constant in `tools/run_ci.py`.
- **Rollback after the run.** Remove the server override and restore its 1800s
  ceiling after confirming the service is idle. The repository's 4800s helper
  budget remains clearly temporary until measured results support a separately
  reviewed final setting. It cannot extend the restored server limit: another
  full run requires a coherent server/helper budget and fresh approval.
- **Not functional normal.** A 90-minute CI window is not the duration ordinary
  work should wait on. It is bounded, it is labelled, and it is expected to
  shrink again; the documents should not be read as claiming that a 90-minute
  full gate is the intended steady state.
- **Unchanged by this budget.** The workflow files, the image, the guest, the
  cleanup reserve, the handoff margin, and the fixed per-stage ceilings are
  exactly as they were. This is a duration change only.

When a command exceeds its bound, the helper prints that it is about to clean up
**before** it starts the risky part, so a run killed during cleanup still leaves
the reason in the log. It then creates **one** monotonic cleanup deadline of
60s and spends it on everything that follows: the cooperative `SIGTERM`, the
forced `SIGKILL`, the reap, the output drain, and the container listing,
removal and confirmation. Every wait and every Docker call inside that window is
capped by the time still left on it, and every poll is truncated to it, so no
step can run past the deadline. A step that no longer fits is skipped and
reported instead. A timed-out stage is never retried.

The stop is only *claimed* when the exit was confirmed. If the group could not
be reaped inside the deadline — a process stuck in an uninterruptible sleep, for
instance — the helper reports `cleanup unconfirmed` and says that anything the
stage started in the Docker daemon may still exist. A reaped process is also
**not** treated as proof that its containers are gone: resource cleanup is
reported on its own evidence.

The Docker calls the cleanup makes run through the same runner, and they are
handed the *same* deadline rather than a fresh allowance, so a listing or
removal that itself times out spends the time that is left — including its own
stop and drain — and cannot extend the cleanup past its end.

What this cannot promise: if the helper *itself* is killed with `SIGKILL`, no
cleanup code runs at all. Nothing here claims otherwise; a container left that
way is identifiable by its label (see below) and is not swept by a later run.

When the budget itself runs out the helper prints the stage it was about to
run and how little time was left, and that the stage and every later one
did not run. That is a **bounded failure, not a stage verdict**: even 4800s may
not cover a cold container build plus the whole gate, and the honest result is
that the remaining stages were not attempted. Raising the workflow duration
further is a separate change that needs its own approval; this helper never
raises its own limit, and nothing here changes the workflow files.

Command output is **streamed live** as the child writes it, so a step that is
still running is visible and a workflow that kills the helper still leaves the
log behind. Volume is bounded twice:

- at most 256 KiB is echoed live per command;
- at most a 64 KiB head and a 64 KiB tail are retained in memory for the
  failure diagnostic, and the number of bytes withheld is always reported.

Nothing is spooled to an unbounded temp file, and nothing is printed twice: the
diagnostic prints only the retained tail that the live echo did not already
show, which is also what keeps a nested log (the inner `_container` helper
inside the outer `docker run`) from being replayed. Output is decoded
incrementally, so a multi-byte character split across two reads is echoed whole;
the returned head and tail are cut at byte offsets and can still split a
character at the truncation point, which is why the withheld byte count is
reported. The helper never prints the environment.

### Per-stage timing and read-only resource snapshots

Every stage prints one line when it returns:

```
[run_ci] elapsed 512.3s status=0 'make check-all'
```

`elapsed` is wall time on one monotonic clock, from just before the child was
started to the moment the helper finished with it. For a stage that had to be
stopped it therefore includes the bounded cleanup, and it is the stage's own
wall time — not the child's CPU time, and not a per-command figure for the
targets a stage runs inside itself. A stopped stage adds `(timed out)` and
reports `status=124`.

The stage this helper marks as measured — `make check-all`, named in
`MEASURED_ARGV` next to the command definitions — prints one more line at each
edge of its window. `[run_ci] resources before 'make check-all': …` appears
before the child starts, and the same allowlist is appended to the elapsed line
as `resources:` with a `delta:` of the counters that moved:

```
[run_ci] elapsed 512.3s status=0 'make check-all' resources: psi.cpu.some.avg60=0.02 … delta: psi.cpu.some.total=+4200 cgroup.memory.events.oom_kill=+0
```

Only these fixed paths are read, and only these keys of them:

| Path | Keys reported |
| --- | --- |
| `/proc/pressure/cpu`, `io`, `memory` | `some` and `full`: `avg10`, `avg60`, `avg300`, `total` |
| `/sys/fs/cgroup/{cpu,io,memory}.pressure` | the same shape, for the visible cgroup mount root |
| `/sys/fs/cgroup/memory.events` | `low`, `high`, `max`, `oom`, `oom_kill` |
| `/sys/fs/cgroup/cpu.stat` | `usage_usec`, `nr_periods`, `nr_throttled`, `throttled_usec` |
| `/sys/fs/cgroup/memory.current`, `memory.max` | the value, where `max` means no limit |

Cgroup fields describe the visible mount root, which is container-local only
when the runner provides a private cgroup namespace. Otherwise they may describe
an ancestor or host root; these measurements do not establish that namespace.

Nothing about the read is discoverable or operator-chosen: no path is globbed,
listed, or taken from an argument or the environment, so no input can redirect
it. Each read is a single synchronous `os.open` with `O_NOFOLLOW` — a symlink is
refused rather than followed — capped at 4096 bytes, and a capped read that ends
mid-value drops that line instead of reporting a number the kernel never wrote.
The rendered block is capped at 4096 characters and states how many fields were
withheld; a file that cannot be read contributes no fields, and when none can be
read the line says `resources: unavailable`. Two synchronous snapshots at a
stage's edges are the whole cost: there is no sampler, no background thread, no
network call, and no telemetry.

Only a value that is a plain number is reported, so free text from a file — a
name, a path, a host, an address — cannot reach the log. What the numbers *are*
is what the kernel reported at those two instants: they describe the load
present around the stage, and the `delta:` is the movement of monotonic counters
across that window. Neither is evidence that the stage caused that load, and no
verdict is drawn from them. They exist so a bounded timeout can be read next to
the pressure that accompanied it.

This reporting adds nothing to `RunOutcome` or to any record: an injected runner
still unpacks as `(code, output)`, and only the log gained lines. The budgets
above are unchanged — a measured stage costs the same time it did before.

### Container ownership and teardown

Killing the local `docker` client does not stop the container the daemon
started, and `--rm` only removes a container that exits, so neither one is
enough on its own. Every container this helper starts is created with both
ownership marks:

- a unique name, `ultimate-tool-ci-<action>-<pid>-<12 hex>`;
- the labels `ultimate-tool-ci.run=<pid>-<12 hex>` and
  `ultimate-tool-ci.stage=<action>`.

The token is generated per invocation, so no two runs can address each other's
containers. It is a namespace, not a credential: it selects containers, and it
authorises nothing.

Removal happens in a `finally` block, independently of whether the client
exited cleanly, failed, or was killed, and inside the single cleanup deadline:

1. list the containers carrying *this* invocation's label
   (`docker ps --all --quiet --no-trunc --filter label=ultimate-tool-ci.run=<token>`);
2. remove exactly the identifiers that listing printed
   (`docker rm --force <id>...`), accepting only values that match an
   identifier pattern and never a guess;
3. re-probe until nothing carrying the token remains, then report `removed`.

The outcome is `removed`, `absent`, or `unconfirmed`, and only `removed` and
`absent` are treated as confirmed. `unconfirmed` is an error (exit 3), not a
warning, and it is what you get when:

- the daemon cannot be asked, or its listing is truncated or contains something
  that is not an identifier (fail closed: nothing is removed on a guess);
- a `docker rm` returns an ambiguous status — it is **never retried**, because a
  second attempt could remove a container that appeared after the first;
- the process that owned the containers could not be confirmed stopped;
- the command timed out at all. A confirmed, unforced stop is a clean *process*
  fact, not a clean *resource* fact: the run may have been killed with a
  container half-created;
- an observation window closes without ever seeing a container. A container
  killed mid-create can appear later, so "nothing listed yet" is a bounded
  observation, not proof of absence.

There is **no `docker system prune`, no filter wildcard, and no cross-invocation
cleanup**: the only identifiers this can touch are ones the daemon printed for
this invocation's own random label. A label prefix a teardown may filter on must
be on the helper's allowlist, so a typo cannot turn into a broad sweep.

### The runtime harness's own containers

`tools/verify_release_local.py` labels the containers it starts with the
key-only label `ultimate-tool-verify-local.<id>`. Its id is normally minted
inside its own process, which the caller cannot know — so the CI action supplies
one and makes the division of ownership explicit:

- `--invocation-id <token>`: the caller-chosen id. It is validated (1-64
  characters of `a-z`, `0-9` and `-`) **before any journal, scratch directory,
  image or container exists**, so an unacceptable id cannot leave a resource
  behind under a name it chose. Omitted, the harness generates its own id
  exactly as before.
- `--external-container-cleanup`: requires `--mode runtime` and an explicit
  `--invocation-id`. In this mode the supervisor is the **sole remover**: the
  harness skips its three container-removal paths (pre-clean, the registered
  cleanup action, and the post-run sweep) and keeps its read-only checks and its
  scratch-directory cleanup. Two removers would make every `docker rm` result
  ambiguous, which is exactly what this switch prevents. Standalone runs are
  unchanged.

The CI helper therefore passes both flags, keeps the harness's scratch `HOME`
alive until the teardown has finished, and removes exactly
`ultimate-tool-verify-local.<token>` inside the same cleanup deadline that
stopped the harness. A harness that had to be forced, or whose stop could not be
confirmed, leaves its resource cleanup explicitly unconfirmed even when the
listing is empty.

## 7. Source manifest gate

The release source manifest denies the maintainer development manifest at the
extracted source: `tools/release_source_manifest.json` lists `c64dev.toml` under
`source_paths.deny` (the packaged first-run example manifests replace it). The
structural tests assert that deny entry still exists, so the gate cannot be
silently dropped.

## 8. Secrets and environment

- The workflows declare no `environment` block and no secrets. Nothing
  credential-shaped appears in any workflow file, and the tests enforce that.
- Child processes receive only an allowlist of environment variables
  (`PATH`, `HOME`, `LANG`, `LC_ALL`, `LC_CTYPE`, `TERM`, `TMPDIR`, `CI`, plus
  `TANGLED_*`). Anything whose name looks like a password, token, secret,
  credential, or API key is dropped, and the whole `C64U_*` namespace is
  dropped. `C64U_PASSWORD` therefore never reaches a child process.
- Commands are argv arrays. There is no `shell=True`, no `eval`, and no
  command assembled by string concatenation.
- The helper refuses any step whose `--target` is `ultimate`; CI runs only the
  fake and VICE targets.

## 9. Discovery and pilot gate

Read-only inspection established the `nixos` image name and its resource spec.
The image requests 4,096 MiB plus the engine's 96 MiB runner allowance. Both
server memory caps were separately approved and raised to 4,192 MiB; CPU, disk
and timeout were unchanged. The immediate service-only runtime override is
volatile across reboot, while the Nix configuration source records the two
settings for a later separately approved normal rebuild. The per-step timeout
has since moved separately from that change: the qualification budget in
section 6.1 needs its own volatile server-side override and reverts to 1800s
after the qualification run.

Repository linkage and HTTPS cloning were verified after a bounded approved
DHCP recovery. The successful manual smoke is recorded above; those results
justify preparing the full check, not claiming that it has passed. Automatic
trigger activation remains separately gated.

## 10. What this scaffold does not do

- It does not configure a spindle, register a workflow, or trigger a run.
- The pilot selects an observed installed image name, not a reproducible image digest; it does not certify hosted execution.
- It does not publish a release or a release artifact.
- It does not touch a C64 Ultimate device.
- It does not claim that any check has passed on a hosted runner. The
  structural tests in `tests/python/test_ci_workflow.py` check the workflow
  and helper *specifications*; they never execute Docker, a workflow, or any
  infrastructure.