# Infrastructure

The machines the generation loop runs on, and how work reaches them. The loop itself is
in `docs/generation-loop.md`; measured rates are in `docs/rig-capacity.md`. Hardware is
named by role and class, never by hostname, address or account — this repository is
assumed public. Real names live in the gitignored `bench/*.local.json` and the local rig
notes.

## In brief

- **The loop.** One engine function per spec algorithm (for example the CSS 2.1 width
  calculation), written by a model, checked against cached browser results.
- **Sampling tier.** GPU cards running mid-sized models write N versions of a function
  from the spec text, the engine's Rust interfaces and the currently failing cases. Many
  cheap tries beat one careful one, because the browsers say which tries are right.
- **Capacity tier.** When every version fails, a larger model gets the spec, the best
  failing versions and the exact per-box geometry differences.
- **Verification tier.** CPU servers compile candidates in one batch and run each inside
  the engine over the cached corpus, under a millisecond per page. The same tier builds
  that cache by running the three browsers once per page, and runs conformance suites.
- **Outcomes.** Keep the passing version, sample again with the new failures, or
  escalate. What is kept is ordinary Rust with a provenance record; no model runs in the
  shipped engine.

Environment variables are named `WEBENGINE_*`. Older records use `NEZE_*`
(`docs/rename-to-bez.md`), `NEV_*` (`docs/rename-to-neze.md`) or `GENERA_*`
(`docs/rename-to-nev.md`); nothing reads any of them any more.

## Detail

### The full cycle

```mermaid
flowchart TD
  browsers["Chromium, Firefox, WebKit<br/>once per page (CPU)"] --> cache["cached ground truth<br/>three-engine vote"]
  spec["spec text<br/>one algorithm"] --> sample["SAMPLE (GPU pool)<br/>N versions of one function"]
  sample --> verify["VERIFY (CPU)<br/>dedupe, batch compile,<br/>run over the corpus"]
  cache --> verify
  verify -- "all pass" --> accept["commit Rust source<br/>+ provenance record"]
  verify -- "some fail" --> sample
  verify -- "all fail repeatedly" --> escalate["ESCALATE<br/>larger model + geometry diffs"]
  escalate --> verify
```

Nothing is accepted on a model's word: escalated output goes through the same check.
Writing candidates is the binding constraint; verification has a large margin over it.

### What comes out, and how the engine uses it

The cycle outputs **ordinary Rust source**, committed next to the hand-written code it
plugs into, with spec steps as comments and a **provenance record** (`roadmap.md` →
"Provenance is a first-class artifact"). The engine compiles it like any other code; the
GPU and CPU fleets exist only at build time.

**Provenance drives maintenance.** When a spec section or a definition it uses changes,
the provenance index finds exactly the functions to regenerate (`roadmap.md` → "A
specification changes"); a browser update re-dumps the ground truth instead. The same
index is the bill of materials for a configurable build (`roadmap.md` → "Phase 4").

## Measured baseline

One oracle process, three engines, 235 documents including browser startup: **17.9 s**,
about **13 documents per second per engine**, using about two cores. Scaled to a few
hundred cores that would be about 3,000 documents per second, a ten-million-case corpus in
about an hour — but per-core throughput on the contended 16-core development box fell from
1.48 to 0.48 docs/s/core between 1 and 16 workers (`docs/rig-capacity.md` → "Oracle
corpus building"), so the figure on dedicated hardware is still to be measured. GPU-side
figures: `docs/rig-capacity.md` → "GPU tier".

## Three tiers

**Sampling tier — high memory bandwidth GPUs.** Bulk best-of-N generation,
throughput-bound. Decode speed tracks *active* parameters, so a mixture-of-experts model
with a small active fraction gives several times the candidates of a dense model. Ampere
cards have no fp8, so quantization is 4-bit weight-only.

**Capacity tier — large unified-memory accelerators.** About a third of the bandwidth,
several times the memory: for the few hard calls and any teacher model. About 100B total
with ~12B active is the realistic ceiling.

**Verification tier — many CPU cores.** Corpus building, candidate compilation and the
inner loop. It must never spawn a browser per case.

## As discovered

Every host advertises itself over mDNS and answers an HTTP inventory; a collector
aggregates it. What that shows, by role:

- **The model gateway** is a LiteLLM instance fronting one alias per model pool. The
  aliases, their pools, how many requests each takes at once, and which are retired are
  declared in `bench/folknet.json` (the two GLM-5.3 Flash aliases, without backends since
  2026-09-22, were removed from the gateway on 2026-09-25); whether each one answers
  right now is in each lane's snapshot, which a probe refreshes every 5 minutes and
  `bench/folknet-status.sh [--lane N]` prints (status, how long it has held it, last
  answer, snapshot age). Design: `docs/folknet-inventory.md`. Virtual keys carry per-key
  allowlists, so a new alias must be granted to the caller's key. Each pool's HAProxy
  reports liveness, not readiness: a `/v1/models` answer does not prove a completion
  will return.
- **Four ARM unified-memory nodes (DGX Spark class)** — `aarch64`, one GB10 GPU, 20
  cores, 121 GB RAM, 3.7 TB NVMe each. They served GLM-5.3 Flash as two independent
  two-node pairs until 2026-09-22. Qwen3.8-Flash-Next did not run under any vLLM build on
  the rig (the architecture was too new for the rig's EXL3 fork, and upstream has no EXL3
  backend), so it serves through the checkpoint's own engine (TabbyAPI) instead, one
  replica per node, about 75 GB plus ~45 GB of page cache each. Two of the four now serve
  DeepSeek-V4.1-Flash. Full account: `docs/rig-capacity.md` → "Attempted —
  Qwen3.8-Flash-Next".
- **The eight-card host** — eight 24 GB consumer cards on a 64-core EPYC with ~500 GB
  RAM, bare Linux, six serving containers using every card (four behind `-fast`, two
  behind `-long`). The closest thing to the sampling tier as designed. No spare GPU.
- **The verification tier** — ten Proxmox 9.x nodes (EPYC 7742, 7B13, 7K62 and 7452; 32
  to 64 physical cores, 125 GB to ~500 GB RAM, 1–2 TB NVMe), shared with tenant VMs;
  five pass thirteen consumer cards through to tenant machines. A VM sized for browser
  builds sits on the GLM-pool proxy host (open: its core count and whether it should be
  running). Free capacity is host by host.
- **A downloader host** pulls checkpoint shards in parallel across the site's four WAN
  uplinks onto the NAS, verifies them and publishes a pinned revision for nodes to sync.
- **Access.** Root with the jump host's key works on every physical host; the ARM nodes
  use their own admin key. The gateway's own guest refuses, and is not a compute host.

## Where each piece runs

Every role is a guest or a container; nothing is installed on a Proxmox node's own OS.

```mermaid
flowchart LR
  dev["development box<br/>16 cores, shared"] -- "ssh via jump host" --> lanes["2 build lanes<br/>(LXC, 96 threads each)"]
  dev -- "port forward" --> gw["model gateway<br/>(LiteLLM)"]
  lanes --> gw
  gw --> fast["eight-card host<br/>-fast, -long"]
  gw --> vm["four-card VM<br/>-fast-2"]
  gw --> fn["2 ARM nodes<br/>flash-next"]
  gw --> ds["2 ARM nodes<br/>deepseek pair"]
  lanes --> helpers["compile helpers<br/>(sccache-dist)"]
```

- **Development box.** No GPU, no rig role; holds the port forwards to the gateway. Each
  forward is an enabled systemd user unit (`rig-gateway-tunnel`, `rig-forge-tunnel`,
  `rig-lightwall-tunnel`, `rig-tunnel`, `webengine-job-events-tunnel`) with `Restart=always`
  every 10 s; user lingering is on, so the units start at boot with no login.
- **Jump host.** A VM on one node; the only route in. Nothing of this project on it.
- **Build lanes.** One unprivileged LXC on each of two nodes (`bench/lane-create-lxc.sh`,
  `bench/lane-provision.sh`), started at node boot. A gate running when a lane stops
  fails.
- **Compile helpers and scheduler.** Four more containers plus builders on the two lanes;
  scheduler, redis cache and a loopback CONNECT proxy on one lane (`bench/lane-dist.sh`).
- **Eight-card host.** Docker containers under a templated unit
  `folknet-qwen38-vllm-worker@<profile>`, behind that host's own HAProxy on port 4000.
  Not edited by this project.
- **ARM nodes, split two ways since 2026-09-24.**
  - Two serve `qwen3.8-flash-next`, one systemd unit and container each on port 5000
    (`bench/sparks/`), behind the proxy host's HAProxy (least connections, eight in flight
    per replica) — so the alias takes **16 in flight, not 32**; a batch at concurrency 32
    queues the rest. Either replica can stop alone.
  - Two, joined by a direct cable, serve `deepseek-v4.1-flash` as one tensor-parallel
    pair: one oneshot unit on the head node
    (`bench/sparks/dsv41/folknet-dsv41-exl3-pair.service`) starts both containers and
    returns once the API is healthy (about 7.5 minutes cold); stop and start only through
    that unit, never one node alone. HAProxy `maxconn 2`.
  - Units are stopped and started through a privileged container that chroots into the
    host, because the admin account has no sudo. GLM units are disabled on all four
    (since 2026-09-29; before that `folknet-glm53-vllm` was still enabled everywhere).
- **Four-card VM (`qwen3.8-27b-fast-2`).** Four 3090s borrowed from a stopped tenant VM on
  the same node (only one of the two may run; `bench/gpu-vm/`). Four units
  `folknet-qwen38-vllm-worker@fast-{0..3}` on ports 8101–8104, the eight-card host's
  `-fast` recipe, HAProxy `maxconn 4` per worker (16 in flight). A separate alias because
  the `-fast` load balancer lives on the eight-card host.

**Open.** Which host carries the four-card VM behind `qwen3.8-27b` (the measured Qwen
server), and how it is started and stopped; whether anything restarts the eight-card
host's HAProxy or re-pulls its image; and the gateway config
is generated from a site file outside the rig, which needs every hand-added alias too.

## Sharing the hardware

The Proxmox nodes are shared with other tenants. Three rules:

- **Nothing of this project goes on a node's own OS**, so a guest can be stopped and its
  hardware handed back without touching the node.
- **GPU work runs in a VM with passthrough, never an LXC** (an LXC would need the vendor
  driver on the node). CPU work runs in an unprivileged LXC, or in docker on non-Proxmox
  hosts.
- **Stopping a project guest frees what it held**, and a lane competes with tenants as an
  equal under contention. Nothing checks this; it is how the rig is laid out.

Limits today: the ARM replicas' HAProxy queues a request at most 30 minutes; power
budgets are modelled (`bench/capacity.py`) but not enforced.

**Not built yet:** a per-role start/stop/drain wrapper; a drain (a replica's stop gives
in-flight replies only 60 s); taking a backend out of HAProxy before stopping it; a narrow
sudo rule for the ARM nodes' admin account.

## Build lane

Two unprivileged containers (96 logical CPUs — three quarters of a 64-core, 128-thread
node — 96 GB, 200 GB disk), each a build lane and oracle host: the Rust toolchain, Node
22, the repository cloned from the rig's forge ("Where the repository is stored" below),
Playwright's three browsers, and a PulseAudio null sink so a realtime `AudioContext` runs. The scripts are the record and are
re-runnable: `bench/lane-create-lxc.sh`, `bench/lane-provision.sh`, `bench/lane-nas.sh`.
When folknet is down, the development box can serve as a lane with its own forge and Kimi
as the model: `docs/local-lane.md`.

**Running a gate there.**

```mermaid
sequenceDiagram
  participant W as worktree (dev box)
  participant L as lane
  participant F as rig forge
  W->>L: ssh: git bundle of unpushed commits + diff + untracked files
  L->>F: fetch the lane's mirror (only if needed, or older than 10 min)
  L->>L: lane-apply.sh: check out exact commit, apply diff, run command
  L-->>W: streamed output (and --pull directories)
```

`bench/remote-lane.sh <worktree> <command...>` does this. Each lane root keeps one bare
mirror (`<root>/.git-mirror`) with one git worktree per track; it fetches only when the
bundle cannot supply the commit or the last fetch is older than `WEBENGINE_LANE_FETCH_MAX_AGE`
(600 s), because the public forge the mirrors first fetched from rate-limits. The lane
address comes from `WEBENGINE_LANE_SSH` or `bench/lane.local.json`; the mirror URL from `WEBENGINE_LANE_GIT_URL`, that file, or the
worktree's `origin`. Examples:

```
bench/remote-lane.sh . cargo test --workspace
bench/remote-lane.sh --pull corpus/absolute-ground-truth . \
  cargo run --release -p generation --bin absolute_ground_truth_cache
```

Why a bundle rather than rsync: uploads from the development box to the lane run at about
18.6 KB/s (downloads 435 KB/s), so rsyncing a worktree took ~315 s of a ~424 s run.

**Where the repository is stored.** Two forges, with different jobs:

| forge | role | holds |
|---|---|---|
| the rig's Forgejo | intermediate storage, on the rig network | everything the lanes need: `main`, job branches `job/<name>` and their results `result/<name>` (`tasks/job-submit-land.md`) |
| tangled | persistent storage, public | `main`, the `spec/<name>/<sha>` landing tags, and every polled job run's result as the tag `result/<name>/<run id>` (`jobs/scripts/poll-and-land.sh` `keep_result`) |

The Forgejo repository is a regular repository, not a mirror of tangled. The lanes clone
and fetch from it: each lane root's `.git-mirror` and the lane user's clone fetch from
`http://<forge-user>:<forge-token>@<forgejo-address>/<owner>/<repo>.git`, and
`bench/lane.local.json` `git_url` (or `WEBENGINE_LANE_GIT_URL`) names the same URL. The user and
token come from the primary checkout's `.env` (`WEBENGINE_FORGE_USER`, `WEBENGINE_FORGE_TOKEN`). The development box reaches Forgejo
through a local port forward, as the remote `rig`
(`http://<forge-user>:<forge-token>@127.0.0.1:<forward-port>/<owner>/<repo>.git`), which
the job scripts use by default (`JOBS_FORGE_REMOTE`). `origin` fetches from tangled and has
push URLs for both forges, so `git push origin main` updates both; job and result branches
go to `rig` only and are deleted there after a landing. Each of those URLs carries the
token: they live in `.git/config` and the gitignored files, never in a committed file, a
log line or a message.

**Outside sources.** Every outside git repository is fetched from a pull mirror on the
rig's Forgejo, never from GitHub or another upstream, so a lane or the development box
neither meets GitHub's rate limits nor depends on it being up. Each mirror is a Forgejo
pull mirror (all history and branches, so any pinned commit stays fetchable) owned by
the same owner as the project's own Forgejo repository, and synced from upstream every
8 hours, the interval the forge's other mirrors use:

| source | pinned in | mirror repository | size on the forge |
|---|---|---|---|
| `https://github.com/web-platform-tests/wpt.git` | `oracle/suites/suites.json` `wpt` | `<owner>/web-platform-tests.wpt` | 3.0 GiB |
| `https://github.com/KhronosGroup/WebGL.git` | `oracle/suites/suites.json` `webgl` | `<owner>/KhronosGroup.WebGL` | 43 MiB |
| `https://github.com/gpuweb/cts.git` | `webgpu-cts/cts.json` | `<owner>/gpuweb.cts` | 623 MiB |

The pin files keep the upstream URL as the source's identity; `oracle/suites/mirror.mjs`
`resolveSource()` turns it into the mirror's URL, `<forge>/<owner>/<a>.<b>.git` for
upstream `…/<a>/<b>.git`. The forge and owner come from the first of
`WEBENGINE_GIT_MIRRORS` (the owner's URL, `http://<forgejo-address>/<owner>`), the
checkout's `rig` remote (the development box), `WEBENGINE_LANE_GIT_URL`, and the
checkout's `origin` remote when its URL carries credentials (a lane's worktree of its
`.git-mirror`). The mirror URL never carries credentials: the user and token (from that
URL, else `WEBENGINE_FORGE_USER` / `WEBENGINE_FORGE_TOKEN` in the environment or the
primary checkout's `.env`) reach git only as an `http.<forge>/.extraHeader` in the
environment of the one git command, so a checkout's `.git/config` on the share holds the
address and nothing else. Every fetch prints which mirror it used.
`WEBENGINE_UPSTREAM_DIRECT=1` fetches from the upstream URL instead (a source whose
mirror is missing, or the forge down) and says so on every fetch; with neither a forge
setting nor the override, a fetch fails naming both.

Callers: `oracle/suites/fetch.mjs` (and through it `bench/nas-refresh.sh suites`),
`bench/nas-refresh.sh webgpu-cts`, `oracle/suites/fetch-checklayout.sh`, and `crates/wpt-votes/src/features.rs` (its WPT tree
cache). Shell and Rust callers run git through the script:
`node oracle/suites/mirror.mjs git <upstream> <git args…>`, an argument `{url}` standing
for the mirror URL; `mirror.mjs url <upstream>` prints the URL. `fetch.mjs` points a
checkout's `origin` at the mirror before anything that reaches the network (a fetch, or a
new sparse path whose blobs a blobless checkout fetches), so a checkout made from
upstream is switched in place, never cloned again; a checkout already at its pin needs no
forge settings. `oracle/suites/mirror.test.mjs` covers the resolution;
`oracle/suites/fetch.test.mjs` and `bench/nas-refresh-test.sh` fetch through a local
directory standing in for the forge.

Adding an outside source: create its mirror before the first script that reads it lands,
with the forge's migrate API and a token holding the repository scope —
`POST /api/v1/repos/migrate` with `clone_addr` the upstream URL, `repo_owner` the
owner above, `repo_name` from `node oracle/suites/mirror.mjs name <upstream>`,
`mirror: true`, `mirror_interval: "8h0m0s"`, `service: "git"` (WPT took about 7
minutes; the call returns when the first sync is done). A pin moved to a commit newer than the mirror's last sync needs
`POST /api/v1/repos/<owner>/<name>/mirror-sync` first.

npm packages and crates are plain files on the NAS share, filled only by the scheduled
refresh (`bench/nas-refresh.sh`, below), and every install and build reads only those
files: nothing contacts npmjs or crates.io at build time, and nothing falls back to them.
A package or crate missing from the share fails the install or build, naming it.

- npm: the share's `npm-cache/`, filled by `bench/nas-refresh.sh npm`.
  `bench/oracle-install.sh` runs `npm ci --offline` with it as npm's cache; the share is
  `WEBENGINE_LANE_SHARED` on a lane or in a job and `$HOME/lane-shared` otherwise.
  `bench/lane-apply.sh` installs with `--offline` against it as well. npm's `integrity`
  hashes are checked on every install from the cache. The `web-features` data
  (`data.json`, which the web-features map and the feature loop read for each feature's
  keys and specification URLs) is the installed package's file,
  `oracle/node_modules/web-features/data.json`, pinned in `oracle/package-lock.json`;
  `crates/wpt-votes/src/features.rs` `WebFeatures::load` reads it, checks the installed
  version against the pin, and fails naming `bench/oracle-install.sh` when the package is
  missing. It does not fetch from a CDN. `browser-compat-data` is read the same way:
  `crates/content-analyser/src/bcd.rs` `Bcd::load` reads
  `oracle/node_modules/@mdn/browser-compat-data/data.json`.
- Crates: the share's `crates/`, a vendored directory source (one `<name>-<version>/` per
  crate) filled by `bench/nas-refresh.sh crates` from every crate `Cargo.lock` and
  `bench/Cargo.lock` name (`cargo vendor --locked --versioned-dirs`, with its own
  `CARGO_HOME` holding no source replacement). Crates are only added, never removed. Each
  machine's own `config.toml` in its Cargo home replaces crates.io with that directory:
  `bench/lane-dist.sh` with `DIST_ROLE=registries` writes the block. On a lane it reads
  `LANE_SHARED_DIR` (default `/srv/lane-shared`) and writes the lane user's
  `~/.cargo/config.toml`. The development box's user builds other projects whose crates are
  not on the share, so its block goes into a Cargo home used only for this repository,
  `$HOME/.local/share/webengine/cargo-home`, from `$HOME/lane-shared`;
  `bench/devbox-cargo-home.sh` installs it (it runs the registries role with
  `LANE_CARGO_HOME`). A cargo wrapper (`$HOME/.local/share/webengine/bin/cargo`, linked from
  `$HOME/.local/bin/cargo`, which comes before `~/.cargo/bin` on `PATH`) sets `CARGO_HOME` to
  it when cargo runs inside a checkout of this repository (a directory above holding both
  `Cargo.lock` and `jobs/src/manifest.rs`) and `CARGO_HOME` is unset; elsewhere it changes
  nothing. Dagu job steps (`bench/dagu/*.yaml`) put the wrapper's directory before
  `~/.cargo/bin`; `bench/devbox-cargo-home-test.sh` covers the wrapper and that ordering.
  `~/.cargo/config.toml` settings (sccache) still apply, because cargo reads the
  `.cargo/config.toml` of every directory above where it runs. Not caught: `cargo
  --manifest-path` or `-C` from outside a checkout. Cargo takes no `[source]` replacement
  from environment variables, so the setting lives in each machine's Cargo home, not in the
  repository. Each vendored crate carries a `.cargo-checksum.json`; cargo checks its
  package checksum against `Cargo.lock` (a mismatch stops the build) and each file against
  its recorded hash. `bench/lane-dist-registries-test.sh` covers the role. `experiments/` lockfiles are not
  vendored; building them needs a Cargo home without the replacement.

The development box cannot mount the NAS, so `$HOME/lane-shared` holds its own copy of
both, refreshed the same way (`WEBENGINE_LANE_SHARED=$HOME/lane-shared
bench/nas-refresh.sh crates npm`, or the scheduled refresh on the box's own lane entry).

Other outside downloads are not yet mirrored: Playwright's browsers, specification sections (`fetch_specs`, over HTTP),
wpt.fyi run listings and raw results (`crates/wpt-votes` `runs.rs`, `build_table.rs`;
`oracle/suites/compare-wptfyi.mjs`), and release binaries from
GitHub for Dagu (`bench/dagu-lane.sh`) and sccache (`bench/lane-dist.sh`), plus the
toolchain installers `bench/lane-provision.sh` runs.

The wpt.fyi readers stay direct and are not mirrored: only by-hand reports use them, and
nothing in the feature pipeline does (`jobs/scripts/feature-triage.sh` and
`derive_feature_file` read the WPT mirror, and the browser vote comes from the rig's own
browser runs), while a copy on the share would cost about 2.4 GB per WPT revision.

**The track gate runs only on the build lane.** `tracks.json` → `verify` runs
`bench/verify.sh`, which hands itself to `bench/on-lane.sh --lane-only` and, once there,
runs `cargo test --workspace`, the job event store's tests (`jobs/events/*.test.mjs` under
`node --test`, then `bench/job-events-lane-test.sh`), the page selector's tests
(`oracle/suites/feature-pages.test.mjs` under `node --test`, on committed fixture pages
with no browser), and then `bench/generated-scores.sh`. The other `*.test.mjs` and
`bench/*-test.sh` files are run by hand, not by the gate. With no lane
configured or none answering (`ssh -o BatchMode=yes -o ConnectTimeout=5` fails) the gate
exits 3 and runs nothing on the development box; a lane that answers but fails the tests or
a score is a failed gate — no local retry. Measured before the scores were added: a new
worktree's first test run 106 s (124 s with a fresh mirror), a warm rerun 22 s, local
fallback 78 s. A clean `cargo test --workspace` compiles in 61 s on a lane and 57 s on the
development box — the build is bound by the dependency chain through Stylo, not by cores;
the lane's value is that it takes load off the shared box and runs several gates at once.

**Unattended landings never fall back to the development box.** `bench/on-lane.sh`
without `--lane-only` runs its command locally when no lane answers, and `bench/land.sh`
runs each `landChecks` command that way, and `verify` directly. Setting
`WEBENGINE_LANE_STRICT=1` in `bench/land.sh`'s environment (the scheduled
`jobs/scripts/poll-and-land.sh` gets it from `tracks.json`'s `specWatch.pollEnv`) makes
`verify` go through `bench/on-lane.sh --lane-only` too and every check `--lane-only`, so
with no lane answering the landing is refused with "verify failed on the tip (exit 3: no
build lane answered ...)" and nothing of the gate runs here. Unset, as in interactive
use, a check keeps its local fallback. `bench/verify.sh` itself is lane-only either way;
the one thing that would run it on the development box is `WEBENGINE_ON_LANE` set there by hand.

**`bench/on-lane.sh` probes the lane up to 3 times before calling it unreachable.** The probe
(`bench/lane-probe.sh`, 10 s connect timeout, 2 s between tries) fails now and then through the
jump host on a healthy lane; `WEBENGINE_LANE_PROBE_TRIES` changes the count, and each retry
prints one line to stderr.

**Already on a lane, `bench/on-lane.sh` runs the command in place.** With `WEBENGINE_ON_LANE`
set, `bench/on-lane.sh` (and `bench/remote-lane.sh`) runs its command in the named
worktree on the machine it is on, `--lane-only` or not, reads no lane settings, and
ignores `--pull`, since the outputs are already where they belong; the command's exit
status is the run's. This is the same check `bench/verify.sh` makes, so a spec's
`bench/on-lane.sh --lane-only "$PWD" ...` works unchanged inside a Dagu job, whose clone
has no `.env` and no `bench/lane.local.json` and whose writer gets `WEBENGINE_ON_LANE=1` from
`jobs/scripts/pi-session.sh`. As with `bench/verify.sh`, a `WEBENGINE_ON_LANE` set by hand on the
development box would run the command there. `bench/on-lane-test.sh` covers both sides.
The cost: with both lanes down nothing lands until one answers. `bench/land-test.sh`
scenario `s1` covers both settings.

**The gate scores every generated function against its own corpus.** A committed file under
`crates/layout/src/generated/` can be wrong while every test passes, because the tests
exercise in-tree reference rules, not the generated function. `bench/generated-scores.sh`
clears and rebuilds each corpus with its `*_ground_truth_cache` binary (the browser oracle
runs on the lane), then `generated_score_check` scores each generated function through its
own runner and fails it against its floor. The floor is two numbers in
`crates/layout/src/generated/<name>.floor.json` beside the function's provenance record:
`cases_passed`, how many cases it passes, and `cases_run`, how many cases its corpus holds
(only cases three engines agreed on). It fails when the passing count drops, when the corpus
shrinks even though the passing count holds, and when a function passes no case at all (a
lane missing a browser leaves fewer than three engines to vote, so every case drops; a floor
of zero cannot hide a function that scores nothing). The provenance record describes the
model batch that admitted the function and is never rewritten; the floor is never below its
`verification.cases_passed` when the record names `verification.scoring` as `browser-measured
inputs` (how the runner scores now, `docs/generation-loop.md` -> "Scoring a function:
browser-measured inputs"); a record with no such field was scored another way and binds
nothing, so its floor is the floor file alone. A function with no floor file yet is held to the
record alone.
That is where a function a feature job admits starts: the job's `admit` stage
(`jobs/scripts/admit.py` `store_main`) writes the provenance record and no floor file, since
`bench/land.sh` refuses a branch that touches one, and the first floor comes from the raise
after the landing (below). A
placeholder record, such as the hand-written `table` one (0 recorded), therefore floors at
its real score once a floor file exists.

**Floors rise by themselves, after a landing.** `bench/after-land.sh` ends with
`bench/generated-floors-update.sh`, which runs `bench/generated-scores.sh --print-raises` on
the build lane, writes each floor that the fresh score is above (a function with no floor
file gets one), and commits only those files as one commit on `main`, the way the map and
dashboard commits are made. It refuses on uncommitted changes, writes nothing when the lane
does not answer or a score fails, never lowers a number, and a failure is one line and does
not change the landing. Raising happens after the landing rather than in the gate because
the gate runs on the lane's copy, where a commit cannot reach the track, and because a track
never edits a floor: `bench/land.sh` refuses a diff that touches any `*.floor.json`, so an
unattended track cannot lower one. The gate scores with `--print-raises` too, and a gate
that passes on a checkout with nothing uncommitted or untracked keeps its raise lines keyed
by the git tree it scored (`bench/generated-raises.sh`, in the repository's shared git
directory). A landing is a fast-forward to exactly that tree, so the raise after it takes
those lines instead of scoring again, as long as `main` has changed since only in
`README.md` and under `docs/` (the dashboard and map commits that run first). A gate on a
dirty checkout, or a landing whose tree no passing gate scored, still costs a second
scoring run on the lane. Either way there is a window between a landing that improves a
function and the raise in which the old floor holds. A floor comes down only by a person's
reviewed edit of its file.

**One place to add a generated function.** A function is discovered from its
`<name>.provenance.json`; its runner and the cache binary that writes its corpus are one
entry in `FUNCTIONS` in `crates/generation/src/bin/generated_score_check.rs`, and a
provenance record with no entry fails the gate with that instruction. `bench/generated-scores.sh`
takes the caches to rebuild from `generated_score_check --list-caches`, so nothing under
`bench/` changes and a track adding a function can land unattended.
`bench/generated-scores-test.sh` exercises the scripts, including the raise, without a
browser. `bench/score-generated-oracle-test.sh` runs `jobs/scripts/score-generated.sh`,
what a feature job's `admit` stage scores with, against the real browser oracle on the
box-generation train pages (generated, no network; about two minutes), on the development box
only. It and `bench/feature-job-box-test.sh` (`docs/local-lane.md` → "Before a folknet
send") run in the job-system gate, `bench/e2e-gate.sh`, on every landing that changes the
job system, `crates/generation/`, `crates/provenance/`, the harness's case modules or a
`specs/*.job.json`, and fail it when either fails (`docs/local-lane.md` → "Scenario jobs").

**Model batches run only on a lane.** `bench/absolute-mixed-batch.sh` and
`bench/overflow-mixed-run.sh` are started from the development box like any script
(`WEBENGINE_MODELS=qwen3.8-27b-fast:32:32 bench/absolute-mixed-batch.sh`) and re-run themselves
on the lane through `bench/on-lane.sh --lane-only` (`bench/batch-lane.sh`): the `WEBENGINE_*`
settings go along, credentials and lane settings excepted, and the batch's results come
back with `--pull`. With no lane configured or none answering the script exits 3 and runs
nothing locally; `bench/lane-apply.sh` exports `WEBENGINE_ON_LANE=1`, which is how a script
knows it is already there. Each run also records its phase timings and how many of the
lane's CPUs each phase kept busy (`bench/cpu-sample.sh`, `bench/cpu-phases.py`;
`docs/rig-capacity.md` → "A real batch on a build lane").

**A stray directory at a worktree's lane path breaks the gate.** `bench/lane-apply.sh`
tests only for `<lane-root>/<name>/.git`; a leftover directory without it makes `git
worktree add` fail, and the gate cannot pass until it is removed by hand (check `git -C
<mirror> worktree list` first). Not yet fixed in the script.

**Cleanup.** Every worktree run on a lane leaves its copy (`<root>/<name>`, `target/`
and all, 3–12 GB) and `<root>/.payload-<name>` there; left alone they filled a lane's
disk. `bench/lane-sweep.sh` removes, on every lane in `bench/lane.local.json` `lanes[]`,
the copies whose track has no worktree in the primary checkout (`--dry-run` lists them
with sizes). It keeps the primary's own copy, any copy with a process running in it (a
spec writer runs inside its copy), and directories that are not worktrees of the lane's
mirror. Untracked data under a copy's `corpus/` (links excepted: they point into the
NAS already) goes to the NAS first, as one tar file at
`archive/corpus/<name>/<time>.tar`; with no NAS directory the copy is kept. A branch a
copy was on stays in the mirror. It runs at four points: by hand; from
`bench/after-land.sh` after every landing, by hand or unattended, over every copy (a
failure is one line on stderr and changes nothing else); from `bench/land.sh` after a
landing, once the track's worktree is gone, for that track only (a failure is named on
its line and undoes nothing); and from `bench/remote-lane.sh` when the lane's root filesystem has less than
`WEBENGINE_LANE_MIN_FREE_GB` (20) free before a run, which then refuses with exit 4 if the
sweep did not free enough, naming the lane by its place in the list, the free space
and the largest copies. A hand landing leaves no worktree, so the next sweep takes its
copies. Scenarios: `bench/lane-sweep-test.sh`.

While `KEEP_JOB_OUTPUT` is yes (the default; the environment, else `~/.lane-shared-env`
on the lane), the sweep keeps the most recently used of the copies it would remove (the
later change time of the copy and its payload directory, worked out over the whole lane
even with `--track`) and removes the older ones as above, whatever the free space: only
the latest run's output is ever read again. Of the copy it keeps it touches nothing that
cannot be rebuilt: sources, `corpus/`, candidates, `jobs-work/`, runs and archives stay.
When the lane's root filesystem has less than `WEBENGINE_LANE_MIN_FREE_GB` free, it
deletes only that copy's build output (a Cargo `target/` beside a `Cargo.toml`,
`node_modules/`), on a `freed:` line with the size; otherwise a `kept:` line says why.
`bench/remote-lane.sh` still stops with exit 4 on a lane that is short after that, so a
person frees or adds space. The newest copy keeps its job output because the first
aspect-ratio run's candidate sources, reports, prompts and compile errors could not be
read afterwards and the failure was diagnosed from sizes in `batch.log` instead of from
code. `KEEP_JOB_OUTPUT=no` removes that copy too.

**Retention of job-run folders.** One rule covers every folder a job run leaves behind,
on the lanes and on the NAS share: of each group the newest two stay and the rest go,
with no age floor. `jobs/scripts/retention.sh` holds the rule and, in its header, the
list of folders and their groups:

| folder | group |
|---|---|
| `archive/job-logs/<job>/<Dagu run id>/` | job |
| `archive/job-work/<job>/<Dagu run id>/` | job |
| `archive/job-work/<job>/restarted-<time>.tar.gz` | job (apart from its runs) |
| `archive/job-corpus/<job>/<spec blob, else run id>/` | job |
| `archive/stage-cache/<job>/<stage>/<key>/` | job and stage |
| `archive/gate-logs/<worktree>/<time>-<guest>-<commit>.log` | worktree |
| `archive/corpus/<lane copy>/<time>.tar` | lane copy |
| `~/dagu/runs/.restarted-<name>-<time>/` | name |

Newest is the modification time, except in the stage cache (an entry's last use) and for
a moved-aside run directory (the time in its name). It runs at two points. The exit handler
of `bench/dagu/job.yaml` runs it at the end of every job for that job's groups only, and
never removes the folders of the run itself (its run id, its staging folders and its spec
blob). `bench/lane-sweep.sh` runs it over every group on every lane it sweeps (`--dry-run`
lists what would go). A leftover of an interrupted copy (`<run id>.partial-<pid>`,
`*.tar.partial`, `*.tar.part`) does not count among the two and goes once nothing under it
has changed for an hour, since a copy on another lane may still be writing it. Each
earlier location had its own count-plus-age rule, or none; with daily runs the age floor
kept everything, and on 2026-10-12 one job's `archive/job-work/` held 25 runs (34 GB).

A job directory is never moved aside: a restart (`clean-jobs`, or `prepare` on RESET=yes
or a new base or spec) archives it as a restart tar and removes it (see "Dagu on a lane"),
so no folder can be left without an `archived-*` marker. `--keep-output` is still
accepted and has no effect.

Left out of the rule, because each is bounded otherwise or is not per run: the live job
directory `~/dagu/jobs-work/<job>/` (the run still writing); `~/dagu/src-<id>/` and
`~/dagu/scripts-<id>/` (`prune_sources()`); lane copies (the sweep above); the per-worktree
`archive/suite-results/` and `archive/webgpu-cts-results/` (overwritten in place);
`ledger/<job>.jsonl` (one append-only file per job); Dagu's run history (7 days); and the
fixture workflow's `~/dagu/runs/<run id>/`. `jobs/tests/retention_coverage.rs` fails when
the workflow's code names a folder under the share's `archive/`, at the top of the share,
or in `jobs-work/` that is neither in the list nor among these exclusions;
`jobs/tests/retention_share.rs` and `jobs/tests/retention_moved_aside.rs` test the rule
itself.

**Test-run memory.** A cold `cargo test --workspace` on the development box adds about
1.1 GB at peak (10.8 → 12.0 GB), a warm one 67 MB; seven concurrent tracks fit in its 30
GB if their peaks do not coincide.

**Distributed compilation.** `sccache-dist` builders on six nodes (416 cores registered)
with redis as the shared cache. After dropping one builder on a node with slow outbound
network (failures 84 → 1), it still lost on wall time: a clean `-p dom` release build took
82 s distributed against 45 s local on the same lane. The dependency graph is too narrow
to use the cores. A wide build fares no better: a batch's candidate probe (one
`--keep-going` build of 32–64 small crates) took 14–45 s through sccache-dist against
1.4–1.6 s with no compiler wrapper on the lane, just as slow on a second pass, so each
compile through the wrapper carries a fixed delay far larger than the compile itself
(`docs/rig-capacity.md`, "A real batch on a build lane"). The delay is the input
upload: for a Rust compile the sccache client packages every library in the
`-L dependency=` directory (`target/release/deps`, 225 rlibs, 336 MB), not only the
crates the compile loads. One probe library compile (0.1 s with plain `rustc`) took
4.4 s distributed, sent 121.9 MB (the gzip size of that whole directory) and spent 2.3 s
of client CPU compressing it. The cost grows with the dependency directory, not with the
crate, and is paid again on every compile, which likely explains the `-p dom` result
too (not measured). The probe therefore runs without the wrapper, on the lane's own cores; the
development box's sccache config stays untouched.

**Shared data on the NAS.** Each lane bind-mounts one NAS directory at
`/srv/lane-shared` (`WEBENGINE_LANE_SHARED`, also the npm cache). Login shells get the npm cache
from `~/.lane-env`; the Dagu job's `prepare` and `run` steps (`bench/dagu/job.yaml`) read
only `WEBENGINE_LANE_SHARED` and `npm_config_cache` from `~/.lane-shared-env`, and `bench/oracle-install.sh` runs
`npm ci --prefer-offline` (`--offline` when it sees the share; below), so a package is downloaded from the registry once and every
lane and job after that installs it from the share. The NAS is fast from the lanes
(about 92 MB/s write, 114 MB/s read) but slow for small files (~38 ms each), so `target/`
stays local. `bench/lane-apply.sh` links each worktree into it on every run:

| in the worktree | on the share |
|---|---|
| `oracle/suites/checkouts/<suite>` | `suites/<suite>-<pinned commit>/` |
| `webgpu-cts/cts` | `suites/webgpu-cts/` |
| `corpus/specs` (specification sections; a worktree's own earlier cache is merged in first) | `specs/` |
| `corpus/<name>` for every `*ground-truth` cache the code names | `ground-truth/<name>/` |
| `oracle/suites/results` | `archive/suite-results/<worktree>/` |
| `webgpu-cts/results` | `archive/webgpu-cts-results/<worktree>/` |
| (the command's output, copied) | `archive/gate-logs/<worktree>/<time>-<guest>-<commit>.log` |
| (one line per model batch, `bench/ledger.py`) | `ledger/<worktree>.jsonl` |
| (a Dagu job's full logs, `bench/dagu/job.yaml`, scrubbed) | `archive/job-logs/<job name>/<Dagu run id>/` |
| (a Dagu job's gitignored `keep_dirs`, tarred, unscrubbed) | `archive/job-corpus/<job name>/<spec blob>/<dir>.tar` |
| (a Dagu job's work tree and candidates, unscrubbed) | `archive/job-work/<job name>/<Dagu run id>/` |

The last row comes from the Dagu job workflow, not `bench/lane-apply.sh`: it copies each job's stage logs,
`state.json`, the review, each pi session's prompt and `events.jsonl` (its turns and
tokens) and `oracle-install.log` there when a run ends,
whether it passed, failed or was stopped, with an `archive.txt` listing every file and its
line count; the step log prints the path. Credentials, URLs, addresses and token-shaped
values are replaced on the way (`bench/dagu/job.yaml`, "handler_on").

The two job rows after it are written by the same handler, for every final status, while
`KEEP_JOB_OUTPUT` is yes: `archive/job-work/<job name>/<Dagu run id>/` holds `repo.tar`
(the job's repo tree), `writer-<n>-tree.tar` for each fan-out writer's tree, and
`candidates/` (the repo's `corpus/candidates` as plain files: sources, replies, compile
errors, reports, prompts, readable without extracting anything). `target/`,
`oracle/node_modules` and `.git/config` (it holds the forge URL with its credential) are
left out. These copies are not scrubbed, like the `keep_dirs` tars. Of each job's
folders under `archive/job-logs/`, `archive/job-corpus/` and `archive/job-work/`, and of
each worktree's `archive/gate-logs/`, the newest two stay (see "Retention of job-run
folders" above). `bench/job-archive-test.sh` checks that a finished run and a failed run
each leave their candidate sources under `archive/job-work/`.

**A share that is not the NAS.** The development box cannot mount the NAS (it reaches the
rig only over ssh through the jump host), so its `WEBENGINE_LANE_SHARED` is a local
directory, `$HOME/lane-shared` (`docs/local-lane.md`). Every archive still ends up on the
NAS: its `~/.lane-shared-env` also sets `WEBENGINE_ARCHIVE_NAS=<lane ssh destination>:/srv/lane-shared/archive`
and `WEBENGINE_ARCHIVE_NAS_SSH_OPTS` (lane 0 of `bench/lane.local.json`, whose
`/srv/lane-shared` is the NAS), and `jobs/scripts/archive-to-nas.sh` moves what the
writers above put in the local `archive/` there over rsync. A lane whose share is the NAS
leaves `WEBENGINE_ARCHIVE_NAS` unset, and the script does nothing there. Each file is
copied with `--ignore-existing`, compared with its NAS copy by content (`rsync
--checksum`), and deleted locally only when the two match, so a failed or partial copy
leaves the local file. A file the NAS already holds under that name with other content
is never replaced: the copy goes beside it with the UTC time before its extension, and a
`kept both:` line names the two. It moves the write-once groups (`job-logs/`, `job-work/`,
`job-corpus/`, `gate-logs/` and `corpus/`). It leaves local whatever is still being written
(`*.partial*` and `*.part` staging copies, and gate logs changed within the hour, which
`bench/lane-apply.sh` is still appending to), an e2e scenario's `e2e-*` folders (removed
locally by `bench/e2e-lane-clean.sh`), and the folders this machine reads back or links:
`stage-cache/`, `suite-results/` and `webgpu-cts-results/`.

It runs at three points, with nothing to start by hand:

- the exit handler of `bench/dagu/job.yaml`, for the run's own archive (`--only` its
  `job-logs/`, `job-corpus/` and `job-work/` folders), before it writes
  `archived-<Dagu run id>`: the marker is written, naming the NAS copy, only when every
  file is verified there. A failed copy keeps it all local, writes no marker, prints a line
  starting `JOB ARCHIVE NOT ON THE NAS`, and does not fail the run. After its retention
  pass the handler moves whatever else is waiting, such as an earlier failed copy, gate
  logs and corpus tars;
- a restart, for the restart tar, before the old work is removed: `bench/dagu-lane.sh
  clean-jobs`, and the `prepare` step of `bench/dagu/job.yaml` (see "Dagu on a lane");
- `bench/lane-sweep.sh`, after its retention pass on every lane (so after every landing,
  through `bench/after-land.sh`).

One move runs at a time (`flock` on `~/.archive-to-nas.lock`). Tests:
`bench/archive-to-nas-test.sh` covers the script, and `bench/job-archive-test.sh` and
`bench/lane-sweep-test.sh` cover its callers, each with a stand-in `ssh` and a local
directory standing in for the NAS.

Specification sections come only from the share on a lane or in a job. With
`WEBENGINE_LANE_SHARED` set and `WEBENGINE_NAS_REFRESH` unset, `fetch_specs`
(`crates/provenance/src/fetch.rs` `fetch_section()`) fetches nothing: a section missing
from `specs/` fails naming `bench/nas-refresh.sh specs`, the refresh that fills it.
`jobs/scripts/feature-specification.sh` passes the share's `specs/` as `--cache` to both
`fetch_specs` and `specification_text`. The development box sets neither variable and
keeps fetching into its own `corpus/specs`. `bench/nas-refresh.sh specs` (with
`WEBENGINE_LANE_SHARED` naming the share) sets `WEBENGINE_NAS_REFRESH=1` and runs
`fetch_specs` once for each section the checkout cites: every `pub const SPEC_URL` under
`crates/` and every `oracle/suites/features/*.json` `specification` value. A section
already on the share is kept as it is, so a run adds only what is newly cited. It prints
`nas-refresh specs ok|failed <detail>`, exits 1 when a section could not be fetched (the
others are still fetched), and holds a local `flock` on `nas-refresh.lock` in the
checkout's git directory; `bench/nas-refresh-test.sh` covers it with a stand-in
`fetch_specs`.

Conformance suite checkouts follow the same rule. Under the same two variables,
`oracle/suites/fetch.mjs` `fetchSuite()` fetches nothing: under its lock it checks that
`suites/<suite>-<pinned commit>/` is at the pinned commit and holds every pinned sparse
path, and otherwise fails naming `bench/nas-refresh.sh suites`. That source runs
`fetch.mjs` for each suite in `oracle/suites/suites.json` against the share's directory
(through a temporary link, as `bench/lane-apply.sh` links it), prints
`nas-refresh suites ok|failed <detail>`, and fails one suite without stopping the others.
A new checkout is built in a `<dir>.staging-<token>` sibling and renamed onto the pinned
directory, so a reader finds it empty or finished; one already at the pin is left alone.
On the NFS share, `git sparse-checkout set` adding several paths fills one, leaves the next
as an empty directory and fails with "Is a directory"; `fetchSuite()` then removes the empty
directories at the new paths and sets again, at most once per new path.
`oracle/suites/fetch.test.mjs` and `bench/nas-refresh-test.sh` cover both sides against a
local upstream repository.

npm packages too. Under the same two variables `bench/oracle-install.sh` runs `npm ci
--offline` and `bench/lane-apply.sh` runs `npm install --offline`, both against the
share's `npm-cache/`, so a package missing there fails the install naming
`bench/nas-refresh.sh npm`. That source copies `package.json` and `package-lock.json` of
`oracle/` and `webgpu-cts/` to a temporary directory and runs `npm ci --prefer-offline
--ignore-scripts` there with the share's `npm-cache/` as npm's cache, so a package already
cached is not downloaded again and the checkout is untouched; it prints `nas-refresh npm
ok|failed <detail>`. The Dagu job's `prepare` step reads `WEBENGINE_LANE_SHARED` from
`~/.lane-shared-env` and passes it to `bench/oracle-install.sh`, with `npm_config_cache`
set to the share's `npm-cache/` and `npm_config_offline=true` for a base whose
`bench/oracle-install.sh` predates the refusal, so a job's install is offline too; a
failure is reported in the step log and left to the writer, as before.
`jobs/tests/job_prepare_install.rs` runs that part of the step against a stand-in npm.

The WebGPU conformance suite is pinned in `webgpu-cts/cts.json`, and nothing on a lane
fetches it: `webgpu-cts/cts` is linked to the share's `suites/webgpu-cts`, which
`bench/nas-refresh.sh webgpu-cts` keeps as a link to `suites/webgpu-cts-<pinned commit>/`.
That source makes a shallow checkout of the pin in a staging sibling, runs `npm ci` in it
against the share's `npm-cache/` (the suite's `tools/run_node` and dev server need its own
packages), renames it into place, and then replaces the link by a rename, so a lane sees
the old suite or the new one. A checkout already at the pin is left alone, and an earlier
pin's checkout is kept. An empty directory where the link goes (what `bench/lane-apply.sh`
creates when the share lacks it) is replaced; any other directory there fails the source,
naming it, since a lane may be running from it. It prints `nas-refresh webgpu-cts
ok|failed <detail>`; `bench/nas-refresh-test.sh` covers it against a local upstream.

The refresh runs on a schedule from one lane, never from a track's copy (with the one
exception below):
`bench/nas-refresh-install.sh [--lane <index>] [--schedule '<cron fields>']` copies
`bench/nas-refresh-cron.sh` to `~/nas-refresh/run.sh` on the lane and adds a crontab line
(marker `# webengine-nas-refresh`) that runs it under `flock -n`, appending to
`~/nas-refresh/refresh.log`. Each run fetches the lane root's `.git-mirror` (skipped when it
fetched in the last 10 minutes, as in `bench/lane-apply.sh`), moves the detached checkout
`~/nas-refresh/checkout` (a worktree of that mirror outside the lane root, so
`bench/lane-sweep.sh` leaves it alone) to the mirror's `main`, and runs every source from
it with the share from `~/.lane-env`. The default lane is the `lanes[]` entry whose ssh host
is localhost (the development box's own, `docs/local-lane.md`) and the default schedule
hourly at minute 17. That entry is for validating the refresh against the box's stand-in
share; the production schedule runs on a folknet lane once it is validated there, which is
the same script with `--lane <folknet index>` (and `--schedule` if hourly does not suit),
followed by `remove` on the box's lane. `run` refreshes once now, `status` and `remove` do
what they say.
`bench/nas-refresh-install-test.sh` covers it with stand-in `ssh` and `crontab` and a local
upstream.

The one exception is a branch that changes `wpt.sparse` (or another suite's `sparse`) in
`oracle/suites/suites.json`: the lanes' check fails until the share holds the new paths, and
the scheduled refresh reads `main`. Such a branch runs the suites source once from its own
worktree before its gate, on a build lane with
`bench/on-lane.sh --lane-only <worktree> bash bench/nas-refresh.sh suites` and on the
development box with `WEBENGINE_LANE_SHARED=$HOME/lane-shared bench/nas-refresh.sh suites`.
Adding paths is safe for other readers, which only check that their own paths are present.
The hourly refresh from `main` may narrow the list back until the branch lands, so the
branch's refresh runs just before the gate.

The same holds for a branch that adds or upgrades a crate (`Cargo.lock`,
`bench/Cargo.lock`) or an npm package (`oracle/package-lock.json`,
`webgpu-cts/package-lock.json`): builds read only the share, so the branch's gate fails
until the share holds them. It runs the matching sources once from its own worktree
before its gate: `bench/on-lane.sh --lane-only <worktree> bash bench/nas-refresh.sh crates
npm` for the lanes' share, and `WEBENGINE_LANE_SHARED=$HOME/lane-shared
bench/nas-refresh.sh crates npm` for the development box's copy. Both sources only add,
so the hourly refresh from `main` never takes the branch's packages away again, and
other readers are unaffected.

A lane never starts without the share. The node mounts it through `/etc/fstab` with
`nofail`, so a NAS that is down at boot leaves the node and its other tenants' guests
starting; a systemd drop-in on the lane's own container unit
(`pve-container@<vmid>.service.d/lane-nas.conf`, `RequiresMountsFor`) makes each start of
the lane retry the mount and fail when it fails. Without it, a boot with the NAS down
left a lane running on the node's empty local directory, and every lane run failed at
`mkdir /srv/lane-shared/suites: Permission denied` (2026-09-29, after a maintenance
window). After such a boot, `pct start <vmid>` on the node once the NAS answers brings
the lane up. `bench/lane-nas.sh` writes the drop-in.

**After an outage or maintenance window,** run `bench/folknet-check.sh` from the
development box, then `bench/folknet-check.sh --fix`. It checks every node, lane, build
guest and tunnel the project depends on against a private inventory
(`bench/folknet.local.json`, shape in `bench/folknet.example.json`): root login by key on
each node, each guest on its declared node, running, at its declared address, each lane's
NAS mount and container unit guard, and each development-box tunnel. `--fix` starts
stopped guests, reruns `bench/lane-nas.sh` for a lane missing the NAS, and restarts
tunnels; a moved guest, a changed address or a node that lost its key is reported for a
person, with every Proxmox node on the subnet listed by the name in its web certificate.
It also checks what a job needs above the machines: each folknet lane's Dagu answering
through its port forward, each lane's inventory probe entry and a snapshot under 15
minutes old, every model a `specs/*.job.json` names being ready in the snapshot of the lane
jobs go to (`bench/dagu-pick.sh`), the job event store, the
announcement bot, both forges, and the scheduled poller having run in the last 15 minutes.
A lane in `dagu_lanes` must have Dagu installed and its autostart on, and `--fix` installs
the probe and turns on the autostart there when they are missing. Jobs do not wait for a
dead lane: after 30 minutes without an answer a run moves to the next listed lane that is
up, as described under "Dagu on a lane".

**Alerting.** `bench/folknet-watch.sh install` puts a crontab entry on the development box
that runs `bench/folknet-check.sh --fix` every 5 minutes and posts to #bez through the
announcement bot, and nowhere else, when a check starts failing, when it recovers, when
`--fix` repaired something, and every 6 hours while anything still fails. Addresses are
masked in every post. `bench/folknet-watch.sh status` shows what is failing now.

The watch itself stopping, as it does when the development box is down, is noticed from
lane 0. Each pass sends a heartbeat to the job event store (a step event of the feature
`folknet-watch`, which also shows as a row on the job dashboard; the store keeps every one,
about 100 bytes every 5 minutes). `bench/folknet-watch-heartbeat-check.sh` runs on lane 0
from the lane user's crontab every 5 minutes, reads the feature's newest time from the
store's `GET /features`, and posts to #bez through the announcement bot when it is over 15
minutes old ("has not reported for N minutes"), then once more when heartbeats resume. It
posts once per change and nothing while the state holds; a store that is down or holds no
heartbeat yet posts nothing, since the store's own check reports the first. Its state
(`ok` or `stale`) and log sit in `~/folknet/watch-heartbeat/` on the lane. The bot's URL and
token live there too, as `announce-url` and `announce-token`, mode 600; only the check
reads them, handing the token to curl on stdin. `bench/folknet-watch-heartbeat-install.sh
--lane 0` copies the check and adds the crontab entry (`install`, `remove`, `status`), and
`bot` sends `WEBENGINE_ANNOUNCE_URL` and `WEBENGINE_ANNOUNCE_TOKEN` from the environment or
`.env` over ssh on stdin. The development box needs `WEBENGINE_JOB_EVENTS_URL` and
`WEBENGINE_JOB_EVENTS_TOKEN` in `.env`, as for the job scripts.
`bench/folknet-watch-heartbeat-test.sh` tests all three against the real store and a
stand-in bot.

The ledger is what a landing's announcement to the community Discord reads for its
numbers (`docs/web-features-map.md` → "Announcements"): each batch's start and end time,
lane CPU-seconds, and per model the candidates, compiled count, best score, tokens and
draw seconds, with no host names or addresses. A batch already in the file (same track,
start and end) is not appended again, and a landing's announcement leaves a marker file
named for the landing's tip commit and job in `~/.local/state/webengine/announced`
(`WEBENGINE_STATE_DIR` moves it), so a rerun of `bench/after-land.sh` or the batch script
neither posts nor records twice; a failed post leaves no marker.

An existing real directory is left alone. A ground-truth cache written by one branch is
read by every other, so a recipe change should regenerate its cache. The shared copy of a
ground-truth cache mirrors the checkout's exactly: a cache binary writes case files and
never removes one, so when `bench/lane-apply.sh` finds a cache rebuilt in a real
directory after a successful command it runs `bench/lane-mirror-cache.sh`
(`rsync --delete`), which also removes the case files the new cache lacks. Without that,
a reference that passed against the job's own cases failed the landing gate against the
extra case files earlier runs had left in the shared copy. Only a directory named
`<name>-ground-truth` is mirrored onto the shared directory of the same name; any other
directory is refused and left alone (`bench/lane-mirror-cache-test.sh`). The export is open
to the whole subnet: nothing secret goes on it. Measured: fetching the suites into an
empty share took 6 min 19 s; a second lane then found them in 13 s.

**Reaching the model gateway.** A lane reaches the gateway at its rig-network address,
not the development box's loopback forward. Each lane keeps that address in
`~lane/.lane-model-env` as `WEBENGINE_LANE_MODEL_URL` (written by `bench/lane-provision.sh`
from `LANE_MODEL_URL`), and `bench/lane-apply.sh` swaps it into the worktree `.env`'s
`WEBENGINE_MODEL_URL` line on every run, leaving the key as copied. The address never enters
the repository (`bench/lane-apply-test.sh`).

**No credentials left behind.** The worktree's `.env` (the gateway key, usually the forge
token) is copied into the lane copy for the run only. When the run ends -- success,
failure or interrupt -- `bench/lane-apply.sh` removes the lane copy's `.env`, its
temporary rewrite and the payload's `.env`, and `bench/remote-lane.sh` removes the same
three over its own ssh, which covers a run whose connection dropped first. Nothing else in
the lane copy is touched. `WEBENGINE_LANE_KEEP_ENV=1` keeps them, and
`bench/lane-writer-launch.sh` sets it, because the writer it starts outlives the launch
and its model batches read that `.env`. Two known gaps: an interrupted run exits 130
(SIGINT) or 143 (SIGTERM) locally but does not stop the command on the lane, which runs
until it finishes or next writes to the dropped connection; and the git mirror's clone
URL can carry the forge token, which then shows in the lane's process list as an argument
to `lane-apply.sh`, a copy of that secret outside `.env`.

**Running writers on a lane.** No script here names an agent harness; the harness is a
declaration. `tracks.json` → `specWatch.lanes.tool` names it (`TRACKS_TOOL` in the
environment is the fallback, and a launch with neither is refused), and pi-tracks'
writer command prepares that harness on the lane and starts the session.
`bench/lane-provision.sh`'s writer-host section (enabled by `LANE_PI_TRACKS_URL`) clones
pi-tracks on the guest and prints `changed:`/`unchanged:` per step, so a rerun is
idempotent. What a particular harness needs installed ahead of time is a command the
operator supplies as `LANE_WRITER_SETUP`, run as `lane` with `LANE_WRITER_TOOL` and
`LANE_WRITER_DIRS` (the lane roots and mirrors, relative to the home directory) in its
environment; with none set the step does nothing. A harness login is a person's step
over ssh as `lane`. `bench/lane-writer-launch.sh <worktree> <lane-index> <model>
<brief-file>` syncs a worktree, checks out its branch, pulls pi-tracks and runs
`"$PI_TRACKS_HOME/bin/writer-exec.sh" --tool <name> --model <model> --brief-file <path>`
there (`PI_TRACKS_HOME` defaults to `$HOME/.local/share/pi-tracks`), refusing over a
dead writer's unfetched commits and failing when that command exits nonzero; the brief
lands in `~/.cache/lane-writer-briefs/` on the lane, outside the worktree.
`bench/lane-writer-fetch.sh` brings commits back fast-forward only. `tracks.json` →
`specWatch.lanes` declares both. `bench/lane-nas.sh` decides a guest is busy from the
process names in `LANE_BUSY_COMMANDS` (default `cargo|rustc|node`); add the harness's
process name when a writer runs under a different one. `crates/harness/tests/no_harness_name.rs`
fails the workspace tests when a `bench/` script, `tracks.json` (other than the `tool`
entry) or `.gitignore` names a harness. `bench/lane-writer-launch-test.sh` tests the
launcher against a stand-in writer command; a real launch on a lane is unverified until
the pi-tracks writer command is on its `main`.

**Dagu on a lane.** A job (`tasks/job-submit-land.md`) runs as one run of the Dagu workflow
`bench/dagu/job.yaml`, on one lane (an index into `bench/lane.local.json`; Dagu's
coordinator is off, so a run stays on the lane it started on). Which lane: `WEBENGINE_DAGU_LANE`
when set, with no probe; otherwise, of the lanes in the lane file's `dagu_lanes` (an ordered
list of the lanes that may run jobs, see `bench/lane.example.json`) whose
`bench/dagu-lane.sh --lane N check` passes, the one with the fewest open jobs in the job
records, the first listed on a tie, printed by `bench/dagu-pick.sh` (tests:
`bench/dagu-pick-test.sh`). Absent, the list
is `[0]`; a list of one is that lane with no probe. The development-box lane is never listed
by default, because a job run there only tests the pipeline. When no listed lane answers,
`bench/dagu-pick.sh` names each lane and why and exits 1, and `jobs/scripts/submit.sh` exits 3
with nothing sent.

A job whose lane stops answering moves on by itself. `jobs/scripts/poll-and-land.sh` keeps
the time of the first pass at which the lane did not answer about a run (`lost_since` in the
job's record, `jobs/scripts/records.py`). After `JOBS_LANE_LOST_SECONDS` (default 1800) of
silence the record is marked `lost`, and the run is resent, uncounted like the other
infrastructure resends, to the picked lane. It resumes there in a fresh clone: `prepare`
clones the sources and `adopt_stage_branch` fetches `job-stages/<name>` from the forge. A
lost record is skipped by `submit.sh`'s open-runs check, and `submit.sh --restart` does not
ask a lost lane to clean up. If no lane is up the mark stays and the next pass tries again.
A lane that comes back keeps its stale `jobs-work/<name>` directory.
`jobs/scripts/retention.sh` never removes a job directory, so `bench/lane-sweep.sh` does: for
each job record on the development box (`records.py lanes`) it runs
`bench/dagu-lane.sh --lane N clean-jobs <name>` on every lane N other than the one the
record's current run is on, and never on that one.

`bench/dagu-lane.sh` installs, starts, reaches and cleans up after it; its header lists the
actions and settings. Without `--lane` it takes `WEBENGINE_DAGU_LANE`, else the pick when
`dagu_lanes` lists several lanes, else lane 0,
unless `WEBENGINE_DAGU_LANE` is unset and `WEBENGINE_LANE_SSH` names a lane, which then wins.
`jobs/scripts/submit.sh` and `jobs/scripts/poll-and-land.sh` check
manifests and results with the `job-check` binary from `$JOBS_STATE_DIR/bin` (default
`~/.local/state/webengine-jobs/bin`) on the development box; `jobs/scripts/install-job-check.sh`
builds it on a lane (`bench/on-lane.sh --lane-only --pull`) and installs it there;
`bench/after-land.sh` runs it after every landing, so the binary follows `main`.

- *The requirements check.* Before the `run` step's `job-run` starts the first stage that is
  not `passed`, on every start and every resume, it checks that the lane has what the
  manifest is about to need -- `git`, `python3`, `node` and `kill` resolvable, `cargo` and
  the oracle's Playwright browsers when a writer is in the stage list, a reachable model
  gateway when a stage runs a pi session, 20 GiB free on the workdir's filesystem, the
  shared archive directory, `WEBENGINE_ON_LANE` when a stage calls `bench/on-lane.sh`,
  `bench/verify.sh` or `run-batch.sh`, and at least 600 s left of the wall-clock limit
  across resumes (`jobs/src/preflight.rs`). One unmet requirement prints every reason to
  stderr, records them in state.json's `preflight` field, runs no stage and exits 9 --
  distinct from 7 (publish failed) and 8 (stopped), and not resent by
  `jobs/scripts/poll-and-land.sh`, since a missing requirement is still missing a minute
  later. The exception is a dependency that is only down: when the model gateway does not
  answer (connection refused, a timeout, a 5xx) or the NAS share is not mounted or not
  writable, and nothing else is unmet, the check waits (10 tries, 15 s doubling to 75 s,
  about ten minutes; `JOBS_PREFLIGHT_WAIT_TRIES` and `JOBS_PREFLIGHT_WAIT_SECONDS`) and then
  exits 12, which the poller resends uncounted like 7 and 3 (`docs/autonomous-loop.md` ->
  "What a job does when a dependency drops out"). A gateway that answers 401, 403 or 404,
  a share without its directory and every other unmet requirement stay 9. Run the same
  check by hand with `job-check preflight
  <manifest> [--workdir <dir>] [--state <dir>]`, exit 0 when every requirement is met, 9
  otherwise (it waits the same way); `--workdir` defaults to the current directory and `{tools}` in a stage's
  command resolves under its `jobs/scripts`. This check only asks what the lane itself can
  answer right now (a binary, a socket, a directory); whether a model or provider is
  currently available on the rig is the separate inventory check that follows it.

- *The inventory check.* Once the requirements check passes, the `run` step's `job-run
  --inventory ~/folknet/inventory.json` copies the lane's snapshot into the state
  directory (again on every resume), hands every stage `JOB_INVENTORY` (the copy) and
  `JOB_INVENTORY_LIVE` (the lane's own path), and checks the manifest against it and the
  declared file `tools/folknet.json` (the submitting checkout's `bench/folknet.json`). A
  model that is not ready, or a snapshot older than 15 minutes or missing, prints every
  reason, records them in state.json's `inventory` field, commits a `not-run` result
  locally (not published), runs no stage and exits 3, which
  `jobs/scripts/poll-and-land.sh` resends uncounted. A refusal that never clears by itself
  -- a model retired or not declared, a provider not declared or not allowed -- is
  reported the same way but commits a `failed` result and exits 10, which the poller
  records as `failed` without resending. The same check by hand: `job-check
  inventory <manifest> <snapshot> [--declared <file>]` (`docs/folknet-inventory.md` ->
  "The check before the first stage").

- *Installed as a plain binary.* The lanes have no container runtime (unprivileged LXC,
  no `docker` or `podman`, no passwordless `sudo`; rootless podman would also need
  `/dev/fuse` and `/dev/net/tun` added to the guest's config on its node), so Dagu runs as
  the lane user from `~/dagu`: `bench/dagu-lane.sh --lane N up` installs the pinned
  release (its sha256 checked against the pin and the release's own `checksums.txt`),
  makes the lane's repository mirror (`~/lanes/.git-mirror`, cloned from the lane's
  `git_url` by `bench/lane-mirror-init.sh`, the script `bench/lane-apply.sh` runs too; the
  `prepare` step refuses without it; `dagu-lane.sh mirror` makes it alone and
  `bench/folknet-check.sh --fix` makes it on any job lane that lacks it), pushes the
  gateway key, starts it and checks it. `stop`, then deleting `~/dagu`, removes
  it. `autostart on` adds a lane-user `@reboot` crontab line, which has brought Dagu back
  after a guest restart.
- *Reached only through ssh.* Dagu binds to the lane's loopback (`WEBENGINE_DAGU_PORT`, default
  18431) with basic auth. The password is generated on the lane, kept in
  `~/dagu/home/config.yaml` (mode 600) and read over ssh when needed, never stored on the
  development box. `tunnel` holds a port forward open for the web UI, `/api/v1` and
  `/mcp`. The MCP server is the `/mcp` route of `dagu start-all` (Dagu 2.x has no
  `dagu mcp` command). The one MCP client is the curl-based one in `bench/dagu-lane.sh`
  (`run`, `stop-run`, `watch`).
- *Secrets never go through run parameters.* Dagu keeps a run's parameters in its run
  records on disk, so `run` refuses a parameter whose name contains KEY, TOKEN, SECRET,
  PASSWORD, PASSWD or CREDENTIAL. The gateway key goes by `key` over ssh on stdin to
  `~/dagu/home/model-key` (mode 600); `start` exports it as `WEBENGINE_MODEL_API_KEY` with
  `DAGU_ENV_PASSTHROUGH=WEBENGINE_MODEL_API_KEY`, so every step sees that one variable and it
  lands in no run record or log. Steps start from a non-login shell and inherit nothing
  from `~/.lane-env`: the gateway address comes from `~/.lane-model-env`, the npm cache
  and the share from `~/.lane-shared-env` (one variable at a time), and the forge through
  the lane mirror's origin URL, which carries the credential; git messages from forge
  commands pass through a filter that replaces anything shaped like a URL.
- *How Dagu 2.17.2 behaves.* Workflow keys are snake_case (`retry_policy`,
  `hist_retention_days`); camelCase is refused. One `dagu_execute` call waits at most
  300 s, so `run` follows a longer run by reading its status every 15 s until it ends or
  `WEBENGINE_DAGU_WAIT` seconds pass. Dagu records no exit code for a step: `watch --once
  --step <step>` takes 0 for a succeeded step and otherwise the N of the step's error text
  `exit status N`, and `-` when there is none (a step killed by a signal). The exit
  handler (`handler_on: exit`) runs after every run, succeeded, failed or aborted, but not
  after a run cut off by a lane restart or a KILL of Dagu.
- *The queue.* A workflow's own `max_active_runs` above 1 is ignored for local queues; the
  limit is a global queue in `~/dagu/home/config.yaml` (`queues.config[].max_concurrency`,
  written by `set_queue()`: queue `jobs`, `JOB_QUEUE_LIMIT` in `bench/dagu-lane.sh` at once,
  1 today, for the reason in `docs/job-concurrency.md`), read only when the scheduler
  starts, so a change needs `stop`, `start` and `autostart on` again (the last rewrites the
  limit `~/dagu/autostart.sh` passes after a lane restart). A lane's `~/dagu/home/queue-limit`
  holds the limit it was last started with; a lane without that file was started before
  `set_queue()` wrote it, and its `config.yaml` may still hold an older limit. Only enqueued runs pass through it (`dagu_execute`
  `action=start` runs at once), so `run` enqueues any workflow file with a top-level
  `queue:` key. Every submission sends its own copy of the sources: `run` hashes the files it
  sends into a 16-digit id, copies them to `~/dagu/src-<id>` and `~/dagu/scripts-<id>`
  (`sync_source()`, `sync_scripts()`) and passes the id as the run parameter `SRC_ID`, so a
  later submission never replaces the copy under an earlier run's build. The same files give
  the same id and reuse the copies. `prune_sources()` removes a copy that no job directory
  names (`jobs-work/<name>/src_id`) and that was last used more than a day ago
  (`WEBENGINE_DAGU_SRC_GRACE_MIN`).
- *What it keeps.* Run history 7 days (`hist_retention_days`, set in
  `~/dagu/home/base.yaml` by `set_retention()` and in `job.yaml`): the forge holds each
  job's result. A job's directory `~/dagu/jobs-work/<name>/` stays until the job passes,
  so a new run with the same parameters resumes it, and after a pass it stays too when
  its logs did not reach the NAS share (step `cleanup` removes it only once the archive
  wrote its marker `archived-<Dagu run id>` there, and says in its log when it kept it);
  while `KEEP_JOB_OUTPUT` is yes (the default, see "Cleanup") it keeps the directory after
  a pass too. Nothing of a job is moved aside. A restart archives the old work to the NAS
  share as `archive/job-work/<name>/restarted-<time>.tar.gz` (without `target/`,
  `node_modules/`, any `.git/config` or pi's `auth.json`; on to the NAS through
  `jobs/scripts/archive-to-nas.sh` on a machine whose share is a local directory) and only
  then removes it, at two points: `clean-jobs <name>` (which `jobs/scripts/submit.sh
  --restart` runs) for the whole job directory, refusing the restart with exit 3 and a line
  starting `RESTART REFUSED` when the copy fails; and `prepare` for the repo, state and
  manifest it clears on RESET=yes or a new base or spec, failing the step with a line
  starting `PREPARE FAILED` when the copy fails. Either way a failed copy keeps the
  directory in place. With `KEEP_JOB_OUTPUT=no`, `prepare` removes without archiving.
  `clean-jobs` still moves `runs/<name>/` to `runs/.restarted-<name>-<time>/`. Of the
  restart tars and of those, the newest two per name stay (`jobs/scripts/retention.sh`,
  see "Retention of job-run folders"). The job tools are built from the copy of the sources `run` sends to `~/dagu/src-<id>`
  (`sync_source()`), not from the job's base commit.
- *The mirror keeps deleted job refs.* The lane mirror's fetch (refspec `+refs/*:refs/*`,
  no prune) copies every `job/*` and `result/*` branch from the forge and keeps it after
  the forge deletes it. `prune-mirror` fetches and prunes those two namespaces only;
  `bench/dagu/fixture-base.sh --delete` runs it.

## Landing approved spec tracks from a tick

`bench/land.sh <spec-name> <spec-sha> <branch-tip>` lands an approved track without a
person; the spec-watch tick calls it (`tracks.json` → `specWatch.landScript`). Exit codes:
0 landed, 1 not landed (reason on one line), 2 rebase conflict outside `handoff.md`. The
contract: `~/.claude/docs/working-modes.md` → "Let a tick land tracks".

```mermaid
flowchart TD
  start["tip and spec sha match<br/>what was reviewed?"] -- no --> refuse["refuse"]
  start -- yes --> scope["diff inside the spec's touches,<br/>no protected file?"]
  scope -- no --> refuse
  scope -- yes --> rebase["rebase onto main<br/>(handoff.md → main's roster)"]
  rebase --> gate["verify + landChecks<br/>(tests, 227 floor, per-function floors)"]
  gate -- fail --> refuse
  gate -- pass --> merge["fast-forward main<br/>(retry if main moved, max 3)"]
  merge --> after["tag spec/name/sha, remove worktree<br/>and branch, update roster"]
```

- **Protected files.** A diff touching `bench/`, `tracks.json`, `.github/`, `specs/`, `tasks/`, or at
  any depth `.cargo/`, `rust-toolchain*`, `.env*`, `.gitattributes`, `.gitmodules` or
  `*.floor.json` (a generated function's score floor) is refused whatever `touches` says,
  since those change the gate itself. So is a spec whose
  `touches` matches everything or reaches those paths. Case-insensitive, no rename
  detection. Such a track waits for a person's `/land`.
- **What the gate still trusts.** The track's own crates, tests, `build.rs`, `oracle/` and
  `Cargo.toml`/`Cargo.lock`. The gate catches regressions and honest mistakes, not a track
  built to fool it.
- **The checks.** `verify` (`bench/verify.sh`: the tests, then every generated function
  scored against its own corpus, each at or above its floor in `<name>.floor.json`), then
  `landChecks`: `bench/fragment-floor.sh` (`fragment_check --random` must pass 227), which
  runs through `bench/on-lane.sh` (`--lane-only`, like `verify`, when `WEBENGINE_LANE_STRICT=1`;
  see "Build lane"). The four hand-written floors that `bench/seam-floor.sh`
  used to hold (floats, stacking order, relative positioning, block height) are floor files
  like the rest, and that script and its checker are gone. Floors are raised after a
  landing, never lowered by a track: a diff that touches a `*.floor.json` is refused.
- **Safety.** Pinned SHAs re-checked right before a `--ff-only` merge; a lock on the git
  directory; signals kill the gate's process tree and restore the branch. A
  `land-pending/<name>` marker makes a rerun finish anything owed after a merge. Nothing
  is pushed.
- **`handoff.md`.** A landing restores `main`'s roster on the branch before the gate, so a
  track's working note never overwrites the roster.
- **Install after a landing.** If `oracle/package*.json` changed or a crate was added, only
  `bench/oracle-install.sh` runs, staged and swapped in on success.
- **Testing.** `bench/land-test.sh` drives the script through about 130 assertions in a
  throwaway clone. `bench/land.sh --check ...` runs every condition and stops before the
  merge.

## Secrets

The source copy of each secret is an item in the Proton Pass vault `folknet`; `.env`
(gitignored) is the working copy every script and cron job reads, so nothing that reads
it changes. `bench/env-from-vault.sh` rewrites the six secret values in `.env` from the
vault and keeps every other line; `--check` reports drift without writing, and neither
prints a value. Rotate a secret in the vault, then run the script. It needs a logged-in
`pass-cli`, which keeps its local key in a file (`PROTON_PASS_KEY_PROVIDER=fs`, exported
by `~/.profile` and `~/.bashrc` above its interactive check, and set by the script)
because the kernel keyring it uses by default is revoked when the login session ends; a
shell that lacks the variable creates a database keyed from the keyring, and the next
call under the file store cannot open it ("file is not a database"). When the session is
unusable the script resets it and logs in again from a personal access token kept in a
mode 600 file (`~/.config/folknet/pass-pat`, or `FOLKNET_PASS_PAT_FILE`), if that file
exists; otherwise a person runs `pass-cli logout --force` and `pass-cli login`.
`bench/env-from-vault-test.sh` covers the script with a stub.

## Training limits

Full fine-tuning is out of reach on 24 GB Ampere cards; LoRA or QLoRA on a 30B-class model
across eight GPUs is feasible, so any supervised step is a LoRA.

## Sharding without a broker

Case generation is deterministic from a seed, so worker *i* of *n* takes seeds
`i, i+n, i+2n, …`: no queue, no coordinator, and any worker can restart without losing or
duplicating work. Output is content-addressed and written once; ground truth is immutable
per engine version, so there is nothing to invalidate.

## Storage

About 10 KB per document for three engines, so ten million cases is about 100 GB. The
style subset dominates and is redundant; compress it, sample it, or store it as a delta.

## Virtual machine template

One image, duplicated: WebKit's 188 system packages, pinned Playwright browsers, Rust with
sccache, the repository. Workers are stateless. Pin one VM per NUMA node rather than one
large VM per host.

## Engine versions pin the ground truth

Ground truth is immutable only for a given set of browser versions, so every dump records
`engine_version` (`docs/dump-format.md` → "Dump") and a corpus is keyed by that version
set. A browser update invalidates only what it touches, never silently.

## Rig change log

Every change tried on the rig's hardware, worked or not, newest last. Hosts by role and
class only. Before a change, and again after each change that works,
`bench/rig-snapshot.sh snapshot <label>` records the serving configs of every role (proxy
host HAProxy, gateway container config and key database, each ARM node's units and
TabbyAPI configs, the four-card VM's Proxmox config and units) on the host itself and in
the local rig notes; `restore <label>` puts them back. The label named in an entry holds
the working state after it. Snapshots contain the gateway's master key and never go into
git or onto the NAS.

**2026-09-24**

- *Gateway key for self-hosted writers.* A new virtual key on the gateway, alias
  `nev-folknet-writer`, allowed only `qwen3.8-27b-long` and `qwen3.8-27b-fast`, so
  writer traffic is accounted apart from batch traffic and cannot reach any other alias.
  No serving config changed. The key itself lives outside every repository, as
  `FOLKNET_WRITER_API_KEY` in `$HOME/.config/folknet-writer/env` (mode 600) on the
  development box. First used by stage 0 of `docs/folknet-writer-design.md`
  (`docs/writer-harness-spike.md` → "Rerun through Pi"). Worked.
- *Snapshot and restore.* `bench/rig-snapshot.sh` written; `known-good-2026-09-24`
  taken before any serving change. `restore known-good-2026-09-24 haproxy` run once as the
  round-trip test: config check passed, HAProxy reloaded, all four ARM replicas still up.
  Worked.
- *Four-card VM.* A new VM on a Proxmox node, given the four 3090s of a stopped tenant
  test VM (same `hostpci` entries, q35, OVMF, 256 GB 64-bit PCI hole), 32 cores, 160 GB
  RAM, a 120 GB OS disk, DHCP on the main LAN (`bench/gpu-vm/create-vm.sh`). The tenant
  VM's disks were not touched. Worked; four 3090s visible.
- *Provisioning, first two attempts failed.* First: the fresh cloud image was still
  running cloud-init's package upgrade, which held the dpkg lock and then rebooted the
  guest. Second: the rig's apt cache answers `403 CONNECT denied` for HTTPS repositories,
  so docker's and NVIDIA's apt repositories could not be reached through it. Fixed in
  `bench/gpu-vm/provision.sh`: wait for cloud-init and the dpkg lock, send HTTPS
  repositories direct, take docker from the Ubuntu archive (`docker.io`, which the cache
  serves). Third run worked: driver 595 (open kernel modules) loaded without a reboot,
  `nvidia-smi` shows four 3090s, docker has the NVIDIA runtime.
- *Model choice for the four cards, by size (no download attempted).*
  - `qwen3.8-flash-next`, EXL3 4.05 bpw (the ARM nodes' checkpoint): 101 GB on disk, of
    which 39 GB is the hashed n-gram table that exllamav3 keeps in host RAM or on disk,
    so about 62 GB of weights would sit on the cards. That the checkpoint does not fit
    96 GB was assumed, never tested. A 3.05 bpw branch is also published (85 GB on disk,
    32.6 GB of it n-gram table). Not pursued: the target moved to a different model family
    for variety.
  - DeepSeek-V4-Flash (the July release, 284B parameters, 13B active): 167 GB native
    (MXFP4 experts). Mainline llama.cpp with CUDA serves it on sm86 from GGUF. The
    smallest quant its quantizer recommends (3-bit, 104 GB) does not fit 96 GB of VRAM;
    2-bit quants (87–91 GB) fit with only a few GB left for cache and compute buffers,
    and re-quantizing the native 4-bit experts to 2 bits costs a lot of accuracy
    (published KL divergence around 0.4 at 2 bits against 0.01 at 4 bits). Download
    straight from Hugging Face inside the VM measured about 250 KB/s. Not tried.
  - DeepSeek-V4.1-Flash is a different, larger model, not a revision of the one above: a
    new architecture (`deepseek_v41`, 384 experts over 40 layers, n-gram memory tables),
    510 GB native. It does not fit the four cards. The first reading, that it fits no
    pair of ARM nodes either, was wrong: it looked only at GGUF quants (smallest about
    260 GB) and missed a published EXL3 recipe for exactly two GB10s — a 2.9 bpw EXL3
    checkpoint (~197 GiB, about 99.5 GiB of weights per node at tensor-parallel 2) plus
    the model's unquantized n-gram ("Engram") tables, which stay on disk and are read
    per lookup rather than held in memory (two native shards, ~190 GiB), served by a
    vLLM build that carries the `deepseek_v41` architecture. That recipe is what the
    ARM pair uses (entries below).
  - qwen3.8-27b, W4A16 AutoRound with DFlash2 speculative decoding, the eight-card
    host's `-fast` recipe: about 22 GB per card, one worker per card. Chosen.
- *qwen3.8-27b on the four cards.* The eight-card host's image and prepared checkpoint
  (24 GB) were streamed to the VM read-only (`tar` and `docker save` piped over ssh,
  nothing on that host changed). One worker per card came up in about 6.5 minutes from
  cold; all four report healthy and answer chat completions. Worked; snapshot
  `vm-qwen27b-workers-up`.
- *Proxy and gateway for the VM.* A new HAProxy frontend and backend on the proxy host
  (the existing backends untouched; backup `haproxy.cfg.pre-vm-qwen27b-20260924` beside
  the config) and a gateway alias `qwen3.8-27b-fast-2` (backup
  `config.yaml.pre-qwen27b-vm-20260924`), granted to the project's key only. The first
  gateway attempt failed with "Missing credentials": an OpenAI-style entry needs some
  `api_key` even when the upstream checks none; `sk-none`, as the flash-next entry uses,
  fixed it. Worked: the alias is listed and returns code through the gateway; snapshot
  `vm-qwen27b-fast-2-alias`.
- *Measurement batch at `qwen3.8-27b-fast-2`* on a build lane: one 32-candidate
  absolute-positioning batch, 2/32 compiled, best 74/108 (`docs/model-first-cost.md`, row
  `vm-batch 1`). Stopping the loop after batch 1 with `pkill -f` on the shared lane also
  killed another track's run of the same binary in its own lane worktree (a mixed-model
  batch at 32 of 64 draws). On a shared lane, stop a run by the pid whose
  `/proc/<pid>/cwd` is this worktree's lane copy, never by name.
- *Checkpoint fetch for an ARM pair.* `bench/sparks/hf-to-nas.sh` (a runtime bundle and
  oneshot unit on the downloader host, reusing that host's multi-WAN scripts) started
  DeepSeek-V4-Flash (July release, 167 GB); stopped after 14 GB when the ARM target
  changed to DeepSeek-V4.1-Flash. The partial staging directory is left on the NAS. Then
  both parts of the V4.1 recipe, pinned by revision: the EXL3 2.9 bpw checkpoint
  (42 files, 210.7 GB) and the two Engram shards plus index and config of the native
  checkpoint (203.1 GB), both through the multi-WAN downloader at about 130 MB/s
  combined. Snapshot before: `pre-deepseek-sparks`.
- *Which two ARM nodes.* LLDP shows each node's first 200G port on the switch and its
  second port cabled straight to one other node, in two cabled pairs, each crossing
  the old GLM pairs (which ran over the switch). The DeepSeek pair is one of the cabled
  pairs; the other cabled pair keeps flash-next. Its tensor-parallel traffic is set to
  run over the switch port with the RoCE settings the GLM pairs used, because the direct
  cable carries no addresses today; moving it onto the cable means addressing that
  port on both nodes (not done).
- *Staging the pair (nothing serving yet).* The recipe is cloned at a pinned commit on
  the head node, its `.env` written from `bench/sparks/dsv41/env.template` (weights from
  local disk, no Hugging Face download, `AUTO_DOWNLOAD=0`), an ssh alias for the worker
  added to the head's admin account (`bench/sparks/dsv41/ssh-config-worker`; previous
  config kept as `config.pre-dsv41`), `/srv/dsv41` created on both nodes, and the
  recipe image pulled on both (the `2.9bpw` tag, 22.5 GB; digest in the local rig
  notes). Both nodes still serve flash-next. Snapshot `dsv41-staged`.
- *Throughput sweep of `qwen3.8-27b-fast-2`* (`bench/rig_throughput.py`) on a build lane:
  failed twice before sending a request — the lane's Python has no `requests` module,
  and `python3 -m venv` fails there without the `python3.12-venv` package. Not run.
- *Four-card VM reboot.* All four workers healthy three minutes after `reboot`, same
  DHCP address, HAProxy backends back UP without intervention. Worked.
- *Second measurement batch at `qwen3.8-27b-fast-2`*, after the prompt fixes, run with
  `bench/absolute-mixed-batch.sh` on a build lane: 8/32 compiled, best 82/108, 233 s for
  the batch (`docs/model-first-cost.md`, row `vm-batch 2`). The run ended by itself
  (`GENERA_MAX_BATCHES=1`); nothing needed stopping. No serving change.
- *Throughput sweep of `qwen3.8-27b-fast-2`, second attempt*, run from the dev box
  (which has `requests`) through the gateway, with the prompt generated on a build lane:
  worked, 819 output tok/s at 16 in flight (`docs/rig-capacity.md` → "Measured — the
  four-card VM"). No serving change.
- *Abandoned DeepSeek-V4-Flash download removed* from the downloader host: the 14 GB
  partial staging directory on the NAS, its oneshot unit and runtime bundle. Worked; the
  two V4.1 units and their published trees are untouched.
- *DeepSeek-V4.1-Flash weights to local disk on the ARM pair.* Both NAS downloads had
  finished and published. `rsync -a` from the NAS mount to local NVMe on both nodes, the
  EXL3 checkpoint (39 shards, 197 GB) to each, the two Engram shards with index and
  config (190 GB) to the head's engram source directory and to the worker's engram
  directory. The worker gets no second engram tree: the recipe's `start.sh` builds the
  slim engram set on the head and rsyncs it to the worker, and with the NAS mtimes kept,
  that rsync finds the shards already in place. About 100 MB/s per node, both done in
  about 55 minutes. Both nodes kept serving flash-next throughout, all backends UP
  after. Worked; no serving change, so no new snapshot (`dsv41-staged` still holds the
  serving state).
- *Two ARM nodes out of the flash-next pool.* Snapshot before: `pre-dsv41-pair-move`.
  Their two server lines were removed from the flash-next HAProxy backend (backup
  `haproxy.cfg.pre-dsv41-pair-20260924` beside the config; config check, reload; the
  other two replicas UP), then their flash-next units disabled and stopped; unit,
  config and checkpoint stay on disk. Nothing was drawing from the alias at the time.
  The alias now takes 16 requests in flight (two replicas × 8), not 32. Worked; snapshot
  `flashnext-two-replicas`.
- *DeepSeek-V4.1-Flash on the pair.* The published recipe image carries the recipe's
  own build stamp from four commits before the pinned one, so the pinned `start.sh`
  would have compiled the image on the node. The only image-side change in between is
  one Python patch, which that `start.sh` bind-mounts into both containers anyway, so the
  unit runs the published image as-is (`SKIP_BUILD=1`, reason in the unit file). Unit
  installed on the head and started: preflight passed (memory headroom 117.6 GiB
  available per node against 111.5 needed, RoCE GID accepted), the weight sync to the
  worker found everything already in place and took 3 s, weights loaded and warmed up,
  API healthy about 7.5 minutes after the start; the README's `17*19` request returned
  `323`. The recipe's own warmup reports one sampler-kernel variant not pre-compiled
  ("may JIT mid-serve"); harmless so far. Worked; snapshot `dsv41-pair-serving`.
- *Proxy and gateway for the pair.* A new HAProxy frontend and backend on the proxy
  host, one server, `maxconn 2` (backup `haproxy.cfg.pre-dsv41-backend-20260924`), and
  a gateway alias `deepseek-v4.1-flash` → the recipe's served model name, `api_key:
  sk-none` (backup `config.yaml.pre-dsv41-20260924`), granted to the project's key only.
  Worked: the alias is listed through the gateway and a genera-shaped chat completion
  (thinking off) returned a one-function Rust code block in 2.7 s with no reasoning in
  `content`. Snapshot `dsv41-alias-serving`.
- *First measuring batch at `deepseek-v4.1-flash`* on a build lane, 8 candidates at 2 in
  flight: 4/8 compiled, best 82/108, 1,731 s (`docs/model-first-cost.md`, row
  `dsv-batch 1`). The first launch failed before drawing anything: the lane's shared git
  mirror refused a ref update (`cannot lock ref 'refs/heads/main'`) because another run
  had moved `main` at the same moment; a plain retry worked. No serving change.

**2026-09-25**

- *Dead GLM aliases removed from the gateway.* `glm-5.3-flash` and `glm-5.3-flash-exl3`
  were still listed by `/v1/models` three days after their backends stopped, so a caller
  could pick one and get a 503 ("No server is available to handle this request"), as a
  job on another track did. Snapshot before: `pre-glm-alias-removal`. Both model entries
  deleted from the gateway config (backup `config.yaml.pre-glm-removal-20260925` beside
  it; no fallback named them) and the gateway service restarted, live again in 18 s. The
  two names were then dropped from the allowlists of the three virtual keys that carried
  them, each keeping its other models. Worked: `/v1/models` through the dev-box tunnel
  lists neither name; one-token completions return 200 for `qwen3.8-27b-fast`,
  `deepseek-v4.1-flash`, `qwen3.8-27b-fast-2` and `qwen3.8-flash-next`; a request for
  `glm-5.3-flash` is refused with 403. Snapshot `glm-aliases-removed`. The site file the
  gateway config is generated from still carries the GLM entries, so a regeneration
  would bring them back.

**2026-09-29**

- *`deepseek-v4.1-flash` back after the outage reboot.* It had answered 503 since
  2026-09-28: the pair unit ran before the head's fabric address came up, `start.sh`
  refused ("HEAD_IP ... is not assigned"), and a oneshot unit is not retried. Snapshot
  before: `before-dsv41-fix`. `folknet-glm53-vllm` was enabled on all four ARM nodes and
  crash-looping; disabled and stopped on all four. The pair unit gained an
  `ExecStartPre` that waits up to 300 s for the `HEAD_IP` its `.env` names
  (`bench/sparks/dsv41/folknet-dsv41-exl3-pair.service`), installed on the head and
  started: preflight saw 117 GiB free on both nodes, and the unit finished in 7 minutes.
  Worked: a completion for `deepseek-v4.1-flash` through the gateway returned 200 in
  1.6 s. Snapshot `dsv41-pair-waits-for-head-ip`. *Unverified*: the wait has not yet
  run across a real reboot.

- 2026-10-23, build lanes: the sccache client's local disk cache is capped at 8 GiB
  (`bench/lane-dist.sh` `DIST_DISK_CACHE_BYTES`; it was 20 GiB and filled a lane with 21 GB)
  so a 64 GB root keeps room for a job's build output; misses fall through to the shared
  redis. Applied to the lane whose client config exists and answered; a lane that did not
  answer keeps the old cap until `lane-dist.sh` runs on it again.

- 2026-10-23, build lanes: three more builder containers became build lanes
  (`bench/lane-provision.sh`, then `bench/lane-nas.sh` on each container's node, then
  `bench/lane-dist.sh` role `registries`), appended to `bench/lane.local.json` `lanes[]`
  and not to `dagu_lanes`. Each built the workspace from main through
  `bench/remote-lane.sh` in about 50 s. They have no sccache client, like the other two
  builder lanes, because the client tokens are not on the development box. Found on the
  way: `bench/folknet-check.sh --fix` never ran `lane-nas.sh`, since `node()` replaces
  the command's input with `/dev/null`; the fix goes through `node_script()`. Worked.
  *Unverified*: peak disk use of a feature job on these 64 GB roots.