# Infrastructure

The machines the generation loop runs on, and how work reaches them. The loop itself is
in `docs/generation-loop.md`; measured rates are in `docs/rig-capacity.md`. Hardware is
named by role and class, never by hostname, address or account — this repository is
assumed public. Real names live in the gitignored `bench/*.local.json` and the local rig
notes.

## In brief

- **The loop.** One engine function per spec algorithm (for example the CSS 2.1 width
  calculation), written by a model, checked against cached browser results.
- **Sampling tier.** GPU cards running mid-sized models write N versions of a function
  from the spec text, the engine's Rust interfaces and the currently failing cases. Many
  cheap tries beat one careful one, because the browsers say which tries are right.
- **Capacity tier.** When every version fails, a larger model gets the spec, the best
  failing versions and the exact per-box geometry differences.
- **Verification tier.** CPU servers compile candidates in one batch and run each inside
  the engine over the cached corpus, under a millisecond per page. The same tier builds
  that cache by running the three browsers once per page, and runs conformance suites.
- **Outcomes.** Keep the passing version, sample again with the new failures, or
  escalate. What is kept is ordinary Rust with a provenance record; no model runs in the
  shipped engine.

Environment variables are named `NEV_*`; older records use `GENERA_*`, which every reader
still accepts (`docs/rename-to-nev.md`).

## Detail

### The full cycle

```mermaid
flowchart TD
  browsers["Chromium, Firefox, WebKit<br/>once per page (CPU)"] --> cache["cached ground truth<br/>three-engine vote"]
  spec["spec text<br/>one algorithm"] --> sample["SAMPLE (GPU pool)<br/>N versions of one function"]
  sample --> verify["VERIFY (CPU)<br/>dedupe, batch compile,<br/>run over the corpus"]
  cache --> verify
  verify -- "all pass" --> accept["commit Rust source<br/>+ provenance record"]
  verify -- "some fail" --> sample
  verify -- "all fail repeatedly" --> escalate["ESCALATE<br/>larger model + geometry diffs"]
  escalate --> verify
```

Nothing is accepted on a model's word: escalated output goes through the same check.
Writing candidates is the binding constraint; verification has a large margin over it.

### What comes out, and how the engine uses it

The cycle outputs **ordinary Rust source**, committed next to the hand-written code it
plugs into, with spec steps as comments and a **provenance record** (`roadmap.md` →
"Provenance is a first-class artifact"). The engine compiles it like any other code; the
GPU and CPU fleets exist only at build time.

**Provenance drives maintenance.** When a spec section or a definition it uses changes,
the provenance index finds exactly the functions to regenerate (`roadmap.md` → "A
specification changes"); a browser update re-dumps the ground truth instead. The same
index is the bill of materials for a configurable build (`roadmap.md` → "Phase 4").

## Measured baseline

One oracle process, three engines, 235 documents including browser startup: **17.9 s**,
about **13 documents per second per engine**, using about two cores. Scaled to a few
hundred cores that would be about 3,000 documents per second, a ten-million-case corpus in
about an hour — but per-core throughput on the contended 16-core development box fell from
1.48 to 0.48 docs/s/core between 1 and 16 workers (`docs/rig-capacity.md` → "Oracle
corpus building"), so the figure on dedicated hardware is still to be measured. GPU-side
figures: `docs/rig-capacity.md` → "GPU tier".

## Three tiers

**Sampling tier — high memory bandwidth GPUs.** Bulk best-of-N generation,
throughput-bound. Decode speed tracks *active* parameters, so a mixture-of-experts model
with a small active fraction gives several times the candidates of a dense model. Ampere
cards have no fp8, so quantization is 4-bit weight-only.

**Capacity tier — large unified-memory accelerators.** About a third of the bandwidth,
several times the memory: for the few hard calls and any teacher model. About 100B total
with ~12B active is the realistic ceiling.

**Verification tier — many CPU cores.** Corpus building, candidate compilation and the
inner loop. It must never spawn a browser per case.

## As discovered

Every host advertises itself over mDNS and answers an HTTP inventory; a collector
aggregates it. What that shows, by role:

- **The model gateway** is a LiteLLM instance fronting one alias per model pool. The
  aliases, their pools, how many requests each takes at once, and which are retired are
  declared in `bench/folknet.json` (the two GLM-5.3 Flash aliases, without backends since
  2026-09-22, were removed from the gateway on 2026-09-25); whether each one answers
  right now is in each lane's snapshot, which a probe refreshes every 5 minutes and
  `bench/folknet-status.sh [--lane N]` prints (status, how long it has held it, last
  answer, snapshot age). Design: `docs/folknet-inventory.md`. Virtual keys carry per-key
  allowlists, so a new alias must be granted to the caller's key. Each pool's HAProxy
  reports liveness, not readiness: a `/v1/models` answer does not prove a completion
  will return.
- **Four ARM unified-memory nodes (DGX Spark class)** — `aarch64`, one GB10 GPU, 20
  cores, 121 GB RAM, 3.7 TB NVMe each. They served GLM-5.3 Flash as two independent
  two-node pairs until 2026-09-22. Qwen3.8-Flash-Next did not run under any vLLM build on
  the rig (the architecture was too new for the rig's EXL3 fork, and upstream has no EXL3
  backend), so it serves through the checkpoint's own engine (TabbyAPI) instead, one
  replica per node, about 75 GB plus ~45 GB of page cache each. Two of the four now serve
  DeepSeek-V4.1-Flash. Full account: `docs/rig-capacity.md` → "Attempted —
  Qwen3.8-Flash-Next".
- **The eight-card host** — eight 24 GB consumer cards on a 64-core EPYC with ~500 GB
  RAM, bare Linux, six serving containers using every card (four behind `-fast`, two
  behind `-long`). The closest thing to the sampling tier as designed. No spare GPU.
- **The verification tier** — ten Proxmox 9.x nodes (EPYC 7742, 7B13, 7K62 and 7452; 32
  to 64 physical cores, 125 GB to ~500 GB RAM, 1–2 TB NVMe), shared with tenant VMs;
  five pass thirteen consumer cards through to tenant machines. A VM sized for browser
  builds sits on the GLM-pool proxy host (open: its core count and whether it should be
  running). Free capacity is host by host.
- **A downloader host** pulls checkpoint shards in parallel across the site's four WAN
  uplinks onto the NAS, verifies them and publishes a pinned revision for nodes to sync.
- **Access.** Root with the jump host's key works on every physical host; the ARM nodes
  use their own admin key. The gateway's own guest refuses, and is not a compute host.

## Where each piece runs

Every role is a guest or a container; nothing is installed on a Proxmox node's own OS.

```mermaid
flowchart LR
  dev["development box<br/>16 cores, shared"] -- "ssh via jump host" --> lanes["2 build lanes<br/>(LXC, 96 threads each)"]
  dev -- "port forward" --> gw["model gateway<br/>(LiteLLM)"]
  lanes --> gw
  gw --> fast["eight-card host<br/>-fast, -long"]
  gw --> vm["four-card VM<br/>-fast-2"]
  gw --> fn["2 ARM nodes<br/>flash-next"]
  gw --> ds["2 ARM nodes<br/>deepseek pair"]
  lanes --> helpers["compile helpers<br/>(sccache-dist)"]
```

- **Development box.** No GPU, no rig role; holds the port forwards to the gateway.
- **Jump host.** A VM on one node; the only route in. Nothing of this project on it.
- **Build lanes.** One unprivileged LXC on each of two nodes (`bench/lane-create-lxc.sh`,
  `bench/lane-provision.sh`), started at node boot. A gate running when a lane stops
  fails.
- **Compile helpers and scheduler.** Four more containers plus builders on the two lanes;
  scheduler, redis cache and a loopback CONNECT proxy on one lane (`bench/lane-dist.sh`).
- **Eight-card host.** Docker containers under a templated unit
  `folknet-qwen38-vllm-worker@<profile>`, behind that host's own HAProxy on port 4000.
  Not edited by this project.
- **ARM nodes, split two ways since 2026-09-24.**
  - Two serve `qwen3.8-flash-next`, one systemd unit and container each on port 5000
    (`bench/sparks/`), behind the proxy host's HAProxy (least connections, eight in flight
    per replica) — so the alias takes **16 in flight, not 32**; a batch at concurrency 32
    queues the rest. Either replica can stop alone.
  - Two, joined by a direct cable, serve `deepseek-v4.1-flash` as one tensor-parallel
    pair: one oneshot unit on the head node
    (`bench/sparks/dsv41/folknet-dsv41-exl3-pair.service`) starts both containers and
    returns once the API is healthy (about 7.5 minutes cold); stop and start only through
    that unit, never one node alone. HAProxy `maxconn 2`.
  - Units are stopped and started through a privileged container that chroots into the
    host, because the admin account has no sudo. GLM units are disabled on all four.
- **Four-card VM (`qwen3.8-27b-fast-2`).** Four 3090s borrowed from a stopped tenant VM on
  the same node (only one of the two may run; `bench/gpu-vm/`). Four units
  `folknet-qwen38-vllm-worker@fast-{0..3}` on ports 8101–8104, the eight-card host's
  `-fast` recipe, HAProxy `maxconn 4` per worker (16 in flight). A separate alias because
  the `-fast` load balancer lives on the eight-card host.

**Open.** Which host carries the four-card VM behind `qwen3.8-27b` (the measured Qwen
server), and how it is started and stopped; whether anything restarts the eight-card
host's HAProxy or re-pulls its image;
what keeps the development box's port forwards up after a reboot; and the gateway config
is generated from a site file outside the rig, which needs every hand-added alias too.

## Sharing the hardware

The Proxmox nodes are shared with other tenants. Three rules:

- **Nothing of this project goes on a node's own OS**, so a guest can be stopped and its
  hardware handed back without touching the node.
- **GPU work runs in a VM with passthrough, never an LXC** (an LXC would need the vendor
  driver on the node). CPU work runs in an unprivileged LXC, or in docker on non-Proxmox
  hosts.
- **Stopping a project guest frees what it held**, and a lane competes with tenants as an
  equal under contention. Nothing checks this; it is how the rig is laid out.

Limits today: the ARM replicas' HAProxy queues a request at most 30 minutes; power
budgets are modelled (`bench/capacity.py`) but not enforced.

**Not built yet:** a per-role start/stop/drain wrapper; a drain (a replica's stop gives
in-flight replies only 60 s); taking a backend out of HAProxy before stopping it; a narrow
sudo rule for the ARM nodes' admin account.

## Build lane

Two unprivileged containers (96 logical CPUs — three quarters of a 64-core, 128-thread
node — 96 GB, 200 GB disk), each a build lane and oracle host: the Rust toolchain, Node
22, the repository cloned from the rig's forge ("Where the repository is stored" below),
Playwright's three browsers, and a PulseAudio null sink so a realtime `AudioContext` runs. The scripts are the record and are
re-runnable: `bench/lane-create-lxc.sh`, `bench/lane-provision.sh`, `bench/lane-nas.sh`.

**Running a gate there.**

```mermaid
sequenceDiagram
  participant W as worktree (dev box)
  participant L as lane
  participant F as rig forge
  W->>L: ssh: git bundle of unpushed commits + diff + untracked files
  L->>F: fetch the lane's mirror (only if needed, or older than 10 min)
  L->>L: lane-apply.sh: check out exact commit, apply diff, run command
  L-->>W: streamed output (and --pull directories)
```

`bench/remote-lane.sh <worktree> <command...>` does this. Each lane root keeps one bare
mirror (`<root>/.git-mirror`) with one git worktree per track; it fetches only when the
bundle cannot supply the commit or the last fetch is older than `NEV_LANE_FETCH_MAX_AGE`
(600 s), because the public forge the mirrors first fetched from rate-limits. The lane
address comes from `NEV_LANE_SSH` or `bench/lane.local.json`; the mirror URL from `NEV_LANE_GIT_URL`, that file, or the
worktree's `origin`. Examples:

```
bench/remote-lane.sh . cargo test --workspace
bench/remote-lane.sh --pull corpus/absolute-ground-truth . \
  cargo run --release -p generation --bin absolute_ground_truth_cache
```

Why a bundle rather than rsync: uploads from the development box to the lane run at about
18.6 KB/s (downloads 435 KB/s), so rsyncing a worktree took ~315 s of a ~424 s run.

**Where the repository is stored.** Two forges, with different jobs:

| forge | role | holds |
|---|---|---|
| the rig's Forgejo | intermediate storage, on the rig network | everything the lanes need: `main`, job branches `job/<name>` and their results `result/<name>` (`specs/job-submit-land.md`) |
| tangled | persistent storage, public | `main` and the `spec/<name>/<sha>` landing tags only |

The Forgejo repository is a regular repository, not a mirror of tangled. The lanes clone
and fetch from it: each lane root's `.git-mirror` and the lane user's clone fetch from
`http://<forge-user>:<forge-token>@<forgejo-address>/<owner>/<repo>.git`, and
`bench/lane.local.json` `git_url` (or `NEV_LANE_GIT_URL`) names the same URL. The user and
token come from the primary checkout's `.env` (`GENERA_FORGE_USER`, `GENERA_FORGE_TOKEN`,
still under the old prefix; `docs/rename-to-nev.md`). The development box reaches Forgejo
through a local port forward, as the remote `rig`
(`http://<forge-user>:<forge-token>@127.0.0.1:<forward-port>/<owner>/<repo>.git`), which
the job scripts use by default (`JOBS_FORGE_REMOTE`). `origin` fetches from tangled and has
push URLs for both forges, so `git push origin main` updates both; job and result branches
go to `rig` only and are deleted there after a landing. Each of those URLs carries the
token: they live in `.git/config` and the gitignored files, never in a committed file, a
log line or a message.

**The track gate runs only on the build lane.** `tracks.json` → `verify` runs
`bench/verify.sh`, which hands itself to `bench/on-lane.sh --lane-only` and, once there,
runs `cargo test --workspace`, the job event store's tests (`jobs/events/*.test.mjs` under
`node --test`, then `bench/job-events-lane-test.sh`), and then
`bench/generated-scores.sh`. The other `*.test.mjs` and `bench/*-test.sh` files are run by
hand, not by the gate. With no lane
configured or none answering (`ssh -o BatchMode=yes -o ConnectTimeout=5` fails) the gate
exits 3 and runs nothing on the development box; a lane that answers but fails the tests or
a score is a failed gate — no local retry. Measured before the scores were added: a new
worktree's first test run 106 s (124 s with a fresh mirror), a warm rerun 22 s, local
fallback 78 s. A clean `cargo test --workspace` compiles in 61 s on a lane and 57 s on the
development box — the build is bound by the dependency chain through Stylo, not by cores;
the lane's value is that it takes load off the shared box and runs several gates at once.

**Unattended landings never fall back to the development box.** `bench/on-lane.sh`
without `--lane-only` runs its command locally when no lane answers, and `bench/land.sh`
runs each `landChecks` command that way, and `verify` directly. Setting
`NEV_LANE_STRICT=1` in `bench/land.sh`'s environment (the scheduled
`jobs/scripts/poll-and-land.sh` gets it from `tracks.json`'s `specWatch.pollEnv`) makes
`verify` go through `bench/on-lane.sh --lane-only` too and every check `--lane-only`, so
with no lane answering the landing is refused with "verify failed on the tip (exit 3: no
build lane answered ...)" and nothing of the gate runs here. Unset, as in interactive
use, a check keeps its local fallback. `bench/verify.sh` itself is lane-only either way;
the one thing that would run it on the development box is `NEV_ON_LANE` set there by hand.

**Already on a lane, `bench/on-lane.sh` runs the command in place.** With `NEV_ON_LANE`
set, `bench/on-lane.sh` (and `bench/remote-lane.sh`) runs its command in the named
worktree on the machine it is on, `--lane-only` or not, reads no lane settings, and
ignores `--pull`, since the outputs are already where they belong; the command's exit
status is the run's. This is the same check `bench/verify.sh` makes, so a spec's
`bench/on-lane.sh --lane-only "$PWD" ...` works unchanged inside a Dagu job, whose clone
has no `.env` and no `bench/lane.local.json` and whose writer gets `NEV_ON_LANE=1` from
`jobs/scripts/pi-session.sh`. As with `bench/verify.sh`, a `NEV_ON_LANE` set by hand on the
development box would run the command there. `bench/on-lane-test.sh` covers both sides.
The cost: with both lanes down nothing lands until one answers. `bench/land-test.sh`
scenario `s1` covers both settings.

**The gate scores every generated function against its own corpus.** A committed file under
`crates/layout/src/generated/` can be wrong while every test passes, because the tests
exercise in-tree reference rules, not the generated function. `bench/generated-scores.sh`
clears and rebuilds each corpus with its `*_ground_truth_cache` binary (the browser oracle
runs on the lane), then `generated_score_check` scores each generated function through its
own runner and fails it against its floor. The floor is two numbers in
`crates/layout/src/generated/<name>.floor.json` beside the function's provenance record:
`cases_passed`, how many cases it passes, and `cases_run`, how many cases its corpus holds
(only cases three engines agreed on). It fails when the passing count drops, when the corpus
shrinks even though the passing count holds, and when a function passes no case at all (a
lane missing a browser leaves fewer than three engines to vote, so every case drops; a floor
of zero cannot hide a function that scores nothing). The provenance record describes the
model batch that admitted the function and is never rewritten; the floor is never below its
`verification.cases_passed`, and a function with no floor file yet is held to that alone. A
placeholder record, such as the hand-written `table` one (0 recorded), therefore floors at
its real score once a floor file exists.

**Floors rise by themselves, after a landing.** `bench/after-land.sh` ends with
`bench/generated-floors-update.sh`, which runs `bench/generated-scores.sh --print-raises` on
the build lane, writes each floor that the fresh score is above (a function with no floor
file gets one), and commits only those files as one commit on `main`, the way the map and
dashboard commits are made. It refuses on uncommitted changes, writes nothing when the lane
does not answer or a score fails, never lowers a number, and a failure is one line and does
not change the landing. Raising happens after the landing rather than in the gate because
the gate runs on the lane's copy, where a commit cannot reach the track, and because a track
never edits a floor: `bench/land.sh` refuses a diff that touches any `*.floor.json`, so an
unattended track cannot lower one. The cost is a second scoring run on the lane per landing,
and a window between a landing that improves a function and the raise in which the old floor
holds. A floor comes down only by a person's reviewed edit of its file.

**One place to add a generated function.** A function is discovered from its
`<name>.provenance.json`; its runner and the cache binary that writes its corpus are one
entry in `FUNCTIONS` in `crates/generation/src/bin/generated_score_check.rs`, and a
provenance record with no entry fails the gate with that instruction. `bench/generated-scores.sh`
takes the caches to rebuild from `generated_score_check --list-caches`, so nothing under
`bench/` changes and a track adding a function can land unattended.
`bench/generated-scores-test.sh` exercises the scripts, including the raise, without a
browser.

**Model batches run only on a lane.** `bench/absolute-mixed-batch.sh` and
`bench/overflow-mixed-run.sh` are started from the development box like any script
(`NEV_MODELS=qwen3.8-27b-fast:32:32 bench/absolute-mixed-batch.sh`) and re-run themselves
on the lane through `bench/on-lane.sh --lane-only` (`bench/batch-lane.sh`): the `NEV_*`
settings go along, credentials and lane settings excepted, and the batch's results come
back with `--pull`. With no lane configured or none answering the script exits 3 and runs
nothing locally; `bench/lane-apply.sh` exports `NEV_ON_LANE=1`, which is how a script
knows it is already there. Each run also records its phase timings and how many of the
lane's CPUs each phase kept busy (`bench/cpu-sample.sh`, `bench/cpu-phases.py`;
`docs/rig-capacity.md` → "A real batch on a build lane").

**A stray directory at a worktree's lane path breaks the gate.** `bench/lane-apply.sh`
tests only for `<lane-root>/<name>/.git`; a leftover directory without it makes `git
worktree add` fail, and the gate cannot pass until it is removed by hand (check `git -C
<mirror> worktree list` first). Not yet fixed in the script.

**Cleanup.** Every worktree run on a lane leaves its copy (`<root>/<name>`, `target/`
and all, 3–12 GB) and `<root>/.payload-<name>` there; left alone they filled a lane's
disk. `bench/lane-sweep.sh` removes, on every lane in `bench/lane.local.json` `lanes[]`,
the copies whose track has no worktree in the primary checkout (`--dry-run` lists them
with sizes). It keeps the primary's own copy, any copy with a process running in it (a
spec writer runs inside its copy), and directories that are not worktrees of the lane's
mirror. Untracked data under a copy's `corpus/` (links excepted: they point into the
NAS already) goes to the NAS first, as one tar file at
`archive/corpus/<name>/<time>.tar`; with no NAS directory the copy is kept. A branch a
copy was on stays in the mirror. It runs at three points: by hand; from `bench/land.sh`
after a landing, for that track only (a failure is named on its line and undoes
nothing); and from `bench/remote-lane.sh` when the lane's root filesystem has less than
`NEV_LANE_MIN_FREE_GB` (20) free before a run, which then refuses with exit 4 if the
sweep did not free enough, naming the lane by its place in the list, the free space
and the largest copies. A hand landing leaves no worktree, so the next sweep takes its
copies. Scenarios: `bench/lane-sweep-test.sh`.

**Test-run memory.** A cold `cargo test --workspace` on the development box adds about
1.1 GB at peak (10.8 → 12.0 GB), a warm one 67 MB; seven concurrent tracks fit in its 30
GB if their peaks do not coincide.

**Distributed compilation.** `sccache-dist` builders on six nodes (416 cores registered)
with redis as the shared cache. After dropping one builder on a node with slow outbound
network (failures 84 → 1), it still lost on wall time: a clean `-p dom` release build took
82 s distributed against 45 s local on the same lane. The dependency graph is too narrow
to use the cores. A wide build fares no better: a batch's candidate probe (one
`--keep-going` build of 32–64 small crates) took 14–45 s through sccache-dist against
1.4–1.6 s with no compiler wrapper on the lane, just as slow on a second pass, so each
compile through the wrapper carries a fixed delay far larger than the compile itself
(`docs/rig-capacity.md`, "A real batch on a build lane"). The delay is the input
upload: for a Rust compile the sccache client packages every library in the
`-L dependency=` directory (`target/release/deps`, 225 rlibs, 336 MB), not only the
crates the compile loads. One probe library compile (0.1 s with plain `rustc`) took
4.4 s distributed, sent 121.9 MB (the gzip size of that whole directory) and spent 2.3 s
of client CPU compressing it. The cost grows with the dependency directory, not with the
crate, and is paid again on every compile, which likely explains the `-p dom` result
too (not measured). The probe therefore runs without the wrapper, on the lane's own cores; the
development box's sccache config stays untouched.

**Shared data on the NAS.** Each lane bind-mounts one NAS directory at
`/srv/lane-shared` (`NEV_LANE_SHARED`, also the npm cache). Login shells get the npm cache
from `~/.lane-env`; the Dagu job's `prepare` and `run` steps (`bench/dagu/job.yaml`) read
only `npm_config_cache` from `~/.lane-shared-env`, and `bench/oracle-install.sh` runs
`npm ci --prefer-offline`, so a package is downloaded from the registry once and every
lane and job after that installs it from the share. The NAS is fast from the lanes
(about 92 MB/s write, 114 MB/s read) but slow for small files (~38 ms each), so `target/`
stays local. `bench/lane-apply.sh` links each worktree into it on every run:

| in the worktree | on the share |
|---|---|
| `oracle/suites/checkouts/<suite>` | `suites/<suite>-<pinned commit>/` |
| `webgpu-cts/cts` | `suites/webgpu-cts/` |
| `corpus/<name>` for every `*ground-truth` cache the code names | `ground-truth/<name>/` |
| `oracle/suites/results` | `archive/suite-results/<worktree>/` |
| `webgpu-cts/results` | `archive/webgpu-cts-results/<worktree>/` |
| (the command's output, copied) | `archive/gate-logs/<worktree>/<time>-<guest>-<commit>.log` |
| (one line per model batch, `bench/ledger.py`) | `ledger/<worktree>.jsonl` |
| (a Dagu job's full logs, `bench/dagu/job.yaml`, scrubbed) | `archive/job-logs/<job name>/<Dagu run id>/` |

The last row comes from the Dagu job workflow, not `bench/lane-apply.sh`: it copies each job's stage logs,
`state.json`, the review, each pi session's prompt and `events.jsonl` (its turns and
tokens) and `oracle-install.log` there when a run ends,
whether it passed, failed or was stopped, with an `archive.txt` listing every file and its
line count; the step log prints the path. Credentials, URLs, addresses and token-shaped
values are replaced on the way (`bench/dagu/job.yaml`, "handler_on"). Neither
`archive/gate-logs/` nor `archive/job-logs/` is pruned: both grow until removed by hand
(`find <share>/archive/job-logs -mindepth 2 -maxdepth 2 -type d -mtime +30 -exec rm -rf
{} +` drops job runs older than 30 days).

The ledger is what a landing's announcement to the community Discord reads for its
numbers (`docs/web-features-map.md` → "Announcements"): each batch's start and end time,
lane CPU-seconds, and per model the candidates, compiled count, best score, tokens and
draw seconds, with no host names or addresses.

An existing real directory is left alone. A ground-truth cache written by one branch is
read by every other, so a recipe change should regenerate its cache. The export is open
to the whole subnet: nothing secret goes on it. Measured: fetching the suites into an
empty share took 6 min 19 s; a second lane then found them in 13 s.

**Reaching the model gateway.** A lane reaches the gateway at its rig-network address,
not the development box's loopback forward. Each lane keeps that address in
`~lane/.lane-model-env` as `NEV_LANE_MODEL_URL` (written by `bench/lane-provision.sh`
from `LANE_MODEL_URL`), and `bench/lane-apply.sh` swaps it into the worktree `.env`'s
`NEV_MODEL_URL` line on every run, leaving the key as copied. The address never enters
the repository (`bench/lane-apply-test.sh`).

**No credentials left behind.** The worktree's `.env` (the gateway key, usually the forge
token) is copied into the lane copy for the run only. When the run ends -- success,
failure or interrupt -- `bench/lane-apply.sh` removes the lane copy's `.env`, its
temporary rewrite and the payload's `.env`, and `bench/remote-lane.sh` removes the same
three over its own ssh, which covers a run whose connection dropped first. Nothing else in
the lane copy is touched. `NEV_LANE_KEEP_ENV=1` keeps them, and
`bench/lane-writer-launch.sh` sets it, because the writer it starts outlives the launch
and its model batches read that `.env`. Two known gaps: an interrupted run exits 130
(SIGINT) or 143 (SIGTERM) locally but does not stop the command on the lane, which runs
until it finishes or next writes to the dropped connection; and the git mirror's clone
URL can carry the forge token, which then shows in the lane's process list as an argument
to `lane-apply.sh`, a copy of that secret outside `.env`.

**Running writers on a lane.** No script here names an agent harness; the harness is a
declaration. `tracks.json` → `specWatch.lanes.tool` names it (`TRACKS_TOOL` in the
environment is the fallback, and a launch with neither is refused), and pi-tracks'
writer command prepares that harness on the lane and starts the session.
`bench/lane-provision.sh`'s writer-host section (enabled by `LANE_PI_TRACKS_URL`) clones
pi-tracks on the guest and prints `changed:`/`unchanged:` per step, so a rerun is
idempotent. What a particular harness needs installed ahead of time is a command the
operator supplies as `LANE_WRITER_SETUP`, run as `lane` with `LANE_WRITER_TOOL` and
`LANE_WRITER_DIRS` (the lane roots and mirrors, relative to the home directory) in its
environment; with none set the step does nothing. A harness login is a person's step
over ssh as `lane`. `bench/lane-writer-launch.sh <worktree> <lane-index> <model>
<brief-file>` syncs a worktree, checks out its branch, pulls pi-tracks and runs
`"$PI_TRACKS_HOME/bin/writer-exec.sh" --tool <name> --model <model> --brief-file <path>`
there (`PI_TRACKS_HOME` defaults to `$HOME/.local/share/pi-tracks`), refusing over a
dead writer's unfetched commits and failing when that command exits nonzero; the brief
lands in `~/.cache/lane-writer-briefs/` on the lane, outside the worktree.
`bench/lane-writer-fetch.sh` brings commits back fast-forward only. `tracks.json` →
`specWatch.lanes` declares both. `bench/lane-nas.sh` decides a guest is busy from the
process names in `LANE_BUSY_COMMANDS` (default `cargo|rustc|node`); add the harness's
process name when a writer runs under a different one. `crates/harness/tests/no_harness_name.rs`
fails the workspace tests when a `bench/` script, `tracks.json` (other than the `tool`
entry) or `.gitignore` names a harness. `bench/lane-writer-launch-test.sh` tests the
launcher against a stand-in writer command; a real launch on a lane is unverified until
the pi-tracks writer command is on its `main`.

**Dagu on a lane.** A job (`specs/job-submit-land.md`) runs as one run of the Dagu workflow
`bench/dagu/job.yaml`, on the lane `NEV_DAGU_LANE` names (an index into
`bench/lane.local.json`, default 0; Dagu's coordinator is off, so one lane).
`bench/dagu-lane.sh` installs, starts, reaches and cleans up after it; its header lists the
actions and settings. Without `--lane` it takes the same lane, `NEV_DAGU_LANE` or 0,
unless `NEV_DAGU_LANE` is unset and `NEV_LANE_SSH` names a lane, which then wins.
`jobs/scripts/submit.sh` and `jobs/scripts/poll-and-land.sh` check
manifests and results with the `job-check` binary from `$JOBS_STATE_DIR/bin` (default
`~/.local/state/nev-jobs/bin`) on the development box; `jobs/scripts/install-job-check.sh`
builds it on a lane (`bench/on-lane.sh --lane-only --pull`) and installs it there, and is
rerun after a change to `jobs/`.

- *The requirements check.* Before the `run` step's `job-run` starts the first stage that is
  not `passed`, on every start and every resume, it checks that the lane has what the
  manifest is about to need -- `git`, `python3`, `node` and `kill` resolvable, `cargo` and
  the oracle's Playwright browsers when a writer is in the stage list, a reachable model
  gateway when a stage runs a pi session, 20 GiB free on the workdir's filesystem, the
  shared archive directory, `NEV_ON_LANE` when a stage calls `bench/on-lane.sh`,
  `bench/verify.sh` or `run-batch.sh`, and at least 600 s left of the wall-clock limit
  across resumes (`jobs/src/preflight.rs`). One unmet requirement prints every reason to
  stderr, records them in state.json's `preflight` field, runs no stage and exits 9 --
  distinct from 7 (publish failed) and 8 (stopped), and never resent by
  `jobs/scripts/poll-and-land.sh` the way 7 and 3 are, since a missing requirement is still
  missing a minute later. Run the same check by hand with `job-check preflight
  <manifest> [--workdir <dir>] [--state <dir>]`, exit 0 when every requirement is met, 9
  otherwise; `--workdir` defaults to the current directory and `{tools}` in a stage's
  command resolves under its `jobs/scripts`. This check only asks what the lane itself can
  answer right now (a binary, a socket, a directory); whether a model or provider is
  currently available on the rig is the separate inventory check that follows it.

- *The inventory check.* Once the requirements check passes, the `run` step's `job-run
  --inventory ~/folknet/inventory.json` copies the lane's snapshot into the state
  directory (again on every resume), hands every stage `JOB_INVENTORY` (the copy) and
  `JOB_INVENTORY_LIVE` (the lane's own path), and checks the manifest against it and the
  declared file `tools/folknet.json` (the submitting checkout's `bench/folknet.json`). A
  model that is not ready, or a snapshot older than 15 minutes or missing, prints every
  reason, records them in state.json's `inventory` field, commits a `not-run` result
  locally (not published), runs no stage and exits 3, which
  `jobs/scripts/poll-and-land.sh` resends uncounted. A refusal that never clears by itself
  -- a model retired or not declared, a provider not declared or not allowed -- is
  reported the same way but commits a `failed` result and exits 10, which the poller
  records as `failed` without resending. The same check by hand: `job-check
  inventory <manifest> <snapshot> [--declared <file>]` (`docs/folknet-inventory.md` ->
  "The check before the first stage").

- *Installed as a plain binary.* The lanes have no container runtime (unprivileged LXC,
  no `docker` or `podman`, no passwordless `sudo`; rootless podman would also need
  `/dev/fuse` and `/dev/net/tun` added to the guest's config on its node), so Dagu runs as
  the lane user from `~/dagu`: `bench/dagu-lane.sh --lane N up` installs the pinned
  release (its sha256 checked against the pin and the release's own `checksums.txt`),
  pushes the gateway key, starts it and checks it. `stop`, then deleting `~/dagu`, removes
  it. `autostart on` adds a lane-user `@reboot` crontab line, which has brought Dagu back
  after a guest restart.
- *Reached only through ssh.* Dagu binds to the lane's loopback (`NEV_DAGU_PORT`, default
  18431) with basic auth. The password is generated on the lane, kept in
  `~/dagu/home/config.yaml` (mode 600) and read over ssh when needed, never stored on the
  development box. `tunnel` holds a port forward open for the web UI, `/api/v1` and
  `/mcp`. The MCP server is the `/mcp` route of `dagu start-all` (Dagu 2.x has no
  `dagu mcp` command). The one MCP client is the curl-based one in `bench/dagu-lane.sh`
  (`run`, `stop-run`, `watch`).
- *Secrets never go through run parameters.* Dagu keeps a run's parameters in its run
  records on disk, so `run` refuses a parameter whose name contains KEY, TOKEN, SECRET,
  PASSWORD, PASSWD or CREDENTIAL. The gateway key goes by `key` over ssh on stdin to
  `~/dagu/home/model-key` (mode 600); `start` exports it as `NEV_MODEL_API_KEY` with
  `DAGU_ENV_PASSTHROUGH=NEV_MODEL_API_KEY`, so every step sees that one variable and it
  lands in no run record or log. Steps start from a non-login shell and inherit nothing
  from `~/.lane-env`: the gateway address comes from `~/.lane-model-env`, the npm cache
  and the share from `~/.lane-shared-env` (one variable at a time), and the forge through
  the lane mirror's origin URL, which carries the credential; git messages from forge
  commands pass through a filter that replaces anything shaped like a URL.
- *How Dagu 2.17.2 behaves.* Workflow keys are snake_case (`retry_policy`,
  `hist_retention_days`); camelCase is refused. One `dagu_execute` call waits at most
  300 s, so `run` follows a longer run by reading its status every 15 s until it ends or
  `NEV_DAGU_WAIT` seconds pass. Dagu records no exit code for a step: `watch --once
  --step <step>` takes 0 for a succeeded step and otherwise the N of the step's error text
  `exit status N`, and `-` when there is none (a step killed by a signal). The exit
  handler (`handler_on: exit`) runs after every run, succeeded, failed or aborted, but not
  after a run cut off by a lane restart or a KILL of Dagu.
- *The queue.* A workflow's own `max_active_runs` above 1 is ignored for local queues; the
  limit is a global queue in `~/dagu/home/config.yaml` (`queues.config[].max_concurrency`,
  written by `set_queue()`: queue `jobs`, 2 at once), read only when the scheduler starts,
  so a change needs `stop` then `start`. Only enqueued runs pass through it (`dagu_execute`
  `action=start` runs at once), so `run` enqueues any workflow file with a top-level
  `queue:` key. A second submission while one is building passes `--no-sync`, so the source
  copy under the first run's build is not replaced.
- *What it keeps.* Run history 7 days (`hist_retention_days`, set in
  `~/dagu/home/base.yaml` by `set_retention()` and in `job.yaml`): the forge holds each
  job's result. A job's directory `~/dagu/jobs-work/<name>/` stays until the job passes,
  so a new run with the same parameters resumes it, and after a pass it stays too when
  its logs did not reach the NAS share (step `cleanup` removes it only once the archive
  wrote its marker `archived-<Dagu run id>` there, and says in its log when it kept it);
  `clean-jobs <name>` removes one by hand. The job tools are built from the copy of the sources `run` sends to `~/dagu/src`
  (`sync_source()`), not from the job's base commit.
- *The mirror keeps deleted job refs.* The lane mirror's fetch (refspec `+refs/*:refs/*`,
  no prune) copies every `job/*` and `result/*` branch from the forge and keeps it after
  the forge deletes it. `prune-mirror` fetches and prunes those two namespaces only;
  `bench/dagu/fixture-base.sh --delete` runs it.

## Landing approved spec tracks from a tick

`bench/land.sh <spec-name> <spec-sha> <branch-tip>` lands an approved track without a
person; the spec-watch tick calls it (`tracks.json` → `specWatch.landScript`). Exit codes:
0 landed, 1 not landed (reason on one line), 2 rebase conflict outside `handoff.md`. The
contract: `~/.claude/docs/working-modes.md` → "Let a tick land tracks".

```mermaid
flowchart TD
  start["tip and spec sha match<br/>what was reviewed?"] -- no --> refuse["refuse"]
  start -- yes --> scope["diff inside the spec's touches,<br/>no protected file?"]
  scope -- no --> refuse
  scope -- yes --> rebase["rebase onto main<br/>(handoff.md → main's roster)"]
  rebase --> gate["verify + landChecks<br/>(tests, 227 floor, per-function floors)"]
  gate -- fail --> refuse
  gate -- pass --> merge["fast-forward main<br/>(retry if main moved, max 3)"]
  merge --> after["tag spec/name/sha, remove worktree<br/>and branch, update roster"]
```

- **Protected files.** A diff touching `bench/`, `tracks.json`, `.github/`, `specs/`, or at
  any depth `.cargo/`, `rust-toolchain*`, `.env*`, `.gitattributes`, `.gitmodules` or
  `*.floor.json` (a generated function's score floor) is refused whatever `touches` says,
  since those change the gate itself. So is a spec whose
  `touches` matches everything or reaches those paths. Case-insensitive, no rename
  detection. Such a track waits for a person's `/land`.
- **What the gate still trusts.** The track's own crates, tests, `build.rs`, `oracle/` and
  `Cargo.toml`/`Cargo.lock`. The gate catches regressions and honest mistakes, not a track
  built to fool it.
- **The checks.** `verify` (`bench/verify.sh`: the tests, then every generated function
  scored against its own corpus, each at or above its floor in `<name>.floor.json`), then
  `landChecks`: `bench/fragment-floor.sh` (`fragment_check --random` must pass 227), which
  runs through `bench/on-lane.sh` (`--lane-only`, like `verify`, when `NEV_LANE_STRICT=1`;
  see "Build lane"). The four hand-written floors that `bench/seam-floor.sh`
  used to hold (floats, stacking order, relative positioning, block height) are floor files
  like the rest, and that script and its checker are gone. Floors are raised after a
  landing, never lowered by a track: a diff that touches a `*.floor.json` is refused.
- **Safety.** Pinned SHAs re-checked right before a `--ff-only` merge; a lock on the git
  directory; signals kill the gate's process tree and restore the branch. A
  `land-pending/<name>` marker makes a rerun finish anything owed after a merge. Nothing
  is pushed.
- **`handoff.md`.** A landing restores `main`'s roster on the branch before the gate, so a
  track's working note never overwrites the roster.
- **Install after a landing.** If `oracle/package*.json` changed or a crate was added, only
  `bench/oracle-install.sh` runs, staged and swapped in on success.
- **Testing.** `bench/land-test.sh` drives the script through about 130 assertions in a
  throwaway clone. `bench/land.sh --check ...` runs every condition and stops before the
  merge.

## Training limits

Full fine-tuning is out of reach on 24 GB Ampere cards; LoRA or QLoRA on a 30B-class model
across eight GPUs is feasible, so any supervised step is a LoRA.

## Sharding without a broker

Case generation is deterministic from a seed, so worker *i* of *n* takes seeds
`i, i+n, i+2n, …`: no queue, no coordinator, and any worker can restart without losing or
duplicating work. Output is content-addressed and written once; ground truth is immutable
per engine version, so there is nothing to invalidate.

## Storage

About 10 KB per document for three engines, so ten million cases is about 100 GB. The
style subset dominates and is redundant; compress it, sample it, or store it as a delta.

## Virtual machine template

One image, duplicated: WebKit's 188 system packages, pinned Playwright browsers, Rust with
sccache, the repository. Workers are stateless. Pin one VM per NUMA node rather than one
large VM per host.

## Engine versions pin the ground truth

Ground truth is immutable only for a given set of browser versions, so every dump records
`engine_version` (`docs/dump-format.md` → "Dump") and a corpus is keyed by that version
set. A browser update invalidates only what it touches, never silently.

## Rig change log

Every change tried on the rig's hardware, worked or not, newest last. Hosts by role and
class only. Before a change, and again after each change that works,
`bench/rig-snapshot.sh snapshot <label>` records the serving configs of every role (proxy
host HAProxy, gateway container config and key database, each ARM node's units and
TabbyAPI configs, the four-card VM's Proxmox config and units) on the host itself and in
the local rig notes; `restore <label>` puts them back. The label named in an entry holds
the working state after it. Snapshots contain the gateway's master key and never go into
git or onto the NAS.

**2026-09-24**

- *Gateway key for self-hosted writers.* A new virtual key on the gateway, alias
  `nev-folknet-writer`, allowed only `qwen3.8-27b-long` and `qwen3.8-27b-fast`, so
  writer traffic is accounted apart from batch traffic and cannot reach any other alias.
  No serving config changed. The key itself lives outside every repository, as
  `FOLKNET_WRITER_API_KEY` in `$HOME/.config/folknet-writer/env` (mode 600) on the
  development box. First used by stage 0 of `docs/folknet-writer-design.md`
  (`docs/writer-harness-spike.md` → "Rerun through Pi"). Worked.
- *Snapshot and restore.* `bench/rig-snapshot.sh` written; `known-good-2026-09-24`
  taken before any serving change. `restore known-good-2026-09-24 haproxy` run once as the
  round-trip test: config check passed, HAProxy reloaded, all four ARM replicas still up.
  Worked.
- *Four-card VM.* A new VM on a Proxmox node, given the four 3090s of a stopped tenant
  test VM (same `hostpci` entries, q35, OVMF, 256 GB 64-bit PCI hole), 32 cores, 160 GB
  RAM, a 120 GB OS disk, DHCP on the main LAN (`bench/gpu-vm/create-vm.sh`). The tenant
  VM's disks were not touched. Worked; four 3090s visible.
- *Provisioning, first two attempts failed.* First: the fresh cloud image was still
  running cloud-init's package upgrade, which held the dpkg lock and then rebooted the
  guest. Second: the rig's apt cache answers `403 CONNECT denied` for HTTPS repositories,
  so docker's and NVIDIA's apt repositories could not be reached through it. Fixed in
  `bench/gpu-vm/provision.sh`: wait for cloud-init and the dpkg lock, send HTTPS
  repositories direct, take docker from the Ubuntu archive (`docker.io`, which the cache
  serves). Third run worked: driver 595 (open kernel modules) loaded without a reboot,
  `nvidia-smi` shows four 3090s, docker has the NVIDIA runtime.
- *Model choice for the four cards, by size (no download attempted).*
  - `qwen3.8-flash-next`, EXL3 4.05 bpw (the ARM nodes' checkpoint): 101 GB on disk, of
    which 39 GB is the hashed n-gram table that exllamav3 keeps in host RAM or on disk,
    so about 62 GB of weights would sit on the cards. That the checkpoint does not fit
    96 GB was assumed, never tested. A 3.05 bpw branch is also published (85 GB on disk,
    32.6 GB of it n-gram table). Not pursued: the target moved to a different model family
    for variety.
  - DeepSeek-V4-Flash (the July release, 284B parameters, 13B active): 167 GB native
    (MXFP4 experts). Mainline llama.cpp with CUDA serves it on sm86 from GGUF. The
    smallest quant its quantizer recommends (3-bit, 104 GB) does not fit 96 GB of VRAM;
    2-bit quants (87–91 GB) fit with only a few GB left for cache and compute buffers,
    and re-quantizing the native 4-bit experts to 2 bits costs a lot of accuracy
    (published KL divergence around 0.4 at 2 bits against 0.01 at 4 bits). Download
    straight from Hugging Face inside the VM measured about 250 KB/s. Not tried.
  - DeepSeek-V4.1-Flash is a different, larger model, not a revision of the one above: a
    new architecture (`deepseek_v41`, 384 experts over 40 layers, n-gram memory tables),
    510 GB native. It does not fit the four cards. The first reading, that it fits no
    pair of ARM nodes either, was wrong: it looked only at GGUF quants (smallest about
    260 GB) and missed a published EXL3 recipe for exactly two GB10s — a 2.9 bpw EXL3
    checkpoint (~197 GiB, about 99.5 GiB of weights per node at tensor-parallel 2) plus
    the model's unquantized n-gram ("Engram") tables, which stay on disk and are read
    per lookup rather than held in memory (two native shards, ~190 GiB), served by a
    vLLM build that carries the `deepseek_v41` architecture. That recipe is what the
    ARM pair uses (entries below).
  - qwen3.8-27b, W4A16 AutoRound with DFlash2 speculative decoding, the eight-card
    host's `-fast` recipe: about 22 GB per card, one worker per card. Chosen.
- *qwen3.8-27b on the four cards.* The eight-card host's image and prepared checkpoint
  (24 GB) were streamed to the VM read-only (`tar` and `docker save` piped over ssh,
  nothing on that host changed). One worker per card came up in about 6.5 minutes from
  cold; all four report healthy and answer chat completions. Worked; snapshot
  `vm-qwen27b-workers-up`.
- *Proxy and gateway for the VM.* A new HAProxy frontend and backend on the proxy host
  (the existing backends untouched; backup `haproxy.cfg.pre-vm-qwen27b-20260924` beside
  the config) and a gateway alias `qwen3.8-27b-fast-2` (backup
  `config.yaml.pre-qwen27b-vm-20260924`), granted to the project's key only. The first
  gateway attempt failed with "Missing credentials": an OpenAI-style entry needs some
  `api_key` even when the upstream checks none; `sk-none`, as the flash-next entry uses,
  fixed it. Worked: the alias is listed and returns code through the gateway; snapshot
  `vm-qwen27b-fast-2-alias`.
- *Measurement batch at `qwen3.8-27b-fast-2`* on a build lane: one 32-candidate
  absolute-positioning batch, 2/32 compiled, best 74/108 (`docs/model-first-cost.md`, row
  `vm-batch 1`). Stopping the loop after batch 1 with `pkill -f` on the shared lane also
  killed another track's run of the same binary in its own lane worktree (a mixed-model
  batch at 32 of 64 draws). On a shared lane, stop a run by the pid whose
  `/proc/<pid>/cwd` is this worktree's lane copy, never by name.
- *Checkpoint fetch for an ARM pair.* `bench/sparks/hf-to-nas.sh` (a runtime bundle and
  oneshot unit on the downloader host, reusing that host's multi-WAN scripts) started
  DeepSeek-V4-Flash (July release, 167 GB); stopped after 14 GB when the ARM target
  changed to DeepSeek-V4.1-Flash. The partial staging directory is left on the NAS. Then
  both parts of the V4.1 recipe, pinned by revision: the EXL3 2.9 bpw checkpoint
  (42 files, 210.7 GB) and the two Engram shards plus index and config of the native
  checkpoint (203.1 GB), both through the multi-WAN downloader at about 130 MB/s
  combined. Snapshot before: `pre-deepseek-sparks`.
- *Which two ARM nodes.* LLDP shows each node's first 200G port on the switch and its
  second port cabled straight to one other node, in two cabled pairs, each crossing
  the old GLM pairs (which ran over the switch). The DeepSeek pair is one of the cabled
  pairs; the other cabled pair keeps flash-next. Its tensor-parallel traffic is set to
  run over the switch port with the RoCE settings the GLM pairs used, because the direct
  cable carries no addresses today; moving it onto the cable means addressing that
  port on both nodes (not done).
- *Staging the pair (nothing serving yet).* The recipe is cloned at a pinned commit on
  the head node, its `.env` written from `bench/sparks/dsv41/env.template` (weights from
  local disk, no Hugging Face download, `AUTO_DOWNLOAD=0`), an ssh alias for the worker
  added to the head's admin account (`bench/sparks/dsv41/ssh-config-worker`; previous
  config kept as `config.pre-dsv41`), `/srv/dsv41` created on both nodes, and the
  recipe image pulled on both (the `2.9bpw` tag, 22.5 GB; digest in the local rig
  notes). Both nodes still serve flash-next. Snapshot `dsv41-staged`.
- *Throughput sweep of `qwen3.8-27b-fast-2`* (`bench/rig_throughput.py`) on a build lane:
  failed twice before sending a request — the lane's Python has no `requests` module,
  and `python3 -m venv` fails there without the `python3.12-venv` package. Not run.
- *Four-card VM reboot.* All four workers healthy three minutes after `reboot`, same
  DHCP address, HAProxy backends back UP without intervention. Worked.
- *Second measurement batch at `qwen3.8-27b-fast-2`*, after the prompt fixes, run with
  `bench/absolute-mixed-batch.sh` on a build lane: 8/32 compiled, best 82/108, 233 s for
  the batch (`docs/model-first-cost.md`, row `vm-batch 2`). The run ended by itself
  (`GENERA_MAX_BATCHES=1`); nothing needed stopping. No serving change.
- *Throughput sweep of `qwen3.8-27b-fast-2`, second attempt*, run from the dev box
  (which has `requests`) through the gateway, with the prompt generated on a build lane:
  worked, 819 output tok/s at 16 in flight (`docs/rig-capacity.md` → "Measured — the
  four-card VM"). No serving change.
- *Abandoned DeepSeek-V4-Flash download removed* from the downloader host: the 14 GB
  partial staging directory on the NAS, its oneshot unit and runtime bundle. Worked; the
  two V4.1 units and their published trees are untouched.
- *DeepSeek-V4.1-Flash weights to local disk on the ARM pair.* Both NAS downloads had
  finished and published. `rsync -a` from the NAS mount to local NVMe on both nodes, the
  EXL3 checkpoint (39 shards, 197 GB) to each, the two Engram shards with index and
  config (190 GB) to the head's engram source directory and to the worker's engram
  directory. The worker gets no second engram tree: the recipe's `start.sh` builds the
  slim engram set on the head and rsyncs it to the worker, and with the NAS mtimes kept,
  that rsync finds the shards already in place. About 100 MB/s per node, both done in
  about 55 minutes. Both nodes kept serving flash-next throughout, all backends UP
  after. Worked; no serving change, so no new snapshot (`dsv41-staged` still holds the
  serving state).
- *Two ARM nodes out of the flash-next pool.* Snapshot before: `pre-dsv41-pair-move`.
  Their two server lines were removed from the flash-next HAProxy backend (backup
  `haproxy.cfg.pre-dsv41-pair-20260924` beside the config; config check, reload; the
  other two replicas UP), then their flash-next units disabled and stopped; unit,
  config and checkpoint stay on disk. Nothing was drawing from the alias at the time.
  The alias now takes 16 requests in flight (two replicas × 8), not 32. Worked; snapshot
  `flashnext-two-replicas`.
- *DeepSeek-V4.1-Flash on the pair.* The published recipe image carries the recipe's
  own build stamp from four commits before the pinned one, so the pinned `start.sh`
  would have compiled the image on the node. The only image-side change in between is
  one Python patch, which that `start.sh` bind-mounts into both containers anyway, so the
  unit runs the published image as-is (`SKIP_BUILD=1`, reason in the unit file). Unit
  installed on the head and started: preflight passed (memory headroom 117.6 GiB
  available per node against 111.5 needed, RoCE GID accepted), the weight sync to the
  worker found everything already in place and took 3 s, weights loaded and warmed up,
  API healthy about 7.5 minutes after the start; the README's `17*19` request returned
  `323`. The recipe's own warmup reports one sampler-kernel variant not pre-compiled
  ("may JIT mid-serve"); harmless so far. Worked; snapshot `dsv41-pair-serving`.
- *Proxy and gateway for the pair.* A new HAProxy frontend and backend on the proxy
  host, one server, `maxconn 2` (backup `haproxy.cfg.pre-dsv41-backend-20260924`), and
  a gateway alias `deepseek-v4.1-flash` → the recipe's served model name, `api_key:
  sk-none` (backup `config.yaml.pre-dsv41-20260924`), granted to the project's key only.
  Worked: the alias is listed through the gateway and a genera-shaped chat completion
  (thinking off) returned a one-function Rust code block in 2.7 s with no reasoning in
  `content`. Snapshot `dsv41-alias-serving`.
- *First measuring batch at `deepseek-v4.1-flash`* on a build lane, 8 candidates at 2 in
  flight: 4/8 compiled, best 82/108, 1,731 s (`docs/model-first-cost.md`, row
  `dsv-batch 1`). The first launch failed before drawing anything: the lane's shared git
  mirror refused a ref update (`cannot lock ref 'refs/heads/main'`) because another run
  had moved `main` at the same moment; a plain retry worked. No serving change.

**2026-09-25**

- *Dead GLM aliases removed from the gateway.* `glm-5.3-flash` and `glm-5.3-flash-exl3`
  were still listed by `/v1/models` three days after their backends stopped, so a caller
  could pick one and get a 503 ("No server is available to handle this request"), as a
  job on another track did. Snapshot before: `pre-glm-alias-removal`. Both model entries
  deleted from the gateway config (backup `config.yaml.pre-glm-removal-20260925` beside
  it; no fallback named them) and the gateway service restarted, live again in 18 s. The
  two names were then dropped from the allowlists of the three virtual keys that carried
  them, each keeping its other models. Worked: `/v1/models` through the dev-box tunnel
  lists neither name; one-token completions return 200 for `qwen3.8-27b-fast`,
  `deepseek-v4.1-flash`, `qwen3.8-27b-fast-2` and `qwen3.8-flash-next`; a request for
  `glm-5.3-flash` is refused with 403. Snapshot `glm-aliases-removed`. The site file the
  gateway config is generated from still carries the GLM entries, so a regeneration
  would bring them back.