Infrastructure #

The machines the generation loop runs on, and how work reaches them. The loop itself is in docs/generation-loop.md; measured rates are in docs/rig-capacity.md. Hardware is named by role and class, never by hostname, address or account — this repository is assumed public. Real names live in the gitignored bench/*.local.json and the local rig notes.

In brief #

• The loop. One engine function per spec algorithm (for example the CSS 2.1 width calculation), written by a model, checked against cached browser results.
• Sampling tier. GPU cards running mid-sized models write N versions of a function from the spec text, the engine's Rust interfaces and the currently failing cases. Many cheap tries beat one careful one, because the browsers say which tries are right.
• Capacity tier. When every version fails, a larger model gets the spec, the best failing versions and the exact per-box geometry differences.
• Verification tier. CPU servers compile candidates in one batch and run each inside the engine over the cached corpus, under a millisecond per page. The same tier builds that cache by running the three browsers once per page, and runs conformance suites.
• Outcomes. Keep the passing version, sample again with the new failures, or escalate. What is kept is ordinary Rust with a provenance record; no model runs in the shipped engine.

Environment variables are named WEBENGINE_*. Older records use NEZE_* (docs/rename-to-bez.md), NEV_* (docs/rename-to-neze.md) or GENERA_* (docs/rename-to-nev.md); nothing reads any of them any more.

Detail #

The full cycle #

flowchart TD
  browsers["Chromium, Firefox, WebKit<br/>once per page (CPU)"] --> cache["cached ground truth<br/>three-engine vote"]
  spec["spec text<br/>one algorithm"] --> sample["SAMPLE (GPU pool)<br/>N versions of one function"]
  sample --> verify["VERIFY (CPU)<br/>dedupe, batch compile,<br/>run over the corpus"]
  cache --> verify
  verify -- "all pass" --> accept["commit Rust source<br/>+ provenance record"]
  verify -- "some fail" --> sample
  verify -- "all fail repeatedly" --> escalate["ESCALATE<br/>larger model + geometry diffs"]
  escalate --> verify

Nothing is accepted on a model's word: escalated output goes through the same check. Writing candidates is the binding constraint; verification has a large margin over it.

What comes out, and how the engine uses it #

The cycle outputs ordinary Rust source, committed next to the hand-written code it plugs into, with spec steps as comments and a provenance record (roadmap.md → "Provenance is a first-class artifact"). The engine compiles it like any other code; the GPU and CPU fleets exist only at build time.

Provenance drives maintenance. When a spec section or a definition it uses changes, the provenance index finds exactly the functions to regenerate (roadmap.md → "A specification changes"); a browser update re-dumps the ground truth instead. The same index is the bill of materials for a configurable build (roadmap.md → "Phase 4").

Measured baseline #

One oracle process, three engines, 235 documents including browser startup: 17.9 s, about 13 documents per second per engine, using about two cores. Scaled to a few hundred cores that would be about 3,000 documents per second, a ten-million-case corpus in about an hour — but per-core throughput on the contended 16-core development box fell from 1.48 to 0.48 docs/s/core between 1 and 16 workers (docs/rig-capacity.md → "Oracle corpus building"), so the figure on dedicated hardware is still to be measured. GPU-side figures: docs/rig-capacity.md → "GPU tier".

Three tiers #

Sampling tier — high memory bandwidth GPUs. Bulk best-of-N generation, throughput-bound. Decode speed tracks active parameters, so a mixture-of-experts model with a small active fraction gives several times the candidates of a dense model. Ampere cards have no fp8, so quantization is 4-bit weight-only.

Capacity tier — large unified-memory accelerators. About a third of the bandwidth, several times the memory: for the few hard calls and any teacher model. About 100B total with ~12B active is the realistic ceiling.

Verification tier — many CPU cores. Corpus building, candidate compilation and the inner loop. It must never spawn a browser per case.

As discovered #

Every host advertises itself over mDNS and answers an HTTP inventory; a collector aggregates it. What that shows, by role:

• The model gateway is a LiteLLM instance fronting one alias per model pool. The aliases, their pools, how many requests each takes at once, and which are retired are declared in bench/folknet.json (the two GLM-5.3 Flash aliases, without backends since 2026-09-22, were removed from the gateway on 2026-09-25); whether each one answers right now is in each lane's snapshot, which a probe refreshes every 5 minutes and bench/folknet-status.sh [--lane N] prints (status, how long it has held it, last answer, snapshot age). Design: docs/folknet-inventory.md. Virtual keys carry per-key allowlists, so a new alias must be granted to the caller's key. Each pool's HAProxy reports liveness, not readiness: a /v1/models answer does not prove a completion will return.
• Four ARM unified-memory nodes (DGX Spark class) — aarch64, one GB10 GPU, 20 cores, 121 GB RAM, 3.7 TB NVMe each. They served GLM-5.3 Flash as two independent two-node pairs until 2026-09-22. Qwen3.8-Flash-Next did not run under any vLLM build on the rig (the architecture was too new for the rig's EXL3 fork, and upstream has no EXL3 backend), so it serves through the checkpoint's own engine (TabbyAPI) instead, one replica per node, about 75 GB plus ~45 GB of page cache each. Two of the four now serve DeepSeek-V4.1-Flash. Full account: docs/rig-capacity.md → "Attempted — Qwen3.8-Flash-Next".
• The eight-card host — eight 24 GB consumer cards on a 64-core EPYC with ~500 GB RAM, bare Linux, six serving containers using every card (four behind -fast, two behind -long). The closest thing to the sampling tier as designed. No spare GPU.
• The verification tier — ten Proxmox 9.x nodes (EPYC 7742, 7B13, 7K62 and 7452; 32 to 64 physical cores, 125 GB to ~500 GB RAM, 1–2 TB NVMe), shared with tenant VMs; five pass thirteen consumer cards through to tenant machines. A VM sized for browser builds sits on the GLM-pool proxy host (open: its core count and whether it should be running). Free capacity is host by host.
• A downloader host pulls checkpoint shards in parallel across the site's four WAN uplinks onto the NAS, verifies them and publishes a pinned revision for nodes to sync.
• Access. Root with the jump host's key works on every physical host; the ARM nodes use their own admin key. The gateway's own guest refuses, and is not a compute host.

Where each piece runs #

Every role is a guest or a container; nothing is installed on a Proxmox node's own OS.

flowchart LR
  dev["development box<br/>16 cores, shared"] -- "ssh via jump host" --> lanes["2 build lanes<br/>(LXC, 96 threads each)"]
  dev -- "port forward" --> gw["model gateway<br/>(LiteLLM)"]
  lanes --> gw
  gw --> fast["eight-card host<br/>-fast, -long"]
  gw --> vm["four-card VM<br/>-fast-2"]
  gw --> fn["2 ARM nodes<br/>flash-next"]
  gw --> ds["2 ARM nodes<br/>deepseek pair"]
  lanes --> helpers["compile helpers<br/>(sccache-dist)"]

• Development box. No GPU, no rig role; holds the port forwards to the gateway. Each forward is an enabled systemd user unit (rig-gateway-tunnel, rig-forge-tunnel, rig-lightwall-tunnel, rig-tunnel, webengine-job-events-tunnel) with Restart=always every 10 s; user lingering is on, so the units start at boot with no login.
• Jump host. A VM on one node; the only route in. Nothing of this project on it.
• Build lanes. One unprivileged LXC on each of two nodes (bench/lane-create-lxc.sh, bench/lane-provision.sh), started at node boot. A gate running when a lane stops fails.
• Compile helpers and scheduler. Four more containers plus builders on the two lanes; scheduler, redis cache and a loopback CONNECT proxy on one lane (bench/lane-dist.sh).
• Eight-card host. Docker containers under a templated unit folknet-qwen38-vllm-worker@<profile>, behind that host's own HAProxy on port 4000. Not edited by this project.
• ARM nodes, split two ways since 2026-09-24.
  • Two serve qwen3.8-flash-next, one systemd unit and container each on port 5000 (bench/sparks/), behind the proxy host's HAProxy (least connections, eight in flight per replica) — so the alias takes 16 in flight, not 32; a batch at concurrency 32 queues the rest. Either replica can stop alone.
  • Two, joined by a direct cable, serve deepseek-v4.1-flash as one tensor-parallel pair: one oneshot unit on the head node (bench/sparks/dsv41/folknet-dsv41-exl3-pair.service) starts both containers and returns once the API is healthy (about 7.5 minutes cold); stop and start only through that unit, never one node alone. HAProxy maxconn 2.
  • Units are stopped and started through a privileged container that chroots into the host, because the admin account has no sudo. GLM units are disabled on all four (since 2026-09-29; before that folknet-glm53-vllm was still enabled everywhere).
• Four-card VM (qwen3.8-27b-fast-2). Four 3090s borrowed from a stopped tenant VM on the same node (only one of the two may run; bench/gpu-vm/). Four units folknet-qwen38-vllm-worker@fast-{0..3} on ports 8101–8104, the eight-card host's -fast recipe, HAProxy maxconn 4 per worker (16 in flight). A separate alias because the -fast load balancer lives on the eight-card host.

Open. Which host carries the four-card VM behind qwen3.8-27b (the measured Qwen server), and how it is started and stopped; whether anything restarts the eight-card host's HAProxy or re-pulls its image; and the gateway config is generated from a site file outside the rig, which needs every hand-added alias too.

Sharing the hardware #

The Proxmox nodes are shared with other tenants. Three rules:

• Nothing of this project goes on a node's own OS, so a guest can be stopped and its hardware handed back without touching the node.
• GPU work runs in a VM with passthrough, never an LXC (an LXC would need the vendor driver on the node). CPU work runs in an unprivileged LXC, or in docker on non-Proxmox hosts.
• Stopping a project guest frees what it held, and a lane competes with tenants as an equal under contention. Nothing checks this; it is how the rig is laid out.

Limits today: the ARM replicas' HAProxy queues a request at most 30 minutes; power budgets are modelled (bench/capacity.py) but not enforced.

Not built yet: a per-role start/stop/drain wrapper; a drain (a replica's stop gives in-flight replies only 60 s); taking a backend out of HAProxy before stopping it; a narrow sudo rule for the ARM nodes' admin account.

Build lane #

Two unprivileged containers (96 logical CPUs — three quarters of a 64-core, 128-thread node — 96 GB, 200 GB disk), each a build lane and oracle host: the Rust toolchain, Node 22, the repository cloned from the rig's forge ("Where the repository is stored" below), Playwright's three browsers, and a PulseAudio null sink so a realtime AudioContext runs. The scripts are the record and are re-runnable: bench/lane-create-lxc.sh, bench/lane-provision.sh, bench/lane-nas.sh. When folknet is down, the development box can serve as a lane with its own forge and Kimi as the model: docs/local-lane.md.

Running a gate there.

sequenceDiagram
  participant W as worktree (dev box)
  participant L as lane
  participant F as rig forge
  W->>L: ssh: git bundle of unpushed commits + diff + untracked files
  L->>F: fetch the lane's mirror (only if needed, or older than 10 min)
  L->>L: lane-apply.sh: check out exact commit, apply diff, run command
  L-->>W: streamed output (and --pull directories)

bench/remote-lane.sh <worktree> <command...> does this. Each lane root keeps one bare mirror (<root>/.git-mirror) with one git worktree per track; it fetches only when the bundle cannot supply the commit or the last fetch is older than WEBENGINE_LANE_FETCH_MAX_AGE (600 s), because the public forge the mirrors first fetched from rate-limits. The lane address comes from WEBENGINE_LANE_SSH or bench/lane.local.json; the mirror URL from WEBENGINE_LANE_GIT_URL, that file, or the worktree's origin. Examples:

bench/remote-lane.sh . cargo test --workspace
bench/remote-lane.sh --pull corpus/absolute-ground-truth . \
  cargo run --release -p generation --bin absolute_ground_truth_cache

Why a bundle rather than rsync: uploads from the development box to the lane run at about 18.6 KB/s (downloads 435 KB/s), so rsyncing a worktree took ~315 s of a ~424 s run.

Where the repository is stored. Two forges, with different jobs:

forge	role	holds
the rig's Forgejo	intermediate storage, on the rig network	everything the lanes need: main, job branches job/<name> and their results result/<name> (tasks/job-submit-land.md)
tangled	persistent storage, public	main, the spec/<name>/<sha> landing tags, and every polled job run's result as the tag result/<name>/<run id> (jobs/scripts/poll-and-land.sh keep_result)

The Forgejo repository is a regular repository, not a mirror of tangled. The lanes clone and fetch from it: each lane root's .git-mirror and the lane user's clone fetch from http://<forge-user>:<forge-token>@<forgejo-address>/<owner>/<repo>.git, and bench/lane.local.json git_url (or WEBENGINE_LANE_GIT_URL) names the same URL. The user and token come from the primary checkout's .env (WEBENGINE_FORGE_USER, WEBENGINE_FORGE_TOKEN). The development box reaches Forgejo through a local port forward, as the remote rig (http://<forge-user>:<forge-token>@127.0.0.1:<forward-port>/<owner>/<repo>.git), which the job scripts use by default (JOBS_FORGE_REMOTE). origin fetches from tangled and has push URLs for both forges, so git push origin main updates both; job and result branches go to rig only and are deleted there after a landing. Each of those URLs carries the token: they live in .git/config and the gitignored files, never in a committed file, a log line or a message.

Outside sources. Every outside git repository is fetched from a pull mirror on the rig's Forgejo, never from GitHub or another upstream, so a lane or the development box neither meets GitHub's rate limits nor depends on it being up. Each mirror is a Forgejo pull mirror (all history and branches, so any pinned commit stays fetchable) owned by the same owner as the project's own Forgejo repository, and synced from upstream every 8 hours, the interval the forge's other mirrors use:

source	pinned in	mirror repository	size on the forge
https://github.com/web-platform-tests/wpt.git	oracle/suites/suites.json wpt	<owner>/web-platform-tests.wpt	3.0 GiB
https://github.com/KhronosGroup/WebGL.git	oracle/suites/suites.json webgl	<owner>/KhronosGroup.WebGL	43 MiB
https://github.com/gpuweb/cts.git	webgpu-cts/cts.json	<owner>/gpuweb.cts	623 MiB

The pin files keep the upstream URL as the source's identity; oracle/suites/mirror.mjs resolveSource() turns it into the mirror's URL, <forge>/<owner>/<a>.<b>.git for upstream …/<a>/<b>.git. The forge and owner come from the first of WEBENGINE_GIT_MIRRORS (the owner's URL, http://<forgejo-address>/<owner>), the checkout's rig remote (the development box), WEBENGINE_LANE_GIT_URL, and the checkout's origin remote when its URL carries credentials (a lane's worktree of its .git-mirror). The mirror URL never carries credentials: the user and token (from that URL, else WEBENGINE_FORGE_USER / WEBENGINE_FORGE_TOKEN in the environment or the primary checkout's .env) reach git only as an http.<forge>/.extraHeader in the environment of the one git command, so a checkout's .git/config on the share holds the address and nothing else. Every fetch prints which mirror it used. WEBENGINE_UPSTREAM_DIRECT=1 fetches from the upstream URL instead (a source whose mirror is missing, or the forge down) and says so on every fetch; with neither a forge setting nor the override, a fetch fails naming both.

Callers: oracle/suites/fetch.mjs (and through it bench/nas-refresh.sh suites), bench/nas-refresh.sh webgpu-cts, oracle/suites/fetch-checklayout.sh, and crates/wpt-votes/src/features.rs (its WPT tree cache). Shell and Rust callers run git through the script: node oracle/suites/mirror.mjs git <upstream> <git args…>, an argument {url} standing for the mirror URL; mirror.mjs url <upstream> prints the URL. fetch.mjs points a checkout's origin at the mirror before anything that reaches the network (a fetch, or a new sparse path whose blobs a blobless checkout fetches), so a checkout made from upstream is switched in place, never cloned again; a checkout already at its pin needs no forge settings. oracle/suites/mirror.test.mjs covers the resolution; oracle/suites/fetch.test.mjs and bench/nas-refresh-test.sh fetch through a local directory standing in for the forge.

Adding an outside source: create its mirror before the first script that reads it lands, with the forge's migrate API and a token holding the repository scope — POST /api/v1/repos/migrate with clone_addr the upstream URL, repo_owner the owner above, repo_name from node oracle/suites/mirror.mjs name <upstream>, mirror: true, mirror_interval: "8h0m0s", service: "git" (WPT took about 7 minutes; the call returns when the first sync is done). A pin moved to a commit newer than the mirror's last sync needs POST /api/v1/repos/<owner>/<name>/mirror-sync first.

npm packages and crates are plain files on the NAS share, filled only by the scheduled refresh (bench/nas-refresh.sh, below), and every install and build reads only those files: nothing contacts npmjs or crates.io at build time, and nothing falls back to them. A package or crate missing from the share fails the install or build, naming it.

• npm: the share's npm-cache/, filled by bench/nas-refresh.sh npm. bench/oracle-install.sh runs npm ci --offline with it as npm's cache; the share is WEBENGINE_LANE_SHARED on a lane or in a job and $HOME/lane-shared otherwise. bench/lane-apply.sh installs with --offline against it as well. npm's integrity hashes are checked on every install from the cache.
• Crates: the share's crates/, a vendored directory source (one <name>-<version>/ per crate) filled by bench/nas-refresh.sh crates from every crate Cargo.lock and bench/Cargo.lock name (cargo vendor --locked --versioned-dirs, with its own CARGO_HOME holding no source replacement). Crates are only added, never removed. Each machine's own config.toml in its Cargo home replaces crates.io with that directory: bench/lane-dist.sh with DIST_ROLE=registries writes the block. On a lane it reads LANE_SHARED_DIR (default /srv/lane-shared) and writes the lane user's ~/.cargo/config.toml. The development box's user builds other projects whose crates are not on the share, so its block goes into a Cargo home used only for this repository, $HOME/.local/share/webengine/cargo-home, from $HOME/lane-shared; bench/devbox-cargo-home.sh installs it (it runs the registries role with LANE_CARGO_HOME). A cargo wrapper ($HOME/.local/share/webengine/bin/cargo, linked from $HOME/.local/bin/cargo, which comes before ~/.cargo/bin on PATH) sets CARGO_HOME to it when cargo runs inside a checkout of this repository (a directory above holding both Cargo.lock and jobs/src/manifest.rs) and CARGO_HOME is unset; elsewhere it changes nothing. Dagu job steps (bench/dagu/*.yaml) put the wrapper's directory before ~/.cargo/bin; bench/devbox-cargo-home-test.sh covers the wrapper and that ordering. ~/.cargo/config.toml settings (sccache) still apply, because cargo reads the .cargo/config.toml of every directory above where it runs. Not caught: cargo --manifest-path or -C from outside a checkout. Cargo takes no [source] replacement from environment variables, so the setting lives in each machine's Cargo home, not in the repository. Each vendored crate carries a .cargo-checksum.json; cargo checks its package checksum against Cargo.lock (a mismatch stops the build) and each file against its recorded hash. bench/lane-dist-registries-test.sh covers the role. experiments/ lockfiles are not vendored; building them needs a Cargo home without the replacement.

The development box cannot mount the NAS, so $HOME/lane-shared holds its own copy of both, refreshed the same way (WEBENGINE_LANE_SHARED=$HOME/lane-shared bench/nas-refresh.sh crates npm, or the scheduled refresh on the box's own lane entry).

Other outside downloads are not yet mirrored: Playwright's browsers, specification sections (fetch_specs, over HTTP), browser-compat-data and web-features data from a CDN (crates/content-analyser, crates/wpt-votes), wpt.fyi results (crates/wpt-votes), and release binaries from GitHub for Dagu (bench/dagu-lane.sh) and sccache (bench/lane-dist.sh), plus the toolchain installers bench/lane-provision.sh runs.

The track gate runs only on the build lane. tracks.json → verify runs bench/verify.sh, which hands itself to bench/on-lane.sh --lane-only and, once there, runs cargo test --workspace, the job event store's tests (jobs/events/*.test.mjs under node --test, then bench/job-events-lane-test.sh), the page selector's tests (oracle/suites/feature-pages.test.mjs under node --test, on committed fixture pages with no browser), and then bench/generated-scores.sh. The other *.test.mjs and bench/*-test.sh files are run by hand, not by the gate. With no lane configured or none answering (ssh -o BatchMode=yes -o ConnectTimeout=5 fails) the gate exits 3 and runs nothing on the development box; a lane that answers but fails the tests or a score is a failed gate — no local retry. Measured before the scores were added: a new worktree's first test run 106 s (124 s with a fresh mirror), a warm rerun 22 s, local fallback 78 s. A clean cargo test --workspace compiles in 61 s on a lane and 57 s on the development box — the build is bound by the dependency chain through Stylo, not by cores; the lane's value is that it takes load off the shared box and runs several gates at once.

Unattended landings never fall back to the development box. bench/on-lane.sh without --lane-only runs its command locally when no lane answers, and bench/land.sh runs each landChecks command that way, and verify directly. Setting WEBENGINE_LANE_STRICT=1 in bench/land.sh's environment (the scheduled jobs/scripts/poll-and-land.sh gets it from tracks.json's specWatch.pollEnv) makes verify go through bench/on-lane.sh --lane-only too and every check --lane-only, so with no lane answering the landing is refused with "verify failed on the tip (exit 3: no build lane answered ...)" and nothing of the gate runs here. Unset, as in interactive use, a check keeps its local fallback. bench/verify.sh itself is lane-only either way; the one thing that would run it on the development box is WEBENGINE_ON_LANE set there by hand.

bench/on-lane.sh probes the lane up to 3 times before calling it unreachable. The probe (bench/lane-probe.sh, 10 s connect timeout, 2 s between tries) fails now and then through the jump host on a healthy lane; WEBENGINE_LANE_PROBE_TRIES changes the count, and each retry prints one line to stderr.

Already on a lane, bench/on-lane.sh runs the command in place. With WEBENGINE_ON_LANE set, bench/on-lane.sh (and bench/remote-lane.sh) runs its command in the named worktree on the machine it is on, --lane-only or not, reads no lane settings, and ignores --pull, since the outputs are already where they belong; the command's exit status is the run's. This is the same check bench/verify.sh makes, so a spec's bench/on-lane.sh --lane-only "$PWD" ... works unchanged inside a Dagu job, whose clone has no .env and no bench/lane.local.json and whose writer gets WEBENGINE_ON_LANE=1 from jobs/scripts/pi-session.sh. As with bench/verify.sh, a WEBENGINE_ON_LANE set by hand on the development box would run the command there. bench/on-lane-test.sh covers both sides. The cost: with both lanes down nothing lands until one answers. bench/land-test.sh scenario s1 covers both settings.

The gate scores every generated function against its own corpus. A committed file under crates/layout/src/generated/ can be wrong while every test passes, because the tests exercise in-tree reference rules, not the generated function. bench/generated-scores.sh clears and rebuilds each corpus with its *_ground_truth_cache binary (the browser oracle runs on the lane), then generated_score_check scores each generated function through its own runner and fails it against its floor. The floor is two numbers in crates/layout/src/generated/<name>.floor.json beside the function's provenance record: cases_passed, how many cases it passes, and cases_run, how many cases its corpus holds (only cases three engines agreed on). It fails when the passing count drops, when the corpus shrinks even though the passing count holds, and when a function passes no case at all (a lane missing a browser leaves fewer than three engines to vote, so every case drops; a floor of zero cannot hide a function that scores nothing). The provenance record describes the model batch that admitted the function and is never rewritten; the floor is never below its verification.cases_passed, and a function with no floor file yet is held to that alone. That is where a function a feature job admits starts: the job's admit stage (jobs/scripts/admit.py store_main) writes the provenance record and no floor file, since bench/land.sh refuses a branch that touches one, and the first floor comes from the raise after the landing (below). A placeholder record, such as the hand-written table one (0 recorded), therefore floors at its real score once a floor file exists.

Floors rise by themselves, after a landing. bench/after-land.sh ends with bench/generated-floors-update.sh, which runs bench/generated-scores.sh --print-raises on the build lane, writes each floor that the fresh score is above (a function with no floor file gets one), and commits only those files as one commit on main, the way the map and dashboard commits are made. It refuses on uncommitted changes, writes nothing when the lane does not answer or a score fails, never lowers a number, and a failure is one line and does not change the landing. Raising happens after the landing rather than in the gate because the gate runs on the lane's copy, where a commit cannot reach the track, and because a track never edits a floor: bench/land.sh refuses a diff that touches any *.floor.json, so an unattended track cannot lower one. The gate scores with --print-raises too, and a gate that passes on a checkout with nothing uncommitted or untracked keeps its raise lines keyed by the git tree it scored (bench/generated-raises.sh, in the repository's shared git directory). A landing is a fast-forward to exactly that tree, so the raise after it takes those lines instead of scoring again, as long as main has changed since only in README.md and under docs/ (the dashboard and map commits that run first). A gate on a dirty checkout, or a landing whose tree no passing gate scored, still costs a second scoring run on the lane. Either way there is a window between a landing that improves a function and the raise in which the old floor holds. A floor comes down only by a person's reviewed edit of its file.

One place to add a generated function. A function is discovered from its <name>.provenance.json; its runner and the cache binary that writes its corpus are one entry in FUNCTIONS in crates/generation/src/bin/generated_score_check.rs, and a provenance record with no entry fails the gate with that instruction. bench/generated-scores.sh takes the caches to rebuild from generated_score_check --list-caches, so nothing under bench/ changes and a track adding a function can land unattended. bench/generated-scores-test.sh exercises the scripts, including the raise, without a browser. bench/score-generated-oracle-test.sh runs jobs/scripts/score-generated.sh, what a feature job's admit stage scores with, against the real browser oracle on the box-generation train pages (generated, no network; about two minutes), on the development box only. It and bench/feature-job-box-test.sh (docs/local-lane.md → "Before a folknet send") run in the job-system gate, bench/e2e-gate.sh, on every landing that changes the job system, crates/generation/, crates/provenance/, the harness's case modules or a specs/*.job.json, and fail it when either fails (docs/local-lane.md → "Scenario jobs").

Model batches run only on a lane. bench/absolute-mixed-batch.sh and bench/overflow-mixed-run.sh are started from the development box like any script (WEBENGINE_MODELS=qwen3.8-27b-fast:32:32 bench/absolute-mixed-batch.sh) and re-run themselves on the lane through bench/on-lane.sh --lane-only (bench/batch-lane.sh): the WEBENGINE_* settings go along, credentials and lane settings excepted, and the batch's results come back with --pull. With no lane configured or none answering the script exits 3 and runs nothing locally; bench/lane-apply.sh exports WEBENGINE_ON_LANE=1, which is how a script knows it is already there. Each run also records its phase timings and how many of the lane's CPUs each phase kept busy (bench/cpu-sample.sh, bench/cpu-phases.py; docs/rig-capacity.md → "A real batch on a build lane").

A stray directory at a worktree's lane path breaks the gate. bench/lane-apply.sh tests only for <lane-root>/<name>/.git; a leftover directory without it makes git worktree add fail, and the gate cannot pass until it is removed by hand (check git -C <mirror> worktree list first). Not yet fixed in the script.

Cleanup. Every worktree run on a lane leaves its copy (<root>/<name>, target/ and all, 3–12 GB) and <root>/.payload-<name> there; left alone they filled a lane's disk. bench/lane-sweep.sh removes, on every lane in bench/lane.local.json lanes[], the copies whose track has no worktree in the primary checkout (--dry-run lists them with sizes). It keeps the primary's own copy, any copy with a process running in it (a spec writer runs inside its copy), and directories that are not worktrees of the lane's mirror. Untracked data under a copy's corpus/ (links excepted: they point into the NAS already) goes to the NAS first, as one tar file at archive/corpus/<name>/<time>.tar; with no NAS directory the copy is kept. A branch a copy was on stays in the mirror. It runs at four points: by hand; from bench/after-land.sh after every landing, by hand or unattended, over every copy (a failure is one line on stderr and changes nothing else); from bench/land.sh after a landing, once the track's worktree is gone, for that track only (a failure is named on its line and undoes nothing); and from bench/remote-lane.sh when the lane's root filesystem has less than WEBENGINE_LANE_MIN_FREE_GB (20) free before a run, which then refuses with exit 4 if the sweep did not free enough, naming the lane by its place in the list, the free space and the largest copies. A hand landing leaves no worktree, so the next sweep takes its copies. Scenarios: bench/lane-sweep-test.sh.

While KEEP_JOB_OUTPUT is yes (the default; the environment, else ~/.lane-shared-env on the lane), the sweep keeps the most recently used of the copies it would remove (the later change time of the copy and its payload directory, worked out over the whole lane even with --track) and removes the older ones as above, whatever the free space: only the latest run's output is ever read again. Of the copy it keeps it touches nothing that cannot be rebuilt: sources, corpus/, candidates, jobs-work/, runs and archives stay. When the lane's root filesystem has less than WEBENGINE_LANE_MIN_FREE_GB free, it deletes only that copy's build output (a Cargo target/ beside a Cargo.toml, node_modules/), on a freed: line with the size; otherwise a kept: line says why. bench/remote-lane.sh still stops with exit 4 on a lane that is short after that, so a person frees or adds space. The newest copy keeps its job output because the first aspect-ratio run's candidate sources, reports, prompts and compile errors could not be read afterwards and the failure was diagnosed from sizes in batch.log instead of from code. KEEP_JOB_OUTPUT=no removes that copy too.

Retention of job-run folders. One rule covers every folder a job run leaves behind, on the lanes and on the NAS share: of each group the newest two stay and the rest go, with no age floor. jobs/scripts/retention.sh holds the rule and, in its header, the list of folders and their groups:

folder	group
archive/job-logs/<job>/<Dagu run id>/	job
archive/job-work/<job>/<Dagu run id>/	job
archive/job-work/<job>/restarted-<time>.tar.gz	job (apart from its runs)
archive/job-corpus/<job>/<spec blob, else run id>/	job
archive/stage-cache/<job>/<stage>/<key>/	job and stage
archive/gate-logs/<worktree>/<time>-<guest>-<commit>.log	worktree
archive/corpus/<lane copy>/<time>.tar	lane copy
~/dagu/runs/.restarted-<name>-<time>/	name

Newest is the modification time, except in the stage cache (an entry's last use) and for a moved-aside run directory (the time in its name). It runs at two points. The exit handler of bench/dagu/job.yaml runs it at the end of every job for that job's groups only, and never removes the folders of the run itself (its run id, its staging folders and its spec blob). bench/lane-sweep.sh runs it over every group on every lane it sweeps (--dry-run lists what would go). A leftover of an interrupted copy (<run id>.partial-<pid>, *.tar.partial, *.tar.part) does not count among the two and goes once nothing under it has changed for an hour, since a copy on another lane may still be writing it. Each earlier location had its own count-plus-age rule, or none; with daily runs the age floor kept everything, and on 2026-10-12 one job's archive/job-work/ held 25 runs (34 GB).

A job directory is never moved aside: a restart (clean-jobs, or prepare on RESET=yes or a new base or spec) archives it as a restart tar and removes it (see "Dagu on a lane"), so no folder can be left without an archived-* marker. --keep-output is still accepted and has no effect.

Left out of the rule, because each is bounded otherwise or is not per run: the live job directory ~/dagu/jobs-work/<job>/ (the run still writing); ~/dagu/src-<id>/ and ~/dagu/scripts-<id>/ (prune_sources()); lane copies (the sweep above); the per-worktree archive/suite-results/ and archive/webgpu-cts-results/ (overwritten in place); ledger/<job>.jsonl (one append-only file per job); Dagu's run history (7 days); and the fixture workflow's ~/dagu/runs/<run id>/. jobs/tests/retention_coverage.rs fails when the workflow's code names a folder under the share's archive/, at the top of the share, or in jobs-work/ that is neither in the list nor among these exclusions; jobs/tests/retention_share.rs and jobs/tests/retention_moved_aside.rs test the rule itself.

Test-run memory. A cold cargo test --workspace on the development box adds about 1.1 GB at peak (10.8 → 12.0 GB), a warm one 67 MB; seven concurrent tracks fit in its 30 GB if their peaks do not coincide.

Distributed compilation. sccache-dist builders on six nodes (416 cores registered) with redis as the shared cache. After dropping one builder on a node with slow outbound network (failures 84 → 1), it still lost on wall time: a clean -p dom release build took 82 s distributed against 45 s local on the same lane. The dependency graph is too narrow to use the cores. A wide build fares no better: a batch's candidate probe (one --keep-going build of 32–64 small crates) took 14–45 s through sccache-dist against 1.4–1.6 s with no compiler wrapper on the lane, just as slow on a second pass, so each compile through the wrapper carries a fixed delay far larger than the compile itself (docs/rig-capacity.md, "A real batch on a build lane"). The delay is the input upload: for a Rust compile the sccache client packages every library in the -L dependency= directory (target/release/deps, 225 rlibs, 336 MB), not only the crates the compile loads. One probe library compile (0.1 s with plain rustc) took 4.4 s distributed, sent 121.9 MB (the gzip size of that whole directory) and spent 2.3 s of client CPU compressing it. The cost grows with the dependency directory, not with the crate, and is paid again on every compile, which likely explains the -p dom result too (not measured). The probe therefore runs without the wrapper, on the lane's own cores; the development box's sccache config stays untouched.

Shared data on the NAS. Each lane bind-mounts one NAS directory at /srv/lane-shared (WEBENGINE_LANE_SHARED, also the npm cache). Login shells get the npm cache from ~/.lane-env; the Dagu job's prepare and run steps (bench/dagu/job.yaml) read only WEBENGINE_LANE_SHARED and npm_config_cache from ~/.lane-shared-env, and bench/oracle-install.sh runs npm ci --prefer-offline (--offline when it sees the share; below), so a package is downloaded from the registry once and every lane and job after that installs it from the share. The NAS is fast from the lanes (about 92 MB/s write, 114 MB/s read) but slow for small files (~38 ms each), so target/ stays local. bench/lane-apply.sh links each worktree into it on every run:

in the worktree	on the share
oracle/suites/checkouts/<suite>	suites/<suite>-<pinned commit>/
webgpu-cts/cts	suites/webgpu-cts/
corpus/specs (specification sections; a worktree's own earlier cache is merged in first)	specs/
corpus/<name> for every *ground-truth cache the code names	ground-truth/<name>/
oracle/suites/results	archive/suite-results/<worktree>/
webgpu-cts/results	archive/webgpu-cts-results/<worktree>/
(the command's output, copied)	archive/gate-logs/<worktree>/<time>-<guest>-<commit>.log
(one line per model batch, bench/ledger.py)	ledger/<worktree>.jsonl
(a Dagu job's full logs, bench/dagu/job.yaml, scrubbed)	archive/job-logs/<job name>/<Dagu run id>/
(a Dagu job's gitignored keep_dirs, tarred, unscrubbed)	archive/job-corpus/<job name>/<spec blob>/<dir>.tar
(a Dagu job's work tree and candidates, unscrubbed)	archive/job-work/<job name>/<Dagu run id>/

The last row comes from the Dagu job workflow, not bench/lane-apply.sh: it copies each job's stage logs, state.json, the review, each pi session's prompt and events.jsonl (its turns and tokens) and oracle-install.log there when a run ends, whether it passed, failed or was stopped, with an archive.txt listing every file and its line count; the step log prints the path. Credentials, URLs, addresses and token-shaped values are replaced on the way (bench/dagu/job.yaml, "handler_on").

The two job rows after it are written by the same handler, for every final status, while KEEP_JOB_OUTPUT is yes: archive/job-work/<job name>/<Dagu run id>/ holds repo.tar (the job's repo tree), writer-<n>-tree.tar for each fan-out writer's tree, and candidates/ (the repo's corpus/candidates as plain files: sources, replies, compile errors, reports, prompts, readable without extracting anything). target/, oracle/node_modules and .git/config (it holds the forge URL with its credential) are left out. These copies are not scrubbed, like the keep_dirs tars. Of each job's folders under archive/job-logs/, archive/job-corpus/ and archive/job-work/, and of each worktree's archive/gate-logs/, the newest two stay (see "Retention of job-run folders" above). bench/job-archive-test.sh checks that a finished run and a failed run each leave their candidate sources under archive/job-work/.

A share that is not the NAS. The development box cannot mount the NAS (it reaches the rig only over ssh through the jump host), so its WEBENGINE_LANE_SHARED is a local directory, $HOME/lane-shared (docs/local-lane.md). Every archive still ends up on the NAS: its ~/.lane-shared-env also sets WEBENGINE_ARCHIVE_NAS=<lane ssh destination>:/srv/lane-shared/archive and WEBENGINE_ARCHIVE_NAS_SSH_OPTS (lane 0 of bench/lane.local.json, whose /srv/lane-shared is the NAS), and jobs/scripts/archive-to-nas.sh moves what the writers above put in the local archive/ there over rsync. A lane whose share is the NAS leaves WEBENGINE_ARCHIVE_NAS unset, and the script does nothing there. Each file is copied with --ignore-existing, compared with its NAS copy by content (rsync --checksum), and deleted locally only when the two match, so a failed or partial copy leaves the local file. A file the NAS already holds under that name with other content is never replaced: the copy goes beside it with the UTC time before its extension, and a kept both: line names the two. It moves the write-once groups (job-logs/, job-work/, job-corpus/, gate-logs/ and corpus/). It leaves local whatever is still being written (*.partial* and *.part staging copies, and gate logs changed within the hour, which bench/lane-apply.sh is still appending to), an e2e scenario's e2e-* folders (removed locally by bench/e2e-lane-clean.sh), and the folders this machine reads back or links: stage-cache/, suite-results/ and webgpu-cts-results/.

It runs at three points, with nothing to start by hand:

• the exit handler of bench/dagu/job.yaml, for the run's own archive (--only its job-logs/, job-corpus/ and job-work/ folders), before it writes archived-<Dagu run id>: the marker is written, naming the NAS copy, only when every file is verified there. A failed copy keeps it all local, writes no marker, prints a line starting JOB ARCHIVE NOT ON THE NAS, and does not fail the run. After its retention pass the handler moves whatever else is waiting, such as an earlier failed copy, gate logs and corpus tars;
• a restart, for the restart tar, before the old work is removed: bench/dagu-lane.sh clean-jobs, and the prepare step of bench/dagu/job.yaml (see "Dagu on a lane");
• bench/lane-sweep.sh, after its retention pass on every lane (so after every landing, through bench/after-land.sh).

One move runs at a time (flock on ~/.archive-to-nas.lock). Tests: bench/archive-to-nas-test.sh covers the script, and bench/job-archive-test.sh and bench/lane-sweep-test.sh cover its callers, each with a stand-in ssh and a local directory standing in for the NAS.

Specification sections come only from the share on a lane or in a job. With WEBENGINE_LANE_SHARED set and WEBENGINE_NAS_REFRESH unset, fetch_specs (crates/provenance/src/fetch.rs fetch_section()) fetches nothing: a section missing from specs/ fails naming bench/nas-refresh.sh specs, the refresh that fills it. jobs/scripts/feature-specification.sh passes the share's specs/ as --cache to both fetch_specs and specification_text. The development box sets neither variable and keeps fetching into its own corpus/specs. bench/nas-refresh.sh specs (with WEBENGINE_LANE_SHARED naming the share) sets WEBENGINE_NAS_REFRESH=1 and runs fetch_specs once for each section the checkout cites: every pub const SPEC_URL under crates/ and every oracle/suites/features/*.json specification value. A section already on the share is kept as it is, so a run adds only what is newly cited. It prints nas-refresh specs ok|failed <detail>, exits 1 when a section could not be fetched (the others are still fetched), and holds a local flock on nas-refresh.lock in the checkout's git directory; bench/nas-refresh-test.sh covers it with a stand-in fetch_specs.

Conformance suite checkouts follow the same rule. Under the same two variables, oracle/suites/fetch.mjs fetchSuite() fetches nothing: under its lock it checks that suites/<suite>-<pinned commit>/ is at the pinned commit and holds every pinned sparse path, and otherwise fails naming bench/nas-refresh.sh suites. That source runs fetch.mjs for each suite in oracle/suites/suites.json against the share's directory (through a temporary link, as bench/lane-apply.sh links it), prints nas-refresh suites ok|failed <detail>, and fails one suite without stopping the others. A new checkout is built in a <dir>.staging-<token> sibling and renamed onto the pinned directory, so a reader finds it empty or finished; one already at the pin is left alone. On the NFS share, git sparse-checkout set adding several paths fills one, leaves the next as an empty directory and fails with "Is a directory"; fetchSuite() then removes the empty directories at the new paths and sets again, at most once per new path. oracle/suites/fetch.test.mjs and bench/nas-refresh-test.sh cover both sides against a local upstream repository.

npm packages too. Under the same two variables bench/oracle-install.sh runs npm ci --offline and bench/lane-apply.sh runs npm install --offline, both against the share's npm-cache/, so a package missing there fails the install naming bench/nas-refresh.sh npm. That source copies package.json and package-lock.json of oracle/ and webgpu-cts/ to a temporary directory and runs npm ci --prefer-offline --ignore-scripts there with the share's npm-cache/ as npm's cache, so a package already cached is not downloaded again and the checkout is untouched; it prints nas-refresh npm ok|failed <detail>. The Dagu job's prepare step reads WEBENGINE_LANE_SHARED from ~/.lane-shared-env and passes it to bench/oracle-install.sh, with npm_config_cache set to the share's npm-cache/ and npm_config_offline=true for a base whose bench/oracle-install.sh predates the refusal, so a job's install is offline too; a failure is reported in the step log and left to the writer, as before. jobs/tests/job_prepare_install.rs runs that part of the step against a stand-in npm.

The WebGPU conformance suite is pinned in webgpu-cts/cts.json, and nothing on a lane fetches it: webgpu-cts/cts is linked to the share's suites/webgpu-cts, which bench/nas-refresh.sh webgpu-cts keeps as a link to suites/webgpu-cts-<pinned commit>/. That source makes a shallow checkout of the pin in a staging sibling, runs npm ci in it against the share's npm-cache/ (the suite's tools/run_node and dev server need its own packages), renames it into place, and then replaces the link by a rename, so a lane sees the old suite or the new one. A checkout already at the pin is left alone, and an earlier pin's checkout is kept. An empty directory where the link goes (what bench/lane-apply.sh creates when the share lacks it) is replaced; any other directory there fails the source, naming it, since a lane may be running from it. It prints nas-refresh webgpu-cts ok|failed <detail>; bench/nas-refresh-test.sh covers it against a local upstream.

The refresh runs on a schedule from one lane, never from a track's copy (with the one exception below): bench/nas-refresh-install.sh [--lane <index>] [--schedule '<cron fields>'] copies bench/nas-refresh-cron.sh to ~/nas-refresh/run.sh on the lane and adds a crontab line (marker # webengine-nas-refresh) that runs it under flock -n, appending to ~/nas-refresh/refresh.log. Each run fetches the lane root's .git-mirror (skipped when it fetched in the last 10 minutes, as in bench/lane-apply.sh), moves the detached checkout ~/nas-refresh/checkout (a worktree of that mirror outside the lane root, so bench/lane-sweep.sh leaves it alone) to the mirror's main, and runs every source from it with the share from ~/.lane-env. The default lane is the lanes[] entry whose ssh host is localhost (the development box's own, docs/local-lane.md) and the default schedule hourly at minute 17. That entry is for validating the refresh against the box's stand-in share; the production schedule runs on a folknet lane once it is validated there, which is the same script with --lane <folknet index> (and --schedule if hourly does not suit), followed by remove on the box's lane. run refreshes once now, status and remove do what they say. bench/nas-refresh-install-test.sh covers it with stand-in ssh and crontab and a local upstream.

The one exception is a branch that changes wpt.sparse (or another suite's sparse) in oracle/suites/suites.json: the lanes' check fails until the share holds the new paths, and the scheduled refresh reads main. Such a branch runs the suites source once from its own worktree before its gate, on a build lane with bench/on-lane.sh --lane-only <worktree> bash bench/nas-refresh.sh suites and on the development box with WEBENGINE_LANE_SHARED=$HOME/lane-shared bench/nas-refresh.sh suites. Adding paths is safe for other readers, which only check that their own paths are present. The hourly refresh from main may narrow the list back until the branch lands, so the branch's refresh runs just before the gate.

The same holds for a branch that adds or upgrades a crate (Cargo.lock, bench/Cargo.lock) or an npm package (oracle/package-lock.json, webgpu-cts/package-lock.json): builds read only the share, so the branch's gate fails until the share holds them. It runs the matching sources once from its own worktree before its gate: bench/on-lane.sh --lane-only <worktree> bash bench/nas-refresh.sh crates npm for the lanes' share, and WEBENGINE_LANE_SHARED=$HOME/lane-shared bench/nas-refresh.sh crates npm for the development box's copy. Both sources only add, so the hourly refresh from main never takes the branch's packages away again, and other readers are unaffected.

A lane never starts without the share. The node mounts it through /etc/fstab with nofail, so a NAS that is down at boot leaves the node and its other tenants' guests starting; a systemd drop-in on the lane's own container unit (pve-container@<vmid>.service.d/lane-nas.conf, RequiresMountsFor) makes each start of the lane retry the mount and fail when it fails. Without it, a boot with the NAS down left a lane running on the node's empty local directory, and every lane run failed at mkdir /srv/lane-shared/suites: Permission denied (2026-09-29, after a maintenance window). After such a boot, pct start <vmid> on the node once the NAS answers brings the lane up. bench/lane-nas.sh writes the drop-in.

After an outage or maintenance window, run bench/folknet-check.sh from the development box, then bench/folknet-check.sh --fix. It checks every node, lane, build guest and tunnel the project depends on against a private inventory (bench/folknet.local.json, shape in bench/folknet.example.json): root login by key on each node, each guest on its declared node, running, at its declared address, each lane's NAS mount and container unit guard, and each development-box tunnel. --fix starts stopped guests, reruns bench/lane-nas.sh for a lane missing the NAS, and restarts tunnels; a moved guest, a changed address or a node that lost its key is reported for a person, with every Proxmox node on the subnet listed by the name in its web certificate. It also checks what a job needs above the machines: each folknet lane's Dagu answering through its port forward, each lane's inventory probe entry and a snapshot under 15 minutes old, every model a specs/*.job.json names being ready, the job event store, the announcement bot, both forges, and the scheduled poller having run in the last 15 minutes.

Alerting. bench/folknet-watch.sh install puts a crontab entry on the development box that runs bench/folknet-check.sh --fix every 5 minutes and posts to #bez through the announcement bot, and nowhere else, when a check starts failing, when it recovers, when --fix repaired something, and every 6 hours while anything still fails. Addresses are masked in every post. bench/folknet-watch.sh status shows what is failing now.

The watch itself stopping, as it does when the development box is down, is noticed from lane 0. Each pass sends a heartbeat to the job event store (a step event of the feature folknet-watch, which also shows as a row on the job dashboard; the store keeps every one, about 100 bytes every 5 minutes). bench/folknet-watch-heartbeat-check.sh runs on lane 0 from the lane user's crontab every 5 minutes, reads the feature's newest time from the store's GET /features, and posts to #bez through the announcement bot when it is over 15 minutes old ("has not reported for N minutes"), then once more when heartbeats resume. It posts once per change and nothing while the state holds; a store that is down or holds no heartbeat yet posts nothing, since the store's own check reports the first. Its state (ok or stale) and log sit in ~/folknet/watch-heartbeat/ on the lane. The bot's URL and token live there too, as announce-url and announce-token, mode 600; only the check reads them, handing the token to curl on stdin. bench/folknet-watch-heartbeat-install.sh --lane 0 copies the check and adds the crontab entry (install, remove, status), and bot sends WEBENGINE_ANNOUNCE_URL and WEBENGINE_ANNOUNCE_TOKEN from the environment or .env over ssh on stdin. The development box needs WEBENGINE_JOB_EVENTS_URL and WEBENGINE_JOB_EVENTS_TOKEN in .env, as for the job scripts. bench/folknet-watch-heartbeat-test.sh tests all three against the real store and a stand-in bot.

The ledger is what a landing's announcement to the community Discord reads for its numbers (docs/web-features-map.md → "Announcements"): each batch's start and end time, lane CPU-seconds, and per model the candidates, compiled count, best score, tokens and draw seconds, with no host names or addresses. A batch already in the file (same track, start and end) is not appended again, and a landing's announcement leaves a marker file named for the landing's tip commit and job in ~/.local/state/webengine/announced (WEBENGINE_STATE_DIR moves it), so a rerun of bench/after-land.sh or the batch script neither posts nor records twice; a failed post leaves no marker.

An existing real directory is left alone. A ground-truth cache written by one branch is read by every other, so a recipe change should regenerate its cache. The shared copy of a ground-truth cache mirrors the checkout's exactly: a cache binary writes case files and never removes one, so when bench/lane-apply.sh finds a cache rebuilt in a real directory after a successful command it runs bench/lane-mirror-cache.sh (rsync --delete), which also removes the case files the new cache lacks. Without that, a reference that passed against the job's own cases failed the landing gate against the extra case files earlier runs had left in the shared copy. Only a directory named <name>-ground-truth is mirrored onto the shared directory of the same name; any other directory is refused and left alone (bench/lane-mirror-cache-test.sh). The export is open to the whole subnet: nothing secret goes on it. Measured: fetching the suites into an empty share took 6 min 19 s; a second lane then found them in 13 s.

Reaching the model gateway. A lane reaches the gateway at its rig-network address, not the development box's loopback forward. Each lane keeps that address in ~lane/.lane-model-env as WEBENGINE_LANE_MODEL_URL (written by bench/lane-provision.sh from LANE_MODEL_URL), and bench/lane-apply.sh swaps it into the worktree .env's WEBENGINE_MODEL_URL line on every run, leaving the key as copied. The address never enters the repository (bench/lane-apply-test.sh).

No credentials left behind. The worktree's .env (the gateway key, usually the forge token) is copied into the lane copy for the run only. When the run ends -- success, failure or interrupt -- bench/lane-apply.sh removes the lane copy's .env, its temporary rewrite and the payload's .env, and bench/remote-lane.sh removes the same three over its own ssh, which covers a run whose connection dropped first. Nothing else in the lane copy is touched. WEBENGINE_LANE_KEEP_ENV=1 keeps them, and bench/lane-writer-launch.sh sets it, because the writer it starts outlives the launch and its model batches read that .env. Two known gaps: an interrupted run exits 130 (SIGINT) or 143 (SIGTERM) locally but does not stop the command on the lane, which runs until it finishes or next writes to the dropped connection; and the git mirror's clone URL can carry the forge token, which then shows in the lane's process list as an argument to lane-apply.sh, a copy of that secret outside .env.

Running writers on a lane. No script here names an agent harness; the harness is a declaration. tracks.json → specWatch.lanes.tool names it (TRACKS_TOOL in the environment is the fallback, and a launch with neither is refused), and pi-tracks' writer command prepares that harness on the lane and starts the session. bench/lane-provision.sh's writer-host section (enabled by LANE_PI_TRACKS_URL) clones pi-tracks on the guest and prints changed:/unchanged: per step, so a rerun is idempotent. What a particular harness needs installed ahead of time is a command the operator supplies as LANE_WRITER_SETUP, run as lane with LANE_WRITER_TOOL and LANE_WRITER_DIRS (the lane roots and mirrors, relative to the home directory) in its environment; with none set the step does nothing. A harness login is a person's step over ssh as lane. bench/lane-writer-launch.sh <worktree> <lane-index> <model> <brief-file> syncs a worktree, checks out its branch, pulls pi-tracks and runs "$PI_TRACKS_HOME/bin/writer-exec.sh" --tool <name> --model <model> --brief-file <path> there (PI_TRACKS_HOME defaults to $HOME/.local/share/pi-tracks), refusing over a dead writer's unfetched commits and failing when that command exits nonzero; the brief lands in ~/.cache/lane-writer-briefs/ on the lane, outside the worktree. bench/lane-writer-fetch.sh brings commits back fast-forward only. tracks.json → specWatch.lanes declares both. bench/lane-nas.sh decides a guest is busy from the process names in LANE_BUSY_COMMANDS (default cargo|rustc|node); add the harness's process name when a writer runs under a different one. crates/harness/tests/no_harness_name.rs fails the workspace tests when a bench/ script, tracks.json (other than the tool entry) or .gitignore names a harness. bench/lane-writer-launch-test.sh tests the launcher against a stand-in writer command; a real launch on a lane is unverified until the pi-tracks writer command is on its main.

Dagu on a lane. A job (tasks/job-submit-land.md) runs as one run of the Dagu workflow bench/dagu/job.yaml, on the lane WEBENGINE_DAGU_LANE names (an index into bench/lane.local.json, default 0; Dagu's coordinator is off, so one lane). bench/dagu-lane.sh installs, starts, reaches and cleans up after it; its header lists the actions and settings. Without --lane it takes the same lane, WEBENGINE_DAGU_LANE or 0, unless WEBENGINE_DAGU_LANE is unset and WEBENGINE_LANE_SSH names a lane, which then wins. jobs/scripts/submit.sh and jobs/scripts/poll-and-land.sh check manifests and results with the job-check binary from $JOBS_STATE_DIR/bin (default ~/.local/state/webengine-jobs/bin) on the development box; jobs/scripts/install-job-check.sh builds it on a lane (bench/on-lane.sh --lane-only --pull) and installs it there; bench/after-land.sh runs it after every landing, so the binary follows main.

• The requirements check. Before the run step's job-run starts the first stage that is not passed, on every start and every resume, it checks that the lane has what the manifest is about to need -- git, python3, node and kill resolvable, cargo and the oracle's Playwright browsers when a writer is in the stage list, a reachable model gateway when a stage runs a pi session, 20 GiB free on the workdir's filesystem, the shared archive directory, WEBENGINE_ON_LANE when a stage calls bench/on-lane.sh, bench/verify.sh or run-batch.sh, and at least 600 s left of the wall-clock limit across resumes (jobs/src/preflight.rs). One unmet requirement prints every reason to stderr, records them in state.json's preflight field, runs no stage and exits 9 -- distinct from 7 (publish failed) and 8 (stopped), and not resent by jobs/scripts/poll-and-land.sh, since a missing requirement is still missing a minute later. The exception is a dependency that is only down: when the model gateway does not answer (connection refused, a timeout, a 5xx) or the NAS share is not mounted or not writable, and nothing else is unmet, the check waits (10 tries, 15 s doubling to 75 s, about ten minutes; JOBS_PREFLIGHT_WAIT_TRIES and JOBS_PREFLIGHT_WAIT_SECONDS) and then exits 12, which the poller resends uncounted like 7 and 3 (docs/autonomous-loop.md -> "What a job does when a dependency drops out"). A gateway that answers 401, 403 or 404, a share without its directory and every other unmet requirement stay 9. Run the same check by hand with job-check preflight <manifest> [--workdir <dir>] [--state <dir>], exit 0 when every requirement is met, 9 otherwise (it waits the same way); --workdir defaults to the current directory and {tools} in a stage's command resolves under its jobs/scripts. This check only asks what the lane itself can answer right now (a binary, a socket, a directory); whether a model or provider is currently available on the rig is the separate inventory check that follows it.

• The inventory check. Once the requirements check passes, the run step's job-run --inventory ~/folknet/inventory.json copies the lane's snapshot into the state directory (again on every resume), hands every stage JOB_INVENTORY (the copy) and JOB_INVENTORY_LIVE (the lane's own path), and checks the manifest against it and the declared file tools/folknet.json (the submitting checkout's bench/folknet.json). A model that is not ready, or a snapshot older than 15 minutes or missing, prints every reason, records them in state.json's inventory field, commits a not-run result locally (not published), runs no stage and exits 3, which jobs/scripts/poll-and-land.sh resends uncounted. A refusal that never clears by itself -- a model retired or not declared, a provider not declared or not allowed -- is reported the same way but commits a failed result and exits 10, which the poller records as failed without resending. The same check by hand: job-check inventory <manifest> <snapshot> [--declared <file>] (docs/folknet-inventory.md -> "The check before the first stage").

• Installed as a plain binary. The lanes have no container runtime (unprivileged LXC, no docker or podman, no passwordless sudo; rootless podman would also need /dev/fuse and /dev/net/tun added to the guest's config on its node), so Dagu runs as the lane user from ~/dagu: bench/dagu-lane.sh --lane N up installs the pinned release (its sha256 checked against the pin and the release's own checksums.txt), pushes the gateway key, starts it and checks it. stop, then deleting ~/dagu, removes it. autostart on adds a lane-user @reboot crontab line, which has brought Dagu back after a guest restart.

• Reached only through ssh. Dagu binds to the lane's loopback (WEBENGINE_DAGU_PORT, default 18431) with basic auth. The password is generated on the lane, kept in ~/dagu/home/config.yaml (mode 600) and read over ssh when needed, never stored on the development box. tunnel holds a port forward open for the web UI, /api/v1 and /mcp. The MCP server is the /mcp route of dagu start-all (Dagu 2.x has no dagu mcp command). The one MCP client is the curl-based one in bench/dagu-lane.sh (run, stop-run, watch).

• Secrets never go through run parameters. Dagu keeps a run's parameters in its run records on disk, so run refuses a parameter whose name contains KEY, TOKEN, SECRET, PASSWORD, PASSWD or CREDENTIAL. The gateway key goes by key over ssh on stdin to ~/dagu/home/model-key (mode 600); start exports it as WEBENGINE_MODEL_API_KEY with DAGU_ENV_PASSTHROUGH=WEBENGINE_MODEL_API_KEY, so every step sees that one variable and it lands in no run record or log. Steps start from a non-login shell and inherit nothing from ~/.lane-env: the gateway address comes from ~/.lane-model-env, the npm cache and the share from ~/.lane-shared-env (one variable at a time), and the forge through the lane mirror's origin URL, which carries the credential; git messages from forge commands pass through a filter that replaces anything shaped like a URL.

• How Dagu 2.17.2 behaves. Workflow keys are snake_case (retry_policy, hist_retention_days); camelCase is refused. One dagu_execute call waits at most 300 s, so run follows a longer run by reading its status every 15 s until it ends or WEBENGINE_DAGU_WAIT seconds pass. Dagu records no exit code for a step: watch --once --step <step> takes 0 for a succeeded step and otherwise the N of the step's error text exit status N, and - when there is none (a step killed by a signal). The exit handler (handler_on: exit) runs after every run, succeeded, failed or aborted, but not after a run cut off by a lane restart or a KILL of Dagu.

• The queue. A workflow's own max_active_runs above 1 is ignored for local queues; the limit is a global queue in ~/dagu/home/config.yaml (queues.config[].max_concurrency, written by set_queue(): queue jobs, JOB_QUEUE_LIMIT in bench/dagu-lane.sh at once, 1 today, for the reason in docs/job-concurrency.md), read only when the scheduler starts, so a change needs stop, start and autostart on again (the last rewrites the limit ~/dagu/autostart.sh passes after a lane restart). A lane's ~/dagu/home/queue-limit holds the limit it was last started with; a lane without that file was started before set_queue() wrote it, and its config.yaml may still hold an older limit. Only enqueued runs pass through it (dagu_execute action=start runs at once), so run enqueues any workflow file with a top-level queue: key. Every submission sends its own copy of the sources: run hashes the files it sends into a 16-digit id, copies them to ~/dagu/src-<id> and ~/dagu/scripts-<id> (sync_source(), sync_scripts()) and passes the id as the run parameter SRC_ID, so a later submission never replaces the copy under an earlier run's build. The same files give the same id and reuse the copies. prune_sources() removes a copy that no job directory names (jobs-work/<name>/src_id) and that was last used more than a day ago (WEBENGINE_DAGU_SRC_GRACE_MIN).

• What it keeps. Run history 7 days (hist_retention_days, set in ~/dagu/home/base.yaml by set_retention() and in job.yaml): the forge holds each job's result. A job's directory ~/dagu/jobs-work/<name>/ stays until the job passes, so a new run with the same parameters resumes it, and after a pass it stays too when its logs did not reach the NAS share (step cleanup removes it only once the archive wrote its marker archived-<Dagu run id> there, and says in its log when it kept it); while KEEP_JOB_OUTPUT is yes (the default, see "Cleanup") it keeps the directory after a pass too. Nothing of a job is moved aside. A restart archives the old work to the NAS share as archive/job-work/<name>/restarted-<time>.tar.gz (without target/, node_modules/, any .git/config or pi's auth.json; on to the NAS through jobs/scripts/archive-to-nas.sh on a machine whose share is a local directory) and only then removes it, at two points: clean-jobs <name> (which jobs/scripts/submit.sh --restart runs) for the whole job directory, refusing the restart with exit 3 and a line starting RESTART REFUSED when the copy fails; and prepare for the repo, state and manifest it clears on RESET=yes or a new base or spec, failing the step with a line starting PREPARE FAILED when the copy fails. Either way a failed copy keeps the directory in place. With KEEP_JOB_OUTPUT=no, prepare removes without archiving. clean-jobs still moves runs/<name>/ to runs/.restarted-<name>-<time>/. Of the restart tars and of those, the newest two per name stay (jobs/scripts/retention.sh, see "Retention of job-run folders"). The job tools are built from the copy of the sources run sends to ~/dagu/src-<id> (sync_source()), not from the job's base commit.

• The mirror keeps deleted job refs. The lane mirror's fetch (refspec +refs/*:refs/*, no prune) copies every job/* and result/* branch from the forge and keeps it after the forge deletes it. prune-mirror fetches and prunes those two namespaces only; bench/dagu/fixture-base.sh --delete runs it.

Landing approved spec tracks from a tick #

bench/land.sh <spec-name> <spec-sha> <branch-tip> lands an approved track without a person; the spec-watch tick calls it (tracks.json → specWatch.landScript). Exit codes: 0 landed, 1 not landed (reason on one line), 2 rebase conflict outside handoff.md. The contract: ~/.claude/docs/working-modes.md → "Let a tick land tracks".

flowchart TD
  start["tip and spec sha match<br/>what was reviewed?"] -- no --> refuse["refuse"]
  start -- yes --> scope["diff inside the spec's touches,<br/>no protected file?"]
  scope -- no --> refuse
  scope -- yes --> rebase["rebase onto main<br/>(handoff.md → main's roster)"]
  rebase --> gate["verify + landChecks<br/>(tests, 227 floor, per-function floors)"]
  gate -- fail --> refuse
  gate -- pass --> merge["fast-forward main<br/>(retry if main moved, max 3)"]
  merge --> after["tag spec/name/sha, remove worktree<br/>and branch, update roster"]

• Protected files. A diff touching bench/, tracks.json, .github/, specs/, tasks/, or at any depth .cargo/, rust-toolchain*, .env*, .gitattributes, .gitmodules or *.floor.json (a generated function's score floor) is refused whatever touches says, since those change the gate itself. So is a spec whose touches matches everything or reaches those paths. Case-insensitive, no rename detection. Such a track waits for a person's /land.
• What the gate still trusts. The track's own crates, tests, build.rs, oracle/ and Cargo.toml/Cargo.lock. The gate catches regressions and honest mistakes, not a track built to fool it.
• The checks. verify (bench/verify.sh: the tests, then every generated function scored against its own corpus, each at or above its floor in <name>.floor.json), then landChecks: bench/fragment-floor.sh (fragment_check --random must pass 227), which runs through bench/on-lane.sh (--lane-only, like verify, when WEBENGINE_LANE_STRICT=1; see "Build lane"). The four hand-written floors that bench/seam-floor.sh used to hold (floats, stacking order, relative positioning, block height) are floor files like the rest, and that script and its checker are gone. Floors are raised after a landing, never lowered by a track: a diff that touches a *.floor.json is refused.
• Safety. Pinned SHAs re-checked right before a --ff-only merge; a lock on the git directory; signals kill the gate's process tree and restore the branch. A land-pending/<name> marker makes a rerun finish anything owed after a merge. Nothing is pushed.
• handoff.md. A landing restores main's roster on the branch before the gate, so a track's working note never overwrites the roster.
• Install after a landing. If oracle/package*.json changed or a crate was added, only bench/oracle-install.sh runs, staged and swapped in on success.
• Testing. bench/land-test.sh drives the script through about 130 assertions in a throwaway clone. bench/land.sh --check ... runs every condition and stops before the merge.

Secrets #

The source copy of each secret is an item in the Proton Pass vault folknet; .env (gitignored) is the working copy every script and cron job reads, so nothing that reads it changes. bench/env-from-vault.sh rewrites the six secret values in .env from the vault and keeps every other line; --check reports drift without writing, and neither prints a value. Rotate a secret in the vault, then run the script. It needs a logged-in pass-cli, which keeps its local key in a file (PROTON_PASS_KEY_PROVIDER=fs, exported by ~/.profile and ~/.bashrc above its interactive check, and set by the script) because the kernel keyring it uses by default is revoked when the login session ends; a shell that lacks the variable creates a database keyed from the keyring, and the next call under the file store cannot open it ("file is not a database"). When the session is unusable the script resets it and logs in again from a personal access token kept in a mode 600 file (~/.config/folknet/pass-pat, or FOLKNET_PASS_PAT_FILE), if that file exists; otherwise a person runs pass-cli logout --force and pass-cli login. bench/env-from-vault-test.sh covers the script with a stub.

Training limits #

Full fine-tuning is out of reach on 24 GB Ampere cards; LoRA or QLoRA on a 30B-class model across eight GPUs is feasible, so any supervised step is a LoRA.

Sharding without a broker #

Case generation is deterministic from a seed, so worker i of n takes seeds i, i+n, i+2n, …: no queue, no coordinator, and any worker can restart without losing or duplicating work. Output is content-addressed and written once; ground truth is immutable per engine version, so there is nothing to invalidate.

Storage #

About 10 KB per document for three engines, so ten million cases is about 100 GB. The style subset dominates and is redundant; compress it, sample it, or store it as a delta.

Virtual machine template #

One image, duplicated: WebKit's 188 system packages, pinned Playwright browsers, Rust with sccache, the repository. Workers are stateless. Pin one VM per NUMA node rather than one large VM per host.

Engine versions pin the ground truth #

Ground truth is immutable only for a given set of browser versions, so every dump records engine_version (docs/dump-format.md → "Dump") and a corpus is keyed by that version set. A browser update invalidates only what it touches, never silently.

Rig change log #

Every change tried on the rig's hardware, worked or not, newest last. Hosts by role and class only. Before a change, and again after each change that works, bench/rig-snapshot.sh snapshot <label> records the serving configs of every role (proxy host HAProxy, gateway container config and key database, each ARM node's units and TabbyAPI configs, the four-card VM's Proxmox config and units) on the host itself and in the local rig notes; restore <label> puts them back. The label named in an entry holds the working state after it. Snapshots contain the gateway's master key and never go into git or onto the NAS.

2026-09-24

• Gateway key for self-hosted writers. A new virtual key on the gateway, alias nev-folknet-writer, allowed only qwen3.8-27b-long and qwen3.8-27b-fast, so writer traffic is accounted apart from batch traffic and cannot reach any other alias. No serving config changed. The key itself lives outside every repository, as FOLKNET_WRITER_API_KEY in $HOME/.config/folknet-writer/env (mode 600) on the development box. First used by stage 0 of docs/folknet-writer-design.md (docs/writer-harness-spike.md → "Rerun through Pi"). Worked.
• Snapshot and restore. bench/rig-snapshot.sh written; known-good-2026-09-24 taken before any serving change. restore known-good-2026-09-24 haproxy run once as the round-trip test: config check passed, HAProxy reloaded, all four ARM replicas still up. Worked.
• Four-card VM. A new VM on a Proxmox node, given the four 3090s of a stopped tenant test VM (same hostpci entries, q35, OVMF, 256 GB 64-bit PCI hole), 32 cores, 160 GB RAM, a 120 GB OS disk, DHCP on the main LAN (bench/gpu-vm/create-vm.sh). The tenant VM's disks were not touched. Worked; four 3090s visible.
• Provisioning, first two attempts failed. First: the fresh cloud image was still running cloud-init's package upgrade, which held the dpkg lock and then rebooted the guest. Second: the rig's apt cache answers 403 CONNECT denied for HTTPS repositories, so docker's and NVIDIA's apt repositories could not be reached through it. Fixed in bench/gpu-vm/provision.sh: wait for cloud-init and the dpkg lock, send HTTPS repositories direct, take docker from the Ubuntu archive (docker.io, which the cache serves). Third run worked: driver 595 (open kernel modules) loaded without a reboot, nvidia-smi shows four 3090s, docker has the NVIDIA runtime.
• Model choice for the four cards, by size (no download attempted).
  • qwen3.8-flash-next, EXL3 4.05 bpw (the ARM nodes' checkpoint): 101 GB on disk, of which 39 GB is the hashed n-gram table that exllamav3 keeps in host RAM or on disk, so about 62 GB of weights would sit on the cards. That the checkpoint does not fit 96 GB was assumed, never tested. A 3.05 bpw branch is also published (85 GB on disk, 32.6 GB of it n-gram table). Not pursued: the target moved to a different model family for variety.
  • DeepSeek-V4-Flash (the July release, 284B parameters, 13B active): 167 GB native (MXFP4 experts). Mainline llama.cpp with CUDA serves it on sm86 from GGUF. The smallest quant its quantizer recommends (3-bit, 104 GB) does not fit 96 GB of VRAM; 2-bit quants (87–91 GB) fit with only a few GB left for cache and compute buffers, and re-quantizing the native 4-bit experts to 2 bits costs a lot of accuracy (published KL divergence around 0.4 at 2 bits against 0.01 at 4 bits). Download straight from Hugging Face inside the VM measured about 250 KB/s. Not tried.
  • DeepSeek-V4.1-Flash is a different, larger model, not a revision of the one above: a new architecture (deepseek_v41, 384 experts over 40 layers, n-gram memory tables), 510 GB native. It does not fit the four cards. The first reading, that it fits no pair of ARM nodes either, was wrong: it looked only at GGUF quants (smallest about 260 GB) and missed a published EXL3 recipe for exactly two GB10s — a 2.9 bpw EXL3 checkpoint (~197 GiB, about 99.5 GiB of weights per node at tensor-parallel 2) plus the model's unquantized n-gram ("Engram") tables, which stay on disk and are read per lookup rather than held in memory (two native shards, ~190 GiB), served by a vLLM build that carries the deepseek_v41 architecture. That recipe is what the ARM pair uses (entries below).
  • qwen3.8-27b, W4A16 AutoRound with DFlash2 speculative decoding, the eight-card host's -fast recipe: about 22 GB per card, one worker per card. Chosen.
• qwen3.8-27b on the four cards. The eight-card host's image and prepared checkpoint (24 GB) were streamed to the VM read-only (tar and docker save piped over ssh, nothing on that host changed). One worker per card came up in about 6.5 minutes from cold; all four report healthy and answer chat completions. Worked; snapshot vm-qwen27b-workers-up.
• Proxy and gateway for the VM. A new HAProxy frontend and backend on the proxy host (the existing backends untouched; backup haproxy.cfg.pre-vm-qwen27b-20260924 beside the config) and a gateway alias qwen3.8-27b-fast-2 (backup config.yaml.pre-qwen27b-vm-20260924), granted to the project's key only. The first gateway attempt failed with "Missing credentials": an OpenAI-style entry needs some api_key even when the upstream checks none; sk-none, as the flash-next entry uses, fixed it. Worked: the alias is listed and returns code through the gateway; snapshot vm-qwen27b-fast-2-alias.
• Measurement batch at qwen3.8-27b-fast-2 on a build lane: one 32-candidate absolute-positioning batch, 2/32 compiled, best 74/108 (docs/model-first-cost.md, row vm-batch 1). Stopping the loop after batch 1 with pkill -f on the shared lane also killed another track's run of the same binary in its own lane worktree (a mixed-model batch at 32 of 64 draws). On a shared lane, stop a run by the pid whose /proc/<pid>/cwd is this worktree's lane copy, never by name.
• Checkpoint fetch for an ARM pair. bench/sparks/hf-to-nas.sh (a runtime bundle and oneshot unit on the downloader host, reusing that host's multi-WAN scripts) started DeepSeek-V4-Flash (July release, 167 GB); stopped after 14 GB when the ARM target changed to DeepSeek-V4.1-Flash. The partial staging directory is left on the NAS. Then both parts of the V4.1 recipe, pinned by revision: the EXL3 2.9 bpw checkpoint (42 files, 210.7 GB) and the two Engram shards plus index and config of the native checkpoint (203.1 GB), both through the multi-WAN downloader at about 130 MB/s combined. Snapshot before: pre-deepseek-sparks.
• Which two ARM nodes. LLDP shows each node's first 200G port on the switch and its second port cabled straight to one other node, in two cabled pairs, each crossing the old GLM pairs (which ran over the switch). The DeepSeek pair is one of the cabled pairs; the other cabled pair keeps flash-next. Its tensor-parallel traffic is set to run over the switch port with the RoCE settings the GLM pairs used, because the direct cable carries no addresses today; moving it onto the cable means addressing that port on both nodes (not done).
• Staging the pair (nothing serving yet). The recipe is cloned at a pinned commit on the head node, its .env written from bench/sparks/dsv41/env.template (weights from local disk, no Hugging Face download, AUTO_DOWNLOAD=0), an ssh alias for the worker added to the head's admin account (bench/sparks/dsv41/ssh-config-worker; previous config kept as config.pre-dsv41), /srv/dsv41 created on both nodes, and the recipe image pulled on both (the 2.9bpw tag, 22.5 GB; digest in the local rig notes). Both nodes still serve flash-next. Snapshot dsv41-staged.
• Throughput sweep of qwen3.8-27b-fast-2 (bench/rig_throughput.py) on a build lane: failed twice before sending a request — the lane's Python has no requests module, and python3 -m venv fails there without the python3.12-venv package. Not run.
• Four-card VM reboot. All four workers healthy three minutes after reboot, same DHCP address, HAProxy backends back UP without intervention. Worked.
• Second measurement batch at qwen3.8-27b-fast-2, after the prompt fixes, run with bench/absolute-mixed-batch.sh on a build lane: 8/32 compiled, best 82/108, 233 s for the batch (docs/model-first-cost.md, row vm-batch 2). The run ended by itself (GENERA_MAX_BATCHES=1); nothing needed stopping. No serving change.
• Throughput sweep of qwen3.8-27b-fast-2, second attempt, run from the dev box (which has requests) through the gateway, with the prompt generated on a build lane: worked, 819 output tok/s at 16 in flight (docs/rig-capacity.md → "Measured — the four-card VM"). No serving change.
• Abandoned DeepSeek-V4-Flash download removed from the downloader host: the 14 GB partial staging directory on the NAS, its oneshot unit and runtime bundle. Worked; the two V4.1 units and their published trees are untouched.
• DeepSeek-V4.1-Flash weights to local disk on the ARM pair. Both NAS downloads had finished and published. rsync -a from the NAS mount to local NVMe on both nodes, the EXL3 checkpoint (39 shards, 197 GB) to each, the two Engram shards with index and config (190 GB) to the head's engram source directory and to the worker's engram directory. The worker gets no second engram tree: the recipe's start.sh builds the slim engram set on the head and rsyncs it to the worker, and with the NAS mtimes kept, that rsync finds the shards already in place. About 100 MB/s per node, both done in about 55 minutes. Both nodes kept serving flash-next throughout, all backends UP after. Worked; no serving change, so no new snapshot (dsv41-staged still holds the serving state).
• Two ARM nodes out of the flash-next pool. Snapshot before: pre-dsv41-pair-move. Their two server lines were removed from the flash-next HAProxy backend (backup haproxy.cfg.pre-dsv41-pair-20260924 beside the config; config check, reload; the other two replicas UP), then their flash-next units disabled and stopped; unit, config and checkpoint stay on disk. Nothing was drawing from the alias at the time. The alias now takes 16 requests in flight (two replicas × 8), not 32. Worked; snapshot flashnext-two-replicas.
• DeepSeek-V4.1-Flash on the pair. The published recipe image carries the recipe's own build stamp from four commits before the pinned one, so the pinned start.sh would have compiled the image on the node. The only image-side change in between is one Python patch, which that start.sh bind-mounts into both containers anyway, so the unit runs the published image as-is (SKIP_BUILD=1, reason in the unit file). Unit installed on the head and started: preflight passed (memory headroom 117.6 GiB available per node against 111.5 needed, RoCE GID accepted), the weight sync to the worker found everything already in place and took 3 s, weights loaded and warmed up, API healthy about 7.5 minutes after the start; the README's 17*19 request returned 323. The recipe's own warmup reports one sampler-kernel variant not pre-compiled ("may JIT mid-serve"); harmless so far. Worked; snapshot dsv41-pair-serving.
• Proxy and gateway for the pair. A new HAProxy frontend and backend on the proxy host, one server, maxconn 2 (backup haproxy.cfg.pre-dsv41-backend-20260924), and a gateway alias deepseek-v4.1-flash → the recipe's served model name, api_key: sk-none (backup config.yaml.pre-dsv41-20260924), granted to the project's key only. Worked: the alias is listed through the gateway and a genera-shaped chat completion (thinking off) returned a one-function Rust code block in 2.7 s with no reasoning in content. Snapshot dsv41-alias-serving.
• First measuring batch at deepseek-v4.1-flash on a build lane, 8 candidates at 2 in flight: 4/8 compiled, best 82/108, 1,731 s (docs/model-first-cost.md, row dsv-batch 1). The first launch failed before drawing anything: the lane's shared git mirror refused a ref update (cannot lock ref 'refs/heads/main') because another run had moved main at the same moment; a plain retry worked. No serving change.

2026-09-25

• Dead GLM aliases removed from the gateway. glm-5.3-flash and glm-5.3-flash-exl3 were still listed by /v1/models three days after their backends stopped, so a caller could pick one and get a 503 ("No server is available to handle this request"), as a job on another track did. Snapshot before: pre-glm-alias-removal. Both model entries deleted from the gateway config (backup config.yaml.pre-glm-removal-20260925 beside it; no fallback named them) and the gateway service restarted, live again in 18 s. The two names were then dropped from the allowlists of the three virtual keys that carried them, each keeping its other models. Worked: /v1/models through the dev-box tunnel lists neither name; one-token completions return 200 for qwen3.8-27b-fast, deepseek-v4.1-flash, qwen3.8-27b-fast-2 and qwen3.8-flash-next; a request for glm-5.3-flash is refused with 403. Snapshot glm-aliases-removed. The site file the gateway config is generated from still carries the GLM entries, so a regeneration would bring them back.

2026-09-29

• deepseek-v4.1-flash back after the outage reboot. It had answered 503 since 2026-09-28: the pair unit ran before the head's fabric address came up, start.sh refused ("HEAD_IP ... is not assigned"), and a oneshot unit is not retried. Snapshot before: before-dsv41-fix. folknet-glm53-vllm was enabled on all four ARM nodes and crash-looping; disabled and stopped on all four. The pair unit gained an ExecStartPre that waits up to 300 s for the HEAD_IP its .env names (bench/sparks/dsv41/folknet-dsv41-exl3-pair.service), installed on the head and started: preflight saw 117 GiB free on both nodes, and the unit finished in 7 minutes. Worked: a completion for deepseek-v4.1-flash through the gateway returned 200 in 1.6 s. Snapshot dsv41-pair-waits-for-head-ip. Unverified: the wait has not yet run across a real reboot.