This repository has no description

Fork 0

bez docs infrastructure.md

56 kB

Markdown

at track/substrate-composition

dietrich ayala · Keep a passed job's folder until its logs are archived, and archive events.jsonl · 3w ago

0099e3d8

Infrastructure #

The machines the generation loop runs on, and how work reaches them. The loop itself is in docs/generation-loop.md; measured rates are in docs/rig-capacity.md. Hardware is named by role and class, never by hostname, address or account — this repository is assumed public. Real names live in the gitignored bench/*.local.json and the local rig notes.

In brief #

The loop. One engine function per spec algorithm (for example the CSS 2.1 width calculation), written by a model, checked against cached browser results.
Sampling tier. GPU cards running mid-sized models write N versions of a function from the spec text, the engine's Rust interfaces and the currently failing cases. Many cheap tries beat one careful one, because the browsers say which tries are right.
Capacity tier. When every version fails, a larger model gets the spec, the best failing versions and the exact per-box geometry differences.
Verification tier. CPU servers compile candidates in one batch and run each inside the engine over the cached corpus, under a millisecond per page. The same tier builds that cache by running the three browsers once per page, and runs conformance suites.
Outcomes. Keep the passing version, sample again with the new failures, or escalate. What is kept is ordinary Rust with a provenance record; no model runs in the shipped engine.

Environment variables are named NEV_*; older records use GENERA_*, which every reader still accepts (docs/rename-to-nev.md).

Detail #

The full cycle #

flowchart TD
  browsers["Chromium, Firefox, WebKit<br/>once per page (CPU)"] --> cache["cached ground truth<br/>three-engine vote"]
  spec["spec text<br/>one algorithm"] --> sample["SAMPLE (GPU pool)<br/>N versions of one function"]
  sample --> verify["VERIFY (CPU)<br/>dedupe, batch compile,<br/>run over the corpus"]
  cache --> verify
  verify -- "all pass" --> accept["commit Rust source<br/>+ provenance record"]
  verify -- "some fail" --> sample
  verify -- "all fail repeatedly" --> escalate["ESCALATE<br/>larger model + geometry diffs"]
  escalate --> verify

Nothing is accepted on a model's word: escalated output goes through the same check. Writing candidates is the binding constraint; verification has a large margin over it.

What comes out, and how the engine uses it #

The cycle outputs ordinary Rust source, committed next to the hand-written code it plugs into, with spec steps as comments and a provenance record (roadmap.md → "Provenance is a first-class artifact"). The engine compiles it like any other code; the GPU and CPU fleets exist only at build time.

Provenance drives maintenance. When a spec section or a definition it uses changes, the provenance index finds exactly the functions to regenerate (roadmap.md → "A specification changes"); a browser update re-dumps the ground truth instead. The same index is the bill of materials for a configurable build (roadmap.md → "Phase 4").

Measured baseline #

One oracle process, three engines, 235 documents including browser startup: 17.9 s, about 13 documents per second per engine, using about two cores. Scaled to a few hundred cores that would be about 3,000 documents per second, a ten-million-case corpus in about an hour — but per-core throughput on the contended 16-core development box fell from 1.48 to 0.48 docs/s/core between 1 and 16 workers (docs/rig-capacity.md → "Oracle corpus building"), so the figure on dedicated hardware is still to be measured. GPU-side figures: docs/rig-capacity.md → "GPU tier".

Three tiers #

Sampling tier — high memory bandwidth GPUs. Bulk best-of-N generation, throughput-bound. Decode speed tracks active parameters, so a mixture-of-experts model with a small active fraction gives several times the candidates of a dense model. Ampere cards have no fp8, so quantization is 4-bit weight-only.

Capacity tier — large unified-memory accelerators. About a third of the bandwidth, several times the memory: for the few hard calls and any teacher model. About 100B total with ~12B active is the realistic ceiling.

Verification tier — many CPU cores. Corpus building, candidate compilation and the inner loop. It must never spawn a browser per case.

As discovered #

Every host advertises itself over mDNS and answers an HTTP inventory; a collector aggregates it. What that shows, by role:

The model gateway is a LiteLLM instance fronting one alias per model pool. The aliases, their pools, how many requests each takes at once, and which are retired are declared in bench/folknet.json (the two GLM-5.3 Flash aliases, without backends since 2026-09-22, were removed from the gateway on 2026-09-25); whether each one answers right now is in each lane's snapshot, which a probe refreshes every 5 minutes and bench/folknet-status.sh [--lane N] prints (status, how long it has held it, last answer, snapshot age). Design: docs/folknet-inventory.md. Virtual keys carry per-key allowlists, so a new alias must be granted to the caller's key. Each pool's HAProxy reports liveness, not readiness: a /v1/models answer does not prove a completion will return.
Four ARM unified-memory nodes (DGX Spark class) — aarch64, one GB10 GPU, 20 cores, 121 GB RAM, 3.7 TB NVMe each. They served GLM-5.3 Flash as two independent two-node pairs until 2026-09-22. Qwen3.8-Flash-Next did not run under any vLLM build on the rig (the architecture was too new for the rig's EXL3 fork, and upstream has no EXL3 backend), so it serves through the checkpoint's own engine (TabbyAPI) instead, one replica per node, about 75 GB plus ~45 GB of page cache each. Two of the four now serve DeepSeek-V4.1-Flash. Full account: docs/rig-capacity.md → "Attempted — Qwen3.8-Flash-Next".
The eight-card host — eight 24 GB consumer cards on a 64-core EPYC with ~500 GB RAM, bare Linux, six serving containers using every card (four behind -fast, two behind -long). The closest thing to the sampling tier as designed. No spare GPU.
The verification tier — ten Proxmox 9.x nodes (EPYC 7742, 7B13, 7K62 and 7452; 32 to 64 physical cores, 125 GB to ~500 GB RAM, 1–2 TB NVMe), shared with tenant VMs; five pass thirteen consumer cards through to tenant machines. A VM sized for browser builds sits on the GLM-pool proxy host (open: its core count and whether it should be running). Free capacity is host by host.
A downloader host pulls checkpoint shards in parallel across the site's four WAN uplinks onto the NAS, verifies them and publishes a pinned revision for nodes to sync.
Access. Root with the jump host's key works on every physical host; the ARM nodes use their own admin key. The gateway's own guest refuses, and is not a compute host.

Where each piece runs #

Every role is a guest or a container; nothing is installed on a Proxmox node's own OS.

flowchart LR
  dev["development box<br/>16 cores, shared"] -- "ssh via jump host" --> lanes["2 build lanes<br/>(LXC, 96 threads each)"]
  dev -- "port forward" --> gw["model gateway<br/>(LiteLLM)"]
  lanes --> gw
  gw --> fast["eight-card host<br/>-fast, -long"]
  gw --> vm["four-card VM<br/>-fast-2"]
  gw --> fn["2 ARM nodes<br/>flash-next"]
  gw --> ds["2 ARM nodes<br/>deepseek pair"]
  lanes --> helpers["compile helpers<br/>(sccache-dist)"]

Development box. No GPU, no rig role; holds the port forwards to the gateway.
Jump host. A VM on one node; the only route in. Nothing of this project on it.
Build lanes. One unprivileged LXC on each of two nodes (bench/lane-create-lxc.sh, bench/lane-provision.sh), started at node boot. A gate running when a lane stops fails.
Compile helpers and scheduler. Four more containers plus builders on the two lanes; scheduler, redis cache and a loopback CONNECT proxy on one lane (bench/lane-dist.sh).
Eight-card host. Docker containers under a templated unit folknet-qwen38-vllm-worker@<profile>, behind that host's own HAProxy on port 4000. Not edited by this project.
ARM nodes, split two ways since 2026-09-24.
- Two serve qwen3.8-flash-next, one systemd unit and container each on port 5000 (bench/sparks/), behind the proxy host's HAProxy (least connections, eight in flight per replica) — so the alias takes 16 in flight, not 32; a batch at concurrency 32 queues the rest. Either replica can stop alone.
- Two, joined by a direct cable, serve deepseek-v4.1-flash as one tensor-parallel pair: one oneshot unit on the head node (bench/sparks/dsv41/folknet-dsv41-exl3-pair.service) starts both containers and returns once the API is healthy (about 7.5 minutes cold); stop and start only through that unit, never one node alone. HAProxy maxconn 2.
- Units are stopped and started through a privileged container that chroots into the host, because the admin account has no sudo. GLM units are disabled on all four.
Four-card VM (qwen3.8-27b-fast-2). Four 3090s borrowed from a stopped tenant VM on the same node (only one of the two may run; bench/gpu-vm/). Four units folknet-qwen38-vllm-worker@fast-{0..3} on ports 8101–8104, the eight-card host's -fast recipe, HAProxy maxconn 4 per worker (16 in flight). A separate alias because the -fast load balancer lives on the eight-card host.

Open. Which host carries the four-card VM behind qwen3.8-27b (the measured Qwen server), and how it is started and stopped; whether anything restarts the eight-card host's HAProxy or re-pulls its image; what keeps the development box's port forwards up after a reboot; and the gateway config is generated from a site file outside the rig, which needs every hand-added alias too.

The Proxmox nodes are shared with other tenants. Three rules:

Nothing of this project goes on a node's own OS, so a guest can be stopped and its hardware handed back without touching the node.
GPU work runs in a VM with passthrough, never an LXC (an LXC would need the vendor driver on the node). CPU work runs in an unprivileged LXC, or in docker on non-Proxmox hosts.
Stopping a project guest frees what it held, and a lane competes with tenants as an equal under contention. Nothing checks this; it is how the rig is laid out.

Limits today: the ARM replicas' HAProxy queues a request at most 30 minutes; power budgets are modelled (bench/capacity.py) but not enforced.

Not built yet: a per-role start/stop/drain wrapper; a drain (a replica's stop gives in-flight replies only 60 s); taking a backend out of HAProxy before stopping it; a narrow sudo rule for the ARM nodes' admin account.

Build lane #

Two unprivileged containers (96 logical CPUs — three quarters of a 64-core, 128-thread node — 96 GB, 200 GB disk), each a build lane and oracle host: the Rust toolchain, Node 22, the repository cloned from the rig's forge ("Where the repository is stored" below), Playwright's three browsers, and a PulseAudio null sink so a realtime AudioContext runs. The scripts are the record and are re-runnable: bench/lane-create-lxc.sh, bench/lane-provision.sh, bench/lane-nas.sh.

Running a gate there.

sequenceDiagram
  participant W as worktree (dev box)
  participant L as lane
  participant F as rig forge
  W->>L: ssh: git bundle of unpushed commits + diff + untracked files
  L->>F: fetch the lane's mirror (only if needed, or older than 10 min)
  L->>L: lane-apply.sh: check out exact commit, apply diff, run command
  L-->>W: streamed output (and --pull directories)

bench/remote-lane.sh <worktree> <command...> does this. Each lane root keeps one bare mirror (<root>/.git-mirror) with one git worktree per track; it fetches only when the bundle cannot supply the commit or the last fetch is older than NEV_LANE_FETCH_MAX_AGE (600 s), because the public forge the mirrors first fetched from rate-limits. The lane address comes from NEV_LANE_SSH or bench/lane.local.json; the mirror URL from NEV_LANE_GIT_URL, that file, or the worktree's origin. Examples:

bench/remote-lane.sh . cargo test --workspace
bench/remote-lane.sh --pull corpus/absolute-ground-truth . \
  cargo run --release -p generation --bin absolute_ground_truth_cache

Why a bundle rather than rsync: uploads from the development box to the lane run at about 18.6 KB/s (downloads 435 KB/s), so rsyncing a worktree took ~315 s of a ~424 s run.

Where the repository is stored. Two forges, with different jobs:

forge	role	holds
the rig's Forgejo	intermediate storage, on the rig network	everything the lanes need: `main`, job branches `job/<name>` and their results `result/<name>` (`specs/job-submit-land.md`)
tangled	persistent storage, public	`main` and the `spec/<name>/<sha>` landing tags only

The Forgejo repository is a regular repository, not a mirror of tangled. The lanes clone and fetch from it: each lane root's .git-mirror and the lane user's clone fetch from http://<forge-user>:<forge-token>@<forgejo-address>/<owner>/<repo>.git, and bench/lane.local.json git_url (or NEV_LANE_GIT_URL) names the same URL. The user and token come from the primary checkout's .env (GENERA_FORGE_USER, GENERA_FORGE_TOKEN, still under the old prefix; docs/rename-to-nev.md). The development box reaches Forgejo through a local port forward, as the remote rig (http://<forge-user>:<forge-token>@127.0.0.1:<forward-port>/<owner>/<repo>.git), which the job scripts use by default (JOBS_FORGE_REMOTE). origin fetches from tangled and has push URLs for both forges, so git push origin main updates both; job and result branches go to rig only and are deleted there after a landing. Each of those URLs carries the token: they live in .git/config and the gitignored files, never in a committed file, a log line or a message.

The track gate runs only on the build lane. tracks.json → verify runs bench/verify.sh, which hands itself to bench/on-lane.sh --lane-only and, once there, runs cargo test --workspace, the job event store's tests (jobs/events/*.test.mjs under node --test, then bench/job-events-lane-test.sh), and then bench/generated-scores.sh. The other *.test.mjs and bench/*-test.sh files are run by hand, not by the gate. With no lane configured or none answering (ssh -o BatchMode=yes -o ConnectTimeout=5 fails) the gate exits 3 and runs nothing on the development box; a lane that answers but fails the tests or a score is a failed gate — no local retry. Measured before the scores were added: a new worktree's first test run 106 s (124 s with a fresh mirror), a warm rerun 22 s, local fallback 78 s. A clean cargo test --workspace compiles in 61 s on a lane and 57 s on the development box — the build is bound by the dependency chain through Stylo, not by cores; the lane's value is that it takes load off the shared box and runs several gates at once.

Unattended landings never fall back to the development box. bench/on-lane.sh without --lane-only runs its command locally when no lane answers, and bench/land.sh runs each landChecks command that way, and verify directly. Setting NEV_LANE_STRICT=1 in bench/land.sh's environment (the scheduled jobs/scripts/poll-and-land.sh gets it from tracks.json's specWatch.pollEnv) makes verify go through bench/on-lane.sh --lane-only too and every check --lane-only, so with no lane answering the landing is refused with "verify failed on the tip (exit 3: no build lane answered ...)" and nothing of the gate runs here. Unset, as in interactive use, a check keeps its local fallback. bench/verify.sh itself is lane-only either way; the one thing that would run it on the development box is NEV_ON_LANE set there by hand.

Already on a lane, bench/on-lane.sh runs the command in place. With NEV_ON_LANE set, bench/on-lane.sh (and bench/remote-lane.sh) runs its command in the named worktree on the machine it is on, --lane-only or not, reads no lane settings, and ignores --pull, since the outputs are already where they belong; the command's exit status is the run's. This is the same check bench/verify.sh makes, so a spec's bench/on-lane.sh --lane-only "$PWD" ... works unchanged inside a Dagu job, whose clone has no .env and no bench/lane.local.json and whose writer gets NEV_ON_LANE=1 from jobs/scripts/pi-session.sh. As with bench/verify.sh, a NEV_ON_LANE set by hand on the development box would run the command there. bench/on-lane-test.sh covers both sides. The cost: with both lanes down nothing lands until one answers. bench/land-test.sh scenario s1 covers both settings.

The gate scores every generated function against its own corpus. A committed file under crates/layout/src/generated/ can be wrong while every test passes, because the tests exercise in-tree reference rules, not the generated function. bench/generated-scores.sh clears and rebuilds each corpus with its *_ground_truth_cache binary (the browser oracle runs on the lane), then generated_score_check scores each generated function through its own runner and fails it against its floor. The floor is two numbers in crates/layout/src/generated/<name>.floor.json beside the function's provenance record: cases_passed, how many cases it passes, and cases_run, how many cases its corpus holds (only cases three engines agreed on). It fails when the passing count drops, when the corpus shrinks even though the passing count holds, and when a function passes no case at all (a lane missing a browser leaves fewer than three engines to vote, so every case drops; a floor of zero cannot hide a function that scores nothing). The provenance record describes the model batch that admitted the function and is never rewritten; the floor is never below its verification.cases_passed, and a function with no floor file yet is held to that alone. A placeholder record, such as the hand-written table one (0 recorded), therefore floors at its real score once a floor file exists.

Floors rise by themselves, after a landing. bench/after-land.sh ends with bench/generated-floors-update.sh, which runs bench/generated-scores.sh --print-raises on the build lane, writes each floor that the fresh score is above (a function with no floor file gets one), and commits only those files as one commit on main, the way the map and dashboard commits are made. It refuses on uncommitted changes, writes nothing when the lane does not answer or a score fails, never lowers a number, and a failure is one line and does not change the landing. Raising happens after the landing rather than in the gate because the gate runs on the lane's copy, where a commit cannot reach the track, and because a track never edits a floor: bench/land.sh refuses a diff that touches any *.floor.json, so an unattended track cannot lower one. The cost is a second scoring run on the lane per landing, and a window between a landing that improves a function and the raise in which the old floor holds. A floor comes down only by a person's reviewed edit of its file.

One place to add a generated function. A function is discovered from its <name>.provenance.json; its runner and the cache binary that writes its corpus are one entry in FUNCTIONS in crates/generation/src/bin/generated_score_check.rs, and a provenance record with no entry fails the gate with that instruction. bench/generated-scores.sh takes the caches to rebuild from generated_score_check --list-caches, so nothing under bench/ changes and a track adding a function can land unattended. bench/generated-scores-test.sh exercises the scripts, including the raise, without a browser.

Model batches run only on a lane. bench/absolute-mixed-batch.sh and bench/overflow-mixed-run.sh are started from the development box like any script (NEV_MODELS=qwen3.8-27b-fast:32:32 bench/absolute-mixed-batch.sh) and re-run themselves on the lane through bench/on-lane.sh --lane-only (bench/batch-lane.sh): the NEV_* settings go along, credentials and lane settings excepted, and the batch's results come back with --pull. With no lane configured or none answering the script exits 3 and runs nothing locally; bench/lane-apply.sh exports NEV_ON_LANE=1, which is how a script knows it is already there. Each run also records its phase timings and how many of the lane's CPUs each phase kept busy (bench/cpu-sample.sh, bench/cpu-phases.py; docs/rig-capacity.md → "A real batch on a build lane").

A stray directory at a worktree's lane path breaks the gate. bench/lane-apply.sh tests only for <lane-root>/<name>/.git; a leftover directory without it makes git worktree add fail, and the gate cannot pass until it is removed by hand (check git -C <mirror> worktree list first). Not yet fixed in the script.

Cleanup. Every worktree run on a lane leaves its copy (<root>/<name>, target/ and all, 3–12 GB) and <root>/.payload-<name> there; left alone they filled a lane's disk. bench/lane-sweep.sh removes, on every lane in bench/lane.local.json lanes[], the copies whose track has no worktree in the primary checkout (--dry-run lists them with sizes). It keeps the primary's own copy, any copy with a process running in it (a spec writer runs inside its copy), and directories that are not worktrees of the lane's mirror. Untracked data under a copy's corpus/ (links excepted: they point into the NAS already) goes to the NAS first, as one tar file at archive/corpus/<name>/<time>.tar; with no NAS directory the copy is kept. A branch a copy was on stays in the mirror. It runs at three points: by hand; from bench/land.sh after a landing, for that track only (a failure is named on its line and undoes nothing); and from bench/remote-lane.sh when the lane's root filesystem has less than NEV_LANE_MIN_FREE_GB (20) free before a run, which then refuses with exit 4 if the sweep did not free enough, naming the lane by its place in the list, the free space and the largest copies. A hand landing leaves no worktree, so the next sweep takes its copies. Scenarios: bench/lane-sweep-test.sh.

Test-run memory. A cold cargo test --workspace on the development box adds about 1.1 GB at peak (10.8 → 12.0 GB), a warm one 67 MB; seven concurrent tracks fit in its 30 GB if their peaks do not coincide.

Distributed compilation. sccache-dist builders on six nodes (416 cores registered) with redis as the shared cache. After dropping one builder on a node with slow outbound network (failures 84 → 1), it still lost on wall time: a clean -p dom release build took 82 s distributed against 45 s local on the same lane. The dependency graph is too narrow to use the cores. A wide build fares no better: a batch's candidate probe (one --keep-going build of 32–64 small crates) took 14–45 s through sccache-dist against 1.4–1.6 s with no compiler wrapper on the lane, just as slow on a second pass, so each compile through the wrapper carries a fixed delay far larger than the compile itself (docs/rig-capacity.md, "A real batch on a build lane"). The delay is the input upload: for a Rust compile the sccache client packages every library in the -L dependency= directory (target/release/deps, 225 rlibs, 336 MB), not only the crates the compile loads. One probe library compile (0.1 s with plain rustc) took 4.4 s distributed, sent 121.9 MB (the gzip size of that whole directory) and spent 2.3 s of client CPU compressing it. The cost grows with the dependency directory, not with the crate, and is paid again on every compile, which likely explains the -p dom result too (not measured). The probe therefore runs without the wrapper, on the lane's own cores; the development box's sccache config stays untouched.

Shared data on the NAS. Each lane bind-mounts one NAS directory at /srv/lane-shared (NEV_LANE_SHARED, also the npm cache). Login shells get the npm cache from ~/.lane-env; the Dagu job's prepare and run steps (bench/dagu/job.yaml) read only npm_config_cache from ~/.lane-shared-env, and bench/oracle-install.sh runs npm ci --prefer-offline, so a package is downloaded from the registry once and every lane and job after that installs it from the share. The NAS is fast from the lanes (about 92 MB/s write, 114 MB/s read) but slow for small files (~38 ms each), so target/ stays local. bench/lane-apply.sh links each worktree into it on every run:

in the worktree	on the share
`oracle/suites/checkouts/<suite>`	`suites/<suite>-<pinned commit>/`
`webgpu-cts/cts`	`suites/webgpu-cts/`
`corpus/<name>` for every `*ground-truth` cache the code names	`ground-truth/<name>/`
`oracle/suites/results`	`archive/suite-results/<worktree>/`
`webgpu-cts/results`	`archive/webgpu-cts-results/<worktree>/`
(the command's output, copied)	`archive/gate-logs/<worktree>/<time>-<guest>-<commit>.log`
(one line per model batch, `bench/ledger.py`)	`ledger/<worktree>.jsonl`
(a Dagu job's full logs, `bench/dagu/job.yaml`, scrubbed)	`archive/job-logs/<job name>/<Dagu run id>/`

The last row comes from the Dagu job workflow, not bench/lane-apply.sh: it copies each job's stage logs, state.json, the review, each pi session's prompt and events.jsonl (its turns and tokens) and oracle-install.log there when a run ends, whether it passed, failed or was stopped, with an archive.txt listing every file and its line count; the step log prints the path. Credentials, URLs, addresses and token-shaped values are replaced on the way (bench/dagu/job.yaml, "handler_on"). Neither archive/gate-logs/ nor archive/job-logs/ is pruned: both grow until removed by hand (find <share>/archive/job-logs -mindepth 2 -maxdepth 2 -type d -mtime +30 -exec rm -rf {} + drops job runs older than 30 days).

The ledger is what a landing's announcement to the community Discord reads for its numbers (docs/web-features-map.md → "Announcements"): each batch's start and end time, lane CPU-seconds, and per model the candidates, compiled count, best score, tokens and draw seconds, with no host names or addresses.

An existing real directory is left alone. A ground-truth cache written by one branch is read by every other, so a recipe change should regenerate its cache. The export is open to the whole subnet: nothing secret goes on it. Measured: fetching the suites into an empty share took 6 min 19 s; a second lane then found them in 13 s.

Reaching the model gateway. A lane reaches the gateway at its rig-network address, not the development box's loopback forward. Each lane keeps that address in ~lane/.lane-model-env as NEV_LANE_MODEL_URL (written by bench/lane-provision.sh from LANE_MODEL_URL), and bench/lane-apply.sh swaps it into the worktree .env's NEV_MODEL_URL line on every run, leaving the key as copied. The address never enters the repository (bench/lane-apply-test.sh).

No credentials left behind. The worktree's .env (the gateway key, usually the forge token) is copied into the lane copy for the run only. When the run ends -- success, failure or interrupt -- bench/lane-apply.sh removes the lane copy's .env, its temporary rewrite and the payload's .env, and bench/remote-lane.sh removes the same three over its own ssh, which covers a run whose connection dropped first. Nothing else in the lane copy is touched. NEV_LANE_KEEP_ENV=1 keeps them, and bench/lane-writer-launch.sh sets it, because the writer it starts outlives the launch and its model batches read that .env. Two known gaps: an interrupted run exits 130 (SIGINT) or 143 (SIGTERM) locally but does not stop the command on the lane, which runs until it finishes or next writes to the dropped connection; and the git mirror's clone URL can carry the forge token, which then shows in the lane's process list as an argument to lane-apply.sh, a copy of that secret outside .env.

Running writers on a lane. No script here names an agent harness; the harness is a declaration. tracks.json → specWatch.lanes.tool names it (TRACKS_TOOL in the environment is the fallback, and a launch with neither is refused), and pi-tracks' writer command prepares that harness on the lane and starts the session. bench/lane-provision.sh's writer-host section (enabled by LANE_PI_TRACKS_URL) clones pi-tracks on the guest and prints changed:/unchanged: per step, so a rerun is idempotent. What a particular harness needs installed ahead of time is a command the operator supplies as LANE_WRITER_SETUP, run as lane with LANE_WRITER_TOOL and LANE_WRITER_DIRS (the lane roots and mirrors, relative to the home directory) in its environment; with none set the step does nothing. A harness login is a person's step over ssh as lane. bench/lane-writer-launch.sh <worktree> <lane-index> <model> <brief-file> syncs a worktree, checks out its branch, pulls pi-tracks and runs "$PI_TRACKS_HOME/bin/writer-exec.sh" --tool <name> --model <model> --brief-file <path> there (PI_TRACKS_HOME defaults to $HOME/.local/share/pi-tracks), refusing over a dead writer's unfetched commits and failing when that command exits nonzero; the brief lands in ~/.cache/lane-writer-briefs/ on the lane, outside the worktree. bench/lane-writer-fetch.sh brings commits back fast-forward only. tracks.json → specWatch.lanes declares both. bench/lane-nas.sh decides a guest is busy from the process names in LANE_BUSY_COMMANDS (default cargo|rustc|node); add the harness's process name when a writer runs under a different one. crates/harness/tests/no_harness_name.rs fails the workspace tests when a bench/ script, tracks.json (other than the tool entry) or .gitignore names a harness. bench/lane-writer-launch-test.sh tests the launcher against a stand-in writer command; a real launch on a lane is unverified until the pi-tracks writer command is on its main.

Dagu on a lane. A job (specs/job-submit-land.md) runs as one run of the Dagu workflow bench/dagu/job.yaml, on the lane NEV_DAGU_LANE names (an index into bench/lane.local.json, default 0; Dagu's coordinator is off, so one lane). bench/dagu-lane.sh installs, starts, reaches and cleans up after it; its header lists the actions and settings. Without --lane it takes the same lane, NEV_DAGU_LANE or 0, unless NEV_DAGU_LANE is unset and NEV_LANE_SSH names a lane, which then wins. jobs/scripts/submit.sh and jobs/scripts/poll-and-land.sh check manifests and results with the job-check binary from $JOBS_STATE_DIR/bin (default ~/.local/state/nev-jobs/bin) on the development box; jobs/scripts/install-job-check.sh builds it on a lane (bench/on-lane.sh --lane-only --pull) and installs it there, and is rerun after a change to jobs/.

The requirements check. Before the run step's job-run starts the first stage that is not passed, on every start and every resume, it checks that the lane has what the manifest is about to need -- git, python3, node and kill resolvable, cargo and the oracle's Playwright browsers when a writer is in the stage list, a reachable model gateway when a stage runs a pi session, 20 GiB free on the workdir's filesystem, the shared archive directory, NEV_ON_LANE when a stage calls bench/on-lane.sh, bench/verify.sh or run-batch.sh, and at least 600 s left of the wall-clock limit across resumes (jobs/src/preflight.rs). One unmet requirement prints every reason to stderr, records them in state.json's preflight field, runs no stage and exits 9 -- distinct from 7 (publish failed) and 8 (stopped), and never resent by jobs/scripts/poll-and-land.sh the way 7 and 3 are, since a missing requirement is still missing a minute later. Run the same check by hand with job-check preflight <manifest> [--workdir <dir>] [--state <dir>], exit 0 when every requirement is met, 9 otherwise; --workdir defaults to the current directory and {tools} in a stage's command resolves under its jobs/scripts. This check only asks what the lane itself can answer right now (a binary, a socket, a directory); whether a model or provider is currently available on the rig is the separate inventory check that follows it.
The inventory check. Once the requirements check passes, the run step's job-run --inventory ~/folknet/inventory.json copies the lane's snapshot into the state directory (again on every resume), hands every stage JOB_INVENTORY (the copy) and JOB_INVENTORY_LIVE (the lane's own path), and checks the manifest against it and the declared file tools/folknet.json (the submitting checkout's bench/folknet.json). A model that is not ready, or a snapshot older than 15 minutes or missing, prints every reason, records them in state.json's inventory field, commits a not-run result locally (not published), runs no stage and exits 3, which jobs/scripts/poll-and-land.sh resends uncounted. A refusal that never clears by itself -- a model retired or not declared, a provider not declared or not allowed -- is reported the same way but commits a failed result and exits 10, which the poller records as failed without resending. The same check by hand: job-check inventory <manifest> <snapshot> [--declared <file>] (docs/folknet-inventory.md -> "The check before the first stage").
Installed as a plain binary. The lanes have no container runtime (unprivileged LXC, no docker or podman, no passwordless sudo; rootless podman would also need /dev/fuse and /dev/net/tun added to the guest's config on its node), so Dagu runs as the lane user from ~/dagu: bench/dagu-lane.sh --lane N up installs the pinned release (its sha256 checked against the pin and the release's own checksums.txt), pushes the gateway key, starts it and checks it. stop, then deleting ~/dagu, removes it. autostart on adds a lane-user @reboot crontab line, which has brought Dagu back after a guest restart.
Reached only through ssh. Dagu binds to the lane's loopback (NEV_DAGU_PORT, default 18431) with basic auth. The password is generated on the lane, kept in ~/dagu/home/config.yaml (mode 600) and read over ssh when needed, never stored on the development box. tunnel holds a port forward open for the web UI, /api/v1 and /mcp. The MCP server is the /mcp route of dagu start-all (Dagu 2.x has no dagu mcp command). The one MCP client is the curl-based one in bench/dagu-lane.sh (run, stop-run, watch).
Secrets never go through run parameters. Dagu keeps a run's parameters in its run records on disk, so run refuses a parameter whose name contains KEY, TOKEN, SECRET, PASSWORD, PASSWD or CREDENTIAL. The gateway key goes by key over ssh on stdin to ~/dagu/home/model-key (mode 600); start exports it as NEV_MODEL_API_KEY with DAGU_ENV_PASSTHROUGH=NEV_MODEL_API_KEY, so every step sees that one variable and it lands in no run record or log. Steps start from a non-login shell and inherit nothing from ~/.lane-env: the gateway address comes from ~/.lane-model-env, the npm cache and the share from ~/.lane-shared-env (one variable at a time), and the forge through the lane mirror's origin URL, which carries the credential; git messages from forge commands pass through a filter that replaces anything shaped like a URL.
How Dagu 2.17.2 behaves. Workflow keys are snake_case (retry_policy, hist_retention_days); camelCase is refused. One dagu_execute call waits at most 300 s, so run follows a longer run by reading its status every 15 s until it ends or NEV_DAGU_WAIT seconds pass. Dagu records no exit code for a step: watch --once --step <step> takes 0 for a succeeded step and otherwise the N of the step's error text exit status N, and - when there is none (a step killed by a signal). The exit handler (handler_on: exit) runs after every run, succeeded, failed or aborted, but not after a run cut off by a lane restart or a KILL of Dagu.
The queue. A workflow's own max_active_runs above 1 is ignored for local queues; the limit is a global queue in ~/dagu/home/config.yaml (queues.config[].max_concurrency, written by set_queue(): queue jobs, 2 at once), read only when the scheduler starts, so a change needs stop then start. Only enqueued runs pass through it (dagu_execute action=start runs at once), so run enqueues any workflow file with a top-level queue: key. A second submission while one is building passes --no-sync, so the source copy under the first run's build is not replaced.
What it keeps. Run history 7 days (hist_retention_days, set in ~/dagu/home/base.yaml by set_retention() and in job.yaml): the forge holds each job's result. A job's directory ~/dagu/jobs-work/<name>/ stays until the job passes, so a new run with the same parameters resumes it, and after a pass it stays too when its logs did not reach the NAS share (step cleanup removes it only once the archive wrote its marker archived-<Dagu run id> there, and says in its log when it kept it); clean-jobs <name> removes one by hand. The job tools are built from the copy of the sources run sends to ~/dagu/src (sync_source()), not from the job's base commit.
The mirror keeps deleted job refs. The lane mirror's fetch (refspec +refs/*:refs/*, no prune) copies every job/* and result/* branch from the forge and keeps it after the forge deletes it. prune-mirror fetches and prunes those two namespaces only; bench/dagu/fixture-base.sh --delete runs it.

Landing approved spec tracks from a tick #

bench/land.sh <spec-name> <spec-sha> <branch-tip> lands an approved track without a person; the spec-watch tick calls it (tracks.json → specWatch.landScript). Exit codes: 0 landed, 1 not landed (reason on one line), 2 rebase conflict outside handoff.md. The contract: ~/.claude/docs/working-modes.md → "Let a tick land tracks".

flowchart TD
  start["tip and spec sha match<br/>what was reviewed?"] -- no --> refuse["refuse"]
  start -- yes --> scope["diff inside the spec's touches,<br/>no protected file?"]
  scope -- no --> refuse
  scope -- yes --> rebase["rebase onto main<br/>(handoff.md → main's roster)"]
  rebase --> gate["verify + landChecks<br/>(tests, 227 floor, per-function floors)"]
  gate -- fail --> refuse
  gate -- pass --> merge["fast-forward main<br/>(retry if main moved, max 3)"]
  merge --> after["tag spec/name/sha, remove worktree<br/>and branch, update roster"]

Protected files. A diff touching bench/, tracks.json, .github/, specs/, or at any depth .cargo/, rust-toolchain*, .env*, .gitattributes, .gitmodules or *.floor.json (a generated function's score floor) is refused whatever touches says, since those change the gate itself. So is a spec whose touches matches everything or reaches those paths. Case-insensitive, no rename detection. Such a track waits for a person's /land.
What the gate still trusts. The track's own crates, tests, build.rs, oracle/ and Cargo.toml/Cargo.lock. The gate catches regressions and honest mistakes, not a track built to fool it.
The checks. verify (bench/verify.sh: the tests, then every generated function scored against its own corpus, each at or above its floor in <name>.floor.json), then landChecks: bench/fragment-floor.sh (fragment_check --random must pass 227), which runs through bench/on-lane.sh (--lane-only, like verify, when NEV_LANE_STRICT=1; see "Build lane"). The four hand-written floors that bench/seam-floor.sh used to hold (floats, stacking order, relative positioning, block height) are floor files like the rest, and that script and its checker are gone. Floors are raised after a landing, never lowered by a track: a diff that touches a *.floor.json is refused.
Safety. Pinned SHAs re-checked right before a --ff-only merge; a lock on the git directory; signals kill the gate's process tree and restore the branch. A land-pending/<name> marker makes a rerun finish anything owed after a merge. Nothing is pushed.
handoff.md. A landing restores main's roster on the branch before the gate, so a track's working note never overwrites the roster.
Install after a landing. If oracle/package*.json changed or a crate was added, only bench/oracle-install.sh runs, staged and swapped in on success.
Testing. bench/land-test.sh drives the script through about 130 assertions in a throwaway clone. bench/land.sh --check ... runs every condition and stops before the merge.

Training limits #

Full fine-tuning is out of reach on 24 GB Ampere cards; LoRA or QLoRA on a 30B-class model across eight GPUs is feasible, so any supervised step is a LoRA.

Sharding without a broker #

Case generation is deterministic from a seed, so worker i of n takes seeds i, i+n, i+2n, …: no queue, no coordinator, and any worker can restart without losing or duplicating work. Output is content-addressed and written once; ground truth is immutable per engine version, so there is nothing to invalidate.

Storage #

About 10 KB per document for three engines, so ten million cases is about 100 GB. The style subset dominates and is redundant; compress it, sample it, or store it as a delta.

Virtual machine template #

One image, duplicated: WebKit's 188 system packages, pinned Playwright browsers, Rust with sccache, the repository. Workers are stateless. Pin one VM per NUMA node rather than one large VM per host.

Engine versions pin the ground truth #

Ground truth is immutable only for a given set of browser versions, so every dump records engine_version (docs/dump-format.md → "Dump") and a corpus is keyed by that version set. A browser update invalidates only what it touches, never silently.

Rig change log #

Every change tried on the rig's hardware, worked or not, newest last. Hosts by role and class only. Before a change, and again after each change that works, bench/rig-snapshot.sh snapshot <label> records the serving configs of every role (proxy host HAProxy, gateway container config and key database, each ARM node's units and TabbyAPI configs, the four-card VM's Proxmox config and units) on the host itself and in the local rig notes; restore <label> puts them back. The label named in an entry holds the working state after it. Snapshots contain the gateway's master key and never go into git or onto the NAS.

2026-09-24

Gateway key for self-hosted writers. A new virtual key on the gateway, alias nev-folknet-writer, allowed only qwen3.8-27b-long and qwen3.8-27b-fast, so writer traffic is accounted apart from batch traffic and cannot reach any other alias. No serving config changed. The key itself lives outside every repository, as FOLKNET_WRITER_API_KEY in $HOME/.config/folknet-writer/env (mode 600) on the development box. First used by stage 0 of docs/folknet-writer-design.md (docs/writer-harness-spike.md → "Rerun through Pi"). Worked.
Snapshot and restore. bench/rig-snapshot.sh written; known-good-2026-09-24 taken before any serving change. restore known-good-2026-09-24 haproxy run once as the round-trip test: config check passed, HAProxy reloaded, all four ARM replicas still up. Worked.
Four-card VM. A new VM on a Proxmox node, given the four 3090s of a stopped tenant test VM (same hostpci entries, q35, OVMF, 256 GB 64-bit PCI hole), 32 cores, 160 GB RAM, a 120 GB OS disk, DHCP on the main LAN (bench/gpu-vm/create-vm.sh). The tenant VM's disks were not touched. Worked; four 3090s visible.
Provisioning, first two attempts failed. First: the fresh cloud image was still running cloud-init's package upgrade, which held the dpkg lock and then rebooted the guest. Second: the rig's apt cache answers 403 CONNECT denied for HTTPS repositories, so docker's and NVIDIA's apt repositories could not be reached through it. Fixed in bench/gpu-vm/provision.sh: wait for cloud-init and the dpkg lock, send HTTPS repositories direct, take docker from the Ubuntu archive (docker.io, which the cache serves). Third run worked: driver 595 (open kernel modules) loaded without a reboot, nvidia-smi shows four 3090s, docker has the NVIDIA runtime.
Model choice for the four cards, by size (no download attempted).
- qwen3.8-flash-next, EXL3 4.05 bpw (the ARM nodes' checkpoint): 101 GB on disk, of which 39 GB is the hashed n-gram table that exllamav3 keeps in host RAM or on disk, so about 62 GB of weights would sit on the cards. That the checkpoint does not fit 96 GB was assumed, never tested. A 3.05 bpw branch is also published (85 GB on disk, 32.6 GB of it n-gram table). Not pursued: the target moved to a different model family for variety.
- DeepSeek-V4-Flash (the July release, 284B parameters, 13B active): 167 GB native (MXFP4 experts). Mainline llama.cpp with CUDA serves it on sm86 from GGUF. The smallest quant its quantizer recommends (3-bit, 104 GB) does not fit 96 GB of VRAM; 2-bit quants (87–91 GB) fit with only a few GB left for cache and compute buffers, and re-quantizing the native 4-bit experts to 2 bits costs a lot of accuracy (published KL divergence around 0.4 at 2 bits against 0.01 at 4 bits). Download straight from Hugging Face inside the VM measured about 250 KB/s. Not tried.
- DeepSeek-V4.1-Flash is a different, larger model, not a revision of the one above: a new architecture (deepseek_v41, 384 experts over 40 layers, n-gram memory tables), 510 GB native. It does not fit the four cards. The first reading, that it fits no pair of ARM nodes either, was wrong: it looked only at GGUF quants (smallest about 260 GB) and missed a published EXL3 recipe for exactly two GB10s — a 2.9 bpw EXL3 checkpoint (~197 GiB, about 99.5 GiB of weights per node at tensor-parallel 2) plus the model's unquantized n-gram ("Engram") tables, which stay on disk and are read per lookup rather than held in memory (two native shards, ~190 GiB), served by a vLLM build that carries the deepseek_v41 architecture. That recipe is what the ARM pair uses (entries below).
- qwen3.8-27b, W4A16 AutoRound with DFlash2 speculative decoding, the eight-card host's -fast recipe: about 22 GB per card, one worker per card. Chosen.
qwen3.8-27b on the four cards. The eight-card host's image and prepared checkpoint (24 GB) were streamed to the VM read-only (tar and docker save piped over ssh, nothing on that host changed). One worker per card came up in about 6.5 minutes from cold; all four report healthy and answer chat completions. Worked; snapshot vm-qwen27b-workers-up.
Proxy and gateway for the VM. A new HAProxy frontend and backend on the proxy host (the existing backends untouched; backup haproxy.cfg.pre-vm-qwen27b-20260924 beside the config) and a gateway alias qwen3.8-27b-fast-2 (backup config.yaml.pre-qwen27b-vm-20260924), granted to the project's key only. The first gateway attempt failed with "Missing credentials": an OpenAI-style entry needs some api_key even when the upstream checks none; sk-none, as the flash-next entry uses, fixed it. Worked: the alias is listed and returns code through the gateway; snapshot vm-qwen27b-fast-2-alias.
Measurement batch at qwen3.8-27b-fast-2 on a build lane: one 32-candidate absolute-positioning batch, 2/32 compiled, best 74/108 (docs/model-first-cost.md, row vm-batch 1). Stopping the loop after batch 1 with pkill -f on the shared lane also killed another track's run of the same binary in its own lane worktree (a mixed-model batch at 32 of 64 draws). On a shared lane, stop a run by the pid whose /proc/<pid>/cwd is this worktree's lane copy, never by name.
Checkpoint fetch for an ARM pair. bench/sparks/hf-to-nas.sh (a runtime bundle and oneshot unit on the downloader host, reusing that host's multi-WAN scripts) started DeepSeek-V4-Flash (July release, 167 GB); stopped after 14 GB when the ARM target changed to DeepSeek-V4.1-Flash. The partial staging directory is left on the NAS. Then both parts of the V4.1 recipe, pinned by revision: the EXL3 2.9 bpw checkpoint (42 files, 210.7 GB) and the two Engram shards plus index and config of the native checkpoint (203.1 GB), both through the multi-WAN downloader at about 130 MB/s combined. Snapshot before: pre-deepseek-sparks.
Which two ARM nodes. LLDP shows each node's first 200G port on the switch and its second port cabled straight to one other node, in two cabled pairs, each crossing the old GLM pairs (which ran over the switch). The DeepSeek pair is one of the cabled pairs; the other cabled pair keeps flash-next. Its tensor-parallel traffic is set to run over the switch port with the RoCE settings the GLM pairs used, because the direct cable carries no addresses today; moving it onto the cable means addressing that port on both nodes (not done).
Staging the pair (nothing serving yet). The recipe is cloned at a pinned commit on the head node, its .env written from bench/sparks/dsv41/env.template (weights from local disk, no Hugging Face download, AUTO_DOWNLOAD=0), an ssh alias for the worker added to the head's admin account (bench/sparks/dsv41/ssh-config-worker; previous config kept as config.pre-dsv41), /srv/dsv41 created on both nodes, and the recipe image pulled on both (the 2.9bpw tag, 22.5 GB; digest in the local rig notes). Both nodes still serve flash-next. Snapshot dsv41-staged.
Throughput sweep of qwen3.8-27b-fast-2 (bench/rig_throughput.py) on a build lane: failed twice before sending a request — the lane's Python has no requests module, and python3 -m venv fails there without the python3.12-venv package. Not run.
Four-card VM reboot. All four workers healthy three minutes after reboot, same DHCP address, HAProxy backends back UP without intervention. Worked.
Second measurement batch at qwen3.8-27b-fast-2, after the prompt fixes, run with bench/absolute-mixed-batch.sh on a build lane: 8/32 compiled, best 82/108, 233 s for the batch (docs/model-first-cost.md, row vm-batch 2). The run ended by itself (GENERA_MAX_BATCHES=1); nothing needed stopping. No serving change.
Throughput sweep of qwen3.8-27b-fast-2, second attempt, run from the dev box (which has requests) through the gateway, with the prompt generated on a build lane: worked, 819 output tok/s at 16 in flight (docs/rig-capacity.md → "Measured — the four-card VM"). No serving change.
Abandoned DeepSeek-V4-Flash download removed from the downloader host: the 14 GB partial staging directory on the NAS, its oneshot unit and runtime bundle. Worked; the two V4.1 units and their published trees are untouched.
DeepSeek-V4.1-Flash weights to local disk on the ARM pair. Both NAS downloads had finished and published. rsync -a from the NAS mount to local NVMe on both nodes, the EXL3 checkpoint (39 shards, 197 GB) to each, the two Engram shards with index and config (190 GB) to the head's engram source directory and to the worker's engram directory. The worker gets no second engram tree: the recipe's start.sh builds the slim engram set on the head and rsyncs it to the worker, and with the NAS mtimes kept, that rsync finds the shards already in place. About 100 MB/s per node, both done in about 55 minutes. Both nodes kept serving flash-next throughout, all backends UP after. Worked; no serving change, so no new snapshot (dsv41-staged still holds the serving state).
Two ARM nodes out of the flash-next pool. Snapshot before: pre-dsv41-pair-move. Their two server lines were removed from the flash-next HAProxy backend (backup haproxy.cfg.pre-dsv41-pair-20260924 beside the config; config check, reload; the other two replicas UP), then their flash-next units disabled and stopped; unit, config and checkpoint stay on disk. Nothing was drawing from the alias at the time. The alias now takes 16 requests in flight (two replicas × 8), not 32. Worked; snapshot flashnext-two-replicas.
DeepSeek-V4.1-Flash on the pair. The published recipe image carries the recipe's own build stamp from four commits before the pinned one, so the pinned start.sh would have compiled the image on the node. The only image-side change in between is one Python patch, which that start.sh bind-mounts into both containers anyway, so the unit runs the published image as-is (SKIP_BUILD=1, reason in the unit file). Unit installed on the head and started: preflight passed (memory headroom 117.6 GiB available per node against 111.5 needed, RoCE GID accepted), the weight sync to the worker found everything already in place and took 3 s, weights loaded and warmed up, API healthy about 7.5 minutes after the start; the README's 17*19 request returned 323. The recipe's own warmup reports one sampler-kernel variant not pre-compiled ("may JIT mid-serve"); harmless so far. Worked; snapshot dsv41-pair-serving.
Proxy and gateway for the pair. A new HAProxy frontend and backend on the proxy host, one server, maxconn 2 (backup haproxy.cfg.pre-dsv41-backend-20260924), and a gateway alias deepseek-v4.1-flash → the recipe's served model name, api_key: sk-none (backup config.yaml.pre-dsv41-20260924), granted to the project's key only. Worked: the alias is listed through the gateway and a genera-shaped chat completion (thinking off) returned a one-function Rust code block in 2.7 s with no reasoning in content. Snapshot dsv41-alias-serving.
First measuring batch at deepseek-v4.1-flash on a build lane, 8 candidates at 2 in flight: 4/8 compiled, best 82/108, 1,731 s (docs/model-first-cost.md, row dsv-batch 1). The first launch failed before drawing anything: the lane's shared git mirror refused a ref update (cannot lock ref 'refs/heads/main') because another run had moved main at the same moment; a plain retry worked. No serving change.

2026-09-25

Dead GLM aliases removed from the gateway. glm-5.3-flash and glm-5.3-flash-exl3 were still listed by /v1/models three days after their backends stopped, so a caller could pick one and get a 503 ("No server is available to handle this request"), as a job on another track did. Snapshot before: pre-glm-alias-removal. Both model entries deleted from the gateway config (backup config.yaml.pre-glm-removal-20260925 beside it; no fallback named them) and the gateway service restarted, live again in 18 s. The two names were then dropped from the allowlists of the three virtual keys that carried them, each keeping its other models. Worked: /v1/models through the dev-box tunnel lists neither name; one-token completions return 200 for qwen3.8-27b-fast, deepseek-v4.1-flash, qwen3.8-27b-fast-2 and qwen3.8-flash-next; a request for glm-5.3-flash is refused with 403. Snapshot glm-aliases-removed. The site file the gateway config is generated from still carries the GLM entries, so a regeneration would bring them back.