# Running a web feature with no session editing
The aim: a web feature goes from a spec to a landed commit and a community announcement on
folknet hardware with nobody editing anything. A person or an interactive session that
edits files inside a feature's track is a failure of the system, not a step in it.
This document surveys how far the repository is from that, ranks what stops it, and
proposes a design. Nothing here is built. Evidence is cited as file plus symbol, or as
the overnight run of 2026-09-24/25 (the roster `handoff.md` on `main` → "Overnight
ramp", and the tick log `$HOME/.local/state/spec-watch/spec-watch.log`). Every claim not
checked against code or a run is marked *unverified*. Hosts and addresses stay in
`$HOME/rig-notes`.
## The principle
The whole lifecycle of one feature is packaged as one job. The job is shipped to a build
lane, started there, and runs to the end there. The only thing watched from outside is a
commit appearing on a result branch. A live writer session in a development-box worktree,
one per feature, is the failure state, because it needs something to keep it alive
(a tick, a relaunch, a local rebase) and every such touch is a place a person is pulled
in.
The development box does three things: submit a job, poll for its result commit, and
land it.
The loop starts at an enabled spec and ends at a result commit. It does not write or enable
specs: a person or a coordinator session does that, outside the loop, so spec authorship
stays with a reviewer. Finished features are reviewed afterwards, asynchronously, and a
review that finds problems sends the feature back through the loop as a new or amended
spec.
## The loop as it stands
```mermaid
flowchart TD
spec["1. spec written/` tag to `JOBS_TAG_REMOTE` (default `origin`), not `main` | development box | yes for `main`, after each batch of landings; also for the tag of a landing the poller did not make |
| 13 | Floors, map rules for what landed | floors: `bench/generated-floors-update.sh` from `bench/after-land.sh`; map rules: a job's `feature-map` stage from its manifest's `feature_rules`, with a `NOT ANNOUNCED` line from `bench/expected-features-check.py` after landing when a declared feature did not move | lane (floors, the job's stage); development box (the line) | no for a job; yes, by hand, for a spec landed outside a job |
| 14 | Review the finished feature; on problems, write a new or amended spec that re-enters at step 1 | a person or a coordinator session, outside the loop, after the result commit and without holding a landing | development box | yes, by design: spec authorship stays with a reviewer |
Which parts already work unattended: the overnight run had the tick write, review and land
seven specs with no hand intervention (`dashboard-history-graph`, `text-metrics-oracle`,
`testdriver-playwright-input`, `transforms2d`, `audio-null-sink`, `inline-atomic`,
`tables-fixed-layout`), and `fixed-positioning` landed by hand after `bench/land.sh`
refused it. The independent review found two real defects in `aspect-ratio`. So steps 2 to
11 run without a person when a spec is well formed; what fails is the edges (steps 12 and
13) and the checking inside the steps. Steps 1 and 14 need a person or a coordinator
session by design.
One correction to older text: the roster's "Resume Here" says an owning interactive
session's loop drives ticks. `spec-watch-tick.sh`'s header says no interactive session
drives ticks, and the crontab runs it every five minutes for genera. The roster line is
out of date.
## Reviewing, stopping and rewinding a job's stages
A job's stage results can be read and re-run from the development box without watching a lane
directory.
- **A commit per passed stage.** After each stage passes, `job-run` commits the work tree as
`stage: ` on the branch `job-stages/` and pushes it to the `--publish` remote
(`result.rs` `commit_stage`, `publish_stage`). Each commit's parent is the previous stage's
commit, and the first stage's parent is the job branch head. The tree is the work tree minus
git-ignored files and the manifest's `keep_dirs`; a kept directory appears as a
`Kept-dir: ` line in the message, because corpus caches are large. A
landing keeps the branch on the tag remote as the tag `stages//` before it
deletes it from the forge (`jobs/scripts/poll-and-land.sh`); when the tag cannot be pushed,
the branch stays on the forge.
- **Output under ignored paths.** A stage's declared `outputs` (the stage cache's field) and
its `output_dirs` (`jobs/src/manifest.rs`) are force-added to that stage's commit and kept
in every later stage commit, even where git ignores them or they lie in a kept directory, so
each stage commit holds what the stages so far produced (for `aspect-ratio`, a few MB of
`corpus/` per run). The `dump` stage of `jobs/scripts/e2e-fixtures.py`
`stage_commits_stages()` names `target/e2e-stage-dump` in `output_dirs`. Until 2026-10-08
no step-pipeline run had a stage commit: `git add` refused the exclude of a kept directory
that git ignores, and every stage logged "no stage commit".
- **Why a separate branch.** Landing fast-forwards the base branch to the tip of `job/`
and takes every commit reachable from it, so generated output on that branch would reach
`main`. `job-stages/` is never an ancestor of the job branch, and a landing deletes it
with `job/` and `result/`.
- **Stopping.** `JOBS_STOP_AFTER= jobs/scripts/submit.sh ` makes `job-run` stop
once that stage has passed: exit 13, state saved, job branch pushed, no result written. The
poller records the job as `stopped` and does not resend it. A resend by the poller keeps the
stop point (record field `stop_after`), so a dependency dropping out cannot carry a held job
past it. A plain `submit.sh ` clears it and resumes at the next stage. A resend by the
poller likewise keeps the run's other settings from its record (`model`, `batch_models`,
`rerun`, `pass_failing_cases`), never the poller's own environment. A run stopped in the
middle with `bench/dagu-lane.sh stop-run job ` is recorded `stopped` too and not
resent.
- **Rewinding.** `JOBS_REWIND_TO=` (`job-run --rewind-to`) restores the work tree from
that stage's commit, resets the job branch to the `Job-head:` trailer the commit carries and
makes the later stages pending. It applies once, on the send that asked for it: a resend by
the poller resumes where the rewound run got to, and a send that died before `job-run`
started loses its rewind and must be sent again. It exits 2 for an unknown stage, a stage
that has not passed, or one with no commit, and combines with `JOBS_STOP_AFTER` only when
the stop stage comes after the rewind stage.
- **Stages with no commit.** A job that resumes from a run that began before stage commits
existed has passed stages with no commit. None is invented from the current tree, since that
would put later content under an earlier stage's name; the next stage commit lists them as
`No-commit: `. A resume in a fresh clone first fetches `job-stages/` from the
publish remote (`result.rs` `adopt_stage_branch`), so the earlier commits are chained on,
not overwritten.
- **Reusing a stage whose inputs did not change.** A script stage may declare `inputs` (repo
paths or globs it reads; a path inside an earlier stage's `outputs` stands for that stage's
output), `env` (variable names whose values it depends on) and `outputs` (everything it
writes). Before such a stage runs, `job-run` keys it by those inputs, its command, the
`{tools}` files it names and its agent (`jobs/src/cache.rs` `stage_key`), and when the share
(`WEBENGINE_LANE_SHARED`) holds an entry for that key under `archive/stage-cache/`, it
restores the stored outputs and passes the stage without running it (message
`reused `). A stage that runs and passes stores its outputs there.
`job-run --rerun ` forces a run and replaces the entry. A reuse rewrites the entry's
`last-used` file (`cache.rs` `mark_used()`), and right after each store `job-run` removes
that stage's old entries (`cache.rs` `prune()`): per job and stage it keeps the 2 most
recently used, with no age floor, and removes the rest along with `.tmp-*` leftovers
older than 14 days. This is the one retention rule every job-run folder follows
(`jobs/scripts/retention.sh`, `docs/infrastructure.md` → "Retention of job-run
folders"), which also applies it at the end of every job and on each lane sweep. An
entry's last use is the newest modification time of its directory, `meta.json` and
`last-used`; on the share that time comes from the NAS clock. A removed entry is renamed
aside first, so another lane's lookup sees it whole or not at all. The prune runs in
`job-run` too, rather than only at the end of the job, because a store is the only thing
that adds entries; a prune that fails is logged and the stage still passes. `job-check manifest` checks
that every repo path the command names lies inside an input or an output, that `--out` lies
inside an output, and that no two stages' outputs overlap; what a script reads without
naming it cannot be checked. `JOBS_RERUN=[,]` on `submit.sh` passes
`--rerun` through `bench/dagu/job.yaml` (`RERUN`). A review is never restored; in a
manifest with no writer stage it is passed without running when nothing was committed
since the base commit (a `store` that found its candidate already stored), with the
message `nothing committed since base; no diff to review`, and `job-run` writes the
`review.json` a reviewer would have, a `pass` with that message as its notes, so the
result passes `job-check result` (`jobs/src/runner.rs` rule 6). A script stage that
declares `outputs` without `inputs` always runs and is never stored.
- **Starting afresh instead of resuming.** A resend whose spec changed, or of a job whose
manifest gives any stage `inputs` when the manifest or a path some stage lists as an input
differs between the recorded base and the base tip, goes at the base tip and starts the
job afresh (`jobs/scripts/submit.sh`, "The base sent"): `submit.sh` tags and deletes the
forge's `job/` from the older base, and the lane's `prepare` moves the old job
directory aside and clones again, so every stage runs again except those the stage cache
restores. Any other resend resumes in place at the recorded base, as before; a
needs-person resend with the spec unchanged always does, since it carries a person's fix.
`prepare` no longer restores `keep_dirs` from the newest archived tar, which said nothing
of what the directory was made from. `specs/aspect-ratio.job.json` declares `inputs` and
`outputs` on every script stage but `wpt` (its checkout is pinned by
`oracle/suites/suites.json`, which keys the stages that read it), `store` (it commits),
`findings-pages-prompt` and `findings-pages-keep`, which declare `outputs` only:
`findings-pages-prompt` reads the round's rework file, which no path names, and
`findings-pages-keep` reads its own earlier output to carry the pages earlier rounds kept,
so both run every time, and the stages reading their output are keyed by what they wrote. That manifest has no writer stage; its `rework_from` names
`findings-pages-prompt`, so a review `fail` that names the inputs on which the candidate is
wrong turns them into browser-measured train cases and repairs again, up to
`failed_rounds` (`tasks/job-review-rework.md`). A `fail` that names no such input gets
`"rework": false` from `pi-review.sh` and starts no round; its review is advisory, so once
no round follows the job lands the stored candidate on the score bar (`jobs/src/runner.rs`
rule 22).
`repair` lists every folder and file `bench/repair-loop.sh` leaves beside the batch, for
each of the twelve rounds its `--rounds 12` allows (`outputs` take no globs, so a change
to `--rounds` changes the list with it); the loop only reads the batch itself (each round's `report.json` goes in
that round's `-work` folder), so the `candidates` stage's output stays its own.
`bench/e2e-jobs.sh` has two scenarios for this: `e2e-stage-commits` covers stage commits,
`output_dirs`, rewind through `submit.sh` and the stage branch being gone after a landing;
`e2e-stage-cache` edits one stage's input on the base after a failed run and expects the
resend to start afresh with only that stage, the stage that reads its output and the stages
without `inputs` run again. Unless given `--keep`, the run's cleanup
(`bench/e2e-lane-clean.sh`) removes each scenario's stage cache entries
(`archive/stage-cache//`), so gate runs leave none on the share.
## Gaps, ranked by how often they stop or degrade the loop
Counts come from the overnight run (eight specs, seven landed by the tick, one by hand)
and are the roster's, not rechecked against the tick log line by line.
### 1. Specs are written by hand (stops every feature)
Nothing creates a spec, and the loop is not meant to: specs are written by a person or a
coordinator session outside the loop, so the gap to close is a malformed spec, not the
lack of an author. The roster's "Queue" is drafted by a person or a session, and the
drafting rules live as prose there: single-line front matter (`bench/land.sh` `touchesOf()`
reads a multi-line list as empty), `touches` must name the spec's own test files, its
provenance module, `crates/provenance/src/lib.rs`, `docs/generated-functions.md` and
`docs/feature-map.json`, and a `depends-on` on the previous generated-function spec because
they all edit `fragment.rs`. No check enforces any of it. A drafted spec cannot arrive by
a track either: `bench/land.sh` refuses any diff under `specs/` or `tasks/`.
Every feature job carries the same two things, and these are checked. Its manifest
(`specs/.job.json`) lists the web-features ids it moves as `expected_features` and
the exact browser-compat-data keys it implements as `feature_rules`, each with the
`status` and `evidence` of a `docs/feature-map.json` rule, and only keys the spec's scope
covers (a feature's keys are listed in the pinned `web-features` package under
`oracle/node_modules`; `aspect-ratio`'s spec leaves out the `` keys, so
its manifest does not list them). Its spec's `touches` name
`docs/feature-map.json`, since the job commits the rules there, and its prose must not
forbid that file. `job-check manifest` refuses a manifest with features and no rules, or
rules and no features; `job-check manifest --spec`, which `jobs/scripts/submit.sh` runs,
refuses one whose spec's `touches` omit the map (gap 2).
### 2. Announcements needed hand-added map rules (closed for jobs)
A feature moves past `unknown` only through an exact-key rule in `docs/feature-map.json`
(`docs/web-features-map.md` → "The status rule"). The generated-function specs' prose
forbade editing that file, so none of the seven landings announced anything until rules
were added by hand at `6a3de27`, and `bench/after-land.sh` could not tell "nothing moved"
from "no rule exists", so a missing rule looked like a quiet success.
For a job, the rules now come from its manifest (`feature_rules`, gap 1), and no writer
has to choose to add them. `job-run` adds a `feature-map` stage before review
(`jobs/src/runner.rs` `add_feature_map_stage`): `job-check feature-map` writes the rules
into `docs/feature-map.json` through `job-check feature-rules` and commits them, runs
`bench/web-features-update.sh --no-commit`, and fails the job when a listed key or expected
feature does not end partial or complete. After landing, `bench/after-land.sh` runs
`bench/expected-features-check.py`, which prints a `NOT ANNOUNCED` line naming each
declared feature no event moved that is not partial or complete, with the fix. The
announcement no longer depends on the map: a landed job posts once whether or not a
feature moved, and a map that did not move is one line of that post giving the reason
(`docs/web-features-map.md` → "Announcements"). Nothing is
owed for `aspect-ratio`, which ran before the stage existed: its job definition is removed,
and a new one writes its rules through the stage.
### 3. Floors are added by hand (silent on every generated function)
Resolved by the `floors-follow-the-corpus` track. `bench/seam-floor.sh` used to have four
floors (floats, stacking order, relative positioning, block height); containing block,
transform and inline atomic had none, and each landed without one, because adding a floor
needed a change in `bench/` (refused by `bench/land.sh` for a track), a score line in
`seam_floor_check.rs` and a cache rebuild in the script's loop. Now every generated function
is scored by `bench/generated-scores.sh` against `.floor.json`, a new function is one
entry in `generated_score_check.rs` `FUNCTIONS`, and `bench/generated-floors-update.sh` raises
the floors after each landing.
### 4. Steps the spec prescribes in prose are not enforced (silent, once in seven)
`tables-fixed-layout` landed with a hand-written `crates/layout/src/generated/table.rs`
whose `table.provenance.json` records `"model": "hand-written"` and `"cases_run": 0`,
because its three lane runs never happened, and the review passed it. The review's four
questions (`prompts/spec-watch.md` → "Review work") ask whether the diff implements the
spec, not whether the runs the spec requires happened, and `bench/land.sh` reads no
provenance file. The map already treats such a record as not complete
(`docs/web-features-map.md`: a complete key needs at least one case run and every case
passed), so the evidence exists and nothing reads it at landing.
### 5. Model batches bypass the batch script (loses the numbers on every landing)
`transforms2d`'s spec prescribes `cargo run --release -p generation --bin
transform_real_model_run` (`tasks/transforms2d.md`), not `bench/absolute-mixed-batch.sh`
or `bench/overflow-mixed-run.sh`. Only those two scripts call `bench/ledger.py`
(through `bench/batch-lane.sh` `batch_measure_stop`), so a batch run the spec's way writes
no ledger line, and the announcement has no candidates, tokens or CPU-hours to post. The
roster says no real batch has written a ledger line yet. The tick log says the
`transforms2d` batch "ran locally"; the log also says its writer started on lane 1, so the
batch most likely ran on the lane inside the writer's copy, not on the development box
(*unverified*: which machine ran it). Either way it did not go through
`bench/batch-lane.sh`, which is what enforces "a model batch fails rather than fall back to
a local run".
### 6. Landing refusals hold a slot forever (stops one feature and its dependents)
`bench/land.sh` refuses committed files outside `touches`, any diff under `bench/`,
`tracks.json`, `.github/`, `specs/`, `tasks/`, `.cargo/`, `rust-toolchain*`, `.env*`,
`.gitattributes`, `.gitmodules`, and a `**`-broad `touches` (header of `bench/land.sh`;
`isDenied()`). A refused track stays `awaiting-approval`, holds a live slot and blocks its
dependents. `fixed-positioning` hit this and landed by hand. The `touches` omissions in
sandbox specs (`ring-buffer`) are the same failure. A person's `/land` is the only way
out.
### 7. The tick is a model session on the development box that edits worktrees
The tick creates worktrees, runs `npm install` for `oracle/`, writes and commits
`handoff.md`, rebases, resolves `handoff.md` conflicts, and relaunches a writer on the
development box when a rebase conflicts outside `handoff.md` ("Rebases are kept on this
machine by design": a lane copy carries only `track/*` branches, so a lane writer has
nothing to rebase onto). It also falls back to a local writer when every lane refuses.
Recorded failures: a lane launch read as a failure and started three writers on one track
(fixed in pi-tracks); a tick died with no exit line twice; a lane writer waiting on a
silent subagent stayed `working` for over 30 minutes, and the stall check did not fire;
lane copies diverge after a local rebase. These are the roster's "Spike findings", several
marked fixed. Each is a place where a dev-box process has to stay healthy for a feature
to finish.
### 8. Gates can fall back to the development box
`bench/on-lane.sh` runs a command locally when no lane answers, unless `--lane-only`.
`bench/land.sh` runs each `landChecks` command through it without `--lane-only`, and runs
`verify` directly (`tracks.json`'s `verify` even falls back to a local `cargo test` when
`bench/verify.sh` is missing), so with both lanes down the floors could run on the
development box. That contradicts the root `AGENTS.md` for everything except model
batches and `bench/verify.sh`, which is lane-only already. *Unverified:* that it has ever
happened. **Fixed in `bench/land.sh`:** with `WEBENGINE_LANE_STRICT=1` the gate and every
check go `--lane-only` (`docs/infrastructure.md` → "Build lane"). The scheduled poller
gets the variable from `tracks.json`'s `specWatch.pollEnv` ("Hand changes still needed",
item 5); a landing run by hand, or by the model tick's `specWatch.landScript`, does not
set it.
### 9. Pushing `main` is by hand
The tick lands on `main` without pushing. The poller (`jobs/scripts/poll-and-land.sh`)
pushes the `spec//` tag of each job it lands to `JOBS_TAG_REMOTE` (default
`origin`), but not `main`. A person pushes `main`, and any `spec/*` tag the poller did not
push, after each batch (`docs/coordinator-on-folknet.md` → "Who pushes"). This does not
stop the loop, since a lane gets unpushed commits in a bundle (`bench/remote-lane.sh`
bundles `origin/main..HEAD`), but the bundle grows with every unpushed landing and the
public repository lags.
### 10. Smaller items
- One review round: `maxRounds` is 1, so the second failed review parks a track
(`aspect-ratio` used its one round overnight; a second failure parks it).
- `awaiting-approval` tracks were listed as waiting on a person in repositories that land
them automatically (roster "Spike findings"; the prompt now excludes them).
- The coordinator's move to a folknet guest (`docs/coordinator-on-folknet.md`) is designed
and not done, and needs a person.
- Not yet exercised: subscription-limit backoff, the stalled-writer relaunch, and lane
capacity under many writers (roster).
- An open decision: how wide a feature spec's generated scope should be (roster, "Blocked
on"). It shapes what a spec asks for, so it shapes the job's stages.
## Proposed design
### What a job contains
One job is a branch plus a manifest, and everything the run needs is either in it or
pinned by sha.
| Part | Content | Today |
|---|---|---|
| Spec | `specs/.md` at a pinned sha; its `touches`, `depends-on`, writer `model` | the same, read by `spec-status.mjs` |
| Base | the sha of `main` the job starts from; the lane carries this commit so a rebase can run there | not carried: lane copies hold `track/*` only |
| Plan | the brief for the writer stages, and the ordered stage list below | `handoff.md` plus prose in the spec |
| Model and batch settings | provider (never metered), model pool string, per-model counts and concurrency, stop rule (two flat batches or six), escalation | prose inside the spec (`tasks/transforms2d.md`) |
| Floors | per generated function: which corpus binary rebuilds its cache, the score it must keep | `generated_score_check.rs` `FUNCTIONS` and `.floor.json`, raised automatically |
| Announcement | the features expected to move, and the exact-key rules that move them; the landing itself triggers the post, so a job that moves no feature still posts, with the reason in its map line | the manifest's `expected_features` and `feature_rules`, written into `docs/feature-map.json` by the job's `feature-map` stage; `bench/after-land.sh` passes the landed job's `result.json` to `bench/feature-announce.py` |
| Limits | wall-clock cap, writer relaunches, review rounds | `maxRounds` in `tracks.json`, three relaunches in `prompts/spec-watch.md` |
The manifest is one file beside the spec, read by a script, so nothing depends on prose.
Stages, in order. A stage is either a writer stage (a Claude session on the lane,
subscription-billed, doing work that needs judgment) or a script stage (a fixed command
whose output is checked). Only stages 1 and 6 and the review need a model; the rest are
commands, because the run must not depend on a writer choosing to do them.
1. Writer: hand-written substrate, types, baseline, corpus recipes.
2. Script: ground-truth cache from three browsers (`*_ground_truth_cache`). Check: case
count and drops recorded.
3. Script: the generation loop over the baseline (`*_generation_loop`), which seeds the
regression set. Candidates are scored against the browser-measured cases alone, with
no hand-written reference and no deliberately wrong fixtures (`docs/generation-loop.md`
step 2 gives the reason).
4. Script: the model batches, always through a batch wrapper that records a ledger line.
Check: at least one ledger line for the job's name; provider is not metered.
5. Script: admission and commit of the winner with its provenance record. Check: the
record's model is not `hand-written` unless an escalation is recorded, `cases_run` is
above 0, `cases_passed` equals `cases_run`, `regression_free` is true.
6. Writer: wiring, docs, floor entry. Not the `docs/feature-map.json` rules: stage 8
writes those from the manifest's `feature_rules`.
7. Script: rebase onto the carried base; `verify`; `landChecks`; the new function's floor.
A conflict resumes writer stage 6 on the lane with the conflict named.
8. Script: the `feature-map` stage, which `job-run` adds before review when the manifest
has `feature_rules` (`job-check feature-map`). It writes and commits the rules
(`job-check feature-rules`), then runs the map update as a dry run
(`bench/web-features-update.sh --no-commit`) and puts the snapshot and event log back.
Check: every listed key ends partial or complete through its own rule, no further back
than before, and every `expected_features` entry ends partial or complete; it need not
move, so a job that extends an already partial feature passes. After landing,
`bench/expected-features-check.py` prints a `NOT ANNOUNCED` line for a declared
feature that still did not move.
9. Review: a fresh read-only session on the lane, given the spec, the diff and the stage
results. It answers the tick's four questions plus "did each script stage produce its
artifact".
10. Result: a commit on `result/` in the lane's copy holding `result.json`: spec
sha, base sha, branch tip, per-stage status and artifact hashes, review verdict, ledger
lines, expected events. Written last, and only when every check passed. A failed job
writes the same file with the failing stage, so a poll can tell "failed" from "still
running". The poller keeps each result it reads (a landing, a failed stage, a job
waiting on a person) on tangled as the tag `result//`, and deletes the
forge's `job/` and `result/` after a landing only once that tag is pushed.
The runner and the stage checks are copied from `main` at submit time and run from that
copy, never from the job's own tree. A job that edits them affects later jobs, not its own
run, which keeps the reason `bench/land.sh` gives for protecting its own files (a track
must not rewrite the judge it is judged by) without putting the runner under `bench/`.
Before that first stage, on every start and every resume, the runner checks that the lane
actually has what the manifest is about to need: a resolvable `git`, `python3`, `node` and
`kill`, `cargo` and the oracle's Playwright browsers when a writer is in the stage list, a
reachable model gateway when a stage runs a pi session, disk space, the shared archive
directory, and enough wall-clock time left across resumes to be worth starting another
stage (`jobs/src/preflight.rs`). One unmet requirement prints every reason, records them in
state.json, runs no stage and exits 9 rather than let a writer spend a long stage only to
lose the next one to something as small as a missing `PATH` entry. A gateway or NAS share
that is only down exits 12 instead, after a bounded wait, and is resent by the poller.
An agent of kind `command` that runs `pi-writer.sh` or `pi-review.sh` may set `thinking`
(`off`, `minimal`, `low`, `medium`, `high`, `xhigh` or `max`); the runner passes it as
`--thinking`, a `fan_out` block's writers inherit it, and the stage's `thinking` in
`result.json` reports it (`tasks/job-agents-setting.md`). An agent that can run a
`deepseek*` model may use only `low`, `high`, `xhigh` or `max`; the gateway answers the
other levels with HTTP 400, so the manifest check refuses them.
A job survives a change in which models the gateway serves. An agent of kind `command` may
list `fallback` models after its `model`, and a `fan_out` block may list `fallback` spares
for its `models` (`jobs/src/fallback.rs`; no manifest on `main` lists any). At
each start the runner picks, from the lane's snapshot, the first model of each list that is
declared, not retired, listed and ready, and runs every stage of the role on it; the job is
refused only when no model of a list is usable (exit 10 when every one is out for good,
else exit 3 so it is resent). If a stage's command then exits 66 (`pi-session.sh` does when
its first request is refused with 401, 403 or 404), the role moves to its next model
and the stage runs again, without counting a failed attempt. `result.json` has
`model_choices`: the chosen model (or a `fan_out` stage's resolved `models`) and each model
passed over, with the reason and whether it was at start or while running; each stage's
`model` is the one it ran on. The `candidates` batch's `models` is a set run together, not
a preference list, and has no fallback.
### What a job does when a dependency drops out
A run pauses and continues by itself when a dependency it needs is down for a while. The
poller's resends for these are uncounted (they do not use up the three resends) and capped:
`infra_resends` in the job's record counts them in a row, `JOBS_INFRA_RESEND_CAP` (default 3)
is the limit, and the next one records `failed` with `failure=infra` and a message saying
so, which a person resends with `jobs/scripts/submit.sh ` once the dependency is back
(that resend is uncounted too). The count starts again when a run ends any other way.
| Dependency | How it shows | What happens | Caps |
|---|---|---|---|
| Forge, mirror or lane while `prepare` runs | `git ls-remote` or `git fetch` fails in `bench/dagu/job.yaml` `prepare` | `retry()` runs it again after a pause | 10 tries, 15 s doubling to 75 s (`PREPARE_TRIES`, `PREPARE_PAUSE`); then the step fails and the run ends `-` before the `run` step began, a failure marked `infra` that a person resends |
| Gateway or NAS share at the start | `job-run` preflight, exit 12 | the check waits, then the poller resends uncounted | 10 tries, about ten minutes (`JOBS_PREFLIGHT_WAIT_TRIES`, `JOBS_PREFLIGHT_WAIT_SECONDS`), then 3 resends in a row |
| Gateway during a stage | the stage's command (`pi-session.sh`) exits 67 on connection refused or reset, a timeout, 500, 502, 503 or 504 | `job-run` waits for the gateway (the same bounded wait) and runs the stage again, with no failure and no writer relaunch counted | 3 reruns per stage; a gateway that does not answer, or a fourth outage, is an ordinary failed attempt |
| Model not served | `pi-session.sh` exits 66 on a first request refused with 401, 403 or 404 | the role moves to the next model of its `fallback` list | the list |
| Forge at publish | `job-run` exit 7 | the poller resends uncounted | 3 in a row |
| Inventory snapshot stale or a model not ready | `job-run` exit 3 | the poller resends uncounted | 3 in a row |
| Runner stopped (other than by `stop-run`, below) or killed after the `run` step began | exit 8, or no exit code with `run` started | the poller resends uncounted; `job-run` resumes the interrupted stage (`JOB_RESUMED`) | 3 in a row |
| Anything permanent (a missing binary or variable, a retired model, a stage's own failure, exit 1, 2, 6, 9, 10, 11) | those exit codes | recorded `failed` or `needs-person` for a person; not resent | none |
A run a person stops with `bench/dagu-lane.sh stop-run job ` is not resent: before
asking Dagu to stop it, `stop-run` leaves the file `stopped-runs/` in the job state
directory (`JOBS_STATE_DIR`, default `$HOME/.local/state/webengine-jobs`), and the poller,
finding it, records the job `stopped` with the line "resend with jobs/scripts/submit.sh
" whether the run ended with exit 8 or with no exit code, then deletes the file. The
file is written on the machine that runs `stop-run`, so this holds only when that is the
machine the poller runs on (the box). A stop any other way after the `run` step began is
still taken as a killed runner and resent as in the table.
Every resend by the poller goes with the per-run settings the job's record keeps (`stop_after`,
`model`, `batch_models`, `rerun`, `pass_failing_cases`) and never with a rewind, which the
run that asked for it has already done.
A writer stage with a `fan_out` block runs several writers on several models, each in its
own checkout. The runner starts at most `WEBENGINE_JOB_WRITERS_AT_ONCE` of them at a time
(default: the lane's CPU count divided by 16, at least 1) and queues the rest; a killed
runner reruns only the writers that had not ended. The branches of the writers not chosen
are kept as `archive/job-writer//-` and pushed with the result.
The block's `choose` says which writer continues: `first-pass` (the first to pass the
stages through `through`); `score` (the block's `score` command runs in each writer's
checkout and prints one number, its last numeric line; the highest continues, a tie goes to
the smaller diff, and a writer whose command fails or prints none is out); and, on a review
stage, `majority` (each reviewer runs the review in the job's checkout with its own state
directory; the review fails when more than half fail, and the findings handed back are those
two reviewers report in the same words apart from case, spacing and a final full stop, plus
each finding of a failed review that names a file and a symbol). The step's `detail` in the
progress events names the writer that continued and its score, or how many reviewers
failed.
`merge` (a writer stage; it needs `score`) scores the writers as `score` does, then, when at
least two passed and were scored, runs one more session on the best writer's model in a
checkout of the job branch's tip. It is handed a file (`JOB_MERGE_FILE`) listing every branch
with its score and diff, and integrates them. Its result is scored the same way and
continues when it passed and scores at least as well as the best single branch; otherwise the
best branch continues and the merge's branch is kept as `archive/job-writer//-merge`.
The merge session is recorded as writer 0 in the stage's `writers`. With `score` or `merge`,
when the writers' scores differ or some changed files others did not, the runner writes
`writer-notes.md` in the job's state directory and every later stage gets `JOB_WRITER_NOTES`
naming it.
Each writer (and reviewer, and the merge session) of a `fan_out` stage sends its own events
(`docs/job-events.md`: a `writer` event when it starts and when it passes or fails, with its
number, the stage's writer count and its model; `pi-writer.sh` passes the writer's number to
`pi-session.sh`, which sends the progress heartbeat under it). The dashboard lists each writer of the current attempt with its
state, turns and last activity, and `jobs/scripts/status-line.sh` prints the same for the
running stage; a running writer with no activity for 10 minutes is shown as stalled. In
`result.json` each stage has `duration_seconds`, and each writer of a fan-out stage has
`started_at`, `ended_at`, `duration_seconds`, `turns`, `tool_calls` and `tokens` (input plus
output, totalled from the `usage.jsonl` that `pi-session.sh` appends to in the writer's state
directory; a reviewer's sessions are not counted, so its usage is 0).
A stage that failed also has `log_tail`: the last 40 lines of its log (`logs/.log`),
cut from the front to 4,000 bytes, so the reason a stage stopped is in the record without the
log. A stage that passed has none. `poll-and-land.sh` keeps the stage commits of a run that
failed or waits for a person as the tag `stages//`, as a landing does for a
passed run, and leaves `job-stages/` on the forge for a resend: the findings stages'
pages and their browser-measured ground truth (`corpus/-findings-pages`,
`corpus/-findings-ground-truth`, outputs of their stages) are in those commits.
`result.json` also records which model did what. Each stage that ran agent sessions has
`sessions` (one entry per session: its stage, `model`, `thinking` level, fan-out `writer`
and `round`, with turns, tool calls and tokens) and `thinking` (the distinct levels, `default`
when pi's own was used; a DeepSeek model defaults to `high`). `summary` counts sessions by
stage and by model (writers' included), the rounds used against the failed-round limit,
stage retries, the batch stage's candidates and compiled per model, and the run's turns,
tool calls, tokens, wall-clock seconds and energy. `authorship` lists the job branch's
commits since `base_sha` with the model of each (the `Model:` trailer a hook in
`pi-session.sh` adds to every commit a session makes). A commit a script stage makes carries a
`Stage:` trailer instead (the feature-map stage's commit has `Stage: feature-map`) and is listed
as `{"model": null, "stage": "feature-map", "kind": "script"}`, counted as `script:feature-map`;
a commit with neither trailer is `unknown`. Each entry has `sha`, `model`, `stage` and `kind`
(`model`, `script` or `unknown`). `authorship` also gives the commit count per model (or
`script:`), and per file the same keys for the commits that touched it. These are per commit and
file, not per line.
A feature job may split its `substrate` writer stage into phases, each its own named, timed
stage: `test-cases`, `test-fixtures` and `test-ground-truth-binary` (the test cases, the
baseline fixture, and the ground-truth binary, one writer stage each), `ground-truth` (script),
`runner` (generation runner, prompt, admission record, the loop and batch binaries),
`generation-loop` (script), `build` (script, `cargo test --workspace --locked --lib --bins`)
and `gate` (script, `bench/verify.sh`), then `batch`, `admit`, `test` and `review` as before.
A failed `generation-loop`, `build` or `gate` has `on_fail: "rework"`, so it goes back to the
nearest writer before it, `runner`. A manifest may keep one `substrate` stage; no manifest
on `main` uses the split. No stage writes a hand-written reference implementation or
deliberately wrong candidates: candidates are scored against the browser-measured cases
alone (`docs/generation-loop.md` step 2 gives the reason).
### The model prompt and its check
What the model prompt of a generated function carries. The `model_prompt` command builds it
from three stored inputs, with no model session and nothing written per feature
(`crates/generation/src/model_prompt.rs` `build()`, over the parts in
`crates/generation/src/prompt_parts.rs`; `docs/geometry-verification.md` → "Cases from a
feature's existing WPT pages" has the command and its arguments). The first aspect-ratio
run sent a paraphrase, so the rule that a minimum beats a maximum, in CSS 2.1 sections 10.4
and 10.7 and referenced by CSS Box Sizing 4, never reached a model and 41 of 43 candidates
got it wrong. A prompt holds, in this order:
- The whole readable text of the specification section the function implements, never an
excerpt or a paraphrase, with the part that holds the rule being implemented between
`RELEVANT_BEGIN` and `RELEVANT_END` (`SpecificationBundle::render()`).
- Each cited section whole (the section only, not the specification around it), under its
own heading. The cited sections are the `references/` files of the folder the
`specification_text` command writes, one per `--reference` argument given to it
(`docs/provenance.md` → "The text files of one function"). That command reads the
project's copy under `corpus/specs` and never the network, and refuses a section that is
not cached. `SpecificationBundle::from_dir()` reads the folder: `specification.txt`,
`relevant.txt` and every `references/` file its `sources.json` lists, in the order listed,
each checked against the SHA-256 recorded for it (`read_listed()`). A listed file that is
missing or altered is an error naming the file, never an omission. A prompt holds at most
6 cited sections and 60000 characters of their text (`MAX_REFERENCES`,
`MAX_REFERENCE_CHARS`); `SpecificationBundle::new()` refuses a larger set instead of
trimming it, so a smaller set is named to `specification_text`. The folder's
`sources.json` (each text's URL with its anchor, and its hashes) is copied beside the
prompt unchanged, so a run can be reproduced.
- The types file, byte for byte, under `# Implement this exact signature`. Every field's doc
states what the input means: its unit, which box it measures (content box, or the box's own
box-sizing box with padding and border), whether it may be absent or `auto`, and whether it
may exceed another input. A doc never contradicts the specification and states no rule
about which of two limits wins; that is the function's job.
- `# Contract` (`model_prompt.rs` `contract()`), the one text that is not an input: the
module a candidate imports the types from, that it redefines none of them, that a path the
types file writes as `crate::...` is `::...` in a candidate, and the name and type
of the one function called. The names come from the command's `--function` and
`--resolver` arguments (`Function::new()`).
- Every kept case of the cases folder, in case id order (`groundtruth.rs` `load_all()`),
each shown as, for each box the engine calls the function for, the inputs it hands the
function and the browsers' content box for that box as the expected output
(`render_function_case()`; `docs/geometry-verification.md` → "Building the prompt"
says how the calls are recorded), with the case's page before the calls unless the
command is given `--no-page-html` (`prompt_parts.rs` `PageHtml`). The feature jobs give
it: the function never sees a page, only its inputs, and a WPT page carries text that
points away from the rule (a ` ` (`jobs/src/checks.rs` `prompt()`)
fails the job before any model call when the prompt lacks a specification text or an
expected result. It reads `prompt.txt` and `expected.txt` from the first folder, where
`model_prompt` writes them, and the specification texts from the second, which is the
folder `specification_text` wrote: no copy of a specification text is kept beside the
prompt. It fails (exit 1) when:
- the specification folder is not one to compare with (`specification_texts()`):
`sources.json` is missing, is not JSON or has no `texts` list; an entry lacks `file` or
`text_sha256`, names a file that is not `specification.txt`, `relevant.txt` or a file
directly under `references/`, or names a file twice; a listed file is missing or does not
hash to its `text_sha256`; or `specification.txt` or `relevant.txt` is not listed;
- the prompt lacks the whole of `specification.txt`, does not hold `relevant.txt` between
`RELEVANT_BEGIN` and `RELEVANT_END`, or lacks the whole text of any cited section the
folder lists (the message names the `references/` file);
- the prompt lacks an expected-result block of `expected.txt`, or `expected.txt` holds no
block.
It cannot run (exit 2) when `prompt.txt` or `expected.txt` cannot be read. Whitespace is
ignored in every comparison (`squeeze()`). A stage's `checks` entry
`{"check": "prompt", "args": ["", ""]}` runs the check
when the stage passes, each argument resolved against the job's working directory
(`jobs/src/runner.rs` `run_checks()`; `jobs/src/manifest.rs` `CHECKS` gives the check its two
arguments). `job-check manifest` refuses a manifest whose model batch (a stage running
`run-batch.sh`) has no earlier stage with that check (`manifest.rs`
`prompt_check_violations()`). No manifest under `specs/` runs a model batch, so none carries
the check.
Two things the check does not do. It compares text, so it cannot tell whether a types
file's docs agree with the specification; the review carries that. And it compares the prompt with the specification folder it is given: the
`sources.json` beside the prompt records, by URL and SHA-256, the texts the prompt was built
from, and the check does not read it.
**The types file's check.** `job-check types ` (`jobs/src/checks.rs` `types()`,
rules in `jobs/src/types_file.rs` `check()`) is the output test of the step that writes a
generated function's types file, the part that needs no checkout. The file must parse as
Rust (`syn`); declare `pub` types and exactly one resolver type (`pub type X = fn(&Inputs)
-> Outputs`) and nothing that runs (a `fn` with a body, an `impl`, a `mod`, a macro, a
`const` or `static` each fail, named with their line); open with a module doc that states
`Scope` with an `In:` and an `Out:` list; and document every struct, enum, variant and
field, where a number's doc names its unit (`px`, `unitless`, or "the same unit ... as"
another field) and a flag's doc opens with `Whether`. The stored `aspect-ratio` types file
passes (2 structs, 17 fields). Not checked here, because they need more than the file:
that it compiles against the engine (`cargo check -p layout` with the file in place, the
engine-wiring step's test) and that no sentence in it contradicts a measured case (a
reading). A stage may name it as `{"check": "types", "args": [""]}`.
**Request and score, each alone from stored inputs.** `request_candidates --prompt
--count --out ` (`crates/generation/src/bin/request_candidates.rs`)
sends a stored prompt to the gateway the environment names (`WEBENGINE_MODEL_URL`,
`WEBENGINE_MODEL` or `WEBENGINE_MODELS`, `WEBENGINE_MODEL_API_KEY`), one request per seed,
and writes each reply as it came under `replies/`, the Rust it carries under `sources/`
(`model.rs` `extract_rust_source()`) and `requests.json` with each request's model, seed,
tokens, wall clock and finish reason or error. `score_candidates --function aspect-ratio
--sources --cases --work --out `
(`crates/generation/src/bin/score_candidates.rs`) takes such a `sources/` folder, or any
folder of `.rs` candidates, writes them into one crate (`candidate_crate.rs`
`Seam::AspectRatio`), compiles it and scores every candidate over the measured cases with
`aspect_ratio_runner.rs` `evaluate()`, every other generated function running as landed;
the report is `runner::Report`. Two hand-written candidates are stored under
`crates/generation/tests/fixtures/aspect-ratio-candidates/` (its `NOTICE`) so the scoring
step runs with no model: over the 33 measured train cases the baseline passes 12 and a
first attempt at CSS Box Sizing 4 §4.2 and §4.4 passes 32. Before the crate,
`score_candidates` compiles each candidate on its own (`candidate_crate::probe`); one that
does not compile is left out of the report and its compiler messages go to
`/compile-errors/.txt`, so each batch wants a `--work` folder of its own.
**Repair, alone from stored inputs.** `repair_candidates`
(`crates/generation/src/bin/repair_candidates.rs`) asks the model once more for each
candidate that is wrong, in one user turn (`compile_repair::user_turn`): the prompt, the
candidate's source alone in a `rust` block as code to fix (`compile_repair::source_block`;
not the reply it came from, whose reasoning can run to ten times the code), and what is
wrong (`compile_repair.rs`). Sent as the model's own earlier answer instead, the source came
back from Kimi unchanged in most rounds. With `--errors /compile-errors` it repairs
every candidate that did not compile, showing its compiler errors by line and column
(`compile_repair::message`); with `--report --cases --function
--resolver ` it repairs every candidate that fails a case, showing each failed case's
boxes that come out off the browsers' majority and, for a box the function is called for,
the call's inputs and expected output (`compile_repair::case_message`; the scorer fills
`CandidateReport::failing_case_diffs`, `aspect_ratio_runner.rs` `vote_diff()`). It writes
`replies/`, `sources/` and `requests.json` as `request_candidates` does, each candidate under
its own name, so its output is the next `score_candidates --sources` and the next repair's
`--batch`; given both `--errors` and `--report`, one run repairs both kinds. Each request
moves the candidate's seed on (`compile_repair::next_seed`), since the same request with the
same seed comes back the same, and each record in `requests.json` lists the keys of every
repair request the candidate was sent (`asked`, `compile_repair::request_key` of the source
and the message); a candidate whose next request would repeat one of them is dropped, since
its last repair came back failing the same way. A repair reply's source is its closed `rust`
block (`model.rs` `extract_tagged_rust_source()`); a reply with none keeps the candidate's
previous source rather than compiling its prose, and a compile repair shows at most the
first `compile_repair::MAX_ERRORS_SHOWN` errors with a count of the rest. A case repair
often breaks compiling (5 of 12 on `aspect-ratio`), so each one is followed by a compile
repair. `bench/repair-loop.sh` runs the rounds: it scores the latest folder, stops when a
candidate passes every train case, when the best train score has not risen for 3 rounds
(box run `034eWRT4E70xnVxCtwTumo` reached 60 of 62 at round 8 and rounds 9 to 12 added
nothing), or when a round limit is met, and otherwise repairs every
wrong candidate into `-round`, keeping each round's folders (its score report in
`-round-work/report.json`, the batch's in `-work/`); the held-out cases
are scored for the final candidate alone. Each round repairs from the better of every
candidate's previous and repaired versions: a repair that passes fewer train cases, or
breaks compiling, is set aside (a tie keeps the repair) (`-round-work/kept/`), and the next request for
that candidate opens with `compile_repair::SET_ASIDE_NOTE` and shows the next slice of its
failing cases (`compile_repair::next_slice`), so it is not a request already sent. On `aspect-ratio` with `qwen3.8-27b-fast`, a candidate that did
not compile went 23, 30, 32 and 33 of 33 train cases over one compile repair and three case
repairs, and passes 6 of the 7 held-out cases.
A repair asks for the whole source again, not SEARCH/REPLACE edits to it. Edits were
measured on box replays of folknet run `034fcGTHF0EaocdU3L3eUC`'s first batch with
`kimi-for-coding` (53 train cases, the engine's function 50): whole source reached 49 train
and 8 of 10 held out; edits only, skipping an edit that did not match, 48 and 5 of 10
(edits applied in 35 of 36 replies); edits with whole source allowed, 46 and 8 of 10. In
every form a repair of a source at 45 or better came back at 33 to 47, from a wrong fix
rather than from rewriting lines it was not asked to touch, so edits confine nothing that
matters. Not measured with a job model on folknet.
A job lands a candidate that improves on the function the engine has, even when it misses
train cases: getting a partial function landed for a feature comes first, and later jobs
raise it. Before round 0 the loop scores the engine's current function on the job's own
train cases (`crates//src/generated/.rs` for `--resolver
::::`, or `--current`); when that file is missing or does not compile
against the types file in place, it scores the hand-written fallback instead
(`::::resolve_baseline` when that module declares it, or `--fallback`).
`-work/current.json` records which one set the bar (`scored`: `current`, `fallback`,
or `none` when neither could be scored, which counts as passing no case) and the cases it
passes. A candidate clears this landing bar when it fails no train case the scored function
passes and passes more train cases than it, or every one. The `.floor.json` floors are
not the bar: they count cases on `generated_score_check`'s corpus, not the job's train cases.
The final candidate is the one clearing the bar with the most train cases, or, when none
clears it, the one with the most. The loop exits 0 when the final candidate clears the bar
and 1 when none does, and `jobs/scripts/store-candidate.sh` refuses (exit 1) a final
candidate below it, so a job whose repairs fall short stops at `repair`. A candidate that
clears the bar but misses train cases is stored as partial: its commit subject ends
`; partial, train X of Y, current function B`, its body has a `Known failures: `
line, and it writes `-store.json` for the review, whose `--known-failures` turns a
`fail` resting only on those cases' inputs into a `pass` with `verdict_was` `fail`, since no
round is needed for it (`specs/aspect-ratio.md`, the `review` stage). The review does not
decide the landing of a feature job: its review stage is advisory (`"advisory": true`,
`jobs/src/runner.rs` rule 22), so a `fail` that stands starts a round while one can add a
case, and otherwise the job lands the candidate on the score bar, with the review's findings
committed beside the function as `known-failures/.json` (each wrong input, the
failing train cases it matches, and whether it needs an input the inputs struct lacks), a
list later jobs work through. Landing on the score bar alone puts nothing into the engine:
the job's `admit` stage, after the review, runs only when the review ran and passed
(`jobs/src/runner.rs` rule 23), and is passed without running otherwise. When it runs,
`jobs/scripts/admit.sh --stored-dir` scores the stored candidate on the score check's own
corpus against the committed function, and when it passes more cases commits it as the
generated function with its types file, provenance record and doc rows (no floor file,
which `bench/land.sh` would refuse); the
`feature-map` stage after it writes the job's `generated` rules. A later round whose repair or store falls below the bar lands
the candidate the earlier round stored. In a review round after the first, `store` replaces
the candidate an earlier round of the same job stored: the paths under its folder that the
job's earlier `Stage: store` and `Stage: review` commits (those beyond `JOB_BASE_SHA`)
touched, and `NOTICE`, go back to the
job's base before the new candidate is written, and the commit body has a `Replaces:` line,
so a job branch carries one candidate, the one its last review judged. When such a run lands, `jobs/scripts/poll-and-land.sh` adds
`partial, train X of Y, N known failures` to the `landed` record's message (and, after an
advisory review that failed, `review failed, landed on the score bar, N known failure(s)
from the review`), the run's
history record `runs//.json` carries `candidate` (`train_passed`,
`train_run`, `misses`, `known_failures`), and `bench/after-land.sh` raises the function's
floor, or writes its first (`bench/generated-floors-update.sh`), so a later job cannot lower
it.
A job sent from the development box with its models replaced may go on below the bar:
`JOBS_PASS_FAILING_CASES=yes` on `submit.sh` (refused without `JOBS_MODEL` or
`JOBS_BATCH_MODELS`, sent as `PASS_FAILING_CASES=yes` and exported to the stages by
`bench/dagu/job.yaml`) means "store a candidate below the landing bar, never land the run".
The loop then exits 0 whenever there is a final candidate, and `store` commits one below the
bar with a `NOTICE` paragraph and a commit subject that say it misses cases (train X of Y)
and was stored under `JOBS_PASS_FAILING_CASES`. Those runs, against Kimi with reasoning off,
test the job's later stages and its lifecycle, and their landing is checked, not done, even
when the candidate cleared the bar: `jobs/scripts/poll-and-land.sh` runs `bench/land.sh
--check` for a passing run whose record has `pass_failing_cases`, merges and tags nothing,
and closes the record as `checked`. A function goes into the engine only from a folknet
job, which never carries the setting. A loop with no final candidate (none compiled) still
exits 1; one stopped by a round that gave no source exits by the same rule as any other
stop, on its final candidate so far.
### How it is shipped and started
Existing pieces, and where each falls short of an unattended run:
| Existing | What it does | Falls short |
|---|---|---|
| `bench/remote-lane.sh` | ships a bundle of commits not on `origin/main`, a diff and untracked files in one ssh stream, and runs a command in a lane copy | carries the worktree's HEAD only, not a base ref; the command is foreground and ends with the connection unless the command detaches |
| `bench/lane-writer-launch.sh` | syncs a worktree, `git checkout -B`, pulls pi-tracks, runs its writer command (`--tool` from `specWatch.lanes.tool`) with the brief, prints `{ssh, sshOpts, path}` | starts one open-ended writer session with a brief, not a runner with stages; needs a clean local worktree (exit 5), so a dev-box worktree must exist first; refuses over unfetched lane commits, which forces a fetch before any relaunch |
| `bench/lane-writer-fetch.sh` | fetches the branch from the lane copy and fast-forwards | fast-forward only, so a local rebase makes every later fetch report "diverged"; fetches the working branch, so "done" is inferred from liveness and transcript age (`spec-status.mjs` `writerFor()`, `isStalled()`) instead of read from a commit |
| `bench/lane-apply.sh` | checks out the exact commit in a lane copy, applies the diff, exports `WEBENGINE_ON_LANE=1` | nothing stops a lane run from being cleaned under a live job (`git clean -fdx`); the writer launch guards it, a runner would need the same |
| `spec-watch-launch.mjs` | picks a lane, falls back to local | the local fallback is the failure state; exit 4 exists only because of it |
The proposed submit does, from the development box and in this order: verify the spec sha
is the one on `main`, verify `depends-on` specs are landed, bundle the job branch and the
base tip, pick the lane with the lowest load, send the bundle and the manifest through the
same single ssh stream `bench/remote-lane.sh` uses, and start the runner detached on the
lane (its own session, a pid file and a lease file it touches every minute). Submit prints
the lane path and returns. It never starts anything locally: with no lane answering, it
exits non-zero and the job stays unsubmitted (the same rule `bench/on-lane.sh --lane-only`
gives a model batch).
The runner is resumable: each stage writes its status into a state file in the lane copy,
so a lane restart or a killed runner is handled by starting the runner again on the same
copy, and it continues from the first unfinished stage.
### What the development box does
```mermaid
sequenceDiagram
participant D as development box (script, no model)
participant L as lane
participant P as primary checkout
D->>L: submit: bundle, manifest, start runner
loop every few minutes
D->>L: poll: lease, then git ls-remote result/name
end
L-->>D: result commit (fetch)
D->>P: land: bench/land.sh name spec-sha tip
P->>P: fast-forward, tag, map update, announcement
```
- **Submit.** One script per enabled spec whose dependencies are landed and whose slot
count allows it.
- **Poll.** One ssh call per running job: read the lease, and `git ls-remote` the result
branch. No lease refresh for ten minutes and no result: mark the job lost and submit it
again (resumes). A result that says failed: record the stage, park the job.
- **Land.** Fetch the result branch and the job branch, check the result's spec sha, base
sha and tip against what `bench/land.sh` will read, create the local worktree
`bench/land.sh` needs from the fetched branch, run `bench/land.sh
`, then `bench/after-land.sh` runs as it does today. `bench/land.sh` keeps its own
gate rerun and its own protected-file list, so a wrong result cannot land anything the
gate rejects.
The worktree in the primary checkout exists only for the length of the landing. Nothing
writes a file there, so there is no `handoff.md` to conflict with the roster, no local
rebase, and no local writer. This step needs no model session, so it is a plain script
(cron) rather than a `claude -p` tick.
Left outside the job on purpose: writing and enabling specs (gap 1), which stays with a
person or a coordinator session, and pushing `main` (gap 9), which needs a decision below.
### How the gaps map onto the shape
| Gap | What it blocks in the shape | Closed by |
|---|---|---|
| 1 Hand-written specs | a malformed spec gives the job nothing to run | a spec lint; specs stay outside the loop (below) |
| 2 Map rules | the announcement stage | stage 8, and the manifest's `feature_rules` (done) |
| 3 Floors by hand | stage 7 for every function | floors as data read by the checker |
| 4 Prose not enforced | stages 2 to 5 being real | script stages with artifact checks |
| 5 Batch bypass | the ledger, so the post's numbers | stage 4 always through the batch wrapper |
| 6 Refusals | landing | lint on `touches` at submit; a job that would touch `bench/` is not submitted |
| 7 Dev-box tick | "only submit, poll, land" | the runner on the lane, review inside the job |
| 8 Local fallback | "nothing heavy on the development box" | strict mode for unattended gates |
| 9 Push by hand | public visibility only | a decision, not a blocker |
### Settled
- **The runner and its scripts may live outside `bench/`.** The loop has to be unattended,
and `bench/land.sh` refuses any diff under `bench/`, so a runner kept there could never
be landed or changed without a person. The split is accepted: some scripts stay in
`bench/`, the runner and its stage checks live elsewhere.
- **A running loop does not write or enable specs.** Specs are written by a person or a
coordinator session, outside the loop. Finished features are reviewed afterwards,
asynchronously, and a review that finds problems sends the feature back through the loop
as a new or amended spec (a new spec sha, so a new job). Spec authorship stays with a
reviewer; the loop's job is to take an enabled spec to a result commit. There is
therefore no drafting stage in the job and no path for adding to `specs/` from a track.
- **Dagu ships, queues and supervises a job; `job-run` runs its stages as one step.** A
trial on a lane ran fixture jobs end to end through Dagu, and a real spec has since run
the same way ("First real run" below) (`bench/dagu/job.yaml`,
`docs/infrastructure.md` → "Dagu on a lane"): sent over its MCP server from the
development box, working copy from the lane's mirror at a base on the forge, `job/`
and `result/` published to the forge, a queue limit, a resume after a stop, and a
restart of the lane. So the detached start, pid and lease handshake, ssh lease probe and
`lost` status drafted in "How it is shipped and started" and "What the development box
does" above are not built: Dagu starts and queues the run, its status replaces the lease
probe, and a resend with the same parameters resumes the job. The queue's limit replaces
`specWatch.maxLive`. `job-run` stays a single step rather than one Dagu step per stage,
because a step per stage would rebuild in workflow YAML what `jobs/src/runner.rs`
already does under tests: pinned tools, compiled-in checks, the per-role environment,
the writer's relaunches, the summed wall-clock limit, the review verdict, the state file
and the result commit and its push. What stays on the development box is
`tasks/job-submit-land.md`: the repository's refusal checks before a submit, and the
fetch, `job-check result` and `bench/land.sh` at landing.
### First real run
A real spec has since run the loop end to end, not only the fixture job the Dagu trial
above used. Spec `viewport-units` (manifest `specs/viewport-units.job.json`: pi writer and
review on `deepseek-v4.1-flash` through the gateway; stages write, test, measure, review)
ran as Dagu run `034VMxPcXtPSIim2wljbuk` on lane 0 on 2026-09-26 and ran every stage. The
start-of-job check (`jobs/src/preflight.rs`) passed. `write` made two commits, one of them
a new file, `crates/layout/src/bin/viewport_units_check.rs`, with 69 cases; `test` passed;
`measure` compared all 69 cases across chromium, firefox and webkit, 69 passed and none
dropped. `review` returned `fail` on a real defect: `browser_verdict` compares only
adjacent browser pairs (`names.windows(2)`), so chromium and webkit are never compared
directly, though the spec requires every returned dump to agree. `job-run` exited 1 and
published `job/viewport-units` and a failed `result/viewport-units`; `poll-and-land.sh
--check` reported the review failure. Logs landed in
`archive/job-logs/viewport-units//` on the NAS share with exactly the files the
archive handler in `bench/dagu/job.yaml` lists, and a scan of them found no keys, tokens,
credentials, addresses, hostnames or personal home paths.
A first attempt at the same spec (`034VKGQgP1MDBABX0D5rJB`) wrote for 1899 s of a 2400 s
limit, then failed at stage `test`: the `run` step of `bench/dagu/job.yaml` did not put
`~/.cargo/bin` on `PATH`, and `job-run` starts a script stage's command without a shell
(`jobs/src/runner.rs` `run_command()`). The wall-clock limit counts across resumes, so the
job was restarted from scratch with a 3600 s limit.
The two runs exposed gaps, each fixed since: no `job-check` on the development box
(`jobs/scripts/install-job-check.sh`); `bench/dagu-lane.sh` without `--lane` reached a lane
without Dagu, so it now defaults to `WEBENGINE_DAGU_LANE` or lane 0; `bench/on-lane.sh
--lane-only` refused inside a job, so it and `bench/remote-lane.sh` now run in place under
`WEBENGINE_ON_LANE`; cargo missing from the `run` step's `PATH`; a resend always went at the base
tip while the lane resumes only at the base a job started from, so `submit.sh` now keeps
`base_sha` in the record; requirements went unchecked until a stage needed them, so
`jobs/src/preflight.rs` now exits 9 with every reason, read back by `poll-and-land.sh`
through `bench/dagu-lane.sh step-log`; and a `Cargo.lock` missing a new dependency, which
the job's `--locked` build refused while lane test runs had updated the lock file silently,
so `bench/verify.sh` now tests with `--locked`. A restart from scratch was done by hand
(deleting `job/` and `result/` on the forge, `bench/dagu-lane.sh clean-jobs
` for the lane's `jobs-work//`, `bench/dagu-lane.sh prune-mirror`, and moving
the local record aside), so `jobs/scripts/submit.sh --restart ` now does all four
and sends the job again at the base tip; `clean-jobs` itself never touches the forge.
A resend resumes at the failed stage inside the same round, at the base and spec blob the
job started with, and it may carry a changed `specs/.job.json`: a fixed stage command
no longer forces `--restart`, which would delete `job/` and discard every passed
stage. When the manifest at the base tip differs from the one at the recorded base,
`submit.sh` checks it with `job-check manifest --spec`, then sends `MANIFEST_BASE=/events.jsonl`). The first run's logs, its job record
(`job-record.json`) and these stage times (`baseline.txt`) are under
`archive/job-logs/aspect-ratio/034VztFIHkkBEpA6VFkhuR/` on the NAS share; its
`result.json` went with the forge's `result/aspect-ratio`, deleted at landing, and its
content is in `state/state.json` and `state/review.json` there.
The rerun passed every stage in one round as Dagu run `034WA9rX2fx1UlV9szr3Bd` (base
`03948e5`) and was landed by the scheduled poller at `c362bc4`. The admitted function is
again a model candidate, `model_34` from `qwen3.8-27b-fast`, at 109/109. Stage times, from
the job event store, with the first run's in brackets: substrate 35m28s (2h26m),
ground-truth 5 seconds, generation-loop 50 seconds, batch 5m48s, admit 18m32s (11
minutes), test 36 seconds, review 40m15s (30 minutes); 1h41m37s from start to publish
(3h09m). The review session took 88 turns and 92 tool calls, with 392,574 uncached input,
43,221 output and 5,026,304 cache-read tokens, so the model server reuses the shared start
of the context across turns. Admit grew because the substrate writer left the
`aspect_ratio` entry in `crates/generation/src/bin/generated_score_check.rs` `FUNCTIONS`
and a test in `crates/generation/src/admit.rs` for admit to restore: with them in place,
`bench/verify.sh` would have run the ground-truth binary that the substrate's `--task`
forbids. The writer stages' turns and tokens are not known, because the rerun's logs were
lost: the archive script in `bench/dagu/job.yaml` failed to create its temporary folder on
the NAS share, and step `cleanup` then removed the job folder, the only copy. `cleanup` now
keeps the job folder until the archive succeeds, and the archive keeps each pi session's
`events.jsonl`.
### Proposed changes
Each change lists what it makes harder later, and one tag: **spec now** (can be
commissioned as a spec that lands through `bench/land.sh`), **decision** (needs a person to
choose), **action** (needs a person to do something a track cannot).
1. **Spec lint as a test.** `crates/harness/tests/spec_lint.rs`, in the style of
`no_seam_word.rs`, reads `specs/*.md` and `tasks/*.md` and enforces the drafting rules of gap 1 plus:
`touches` never names `bench/`, `specs/`, `tasks/` or the other protected paths, and a
generated-function spec names `docs/feature-map.json`. Harder later: a wrong rule
rejects every new spec until the test is fixed. Existing specs that break a rule need
an allow-list entry, since a spec's sha must not change. **Spec now.**
2. **Provenance check as a test.** A test over every `*.provenance.json` under
`crates/layout/src/generated/` requiring a model other than `hand-written` unless an
escalation is recorded, and `cases_run` above 0. Harder later: hand-written
generated functions need a recorded escalation field. *Unverified:* whether the older
hand-written modules would pass; an allow-list may be needed. **Spec now.**
3. **Floors as data.** Done differently from the plan below: one `.floor.json` per
generated function, read by `generated_score_check.rs` (which also names each function's
cache binary), raised by `bench/generated-floors-update.sh` after each landing, and
protected from a track by `bench/land.sh` refusing any diff that touches a `*.floor.json`
(which replaces the comparison against `main`'s table). `bench/seam-floor.sh` and
`seam_floor_check.rs` are gone. `tasks/floors-table.md` is superseded.
4. **Runner, stage checks and manifest.** A new top-level directory, not `bench/`, so a
spec can commission and land it, because `bench/land.sh` refuses any diff under
`bench/`. Harder later: two places hold lane scripts, and the pinned-from-`main` rule
must be kept or a track could edit its own judge. **Spec now**, in this order: manifest
and stage checks first (also usable by hand), then the batch wrapper that always writes
a ledger line, then the runner.
5. **Submit and poll-and-land scripts.** Submit reuses `bench/remote-lane.sh`'s payload;
poll-and-land wraps `bench/land.sh`. Harder later: `bench/land.sh` still expects a
local track worktree, so the poller builds one at landing time; a later `bench/land.sh`
change to land from a fetched ref would remove that. **Spec now** if placed with the
runner (item 4); a change to `bench/land.sh` itself is an **action**.
6. **Replace the model tick for genera.** The cron entry runs the poll-and-land script
instead of `spec-watch-tick.sh`. That is a change to pi-tracks (a shared package other
repositories use), so it is opt-in by declaration in `tracks.json`, the way
`specWatch.landScript` is. Harder later: the model tick's review and conflict
handling are gone for these repositories, so every failure mode moves into the job
result. **Action** (a pi-tracks change, and editing `tracks.json`, which
`bench/land.sh` refuses).
7. **Strict lane mode for unattended gates.** `bench/verify.sh` and each `landChecks`
command use `bench/on-lane.sh --lane-only` when a variable the poller sets is present,
and submit refuses a local fallback. Harder later: with both lanes down nothing lands,
which is the intended result. The landing half is done (`WEBENGINE_LANE_STRICT=1`, see
"Hand changes still needed", item 2); the submit half waits for submit itself.
8. **Asynchronous review and send-back.** A person or a coordinator session reviews
finished features after the result commit, outside the loop, and does not hold a
landing. A review that finds problems writes a new or amended spec (linted by item 1),
which re-enters the loop as a new job. Harder later: review lags the loop, so a
problem found late is fixed by a follow-up spec on top of what already landed, not by
holding the landing; and a reviewer has to be available for the loop's output to be
checked at all. Nothing in the loop writes or enables specs, so `specs/` and `tasks/`
stay protected by `bench/land.sh`. **Action** (a person or a coordinator session does the
review; nothing to build in the loop).
9. **Push `main` and tags.** The poller already pushes a landed job's `spec/*` tag;
`main` is still pushed by hand. Harder later: an unattended push publishes code no person
has seen. **Decision** (`docs/coordinator-on-folknet.md` → "Who pushes" and "Open
questions" hold the deploy-key design). Not needed for the loop.
10. **Coordinator on folknet.** With the development box reduced to submit, poll and
land, moving those three duties to a guest is smaller than it was, and still needs a
person to create it and sign in (`docs/coordinator-on-folknet.md` → "Cutover").
**Action.**
11. **Fix the roster's stale tick text.** The roster's "Resume Here" still says an owning
session drives ticks. Nothing is made harder; the roster is on `main` and outside
this track's scope. **Action.**
Suggested order: items 1 to 3 (Rust side) first, since they need no decision and make the
present loop safer at once; then items 4 and 5; then actions 3, 6 and 7 together, since
they change the same scripts. Item 8 runs alongside, as soon as the first job lands. The
only open decision among these changes is item 9, which the loop does not need.
## Drafted specs
Each spec below is a draft with `enabled: false`; a person reviews it before it is enabled.
Each does the least that stands on its own, and none changes an existing spec, a `bench/`
file or `tracks.json`. They are chained with `depends-on` in this order because they share
`crates/harness/`, `crates/generation/` and the workspace manifest.
| Spec | Change above | What it adds |
|---|---|---|
| `tasks/spec-lint.md` | 1 | `crates/harness/tests/spec_lint.rs`, the drafting rules as a test, with a closed allow-list for older specs |
| `tasks/provenance-check.md` | 2 | `crates/harness/tests/provenance_check.rs`, a model and cases-run check over every committed record, with a closed allow-list |
| `tasks/floors-table.md` | 3, Rust side | superseded by `floors-follow-the-corpus` (`.floor.json` files); `enabled: false` |
| `tasks/cutoff-reporting.md` | (from the roster) | per-model counts in `candidate_crate.rs` that separate replies cut off at the token cap from complete replies that failed to compile |
| `tasks/job-manifest-checks.md` | 4, first | a new `jobs/` crate: the manifest and the checks on a stage's output |
| `tasks/job-batch-wrapper.md` | 4, second | `jobs/scripts/run-batch.sh`, a batch that cannot end without a ledger line |
| `tasks/job-runner.md` | 4, third | the resumable runner and the result commit |
| `tasks/job-submit-land.md` | 5 | submit and poll-and-land scripts around Dagu, which starts and supervises the runner |
| `tasks/job-agents-setting.md` | (swappable agents) | the model or agent behind each role as a setting of the job, not code |
The runner lives in a workspace crate at the top level because the track gate is
`cargo test --workspace`, so only a crate's tests run in it, and because `bench/land.sh`
refuses a diff under `bench/`. A job's manifest is `specs/.job.json`, written by a
person with the spec; a feature job's manifest lists the keys it implements as
`feature_rules` and its spec's `touches` name `docs/feature-map.json` (gap 1);
`spec-status.mjs` lists only `specs/*.md`, so it ignores the file
(*unverified* for any other reader of `specs/`).
## Hand changes still needed
Changes that need an edit under `bench/` or to `tracks.json` cannot be a spec, because
`bench/land.sh` refuses those paths in a track. Each is an **action** for a person (or an
interactive session) after the spec named has landed; none is done.
1. **Done: floors are files, not a hand-edited script.** See "Floors as data" above; no
action left.
2. **Done: a strict-lane mode for unattended gates.** With `WEBENGINE_LANE_STRICT=1` in its
environment, `bench/land.sh` runs `verify` through `bench/on-lane.sh --lane-only` and
every `landChecks` command `--lane-only`, so with no lane answering the landing is
refused and nothing of the gate runs on the development box (`docs/infrastructure.md`
→ "Build lane"; `bench/land-test.sh` scenario `s1`). `bench/verify.sh` was lane-only
already. Interactive use, with the variable unset, keeps the checks' fallback. The
unattended caller exports it through `tracks.json`'s `specWatch.pollEnv` (item 5).
3. **One announcement per feature, for landings that are not jobs.** A feature job
carries one feature, so a job landing posts once and that post is the feature's
announcement (`docs/web-features-map.md` → "Announcements"); this item does not apply
to it. A landing that is not a job (a plain track) can move several features, and
`bench/feature-announce.py` bundles them into one message. Post one message per
feature for such a landing, each with the track's numbers once. Why: the first real
post (2026-09-25) bundled several features, which reads as one event.
4. **`bench/ledger.py` carries the token-cap counts.** Read the `compile outcome:` line that
`tasks/cutoff-reporting.md` adds and add `cut_off` and `complete_failed` to each model's
entry in a ledger line, and print them in the announcement. Why: an announcement that
says "16 of 32 compiled" should say how many of the rest were cut off by the cap. Needs
`tasks/cutoff-reporting.md` landed.
5. **The scheduled run is the poller, not the model tick.** `tracks.json` declares
`specWatch.pollScript: "jobs/scripts/poll-and-land.sh"` with
`specWatch.pollEnv: {"WEBENGINE_LANE_STRICT": "1"}`. pi-tracks `bin/spec-watch-tick.sh` reads
the key: for a repo that declares it, each scheduled tick runs that script from the
primary checkout under the tick's per-repo lock, with `pollEnv` exported, and starts no
due check and no model session, so nothing creates worktrees or launches writers any
more. The crontab line `bin/spec-watch-schedule.sh` installs is unchanged (still one per
repo, still naming `spec-watch-tick.sh`); the poller's output lands in
`$HOME/.local/state/spec-watch/spec-watch.log` between `=== poll` and `=== exit` lines.
A malformed declaration, or a script that is missing or not executable, is logged as
refused and runs no model tick in its place. Left, as actions: land the pi-tracks
change (branch `track/poller-cutover` there) together with this `tracks.json`, then
reinstall the schedule and restore this repository's cron line, which is commented out by hand
while jobs run one at a time.
6. **Landing from a fetched ref.** `bench/land.sh` needs a local track worktree and branch;
`jobs/scripts/poll-and-land.sh` builds both at landing time and removes them after a
refusal. A `bench/land.sh` that lands from a fetched ref would remove that step. Optional.
7. **Map rules for specs landed outside a job.** A job writes its own rules (gap 2).
Exact-key rules for anything that landed without that stage are added by hand, or for
a job by `job-check feature-rules specs/.job.json` on `main`; no job is owed this
now that the `aspect-ratio` job definition is removed. The field list in `tasks/job-manifest-checks.md`
does not yet name `feature_rules`.
8. **The roster's stale tick text** (item 11 above) and the ground-truth, generation-loop
and map-update-dry-run stage checks, which `tasks/job-manifest-checks.md` leaves out until
their artifact formats are read. The checks are specs later; the roster line is a
person's edit.
## Later stages (design only)
Three larger pieces come after the job runs a real feature. None is drafted as a spec: each
is either unbounded as it stands or waits on something not yet known, and a spec written now
would commission code against a guess.
### Performance runs
A generated function is chosen on its score against the browsers' answers; two candidates
that both score in full are not told apart. A performance run would tell them apart on cost:
several candidates that are known correct on the same corpus, each measured for CPU time,
wall-clock time and peak memory over many scenarios.
Shape, if built: a runner takes a function's committed candidate and the other admitted
candidates from that function's batches, builds each into its own process, and calls the
function in a loop over a scenario set (the function's own corpus plus randomly generated
inputs from the function's cases module, whose `random(seed, count)` already exists). Each
candidate runs in a fresh process pinned to one CPU with the first run discarded and a
fixed repeat count, and reports user plus system CPU time and peak resident memory from the
process's own accounting, and wall-clock time. It runs on a lane, alone, since a shared
lane's load would be the noise. The output is one table per function.
Why not a spec yet:
- The cost of one call of a leaf function is microseconds, and what matters is its share of
a whole layout, so the right scenarios are whole documents, not isolated inputs. Which
documents represent real pages is not decided (**decision**).
- What the table is for is not decided: choosing between equal candidates, a regression
check on later batches, or a number for the announcement. Each needs a different
tolerance (**decision**).
- Whether losing candidates' sources survive a batch is *unverified*: each batch writes
`sources/` under its run directory on the lane, and nothing shows whether the run
directory is kept.
The bounded first slice, once those are answered: one function, its committed candidate
against the baseline fixture, wall-clock and CPU only, reported and not enforced.
### Adversarial review
The review stage asks whether the diff implements the spec. It does not test whether the
winning candidate is right in cases the corpus does not contain. One model could review
another's winning candidate: given the spec text, the function's signature and the
candidate's source, and not the scores, it proposes inputs it believes the candidate answers
wrongly.
Shape, if built: an opinion never blocks a landing, and only a counterexample the browsers
confirm counts. The reviewer's inputs are run through the same ground-truth path as the
function's corpus (`*_ground_truth_cache`, three browsers) and through the candidate; a
finding is an input where the browsers agree with each other, the candidate disagrees with
them, and the reviewer's claim matches the browsers. Confirmed findings go into the job's
result and are not applied: turning them into corpus cases and a new or amended spec is
for a person or a coordinator session outside the loop, since the loop does not write specs.
The reviewer is a `review` role with its own agent (`tasks/job-agents-setting.md`),
required to differ from the candidates' agent.
Why not a spec yet:
- Each generated function has its own case type in its own `*_cases.rs`, so a reviewer's
output has no common format to be written in. The roster's Queue item `seam-kit`, which
factors the per-function runners into one driven by a table, is where a common case
format would come from; drafting the review before it means writing the format twice.
- Whether a counterexample should ever hold a landing (not only be reported) is a policy
choice (**decision**); reporting only is the reversible default.
- The browser oracle's cost for many extra inputs is not measured (*unverified*).
### The long-running learning loop
The idea: across many features, record what worked (which models, counts and prompts led to
admitted candidates, at what cost) and use it to set the next job's package, so the loop
improves without a person tuning it.
Only the recording half is known enough to build, and the drafted specs already do it: the
ledger line per batch (`tasks/job-batch-wrapper.md`), the token-cap counts
(`tasks/cutoff-reporting.md`), the provenance record per function and the result commit per
job. What to do with the record is not known: no real batch has yet written a ledger line,
there are about a dozen generated functions to learn from, and it is not known which knob
matters (the model pool, the count, the prompt, the token cap). A loop that changed its own
settings would edit a job's manifest under `specs/`, which the loop must not write; so any
tuning is a proposal for a person or a coordinator session to turn into a spec.
Next step, not a spec: after the first several real jobs land, read the ledgers and
results by hand (an **action**) and decide which one number would have changed a decision. A
read-only summary over the same files comes after that.
## What this survey did not check
- **The design was unbuilt when this survey was written.** Items 4 and 5 have since been
built and a real feature has run through them ("First real run" above), which was the
test this survey proposed.
- **The roster's counts** (seven landings, one by hand, no announcements, no ledger line)
are taken from the roster; the tick log was read only for its tail and the
`transforms2d` lines. Rechecking against the whole log needs no decision (**action**,
a read of `$HOME/.local/state/spec-watch/spec-watch.log`).
- **Where the `transforms2d` batch ran** (development box or lane copy): *unverified*.
Settled by reading the writer's transcript on lane 1 (**action**, needs lane access).
- **Whether the lane's mirror can hold a base ref** and be rebased there:
`bench/remote-lane.sh` bundles `origin/main..HEAD` and `bench/lane-apply.sh` checks out
the exact commit, but neither was read for what refs the copy keeps. Settled inside
item 5 (**spec now**).
- **Whether older hand-written provenance records would pass the check in item 2**:
*unverified*, settled by running it (**spec now**, inside item 2).
- **`bench/land.sh` past its option parsing and helper functions** (the gate, rebase and
finish steps) was read from the header comments, not line by line; refusals and their
order are as the header states them.
- **Checks that do not exist yet.** The proposed stage checks name artifacts
(`cases_run`, ledger lines, map events) that are read from committed formats, but no
checker for them exists; their fields were read from
`crates/layout/src/generated/table.provenance.json` and `bench/ledger.py`'s callers only.
- **The drafted specs were not built or run.** They were written against the files they cite,
read while drafting (`bench/land.sh`, `bench/on-lane.sh`, `bench/remote-lane.sh`,
`bench/batch-lane.sh`, `bench/ledger.py`, `candidate_crate.rs`, the provenance records).
Their test lists are proposals; whether each is enough is settled when a track builds it
(**spec now**). Two facts they rely on are *unverified*: that the lane's mirror keeps
`result/` and the job branch fetchable after a lane copy is swept, and that the
lane's shared store is writable for the ledger at the time of a real batch.