burrito.space / bez
This repository has no description
Star
Fork 0
Watch
bez docs autonomous-loop.md
55 kB
Markdown
at track/substrate-composition

Running a web feature with no session editing #

The aim: a web feature goes from a spec to a landed commit and a community announcement on folknet hardware with nobody editing anything. A person or an interactive session that edits files inside a feature's track is a failure of the system, not a step in it.

This document surveys how far the repository is from that, ranks what stops it, and proposes a design. Nothing here is built. Evidence is cited as file plus symbol, or as the overnight run of 2026-09-24/25 (the roster handoff.md on main → "Overnight ramp", and the tick log $HOME/.local/state/spec-watch/spec-watch.log). Every claim not checked against code or a run is marked unverified. Hosts and addresses stay in $HOME/rig-notes.

The principle #

The whole lifecycle of one feature is packaged as one job. The job is shipped to a build lane, started there, and runs to the end there. The only thing watched from outside is a commit appearing on a result branch. A live writer session in a development-box worktree, one per feature, is the failure state, because it needs something to keep it alive (a tick, a relaunch, a local rebase) and every such touch is a place a person is pulled in.

The development box does three things: submit a job, poll for its result commit, and land it.

The loop starts at an enabled spec and ends at a result commit. It does not write or enable specs: a person or a coordinator session does that, outside the loop, so spec authorship stays with a reviewer. Finished features are reviewed afterwards, asynchronously, and a review that finds problems sends the feature back through the loop as a new or amended spec.

The loop as it stands #

flowchart TD
  spec["1. spec written<br/>(person or session, by hand)"]
  tick["2-3. cron tick: model session creates<br/>the worktree, commits handoff.md"]
  launch["4. writer launched on a lane<br/>(local if every lane refuses)"]
  work["5. writer works on the lane:<br/>corpus, ground truth, loop, model batch,<br/>admit, commit, docs"]
  fetch["6. tick fetches the commits back"]
  rebase["7. tick rebases locally if behind"]
  gate["8. verify on a lane<br/>(local if no lane answers)"]
  review["9. spec-reviewer subagent<br/>inside the tick session"]
  land["10. bench/land.sh: gate again on a lane,<br/>fast-forward merge, tag"]
  after["11. bench/after-land.sh: dashboard and map update,<br/>announcement"]
  hand["12-13. by hand: push main;<br/>map rules only for a spec<br/>landed outside a job"]
  late["14. asynchronous review of the finished feature<br/>(person or coordinator session)"]
  spec --> tick --> launch --> work --> fetch --> rebase --> gate --> review --> land --> after --> hand
  after --> late
  late -. "problems: new or amended spec" .-> spec
# Step What runs it Where Person or interactive session needed
1 Write the spec and its touches, depends-on, enabled a person or an interactive session; drafting rules are prose in the roster's "Queue" section development box yes, every feature
2 Decide a tick is due cron, spec-watch-tick.sh then spec-watch-due.mjs (no model) development box no
3 Create the worktree, run tracks.json install, write and commit handoff.md the tick: one claude -p '/spec-watch' session (prompts/spec-watch.md → "Start unstarted or stale work") development box no
4 Start the writer spec-watch-launch.mjs → bench/lane-writer-launch.sh → the declared harness's writer on the lane with the lowest load lane; local when every lane refuses no
5 Build the feature: corpus, ground truth in three browsers, generation loop, model batch, admission, commit, docs the writer, following the spec's prose lane no
6 Bring commits back spec-watch-fetch.mjs → bench/lane-writer-fetch.sh, fast-forward only development box no
7 Rebase onto main when behind the tick; a conflict outside handoff.md relaunches a writer locally ("A rebase conflict") development box no
8 Gate: bench/verify.sh bench/on-lane.sh lane; development box if no lane answers no
9 Independent review spec-reviewer subagent, four questions, inside the tick session; verdict saved in $HOME/.local/state/spec-watch, not in git development box no
10 Land bench/land.sh, one track per tick; reruns verify and landChecks (bench/fragment-floor.sh); bench/after-land.sh then raises the floors gate on a lane; merge in the primary checkout no
11 Dashboard and map update, announcement bench/after-land.sh → bench/dashboard-update.sh, bench/web-features-update.sh, bench/feature-announce.py development box no
12 Push main and spec/* tags bench/land.sh pushes nothing ("Nothing is pushed"); jobs/scripts/poll-and-land.sh pushes a landed job's spec/<name>/<sha> tag to JOBS_TAG_REMOTE (default origin), not main development box yes for main, after each batch of landings; also for the tag of a landing the poller did not make
13 Floors, map rules for what landed floors: bench/generated-floors-update.sh from bench/after-land.sh; map rules: a job's feature-map stage from its manifest's feature_rules, with a NOT ANNOUNCED line from bench/expected-features-check.py after landing when a declared feature did not move lane (floors, the job's stage); development box (the line) no for a job; yes, by hand, for a spec landed outside a job
14 Review the finished feature; on problems, write a new or amended spec that re-enters at step 1 a person or a coordinator session, outside the loop, after the result commit and without holding a landing development box yes, by design: spec authorship stays with a reviewer

Which parts already work unattended: the overnight run had the tick write, review and land seven specs with no hand intervention (dashboard-history-graph, text-metrics-oracle, testdriver-playwright-input, transforms2d, audio-null-sink, inline-atomic, tables-fixed-layout), and fixed-positioning landed by hand after bench/land.sh refused it. The independent review found two real defects in aspect-ratio. So steps 2 to 11 run without a person when a spec is well formed; what fails is the edges (steps 12 and 13) and the checking inside the steps. Steps 1 and 14 need a person or a coordinator session by design.

One correction to older text: the roster's "Resume Here" says an owning interactive session's loop drives ticks. spec-watch-tick.sh's header says no interactive session drives ticks, and the crontab runs it every five minutes for genera. The roster line is out of date.

Gaps, ranked by how often they stop or degrade the loop #

Counts come from the overnight run (eight specs, seven landed by the tick, one by hand) and are the roster's, not rechecked against the tick log line by line.

1. Specs are written by hand (stops every feature) #

Nothing creates a spec, and the loop is not meant to: specs are written by a person or a coordinator session outside the loop, so the gap to close is a malformed spec, not the lack of an author. The roster's "Queue" is drafted by a person or a session, and the drafting rules live as prose there: single-line front matter (bench/land.sh touchesOf() reads a multi-line list as empty), touches must name the spec's own test files, its provenance module, crates/provenance/src/lib.rs, docs/generated-functions.md and docs/feature-map.json, and a depends-on on the previous generated-function spec because they all edit fragment.rs. No check enforces any of it. A drafted spec cannot arrive by a track either: bench/land.sh refuses any diff under specs/.

Every feature job carries the same two things, and these are checked. Its manifest (specs/<name>.job.json) lists the web-features ids it moves as expected_features and the exact browser-compat-data keys it implements as feature_rules, each with the status and evidence of a docs/feature-map.json rule, and only keys the spec's scope covers (a feature's keys are listed in the pinned web-features package under oracle/node_modules; aspect-ratio's spec leaves out the <img> and <video> keys, so its manifest does not list them). Its spec's touches name docs/feature-map.json, since the job commits the rules there, and its prose must not forbid that file. job-check manifest refuses a manifest with features and no rules, or rules and no features; job-check manifest --spec, which jobs/scripts/submit.sh runs, refuses one whose spec's touches omit the map (gap 2).

2. Announcements needed hand-added map rules (closed for jobs) #

A feature moves past unknown only through an exact-key rule in docs/feature-map.json (docs/web-features-map.md → "The status rule"). The generated-function specs' prose forbade editing that file, so none of the seven landings announced anything until rules were added by hand at 6a3de27, and bench/after-land.sh could not tell "nothing moved" from "no rule exists", so a missing rule looked like a quiet success.

For a job, the rules now come from its manifest (feature_rules, gap 1), and no writer has to choose to add them. job-run adds a feature-map stage before review (jobs/src/runner.rs add_feature_map_stage): job-check feature-map writes the rules into docs/feature-map.json through job-check feature-rules and commits them, runs bench/web-features-update.sh --no-commit, and fails the job when a listed key or expected feature does not end partial or complete. After landing, bench/after-land.sh runs bench/expected-features-check.py, which prints a NOT ANNOUNCED line naming each declared feature no event moved that is not partial or complete, with the fix. Still by hand: the backfill for aspect-ratio, which ran before the stage existed (job-check feature-rules specs/aspect-ratio.job.json on main), and a spec amendment for it, since specs/aspect-ratio.md still forbids the file.

3. Floors are added by hand (silent on every generated function) #

Resolved by the floors-follow-the-corpus track. bench/seam-floor.sh used to have four floors (floats, stacking order, relative positioning, block height); containing block, transform and inline atomic had none, and each landed without one, because adding a floor needed a change in bench/ (refused by bench/land.sh for a track), a score line in seam_floor_check.rs and a cache rebuild in the script's loop. Now every generated function is scored by bench/generated-scores.sh against <name>.floor.json, a new function is one entry in generated_score_check.rs FUNCTIONS, and bench/generated-floors-update.sh raises the floors after each landing.

4. Steps the spec prescribes in prose are not enforced (silent, once in seven) #

tables-fixed-layout landed with a hand-written crates/layout/src/generated/table.rs whose table.provenance.json records "model": "hand-written" and "cases_run": 0, because its three lane runs never happened, and the review passed it. The review's four questions (prompts/spec-watch.md → "Review work") ask whether the diff implements the spec, not whether the runs the spec requires happened, and bench/land.sh reads no provenance file. The map already treats such a record as not complete (docs/web-features-map.md: a complete key needs at least one case run and every case passed), so the evidence exists and nothing reads it at landing.

5. Model batches bypass the batch script (loses the numbers on every landing) #

transforms2d's spec prescribes cargo run --release -p generation --bin transform_real_model_run (specs/transforms2d.md), not bench/absolute-mixed-batch.sh or bench/overflow-mixed-run.sh. Only those two scripts call bench/ledger.py (through bench/batch-lane.sh batch_measure_stop), so a batch run the spec's way writes no ledger line, and the announcement has no candidates, tokens or CPU-hours to post. The roster says no real batch has written a ledger line yet. The tick log says the transforms2d batch "ran locally"; the log also says its writer started on lane 1, so the batch most likely ran on the lane inside the writer's copy, not on the development box (unverified: which machine ran it). Either way it did not go through bench/batch-lane.sh, which is what enforces "a model batch fails rather than fall back to a local run".

6. Landing refusals hold a slot forever (stops one feature and its dependents) #

bench/land.sh refuses committed files outside touches, any diff under bench/, tracks.json, .github/, specs/, .cargo/, rust-toolchain*, .env*, .gitattributes, .gitmodules, and a **-broad touches (header of bench/land.sh; isDenied()). A refused track stays awaiting-approval, holds a live slot and blocks its dependents. fixed-positioning hit this and landed by hand. The touches omissions in sandbox specs (ring-buffer) are the same failure. A person's /land is the only way out.

7. The tick is a model session on the development box that edits worktrees #

The tick creates worktrees, runs npm install for oracle/, writes and commits handoff.md, rebases, resolves handoff.md conflicts, and relaunches a writer on the development box when a rebase conflicts outside handoff.md ("Rebases are kept on this machine by design": a lane copy carries only track/* branches, so a lane writer has nothing to rebase onto). It also falls back to a local writer when every lane refuses. Recorded failures: a lane launch read as a failure and started three writers on one track (fixed in pi-tracks); a tick died with no exit line twice; a lane writer waiting on a silent subagent stayed working for over 30 minutes, and the stall check did not fire; lane copies diverge after a local rebase. These are the roster's "Spike findings", several marked fixed. Each is a place where a dev-box process has to stay healthy for a feature to finish.

8. Gates can fall back to the development box #

bench/on-lane.sh runs a command locally when no lane answers, unless --lane-only. bench/land.sh runs each landChecks command through it without --lane-only, and runs verify directly (tracks.json's verify even falls back to a local cargo test when bench/verify.sh is missing), so with both lanes down the floors could run on the development box. That contradicts the root AGENTS.md for everything except model batches and bench/verify.sh, which is lane-only already. Unverified: that it has ever happened. Fixed in bench/land.sh: with NEV_LANE_STRICT=1 the gate and every check go --lane-only (docs/infrastructure.md → "Build lane"). The scheduled poller gets the variable from tracks.json's specWatch.pollEnv ("Hand changes still needed", item 5); a landing run by hand, or by the model tick's specWatch.landScript, does not set it.

9. Pushing main is by hand #

The tick lands on main without pushing. The poller (jobs/scripts/poll-and-land.sh) pushes the spec/<name>/<sha> tag of each job it lands to JOBS_TAG_REMOTE (default origin), but not main. A person pushes main, and any spec/* tag the poller did not push, after each batch (docs/coordinator-on-folknet.md → "Who pushes"). This does not stop the loop, since a lane gets unpushed commits in a bundle (bench/remote-lane.sh bundles origin/main..HEAD), but the bundle grows with every unpushed landing and the public repository lags.

10. Smaller items #

  • One review round: maxRounds is 1, so the second failed review parks a track (aspect-ratio used its one round overnight; a second failure parks it).
  • awaiting-approval tracks were listed as waiting on a person in repositories that land them automatically (roster "Spike findings"; the prompt now excludes them).
  • The coordinator's move to a folknet guest (docs/coordinator-on-folknet.md) is designed and not done, and needs a person.
  • Not yet exercised: subscription-limit backoff, the stalled-writer relaunch, and lane capacity under many writers (roster).
  • An open decision: how wide a feature spec's generated scope should be (roster, "Blocked on"). It shapes what a spec asks for, so it shapes the job's stages.

Proposed design #

What a job contains #

One job is a branch plus a manifest, and everything the run needs is either in it or pinned by sha.

Part Content Today
Spec specs/<name>.md at a pinned sha; its touches, depends-on, writer model the same, read by spec-status.mjs
Base the sha of main the job starts from; the lane carries this commit so a rebase can run there not carried: lane copies hold track/* only
Plan the brief for the writer stages, and the ordered stage list below handoff.md plus prose in the spec
Model and batch settings provider (never metered), model pool string, per-model counts and concurrency, stop rule (two flat batches or six), escalation prose inside the spec (specs/transforms2d.md)
Floors per generated function: which corpus binary rebuilds its cache, the score it must keep generated_score_check.rs FUNCTIONS and <name>.floor.json, raised automatically
Announcement the features expected to move, and the exact-key rules that move them the manifest's expected_features and feature_rules, written into docs/feature-map.json by the job's feature-map stage
Limits wall-clock cap, writer relaunches, review rounds maxRounds in tracks.json, three relaunches in prompts/spec-watch.md

The manifest is one file beside the spec, read by a script, so nothing depends on prose.

Stages, in order. A stage is either a writer stage (a Claude session on the lane, subscription-billed, doing work that needs judgment) or a script stage (a fixed command whose output is checked). Only stages 1 and 6 and the review need a model; the rest are commands, because the run must not depend on a writer choosing to do them.

  1. Writer: hand-written substrate, types, baseline, corpus recipes, wrong fixtures.
  2. Script: ground-truth cache from three browsers (*_ground_truth_cache). Check: case count and drops recorded.
  3. Script: the generation loop over fixtures (*_generation_loop). Check: every seeded wrong fixture rejected.
  4. Script: the model batches, always through a batch wrapper that records a ledger line. Check: at least one ledger line for the job's name; provider is not metered.
  5. Script: admission and commit of the winner with its provenance record. Check: the record's model is not hand-written unless an escalation is recorded, cases_run is above 0, cases_passed equals cases_run, regression_free is true.
  6. Writer: wiring, docs, floor entry. Not the docs/feature-map.json rules: stage 8 writes those from the manifest's feature_rules.
  7. Script: rebase onto the carried base; verify; landChecks; the new function's floor. A conflict resumes writer stage 6 on the lane with the conflict named.
  8. Script: the feature-map stage, which job-run adds before review when the manifest has feature_rules (job-check feature-map). It writes and commits the rules (job-check feature-rules), then runs the map update as a dry run (bench/web-features-update.sh --no-commit) and puts the snapshot and event log back. Check: every listed key ends partial or complete through its own rule, no further back than before, and every expected_features entry ends partial or complete; it need not move, so a job that extends an already partial feature passes. After landing, bench/expected-features-check.py prints a NOT ANNOUNCED line for a declared feature that still did not move.
  9. Review: a fresh read-only session on the lane, given the spec, the diff and the stage results. It answers the tick's four questions plus "did each script stage produce its artifact".
  10. Result: a commit on result/<name> in the lane's copy holding result.json: spec sha, base sha, branch tip, per-stage status and artifact hashes, review verdict, ledger lines, expected events. Written last, and only when every check passed. A failed job writes the same file with the failing stage, so a poll can tell "failed" from "still running".

The runner and the stage checks are copied from main at submit time and run from that copy, never from the job's own tree. A job that edits them affects later jobs, not its own run, which keeps the reason bench/land.sh gives for protecting its own files (a track must not rewrite the judge it is judged by) without putting the runner under bench/.

Before that first stage, on every start and every resume, the runner checks that the lane actually has what the manifest is about to need: a resolvable git, python3, node and kill, cargo and the oracle's Playwright browsers when a writer is in the stage list, a reachable model gateway when a stage runs a pi session, disk space, the shared archive directory, and enough wall-clock time left across resumes to be worth starting another stage (jobs/src/preflight.rs). One unmet requirement prints every reason, records them in state.json, runs no stage and exits 9 rather than let a writer spend a long stage only to lose the next one to something as small as a missing PATH entry.

How it is shipped and started #

Existing pieces, and where each falls short of an unattended run:

Existing What it does Falls short
bench/remote-lane.sh ships a bundle of commits not on origin/main, a diff and untracked files in one ssh stream, and runs a command in a lane copy carries the worktree's HEAD only, not a base ref; the command is foreground and ends with the connection unless the command detaches
bench/lane-writer-launch.sh syncs a worktree, git checkout -B, pulls pi-tracks, runs its writer command (--tool from specWatch.lanes.tool) with the brief, prints {ssh, sshOpts, path} starts one open-ended writer session with a brief, not a runner with stages; needs a clean local worktree (exit 5), so a dev-box worktree must exist first; refuses over unfetched lane commits, which forces a fetch before any relaunch
bench/lane-writer-fetch.sh fetches the branch from the lane copy and fast-forwards fast-forward only, so a local rebase makes every later fetch report "diverged"; fetches the working branch, so "done" is inferred from liveness and transcript age (spec-status.mjs writerFor(), isStalled()) instead of read from a commit
bench/lane-apply.sh checks out the exact commit in a lane copy, applies the diff, exports NEV_ON_LANE=1 nothing stops a lane run from being cleaned under a live job (git clean -fdx); the writer launch guards it, a runner would need the same
spec-watch-launch.mjs picks a lane, falls back to local the local fallback is the failure state; exit 4 exists only because of it

The proposed submit does, from the development box and in this order: verify the spec sha is the one on main, verify depends-on specs are landed, bundle the job branch and the base tip, pick the lane with the lowest load, send the bundle and the manifest through the same single ssh stream bench/remote-lane.sh uses, and start the runner detached on the lane (its own session, a pid file and a lease file it touches every minute). Submit prints the lane path and returns. It never starts anything locally: with no lane answering, it exits non-zero and the job stays unsubmitted (the same rule bench/on-lane.sh --lane-only gives a model batch).

The runner is resumable: each stage writes its status into a state file in the lane copy, so a lane restart or a killed runner is handled by starting the runner again on the same copy, and it continues from the first unfinished stage.

What the development box does #

sequenceDiagram
  participant D as development box (script, no model)
  participant L as lane
  participant P as primary checkout
  D->>L: submit: bundle, manifest, start runner
  loop every few minutes
    D->>L: poll: lease, then git ls-remote result/name
  end
  L-->>D: result commit (fetch)
  D->>P: land: bench/land.sh name spec-sha tip
  P->>P: fast-forward, tag, map update, announcement
  • Submit. One script per enabled spec whose dependencies are landed and whose slot count allows it.
  • Poll. One ssh call per running job: read the lease, and git ls-remote the result branch. No lease refresh for ten minutes and no result: mark the job lost and submit it again (resumes). A result that says failed: record the stage, park the job.
  • Land. Fetch the result branch and the job branch, check the result's spec sha, base sha and tip against what bench/land.sh will read, create the local worktree bench/land.sh needs from the fetched branch, run bench/land.sh <name> <spec-sha> <tip>, then bench/after-land.sh runs as it does today. bench/land.sh keeps its own gate rerun and its own protected-file list, so a wrong result cannot land anything the gate rejects.

The worktree in the primary checkout exists only for the length of the landing. Nothing writes a file there, so there is no handoff.md to conflict with the roster, no local rebase, and no local writer. This step needs no model session, so it is a plain script (cron) rather than a claude -p tick.

Left outside the job on purpose: writing and enabling specs (gap 1), which stays with a person or a coordinator session, and pushing main (gap 9), which needs a decision below.

How the gaps map onto the shape #

Gap What it blocks in the shape Closed by
1 Hand-written specs a malformed spec gives the job nothing to run a spec lint; specs stay outside the loop (below)
2 Map rules the announcement stage stage 8, and the manifest's feature_rules (done)
3 Floors by hand stage 7 for every function floors as data read by the checker
4 Prose not enforced stages 2 to 5 being real script stages with artifact checks
5 Batch bypass the ledger, so the post's numbers stage 4 always through the batch wrapper
6 Refusals landing lint on touches at submit; a job that would touch bench/ is not submitted
7 Dev-box tick "only submit, poll, land" the runner on the lane, review inside the job
8 Local fallback "nothing heavy on the development box" strict mode for unattended gates
9 Push by hand public visibility only a decision, not a blocker

Settled #

  • The runner and its scripts may live outside bench/. The loop has to be unattended, and bench/land.sh refuses any diff under bench/, so a runner kept there could never be landed or changed without a person. The split is accepted: some scripts stay in bench/, the runner and its stage checks live elsewhere.
  • A running loop does not write or enable specs. Specs are written by a person or a coordinator session, outside the loop. Finished features are reviewed afterwards, asynchronously, and a review that finds problems sends the feature back through the loop as a new or amended spec (a new spec sha, so a new job). Spec authorship stays with a reviewer; the loop's job is to take an enabled spec to a result commit. There is therefore no drafting stage in the job and no path for adding to specs/ from a track.
  • Dagu ships, queues and supervises a job; job-run runs its stages as one step. A trial on a lane ran fixture jobs end to end through Dagu, and a real spec has since run the same way ("First real run" below) (bench/dagu/job.yaml, docs/infrastructure.md → "Dagu on a lane"): sent over its MCP server from the development box, working copy from the lane's mirror at a base on the forge, job/<name> and result/<name> published to the forge, a queue limit, a resume after a stop, and a restart of the lane. So the detached start, pid and lease handshake, ssh lease probe and lost status drafted in "How it is shipped and started" and "What the development box does" above are not built: Dagu starts and queues the run, its status replaces the lease probe, and a resend with the same parameters resumes the job. The queue's limit replaces specWatch.maxLive. job-run stays a single step rather than one Dagu step per stage, because a step per stage would rebuild in workflow YAML what jobs/src/runner.rs already does under tests: pinned tools, compiled-in checks, the per-role environment, the writer's relaunches, the summed wall-clock limit, the review verdict, the state file and the result commit and its push. What stays on the development box is specs/job-submit-land.md: the repository's refusal checks before a submit, and the fetch, job-check result and bench/land.sh at landing.

First real run #

A real spec has since run the loop end to end, not only the fixture job the Dagu trial above used. Spec viewport-units (manifest specs/viewport-units.job.json: pi writer and review on deepseek-v4.1-flash through the gateway; stages write, test, measure, review) ran as Dagu run 034VMxPcXtPSIim2wljbuk on lane 0 on 2026-09-26 and ran every stage. The start-of-job check (jobs/src/preflight.rs) passed. write made two commits, one of them a new file, crates/layout/src/bin/viewport_units_check.rs, with 69 cases; test passed; measure compared all 69 cases across chromium, firefox and webkit, 69 passed and none dropped. review returned fail on a real defect: browser_verdict compares only adjacent browser pairs (names.windows(2)), so chromium and webkit are never compared directly, though the spec requires every returned dump to agree. job-run exited 1 and published job/viewport-units and a failed result/viewport-units; poll-and-land.sh --check reported the review failure. Logs landed in archive/job-logs/viewport-units/<run id>/ on the NAS share with exactly the files the archive handler in bench/dagu/job.yaml lists, and a scan of them found no keys, tokens, credentials, addresses, hostnames or personal home paths.

A first attempt at the same spec (034VKGQgP1MDBABX0D5rJB) wrote for 1899 s of a 2400 s limit, then failed at stage test: the run step of bench/dagu/job.yaml did not put ~/.cargo/bin on PATH, and job-run starts a script stage's command without a shell (jobs/src/runner.rs run_command()). The wall-clock limit counts across resumes, so the job was restarted from scratch with a 3600 s limit.

The two runs exposed gaps, each fixed since: no job-check on the development box (jobs/scripts/install-job-check.sh); bench/dagu-lane.sh without --lane reached a lane without Dagu, so it now defaults to NEV_DAGU_LANE or lane 0; bench/on-lane.sh --lane-only refused inside a job, so it and bench/remote-lane.sh now run in place under NEV_ON_LANE; cargo missing from the run step's PATH; a resend always went at the base tip while the lane resumes only at the base a job started from, so submit.sh now keeps base_sha in the record; requirements went unchecked until a stage needed them, so jobs/src/preflight.rs now exits 9 with every reason, read back by poll-and-land.sh through bench/dagu-lane.sh step-log; and a Cargo.lock missing a new dependency, which the job's --locked build refused while lane test runs had updated the lock file silently, so bench/verify.sh now tests with --locked. A restart from scratch was done by hand (deleting job/<name> and result/<name> on the forge, bench/dagu-lane.sh clean-jobs <name> for the lane's jobs-work/<name>/, bench/dagu-lane.sh prune-mirror, and moving the local record aside), so jobs/scripts/submit.sh --restart <name> now does all four and sends the job again at the base tip; clean-jobs itself never touches the forge.

Left open: jobs/tests/agents.rs reportedly fails inside a job because job-run exports JOB_AGENT_NAME, JOB_AGENT_MODEL and JOB_AGENT_KIND to every stage (unverified beyond the writer's own report); a failed review ends the job with review_rounds at 1, and a resend resumes at review, reviewing the same code again, so a finding does not yet reach a second writer round; submit.sh --restart has not yet restarted a real job; and the writer's claim to have run bench/verify.sh inside the job is unverified, since pi runs with --no-session and keeps only its closing text.

aspect-ratio: the first generated-function run, kept as a baseline #

Spec aspect-ratio (manifest specs/aspect-ratio.job.json: substrate, ground-truth, generation-loop, batch, admit, test, review) passed every stage in one round as Dagu run 034VztFIHkkBEpA6VFkhuR (base 34c51d5) and was landed by the scheduled poller at 96fe266, with its floor raised by bench/after-land.sh at ba2a910. The admitted function is a model candidate, model_28 from qwen3.8-27b-fast, at 109/109. Stage times: substrate 2h26m, ground-truth, generation-loop and batch under a minute together, admit 11 minutes, test 36 seconds, review 30 minutes; 3h09m from start to publish. The substrate took that long because its writer followed the spec's "Run, in order" list and ran the ground truth, the generation loop and the whole model batch itself (772 s of model time) before the job's own stages ran them again.

That run's commits (7d48c22 through ba2a910) are kept under tag archive/aspect-ratio-run1 and reverted on main, so the spec runs again from scratch as a comparison against writer stages that pass --task (jobs/tests/writer_task.rs), a brief that asks for narrow reading (jobs/scripts/pi-writer.sh BRIEF) and sessions that record their turns and tokens (jobs/scripts/pi-session.sh, events under state/pi/<stage>/events.jsonl). The first run's logs, its job record (job-record.json) and these stage times (baseline.txt) are under archive/job-logs/aspect-ratio/034VztFIHkkBEpA6VFkhuR/ on the NAS share; its result.json went with the forge's result/aspect-ratio, deleted at landing, and its content is in state/state.json and state/review.json there.

The rerun passed every stage in one round as Dagu run 034WA9rX2fx1UlV9szr3Bd (base 03948e5) and was landed by the scheduled poller at c362bc4. The admitted function is again a model candidate, model_34 from qwen3.8-27b-fast, at 109/109. Stage times, from the job event store, with the first run's in brackets: substrate 35m28s (2h26m), ground-truth 5 seconds, generation-loop 50 seconds, batch 5m48s, admit 18m32s (11 minutes), test 36 seconds, review 40m15s (30 minutes); 1h41m37s from start to publish (3h09m). The review session took 88 turns and 92 tool calls, with 392,574 uncached input, 43,221 output and 5,026,304 cache-read tokens, so the model server reuses the shared start of the context across turns. Admit grew because the substrate writer left the aspect_ratio entry in crates/generation/src/bin/generated_score_check.rs FUNCTIONS and a test in crates/generation/src/admit.rs for admit to restore: with them in place, bench/verify.sh would have run the ground-truth binary that the substrate's --task forbids. The writer stages' turns and tokens are not known, because the rerun's logs were lost: the archive script in bench/dagu/job.yaml failed to create its temporary folder on the NAS share, and step cleanup then removed the job folder, the only copy. cleanup now keeps the job folder until the archive succeeds, and the archive keeps each pi session's events.jsonl.

Proposed changes #

Each change lists what it makes harder later, and one tag: spec now (can be commissioned as a spec that lands through bench/land.sh), decision (needs a person to choose), action (needs a person to do something a track cannot).

  1. Spec lint as a test. crates/harness/tests/spec_lint.rs, in the style of no_seam_word.rs, reads specs/*.md and enforces the drafting rules of gap 1 plus: touches never names bench/, specs/ or the other protected paths, and a generated-function spec names docs/feature-map.json. Harder later: a wrong rule rejects every new spec until the test is fixed. Existing specs that break a rule need an allow-list entry, since a spec's sha must not change. Spec now.
  2. Provenance check as a test. A test over every *.provenance.json under crates/layout/src/generated/ requiring a model other than hand-written unless an escalation is recorded, and cases_run above 0. Harder later: hand-written generated functions need a recorded escalation field. Unverified: whether the older hand-written modules would pass; an allow-list may be needed. Spec now.
  3. Floors as data. Done differently from the plan below: one <name>.floor.json per generated function, read by generated_score_check.rs (which also names each function's cache binary), raised by bench/generated-floors-update.sh after each landing, and protected from a track by bench/land.sh refusing any diff that touches a *.floor.json (which replaces the comparison against main's table). bench/seam-floor.sh and seam_floor_check.rs are gone. specs/floors-table.md is superseded.
  4. Runner, stage checks and manifest. A new top-level directory, not bench/, so a spec can commission and land it, because bench/land.sh refuses any diff under bench/. Harder later: two places hold lane scripts, and the pinned-from-main rule must be kept or a track could edit its own judge. Spec now, in this order: manifest and stage checks first (also usable by hand), then the batch wrapper that always writes a ledger line, then the runner.
  5. Submit and poll-and-land scripts. Submit reuses bench/remote-lane.sh's payload; poll-and-land wraps bench/land.sh. Harder later: bench/land.sh still expects a local track worktree, so the poller builds one at landing time; a later bench/land.sh change to land from a fetched ref would remove that. Spec now if placed with the runner (item 4); a change to bench/land.sh itself is an action.
  6. Replace the model tick for genera. The cron entry runs the poll-and-land script instead of spec-watch-tick.sh. That is a change to pi-tracks (a shared package other repositories use), so it is opt-in by declaration in tracks.json, the way specWatch.landScript is. Harder later: the model tick's review and conflict handling are gone for these repositories, so every failure mode moves into the job result. Action (a pi-tracks change, and editing tracks.json, which bench/land.sh refuses).
  7. Strict lane mode for unattended gates. bench/verify.sh and each landChecks command use bench/on-lane.sh --lane-only when a variable the poller sets is present, and submit refuses a local fallback. Harder later: with both lanes down nothing lands, which is the intended result. The landing half is done (NEV_LANE_STRICT=1, see "Hand changes still needed", item 2); the submit half waits for submit itself.
  8. Asynchronous review and send-back. A person or a coordinator session reviews finished features after the result commit, outside the loop, and does not hold a landing. A review that finds problems writes a new or amended spec (linted by item 1), which re-enters the loop as a new job. Harder later: review lags the loop, so a problem found late is fixed by a follow-up spec on top of what already landed, not by holding the landing; and a reviewer has to be available for the loop's output to be checked at all. Nothing in the loop writes or enables specs, so specs/ stays protected by bench/land.sh. Action (a person or a coordinator session does the review; nothing to build in the loop).
  9. Push main and tags. The poller already pushes a landed job's spec/* tag; main is still pushed by hand. Harder later: an unattended push publishes code no person has seen. Decision (docs/coordinator-on-folknet.md → "Who pushes" and "Open questions" hold the deploy-key design). Not needed for the loop.
  10. Coordinator on folknet. With the development box reduced to submit, poll and land, moving those three duties to a guest is smaller than it was, and still needs a person to create it and sign in (docs/coordinator-on-folknet.md → "Cutover"). Action.
  11. Fix the roster's stale tick text. The roster's "Resume Here" still says an owning session drives ticks. Nothing is made harder; the roster is on main and outside this track's scope. Action.

Suggested order: items 1 to 3 (Rust side) first, since they need no decision and make the present loop safer at once; then items 4 and 5; then actions 3, 6 and 7 together, since they change the same scripts. Item 8 runs alongside, as soon as the first job lands. The only open decision among these changes is item 9, which the loop does not need.

Drafted specs #

Each spec below is a draft with enabled: false; a person reviews it before it is enabled. Each does the least that stands on its own, and none changes an existing spec, a bench/ file or tracks.json. They are chained with depends-on in this order because they share crates/harness/, crates/generation/ and the workspace manifest.

Spec Change above What it adds
specs/spec-lint.md 1 crates/harness/tests/spec_lint.rs, the drafting rules as a test, with a closed allow-list for older specs
specs/provenance-check.md 2 crates/harness/tests/provenance_check.rs, a model and cases-run check over every committed record, with a closed allow-list
specs/floors-table.md 3, Rust side superseded by floors-follow-the-corpus (<name>.floor.json files); enabled: false
specs/cutoff-reporting.md (from the roster) per-model counts in candidate_crate.rs that separate replies cut off at the token cap from complete replies that failed to compile
specs/job-manifest-checks.md 4, first a new jobs/ crate: the manifest and the checks on a stage's output
specs/job-batch-wrapper.md 4, second jobs/scripts/run-batch.sh, a batch that cannot end without a ledger line
specs/job-runner.md 4, third the resumable runner and the result commit
specs/job-submit-land.md 5 submit and poll-and-land scripts around Dagu, which starts and supervises the runner
specs/job-agents-setting.md (swappable agents) the model or agent behind each role as a setting of the job, not code

The runner lives in a workspace crate at the top level because the track gate is cargo test --workspace, so only a crate's tests run in it, and because bench/land.sh refuses a diff under bench/. A job's manifest is specs/<name>.job.json, written by a person with the spec; a feature job's manifest lists the keys it implements as feature_rules and its spec's touches name docs/feature-map.json (gap 1); spec-status.mjs lists only specs/*.md, so it ignores the file (unverified for any other reader of specs/).

Hand changes still needed #

Changes that need an edit under bench/ or to tracks.json cannot be a spec, because bench/land.sh refuses those paths in a track. Each is an action for a person (or an interactive session) after the spec named has landed; none is done.

  1. Done: floors are files, not a hand-edited script. See "Floors as data" above; no action left.
  2. Done: a strict-lane mode for unattended gates. With NEV_LANE_STRICT=1 in its environment, bench/land.sh runs verify through bench/on-lane.sh --lane-only and every landChecks command --lane-only, so with no lane answering the landing is refused and nothing of the gate runs on the development box (docs/infrastructure.md → "Build lane"; bench/land-test.sh scenario s1). bench/verify.sh was lane-only already. Interactive use, with the variable unset, keeps the checks' fallback. The unattended caller exports it through tracks.json's specWatch.pollEnv (item 5).
  3. One announcement per feature. bench/feature-announce.py bundles every feature a landing moved forward into one message. Post one message per feature, each with the track's numbers once. Why: the first real post (2026-09-25) bundled several features, which reads as one event.
  4. bench/ledger.py carries the token-cap counts. Read the compile outcome: line that specs/cutoff-reporting.md adds and add cut_off and complete_failed to each model's entry in a ledger line, and print them in the announcement. Why: an announcement that says "16 of 32 compiled" should say how many of the rest were cut off by the cap. Needs specs/cutoff-reporting.md landed.
  5. The scheduled run is the poller, not the model tick. tracks.json declares specWatch.pollScript: "jobs/scripts/poll-and-land.sh" with specWatch.pollEnv: {"NEV_LANE_STRICT": "1"}. pi-tracks bin/spec-watch-tick.sh reads the key: for a repo that declares it, each scheduled tick runs that script from the primary checkout under the tick's per-repo lock, with pollEnv exported, and starts no due check and no model session, so nothing creates worktrees or launches writers any more. The crontab line bin/spec-watch-schedule.sh installs is unchanged (still one per repo, still naming spec-watch-tick.sh); the poller's output lands in $HOME/.local/state/spec-watch/spec-watch.log between === poll and === exit lines. A malformed declaration, or a script that is missing or not executable, is logged as refused and runs no model tick in its place. Left, as actions: land the pi-tracks change (branch track/poller-cutover there) together with this tracks.json, then reinstall the schedule and restore the nev cron line, which is commented out by hand while jobs run one at a time.
  6. Landing from a fetched ref. bench/land.sh needs a local track worktree and branch; jobs/scripts/poll-and-land.sh builds both at landing time and removes them after a refusal. A bench/land.sh that lands from a fetched ref would remove that step. Optional.
  7. Map rules for specs landed outside a job. A job writes its own rules (gap 2). Exact-key rules for anything that landed without that stage are added by hand, or for a job by job-check feature-rules specs/<name>.job.json on main; aspect-ratio is the one job still owed this, together with a specs/aspect-ratio.md amendment whose touches name docs/feature-map.json. The field list in specs/job-manifest-checks.md does not yet name feature_rules.
  8. The roster's stale tick text (item 11 above) and the ground-truth, generation-loop and map-update-dry-run stage checks, which specs/job-manifest-checks.md leaves out until their artifact formats are read. The checks are specs later; the roster line is a person's edit.

Later stages (design only) #

Three larger pieces come after the job runs a real feature. None is drafted as a spec: each is either unbounded as it stands or waits on something not yet known, and a spec written now would commission code against a guess.

Performance runs #

A generated function is chosen on its score against the browsers' answers; two candidates that both score in full are not told apart. A performance run would tell them apart on cost: several candidates that are known correct on the same corpus, each measured for CPU time, wall-clock time and peak memory over many scenarios.

Shape, if built: a runner takes a function's committed candidate and the other admitted candidates from that function's batches, builds each into its own process, and calls the function in a loop over a scenario set (the function's own corpus plus randomly generated inputs from the function's cases module, whose random(seed, count) already exists). Each candidate runs in a fresh process pinned to one CPU with the first run discarded and a fixed repeat count, and reports user plus system CPU time and peak resident memory from the process's own accounting, and wall-clock time. It runs on a lane, alone, since a shared lane's load would be the noise. The output is one table per function.

Why not a spec yet:

  • The cost of one call of a leaf function is microseconds, and what matters is its share of a whole layout, so the right scenarios are whole documents, not isolated inputs. Which documents represent real pages is not decided (decision).
  • What the table is for is not decided: choosing between equal candidates, a regression check on later batches, or a number for the announcement. Each needs a different tolerance (decision).
  • Whether losing candidates' sources survive a batch is unverified: each batch writes sources/ under its run directory on the lane, and nothing shows whether the run directory is kept.

The bounded first slice, once those are answered: one function, its committed candidate against the baseline fixture, wall-clock and CPU only, reported and not enforced.

Adversarial review #

The review stage asks whether the diff implements the spec. It does not test whether the winning candidate is right in cases the corpus does not contain. One model could review another's winning candidate: given the spec text, the function's signature and the candidate's source, and not the scores, it proposes inputs it believes the candidate answers wrongly.

Shape, if built: an opinion never blocks a landing, and only a counterexample the browsers confirm counts. The reviewer's inputs are run through the same ground-truth path as the function's corpus (*_ground_truth_cache, three browsers) and through the candidate; a finding is an input where the browsers agree with each other, the candidate disagrees with them, and the reviewer's claim matches the browsers. Confirmed findings go into the job's result and are not applied: turning them into corpus cases and a new or amended spec is for a person or a coordinator session outside the loop, since the loop does not write specs. The reviewer is a review role with its own agent (specs/job-agents-setting.md), required to differ from the candidates' agent.

Why not a spec yet:

  • Each generated function has its own case type in its own *_cases.rs, so a reviewer's output has no common format to be written in. The roster's Queue item seam-kit, which factors the per-function runners into one driven by a table, is where a common case format would come from; drafting the review before it means writing the format twice.
  • Whether a counterexample should ever hold a landing (not only be reported) is a policy choice (decision); reporting only is the reversible default.
  • The browser oracle's cost for many extra inputs is not measured (unverified).

The long-running learning loop #

The idea: across many features, record what worked (which models, counts and prompts led to admitted candidates, at what cost) and use it to set the next job's package, so the loop improves without a person tuning it.

Only the recording half is known enough to build, and the drafted specs already do it: the ledger line per batch (specs/job-batch-wrapper.md), the token-cap counts (specs/cutoff-reporting.md), the provenance record per function and the result commit per job. What to do with the record is not known: no real batch has yet written a ledger line, there are about a dozen generated functions to learn from, and it is not known which knob matters (the model pool, the count, the prompt, the token cap). A loop that changed its own settings would edit a job's manifest under specs/, which the loop must not write; so any tuning is a proposal for a person or a coordinator session to turn into a spec.

Next step, not a spec: after the first several real jobs land, read the ledgers and results by hand (an action) and decide which one number would have changed a decision. A read-only summary over the same files comes after that.

What this survey did not check #

  • The design was unbuilt when this survey was written. Items 4 and 5 have since been built and a real feature has run through them ("First real run" above), which was the test this survey proposed.
  • The roster's counts (seven landings, one by hand, no announcements, no ledger line) are taken from the roster; the tick log was read only for its tail and the transforms2d lines. Rechecking against the whole log needs no decision (action, a read of $HOME/.local/state/spec-watch/spec-watch.log).
  • Where the transforms2d batch ran (development box or lane copy): unverified. Settled by reading the writer's transcript on lane 1 (action, needs lane access).
  • Whether the lane's mirror can hold a base ref and be rebased there: bench/remote-lane.sh bundles origin/main..HEAD and bench/lane-apply.sh checks out the exact commit, but neither was read for what refs the copy keeps. Settled inside item 5 (spec now).
  • Whether older hand-written provenance records would pass the check in item 2: unverified, settled by running it (spec now, inside item 2).
  • bench/land.sh past its option parsing and helper functions (the gate, rebase and finish steps) was read from the header comments, not line by line; refusals and their order are as the header states them.
  • Checks that do not exist yet. The proposed stage checks name artifacts (cases_run, ledger lines, map events) that are read from committed formats, but no checker for them exists; their fields were read from crates/layout/src/generated/table.provenance.json and bench/ledger.py's callers only.
  • The drafted specs were not built or run. They were written against the files they cite, read while drafting (bench/land.sh, bench/on-lane.sh, bench/remote-lane.sh, bench/batch-lane.sh, bench/ledger.py, candidate_crate.rs, the provenance records). Their test lists are proposals; whether each is enough is settled when a track builds it (spec now). Two facts they rely on are unverified: that the lane's mirror keeps result/<name> and the job branch fetchable after a lane copy is swept, and that the lane's shared store is writable for the ledger at the time of a real batch.