enabled: false touches: [jobs/scripts/run-batch.sh, jobs/tests/run_batch.rs, jobs/fixtures/run-batch/**] depends-on: [job-manifest-checks] #
job-batch-wrapper: a model batch that cannot finish without a ledger line #
The overnight run's transforms2d batch was started the way its spec said, cargo run --release -p generation --bin transform_real_model_run, and wrote no ledger line, so the
announcement had no candidates, tokens or CPU-hours to post. Only
bench/absolute-mixed-batch.sh and bench/overflow-mixed-run.sh record a batch, through
bench/batch-lane.sh batch_measure_stop, which calls bench/ledger.py. That script is
written never to fail a batch: with no ledger directory it prints one line on stderr,
exits 0 and writes nothing, so a run that skips it and a run whose write silently failed
look the same. The prose that tells a writer to go through a script is not a check. This
spec builds the one script every job's batch stage goes through, so that a batch either
leaves a ledger line or the stage fails.
Where. jobs/scripts/run-batch.sh, in the crate that job-manifest-checks adds, with its
test jobs/tests/run_batch.rs and fixtures jobs/fixtures/run-batch/. It calls the existing
bench/batch-lane.sh functions and bench/ledger.py and edits neither: both are under
bench/, which bench/land.sh refuses in a track. Those bench/ scripts are read from the
worktree the wrapper runs in, as the other batch scripts read them; a job whose writer edits
one gets its landing refused, so the wrapper needs no pinned copy of them.
Use.
jobs/scripts/run-batch.sh --job <name> --measure-dir <dir> -- <command...>
<command...> is the batch itself, taken from the job's manifest (for example cargo run --release -p generation --bin table_real_model_run), so one wrapper serves every
generated function. The wrapper does, in order:
- Refuses, exit 2 and without running the command, when
NEV_PROVIDERis set to anything butclaude-code. The metered Anthropic API is never used unattended. batch_on_lane jobs/scripts/run-batch.sh --pull <measure-dir>/ -- "$@": on a lane it returns, anywhere else it replaces itself withbench/on-lane.sh --lane-onlyand exits 3 when no lane answers. The batch never falls back to the development box.batch_load_env, so the model address in.envreaches the batch under anything the environment already sets. Nothing from.envis printed.- Resolves the ledger directory the way
bench/ledger.pyledger_dir()does (NEV_LEDGER_DIR, else<NEV_LANE_SHARED or GENERA_LANE_SHARED>/ledgerwhen that shared directory exists) and exits 4 before running the command when neither gives one. This is the checkbench/ledger.pycannot make: it learns there is nowhere to write only after the batch has run and its cost is spent. - Exports
NEV_LEDGER_TRACK=<name>, so the ledger file is<name>.jsonland the job's ledger check finds it, and records the file's line count. batch_measure_start <measure-dir>, runs the command with its output throughbatch_stamp, and takes the command's exit status.batch_measure_stopruns on every exit, through a trap that covers the command failing and the wrapper being interrupted (INT,TERM), so a batch that failed halfway still leaves the line for what it spent.- Compares the ledger's line count with the one recorded. When it did not grow, prints one line saying no ledger line was written and exits 5, unless the command itself failed, in which case the command's exit status wins and the same line is still printed. Otherwise exits with the command's status.
Why 5 and 4 are different. Exit 4 means the batch was never started, so nothing is spent and the job can be resubmitted after the store is mounted. Exit 5 means a batch ran and its cost is unrecorded, which the job's ledger check then reports as a failed stage with the batch's output kept in the measure directory.
Tests. jobs/tests/run_batch.rs runs the wrapper with std::process::Command, with the
environment cleared and only PATH, HOME, the lane variables and a temporary ledger
directory set, and a fixture command that prints the log lines bench/ledger.py reads
(timing:, A/B candidate(s) compiled, received, K completion tokens, scored, best) and
touches a marker file. It sets NEV_ON_LANE=1 so the run is local to the test and no lane is
needed; it never sets a real lane address. Cases: a passing batch leaves one new line for
the job name with the fixture's model counts; a failing command's status is returned and a
line is still written; a batch interrupted with SIGTERM still writes a line; NEV_PROVIDER=anthropic
exits 2 and the marker file is absent; no ledger directory exits 4 and the marker is absent;
a ledger directory that cannot be written (a path under a regular file) exits 5; with
NEV_ON_LANE unset and NEV_LANE_SSH set to an unresolvable name the wrapper exits 3 and
the marker is absent (an environment variable outranks bench/lane.local.json, so a
worktree's own lane file does not change the result); with NEV_LEDGER_DIR and a shared
directory both set, NEV_LEDGER_DIR is written. The test takes several seconds, because
batch_measure_stop waits one sample interval.
Gates. cargo test -p jobs --test run_batch then cargo test --workspace, on a build
lane (bench/on-lane.sh --lane-only), never on the development box. The batch itself runs
no model, so this spec draws nothing and has no provenance record or floor.
Not covered. Whether the ledger's shared store is writable from a job's lane at the moment of a real batch: the tests write to a temporary directory. The first real job confirms it; until then it is unverified.
Files. touches is the wrapper, its test and its fixtures. It follows
job-manifest-checks because the test lives in that crate.