This repository has no description
bez specs job-batch-wrapper.md
5.9 kB


enabled: false touches: [jobs/scripts/run-batch.sh, jobs/tests/run_batch.rs, jobs/fixtures/run-batch/**] depends-on: [job-manifest-checks] #

job-batch-wrapper: a model batch that cannot finish without a ledger line #

The overnight run's transforms2d batch was started the way its spec said, cargo run --release -p generation --bin transform_real_model_run, and wrote no ledger line, so the announcement had no candidates, tokens or CPU-hours to post. Only bench/absolute-mixed-batch.sh and bench/overflow-mixed-run.sh record a batch, through bench/batch-lane.sh batch_measure_stop, which calls bench/ledger.py. That script is written never to fail a batch: with no ledger directory it prints one line on stderr, exits 0 and writes nothing, so a run that skips it and a run whose write silently failed look the same. The prose that tells a writer to go through a script is not a check. This spec builds the one script every job's batch stage goes through, so that a batch either leaves a ledger line or the stage fails.

Where. jobs/scripts/run-batch.sh, in the crate that job-manifest-checks adds, with its test jobs/tests/run_batch.rs and fixtures jobs/fixtures/run-batch/. It calls the existing bench/batch-lane.sh functions and bench/ledger.py and edits neither: both are under bench/, which bench/land.sh refuses in a track. Those bench/ scripts are read from the worktree the wrapper runs in, as the other batch scripts read them; a job whose writer edits one gets its landing refused, so the wrapper needs no pinned copy of them.

Use.

jobs/scripts/run-batch.sh --job <name> --measure-dir <dir> -- <command...>

<command...> is the batch itself, taken from the job's manifest (for example cargo run --release -p generation --bin table_real_model_run), so one wrapper serves every generated function. The wrapper does, in order:

  1. Refuses, exit 2 and without running the command, when NEV_PROVIDER is set to anything but claude-code. The metered Anthropic API is never used unattended.
  2. batch_on_lane jobs/scripts/run-batch.sh --pull <measure-dir>/ -- "$@": on a lane it returns, anywhere else it replaces itself with bench/on-lane.sh --lane-only and exits 3 when no lane answers. The batch never falls back to the development box.
  3. batch_load_env, so the model address in .env reaches the batch under anything the environment already sets. Nothing from .env is printed.
  4. Resolves the ledger directory the way bench/ledger.py ledger_dir() does (NEV_LEDGER_DIR, else <NEV_LANE_SHARED or GENERA_LANE_SHARED>/ledger when that shared directory exists) and exits 4 before running the command when neither gives one. This is the check bench/ledger.py cannot make: it learns there is nowhere to write only after the batch has run and its cost is spent.
  5. Exports NEV_LEDGER_TRACK=<name>, so the ledger file is <name>.jsonl and the job's ledger check finds it, and records the file's line count.
  6. batch_measure_start <measure-dir>, runs the command with its output through batch_stamp, and takes the command's exit status. batch_measure_stop runs on every exit, through a trap that covers the command failing and the wrapper being interrupted (INT, TERM), so a batch that failed halfway still leaves the line for what it spent.
  7. Compares the ledger's line count with the one recorded. When it did not grow, prints one line saying no ledger line was written and exits 5, unless the command itself failed, in which case the command's exit status wins and the same line is still printed. Otherwise exits with the command's status.

Why 5 and 4 are different. Exit 4 means the batch was never started, so nothing is spent and the job can be resubmitted after the store is mounted. Exit 5 means a batch ran and its cost is unrecorded, which the job's ledger check then reports as a failed stage with the batch's output kept in the measure directory.

Tests. jobs/tests/run_batch.rs runs the wrapper with std::process::Command, with the environment cleared and only PATH, HOME, the lane variables and a temporary ledger directory set, and a fixture command that prints the log lines bench/ledger.py reads (timing:, A/B candidate(s) compiled, received, K completion tokens, scored, best) and touches a marker file. It sets NEV_ON_LANE=1 so the run is local to the test and no lane is needed; it never sets a real lane address. Cases: a passing batch leaves one new line for the job name with the fixture's model counts; a failing command's status is returned and a line is still written; a batch interrupted with SIGTERM still writes a line; NEV_PROVIDER=anthropic exits 2 and the marker file is absent; no ledger directory exits 4 and the marker is absent; a ledger directory that cannot be written (a path under a regular file) exits 5; with NEV_ON_LANE unset and NEV_LANE_SSH set to an unresolvable name the wrapper exits 3 and the marker is absent (an environment variable outranks bench/lane.local.json, so a worktree's own lane file does not change the result); with NEV_LEDGER_DIR and a shared directory both set, NEV_LEDGER_DIR is written. The test takes several seconds, because batch_measure_stop waits one sample interval.

Gates. cargo test -p jobs --test run_batch then cargo test --workspace, on a build lane (bench/on-lane.sh --lane-only), never on the development box. The batch itself runs no model, so this spec draws nothing and has no provenance record or floor.

Not covered. Whether the ledger's shared store is writable from a job's lane at the moment of a real batch: the tests write to a temporary directory. The first real job confirms it; until then it is unverified.

Files. touches is the wrapper, its test and its fixtures. It follows job-manifest-checks because the test lives in that crate.