Something went wrong. Try again.
This repository has no description
Something went wrong. Try again.
Shell
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465#!/usr/bin/env bash# One measuring batch of absolute_real_model_run from a model list, from the same# starting point every recorded absolute batch had, plus a compiler error file per# candidate so failures can be classified afterwards (docs/model-first-cost.md -># "Why candidates fail to compile").## NEV_MODELS=qwen3.8-flash-next:32:32,qwen3.8-27b-fast:32:32 bench/absolute-mixed-batch.sh## Runs on a build lane, never on the development box: started anywhere else, it re-runs# itself on the lane through bench/on-lane.sh --lane-only (bench/batch-lane.sh), passing# the NEV_* settings along, and copies batch-1/ and the committed absolute candidate# back afterwards. With no lane answering it fails instead. On the lane, .env points at# the gateway (docs/infrastructure.md -> "Reaching the model gateway"); the binaries read# the model endpoint from the environment, so .env is loaded here, under anything the# environment already sets; nothing from it is printed. The ground-truth cache is built# only when missing or empty; the regression set is always reseeded from the baseline# (absolute_generation_loop), so every measured batch samples its prompt from the same# failing cases.## Leaves corpus/candidates/absolute-model-run/batch-1/: prompt.txt, sources/,# replies/ (raw replies), report.json, run.log, and compile-errors/<candidate>.txt with# compile-errors/_status.txt (one "<candidate> exit <code>" line each); and the phase# timings and CPU use of the run (bench/batch-lane.sh -> batch_measure_start):# cpu.tsv, phases.tsv, run.stamped.log, cpu-phases.txt. A winner that beats the# committed candidate rewrites crates/layout/src/generated/absolute.rs and its# provenance in place, which is why those two files come back too.set -uo pipefailBENCH_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"# shellcheck source=bench/batch-lane.shsource "$BENCH_DIR/batch-lane.sh"RUN=corpus/candidates/absolute-model-runbatch_on_lane bench/absolute-mixed-batch.sh \ --pull "$RUN/batch-1/" \ --pull crates/layout/src/generated/absolute.rs \ --pull crates/layout/src/generated/absolute.provenance.json \ -- "$@"
cd "$BENCH_DIR/.."# GENERA_MODELS, the name before the project was renamed, is still read.: "${NEV_MODELS:-${GENERA_MODELS:?set NEV_MODELS, e.g. qwen3.8-flash-next:32:32,qwen3.8-27b-fast:32:32}}"batch_load_envmkdir -p "$RUN"MEASURE="$RUN/measure"rm -rf "$MEASURE"batch_measure_start "$MEASURE"# A lane links corpus/absolute-ground-truth to a shared directory that can exist and be# empty, so an empty directory counts as missing.if [ -z "$(ls -A corpus/absolute-ground-truth/ 2>/dev/null)" ]; then batch_phase ground-truth cargo run --release -p generation --bin absolute_ground_truth_cache || exit 1fibatch_phase generation-looprm -f corpus/absolute-regression-set.jsoncargo run --release -p generation --bin absolute_generation_loop || exit 1rm -rf "$RUN/batch-1"batch_phase build-run-binarycargo build --release -p generation --bin absolute_real_model_run || exit 1batch_phase model-runNEV_MAX_BATCHES=1 cargo run --release -p generation --bin absolute_real_model_run 2>&1 \ | batch_stamp | tee "$RUN/run.log"mkdir -p "$RUN/batch-1"cp "$RUN/run.log" "$RUN/batch-1/run.log"batch_measure_stopcp "$MEASURE"/* "$RUN/batch-1/"