enabled: false touches: [crates/generation/src/candidate_crate.rs] depends-on: [floors-table] #

cutoff-reporting: separate replies cut off at the token cap from replies that were complete and failed to compile #

A batch's report says how many candidates compiled and drops the rest into one count. Two different things sit in that count. A reply that ended because the model reached the token cap (crates/generation/src/model.rs MAX_COMPLETION_TOKENS, 12000, ending with finish_reason: "length" on the gateway or stop_reason: "max_tokens" on the Claude API) stops in the middle of a function and is a certain compile failure that says nothing about the model's answer. A reply that finished and still failed to compile is a wrong or malformed answer. The cap was raised from 8000 to 12000 for exactly this: in one 32-candidate batch four replies were cut off and counted as failures. Today the only way to tell the two apart is to read each candidate's stop_reason in the run's candidate listing by hand (docs/model-first-cost.md → "Why candidates fail to compile"). A batch that loses half its candidates to the cap reads the same as one whose model cannot write the function, and an unattended run then draws the wrong conclusion from a flat batch. This spec counts them apart.

Where. crates/generation/src/candidate_crate.rs, the one file that holds both Candidate outcomes and the compile probe. probe_batch is the single call every batch binary makes after drawing candidates, and it already has the candidates and the probe outcomes side by side, so the count goes there and no *_real_model_run binary changes.

• FailureTally { drawn, compiled, cut_off, complete_failed }, all usize, and failure_tally(candidates, outcomes) -> BTreeMap<String, FailureTally>, keyed by the candidate's model (unknown when it has none: a hand-written candidate). Outcomes are matched to candidates by name.
• A candidate is cut off when its stop_reason is length or max_tokens (compared without case), the two names the two providers use for the token cap, and it did not compile. A candidate that compiled counts as compiled whatever its stop reason. Any other candidate that did not compile is complete and failed to compile. A candidate with no stop_reason (a CLI completion reports none, as Candidate::stop_reason says) is counted as complete, since nothing says otherwise; the report says so in a comment and the caveat below.
• compiled + cut_off + complete_failed == drawn for every model, and a unit test asserts it.
• probe_batch prints one line per model after probing, to stdout with the batch's other progress lines: compile outcome: <model>: <compiled> compiled, <cut_off> cut off at the token cap, <complete_failed> complete and failed to compile, of <drawn>. The wording avoids the shapes bench/ledger.py reads (<model>: A/B candidate(s) compiled, timing: ..., ... received, K completion tokens, ... scored, best ...), so the ledger line a batch writes is unchanged.

The token cap, the probe, the compiler messages, write_compile_errors and its _status.txt layout stay as they are. Nothing here retries a cut-off candidate, raises the cap or changes what counts as admitted; a cut-off candidate is still dropped from scoring. The report only stops calling it a model failure.

Tests. Unit tests in the file's existing mod tests, on in-memory Candidate and ProbeOutcome values with no compiler run: a mixed batch of two models with each of the four outcomes; both spellings of the cap; a capped candidate that compiled; no stop_reason; an outcome with no matching candidate name (ignored); an empty batch (an empty map, no line printed).

Gates. cargo test -p generation then cargo test --workspace, on a build lane (bench/on-lane.sh --lane-only), never on the development box. This spec runs no model batch and adds no generated function, so it has no provenance record and no floor.

Left to a hand change. Teaching bench/ledger.py to carry the two counts into a ledger line, so an announcement can say how many candidates the cap cost, edits bench/, which bench/land.sh refuses in a track. It is listed in docs/autonomous-loop.md → "Hand changes still needed".

Caveat. A candidate from the Claude CLI (claude-code) has no stop_reason, so a cut-off reply from that source is counted as complete and failed. This spec does not fix that: the CLI reports nothing to read. Recording it here keeps a claude-code batch's "complete and failed" number from being over-read.

Files. touches is the one source file (its tests are in it). It shares crates/generation/ with every generated-function spec and with floors-table, so it follows the latter.