---
enabled: false
touches: [crates/generation/src/candidate_crate.rs]
depends-on: [floors-table]
---

# `cutoff-reporting`: separate replies cut off at the token cap from replies that were complete and failed to compile

A batch's report says how many candidates compiled and drops the rest into one count. Two
different things sit in that count. A reply that ended because the model reached the token
cap (`crates/generation/src/model.rs` `MAX_COMPLETION_TOKENS`, 12000, ending with
`finish_reason: "length"` on the gateway or `stop_reason: "max_tokens"` on the Claude API)
stops in the middle of a function and is a certain compile failure that says nothing about
the model's answer. A reply that finished and still failed to compile is a wrong or
malformed answer. The cap was raised from 8000 to 12000 for exactly this: in one 32-candidate
batch four replies were cut off and counted as failures. Today the only way to tell the two
apart is to read each candidate's `stop_reason` in the run's candidate listing by hand
(`docs/model-first-cost.md` → "Why candidates fail to compile"). A batch that loses half its
candidates to the cap reads the same as one whose model cannot write the function, and an
unattended run then draws the wrong conclusion from a flat batch. This spec counts them
apart.

**Where.** `crates/generation/src/candidate_crate.rs`, the one file that holds both
`Candidate` outcomes and the compile probe. `probe_batch` is the single call every batch
binary makes after drawing candidates, and it already has the candidates and the probe
outcomes side by side, so the count goes there and no `*_real_model_run` binary changes.

- `FailureTally { drawn, compiled, cut_off, complete_failed }`, all `usize`, and
  `failure_tally(candidates, outcomes) -> BTreeMap<String, FailureTally>`, keyed by the
  candidate's `model` (`unknown` when it has none: a hand-written candidate). Outcomes are
  matched to candidates by name.
- A candidate is cut off when its `stop_reason` is `length` or `max_tokens`
  (compared without case), the two names the two providers use for the token cap, and it
  did not compile. A candidate that compiled counts as compiled whatever its stop reason.
  Any other candidate that did not compile is complete and failed to compile. A candidate
  with no `stop_reason` (a CLI completion reports none, as `Candidate::stop_reason` says)
  is counted as complete, since nothing says otherwise; the report says so in a comment
  and the caveat below.
- `compiled + cut_off + complete_failed == drawn` for every model, and a unit test
  asserts it.
- `probe_batch` prints one line per model after probing, to stdout with the batch's other
  progress lines:
  `compile outcome: <model>: <compiled> compiled, <cut_off> cut off at the token cap, <complete_failed> complete and failed to compile, of <drawn>`.
  The wording avoids the shapes `bench/ledger.py` reads (`<model>: A/B candidate(s) compiled`,
  `timing: ...`, `... received, K completion tokens`, `... scored, best ...`), so the ledger
  line a batch writes is unchanged.

The token cap, the probe, the compiler messages, `write_compile_errors` and its
`_status.txt` layout stay as they are. Nothing here retries a cut-off candidate, raises the
cap or changes what counts as admitted; a cut-off candidate is still dropped from scoring.
The report only stops calling it a model failure.

**Tests.** Unit tests in the file's existing `mod tests`, on in-memory `Candidate` and
`ProbeOutcome` values with no compiler run: a mixed batch of two models with each of the
four outcomes; both spellings of the cap; a capped candidate that compiled; no `stop_reason`;
an outcome with no matching candidate name (ignored); an empty batch (an empty map, no
line printed).

**Gates.** `cargo test -p generation` then `cargo test --workspace`, on a build lane
(`bench/on-lane.sh --lane-only`), never on the development box. This spec runs no model
batch and adds no generated function, so it has no provenance record and no floor.

**Left to a hand change.** Teaching `bench/ledger.py` to carry the two counts into a ledger
line, so an announcement can say how many candidates the cap cost, edits `bench/`, which
`bench/land.sh` refuses in a track. It is listed in `docs/autonomous-loop.md` → "Hand
changes still needed".

**Caveat.** A candidate from the Claude CLI (`claude-code`) has no `stop_reason`, so a
cut-off reply from that source is counted as complete and failed. This spec does not fix
that: the CLI reports nothing to read. Recording it here keeps a claude-code batch's
"complete and failed" number from being over-read.

**Files.** `touches` is the one source file (its tests are in it). It shares
`crates/generation/` with every generated-function spec and with `floors-table`, so it
follows the latter.