Model-first cost: the absolute-positioning function #
Metering for track/model-first (handoff.md): CSS 2.1 absolute positioning, the first
generated function attempted by the model before any hand-written candidate exists
(roadmap.md -> "Proof of concept", claim 3). This table is appended to as each phase
completes; it is not yet complete (handoff.md -> "Next steps").
Per-phase wall-clock #
Phase boundaries below are read off each new file's own last-modified timestamp, not a
stopwatch kept running through the session -- the session did not call out to a clock at
the start of each phase, only at its end, so the first phase's own start (reading the
percentage/width/margin functions as a template, handoff.md -> "Design") has no marker and
its duration is folded into "template reading" below rather than invented. Every phase
from the substrate accessors on is bounded by two real timestamps.
| Phase | Wall-clock | Model tokens | Notes |
|---|---|---|---|
Template reading (percentage/width/margin functions, dom::computed, cases.rs, fragment.rs, plus one routine-explorer dispatch to find Stylo's Inset/PositionProperty types) |
not separately timed; folded into the substrate-accessors row below | none (no model calls; a routine-explorer subagent ran, its own tokens are that agent's, not this function's generation cost) |
Ends at crates/dom/src/computed.rs's edit, 06:04. |
Substrate: dom::computed::position/top/right/bottom/left and Position |
~2 min (to 06:06) | none | Hand-written accessors, mirroring margin_top et al.; not part of the metered generation loop. |
Generated-function types: crates/layout/src/absolute.rs (AbsoluteInputs/AbsoluteOutputs/AbsoluteResolver/resolve_baseline) |
~12 min (to 06:18, includes registering the module and a first cargo test -p layout) |
none | Hand-written function types and its deliberately-wrong baseline (handoff.md's hard rule: no correct implementation yet). |
Structural half: crates/layout/src/fragment.rs (stack_children, layout_absolute_box, PendingAbsolute, attach_absolute, relative-shift) |
folded into the types row above through 06:18 | none | The larger of the two hand-written pieces; cargo test -p layout passed on the first build after this landed. |
Corpus: crates/harness/src/absolute_cases.rs (12 categories, enumerated/random) |
~5 min (to 06:23) | none | Reuses crate::cases's Elem/Source/px/document/RESET (newly pub(crate)); does not touch crate::cases::RECIPES. |
Provenance spec text: crates/provenance/src/css21_absolute_positioning.rs |
~1 min (to 06:24) | none | CSS 2.1 §9.3.1/§9.3.2/§10.1/§10.3.7/§10.6.4, narrowed the same way css21_percentage_resolution.rs narrows its own two sections. |
Runner/prompt/admit/candidate-crate wiring: absolute_runner.rs, prompt::build_absolute_positioning, admit::record_for_absolute_positioning, candidate_crate::Seam::AbsolutePositioning |
~5 min (to 06:29, includes the two seeded-wrong fixture candidates) | none | cargo test -p generation passed at 55/55 including the new tests. |
Ground-truth cache and generation-loop binaries: bin/absolute_ground_truth_cache.rs, bin/absolute_generation_loop.rs |
~3 min (to 06:32) | none | None of the three binaries in this row and the next have been run yet -- each either starts a real browser oracle or reads its cache, out of scope for this stopping point (handoff.md -> "Next steps"). |
Workspace verification (first pass): cargo build --workspace (one Seam match-exhaustiveness fix in bin/feature_model_run.rs), cargo test --workspace, cargo run --release -p layout --bin fragment_check -- --random |
~4 min (06:32 to 06:35) | none | All green: 0 failed across the workspace; fragment_check --random reports 94/227 (25/25 fixed-box), unchanged from main. |
Real-model-run binary: bin/absolute_real_model_run.rs, plus a second cargo build --workspace/cargo test --workspace pass to confirm it |
~3 min (to 06:38) | none | Not run; the binary itself exits early and successfully if no model credentials are configured (its own module doc), so it is safe to leave written but unrun. |
Total elapsed for everything above (first substrate edit to the last verification command):
06:04 to 06:38, about 34 minutes, plus an untimed template-reading phase before it. No
model batch has run yet, so the "model tokens" column is empty throughout -- the whole
table so far is the hand-written scaffolding docs/platform-economics.md -> "Per-feature
cost" already flags as the uncounted cost sitting underneath a cheap generation step
("the scaffolding each new generated function needs ... was built by agent sessions whose tokens were
not metered, and on this evidence it costs more than the generation it enables").
Batch scores and token totals #
absolute_ground_truth_cache wrote 108 cases to corpus/absolute-ground-truth/ (0
dropped for engine disagreement). absolute_generation_loop then seeded the permanent
regression set with the 6 cases resolve_baseline already passes (all StaticPosition)
and confirmed the gate rejects both seeded-wrong fixtures
(ignores_auto_margin_centering: 57/108, treats_right_as_left: 0/108, both
admitted=false). Each row below is one absolute_real_model_run batch
(NEV_PROVIDER=anthropic, model claude-opus-5, 8 candidates drawn per batch); the
per-category tally is the admitted (best) candidate's.
| batch | compiled/drawn | admitted candidate | best score | per-category tally (best) | prompt tokens | output tokens |
|---|---|---|---|---|---|---|
| 1 | 8/8 | model_0 |
74/108 | BottomHeightTopAuto 0/6, LeftWidthRightAuto 8/8, MarginAutoCentering 11/11, NegativeOffset 13/13, OffsetsDefineSize 0/12, OverConstrained 12/12, OverConstrainedRtl 0/8, PercentageOffsetsAndSizes 0/8, RelativeAncestorPadding 11/11, RightWidthLeftAuto 9/9, StaticPosition 6/6, TopHeightBottomAuto 4/4 | 80,184 | 102,955 |
| 2 | 7/8 (model_2 dropped: Anthropic API returned "credit balance is too low") |
model_0 |
74/108 | identical to batch 1 — BottomHeightTopAuto 0/6, LeftWidthRightAuto 8/8, MarginAutoCentering 11/11, NegativeOffset 13/13, OffsetsDefineSize 0/12, OverConstrained 12/12, OverConstrainedRtl 0/8, PercentageOffsetsAndSizes 0/8, RelativeAncestorPadding 11/11, RightWidthLeftAuto 9/9, StaticPosition 6/6, TopHeightBottomAuto 4/4 | 60,767 (7 candidates; 8th dropped before billing) | 92,451 |
| 3 | 0/8 — every request refused | none | n/a | n/a | 0 | 0 |
loop 1 (claude-code, resolved to claude-sonnet-5) |
8/8 | model_0 (all 8 tied at 74/108) |
74/108 | identical to batch 1 above | n/a — the CLI reports no token counts; batch wall-clock 251 s | n/a |
loop 2 (claude-code) |
8/8 (model_2 compiled but was rejected: 63/108, 11 regressions, MarginAutoCentering 0/11) |
model_0 (7 tied at 74/108) |
74/108 | identical to batch 1 above | n/a; batch wall-clock 242 s | n/a |
loop 3 (claude-code) |
7/8 (model_3 failed to compile: a non-exhaustive match) |
model_0 (all 7 tied at 74/108) |
74/108 | identical to batch 1 above | n/a; batch wall-clock 243 s | n/a |
rig-batch 1 (qwen3.8-27b-fast, 32 candidates, concurrency 8) |
4/32 | model_8 (tied with model_16 at 63/108) |
63/108 | BottomHeightTopAuto 0/6, LeftWidthRightAuto 8/8, MarginAutoCentering 0/11, NegativeOffset 13/13, OffsetsDefineSize 0/12, OverConstrained 12/12, OverConstrainedRtl 0/8, PercentageOffsetsAndSizes 0/8, RelativeAncestorPadding 11/11, RightWidthLeftAuto 9/9, StaticPosition 6/6, TopHeightBottomAuto 4/4 | 229,760 (0/32 null-content) | 179,000; wall-clock 414 s |
rig-batch 2 (qwen3.8-27b-fast) |
2/32 | model_31 (tied with a rejected model_7) |
63/108 | identical to rig-batch 1 above — a non-admitted model_7 reached 73/108 with OverConstrainedRtl 8/8, but 9 regressed cases kept it out of admission |
198,240 (0/32 null-content) | 151,547; wall-clock 266 s |
rig-batch 3 (qwen3.8-27b-fast) |
4/32 | model_2 (tied with model_27, model_29 at 63/108) |
63/108 | identical to rig-batch 1 above | 198,240 (0/32 null-content) | 170,688; wall-clock 293 s |
flash-batch 1 (qwen3.8-flash-next, 32 candidates, concurrency 32) |
14/32 (12 scored; model_16 and model_25 panicked mid-corpus and were dropped) |
model_3 |
74/108 | BottomHeightTopAuto 0/6, LeftWidthRightAuto 8/8, MarginAutoCentering 11/11, NegativeOffset 13/13, OffsetsDefineSize 0/12, OverConstrained 12/12, OverConstrainedRtl 0/8, PercentageOffsetsAndSizes 0/8, RelativeAncestorPadding 11/11, RightWidthLeftAuto 9/9, StaticPosition 6/6, TopHeightBottomAuto 4/4 | not reported by this alias (0/32 null-content) | not reported; wall-clock 1,158 s, slowest single reply 595.7 s |
flash-batch 2 (qwen3.8-flash-next) |
12/32 (12 scored) | model_1 |
74/108 | identical to flash-batch 1 above | not reported (0/32 null-content) | not reported; wall-clock 1,014 s, slowest single reply 593.8 s |
flash-batch 3 (qwen3.8-flash-next) |
10/32 (9 scored; model_17 panicked) |
model_31 — committed |
82/108 | flash-batch 1's tally with OverConstrainedRtl 0/8 → 8/8; the other three target categories unchanged at 0 | not reported (0/32 null-content) | not reported; wall-clock 1,152 s, slowest single reply 594.8 s |
flash-batch 4 (qwen3.8-flash-next) |
20/32 (17 scored; model_9, model_16, model_19 panicked) |
model_5 |
82/108 | identical to flash-batch 3 above | not reported (0/32 null-content) | not reported; wall-clock 966 s, slowest single reply 584.5 s |
flash-batch 5 (qwen3.8-flash-next) |
19/32 (18 scored; model_25 panicked) |
model_11 |
82/108 | identical to flash-batch 3 above | not reported (0/32 null-content) | not reported; wall-clock 1,056 s, slowest single reply 599.4 s |
diagnosis batch (mixed, 16 qwen3.8-flash-next + 16 qwen3.8-27b-fast, before the fixes below) |
11/32: flash-next 10/16 (6 scored, 4 panicked), 27b-fast 1/16 | model_2 (flash-next) |
74/108 | flash-next best 74/108, flash-batch 1's tally; 27b-fast best 63/108, rig-batch 1's tally | 27b-fast only, not summed | not summed; 27b-fast replies 37–104 s, flash-next 332–484 s |
| mixed-batch 1 (32 flash-next + 32 27b-fast, both at 32 in flight) | 23/64: flash-next 13/19 (13 of its 32 requests dropped, see below; 11 scored), 27b-fast 10/32 (10 scored) | model_40 (27b-fast) |
82/108 | 27b-fast best 82/108, flash-batch 3's tally; flash-next best 74/108, flash-batch 1's tally | 27b-fast only, not summed | not summed; wall-clock 737.9 s |
| mixed-batch 2 (same shape, three-import contract) | 29/64: flash-next 25/32 (22 scored), 27b-fast 4/32 (3 scored) | model_58 (27b-fast) |
82/108 | as mixed-batch 1: 27b-fast best 82/108 with OverConstrainedRtl 8/8, flash-next best 74/108 |
6,624 per 27b-fast request | not summed; wall-clock 849.8 s, slowest flash-next reply 760.2 s |
vm-batch 1 (qwen3.8-27b-fast-2, 32 candidates, concurrency 16, before the prompt fixes) |
2/32 | model_13 (tied with model_18 at 74/108) |
74/108 | identical to flash-batch 1 above | 212,160 (0/32 null-content) | 197,661 (12 of 32 replies stopped at the 8,000-token cap); slowest single reply 106.5 s |
vm-batch 2 (qwen3.8-27b-fast-2, same shape, after the prompt fixes) |
8/32 (8 scored) | model_4 (tied with model_5 at 82/108) |
82/108 | flash-batch 3's tally: OverConstrainedRtl 8/8, MarginAutoCentering 11/11 |
211,968 (6,624 per request) | 175,004 (0 of 32 at the 12,000-token cap, longest 11,788); wall-clock 233.1 s, slowest single reply 107.9 s |
dsv-batch 1 (deepseek-v4.1-flash, 8 candidates, concurrency 2) |
4/8 (4 scored) | model_7 |
82/108 | flash-batch 3's tally: OverConstrainedRtl 8/8, MarginAutoCentering 11/11 |
50,496 (6,312 per request) | 76,115 (5 of 8 at the 12,000-token cap); wall-clock 1,731.3 s, replies 143–512 s |
dsv-batch 2 (deepseek-v4.1-flash, 32 candidates, concurrency 2) |
16/32 (15 scored; model_6 panicked) |
model_19 (tied with model_21, model_23 at 82/108) |
82/108 | flash-batch 3's tally: OverConstrainedRtl 8/8, MarginAutoCentering 11/11 |
201,984 (6,312 per request) | 306,315 (19 of 32 at the 12,000-token cap); wall-clock 6,572.4 s, replies 83–530 s |
cb-batch 1 (containing-block function, 16 qwen3.8-flash-next + 32 qwen3.8-27b-fast + 32 qwen3.8-27b-fast-2 + 2 deepseek-v4.1-flash, one mixed batch) |
77/82: flash-next 16/16, 27b-fast 30/32, 27b-fast-2 30/32, deepseek 1/2 | model_0 (flash-next) — committed |
105/105 | every category 100%: AbsoluteNestedPositionedAncestors 9/9, AbsoluteNoPositionedAncestor 12/12, FixedAtTopLevel 10/10, FixedBottomRightOnly 11/11, FixedEscapingNestedAncestors 12/12, FixedInsideAbsoluteAncestor 14/14, FixedInsideRelativeAncestor 12/12, FixedPercentageHeight 14/14, StaticInPaddedBlockContainer 11/11 | not reported by these aliases | 47,613 completion tokens reported (27b-fast 20,201 over 32 replies, 27b-fast-2 24,262 over 32, deepseek 3,150 over 2, flash-next reports none); wall-clock 677.4 s, slowest single reply 71.8 s |
The cb-batch 1 row is the whole of the containing-block function's model run
(specs/fixed-positioning.md, containing_block_real_model_run): one batch, and the loop
stopped itself because no failing case was left to repair rather than on a plateau or the
six-batch cap. It is the cheapest generated function so far by a wide margin, and the most lopsided
result: 77 of 82 candidates compiled, and every one that compiled and scored reached
105/105 except a single qwen3.8-27b-fast-2 candidate at 30/105 -- the baseline's own
score, reached by treating a fixed box as a static one. CSS 2.1 10.1 is four rules with no
arithmetic in them, and the function's inputs hand a candidate the ancestor chain already built,
so there is little left to get wrong: the compile rate (94%) is far above the absolute
function's best (75 of 160 at the same flash-next alias) because the contract is two imports and
one enum rather than three imports and a four-variant length type. What that says about cost
is that the expensive generated functions are the ones with equations in them, not the ones with rules.
Every number in the cb-batch 1 row is against the corpus as it stood at that batch, 105
cases. The corpus is 106 now: one recipe was added for a relative ancestor moved with
top/left rather than with margin, the case no earlier recipe covered, and the
committed candidate scores 106/106 over it with that category at 13/13
(containing_block_generation_loop re-scores the committed candidate for exactly this
reason — the corpus is gitignored, so a quoted score is only checkable if something
recomputes it). Nothing was re-drawn from any model to get that; the batch above is still
the whole of this function's model spend.
The three rig-batch N rows are one unattended run of absolute_real_model_run against the
rig's free qwen3.8-27b-fast alias (NEV_MODEL=qwen3.8-27b-fast NEV_CANDIDATES=32 NEV_CONCURRENCY=8, specs/rig-batch-absolute.md), drawing four times as many candidates
per batch as every batch above. Every one of the 96 draws across the three batches returned
real content (0 null-content replies at this alias, unlike docs/rig-capacity.md's earlier
8-candidate qwen3.8-27b-fast batch, which saw 3 of 8 come back content: null). The compile
rate stayed low (4/32, 2/32, 4/32 — 10 of 96 total, close to the 1-of-5-real-replies rate
docs/rig-capacity.md's width-function batch saw from the same alias) and every admitted
candidate plateaued at 63/108 with the identical per-category tally, never touching
BottomHeightTopAuto, OffsetsDefineSize, OverConstrainedRtl or PercentageOffsetsAndSizes
— the same four categories the committed 74/108 candidate leaves at 0. The loop's own stop
condition (two consecutive batches with no per-category improvement,
orchestrate::improved_any_category) ended it after rig-batch 3, short of the six-batch cap;
admit::beats_committed never fired, so crates/layout/src/generated/absolute.rs and its
provenance file were untouched throughout. Prompt/output token totals across all three batches:
626,240 prompt, 501,235 output tokens, 973.1 s wall-clock — see "Free volume versus a paid
batch" below for what that bought against the paid batches' own tally.
The five flash-batch N rows are one unattended run of the same binary at the rig's second
free alias, qwen3.8-flash-next (NEV_MODEL=qwen3.8-flash-next NEV_CANDIDATES=32 NEV_CONCURRENCY=32, specs/rig-batch-absolute.md as amended). This alias is served by
one TabbyAPI replica per Spark across four Sparks, each saturating at eight requests in
flight, so a 32-candidate batch draws in one wave instead of serialising through an 8-wide
pipe the way the rig-batch N rows above did. Five batches, 160 draws, 5,345.3 s (89 m 5 s)
total; the loop stopped itself after batch 5 on two consecutive batches with no per-category
improvement, short of the six-batch cap.
The vm-batch 1 row is one batch of the same binary at qwen3.8-27b-fast-2
(GENERA_MODEL=qwen3.8-27b-fast-2 GENERA_CANDIDATES=32 GENERA_CONCURRENCY=16), the same
model and serving recipe as qwen3.8-27b-fast on four more cards of the same class, one
worker per card, four requests each (docs/infrastructure.md → "Where each piece
runs"). It started from the same state as flash-batch 1 (the 6-case seed regression
set, the same 22,959-byte prompt) and was run to measure the new alias against the
recorded ones, so it was stopped after batch 1 rather than left to loop. Its compile rate
matches the older alias (2 of 32, against 10 of 96 in the three rig-batch rows) and is
far below qwen3.8-flash-next's (14 of 32 from the same starting state); its best, 74/108
with MarginAutoCentering 11/11, is above every rig-batch candidate (63/108) and equal
to flash-batch 1's, short of the committed 82/108, so nothing was committed. Draws were
fast: all 32 replies back within 106.5 s of their own start, against 584–600 s for the
slowest flash-next reply, but 12 of 32 ran into the 8,000-token output cap.
The vm-batch 2 row reruns that batch after the prompt fixes in "Why candidates fail to
compile" below (layout source files shown with layout:: paths, three import lines, a 12,000-token
cap): GENERA_MODELS=qwen3.8-27b-fast-2:32:16 bash bench/absolute-mixed-batch.sh on a
build lane, the same 32 candidates at 16 in flight, from the same reseeded 6-case regression
set (a 22,944-byte prompt). 8 of 32 compiled, four times vm-batch 1's 2, and none
panicked while scoring. Two candidates reached 82/108 with OverConstrainedRtl 8/8 and
MarginAutoCentering 11/11, tying the committed candidate, so nothing was committed; the
other six scored 74, 74, 74, 73, 63 and 51. No reply reached the new cap (longest 11,788
tokens, every one finish_reason: "stop") and the whole batch took 233.1 s. The 24
failures are model-side, the same kinds as mixed-batch 2's 27b-fast failures: 11
candidates have a type mismatch (E0308), 6 declare fn resolve without pub (E0603;
for one of them that is the only error), 5 read a name out of scope (E0425), 4 call a
method that does not exist (E0599), 3 assign twice to an immutable binding (E0384),
and one has a syntax error. None imports crate::width or leaves out Direction. This
matches the older alias after the same fixes (mixed-batch 1 and 2: 10/32 and 4/32
compiled, best 82/108 both times), as expected of the same model and recipe on the same
card class.
The dsv-batch 1 row is the first batch from a different model family:
DeepSeek-V4.1-Flash (EXL3 2.9 bpw, two ARM nodes at tensor-parallel 2, alias
deepseek-v4.1-flash), GENERA_MODELS=deepseek-v4.1-flash:8:2 bash bench/absolute-mixed-batch.sh, 8 candidates at 2 in flight (the pair runs two sequences
at once), same reseeded 6-case regression set. 4 of 8 compiled and none panicked; three
scored 74/108 and model_7 82/108 with OverConstrainedRtl 8/8, tying the committed
candidate, so nothing was committed. It is slow: replies took 143–512 s each at about 26
output tokens/s per request, 1,731 s for the batch including the build, and the model's
built-in speculative decoding accepted almost no drafted tokens on this prompt (mean
acceptance length 1.02). The request's enable_thinking: false is honoured (the
template reads it, and a smoke request with it reported 0 reasoning tokens), but with thinking off this
model still reasons at length in prose inside content, writing and revising code
blocks as it goes: six replies did that, and five of them hit the 12,000-token cap
mid-thought. extract_rust_source still found a compiling block in three of those five.
The two that answered with one code block and nothing else (3,049 and 3,909 tokens) both
failed to compile, one on a missing mut, one on possibly-uninitialized bindings. Its
failures differ from the Qwen models': no path or import faults and no private fn resolve; the other two failures are replies cut off at the cap, whose extracted block
was a fragment. At 2 in flight a
32-candidate batch would take about two hours.
The dsv-batch 2 row is that 32-candidate batch: GENERA_MODELS=deepseek-v4.1-flash:32:2 bash bench/absolute-mixed-batch.sh, same alias, same 2 in flight, same reseeded 6-case
regression set and the same 22,944-byte prompt. It took 6,572.4 s (1 h 50 m), about 205 s
per candidate against dsv-batch 1's 216 s, so the two-hour estimate held. 16 of 32
compiled, the same half as the 8-candidate batch; one of them (model_6) panicked while
scoring on an Option::unwrap() of None. Of the 15 scored, twelve reached 74/108 and
three (model_19, model_21, model_23) 82/108 with OverConstrainedRtl 8/8 and
MarginAutoCentering 11/11, tying the committed candidate, so nothing was committed; no
candidate touched BottomHeightTopAuto, OffsetsDefineSize or
PercentageOffsetsAndSizes. 19 of 32 replies hit the 12,000-token cap (5 of 8 in
dsv-batch 1), for 306,315 output tokens, about 9,570 per request, the same as the smaller
batch's 9,514. The cap decides most of the outcome. Of the 19 capped replies, 6 still
yielded a compiling block and 13 did not; 11 of those 13 extracted a fragment (a later
partial rewrite without the import lines or the helpers defined in an earlier block, or
code cut off mid-token), and the other two were complete but wrong (a type annotation
needed, an immutable binding assigned twice). Of the 13 replies that stopped on their own,
10 compiled, and all three 82/108 candidates are among them (2,177, 8,028 and 5,984
tokens). The three that stopped on their own and failed are ordinary model errors: a
missing type annotation, a private fn resolve with a non-exhaustive match, and result
tuples left as the value of an if let block with no else, where return was needed. Four times the
draws bought three 82/108 candidates instead of one, and no new category.
Three things separate qwen3.8-flash-next from qwen3.8-27b-fast. Its compile rate is about five
times higher — 75 of 160 candidates compiled (14, 12, 10, 20, 19 per batch) against 10 of 96
at the old alias. Its ceiling is higher: batch 3's model_31 scored 82/108 and was committed
over the paid claude-opus-5 candidate's 74/108 through admit::beats_committed, the first
time any batch at either free alias beat the committed score. And the whole gain is one
category — OverConstrainedRtl, 0/8 to 8/8 — with BottomHeightTopAuto 0/6,
OffsetsDefineSize 0/12 and PercentageOffsetsAndSizes 0/8 untouched in every admitted
candidate across all five batches. Batches 4 and 5 re-derived the same 82/108 tally
independently, which is what stopped the loop. The regression set grew from 6 to 74 cases
after batch 1 and to 82 after batch 3, where it stayed. Every one of the 160 draws returned
real content (0 null-content replies), as at the old alias.
Two harness gaps this run surfaced, both fixed in the same track rather than routed around.
absolute_runner::evaluate scored every candidate inside one loop, so a candidate that
panicked partway through the corpus — an unwrap on an input shape the compile probe cannot
catch, since a panicking candidate compiles cleanly — unwound out of the batch and took every
other candidate's score with it; it now catches and drops just that candidate, the way a
compile failure drops one, and seven candidates across the five batches were dropped that way.
And bin/absolute_real_model_run.rs committed a winner's source verbatim, while the prompt
shows the function's types as layout::absolute::… — correct for a candidate crate that has
layout as a dependency, and unresolvable inside crates/layout itself, where the committed
copy lands. Those paths are now rewritten to crate:: on the way in, the form
generated/float.rs, generated/width.rs and generated/stacking.rs already carry.
Unlike the vLLM-backed qwen3.8-27b-fast alias, qwen3.8-flash-next returned no usage
block on any of the 160 replies, so HttpModelClient reported 0 prompt and 0 completion
tokens for every candidate and the token columns above cannot be filled for these five rows.
Prompt sizes are known from the client side (22,959 bytes for batch 1, 20,063 for batches 2-3,
19,642 for batches 4-5) and nothing about the run depended on the counts, since this alias
costs nothing per token; the gap is in what the serving stack reports, not in the loop.
The three loop N rows are one unattended run of the outer batch loop below
(NEV_PROVIDER=claude-code absolute_real_model_run, NEV_CLAUDE_CLI_MODEL unset so
the CLI's own default applied — a claude -p --output-format json probe on the same
machine reports that default as claude-sonnet-5; the provenance record only names the
provider, claude-code:default). The loop stopped itself after loop 3: two consecutive
batches with no per-category improvement. Every admitted candidate tied the committed
model_0 at 74/108 with the identical per-category tally, so admit::beats_committed
left crates/layout/src/generated/absolute.rs untouched; the run's regression set grew
from 6 to 74 cases. 735 s for all three batches; 24 candidates drawn, 23 compiled, 22
admitted.
Batch 2 reproduced batch 1's exact 74/108 score and per-category tally byte-for-byte
across a differently-sampled prompt (different failing-case sample, same function), so the
per-category tally stopped improving across these two consecutive batches — the spec's
stated stopping condition — independent of what follows. Batch 3 was attempted anyway to
confirm the plateau and instead hit an external block: every one of its 8 requests to the
Anthropic API failed with invalid_request_error: Your credit balance is too low to access the Anthropic API. This is a refusal by the API account, not a model or harness failure —
recorded here rather than routed around (no other provider/key was substituted). model_0
(from batch 1, re-derived identically in batch 2; provenance in
corpus/candidates/absolute-model-run/provenance/model_0.provenance.json) is admitted as
the final candidate: two consecutive non-improving batches already satisfied the spec's
stop condition, so the billing block did not change the outcome, only foreclosed a third
confirmation batch. Remaining gaps at 74/108: BottomHeightTopAuto, OffsetsDefineSize,
OverConstrainedRtl, PercentageOffsetsAndSizes — all 0%.
Floats (specs/floats-bfc.md) #
float_ground_truth_cache wrote 109 cases to corpus/float-ground-truth/ (13 enumerated
plus 96 seeded-random draws over crates/harness/src/float_cases.rs's eight categories; 0
dropped for engine disagreement). float_generation_loop seeds the permanent regression
set from the union of every hand-written fixture's passing cases -- 97 -- not the
baseline's alone: the baseline (41/109) ignores clear, so a fixture that places floats
correctly but ignores clear too (ignores_clearance, 82/109) strictly improves on it
and a no-loss gate seeded from the baseline admitted it on the first run. No corpus case
can separate two resolvers that treat clear identically, so the loop changed, not the
fixture (bin/float_generation_loop.rs, the comment at the seeding site). Against the
union-seeded set both seeded-wrong fixtures are rejected: ignores_clearance 82/109
losing 15, pins_every_float_left 66/109 losing 31. Each row below is one
float_real_model_run batch (NEV_PROVIDER=claude-code, NEV_CLAUDE_CLI_MODEL
unset so the CLI's default applied, 8 candidates drawn per batch); the CLI reports no token
counts, so wall-clock stands in.
| batch | compiled/drawn | admitted candidate | best score | per-category tally (best) | wall-clock |
|---|---|---|---|---|---|
| 1 | 8/8 | model_0 (all 8 tied at 109/109) |
109/109 | BfcSiblingBesideFloat 24/24, ClearedBlock 27/27, OppositeSides 14/14, SameSideWrap 10/10, ShrinkToFitWidth 8/8, SingleLeftFloat 7/7, SingleRightFloat 7/7, TallerThanContainer 12/12 | 290 s |
The loop stopped after batch 1 on its own: no failing case remained to sample a second
prompt from. model_0 is committed as crates/layout/src/generated/float.rs with
float.provenance.json beside it (the one import line rewritten from layout::float to
crate::float, as generated/absolute.rs and generated/width.rs were). No hand-written
escalation: this is the first generated function admitted with no hand-written correct candidate
anywhere in the tree, and the first at 100% of its corpus on the first batch.
Stacking order (specs/stacking-order.md) #
The first generated function whose output is an ordering rather than a rectangle set, and the second
built model-first: stacking_ground_truth_cache wrote 105 cases (9 enumerated plus 96
seeded-random) to corpus/stacking-ground-truth/, with 105/105 probe points agreed
exactly across all three engines (document.elementsFromPoint, docs/dump-format.md →
"Stacking probe") and 0 dropped. stacking_generation_loop seeded the permanent
regression set with the 64 cases the tree-order baseline already passes and confirmed
the gate rejects both seeded-wrong fixtures (global_z_index_sort: 26/105, 47
regressions; negative_z_above_in_flow: 85/105, 9 regressions — the two corpus
categories added for exactly this, crates/harness/src/stacking_cases.rs). One
stacking_real_model_run batch through NEV_PROVIDER=claude-code (the CLI's own
default model, NEV_CLAUDE_CLI_MODEL unset), 8 candidates; the CLI reports no token
counts, so wall-clock stands in.
| batch | compiled/drawn | admitted candidate | best score | per-category tally (best) | draw wall-clock per candidate | batch wall-clock |
|---|---|---|---|---|---|---|
1 (claude-code) |
8/8 | model_0 (all 8 tied at 105/105) |
105/105 | NegativeZIndex 11/11, NegativeZIndexBeforeInFlow 9/9, NestedContextsThreeDeep 16/16, OpacityCreatesContext 12/12, OverlappingSiblingsTreeOrder 17/17, PositionedOverStatic 9/9, PositiveZIndexOrdering 9/9, ZIndexIgnoredOnStatic 13/13, ZIndexTrappedInChildContext 9/9 | 159–202 s each, drawn concurrently | 321 s |
The loop stopped itself after batch 1 with no failing case left to sample: every one of
the 8 candidates scored the full 105/105 with zero regressions, and model_0 (first
among ties, 141 lines, a direct Appendix E walk: collect a context's members without
descending into child contexts, then paint background, negative contexts, in-flow
descendants, level-0 members, positive contexts) is committed at
crates/layout/src/generated/stacking.rs with stacking.provenance.json beside it;
layout::paint_order() defaults to it. No hand-written candidate was needed, so no
escalation is recorded. The prompt was 22,807 bytes: the spec text (CSS 2.1 §9.9.1,
Appendix E.2 and CSS Color 3's opacity rule), the function's source, a contract paragraph, and
the four failing cases (one per failing category) each shown with the classes every
engine listed under its probe point, front to back — the one departure from the geometry
functions' prompts, which show a case's document but not its ground truth.
Block height (specs/block-height-equation.md) #
The first generated function extracted from working hand-written code rather than built model-first,
and so the first whose committed candidate going into its model run is already correct.
CSS 2.1 §10.6.3's auto-height rule, its §10.6.7 float extension and the §10.5/§10.6.4
definite-height test had never been pulled out of crates/layout/src/fragment.rs; they sat
inline in layout_box and, a second time, in layout_absolute_box. Stage 1 extracted them
into crates/layout/src/height.rs (HeightInputs/HeightOutputs/HeightResolver/
resolve_baseline, plus definite_content_height) with no behavior change:
cargo test --workspace passed and fragment_check --random held at 227/227 cases, 25/25
fixed-box, 0 shape mismatches, with a per-category tally identical to the pre-change run.
height_ground_truth_cache wrote 105 cases to corpus/block-height-ground-truth/ (9
enumerated plus 96 seeded-random draws over crates/harness/src/block_height_cases.rs's
nine categories; 0 dropped for engine disagreement). height_generation_loop then seeded
the permanent regression set from the Stage-1 baseline's own passing cases — which, unlike
every other function, is the whole corpus, since that baseline is the correct extracted rule
rather than a deliberately-wrong placeholder. Both seeded-wrong fixtures were rejected
against real ground truth: ignores_float_extension 94/105 with 11 regressions
(AutoHeightFloatExtension 0/11, every other category perfect) and
pins_height_to_lowest_float 0/105 with 105 regressions. That loop is also the only one
that can run the positive half of the discrimination proof — it asserts the correct
baseline comes back admitted — and it is what records the incumbent's score at
crates/layout/src/generated/height.provenance.json, without which
admit::beats_committed would have had nothing to protect.
| batch | compiled/drawn | admitted candidate | best score | per-category tally (best) | wall-clock |
|---|---|---|---|---|---|
1 (NEV_PROVIDER=claude-code, claude-code:default) |
8/8 | model_0 (all 8 tied at 105/105) |
105/105 | AutoHeightFloatExtension 11/11, AutoHeightFloatShorterThanChildren 14/14, AutoHeightSingleChild 8/8, AutoHeightStackedChildren 14/14, BorderBoxExplicitHeight 9/9, ExplicitHeight 15/15, NestedAutoHeights 9/9, PercentageHeightDefiniteParent 9/9, PercentageHeightIndefiniteParent 16/16 | 112.4 s |
Every one of the 8 candidates compiled and every one reached a perfect 105/105 with zero
regressions — but none scored strictly higher than the committed Stage-1 extraction's own
105/105, so admit::beats_committed left crates/layout/src/generated/height.rs
unchanged. That is this spec's stated expected outcome, not a failure: losing to a
correct-by-construction baseline is what a tie against a perfect incumbent looks like. The
prompt was 22,418 bytes; the CLI reports no token counts, so wall-clock stands in.
Two things this function surfaced that are worth carrying forward.
The run needed a loop shape no other function does. Every other *_real_model_run samples its
prompt from the cases the committed candidate still fails, and exits when none remain. Here
none ever fail, so the loop would have drawn nothing and answered nothing. bin/ height_real_model_run.rs now draws one challenge batch in that situation, sampling one
case per category over the whole corpus — the question this spec asks is whether a model
beats a correct incumbent, and that needs a batch to actually run. Exactly one batch,
because no category can improve past a perfect tally and the regression set cannot grow.
And the corpus re-taught the relative-positioning function's lesson a second time, from a
different direction. The first height_generation_loop run scored the correct baseline at
93/105, losing AutoHeightSingleChild 3/8 and NestedAutoHeights 2/9 — both categories
give a child a bottom margin, and whether CSS 2.1 §10.6.3 measures to that margin edge at
all depends on whether §8.3.1 collapsed it into the parent's. height_runner::candidate_dump
was holding margin collapsing at the no-collapsing baseline, which made those cases
unwinnable by any height candidate whatsoever. Running margin collapsing, percentage
resolution, floats, absolute positioning and relative positioning through their landed
candidates took the baseline to 105/105. What makes this a sharper data point than the
relative-positioning case: there, the gap showed up as categories no model candidate could
win, which reads like a model limitation; here it showed up as a known-correct candidate
scoring less than perfect, which reads only one way. A generated function whose baseline is correct by
construction is its own harness test, and that is an argument for extracting working code
into a generated function before generating against it rather than after.
Free volume versus a paid batch #
specs/rig-batch-absolute.md's question: does drawing four times as many candidates from
free rig hardware, aimed only at the four categories every eight-candidate batch left at
0%, buy anything a small paid batch did not. Three sides answer it, not two: the paid
eight-candidate batches, the free 32-candidate batches at qwen3.8-27b-fast, and the free
32-candidate batches at qwen3.8-flash-next. Every figure below names its alias, because
the two free aliases differ more from each other than either differs from the paid side.
Paid side (eight candidates per batch, claude-opus-5 and claude-code's
claude-sonnet-5): best per-category tally 74/108 —
BottomHeightTopAuto 0/6, LeftWidthRightAuto 8/8, MarginAutoCentering 11/11, NegativeOffset
13/13, OffsetsDefineSize 0/12, OverConstrained 12/12, OverConstrainedRtl 0/8,
PercentageOffsetsAndSizes 0/8, RelativeAncestorPadding 11/11, RightWidthLeftAuto 9/9,
StaticPosition 6/6, TopHeightBottomAuto 4/4 — reached on the first metered batch and tied
by every batch since, metered and subscription-billed alike. Metered cost for that first
batch: 80,184 prompt + 102,955 output tokens (batch 1, "Batch scores and token totals"
above); the metered batches' own wall-clock was not recorded, but the same 8-candidate
shape run three times over the subscription-billed CLI (identical 74/108 tally every
time) took 735 s total, about 245 s per batch.
First free side (32 candidates per batch, qwen3.8-27b-fast through the rig's gateway at
NEV_CONCURRENCY=8, three batches, "Batch scores and token totals" above): best
per-category tally 63/108 — the same twelve categories, but 11/11 short of the paid side on
MarginAutoCentering (0/11 against the free side's own compiled candidates never
re-deriving the auto-margin centering fix the paid model_0 happened to include) and
otherwise identical zero scores in the four target categories. 626,240 prompt + 501,235
output tokens across the three batches, 973.1 s wall-clock total (about 324 s per batch) —
more tokens and more wall-clock than the single paid batch that reached 74/108, for a worse
score. 10 of 96 candidates compiled.
Second free side (32 candidates per batch, qwen3.8-flash-next through the same gateway at
NEV_CONCURRENCY=32, five batches, "Batch scores and token totals" above): best
per-category tally 82/108 — BottomHeightTopAuto 0/6, LeftWidthRightAuto 8/8,
MarginAutoCentering 11/11, NegativeOffset 13/13, OffsetsDefineSize 0/12, OverConstrained
12/12, OverConstrainedRtl 8/8, PercentageOffsetsAndSizes 0/8, RelativeAncestorPadding
11/11, RightWidthLeftAuto 9/9, StaticPosition 6/6, TopHeightBottomAuto 4/4 — eight cases
above the paid side, the whole difference being OverConstrainedRtl. 75 of 160 candidates
compiled. 5,345.3 s wall-clock across five batches (about 1,069 s per batch), roughly four
times the wall-clock the three qwen3.8-27b-fast batches took and about twenty times the
paid side's per-batch 245 s. Token cost is unknown rather than zero: this alias returns no
usage block, so nothing was counted, and nothing was billed either.
Free volume solved one of the four target categories, and only at the second alias.
OverConstrainedRtl went from 0/8 to 8/8, lifting the committed candidate from the paid
batches' 74/108 to 82/108 — the first time admit::beats_committed fired for this function
since the original claude-opus-5 batch. BottomHeightTopAuto, OffsetsDefineSize and
PercentageOffsetsAndSizes stayed at 0/N in every admitted candidate across all eight free
32-candidate batches at both aliases, as in every paid eight-candidate batch before them.
So volume alone did not do it: before the prompt fixes in "Why candidates fail to
compile", the same 32-candidate shape at qwen3.8-27b-fast bought nothing at all, and
what changed at qwen3.8-flash-next was the model behind the alias — a five-fold higher
compile rate and a higher ceiling — bought with about four times the wall-clock and no
token cost either way. After those fixes qwen3.8-27b-fast also reaches 82/108 and passes
OverConstrainedRtl where no qwen3.8-flash-next candidate in the same batch did ("Mixed
batches" below), so its 63/108 here measures the old prompt, not the model's ceiling.
Why candidates fail to compile #
Before the three diagnosis/mixed-batch rows above, 86 of 96 qwen3.8-27b-fast
candidates and 85 of 160 qwen3.8-flash-next ones had failed to compile with nothing
recorded about why: the run kept only the extracted source, not the reply, the finish
reason or the compiler's output. Each candidate now carries its model, the reply's
finish_reason and the raw reply (generation::model::Candidate), each batch saves the
replies under replies/, and each batch keeps one compiler error file per candidate
under compile-errors/ (written by the probe step itself, candidate_crate::probe). The diagnosis batch (16 candidates from each model, 8000-token cap)
classified its 21 failures like this:
| cause | 27b-fast (15 failed) | flash-next (6 failed) | fixed on this side? |
|---|---|---|---|
imports crate::width::Direction, copied from the layout source file the prompt shows verbatim; no candidate crate can resolve crate::width |
10 (7 with nothing else wrong) | 0 | yes: layout source files are shown with layout:: paths (prompt.rs layout_seam_as_a_candidate_sees_it) |
reply cut off at the 8000-token cap (finish_reason: "length") mid-function; not looping, just long |
2 | 2 | yes: cap raised to 12000, request timeout to 1200 s (model.rs MAX_COMPLETION_TOKENS, REQUEST_TIMEOUT) |
wrong extraction by extract_rust_source |
0 | 0 | nothing to fix: every reply was essentially all code, 0 null replies, no reasoning tags |
genuinely wrong Rust (type mismatch, variable out of scope, unbalanced brace, a ?: ternary, a bad pattern) |
3, plus 3 of the path failures | 4 | no |
Replies are long because the models reason in comments: 12–34 KB of source per candidate.
Four flash-next candidates compiled and then panicked while scoring (three unwrap on a
None, one explicit panic on a non-Length value), dropped the way a compile failure is.
After those two fixes the first mixed batch surfaced a third prompt fault: the absolute
contract told candidates to write "exactly these two import lines", which leave out
Direction, the type of AbsoluteInputs::direction; 4 of 22 failed 27b-fast candidates
named Direction without importing it. The contract now lists three lines. In
mixed-batch 1 no reply hit the new 12000-token cap (nine 27b-fast replies ran past 8000
tokens and finished); in mixed-batch 2 one did. What remains is model-side:
mixed-batch 2's 28 27b-fast failures are mostly type mismatches (E0308), a local read
before assignment or out of scope (E0381, E0425), a second assignment to an immutable
binding (E0384) and float literals of ambiguous type (E0689) -- and 6 declared
fn resolve without pub (E0603), which the contract could still forbid outright.
Mixed batches: qwen3.8-flash-next and qwen3.8-27b-fast together #
NEV_MODELS=qwen3.8-flash-next:32:32,qwen3.8-27b-fast:32:32 draws one batch from both
aliases at once (generation::model::parse_model_list; NEV_MAX_BATCHES=1, one batch
per run, regression set reseeded from the baseline so the prompt samples the same failing
cases the recorded batches did). The run prints each model's compile count, best score and
the categories its best candidate passes that no other model's best does
(orchestrate::categories_only_one_model_passes).
| batch | model | compiled/drawn | best | categories only this model's best passes |
|---|---|---|---|---|
| mixed-batch 1 | qwen3.8-27b-fast |
10/32 | 82/108 | 1: OverConstrainedRtl |
| mixed-batch 1 | qwen3.8-flash-next |
13/19 received (of 32 requested) | 74/108 | 0 |
| mixed-batch 2 | qwen3.8-27b-fast |
4/32 | 82/108 | 1: OverConstrainedRtl |
| mixed-batch 2 | qwen3.8-flash-next |
25/32 | 74/108 | 0 |
With the prompt faults fixed, qwen3.8-27b-fast stopped being dead weight: in both
batches its best candidate reached 82/108, tying the committed flash-next candidate and
the best any single-model batch at either alias ever reached, and it was the only model in
the batch to solve OverConstrainedRtl -- flash-next's best in the same batches stayed at
74/108. The two batches' 27b-fast compile rates (10/32, then 4/32) bracket the recorded
2–4/32 on the high side and vary a lot between draws; flash-next's 25/32 in mixed-batch 2
is above every recorded flash-only batch (10–20/32). Neither batch beat 82/108, so
crates/layout/src/generated/absolute.rs is unchanged; BottomHeightTopAuto,
OffsetsDefineSize and PercentageOffsetsAndSizes stay at 0 in every candidate.
Drawing both at 32 in flight did not slow flash-next down: a mixed batch took 738–850 s, inside the recorded flash-only 966–1,158 s, since 27b-fast's replies (47–256 s each) land long before flash-next's (430–760 s). In mixed-batch 1, 13 flash-next requests still in flight at about 670 s lost their connection to the gateway ("peer disconnected") and every retry was refused over the next 7 s; those 13 were dropped, which is why that row's flash-next count is out of 19. A later attempt at mixed-batch 2 was stopped by a SIGTERM on the lane about 7 minutes in, with no record of what sent it; it is not a row.
How the list behaves: each entry is model[:count[:concurrency[:timeout_secs]]], and
NEV_MODELS takes precedence over NEV_MODEL. A malformed list is an error, not a
fallback to one model, because a mixed batch silently drawn from one model would measure
the wrong thing. Candidate names stay model_<n>, numbered across the whole batch, and
the model is carried on Candidate::model, so names stay valid module names whatever the
model is called. Each model's requests are seeded from 0, as in a batch of its own.
absolute_real_model_run scores each model's share on its own thread as soon as that
model's last reply lands (score_share, then conclude_batch picks the winner across
shares) and records the winner's own model in provenance, and overflow_real_model_run
does the same through the shared orchestrate::print_per_model_summary and
orchestrate::count_per_model; the other *_real_model_run binaries accept the list but
record the joined label (a+b) and score after the last reply. Per-share scoring has run
live on the overflow function ("Overflow: a mixed batch" below). On the absolute function it
has not: both batches above ran before it existed (bench/absolute-mixed-batch.sh).
Open prompt fault: 6 of the 28 failed qwen3.8-27b-fast candidates in mixed-batch 2
declared fn resolve without pub (E0603); prompt::ABSOLUTE_SEAM_CONTRACT could
require pub fn resolve in so many words. prompt::build_relative_positioning has the
same "two import lines" wording that left out Direction on the absolute function; that function
is at 104/104, so it is unchanged.
The outer batch loop: specs/automated-repair-iteration.md #
The three batches above were driven by hand: a person reran absolute_real_model_run,
read the two per-category printouts, and judged the plateau by eye. bin/ absolute_real_model_run.rs::main() is now a loop instead of a single shot: each
iteration reloads the regression set, resamples the failing cases, rebuilds the prompt,
draws a batch, evaluates, and — on admission — extends the regression set and (only if
admit::beats_committed says so) overwrites crates/layout/src/generated/absolute.rs
before the next iteration reads the shrunken failing set. It stops itself after two
consecutive batches with no per-category improvement
(orchestrate::improved_any_category) or after six batches, whichever comes first,
reproducing what the three hand-run batches above did by eye.
The loop first ran end to end under NEV_PROVIDER=claude-code after a fresh
absolute_ground_truth_cache (108 cases, 0 dropped — the same count as the hand-run
batches) and absolute_generation_loop (6 cases seeded, both seeded-wrong fixtures
rejected at 57/108 and 0/108 as before): the loop 1–loop 3 rows above. It terminated
on its own stop condition (two consecutive batches with no per-category improvement)
after three batches and 735 s, admitted a 74/108 candidate in every batch, and never
overwrote the committed candidate because none scored strictly higher — the same plateau
the hand-run batches reached, reproduced without a person between batches. Its second
end-to-end run, against qwen3.8-flash-next at 32 candidates per batch (the
flash-batch 1–flash-batch 5 rows above), terminated on the same condition after five
batches and 5,345.3 s, and did overwrite the committed candidate, at 82/108.
cargo test --workspace also covers improved_any_category directly with hand-built
tallies (crates/generation/src/orchestrate.rs's #[cfg(test)] mod tests).
The run exposed two defects in the CLI provider, both fixed in
crates/generation/src/model.rs ClaudeCliClient::cli_command with unit tests beside
the existing model.rs tests: the subprocess inherited ANTHROPIC_API_KEY from the
sourced .env and so authenticated against the metered account (16/16 calls refused
with "Credit balance is too low" before the fix); and, run from the worktree, the CLI
loaded the repository's and the user's CLAUDE.md and session hooks and answered as a
coding agent instead of returning source (0/8 compiled before the fix). See the
subscription-billed section below for what the subprocess now looks like.
Admission and escalation status #
model_31(qwen3.8-flash-next, flash-batch 3) is the committed candidate: 82/108 on the absolute-ground-truth corpus, regression-free, committed ascrates/layout/src/generated/absolute.rswithabsolute.provenance.jsonbeside it. It replacedmodel_0(claude-opus-5, batch 1) at 74/108 throughadmit::beats_committed, the only time this function's committed score has moved since that first batch. Not 100% of every category — three categories (BottomHeightTopAuto,OffsetsDefineSize,PercentageOffsetsAndSizes) remain at 0/N — but the loop's stopping condition (the per-category tally stops improving across two consecutive batches) was met at flash-batch 5, with flash-batches 4 and 5 independently re-deriving the same 82/108 tally.model_0(claude-opus-5, batch 1) held the committed slot before that at 74/108, with four categories at 0/N. Its own stopping condition was met at batch 2, and a third confirmation batch was blocked by the Anthropic account running out of credit (see "Batch scores and token totals").- The generation loop's discrimination proof ran and passed: both seeded-wrong fixtures
(
treats_right_as_left.rs: 0/108,ignores_auto_margin_centering.rs: 57/108) were rejected by the gate against real ground truth. - No hand-written escalation was needed: a model candidate was admitted on the first
batch, so
handoff.md's hard rule (hand-write only after every model batch fails) was never reached. - The outer batch loop (
specs/automated-repair-iteration.md, "The outer batch loop" above) has run end to end once, unattended, underNEV_PROVIDER=claude-code: three batches, stopped by its own two-consecutive-non-improving-batches condition (the six-batch cap was not reached), one candidate admitted per batch, every one at 74/108 with the identical per-category tally — BottomHeightTopAuto 0/6, LeftWidthRightAuto 8/8, MarginAutoCentering 11/11, NegativeOffset 13/13, OffsetsDefineSize 0/12, OverConstrained 12/12, OverConstrainedRtl 0/8, PercentageOffsetsAndSizes 0/8, RelativeAncestorPadding 11/11, RightWidthLeftAuto 9/9, StaticPosition 6/6, TopHeightBottomAuto 4/4. The then-committedmodel_0was unchanged by that run: the loop's committed-score guard (admit::beats_committed) replaces it only with something that scores strictly higher on the same corpus, and nothing in those three batches did. - The same loop has since run end to end a second time, unattended, against the rig's
qwen3.8-flash-nextalias at 32 candidates per batch: five batches, stopped by the same condition, and this time the guard did fire — flash-batch 3'smodel_31at 82/108 is whatcrates/layout/src/generated/absolute.rsnow holds.
A subscription-billed alternative: NEV_PROVIDER=claude-code #
Batch 3's outright refusal above (account credit exhausted mid-run) is the kind of
failure the metered Anthropic Messages API path can hit at any point, since it bills
per token regardless of how the request was made. crates/generation/src/model.rs
now also has ClaudeCliCandidateSource, which draws each candidate from claude -p
run headless as a subprocess instead of an HTTP request to /v1/messages — billed
under the interactive subscription the spec-watch tick and its writers already run on,
not metered per-token. Selected the same way as the existing Anthropic path, with
NEV_PROVIDER=claude-code instead of NEV_PROVIDER=anthropic;
NEV_CLAUDE_CLI_MODEL optionally overrides the CLI's own default model, the same
role NEV_CLAUDE_MODEL plays for AnthropicClient. Telemetry.prompt_tokens/
completion_tokens have no CLI equivalent and are always 0 — there is nothing to
total in a table shaped like "Batch scores and token totals" above, only wall-clock.
The subprocess (ClaudeCliClient::cli_command) runs claude -p <prompt> --output-format text --tools "" --strict-mcp-config --setting-sources "" --disable-slash-commands --system-prompt <a plain code-generator prompt>, from an empty
directory under the system temp dir, with ANTHROPIC_API_KEY and ANTHROPIC_AUTH_TOKEN
removed from its environment. Each of those is load-bearing: the sourced .env carries
the metered API key and the CLI prefers a key over its subscription login, and without
the isolation flags the CLI loads the repository's and user's CLAUDE.md and hooks and
answers as a coding agent rather than as a completion. --bare would isolate in one
flag but authenticates only with ANTHROPIC_API_KEY, so it cannot be used here.
Observed over one three-batch run (the loop N rows above): all 24 responses the CLI
returned were extracted by the same extract_rust_source path as the API providers —
every one began with a fenced Rust block, and every extracted source starts at its
first use line; 23 of 24 compiled, the one failure a genuine Rust error in the
candidate (a non-exhaustive match), not an extraction artifact. Wall-clock per
candidate ranged 119–241 s (mean about 187 s) with 8 subprocesses in flight at once;
a batch of 8 including its compile and evaluation took 242–251 s. The CLI's default
model resolved to claude-sonnet-5.
Escalation: one larger-model batch on a geometry diff #
The outer batch loop no longer ends the moment its plateau condition fires. The first
time consecutive_no_improvement reaches STOP_AFTER_CONSECUTIVE_NO_IMPROVEMENT
(bin/absolute_real_model_run.rs), the loop draws one more batch — from the
subscription-billed CLI provider above, whatever the sampling tier was — and only stops
for real if that batch also improves no category. A second trigger stops the run without
escalating again, so escalation is spent once per run rather than once per plateau
(orchestrate::stall_action). The escalated batch is probed, compiled, scored, admitted
and committed through exactly the same path as an ordinary batch; only the prompt and
the provider differ.
Two things make it a real escalation rather than a redraw:
- A genuinely larger model. The rig serves
qwen3.8-flash-nexton all four Sparks, for sampling and capacity alike, and GLM is stopped and disabled there, so escalating on the rig would redraw from the model that just failed. The CLI provider's default resolves toclaude-sonnet-5and costs nothing beyond the subscription already paying for writer sessions. The rig side of escalation — a second, larger model actually deployed — remains unbuilt. - A geometry-diff prompt instead of the failing documents.
prompt::build_absolute_positioning_geometry_diffkeeps the spec text and the function contract but replaces each failing case's full HTML with the measured per-node error the previous attempt still has: node index and tag, which of width/height/offset_x/ offset_y is outside tolerance, and the signed delta in pixels. Nodes that already agree are left out. That error is computed byharness::compareon every batch and used to be discarded inside the runner crate; it now rides out inrunner::CandidateReport::failing_case_diffs, taken from the batch's highest-scoring candidate.
No numbers here yet: this has unit tests (cargo test --workspace) but has not run
live, so there is no wall-clock, no compile rate and no evidence that escalation
recovers cases a plateaued sampling-tier run could not. That measurement is separate
work, standing to this the way repair-loop-live-run stands to
automated-repair-iteration.
Relative positioning: a second data point #
specs/relative-positioning.md builds a second generated function the same model-first way as
absolute positioning above, specifically so the cost of building its scaffolding gets a
second measurement next to it (track/spec-relative-positioning's own handoff.md). The
function
(crates/layout/src/relpos.rs — RelposInputs/RelposOutputs/RelposResolver/
resolve_baseline) isolates CSS 2.1 9.4.3's shift behind one plain-data function over an
eight-category corpus (crates/harness/src/relpos_cases.rs: pixel offsets, percentage
offsets, auto on one side, no offsets at all, LTR and RTL over-constraint, nested
relative boxes, a relative box containing an absolute one).
relpos_ground_truth_cache wrote 104 cases to corpus/relpos-ground-truth/ (0 dropped
for engine disagreement). relpos_generation_loop seeded the permanent regression set
with the 8 NoOffsets cases resolve_baseline already passes (the only category where a
permanently-zero shift is correct) and confirmed the gate rejects both seeded-wrong
fixtures: treats_right_as_left 8/104 and ignores_direction 89/104 (fails only
OverConstrainedRtl, 0/15 — it always lets left win instead of reading
inputs.direction), both admitted=false.
| batch | compiled/drawn | admitted candidate | best score | per-category tally (best) | wall-clock |
|---|---|---|---|---|---|
1 (NEV_PROVIDER=claude-code, claude-code:default) |
8/8 | model_0 |
104/104 | AutoOnOneSide 11/11, NestedRelativeBoxes 15/15, NoOffsets 8/8, OverConstrainedLtr 19/19, OverConstrainedRtl 15/15, PercentageOffsets 13/13, PixelOffsets 11/11, RelativeContainingAbsolute 12/12 | 41.8 s |
relpos_real_model_run stopped after this one batch: "no failing cases remain; nothing
left to repair". All 8 drawn candidates tied at a perfect 104/104 (the CLI does not
report token counts, matching the claude-code provider note above). Committed at
crates/layout/src/generated/relpos.rs, provenance relpos.provenance.json; layout()
and layout_with_resolver() now default to it. No remaining gaps.
Unlike the width/margin/percentage/absolute functions, this corpus is the first to depend on
two other generated functions' own real (non-baseline) behavior at scoring time — a percentage
offset (PercentageOffsets) and a nested absolute descendant
(RelativeContainingAbsolute) — and that dependency surfaced three real bugs before any
batch could be scored honestly, each fixed in this same track rather than routed around
(recorded in track/spec-relative-positioning's own handoff.md → "Decisions"):
relpos_runner::candidate_dump held percentage resolution and absolute positioning at
their own bootstrap baselines despite this function's corpus needing both resolved for real,
making those two categories unwinnable by any candidate regardless of quality;
fragment.rs's over-constraint rule read the shifted box's own direction rather than
its containing block's, the property CSS 2.1 9.4.3 actually branches on; and
relpos_ground_truth_cache never cleared its output directory before a fresh run, so a
case whose recipe-selection index shifted (adding the NoOffsets category re-numbers
every later random draw) left its old id's stale dump on disk forever, silently mixed
into a category's score under a new one's name. None of the three would have been caught
by cargo test --workspace or fragment_check alone — only running the pipeline
end to end against live ground truth surfaced them, which is itself a data point for
"the cost of building each generated function's scaffolding is the unmeasured dominant
term" above: this second function's
scaffolding cost included debugging its own evaluation harness, not just writing it.
Overflow: the same prompt against two providers #
specs/overflow-scrollbars.md builds the overflow function
(crates/layout/src/overflow.rs — OverflowInputs/OverflowOutputs/
OverflowResolver/resolve_baseline) over a nine-category corpus
(crates/harness/src/overflow_cases.rs). The function decides one boolean per axis — does a
box scroll — so generation::overflow_runner::evaluate scores scrollbar presence against
the browsers' unanimous verdict and ignores geometry, which every candidate in a batch
produces identically.
overflow_ground_truth_cache wrote all 105 cases to corpus/overflow-ground-truth/
(9 enumerated plus 96 random at seed 0xBAD0FFEE), 0 dropped: Chromium, Firefox and
WebKit agree about every node's scrollbar in every case, so the spec's provision for
recording a three-way disagreement as a finding has nothing to record here.
overflow_generation_loop seeded the permanent regression set with the 20 cases
resolve_baseline already passes (AutoFitting 11 and ScrollFitting 9 — the
categories where nothing scrolls and a permanently-false answer is right) and confirmed
the gate rejects both seeded-wrong fixtures: decides_both_axes_together 68/105 and
only_clipping_boxes_overflow 82/105.
This is the first generated function measured against two providers on a byte-identical prompt — both
runs report prompt_hash 51e0b754, so the two scores differ only by the model
answering. The gateway alias served 32 candidates per batch
(NEV_MODEL=qwen3.8-flash-next NEV_CANDIDATES=32 NEV_CONCURRENCY=32,
NEV_PROVIDER unset):
| batch | compiled/drawn | admitted candidate | best score | per-category tally (best) | slowest reply |
|---|---|---|---|---|---|
| 1 | 32/32 | model_4 |
70/105 | AutoFitting 11/11, AutoOverflowing 17/17, HiddenOverflowing 0/10, NestedScrollContainers 0/10, OneAxisOnly 14/14, RootViewportOverflow 8/8, ScrollFitting 9/9, ScrollOverflowing 11/11, VisibleOverflowing 0/15 | 434 s |
| 2 | 30/32 | model_5 |
90/105 | AutoFitting 11/11, AutoOverflowing 17/17, HiddenOverflowing 10/10, NestedScrollContainers 10/10, OneAxisOnly 14/14, RootViewportOverflow 8/8, ScrollFitting 9/9, ScrollOverflowing 11/11, VisibleOverflowing 0/15 | 420 s |
| 3 | 27/32 | none (best model_25 85/105 regressed 20) |
85/105 | AutoFitting 11/11, AutoOverflowing 17/17, HiddenOverflowing 0/10, NestedScrollContainers 0/10, OneAxisOnly 14/14, RootViewportOverflow 8/8, ScrollFitting 9/9, ScrollOverflowing 11/11, VisibleOverflowing 15/15 | 481 s |
| 4 | 29/32 | none (best model_31 85/105 regressed 20) |
85/105 | identical to batch 3 | 304 s |
Four batches in 3194 s total, ended by the two-consecutive-batches-without-improvement
rule rather than the six-batch cap. Token columns are absent on purpose: the gateway
returns no usage object on a completion, so Telemetry::prompt_tokens/
completion_tokens are 0 for every one of the 128 replies
(model::HttpModelClient::complete takes them from the response's own usage field and
defaults them when it is missing). Only wall-clock is measurable here, the same gap the
claude-code provider has for a different reason.
The committed candidate is batch 2's model_5 at 90/105
(crates/layout/src/generated/overflow.rs, provenance overflow.provenance.json);
layout::scrollbars already defaulted to generated::overflow::resolve, so nothing in
fragment.rs had to change to put it in the engine's path.
The unsolved category is the interesting result. VisibleOverflowing stayed 0/15 in both
admitted batches, and batches 3 and 4 independently converged on an 85/105 candidate that
wins all 15 of those cases and loses all 20 clipping cases (HiddenOverflowing,
NestedScrollContainers) doing it. No candidate in 128 drawn held both at once, and the
regression gate correctly refused the trade. The two rules pull against each other: a box
reports overflow when its content spills past its padding box whatever its own
overflow value, visible included, while a box that clips keeps its own spill out of
its ancestors' extents. A candidate that reads overflow as "only non-visible boxes
report" wins the clipping cases; one that ignores overflow entirely wins the visible
ones; the gateway model wrote one or the other and never both.
NEV_PROVIDER=claude-code on the same prompt reached 105/105 in one batch, with all
8 candidates tied there — every category solved, VisibleOverflowing included. That
candidate is parked, unmerged, on branch side/overflow-claude-candidate with
"model": "claude-code:default" in its provenance; the branch this function lands on carries
the gateway's 90/105. On this function the free alias is therefore 15 cases and four batches
short of the subscription-billed CLI's single batch, the opposite of the absolute function's
result above, where qwen3.8-flash-next beat claude-code 82/108 to 74/108. One function
does not generalize, but the pairing says the alias's advantage there was not a property
of the alias — it was a property of that function.
Overflow: a mixed batch #
specs/overflow-scrollbars.md Stage 2 reran the function with
NEV_MODELS=qwen3.8-flash-next:32:32,qwen3.8-27b-fast:32:32 (bench/overflow-mixed-run.sh
on a lane: ground truth rebuilt, 105 cases, 0 dropped; regression set reseeded from the
baseline's 20 cases; the same prompt_hash 51e0b754 as every recorded overflow batch).
The run stopped after one batch: its winner passed every case, so "no failing cases
remain" ended the loop before the two-flat-batches rule or the six-batch cap could.
| batch | model | compiled/drawn | best (admitted) | categories only this model's best passes | prompt tokens | output tokens | replies |
|---|---|---|---|---|---|---|---|
| overflow mixed-batch 1 | qwen3.8-27b-fast |
25/32 | model_55 105/105 — committed |
2: HiddenOverflowing, NestedScrollContainers |
152,288 | 28,216 | 10.8–171.8 s |
| overflow mixed-batch 1 | qwen3.8-flash-next |
31/32 | model_25 85/105 |
0 | not reported | not reported | 39.4–189.4 s |
The whole batch, drawing and scoring both shares, took 344.1 s — about 2,850 s less than
the four flash-only batches that produced the previous 90/105. model_55 solves all nine
categories, VisibleOverflowing included, which no candidate in the 128 flash-only draws
did while also keeping the clipping cases. flash-next's best is the same 85/105 trade
recorded above (VisibleOverflowing 15/15, HiddenOverflowing and
NestedScrollContainers 0/10); with only the baseline's 20 cases in the regression set it
is admitted here rather than rejected. The rest of each share clusters on the same few
scores: flash-next 70 (3), 62 (2), 49 (1, rejected), 47 (17, rejected), 39 (7, rejected);
27b-fast 70 (2), 62 (1), 49 (1, rejected), 47 (4, rejected), 39 (15, rejected), 20 (1). Only
one 27b-fast candidate in 32 reached 105, so the result is a single draw, not a rate.
Compile failures, 8 of 64: flash-next 1 (model_2, assigning twice to an immutable
binding); 27b-fast 7 — 4 matching the Overflow enum as if it were a struct (E0574),
2 replies that were prose rather than code (one of them cut at the 12,000-token limit, the
batch's only stop_reason=length), and 1 with no resolve the scoring crate could name.
Unlike the flash-only overflow batches, the gateway reported a usage object for every
27b-fast reply here; flash-next's are still all zero.
crates/layout/src/generated/overflow.rs is now model_55 with "model": "qwen3.8-27b-fast" in overflow.provenance.json, replacing flash-next's 90/105 on the
free pool alone — the claude-code 105/105 on side/overflow-claude-candidate is no longer
needed for this function. fragment_check --random stays 227/227. held_out_check --grammar
moves from 42 to 61 of 164 compared cases passed (36 discarded as untrustworthy both
times), and the failures that differ only in scrollbar presence drop from 23 to 4.
Transforms: the whole corpus on the first batch #
The transform function (specs/transforms2d.md, crates/layout/src/transform.rs) is the
first drawn from a spec other than CSS 2.1, and the first whose answer is a matrix rather
than a length, a box or an ordering.
transform_ground_truth_cache wrote 110 cases to corpus/transform-ground-truth/ (14
enumerated plus 96 seeded-random draws over crates/harness/src/transform_cases.rs's
fourteen categories), and dropped 0 for engine disagreement — including every
arbitrary-angle rotation and every skew. That was the open question going in: rotations and
skews land off every engine's pixel grid, Firefox rounds rects onto its own 1/60 px grid
and Chromium and WebKit onto 1/64, and specs/transforms2d.md allowed for keeping only the
angles the engines agree on. None had to be dropped. Per-category kept counts:
BetweenUntransformedSiblings 10, Matrix 9, NestedTransform 8, OrderRotateThenTranslate 6,
OrderTranslateThenRotate 9, OriginKeyword 4, OriginLength 6, OriginPercentage 13,
RotateArbitrary 8, RotateRightAngle 8, Scale 8, Skew 7, TranslatePercentage 6,
TranslatePx 8.
transform_generation_loop seeds the permanent regression set from the union of every
hand-written fixture's passing cases — 102 — not the baseline's alone, the same change
float_generation_loop already carries and for the same reason. The first run seeded from
the baseline and admitted a seeded-wrong fixture: the baseline returns the identity, so the
only three cases it passes are the rotate(180deg) ones, whose bounding box is the
untransformed one, and composes_right_to_left passes all three plus 84 more, leaving a
no-loss gate nothing to catch it on at 87/110. Two changes fixed it, neither to a fixture:
the two order recipes pin transform-origin to 0 0, so those categories turn on
composition order alone rather than also on the origin rule, and the seeding takes the
union. Against the union-seeded set every fixture is rejected — baseline 3/110 losing 99,
composes_right_to_left 87/110 losing the 15 order cases, ignores_transform_origin
41/110 losing 61.
One batch, four gateway models at once (NEV_MODELS as specs/transforms2d.md gives it;
NEV_PROVIDER unset):
| batch | compiled/drawn | admitted candidate | best score | per-category tally (best) | prompt tokens | output tokens |
|---|---|---|---|---|---|---|
| 1 | 20/82: 27b-fast 8/32, 27b-fast-2 10/32, deepseek-v4.1-flash 2/2, flash-next 0/16 | model_22 (27b-fast; 12 of the 20 scored tied at 110/110) |
110/110 | every category full: BetweenUntransformedSiblings 10/10, Matrix 9/9, NestedTransform 8/8, OrderRotateThenTranslate 6/6, OrderTranslateThenRotate 9/9, OriginKeyword 4/4, OriginLength 6/6, OriginPercentage 13/13, RotateArbitrary 8/8, RotateRightAngle 8/8, Scale 8/8, Skew 7/7, TranslatePercentage 6/6, TranslatePx 8/8 | 349,478 (27b-fast 169,504; 27b-fast-2 169,504; deepseek 10,470; flash-next reported none) | 141,012; batch wall-clock 308 s, slowest single reply 206.6 s (deepseek) |
The loop stopped after batch 1 on its own: no failing case remained to sample a second prompt from. This is the second generated function admitted at 100% of its corpus on the first batch, after floats, and the first where the free gateway pool did it unaided — twelve independent candidates from three different models all reached 110/110.
qwen3.8-flash-next contributed nothing here: all 16 replies came back with
finish_reason=tool_calls and zero completion tokens, so the alias emitted no code to
extract. That is a rig-side behaviour of that alias, not a property of this function's
prompt, and it is the same alias that carried the absolute-positioning batches.
The candidates that failed to compile failed the way earlier functions' did ("Why
candidates fail to compile" above): redefining TransformInputs or TransformFunction
locally instead of importing them, and non-exhaustive matches over TransformFunction's
six variants.
One pipeline change came out of this function rather than out of its corpus.
admit::rewrite_layout_imports rewrote only a line-leading use layout::, which was
enough for every earlier generated function because each imports types and calls nothing.
crates/layout/src/transform.rs exports IDENTITY and compose beside its types, and the
first batch's winner called them through the qualified path the prompt shows it
(layout::transform::compose) — which compiles in the probe crate and does not compile
once the file sits inside the layout crate. The rewrite now covers a qualified path
anywhere in a line, and the batch was re-run so the committed file came through the fixed
path.
Inline atomic: a corpus two wrong fixtures cover exactly #
The inline-atomic function (specs/inline-atomic.md,
crates/layout/src/inline_atomic.rs) is the first whose answer is a set of positions
rather than one value: a placement for every atomic inline the container holds, the line
boxes they sit on, the stack's total height, and the last line's baseline.
inline_atomic_ground_truth_cache wrote 111 cases to
corpus/inline-atomic-ground-truth/ (15 enumerated plus 96 seeded-random draws over
crates/harness/src/inline_atomic_cases.rs's fifteen categories) and dropped 0 for engine
disagreement. That is the answer to the question the corpus was designed around. CSS 2.1
10.8 starts every line box with a strut carrying the container's font, so a line box's
height and baseline normally depend on font metrics the three engines need not share; the
corpus sets font-size: 0 with an explicit line-height in px on every container so the
strut's ascent and descent are zero and its baseline sits at the middle of a known height.
Whether Chromium, Firefox and WebKit would actually agree on that was untested until this
run. They do, on all fifteen categories, AlignMiddle and StrutTallerThanContent
included — the two that turn on the strut alone. Per-category kept counts: AlignBottom 9,
AlignLength 9, AlignMiddle 3, AlignPercentage 9, AlignTop 8, BaselineDifferentHeights 3,
EdgesOnAtomicInlines 10, LineBreak 12, NestedInlineBlocks 5, OneLineEqualHeights 8,
OverflowHiddenBaseline 7, StrutTallerThanContent 3, TextAlignCenter 8, TextAlignRight 12,
WiderThanContainer 5.
inline_atomic_generation_loop seeds the permanent regression set from the union of every
hand-written fixture's passing cases, the same change float_generation_loop and
transform_generation_loop already carry. Here the union came out complete — 111 of 111 —
because the two seeded-wrong fixtures are wrong about disjoint halves of the function and
right about everything else, so between them they pass every case:
| fixture | score | what it gets wrong | categories at 0 |
|---|---|---|---|
baseline |
0/111 | everything: one box per line at the left content edge, the vertical stacking the engine did before the function existed | all fifteen |
aligns_every_bottom_edge |
61/111, 50 regressions | every box's bottom margin edge on the line's baseline, vertical-align never read |
AlignBottom, AlignLength, AlignMiddle, AlignPercentage, AlignTop, NestedInlineBlocks, OverflowHiddenBaseline |
never_breaks_a_line |
86/111, 25 regressions | one line box however far past the content width it runs | LineBreak, TextAlignCenter, WiderThanContainer |
Both are rejected, and the shape of the rejection is what makes the corpus trustworthy: each fixture loses exactly the categories the other passes, so neither can be admitted by being right about the easy half. A complete regression set also means the gate is as strict as it can be — a candidate must pass all 111 to be admitted at all, not merely avoid losing ground.
The complete seed cost one change to the batch loop. Every other *_real_model_run samples
its prompt from the cases outside the regression set, which for every other function is the
same thing as the cases the committed candidate still fails, because the set was seeded
from that candidate's own passes. Here the two differ completely: the set covers the corpus
from the first batch while the committed candidate passes none of it, so the first run of
inline_atomic_real_model_run sampled an empty difference, drew nothing and exited in 0.0
seconds. It now samples over the whole corpus when the difference is empty — the same dead
end bin/height_real_model_run.rs reached from the opposite direction, where the committed
candidate passed everything, and the same answer. The ordinary stop condition still applies
here, unlike there: a batch against a committed candidate at 0/111 can genuinely improve on
the one before it.
One batch, four gateway models at once (NEV_MODELS as specs/inline-atomic.md gives it;
NEV_PROVIDER unset):
| batch | compiled/drawn | admitted candidate | best score | per-category tally (best) | prompt tokens | output tokens |
|---|---|---|---|---|---|---|
| 1 | 18/82: flash-next 6/16, 27b-fast 4/32, 27b-fast-2 7/32, deepseek-v4.1-flash 1/2 | model_0 (flash-next; all four models produced a 111/111 candidate) |
111/111 | every category full: AlignBottom 9/9, AlignLength 9/9, AlignMiddle 3/3, AlignPercentage 9/9, AlignTop 8/8, BaselineDifferentHeights 3/3, EdgesOnAtomicInlines 10/10, LineBreak 12/12, NestedInlineBlocks 5/5, OneLineEqualHeights 8/8, OverflowHiddenBaseline 7/7, StrutTallerThanContent 3/3, TextAlignCenter 8/8, TextAlignRight 12/12, WiderThanContainer 5/5 | 675,794 (27b-fast 328,032; 27b-fast-2 328,032; deepseek 19,730; flash-next reported none) | 372,707; slowest single reply 489.4 s (flash-next) |
Four candidates reached 111/111 and every one of them came from a different model —
model_0 from qwen3.8-flash-next, model_40 from qwen3.8-27b-fast, model_51 from
qwen3.8-27b-fast-2, model_81 from deepseek-v4.1-flash. model_62 missed by two
cases at 109/111, and the rest of the compiled field collapsed: 93/111 once, then 20, 15,
3, 3, 3, 3, 0, 0. That gap is the shape of the function — line boxes are either built or
they are not, and a candidate that misreads the strut or the baseline loses whole
categories at once rather than drifting by a pixel.
qwen3.8-flash-next reported zero prompt and zero completion tokens on all 16 replies
while returning real source (the winner's own reply was 24,957 bytes), so its share is
missing from the token column. That is the same alias, and the same missing usage block,
recorded for the absolute-positioning batches; it behaved differently here from the
transform batch, where it returned no content at all.
Two candidates panicked partway through the corpus and were caught and dropped rather than
taking the batch with them — model_42 and model_9, both indexing their own output.
That failure mode is more available here than in any earlier function: a candidate returns
one vector of placements and another of line boxes, and getting their lengths out of step
with each other compiles cleanly and panics on the first case that breaks a line. The
isolation crate::transform_runner::evaluate introduced is what kept the other sixteen
candidates' scores.
Fixed table layout: the scaffolding measured on its own #
The fixed-table-layout function (specs/tables-fixed-layout.md,
crates/layout/src/table.rs) is the first whose structural half was entirely new. Every
generated function before it had most of its substrate already standing: a box tree that
already produced the right boxes, a walk in fragment.rs that already reached them, a
display value the engine already understood. None of that existed for tables. So this
section is the data point roadmap.md → "Phase 1 — Engine bootstrap" asks for, on the term
it calls the dominant unmeasured one — what the hand-written scaffolding under a generated
function costs, measured on its own rather than inferred.
Per-phase wall-clock #
Times are file modification timestamps, not a stopwatch, and not commit timestamps: one commit holds the whole change, so its own timestamp says nothing about the phases inside it. A phase's end is the last modification of the file that phase wrote, which means a file touched again later has its boundary pushed to that later touch — two rows below say where that happened, and neither is a phase whose own duration can be read off the table as a result. The start, 00:09:03, is the worktree's own checkout.
| Phase | Ends | Wall-clock | Model tokens | Notes |
|---|---|---|---|---|
Substrate: Table/TableRow/TableCell on dom::computed::Display, plus table_layout() and border_spacing() |
00:16:59 | ~8 min | none | Additions only; three new accessor tests, and the one existing test that asserted display: table reads as Other had to change. |
The function surface and its two deliberately wrong baselines (crates/layout/src/table.rs) |
00:22:25 | ~5 min | none | Two functions, not one, split on the axis §17.5.2.1 splits the algorithm on. |
Anonymous table objects in boxtree.rs, and the starting candidate under generated/table.rs |
00:26:47 | ~4 min | none | §17.2.1's two in-scope rules, generated only inside a fixed-layout table. Seven new box-tree tests. |
The table walk in fragment.rs, and two more resolver arguments on the three layout* entry points |
00:35:50 | ~9 min | none | The largest hand-written piece: the walk itself, the anonymous-box splice, the used-width override, and eleven tests over hand-computed geometry. |
The runner (crates/generation/src/table_runner.rs) |
00:40:32 | ~5 min | none | Panic isolation and a short-answer guard, both carried over from the inline-atomic runner. |
Generation-crate wiring: the Seam variant, prompt::build_table_fixed_layout, admit::record_for_table_fixed_layout, the provenance spec-text module |
00:44:01 | ~3 min | none | Plus the same two-line addition to each of the eight other runners that call layout_with_resolvers. |
| The baseline and two seeded-wrong fixtures | 00:45:54 | ~2 min | none | |
The three binaries (table_ground_truth_cache, table_generation_loop, table_real_model_run) |
00:48:51 | ~3 min | none | None of the three has been run: the build lane was unreachable for the whole session. |
The corpus (crates/harness/src/table_cases.rs, seventeen categories) |
00:50:18 | not separately bounded | none | Written between the table walk and the runner, but touched again at 00:50:18 to add the check that no recipe in cases.rs writes a table display value, so its own boundary is not readable here. |
| Workspace verification, and the provenance record's own guard test | 00:53:20 | ~3 min | none | cargo test --workspace green through bench/verify.sh; admit.rs touched last for the test that keeps the committed record's hashes honest. |
Hand-written total, first substrate edit to the last verification command: 00:09:03 to
00:53:20, about 44 minutes, against a model-run total of zero — no batch has been
drawn for this function at all, so there is no row for it in "Batch scores and token totals"
and no token count to put beside the 44 minutes. The build lane was unreachable for the
whole session (bench/on-lane.sh --lane-only exits 3), and a model batch fails rather than
fall back to a local run, so the three binaries were written and compiled but never
executed. The comparison this section exists to make is therefore still one-sided: what it
establishes is the cost of the scaffolding, not yet the ratio.
That 44 minutes is the whole hand-written half of a generated function whose substrate did
not exist — a display value the engine had never modelled, anonymous boxes the box tree's
own doc comment used to say were out of scope, and a second formatting context in
fragment.rs. For comparison, the absolute-positioning function's own scaffolding, the
first measured this way, took about 34 minutes over a substrate that already gave every
absolutely positioned element a box.
What the structural half actually cost, by kind of work #
Worth separating, because the phases above mix two things a later function will pay very
differently for. Roughly 26 of the 44 minutes went on the four phases that are genuinely
about tables: the accessors, the function surface, the anonymous boxes and the table walk.
The other ~18 went on wiring that every generated function pays again — a runner that is the
previous runner with two type names changed, a Seam variant, a prompt contract, a
provenance module, three binaries that differ from the inline-atomic three by a corpus path
and a section number, and one two-line addition to each of eight other runners because the
layout* entry points gained two arguments. The second figure is the one that should be
falling and is not.