How layout output is checked against real browsers #

The engine's layout is judged by one question: for a given page, does every box come out the same size and in the same place as in real browsers? This page explains how that is answered, starting from nothing. The exact wire format is in docs/dump-format.md.

The idea in one picture #

flowchart LR
  page["test page<br/>(HTML + CSS)"] --> chr["Chromium"]
  page --> ff["Firefox"]
  page --> wk["WebKit"]
  page --> bez["this engine"]
  chr --> vote["vote per<br/>measurement"]
  ff --> vote
  wk --> vote
  vote --> cmp{"engine within<br/>1/60 px of the vote?"}
  bez --> cmp
  cmp --> result["pass / fail<br/>+ per-box error"]

The browser side runs once per page and is cached; only the engine side runs on every candidate.

Step by step #

1. Dump. Each browser loads the page through Playwright (oracle/oracle.mjs) at a fixed 800×600 viewport, device pixel ratio 1, and walks every element in document order. For each it records the element's box rectangles (getClientRects()), a subset of computed style, and whether its content overflows. That list is a dump. The engine produces the same dump from its own fragment tree (crates/harness/src/dump.rs).

2. Compare shape first. Both dumps must list the same elements in the same tree shape (tag, depth, parent). If they do not, the difference is in parsing, not layout, and the case stops there.

3. Compare sizes and relative offsets. For each box: its width and height, and its position relative to its parent. Relative offsets mean one misplaced box shows up as one wrong number instead of shifting every box below it, so the first wrong box names the rule that is wrong.

4. Vote. With three browsers, each measurement is classified (crates/harness/src/compare.rs vote()):

flowchart TD
  m["one measurement,<br/>three browser values"] --> q1{"all three agree<br/>within 1/60 px?"}
  q1 -- yes --> u["unanimous:<br/>that value is the reference"]
  q1 -- no --> q2{"two agree,<br/>one differs?"}
  q2 -- yes --> o["2–1: the pair's value is the<br/>reference, the third is named outlier"]
  q2 -- no --> s["split: no reference,<br/>measurement skipped"]

Two browsers could only report "they disagree" and throw the case away; three can keep it and name who is off.

5. Discard what cannot be trusted. A whole case is dropped rather than scored when the browsers disagree about whether some box has a scrollbar, because a scrollbar takes a platform-dependent width from everything inside it (compare::scrollbar_dispute, docs/dump-format.md → "Scrollbar probe"). Unanimous agreement that a scrollbar is there does not discard the case.

6. Score. A case passes when every trustworthy measurement of the engine is within 1/60 px of the reference. 1/60 px is the coarsest grid any of the browsers rounds to (Firefox's); Chromium and WebKit use 1/64 px. The raw difference is kept too, as a finer-grained signal than pass/fail.

A worked example #

Twelve nested boxes, each width: 83.333% of its parent, starting from 800 px (the nesting-depth-12 recipe). At the innermost box:

source	width at depth 12
Chromium	89.671875 px
WebKit	89.671875 px
Firefox	89.66667175292969 px (its own 1/60 px grid)
vote	unanimous — all three within 1/60 px; reference 89.671875 px
engine, exact arithmetic	89.72098558887357 px — off by 0.049 px, fail
engine, rounding to nearest 1/64 px at each level	89.71875 px — fail
engine, truncating to 1/64 px at each level	89.671875 px — exact, pass

The browsers' shared answer told the engine exactly how they round: truncate toward zero on a 1/64 px grid at every step (crates/dom/src/px64.rs, tasks/device-pixel-rounding.md). On other boxes of the same shape Firefox's coarser grid drifts up to 0.0375 px from the other two, past the tolerance; there the vote is 2–1 and Firefox is named the outlier. That drift turned out to be a real web-compatibility bug (docs/firefox-app-units.md).

Where the test pages come from #

flowchart LR
  recipes["recipe corpus<br/>227 cases"] --> gate["development signal<br/>and landing floor"]
  seamc["per-function corpora<br/>104–109 cases each"] --> gate
  grammar["grammar corpus<br/>200 cases"] --> held["held-out exam"]
  wpt["WPT normal-flow pages<br/>7 usable"] --> held
  fwpt["a feature's own WPT pages<br/>aspect-ratio: 40 cases"] -. "four pages in five: 33" .-> gen["cases for generating<br/>that feature"]
  fwpt -. "one page in five: 8" .-> held

The dotted arrows are built and measured, and no feature job reads those cases yet ("Cases from a feature's existing WPT pages" below).

• Recipe corpus (crates/harness/src/cases.rs). About thirty recipes, each a small page generator for one branch of the spec's case analysis (auto margins on one side, a negative margin collapsing with a positive one, a percentage against an auto-height parent...). Each recipe is a category, so scores are reported per category. 27 fixed cases plus 200 seeded-random draws from the same recipes (seed 0xC0FFEE) make 227. Values are deliberately fractional (33.333%, 1/64 px steps), because round numbers hide rounding bugs.
• Per-function corpora (absolute_cases.rs, float_cases.rs, relpos_cases.rs, stacking_cases.rs, block_height_cases.rs, overflow_cases.rs). The same recipe idea, one set per generated function, kept separate so adding cases does not reshuffle everyone else's random draw.
• Grammar corpus (crates/harness/src/grammar.rs). Pages composed freely from the same properties, with no recipe steering them, so it contains shapes no recipe thought of. Never used to generate code — only to check it.
• WPT normal-flow pages. The css/CSS2/normal-flow pages that the engine can read (no text, no external stylesheet; a no-break space counts as text): 7 of 1056 (oracle/suites/normal-flow-pages.mjs). An exam only: scored the same way, and no prompt or case set for generation holds one.
• A feature's own WPT pages. The pages of the WPT folder written for one feature, selected and measured by code. Four in five are cases for generating that feature; the fifth is held out and only ever scored. The next section is the rule, the two commands that make the cases and the command that builds a prompt from them.
• SMT-sampled cases (crates/harness/src/smt.rs). A handful of margin-collapse cases whose values a solver chose to hit rule boundaries.

Cases from a feature's existing WPT pages #

A feature's WPT folder is a source of cases for generating that feature, not only an exam for it. Two commands turn the folder into ground-truth cases, a third builds the model prompt from the cases for generation, and none of the three runs a model:

flowchart LR
  folder["WPT folder of one feature<br/>pinned checkout"] --> select["feature-pages.mjs<br/>no browser, no model"]
  select --> pages["corpus/NAME-pages/<br/>rewritten pages + manifest.json"]
  pages --> train["pages_ground_truth_cache<br/>--list train"]
  pages --> heldout["pages_ground_truth_cache<br/>--list held-out"]
  train --> gt["corpus/NAME-wpt-ground-truth/"]
  heldout --> hgt["corpus/NAME-wpt-held-out-ground-truth/"]
  gt --> build["model_prompt<br/>no browser, no model"]
  text["specification_text folder"] --> build
  types["types file"] --> build
  build --> out["prompt.txt, expected.txt,<br/>sources.json"]

The rule, and why:

• The pages that already exist are used before any case is written. In aspect-ratio job run 034aWbHrNquwSP4TyfOSr5 a model session took about 40 minutes to write a 993-line case generator, while the feature's WPT folder held 289 pages; selecting and measuring those pages takes under ten seconds and gives 40 cases.
• A page supplies its markup and nothing else. The expected result of a case is what the three browsers measure on the page as it was rewritten, by the same vote as every other corpus. A WPT assertion or reference page is never read, because it says whether one test passes, not where every box is, and because the page measured is no longer the page the assertion was written for once its text is removed.
• One page in five is held out, so each feature keeps an exam of pages that no prompt and no case set for generation has held. A page is held out when the first byte of the SHA-256 of its file name is divisible by 5 (feature-pages.mjs isHeldOut()), so the choice is the same on every run and does not move when the deny list changes. The two lists are measured into two folders, so nothing that reads the first can see a held-out page.
• The pages are what the suite's authors wrote. They are not stratified by the specification's case analysis and not chosen for fractional values; the recipe corpora above are what covers those.

No feature job reads the two case folders or runs the third command yet, and aspect-ratio has no job definition: the one that named a writer-built case generator and corpus/aspect-ratio-ground-truth/ is removed (tag archive/job-aspect-ratio holds its last run's commits).

Selecting pages. No browser and no model:

node oracle/suites/fetch.mjs wpt
node oracle/suites/feature-pages.mjs --feature oracle/suites/features/aspect-ratio.json \
  --out corpus/aspect-ratio-pages

The first command lands the pinned WPT checkout; the feature's folder must be listed in oracle/suites/suites.json wpt.sparse. The feature file (oracle/suites/features/<name>.json) holds source, the folder's path inside WPT; deny, regular expressions for what the feature's job definition puts out of scope (for aspect-ratio: flex, grid, absolute and fixed positioning, floats, writing-mode, every inline and table display, multi-column, contain-intrinsic-*, the intrinsic sizing keywords min-content, max-content and fit-content, which the engine has no sizing for, and scripts); and exclude, optional, page name to reason, for a page out of scope for a reason its markup does not state (for aspect-ratio: the two pages where the automatic minimum width grows a ratio-derived width to a child's, which needs a min-content width the engine does not compute). Each page goes through five steps, and a page is recorded under the first step that rejects it:

1. A page with no doctype is rejected (no-doctype): browsers lay it out in quirks mode, and the engine always lays out in no-quirks mode (crates/dom/src/parse.rs set_quirks_mode()).
2. Every <p> holding the test's pass instruction ("Test passes if...") is removed, because the engine lays out no text and step 5 rejects a page with any. The browsers measure the page after this rewrite, so the expected result is true of the page the engine is given.
3. A page named in exclude is rejected with its reason (excluded:<reason>).
4. A page matching a deny pattern is rejected (denied:<pattern>).
5. The rules of oracle/suites/normal-flow-pages.mjs classify() apply: no text, no external stylesheet, no image, no script in the body, no element other than html, body, div, p and span, and no span that is not shown to be display: block.

Reference pages (-ref, -notref) and the support/, resources/ and reference/ folders are skipped, not rejected. The output folder holds the accepted pages, flat (a page in a subfolder takes its path with / replaced by _), and manifest.json: source, wpt_commit, the train and held_out lists, and rejected (page name to reason). The folder is removed and rebuilt on every run, and refused when it exists and holds no manifest.json, so a mistyped path deletes nothing else.

Measuring pages. The three browsers, no model; once per list:

cargo run --release -p generation --bin pages_ground_truth_cache -- \
  --pages corpus/aspect-ratio-pages --list train --out corpus/aspect-ratio-wpt-ground-truth
cargo run --release -p generation --bin pages_ground_truth_cache -- \
  --pages corpus/aspect-ratio-pages --list held-out \
  --out corpus/aspect-ratio-wpt-held-out-ground-truth

The flags are read in that order. Each page of the list is dumped in every browser and written as one case in the format every *_ground_truth_cache binary writes (crates/generation/src/groundtruth.rs CachedCase), so load_all() and prompt_parts.rs render_case() read the folder unchanged. A case's id is the page's file name (block-aspect-ratio-001.html), its seed is 0 and its one category is wpt. groundtruth.rs drop_reason() decides whether a measured page is kept; the text before a reason's first : names the rule:

rule	a page is dropped when
too-few-engines	fewer than three browsers produced a dump, so there is no vote
shape-mismatch	the browsers list different elements, so there is no vote
split	on some measurement no two browsers agree
scrollbar	the browsers disagree whether some box's content overflows it (compare::scrollbar_dispute)

A measurement two browsers agree on and the third does not is kept: the pair's value is the expected one. A dropped page is still written in full, as a case file under <out>/dropped/, with every reason in <out>/dropped/reasons.json, so the numbers behind a drop can be read; load_all() skips subfolders and so reads the kept cases only. The output folder is removed and rebuilt on every run, and refused when it exists and holds no dropped/reasons.json.

The scrollbar rule drops a page on any such disagreement, including one about a box that cannot scroll: scrollbar_dispute compares content width with box width on every element whatever its overflow value. Seven aspect-ratio pages were dropped that way before the intrinsic sizing keywords were denied, and one (block-aspect-ratio-038.html) still is, each because Firefox alone reports content wider than a box whose overflow-x is visible in all three browsers. The rule is kept as it is because the cases left are enough to bring up the steps that will read them; counting only boxes that can scroll would keep that page.

Measured on aspect-ratio (css/css-sizing/aspect-ratio/ at the commit suites.json pins; Chromium 153, Firefox 155, WebKit 26.6; the commands above, on the development box):

command	in	out	wall time
feature-pages.mjs	289 pages scanned	42 selected (35 for generation, 7 held out), 247 rejected	0.08 s
pages_ground_truth_cache --list train	35 pages	33 cases, 2 dropped	about 4 s
pages_ground_truth_cache --list held-out	7 pages	7 cases, 0 dropped	about 3 s

So 289 pages give 40 measured cases, 33 for generation and 7 held out, in about 7 s of measuring, of which about 2 s per run is starting the browsers. The binary builds in 1 min 26 s from an empty target folder. The 247 rejections are mostly layout modes outside the job's scope (flex 57, grid 42, intrinsic sizing keywords 35, absolute or fixed positioning 26) and replaced elements (18 with an image). Before the intrinsic sizing keywords were denied, the same run selected 63 pages and measured 55 cases, 47 for generation and 8 held out. Seven held out of 42 is 17%, under one in five, which follows from these file names. Of the 2 dropped pages, one is a scrollbar page above (block-aspect-ratio-038.html) and one is a split (small-aspect-ratio-crash.html: the root's height is 8, 16 and 33554432 px in the three browsers). All 40 kept cases are unanimous: none is kept on a 2–1 vote.

Building the prompt. No browser, no model, no network; it reads three stored inputs:

cargo run -p generation --bin model_prompt -- \
  --specification crates/provenance/tests/fixtures/specification-text/expected-output \
  --types crates/generation/tests/fixtures/model-prompt/aspect_ratio_types.rs \
  --cases crates/generation/tests/fixtures/pages-ground-truth/train \
  --function resolve --resolver layout::aspect_ratio::AspectRatioResolver \
  --out <prompt folder>

Each case is shown as, for each box the engine calls the function for, the inputs it hands the function and the content box the browsers measure for that box, as the expected output (prompt_parts.rs render_function_case()), with the page before the calls unless --no-page-html is given, as the feature jobs give it (docs/autonomous-loop.md → "The model prompt and its check" has the measurement). The calls are recorded with the baseline answering, never the resolver the engine ships (layout::fragment::layout_recording_aspect_ratio_calls(), aspect_ratio_runner.rs record_calls()). A box whose height is not definite is asked twice, before and after its children; when the children's height depends on the first answer (a child's percentage height against it), the case shows both calls, the first with no expected output and the second with that height written as a rule in the first answer, measured by laying the page out with the first answer held at three values. The module doc of aspect_ratio_runner.rs says why ("A box asked twice"). For aspect-ratio the 33 train cases have 33 ratio boxes, every one with an agreed browser size, two of them shown with both calls, and the baseline already gives the expected output for 8 of them. A case was once also shown as its whole page with every node's box under it; with that form qwen3.8-27b-fast wrote no candidate that compiled (0 of 13, against 4 of 11 with the calls), so the command no longer builds it.

Those are the inputs stored for the tests; a feature's own run names its specification_text folder, its types file and its corpus/<name>-wpt-ground-truth folder. All six arguments are required, in any order, and none may be given twice; --no-page-html takes no value (crates/generation/src/bin/model_prompt.rs main()):

argument	what it names
--specification <dir>	A folder the specification_text command wrote (docs/provenance.md → "The text files of one function"): specification.txt, relevant.txt, the cited sections under references/, and the sources.json that lists them.
--types <file>	The function's types file. The prompt carries it byte for byte.
--cases <dir>	A folder of measured cases, such as one pages_ground_truth_cache wrote. Every kept case is shown, in case id order (groundtruth.rs load_all()); the cases under dropped/ are not read.
--function <name>	The name a candidate gives the function (resolve).
--resolver <crate>::<module>::<Type>	The full path of the function's type as a candidate crate names it: the module a candidate imports the types from, then the type's name (model_prompt.rs Function::new()). Only a function whose calls the engine can record is accepted (recorder_for(): aspect-ratio).
--out <dir>	The folder the three files below are written into; it is created when missing.

crates/generation/src/model_prompt.rs write() writes three files into --out, in this order:

file	content
prompt.txt	The prompt (build()): the specification text with its relevant part marked, every cited section, the types file, the contract and every case with its calls and their expected outputs. docs/autonomous-loop.md → "The model prompt and its check" describes each part.
expected.txt	Each call's expected output followed by a line --- (prompt_parts.rs expected_calls_file()).
sources.json	The --specification folder's own, byte for byte: which texts, by URL and SHA-256, the prompt carries.

No specification text is copied into --out: cargo run -p jobs --bin job-check -- prompt <prompt folder> <specification folder> reads prompt.txt and expected.txt from the first folder and the texts from the second, and fails when the prompt lacks one.

Every input is read and checked before the first file is written, so a refused run writes nothing. Refused: a case with no three-browser vote (the message names the case), a --cases folder that is missing or holds no case, a types file with no text, a specification folder that prompt_parts.rs SpecificationBundle::from_dir() refuses (a listed text missing or not hashing to its recorded SHA-256, among others), and a --function or --resolver that is not a name or not a path with a module. A file of another name already in --out is left there.

--cases is given the folder of the train list. The command reads whichever folder it is given and does not refuse a held-out one.

For the stored inputs (one cited section, one case) the prompt is 22,764 characters and the built binary writes the three files in under 0.01 s on the development box.

Tests. node --test oracle/suites/feature-pages.test.mjs runs the selector on real WPT pages stored under oracle/suites/fixtures/feature-pages/, one per outcome, and compares the manifest, the rewritten page and the printed summary; bench/verify.sh runs it. crates/generation/tests/pages_ground_truth.rs tests the keep-or-drop decision on stored cases with no browser, under cargo test. bench/pages-ground-truth-test.sh measures the fixture pages again in the real browsers and compares pages kept, pages dropped with their rule, and each kept page's majority boxes with the stored cases; it needs the three browsers and is run by name, not by the gate. crates/generation/tests/model_prompt_command.rs runs the model_prompt binary on the stored inputs under cargo test: the order of the prompt's parts, the three files written, two runs writing the same files, and each refusal. Its the_stored_prompt_the_prompt_check_is_tested_on_is_what_the_command_writes() fails when the binary's prompt.txt and expected.txt differ by a byte from jobs/fixtures/prompt/model-prompt/, the folder jobs/tests/prompt_check.rs runs the prompt check on; that folder's NOTICE holds the command that writes the two files again.

Floors #

A floor is a recorded score that a change may not go below; landing on main runs them (bench/land.sh):

• bench/fragment-floor.sh — the recipe corpus through layout() must pass 227 of 227.
• bench/verify.sh (bench/generated-scores.sh) — every generated function must pass at least as many cases of its own corpus as its <name>.floor.json holds, and the corpus must not shrink.

Floors are raised when more cases pass (by bench/generated-floors-update.sh after a landing), never lowered to let a change through.

Current numbers #

• Recipe corpus: 227/227, worst per-box error 0.0000 px.
• Held-out WPT normal-flow pages: 7/7. Grammar corpus: 61/164 scored, 36 discarded for scrollbar disagreement. The gap between 227/227 and 61/164 is the honest measure of how much the recipes still miss.
• Browser agreement: 699 of 705 browser pairs agreed across 235 documents; all six disagreements were the twelve-deep percentage shape above.

History of these numbers: roadmap.md → "Proof of concept".