This repository has no description
bez docs generation-loop.md
20 kB
Markdown
at main

The generation loop #

How one layout rule goes from spec text to committed engine code. The unit of work is a generated function: one rule pulled out of the hand-written engine into a small plain-data function (docs/generated-functions.md). Everything below happens once per generated function, and again whenever the spec or the browsers change.

flowchart TD
  spec["1. Spec section<br/>(e.g. CSS 2.1 §10.3.3)"] --> types["2. Types and corpus<br/>(hand-built)"]
  types --> truth["3. Ground truth<br/>3 browsers, cached once"]
  types --> prompt["4. Prompt"]
  prompt --> gen["5. Model pool writes<br/>N candidates"]
  gen --> score["6. Compile and score<br/>against the cache"]
  truth --> score
  score --> admit{"7. Admit?"}
  admit -- "no: failures go<br/>into the next prompt" --> prompt
  admit -- yes --> commit["8. Commit source<br/>+ provenance record"]
  commit --> land["9. Land on main<br/>behind score floors"]
  1. Spec section. One algorithm from one spec section is the unit, because a whole algorithm is the smallest thing a browser can check. The spec text travels with the generated function and is hashed, so a later spec change can find the code derived from it (docs/provenance.md).
  2. Types and corpus. A person (or agent) writes the function's input and output types, a baseline that reproduces the old behaviour, and a set of cases: existing WPT pages for the feature, or recipes (small HTML page generators, one per case the spec distinguishes). Candidates are scored against these browser-measured cases alone; no hand-written reference implementation and no deliberately wrong candidates take part. Neither is used, because on aspect ratio a hand-written reference passed the 161 cases its own author wrote but missed 13 of 47 measured WPT cases, a prompt carrying it drew a copy of it rather than a function derived from the spec, writing and checking it took about 2.4 of a 4.5-hour job, and no deliberately wrong fixture anticipated the min/max clamp-order bug the admitted function had.
  3. Ground truth. Every corpus page is rendered once in Chromium, Firefox and WebKit, and each page's box geometry is saved to a gitignored cache (crates/generation/src/groundtruth.rs). Nothing after this starts a browser. How the three answers become one reference is in docs/geometry-verification.md.
  4. Prompt. The spec text, the function's Rust types, and one failing case per failing category (crates/generation/src/prompt.rs, one builder per function). The model_prompt command builds a prompt with no builder written for the function, from three stored inputs: the whole specification text with every section it cites, the types file, and a folder of measured cases, each shown with its expected result (docs/autonomous-loop.md → "The model prompt and its check").
  5. Candidates. A model draws N independent versions of the function — 8 per batch from a paid API, 32 at once from the self-hosted model pool (crates/generation/src/model.rs, docs/rig-capacity.md). Many cheap tries beat one careful one, because the browsers say which tries are right.
  6. Compile and score. Each candidate is compiled into a small probe crate against the function's types and run over every cached case inside the real engine, in under a millisecond per case. A case passes when every measurement the function decides matches the browsers' vote within 1/60 px, with the inputs its neighbours decide taken from the browsers too ("Scoring a function: browser-measured inputs" below).
  7. Admit. Two gates, both required (next diagram). A rejected batch's failures seed the next prompt; a batch loop stops itself when per-category scores stop improving or after six batches, and may escalate once to a stronger model first (tasks/automated-repair-iteration.md, tasks/escalation-tier.md).
  8. Commit. The winner is written under crates/layout/src/generated/ with a provenance record (spec URL, text hash, model, prompt hash, seed, score). From then on it is ordinary engine source; no model runs at build or run time.
  9. Land. The track merges to main through bench/land.sh, which runs cargo test --workspace and the score of every generated function against its own floor (bench/verify.sh), and bench/fragment-floor.sh (the recipe corpus must stay at 227/227). A generated function's floor is its <name>.floor.json, which bench/after-land.sh raises after a landing and nothing lowers.

Scoring a function: browser-measured inputs #

A runner scores one generated function. Every input of that function that another rule decides (a neighbour's size or offsets, the width a box's containing block gives it, where an ancestor's box sits) is taken from the browsers' dumps of the same page, and the function is judged only on its own outputs against those dumps. The engine's other generated functions take no part in the score, so any function can be generated alone and in any order.

The reason: with the hand-written web rules removed, the neighbours' starting candidates are deliberately wrong (every child at the content origin, every box zero tall), and a page laid out through them measures nothing about the function under test. Before the rules were removed the neighbours were hand rules that were right on the corpus, which is the only reason a page laid out through them scored the function itself.

How: crates/layout/src/pins.rs holds a layout to measured geometry, and crates/generation/src/pins.rs builds it from a cached case (the vote's majority for each box's width, height and offset from its parent). A runner lays the page out inside pins::scoped and names what it leaves free (layout::pins::Free): the metrics of the boxes the function decides. Where layout would take a pinned box's width, height or offset from a function's answer it takes the measured value; a free metric comes out of the functions as they are, and the score is the comparison against the vote that every runner already makes. Pinned metrics match by construction, free ones are the function's own answer. A function whose answer is not a box metric (overflow: scrollbars) is scored on that answer with every metric pinned.

A function whose input has no browser-measured counterpart is not scored this way and no stand-in rule is written for it: its runner stays as it was, its floor file is a placeholder (zero passing cases) and says in blocked_on what it waits for. What is missing today, and for which function, is in each crates/layout/src/generated/<name>.floor.json. The common causes: a function that block flow or the formatting-context function reaches the engine through (float, margin_classify, margin_combine, inline_atomic, table) cannot be asked until a candidate for that function exists; the answers of block_flow that feed another function (height's content height from its children, absolute's static position) are not measured by a browser apart from the size or offset they produce.

Scorers for the structure functions. Each of the six structure functions added when the hand-written rules were removed is scored on a browser measurement of its own answer where one exists; none is scored by reading a rule's consequence out of the geometry, since that reading is itself a hand-written rule.

  • intrinsic_width: a measurement op in oracle/oracle.mjs, beside textProbe and stackingProbe, that sets width: min-content and then width: max-content on each element in turn and reads its border-box width, restoring the style between reads. The answers are cached beside the case's dumps. A box's items' own widths are pinned from the same measurement, so the function is scored alone.
  • grid: the resolved grid-template-columns and grid-template-rows (browsers serialize a grid container's used track sizes in px) join the dump's style subset. columns and rows are scored on those track sizes; item on item offsets with the tracks pinned. Item intrinsic widths come from the intrinsic_width measurement.
  • white_space: the text probe also records each character's Range.getClientRects(). A character with no rect, or only zero-width ones, was collapsed; the function's output text is scored against the characters every browser kept.

Shipped for the first three (intrinsic_width_runner, grid_runner, white_space_runner, with structure_scoring the shared case loop and structure_pages the page lists; caches corpus/intrinsic-width-ground-truth, grid-ground-truth, white-space-ground-truth from the *_ground_truth_cache binaries). Each has a floor file and no provenance record, since the function is still the starting candidate; generated_score_check scores an entry that has either file. Limits of what is measured:

  • Intrinsic widths are the forced min-content and max-content width of a box, not its contribution to its parent. Anonymous boxes have no element to measure. The pages hold no text items, percentages or declared widths on the measured boxes.

  • Grid track sizes are scored for containers with a definite width. rows and item need item placement, and no browser reports item areas, so they run through the engine's columns function and cannot pass until a columns candidate is admitted. The grid floor is a placeholder for that reason.

  • White space is scored on the characters kept. Whether a line feed is preserved is not visible (a kept line feed has no width), and the pages avoid wrapping and non-atomic inline elements.

  • block_flow, margin_classify, margin_combine: no browser measures which margins adjoin or what they combine to apart from the offsets block flow produces, so the three are one scoring group, generated by one feature job. Everything outside the group is pinned, float's placements among it (float boxes are in the dump); the score is the children's offsets and the container's height. Each function keeps its own provenance record and floor file.

    The scorer is crates/generation/src/block_flow_runner.rs. Every box's width and height is pinned from the browsers' dump, and only the offsets of in-flow, non-float boxes are free (layout::pins::Free::BLOCK_FLOW); floats are answered from the browsers' measured positions, so block_flow sees realistic floats. A container's height is scored from the content_height block_flow answers, against the measured border-box height less padding and border, for containers whose height is auto and whose children are block-level. The pages are the margin-collapsing recipes and the float recipes (structure_pages.rs block_flow_pages, cache corpus/block-flow-ground-truth/ built by block_flow_ground_truth_cache), 86 pages after the dump vote. All three functions are registered in generated_score_check with the same cache and the same group score, so a candidate for any one is scored with the committed other two. The starting candidates pass 0 of 86.

    A manifest names a group with group, a list of at least two function modules (jobs/src/manifest.rs, group_violations); a manifest without it is a one-function job as before.

    What handles a group today: the candidate crate (candidate_crate.rs Seam::BlockFlow: one source with a pub mod per member, each with resolve, registered together through block_flow_runner::main), the model prompt (model_prompt.rs BLOCK_FLOW_GROUP: one reply carries a module per member), the recorder (block_flow_runner.rs function_calls) and jobs/scripts/admit.py --group a,b,c (split_group_source: each member goes into its own generated file with its own record, mod.rs line, re-export and table row; every record carries the one score the group gets). block_flow's types are fixed by the engine (block_flow_types.rs, block_flow_engine.rs, block_flow_call_site.txt), not written by a model, and oracle/suites/features/block-flow.json names its WPT pages.

    A group job has a stage list of its own, jobs/src/feature_group_job.template.json, which job-check new-feature-job picks when the short form has a group (the functions with hyphens, function among them); a short form without group expands from feature_job.template.json exactly as before. The single-function stage list has a model write the function's types (types-prompt, types-candidates, types-keep, types-use), add coverage and findings pages, and take its cases from pages_ground_truth_cache. A group has none of these, because its types are fixed by the engine (its inputs hold borrows and closures, which a model cannot be asked to restate) and its cases are the pages of structure_pages::block_flow_pages, measured by block_flow_ground_truth_cache, the cache its scorer reads. So the group template runs ground-truth (--list train) and held-out-ground-truth (--list held-out, the pages whose id hashes into one in five, structure_pages::split_pages), then fetch-specs, specification-text, prompt, candidates, repair, store, review and admit. repair, store and admit take --group <module>,<module>,...: bench/repair-loop.sh scores the committed file of every member, each wrapped in its pub mod, as the landing bar and has no hand-written fallback; jobs/scripts/store-candidate.sh stores the one source and refuses one missing a member; admit.py splits it. The prompt takes the group from the resolver (layout::block_flow::BlockFlowResolver), not from a flag. The group manifest has no rework_from, since nothing in it turns a wrong input into a page. specs/block-flow.md and specs/block-flow.job.json are the expansion for the block-flow group.

  • formatting_context: display is in the dump, but which walk a box takes and whether it establishes a block formatting context are visible only as consequences (floats avoided, margins not collapsing through). It is scored on whole pages after the block-flow group has an admitted candidate, through that candidate.

  • anonymous_table: anonymous boxes appear in no measurement. It stays unscored and is scored with the table functions once the table walk is structure only.

Floors and records. A provenance record names the method in verification.scoring (browser-measured inputs, or engine's other functions). A floor compares with a record's passing count only when the record was scored the way the check scores now; a record with no scoring was scored through the other functions and holds a number that scoring no longer produces, so its floor is its floor file alone. Floor files are written from the new measurements; records are never rewritten (generated_score_check comparable_passes).

Admission #

flowchart LR
  cand["candidate<br/>scores"] --> reg{"loses any case in<br/>the regression set?"}
  reg -- yes --> reject["rejected"]
  reg -- no --> best{"beats the committed<br/>candidate's score?"}
  best -- "no (tie or lower)" --> keep["committed one stays"]
  best -- yes --> admit["admitted and<br/>committed"]
  • Regression set. Every case any admitted candidate has ever passed. A new candidate may not lose one, so the loop cannot trade old correctness for new (crates/generation/src/admit.rs).
  • Beats the committed candidate. Strictly more cases passed; a tie keeps what is already committed (admit::beats_committed).
  • Margin collapsing only: a third gate checks the candidate's intermediate decisions against Datalog rules in crates/logic, because right geometry can come from wrong reasoning (docs/margin-collapse-experiment.md).

Where it runs #

flowchart LR
  subgraph cpu["CPU servers"]
    oracle["browser oracle<br/>(builds the cache)"]
    verify["compile and score"]
  end
  subgraph gpu["GPU model pool"]
    sample["candidate generation"]
  end
  sample -- "candidate source" --> verify
  verify -- "failing cases" --> sample
  oracle -- "cached ground truth" --> verify

Scoring is fast (272,930 cases/s at sixteen threads, docs/rig-capacity.md); writing candidates is the bottleneck. Hardware, gateway and tiers: docs/infrastructure.md.

Prompts fit the model's window #

A request's prompt and reply share one context window. The padding job's types prompt was 318 KB, about 90k tokens, and asked for a 64000-token reply from qwen3.8-27b-fast-2, whose window is 131072 tokens: every request got HTTP 400.

  • The reply is sized to the room left. HttpModelClient asks for the smaller of its usual max_tokens and the window minus the estimated prompt minus a 2048-token margin (model.rs fit_max_tokens). The estimate is one token per 3 bytes, above the measured 3.5, so a prompt is never counted short. The window is read from the gateway (/v1/models max_model_len on vLLM, else /v1/model/info max_input_tokens or max_tokens on LiteLLM), else WEBENGINE_MODEL_WINDOW, else 131072, the smallest window in use. A hosted plan with a larger window that reports none is therefore treated as 131072 unless that variable is set.
  • A prompt with under 2048 tokens of room fails before the request, with a message naming the prompt's size and the window, since the server would only answer 400.
  • The types prompt and the function prompt are bounded to 288,768 bytes (model::PROMPT_BYTE_BUDGET): the 131072-token window, less a 32768-token reply (a candidate or types file is some 8,000 to 12,000 tokens, so this is generous), less the margin, at 3 bytes per token. Only the cases give way. They are chosen spread evenly over the list sorted by id (types_prompt::choose_cases), so a rerun shows the same pages, and the types prompt states how many of how many it shows. The function prompt's expected.txt lists the same cases it shows.
  • Not bounded: the pages prompts (coverage_pages.rs, findings_pages.rs) show a fixed number of examples, but their list of uncovered pairs or wrong inputs grows with the review; the request-time check above still stops one that no longer fits.
  • A pages prompt shows a train case even when no call of it has an expected output. For a function that has no admitted candidate yet, the engine's stand-in answers wrongly on the pages that most need it, so an absolute call on a fit-content page has no expected output (absolute_page_runner), and the only train pages that reach the function would be refused. The examples exist to show which markup gives which inputs, so coverage_pages::choose_examples takes every case with a call and prompt_parts::render_example_case says no expected output is shown for it.
  • model_prompt leaves out a case the browsers agree on no box of, and says so. It reports the ids (Written::skipped) and fails only when none is left. Block-flow shows no expected output for any call by design (the browsers measure boxes, not the function's answer), so for it no case is refused (prompt_parts::render_case_without_expected).
  • merge_cases --drop-outside-function takes any function name with a call recorder (candidate_crate::Seam::recorder). aspect-ratio keeps its finer test (the page fails even with the browsers' answers); any other function drops a page it is never called on.
  • A types file may not use a lifetime its struct does not declare. The engine's containing-block values borrow the ancestor chain as &'a [AncestorBox]; a model that copies the field type without <'a> writes a file that does not compile. job-check types-keep now refuses it with the struct to change (types_file::undeclared_lifetimes).

When things change #

A spec edit, a browser release or a WPT update re-enters this loop at different points: see roadmap.md → "Ongoing development lifecycle".