Feature-cost experiment #

Measures the cost of generating one non-geometry, in-page feature end to end, to replace the modelled range in docs/platform-economics.md -> "Per-feature cost" with a measurement (docs/platform-economics.md -> "Experiments that would move the estimate most"). The feature is HTML [Reflect] attributes plus DOM tree mutation, chosen over dialog and popover because it is mechanical and well specified, which isolates the loop's cost from spec ambiguity.

Result: 234k model tokens, $4.35, about 30 minutes of wall-clock for three 8-candidate batches; every admitted candidate passes every three-engine-voted case. Not measured: the unmetered agent sessions that built the corpus, oracle operation, function types and prompts — which cost more than the generation — and any human review.

flowchart LR
  wpt["WPT html/dom,<br/>dom/nodes"] --> corpus["91 reflection +<br/>23 mutation cases"]
  corpus --> oracle["3-engine vote<br/>reflect op / script dump"]
  oracle --> cache["ground-truth cache"]
  cache --> score["score candidates:<br/>values / tree shape"]
  model["claude-opus-5<br/>8 per batch"] --> score
  score --> gate["regression-set gate"]

1. WPT subset #

Added to oracle/suites/suites.json's wpt list:

/html/dom/ — reflection.js (ReflectionTests.reflects(type, idlName, element, domName)) keys its type table by exactly the five types crates/dom::reflect::ReflectedType covers: "string", "boolean", "long", "enum" and "url"; reflection-forms.html has call sites for all five, including content/IDL name mismatches ("className"/"class").
/dom/nodes/ — Node-appendChild.html, Node-insertBefore.html and neighbours, the three algorithms crates/dom::mutable::MutableTree implements.

The harnesses were read to confirm vocabulary and borrow cases, not run verbatim.

2. Reflection / JS-state dump #

Two additive oracle changes (docs/dump-format.md):

DOM tree mutation reuses the geometry dump: an optional script field runs after html loads and before the dump walk (docs/dump-format.md -> "Oracle protocol" -> "Request"), so a mutation's ground truth is an ordinary Dump.
Reflection is a new "reflect" operation (docs/dump-format.md -> "Reflection protocol"), with crates/harness::reflect wire types and a three-engine vote() on exact equality.

3. The evaluation path #

Two generated functions in crates/dom, each shaped like layout::width::WidthResolver: plain types, a function-pointer resolver, a spec-literal baseline, unit tests.

Reflection (crates/dom::reflect): ReflectedType, ReflectedValue, GetInputs/SetInputs, ReflectGetResolver/ReflectSetResolver, baselines reflect_get_baseline/reflect_set_baseline. Pure functions. A case that sets the content attribute passes when get_idl(type, observed_attr) == voted_idl_value; one that sets the IDL property passes when set_idl(type, assigned_value) == voted_attr_value, using the value the script assigned (a URL's read-back value is already resolved). No JavaScript interpreter is needed on the Rust side.
Tree mutation (crates/dom::mutable): MutableTree (a separate mutable arena; the parsed Document stays immutable), TreeOp (AppendChild/InsertBefore/RemoveChild by #id), MutationResolver, baseline apply_mutation_baseline. A candidate passes when its tree's (tag, depth, parent) triples (generation::feature_runner::tree_shape) equal the vote's; compared directly, since harness::compare::compare checks overflow before shape.

What is built #

Corpus (generation::feature_corpus): 91 reflection cases (five types, both directions, attributes from WPT's html/dom/elements-*.js, values at each algorithm's edges) and 23 mutation cases (including multi-op sequences). A mutation case's script is generated from its TreeOp list (ops_to_script) so the two cannot drift. Error paths are not covered.
Ground-truth cache (generation::feature_groundtruth, bin/feature_ground_truth_cache) under corpus/feature-ground-truth/{reflect,mutation}/.
Scorer (generation::feature_runner evaluate_reflect/evaluate_mutation), sharing the width function's admission gate and orchestrate::run_candidates.
Candidate plumbing: candidate_crate::write_for takes a Seam.
Fixture loop (bin/feature_loop, crates/generation/fixtures/feature-candidates/) and bin/feature_baseline_check.

Results as of this entry:

Vote: 89 of 91 reflection cases unanimous; on 2, Chromium is the outlier, reporting img.loading as "auto" where the spec, Firefox and WebKit say "eager". Every mutation case is unanimous on tree shape.
Baselines: reflection 91/91, mutation 23/23.
Gate: correct admitted on both; seeded-wrong boolean_by_value (90/91), enum_case_preserved (86/91) and insert_after_reference (18/23) rejected. The last still passes 2 of 7 insert cases, because (tag, depth, parent) cannot tell same-tag siblings apart.

Model-driven run #

Prompts (prompt::build_reflect, prompt::build_mutation): spec text, exact signatures, a contract forbidding redefinition of crates/dom types, and sampled cases with their voted results (one passing case per type or op, since the baselines pass everything).
Spec text (provenance::html_reflect::SPEC_TEXT, provenance::dom_mutation::SPEC_TEXT): verbatim WHATWG prose (CC BY 4.0), trimmed to what the cases exercise.
Runner: bin/feature_model_run --seam reflect|mutation (--dry-run over fixtures). No provenance record is written, since admit assumes the width function's closure.
The url crate is in the candidate crate and named in the reflection prompt; without it candidates hand-write a URL parser (batch 1 below).

claude-opus-5, 8 candidates per batch, max_tokens 16000, sequential; artifacts under corpus/feature-candidates/model-run/ (gitignored):

Batch	Input tokens	Output tokens	Cost	Wall-clock	Compiled	Admitted
reflection 1 (no `url` crate)	29,216	108,020	$2.85	1,175 s	5/8	0/8
reflection 2	30,200	35,714	$1.04	382 s	8/8	8/8 (91/91 each)
mutation	15,872	15,114	$0.46	250 s	5/8	5/8 (23/23 each)
total	75,288	158,848	$4.35	1,807 s

At $5 / $25 per million input / output tokens; wall-clock includes compiling and scoring.

Reflection batch 1: three candidates ran out of tokens mid-way through a hand-written URL parser; the rest passed every non-URL case. The harness's fault — no parser in reach. Batch 2 differs only by the url dependency and one prompt paragraph.
Mutation: 3 of 8 requests ended stop_reason: "refusal" on an ordinary DOM prompt, no apparent cause; every other completion passed 23/23.

4. Ledger #

Human minutes are counted for every step; an agent session's time is not human time and is recorded as 0.

Date	Step	Tokens	Wall-clock	Human minutes	Oracle disagreements	Notes
2026-09-21	1-3 (design, protocol, function types, unit tests)	0 (no model call; hand-written)	~1 session	0 (agent session; not human time)	0 (no three-engine vote run yet -- single-engine smoke test only)	`cargo test -p dom` 34/34, `cargo test -p harness` 31/31, `node oracle/selftest.mjs --engines=chromium` 34/34.
2026-09-21	corpus, cache, scorer, candidate plumbing, fixture loop	0 (no model call; hand-written)	~1 session (cache build for 114 cases on three engines not timed)	0 (agent session; not human time)	2 of 114 cases (both Chromium, `img.loading` default); 0 on mutation shape	`feature_baseline_check`: reflection 91/91, mutation 23/23. `feature_loop`: gate admits `correct`, rejects all three seeded-wrong fixtures.
2026-09-21	prompts, spec text, `feature_model_run`	not metered (agent sessions)	~1 session	0 (agent session; not human time)	none new	`--dry-run` reproduces the fixture gate results on both functions.
2026-09-21	reflection batch 1, `claude-opus-5`	137,236	1,175 s	0	2 of 91 (unchanged, `img.loading`)	0/8 admitted; no `url` crate in the candidate crate.
2026-09-21	reflection batch 2, `claude-opus-5`	65,914	382 s	0	2 of 91 (unchanged)	8/8 admitted at 91/91; `model_0` taken.
2026-09-21	mutation batch, `claude-opus-5`	30,986	250 s	0	0 of 23	5/8 admitted at 23/23; 3 refusals; `model_1` taken.

"Tokens" is model.rs Telemetry for model-driven rows.