This repository has no description

Fork 0

bez docs dump-format.md

18 kB

Markdown

at track/rig-throughput

dietrich ayala · Feature-cost: WPT subset, reflection/mutation oracle protocol, and dom seams · 1mo ago

ca6fd493

Dump format #

The contract between the oracle service, the comparator, and eventually the engine under test. Frozen: changing it requires changing every consumer at once.

Oracle protocol #

The oracle is a long-lived process speaking newline-delimited JSON on stdio. One JSON object per line in each direction, no pretty-printing, no interleaved text on stdout. Diagnostics go to stderr.

Browsers stay open for the process lifetime. Nothing spawns a browser per case.

Request #

{"id": "case-0001", "html": "<!doctype html>...", "engines": ["chromium", "firefox", "webkit"]}

engines is optional; omitted means every engine the service launched. script is additive (docs/feature-cost-experiment.md): optional JavaScript run in the page after html loads and before the dump walk. Omitted, a request behaves exactly as before -- this is what lets DOM tree-mutation ground truth reuse the ordinary dump response unchanged, rather than needing a new shape: the mutation happens, then the same domDump walk that has always run answers the request.

{"id": "case-0002", "html": "<div id=a><span id=b></span></div>", "script": "document.getElementById('a').appendChild(document.createElement('b'));"}

Response #

{"id": "case-0001", "dumps": {"chromium": {...}, "firefox": {...}}, "errors": {"webkit": "message"}}

dumps holds one dump per engine that succeeded. errors holds a message per engine that failed, and is omitted when empty. A response always carries the id it answers.

Control #

{"id": "...", "op": "ping"} responds {"id": "...", "ok": true, "engines": [...], "versions": {"chromium": "140.0.7339.5", ...}, "profile": "default"}. versions maps each launched engine's name to its engine_version (see below); profile is the launch profile active for the process (see "Launch profiles" below). {"id": "...", "op": "shutdown"} closes browsers and exits.

Dump #

{
  "engine": "chromium",
  "engine_version": "140.0.7339.5",
  "viewport": {"width": 800, "height": 600},
  "overflowed": false,
  "nodes": [
    {
      "index": 0,
      "parent": -1,
      "tag": "HTML",
      "depth": 0,
      "bounding": {"x": 0.0, "y": 0.0, "w": 800.0, "h": 100.0},
      "rects": [{"x": 0.0, "y": 0.0, "w": 800.0, "h": 100.0}],
      "style": {"display": "block", "...": "..."}
    }
  ]
}

engine_version is the exact browser build that produced the dump, from Playwright's browser.version() at launch — not from navigator.userAgent, which the in-page walk could report but which is imprecise. Required, since ground truth without a version is exactly what this field exists to rule out.
nodes is every element in document order, starting at <html> at index 0. Document order means index increases monotonically and a parent always precedes its children.
parent is an index into nodes, or -1 for the root.
tag is the uppercase tag name as the DOM reports it.
rects is getClientRects(), one entry per box fragment, in content order. An empty array means the element generates no box — that is the signal, not a null bounding.
bounding is getBoundingClientRect(), always present, zeroed when rects is empty.
Coordinates are viewport-relative CSS pixels as doubles, serialized at full precision.
overflowed is true when the document scrolls in either axis. Overflow means a scrollbar may have stolen viewport width by a platform-dependent amount, so comparisons on an overflowed case are not trustworthy and the case is discarded.

Style subset #

A fixed, ordered list. Present on every node, values exactly as getComputedStyle returns them.

display position box-sizing direction writing-mode float clear
width height min-width max-width min-height max-height
margin-top margin-right margin-bottom margin-left
padding-top padding-right padding-bottom padding-left
border-top-width border-right-width border-bottom-width border-left-width
overflow-x overflow-y

A mismatch here indicts the style pipeline rather than layout, which is the second attribution axis.

Comparison #

Comparison never uses absolute coordinates.

Shape. The sequence of (tag, depth, parent) must match. A mismatch is a parse divergence and the case reports that and stops — no geometry is compared.
Size. Each node's bounding width and height, and each fragment's width and height pairwise.
Relative offset. Each node's bounding origin minus its parent's bounding origin. The root compares against the viewport origin. A single wrong offset then appears once rather than corrupting every descendant.

Tolerance is 1/60 px (0.01667), the coarsest engine quantization — Gecko app units. Blink and WebKit quantize at 1/64. Differences below tolerance are quantization noise.

Both an exact verdict (within tolerance or not) and the raw signed delta are reported. The verdict gates pass/fail; the delta is the dense reward signal and needs no threshold.

Trace protocol #

Additive: everything above (request, response, dump, style subset, comparison) is unchanged. trace is a new request operation with its own response shape, for what script can observe beyond geometry — return values, thrown errors, event order, promise settlement order, object structure, pixel and audio buffers — the oracle for docs/oracle-coverage.md → "Four kinds of oracle that need no geometry vote" item 4.

Origin #

Engine.run() (the geometry path) uses page.setContent, which leaves the page on about:blank; many APIs this unlocks need a secure context or a real origin. The oracle starts one local HTTP server on 127.0.0.1 for its own lifetime and serves trace pages from http://localhost:<port>/trace/<id>. Browsers treat localhost as a secure context regardless of scheme, so isSecureContext is true without TLS. Every trace records the page's origin and isSecureContext.

Request #

{"id": "t1", "op": "trace", "script": "trace.value('x', 1); trace.done();", "html": "<button id=btn></button>", "engines": ["chromium", "firefox", "webkit"]}

script is required: JavaScript text run in the served page, after the recorder (see below) is installed and after html (optional; omitted means an empty body) has loaded. engines is optional, same meaning as the geometry request.

Response #

{"id": "t1", "traces": {"chromium": {"...": "..."}}, "errors": {"webkit": "message"}}

traces holds one trace per engine that completed; errors holds a message per engine that failed (including "engine not launched"), omitted when empty.

Trace #

{
  "engine": "chromium",
  "engine_version": "153.0.8010.12",
  "profile": "default",
  "origin": "http://localhost:41673",
  "is_secure_context": true,
  "timed_out": false,
  "records": [
    {"seq": 0, "kind": "value", "label": "x", "value": 1}
  ]
}

profile is the launch profile active for every engine in this run (see "Launch profiles" below).
timed_out is true when the recorder's fixed completion timeout ended the trace rather than the script itself calling trace.done().
records is every observation the recorder pushed, in the order it pushed them; seq is that order as an explicit integer. Order is part of the result: two traces with the same records in a different order are not the same trace.

The recorder #

A script installed via Playwright's context.addInitScript, so it runs before any script the trace page's own content carries — including inline <script> tags, which load in document order after it. It defines window.trace:

trace.value(label, v) — records a canonicalized value.
trace.event(target, types) — listens for types (a string or array of strings) on target and records one event per firing, labeled from target's identity (window, document, or TAGNAME / TAGNAME#id for an element).
trace.settle(label, promise) — records how and when promise settles.
trace.error(label, err) — records a caught error directly.
trace.done() — ends the trace early. A trace that never calls this still ends, at a fixed completion timeout (5000ms, not configurable per request), with timed_out: true.

Canonicalization #

Values leave the page as JSON, so the recorder canonicalizes each one before recording it:

Booleans and strings serialize as-is.
Numbers serialize as-is, except NaN, Infinity, -Infinity and -0, which have no JSON representation and are encoded explicitly as {"$num": "NaN"} (and so on).
undefined becomes {"$undefined": true}; a bigint becomes {"$bigint": "<digits>"}.
A thrown Error (wherever one ends up — a trace.error call, a rejected trace.settle) reduces to its name: {"$error": true, "name": "TypeError"}. The message is not recorded; the name is what a test asserts on.
A typed array, DataView or ArrayBuffer reduces to base64 of its bytes plus element type and length: {"$buffer": true, "elementType": "Float32Array", "length": 2, "base64": "..."}.
ImageData reduces the same way, plus width/height; the channel data is Uint8ClampedArray. AudioBuffer reduces to {"$audioBuffer": true, "numberOfChannels", "length", "sampleRate", "channels": [...]}, each channel a Float32Array-shaped buffer.
Anything else (a plain object, an array, an IDL result) reduces to an own-property key/type structure walk, bounded to depth 4 and 64 array elements per level, so an unbounded or circular object cannot hang the trace: {"$shape": "object", "keys": [...], "props": {"k": {"type": "...", "value": ...}}} for objects, {"$shape": "array", "length": N, "elements": [...]} for arrays.

Comparison #

crates/harness/src/trace.rs compares two traces record by record, in order: an unequal record count is reported outright (CountMismatch), since there is no sound way to pair up the remainder. Otherwise every position is compared: kind and label exactly; an Event's type and an Error's name exactly; a Value or Settle's value structurally exactly (object keys, array lengths, non-numeric leaves) except a $buffer/$imageData/$audioBuffer payload, whose samples are decoded and compared within a tolerance (BufferTolerance: relative and absolute, looser than the geometry tolerance since two engines' DSP or rasterizer rounding differs more than their layout). The three-engine vote (trace::vote) mirrors compare::vote(): pairwise-agreement clustering per record, naming an outlier on a 2-1 split, Split (untrustworthy) on a three-way disagreement.

Launch profiles #

Chosen once at oracle startup with --profile=<name> (default: default), applied to every engine the process launches, reported in the ping reply's profile field and in every trace's profile field.

default — no extra launch args, no prefs, no permission grants. Exactly what geometry dumps have always used; cached ground truth keeps its meaning.
media — makes hardware-backed APIs deterministic:
- Chromium: --use-fake-device-for-media-stream, --use-fake-ui-for-media-stream, --enable-unsafe-swiftshader, plus context.grantPermissions(['camera', 'microphone']) (needed even with the fake device flags, or getUserMedia rejects with NotAllowedError).
- Firefox: prefs media.navigator.streams.fake, media.navigator.permission.disabled.
- WebKit: no launch args (Playwright's WebKit build has none for this); the same grantPermissions(['camera', 'microphone']) as Chromium.
- Verified by observation, not assumed: getUserMedia({video: true, audio: true}) resolves in all three engines under media, each delivering a fake track whose getSettings() reports a plausible fixed resolution/frame rate/sample rate (the key set differs per engine — that is a real platform difference probe's fake-camera-track-settings result surfaces, not a harness bug).
- Known gap: the profile also sets Firefox's webgl.disable-fail-if-major-performance-caveat, meant to allow WebGL on software rendering. Observed on the development host, this is not sufficient: Firefox's WebGL context creation still fails ("exhausted GL driver options"), where Chromium and WebKit both succeed via their bundled software renderer. Left in place as directionally correct and harmless; not claimed to work. A host that lets Firefox's GPU process start may not hit this.
CDP-only emulation (virtual sensors, virtual authenticator, Bluetooth) is Chromium-only and is not implemented; it is a later step, not a partial one.

Probes #

crates/harness/src/bin/probe.rs, modeled on crosscheck.rs: hand-written scripts across OfflineAudioContext rendering, a canvas getImageData readback, DOM event bubble order, promise/microtask settlement order, an IDL shape walk (URL's prototype getters), and a fake-camera track's settings, each run through every engine, compared pairwise and voted three-way, reporting agreement per probe.

Not yet supported #

CDP-only emulation (virtual sensors, virtual authenticator, Bluetooth). These exist only in Chromium, so they need their own launch profile or a per-request flag, and the trace response needs a field saying which engines applied the emulation. Without it, a three-engine vote compares emulated Chromium output against unemulated Firefox and WebKit output.
Trace inputs mined from suites. Scripts drawn from WPT, Chromium web_tests and WebKit LayoutTests, seeded by the text baselines identical between Chromium and WebKit (docs/oracle-coverage.md), would run through Oracle::trace() and trace::vote(), with mismatches routed to test-case generation.

Reflection protocol #

Additive, the same way the trace protocol is: everything above is unchanged, and a new request operation carries its own response shape. This is the reflection half of docs/feature-cost-experiment.md's measurement -- content-attribute and reflected-IDL state -- where "Request" above's additive script field is the DOM tree-mutation half.

Request #

{"id": "r1", "op": "reflect", "html": "<input id=a>", "script": "document.getElementById('a').setAttribute('value', '5');", "probes": [{"selector": "#a", "attr": "value", "idl": "value"}], "engines": ["chromium", "firefox", "webkit"]}

html is optional body markup, same meaning as a trace request's (omitted means an empty body). script is optional: JavaScript run after html loads and before the probe walk -- the mutation sequence, when the case has one. probes is required, one entry per (selector, content attribute, IDL property) triple to read: selector is a CSS selector (every case this repository scores against restricts it to a bare #id, docs/feature-cost-experiment.md); attr is the content attribute name; idl is the IDL property name and defaults to attr when omitted (most reflected attributes share their content and IDL name; for/htmlFor, class/className and similar exceptions name it explicitly). engines is optional, same meaning as the geometry request's.

Response #

{"id": "r1", "states": {"chromium": {...}}, "errors": {"webkit": "message"}}

states holds one reflection dump per engine that succeeded; errors holds a message per engine that failed (including a probe's selector not resolving), omitted when empty.

Reflection dump #

{
  "engine": "chromium",
  "engine_version": "153.0.8010.12",
  "observations": [
    {"selector": "#a", "attr": "value", "attr_value": "5", "idl": "value", "idl_value": "5"}
  ]
}

One observation per probe, in the request's probe order. attr_value is the content attribute's string value, or null when absent. idl_value is the IDL property's value, canonicalized: booleans, strings and finite numbers as-is; NaN/Infinity/-Infinity encoded explicitly the same way the trace protocol's canonicalization does ({"$num": "NaN"} and so on) since they have no JSON representation; a property the element does not carry (an IDL name typoed, or the wrong element for that attribute) comes back as {"$undefined": true} rather than null, so a probe naming a property that does not exist is visible rather than indistinguishable from one that resolved to null. A reflected IDL property is scalar by construction, so this is a much smaller canonicalization than the trace protocol's general one -- no buffers, no object shape walk.

Comparison #

crates/harness/src/reflect.rs votes three-way per observation: attr_value and idl_value must both match, exactly -- reflection has no quantization to tolerate the way geometry does. The clustering itself mirrors compare::vote() and trace::vote(): pairwise agreement per observation, naming an outlier on a 2-1 split, Split (untrustworthy) on a three-way disagreement.

Scope decisions #

#id only. Every probe's selector in this repository's own cases is a bare #id; general CSS selector matching is not needed for the two attribute types and one mutation sequence step 1 chose, and crates/dom::mutable::MutableTree (the candidate's side of the same cases) implements only #id lookup to match.
No JS interpreter on the candidate side. A case's script is hand-written JavaScript for the oracle to run; the equivalent instruction for a Rust candidate to replay against crates/dom::mutable::MutableTree is a separate, hand-written crates/dom::mutable::TreeOp value in the same case, not derived from script automatically. Keeping both in sync by hand is the cost this scope decision accepts, in exchange for not building a JavaScript-to-Rust mutation translator for a five-case experiment.