burrito.space / bez
This repository has no description
Star
Fork 0
Watch
bez docs dump-format.md
28 kB
Markdown
at main

Dump format #

The contract between the oracle service, the comparator, and eventually the engine under test. Frozen: changing it requires changing every consumer at once. What the comparison means, explained from zero, is in docs/geometry-verification.md.

flowchart LR
  req["request line<br/>(stdin, JSON)"] --> op{"op"}
  op -- "none: geometry" --> dump["Dump per engine<br/>rects, style, scroll"]
  op -- "trace" --> trace["Trace per engine<br/>recorded JS observations"]
  op -- "reflect" --> refl["Reflection dump<br/>attribute + IDL values"]
  op -- "stackingProbe" --> stack["ids at points,<br/>front to back"]
  dump --> vote["compare.rs vote<br/>1/60 px tolerance"]
  trace --> tvote["trace::vote"]
  refl --> rvote["reflect::vote<br/>exact"]
  stack --> svote["vote_stacking<br/>exact, unanimous only"]

Oracle protocol #

The oracle is a long-lived process speaking newline-delimited JSON on stdio. One JSON object per line in each direction, no pretty-printing, no interleaved text on stdout. Diagnostics go to stderr.

Browsers stay open for the process lifetime. Nothing spawns a browser per case.

Request #

{"id": "case-0001", "html": "<!doctype html>...", "engines": ["chromium", "firefox", "webkit"]}

engines is optional; omitted means every engine the service launched. script is optional JavaScript run after html loads and before the dump walk (docs/feature-cost-experiment.md), so DOM tree-mutation ground truth is an ordinary dump.

{"id": "case-0002", "html": "<div id=a><span id=b></span></div>", "script": "document.getElementById('a').appendChild(document.createElement('b'));"}

Response #

{"id": "case-0001", "dumps": {"chromium": {...}, "firefox": {...}}, "errors": {"webkit": "message"}}

dumps holds one dump per engine that succeeded. errors holds a message per engine that failed, and is omitted when empty. A response always carries the id it answers.

Control #

{"id": "...", "op": "ping"} responds {"id": "...", "ok": true, "engines": [...], "versions": {"chromium": "140.0.7339.5", ...}, "profile": "default"}. versions maps each launched engine's name to its engine_version (see below); profile is the launch profile active for the process (see "Launch profiles" below). {"id": "...", "op": "shutdown"} closes browsers and exits.

Dump #

{
  "engine": "chromium",
  "engine_version": "140.0.7339.5",
  "viewport": {"width": 800, "height": 600},
  "overflowed": false,
  "nodes": [
    {
      "index": 0,
      "parent": -1,
      "tag": "HTML",
      "depth": 0,
      "bounding": {"x": 0.0, "y": 0.0, "w": 800.0, "h": 100.0},
      "rects": [{"x": 0.0, "y": 0.0, "w": 800.0, "h": 100.0}],
      "scroll": {"client_width": 800.0, "client_height": 100.0, "scroll_width": 800.0, "scroll_height": 100.0},
      "style": {"display": "block", "...": "..."}
    }
  ]
}
  • engine_version is the exact browser build that produced the dump, from Playwright's browser.version() at launch — not from navigator.userAgent, which the in-page walk could report but which is imprecise. Required, since ground truth without a version is exactly what this field exists to rule out.
  • nodes is every element in document order, starting at <html> at index 0. Document order means index increases monotonically and a parent always precedes its children.
  • parent is an index into nodes, or -1 for the root.
  • tag is the uppercase tag name as the DOM reports it.
  • rects is getClientRects(), one entry per box fragment, in content order. An empty array means the element generates no box — that is the signal, not a null bounding.
  • bounding is getBoundingClientRect(), always present, zeroed when rects is empty.
  • Coordinates are viewport-relative CSS pixels as doubles, serialized at full precision.
  • overflowed is true when the document scrolls in either axis. It is a report, not a verdict: trustworthiness is decided per node from scroll (see "Scrollbar probe" below), because one document-level boolean from one engine cannot say whether the engines agree.

Scrollbar probe #

Additive: everything above (request, response, the rest of the dump, style subset) is unchanged. scroll is a new optional per-node object, alongside bounding and rects, carrying the element's client and scroll dimensions as CSSOM-View reports them (clientWidth/clientHeight, scrollWidth/scrollHeight).

"scroll": {"client_width": 200.0, "client_height": 100.0, "scroll_width": 400.0, "scroll_height": 100.0}
  • The two differ on an axis exactly when the element's content spills past its padding box on that axis — here, horizontally. This holds whatever the element's own overflow value is: a visible box whose content spills reports differing dimensions too, as Chromium, Firefox and WebKit all do. What overflow changes is the other direction: a box that clips (hidden, scroll or auto) keeps its own spill out of its ancestors' scroll dimensions.
  • The signal is therefore "this element's content overflows it", not "a scrollbar is painted": a rendered scrollbar follows from it for scroll and auto, a hidden box clips instead, and a visible box simply spills. The dump cannot tell the three apart, and deliberately does not try — the three engines agree on the overflow and would not agree on a painted widget's width.
  • The root element's client box is the viewport's, so the root reports overflow whenever the page is larger than the viewport, whatever its own overflow value. That is the condition the whole-document overflowed boolean reports on its own.
  • Omitted from dumps taken before the probe existed, and from an engine under test with no scrolling model. Consumers treat an absent scroll as unknown, never as "no scrollbar" — except when scoring an engine under test against the browsers, where an absent probe is exactly the engine reporting no scrollbar.
  • The pixel width a scrollbar steals is platform- and theme-dependent and is deliberately not part of this contract. Only presence is.

Comparison uses it as the discard rule: a case is untrustworthy when three or more engines disagree about whether some node overflows on some axis, clustered by the same pairwise-agreement rule as the geometry vote (harness::compare::scrollbar_dispute). Unanimous agreement that a scrollbar is present does not discard the case — it is a case an engine that does not model overflow is expected to fail.

Style subset #

A fixed, ordered list. Present on every node, values exactly as getComputedStyle returns them.

display position box-sizing direction writing-mode float clear
width height min-width max-width min-height max-height
margin-top margin-right margin-bottom margin-left
padding-top padding-right padding-bottom padding-left
border-top-width border-right-width border-bottom-width border-left-width
overflow-x overflow-y

A mismatch here indicts the style pipeline rather than layout, which is the second attribution axis.

Comparison #

Comparison never uses absolute coordinates.

  1. Shape. The sequence of (tag, depth, parent) must match. A mismatch is a parse divergence and the case reports that and stops — no geometry is compared.
  2. Size. Each node's bounding width and height, and each fragment's width and height pairwise.
  3. Relative offset. Each node's bounding origin minus its parent's bounding origin. The root compares against the viewport origin. A single wrong offset then appears once rather than corrupting every descendant.

Tolerance is 1/60 px (0.01667), the coarsest engine quantization — Gecko app units. Blink and WebKit quantize at 1/64. Differences below tolerance are quantization noise.

Both an exact verdict (within tolerance or not) and the raw signed delta are reported. The verdict gates pass/fail; the delta is the dense reward signal and needs no threshold.

Trace protocol #

Additive: everything above (request, response, dump, style subset, comparison) is unchanged. trace is a new request operation with its own response shape, for what script can observe beyond geometry — return values, thrown errors, event order, promise settlement order, object structure, pixel and audio buffers — the oracle for docs/oracle-coverage.md → "Four kinds of oracle that need no geometry vote" item 4.

Origin #

Engine.run() (the geometry path) uses page.setContent, which leaves the page on about:blank; many APIs this unlocks need a secure context or a real origin. The oracle starts one local HTTP server on 127.0.0.1 for its own lifetime and serves trace pages from http://localhost:<port>/trace/<id>. Browsers treat localhost as a secure context regardless of scheme, so isSecureContext is true without TLS. Every trace records the page's origin and isSecureContext.

Request #

{"id": "t1", "op": "trace", "script": "trace.value('x', 1); trace.done();", "html": "<button id=btn></button>", "engines": ["chromium", "firefox", "webkit"]}

script is required: JavaScript text run in the served page, after the recorder (see below) is installed and after html (optional; omitted means an empty body) has loaded. engines is optional, same meaning as the geometry request.

Response #

{"id": "t1", "traces": {"chromium": {"...": "..."}}, "errors": {"webkit": "message"}}

traces holds one trace per engine that completed; errors holds a message per engine that failed (including "engine not launched"), omitted when empty.

Trace #

{
  "engine": "chromium",
  "engine_version": "153.0.8010.12",
  "profile": "default",
  "origin": "http://localhost:41673",
  "is_secure_context": true,
  "timed_out": false,
  "records": [
    {"seq": 0, "kind": "value", "label": "x", "value": 1}
  ]
}
  • profile is the launch profile active for every engine in this run (see "Launch profiles" below).
  • timed_out is true when the recorder's fixed completion timeout ended the trace rather than the script itself calling trace.done().
  • records is every observation the recorder pushed, in the order it pushed them; seq is that order as an explicit integer. Order is part of the result: two traces with the same records in a different order are not the same trace.

The recorder #

A script installed via Playwright's context.addInitScript, so it runs before any script the trace page's own content carries — including inline <script> tags, which load in document order after it. It defines window.trace:

  • trace.value(label, v) — records a canonicalized value.
  • trace.event(target, types) — listens for types (a string or array of strings) on target and records one event per firing, labeled from target's identity (window, document, or TAGNAME / TAGNAME#id for an element).
  • trace.settle(label, promise) — records how and when promise settles.
  • trace.error(label, err) — records a caught error directly.
  • trace.done() — ends the trace early. A trace that never calls this still ends, at a fixed completion timeout (5000ms, not configurable per request), with timed_out: true.

Canonicalization #

Values leave the page as JSON, so the recorder canonicalizes each one before recording it:

  • Booleans and strings serialize as-is.
  • Numbers serialize as-is, except NaN, Infinity, -Infinity and -0, which have no JSON representation and are encoded explicitly as {"$num": "NaN"} (and so on).
  • undefined becomes {"$undefined": true}; a bigint becomes {"$bigint": "<digits>"}.
  • A thrown Error (wherever one ends up — a trace.error call, a rejected trace.settle) reduces to its name: {"$error": true, "name": "TypeError"}. The message is not recorded; the name is what a test asserts on.
  • A typed array, DataView or ArrayBuffer reduces to base64 of its bytes plus element type and length: {"$buffer": true, "elementType": "Float32Array", "length": 2, "base64": "..."}.
  • ImageData reduces the same way, plus width/height; the channel data is Uint8ClampedArray. AudioBuffer reduces to {"$audioBuffer": true, "numberOfChannels", "length", "sampleRate", "channels": [...]}, each channel a Float32Array-shaped buffer.
  • Anything else (a plain object, an array, an IDL result) reduces to an own-property key/type structure walk, bounded to depth 4 and 64 array elements per level, so an unbounded or circular object cannot hang the trace: {"$shape": "object", "keys": [...], "props": {"k": {"type": "...", "value": ...}}} for objects, {"$shape": "array", "length": N, "elements": [...]} for arrays.

Comparison #

crates/harness/src/trace.rs compares two traces record by record, in order: an unequal record count is reported outright (CountMismatch), since there is no sound way to pair up the remainder. Otherwise every position is compared: kind and label exactly; an Event's type and an Error's name exactly; a Value or Settle's value structurally exactly (object keys, array lengths, non-numeric leaves) except a $buffer/$imageData/$audioBuffer payload, whose samples are decoded and compared within a tolerance (BufferTolerance: relative and absolute, looser than the geometry tolerance since two engines' DSP or rasterizer rounding differs more than their layout). The three-engine vote (trace::vote) mirrors compare::vote(): pairwise-agreement clustering per record, naming an outlier on a 2-1 split, Split (untrustworthy) on a three-way disagreement.

Launch profiles #

Chosen once at oracle startup with --profile=<name> (default: default), applied to every engine the process launches, reported in the ping reply's profile field and in every trace's profile field.

  • default — no extra launch args, no prefs, no permission grants. Exactly what geometry dumps have always used; cached ground truth keeps its meaning.
  • media — makes hardware-backed APIs deterministic:
    • Chromium: --use-fake-device-for-media-stream, --use-fake-ui-for-media-stream, --enable-unsafe-swiftshader, plus context.grantPermissions(['camera', 'microphone']) (needed even with the fake device flags, or getUserMedia rejects with NotAllowedError).
    • Firefox: prefs media.navigator.streams.fake, media.navigator.permission.disabled.
    • WebKit: no launch args (Playwright's WebKit build has none for this); the same grantPermissions(['camera', 'microphone']) as Chromium.
    • Verified by observation, not assumed: getUserMedia({video: true, audio: true}) resolves in all three engines under media, each delivering a fake track whose getSettings() reports a plausible fixed resolution/frame rate/sample rate (the key set differs per engine — that is a real platform difference probe's fake-camera-track-settings result surfaces, not a harness bug).
    • Known gap: the profile's Firefox pref webgl.disable-fail-if-major-performance-caveat does not make WebGL work on this host ("exhausted GL driver options"); kept as harmless, not claimed to work.
  • CDP-only emulation (virtual sensors, virtual authenticator, Bluetooth) is Chromium-only and is not implemented; it is a later step, not a partial one.

Probes #

crates/harness/src/bin/probe.rs, modeled on crosscheck.rs: hand-written scripts across OfflineAudioContext rendering, a canvas getImageData readback, DOM event bubble order, promise/microtask settlement order, an IDL shape walk (URL's prototype getters), and a fake-camera track's settings, each run through every engine, compared pairwise and voted three-way, reporting agreement per probe.

Not yet supported #

  • CDP-only emulation (virtual sensors, virtual authenticator, Bluetooth). These exist only in Chromium, so they need their own launch profile or a per-request flag, and the trace response needs a field saying which engines applied the emulation. Without it, a three-engine vote compares emulated Chromium output against unemulated Firefox and WebKit output.
  • Trace inputs mined from suites. Scripts drawn from WPT, Chromium web_tests and WebKit LayoutTests, seeded by the text baselines identical between Chromium and WebKit (docs/oracle-coverage.md), would run through Oracle::trace() and trace::vote(), with mismatches routed to test-case generation.

Reflection protocol #

Additive, the same way the trace protocol is: everything above is unchanged, and a new request operation carries its own response shape. This is the reflection half of docs/feature-cost-experiment.md's measurement -- content-attribute and reflected-IDL state -- where "Request" above's additive script field is the DOM tree-mutation half.

Request #

{"id": "r1", "op": "reflect", "html": "<input id=a>", "script": "document.getElementById('a').setAttribute('value', '5');", "probes": [{"selector": "#a", "attr": "value", "idl": "value"}], "engines": ["chromium", "firefox", "webkit"]}

html is optional body markup, same meaning as a trace request's (omitted means an empty body). script is optional: JavaScript run after html loads and before the probe walk -- the mutation sequence, when the case has one. probes is required, one entry per (selector, content attribute, IDL property) triple to read: selector is a CSS selector (every case this repository scores against restricts it to a bare #id, docs/feature-cost-experiment.md); attr is the content attribute name; idl is the IDL property name and defaults to attr when omitted (most reflected attributes share their content and IDL name; for/htmlFor, class/className and similar exceptions name it explicitly). engines is optional, same meaning as the geometry request's.

Response #

{"id": "r1", "states": {"chromium": {...}}, "errors": {"webkit": "message"}}

states holds one reflection dump per engine that succeeded; errors holds a message per engine that failed (including a probe's selector not resolving), omitted when empty.

Reflection dump #

{
  "engine": "chromium",
  "engine_version": "153.0.8010.12",
  "observations": [
    {"selector": "#a", "attr": "value", "attr_value": "5", "idl": "value", "idl_value": "5"}
  ]
}

One observation per probe, in the request's probe order. attr_value is the content attribute's string value, or null when absent. idl_value is the IDL property's value, canonicalized: booleans, strings and finite numbers as-is; NaN/Infinity/-Infinity encoded explicitly the same way the trace protocol's canonicalization does ({"$num": "NaN"} and so on) since they have no JSON representation; a property the element does not carry (an IDL name typoed, or the wrong element for that attribute) comes back as {"$undefined": true} rather than null, so a probe naming a property that does not exist is visible rather than indistinguishable from one that resolved to null. A reflected IDL property is scalar by construction, so this is a much smaller canonicalization than the trace protocol's general one -- no buffers, no object shape walk.

Comparison #

crates/harness/src/reflect.rs votes three-way per observation: attr_value and idl_value must both match, exactly -- reflection has no quantization to tolerate the way geometry does. The clustering itself mirrors compare::vote() and trace::vote(): pairwise agreement per observation, naming an outlier on a 2-1 split, Split (untrustworthy) on a three-way disagreement.

Scope decisions #

  • #id only. Every probe selector is a bare #id, and crates/dom::mutable::MutableTree implements only #id lookup to match.
  • No JS interpreter on the candidate side. A case carries both the JavaScript the oracle runs and a crates/dom::mutable::TreeOp the Rust candidate replays; the corpus generates the script from the ops (ops_to_script), so they cannot drift.

Stacking probe #

Additive, the same way the trace and reflection protocols are: everything above is unchanged, and stackingProbe is a new request operation with its own response shape. Scope: CSS 2.1 §9.9/Appendix E paint order for block boxes, absolutely positioned boxes, and opacity-triggered stacking contexts (tasks/stacking-order.md).

Request #

{"id": "s1", "op": "stackingProbe", "html": "...", "points": [{"x": 40, "y": 60}], "engines": ["chromium", "firefox", "webkit"]}
  • html (required): the document to render, same as the base request's own html.
  • points (required): viewport-relative {x, y} pairs, in CSS pixels against the oracle's fixed 800x600 viewport, one per region the caller wants an ordering for.
  • engines (optional): as in the base request -- omitted means every launched engine.

Response #

{"id": "s1", "results": {"chromium": {"engine": "chromium", "engine_version": "...", "points": [{"x": 40, "y": 60, "ids": ["b2", "b1"]}]}}, "errors": {"webkit": "message"}}

results is keyed by engine name, one entry per engine that answered; errors (omitted when empty) carries a message per engine that did not.

Per-point result #

{"x": 40, "y": 60, "ids": ["b2", "b1"]}
  • x/y: echoes the request point.
  • ids: the identifiers document.elementsFromPoint(x, y) returns, front to back (the ordering elementsFromPoint itself returns -- topmost paint first). An identifier is the matched element's class attribute, not its DOM id -- the generated documents this probe runs against never set a DOM id, but every box the harness generates already carries a unique per-box class (crates/harness/src/cases.rs write_body, b0, b1, ...), which this probe reuses rather than adding a second identifier scheme. An element with no class (the implicit <html>/<body> wrapper) is dropped from the list.

Comparison #

crates/harness/src/trace.rs vote_stacking(): a point's ordering is ground truth only when every engine's ids list for that point is identical, element for element -- CSS paint order at a point is either exactly right or wrong, so there is no tolerance band and no named-outlier majority the way compare::vote/trace::vote allow for geometry (agreement is exact-match, not a numeric closeness check). A point without unanimous agreement is dropped and counted, not guessed at.

Text probe #

Additive, the same way the stacking probe is: everything above is unchanged, and textProbe is a new request operation with its own response shape. It exists because every other corpus here is text-free by construction, so nothing had measured whether the engines agree on geometry once glyphs are involved (tasks/text-metrics-oracle.md, docs/text-metrics.md). Scope: Latin text in one writing mode. Out: shaping beyond Latin, bidi, vertical writing modes, system fonts and text painting.

Request #

{"id": "t1", "op": "textProbe", "html": "...", "font": "ahem", "engines": ["chromium", "firefox", "webkit"]}
  • html (required): the document to render, same as the base request's own html. It must carry a <head>; see "The font" below.
  • font (required): which pinned font to measure in, ahem or lato. An unknown key, or a key whose file is not in the checkout, fails the whole request rather than one engine, and the message names node oracle/suites/fetch.mjs wpt.
  • engines (optional): as in the base request -- omitted means every launched engine.

The font #

The oracle reads the font's bytes from the WPT checkout oracle/suites/suites.json pins, whose sparse list already fetches /fonts/, and inserts an @font-face rule carrying them as a data: URL directly after the document's <head>, so the page stays self-contained under page.setContent with no network fetch. The rule goes inside the head rather than ahead of the document string because a <style> before <!doctype html> would make the doctype content and drop the page into quirks mode, changing the geometry being measured.

The family is always probe, whichever font key was requested, so a case's markup is identical in both fonts and the only thing that varies is the bytes. Before any measurement the oracle calls document.fonts.load(), waits for document.fonts.ready, and fails that engine's request unless a FontFace for probe exists in the document and every such face reports status: "loaded". document.fonts.check() is checked too but cannot carry this alone: for a family with no matching @font-face it reports the system fallback as available, so it answers "true" in exactly the case that must fail. A silently substituted fallback font would make every agreement figure meaningless, so an engine that fell back appears in errors, never in results.

Response #

{"id": "t1", "results": {"chromium": {"engine": "chromium", "engine_version": "...", "font": "ahem", "font_sha256": "b719ec...", "font_family": "probe", "viewport": {"width": 800, "height": 600}, "overflowed": false, "nodes": [], "texts": []}}, "errors": {"webkit": "message"}}

results is keyed by engine name, one entry per engine that answered; errors (omitted when empty) carries a message per engine that did not.

  • nodes, viewport and overflowed are exactly what an ordinary dump carries, with the same meaning, produced by the same walk. crates/harness/src/dump.rs TextDump::geometry() hands that half back as a plain Dump, so element bounding boxes go through compare::vote() unchanged.
  • font echoes the request's key; font_sha256 is the sha256 of the file actually embedded, so a figure is tied to exact bytes rather than to a file name.
  • texts is every text node in the document, in document order.

Per-text-node result #

{"parent": 5, "text": "alpha bravo", "truncated": false, "rects": [{"x": 0, "y": 0, "w": 176, "h": 16}]}
  • parent: the index into nodes of the element this text node is a child of. It is the same index the element dump uses, recomputed by the same ordering rule (pre-order from document.documentElement, children in order).
  • text: the node's data, truncated to 64 characters with truncated set when it was. Only the rects are measured; the text is carried so a stored response can be read, and a text probe's document holds the font's several-hundred-kilobyte data: URL as one text node.
  • rects: Range.getClientRects() over the node's contents -- one rect per line fragment, in line order. Empty for a text node that generates no line box, which includes the contents of every <style> in the document, the injected @font-face rule among them.

Comparison #

crates/harness/src/trace.rs vote_text(), using compare::Consensus and dump::TOLERANCE_PX = 1/60 px: the same pairwise rule element geometry is voted by, over values that are not Dump nodes and so cannot go through compare::vote() directly. Fragment count is voted for every text node; when it disagrees there is no aligned pairing, so the per-fragment x, y, w and h are skipped for that node and only the count is reported. Engines that disagree on how many text nodes the document has at all are a parse-tree difference rather than a geometry one, and are rejected (TextVoteError::TextCountMismatch) rather than absorbed.

crates/harness/src/bin/text_metrics.rs is what runs this over a corpus; it drops a text node that produced no line box in any engine rather than counting it as agreement about geometry that does not exist.