Oracle coverage beyond geometry #

How much of the web platform has some automated way to check a generated implementation, not only the three-browser geometry vote. Revises docs/platform-economics.md's first estimate, which counted about 30% of engine-relevant entries as having no practical oracle. Research as of September 2026. Per-domain coverage is judgement except Canvas 2D, WebGL, Web Audio and WebGPU, which are measured (docs/conformance-oracles.md, docs/webgpu-cts.md).

Verdict #

The share with no usable oracle falls from 27–32% to roughly 8–18% (8–12% counting spec-authored suites and fake-device hooks, 12–18% counting only what can reach a two-engine majority). The four measured domains move the band by −0.7 to +0.2 points (docs/platform-economics.md → "Measured and modelled inputs"). A random sample of 200 compat keys (below) puts the not-probe-able share at 17.5% (12.9–23.4%) of all keys and 10.1% (6.1–16.3%) of keys all three engines ship.

pie title Sampled compat keys (n = 200)
  "probe-able now" : 128
  "needs one new oracle kind" : 37
  "not probe-able" : 35

Two discounts:

Oracled is not generatable. For GPU, codecs and real-time communication, the oracle checks a generated binding and state layer over a linked ANGLE, Dawn/wgpu, ffmpeg, dav1d or libwebrtc.
About 900 reclassified entries are Chromium-only (WebNN, WebXR, USB, HID, serial, NFC, Bluetooth, sensors, FedCM, Secure Payment Confirmation); their assertions mostly come from Chromium's WPT export, so their errors correlate with Chromium's.

Four kinds of oracle that need no geometry vote #

Spec-authored assertion suites with tolerances — expected values are in the test.
Fake-device hooks — testdriver.js, WebDriver BiDi extensions, CDP emulation, the WebXR Test API — turn hardware APIs into deterministic state machines.
Independent reference libraries — codec bitstreams with reference MD5s, an independent WebAuthn verifier, RFC 8291 Web Push decryption.
Script-observable traces voted across three engines — offline audio buffers, decoded frames, pixel readback, loopback WebRTC state, IDL shape. Landed for hand-written probes: 15 of 18 engine pairs agree under the media profile (docs/dump-format.md → "Trace protocol", → "Probes").

By domain #

Domain	Best oracles	Strength	Coverage (judgement)
Canvas 2D (~290 entries)	WPT `html/canvas`: 4,314 files, generated, `_assertPixel` / `_assertPixelApprox`, fuzzy reftests	exact, tolerance	measured: 82.6% of tests usable (92.7% excluding tentative); 80.0% of 270 entries (93.5% of the 231 outside the CSS Paint API)
WebGL (~540)	Khronos conformance suite, 2,148 pages; GraphicsFuzz-style metamorphic shader transforms	exact, tolerance	measured: 99.6% of tests usable, with dEQP; 97.8% of 495 entries
WebGPU (~330)	gpuweb CTS, 697 spec files; dawn.node as a browser-free runner; cross-runtime agreement against Chromium/SwiftShader; WebGlitch differential testing	exact, ULP	~85% exact (validation), ~50% ULP (numeric execution)
Web Audio (~305)	WPT `webaudio`, 347 files, offline renders against analytic buffers; cross-engine offline diffs; metamorphic identities (gain composition, filter linearity, resample round trip)	tolerance	measured: 74.4% of tests usable (85.6% excluding tentative, 92.7% offline); 83.3% of 257 entries
Media elements, MSE, EME (~380)	WPT media-elements 342, media-source 100, encrypted-media 156 (ClearKey is deterministic); frame-numbered clips drawn at seek times	structural, tolerance	~60%
WebCodecs (~130)	WPT 83 files; Fluster conformance vectors with MD5s (VP9, AVC, HEVC, VVC, AV1); encode-decode PSNR floors	exact, tolerance	~65%
WebRTC and capture (~740)	WPT webrtc 243 plus capture ~100; loopback peer connections; fake capture devices; `getStats` against the webrtc-stats dictionaries	structural	~60%
WebNN (~170)	WPT 182 files, per-operator ULP/ATOL	tolerance	~80%, Chromium only
WebXR (~270)	WebXR Test API, WPT 171 files	structural	~60%; the Test API must be implemented by bez
Devices (~350)	BiDi Bluetooth module and CDP BluetoothEmulation; WPT's Mojo-coupled USB/HID/serial/NFC mocks; Linux virtual devices (dummy_hcd, uhid, uinput, pty, hci_vhci) in a VM	structural, exact	~64%
Sensors, geolocation, screens (~160)	virtual sensors and pressure sources; geolocation override (all three engines via Playwright); CDP screen emulation	exact	~85%
Identity, payments, push (~300)	virtual authenticator plus independent verifier; FedCM testdriver plus wptserve identity providers; local autopush-rs with RFC 8291 decryption; `set_permission`	exact, structural	40–85%
Timing (~250)	ordering, monotonicity and coarsening bounds; virtual time; ~420 WPT files	structural	~80%
Speech (~110)	26 WPT files; espeak-ng; synthesis-to-recognition word-error-rate round trip	smoke	~25%

Canvas was misfiled in docs/platform-economics.md: it is one of the best-tested areas of WPT.

wpt.fyi as a ready-made vote table #

wpt.fyi publishes aligned per-subtest results for Chrome, Firefox and Safari (stable and experimental), Servo and Ladybird. A first hand estimate from the 2026-09-20 stable run, in areas previously counted unoracled:

Area	Subtests	Pass in two or more engines
Media (webaudio, webcodecs, MSE, EME, media elements, webvtt, speech)	14,257	74%
Timing	4,529	78%
WebRTC, capture, WebTransport	5,631	72%
Security and permission UI	2,694	66%
GPU and canvas (webgl, webgpu, canvas, webnn, webxr)	23,005	43%; 77% without WebNN
Devices and sensors	3,609	22%
Total	53,725	57%; 23,729 pass in all three

About 11k of the majority subtests are IDL-shape checks. Superseded by crates/wpt-votes, which rebuilds this from raw results and lands within 0.7% of the total (54,094 subtests); docs/wpt-votes.md is the reproducible figure.

Sampled probe-ability of compat keys #

The other direction: 200 individual compat keys drawn at random and each read against a fixed rubric. No browser ran. node oracle/suites/compat-sample.mjs --seed 20260922 regenerates corpus/compat-sample/sample-20260922.json byte for byte; the rubric, the draw and every verdict with its one-line reason live in that script, and the rubric was committed before any key was classified.

Population. 14,522 leaf keys from browser-compat-data 8.0.4 (api 8,562, css 3,354, javascript.builtins 1,027, html 675, svg 370, http 368, mathml 107, webassembly 59), drawn uniformly with a mulberry32 seed. 138 of the 200 ship unflagged and unprefixed in current stable Chrome 149, Firefox 152 and Safari 26.5.
The rubric in short. Probe-able: script on a fixed page at localhost can observe the behaviour (computed style, geometry, DOM shape, return values, exceptions, event order, promise settlement, read-back buffers within a tolerance) identically across the three engines; IDL shape alone does not count. Probe-able with new oracle work: needs exactly one missing kind — pixels, audio, input, network or timing. Not probe-able: decided by a device, a person, a vendor service, browser UI or values the spec leaves open; fake-device hooks do not lift a key out of this class.

slice	n	probe-able	probe-able with new oracle work	not probe-able
all keys	200	128 (64.0%, 57.1%–70.3%)	37 (18.5%, 13.7%–24.5%)	35 (17.5%, 12.9%–23.4%)
stable, unflagged, standard-track in all three engines	138	105 (76.1%, 68.3%–82.4%)	19 (13.8%, 9.0%–20.5%)	14 (10.1%, 6.1%–16.3%)
api	116	75 (64.7%, 55.6%–72.8%)	14 (12.1%, 7.3%–19.2%)	27 (23.3%, 16.5%–31.7%)
css	51	32 (62.7%, 49.0%–74.7%)	14 (27.5%, 17.1%–40.9%)	5 (9.8%, 4.3%–21.0%)
html	6	5 (83.3%, 43.6%–97.0%)	1 (16.7%, 3.0%–56.4%)	0 (0.0%, 0.0%–39.0%)
javascript.builtins	20	16 (80.0%, 58.4%–91.9%)	2 (10.0%, 2.8%–30.1%)	2 (10.0%, 2.8%–30.1%)
svg	2	0 (0.0%, 0.0%–65.8%)	2 (100.0%, 34.2%–100.0%)	0 (0.0%, 0.0%–65.8%)
webassembly	0	–	–	–
http	4	0 (0.0%, 0.0%–49.0%)	3 (75.0%, 30.1%–95.4%)	1 (25.0%, 4.6%–69.9%)
mathml	1	0 (0.0%, 0.0%–79.3%)	1 (100.0%, 20.7%–100.0%)	0 (0.0%, 0.0%–79.3%)

Intervals are 95% Wilson. New oracle work by kind: pixels 16, network 11, input 5, timing 4, audio 1. Only the first four rows have usable intervals.

21 of the 35 not-probe-able keys are outside the three-engine slice — devices, XR, WebNN, FedCM, speech and Privacy Sandbox live in the unshipped tail.
A screenshot vote and a header-setting fixture server would move roughly 13 points into probe-able.
The rubric is strict: under docs/platform-economics.md's looser reading (computed style counts even for paint-only CSS), 13 more CSS keys would pass, about 70% overall.
The earlier judgement (docs/platform-economics.md → "The three fractions") was ~63% in-page, ~6–8% needing wptserve, ~27–32% none or weak; the sample reproduces the in-page figure almost exactly.

Browser engines' own suites #

Suite	Size	Hook dependence
Chromium `web_tests` (own tests)	~33k test files; 14.3k text and 1.7k image baselines in-tree, plus ~28k platform baselines	`testRunner`, `internals`, `eventSender`
WebKit `LayoutTests` (own tests)	~56.7k test files, 40.7k text baselines	`testRunner` ~15.7k files, `internals` ~5.7k, `eventSender` ~1.9k
Gecko	~12.9k mochitests, 335 reftest manifests, ~5.1k crashtests	`SpecialPowers` in ~2.4k dom files
Ladybird `Tests/LibWeb`	~8.6k text (4.8k imported from WPT), ~1.1k layout, ~1.9k ref	`internals`

Roughly 60–75% of these files use plain web API; testRunner, eventSender and SpecialPowers.pushPrefEnv shim easily, most of internals cannot.
Beyond WPT, what is new is the Khronos and WebGPU suites plus about 3–5k media and audio tests.
Testharness baselines record known failures, not truth: negative labels.
2,058 text baselines are byte-identical between Chromium and WebKit (of 6,840 shared paths): a two-engine agreement set with expected output in hand, mostly fast, editing, http and svg.

The single-engine frontier #

72,014 WPT subtests pass in exactly one stable engine (Chrome-only 41,132, Firefox-only 20,834, Safari-only 10,048) when a missing run counts as a fail. crates/wpt-votes requires all three to have run and gets 54,421 (31,515 / 15,951 / 6,955); README cites that (docs/wpt-votes.md → "Reconciling the frontier count").
Experimental channels rescue 17.1% (9,309 of 54,421) of that frontier (docs/wpt-votes.md → "Summary"); a hand estimate for the Chrome-only slice alone was 8.5%.
Polyfills and reference implementations give a real second vote in places: whatwg-url, parse5, jsdom, the Temporal polyfill with test262, and polyfills for anchor positioning, popover and container queries.
Waiting works: the single-engine share of a year's shipped entries falls from about 62% to about 33% after two years.

What stays unoracled, and why #

Adaptive real-time behaviour (congestion control, echo cancellation, rate control); implementation-defined numerics (compressor, HRTF, anti-aliasing, smoothing quality); capability answers (GPU limits, isConfigSupported, voice lists); wall-clock and compositor behaviour (frame cadence, A/V sync, realtime latency); browser UI; vendor infrastructure (push services, DRM, recognition accuracy); real hardware beyond Linux virtual devices.

Moves, by value per effort #

wpt.fyi vote table joined to compat data. Done: crates/wpt-votes.
WPT canvas, WPT webaudio and Khronos WebGL in the browser pool. Done: docs/conformance-oracles.md.
Trace dump mode and launch profiles. Done for hand-written probes and the default/media profiles; CDP-only emulation and suite-mined inputs open (docs/dump-format.md → "Not yet supported"). Seed with the 2,058 identical baselines.
A dawn.node WebGPU CTS runner. Done: docs/webgpu-cts.md.
Linux virtual devices in a VM, only if device parity is needed. Not landed.

How WPT may be used #

WPT supplies test inputs; expected outputs come from live-browser runs, never from WPT's assertions or reference pages. A feature's own WPT pages are case input for generating that feature, because they already exist and need no model to write them: the pages are selected by oracle/suites/feature-pages.mjs and measured in the three browsers by pages_ground_truth_cache (289 aspect-ratio pages give 40 cases in about 7 s). One page in five, chosen by a hash of its file name, is a fixed held-out partition used only for scoring, so an exam of pages that generation has not seen remains. The rule, the two commands and the drop rules: docs/geometry-verification.md → "Cases from a feature's existing WPT pages".