# Oracle coverage beyond geometry

How much of the web platform has *some* automated way to check a generated
implementation, not only the three-browser geometry vote. Revises
`docs/platform-economics.md`'s first estimate, which counted about 30% of engine-relevant
entries as having no practical oracle. Research as of September 2026. Per-domain coverage
is judgement except Canvas 2D, WebGL, Web Audio and WebGPU, which are measured
(`docs/conformance-oracles.md`, `docs/webgpu-cts.md`).

## Verdict

The share with no usable oracle falls from 27–32% to roughly **8–18%** (8–12% counting
spec-authored suites and fake-device hooks, 12–18% counting only what can reach a
two-engine majority). The four measured domains move the band by −0.7 to +0.2 points
(`docs/platform-economics.md` → "Measured and modelled inputs"). A random sample of 200
compat keys (below) puts the not-probe-able share at 17.5% (12.9–23.4%) of all keys and
10.1% (6.1–16.3%) of keys all three engines ship.

```mermaid
pie title Sampled compat keys (n = 200)
  "probe-able now" : 128
  "needs one new oracle kind" : 37
  "not probe-able" : 35
```

Two discounts:

- **Oracled is not generatable.** For GPU, codecs and real-time communication, the oracle
  checks a generated binding and state layer over a linked ANGLE, Dawn/wgpu, ffmpeg,
  dav1d or libwebrtc.
- **About 900 reclassified entries are Chromium-only** (WebNN, WebXR, USB, HID, serial,
  NFC, Bluetooth, sensors, FedCM, Secure Payment Confirmation); their assertions mostly
  come from Chromium's WPT export, so their errors correlate with Chromium's.

## Four kinds of oracle that need no geometry vote

1. **Spec-authored assertion suites with tolerances** — expected values are in the test.
2. **Fake-device hooks** — `testdriver.js`, WebDriver BiDi extensions, CDP emulation, the
   WebXR Test API — turn hardware APIs into deterministic state machines.
3. **Independent reference libraries** — codec bitstreams with reference MD5s, an
   independent WebAuthn verifier, RFC 8291 Web Push decryption.
4. **Script-observable traces voted across three engines** — offline audio buffers,
   decoded frames, pixel readback, loopback WebRTC state, IDL shape. Landed for
   hand-written probes: 15 of 18 engine pairs agree under the `media` profile
   (`docs/dump-format.md` → "Trace protocol", → "Probes").

## By domain

| Domain | Best oracles | Strength | Coverage (judgement) |
|---|---|---|---|
| Canvas 2D (~290 entries) | WPT `html/canvas`: 4,314 files, generated, `_assertPixel` / `_assertPixelApprox`, fuzzy reftests | exact, tolerance | measured: 82.6% of tests usable (92.7% excluding tentative); 80.0% of 270 entries (93.5% of the 231 outside the CSS Paint API) |
| WebGL (~540) | Khronos conformance suite, 2,148 pages; GraphicsFuzz-style metamorphic shader transforms | exact, tolerance | measured: 99.6% of tests usable, with dEQP; 97.8% of 495 entries |
| WebGPU (~330) | gpuweb CTS, 697 spec files; dawn.node as a browser-free runner; cross-runtime agreement against Chromium/SwiftShader; WebGlitch differential testing | exact, ULP | ~85% exact (validation), ~50% ULP (numeric execution) |
| Web Audio (~305) | WPT `webaudio`, 347 files, offline renders against analytic buffers; cross-engine offline diffs; metamorphic identities (gain composition, filter linearity, resample round trip) | tolerance | measured: 74.4% of tests usable (85.6% excluding tentative, 92.7% offline); 83.3% of 257 entries |
| Media elements, MSE, EME (~380) | WPT media-elements 342, media-source 100, encrypted-media 156 (ClearKey is deterministic); frame-numbered clips drawn at seek times | structural, tolerance | ~60% |
| WebCodecs (~130) | WPT 83 files; Fluster conformance vectors with MD5s (VP9, AVC, HEVC, VVC, AV1); encode-decode PSNR floors | exact, tolerance | ~65% |
| WebRTC and capture (~740) | WPT webrtc 243 plus capture ~100; loopback peer connections; fake capture devices; `getStats` against the webrtc-stats dictionaries | structural | ~60% |
| WebNN (~170) | WPT 182 files, per-operator ULP/ATOL | tolerance | ~80%, Chromium only |
| WebXR (~270) | WebXR Test API, WPT 171 files | structural | ~60%; the Test API must be implemented by bez |
| Devices (~350) | BiDi Bluetooth module and CDP BluetoothEmulation; WPT's Mojo-coupled USB/HID/serial/NFC mocks; Linux virtual devices (dummy_hcd, uhid, uinput, pty, hci_vhci) in a VM | structural, exact | ~64% |
| Sensors, geolocation, screens (~160) | virtual sensors and pressure sources; geolocation override (all three engines via Playwright); CDP screen emulation | exact | ~85% |
| Identity, payments, push (~300) | virtual authenticator plus independent verifier; FedCM testdriver plus wptserve identity providers; local autopush-rs with RFC 8291 decryption; `set_permission` | exact, structural | 40–85% |
| Timing (~250) | ordering, monotonicity and coarsening bounds; virtual time; ~420 WPT files | structural | ~80% |
| Speech (~110) | 26 WPT files; espeak-ng; synthesis-to-recognition word-error-rate round trip | smoke | ~25% |

Canvas was misfiled in `docs/platform-economics.md`: it is one of the best-tested areas of
WPT.

## wpt.fyi as a ready-made vote table

wpt.fyi publishes aligned per-subtest results for Chrome, Firefox and Safari (stable and
experimental), Servo and Ladybird. A first hand estimate from the 2026-09-20 stable run, in
areas previously counted unoracled:

| Area | Subtests | Pass in two or more engines |
|---|---|---|
| Media (webaudio, webcodecs, MSE, EME, media elements, webvtt, speech) | 14,257 | 74% |
| Timing | 4,529 | 78% |
| WebRTC, capture, WebTransport | 5,631 | 72% |
| Security and permission UI | 2,694 | 66% |
| GPU and canvas (webgl, webgpu, canvas, webnn, webxr) | 23,005 | 43%; 77% without WebNN |
| Devices and sensors | 3,609 | 22% |
| **Total** | **53,725** | **57%**; 23,729 pass in all three |

About 11k of the majority subtests are IDL-shape checks. Superseded by `crates/wpt-votes`,
which rebuilds this from raw results and lands within 0.7% of the total (54,094
subtests); `docs/wpt-votes.md` is the reproducible figure.

## Sampled probe-ability of compat keys

The other direction: 200 individual compat keys drawn at random and each read against a
fixed rubric. No browser ran. `node oracle/suites/compat-sample.mjs --seed 20260922`
regenerates `corpus/compat-sample/sample-20260922.json` byte for byte; the rubric, the
draw and every verdict with its one-line reason live in that script, and the rubric was
committed before any key was classified.

- **Population.** 14,522 leaf keys from browser-compat-data 8.0.4 (`api` 8,562, `css`
  3,354, `javascript.builtins` 1,027, `html` 675, `svg` 370, `http` 368, `mathml` 107,
  `webassembly` 59), drawn uniformly with a mulberry32 seed. 138 of the 200 ship
  unflagged and unprefixed in current stable Chrome 149, Firefox 152 and Safari 26.5.
- **The rubric in short.** *Probe-able*: script on a fixed page at localhost can observe
  the behaviour (computed style, geometry, DOM shape, return values, exceptions, event
  order, promise settlement, read-back buffers within a tolerance) identically across
  the three engines; IDL shape alone does not count. *Probe-able with new oracle work*:
  needs exactly one missing kind — pixels, audio, input, network or timing. *Not
  probe-able*: decided by a device, a person, a vendor service, browser UI or values the
  spec leaves open; fake-device hooks do not lift a key out of this class.

| slice | n | probe-able | probe-able with new oracle work | not probe-able |
|---|---:|---|---|---|
| all keys | 200 | 128 (64.0%, 57.1%–70.3%) | 37 (18.5%, 13.7%–24.5%) | 35 (17.5%, 12.9%–23.4%) |
| stable, unflagged, standard-track in all three engines | 138 | 105 (76.1%, 68.3%–82.4%) | 19 (13.8%, 9.0%–20.5%) | 14 (10.1%, 6.1%–16.3%) |
| api | 116 | 75 (64.7%, 55.6%–72.8%) | 14 (12.1%, 7.3%–19.2%) | 27 (23.3%, 16.5%–31.7%) |
| css | 51 | 32 (62.7%, 49.0%–74.7%) | 14 (27.5%, 17.1%–40.9%) | 5 (9.8%, 4.3%–21.0%) |
| html | 6 | 5 (83.3%, 43.6%–97.0%) | 1 (16.7%, 3.0%–56.4%) | 0 (0.0%, 0.0%–39.0%) |
| javascript.builtins | 20 | 16 (80.0%, 58.4%–91.9%) | 2 (10.0%, 2.8%–30.1%) | 2 (10.0%, 2.8%–30.1%) |
| svg | 2 | 0 (0.0%, 0.0%–65.8%) | 2 (100.0%, 34.2%–100.0%) | 0 (0.0%, 0.0%–65.8%) |
| webassembly | 0 | – | – | – |
| http | 4 | 0 (0.0%, 0.0%–49.0%) | 3 (75.0%, 30.1%–95.4%) | 1 (25.0%, 4.6%–69.9%) |
| mathml | 1 | 0 (0.0%, 0.0%–79.3%) | 1 (100.0%, 20.7%–100.0%) | 0 (0.0%, 0.0%–79.3%) |

Intervals are 95% Wilson. New oracle work by kind: pixels 16, network 11, input 5, timing
4, audio 1. Only the first four rows have usable intervals.

- 21 of the 35 not-probe-able keys are outside the three-engine slice — devices, XR,
  WebNN, FedCM, speech and Privacy Sandbox live in the unshipped tail.
- A screenshot vote and a header-setting fixture server would move roughly 13 points into
  probe-able.
- The rubric is strict: under `docs/platform-economics.md`'s looser reading (computed
  style counts even for paint-only CSS), 13 more CSS keys would pass, about 70% overall.
- The earlier judgement (`docs/platform-economics.md` → "The three fractions") was ~63%
  in-page, ~6–8% needing wptserve, ~27–32% none or weak; the sample reproduces the
  in-page figure almost exactly.

## Browser engines' own suites

| Suite | Size | Hook dependence |
|---|---|---|
| Chromium `web_tests` (own tests) | ~33k test files; 14.3k text and 1.7k image baselines in-tree, plus ~28k platform baselines | `testRunner`, `internals`, `eventSender` |
| WebKit `LayoutTests` (own tests) | ~56.7k test files, 40.7k text baselines | `testRunner` ~15.7k files, `internals` ~5.7k, `eventSender` ~1.9k |
| Gecko | ~12.9k mochitests, 335 reftest manifests, ~5.1k crashtests | `SpecialPowers` in ~2.4k dom files |
| Ladybird `Tests/LibWeb` | ~8.6k text (4.8k imported from WPT), ~1.1k layout, ~1.9k ref | `internals` |

- Roughly 60–75% of these files use plain web API; `testRunner`, `eventSender` and
  `SpecialPowers.pushPrefEnv` shim easily, most of `internals` cannot.
- Beyond WPT, what is new is the Khronos and WebGPU suites plus about 3–5k media and audio
  tests.
- Testharness baselines record known failures, not truth: negative labels.
- **2,058 text baselines are byte-identical between Chromium and WebKit** (of 6,840 shared
  paths): a two-engine agreement set with expected output in hand, mostly `fast`,
  `editing`, `http` and `svg`.

## The single-engine frontier

- 72,014 WPT subtests pass in exactly one stable engine (Chrome-only 41,132, Firefox-only
  20,834, Safari-only 10,048) when a missing run counts as a fail. `crates/wpt-votes`
  requires all three to have run and gets 54,421 (31,515 / 15,951 / 6,955); README cites
  that (`docs/wpt-votes.md` → "Reconciling the frontier count").
- Experimental channels rescue 17.1% (9,309 of 54,421) of that frontier
  (`docs/wpt-votes.md` → "Summary"); a hand estimate for the Chrome-only slice alone was
  8.5%.
- Polyfills and reference implementations give a real second vote in places: whatwg-url,
  parse5, jsdom, the Temporal polyfill with test262, and polyfills for anchor positioning,
  popover and container queries.
- Waiting works: the single-engine share of a year's shipped entries falls from about 62%
  to about 33% after two years.

## What stays unoracled, and why

Adaptive real-time behaviour (congestion control, echo cancellation, rate control);
implementation-defined numerics (compressor, HRTF, anti-aliasing, smoothing quality);
capability answers (GPU limits, `isConfigSupported`, voice lists); wall-clock and
compositor behaviour (frame cadence, A/V sync, realtime latency); browser UI; vendor
infrastructure (push services, DRM, recognition accuracy); real hardware beyond Linux
virtual devices.

## Moves, by value per effort

1. **wpt.fyi vote table** joined to compat data. Done: `crates/wpt-votes`.
2. **WPT canvas, WPT webaudio and Khronos WebGL in the browser pool.** Done:
   `docs/conformance-oracles.md`.
3. **Trace dump mode and launch profiles.** Done for hand-written probes and the
   `default`/`media` profiles; CDP-only emulation and suite-mined inputs open
   (`docs/dump-format.md` → "Not yet supported"). Seed with the 2,058 identical baselines.
4. **A dawn.node WebGPU CTS runner.** Done: `docs/webgpu-cts.md`.
5. Linux virtual devices in a VM, only if device parity is needed. Not landed.

## How WPT may be used

WPT supplies test inputs; expected outputs come from live-browser runs, never from WPT's
assertions or reference pages. A feature's own WPT pages are case input for generating
that feature, because they already exist and need no model to write them: the pages are
selected by `oracle/suites/feature-pages.mjs` and measured in the three browsers by
`pages_ground_truth_cache` (289 `aspect-ratio` pages give 40 cases in about 7 s). One page
in five, chosen by a hash of its file name, is a fixed held-out partition used only for
scoring, so an exam of pages that generation has not seen remains. The rule, the two
commands and the drop rules: `docs/geometry-verification.md` → "Cases from a feature's
existing WPT pages".

## Sources

- https://github.com/KhronosGroup/WebGL/tree/main/sdk/tests
- https://github.com/gpuweb/cts, https://dawn.googlesource.com/dawn/+/HEAD/src/dawn/node/README.md
- https://www.doc.ic.ac.uk/~afd/papers/2025/ECOOP-WebGlitch.pdf, https://github.com/google/graphicsfuzz
- https://groups.google.com/a/chromium.org/g/blink-dev/c/yhFguWS_3pM (SwiftShader fallback)
- https://github.com/gpuweb/gpuweb/issues/6331, https://github.com/fluendo/fluster
- https://web-platform-tests.org/writing-tests/testdriver.html
- https://developer.mozilla.org/en-US/docs/Web/WebDriver/Reference/BiDi/Modules/bluetooth
- https://chromedevtools.github.io/devtools-protocol/tot/BluetoothEmulation/
- https://immersive-web.github.io/webxr-test-api/
- https://github.com/web-platform-tests/wpt/tree/master/resources/chromium
- https://bugzilla.mozilla.org/show_bug.cgi?id=1846574, https://github.com/xairy/raw-gadget
- https://wpt.fyi/api/runs?label=master&label=stable&aligned
- https://chromium.googlesource.com/chromium/src/+/HEAD/docs/testing/writing_web_tests.md
- https://github.com/openwebdocs/mdn-bcd-collector
- https://github.com/microsoft/playwright/issues/32577, https://github.com/microsoft/playwright/issues/31185