# Conformance suites as oracles: canvas, Web Audio and WebGL

Canvas 2D, WebGL and Web Audio each have a conformance suite with its own pass/fail
harness, so a generated engine can be checked against them without comparing pixels
across engines. What matters is how much of each suite the three shipping engines agree
on: a test all three pass has a verdict that trusts no single engine; two passing is a
majority; one or none says nothing reliable. Measured for WPT `html/canvas`, WPT
`webaudio` and the Khronos WebGL suite in Chromium, Firefox and WebKit on a GPU-less Linux
host; the `webaudio` numbers come from a build lane, whose PulseAudio null sink gives
realtime audio somewhere to play.

## Usable oracle coverage

| suite | tests run | all three pass | usable (≥2 pass) | usable, excluding tentative |
|---|---:|---:|---:|---:|
| WPT `html/canvas` | 3,889 | 77.5% | 82.5% | 92.7% of 3,435 |
| WPT `webaudio` | 352 | 52.6% | 79.8% | 88.4% of 302 |
| Khronos WebGL, with dEQP | 3,836 | 96.5% | 99.7% | (no tentative tests) |

- **WebGL is the strongest.** Its tests check pixels against values the test computes,
  so software rasterizers agree; nearly every disagreement is a known engine gap.
- **Canvas 2D is strong for what has shipped.** Paths, transforms, state, shadows, line
  styles, images, fills and strokes are 97–100% usable; the unusable part is unshipped
  (`layers`, `filters`, `draw-element-image`). Text sits at 76%.
- **Web Audio is an oracle for offline rendering, and a fair one for realtime**:
  `OfflineAudioContext` tests are 92.7% usable, realtime ones 77.1% with an audio sink
  present.

None of the three fits "no practical oracle" in `docs/platform-economics.md`.

### By browser-compat-data entry

`oracle/suites/entry-coverage.mjs` takes each area's entries from browser-compat-data
8.0.4 by `spec_url` and counts an entry covered when a usable test names it in its own
source (members, constructors, events; interface members only in tests that also name
the interface; context members only in tests of the right kind). Harness and utility
scripts are not scanned.

| area | entries | mentioned by a usable test | mentioned only by tests without a two-engine verdict | not mentioned |
|---|---:|---:|---:|---:|
| Canvas 2D | 270 | 216 (80.0%) | 3 | 51 |
| Canvas 2D without `PaintRenderingContext2D` | 231 | 216 (93.5%) | 3 | 12 |
| WebGL | 495 | 484 (97.8%) | 0 | 11 |
| Web Audio | 257 | 215 (83.7%) | 16 | 26 |

Uncovered: in Canvas 2D, the 39 CSS Paint API context entries (tested elsewhere),
`isContextLost`, `getContextAttributes`, `transferControlToOffscreen`, OffscreenCanvas
context-loss events and seven `Path2D` methods only exercised on the context; in WebGL,
`WebGLContextEvent`, `drawingBufferColorSpace`/`Format`, `webglcontextcreationerror` and
three never-named interface objects; in Web Audio, `AudioPlaybackStats`, `AudioParamMap`,
the output-device entries and `ScriptProcessorNode.bufferSize`.

Read entry figures as a range: a mention is not an assertion (upper bound), unscanned
helpers and worklet modules hide real coverage (lower bound), and one passing test covers
an entry however shallowly. The not-mentioned counts are the firmest figure.

## How it was run

```mermaid
flowchart LR
  fetch["fetch.mjs<br/>pinned sparse checkouts"] --> serve["server.mjs<br/>wptserve subset"]
  serve --> run["run.mjs<br/>one browser per engine,<br/>contexts reused"]
  run --> results["JSON line per<br/>test per engine"]
  results --> sum["summarize.mjs,<br/>compare-wptfyi.mjs,<br/>entry-coverage.mjs"]
```

- `oracle/suites/run.mjs` is separate from the oracle service, so the service's stdout
  protocol is untouched. Each worker navigates one browser context from test to test
  (recycled after a crash, a timeout or 100 tests); each result records
  `browser.version()`. Raw results stay out of git.
- Suites are pinned by commit in `oracle/suites/suites.json`.
- `oracle/suites/server.mjs` implements the `wptserve` subset these directories need
  (`.any.js`/`.window.js`/`.worker.js` wrappers, `.sub.`, `.headers`, the `header()`,
  `status()` and `sub` pipes, an HTTPS twin). Python handlers answer 501 and their tests
  are tagged `uses-py-handler`.
- WPT results come from a replacement `testharnessreport.js`; WebGL results from the
  suite's own `webglTestHarness` hook. Reftests are compared by screenshot, honouring
  `fuzzy`.
- `testdriver.js` tests run: `oracle/suites/server.mjs` serves a `testdriver-vendor.js` (WPT
  ships it empty) that forwards every `test_driver_internal` call to a binding
  `run.mjs` installs on each page, and `oracle/suites/testdriver.mjs` carries `click`,
  `send_keys`, `action_sequence` and `set_permission` out through `page.mouse`,
  `page.keyboard` and the context's permissions. A call it does not implement rejects by
  name rather than hanging. `-manual` tests are still skipped by design.
- Timeouts follow WPT (10 s, 60 s long) and the WebGL runner (20 s idle, 300 s cap).

| engine | version | mode | WebGL renderer |
|---|---|---|---|
| Chromium | 153.0.8010.12 | headless, `--enable-unsafe-swiftshader` | ANGLE on SwiftShader (Vulkan) |
| Firefox | 155.0 | headful on a private Xvfb display | Mesa llvmpipe |
| WebKit | 26.6 | headless, with a GStreamer workaround preloaded (below) | reports "Apple GPU" (the real renderer string is masked) |

Firefox runs headful because headless Firefox on Linux has no GL. Chromium also gets
`--force-color-profile=srgb` and `--disable-lcd-text`, or every canvas grid reftest fails
on text antialiasing differences.

**WebM with a header-only Vorbis track.** A WebGL test video has a Vorbis track with
headers but no packets; GStreamer treats that as fatal and WebKit rejects the whole
element. No GStreamer release avoids it, so suite runs preload
`oracle/suites/gst-empty-vorbis.c` into WebKit (built by `engines.mjs`
`buildVorbisShim()`), which drops that one error. Without a C compiler, 174 video tests
time out. Canvas and Web Audio verdicts are unaffected. `oracle/suites/webm-probe.mjs`
checks video decoding alone.

### Checking the runner against wpt.fyi

`oracle/suites/compare-wptfyi.mjs` compares test verdicts with wpt.fyi at the same WPT
commit (wpt.fyi runs newer builds with experimental features on):

| engine | canvas agreement with wpt.fyi | webaudio agreement with wpt.fyi |
|---|---:|---:|
| Chromium | 3310/3875 (85.4%) | 306/352 (86.9%) |
| Firefox | 3852/3875 (99.4%) | 343/352 (97.4%) |
| WebKit | 3835/3875 (99.0%) | 314/352 (89.2%) |

Chromium's canvas gap is feature flags: with `run.mjs --experimental`, 346 of its 539
wpt.fyi-only passes pass; 176 more need another Blink flag; the rest are version
differences. Web Audio disagreements are mostly the tentative
`media-playback-while-not-visible-permission-policy` tests — 36 of Chromium's 46 and 10 of
WebKit's 38 — with realtime tests making up most of the rest: all 9 of Firefox's, 9 of
Chromium's and 15 of WebKit's. WebKit's other 13 run the opposite way, offline tests that
pass here and fail on wpt.fyi's Safari 27.0 against the 26.6 run here.

## WPT html/canvas

WPT commit `26f417a8c09fbfcc0b4fcf55760f31090ef2b772`. 3,905 tests, 3,889 run in all three
engines, 16 not run (all `-manual`).

| engine | pass | fail | timeout | error |
|---|---:|---:|---:|---:|
| Chromium | 3266 | 538 | 75 | 10 |
| Firefox | 3162 | 643 | 75 | 9 |
| WebKit | 3086 | 717 | 76 | 10 |

| tests | all three pass | two pass | one passes | none pass |
|---|---:|---:|---:|---:|
| all 3,889 | 3013 (77.5%) | 196 (5.0%) | 83 (2.1%) | 597 (15.4%) |
| 3,435 excluding tentative | 2990 (87.0%) | 194 (5.6%) | 78 (2.3%) | 173 (5.0%) |

Two-pass pairs: Chromium–Firefox 129, Chromium–WebKit 61, Firefox–WebKit 6; the lone
passer is Chromium in 63 of 83. Testharness tests are 90.2% usable, reftests 33.2%.

The seven tests the `testdriver.js` bridge recovered are all in `draw-element-image` and
`the-offscreen-canvas`, and their verdicts are about the feature under test rather than
about input: only `content-drawable-no-hit-test.tentative.html` passes, and it passes in
all three engines. The other six fail or time out in all three, every one of them on
`canvas.requestPaint` or `ctx.drawElementImage` being absent — the unshipped
`draw-element-image` proposal — except
`offscreencanvas.transferrable.sw.https.window.html`, which fails on the service-worker
transfer itself (an error in Chromium, a failure in Firefox and WebKit). The bridge
itself carried every call: pointer moves, double clicks, wheel scrolls and `bless`.

| area (element and offscreen together) | tests | usable (≥2 pass) |
|---|---:|---:|
| `path-objects`, `shadows`, `line-styles`, `transformations`, `the-canvas-state`, `canvas-host`, `drawing-rectangles-to-the-canvas`, `drawing-images-to-the-canvas`, `reset` | 1,435 | 100% |
| `canvas-context`, `color-type`, `conformance-requirements` and other small areas | 84 | 100% |
| `fill-and-stroke-styles` | 738 | 97.4% |
| `compositing`, `pixel-manipulation` | 520 | 91.7% |
| `text` | 376 | 76.3% |
| `manual` | 401 | 41.1% |
| `layers` | 242 | 13.6% |
| `filters` | 81 | 7.4% |
| `wide-gamut-canvas` | 12 | 25.0% |

`offscreen/` (2,329 tests) is 86.5% usable, close to `element/`. Subtests, over the 5,888
the full run covered: all three pass 71.2%, two 13.6%, one 3.1%, none 12.2%; the seven
recovered tests add seven more, three passing in all three engines and four in none.
Wall-clock, four in flight per engine, all three at once: Chromium 337 s, Firefox 524 s,
WebKit 337 s.

## WPT webaudio

Same commit. All 352 tests run, none skipped: the 4 that need `testdriver.js` are reached
by the bridge, and the run was taken on a build lane, where a PulseAudio null sink
(`docs/infrastructure.md` → "Build lane") gives realtime audio somewhere to play. All three
engines report `running` for a realtime `AudioContext` there; on a host without a sink
Firefox's stayed `suspended`.

| engine | pass | fail | timeout | error |
|---|---:|---:|---:|---:|
| Chromium | 291 | 54 | 3 | 4 |
| Firefox | 225 | 111 | 14 | 2 |
| WebKit | 257 | 61 | 27 | 7 |

| tests | all three pass | two pass | one passes | none pass |
|---|---:|---:|---:|---:|
| all 352 | 185 (52.6%) | 96 (27.3%) | 26 (7.4%) | 45 (12.8%) |
| 302 excluding tentative | 181 (59.9%) | 86 (28.5%) | 26 (8.6%) | 9 (3.0%) |

Firefox is still the outlier (66 of 96 two-pass tests are Chromium and WebKit), clustered
in `the-audioparam-interface` (26) and `the-audioworklet-interface` (11). The sink helped it
most: the same suite on a host without one gave Firefox 194 passes and 43 timeouts, against
225 and 14 here.

| tests | count | all three pass | usable (≥2 pass) |
|---|---:|---:|---:|
| uses `OfflineAudioContext` | 193 | 119 | 92.7% |
| creates a realtime `AudioContext` | 96 | 49 | 77.1% |

(Tagged by a source scan in `oracle/suites/wpt.mjs`; a test can carry both.) The offline
figure is unchanged by the sink, as expected; the realtime one rose from 67.7% and is now a
measurement rather than a floor.
Subtest names embed computed values, so test-level verdicts are the stable key. Wall-clock:
Chromium 111 s, Firefox 115 s, WebKit 145 s.

## Khronos WebGL conformance suite

KhronosGroup/WebGL commit `714857a28445e8f5d8d6ae1c78498578009534d8`, `sdk/tests`, read
through the suite's own `00_test_list.txt` rules, enumerated as 1.0.4 (`webglVersion=1`)
and 2.0.1 (`webglVersion=2`). One engine at a time, four in flight; timeouts rerun alone
(all six Chromium ones passed, one of Firefox's two, none of WebKit's fourteen).

| engine | pass | fail | timeout |
|---|---:|---:|---:|
| Chromium | 3002 | 10 | 0 |
| Firefox | 2948 | 63 | 1 |
| WebKit | 2963 | 35 | 14 |

| tests | all three pass | two pass | one passes | none pass |
|---|---:|---:|---:|---:|
| all 3,012 | 2904 (96.4%) | 96 (3.2%) | 9 (0.3%) | 3 (0.1%) |
| 940 under WebGL 1 | 916 | 20 | 3 | 1 |
| 2,072 under WebGL 2 | 1988 | 76 | 6 | 2 |

99.6% of the suite has a verdict at least two engines agree on. WebKit's timeouts are
OffscreenCanvas in workers and `khr-parallel-shader-compile`; Firefox's failures are
spread thin (GLSL validation, wide points, `ovr_multiview2`, 16 GB wasm memory).
Wall-clock on a loaded host: Chromium 1,061 s, Firefox 645 s, WebKit 341 s.

### dEQP ports

824 tests (793 WebGL 2), included with `run.mjs --deqp`:

| engine | pass | fail |
|---|---:|---:|
| Chromium | 805 | 19 |
| Firefox | 816 | 8 |
| WebKit | 822 | 2 |

All three pass 796 (96.6%), two 27, one 1, none 0: 99.9% usable. Across both parts,
3,836 tests: 3,700 pass everywhere (96.5%), 3,823 (99.7%) have a two-engine verdict.

## What could not run, and why

- **`-manual` canvas tests** (16): skipped by design, and not recoverable by driving input.
  Each one draws something and asks a person to judge it. Fourteen are generated by WPT's
  `gentest.py` and come in `element/` and `offscreen/` pairs: the two `shadows` blur tests
  (`2d.shadow.blur.high`, `2d.shadow.blur.low`) and five `text` tests
  (`2d.text.draw.fill.basic`, `.fill.maxWidth.large`, `.fill.rtl`, `.kern.consistent`,
  `.stroke.basic`), all of which put a canvas beside a reference PNG for the eye to
  compare, the `offscreen/` half rendering off-DOM with no comparison image at all. The
  other two are `2d.canvas.host.scaled-manual.html`, the same eye-comparison for a
  CSS-scaled canvas, and
  `canvas_focus_drawFocusIfNeeded_AAPI_001-manual.html`, which asks for an accessibility
  API inspector to watch a bounding rectangle move as focus moves. None of the sixteen
  waits on a click or a key, so the `testdriver.js` bridge does not reach them; they would
  need reference images and a pixel comparison, not input.
- **What the `testdriver.js` bridge does not carry.** It implements `click`, `send_keys`,
  `action_sequence` (pointer, key, wheel and null sources) and `set_permission`, and
  `bless` through `click`; every other call rejects with its own name in the message, so a
  test that needs one fails rather than hanging. Coordinates and input go to the page's
  own mouse and keyboard, so a call aimed at a child frame lands in the wrong place:
  cross-frame and cross-origin driving (`test_driver.set_test_context`) is left out, along
  with window management, virtual authenticators and device posture. Within
  `action_sequence`, a touch or pen pointer has no Playwright device, and a sequence's
  sources run one after another within a tick rather than together. `set_permission`
  reaches only the permissions `context.grantPermissions` names, and which of those an
  engine takes differs by engine; the rejection says which engine refused. No canvas test
  calls `set_permission`, so that path is exercised only by
  `oracle/suites/testdriver.test.mjs`.
- **Realtime audio, residual.** The build lanes provide a PulseAudio null sink
  (`docs/infrastructure.md` → "Build lane") and the webaudio numbers above were measured
  there, which closed most of the gap: realtime tests passing on wpt.fyi but failing here
  fell from 20 to 8 for Firefox and from 16 to 13 for WebKit, with 8 for Chromium. The 29
  that remain are not spread evenly — 13 are in `the-audioworklet-interface` (module
  resolution and `MessagePort`, mostly timeouts), 5 are the
  `the-mediaelementaudiosourcenode-interface` CORS tests, which error rather than fail, and
  4 are `audiocontext-suspend-resume` and `suspend-with-navigation`. The other 7 are
  singletons across analyser, destination, convolver, context-construction and
  `MediaStreamAudioSourceNode` routing. Whether a sink with a real output clock would move
  any of them is untested; the cluster shape suggests worklet startup and media loading
  rather than the sink.
- **WebKit OffscreenCanvas-in-worker and `khr-parallel-shader-compile`**: 14 tests hang.
- **Python handlers**: 2 webaudio tests reach one.
- **No external check of WebGL verdicts**: nothing publishes Khronos results per build.
- **Renderer identity**: WebKit masks it.
- **Versions**: Playwright's Chromium 153, Firefox 155, WebKit 26.6; a verdict holds only
  for the recorded build.

## Reproducing

```
cd oracle && npm install
node oracle/suites/fetch.mjs wpt
node oracle/suites/fetch.mjs webgl
node oracle/suites/run.mjs canvas --out=oracle/suites/results/canvas.jsonl
node oracle/suites/summarize.mjs --depth=2 oracle/suites/results/canvas.jsonl
node oracle/suites/compare-wptfyi.mjs oracle/suites/results/canvas.jsonl
node oracle/suites/run.mjs webaudio --out=oracle/suites/results/webaudio.jsonl
node oracle/suites/summarize.mjs --depth=2 oracle/suites/results/webaudio.jsonl
node oracle/suites/compare-wptfyi.mjs oracle/suites/results/webaudio.jsonl
for e in chromium firefox webkit; do
  node oracle/suites/run.mjs webgl --engines=$e --out=oracle/suites/results/webgl-$e.jsonl
done
node oracle/suites/summarize.mjs --depth=2 oracle/suites/results/webgl-*.jsonl
```

Entry coverage, with `data.json` from `npm pack @mdn/browser-compat-data@8.0.4`:

```
node oracle/suites/entry-coverage.mjs --suite=canvas --bcd=DATA_JSON --list oracle/suites/results/canvas.jsonl
node oracle/suites/entry-coverage.mjs --suite=webaudio --bcd=DATA_JSON --list oracle/suites/results/webaudio.jsonl
node oracle/suites/entry-coverage.mjs --suite=webgl --bcd=DATA_JSON --list \
  oracle/suites/results/webgl-{chromium,firefox,webkit}.jsonl \
  oracle/suites/results/webgl-rerun-*.jsonl oracle/suites/results/webgl-deqp-*.jsonl
```

The `webaudio` run belongs on a build lane, whose null sink is what lets a realtime
`AudioContext` leave `suspended`: `bench/remote-lane.sh --pull
oracle/suites/results/webaudio.jsonl . node oracle/suites/run.mjs webaudio
--out=oracle/suites/results/webaudio.jsonl` runs it there and brings the results back.

`run.mjs --engines=firefox` runs one engine; `summarize.mjs` merges files, later records
replacing earlier ones, so a subset rerun (`--ids=FILE`) layers over a full run;
`--resume=FILE` continues an interrupted run.