Conformance suites as oracles: canvas, Web Audio and WebGL #
Canvas 2D, WebGL and Web Audio each have a conformance suite with its own pass/fail
harness, so a generated engine can be checked against them without comparing pixels
across engines. What matters is how much of each suite the three shipping engines agree
on: a test all three pass has a verdict that trusts no single engine; two passing is a
majority; one or none says nothing reliable. Measured for WPT html/canvas, WPT
webaudio and the Khronos WebGL suite in Chromium, Firefox and WebKit on a GPU-less Linux
host; the webaudio numbers come from a build lane, whose PulseAudio null sink gives
realtime audio somewhere to play.
Usable oracle coverage #
| suite | tests run | all three pass | usable (≥2 pass) | usable, excluding tentative |
|---|---|---|---|---|
WPT html/canvas |
3,889 | 77.5% | 82.5% | 92.7% of 3,435 |
WPT webaudio |
352 | 52.6% | 79.8% | 88.4% of 302 |
| Khronos WebGL, with dEQP | 3,836 | 96.5% | 99.7% | (no tentative tests) |
- WebGL is the strongest. Its tests check pixels against values the test computes, so software rasterizers agree; nearly every disagreement is a known engine gap.
- Canvas 2D is strong for what has shipped. Paths, transforms, state, shadows, line
styles, images, fills and strokes are 97–100% usable; the unusable part is unshipped
(
layers,filters,draw-element-image). Text sits at 76%. - Web Audio is an oracle for offline rendering, and a fair one for realtime:
OfflineAudioContexttests are 92.7% usable, realtime ones 77.1% with an audio sink present.
None of the three fits "no practical oracle" in docs/platform-economics.md.
By browser-compat-data entry #
oracle/suites/entry-coverage.mjs takes each area's entries from browser-compat-data
8.0.4 by spec_url and counts an entry covered when a usable test names it in its own
source (members, constructors, events; interface members only in tests that also name
the interface; context members only in tests of the right kind). Harness and utility
scripts are not scanned.
| area | entries | mentioned by a usable test | mentioned only by tests without a two-engine verdict | not mentioned |
|---|---|---|---|---|
| Canvas 2D | 270 | 216 (80.0%) | 3 | 51 |
Canvas 2D without PaintRenderingContext2D |
231 | 216 (93.5%) | 3 | 12 |
| WebGL | 495 | 484 (97.8%) | 0 | 11 |
| Web Audio | 257 | 215 (83.7%) | 16 | 26 |
Uncovered: in Canvas 2D, the 39 CSS Paint API context entries (tested elsewhere),
isContextLost, getContextAttributes, transferControlToOffscreen, OffscreenCanvas
context-loss events and seven Path2D methods only exercised on the context; in WebGL,
WebGLContextEvent, drawingBufferColorSpace/Format, webglcontextcreationerror and
three never-named interface objects; in Web Audio, AudioPlaybackStats, AudioParamMap,
the output-device entries and ScriptProcessorNode.bufferSize.
Read entry figures as a range: a mention is not an assertion (upper bound), unscanned helpers and worklet modules hide real coverage (lower bound), and one passing test covers an entry however shallowly. The not-mentioned counts are the firmest figure.
How it was run #
flowchart LR
fetch["fetch.mjs<br/>pinned sparse checkouts"] --> serve["server.mjs<br/>wptserve subset"]
serve --> run["run.mjs<br/>one browser per engine,<br/>contexts reused"]
run --> results["JSON line per<br/>test per engine"]
results --> sum["summarize.mjs,<br/>compare-wptfyi.mjs,<br/>entry-coverage.mjs"]
oracle/suites/run.mjsis separate from the oracle service, so the service's stdout protocol is untouched. Each worker navigates one browser context from test to test (recycled after a crash, a timeout or 100 tests); each result recordsbrowser.version(). Raw results stay out of git.- Suites are pinned by commit in
oracle/suites/suites.json. oracle/suites/server.mjsimplements thewptservesubset these directories need (.any.js/.window.js/.worker.jswrappers,.sub.,.headers, theheader(),status()andsubpipes, an HTTPS twin). Python handlers answer 501 and their tests are taggeduses-py-handler.- WPT results come from a replacement
testharnessreport.js; WebGL results from the suite's ownwebglTestHarnesshook. Reftests are compared by screenshot, honouringfuzzy. testdriver.jstests run:oracle/suites/server.mjsserves atestdriver-vendor.js(WPT ships it empty) that forwards everytest_driver_internalcall to a bindingrun.mjsinstalls on each page, andoracle/suites/testdriver.mjscarriesclick,send_keys,action_sequenceandset_permissionout throughpage.mouse,page.keyboardand the context's permissions. A call it does not implement rejects by name rather than hanging.-manualtests are still skipped by design.- Timeouts follow WPT (10 s, 60 s long) and the WebGL runner (20 s idle, 300 s cap).
| engine | version | mode | WebGL renderer |
|---|---|---|---|
| Chromium | 153.0.8010.12 | headless, --enable-unsafe-swiftshader |
ANGLE on SwiftShader (Vulkan) |
| Firefox | 155.0 | headful on a private Xvfb display | Mesa llvmpipe |
| WebKit | 26.6 | headless, with a GStreamer workaround preloaded (below) | reports "Apple GPU" (the real renderer string is masked) |
Firefox runs headful because headless Firefox on Linux has no GL. Chromium also gets
--force-color-profile=srgb and --disable-lcd-text, or every canvas grid reftest fails
on text antialiasing differences.
WebM with a header-only Vorbis track. A WebGL test video has a Vorbis track with
headers but no packets; GStreamer treats that as fatal and WebKit rejects the whole
element. No GStreamer release avoids it, so suite runs preload
oracle/suites/gst-empty-vorbis.c into WebKit (built by engines.mjs
buildVorbisShim()), which drops that one error. Without a C compiler, 174 video tests
time out. Canvas and Web Audio verdicts are unaffected. oracle/suites/webm-probe.mjs
checks video decoding alone.
Checking the runner against wpt.fyi #
oracle/suites/compare-wptfyi.mjs compares test verdicts with wpt.fyi at the same WPT
commit (wpt.fyi runs newer builds with experimental features on):
| engine | canvas agreement with wpt.fyi | webaudio agreement with wpt.fyi |
|---|---|---|
| Chromium | 3310/3875 (85.4%) | 306/352 (86.9%) |
| Firefox | 3852/3875 (99.4%) | 343/352 (97.4%) |
| WebKit | 3835/3875 (99.0%) | 314/352 (89.2%) |
Chromium's canvas gap is feature flags: with run.mjs --experimental, 346 of its 539
wpt.fyi-only passes pass; 176 more need another Blink flag; the rest are version
differences. Web Audio disagreements are mostly the tentative
media-playback-while-not-visible-permission-policy tests — 36 of Chromium's 46 and 10 of
WebKit's 38 — with realtime tests making up most of the rest: all 9 of Firefox's, 9 of
Chromium's and 15 of WebKit's. WebKit's other 13 run the opposite way, offline tests that
pass here and fail on wpt.fyi's Safari 27.0 against the 26.6 run here.
WPT html/canvas #
WPT commit 26f417a8c09fbfcc0b4fcf55760f31090ef2b772. 3,905 tests, 3,889 run in all three
engines, 16 not run (all -manual).
| engine | pass | fail | timeout | error |
|---|---|---|---|---|
| Chromium | 3266 | 538 | 75 | 10 |
| Firefox | 3162 | 643 | 75 | 9 |
| WebKit | 3086 | 717 | 76 | 10 |
| tests | all three pass | two pass | one passes | none pass |
|---|---|---|---|---|
| all 3,889 | 3013 (77.5%) | 196 (5.0%) | 83 (2.1%) | 597 (15.4%) |
| 3,435 excluding tentative | 2990 (87.0%) | 194 (5.6%) | 78 (2.3%) | 173 (5.0%) |
Two-pass pairs: Chromium–Firefox 129, Chromium–WebKit 61, Firefox–WebKit 6; the lone passer is Chromium in 63 of 83. Testharness tests are 90.2% usable, reftests 33.2%.
The seven tests the testdriver.js bridge recovered are all in draw-element-image and
the-offscreen-canvas, and their verdicts are about the feature under test rather than
about input: only content-drawable-no-hit-test.tentative.html passes, and it passes in
all three engines. The other six fail or time out in all three, every one of them on
canvas.requestPaint or ctx.drawElementImage being absent — the unshipped
draw-element-image proposal — except
offscreencanvas.transferrable.sw.https.window.html, which fails on the service-worker
transfer itself (an error in Chromium, a failure in Firefox and WebKit). The bridge
itself carried every call: pointer moves, double clicks, wheel scrolls and bless.
| area (element and offscreen together) | tests | usable (≥2 pass) |
|---|---|---|
path-objects, shadows, line-styles, transformations, the-canvas-state, canvas-host, drawing-rectangles-to-the-canvas, drawing-images-to-the-canvas, reset |
1,435 | 100% |
canvas-context, color-type, conformance-requirements and other small areas |
84 | 100% |
fill-and-stroke-styles |
738 | 97.4% |
compositing, pixel-manipulation |
520 | 91.7% |
text |
376 | 76.3% |
manual |
401 | 41.1% |
layers |
242 | 13.6% |
filters |
81 | 7.4% |
wide-gamut-canvas |
12 | 25.0% |
offscreen/ (2,329 tests) is 86.5% usable, close to element/. Subtests, over the 5,888
the full run covered: all three pass 71.2%, two 13.6%, one 3.1%, none 12.2%; the seven
recovered tests add seven more, three passing in all three engines and four in none.
Wall-clock, four in flight per engine, all three at once: Chromium 337 s, Firefox 524 s,
WebKit 337 s.
WPT webaudio #
Same commit. All 352 tests run, none skipped: the 4 that need testdriver.js are reached
by the bridge, and the run was taken on a build lane, where a PulseAudio null sink
(docs/infrastructure.md → "Build lane") gives realtime audio somewhere to play. All three
engines report running for a realtime AudioContext there; on a host without a sink
Firefox's stayed suspended.
| engine | pass | fail | timeout | error |
|---|---|---|---|---|
| Chromium | 291 | 54 | 3 | 4 |
| Firefox | 225 | 111 | 14 | 2 |
| WebKit | 257 | 61 | 27 | 7 |
| tests | all three pass | two pass | one passes | none pass |
|---|---|---|---|---|
| all 352 | 185 (52.6%) | 96 (27.3%) | 26 (7.4%) | 45 (12.8%) |
| 302 excluding tentative | 181 (59.9%) | 86 (28.5%) | 26 (8.6%) | 9 (3.0%) |
Firefox is still the outlier (66 of 96 two-pass tests are Chromium and WebKit), clustered
in the-audioparam-interface (26) and the-audioworklet-interface (11). The sink helped it
most: the same suite on a host without one gave Firefox 194 passes and 43 timeouts, against
225 and 14 here.
| tests | count | all three pass | usable (≥2 pass) |
|---|---|---|---|
uses OfflineAudioContext |
193 | 119 | 92.7% |
creates a realtime AudioContext |
96 | 49 | 77.1% |
(Tagged by a source scan in oracle/suites/wpt.mjs; a test can carry both.) The offline
figure is unchanged by the sink, as expected; the realtime one rose from 67.7% and is now a
measurement rather than a floor.
Subtest names embed computed values, so test-level verdicts are the stable key. Wall-clock:
Chromium 111 s, Firefox 115 s, WebKit 145 s.
Khronos WebGL conformance suite #
KhronosGroup/WebGL commit 714857a28445e8f5d8d6ae1c78498578009534d8, sdk/tests, read
through the suite's own 00_test_list.txt rules, enumerated as 1.0.4 (webglVersion=1)
and 2.0.1 (webglVersion=2). One engine at a time, four in flight; timeouts rerun alone
(all six Chromium ones passed, one of Firefox's two, none of WebKit's fourteen).
| engine | pass | fail | timeout |
|---|---|---|---|
| Chromium | 3002 | 10 | 0 |
| Firefox | 2948 | 63 | 1 |
| WebKit | 2963 | 35 | 14 |
| tests | all three pass | two pass | one passes | none pass |
|---|---|---|---|---|
| all 3,012 | 2904 (96.4%) | 96 (3.2%) | 9 (0.3%) | 3 (0.1%) |
| 940 under WebGL 1 | 916 | 20 | 3 | 1 |
| 2,072 under WebGL 2 | 1988 | 76 | 6 | 2 |
99.6% of the suite has a verdict at least two engines agree on. WebKit's timeouts are
OffscreenCanvas in workers and khr-parallel-shader-compile; Firefox's failures are
spread thin (GLSL validation, wide points, ovr_multiview2, 16 GB wasm memory).
Wall-clock on a loaded host: Chromium 1,061 s, Firefox 645 s, WebKit 341 s.
dEQP ports #
824 tests (793 WebGL 2), included with run.mjs --deqp:
| engine | pass | fail |
|---|---|---|
| Chromium | 805 | 19 |
| Firefox | 816 | 8 |
| WebKit | 822 | 2 |
All three pass 796 (96.6%), two 27, one 1, none 0: 99.9% usable. Across both parts, 3,836 tests: 3,700 pass everywhere (96.5%), 3,823 (99.7%) have a two-engine verdict.
What could not run, and why #
-manualcanvas tests (16): skipped by design, and not recoverable by driving input. Each one draws something and asks a person to judge it. Fourteen are generated by WPT'sgentest.pyand come inelement/andoffscreen/pairs: the twoshadowsblur tests (2d.shadow.blur.high,2d.shadow.blur.low) and fivetexttests (2d.text.draw.fill.basic,.fill.maxWidth.large,.fill.rtl,.kern.consistent,.stroke.basic), all of which put a canvas beside a reference PNG for the eye to compare, theoffscreen/half rendering off-DOM with no comparison image at all. The other two are2d.canvas.host.scaled-manual.html, the same eye-comparison for a CSS-scaled canvas, andcanvas_focus_drawFocusIfNeeded_AAPI_001-manual.html, which asks for an accessibility API inspector to watch a bounding rectangle move as focus moves. None of the sixteen waits on a click or a key, so thetestdriver.jsbridge does not reach them; they would need reference images and a pixel comparison, not input.- What the
testdriver.jsbridge does not carry. It implementsclick,send_keys,action_sequence(pointer, key, wheel and null sources) andset_permission, andblessthroughclick; every other call rejects with its own name in the message, so a test that needs one fails rather than hanging. Coordinates and input go to the page's own mouse and keyboard, so a call aimed at a child frame lands in the wrong place: cross-frame and cross-origin driving (test_driver.set_test_context) is left out, along with window management, virtual authenticators and device posture. Withinaction_sequence, a touch or pen pointer has no Playwright device, and a sequence's sources run one after another within a tick rather than together.set_permissionreaches only the permissionscontext.grantPermissionsnames, and which of those an engine takes differs by engine; the rejection says which engine refused. No canvas test callsset_permission, so that path is exercised only byoracle/suites/testdriver.test.mjs. - Realtime audio, residual. The build lanes provide a PulseAudio null sink
(
docs/infrastructure.md→ "Build lane") and the webaudio numbers above were measured there, which closed most of the gap: realtime tests passing on wpt.fyi but failing here fell from 20 to 8 for Firefox and from 16 to 13 for WebKit, with 8 for Chromium. The 29 that remain are not spread evenly — 13 are inthe-audioworklet-interface(module resolution andMessagePort, mostly timeouts), 5 are thethe-mediaelementaudiosourcenode-interfaceCORS tests, which error rather than fail, and 4 areaudiocontext-suspend-resumeandsuspend-with-navigation. The other 7 are singletons across analyser, destination, convolver, context-construction andMediaStreamAudioSourceNoderouting. Whether a sink with a real output clock would move any of them is untested; the cluster shape suggests worklet startup and media loading rather than the sink. - WebKit OffscreenCanvas-in-worker and
khr-parallel-shader-compile: 14 tests hang. - Python handlers: 2 webaudio tests reach one.
- No external check of WebGL verdicts: nothing publishes Khronos results per build.
- Renderer identity: WebKit masks it.
- Versions: Playwright's Chromium 153, Firefox 155, WebKit 26.6; a verdict holds only for the recorded build.
Reproducing #
cd oracle && npm install
node oracle/suites/fetch.mjs wpt
node oracle/suites/fetch.mjs webgl
node oracle/suites/run.mjs canvas --out=oracle/suites/results/canvas.jsonl
node oracle/suites/summarize.mjs --depth=2 oracle/suites/results/canvas.jsonl
node oracle/suites/compare-wptfyi.mjs oracle/suites/results/canvas.jsonl
node oracle/suites/run.mjs webaudio --out=oracle/suites/results/webaudio.jsonl
node oracle/suites/summarize.mjs --depth=2 oracle/suites/results/webaudio.jsonl
node oracle/suites/compare-wptfyi.mjs oracle/suites/results/webaudio.jsonl
for e in chromium firefox webkit; do
node oracle/suites/run.mjs webgl --engines=$e --out=oracle/suites/results/webgl-$e.jsonl
done
node oracle/suites/summarize.mjs --depth=2 oracle/suites/results/webgl-*.jsonl
Entry coverage, with data.json from npm pack @mdn/browser-compat-data@8.0.4:
node oracle/suites/entry-coverage.mjs --suite=canvas --bcd=DATA_JSON --list oracle/suites/results/canvas.jsonl
node oracle/suites/entry-coverage.mjs --suite=webaudio --bcd=DATA_JSON --list oracle/suites/results/webaudio.jsonl
node oracle/suites/entry-coverage.mjs --suite=webgl --bcd=DATA_JSON --list \
oracle/suites/results/webgl-{chromium,firefox,webkit}.jsonl \
oracle/suites/results/webgl-rerun-*.jsonl oracle/suites/results/webgl-deqp-*.jsonl
The webaudio run belongs on a build lane, whose null sink is what lets a realtime
AudioContext leave suspended: bench/remote-lane.sh --pull oracle/suites/results/webaudio.jsonl . node oracle/suites/run.mjs webaudio --out=oracle/suites/results/webaudio.jsonl runs it there and brings the results back.
run.mjs --engines=firefox runs one engine; summarize.mjs merges files, later records
replacing earlier ones, so a subset rerun (--ids=FILE) layers over a full run;
--resume=FILE continues an interrupted run.