burrito.space / bez
This repository has no description
Star
Fork 0
Watch
bez docs webgpu-cts.md
22 kB
Markdown
at track/rig-throughput

WebGPU CTS as an oracle #

Runs the WebGPU conformance test suite (gpuweb/cts) headlessly, under dawn.node (Dawn's Node.js binding, no browser) and in Playwright browsers, to measure how much of it is usable as an oracle for a future engine's WebGPU binding and validation layer. See docs/oracle-coverage.md → the WebGPU row for the motivation and where this fits among the other domains.

The oracle this produces checks WebGPU's API surface, validation and state machine as a future engine would expose them over a linked GPU layer (Dawn or wgpu). It says nothing about a generated GPU driver; none is planned.

Status: both runs complete. This document has the pinned versions, how to reproduce, dawn.node's full results, the Chromium browser run, and the cross-runtime agreement table between them. Rerun webgpu-cts/aggregate-dawn-node.mjs and webgpu-cts/agreement.mjs to regenerate the numbers below from the raw per-query JSON, which is gitignored.

Pinned versions #

  • Suite: gpuweb/cts commit 81aeeb49e5b15587ec6c1f59b2fdcb4074ffcf63 (2026-09-15), cloned into the gitignored webgpu-cts/cts/.
  • dawn.node: npm package webgpu 0.6.1 (dawn-gpu/node-webgpu, gitHead f2585d1b386b0f8df63b3d080131e35ab448a9bb), its prebuilt Linux x64 binding at webgpu-cts/node_modules/webgpu/dist/linux-x64/dawn.node. The exact upstream Dawn/Chromium git SHA that build embeds was not independently extracted; the npm package version is the reproducibility pin (npm install webgpu@0.6.1 gets the same binary).
  • Playwright: 1.50.0, matching oracle/package.json's pin.
  • Host: 16 cores, Mesa 25.2.8-0ubuntu0.24.04.2, LLVM 20.1.2 (lavapipe/llvmpipe software Vulkan). Adapter reported by dawn.node: vendor: mesa, architecture: software, device: llvmpipe-llvm-20-1-2-256-bits-, description: llvmpipe: Mesa 25.2.8-0ubuntu0.24.04.2 (LLVM 20.1.2).

Host and driver #

The host has no GPU; only software Vulkan drivers are installed (/usr/share/vulkan/icd.d/), alongside ICD files for hardware this host doesn't have. The driver is selected explicitly with VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.json before launching dawn.node or a browser — nothing here trusts a Vulkan loader default.

How to reproduce #

cd webgpu-cts
npm install
git clone https://github.com/gpuweb/cts.git
git -C cts checkout 81aeeb49e5b15587ec6c1f59b2fdcb4074ffcf63
VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.json node adapter-info.cjs   # confirms lavapipe
./run-dawn-node.sh        # full suite, sharded across nproc cores, resumable
node aggregate-dawn-node.mjs   # pass/fail/skip tallies, overall and by area

# Chromium's GPU process needs /dev/dri/renderD128 access; run its steps
# through `sg render -c '...'` if the calling shell isn't already in that
# group (see "Browsers" below for why it's needed and what it does and
# doesn't unlock).
sg render -c './run-browser-shards.sh results/browser-chromium'   # same shape, sharded across Chromium instances
node agreement.mjs dawn-node browser-chromium   # per-case cross-runtime agreement, by area

run-shard.sh derives CTS_DIR and PROVIDER from its own location, so it works from any checkout without editing. gpu-provider.cjs wraps the prebuilt dawn.node as the GPUProviderModule the suite's own tools/run_node --gpu-provider expects; it installs dawn.node's IDL constructors (GPUDevice and friends) onto globalThis before handing control to the runner, because tools/run_node does instanceof GPUDevice checks against those constructors as globals — true in a browser, but dawn.node only returns them from create(). Without that, every test throws ReferenceError: GPUDevice is not defined before it runs.

Query granularity is one CTS query per spec file (list-queries.sh enumerates 697 files as of the pinned commit) rather than hand-picked subtrees, so each shard gets an even, reproducible slice and one process per query means a slow or crashing file only stalls its own query, not the shard. run-shard.sh records each finished query in OUT_DIR/done.txt and skips anything already there, so restarting an interrupted run (same shard count) only redoes what didn't finish.

Throughput #

A single spec file (webgpu:shader,execution,expression,call,builtin,abs:*, 12 cases) ran in about 5 seconds end to end, all passing.

The full sharded dawn.node run (16 shards, one per core, LP_NUM_THREADS=1 per shard so lavapipe's own thread pool doesn't oversubscribe the machine) ran from 2026-09-21T06:30:00+02:00 to 2026-09-21T07:00:22+02:00: 30 minutes 22 seconds for all 697 spec-file queries, expanding to 285,540 individual test cases (about 157 cases/second). The rate per file was highly uneven: most files finish in seconds, but a handful (api,operation,command_buffer,copyTextureToTexture, api,operation,rendering,3d_texture_slices, texture sampling in general) run thousands of parameterized cases each and dominate their shard's wall-clock time — one shard was still on its first-assigned query after more than 5 minutes.

The Chromium run (see "Browsers" below), sharded the same way across parallel browser instances, ran in two passes. The first 12-shard pass (results/browser-chromium/run-manifest.txt) ran from 2026-09-21T07:33:23+02:00 and every shard reported done by 08:19:52+02:00 (about 46 minutes for all 697 queries), but 124 of those queries came back as errors: a hung Chromium instance (page.goto/page.waitForFunction timing out against a dev server that was itself still answering instantly) left the rest of that shard's queries stuck the same way, since the original runner reused one browser for the whole shard and never recovered from a wedged one. webgpu-cts/browser-run-queries.cjs now closes and relaunches a shard's browser on a navigation or result timeout and retries the query once before giving up, so a second, targeted 12-way pass over just the 124 error queries (results/browser-chromium/rerun-manifest.txt, 08:22:51+02:00 to 09:04:34+02:00, about 43 minutes) recovered 113 of them; the remaining 11 are listed in "What could not run, and why" below. Total wall-clock across both passes: about 89 minutes for full coverage of the 697 queries, most of it spent on a handful of large texture-sampling and query files that legitimately (or, for a few, permanently) run close to the 600-second per-query timeout — not representative steady-state throughput, since Chromium/SwiftShader executes most files in seconds like dawn.node does.

Agreement: dawn.node on lavapipe #

Full run, all 697 spec-file queries (aggregate-dawn-node.mjs):

Area Total cases Pass Fail Skip
shader,execution 179,233 95,900 2,360 80,973
api,validation 44,083 33,159 66 10,858
shader,validation 47,535 47,363 1 171
api,operation 10,695 7,591 38 3,066
web_platform,copyToTexture 2,522 0 0 2,522
web_platform,external_texture 162 0 0 162
web_platform,canvas 490 0 1 489
compat,api 608 497 0 111
idl,javascript 94 74 20 0
util,texture 57 39 6 12
idl,constants 28 28 0 0
idl,constructable 8 8 0 0
examples 21 18 0 3
web_platform,worker 3 0 0 3
print_environment 1 1 0 0
Total 285,540 184,678 2,492 98,370

fail here is dawn.node/lavapipe failing its own conformance test, not a cross-runtime disagreement — see "Agreement: dawn.node vs. Chromium" below for that. A 0.87% fail rate against 184,678 passes is dominated by shader,execution (2,360 of the 2,492 total failures, mostly numeric-tolerance cases — plausible for a software rasterizer exercising edge-of-range floating point behavior, not yet root-caused per case); idl,javascript stands out disproportionately at 20 of 94 failing (21%). The large skip counts are mostly feature-gated cases (compressed texture formats, copyToTexture and external_texture external-source tests that need a real video/image pipeline, timestamp queries) that lavapipe or dawn.node report as unsupported and the suite correctly excludes rather than fails.

5 of 697 queries (0.7%) produced no results at all: dawn.node's loader rejected them with "Query ... does not match any cases", even though list-queries.sh's query for each matches that spec file's real path one-for-one. The Chromium run (see "Agreement: dawn.node vs. Chromium" below) hits the identical error on the identical 5 files, ruling out a dawn.node-specific cause; the mismatch is in the suite checkout itself, not yet root-caused further — possibly these files export their test group under a name that doesn't mirror their file path exactly. Affected files: api,operation,command_buffer,render,dynamic_state, api,operation,compute_pipeline,entry_point_name, api,validation,buffer,threading, api,validation,encoding,cmds,render_pass, api,operation,rendering,robust_access_index.

A further 20 of the 692 readable files needed recovery in aggregate-dawn-node.mjs because dawn.node prints shader-compiler warnings ("warning:\nCompilation log for...") straight to stdout ahead of --print-json's own output, corrupting a naive whole-file JSON parse; the aggregator now finds the real results object (always the last {"version": ...} blob dawn.node writes) instead of assuming the file is clean JSON on its own. One query (webgpu:print_environment:*) has the same issue for a different reason: the test itself logs a full JSON environment dump before the runner's own results JSON.

Agreement: dawn.node vs. Chromium #

node agreement.mjs dawn-node browser-chromium joins both runs per case (by full test name, independent of which shard produced it) and reports agree/disagree counts by area. dawn.node has a verdict for all 285,540 cases; Chromium has one for 165,798 — the 119,742 missing are entirely from the 6 files that never completed in Chromium even after one browser relaunch (see "What could not run, and why" below), all shader,execution except one api,operation file.

Area Total Agree Disagree Only dawn.node
api,operation 10,695 8,662 1,769 264
api,validation 44,083 32,399 11,684 0
compat,api 608 515 93 0
examples 21 20 1 0
idl,constants 28 27 1 0
idl,constructable 8 8 0 0
idl,javascript 94 75 19 0
print_environment 1 1 0 0
shader,execution 179,233 38,691 21,064 119,478
shader,validation 47,535 39,246 8,289 0
util,texture 57 55 2 0
web_platform,canvas 490 14 476 0
web_platform,copyToTexture 2,522 456 2,066 0
web_platform,external_texture 162 60 102 0
web_platform,worker 3 0 3 0
Total 285,540 120,229 45,569 119,742

browser-chromium never has a verdict dawn.node lacks (only_browser-chromium is 0 in every area), since dawn.node's run is the complete one.

The 45,569 disagreements are not one thing. Breaking down by the status pair:

Change (dawn.node → Chromium) Count Meaning
pass → skip 26,908 Chromium's SwiftShader adapter reports a feature, format or limit as unsupported that lavapipe supports — the suite correctly skips it there.
skip → pass 9,174 The reverse: lavapipe reports something unsupported that SwiftShader supports.
skip → fail 4,328 A case lavapipe excludes as inapplicable runs on Chromium and fails there.
pass → fail 4,874 Both runtimes attempt the case; Chromium fails where dawn.node passes.
fail → pass 261 Both runtimes attempt the case; Chromium passes where dawn.node fails.
fail → skip 24 Chromium excludes a case dawn.node attempts and fails.

40,410 of the 45,569 (89%) involve a skip on one side — a feature-support gap between lavapipe and Chromium's bundled SwiftShader, not a disagreement about what a supported case's correct behavior is. The remaining 5,135 are true pass/fail conflicts where both runtimes ran the same case to completion and disagreed: 4,874 dawn.node-pass/Chromium-fail and 261 the reverse. The pass/fail conflicts concentrate heavily in shader,execution texture-sampling builtins — textureGatherCompare, textureSampleGrad, textureSampleLevel and similar, varying by depth-texture format and filter mode — consistent with the two software rasterizers differing in sampling precision or comparison-filter behavior at the edges of their tolerance, not yet root-caused per case; api,validation accounts for most of the rest of the non-skip conflicts (3,168 of 91,618 validation cases, 3.5%).

By the split docs/oracle-coverage.md draws between validation and numeric execution (used again in "Oracle coverage estimate" below):

  • Validation (api,validation + shader,validation, exact pass/fail): 91,618 cases, all with a verdict from both runtimes. 78.2% agree; of the 21.8% that disagree, 16,805 (18.3% of the total) are skip-vs-something feature-gap cases and 3,168 (3.5%) are true pass/fail conflicts.
  • Numeric execution (shader,execution + api,operation, ULP-tolerance): 189,928 cases. Only 70,186 (37%) have a verdict from both runtimes — 119,742 (63%) never completed in Chromium. Of the 70,186 with both verdicts, 67.5% agree; 20,886 (11% of the 189,928 total) are skip-vs-something, and 1,947 (1.0%) are true pass/fail conflicts.

Browsers #

webgpu-cts/probe-browser-webgpu.cjs checks whether navigator.gpu initializes in a Playwright browser and what adapter it reports. Results on this host, lavapipe selected via VK_ICD_FILENAMES:

  • Chromium 133.0.6943.16 — navigator.gpu.requestAdapter() succeeds, but reports vendor: google, architecture: swiftshader — Chromium's own bundled software WebGPU implementation, not the host's lavapipe. This was first suspected to be a permissions problem: /dev/dri/renderD128 on this host is owned by group render (crw-rw---- root:render), and the account running these tests was not a member of that group. The account was since added to render (sg render -c '...' grants it for one command without a fresh login, and is what every Chromium invocation in this track uses). The result is unchanged: re-probing under sg render with --use-vulkan=native and the same flags as before still reports vendor: google, architecture: swiftshader. Flags tried across both attempts, none of which changed the outcome: --enable-features=Vulkan, --use-angle=vulkan, --use-gl=angle, --use-vulkan=native, --ignore-gpu-blocklist, --disable-gpu-sandbox, --no-sandbox, and an (undocumented, unconfirmed) --use-webgpu-adapter=vulkan. ANGLE's own verbose log (--enable-logging=stderr --vmodule=*webgpu*=3,*dawn*=3) shows it successfully finding a Vulkan physical device (vulkan_icd.cpp:315 (ChoosePhysicalDevice): Preferred device ICD not found. Using default physicalDevice instead — i.e. it did enumerate lavapipe, just not as a "preferred" device by whatever criteria ANGLE uses), so the Vulkan driver is reachable; nothing in the available logs shows Dawn's own adapter selection reasoning. The most likely remaining explanation, unconfirmed: Chromium's Dawn integration specifically excludes software (VK_PHYSICAL_DEVICE_TYPE_CPU) Vulkan adapters from WebGPU adapter enumeration as a deliberate policy, with its own bundled SwiftShader as the only sanctioned software fallback — separate from the render-node access question, and not something a driver flag or file permission fixes. dawn.node needs no render-node access at all, since llvmpipe/lavapipe is pure CPU rendering with no DRM device involved, which is why it's unaffected by any of this. The suite still runs against Chromium's SwiftShader adapter — see "Agreement: dawn.node vs. Chromium" above and "What could not run, and why" below for what that does and doesn't cover. It is a second adapter, not a second implementation: Chromium's WebGPU is Dawn, the same code dawn.node wraps, so agreement between the two runtimes confirms Dawn against itself on two backends (lavapipe and SwiftShader) rather than against an independent implementation.
  • Firefox 134.0 — navigator.gpu is undefined even with dom.webgpu.enabled: true set as a Firefox user preference; requesting the adapter fails with the browser's own message: "WebGPU is not yet available in Release or Beta builds." This Playwright Firefox build (release/beta channel) does not have WebGPU compiled in.
  • WebKit 18.2 — navigator.gpu is undefined; this Playwright WebKit build does not expose WebGPU.

Chromium is the only one of the three where navigator.gpu initializes at all, so it's the only one the suite runs against (webgpu-cts/run-browser-shards.sh, results under results/browser-chromium/, labeled as Chromium/SwiftShader throughout this document since that's the adapter actually in use, not lavapipe).

What could not run, and why #

  • Firefox and WebKit CTS runs. Neither Playwright build on this host exposes a usable navigator.gpu (see "Browsers" above), and fixing that is outside this track's control — it depends on Playwright shipping a build with WebGPU compiled in.
  • A same-rasterizer browser comparison against lavapipe. Chromium's WebGPU only reaches its own bundled SwiftShader adapter on this host, not lavapipe, for reasons not fully root-caused (see "Browsers" above); the Chromium run in this document is a dawn.node-vs.-SwiftShader comparison, not dawn.node-vs.-lavapipe-in-a-browser.
  • 5 spec files, in both runtimes. Listed in "Agreement: dawn.node on lavapipe" above; the suite's own loader rejects their queries as matching no test cases. The Chromium run hits the exact same 5 files with the identical "does not match any cases" error, which rules out a dawn.node-specific cause — it's a suite-level query/file-path mismatch (not yet root-caused further) that affects any runtime using this suite checkout.
  • 6 spec files that never complete in Chromium. Even after browser-run-queries.cjs closes and relaunches the shard's browser on a timeout and retries once (see "Throughput" above), these 6 time out again on the retry and are recorded as timeouts, not run further: api,operation,command_buffer,queries,timestampQuery, api,operation,command_buffer,queries,occlusionQuery, shader,execution,expression,call,builtin,textureSample, shader,execution,expression,call,builtin,textureSampleGrad, shader,execution,expression,call,builtin,textureSampleLevel, shader,execution,expression,call,builtin,textureGather. All 6 run to completion under dawn.node/lavapipe (see "Agreement: dawn.node vs. Chromium" above for their cases' status there); whether Chromium's SwiftShader adapter genuinely hangs on GPU queries and large texture-sampling parameter sweeps, or is just slow enough to exceed the 600-second per-query timeout twice in a row, is not distinguished here.

Oracle coverage estimate #

Replaces docs/oracle-coverage.md's WebGPU row, which estimated "~90%, exact and ULP" as a judgement call predating this run. The measured split, using the cross-runtime agreement table above and that document's own validation/numeric-execution distinction:

  • Validation (api,validation + shader,validation, exact pass/fail): dawn.node itself produces a verdict (not a feature-gated skip) for 88.0% of cases, and 99.9% of those pass — the API surface and state machine this run can exercise is validated almost without exception. Cross-runtime agreement against Chromium (91,618 cases, all with a verdict from both) is 78.2%; nearly all the gap (16,805 of 19,973 disagreeing cases, 84%) is Chromium's SwiftShader adapter reporting a texture format, limit or feature as unsupported where lavapipe supports it (or vice versa) — a capability difference between the two software adapters, not a validation dispute. True pass/fail conflicts are 3.5% of all validation cases. Net estimate: ~85%, exact (down slightly from the coverage figure alone, to account for that 3.5%).
  • Numeric execution (shader,execution + api,operation, ULP-tolerance): dawn.node produces a verdict for 55.8% of cases (the rest are feature-gated skips — compressed formats, external-texture and timestamp-query paths lavapipe or dawn.node don't support), and 97.7% of those pass. Cross-runtime confirmation is much weaker here: only 37% of numeric-execution cases have a verdict from both runtimes, because the 6 files above never complete in Chromium; of the cases both runtimes did complete, agreement is 67.5%, with true pass/fail conflicts at 1.0% of all numeric cases, concentrated in texture-sampling precision (textureGatherCompare and similar, not yet root-caused per case). Net estimate: ~50%, ULP — closer to dawn.node's own completion rate than to the cross-runtime agreement rate, since the missing Chromium coverage limits how much of that 55.8% is confirmed on a second backend rather than trusted on one runtime alone (both runtimes are Dawn, so this is a second backend, not a second implementation).

Both figures are lower than the prior "~90%, exact and ULP" guess, mainly because the guess didn't account for the suite's own skip rate (cases the suite itself excludes as inapplicable to a given adapter) or for how much of the numeric-execution surface a second runtime can actually reach in a reasonable time budget.