burrito.space / bez
This repository has no description
Star
Fork 0
Watch
bez docs webgpu-cts.md
8.5 kB
Markdown
at track/substrate-composition

WebGPU CTS as an oracle #

Runs the WebGPU conformance test suite (gpuweb/cts) headlessly under dawn.node (Dawn's Node.js binding, no browser) and in Playwright browsers, to measure how much of it can check a future engine's WebGPU binding and validation layer over a linked GPU library (Dawn or wgpu). It says nothing about a generated GPU driver; none is planned. Both runs are complete; webgpu-cts/aggregate-dawn-node.mjs and webgpu-cts/agreement.mjs regenerate every number below from the gitignored per-query JSON.

Oracle coverage estimate #

part dawn.node gives a verdict of those, pass agrees with Chromium true pass/fail conflicts estimate
Validation (api,validation + shader,validation) 88.0% 99.9% 78.2% of 91,618 3.5% ~85%, exact
Numeric execution (shader,execution + api,operation) 55.8% 97.7% 67.5% of the 37% both completed 1.0% ~50%, ULP
  • Most validation disagreement (16,805 of 19,973 cases) is one software adapter reporting a format, limit or feature unsupported where the other supports it — a capability difference, not a dispute about behaviour.
  • Numeric execution is capped by coverage: 63% of cases never completed in Chromium, so much of dawn.node's result is trusted on one runtime.
  • Both runtimes are Dawn, so agreement confirms Dawn on two backends (lavapipe and SwiftShader), not two independent implementations.
  • Both figures are below the earlier "~90%" guess, which ignored the suite's own skip rate and how much a second runtime can reach in reasonable time.
flowchart LR
  cts["gpuweb/cts<br/>697 spec files"] --> dn["dawn.node on lavapipe<br/>285,540 cases, 30 min"]
  cts --> chr["Chromium on SwiftShader<br/>165,798 cases, ~89 min"]
  dn --> agree["agreement.mjs<br/>per case, by area"]
  chr --> agree
  ff["Firefox, WebKit"] -. "no navigator.gpu" .-> x["not run"]

Pinned versions #

  • Suite: gpuweb/cts commit 81aeeb49e5b15587ec6c1f59b2fdcb4074ffcf63 (2026-09-15), cloned into the gitignored webgpu-cts/cts/.
  • dawn.node: npm webgpu 0.6.1 (dawn-gpu/node-webgpu, gitHead f2585d1b386b0f8df63b3d080131e35ab448a9bb); the npm version is the reproducibility pin.
  • Playwright 1.50.0.
  • Host: 16 cores, no GPU, Mesa 25.2.8-0ubuntu0.24.04.2, LLVM 20.1.2 (lavapipe). Driver chosen explicitly with VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.json, never the loader default.

How to reproduce #

cd webgpu-cts
npm install
git clone https://github.com/gpuweb/cts.git
git -C cts checkout 81aeeb49e5b15587ec6c1f59b2fdcb4074ffcf63
VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.json node adapter-info.cjs   # confirms lavapipe
./run-dawn-node.sh        # full suite, sharded across nproc cores, resumable
node aggregate-dawn-node.mjs   # pass/fail/skip tallies, overall and by area

# Chromium's GPU process needs /dev/dri/renderD128 access; run its steps
# through `sg render -c '...'` if the calling shell isn't already in that group.
sg render -c './run-browser-shards.sh results/browser-chromium'
node agreement.mjs dawn-node browser-chromium   # per-case cross-runtime agreement, by area
  • gpu-provider.cjs wraps dawn.node as the suite's GPUProviderModule and installs its IDL constructors on globalThis, because the suite's instanceof GPUDevice checks expect them there (without it every test throws ReferenceError).
  • One CTS query per spec file (list-queries.sh), one process per query, so a slow or crashing file stalls only itself; run-shard.sh records finished queries in OUT_DIR/done.txt so a restart redoes only what did not finish.
  • browser-run-queries.cjs relaunches a shard's browser on a timeout and retries once; without that, one hung Chromium wedged its whole shard.

Agreement: dawn.node on lavapipe #

16 shards, LP_NUM_THREADS=1 each: 30 minutes 22 seconds, 697 queries, 285,540 cases (about 157 cases/s), dominated by a few huge texture files.

Area Total cases Pass Fail Skip
shader,execution 179,233 95,900 2,360 80,973
api,validation 44,083 33,159 66 10,858
shader,validation 47,535 47,363 1 171
api,operation 10,695 7,591 38 3,066
web_platform,copyToTexture 2,522 0 0 2,522
web_platform,external_texture 162 0 0 162
web_platform,canvas 490 0 1 489
compat,api 608 497 0 111
idl,javascript 94 74 20 0
util,texture 57 39 6 12
idl,constants 28 28 0 0
idl,constructable 8 8 0 0
examples 21 18 0 3
web_platform,worker 3 0 0 3
print_environment 1 1 0 0
Total 285,540 184,678 2,492 98,370

fail is dawn.node failing its own test (0.87%, mostly numeric-tolerance cases in shader,execution). Skips are feature-gated cases the suite correctly excludes. 20 result files needed recovery because dawn.node prints shader-compiler warnings to stdout before its JSON; the aggregator takes the last {"version": ...} object.

Agreement: dawn.node vs. Chromium #

Chromium ran in two 12-shard passes (46 minutes, then 43 minutes over 124 errored queries, recovering 113). It has verdicts for 165,798 cases; the missing 119,742 are six files that never completed.

Area Total Agree Disagree Only dawn.node
api,operation 10,695 8,662 1,769 264
api,validation 44,083 32,399 11,684 0
compat,api 608 515 93 0
examples 21 20 1 0
idl,constants 28 27 1 0
idl,constructable 8 8 0 0
idl,javascript 94 75 19 0
print_environment 1 1 0 0
shader,execution 179,233 38,691 21,064 119,478
shader,validation 47,535 39,246 8,289 0
util,texture 57 55 2 0
web_platform,canvas 490 14 476 0
web_platform,copyToTexture 2,522 456 2,066 0
web_platform,external_texture 162 60 102 0
web_platform,worker 3 0 3 0
Total 285,540 120,229 45,569 119,742
Change (dawn.node → Chromium) Count Meaning
pass → skip 26,908 SwiftShader reports unsupported what lavapipe supports
skip → pass 9,174 the reverse
skip → fail 4,328 a case lavapipe excludes runs and fails on Chromium
pass → fail 4,874 both run it; Chromium fails
fail → pass 261 both run it; dawn.node fails
fail → skip 24 Chromium excludes a case dawn.node fails

89% of disagreements involve a skip — a capability gap between adapters. The 5,135 true conflicts concentrate in texture-sampling builtins (textureGatherCompare, textureSampleGrad, textureSampleLevel) by depth format and filter mode, not yet root-caused.

Browsers #

webgpu-cts/probe-browser-webgpu.cjs checks navigator.gpu in each Playwright browser:

  • Chromium 133.0.6943.16 — works, but reports vendor: google, architecture: swiftshader, its own bundled adapter, never the host's lavapipe. Adding the account to the render group and every flag tried (--use-vulkan=native, --use-angle=vulkan, --ignore-gpu-blocklist, and others) changed nothing. Likely, unconfirmed: Chromium's Dawn excludes CPU Vulkan adapters by policy. So the browser run is dawn.node-vs.-SwiftShader.
  • Firefox 134.0 — navigator.gpu undefined even with dom.webgpu.enabled: "WebGPU is not yet available in Release or Beta builds."
  • WebKit 18.2 — navigator.gpu undefined.

What could not run, and why #

  • Firefox and WebKit runs: no WebGPU in these Playwright builds.
  • A browser on lavapipe: Chromium only reaches SwiftShader.
  • 5 spec files in both runtimes, rejected by the suite's loader as matching no cases (a suite-level query/path mismatch): api,operation,command_buffer,render,dynamic_state, api,operation,compute_pipeline,entry_point_name, api,validation,buffer,threading, api,validation,encoding,cmds,render_pass, api,operation,rendering,robust_access_index.
  • 6 files that never complete in Chromium, timing out twice at 600 s: api,operation,command_buffer,queries,timestampQuery, api,operation,command_buffer,queries,occlusionQuery, and the textureSample, textureSampleGrad, textureSampleLevel and textureGather builtins. All complete under dawn.node; whether SwiftShader hangs or is just slow is not distinguished.