# WebGPU CTS as an oracle

Runs the WebGPU conformance test suite (`gpuweb/cts`) headlessly under dawn.node (Dawn's
Node.js binding, no browser) and in Playwright browsers, to measure how much of it can
check a future engine's WebGPU binding and validation layer over a linked GPU library
(Dawn or wgpu). It says nothing about a generated GPU driver; none is planned. Both runs
are complete; `webgpu-cts/aggregate-dawn-node.mjs` and `webgpu-cts/agreement.mjs`
regenerate every number below from the gitignored per-query JSON.

## Oracle coverage estimate

| part | dawn.node gives a verdict | of those, pass | agrees with Chromium | true pass/fail conflicts | estimate |
|---|---:|---:|---:|---:|---|
| Validation (`api,validation` + `shader,validation`) | 88.0% | 99.9% | 78.2% of 91,618 | 3.5% | **~85%, exact** |
| Numeric execution (`shader,execution` + `api,operation`) | 55.8% | 97.7% | 67.5% of the 37% both completed | 1.0% | **~50%, ULP** |

- Most validation disagreement (16,805 of 19,973 cases) is one software adapter
  reporting a format, limit or feature unsupported where the other supports it — a
  capability difference, not a dispute about behaviour.
- Numeric execution is capped by coverage: 63% of cases never completed in Chromium, so
  much of dawn.node's result is trusted on one runtime.
- Both runtimes are Dawn, so agreement confirms Dawn on two backends (lavapipe and
  SwiftShader), not two independent implementations.
- Both figures are below the earlier "~90%" guess, which ignored the suite's own skip
  rate and how much a second runtime can reach in reasonable time.

```mermaid
flowchart LR
  cts["gpuweb/cts<br/>697 spec files"] --> dn["dawn.node on lavapipe<br/>285,540 cases, 30 min"]
  cts --> chr["Chromium on SwiftShader<br/>165,798 cases, ~89 min"]
  dn --> agree["agreement.mjs<br/>per case, by area"]
  chr --> agree
  ff["Firefox, WebKit"] -. "no navigator.gpu" .-> x["not run"]
```

## Pinned versions

- Suite: `gpuweb/cts` commit `81aeeb49e5b15587ec6c1f59b2fdcb4074ffcf63` (2026-09-15),
  cloned into the gitignored `webgpu-cts/cts/`.
- dawn.node: npm `webgpu` 0.6.1 (`dawn-gpu/node-webgpu`, gitHead
  `f2585d1b386b0f8df63b3d080131e35ab448a9bb`); the npm version is the reproducibility pin.
- Playwright 1.50.0.
- Host: 16 cores, no GPU, Mesa 25.2.8-0ubuntu0.24.04.2, LLVM 20.1.2 (lavapipe). Driver
  chosen explicitly with `VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.json`, never
  the loader default.

## How to reproduce

```sh
cd webgpu-cts
npm install
git clone https://github.com/gpuweb/cts.git
git -C cts checkout 81aeeb49e5b15587ec6c1f59b2fdcb4074ffcf63
VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.json node adapter-info.cjs   # confirms lavapipe
./run-dawn-node.sh        # full suite, sharded across nproc cores, resumable
node aggregate-dawn-node.mjs   # pass/fail/skip tallies, overall and by area

# Chromium's GPU process needs /dev/dri/renderD128 access; run its steps
# through `sg render -c '...'` if the calling shell isn't already in that group.
sg render -c './run-browser-shards.sh results/browser-chromium'
node agreement.mjs dawn-node browser-chromium   # per-case cross-runtime agreement, by area
```

- `gpu-provider.cjs` wraps dawn.node as the suite's `GPUProviderModule` and installs its
  IDL constructors on `globalThis`, because the suite's `instanceof GPUDevice` checks
  expect them there (without it every test throws `ReferenceError`).
- One CTS query per spec file (`list-queries.sh`), one process per query, so a slow or
  crashing file stalls only itself; `run-shard.sh` records finished queries in
  `OUT_DIR/done.txt` so a restart redoes only what did not finish.
- `browser-run-queries.cjs` relaunches a shard's browser on a timeout and retries once;
  without that, one hung Chromium wedged its whole shard.

## Agreement: dawn.node on lavapipe

16 shards, `LP_NUM_THREADS=1` each: 30 minutes 22 seconds, 697 queries, 285,540 cases
(about 157 cases/s), dominated by a few huge texture files.

| Area | Total cases | Pass | Fail | Skip |
|---|---|---|---|---|
| shader,execution | 179,233 | 95,900 | 2,360 | 80,973 |
| api,validation | 44,083 | 33,159 | 66 | 10,858 |
| shader,validation | 47,535 | 47,363 | 1 | 171 |
| api,operation | 10,695 | 7,591 | 38 | 3,066 |
| web_platform,copyToTexture | 2,522 | 0 | 0 | 2,522 |
| web_platform,external_texture | 162 | 0 | 0 | 162 |
| web_platform,canvas | 490 | 0 | 1 | 489 |
| compat,api | 608 | 497 | 0 | 111 |
| idl,javascript | 94 | 74 | 20 | 0 |
| util,texture | 57 | 39 | 6 | 12 |
| idl,constants | 28 | 28 | 0 | 0 |
| idl,constructable | 8 | 8 | 0 | 0 |
| examples | 21 | 18 | 0 | 3 |
| web_platform,worker | 3 | 0 | 0 | 3 |
| print_environment | 1 | 1 | 0 | 0 |
| **Total** | **285,540** | **184,678** | **2,492** | **98,370** |

`fail` is dawn.node failing its own test (0.87%, mostly numeric-tolerance cases in
`shader,execution`). Skips are feature-gated cases the suite correctly excludes. 20
result files needed recovery because dawn.node prints shader-compiler warnings to stdout
before its JSON; the aggregator takes the last `{"version": ...}` object.

## Agreement: dawn.node vs. Chromium

Chromium ran in two 12-shard passes (46 minutes, then 43 minutes over 124 errored
queries, recovering 113). It has verdicts for 165,798 cases; the missing 119,742 are six
files that never completed.

| Area | Total | Agree | Disagree | Only dawn.node |
|---|---|---|---|---|
| api,operation | 10,695 | 8,662 | 1,769 | 264 |
| api,validation | 44,083 | 32,399 | 11,684 | 0 |
| compat,api | 608 | 515 | 93 | 0 |
| examples | 21 | 20 | 1 | 0 |
| idl,constants | 28 | 27 | 1 | 0 |
| idl,constructable | 8 | 8 | 0 | 0 |
| idl,javascript | 94 | 75 | 19 | 0 |
| print_environment | 1 | 1 | 0 | 0 |
| shader,execution | 179,233 | 38,691 | 21,064 | 119,478 |
| shader,validation | 47,535 | 39,246 | 8,289 | 0 |
| util,texture | 57 | 55 | 2 | 0 |
| web_platform,canvas | 490 | 14 | 476 | 0 |
| web_platform,copyToTexture | 2,522 | 456 | 2,066 | 0 |
| web_platform,external_texture | 162 | 60 | 102 | 0 |
| web_platform,worker | 3 | 0 | 3 | 0 |
| **Total** | **285,540** | **120,229** | **45,569** | **119,742** |

| Change (dawn.node → Chromium) | Count | Meaning |
|---|---|---|
| pass → skip | 26,908 | SwiftShader reports unsupported what lavapipe supports |
| skip → pass | 9,174 | the reverse |
| skip → fail | 4,328 | a case lavapipe excludes runs and fails on Chromium |
| pass → fail | 4,874 | both run it; Chromium fails |
| fail → pass | 261 | both run it; dawn.node fails |
| fail → skip | 24 | Chromium excludes a case dawn.node fails |

89% of disagreements involve a skip — a capability gap between adapters. The 5,135 true
conflicts concentrate in texture-sampling builtins (`textureGatherCompare`,
`textureSampleGrad`, `textureSampleLevel`) by depth format and filter mode, not yet
root-caused.

## Browsers

`webgpu-cts/probe-browser-webgpu.cjs` checks `navigator.gpu` in each Playwright browser:

- **Chromium 133.0.6943.16** — works, but reports `vendor: google, architecture:
  swiftshader`, its own bundled adapter, never the host's lavapipe. Adding the account to
  the `render` group and every flag tried (`--use-vulkan=native`, `--use-angle=vulkan`,
  `--ignore-gpu-blocklist`, and others) changed nothing. Likely, unconfirmed: Chromium's
  Dawn excludes CPU Vulkan adapters by policy. So the browser run is
  dawn.node-vs.-SwiftShader.
- **Firefox 134.0** — `navigator.gpu` undefined even with `dom.webgpu.enabled`: "WebGPU is
  not yet available in Release or Beta builds."
- **WebKit 18.2** — `navigator.gpu` undefined.

## What could not run, and why

- **Firefox and WebKit runs**: no WebGPU in these Playwright builds.
- **A browser on lavapipe**: Chromium only reaches SwiftShader.
- **5 spec files in both runtimes**, rejected by the suite's loader as matching no cases
  (a suite-level query/path mismatch): `api,operation,command_buffer,render,dynamic_state`,
  `api,operation,compute_pipeline,entry_point_name`, `api,validation,buffer,threading`,
  `api,validation,encoding,cmds,render_pass`, `api,operation,rendering,robust_access_index`.
- **6 files that never complete in Chromium**, timing out twice at 600 s:
  `api,operation,command_buffer,queries,timestampQuery`,
  `api,operation,command_buffer,queries,occlusionQuery`, and the `textureSample`,
  `textureSampleGrad`, `textureSampleLevel` and `textureGather` builtins. All complete
  under dawn.node; whether SwiftShader hangs or is just slow is not distinguished.