burrito.space / bez
This repository has no description
Star
Fork 0
Watch
bez docs js-engine-choice.md
12 kB
Markdown
at main

How much of WPT needs no script, and which JavaScript engine to link #

Answers two open questions behind roadmap.md "Later phases": how far the engine can get through WPT before any JavaScript engine is linked, and which engine to link when one is. Measured on 2026-10-19 at WPT commit 26f417a8c09fbfcc0b4fcf55760f31090ef2b772 (the wpt.commit pinned in oracle/suites/suites.json). Every number below is in docs/js-engine-choice.json; crates/harness/tests/js_engine_choice.rs checks that the engine rows match it and that the WPT counts add up.

Recommendation #

Link nothing yet. 40.1% of WPT test files outside test262 run with no script, and 66.6% of css does: 25350 test files, most of them reftests. The layout work has that many tests to pass before a script engine is the limit. html (6.4%) and dom (0.0%) are where script becomes necessary.

When an engine is linked, link Boa first. It is the only candidate written in Rust: it builds with no C or C++ compiler and downloads no prebuilt library, so a build on any lane is the same build from crates.io sources; it builds for both WebAssembly targets the engine cares about (wasm32-unknown-unknown needs boa_engine's js feature); and at 95.6% of test262 it trails V8 and SpiderMonkey by two to three points, about half of them in intl402, while QuickJS-NG passes 83.5%. Its cost is size: 12.2 MiB over an empty Rust binary, against QuickJS-NG's 1.2 MiB.

What would change it:

  • The WebAssembly build has to stay free of JavaScript glue. crates/webengine builds a plain wasm32-unknown-unknown module with no wasm-bindgen. Boa's js feature reaches the platform's random source through wasm-bindgen imports. If supplying getrandom another way (its custom backend) does not work, QuickJS-NG builds for that target with a wasm-capable clang and no glue, and becomes the first choice.
  • Script speed starts to dominate WPT run time. Boa is a bytecode interpreter with no JIT; QuickJS-NG is a small, quick-starting interpreter. Execution speed was not measured here. A WPT run that is slow because of script points to QuickJS-NG, and a real-site workload to V8 or SpiderMonkey.
  • Binary size gets a budget. At 12.2 MiB, Boa is ten times QuickJS-NG; a budget under that points to QuickJS-NG.
  • Conformance at the top end matters more than a Rust-only build. SpiderMonkey (98.5%) and V8 (97.7%) lead, but both are C++ linked through a prebuilt static library downloaded at build time (or a from-source build of a browser engine), neither builds for either WebAssembly target, and they add 33.2 MiB and 45.6 MiB.
  • QuickJS-NG gains Temporal. Its gap to Boa is mostly the Temporal tests (0 of 6715 pass) and intl402; without intl402 it passes 89.1%.

Method #

WPT without script #

oracle/suites/wpt-script-free.mjs reads a blobless clone of WPT at the pinned commit in a temporary directory (git fetch --depth 1 --filter=blob:none, then git checkout), builds WPT's own manifest there (./wpt manifest), and classifies every test file the manifest lists. It refuses a clone at any other commit and never writes to oracle/suites/checkouts/ or oracle/suites/suites.json.

node oracle/suites/wpt-script-free.mjs /tmp/wpt-clone > counts.json

The rule, the same text as at the top of the script:

  • A test file is a path the manifest lists under a test type. Support files are not tests. A file with several variants (?query, .any.js globals) counts once.
  • testharness, test262, wdspec and aamtest files always need script: their verdict comes from script in the page or from a driver that runs script in it.
  • reftest, print-reftest, crashtest, visual and manual files need script when the test file, or any reference it names (followed through references that name their own references), contains a <script> element or an inline event-handler attribute (onload=, onclick=, ...), or carries the reftest-wait or test-wait class. Text inside <!-- --> comments is ignored. A reference that is not on disk (one a server handler generates) is counted as needing script, since nothing shows it can run without.
  • Everything else about a file is ignored: a frame or image it loads is not followed, so a script-free file whose iframe document has script counts as script-free.

The last point makes the script-free counts an upper bound. One reference was missing on disk at this commit and counts as needing script. Manual tests are counted like the other scanned types, but they need a person to judge them; the 1577 script-free manual files are not reachable by automation either way.

test262 is ECMAScript's own conformance suite, vendored into WPT under third_party/. It measures the JavaScript engine, not the browser engine, so the browser-side fraction is also given without it.

Engines #

Each engine has a standalone probe crate under experiments/js-engine-probe/<engine>/ with its own empty [workspace] table and Cargo.lock, so cargo test --workspace never builds it. Each probe registers one host function written in Rust (hostAdd), evaluates hostAdd(40, 2) and prints the result. experiments/js-engine-probe/empty/ is the same program with no engine, the size baseline.

experiments/js-engine-probe/measure-all.sh <scratch-dir> measures them one after another on the development box (16 cores, rustc 1.97.1). For each probe, experiments/js-engine-probe/measure.sh:

  • fetches crate sources first, so no build counts the crates.io download;
  • builds with cargo build --release --locked from an empty CARGO_TARGET_DIR under the scratch directory, with the compiler cache off (RUSTC_WRAPPER cleared, which also stops cc-rs from using it);
  • records the size of the release binary, stripped by the probe's profile (strip = true), and subtracts the empty probe's size;
  • keeps the lines of build-script output that mention a download, to tell whether the build fetched a prebuilt engine library;
  • tries wasm32-unknown-unknown and wasm32-wasip1 release builds and keeps their logs.

Build times come from experiments/js-engine-probe/time-builds.sh <scratch-dir>: three cold release builds per probe, one at a time, each from a fresh empty CARGO_TARGET_DIR with the compiler cache off, and each started only once the one-minute load average was below 2 (it was between 1.56 and 1.87 at every start). The tables give the median of the three; the samples and start loads are in the JSON.

test262 pass rates are not run here. They come from test262.fyi (https://data.test262.fyi/index.json, per-feature counts from https://data.test262.fyi/meta.json), generated 2026-10-19 against test262 commit d53ebc8. test262.fyi runs newer engine builds than the crates measured (Boa at git 9871775, QuickJS-NG at 081d50c, V8 15.8.21, SpiderMonkey 160.0a1), so a crate's own pass rate can be a little lower. rquickjs bundles QuickJS-NG, not Bellard's QuickJS, so its row uses test262.fyi's qjs_ng.

WPT without script #

Directory Test files Need script No script No script, share
all of WPT 122073 94603 27470 22.5%
all of WPT without test262 68428 40958 27470 40.1%
third_party 53625 53625 0 0.0%
css 38049 12699 25350 66.6%
html 10044 9397 647 6.4%
svg 1887 998 889 47.1%
referrer-policy 1392 1391 1 0.1%
content-security-policy 879 877 2 0.2%
dom 752 752 0 0.0%
webdriver 641 641 0 0.0%
mathml 610 392 218 35.7%
fetch 495 485 10 2.0%
navigation-api 453 453 0 0.0%
wasm 408 408 0 0.0%
mixed-content 388 388 0 0.0%
editing 347 347 0 0.0%
shadow-dom 346 342 4 1.2%
xhr 332 332 0 0.0%
webvtt 323 323 0 0.0%
webaudio 307 307 0 0.0%
service-workers 302 299 3 1.0%
core-aam 277 275 2 0.7%
scroll-animations 276 270 6 2.2%
infrastructure 184 138 46 25.0%
encoding 157 153 4 2.5%
html-ruby-extensions 84 0 84 100.0%
preload 71 70 1 1.4%
forced-colors-mode 63 30 33 52.4%
jpegxl 61 20 41 67.2%
accessibility 58 44 14 24.1%
webauthn 56 55 1 1.8%
quirks 51 26 25 49.0%
png 44 33 11 25.0%
compat 38 17 21 55.3%
html-longdesc 23 1 22 95.7%
xml 23 18 5 21.7%
density-size-correction 22 2 20 90.9%
entries-api 18 17 1 5.6%
html-media-capture 15 12 3 20.0%
content-dpr 6 2 4 66.7%
contenteditable 4 3 1 25.0%
avif 2 1 1 50.0%

The table lists every top-level directory with at least one script-free test file or at least 300 test files. The other 229 directories (8960 test files) have no script-free test file; their counts are in docs/js-engine-choice.json under wpt.directories.

By test type, across all of WPT:

Type Test files Need script No script No script, share
aamtest 98 98 0 0.0%
crashtest 2005 1276 729 36.4%
manual 3047 1470 1577 51.8%
print-reftest 431 22 409 94.9%
reftest 27939 5805 22134 79.2%
test262 53645 53645 0 0.0%
testharness 31552 31552 0 0.0%
visual 2715 94 2621 96.5%
wdspec 641 641 0 0.0%

Within css, 20713 of 24921 reftests (83.1%) need no script; every one of its 7712 testharness files does.

Engines #

Engine Crate License test262 test262 without intl402 Binary over empty Cold release build Prebuilt download wasm32-unknown-unknown wasm32-wasip1
Boa 0.22.0 boa_engine 0.22.0 MIT OR Unlicense 95.6% 96.7% 12.2 MiB 49.2 s no fails by default; builds with feature js builds
QuickJS-NG 0.16.2 rquickjs 0.14.0 MIT 83.5% 89.1% 1.2 MiB 42.8 s no (the wasm32-wasip1 build downloads WASI SDK 24) fails here (no clang); builds with a wasm clang builds
V8 15.2.124 v8 152.2.0 MIT (bindings), BSD-3-Clause (V8) 97.7% 97.6% 45.6 MiB 26.9 s yes fails fails
SpiderMonkey 153.4.0 mozjs 0.26.9 MPL-2.0 98.5% 98.4% 33.2 MiB 16.2 s yes fails fails

Build times are medians of three cold builds with 16 cores. The empty probe is 366352 bytes stripped and builds in 0.2 s. V8 and SpiderMonkey build fastest only because their engine arrives as a prebuilt static library; building either from source was not timed and takes far longer.

WebAssembly build errors, as recorded in the JSON:

  • Boa, wasm32-unknown-unknown: getrandom 0.4.3 refuses the target by default ("you may need to enable the "wasm_js" crate feature"). With boa_engine's js feature the probe builds, and the module then imports wasm-bindgen glue.
  • QuickJS-NG, wasm32-unknown-unknown: the rquickjs-sys build script needs a clang and the development box has none. With CC_wasm32_unknown_unknown pointed at WASI SDK 24's clang the probe builds: rquickjs-sys vendors a wasm libc for this target.
  • V8, both targets: the build script finds no prebuilt binding for the target (HTTP 404), and V8 has no WebAssembly port.
  • SpiderMonkey, both targets: the first failure is libz-sys needing a clang. mozjs_sys does not support wasm32-unknown-unknown; it supports WASI only through WASI_SDK_PATH and a from-source SpiderMonkey build, which was not attempted.

Reproducing #

# WPT counts (writes MANIFEST.json into the clone on first run)
git init /tmp/wpt-clone
node oracle/suites/mirror.mjs git https://github.com/web-platform-tests/wpt.git \
  -C /tmp/wpt-clone fetch --depth 1 --filter=blob:none '{url}' 26f417a8c09fbfcc0b4fcf55760f31090ef2b772
git -C /tmp/wpt-clone checkout FETCH_HEAD
node oracle/suites/wpt-script-free.mjs /tmp/wpt-clone

# Engine probes (needs the wasm32-unknown-unknown and wasm32-wasip1 Rust targets)
experiments/js-engine-probe/measure-all.sh /tmp/js-engine-probe
# Build times: three cold builds per probe, each waiting for a load average below 2
experiments/js-engine-probe/time-builds.sh /tmp/js-engine-timing