The wpt.fyi vote table #

A per-subtest vote table built from wpt.fyi's published results, joined to browser-compat-data through WPT's WEB_FEATURES.yml files and the web-features package, plus the fixed held-out WPT partition that roadmap.md → "WPT changes" reserves for scoring. Motivation: docs/oracle-coverage.md → "wpt.fyi as a ready-made vote table". Crate: crates/wpt-votes.

flowchart LR
  runs["wpt.fyi run listing<br/>8 runs, pinned by revision"] --> raw["raw results<br/>~300 MB each, streamed"]
  raw --> table["table.tsv<br/>one row per test/subtest"]
  wf["WEB_FEATURES.yml<br/>at the run's revision"] --> roll["compat-rollup.tsv"]
  pkg["web-features 3.39.0<br/>data.json"] --> roll
  table --> roll
  table --> area["per-area table"]

Sources and pinned versions #

• Runs: the latest aligned stable and experimental runs for Chrome, Firefox and Safari (wpt_votes::runs::aligned), and Servo and Ladybird runs at either revision (wpt_votes::runs::at_revision). Pinned by WPT revision, browser name and version.
• Raw results: each run's raw_results_url, cached under corpus/wpt-votes/downloads/ (gitignored) and streamed one test at a time (wpt_votes::report), never loaded whole.
• WEB_FEATURES.yml at each run's exact revision, read from a blobless partial clone at corpus/wpt-votes/wpt-repo via git cat-file --batch.
• web-features data.json pinned to 3.39.0 (WEB_FEATURES_PACKAGE_VERSION in crates/wpt-votes/src/bin/join_features.rs); each feature's compat_features lists compat keys directly. It does not record which compat-data snapshot it was built from.

The run set of 2026-09-21:

Table schema #

corpus/wpt-votes/<wpt revision>/table.tsv: tab-separated, one row per (test path, subtest name); subtest is empty for tests without subtests. test and subtest are escaped (wpt_votes::tsv), since subtest names do contain tabs and newlines.

Status codes: P pass/OK, F fail, T timeout, N notrun, E error, C crash, R precondition-failed, S skip, O anything else, - no result.

The raw JSON is never resident; the cross-engine join is (one entry per key, a few hundred MB for 2.28 million keys). A streaming merge over sorted per-engine files is the fallback if WPT outgrows that.

Held-out partition #

wpt_votes::partition::is_held_out(test_path): FNV-1a 64-bit hash of the test path, hash % 100 < 20. By test path, so every subtest of a test lands on the same side. FNV-1a because Rust's DefaultHasher is not promised stable across versions, and a partition that moved would silently invalidate every past score. The hash, the modulus and the 20% are the definition; changing any invalidates every score computed under the old one.

Summary: the 2026-09-21 run (revision bc93a431dd stable, 26f417a8c0 experimental) #

cargo run --release -p wpt-votes --bin build_table, over 2,282,301 distinct keys:

• Stable majority: 2,162,676 (94.8%) — across all of WPT, mostly well-established HTML/DOM, so it answers a different question from docs/oracle-coverage.md's 57% over hard areas.
• Single-engine frontier, all three stable engines having run: 54,421 (chrome-only 31,515, firefox-only 15,951, safari-only 6,955).
• Rescued by experimental: 9,309 of that frontier (17.1%).

Reconciling the frontier count with docs/oracle-coverage.md #

Dropping the "all three ran" requirement reproduces docs/oracle-coverage.md's 72,014 (41,132 / 20,834 / 10,048) exactly. About 10.7% of keys have fewer than three stable engines reporting; the looser rule counts a missing run as a fail. This crate's stricter 54,421 (wpt_votes::vote::StableVote::frontier) means "single-engine, with all three actually tested" — not a correction of the other figure.

Compat-key rollup #

cargo run --release -p wpt-votes --bin join_features corpus/wpt-votes/bc93a431dd bc93a431ddd2e301dece2c6346eb23858aa487a8 writes corpus/wpt-votes/bc93a431dd/compat-rollup.tsv (per compat key: subtests, majority, frontier). 1,765,824 rows (77.4%) land under at least one web-features identifier; the rest have no covering WEB_FEATURES.yml (1,017 such files at this revision, mostly CSS, canvas and web-audio). The join touches 844 of 1,210 web-features identifiers and 12,217 compat keys.

Per-area rollup, against docs/oracle-coverage.md's wpt.fyi table #

wpt_votes::domain rebuilds docs/oracle-coverage.md's six hand-drawn areas from a fixed list of WPT directories per area (html/canvas special-cased). cargo run --release -p wpt-votes --bin domain_table corpus/wpt-votes/bc93a431dd:

The total matches the hand estimate within 0.7% (54,094 against 53,725); individual areas diverge more (Security 43% against 66%) because the directory lists are close but not identical. The total is the load-bearing check.

Negative labels — skipped #

Marking subtests Gecko's testing/web-platform/meta lists as expected failures is not implemented; see Next steps.

Next steps #

1. Negative labels from Gecko's WPT metadata (testing/web-platform/meta/**/*.ini), to tell a known Firefox bug from fresh signal.
2. A streaming merge-join, if the in-memory index stops being cheap.
3. Feed the held-out partition and the rollup into the harness; nothing here is wired into crates/harness yet.

How to verify #

• cargo test --workspace covers the partition rule (stable, within 1% of 20% over 200,000 paths), vote derivation, the streaming reader and the WEB_FEATURES.yml inheritance rules.
• cargo run --release -p wpt-votes --bin list_runs prints the current run set without downloading.
• build_table, join_features and domain_table reproduce the numbers above (they drift as WPT moves; the reconciliation should hold in kind). The first run at a new revision clones WPT blobless and prefetches every WEB_FEATURES.yml blob in one request.
• Spot-check rows of table.tsv against wpt.fyi's viewer for the run ids list_runs prints.