# The wpt.fyi vote table

A per-subtest vote table built from wpt.fyi's published results, joined to
browser-compat-data through WPT's `WEB_FEATURES.yml` files and the `web-features` package,
plus the fixed held-out WPT partition that `roadmap.md` → "WPT changes" reserves for
scoring. Motivation: `docs/oracle-coverage.md` → "wpt.fyi as a ready-made vote table".
Crate: `crates/wpt-votes`.

```mermaid
flowchart LR
  runs["wpt.fyi run listing<br/>8 runs, pinned by revision"] --> raw["raw results<br/>~300 MB each, streamed"]
  raw --> table["table.tsv<br/>one row per test/subtest"]
  wf["WEB_FEATURES.yml<br/>at the run's revision"] --> roll["compat-rollup.tsv"]
  pkg["web-features 3.39.0<br/>data.json"] --> roll
  table --> roll
  table --> area["per-area table"]
```

## Sources and pinned versions

- **Runs**: the latest aligned stable and experimental runs for Chrome, Firefox and Safari
  (`wpt_votes::runs::aligned`), and Servo and Ladybird runs at either revision
  (`wpt_votes::runs::at_revision`). Pinned by WPT revision, browser name and version.
- **Raw results**: each run's `raw_results_url`, cached under `corpus/wpt-votes/downloads/`
  (gitignored) and streamed one test at a time (`wpt_votes::report`), never loaded whole.
- **`WEB_FEATURES.yml`** at each run's exact revision, read from a blobless partial clone
  at `corpus/wpt-votes/wpt-repo` via `git cat-file --batch`.
- **`web-features` `data.json`** pinned to **3.39.0** (`WEB_FEATURES_PACKAGE_VERSION` in
  `crates/wpt-votes/src/bin/join_features.rs`); each feature's `compat_features` lists
  compat keys directly. It does not record which compat-data snapshot it was built from.

The run set of 2026-09-21:

| Column | Browser | Version | WPT revision |
|---|---|---|---|
| `chrome_stable` | Chrome | 153.0.8010.52 | `bc93a431dd` |
| `firefox_stable` | Firefox | 156.0 | `bc93a431dd` |
| `safari_stable` | Safari | 27.0 (22625.1.29.11.27) | `bc93a431dd` |
| `chrome_exp` | Chrome | 156.0.8066.0 | `26f417a8c0` |
| `firefox_exp` | Firefox | 158.0a1 | `26f417a8c0` |
| `safari_exp` | Safari | 252 preview | `26f417a8c0` |
| `servo` | Servo | 0.6.0-b5a1f5e6e | `bc93a431dd` |
| `ladybird` | Ladybird | 1.0-4e806b61a1 | `bc93a431dd` |

## Table schema

`corpus/wpt-votes/<wpt revision>/table.tsv`: tab-separated, one row per `(test path,
subtest name)`; `subtest` is empty for tests without subtests. `test` and `subtest` are
escaped (`wpt_votes::tsv`), since subtest names do contain tabs and newlines.

| Column | Meaning |
|---|---|
| `test`, `subtest` | The WPT key, escaped. |
| `held_out` | `1` if this test path falls in the held-out partition (below). |
| `chrome_stable`, `firefox_stable`, `safari_stable`, `chrome_exp`, `firefox_exp`, `safari_exp`, `servo`, `ladybird` | One status code per run, `-` if that engine reported nothing for this key. |
| `pass_count`, `ran_count` | Out of the three *stable* engines only. |
| `majority` | `1` if at least half of the stable engines that ran agree on pass. |
| `outlier` | The one stable engine that disagrees on a 2–1 split, `-` otherwise. |
| `frontier` | `1` if exactly one of the three stable engines passes **and all three ran** — a missing engine is not evidence it would have failed, so it does not count toward or against frontier status. |
| `frontier_engine` | Which engine, when `frontier` is `1`. |
| `rescued` | `1` if a *different* stable engine's experimental channel passes. |

Status codes: `P` pass/OK, `F` fail, `T` timeout, `N` notrun, `E` error, `C` crash, `R`
precondition-failed, `S` skip, `O` anything else, `-` no result.

The raw JSON is never resident; the cross-engine join is (one entry per key, a few hundred
MB for 2.28 million keys). A streaming merge over sorted per-engine files is the fallback
if WPT outgrows that.

## Held-out partition

`wpt_votes::partition::is_held_out(test_path)`: FNV-1a 64-bit hash of the test path,
`hash % 100 < 20`. By test path, so every subtest of a test lands on the same side. FNV-1a
because Rust's `DefaultHasher` is not promised stable across versions, and a partition
that moved would silently invalidate every past score. The hash, the modulus and the 20%
are the definition; changing any invalidates every score computed under the old one.

## Summary: the 2026-09-21 run (revision `bc93a431dd` stable, `26f417a8c0` experimental)

`cargo run --release -p wpt-votes --bin build_table`, over **2,282,301** distinct keys:

- **Stable majority: 2,162,676 (94.8%)** — across all of WPT, mostly well-established
  HTML/DOM, so it answers a different question from `docs/oracle-coverage.md`'s 57% over
  hard areas.
- **Single-engine frontier, all three stable engines having run:** 54,421 (chrome-only
  31,515, firefox-only 15,951, safari-only 6,955).
- **Rescued by experimental:** 9,309 of that frontier (17.1%).

### Reconciling the frontier count with `docs/oracle-coverage.md`

Dropping the "all three ran" requirement reproduces `docs/oracle-coverage.md`'s 72,014
(41,132 / 20,834 / 10,048) exactly. About 10.7% of keys have fewer than three stable
engines reporting; the looser rule counts a missing run as a fail. This crate's stricter
54,421 (`wpt_votes::vote::StableVote::frontier`) means "single-engine, with all three
actually tested" — not a correction of the other figure.

### Compat-key rollup

`cargo run --release -p wpt-votes --bin join_features corpus/wpt-votes/bc93a431dd bc93a431ddd2e301dece2c6346eb23858aa487a8`
writes `corpus/wpt-votes/bc93a431dd/compat-rollup.tsv` (per compat key: subtests,
majority, frontier). **1,765,824 rows (77.4%) land under at least one `web-features`
identifier**; the rest have no covering `WEB_FEATURES.yml` (1,017 such files at this
revision, mostly CSS, canvas and web-audio). The join touches **844 of 1,210
`web-features` identifiers** and **12,217 compat keys**.

### Per-area rollup, against `docs/oracle-coverage.md`'s wpt.fyi table

`wpt_votes::domain` rebuilds `docs/oracle-coverage.md`'s six hand-drawn areas from a fixed
list of WPT directories per area (`html/canvas` special-cased).
`cargo run --release -p wpt-votes --bin domain_table corpus/wpt-votes/bc93a431dd`:

| Area | Subtests | Pass in two or more engines |
|---|---|---|
| Media | 13,380 | 66% |
| Timing | 4,053 | 85% |
| WebRTC, capture, WebTransport | 6,375 | 72% |
| Security and permission UI | 2,388 | 43% |
| GPU and canvas | 25,340 | 39% |
| Devices and sensors | 2,558 | 25% |
| **Total** | **54,094** | **53%** |

The total matches the hand estimate within 0.7% (54,094 against 53,725); individual areas
diverge more (Security 43% against 66%) because the directory lists are close but not
identical. The total is the load-bearing check.

## Negative labels — skipped

Marking subtests Gecko's `testing/web-platform/meta` lists as expected failures is not
implemented; see Next steps.

## Next steps

1. **Negative labels from Gecko's WPT metadata** (`testing/web-platform/meta/**/*.ini`),
   to tell a known Firefox bug from fresh signal.
2. **A streaming merge-join**, if the in-memory index stops being cheap.
3. **Feed the held-out partition and the rollup into the harness**; nothing here is wired
   into `crates/harness` yet.

## How to verify

- `cargo test --workspace` covers the partition rule (stable, within 1% of 20% over
  200,000 paths), vote derivation, the streaming reader and the `WEB_FEATURES.yml`
  inheritance rules.
- `cargo run --release -p wpt-votes --bin list_runs` prints the current run set without
  downloading.
- `build_table`, `join_features` and `domain_table` reproduce the numbers above (they
  drift as WPT moves; the reconciliation should hold in kind). The first run at a new
  revision clones WPT blobless and prefetches every `WEB_FEATURES.yml` blob in one request.
- Spot-check rows of `table.tsv` against wpt.fyi's viewer for the run ids `list_runs`
  prints.