From ceb9c3637184f1e32aefc653ab4b8d9a3c7230d9 Mon Sep 17 00:00:00 2001 From: Pierre Le Fevre Date: Fri, 29 May 2026 19:59:29 +0200 Subject: [PATCH] e2e: document GPU-dependent golden re-baseline workflow, widen we-golden budgets (isu issue 317) Since the software rasterizer was removed, e2e renders through the production Metal renderer offscreen, so self-referential (we-rendered) goldens are now GPU/driver/OS dependent and drift by a thin band of antialiased edge pixels across drivers. - Add crates/e2e/GOLDENS.md: documents the two golden kinds, why we-rendered goldens drift, the tolerance/max-diff-pct knobs, the re-baseline workflow, and the CI pinning policy. - Widen the two self-referential goldens (border_radius, scaled_image) from the 0.1% default differing-pixel budget to 0.5% with inline rationale. On dev hardware border_radius differs by ~0.06%, leaving little headroom; a real regression changes fills or many edge pixels and stays well above 0.5%. - Link the doc from CLAUDE.md's E2E Harness section. Closes isu issue 317. Co-Authored-By: Claude Opus 4.8 --- .isu/issues.json | 2 +- CLAUDE.md | 6 ++ crates/e2e/GOLDENS.md | 110 ++++++++++++++++++++++++++ crates/e2e/scenarios/border_radius.we | 9 ++- crates/e2e/scenarios/scaled_image.we | 8 +- 5 files changed, 132 insertions(+), 3 deletions(-) create mode 100644 crates/e2e/GOLDENS.md diff --git a/.isu/issues.json b/.isu/issues.json index 7cbe17c..f67d04d 100644 --- a/.isu/issues.json +++ b/.isu/issues.json @@ -3860,7 +3860,7 @@ ], "assigned": [], "author": "piefev", - "state": "open", + "state": "closed", "created_at": "2026-05-29T15:20:25Z" } ] diff --git a/CLAUDE.md b/CLAUDE.md index 7a5ba72..081b036 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -226,6 +226,12 @@ down and starts a fresh one. Output paths are resolved relative to `--out-dir` (default `.`). +Golden PNGs come in two kinds — self-referential (`we`-rendered) and +Chromium-reference. Because e2e now renders through the production Metal +renderer, self-referential goldens are GPU/driver/OS dependent; the tolerance +knobs, the re-baseline workflow, and the CI pinning policy live in +[`crates/e2e/GOLDENS.md`](crates/e2e/GOLDENS.md). + ### Smoke suite `crates/e2e/scenarios/smoke.we` covers basic HTML, CSS box model, typography, diff --git a/crates/e2e/GOLDENS.md b/crates/e2e/GOLDENS.md new file mode 100644 index 0000000..c653abc --- /dev/null +++ b/crates/e2e/GOLDENS.md @@ -0,0 +1,110 @@ +# E2E golden screenshots + +This is the reference for the golden PNGs the e2e harness asserts against with +`assert_screenshot_matches`. It explains the two kinds of golden, why one of +them is now GPU/driver/OS dependent, and how to re-baseline each kind. + +Context: the software rasterizer was removed, so e2e renders through the +production Metal renderer offscreen (`GpuRenderer::render_to_pixels`, +offscreen render + readback). Tests now exercise the real rendering path, but +that also means a `we`-rendered golden is no longer bit-exact across machines +(isu issue 317). + +## Two kinds of golden + +| Kind | Files | Reference | What it proves | +|---|---|---|---| +| **Self-referential (`we`-rendered)** | `crates/e2e/scenarios/*.expected.png` (e.g. `45_border_radius.expected.png`, `46_scaled_image.expected.png`) | a previous `we` render | `we`'s render is stable vs. its own committed baseline (no regression) | +| **Chromium reference** | `crates/e2e/scenarios/real-web/..chromium.expected.png` | a Chromium capture | `we` renders the page like Chromium does (cross-engine parity) | + +The Chromium-reference workflow (capture, `--write-golden`, re-baselining via +`--real-web-online`) is documented in +[`real-web/README.md`](real-web/README.md) and is **not** repeated here. This +file covers the self-referential `we`-rendered goldens. + +## Why `we`-rendered goldens are GPU/driver/OS dependent + +There is a single renderer. With the software rasterizer gone, the golden is +whatever the Metal GPU on the capturing machine produced. Output can drift by a +small number of pixels across: + +- different GPUs (Apple M1 vs M2 vs M3, …), +- different macOS / Metal driver versions, +- antialiasing paths that depend on sub-pixel rounding: analytic SDF / 4x MSAA + rounded-rect and border edges (isu issues 312, 313) and the linear+mipmap + image sampler (isu issue 311). + +The drift is confined to a thin band of edge pixels; fills and geometry do not +move. A genuine regression, by contrast, changes a fill colour or moves an +edge by many pixels, so it stays far above the tolerance budget. + +## The tolerance mitigation + +`assert_screenshot_matches` does not require a bit-exact match. A pixel counts +as *differing* only when some RGBA channel differs by more than `--tolerance` +(absolute per-channel, default `4`); the assertion fails only when the +*fraction* of differing pixels exceeds `--max-diff-pct` (percent of total +pixels, default `0.1`). See `src/screenshot_diff.rs`. + +Two knobs, two jobs: + +- **`--tolerance`** absorbs small per-channel colour drift (gamma/blend + rounding). Keep it small (`4`) so colour regressions are still caught. +- **`--max-diff-pct`** absorbs a thin antialiased edge band drifting across + drivers. Widen this — not `--tolerance` — for AA-heavy self-referential + goldens. + +The self-referential scenarios set an explicit `--max-diff-pct 0.5` (vs the +`0.1` default) for this reason; the rationale is commented inline in each +`.we`. As a calibration point, `border_radius` differs by ~0.06% of pixels on +an Apple-silicon dev machine, so `0.5%` leaves ~8x headroom for cross-driver +drift while a real regression (changed fill / moved edge) is many multiples +above it. + +If a self-referential golden starts failing on a new machine purely from edge +drift (the `.diff.png` shows only a thin magenta outline, no filled regions), +prefer nudging that scenario's `--max-diff-pct` over re-baselining — that keeps +the committed golden meaningful on the original hardware. + +## Re-baselining a `we`-rendered golden + +Do this only when the render *intentionally* changed (you changed the renderer, +the page fixture, or the UA stylesheet) — never to paper over an unexplained +diff. The harness writes the live render to `--out-dir`; re-baselining is just +copying that render over the committed `*.expected.png`. + +```sh +# 1. Render the scenario; the screenshot lands in --out-dir. +cargo run -p we-e2e -- --scenario crates/e2e/scenarios/border_radius.we \ + --out-dir crates/e2e/artifacts + +# 2. Eyeball the render AND the magenta diff before trusting it. +open crates/e2e/artifacts/45_border_radius.png +open crates/e2e/artifacts/45_border_radius.png.diff.png # present on failure + +# 3. If the new render is correct, copy it over the committed golden. +cp crates/e2e/artifacts/45_border_radius.png \ + crates/e2e/scenarios/45_border_radius.expected.png + +# 4. Re-run to confirm it now passes, then commit the new golden. +cargo run -p we-e2e -- --scenario crates/e2e/scenarios/border_radius.we \ + --out-dir crates/e2e/artifacts +``` + +The golden's filename and location come from the scenario's +`assert_screenshot_matches ` line: `` +is resolved under `--out-dir`, `` relative to the scenario file. + +## Pinning / refreshing on CI hardware + +Goldens are only meaningful on hardware comparable to where they were captured. +Policy for CI: + +- Commit `we`-rendered goldens from one canonical capture host (record which in + the commit message), and run the asserting scenarios on matching CI hardware. +- When CI hardware or the macOS/Metal version changes, expect a one-time edge + drift. Re-baseline the affected self-referential goldens in a dedicated + commit (render → eyeball diff → `cp` → re-run), keeping it separate from any + behavioural change so the diff is reviewable. +- If drift recurs across heterogeneous runners rather than as a one-time shift, + widen that scenario's `--max-diff-pct` instead of chasing a moving golden. diff --git a/crates/e2e/scenarios/border_radius.we b/crates/e2e/scenarios/border_radius.we index 12512eb..a5ec189 100644 --- a/crates/e2e/scenarios/border_radius.we +++ b/crates/e2e/scenarios/border_radius.we @@ -1,7 +1,14 @@ # CSS border-radius rendering: rounded backgrounds, capsule borders, circles, # per-corner radii, and a rounded text input. Guards the border-radius support # the HTML/CSS browser chrome depends on. +# +# This golden is self-referential (we-rendered) and therefore GPU/driver/OS +# dependent — see crates/e2e/GOLDENS.md (isu issue 317). The analytic SDF/MSAA +# antialiased edges drift by a handful of sub-pixel-rounded pixels across Metal +# drivers, so the differing-pixel budget is widened to 0.5% (was 0.1%) to +# absorb that drift while still catching real regressions, which change fills +# or many edge pixels at once. viewport 320 360 goto crates/e2e/pages/45_border_radius.html screenshot 45_border_radius.png -assert_screenshot_matches 45_border_radius.png 45_border_radius.expected.png +assert_screenshot_matches 45_border_radius.png 45_border_radius.expected.png --max-diff-pct 0.5 diff --git a/crates/e2e/scenarios/scaled_image.we b/crates/e2e/scenarios/scaled_image.we index f31a324..f051538 100644 --- a/crates/e2e/scenarios/scaled_image.we +++ b/crates/e2e/scenarios/scaled_image.we @@ -2,9 +2,15 @@ # 64x64 fine checkerboard minified to 24px. Locks the linear+mipmap image # sampling path (isu issue 311) so a regression back to a nearest-neighbor # sampler (blocky upscale / aliased downscale) is caught by the screenshot diff. +# +# This golden is self-referential (we-rendered) and therefore GPU/driver/OS +# dependent — see crates/e2e/GOLDENS.md (isu issue 317). The linear+mipmap +# sampler's rounding can shift a few edge pixels across Metal drivers, so the +# differing-pixel budget is widened to 0.5% (was 0.1%). A nearest-neighbor +# regression repaints whole tiles and stays well above that budget. viewport 160 160 goto crates/e2e/pages/46_scaled_image.html screenshot 46_scaled_image.png dump_dom 46_scaled_image.dom.txt assert_dom_contains "checkerboard" -assert_screenshot_matches 46_scaled_image.png 46_scaled_image.expected.png +assert_screenshot_matches 46_scaled_image.png 46_scaled_image.expected.png --max-diff-pct 0.5 -- 2.51.2