diff --git a/.gitignore b/.gitignore index 3151c6e..e4b8960 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,4 @@ /build /.env /crates/e2e/artifacts/ +__pycache__/ diff --git a/.isu/issues.json b/.isu/issues.json index b81fc03..93e585a 100644 --- a/.isu/issues.json +++ b/.isu/issues.json @@ -3255,7 +3255,7 @@ "labels": [], "assigned": [], "author": "piefev", - "state": "open", + "state": "closed", "created_at": "2026-05-22T12:55:54Z" } ] diff --git a/crates/e2e/real-web/README.md b/crates/e2e/real-web/README.md index 48750ce..5659034 100644 --- a/crates/e2e/real-web/README.md +++ b/crates/e2e/real-web/README.md @@ -52,6 +52,17 @@ re-snapshot it against the live site (`--real-web-online`) and treat any divergence either as a real-web regression (file an issue) or as upstream drift (update snapshot + golden in its own commit). +## Comparing against Chrome and Firefox + +`tests/popular-sites/compare.py` drives Playwright to capture Chromium +and Firefox screenshots for any committed `.we` scenario, alongside +`..expected.png`. The reference images are for human +inspection only — the harness still asserts only against +`.expected.png` — and are gitignored so they never accidentally land +in a commit. See +[`tests/popular-sites/README.md`](../../../tests/popular-sites/README.md) +for the install + invocation steps. + ## Running the suite ```sh diff --git a/crates/e2e/scenarios/real-web/.gitignore b/crates/e2e/scenarios/real-web/.gitignore new file mode 100644 index 0000000..2673923 --- /dev/null +++ b/crates/e2e/scenarios/real-web/.gitignore @@ -0,0 +1,6 @@ +# Comparison screenshots generated by tests/popular-sites/compare.py via +# Playwright. They are reference images for human inspection only — not +# consumed by the e2e harness, and they go stale quickly as live sites +# and browser versions change. Regenerate on demand instead of committing. +*.chromium.png +*.firefox.png diff --git a/tests/popular-sites/README.md b/tests/popular-sites/README.md index 6890710..55d5083 100644 --- a/tests/popular-sites/README.md +++ b/tests/popular-sites/README.md @@ -1,8 +1,13 @@ # popular-sites -The single committed input-side script for the Phase 23 real-web soak (see -`PLAN.md` Phase 23). One invocation of `pick.py` produces one random, -safe-as-we-can-make-it popular URL. +Input-side tooling for the Phase 23 real-web soak (see `PLAN.md` Phase 23): + +* `pick.py` — emits one random, safe-as-we-can-make-it popular URL per + invocation. Pure Python 3 stdlib. +* `compare.py` — drives Playwright to capture Chromium/Firefox reference + screenshots for any committed `.we` scenario. See + [Comparison screenshots](#comparison-screenshots-comparepy) below. + Optional tool; the e2e harness does not depend on it. ## Usage @@ -87,6 +92,74 @@ deterministic — which is the layer where determinism actually matters. ## Dependencies -Python 3 standard library only. No `pip install`, no third-party packages, -no shell-out to system tools. Targets the system Python 3 that ships with -the macOS development image. +`pick.py` uses Python 3 standard library only. No `pip install`, no +third-party packages, no shell-out to system tools. Targets the system +Python 3 that ships with the macOS development image. + +`compare.py` is also pure-stdlib Python, but it shells out to +`npx playwright`. Node.js + npx and pre-installed Playwright browser +binaries are required only when actually capturing comparison +screenshots — see below. + +## Comparison screenshots (`compare.py`) + +`compare.py` walks one or more `.we` scenarios, tracks the current +`viewport` / `goto_as` / `screenshot` state, and shells out to +`npx playwright screenshot` so the same page can be re-rendered in +Chromium and Firefox at the same viewport. The PNGs land next to the +existing golden under `crates/e2e/scenarios/real-web/`: + +``` +crates/e2e/scenarios/real-web/ +├── bing.com.we +├── bing.com.desktop.expected.png # we's render — asserted by harness +├── bing.com.desktop.chromium.png # Playwright Chromium — reference +└── bing.com.desktop.firefox.png # Playwright Firefox — reference +``` + +The reference images are for human inspection only — the e2e harness +still asserts `we` against the committed `.expected.png`. They are +listed in +`crates/e2e/scenarios/real-web/.gitignore` so a captured set never +accidentally lands in a commit; regenerate on demand. + +### Setup + +```sh +# Once per machine. Pulls Chromium and Firefox binaries (~200 MiB). +npx playwright install chromium firefox +``` + +### Capture + +```sh +# Live URLs in both browsers, all committed scenarios. +python3 tests/popular-sites/compare.py --all + +# One scenario, just Chromium. +python3 tests/popular-sites/compare.py --browser chromium \ + crates/e2e/scenarios/real-web/bing.com.we + +# Apples-to-apples with the offline snapshot we feed `we` (mostly blank +# for client-side rendered sites, but useful when our render diverges +# from what other engines do on the same exact bytes). +python3 tests/popular-sites/compare.py --source snapshot \ + crates/e2e/scenarios/real-web/bing.com.we + +# Print the capture plan without invoking Playwright. +python3 tests/popular-sites/compare.py --dry-run --all +``` + +`--source url` (the default) navigates to the live `goto_as` URL with a +fixed settle wait, which is the most useful comparison mode for sites +whose UI is rendered by client-side scripts. `--source snapshot` loads +the committed `file://` snapshot instead. + +### Self-tests + +`test_compare.py` covers the scenario parser. Pure stdlib, no Playwright +required: + +```sh +python3 tests/popular-sites/test_compare.py +``` diff --git a/tests/popular-sites/compare.py b/tests/popular-sites/compare.py new file mode 100755 index 0000000..93202d8 --- /dev/null +++ b/tests/popular-sites/compare.py @@ -0,0 +1,362 @@ +#!/usr/bin/env python3 +"""Capture comparison screenshots of real-web scenarios using Playwright. + +For each ``screenshot `` line in a ``.we`` scenario, this script +re-renders the same page in Chromium and Firefox (via Playwright) at the +same viewport and writes the resulting PNGs next to the existing golden +under +``crates/e2e/scenarios/real-web/...png``. + +Two source modes are supported: + +* ``--source url`` (default) navigates Playwright to the *live* URL of + each ``goto_as`` step and waits a few seconds for the page to settle + before capturing. This is the most useful mode for "what does Chrome + render?" comparisons because the snapshot we feed our own engine is + intentionally a static shell and renders mostly blank in any browser. +* ``--source snapshot`` navigates to ``file://`` of the committed + snapshot. This is apples-to-apples with what the offline ``we`` + scenario sees, but for sites whose UI is rendered client-side, the + output is essentially blank. + +The Playwright outputs are reference images for human inspection (do we +look like Chrome at all?). They are not consumed by the e2e harness; +``assert_screenshot_matches`` still compares ``we``'s render against the +committed ``.expected.png``. + +Playwright is not a ``we`` dependency. The script shells out to +``npx playwright`` so the only runtime requirements are Node.js + npx on +PATH plus the browser binaries (``npx playwright install chromium +firefox`` once per machine). + +Usage:: + + # Capture live-URL comparison shots for one scenario. + python3 tests/popular-sites/compare.py crates/e2e/scenarios/real-web/bing.com.we + + # Capture for every committed real-web scenario. + python3 tests/popular-sites/compare.py --all + + # Apples-to-apples against the offline snapshot. + python3 tests/popular-sites/compare.py --source snapshot + + # Restrict to a single browser. + python3 tests/popular-sites/compare.py --browser chromium + + # Skip files that already have output committed. + python3 tests/popular-sites/compare.py --skip-existing --all +""" + +from __future__ import annotations + +import argparse +import shlex +import shutil +import subprocess +import sys +from pathlib import Path +from typing import Iterable + + +REPO_ROOT = Path(__file__).resolve().parents[2] +DEFAULT_SCENARIOS_DIR = REPO_ROOT / "crates" / "e2e" / "scenarios" / "real-web" +SUPPORTED_BROWSERS = ("chromium", "firefox") + + +def _split_scenario_line(line: str) -> list[str]: + """Tokenize a single scenario line, honoring quoted strings. + + Mirrors what the Rust scenario engine does so the script sees the + same arguments. ``#`` starts a comment. + """ + stripped = line.strip() + if not stripped or stripped.startswith("#"): + return [] + no_inline_comment = stripped.split("#", 1)[0].rstrip() + try: + return shlex.split(no_inline_comment, posix=True) + except ValueError as exc: + raise ValueError(f"could not tokenize scenario line: {line!r}: {exc}") from exc + + +def parse_capture_plan(scenario_path: Path) -> list[dict]: + """Return one capture descriptor per ``screenshot`` line in the scenario. + + Each descriptor carries the viewport, the absolute file path of the + most recent ``goto_as`` snapshot, the document URL the snapshot is + served as, and the basename to use for the output (derived from the + ``screenshot`` line's filename without ``.png``). + """ + if not scenario_path.is_file(): + raise FileNotFoundError(scenario_path) + + width = 800 + height = 600 + current_url: str | None = None + current_file: Path | None = None + plan: list[dict] = [] + scenario_dir = scenario_path.parent + + with scenario_path.open("r", encoding="utf-8") as f: + for lineno, raw in enumerate(f, start=1): + tokens = _split_scenario_line(raw) + if not tokens: + continue + cmd, *args = tokens + if cmd == "viewport" and len(args) == 2: + width = int(args[0]) + height = int(args[1]) + elif cmd == "goto_as" and len(args) == 2: + current_url = args[0] + # File paths in scenarios are resolved relative to the + # repo root, matching the Rust harness behavior for + # `goto_as` arguments. + current_file = (REPO_ROOT / args[1]).resolve() + elif cmd == "goto" and len(args) == 1: + target = args[0] + if target.startswith(("http://", "https://", "file://", "data:", "about:")): + current_url = target + current_file = None + else: + current_url = None + current_file = (REPO_ROOT / target).resolve() + elif cmd == "screenshot" and len(args) == 1: + out_basename = Path(args[0]).stem + if current_file is None: + sys.stderr.write( + f"warning: {scenario_path.name}:{lineno}: screenshot " + f"with no preceding goto_as/local goto; skipping\n" + ) + continue + if not current_file.is_file(): + sys.stderr.write( + f"warning: {scenario_path.name}:{lineno}: snapshot " + f"{current_file} does not exist; skipping\n" + ) + continue + plan.append( + { + "scenario": scenario_path, + "scenario_dir": scenario_dir, + "viewport": (width, height), + "snapshot": current_file, + "document_url": current_url, + "viewport_name": out_basename, + "scenario_stem": scenario_path.stem, + } + ) + return plan + + +def _output_path(plan_item: dict, browser: str) -> Path: + name = f"{plan_item['scenario_stem']}.{plan_item['viewport_name']}.{browser}.png" + return plan_item["scenario_dir"] / name + + +def _file_url(path: Path) -> str: + # Playwright accepts file:// URLs; quote the path to handle spaces. + return "file://" + str(path) + + +def run_playwright_screenshot( + target: str, + out_path: Path, + width: int, + height: int, + browser: str, + timeout_ms: int, + wait_ms: int, +) -> None: + """Invoke ``npx playwright screenshot`` for a single capture.""" + out_path.parent.mkdir(parents=True, exist_ok=True) + cmd = [ + "npx", + "--yes", + "playwright", + "screenshot", + f"--browser={browser}", + f"--viewport-size={width},{height}", + f"--timeout={timeout_ms}", + ] + if wait_ms > 0: + cmd.append(f"--wait-for-timeout={wait_ms}") + cmd.append(target) + cmd.append(str(out_path)) + pretty = " ".join(shlex.quote(p) for p in cmd) + print(f" -> {pretty}", flush=True) + proc = subprocess.run(cmd, check=False) + if proc.returncode != 0: + raise RuntimeError( + f"npx playwright screenshot exited {proc.returncode} for " + f"{target} via {browser}" + ) + + +def discover_scenarios(arg_paths: Iterable[str], all_flag: bool) -> list[Path]: + if all_flag: + return sorted(DEFAULT_SCENARIOS_DIR.glob("*.we")) + out = [] + for raw in arg_paths: + p = Path(raw) + if not p.is_absolute(): + p = REPO_ROOT / p + if not p.is_file(): + raise FileNotFoundError(f"scenario not found: {raw}") + out.append(p.resolve()) + return out + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument( + "scenarios", + nargs="*", + help="Scenario .we files to process. Mutually exclusive with --all.", + ) + parser.add_argument( + "--all", + action="store_true", + help="Process every scenario under crates/e2e/scenarios/real-web/.", + ) + parser.add_argument( + "--browser", + choices=SUPPORTED_BROWSERS, + action="append", + help=( + "Browser to capture with. May be repeated. " + "Default: chromium and firefox." + ), + ) + parser.add_argument( + "--source", + choices=("url", "snapshot"), + default="url", + help=( + "Where to point Playwright. 'url' (default) navigates to the " + "live goto_as URL — useful for 'what does Chrome render?'. " + "'snapshot' loads the committed offline snapshot file://, " + "apples-to-apples with the offline we run." + ), + ) + parser.add_argument( + "--timeout-ms", + type=int, + default=60_000, + help="Per-screenshot timeout in milliseconds (default: 60000).", + ) + parser.add_argument( + "--wait-ms", + type=int, + default=None, + help=( + "Settle time before capture in milliseconds. Default: 5000 " + "when --source=url, 500 when --source=snapshot." + ), + ) + parser.add_argument( + "--skip-existing", + action="store_true", + help="Skip captures whose output file already exists.", + ) + parser.add_argument( + "--dry-run", + action="store_true", + help="Print the capture plan but do not invoke Playwright.", + ) + args = parser.parse_args() + + if args.all and args.scenarios: + parser.error("--all is mutually exclusive with explicit scenario paths") + if not args.all and not args.scenarios: + parser.error("provide one or more scenario paths or pass --all") + + browsers = args.browser or list(SUPPORTED_BROWSERS) + wait_ms = args.wait_ms + if wait_ms is None: + wait_ms = 5000 if args.source == "url" else 500 + + if not args.dry_run and shutil.which("npx") is None: + sys.stderr.write( + "error: npx not found on PATH. Install Node.js (which ships " + "npx) and re-run.\n" + ) + return 2 + + try: + scenarios = discover_scenarios(args.scenarios, args.all) + except FileNotFoundError as exc: + sys.stderr.write(f"error: {exc}\n") + return 2 + + if not scenarios: + sys.stderr.write("error: no scenarios to process\n") + return 2 + + total = 0 + captured = 0 + skipped = 0 + failed = 0 + for scenario in scenarios: + try: + plan = parse_capture_plan(scenario) + except (FileNotFoundError, ValueError) as exc: + sys.stderr.write(f"error: {scenario}: {exc}\n") + failed += 1 + continue + if not plan: + print(f"{scenario.name}: no screenshot commands; skipping") + continue + for item in plan: + w, h = item["viewport"] + if args.source == "url": + if not item["document_url"]: + sys.stderr.write( + f"warning: {scenario.name}: {item['viewport_name']}: " + f"--source=url but scenario has no goto_as URL; " + f"skipping\n" + ) + skipped += len(browsers) + total += len(browsers) + continue + target = item["document_url"] + else: + target = _file_url(item["snapshot"]) + for browser in browsers: + total += 1 + out = _output_path(item, browser) + rel = out.relative_to(REPO_ROOT) + print( + f"{scenario.name}: {item['viewport_name']} " + f"({w}x{h}) via {browser} -> {rel}" + ) + if args.skip_existing and out.is_file(): + print(" (exists, skipping)") + skipped += 1 + continue + if args.dry_run: + continue + try: + run_playwright_screenshot( + target=target, + out_path=out, + width=w, + height=h, + browser=browser, + timeout_ms=args.timeout_ms, + wait_ms=wait_ms, + ) + except RuntimeError as exc: + sys.stderr.write(f"error: {exc}\n") + failed += 1 + continue + captured += 1 + + print( + f"\nsummary: planned={total} captured={captured} " + f"skipped={skipped} failed={failed}" + ) + return 0 if failed == 0 else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/popular-sites/test_compare.py b/tests/popular-sites/test_compare.py new file mode 100644 index 0000000..7b98b52 --- /dev/null +++ b/tests/popular-sites/test_compare.py @@ -0,0 +1,146 @@ +#!/usr/bin/env python3 +"""Unit tests for ``compare.py``'s scenario parser. + +Run with ``python3 tests/popular-sites/test_compare.py``. Pure stdlib; +no Playwright required. +""" + +from __future__ import annotations + +import importlib.util +import sys +import tempfile +import unittest +from pathlib import Path + + +def _load_compare(): + here = Path(__file__).resolve().parent + spec = importlib.util.spec_from_file_location("compare", here / "compare.py") + if spec is None or spec.loader is None: + raise RuntimeError("could not load compare.py") + module = importlib.util.module_from_spec(spec) + sys.modules["compare"] = module + spec.loader.exec_module(module) + return module + + +compare = _load_compare() + + +class TokenizeTests(unittest.TestCase): + def test_blank_and_comment_lines_yield_empty(self): + self.assertEqual(compare._split_scenario_line(""), []) + self.assertEqual(compare._split_scenario_line(" "), []) + self.assertEqual(compare._split_scenario_line("# comment"), []) + self.assertEqual(compare._split_scenario_line(" # comment"), []) + + def test_strips_trailing_inline_comment(self): + self.assertEqual( + compare._split_scenario_line("viewport 800 600 # default"), + ["viewport", "800", "600"], + ) + + def test_honors_quoted_arguments(self): + self.assertEqual( + compare._split_scenario_line('assert_dom_contains "Hello world"'), + ["assert_dom_contains", "Hello world"], + ) + + +class CapturePlanTests(unittest.TestCase): + def _write_scenario(self, body: str, tmpdir: Path) -> Path: + scenario = tmpdir / "example.com.we" + scenario.write_text(body, encoding="utf-8") + return scenario + + def _write_snapshot(self, tmpdir: Path, rel: Path) -> Path: + snap = tmpdir / rel + snap.parent.mkdir(parents=True, exist_ok=True) + snap.write_text("", encoding="utf-8") + return snap + + def test_emits_one_entry_per_screenshot(self): + with tempfile.TemporaryDirectory() as td: + tmp = Path(td) + # Force REPO_ROOT to the tempdir so relative paths resolve there. + orig_root = compare.REPO_ROOT + try: + compare.REPO_ROOT = tmp + self._write_snapshot(tmp, Path("snap.html")) + scenario = self._write_scenario( + "viewport 1365 900\n" + "network offline\n" + "goto_as https://example.com/ snap.html\n" + "screenshot out/desktop.png\n" + "viewport 390 844\n" + "goto_as https://example.com/ snap.html\n" + "screenshot out/mobile.png\n", + tmp, + ) + plan = compare.parse_capture_plan(scenario) + self.assertEqual(len(plan), 2) + self.assertEqual(plan[0]["viewport"], (1365, 900)) + self.assertEqual(plan[0]["viewport_name"], "desktop") + self.assertEqual(plan[0]["document_url"], "https://example.com/") + self.assertEqual(plan[1]["viewport"], (390, 844)) + self.assertEqual(plan[1]["viewport_name"], "mobile") + finally: + compare.REPO_ROOT = orig_root + + def test_skips_screenshot_without_goto(self): + with tempfile.TemporaryDirectory() as td: + tmp = Path(td) + orig_root = compare.REPO_ROOT + try: + compare.REPO_ROOT = tmp + scenario = self._write_scenario( + "viewport 800 600\nscreenshot out/orphan.png\n", + tmp, + ) + plan = compare.parse_capture_plan(scenario) + self.assertEqual(plan, []) + finally: + compare.REPO_ROOT = orig_root + + def test_skips_screenshot_with_missing_snapshot(self): + with tempfile.TemporaryDirectory() as td: + tmp = Path(td) + orig_root = compare.REPO_ROOT + try: + compare.REPO_ROOT = tmp + scenario = self._write_scenario( + "viewport 800 600\n" + "goto_as https://example.com/ does-not-exist.html\n" + "screenshot out/x.png\n", + tmp, + ) + plan = compare.parse_capture_plan(scenario) + self.assertEqual(plan, []) + finally: + compare.REPO_ROOT = orig_root + + +class OutputPathTests(unittest.TestCase): + def test_output_path_format(self): + with tempfile.TemporaryDirectory() as td: + tmp = Path(td) + scenario_dir = tmp / "scenarios" + scenario_dir.mkdir() + item = { + "scenario_stem": "bing.com", + "viewport_name": "desktop", + "scenario_dir": scenario_dir, + } + self.assertEqual( + compare._output_path(item, "chromium").name, + "bing.com.desktop.chromium.png", + ) + self.assertEqual( + compare._output_path(item, "firefox").name, + "bing.com.desktop.firefox.png", + ) + + +if __name__ == "__main__": + unittest.main()