From 1785de263b2eb1cc8a59eddc5ed5bcb8a2a9e604 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tao=20Bojl=C3=A9n?= Date: Sat, 20 Jun 2026 12:16:05 +0100 Subject: [PATCH] docs(spec): generalize to a shared Playwright e2e harness The axe a11y gate becomes the first consumer of a general test/e2e/ harness (shared config, server, browser, CI job) so functional/regression specs can reuse the same infrastructure; ship a dialog example as template. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../specs/2026-06-20-a11y-axe-ci-design.md | 165 ++++++++++++------ 1 file changed, 108 insertions(+), 57 deletions(-) diff --git a/docs/superpowers/specs/2026-06-20-a11y-axe-ci-design.md b/docs/superpowers/specs/2026-06-20-a11y-axe-ci-design.md index 1ad6761..6b37561 100644 --- a/docs/superpowers/specs/2026-06-20-a11y-axe-ci-design.md +++ b/docs/superpowers/specs/2026-06-20-a11y-axe-ci-design.md @@ -1,4 +1,4 @@ -# Design: axe a11y conformance gate +# Design: Playwright e2e harness + axe a11y conformance gate **Date:** 2026-06-20 **Branch:** `worktree-a11y-axe-ci` @@ -6,65 +6,103 @@ ## Problem -Every documented Shadix component has a docs page at `/components/:slug`. We want -an automated accessibility check that runs [axe-core](https://www.deque.com/axe/) -against each of those pages and **fails CI loudly on any violation**, so a11y -regressions can't merge. +Two needs, one shared foundation: + +1. **a11y conformance gate (primary):** every documented component has a docs + page at `/components/:slug`. Run [axe-core](https://www.deque.com/axe/) + against each and **fail CI loudly on any violation**, so a11y regressions + can't merge. +2. **General browser regression tests (enabling):** the ability to write + targeted Playwright tests against the docs site for specific behaviors + (e.g. "dialog opens on click and traps focus"), as functional regression + coverage grows over time. + +Both need the same hard infrastructure — a running docs instance, a browser, and +CI wiring. So we build **one general Playwright e2e harness** and make the a11y +gate its first consumer, with a working example regression test as a template for +the second. A stale, never-wired suite already exists at `test/a11y/` (Playwright + axe + -visual-regression). It is broken (navigates to `/storybook/components/`, -a route that no longer exists), bundles visual-regression screenshots we don't +visual-regression). It is broken (navigates to `/storybook/components/`, a +route that no longer exists), bundles visual-regression screenshots we don't want, has a hardcoded slug list that already drifted, and runs in no CI. **We are -rebuilding it fresh, axe-only, and deleting the old suite.** +replacing it with the new `test/e2e/` harness and deleting the old suite.** ## Framing -This is **not** a unit/integration test in the `mix test` sense. It is an +The a11y gate is **not** a unit/integration test in the `mix test` sense. It is an **end-to-end accessibility conformance gate**: it drives the real rendered DOM of a running production docs instance through a real browser and asserts a cross-cutting quality property (every component page passes axe). It crosses the -Elixir↔JS boundary and needs a running server + browser, so it lives **outside** -`mix test` as its own runner — a sibling gate, not part of the Elixir suite. +Elixir↔JS boundary and needs a running server + browser, so the whole e2e harness +lives **outside** `mix test` as its own runner — a sibling gate, not part of the +Elixir suite. The existing ExUnit/Floki tests remain the unit layer (structure assertions on -server-rendered HTML, in-process). This gate sits one layer above: real browser, -real compiled CSS (so axe can check color-contrast), JS hooks live. +server-rendered HTML, in-process). This harness sits one layer above: real +browser, real compiled CSS (so axe can check color-contrast), JS hooks live. ## Decisions (locked with user) -- **Rebuild fresh, axe-only.** Delete `test/a11y/*` (old broken suite + visual - regression). No screenshots. +- **One general e2e harness**, not an a11y-only island. The a11y gate is the first + consumer; functional/regression specs share the same config, server, browser, + and CI job. Ship a working example regression test as a template. +- **Rebuild fresh.** Delete `test/a11y/*` (old broken suite + visual regression). + No screenshots. - **Runner:** Playwright + `@axe-core/playwright` (Node). axe needs a real browser DOM; `mise.toml` already pins Node 24; Tangled nixery can pull `nodejs`. -- **Coverage:** default render **and** open/interactive states. -- **Violation policy:** fail on every violation **except** a documented per-slug - rule-id allowlist for demo-page artifacts. New/unlisted rule → fail. -- **Rule scope:** full default axe rule set (includes best-practice rules) for - maximum strictness; genuine demo artifacts go in the allowlist. +- **a11y coverage:** default render **and** open/interactive states. +- **a11y violation policy:** fail on every violation **except** a documented + per-slug rule-id allowlist for demo-page artifacts. New/unlisted rule → fail. +- **a11y rule scope:** full default axe rule set (includes best-practice rules) + for maximum strictness; genuine demo artifacts go in the allowlist. - **CI target:** a **production instance** of the docs site (the `shadix_docs` OTP release), **never `mix dev`**. ## Architecture +### Directory layout + +``` +test/e2e/ + package.json + playwright.config.mjs # baseURL, global-setup, projects — SHARED + global-setup.mjs # fetch /components.json, stash slug list — SHARED + support/ + axe-fixture.mjs # makeAxeBuilder fixture (a11y-only helper) + slugs.mjs # read the stashed slug list + a11y/ # the conformance gate (this project) + components.spec.mjs # default-render pass (catalog-driven) + interactive.spec.mjs # open-state pass (opener map) + regression/ # functional/behavioral tests + dialog.spec.mjs # the example template (this project) + README.md +``` + +**Running:** `npx playwright test` runs everything. The config defines Playwright +**projects** so a subset can run alone — `--project=a11y` (the gate) vs +`--project=regression` (functional). CI runs all projects; the projects exist so a +developer can iterate on one without the other. + ### Test set is catalog-driven (zero drift) -The slug list must never be hardcoded (it already rotted once). Add a tiny +The a11y slug list must never be hardcoded (it already rotted once). Add a tiny JSON endpoint to the docs router: ``` -GET /components.json -> [{ "slug": "button", "interactive": false }, ...] +GET /components.json -> ["accordion", "alert", "alert_dialog", ...] ``` -Backed by `Shadix.Website.Components.Catalog.all/0`. Playwright's global-setup -fetches it from the running server, so the test set **is** whatever the site +Backed by `Shadix.Website.Components.Catalog.all/0`. The harness's global-setup +fetches it from the running server, so the a11y test set **is** whatever the site serves — it cannot drift from `website/components/catalog.ex`. The endpoint emits **slugs only** (keep it minimal). "Interactive" is decided suite-side: the open-state opener map (below) is the single source of truth, and -the interactive pass simply runs over `slugs ∩ keys(openerMap)`. This means a -new catalog entry automatically gets a default-render scan, and only gains an -open-state scan once someone adds an opener for it. +the interactive pass runs over `slugs ∩ keys(openerMap)`. A new catalog entry +automatically gets a default-render scan, and gains an open-state scan only once +someone adds an opener for it. ### Scoped scanning (test the component, not the chrome) @@ -78,19 +116,20 @@ Add a stable scope hook in `website/components/component_live.ex`: wrap the **preview** section and the **examples** section in a container carrying `data-shadix-preview` (a single attribute; no visual change). axe scans `.include('[data-shadix-preview]')`, so it audits component output only — -excluding site nav, the docs control panel, and the props table. +excluding site nav, the docs control panel, and the props table. (This wrapper is +also a convenient, stable anchor for regression specs.) Rationale for excluding the control panel/props table: those are docs-authored chrome (native inputs, a generated table), not the component. Their a11y is the -website's concern, tracked separately if at all; conflating them with component -conformance produces noise that isn't a component defect. +website's concern; conflating them with component conformance produces noise that +isn't a component defect. -### Two passes per page +### a11y: two passes per page -1. **Default render** — every slug from `/components.json`. Fully data-driven, - no per-component code. Navigate, `waitFor` the preview container, scan. +1. **Default render** — every slug from `/components.json`. Fully data-driven, no + per-component code. Navigate, `waitFor` the preview container, scan. 2. **Open state** — a per-slug **opener map** for interactive components - (overlays/menus/listboxes/toasts). Each opener: perform the interaction + (overlays/menus/listboxes/toasts). Each opener performs the interaction (click trigger / open listbox / hover / right-click / dispatch toast event), then **`await revealed.waitFor()`** the revealed element *before* `analyze()`. This replaces the old suite's flaky `waitForTimeout(600)` with a deterministic @@ -102,7 +141,7 @@ conformance produces noise that isn't a component defect. `context_menu` (right-click trigger), `tooltip, hover_card` (hover), `accordion, collapsible` (click trigger), `sonner` (dispatch `shadix:toast`). -### Violation policy & allowlist +### a11y: violation policy & allowlist ```js // per-slug rule-id allowlist; each entry documents WHY it's a demo artifact, @@ -117,17 +156,24 @@ slug; the remainder must be empty. On failure: `testInfo.attach()` the full axe scan JSON (`{rule, impact, help, nodes[].target}`) so CI logs pinpoint the offending rule and element, not just "violations != []". -### Shared config via a Playwright fixture - A `makeAxeBuilder` fixture (per Playwright docs) centralizes the common config — -`.include('[data-shadix-preview]')` and the full-default rule set — so both spec -files share one configuration and it can't drift between them. +`.include('[data-shadix-preview]')` and the full-default rule set — so both a11y +spec files share one configuration and it can't drift between them. + +### Example regression test (template) + +`regression/dialog.spec.mjs`: navigate to `/components/dialog`, click the trigger, +assert the dialog (`role="dialog"`) becomes visible, press `Escape`, assert it +hides. This is a real behavioral regression candidate (exercises the JS hook), and +doubles as the copy-me template documented in the README for adding more. No axe +involved — proves the harness serves non-a11y tests too. ## CI The existing `.tangled/workflows/ci.yml` (format + `tsc` + `mix test`, MIX_ENV=test) -stays fast and untouched. Add a **separate** workflow `.tangled/workflows/a11y.yml` -that runs in parallel and also gates PRs: +stays fast and untouched. Add a **separate** workflow `.tangled/workflows/e2e.yml` +that runs in parallel and also gates PRs. It runs the **whole** Playwright suite +(a11y + regression), not just axe: 1. Build the production docs artifact (mirrors the Dockerfile build stage): `MIX_ENV=docs`, `mix deps.get`, asset deps (`cd assets && npm install`), @@ -138,7 +184,7 @@ that runs in parallel and also gates PRs: (Release endpoint has `server: true`, `check_origin: false`, no watchers, no code reloader — a genuine production instance.) 3. Wait for `http://localhost:8080/` to return 200 (poll with a timeout). -4. `cd test/a11y && npm ci && npx playwright test` against `baseURL=:8080`. +4. `cd test/e2e && npm ci && npx playwright test` against `baseURL=:8080`. Non-zero exit → red build (loud failure). **Not Docker.** The Tangled engine is `nixery` (builds a nix container per job); @@ -162,38 +208,43 @@ provisioned and can proceed regardless. ## Files **New:** -- `test/a11y/package.json`, `playwright.config.mjs` — runner config, `baseURL`. -- `test/a11y/axe-fixture.mjs` — shared `makeAxeBuilder` fixture. -- `test/a11y/components.spec.mjs` — default-render pass (catalog-driven). -- `test/a11y/interactive.spec.mjs` — open-state pass (opener map). -- `test/a11y/global-setup.mjs` — fetch `/components.json`, expose slug list. -- `test/a11y/README.md` — how to run locally + in CI. -- `.tangled/workflows/a11y.yml` — CI workflow (release + browser + playwright). +- `test/e2e/package.json`, `playwright.config.mjs` — runner config, `baseURL`, + `a11y`/`regression` projects. +- `test/e2e/global-setup.mjs` — fetch `/components.json`, stash slug list. +- `test/e2e/support/axe-fixture.mjs`, `support/slugs.mjs` — shared helpers. +- `test/e2e/a11y/components.spec.mjs` — default-render pass (catalog-driven). +- `test/e2e/a11y/interactive.spec.mjs` — open-state pass (opener map). +- `test/e2e/regression/dialog.spec.mjs` — example functional test (template). +- `test/e2e/README.md` — how to run locally + in CI, and how to add a regression + test (copy the dialog example). +- `.tangled/workflows/e2e.yml` — CI workflow (release + browser + playwright). **Changed:** -- `website/router.ex` — add `GET /components.json`. +- `website/router.ex` — add `GET /components.json` (+ a small handler/controller). - `website/components/component_live.ex` — add `data-shadix-preview` wrapper. -- (new) a small controller/inline route handler for `/components.json`. **Deleted:** -- old `test/a11y/components.spec.mjs`, `interactive.spec.mjs`, - `playwright.config.mjs`, `package-lock.json`, `screenshots/` baselines (the - stale visual-regression suite). +- old `test/a11y/` (stale specs, `playwright.config.mjs`, `package-lock.json`, + `screenshots/` baselines). ## Out of scope - Visual-regression / screenshot testing (explicitly dropped). +- Building out a functional regression suite beyond the single example template — + the user grows that themselves on the harness. - a11y of the docs site chrome (nav, control panel, props table) — website's concern, not component conformance. -- Manual a11y assessment, screen-reader testing — axe catches a subset; this +- Manual a11y assessment / screen-reader testing — axe catches a subset; this gate does not claim full WCAG conformance (axe's own disclaimer). - Running this inside `mix test` — it is intentionally a separate runner. ## Success criteria - A new axe violation on any component page (default or open state) fails CI. -- The slug set is derived from the catalog at runtime (no hardcoded list). -- The suite runs against a production `shadix_docs` instance, not `mix dev`. +- The a11y slug set is derived from the catalog at runtime (no hardcoded list). +- The harness runs against a production `shadix_docs` instance, not `mix dev`. - Demo-only artifacts are documented in an allowlist; genuine component defects are not allowlisted. +- A developer can add a functional regression test by copying the dialog example; + CI runs it in the same browser job. - The fast `ci.yml` job is unaffected (still format + tsc + mix test only). -- 2.51.2