diff --git a/docs/superpowers/plans/2026-06-20-a11y-axe-ci.md b/docs/superpowers/plans/2026-06-20-a11y-axe-ci.md deleted file mode 100644 index d08128e..0000000 --- a/docs/superpowers/plans/2026-06-20-a11y-axe-ci.md +++ /dev/null @@ -1,849 +0,0 @@ -# Playwright e2e harness + axe a11y conformance gate — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** A standalone Playwright suite that runs axe-core against every component docs page (baseline + per-state scenarios) and a production docs release in CI, failing loudly on any new a11y violation — with the same harness reusable for functional regression tests. - -**Architecture:** A `test/e2e/` Node/Playwright project (its own `package.json`, outside `mix test`). A tiny `/components.json` route makes the slug set catalog-driven; a `data-shadix-preview` wrapper scopes axe to component output. Specs run against `mix dev` locally (`:4001`) and a `MIX_ENV=docs` OTP release in CI (`:8080`). The browser is the Nix-provided `chromium` driven via Playwright `executablePath` (recipe confirmed by a green spike). - -**Tech Stack:** Playwright `@playwright/test` + `@axe-core/playwright` (Node 24), Phoenix/LiveView docs site (Elixir 1.20), Tangled nixery CI. - -## Global Constraints - -- **Separate runner:** this suite is NOT part of `mix test`; it has its own `package.json` under `test/e2e/`. -- **Confirmed CI browser recipe (do not re-derive):** nixery deps include `chromium`, `fontconfig`, `dejavu_fonts`; launch via `executablePath=$(command -v chromium)` with args `--no-sandbox --disable-dev-shm-usage`; set `PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1`; provision fonts into `$HOME/.fonts` + a `FONTCONFIG_FILE` before launch (env does not persist across Tangled steps). -- **CI server is a production release, never `mix dev`:** `MIX_ENV=docs mix release shadix_docs`, booted with `SECRET_KEY_BASE` + `PORT=8080` + `PHX_HOST=localhost`. -- **axe scope:** `.include('[data-shadix-preview]')`, full default rule set (no `withTags` filtering). -- **Violation policy:** fail on every violation except a documented per-slug rule-id allowlist; new/unlisted rule fails. -- **Assertions:** use `textContent` / Playwright web-first assertions, never layout-dependent `innerText` (spike finding). -- **Elixir changes:** must pass `mix format` and `mix compile --warnings-as-errors`. Changing a file under `lib/shadix/` requires `mix shadix.gen.registry`; **this plan only changes `website/`**, so no registry regen — keep it that way (prefer `test.fixme()` over editing `lib/` component source). -- **shadcn/base-ui:** never describe from memory; read `vendor/shadcn-ui/` / `vendor/base-ui/` when authoring ARIA-contract scenarios. - ---- - -## File Structure - -**New (Node/Playwright harness):** -- `test/e2e/package.json` — deps + npm scripts. -- `test/e2e/.gitignore` — ignore `node_modules/`, `slugs.json`, `test-results/`, `playwright-report/`. -- `test/e2e/playwright.config.mjs` — `baseURL`, `launchOptions`, `a11y`/`regression` projects. -- `test/e2e/fetch-slugs.mjs` — pretest: GET `/components.json` → write `slugs.json`. -- `test/e2e/support/slugs.mjs` — read `slugs.json` synchronously. -- `test/e2e/support/allowlist.mjs` — per-slug rule-id allowlist + filter helper. -- `test/e2e/support/axe-fixture.mjs` — `makeAxeBuilder` fixture. -- `test/e2e/support/scenarios/index.mjs` — `slug -> [{name, run}]` map. -- `test/e2e/support/scenarios/combobox.mjs` — exemplar scenario file (hand-written). -- `test/e2e/support/scenarios/.mjs` — the rest (workflow-authored, Task 10). -- `test/e2e/a11y/baseline.spec.mjs` — baseline scan per slug. -- `test/e2e/a11y/scenarios.spec.mjs` — named-scenario scans. -- `test/e2e/regression/dialog.spec.mjs` — example functional test (template). -- `test/e2e/README.md` — run instructions (local + CI) + how to add a regression test. -- `.tangled/workflows/e2e.yml` — CI workflow. - -**New (website):** -- `website/catalog_controller.ex` — `GET /components.json` handler. - -**Modified (website):** -- `website/router.ex` — add the JSON route. -- `website/components/component_live.ex:117` — wrap preview + examples in `[data-shadix-preview]`. - -**Deleted (end of plan):** -- `test/a11y/` (stale suite), `test/e2e-spike/`, `.tangled/workflows/a11y-spike.yml`. - ---- - -## Task 1: Scope hook — `data-shadix-preview` wrapper - -**Files:** -- Modify: `website/components/component_live.ex` (the `render/1` `
` body — preview `
` ~line 117 and examples `
` ~line 143) -- Test: `test/shadix/website/component_live_test.exs` (create if absent; otherwise add to existing component_live test) - -**Interfaces:** -- Produces: a single DOM element matching `[data-shadix-preview]` that contains the component preview and the curated examples, and excludes the nav sidebar, the control panel, and the props table. - -- [ ] **Step 1: Write the failing test** - -Check for an existing live-view test first: `ls test/shadix/website/`. If `component_live_test.exs` exists, add this test to it; otherwise create the file: - -```elixir -defmodule Shadix.Website.ComponentLiveTest do - use Shadix.LiveCase, async: true - - test "component page wraps preview + examples in a [data-shadix-preview] scope", %{conn: conn} do - {:ok, _view, html} = live(conn, "/components/button") - doc = Floki.parse_document!(html) - - # Exactly one scope element, and it contains the live preview. - assert [scope] = Floki.find(doc, "[data-shadix-preview]") - assert Floki.find(scope, "[data-slot='button']") != [] - end -end -``` - -- [ ] **Step 2: Run it to verify it fails** - -Run: `mix test test/shadix/website/component_live_test.exs` -Expected: FAIL — `match (=) failed` because `[data-shadix-preview]` is not yet rendered. - -- [ ] **Step 3: Add the wrapper** - -In `website/components/component_live.ex`, wrap the preview `
` and the examples `
` in a single container carrying the attribute. Open the wrapper immediately before the preview `
` and close it immediately after the examples `
` closes. Example shape: - -```heex -
-
-
- {@entry.doc.preview(%{__changed__: nil, props: @props, slot: @slot})} -
- -
- -
- -
-
-``` - -Note: the control panel (`
`) currently lives inside the preview `
`. Leave it where it is — axe scanning the configurator's native inputs is acceptable noise only if they are accessible; if they aren't, they belong to the docs site. Decision for this task: keep the wrapper tight around **preview block + examples**, but EXCLUDE the controls grid by closing the preview wrapper region before the controls. If the controls are structurally inside the same `
`, move the controls grid to sit *after* the `
` close. Verify by reading lines 117–157 and adjusting boundaries so the controls grid and props table are OUTSIDE `[data-shadix-preview]`. - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `mix test test/shadix/website/component_live_test.exs` -Expected: PASS. - -- [ ] **Step 5: Format + compile check** - -Run: `mix format && mix compile --warnings-as-errors` -Expected: no changes needed / no warnings. - -- [ ] **Step 6: Commit** - -```bash -git add website/components/component_live.ex test/shadix/website/component_live_test.exs -git commit -m "feat(website): scope component preview with [data-shadix-preview] for axe" -``` - ---- - -## Task 2: Catalog endpoint — `GET /components.json` - -**Files:** -- Create: `website/catalog_controller.ex` -- Modify: `website/router.ex` -- Test: `test/shadix/website/catalog_controller_test.exs` - -**Interfaces:** -- Produces: `GET /components.json` → `200 application/json`, body is a JSON array of slug strings, equal to `Catalog.all/0 |> Enum.map(& &1.slug)`. - -- [ ] **Step 1: Write the failing test** - -```elixir -defmodule Shadix.Website.CatalogControllerTest do - use Shadix.LiveCase, async: true - - test "GET /components.json returns the catalog slugs as JSON", %{conn: conn} do - conn = get(conn, "/components.json") - assert json_response(conn, 200) - slugs = json_response(conn, 200) - assert is_list(slugs) - assert "button" in slugs - # Must match the catalog exactly. - expected = Enum.map(Shadix.Website.Components.Catalog.all(), & &1.slug) - assert Enum.sort(slugs) == Enum.sort(expected) - end -end -``` - -If `Shadix.LiveCase` doesn't provide `get/2`/`json_response/2`, use `Phoenix.ConnTest` — check `test/support/live_case.ex` and add `import Phoenix.ConnTest` there if missing, or write this as a `use ExUnit.Case` test that builds a conn with `Phoenix.ConnTest.build_conn()`. - -- [ ] **Step 2: Run it to verify it fails** - -Run: `mix test test/shadix/website/catalog_controller_test.exs` -Expected: FAIL — 404 / no route. - -- [ ] **Step 3: Create the controller** - -```elixir -defmodule Shadix.Website.CatalogController do - @moduledoc """ - Exposes the documented-component slug list as JSON, so the Playwright e2e - harness derives its test set from the running site (zero drift vs Catalog). - """ - use Phoenix.Controller, formats: [:json] - - alias Shadix.Website.Components.Catalog - - def index(conn, _params) do - slugs = Enum.map(Catalog.all(), & &1.slug) - json(conn, slugs) - end -end -``` - -- [ ] **Step 4: Add the route** - -In `website/router.ex`, add a JSON pipeline + route inside the existing `scope "/"`: - -```elixir -pipeline :api do - plug(:accepts, ["json"]) -end - -scope "/" do - pipe_through(:api) - get("/components.json", Shadix.Website.CatalogController, :index) -end -``` - -Place this scope alongside the existing `scope "/" do pipe_through(:docs) ... end` (two sibling scopes). Keep `live(...)` routes in the `:docs` scope. - -- [ ] **Step 5: Run the test to verify it passes** - -Run: `mix test test/shadix/website/catalog_controller_test.exs` -Expected: PASS. - -- [ ] **Step 6: Format + compile + commit** - -```bash -mix format && mix compile --warnings-as-errors -git add website/catalog_controller.ex website/router.ex test/shadix/website/catalog_controller_test.exs -git commit -m "feat(website): add GET /components.json catalog endpoint for e2e harness" -``` - ---- - -## Task 3: Harness scaffold + smoke test - -**Files:** -- Create: `test/e2e/package.json`, `test/e2e/.gitignore`, `test/e2e/playwright.config.mjs`, `test/e2e/fetch-slugs.mjs`, `test/e2e/support/slugs.mjs`, `test/e2e/smoke.spec.mjs` (temporary), `test/e2e/README.md` - -**Interfaces:** -- Produces: `support/slugs.mjs` default-exports an array of slug strings (read synchronously from `slugs.json`). `playwright.config.mjs` exposes projects `a11y` and `regression` and a `baseURL` from `PLAYWRIGHT_BASE_URL` (default `http://localhost:4001`). -- Consumes: a running docs server (local `mix dev` on `:4001`, or CI release on `:8080`) and `GET /components.json` (Task 2). - -- [ ] **Step 1: Create `package.json`** - -```json -{ - "name": "shadix-e2e", - "private": true, - "type": "module", - "description": "Playwright e2e harness: axe a11y conformance gate + functional regression tests against the Shadix docs site.", - "scripts": { - "slugs": "node fetch-slugs.mjs", - "test": "node fetch-slugs.mjs && playwright test", - "test:a11y": "node fetch-slugs.mjs && playwright test --project=a11y", - "test:regression": "playwright test --project=regression" - }, - "devDependencies": { - "@playwright/test": "^1.61.0", - "@axe-core/playwright": "^4.10.1" - } -} -``` - -- [ ] **Step 2: Create `.gitignore`** - -```gitignore -node_modules/ -slugs.json -test-results/ -playwright-report/ -``` - -- [ ] **Step 3: Create `playwright.config.mjs`** - -```js -import { defineConfig } from "@playwright/test"; - -// baseURL: local dev runs `mix dev` (:4001); CI sets PLAYWRIGHT_BASE_URL to the -// production release (:8080). CHROMIUM_BIN: unset locally (Playwright's bundled -// browser); CI sets it to the Nix-provided chromium. The two launch args are -// required in CI containers and harmless locally. -export default defineConfig({ - testDir: ".", - fullyParallel: true, - forbidOnly: !!process.env.CI, - reporter: process.env.CI ? "list" : "line", - use: { - baseURL: process.env.PLAYWRIGHT_BASE_URL || "http://localhost:4001", - launchOptions: { - executablePath: process.env.CHROMIUM_BIN || undefined, - args: ["--no-sandbox", "--disable-dev-shm-usage"], - }, - }, - projects: [ - { name: "a11y", testDir: "./a11y" }, - { name: "regression", testDir: "./regression" }, - ], -}); -``` - -- [ ] **Step 4: Create `fetch-slugs.mjs`** - -```js -// Pretest: fetch the catalog slug list from the running docs server and write -// slugs.json, so spec files can build data-driven tests synchronously at import -// time (Playwright loads spec files before globalSetup runs, so we cannot fetch -// inside globalSetup and expect specs to see it). -import { writeFileSync } from "node:fs"; - -const base = process.env.PLAYWRIGHT_BASE_URL || "http://localhost:4001"; -const url = `${base}/components.json`; - -async function main() { - for (let attempt = 1; attempt <= 30; attempt++) { - try { - const res = await fetch(url); - if (!res.ok) throw new Error(`HTTP ${res.status}`); - const slugs = await res.json(); - if (!Array.isArray(slugs) || slugs.length === 0) throw new Error("empty slug list"); - writeFileSync(new URL("./slugs.json", import.meta.url), JSON.stringify(slugs, null, 2)); - console.log(`wrote slugs.json (${slugs.length} components) from ${url}`); - return; - } catch (err) { - if (attempt === 30) { - console.error(`could not fetch ${url} after 30 tries: ${err.message}`); - process.exit(1); - } - await new Promise((r) => setTimeout(r, 1000)); - } - } -} - -await main(); -``` - -- [ ] **Step 5: Create `support/slugs.mjs`** - -```js -import { readFileSync } from "node:fs"; - -// slugs.json is produced by fetch-slugs.mjs (npm `slugs`/`test` scripts run it -// first). Read synchronously so spec files can iterate at import time. -export const slugs = JSON.parse(readFileSync(new URL("../slugs.json", import.meta.url))); -``` - -- [ ] **Step 6: Create a temporary smoke test `smoke.spec.mjs`** - -```js -import { test, expect } from "@playwright/test"; -import { slugs } from "./support/slugs.mjs"; - -test("docs home renders and slug list is non-empty", async ({ page }) => { - expect(slugs.length).toBeGreaterThan(0); - await page.goto("/components/button"); - await expect(page.locator("[data-shadix-preview]")).toBeVisible(); -}); -``` - -- [ ] **Step 7: Install + run locally against `mix dev`** - -In one terminal: `mix dev` (from repo root; serves `:4001`). -In another: - -```bash -cd test/e2e -npm install -npx playwright install chromium # first time only, for local bundled browser -npm run slugs # writes slugs.json -npx playwright test smoke.spec.mjs --project=a11y -``` - -Expected: 1 passed. If `slugs.json` missing error: run `npm run slugs` first (server must be up). - -- [ ] **Step 8: Write `README.md`** - -Document: prerequisites (`mix dev` for local, the CI workflow for prod), `npm install` + `npx playwright install chromium`, `npm test` (all), `npm run test:a11y`, `npm run test:regression`, the `PLAYWRIGHT_BASE_URL`/`CHROMIUM_BIN` env vars, and a "How to add a regression test" section pointing at `regression/dialog.spec.mjs` as the template. - -- [ ] **Step 9: Delete the smoke test and commit** - -```bash -rm test/e2e/smoke.spec.mjs -git add test/e2e/package.json test/e2e/.gitignore test/e2e/playwright.config.mjs \ - test/e2e/fetch-slugs.mjs test/e2e/support/slugs.mjs test/e2e/README.md -git commit -m "feat(e2e): Playwright harness scaffold (config, slug fetch, projects)" -``` - -(The smoke test was scaffolding to verify the harness boots; the real specs in Tasks 5–7 supersede it. `package-lock.json` is generated by `npm install` — commit it too if present.) - ---- - -## Task 4: axe fixture + allowlist - -**Files:** -- Create: `test/e2e/support/axe-fixture.mjs`, `test/e2e/support/allowlist.mjs` - -**Interfaces:** -- Produces: - - `allowlist.mjs`: `export const ALLOW` (`{ [slug]: string[] }`) and `export function unexpected(slug, violations)` → violations whose `id` is not allowlisted for `slug`. - - `axe-fixture.mjs`: `export const test` (Playwright test extended with `makeAxeBuilder: () => AxeBuilder`) and re-exports `expect`. `makeAxeBuilder()` returns an `AxeBuilder` pre-scoped to `[data-shadix-preview]` with the full default rule set. - -- [ ] **Step 1: Create `allowlist.mjs`** - -```js -// Per-slug axe rule-id allowlist for demo-PAGE artifacts that are not component -// defects. Each entry MUST carry a comment justifying it. Any violation whose id -// is not listed here for the given slug fails the test. -export const ALLOW = { - // Two