From 10a8c83721e2a9cdca4b2eecce856622af6c41d5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tao=20Bojl=C3=A9n?= Date: Sat, 20 Jun 2026 12:09:41 +0100 Subject: [PATCH] docs(spec): design for axe a11y conformance CI gate Rebuild the stale test/a11y suite as an axe-only, catalog-driven e2e conformance gate that runs against a production shadix_docs release and fails CI loudly on any new violation. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../specs/2026-06-20-a11y-axe-ci-design.md | 199 ++++++++++++++++++ 1 file changed, 199 insertions(+) create mode 100644 docs/superpowers/specs/2026-06-20-a11y-axe-ci-design.md diff --git a/docs/superpowers/specs/2026-06-20-a11y-axe-ci-design.md b/docs/superpowers/specs/2026-06-20-a11y-axe-ci-design.md new file mode 100644 index 0000000..1ad6761 --- /dev/null +++ b/docs/superpowers/specs/2026-06-20-a11y-axe-ci-design.md @@ -0,0 +1,199 @@ +# Design: axe a11y conformance gate + +**Date:** 2026-06-20 +**Branch:** `worktree-a11y-axe-ci` +**Status:** approved (design); implementation pending plan + +## Problem + +Every documented Shadix component has a docs page at `/components/:slug`. We want +an automated accessibility check that runs [axe-core](https://www.deque.com/axe/) +against each of those pages and **fails CI loudly on any violation**, so a11y +regressions can't merge. + +A stale, never-wired suite already exists at `test/a11y/` (Playwright + axe + +visual-regression). It is broken (navigates to `/storybook/components/`, +a route that no longer exists), bundles visual-regression screenshots we don't +want, has a hardcoded slug list that already drifted, and runs in no CI. **We are +rebuilding it fresh, axe-only, and deleting the old suite.** + +## Framing + +This is **not** a unit/integration test in the `mix test` sense. It is an +**end-to-end accessibility conformance gate**: it drives the real rendered DOM of +a running production docs instance through a real browser and asserts a +cross-cutting quality property (every component page passes axe). It crosses the +Elixir↔JS boundary and needs a running server + browser, so it lives **outside** +`mix test` as its own runner — a sibling gate, not part of the Elixir suite. + +The existing ExUnit/Floki tests remain the unit layer (structure assertions on +server-rendered HTML, in-process). This gate sits one layer above: real browser, +real compiled CSS (so axe can check color-contrast), JS hooks live. + +## Decisions (locked with user) + +- **Rebuild fresh, axe-only.** Delete `test/a11y/*` (old broken suite + visual + regression). No screenshots. +- **Runner:** Playwright + `@axe-core/playwright` (Node). axe needs a real + browser DOM; `mise.toml` already pins Node 24; Tangled nixery can pull + `nodejs`. +- **Coverage:** default render **and** open/interactive states. +- **Violation policy:** fail on every violation **except** a documented per-slug + rule-id allowlist for demo-page artifacts. New/unlisted rule → fail. +- **Rule scope:** full default axe rule set (includes best-practice rules) for + maximum strictness; genuine demo artifacts go in the allowlist. +- **CI target:** a **production instance** of the docs site (the `shadix_docs` + OTP release), **never `mix dev`**. + +## Architecture + +### Test set is catalog-driven (zero drift) + +The slug list must never be hardcoded (it already rotted once). Add a tiny +JSON endpoint to the docs router: + +``` +GET /components.json -> [{ "slug": "button", "interactive": false }, ...] +``` + +Backed by `Shadix.Website.Components.Catalog.all/0`. Playwright's global-setup +fetches it from the running server, so the test set **is** whatever the site +serves — it cannot drift from `website/components/catalog.ex`. + +The endpoint emits **slugs only** (keep it minimal). "Interactive" is decided +suite-side: the open-state opener map (below) is the single source of truth, and +the interactive pass simply runs over `slugs ∩ keys(openerMap)`. This means a +new catalog entry automatically gets a default-render scan, and only gains an +open-state scan once someone adds an opener for it. + +### Scoped scanning (test the component, not the chrome) + +The docs page renders the component inside `
` alongside a nav sidebar, +interactive control panel, and a props table — none of which are the component +under test. There is currently **no stable wrapper** around the component output +(the old suite's `.include('.shadix')` silently fell back to scanning the whole +``). + +Add a stable scope hook in `website/components/component_live.ex`: wrap the +**preview** section and the **examples** section in a container carrying +`data-shadix-preview` (a single attribute; no visual change). axe scans +`.include('[data-shadix-preview]')`, so it audits component output only — +excluding site nav, the docs control panel, and the props table. + +Rationale for excluding the control panel/props table: those are docs-authored +chrome (native inputs, a generated table), not the component. Their a11y is the +website's concern, tracked separately if at all; conflating them with component +conformance produces noise that isn't a component defect. + +### Two passes per page + +1. **Default render** — every slug from `/components.json`. Fully data-driven, + no per-component code. Navigate, `waitFor` the preview container, scan. +2. **Open state** — a per-slug **opener map** for interactive components + (overlays/menus/listboxes/toasts). Each opener: perform the interaction + (click trigger / open listbox / hover / right-click / dispatch toast event), + then **`await revealed.waitFor()`** the revealed element *before* `analyze()`. + This replaces the old suite's flaky `waitForTimeout(600)` with a deterministic + wait, per Playwright's a11y guidance. + + Opener map (initial, mirrors current catalog; revisit during implementation): + `dialog, alert_dialog, sheet, dropdown_menu, popover, command, combobox` + (click trigger), `select` (open listbox via `[data-slot="select-trigger"]`), + `context_menu` (right-click trigger), `tooltip, hover_card` (hover), + `accordion, collapsible` (click trigger), `sonner` (dispatch `shadix:toast`). + +### Violation policy & allowlist + +```js +// per-slug rule-id allowlist; each entry documents WHY it's a demo artifact, +// not a component defect. New/unlisted rule id on any slug fails. +const ALLOW = { + breadcrumb: ['landmark-unique'], // two