Automated Accessibility Tests in the CI Pipeline

1. Why accessibility needs to live in the CI pipeline

A one-time accessibility audit before launch feels like a clean close, but it is only a snapshot in time. Every new feature, every layout tweak and every updated third-party script can introduce new barriers without anyone noticing in a typical review. Without automated checks inside the CI pipeline, regressions creep in unnoticed until the next manual audit uncovers them months later, often only after users have complained or a legal warning letter arrives.

With Germany's Accessibility Strengthening Act (BFSG) and the European Accessibility Act, accessibility is no longer optional for many online shops since 2025, it is a legal requirement. Automated tests that run on every pull request shift responsibility away from a rare, expensive audit toward a continuous safety net that catches problems exactly where regressions originate: in code review, before anything merges into the main branch. That reduces legal risk and cuts the cost of fixing issues that would otherwise surface late in the development cycle.

2. What axe-core catches automatically and what it does not

axe-core is the most widely used open source engine for automated accessibility scanning and underpins practically every popular testing tool, from Lighthouse to WAVE to the browser extensions. The engine checks the rendered DOM against hundreds of rules derived from the WCAG success criteria: missing alternative text, insufficient color contrast, duplicate IDs, invalid ARIA attributes, missing form labels, and structural violations like an incorrect heading hierarchy. These rules are deterministic and reproduce reliably across every test run.

The key fact teams often underestimate: automated scanners like axe-core realistically cover only about a third of all WCAG violations, according to Deque Systems, the makers of the engine. What axe-core can check syntactically, it checks reliably, but the semantic quality of an alt text, the logical order when tabbing, or whether an ARIA live region is actually announced in a meaningful way, all of that escapes a purely structural analysis. An aria-label="click here" is technically valid for axe-core, but worthless for a screen reader user.

<!-- axe-core reliably catches this: button has no accessible name -->
<button class="icon-btn" onclick="removeItem(id)">
  <svg aria-hidden="true"><use href="#icon-trash"></use></svg>
</button>

<!-- axe-core passes this even though the label is meaningless -->
<button aria-label="click here" onclick="removeItem(id)">
  <svg aria-hidden="true"><use href="#icon-trash"></use></svg>
</button>

<!-- Correct: accessible name matches the actual action -->
<button aria-label="Remove item from cart" onclick="removeItem(id)">
  <svg aria-hidden="true"><use href="#icon-trash"></use></svg>
</button>

3. Integrating axe-core into an existing E2E suite

The most pragmatic starting point is not a separate accessibility test suite, but extending the end-to-end tests you already have. If you already use Playwright or Cypress to test checkout flows, product pages and login, you can attach an additional axe-core scan right inside those same tests, after the page has loaded and become interactive. The @axe-core/playwright package provides an AxeBuilder for this, which scans the full visible DOM tree against the desired WCAG conformance levels and returns a structured list of violations.

Placement of the scan matters: run it only after Alpine.js components have initialized and dynamic content like cart counters or loading spinners has finished rendering, otherwise axe-core inspects an incomplete intermediate state. .withTags() lets you restrict the ruleset to the relevant WCAG conformance levels, and .exclude() lets you deliberately skip known third-party widgets you do not maintain yourself, without disabling the entire check.

// tests/accessibility/checkout.spec.js
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test.describe('Accessibility: checkout flow', () => {
  test('cart page has no new axe violations', async ({ page }) => {
    await page.goto('/checkout/cart');

    // Reuse the same page object as the functional E2E suite
    await page.waitForSelector('[data-testid="cart-summary"]');

    const results = await new AxeBuilder({ page })
      .withTags(['wcag2a', 'wcag2aa', 'wcag22aa'])
      .exclude('#hyva-cookie-banner') // known third-party widget, tracked separately
      .analyze();

    expect(results.violations).toEqual([]);
  });
});

4. Baseline strategy: block new violations, not old ones

The most common reason teams abandon automated accessibility tests after a few weeks: the first scan of a grown shop reports dozens or hundreds of violations at once, the build turns red, and nobody can realistically fix every legacy issue in a single pull request. The fix is not a per-test exception, it is a baseline file: a version-controlled JSON document that records the current state of known violations, complete with ticket references and rationale.

Every CI run compares the current scan results against this baseline. Violations that are already documented let the build stay green, but any new, previously unknown violation triggers a failure. That prevents two extremes at once: a permanently red build nobody takes seriously anymore, and a pipeline that silently waves new barriers through. The baseline itself is only updated deliberately, once a known issue has actually been fixed, ideally through the same pull request review process as any other code change.

{
  "baseline": {
    "generatedAt": "2026-06-01T09:00:00Z",
    "url": "/checkout/cart",
    "knownViolations": [
      {
        "id": "color-contrast",
        "impact": "serious",
        "nodes": 2,
        "ticket": "A11Y-142",
        "note": "Legacy price badge, fix scheduled for Q3"
      },
      {
        "id": "aria-required-children",
        "impact": "critical",
        "nodes": 1,
        "ticket": "A11Y-118",
        "note": "Third-party review widget markup"
      }
    ]
  }
}

5. Practical example: a CI pipeline for accessibility checks

In practice the accessibility job runs as a standalone step alongside the functional E2E tests, typically on every pull request against the main branch. The job installs dependencies, starts Playwright with Chromium, runs the accessibility suite against a staging environment, and writes the raw results out as a JSON report. A separate script then diffs that report against the baseline file from the previous section and decides whether the job succeeds or fails.

What matters for team buy-in is uploading the report as an artifact regardless of whether the job passed. That way every developer can trace the concrete violations, including the affected DOM nodes, straight from the pipeline output instead of having to re-run the test locally. GitHub Actions' annotation syntax ::error:: additionally surfaces new violations as inline comments directly on the pull request.

# .github/workflows/accessibility.yml
name: Accessibility Tests
on:
  pull_request:
    branches: [main]

jobs:
  axe-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Run accessibility suite against staging
        run: npx playwright test tests/accessibility --reporter=json > axe-results.json
      - name: Diff against known baseline
        run: node scripts/diff-axe-baseline.js axe-results.json baseline/accessibility.json
        # Exits non-zero only when NEW violations are found
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: axe-report
          path: axe-results.json

6. What manual review still has to cover

Automated scans do not replace hands-on keyboard testing. Only a human who consistently navigates a page with Tab reliably notices whether focus moves in a logical order, whether a modal correctly traps focus and releases it again on close, or whether a dropdown is a keyboard trap you cannot escape. axe-core can check whether an element is focusable, but not whether the focus order actually makes sense.

Testing with a real screen reader such as NVDA, JAWS or VoiceOver remains just as essential: is a form error actually announced the moment it appears? Is the order in which content gets read out sensible for the context? On top of that you need cognitive and motor perspectives, for example whether time limits can be extended, or whether click targets are large enough for people with limited fine motor control. A realistic test plan therefore combines continuous CI checks with a fixed manual review cycle, for instance ahead of every major release.

7. Specifics for Magento and Hyva shops

Hyva shops render large parts of the interactivity client-side through Alpine.js: mini cart, filter overlays, quantity selectors and checkout steps all build up further after the initial HTML response. An axe-core scan that runs too early inspects an incomplete state with x-cloak elements that should already be visible, or components that only set their final ARIA attributes after Alpine initializes. In Playwright tests it helps to wait for a specific attribute or CSS class that only gets set after full hydration, rather than relying on a fixed timeout.

A second specific challenge comes from third-party components commonly embedded into Magento shops: payment widgets, cookie banners, review plugins or chat widgets often ship their own markup that you do not control and whose violations you cannot fix immediately. These areas should consistently be excluded from the automated scan via .exclude() and addressed through a separate process with the respective vendor instead, so your own baseline does not get inflated with someone else's problems.

8. Reporting, ownership and team workflow

A CI job that only reports "red" or "green" does not help much day to day if nobody knows who owns a given violation. What works well is a diff script that outputs a structured message for every new violation, including rule ID, severity and affected DOM nodes, and that only fails the build when genuinely new problems appear. axe-core automatically classifies every violation with an impact value ranging from minor to critical, which maps directly onto prioritization in a ticket system.

In practice it works well to automatically file critical and serious violations as tickets and assign them to the responsible team, while smaller violations get batched into a weekly review. A trend dashboard that visualizes the number of open violations over time makes progress visible and prevents new legacy issues from quietly accumulating in the baseline, since every baseline addition should be deliberate and justified.

// scripts/diff-axe-baseline.js
import fs from 'node:fs';

const [, , resultsPath, baselinePath] = process.argv;
const results = JSON.parse(fs.readFileSync(resultsPath, 'utf8'));
const baseline = JSON.parse(fs.readFileSync(baselinePath, 'utf8'));

const knownIds = new Set(baseline.baseline.knownViolations.map(v => v.id));
const newViolations = results.violations.filter(v => !knownIds.has(v.id));

if (newViolations.length > 0) {
  for (const v of newViolations) {
    console.log(`::error::New a11y violation "${v.id}" (${v.impact}) on ${v.nodes.length} node(s)`);
  }
  process.exit(1); // fail the build only for regressions
}

console.log(`OK: ${results.violations.length} known violations unchanged, no regressions.`);
process.exit(0);

9. Tools and approaches compared

Several established tools exist for automated accessibility testing, and they differ in depth, integration effort and focus. axe-core usually forms the foundation other tools build on, while manual methods target exactly the gaps no automated scanner can close. The table below sets typical tasks against the unsafe approach and the recommended one.

No single tool covers the full breadth of WCAG criteria, which is why combining an automated scan in the CI pipeline with a fixed manual review cycle has become the practical standard. Relying on axe-core alone creates a false sense of security, relying on manual testing alone loses the continuous regression protection you get from checking every commit.

10. Summary

The core building blocks for automated accessibility tests in the CI pipeline solve a recurring problem: a one-time audit goes stale the moment the next commit merges. axe-core, integrated into the E2E suite you already run with Playwright or Cypress, checks every pull request automatically against the WCAG success criteria and realistically catches roughly a third of all violations. A baseline strategy ensures the build only fails on new, previously unknown violations, while documented legacy issues get paid down in a controlled way instead of blocking the pipeline permanently.

The key point remains: automation is a safety net against regressions, not a substitute for manual review. Keyboard navigation, screen reader testing and the semantic quality of descriptive text still require human judgment. Combining both layers, continuous CI checks and a fixed manual review cycle, turns accessibility into a permanent part of the development process instead of an annual compliance project.

11. FAQ: Automated Accessibility Tests in the CI Pipeline