Selective Testing: Running Only Affected Tests

1. Why running the full suite on every commit does not scale

An E2E suite that runs in five minutes at the start of a project grows with every new feature: 50 Cypress specs quickly turn into 500, and five minutes of runtime turns into forty. When the full suite runs on every commit or every push to a feature branch, the wait time for the first green or red result grows proportionally with the codebase, not with the size of the actual change. Developers who wait forty minutes on CI for a one-line fix lose focus, switch context, and review pull requests less promptly.

The problem intensifies with parallel CI infrastructure: more runners speed up wall-clock time, but costs scale linearly with suite size without solving the underlying problem. On top of that, the statistical probability of at least one flaky test failing grows with test count: with 1000 specs at a 0.1% flake rate each, statistically almost every tenth run fails at least once for no real reason. Teams then start ignoring red pipelines or disabling tests, which undermines the actual safety benefit the E2E suite was supposed to provide.

2. Tagging tests by feature area: Cypress and Playwright patterns

The easiest entry point into selective testing is tagging tests by feature area or business domain. In Playwright, a tag can be attached directly in the test name or via test.describe with an annotation string like @checkout; in Cypress, the community plugin cypress-grep handles the same job through test titles or Cypress.env. The CLI flags --grep or grepTags then let you run only the subset that belongs to a changed feature area: a checkout bug fix triggers only @checkout tests, not catalog logic.

Tagging is quick to implement and immediately readable for humans, but it has a structural weakness: the mapping is manual and drifts apart over time when developers commit new tests without a tag or move code into a different area without adjusting the tags. A CI check that blocks untagged specs or validates tags against a directory schema keeps tag discipline intact long term. Tagging works best as a fast first tier, combined with more precise, automated mechanisms for higher reliability.

// tests/checkout/checkout-flow.spec.ts
import { test, expect } from '@playwright/test';

// Tag this spec with its feature area for selective execution
test.describe('Checkout flow @checkout @critical', () => {
  test('customer can complete checkout with saved address @smoke', async ({ page }) => {
    await page.goto('/checkout');
    await page.getByRole('button', { name: 'Place order' }).click();
    await expect(page.getByText('Thank you for your order')).toBeVisible();
  });

  test('checkout blocks submission with invalid payment data', async ({ page }) => {
    await page.goto('/checkout');
    await page.getByLabel('Card number').fill('0000');
    await page.getByRole('button', { name: 'Place order' }).click();
    await expect(page.getByText('Invalid card number')).toBeVisible();
  });
});

3. Dependency-graph-based test selection

More precise than manual tagging is dependency-graph-based test selection: a tool builds a graph from the code's imports and module dependencies and derives from it which test files are transitively affected by a changed source file. For pure frontend or Node codebases, tools like dependency-cruiser or madge handle building the graph; for Magento backends with PHP modules, an explicit module-to-test manifest that maps directory paths to associated spec files is usually the more practical choice.

The advantage over tags: the graph reflects real code dependencies instead of human judgment, which significantly lowers the false-negative rate, meaning affected tests that get overlooked. The downside is the upfront effort: a dependency graph needs to be maintained, recomputed after refactorings, and validated against actual test results before a team can trust it. In practice, a hybrid approach works well, combining tags for fast manual selection with the graph as an automated, more reliable fallback layer.

{
  "modules": {
    "checkout": {
      "sourcePaths": ["src/checkout", "app/code/Mironsoft/Checkout"],
      "specs": ["tests/checkout/checkout-flow.spec.ts", "tests/checkout/payment.spec.ts"]
    },
    "catalog": {
      "sourcePaths": ["src/catalog", "app/code/Mironsoft/Catalog"],
      "specs": ["tests/catalog/product-listing.spec.ts", "tests/catalog/filters.spec.ts"]
    },
    "cart": {
      "sourcePaths": ["src/cart", "app/code/Mironsoft/Cart"],
      "specs": ["tests/cart/add-to-cart.spec.ts", "tests/cart/mini-cart.spec.ts"]
    },
    "core": {
      "sourcePaths": ["src/shared", "app/code/Mironsoft/Core"],
      "specs": ["*"]
    }
  }
}

4. Git-diff heuristics: from changed files to affected tests

A low-effort alternative to a full dependency graph is a git-diff heuristic: a script uses git diff --name-only to find all files changed relative to the target branch and maps them to affected feature tags or test directories via a simple path-prefix scheme. If a commit only changes files under src/checkout/, only @checkout tags run; if the change touches a shared module, a deliberately conservative fallback to the full suite kicks in.

The critical design mistake many teams make early on: they map files only at the directory level, without accounting for transitive dependencies. A changed shared utility module can then potentially affect dozens of feature areas that a pure path heuristic fails to recognize. Every git-diff heuristic should therefore include an explicit allowlist of critical, shared paths that automatically trigger the full suite instead of a subset when changed, rather than silently skipping tests.

#!/usr/bin/env bash
# scripts/select-affected-tests.sh
# Maps changed files against a base branch to affected E2E specs

set -euo pipefail

BASE_BRANCH="${1:-origin/main}"
CHANGED_FILES=$(git diff --name-only "$BASE_BRANCH"...HEAD)

AFFECTED_TAGS=()

while IFS= read -r file; do
  case "$file" in
    src/checkout/*|app/code/*/Checkout/*)
      AFFECTED_TAGS+=("@checkout")
      ;;
    src/catalog/*|app/code/*/Catalog/*)
      AFFECTED_TAGS+=("@catalog")
      ;;
    src/cart/*|app/code/*/Cart/*)
      AFFECTED_TAGS+=("@cart")
      ;;
    src/shared/*|app/code/*/Core/*)
      # Shared/core changes are high risk: fall back to the full suite
      echo "Shared module changed, running full suite"
      exit 0
      ;;
  esac
done <<< "$CHANGED_FILES"

UNIQUE_TAGS=$(printf '%s\n' "${AFFECTED_TAGS[@]}" | sort -u | paste -sd '|')
echo "Running tests matching: ${UNIQUE_TAGS:-@smoke}"
npx playwright test --grep "${UNIQUE_TAGS:-@smoke}"

5. Selective tests on every PR, full suite on a schedule

The most effective combination in practice deliberately separates two timelines: every pull request runs a selective test set tailored to the actual change, delivering feedback within minutes without pulling developers out of context. The full regression run across all specs, by contrast, happens on a fixed schedule, typically overnight, and is additionally mandatory before every merge into the main branch or before a release tag.

This separation resolves the apparent conflict between speed and completeness: developers get fast feedback on exactly the change they are working on, while the team as a whole still regularly gets the full guarantee that no combination of several small PRs has introduced a regression that no single selective run could have caught on its own. It is important that the nightly run is visibly monitored and that failures do not quietly disappear in a Slack channel.

6. The risk of missed regressions and how to mitigate it

Selective testing has an inherent blind spot: a change can trigger side effects in an area that neither tags, dependency graph, nor git-diff heuristic recognize as affected, such as a global CSS rule, shared state in an Alpine.js store, or a change to a third-party library that touches several feature areas at once. Teams that rely exclusively on selective runs and skip full runs entirely accumulate this risk unnoticed over weeks.

The most effective countermeasure is the already-mentioned periodic full regression run as a safety net, which reliably catches overlooked combinations before they reach production. It also pays off to actively track the selection logic's hit rate: whenever a nightly full run finds a failure that was not included in the corresponding PR selection, that is a clear signal to sharpen the dependency graph or the path heuristic. This feedback loop makes selection quality measurable over time instead of a pure guess.

7. Tooling examples: Nx affected, Turborepo, and custom manifests

In monorepo environments, tools like nx affected and Turborepo's change-detection graph have become established. They were originally built for build and unit-test selection but transfer just as well to E2E test selection: both tools compute from the project's dependency graph which packages are affected by a change, and can be configured via a custom target to trigger a playwright test --grep call with matching tags instead of npm test.

For Magento- and PHP-heavy projects without an Nx or Turborepo monorepo structure, a hand-written JSON manifest is often the more pragmatic solution: a single file maps module directories to associated spec files and gets evaluated by a simple Node or bash script. The advantage over a fully automated graph is transparency: everyone on the team can trace and maintain the mapping in one file without having to understand or debug an additional analysis tool.

# Run only checkout-tagged specs with Playwright
npx playwright test --grep "@checkout"

# Run smoke tests across all feature areas for a fast PR gate
npx playwright test --grep "@smoke"

# Cypress equivalent using the cypress-grep plugin
npx cypress run --env grepTags="@checkout"

# Exclude flaky/quarantined specs while still running the affected set
npx playwright test --grep "@checkout" --grep-invert "@quarantine"

8. Practical CI pipeline design: fast PR checks plus a nightly full regression run

A resilient CI pipeline for E2E tests technically separates the selective PR check from the full regression run into two independent jobs. The PR job checks out the branch with full git history, computes the affected tags via the diff script, and runs only those, typically finishing in under ten minutes. The second job runs exclusively on a cron schedule or as a mandatory gate before merging into main, and runs the complete suite, usually split into several shards.

Sharding is essential for full runs to keep runtime manageable despite full coverage: Playwright's --shard=N/M flag or Cypress's parallelization across multiple runners distribute the total set of specs across parallel CI machines. If the nightly full run fails, the team should be notified automatically, for example via a dedicated Slack channel or a GitHub issue, instead of only noticing the failure the next time someone manually checks the CI history.

# .github/workflows/e2e.yml
name: E2E Tests

on:
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 2 * * *'  # Nightly full regression run at 02:00 UTC

jobs:
  selective-pr-tests:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: npm ci
      - name: Run only affected E2E specs
        run: ./scripts/select-affected-tests.sh origin/main

  full-regression:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - name: Run full E2E suite (sharded)
        run: npx playwright test --shard=${{ matrix.shard }}/4

9. Selective test strategies compared

Each of the selection strategies presented has a specific weakness pattern and a proven counterpart that keeps the risk under control in practice. The table below summarizes exactly what matters for each strategy.

In practice, the most successful setups combine several of these strategies: tags for fast manual categorization, a dependency graph or manifest for precision, git-diff heuristics as a fast first filter, and a periodic full regression run as a non-negotiable safety net against anything the automated selection misses.

10. Summary

Selective testing solves a concrete scaling problem: the bigger an E2E suite grows, the more expensive and slower it becomes to run it in full on every commit, without the insight gained growing proportionally. Feature tags provide a fast entry point, dependency graphs and manifests increase precision, and git-diff heuristics connect both to the actual change in a commit or pull request.

What matters most for long-term success is deliberately combining fast selective PR checks with a non-negotiable, regular full regression run as a safety net. Teams that actively measure the selection logic's hit rate and sharpen it after every missed failure build a test system over time that stays both fast and reliable, instead of having to choose between the two goals.

11. FAQ: Selective Testing for E2E Suites