Running E2E Tests Reliably in GitLab CI

1. Choosing the right Docker image for Cypress and Playwright

The choice of Docker image decides a large share of your CI runtime before the first test even runs. For Cypress, the official cypress/included image is the best choice, because the Cypress binary, Node.js, and all browser dependencies (Chrome, Electron) are already preinstalled, so there is no extra cypress install time. For Playwright, the same principle applies with mcr.microsoft.com/playwright, which ships Chromium, Firefox, and WebKit including all system dependencies, in exactly the version that matches the installed @playwright/test package version.

What matters most is pinning the image version explicitly to the test framework version locked in package.json, instead of using latest. A version drift between the Playwright package and the Playwright Docker image leads to cryptic browser launch failures that are not reproducible in local environments. A simple CI script that checks the installed package version against the image tag version reliably prevents this class of failure before it ever shows up in the pipeline.

# .gitlab-ci.yml: pin the image version exactly to the package version
e2e:playwright:
  image: mcr.microsoft.com/playwright:v1.47.0-jammy
  stage: test
  script:
    - npm ci
    # Make sure the image version matches the package version
    - npx playwright --version
    - npx playwright test --reporter=line

e2e:cypress:
  image: cypress/included:13.14.2
  stage: test
  script:
    - npm ci
    - npx cypress run --browser chrome

2. .gitlab-ci.yml stages and job structure for E2E tests

A clean stage structure separates build, deploy, and test cleanly: build produces the application artifact or Docker image, deploy provisions a test environment, and only then does test run the E2E suite against that environment. This separation lets the build job run once and pass its result to several parallel test jobs via needs, instead of rebuilding the application for every browser or every test shard.

Using needs instead of relying purely on stage ordering speeds up the pipeline further, since GitLab then executes a directed dependency graph instead of a strictly sequential chain of stages. The E2E job itself should always use rules instead of the deprecated only/except, combined with a top-level workflow: rules block, to avoid duplicate pipeline runs for the same commit on branch pushes and merge request events, and to trigger the job only on relevant path changes.

3. Caching node_modules and browser binaries between runs

Without caching, every pipeline run reinstalls node_modules and the browser binaries from scratch, which for Playwright alone means several hundred megabytes of Chromium, Firefox, and WebKit downloads. GitLab's cache directive with a key derived from package-lock.json ensures the cache only rebuilds on actual dependency changes and otherwise gets restored from the object storage backend, typically in a few seconds instead of several minutes.

When using the official Docker images, re-downloading the browser binaries is usually unnecessary anyway, since they are already bundled in the image. If a generic Node image is used instead, the Playwright or Cypress cache directory should additionally be added to cache.paths. It's important to choose policy: pull-push for the job that installs dependencies, and policy: pull for downstream test jobs, to avoid unnecessary cache uploads.

#!/usr/bin/env bash
# Check and prime the Playwright browser cache before running tests
set -euo pipefail

CACHE_DIR="$HOME/.cache/ms-playwright"

if [ -d "$CACHE_DIR" ] && [ "$(ls -A "$CACHE_DIR" 2>/dev/null)" ]; then
  echo "Playwright browser cache found, skipping download"
else
  echo "No cache found, installing browsers with system dependencies"
  npx playwright install --with-deps chromium
fi

du -sh "$CACHE_DIR" || true

4. Review apps versus shared staging: isolation against setup cost

GitLab Review Apps create a dynamic, isolated environment per merge request via environment: name: review/$CI_COMMIT_REF_SLUG, complete with its own database and fixtures. E2E tests then run against an instance nobody else is modifying at the same time, which effectively rules out flakiness caused by parallel test runs hitting the same test accounts. A review app also surfaces real deployment problems, such as broken reverse proxy configuration or indexer runs, that a purely local test environment would never reveal.

The price of this isolation is deploy time and infrastructure: every open merge request ties up its own container stack including a database, which adds up quickly with many parallel MRs. A shared staging environment is cheaper to run and available faster, but comes with risks from data drift: test accounts modified by another pipeline running at the same time, or leftover orders from a previous run that make an otherwise correct test fail. In practice, a combination works best: a lean smoke set against the review app per MR for fast, isolated feedback, and the full regression suite against a regularly reset staging environment.

5. Capturing screenshots, videos, and traces on failure

Without artifacts, a failed E2E test is nearly impossible to debug from job logs alone. artifacts: when: on_failure uploads screenshots, videos, and traces only on actual failures, keeping artifact storage manageable while preserving full context when something breaks. Cypress writes screenshots to cypress/screenshots and videos to cypress/videos by default; Playwright's trace: 'retain-on-failure' setting produces a full trace only for failing tests, which drastically reduces artifact size compared to 'on' for every single run.

expire_in is not a minor detail: without an explicit limit, a project's artifact quota fills up silently, while expire_in: 1 week for E2E failure artifacts strikes a sensible balance between debugging usefulness over the next few days and storage cost. Combined with the "browse artifacts" link right inside the merge request pipeline widget, a reviewer can inspect a failure screenshot without checking out the branch locally.

# .gitlab-ci.yml: upload artifacts only when the job fails
e2e:playwright:
  stage: test
  script:
    - npx playwright test --reporter=line,html
  artifacts:
    when: on_failure
    expire_in: 1 week
    paths:
      - test-results/
      - playwright-report/
    reports:
      junit: test-results/junit.xml

6. Gating merge requests on E2E success without the frustration

Under Settings > Merge requests, "Pipelines must succeed" can be enabled; combined with an E2E job that does not set allow_failure: true, a red E2E stage becomes a hard blocker on the merge button instead of a warning that can be ignored. Merge request approval rules requiring at least one reviewer approval on top of a green pipeline additionally prevent a reviewer from merging past a known-red E2E stage under time pressure.

The real danger appears when the entire regression suite becomes mandatory on every single push: a three-minute unit test pipeline turns into a 25-minute wait, which tempts developers to disable or ignore the check. The fix is not removing the gate, it's precisely controlling what gets gated at which stage of the merge lifecycle, as covered in the next section on parallelization.

7. Parallelization: smoke tests versus full regression

Splitting the suite into a critical smoke test set (login, add to cart, checkout, search) that must pass on every MR push, and a full regression suite that only runs on merge to main or on a nightly scheduled pipeline, is the single most effective lever against long CI wait times. GitLab's parallel keyword combined with Playwright's --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL splits even the smoke set across multiple runners, cutting wall-clock time roughly linearly with the number of runners.

A proven pattern: smoke specs get tagged with @smoke and run via rules filtered to every merge request event, while the nightly or main pipeline runs the entire, unfiltered suite across, say, four parallel shards. This keeps feedback on every push under five minutes while the full suite still runs often enough to catch regressions the smoke set misses.

# .gitlab-ci.yml: smoke set per MR, full regression only on main/nightly
e2e:smoke:
  stage: test
  script:
    - npx playwright test --grep @smoke
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

e2e:regression:
  stage: test
  parallel: 4
  script:
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
    - if: '$CI_PIPELINE_SOURCE == "schedule"'

8. Retry strategies for flaky tests in CI

Flaky tests, meaning tests that fail intermittently for reasons unrelated to an actual regression, such as animation timing, network jitter, or race conditions, undermine trust in the E2E gate faster than almost anything else: a suite that turns red for no reason twice a week guarantees nobody looks closely at the third, real failure. Both Cypress (retries: { runMode: 2, openMode: 0 }) and Playwright (retries: 2 in playwright.config.ts, usually enabled only via process.env.CI) support automatically re-running a failed test before marking it as truly failed.

Retries are a mitigation, not a fix: a test that needs three attempts to pass reliably almost always has a real root cause, most often a missing explicit wait for a network response or a DOM state instead of a fixed sleep. Tracking retry counts per spec over time, for example via the JSON or JUnit reporter output, surfaces the worst offenders for targeted fixes, rather than letting retries silently hide a growing pile of technical debt.

// playwright.config.js: retries and CI-specific parallelism
import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests/e2e',
  fullyParallel: true,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 4 : undefined,
  reporter: process.env.CI
    ? [['junit', { outputFile: 'test-results/junit.xml' }], ['html']]
    : 'list',
  use: {
    trace: 'retain-on-failure',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
    baseURL: process.env.E2E_BASE_URL,
  },
});

9. Managing secrets and environment variables for test credentials

Test credentials, meaning customer logins, admin accounts, or sandbox keys for payment providers, must never live in the repository or as plain-text strings in test code. GitLab CI/CD variables under Settings > CI/CD > Variables, marked Protected (only exposed on protected branches/tags) and Masked (automatically redacted from job logs), are the baseline. For review apps that need per-branch database seeding, a dedicated seed job using variables like E2E_ADMIN_USER and E2E_ADMIN_PASS keeps credentials out of fixture files checked into git.

Masking has limits: a masked variable that gets logged as part of a larger JSON blob or a base64-encoded string may not be caught by GitLab's simple substring redaction, so scripts should never print full request or response bodies that might contain secrets. For teams that need more than plain variables provide, GitLab's integration with HashiCorp Vault via ID tokens and OIDC allows fetching short-lived secrets at job runtime instead of storing long-lived credentials as CI variables at all.

In practice, these six areas reinforce each other: a cleanly pinned image doesn't help much if every run still discards the cache, and a perfect artifact configuration doesn't matter if the pipeline never finishes before the next push cancels it due to missing parallelization. Applying all six levers consistently produces an E2E gate developers actually trust, instead of one they route around.

10. Summary

Reliable E2E tests in GitLab CI don't come from a single measure, but from the interplay of several levers: the right, exactly pinned Docker image saves installation time, a cache key based on package-lock.json cuts repeat runs down to seconds, and deliberately separating isolated review apps for fast feedback from a shared staging environment for full regression resolves the tension between isolation and operational cost. Uploading artifacts only on failure with a sensible expire_in balances storage usage against debugging value.

Merge requests can be hard-gated via required status checks without making the pipeline unbearably slow, as long as a lean smoke set covers every push and the full suite only runs on merge to main or nightly. Retries absorb one-off flakiness but don't replace root-cause analysis, and test credentials belong consistently in protected and masked CI/CD variables or a Vault/OIDC integration, not in the test code itself.

11. FAQ: E2E Tests in GitLab CI