Headless tests, artifacts, and parallelization without flakiness
Cypress tests that pass reliably on your local machine often fail unexpectedly in CI due to resource limits, mismatched viewports, and timing differences on shared runners. This article shows how to run Cypress headless in GitLab CI and GitHub Actions, capture video and screenshot artifacts on failure, cache node_modules and the Cypress binary, and parallelize test runs across multiple jobs.
Table of Contents
- 1. Why locally passing tests turn red in CI
- 2. Running Cypress headless: cypress run in detail
- 3. GitLab CI: a production-ready Cypress job
- 4. GitHub Actions: a Cypress workflow step by step
- 5. Capturing artifacts: video and screenshots on failure
- 6. Cypress Cloud recording vs. self-hosted artifacts
- 7. Caching: speeding up node_modules and the Cypress binary
- 8. CI-only flakiness: causes and mitigations
- 9. Parallelization across multiple CI jobs
- 10. Summary
- 11. FAQ
1. Why locally passing tests turn red in CI
A local development machine differs from a CI runner in almost every respect: more CPU cores, more RAM, a visible browser window instead of a headless renderer, and no competition from other jobs running in parallel on the same host. Cypress itself is deterministic, but the environment it runs in is not. This gap is the most common reason a test passes reliably locally yet fails sporadically in the pipeline.
Typical triggers are hardcoded wait times that are sufficient on a faster local machine but too short on a throttled shared runner, as well as animations and CSS transitions that are timed differently in a headless browser than in a visible window. Understanding these differences early avoids time-consuming debugging sessions and leads to tests that wait for explicit states instead of fixed time spans from the start.
2. Running Cypress headless: cypress run in detail
cypress open is built for interactive development and simply cannot be used in a CI environment without a graphical interface. For automated pipelines, cypress run is the right command: it executes all spec files headlessly in the background, returns a meaningful exit code, and terminates cleanly once every test has finished. A non-zero exit code reliably signals a failed test run to the pipeline, which is essential for the build status logic in both GitLab CI and GitHub Actions.
Explicitly choosing the browser via --browser chrome or --browser electron matters, since CI images often ship multiple browser binaries and the default is not always the fastest option. It's also worth running npx cypress verify before the actual test run: the command checks whether the Cypress binary is correctly installed and runnable, catching broken cache states before they turn into cryptic errors mid-run.
3. GitLab CI: a production-ready Cypress job
For GitLab CI, Cypress provides official Docker images with Chrome, Firefox, and Edge preinstalled, so no manual browser setup is needed inside the job. The job should point CYPRESS_CACHE_FOLDER explicitly into the project directory so GitLab's built-in cache mechanism can reuse the Cypress binary between pipeline runs instead of downloading it fresh every time.
The artifacts configuration with when: on_failure is crucial for debugging: videos and screenshots are only uploaded on actual failures, which saves storage and keeps the pipeline history readable. A limited expire_in window prevents unused artifacts from piling up in GitLab storage over months.
# .gitlab-ci.yml: run Cypress E2E tests headlessly in the pipeline
stages:
- test
cypress:e2e:
stage: test
image: cypress/browsers:node-20.11.0-chrome-123.0.6312.86-1-ff-124.0-edge-123.0.2420.65-1
variables:
CYPRESS_CACHE_FOLDER: "$CI_PROJECT_DIR/.cache/Cypress"
cache:
key:
files:
- package-lock.json
paths:
- node_modules/
- .cache/Cypress/
script:
- npm ci
- npx cypress verify
# Run headless inside the Chrome container, no Cypress Cloud recording
- npx cypress run --browser chrome --headless --record false
artifacts:
when: on_failure
expire_in: 7 days
paths:
- cypress/videos/
- cypress/screenshots/
4. GitHub Actions: a Cypress workflow step by step
The official cypress-io/github-action handles several steps in GitHub Actions with a single command: npm installation, Cypress binary caching, and the actual test run. The build and start parameters let you build and start the application server right before the test run, while wait-on makes sure Cypress only starts once the application is actually reachable, instead of running against a server that isn't ready yet.
Artifact upload uses the separate actions/upload-artifact action, gated by the if: failure() condition. This mirrors the GitLab CI logic exactly: storage costs and extra pipeline runtime from uploading video and screenshot files only occur on an actual failure.
# .github/workflows/cypress.yml: Cypress workflow using the official GitHub Action
name: Cypress E2E Tests
on:
pull_request:
push:
branches: [main]
jobs:
cypress-run:
runs-on: ubuntu-latest
steps:
- name: Check out the repository
uses: actions/checkout@v4
- name: Run Cypress (includes npm install and binary cache)
uses: cypress-io/github-action@v6
with:
browser: chrome
build: npm run build
start: npm run start:ci
wait-on: 'http://localhost:8080'
- name: Upload artifacts on failure
uses: actions/upload-artifact@v4
if: failure()
with:
name: cypress-artifacts
path: |
cypress/videos
cypress/screenshots
retention-days: 7
5. Capturing artifacts: video and screenshots on failure
Without a visible browser, a CI runner is a black box: a failed test without artifacts rarely provides enough context to reliably pin down the cause. Cypress records a video of the entire spec run by default and automatically takes a screenshot of the state at the moment of failure for every failed test, including an error message and stack trace overlay baked into the image.
In practice, it pays to keep videos only for actual failures, since full recordings of successful runs quickly add up to several gigabytes per pipeline. The after:spec hook in the Cypress configuration allows exactly this kind of programmatic filtering, so only the recordings that are actually relevant for debugging are kept, while successful runs stay lean as they move through the pipeline.
// cypress.config.js: configure video and screenshot capture for CI runs
const { defineConfig } = require('cypress');
module.exports = defineConfig({
video: true,
videoCompression: 32,
screenshotOnRunFailure: true,
trashAssetsBeforeRuns: true,
retries: {
runMode: 2, // Retry twice in CI to absorb flakiness
openMode: 0, // No automatic retries locally in interactive mode
},
viewportWidth: 1280,
viewportHeight: 720,
e2e: {
baseUrl: 'http://localhost:8080',
setupNodeEvents(on, config) {
// Keep videos in CI only for failed specs
on('after:spec', (spec, results) => {
if (results && results.video && results.stats.failures === 0) {
return require('fs').promises.unlink(results.video).catch(() => {});
}
});
},
},
});
6. Cypress Cloud recording vs. self-hosted artifacts
Cypress Cloud takes over video and screenshot hosting, provides a searchable test history across all runs, and, using --record together with a project token, automatically merges parallel test runs into a single overall result. For teams with many parallel jobs and frequent merge requests, this significantly reduces the manual effort of managing artifacts, but it costs a monthly fee depending on usage and sends test data to an external service.
The self-hosted alternative, using artifacts uploads in GitLab CI or GitHub Actions, is free, keeps all data inside your own infrastructure perimeter, and works well for projects with strict privacy requirements or no Cypress Cloud budget. The downside: results from multiple parallel jobs have to be merged manually, and no searchable history across many pipeline runs exists without additional tooling.
7. Caching: speeding up node_modules and the Cypress binary
The Cypress binary is a standalone Electron package several hundred megabytes in size, downloaded separately from the npm packages. Without caching, every pipeline run downloads both node_modules and the binary from scratch, which, depending on the runner's network connection, costs several minutes per job before the first test even starts.
A cache key based on the hash of package-lock.json ensures the cache is automatically invalidated when dependencies change, rather than serving stale packages. Both GitLab CI and GitHub Actions ship built-in cache mechanisms; for setups outside these platforms or with a custom runner fleet, the same principle can be replicated with a simple shell script that restores the cache directory before the test run and updates it afterward.
#!/usr/bin/env bash
# scripts/ci-cache-restore.sh: cache node_modules and the Cypress binary explicitly
set -euo pipefail
CACHE_KEY=$(sha256sum package-lock.json | awk '{print $1}')
CACHE_DIR="/cache/cypress-ci/${CACHE_KEY}"
if [ -d "$CACHE_DIR/node_modules" ]; then
echo "Cache hit: restoring node_modules"
cp -r "$CACHE_DIR/node_modules" ./node_modules
cp -r "$CACHE_DIR/cypress-binary" "$HOME/.cache/Cypress"
else
echo "Cache miss: installing npm packages and Cypress binary from scratch"
npm ci
npx cypress install
mkdir -p "$CACHE_DIR"
cp -r ./node_modules "$CACHE_DIR/node_modules"
cp -r "$HOME/.cache/Cypress" "$CACHE_DIR/cypress-binary"
fi
8. CI-only flakiness: causes and mitigations
Three causes explain most cases of flakiness that only show up in CI: first, CPU throttling on shared runners, when multiple jobs run simultaneously on the same host and rendering as well as JavaScript execution become noticeably slower than locally. Second, mismatched viewport and DPI settings: the default headless viewport often differs from the window size used during local development and manual testing, leading to different responsive behavior.
Third, network and timing variance between containers on the same host, which produces different response times even when frontend and backend run in the same pipeline job. The most effective mitigation for all three causes is the same principle: never wait for a fixed time span, but wait for an explicit state using cy.intercept() and automatic retries in assertions, combined with a fixed viewport in the configuration instead of a platform-dependent default.
9. Parallelization across multiple CI jobs
As a test suite grows, the runtime of a single, sequential Cypress job quickly climbs to ten minutes or more. Parallelization distributes spec files across multiple concurrently running jobs or machines and reduces total runtime roughly in proportion to the number of parallel instances. Cypress Cloud handles the distribution automatically and intelligently, based on the historical runtime of each spec file, so jobs stay evenly loaded.
Without Cypress Cloud, parallelization can also be implemented manually using a GitLab CI matrix or a GitHub Actions job matrix with fixed spec groups, for example split by feature area like checkout or catalog. The downside of the manual approach: if runtime is unevenly distributed across groups, the whole pipeline waits on the slowest job, whereas Cypress Cloud's intelligent distribution automatically balances that imbalance out.
{
"scripts": {
"cypress:run": "cypress run --browser chrome --headless",
"cypress:run:parallel": "cypress run --record --parallel --ci-build-id $CI_PIPELINE_ID --group ci-parallel",
"cypress:run:group-checkout": "cypress run --spec cypress/e2e/checkout/**/* --headless",
"cypress:run:group-catalog": "cypress run --spec cypress/e2e/catalog/**/* --headless"
}
}
The table below summarizes how a local test run differs from a run on a shared CI runner across the dimensions that matter most.
| Dimension | Local development | CI (shared runner) | Recommended mitigation |
|---|---|---|---|
| Viewport / resolution | Developer's window size | Mismatched headless default | Fixed viewportWidth/Height in config |
| CPU / RAM | Dedicated workstation | Shared runner, throttling under load | Parallelize jobs instead of stacking serially |
| Network / timing | Local dev server, low latency | Variable container-to-container latency | cy.intercept() instead of fixed waits |
| Artifacts | Interactive debugging in the browser | No direct access to the runner | Upload video + screenshots as artifacts |
| Parallelization | One test run per developer | Multiple jobs/machines possible | Cypress Cloud or job matrix with spec split |
Mironsoft
Cypress CI integration and E2E test automation for Magento and Hyvä stores
Looking for a stable Cypress pipeline?
We set up your Cypress tests for GitLab CI or GitHub Actions, fix CI-only flakiness, and use caching and parallelization to keep your pipelines fast and reliable.
CI pipeline audit
Review of existing GitLab CI or GitHub Actions jobs for weak spots
Flakiness analysis
Root cause analysis for sporadic failures and targeted fixes
Setup & parallelization
Caching, artifact handling, and job matrices for faster pipelines
10. Summary
A reliable Cypress CI integration solves a clearly defined problem: tests that pass locally must also stay reproducibly green on a shared, resource-constrained runner. Headless execution via cypress run in GitLab CI or GitHub Actions forms the foundation, complemented by video and screenshot artifacts that are automatically captured on every failure and make debugging possible at all without direct access to the runner.
Caching node_modules and the Cypress binary noticeably shortens pipeline runtime, while specifically addressing CPU throttling, viewport mismatches, and timing variance eliminates the most common causes of CI-only flakiness. Parallelizing test runs across multiple jobs, whether through Cypress Cloud or a manual job matrix, further reduces total pipeline runtime roughly in proportion to the number of parallel instances and keeps the feedback loop short for the entire team.
Cypress in the CI Pipeline - The Essentials at a Glance
Headless execution
cypress run instead of cypress open, an explicit browser, and cypress verify before the test run.
Capturing artifacts
Upload video and screenshots only on failure, configure a limited retention period.
Caching
Cache node_modules and the Cypress binary via the package-lock.json hash, avoid repeat downloads per run.
Flakiness & parallelization
Explicit states instead of fixed waits, fixed viewports, spec splitting across multiple CI jobs.