Test Retry Strategies Without Masking Flakiness

1. What test flakiness is and why blind retry-until-pass is dangerous

A flaky test is a test that passes sometimes and fails sometimes with no change to the code or the application's behavior. Causes range from genuine race conditions in the application, through network latency and animations, to test cases that interfere with each other via shared state. In larger E2E suites with hundreds of tests, some baseline level of flakiness is nearly unavoidable, but what actually matters is how a team responds to it.

The most obvious, and also the most dangerous, reaction is to raise the retry count globally, for example setting retries: 3 for the entire suite, without anyone checking which tests are actually affected. This automatically turns every intermittent failure into a permanently green status, without the underlying cause ever being investigated. This blind retry-until-pass approach is exactly the core of the problem: it doesn't eliminate flakiness, it just makes it invisible to everyone involved.

2. How retry-until-pass masks real bugs and erodes trust in the suite

A retry mechanism has no way to distinguish between a test that fails because of a brief network hiccup and one that is exposing a genuine race condition in the checkout code. Both get rerun using the exact same pattern until a green run eventually happens. For the application itself, this means a bug that occurs under load or under specific timing conditions stays in the code but is no longer reported by the test suite, simply papered over by repetition.

Over time, this undermines the credibility of the entire suite. Developers quickly learn that a single red run means nothing because it will turn green on the next attempt anyway, and start reflexively rerunning failed pipelines instead of investigating them. This behavior, often compared to the broken windows effect in quality engineering, tends to spread: once a team accepts that red doesn't necessarily mean a failure, the whole suite loses its function as a reliable signal.

3. Retry as a signal and triage tool, not a permanent fix

The decisive shift in perspective: retry is not a fix, it's a diagnostic tool. A test that only passes on the second or third attempt is providing valuable information, namely that something at that point is not deterministic, whether in the application or in the test infrastructure itself. That information must not be silently discarded just because the run ended up green. Instead, every retried-but-passed test should be automatically logged and flagged for later investigation.

In practice, this means retry results are captured separately, for example through a custom reporter or a test fixture that records the number of attempts per test. That data feeds into a dashboard or an automatically generated ticket list showing the responsible team which tests repeatedly only barely passed. Retry buys time for the pipeline, but it never buys relief from the actual investigation.

// Custom fixture: log when a test only passed after a retry
import { test as base, expect } from '@playwright/test';

export const test = base.extend({
  page: async ({ page }, use, testInfo) => {
    await use(page);

    // testInfo.retry is 0 on the first attempt, 1+ on retries
    if (testInfo.retry > 0 && testInfo.status === 'passed') {
      console.warn(
        `[FLAKY-SIGNAL] "${testInfo.title}" passed only after ` +
        `${testInfo.retry} retry(s). Flagging for investigation.`
      );

      // Persist to a triage attachment instead of silently accepting the pass
      await testInfo.attach('flaky-signal', {
        body: JSON.stringify({
          test: testInfo.title,
          file: testInfo.file,
          retries: testInfo.retry,
          project: testInfo.project.name,
        }),
        contentType: 'application/json',
      });
    }
  },
});

export { expect };

4. Retry configuration in Cypress: runMode vs. openMode

Cypress explicitly distinguishes between runMode, the headless CI run, and openMode, the interactive mode used during development, when it comes to retry configuration. This split is not an implementation detail, it's a deliberate design principle: in CI, one or two retries are often a reasonable compromise to absorb known infrastructure latency without turning the pipeline red over every small network stutter. In interactive mode, on the other hand, a developer should see every failure immediately, which is why openMode should almost always be set to 0.

It's important not to treat the retry configuration as a black box. The after:spec event lets you read out, for every test, how many attempts were actually needed before it passed. Logging that information directly during the test run, instead of letting it disappear into the Cypress Dashboard, is the simplest entry point into systematic flaky test tracking without any extra tooling.

const { defineConfig } = require('cypress');

module.exports = defineConfig({
  // Different retry strategy for CI (runMode) vs local debugging (openMode)
  retries: {
    // CI runs headless: absorb known environmental noise automatically
    runMode: 2,
    // Interactive mode: never hide a failure while a developer is watching
    openMode: 0,
  },

  e2e: {
    setupNodeEvents(on, config) {
      // Track retry attempts explicitly instead of letting them pass silently
      on('after:spec', (spec, results) => {
        const flaky = results.tests?.filter(
          (t) => t.attempts.length > 1 && t.state === 'passed'
        );

        if (flaky && flaky.length > 0) {
          flaky.forEach((t) => {
            console.warn(
              `[FLAKY-SIGNAL] ${spec.relative} > "${t.title.join(' > ')}" ` +
              `needed ${t.attempts.length} attempts to pass.`
            );
          });
        }
      });
    },
  },
});

5. Retry configuration in Playwright: global retries and test.describe.configure

Playwright handles retries globally through the retries option in the configuration file, typically enabled only for CI and left at 0 locally, so developers see failures immediately instead of having them retried away. On top of that, test.describe.configure() allows targeted exceptions for individual test blocks, for example a known-unstable checkout flow involving a third-party payment iframe, without raising the retry count for the entire suite.

The real value comes from a custom reporter that checks, on every onTestEnd call, whether result.retry is greater than 0 while the status is still passed. That exact combination marks a test as retried-but-passed and therefore a candidate for the triage list. Without such a reporter, the information about how many retries were needed vanishes irretrievably into Playwright's internal report the moment the pipeline as a whole turns green.

// playwright.config.js
import { defineConfig } from '@playwright/test';

export default defineConfig({
  // Global retry budget, only meaningful in CI
  retries: process.env.CI ? 2 : 0,
  reporter: [
    ['list'],
    ['./reporters/flaky-reporter.js'],
  ],

  projects: [
    {
      name: 'checkout-flow',
      // Known-unstable third-party payment iframe: slightly higher budget
      retries: process.env.CI ? 3 : 0,
    },
  ],
});

// reporters/flaky-reporter.js: flag tests that needed a retry to pass
export default class FlakyReporter {
  onTestEnd(test, result) {
    if (result.retry > 0 && result.status === 'passed') {
      console.warn(
        `[FLAKY-SIGNAL] "${test.title}" passed on retry ${result.retry}. ` +
        'Opening a triage ticket instead of treating this as fixed.'
      );
    }
  }
}

6. Distinguishing environmental flakiness from genuine application bugs

Not every flaky test points to a bug in the application. Environmental flakiness typically comes from network latency to external services, overloaded CI runners, insufficient explicit waits instead of robust selectors, or test-order dependencies caused by shared database state between tests. This class of failure can usually be brought under control with more robust selectors, isolated test data per run, and retry mechanisms with a clearly bounded budget.

Genuine application bugs, on the other hand, often show up as race conditions in the production code itself, for example when a cart update and a price calculation aren't properly synchronized. The most reliable way to tell the two apart: run the affected test in isolation, repeatedly, locally, while reviewing network logs, screenshots, and video, and check whether the failure correlates with a specific deployment. If the failure remains reproducible in isolation under stable conditions, it's almost always a genuine bug rather than an infrastructure hiccup.

7. Quarantine and known-flaky-test workflows

A pragmatic middle ground between fixing something immediately and ignoring it forever is quarantine: a test identified as flaky gets tagged, for example with @flaky, and removed from the blocking main pipeline run, while still continuing to run in a separate, non-blocking suite. This keeps the main pipeline meaningful while the test isn't forgotten, and it keeps producing data for diagnosis.

To stop quarantine from turning into a permanent dumping ground, the CI pipeline needs an automated gate that hard-stops the build once the flake rate crosses a defined threshold or once a certain number of quarantined tests is exceeded. Without that guardrail, quarantined tests pile up indefinitely and the original signal gets lost anyway, just one layer further down.

#!/usr/bin/env bash
# ci/check-flake-rate.sh - fail the build if flake rate exceeds threshold
set -euo pipefail

THRESHOLD=5.0
REPORT_FILE="reports/flaky-tests.json"

if [[ ! -f "$REPORT_FILE" ]]; then
  echo "No flaky-test report found, skipping flake-rate gate."
  exit 0
fi

FLAKE_RATE=$(jq '.flakeRatePercent' "$REPORT_FILE")

echo "Current flake rate: ${FLAKE_RATE}% (threshold: ${THRESHOLD}%)"

if (( $(echo "$FLAKE_RATE > $THRESHOLD" | bc -l) )); then
  echo "FAIL: flake rate ${FLAKE_RATE}% exceeds threshold of ${THRESHOLD}%."
  echo "Quarantine or fix the flagged tests before merging."
  exit 1
fi

echo "PASS: flake rate within acceptable range."

8. Metrics for flaky test management: retry rate and flake rate trend

Without measurement, flaky test management stays a matter of gut feeling. The most important metric at the test level is the retry rate per test: the share of runs in which a given test needed at least one retry to pass. A test with a retry rate of 40% is a far more urgent candidate for investigation than one that fails once a month because of a brief infrastructure blip.

At the suite level, it's worth tracking the flake rate as a trend over time rather than as a single point-in-time value. A gradual increase over several weeks often reveals creeping problems, such as growing test data volume or new shared fixtures, long before the pipeline as a whole becomes noticeably unstable. Both metrics can be extracted from Cypress and Playwright reporter data and consolidated into a simple JSON report that in turn feeds dashboards and automatic quarantine decisions.

{
  "generatedAt": "2026-07-10T06:15:00Z",
  "totalTests": 842,
  "flakeRatePercent": 3.1,
  "flaggedTests": [
    {
      "title": "checkout > applies discount code before payment",
      "file": "e2e/checkout.spec.js",
      "retryRate": 0.42,
      "lastPassedOnRetry": 1,
      "owner": "team-checkout",
      "status": "quarantined",
      "slaDueDate": "2026-07-24"
    },
    {
      "title": "search > shows autocomplete suggestions",
      "file": "e2e/search.spec.js",
      "retryRate": 0.18,
      "lastPassedOnRetry": 1,
      "owner": "team-search",
      "status": "investigating",
      "slaDueDate": "2026-07-19"
    }
  ]
}

9. Organizational process: owner assignment and SLA for flaky tests compared

Technical tooling alone doesn't solve the flakiness problem if nobody is actually responsible for the reported tests. A working process automatically assigns an owner to every test flagged as flaky, usually the team that owns the affected code or feature area, and sets a clear deadline, for example two weeks, to either fix the test or deliberately delete it. Without that deadline, quarantined tests pile up indefinitely because they no longer formally disrupt the pipeline.

The table below compares typical retry approaches with the recommended alternatives and shows exactly what matters at each step.

Ultimately, this process pays off twice over: the pipeline stays credible as a quality signal, and the number of real, undetected bugs drops because flakiness gets consistently investigated instead of repeatedly retried away.

10. Summary

Blind retry-until-pass doesn't solve a flakiness problem, it just moves it out of sight: real bugs stay in the code while the test suite loses its credibility and teams start reflexively rerunning red pipelines instead of investigating them. The sustainable approach treats retry as a diagnostic tool rather than a fix, explicitly logs every retried-but-passed test, and turns that into an assignment for investigation rather than a silent success.

Both Cypress, with its split between runMode and openMode, and Playwright, with global retries and targeted test.describe.configure overrides, provide the technical foundation for this. What actually matters, though, is the process wrapped around it: metrics like retry rate and flake rate trend, a quarantine workflow with a hard CI gate, and clear owner assignments with deadlines are what keep flakiness from turning into permanently ignored background noise.

11. FAQ: Test Retry Strategies