Avoiding Flaky Tests: Causes and Robust Solutions

1. What makes a test "flaky"

A flaky test is a test that passes sometimes and fails other times without any change to the application code and without any change to the test code. That fundamentally distinguishes it from a test that deterministically fails and reveals a genuine bug, and from a test that only breaks in a specific environment. Flakiness is nondeterministic: the same commit, the same test run command, the same test database, and yet the outcome switches between pass and fail. This property makes flaky tests especially hard to debug, because a single failed run cannot be reliably reproduced.

At the system and E2E level, where Cypress and Playwright exercise the entire stack of browser, network, backend and database at once, there are structurally more moving parts than in an isolated PHPUnit test of a single class. It is precisely these additional layers, asynchronous rendering, network latency, shared database state, that are the real source of most flakiness cases. The damage is real: a team that habitually restarts red pipelines because "the test flickers sometimes anyway" eventually loses the ability to tell a genuine regression apart from known noise.

2. Timing and race condition problems

Race conditions arise when a test moves faster than the application itself. A click on a button triggers an asynchronous API call, but the test checks for the success message before the response has even come back. The opposite is just as common: a React or Vue re-render needs one extra tick, a modal only becomes truly interactive after a CSS transition, or an Alpine.js x-init hook only sets the initial state after the first paint. To the test, the element already looks present in the DOM, but it is not yet functionally ready.

The reflexive fix is a hardcoded cy.wait(2000), which papers over the symptom in the short term without addressing the actual cause: on slower CI runners the wait time suddenly is not enough, on faster ones it wastes runtime for nothing. Cypress commands like cy.get() and assertions like should() are already built to retry, polling the application up to a configurable timeout instead of checking just once. Waiting for a concrete state change instead, for example a network alias with cy.wait('@getOrders') or a visible success message with should('be.visible'), couples the test to actual application behavior rather than an estimated span of time.

// BEFORE: fragile, hardcoded wait assumes a fixed timing
describe('checkout', () => {
  it('shows order confirmation', () => {
    cy.get('[data-testid="place-order"]').click();
    cy.wait(2000); // guesswork: too short on slow CI, needlessly long on fast CI
    cy.get('[data-testid="confirmation-message"]').should('contain', 'Thank you');
  });
});

// AFTER: wait for the actual network response, retry-ability handles the rest
describe('checkout', () => {
  it('shows order confirmation', () => {
    cy.intercept('POST', '/rest/V1/carts/*/order').as('placeOrder');

    cy.get('[data-testid="place-order"]').click();
    cy.wait('@placeOrder').its('response.statusCode').should('eq', 200);

    // Cypress retries this assertion automatically until it passes or times out
    cy.get('[data-testid="confirmation-message"]').should('contain', 'Thank you');
  });
});

Playwright pursues the same underlying idea even more consistently: every locator action such as click() or fill() automatically runs actionability checks before execution, verifying that the element is visible, enabled and not covered by another element. This removes the need for manual wait logic for most timing problems from the outset, and tests that still need explicit waits in Cypress often run stably in Playwright without any adjustment.

3. Shared test state as a source of failure

A classic flakiness pattern in E2E suites: test A adds a product to the cart through the UI, test B implicitly assumes the cart from test A still exists, and both share the same test account. If the suite runs sequentially in a fixed order, this works by coincidence. As soon as tests run in parallel across multiple workers or the execution order changes due to sharding, two tests collide on the same record, and whichever test happens to run second fails, even though the code is correct.

The robust approach consistently separates test data per test case: each test creates its own customer, its own order or its own cart, ideally through a direct API call rather than through the UI, because that is faster and does not introduce additional timing dependencies. Unique identifiers, for example via a uuid or a timestamp in the email prefix, prevent collisions between parallel workers. In addition, every test environment should either be reset per test run, or the database changes should be rolled back via a transaction at the end of each test, so that no test depends on side effects from a previous one.

// BEFORE: test relies on UI state left behind by a previous test
test('customer sees their cart total', async ({ page }) => {
  await page.goto('/checkout/cart'); // assumes an item was already added earlier
  await expect(page.getByTestId('cart-total')).toHaveText('$49.90');
});

// AFTER: create an isolated cart via the API before the test
import { test as base, expect } from '@playwright/test';

const test = base.extend({
  seededCart: async ({ request }, use) => {
    const customerEmail = `qa-${Date.now()}@example.com`;
    const response = await request.post('/rest/V1/guest-carts', {
      data: { email: customerEmail, items: [{ sku: 'MS-1234', qty: 1 }] }
    });
    const { cartId } = await response.json();
    await use({ cartId, customerEmail });
  }
});

test('customer sees their cart total', async ({ page, seededCart }) => {
  await page.goto(`/checkout/cart?cart_id=${seededCart.cartId}`);
  await expect(page.getByTestId('cart-total')).toHaveText('$49.90');
});

4. Network dependencies and external services

As soon as an E2E test talks to a genuine third-party API, for example a payment sandbox, a shipping provider or an email provider, the test automatically imports that provider's availability and latency into its own suite. A brief sandbox outage, a rate limit, or simply a slow response on a busy day leads to a failing test even though the application itself works flawlessly. Consent banners or A/B testing frameworks from third parties are particularly treacherous, because they serve different markup depending on IP address or a random number, causing selectors to match inconsistently.

The robust solution is to deliberately stub external dependencies in most tests instead of hitting them live. With cy.intercept() in Cypress or page.route() in Playwright, any HTTP request can be intercepted and answered with a fixed, deterministic response, so the latency and availability of the real services no longer matter. It is important not to treat this as a blanket excuse for full isolation: a small number of genuine smoke tests against the actual integration remains necessary to detect when the third-party API's response structure actually changes and the stubbed fixtures go stale.

// Stub the shipping rate API so the test does not depend on a live sandbox
cy.intercept('GET', '/rest/V1/shipping/rates*', {
  statusCode: 200,
  body: [
    { carrier: 'standard', label: 'Standard Shipping', price: 4.99 },
    { carrier: 'express', label: 'Express Shipping', price: 12.99 }
  ]
}).as('shippingRates');

cy.visit('/checkout/shipping');
cy.wait('@shippingRates');
cy.get('[data-testid="shipping-option-express"]').click();
cy.get('[data-testid="shipping-total"]').should('contain', '12.99');

5. Animations and time-dependent UI

CSS transitions and JavaScript animations are an underestimated source of flakiness, because they produce a state that is neither clearly "done" nor clearly "not done". A modal that fades in over 300 milliseconds already exists in the DOM as far as the test is concerned, but is partially transparent or not yet clickable during the transition, because pointer-events only activate once the animation finishes. Toast notifications that automatically close after three seconds create a narrow window in which an assertion randomly catches the element too late, especially when the CI runner reacts more slowly under load than the local development environment.

The most reliable fix is to disable animations in the test environment altogether, or shorten them drastically, for example via a CSS rule that sets all transition- and animation-duration values to 0ms behind a test query parameter or an environment variable. Where that is not possible, the assertion should not target a visual intermediate state, but a semantic end signal such as an aria-expanded attribute, an added CSS class, or a data-state attribute that the application only sets once the transition has finished. Playwright automatically waits for running CSS transitions to finish before screenshot comparisons, which additionally defuses this class of error in visual tests.

6. Robust selector strategies

Selectors that depend on implementation details are the second most common trigger of flakiness after timing problems. A selector like .flex.items-center.gap-2 breaks with every Tailwind refactor, because utility classes express styling, not a stable identity. nth-child(3) selectors break as soon as the order of elements in the markup changes, for example due to a new feature flag or a conditional display. Text-based selectors break with every copy change or localization effort, because the visible text and the test logic are directly coupled even though both should be free to change independently.

data-testid attributes solve this problem because they form an explicit contract between application code and test code that is completely independent of CSS refactors and copy changes. Playwright goes a step further with role-based locators such as getByRole(), getByLabel() and getByText(): these locators mirror how real users and screen readers find an element, and additionally reward accessible markup, because a button without the correct role suddenly becomes harder to select. In practice, a hierarchy proves itself: role-based locators first, data-testid as a fallback for elements without a meaningful semantic role, and CSS selectors only as a last resort.

// Role-based locators mirror how real users find elements, and wait automatically
await page.getByRole('button', { name: 'Add to cart' }).click();

// Web-first assertions retry automatically until the condition holds or the timeout is hit
await expect(page.getByTestId('cart-count')).toHaveText('1');

// Fallback: data-testid for elements without a meaningful semantic role
await expect(page.getByTestId('mini-cart-total')).toHaveText('$49.90');

// Avoid: fragile selectors that depend on styling or DOM position
// await page.locator('.flex.items-center.gap-2 > div:nth-child(3)').click();

7. Retry mechanisms versus fixing the root cause

Retry mechanisms such as Cypress' retries: { runMode: 2, openMode: 0 } or Playwright's retries configuration re-run a failed test up to a defined number of times before the suite finally marks it as red. That is a legitimate safety net against genuine infrastructure noise, such as a brief DNS hiccup on the CI network, but it becomes a problem once teams use retries as their primary strategy against flakiness. A test that turns green on the second attempt is silently counted as a success, even though the underlying race condition or the fragile selector problem remains untouched in the code.

Retries should be understood as a measurement tool, not a repair. Playwright's HTML report explicitly flags retried tests, and that same information belongs in a dashboard that makes the retry rate per test visible over time. A test that occasionally needs one retry because a CI runner was briefly overloaded is unremarkable. A test that needs a retry every third run has a structural problem that deserves its own ticket, not yet another bump to the retry count. The rule of thumb: retries may cushion the symptoms of a rare, genuine infrastructure disruption, but they must never permanently hide a known, reproducible cause.

8. Quarantining flaky tests instead of ignoring them

Simply disabling a known flaky test with .skip is the beginning of the end for test coverage: the test disappears from the team's awareness, is never touched again, and the functionality it was supposed to protect goes untested in the long run. Quarantine is the deliberate alternative: a test flagged as flaky keeps running, but no longer blocks the main pipeline, and its result is made visible separately instead of turning the entire build red. That keeps the main pipeline trustworthy without actually losing test coverage.

For quarantine not to become a permanent dumping ground, it needs a process: a responsible owner, a ticket with context on the suspected cause, and ideally an expiry date after which the test must either be fixed or removed for good. A dashboard that shows the number of quarantined tests over time turns flakiness into a visible hygiene metric that should actively trend toward zero, instead of disappearing into a forgotten test file. Tag-based marking, for example with an @flaky grep tag in Cypress or a dedicated project in the Playwright configuration, cleanly separates blocking from non-blocking test runs on a technical level without removing the test from the suite.

# .github/workflows/e2e.yml
name: E2E Tests
on: [pull_request]

jobs:
  e2e-stable:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      # Main suite: excludes quarantined specs, blocks the merge on failure
      - name: Run stable E2E suite
        run: npx playwright test --grep-invert @flaky --retries=1

  e2e-quarantine:
    runs-on: ubuntu-latest
    continue-on-error: true # visible, but never blocks the pipeline
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      # Quarantined specs: run and get reported, but are non-blocking
      - name: Run quarantined E2E suite
        run: npx playwright test --grep @flaky --retries=2
      - name: Upload quarantine report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: quarantine-report
          path: playwright-report/

9. Flakiness causes at a glance

Each of the flakiness causes covered in this article has a characteristic fragile pattern and an equally characteristic robust alternative. The following overview summarizes the most important comparisons, sorted by frequency in typical Magento and Hyvä frontends.

In practice, these causes frequently overlap: a test with a fragile selector often fails precisely because it is searching for the wrong element during an animation, or a timing problem is made worse by shared state when two parallel workers manipulate the same record. Consistently combining the robust patterns from the table typically reduces the flakiness rate by an order of magnitude without changing the actual test logic.

10. Summary

Avoiding flaky tests does not mean eliminating randomness from an inherently nondeterministic environment, browser, network, backend state, but reducing the coupling between the test and these uncertain factors. Timing problems are solved by retry-capable assertions instead of hardcoded wait times. Shared state is solved by API-based test isolation with unique data per test run. Network dependencies are solved by deliberately stubbing third-party responses, combined with a small number of genuine smoke tests. Animations lose their flaky character as soon as they are disabled in the test environment or queried through semantic end states instead of visual intermediate states.

Retries and quarantine are not contradictions to this approach, but complementary tools: retries absorb rare, genuine infrastructure noise while simultaneously providing measurement data about which tests have structural problems. Quarantine keeps the main pipeline trustworthy without letting known problem cases silently vanish from the suite. The difference between a test suite that nobody trusts anymore and one that reliably reports genuine regressions rarely lies in a single big rewrite, but in the consistent application of these patterns across all specs.

11. FAQ: Avoiding Flaky Tests