Detecting and Systematically Eliminating Flaky Tests in PHPUnit

1. What makes a test flaky?

A flaky test is a test whose result is not deterministic, it occasionally fails on an identical codebase and identical environment, without any change to production code having taken place. The word "occasionally" is the key part: a test that always fails is a bug that is easy to fix. A test that fails in 5% of runs is a creeping problem that erodes trust in the entire test suite.

The most common causes of flaky tests in PHP projects are: time dependencies (tests that use date() or time() without a controllable clock), shared database state between tests (one test leaves behind data that affects the next test), randomness sources such as random_int() or array_rand() without a fixed seed, dependencies on external HTTP APIs, and race conditions in parallel test runs. Almost every one of these causes has a clear resolution pattern.

The organizational damage caused by flaky tests is bigger than the direct technical one. Teams that regularly dismiss failing tests with "that's just flaky" and simply restart the CI job get used to ignoring red CI output. That is the moment real regressions slip through unnoticed. Flaky tests must therefore be treated consistently as bugs and fixed with priority, never tolerated.

2. Detecting and documenting flaky tests systematically

The first step toward eliminating flaky tests is reliably identifying them. A single failed CI run can have many causes, infrastructure problems, network outages, exhausted file descriptors. Only once the same test fails across multiple independent runs on an identical commit is flakiness the most likely cause. Tools such as PHPUnit's re-run feature (--repeat) or external tracking systems help spot patterns.

For systematic identification, a dedicated CI job that runs the test suite multiple times in a row in the same environment and compares results is recommended. A test that fails at least once and passes at least once across 10 consecutive runs is definitely flaky. With --order=random and a fixed seed (--random-order-seed=12345), PHPUnit offers a way to systematically uncover order dependencies.

<?php
// Flaky test detection: run suite multiple times and compare
// vendor/bin/phpunit --repeat=10 --log-junit results.xml

// Detect order-dependent flakiness:
// vendor/bin/phpunit --order=random --random-order-seed=42
// vendor/bin/phpunit --order=random --random-order-seed=99

// PHPUnit 10+ allows marking known flaky tests explicitly
// while they await repair (NOT a long-term solution):
use PHPUnit\Framework\Attributes\Test;
use PHPUnit\Framework\TestCase;

final class ExampleFlakyTest extends TestCase
{
    // Temporary: document known flaky test so team is aware
    // Use #[Group('flaky')] to exclude from main CI run
    #[Test]
    #[\PHPUnit\Framework\Attributes\Group('flaky')]
    public function searchResultsAreOrderedCorrectly(): void
    {
        // This test is flaky due to non-deterministic sort order
        // from database. Tracked in issue #1234.
        // TODO: add ORDER BY clause or use collection sort assertion.
        $this->markTestSkipped('Known flaky: issue #1234');
    }
}

// CI: run normal suite without flaky group
// vendor/bin/phpunit --exclude-group flaky

3. Encapsulating time dependencies: clock interfaces and fake clocks

Time-dependent tests are the most common cause of flaky tests in business logic. A test that checks whether a coupon is valid "today" may fail shortly before or after midnight, because the test runs at 23:59 and the assertion is evaluated at 00:01. Tests that check expiration dates, booking timestamps or caching timeouts are especially prone to this.

The solution is a clock interface that encapsulates access to the current time. In production code, a SystemClock implementation returns the real system time. In tests, a FrozenClock is used, which always returns the same configured timestamp, fully deterministic, regardless of the actual execution time. The interface is injected into every class that contains time-dependent logic via dependency injection.

<?php
declare(strict_types=1);

namespace Mironsoft\Common;

use DateTimeImmutable;

/** Clock interface for deterministic time in tests. */
interface ClockInterface
{
    public function now(): DateTimeImmutable;
}

/** Production implementation: returns real system time. */
final class SystemClock implements ClockInterface
{
    public function now(): DateTimeImmutable
    {
        return new DateTimeImmutable();
    }
}

/** Test implementation: always returns a fixed, configured point in time. */
final class FrozenClock implements ClockInterface
{
    public function __construct(private readonly DateTimeImmutable $frozenAt) {}

    public static function at(string $dateTime): self
    {
        return new self(new DateTimeImmutable($dateTime));
    }

    public function now(): DateTimeImmutable
    {
        return $this->frozenAt;
    }
}

// In production DI container:
// $clock = new SystemClock();

// In test, deterministic, no flakiness:
// $clock = FrozenClock::at('2026-05-10 12:00:00');
// $service = new CouponValidator($clock, $repository);
// $this->assertTrue($service->isValid($coupon));  // always same result

4. Isolating database state between tests

Shared database state is the second most common cause of flaky tests in integration tests. Test A creates a record, test B assumes the table is empty, if tests run in the wrong order or in parallel, test B fails. The problem is often invisible because tests happen to run correctly in default order and only break once parallelization or a reordering occurs.

Three strategies for database isolation: first, database transactions that are rolled back after every test (fast, but problematic for tests that use transactions themselves). Second, database truncation after every test (reliable, but slower). Third, a fresh in-memory database (SQLite) for every test (very fast, but potentially incompatible with production-specific SQL). Magento integration tests use strategy one automatically.

5. Controlling randomness sources and making them reproducible

PHP functions such as random_int(), array_rand(), shuffle() and uniqid() produce non-deterministic values. Tests that rely on the output of these functions without controlling the seed are structurally flaky. The correct solution is the same as with time dependencies: an abstraction that can be replaced by a controllable implementation in tests.

A RandomizerInterface with methods such as int(int $min, int $max): int and string(int $length): string can be replaced in tests by a predictable sequence. Alternatively, PHP 8.2 allows the Random\Randomizer class with a configurable engine, so you can use new Randomizer(new FixedSizeByteSource('...')) in tests. For simple cases like ID generation, a fake implementation that returns a counter is often enough.

<?php
declare(strict_types=1);

namespace Mironsoft\Common;

/** Randomizer interface for deterministic tests. */
interface RandomizerInterface
{
    /** @throws \ValueError if min > max */
    public function int(int $min, int $max): int;

    /** Returns a cryptographically random hex string of given byte length. */
    public function hexString(int $bytes): string;
}

/** Production implementation using PHP 8.2 Random\Randomizer. */
final class SecureRandomizer implements RandomizerInterface
{
    private readonly \Random\Randomizer $randomizer;

    public function __construct()
    {
        $this->randomizer = new \Random\Randomizer();
    }

    public function int(int $min, int $max): int
    {
        return $this->randomizer->getInt($min, $max);
    }

    public function hexString(int $bytes): string
    {
        return bin2hex($this->randomizer->getBytes($bytes));
    }
}

/** Test implementation: predictable sequence, no randomness. */
final class SequentialRandomizer implements RandomizerInterface
{
    private int $counter = 0;

    public function int(int $min, int $max): int
    {
        return $min + ($this->counter++ % ($max - $min + 1));
    }

    public function hexString(int $bytes): string
    {
        return str_pad((string)$this->counter++, $bytes * 2, '0', STR_PAD_LEFT);
    }
}

6. External dependencies: HTTP, queues and the filesystem

Tests that make real HTTP requests to external APIs are inherently flaky, network latency, API rate limiting, server outages and DNS timeouts are all outside the test suite's control. The solution for unit tests is to fully mock the HTTP client. For integration tests there are HTTP recorder libraries such as php-vcr/php-vcr, which record real HTTP requests on the first run and replay them from the recording on subsequent runs, deterministic and without network access.

Filesystem tests are frequently flaky due to missing cleanup logic: one test creates a file, another test assumes it does not exist. The solution: setUp creates a fresh temporary directory with sys_get_temp_dir(), tearDown deletes it recursively. Never use fixed paths in tests. Queue tests require a testable in-memory queue implementation that does not need a real message broker connection.

7. Test order and hidden state dependencies

Static variables and singletons are a frequent source of test order dependencies. One test modifies static state (registry, cache, singleton), the next test assumes this state has been reset. This works as long as tests run in the expected order, under parallelization or --order=random the tests fall apart.

The solution: in tearDown, reset any static state that was set in setUp or in the test itself. For Magento tests, this particularly concerns the ObjectManager bootstrap and config singletons. PHPUnit offers a built-in way to uncover order dependencies with --order=random. Anyone using a fixed seed can produce reproducible orderings and link a discovered order to an issue tracker.

8. Flaky test root causes at a glance

The following table shows the most common causes of flaky tests in PHP projects, their symptoms and the recommended fix. Most causes have a clear, proven countermeasure, the actual problem is the lack of awareness of the root cause.

The difficulty of fixing the issue depends less on technical complexity than on how deeply the problem has spread through the codebase. A single time() call in one method is quickly replaced by a ClockInterface. But if twenty classes call time() directly, the migration takes more time, though the alternative (permanently flaky CI) is more expensive.

9. Summary

Flaky tests are not an unavoidable fate, but the symptom of uncontrolled dependencies: system time, database state, randomness sources, external APIs and static variables. Each of these causes has a clear resolution pattern, a ClockInterface for time, transaction rollback for database state, a randomizer abstraction for random values, mocking or VCR for external APIs, tearDown cleanup for static state.

The first step is consistent identification: --order=random and repeated runs uncover hidden order dependencies. Known flaky tests are marked with #[Group('flaky')] and excluded from the main CI run, but not ignored. Every flaky test gets an issue and a priority. Anyone who tolerates flaky tests pays with their team's trust in the entire test suite, and that trust is harder to win back than the time it costs to fix the underlying problem.

10. FAQ: Detecting and Eliminating Flaky Tests in PHPUnit