Automatically Retrying Failed Tests: Flaky Test Retry Plugins in PHPUnit

1. What makes a test flaky

A flaky test is one that passes sometimes and fails other times with unchanged production code and unchanged test code. That fundamentally distinguishes it from a real failure, where an actual regression in the production code is the cause. Typical triggers for flakiness include time dependencies, for instance a test that relies on a fixed execution time under one second but takes longer under load, network calls to external services with occasional timeouts, or race conditions in parallel tests accessing the same resource.

The core problem with flaky tests is not just a single red pipeline, but the gradual erosion of trust within the team. Once developers learn that a failed build often just needs a rerun, they start reflexively ignoring real failures and simply restarting the pipeline. That is exactly the moment the test suite loses its actual function as a reliable signal, and a real bug can slip through unnoticed for months because nobody looks closely anymore.

2. How retry mechanisms work technically

A retry plugin for PHPUnit, typically via an attribute or annotation on the test method, ensures that a failed test is not immediately treated as definitively failed, but rerun within the same run up to a configured number of additional times. Only once the last attempt also fails does PHPUnit report the test as truly red. If any retry attempt succeeds, the test as a whole counts as passed, usually with a note in the output indicating that a retry occurred.

It matters to distinguish this from simply restarting the entire pipeline: retry plugins operate at the test level, not the pipeline level. That saves considerable time, because the whole suite does not need to rerun just to repeat a single unstable test, only the affected method itself runs again, often within a few milliseconds to seconds of extra runtime.

use PHPUnit\Framework\Attributes\Test;

final class ExternalWeatherApiTest extends \PHPUnit\Framework\TestCase
{
    // Example with a retry-capable test runner (e.g. dodevo/retry-annotation
    // or a custom PHPUnit extension hook).
    /**
     * @retry 3
     * @retryDelayMethod waitMilliseconds
     */
    public function testCurrentTemperatureIsFetchedSuccessfully(): void
    {
        $client = new ExternalWeatherApiClient();

        $temperature = $client->fetchCurrentTemperature('Hamburg');

        self::assertIsFloat($temperature);
    }

    protected function waitMilliseconds(int $attempt): void
    {
        usleep($attempt * 200_000);
    }
}

3. A lightweight in-house retry solution without extra dependencies

Not every project wants to introduce an additional dependency just for retry logic. A simple alternative can be built with a small helper method that repeats a code block until it either succeeds or the maximum number of attempts is reached. This approach is deliberately manual and explicitly visible in the test code, which has an important advantage: nobody can accidentally overlook that a test is being retried, because it is right there in the test body instead of hidden in an invisible annotation.

The downside of this manual approach is that it requires more code and needs to be adapted per test. For projects with only a handful of known unstable tests, that is an acceptable trade-off, because the visibility in the code is higher than with a globally acting retry annotation that can easily end up applied to too many tests without anyone noticing.

trait RetriesFlakyAssertions
{
    /**
     * Runs a code block up to $maxAttempts times until it completes
     * without exception. Use only for known-unstable, external
     * dependencies, never as a default approach for all tests.
     */
    protected function retryFlaky(callable $callback, int $maxAttempts = 3, int $delayMs = 200): void
    {
        $lastException = null;

        for ($attempt = 1; $attempt <= $maxAttempts; $attempt++) {
            try {
                $callback();
                return;
            } catch (\Throwable $exception) {
                $lastException = $exception;
                usleep($delayMs * 1000);
            }
        }

        throw $lastException;
    }
}

final class ExternalWeatherApiTest extends \PHPUnit\Framework\TestCase
{
    use RetriesFlakyAssertions;

    public function testCurrentTemperatureIsFetchedSuccessfully(): void
    {
        $client = new ExternalWeatherApiClient();

        $this->retryFlaky(function () use ($client): void {
            $temperature = $client->fetchCurrentTemperature('Hamburg');
            self::assertIsFloat($temperature);
        });
    }
}

4. Retry treats symptoms, it does not replace root cause analysis

The most important principle when using retry mechanisms is this: a retry makes an unstable test more convenient, not more stable. The actual root cause, for example an external service without proper timeout handling, a race condition in parallel tests, or a too-tight time window for an asynchronous operation, remains completely unchanged. Retry merely hides the problem from the CI dashboard, it does not solve it.

Retry becomes especially dangerous when it is applied uncritically to more and more tests, simply because it is the easiest short-term fix for a red pipeline. A team that follows this path consistently builds up a test suite over months that is formally green but actually hides many latent problems. When a real production bug then occurs, one that could originally have been caught by a flaky test, the retry logic is indirectly complicit, because it repeatedly drowned out the warning signal.

5. When retry is defensible as a time-boxed stopgap

Retry is not inherently wrong, it is a legitimate stopgap in certain, clearly bounded situations. One example is an integration test against an external third-party service whose occasional brief outages are outside the team's control, while a ticket for a more robust solution, such as a mock or a contract test, already exists. In this case, retry bridges the time until the actual fix, without unnecessarily blocking the pipeline.

What matters is that every use of retry carries a clear justification and an expiration date, for example a linked ticket in a code comment. Without this discipline, a time-boxed stopgap quickly turns into a permanent state, because nobody is actively working to remove the retry once the actual cause has been fixed.

final class ThirdPartyShippingRateApiTest extends \PHPUnit\Framework\TestCase
{
    /**
     * Retry as a time-boxed stopgap.
     * Cause: third-party API has sporadic 2-second timeouts.
     * Tracking: JIRA-4821, planned fix: contract test with mocked client.
     *
     * @retry 2
     */
    public function testShippingRateIsCalculatedForGermany(): void
    {
        $client = new ThirdPartyShippingRateApiClient();

        $rate = $client->calculateRate('DE', 2.5);

        self::assertGreaterThan(0.0, $rate);
    }
}

6. Typical causes and their actual fix

To use retry deliberately and sparingly, it helps to know the most common causes of flakiness and to pursue their proper, sustainable fix in parallel. Time-dependent tests can usually be stabilized by explicitly waiting for a defined state instead of a fixed number of milliseconds, for example with a polling mechanism that repeatedly checks whether a condition is met instead of blindly assuming a fixed wait time.

Race conditions in parallel-running tests often arise from shared resources, such as the same test database or the same file, and can be fixed with isolated fixtures per test or by disabling parallelization for the affected test class. Network-dependent tests against unstable third parties should be replaced in the medium term with contract tests using mocked responses, which verify the interface's contract terms without depending on its actual availability.

7. Common configuration pitfalls when using retry

A common mistake is not applying retry deliberately to individual, known-unstable tests, but enabling it globally for the entire suite, for example through a blanket configuration in the test runner. That may reduce the number of red pipelines in the short term, but it systematically hides which tests actually have a problem, since even a test failing due to a real bug can turn green through a random second attempt if the bug itself does not trigger deterministically.

A second common mistake is too short or missing a delay between retry attempts. If a test is rerun immediately without any delay, the original cause, such as a briefly overloaded external service, often still persists, and the second attempt fails as well. A sensible, slightly increasing delay between attempts raises the chance that a temporary disruption has genuinely resolved itself before the next attempt starts.

8. Making flakiness visible instead of letting retry hide it

To keep retry from becoming permanent concealment, every retry should be logged and evaluated. Many CI systems allow tracking the number of retry events per test over time, for example by parsing JUnit XML reports or through custom logging inside the retry logic itself. A test that repeatedly needs a retry is a clear signal that it should be prioritized for a fix, not tolerated indefinitely.

A useful dashboard shows, for each test, how often it needed a retry over the past weeks, and makes that number visible to the whole team, for example in a weekly review. That way, flakiness remains an actively tracked engineering concern instead of silent background noise that eventually gets accepted as the normal state of things.

9. Team rules for responsible use of retry

A team should clearly define under what conditions retry may be used at all: typically only for tests against external systems outside the team's control, never for unit tests against its own deterministic code. Every retry annotation should be required to carry a comment with justification and a ticket reference, so the measure stays traceable and does not get forgotten.

It is also worth doing a regular cleanup: every few months a team should go through all active retry annotations and check whether the original cause has since been fixed and the retry can be removed. The table below summarizes common causes of flakiness, their sustainable fix, and the role retry should play in each case.

10. Summary

11. FAQ: Retry Plugins for Flaky Tests: The Essentials at a Glance