Debugging Broken PHPUnit Test Suites Systematically

1. Recognizing symptoms: green locally, broken in CI

The classic symptom of a broken test suite is the discrepancy between the local result and the CI result. Locally, all 400 tests pass; in the CI pipeline, 15 fail, and when you rerun the failed tests individually, they pass again. This pattern is an almost certain indicator of test isolation problems, not real bugs in the production code. Understanding this distinction is the first step toward systematic debugging.

A second symptom: tests fail, but the error has nothing to do with the behavior being tested. An assertion about an order total fails because a different test earlier changed the currency configuration. A database query returns wrong results because a previous test did not reset test data. The symptom shows up in one test, but the cause lies in the triggering test, often several test classes earlier in the execution order.

A third pattern: tests fail only on the first run of the suite, but not on the second run. This points to temporary files, cached configuration, or singleton state that was not reset and still contains the correct state from the first run on the second pass. All three patterns share one thing: the test output alone is not enough, the execution order has to be known.

2. Isolation as the first diagnostic tool

The first step in diagnosing a broken test suite is narrowing it down through isolation. PHPUnit offers several mechanisms for running tests under different configurations to narrow down the source of the failure. The PHPUnit flag --process-isolation runs every test in a separate PHP process, which fully isolates global variables, static properties, and singleton state. If all tests pass with --process-isolation but fail without this flag, the cause is clear: shared state between tests.

The next step is narrowing it down to the test classes causing the problem. --filter TestClassName runs individual test classes. Using the @depends mechanism or analyzing the setup methods often reveals which test class leaves behind the state that affects other tests. PHPUnit 11 offers the attribute #[RunTestsInSeparateProcesses] as a declarative way to always run certain test classes in isolation, a useful tool once the source of the shared state is known to be in a specific class.

<?php
// Step 1: Run single test class to confirm it passes in isolation
// vendor/bin/phpunit tests/Unit/Order/OrderServiceTest.php
// Result: PASS

// Step 2: Run full suite, does it fail?
// vendor/bin/phpunit
// Result: FAIL (OrderServiceTest::testCalculateTotal)

// Step 3: Run with process isolation, does it pass now?
// vendor/bin/phpunit --process-isolation
// Result: PASS → confirms shared state is the cause

// Step 4: Use --order=reverse to find if order matters
// vendor/bin/phpunit --order=reverse
// Result: different tests fail → order-dependent test confirmed

// Step 5: PHPUnit attribute to always isolate a specific class
use PHPUnit\Framework\Attributes\RunTestsInSeparateProcesses;

#[RunTestsInSeparateProcesses]
final class LegacySingletonTest extends \PHPUnit\Framework\TestCase
{
    // This class modifies a global registry, must run in isolation
    public function testSomethingWithGlobalRegistry(): void
    {
        // Safe: each test in this class runs in its own process
        \GlobalRegistry::set('key', 'value');
        $this->assertSame('value', \GlobalRegistry::get('key'));
    }
}

3. Uncovering order dependencies

By default, PHPUnit runs tests in the order in which they are defined in the test suite, but this order is not guaranteed and can change due to framework updates, new tests, or changed filesystem ordering. The flag --order=random runs tests in a random order and makes order-dependent tests fail reproducibly. With --random-order-seed=XXXX a specific order can be reproduced to analyze the problem.

Systematically narrowing down the triggering test class is done through a bisection approach: run the first half of the tests before the failing test and check whether the error occurs. Then keep halving that subset further until the triggering test class is found. For large suites this is tedious but unavoidable. Tools like phpunit-randomize-order or custom test runner scripts can automate this process.

4. Global state as the most common cause

In PHP projects there are several sources of global state that corrupt test isolation. Static properties (static $instance in singleton implementations), superglobal variables ($_SERVER, $_ENV, $_SESSION), global configuration registries, and PHP ini settings changed at runtime are the most common candidates. In Magento projects, the ObjectManager, the config cache, and the event manager add further potential sources of shared state.

The solution for singleton problems in tests is either resetting the singleton in tearDown() or replacing it with dependency injection that can be swapped for stubs in tests. For superglobal variables, PHPUnit offers the attribute #[BackupGlobals(true)], which backs up all global variables before the test and restores them afterward. This has a performance cost, but it is a pragmatic solution for legacy code that cannot be refactored right away.

<?php
declare(strict_types=1);

use PHPUnit\Framework\Attributes\Test;
use PHPUnit\Framework\Attributes\BackupGlobals;
use PHPUnit\Framework\Attributes\BackupStaticProperties;
use PHPUnit\Framework\TestCase;

// Problematic singleton that leaks state between tests
class ConfigRegistry
{
    private static array $config = [];
    private static ?self $instance = null;

    public static function getInstance(): self
    {
        return self::$instance ??= new self();
    }

    public static function reset(): void
    {
        self::$instance = null;
        self::$config   = [];
    }

    public function set(string $key, mixed $value): void
    {
        self::$config[$key] = $value;
    }

    public function get(string $key): mixed
    {
        return self::$config[$key] ?? null;
    }
}

// Correct pattern: reset static state in tearDown
final class ConfigRegistryTest extends TestCase
{
    protected function tearDown(): void
    {
        // Mandatory: reset singleton after each test
        ConfigRegistry::reset();
    }

    #[Test]
    public function storesAndRetrievesValue(): void
    {
        ConfigRegistry::getInstance()->set('currency', 'EUR');
        $this->assertSame('EUR', ConfigRegistry::getInstance()->get('currency'));
        // tearDown resets the singleton → next test starts clean
    }
}

// Alternative: PHPUnit attribute for automatic static property backup
#[BackupStaticProperties(enabled: true)]
final class AutomaticBackupTest extends TestCase
{
    #[Test]
    public function modifiesStaticPropertySafely(): void
    {
        ConfigRegistry::getInstance()->set('locale', 'de_DE');
        // PHPUnit restores all static properties after this test
    }
}

5. Keeping database state clean between tests

Integration tests that use a database are especially prone to producing order dependencies. One test writes test data into a table, another test reads from the same table and finds unexpected records. The standard pattern is using database transactions: at the start of each test you begin a transaction, at the end it gets rolled back. This way each test starts with a clean database state without truncating tables, which is significantly faster.

In Magento integration tests, the annotation @magentoDbIsolation enabled does exactly that. In plain PHPUnit projects, you can implement the same pattern with an abstract base test class that begins a transaction in setUp() and rolls it back in tearDown(). Important: this technique does not work with tests that use transactions themselves, or with database operations that run outside the transaction (for example DDL statements in MySQL, which commit implicitly).

6. Misconfigured mocks and stubs

Mocks that are not reset correctly are another common cause of order dependencies. In PHPUnit, mocks created with createMock() or getMockBuilder() are automatically reset after each test, that is the default behavior. Problems arise when mocks are stored in class properties and reused across multiple tests without being reset. A mock whose expects($this->once()) expectation was triggered in one test fails in the next test if it gets called again.

The solution is consistently creating fresh mock objects in setUp() instead of during class initialization. In test classes with many dependencies it is tempting to create mocks once and then adjust them, which leads to hidden coupling between tests. Every setUp() method should define the complete starting state of all dependencies, regardless of what previous tests did with those dependencies.

7. Differences between local environment and CI

When tests only fail in CI and never locally, the first step is aligning the environments. PHP version differences, different PHP extensions (GD, intl, mbstring), different system locales, timezones, and filesystem paths are common candidates. The timezone is especially treacherous: tests that perform date/time calculations and were developed on a machine with Europe/Berlin can fail on a CI server with UTC without the code being logically wrong.

A systematic approach: use Docker containers for local development that have exactly the same configuration as the CI container. In phpunit.xml, explicitly set the PHP ini settings (timezone=Europe/Berlin, memory_limit=256M) so tests do not depend on the system-wide configuration. Environment variables that are set locally in .env but are not present in CI are another frequent cause, phpunit.xml should define all required test environment variables with default values.

8. Debugging tools compared

PHPUnit and the PHP ecosystem offer various tools for diagnosing broken test suites. The right choice depends on the symptom.

9. Preventive measures for stable test suites

The most effective measure against broken test suites is preventive isolation: every test must build its own state completely in setUp() and clean it up in tearDown(). No test method may depend on the state left by another test method, nor on the framework's execution order. That sounds obvious, but in practice it is frequently violated under fast development and time pressure. Code reviews that explicitly check for missing tearDown() methods and for shared member variables between tests reduce the problem structurally.

In phpunit.xml, you can enable beStrictAboutTestsThatDoNotTestAnything="true" and beStrictAboutChangesToGlobalState="true". The latter setting makes tests fail if they change global variables or static properties without resetting them, a strong preventive signal that surfaces broken tests early. Combined with --order=random in the CI pipeline, you get an environment that reliably uncovers order dependencies before they cause problems in production.

<?xml version="1.0" encoding="UTF-8"?>
<!-- phpunit.xml, strict configuration for stable test suites -->
<phpunit
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="vendor/phpunit/phpunit/phpunit.xsd"
    beStrictAboutChangesToGlobalState="true"
    beStrictAboutTestsThatDoNotTestAnything="true"
    failOnRisky="true"
    failOnWarning="true"
    executionOrder="random"
    resolveDependencies="true"
>
  <php>
    <!-- Explicit timezone: never rely on system timezone -->
    <ini name="date.timezone" value="Europe/Berlin"/>
    <ini name="memory_limit" value="256M"/>
    <!-- Test environment variables with safe defaults -->
    <env name="APP_ENV" value="testing"/>
    <env name="DATABASE_URL" value="sqlite::memory:"/>
  </php>
  <testsuites>
    <testsuite name="unit">
      <directory>tests/Unit</directory>
    </testsuite>
    <testsuite name="integration">
      <directory>tests/Integration</directory>
    </testsuite>
  </testsuites>
</phpunit>

10. Summary

Broken test suites almost always arise from global state, missing isolation, or order dependencies, rarely from real bugs in the production code. The systematic diagnostic process starts with --process-isolation to confirm shared state, and --order=random to make order dependencies visible. Singleton resets in tearDown(), database transactions per test, and consistent mock creation in setUp() are the three most important preventive measures.

PHPUnit configuration with strict settings and random execution order in the CI pipeline surfaces isolation problems early. Anyone who also aligns the environment through Docker containers and explicitly defines all test environment variables in phpunit.xml permanently eliminates most "green locally, red in CI" phenomena.

11. FAQ: Debugging Broken Test Suites