Mutation Testing: Do Your Tests Actually Test Anything?

1. How mutation testing works: mutants, kills, and escapes

Mutation testing flips the usual testing logic on its head: instead of checking whether your code produces the correct output for given inputs, it checks whether your tests even notice when the code breaks. A mutation testing tool like Infection takes your PHP source code, systematically generates small, syntactically valid variants of it, so-called mutants, and reruns your entire PHPUnit suite against each individual mutant. A mutant is created by a single targeted change, for example swapping >= for >, negating a return value, or removing an entire method call.

If at least one test fails, the mutant is considered killed, meaning your suite actually noticed the behavioral change. If the suite runs unchanged and green despite the code demonstrably behaving differently, the mutant escaped. Those escaped mutants are the real value of mutation testing: they point to the exact spot in the code where tests execute but make no genuine statement about behavior.

2. The mutation score as a stronger quality signal

Line coverage only answers whether a line of code was executed at all during the test run. The mutation score, reported by Infection as the Mutation Score Indicator, MSI, answers the far more relevant question: if this line behaved differently, would any test notice? The MSI is calculated approximately as the number of killed mutants divided by the total number of generated mutants, minus cases that cannot technically be evaluated, and is reported as a percentage.

In practice, a considerable gap often opens up between the two numbers. A project with 92 percent line coverage can easily have a mutation score of only 55 percent, because many tests run through the code but contain no assertions or only trivial ones. That gap is itself a metric: the bigger the difference between coverage and MSI, the more tests exist only on paper and verify essentially nothing they claim to check.

3. Why 100 percent line coverage still guarantees nothing

A test that calls a method without checking the result with an assertion counts toward line coverage exactly the same as a test that meticulously verifies every detail of the return value. Coverage tools measure only which lines the interpreter passed through, not whether the test then says anything meaningful about the behavior of those lines. This phenomenon is aptly called coverage without verification, and it turns up surprisingly often in grown PHP codebases, especially in tests that were primarily written to satisfy a CI coverage threshold.

Typical symptoms are tests that only call assertNotNull() on an object, even though the object is practically always an object, or tests that catch an exception with an empty catch block, so a real failure never turns the test red. Mutation testing reliably surfaces exactly these tests, because a mutant that changes actual business logic still passes green as soon as the assertion simply can't capture the change.

4. Setting up Infection for PHP and PHPUnit

You install Infection via composer require --dev infection/infection and initialize the configuration interactively with vendor/bin/infection --init, which creates an infection.json5 at the project root. The configuration defines which directories under source.directories count as mutation targets, typically app/code in a Magento 2 installation, and which test runner, usually phpunit, is used for the verification runs. Important thresholds are minMsi for the global mutation score and minCoveredMsi, which only applies to code actually reached by tests and therefore stays fair when parts of the codebase deliberately have no tests yet.

The first run with vendor/bin/infection generates a coverage report first, to know which mutants are even reachable by a test, before the actual mutation runs start. --threads=4 parallelizes mutation execution, which cuts runtime substantially on a multi-core CI machine. For Magento projects it's worth restricting source.directories to individual, well-tested modules, such as a dedicated Mironsoft_SeoSuite module, instead of mutating the entire app/code tree right away.

// infection.json5

{
  "$schema": "vendor/infection/infection/resources/schema.json",
  "source": {
    "directories": [
      "app/code/Mironsoft/SeoSuite/Model",
      "app/code/Mironsoft/SeoSuite/Service"
    ]
  },
  "logs": {
    "text": "var/infection/infection.log",
    "html": "var/infection/infection.html",
    "summary": "var/infection/summary.log"
  },
  "mutators": {
    "@default": true
  },
  "minMsi": 70,
  "minCoveredMsi": 80,
  "testFramework": "phpunit",
  "timeout": 10
}

5. Mutation operators in detail: what Infection actually changes

Infection applies dozens of predefined mutation operators, each producing a specific, minimal code change. The Conditional Boundary operator shifts comparison boundaries, turning >= into > and < into <=, which surfaces off-by-one errors that are especially critical in pricing or quantity logic. Arithmetic Operator Replacement swaps + for -, * for /, and vice versa, which becomes immediately visible in discount or tax calculations as soon as a test actually checks concrete numeric values.

The Return Value Negation operator flips boolean return values, turning return true; into return false;, and is especially revealing for methods like isValid() or canApplyDiscount(), whose entire purpose lies in a single truth value. Every operator gets applied individually at every matching location in the source code, so a 200-line service can easily generate forty or fifty mutants. Infection produces an isolated diff for every mutant, viewable in the HTML report right next to the affected test.

// Original code (app/code/Mironsoft/SeoSuite/Model/DiscountCalculator.php)

public function isEligibleForDiscount(float $orderTotal): bool
{
    return $orderTotal >= 100.0;
}

// Mutant generated by Infection's ConditionalBoundary operator:
//
// public function isEligibleForDiscount(float $orderTotal): bool
// {
//     return $orderTotal > 100.0;
// }
//
// A test that only asserts with $orderTotal = 150.0 cannot tell these
// two implementations apart, the mutant escapes.

6. Catching a weak test in practice

Take a simple discount class that grants a five percent discount from an order total of 100 euros. A naive test calls calculate() with an order total of 150 euros and only checks that the result is a number greater than zero. This test reaches 100 percent line coverage of the method, since every line executes, but says nothing about the actual discount logic. If Infection now mutates the condition from >= 100 to > 100, the test still passes, because 150 euros satisfies both variants of the condition equally, the mutant escapes.

A strengthened test instead checks concrete boundary values: an order total of exactly 100 euros, which must just barely trigger the discount, and 99.99 euros, which must just barely not trigger it, each with an exact assertion against the expected discount amount. Precisely this boundary test reliably kills the Conditional Boundary mutant, because the result at 100 euros genuinely differs between >= and >. This kind of boundary test almost never gets written in practice without the concrete nudge of an escaped mutant.

// app/code/Mironsoft/SeoSuite/Model/DiscountCalculator.php

final class DiscountCalculator
{
    public function calculate(float $orderTotal): float
    {
        if ($orderTotal >= 100.0) {
            return $orderTotal * 0.05;
        }

        return 0.0;
    }
}

// Test/Unit/DiscountCalculatorTest.php (weak, coverage-only test)

final class DiscountCalculatorTest extends TestCase
{
    public function testCalculateReturnsANumber(): void
    {
        $calculator = new DiscountCalculator();
        $result = $calculator->calculate(150.0);

        // Executes every line, but proves almost nothing about behavior
        $this->assertGreaterThan(0, $result);
    }
}

// Test/Unit/DiscountCalculatorTest.php (strengthened, boundary-aware test)

final class DiscountCalculatorTest extends TestCase
{
    public function testDiscountAppliesExactlyAtThreshold(): void
    {
        $calculator = new DiscountCalculator();

        // Boundary value: 100.0 must trigger the discount
        $this->assertEqualsWithDelta(5.0, $calculator->calculate(100.0), 0.001);
    }

    public function testDiscountDoesNotApplyBelowThreshold(): void
    {
        $calculator = new DiscountCalculator();

        // Boundary value: 99.99 must not trigger the discount
        $this->assertSame(0.0, $calculator->calculate(99.99));
    }
}

7. Interpreting Infection results correctly

After the test run, Infection classifies every mutant into one of several states. Killed means at least one test failed, the mutant was successfully caught. Escaped means the suite ran unchanged and green despite the changed behavior, this is the critical case. Timeout occurs when a mutant produces an infinite loop, for example through an inverted loop condition, and is conservatively treated like a killed mutant. Uncovered marks mutants in code that no test reaches at all, here the problem isn't a missing assertion, it's a missing test.

The MSI in the HTML or text report rolls all of that up into a single percentage, but it gives little practical guidance without looking at the individual escaped mutants. In practice it pays to sort escaped mutants by affected class and work through business-critical areas first, such as price calculation, discount logic, or stock checks. Some mutants are semantically equivalent to the original, for example a change inside dead code, and can be excluded from the evaluation via ignoreSourceCodeByRegex instead of artificially lowering the MSI target.

8. Performance costs and CI strategies for running it regularly

The biggest practical downside of mutation testing is runtime: a project with 2000 tests and 800 mutants can, even with test isolation via Infection's built-in coverage filter, which reruns only the tests actually affected per mutant, take anywhere from several minutes to hours. Mutating the entire codebase on every commit simply isn't practical for most teams and would grind any pipeline to a halt. The pragmatic compromise is to mutate only the files changed by the current merge request in the regular CI run, via --git-diff-filter=AM combined with --git-diff-base.

A full mutation run across the entire codebase instead belongs in a nightly scheduled job that doesn't block the pull request pipeline but posts results as a trend report to the team chat. The --min-msi=70 parameter lets the pipeline abort with an error code as soon as the mutation score drops below the defined threshold, turning verified test quality into a hard CI gate condition, similar to how minCoveredMsi already enforces it for covered code. It matters to raise the threshold gradually, a hard 90 percent target on day one just frustrates the team and leads to ignored pipelines.

# .gitlab-ci.yml

mutation-testing:
  stage: test
  image: php:8.4-cli
  script:
    - composer install --no-progress
    - vendor/bin/phpunit --coverage-xml=var/coverage-xml --log-junit=var/junit.xml
    - vendor/bin/infection
        --coverage=var
        --git-diff-filter=AM
        --git-diff-base=origin/main
        --min-msi=70
        --min-covered-msi=80
        --threads=4
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

mutation-testing-nightly:
  stage: test
  image: php:8.4-cli
  script:
    - composer install --no-progress
    - vendor/bin/phpunit --coverage-xml=var/coverage-xml
    - vendor/bin/infection --coverage=var --threads=8
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'

9. Line coverage versus mutation score side by side

The table below compares line coverage and mutation score across the dimensions that actually matter for day-to-day test work.

10. Summary

Mutation testing answers the question line coverage can never ask: do your tests actually notice when the code's behavior changes? Infection systematically generates mutants using operators like Conditional Boundary, Arithmetic Operator Replacement, and Return Value Negation, runs your PHPUnit suite against every single mutant, and classifies the result as killed, escaped, timeout, or uncovered. The resulting Mutation Score Indicator is a significantly stronger quality signal than pure line coverage, because it measures actual verification rather than mere execution.

The biggest pitfall isn't the technology, it's the runtime: a full mutation run across an entire codebase takes too long for every single commit. The practical solution combines fast runs limited to changed files in the merge request pipeline with complete nightly scans and a gradually raised min-msi threshold that only blocks the pipeline once test quality demonstrably drops below the defined level. That turns mutation testing into a fixed, but sustainable, part of your quality assurance, instead of a one-off metric with no consequences.

11. FAQ: Mutation Testing