Mutation Testing with Infection in Symfony: Measuring Test Quality Beyond Coverage

1. Why code coverage is the wrong metric

Code coverage measures which lines of code were executed during a test run, not whether the tests actually verify that these lines do the right thing. A line can run without a single assertion validating its result and still count as "covered". This exact gap is closed by mutation testing: instead of asking whether code ran, it asks whether a bug in that code would be noticed by the test suite.

The tool Infection is the established PHP implementation of mutation testing and works for Symfony projects without major adjustments to the existing PHPUnit configuration. Infection systematically changes production code in small steps, for example turning > into >= or negating a condition, runs the test suite against each of these changes, and logs which changes stay undetected.

This article shows how mutation testing works in detail, how Infection gets integrated into a Symfony project, how to interpret the mutation score indicator, and how a baseline strategy helps put the tool to productive use even in grown codebases.

2. How mutation testing works

The core mechanism of mutation testing is easy to describe but far reaching in effect: Infection creates a so called mutant for every eligible line of code, a minimally altered copy of the original code. A typical mutant changes a comparison operator, removes a method from a chain, or replaces a return value with another plausible value. The full test suite then runs for every mutant, but only against this one altered piece of code.

If at least one test catches the mutant by failing, the mutant is considered killed. If the entire test suite still passes despite the code change, the mutant is considered escaped. A high share of escaped mutants in a given class precisely shows where the test suite has gaps that plain coverage numbers would never reveal, because the line was technically executed, just without a real check.

3. Setting up Infection in a Symfony project

Infection gets installed as a Composer dev dependency and needs a configuration file infection.json5 in the project root, which among other things defines the path to the PHPUnit configuration and the minimum mutation score indicator target. For a typical Symfony project, a lean configuration referencing the existing phpunit.xml.dist and restricting the analysis scope to the src/ directory, without including generated code or Symfony core classes, is enough.

// infection.json5
{
    "$schema": "vendor/infection/infection/resources/schema.json",
    "source": {
        "directories": ["src"],
        "excludes": ["src/Kernel.php", "src/DataFixtures"]
    },
    "logs": {
        "text": "var/infection/infection.log",
        "summary": "var/infection/summary.log",
        "html": "var/infection/infection.html"
    },
    "mutators": {
        "@default": true
    },
    "minMsi": 70,
    "minCoveredMsi": 80
}

#!/usr/bin/env bash
# Run mutation testing with existing PHPUnit coverage cache for speed.
set -euo pipefail

vendor/bin/phpunit --coverage-xml=var/coverage/xml --log-junit=var/coverage/junit.xml
vendor/bin/infection --coverage=var/coverage --threads=4 --min-msi=70

4. Interpreting the mutation score indicator

The mutation score indicator, short MSI, is the percentage of generated mutants actually killed by the test suite. An MSI of 100 percent means every artificially injected code change triggered at least one test failure, an MSI of 60 percent means nearly half of all tested code changes would have gone unnoticed. Infection additionally distinguishes between the global MSI and the "covered MSI", which only considers mutants in lines already reached by coverage.

A low MSI in a particular class is not a reason to panic, but a concrete pointer to exactly where the test suite is blind. Unlike a general coverage percentage, the mutation score indicator shows, per class and even per method, which tests are missing, since the Infection HTML report precisely lists which mutant survived and at which line.

5. Understanding and selecting mutators deliberately

Infection ships with dozens of mutators, grouped into categories such as Arithmetic, Boolean, ConditionalBoundary, and ReturnValue. The ConditionalBoundary mutator, for example, turns $stock > 0 into $stock >= 0, a classic off by one bug that many test suites overlook in practice because they only test the obvious case, not the exact boundary.

For Symfony projects with many Doctrine entities and DTOs, a deliberate selection of mutators is worth it instead of the full default set, since some mutators, for instance ones affecting pure getter methods, rarely deliver real business value and mostly generate noise. A configuration built on @default minus a few mutators identified as irrelevant often delivers a more meaningful signal than the full mutator list.

// infection.json5 — targeting specific mutator categories for pricing logic
{
    "mutators": {
        "@arithmetic": true,
        "@conditional_boundary": true,
        "@boolean": true,
        "PublicVisibility": false, // low signal for a DTO-heavy codebase
        "MethodCallRemoval": true
    }
}

6. Analyzing escaped mutants and sharpening tests

The most valuable step after an Infection run is going through the HTML report for escaped mutants and deciding, one by one, whether a missing test needs to be added. Not every surviving mutant automatically justifies a new test: some mutants actually affect genuinely irrelevant code, such as logging calls with no effect on the result, where an extra test would add little value.

For business relevant mutants, for example an escaped mutant in a price calculation or a discount rule, the surviving mutant is almost always a direct pointer to a missing assertion. Often a single additional row in an existing data provider, such as an edge case with exactly zero or exactly the maximum value, is enough to kill the corresponding mutant on the next run.

7. Baseline strategy for grown codebases

An existing Symfony project with several years of history rarely has an MSI of 90 percent the first time mutation testing gets introduced. A hard --min-msi=90 in the CI pipeline would immediately turn every build red and quickly make the tool unpopular. The pragmatic path is a baseline: the current MSI gets recorded as the starting value, and the CI pipeline initially only requires that the MSI does not drop below this value, instead of enforcing a utopian target right away.

With this baseline strategy, the threshold can be raised step by step as new or improved tests push the MSI up in relevant modules. Infection also supports a mode that only analyzes mutants in files changed within a pull request, which for large legacy codebases in particular focuses attention on new or changed code, instead of re-evaluating the entire historical codebase on every CI run.

#!/usr/bin/env bash
# Mutation testing scoped to files changed in the current merge request.
set -euo pipefail

git diff origin/main --name-only --diff-filter=ACMR -- 'src/*.php' > var/infection/changed-files.txt

vendor/bin/infection \
  --filter=$(paste -sd, var/infection/changed-files.txt) \
  --min-msi=70 \
  --threads=4

8. Common mistakes when using mutation testing

The most common mistake is forcing a threshold that is too high into the CI pipeline right from the start, without planning for a baseline phase. That leads either to permanently red builds that get ignored, or to rushed, superficial tests that only kill the mutant without carrying real business value, for example a test that merely checks that a method got called without validating the result.

<?php

// WRONG: test written only to "kill the mutant", checks nothing meaningful
public function testCalculateDiscount(): void
{
    $result = $this->calculator->applyDiscount(100.0, 10);
    self::assertNotNull($result); // kills some mutants, proves nothing
}

// RIGHT: test asserts the actual expected business value
public function testCalculateDiscount(): void
{
    $result = $this->calculator->applyDiscount(100.0, 10);
    self::assertEqualsWithDelta(90.0, $result, 0.001);
}

A second mistake is running Infection against the entire codebase on every commit. On larger projects, a full mutation testing run can take several hours, since the test suite gets executed again for every single mutant. The diff based strategy shown in the previous section keeps CI pipeline runtime practical, while a full nightly run keeps the overall picture in view.

9. Mutation testing compared to coverage metrics

Code coverage and mutation testing answer different questions and complement each other instead of replacing one another. The following table compares both metrics.

The pragmatic combination: code coverage as a fast, cheap baseline metric on every CI run, mutation testing as a deeper but more expensive check at the pull request diff level and in regular full runs outside the critical path.

10. Summary

Mutation testing with Infection answers a question that code coverage cannot structurally answer: would the test suite actually notice a real bug? By injecting artificial faults into Symfony code and checking whether at least one test fails, the mutation score indicator precisely reveals where assertions are missing, even in lines that classic coverage measurement has long counted as "tested".

For grown Symfony projects, a baseline strategy is the practical entry point: record the current MSI, raise it step by step, and combine diff based runs at the pull request level with full nightly runs. Anyone introducing mutation testing this way gets a precise tool for finding test gaps, without slowing the CI pipeline down to the point of impracticality.

11. FAQ: Mutation testing with Infection in Symfony