Measuring and Enforcing Test Coverage Per Module Instead of Project Wide

1. The problem with a single global coverage number

A project wide coverage figure of seventy percent sounds like a solid test foundation. But it says nothing about how those seventy percent came together. In a typical project, test coverage is rarely distributed evenly: utility classes with trivial logic and plenty of generated or simple getter setter structures reach almost one hundred percent effortlessly, while the actual business logic, such as price calculation, discount rules, or checkout validation, stays stuck at twenty percent without the global number ever revealing it.

This hiding effect is especially dangerous because teams rely on the global figure as a success metric and act under the assumption they are sufficiently protected. A single bug in a poorly tested core module can then reach production despite an overall good coverage number, because the statistic masked the gap at exactly the point where it would have been most costly.

2. A concrete example of the hiding effect

Consider a project with three modules: a utility module with 2000 lines and 95 percent coverage, a reporting module with 1500 lines and 80 percent coverage, and a checkout module with 500 lines and only 20 percent coverage. The project wide, line weighted coverage then comes out to roughly 75 percent, a good looking figure at first glance. The small but business critical checkout module with its thin coverage barely registers in that calculation, even though this is exactly where a bug would cause the greatest financial damage.

If instead a separate threshold is defined per module, say ninety percent for utility, seventy percent for reporting, and seventy five percent for checkout, the checkout module's twenty percent immediately stands out as a clear violation and blocks the build, instead of disappearing into a favorable average. This mechanism is exactly what makes module based thresholds superior to a single global figure.

3. Generating coverage per directory with PHPUnit

PHPUnit itself usually generates coverage reports project wide, for example as a Clover XML file or an HTML report. To get values per module, it makes sense to either run PHPUnit separately per module directory with its own phpunit.xml or its own testsuite entry, or to evaluate the resulting Clover report by directory afterward. Both approaches yield more granular data than a single overall run, and running separately per module has the added benefit that modules can be tested independently and in parallel.

A Clover XML report contains, for every file, the number of covered and total statements, so a simple script can aggregate this data per directory. That script can then check, per module, whether the defined threshold was reached, and end the build with a non zero exit code if a module falls short.

<?php
// tools/coverage-per-module.php
declare(strict_types=1);

$thresholds = [
    'app/code/Vendor/Checkout' => 75.0,
    'app/code/Vendor/Pricing' => 85.0,
    'app/code/Vendor/Reporting' => 60.0,
];

$clover = simplexml_load_file(__DIR__ . '/../var/coverage/clover.xml');
$exitCode = 0;

foreach ($thresholds as $modulePath => $minCoverage) {
    [$covered, $total] = [0, 0];

    foreach ($clover->xpath("//file[contains(@name, '{$modulePath}')]") as $file) {
        $metrics = $file->metrics;
        $covered += (int) $metrics['coveredstatements'];
        $total += (int) $metrics['statements'];
    }

    $actual = $total > 0 ? ($covered / $total) * 100 : 0.0;

    if ($actual < $minCoverage) {
        fwrite(STDERR, sprintf(
            "ERROR: %s has only %.1f%% coverage (minimum %.1f%% required)\n",
            $modulePath, $actual, $minCoverage
        ));
        $exitCode = 1;
    } else {
        fwrite(STDOUT, sprintf("OK: %s has %.1f%% coverage\n", $modulePath, $actual));
    }
}

exit($exitCode);

4. How to set sensible thresholds per module

A blanket threshold for all modules, such as 'eighty percent everywhere', repeats the same mistake as the global figure, just shifted down to module level. It makes more sense to tie the threshold to a module's business risk: payment processing, price calculation, and inventory management justify high values of eighty to ninety percent, while an internal admin UI module made up mostly of declarative layout code can be adequately covered even at forty or fifty percent.

For new modules, it is advisable to set the threshold deliberately high, say eighty percent from the start, since no legacy burden exists here that would justify a lower figure. For existing modules with historically grown, thin coverage, a lower starting value is realistic, then gradually raised through a ratchet principle as new tests are added.

5. Enforcement in the CI pipeline

For module based thresholds to be effective, they need to be anchored as their own, blocking step in the CI pipeline, not just an informational report looked at after the merge. A typical setup runs PHPUnit with coverage collection enabled, generates a Clover report, and then runs a script like the one in the previous section against the defined thresholds. If a module check fails, the pipeline aborts with a clear, module specific error message instead of just printing a generic 'coverage too low' notice.

It matters to run this check for every merge request or pull request, not just periodically on the main branch. Only that prevents a developer from accidentally pushing a module below its threshold without it being caught immediately. A later periodic check only catches the problem after the poorly tested code has already landed on the main branch.

# .gitlab-ci.yml (excerpt)
coverage-per-module:
  stage: test
  script:
    - vendor/bin/phpunit --coverage-clover var/coverage/clover.xml
    - php tools/coverage-per-module.php
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

6. Limits of the module based view

Module based thresholds do not automatically solve the problem of shallow tests that execute lines but make no meaningful assertions. A module can formally reach ninety percent coverage and still contain gross bugs if tests merely call methods without checking return values or side effects. Coverage measures executed code, not verified behavior, and that distinction remains even under a granular, per module view.

It is therefore worthwhile to complement this with mutation testing for the most critical modules, which deliberately introduces small changes to the production code and checks whether at least one test fails as a result. A module with high line coverage but a low mutation score signals that the existing tests execute code but would not reliably catch bugs.

7. How to define module boundaries for coverage measurement

The usefulness of module based thresholds depends directly on how sensibly the module boundaries themselves are drawn. In a Magento project, the individual app/code/Vendor/ModuleName directories offer a natural boundary, since they already form the deployment and ownership unit. A module drawn too broadly, bundling several unrelated business areas together, dilutes the meaning of its threshold in exactly the same way a project wide figure does, just at a smaller scale.

For very large modules, it can make sense to define additional sub levels, for example separate thresholds for model, service, and controller layers within a module, when those layers carry markedly different risk profiles. This fine tuning should be used deliberately sparingly though, since too many individual thresholds make the configuration unwieldy and raise maintenance costs.

8. Creating transparent reports for the team

Beyond pure enforcement in the pipeline, a regularly updated dashboard is worthwhile, showing coverage values for all modules side by side, ideally color coded to indicate which modules sit above, near, or below their threshold. Such a dashboard makes visible which modules genuinely need attention, and prevents the conversation about test coverage from being reduced to a single, not very meaningful number.

A short automated comment on the merge request that lists the coverage change per affected module also helps, for example 'Checkout module: 42% to 47%, Pricing module: unchanged at 88%'. This immediate, contextual feedback influences developer behavior far more strongly than a monthly overall report, which is often only noticed with significant delay.

9. Line coverage versus branch coverage per module

Even a module based threshold can mislead if it is based purely on line coverage. A single line with a complex conditional, such as if ($status === 'vip' && $amount > 100 || $isPromoActive), already counts as covered as soon as it has been executed once in any combination, even if three out of four possible condition paths were never tested. For high risk modules like checkout or pricing, branch coverage is therefore a significantly more meaningful metric than plain line coverage.

Through its underlying Xdebug or PCOV driver integration, PHPUnit also supports path and branch metrics, which can be enabled deliberately in phpunit.xml for the most critical modules, while less risky modules continue to be checked with simple line coverage only, so as not to unnecessarily increase coverage collection time across the whole project.

<!-- phpunit.xml, excerpt for a module requiring branch coverage -->
<coverage>
    <report>
        <html outputDirectory="var/coverage/html"/>
        <clover outputFile="var/coverage/clover.xml"/>
    </report>
    <include>
        <directory suffix=".php">app/code/Vendor/Checkout/Model</directory>
    </include>
</coverage>
<!-- Run with path coverage for the checkout module -->
<!-- vendor/bin/phpunit --path-coverage --coverage-clover var/coverage/checkout-clover.xml -->

10. Summary

11. FAQ: Module Coverage: The Essentials at a Glance