Test Metrics That Actually Matter, Beyond Green or Red

1. The problem with a pure green-red view

A test suite that shows either fully green or fully red on every single run reduces a complex, multi-dimensional reality to a single bit of information, systematically hiding all the gradual but ultimately decisive developments, say a gradually rising flakiness rate or a quietly growing total runtime that only becomes genuinely painful after many months.

This problem is compounded by the fact that a single, currently green run says nothing about how stable that result actually is: a test that's green in nine out of ten runs and only occasionally red looks identical in the current run to one that's reliably always green, even though both deserve a fundamentally different level of trust. Without an aggregated view over time, this difference stays completely invisible to the team until the flaky test eventually fails at exactly the worst moment, say shortly before an important production release.

2. Flakiness rate: the most important early indicator

The flakiness rate measures the share of test runs where a test both succeeded and failed between runs without any code change in between, and can be practically captured by rerunning every test across several consecutive CI runs on the same commit and logging the result variance. A test with a flakiness rate of ten percent concretely means it delivers a different result than the previous run roughly every tenth time, without the underlying code actually having changed.

This metric deserves special attention because it directly undermines the team's trust in the entire test suite: once it's known that certain tests are sometimes green and sometimes red regardless of the actual code state, developers start reflexively rerunning failed runs instead of taking every red test seriously, gradually eroding the entire protective effect of the test suite. A sensible target is a flakiness rate under one percent across the whole suite, with individual tests consistently above five percent prioritized for stabilization or, if necessary, temporarily removed from the pipeline instead of leaving the whole team to live with them permanently.

# Example: run the same commit three times in a row in CI
# to detect flakiness independent of actual code changes
npx playwright test --repeat-each=3 --reporter=json > results.json

# Aggregate the flakiness rate per test from the JSON results
node scripts/flakiness-rate.js results.json

3. Mean-time-to-detect as a measure of response speed

Mean-time-to-detect measures the average time span between a bug entering the codebase and the moment a test first catches it, and is therefore a direct measure of how fast the test suite actually delivers feedback about a change's correctness. A bug only discovered two days after it was introduced by an extensive nightly test run typically causes considerably higher fix costs than the same bug caught within minutes of every single pull request.

This metric can be practically collected by retroactively determining, for every bug actually seen in production, how many commits or how much time passed between introducing the faulty code and the first failing test run that should theoretically have caught it. A systematically high mean-time-to-detect often indicates that important test cases run too rarely, say only nightly instead of on every pull request, or that certain code paths simply aren't sufficiently covered.

4. Test runtime trend instead of a snapshot

The absolute runtime of a single test run is by itself not very meaningful, but its development over time is: a test suite whose total runtime has continuously grown from eight to thirty-five minutes over six months signals a structural problem, even if none of the individual, in-between increases looked dramatic on its own.

A runtime chart reviewed regularly, say weekly, makes such gradual trends visible long before they become an acute, universally felt problem, and allows targeted countermeasures like parallelization, removing redundant tests, or a deliberate split into a fast smoke test set and a less frequently run, full regression suite, before the entire team's daily development speed noticeably suffers from the growing wait time.

5. Code coverage as a secondary metric, not the main goal

Code coverage measures the share of production code executed at least once during test runs, but explicitly says nothing about whether the assertions run along the way actually check the right, meaningful things, which is why optimizing in isolation for a specific coverage percentage easily leads to worthless tests that execute lines but perform no genuine behavioral checks.

More useful than a fixed, company-wide coverage target like "always above 80 percent" is a targeted look at which critical business logic paths, say price calculation at checkout or discount logic, are actually covered by meaningful assertions, instead of relying exclusively on the aggregated percentage, which weights trivial getter methods and business-critical calculations equally and can therefore produce a misleading overall picture.

6. Recognizing and avoiding vanity metrics

A vanity metric is a number that looks impressive and presents well on a management dashboard but delivers no actually actionable information, say the raw count of written test cases without any regard for their quality, flakiness level, or actual defect detection rate. A test suite with two thousand tests, a significant share of which redundantly checks the same trivial functionality multiple times, isn't automatically more valuable than a considerably smaller suite with targeted, well-thought-out test cases.

A practical distinguishing criterion is asking whether a metric would actually influence a concrete action decision if it got worse: a rising flakiness rate sensibly triggers a stabilization priority, while a raw test case count getting worse, meaning fewer tests, often triggers no sensible reaction at all, which is a strong sign that this number alone is a vanity metric rather than an actually decision-relevant one.

7. Metrics as a shared conversation, not an evaluation tool

Once test metrics become visible and get reviewed regularly for the first time, a real risk emerges that they get misread as a basis for evaluating individual developers' performance, say someone getting judged negatively for a flaky test personally attributed to them, even though the actual cause lies in shared test infrastructure or a hard-to-predict external dependency, not in any individual fault of the original test's author.

If metrics are instead consistently treated as a shared, team-owned topic of conversation, say within the regular reviews described later, a considerably more constructive dynamic emerges: a rising flakiness rate gets tackled as a shared technical problem deserving collective prioritization, instead of being perceived as an accusation against a single person, which in practice also raises the willingness to raise problems openly and early, instead of concealing them out of fear of negative judgment or masking them through reflexively rerunning failed runs.

8. Building a compact metrics dashboard

Instead of tracking dozens of individual metrics simultaneously, a compact dashboard with a deliberately small selection of the most important metrics is recommended, say flakiness rate, test runtime trend, and mean-time-to-detect for recent past production bugs, complemented by a short, qualitative assessment of coverage on critical business logic paths instead of a single, aggregated percentage.

For a Magento test suite, such a dashboard can be realized as a weekly, automatically generated report that merges data from Allure histories, CI runtime logs, and a manually maintained list of known production bugs, instead of running a separate, isolated tool for every single metric that nobody on the team consults regularly.

9. Test metrics at a glance

The table below compares the metrics presented.

10. Summary

11. FAQ: Test Metrics: The Essentials at a Glance