Analyzing Historical Test Data: Trends Instead of Snapshots

1. Why a snapshot alone isn't enough

Every single test run by itself only delivers a binary result per test, from which it's impossible to tell whether that result is typical or unusual, nor whether the situation improved or worsened compared to the previous week. Without a comparison baseline over time, every single red test stays isolated and can't be put into perspective, which especially with sporadic, rare failures means genuine patterns only get recognized after a long time, or not at all.

A systematically built history turns these isolated individual results into a searchable, analyzable time series, from which questions can be answered like: has this test always failed occasionally, or did the problem only start three weeks ago with a specific commit? Is the total runtime growing evenly across all test cases, or is a single, recently added test file causing most of the increase?

2. Building a suitable data store for test results

The foundation of any historical analysis is structured, permanent storage of every single test run that goes beyond the ephemeral CI output, for which a simple relational table with one row per test case and run already works well, complemented with timestamp, result, runtime, branch, and commit hash as minimum fields.

For a Magento project, it makes sense to write this data directly after every CI run via a simple export step into a dedicated database, or for an easier start, into a series of structured JSON files in a dedicated storage bucket, instead of keeping the raw data exclusively in the artifacts that the CI platform automatically deletes after a short time anyway.

CREATE TABLE test_results (
    id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    test_name VARCHAR(255) NOT NULL,
    result ENUM('passed', 'failed', 'skipped') NOT NULL,
    duration_ms INT UNSIGNED NOT NULL,
    branch VARCHAR(100) NOT NULL,
    commit_hash CHAR(40) NOT NULL,
    executed_at DATETIME NOT NULL,
    INDEX idx_test_time (test_name, executed_at)
);

3. Analyzing test runtimes over time

With a built history, the development of total runtime over weeks and months can be represented as a time series, where besides the pure overall trend, the question of which individual test cases contribute most to the increase can also be answered specifically, by comparing average runtime per test over the same period.

This test-case-specific view frequently reveals that a general, moderate-looking aggregate runtime increase actually traces back to a small number of individual tests that have become disproportionately slow, say because a specific test has used unnecessarily long, fixed wait times instead of targeted wait conditions since a certain change, which can be fixed specifically once the culprit test case is concretely identified.

4. Identifying recurring flaky tests from history

A single, isolated failure can't be reliably distinguished from genuine, recurring flakiness, but a pattern aggregated over several weeks can: a test that failed five times over the last thirty runs, without the underlying code actually changing at those points, is a clear candidate for targeted stabilization, while a test with a single, isolated failure over the last thirty runs rather points to a one-off, external incident.

A practical analysis pattern is a weekly, automatically generated ranking of the ten tests with the highest failure rate over the past month, giving the team a clear, data-based prioritization instead of relying on subjective assessments, often shaped by the most recently annoying experience, when deciding which flaky test to stabilize next.

SELECT test_name,
       COUNT(*) AS total_runs,
       SUM(CASE WHEN result = 'failed' THEN 1 ELSE 0 END) AS failures,
       ROUND(100.0 * SUM(CASE WHEN result = 'failed' THEN 1 ELSE 0 END) / COUNT(*), 2) AS failure_rate
FROM test_results
WHERE executed_at >= NOW() - INTERVAL 30 DAY
GROUP BY test_name
HAVING failures > 0
ORDER BY failure_rate DESC
LIMIT 10;

5. Aggregation and a sensible retention period

Since the volume of raw data grows quickly with hundreds of tests and several daily CI runs, a tiered retention strategy is recommended: detailed individual results stay fully intact for a limited period, say ninety days, while older data gets condensed into weekly or monthly aggregates (success rate, average runtime, failure count), and the granular individual entries get deleted afterward.

This tiering considerably reduces storage needs without losing the information actually needed for long-term trend analysis, since a question like "has runtime grown over the last six months" only requires monthly averages anyway, not every single, granular data point from the past hundred fifty days.

6. Regular team reviews based on history

Raw data collection alone creates no value as long as nobody actually looks at the processed historical data regularly and derives decisions from it, which is why a short, say biweekly, team review session works well, in which the current ranking of flaky tests, the runtime trend, and notable changes since the last review get discussed together.

Such a session should be deliberately kept short and outcome-oriented, say fifteen minutes with a clear agenda, producing concrete, prioritized actions as its result, instead of drifting into an unstructured, general discussion about test quality whose insights then fizzle out without consequence, unassigned to any responsible person.

7. Statistical anomaly detection beyond plain failures

Not every relevant signal shows up as an actual test failure: a test that keeps passing reliably but whose runtime suddenly doubled within a handful of runs often points to a brewing backend problem, say a missing database index after a recent migration, long before that problem actually escalates into a visible failure or even a production incident.

A simple statistical anomaly detection combined with the historical data store, say based on the rolling average and standard deviation of a test's runtime over the last thirty runs, can automatically flag such outliers without a human having to manually review every single test's runtime chart. If a test's current runtime exceeds, say, three times the standard deviation from the historical mean, that can automatically generate a low-threshold, informational note instead of a full-blown, urgent alarm, giving the team an early heads-up without risking the alert fatigue described in the previous article through exaggerated urgency.

8. Visualization as a bridge between raw data and decisions

Raw tables of historical test data are complete, but unsuited for quick, intuitive interpretation, which is why a simple dashboard visualization is worthwhile, say a line chart for the runtime trend over the past six months and a bar chart for the ten tests with the highest current failure rate, combined into a single, glanceable overview.

Tools like Grafana, which can be connected directly to the same relational database, are well suited for this, since they enable automatically updated dashboards without repeated manual export effort, and can additionally be combined with the same observability tools the team already runs for other purposes, instead of operating a completely separate, isolated visualization solution exclusively for test data.

9. Building blocks of historical test analysis at a glance

The table below summarizes the building blocks presented.

10. Summary

11. FAQ: Historical Test Data: The Essentials at a Glance