Soak Tests: Verifying Long-Term Stability Under Sustained Load

1. Why short load tests systematically miss certain problems

A classic load test with k6 or similar tools typically runs a few minutes up to an hour at most and reliably answers whether a system handles a given number of concurrent users briefly. Certain classes of failure, however, only emerge from cumulative effects over long periods and stay fundamentally invisible in a short test run, even at identical load intensity.

A memory leak where each individual request leaves behind only a few kilobytes of not-properly-freed memory doesn't matter across a thousand requests over five minutes, but adds up across millions of requests in a single day to a memory footprint that eventually genuinely exceeds the available memory of a PHP-FPM worker process, leading to a hard crash or an operating-system-forced restart of the process.

Soak tests close this detection gap by deliberately sustaining moderate, realistic load for hours or even days, letting cumulative effects that stay below the noise floor in short test runs add up over time into a clearly measurable, unambiguously recognizable trend.

2. Systematically hunting down memory leaks

The classic, and economically costliest, use case for soak tests is hunting down memory leaks, where a process continuously claims more memory over time without fully releasing it once requests complete. In PHP applications, such leaks often come from static class variables unintentionally accumulating data across multiple requests, from file handles not properly closed, or from event listeners re-registered on every request but never removed.

During a soak test, the memory consumption of the relevant processes gets recorded continuously over the entire test run, typically via a monitoring tool like Prometheus with a Grafana dashboard. A process with healthy memory behavior shows, after an initial warm-up phase, a largely constant, sawtooth-like pattern from regular garbage collection, while a process with a real leak shows a steadily, usually nearly linearly rising trend that, without intervention, inevitably ends sooner or later in an out-of-memory error.

# Log PHP-FPM worker memory consumption at regular intervals
# during a soak test
while true; do
  echo "$(date +%s) $(ps -eo rss,cmd | grep 'php-fpm: pool' | \
    awk '{sum+=$1} END {print sum}')" >> soak-memory.log
  sleep 60
done

# then check the trend visually, say with gnuplot,
# or import the raw data into Prometheus/Grafana

3. Detecting connection pool exhaustion

Another typical soak test finding is the gradual exhaustion of database or Redis connection pools, where individual connections don't get correctly returned to the pool once a request completes, say because an unhandled exception bypasses the regular cleanup code. At low load, a single orphaned connection is practically unnoticeable, since the pool has plenty of free connections, but over hours these orphaned connections add up until the pool is fully exhausted and new requests fail with a connection error, even though the database itself works completely normally.

This pattern is especially insidious because the error that eventually surfaces in production usually doesn't point to the actual root cause: a "connection pool exhausted" error after twelve hours of stable operation looks at first glance like a sudden, inexplicable incident, but is actually the predictable end result of a resource violation that accumulated gradually over hours, something a soak test deliberately and reproducibly surfaces in advance.

4. Monitoring log growth and disk consumption

Besides memory and connections, uncontrolled log growth is a third, often underestimated soak test finding: an application writing an extra log line meant for debugging on every request creates no noticeable problem at low load, but under sustained production load over a full day it can produce several gigabytes of log data, eventually exhausting available disk capacity and thereby disabling the entire server, not just the application itself.

A soak test should therefore, besides memory and connection metrics, also track the growth of relevant log directories (say, `var/log` in a Magento installation) across the entire test run, to get a realistic picture of actual resource consumption under sustained load instead of blanking out this aspect entirely.

5. Typical durations and load intensity for soak tests

While a regular load test is designed for high but short-lived load, a soak test deliberately targets moderate, realistic everyday load over a considerably longer period, typically between eight and twenty-four hours, in critical cases even several consecutive days. The load intensity itself usually follows the actually observed, average production load, not an artificial maximum value, since the goal isn't reaching a load limit but uncovering effects that only accumulate over time.

For a Magento store, for instance, a soak test with a constant, moderate number of concurrent virtual users running realistic catalog requests, cart actions, and occasional checkouts over twenty-four hours works well, while memory, connection, and log metrics get recorded continuously in parallel.

6. How it differs from short load spike tests

A spike test (see the separate article on this topic) and a soak test deliberately pursue opposite goals: a spike test checks how a system reacts to a sudden, brief, extreme load increase, while a soak test deliberately relies on constant, moderate load over a long time to uncover cumulative rather than momentary effects. Both test types complement each other but cover fundamentally different failure classes and therefore don't replace one another.

A system that easily survives a spike test can still fail a soak test if it correctly absorbs brief load spikes but slowly bleeds resources over hours, an important reason why a complete test portfolio ideally combines both types instead of relying on just one.

7. Automated monitoring and alerting during a soak test

Since a soak test runs for hours or days, manually watching metrics continuously is neither practical nor sensible. Instead, a soak test should come equipped from the start with automated alert thresholds that notify the responsible team as soon as a monitored metric, say memory consumption or the number of open database connections, crosses a defined limit, instead of evaluating the result manually only after the full test run completes.

This alerting also allows investigating a problem while the test is still running, say by deliberately pulling a memory snapshot (heap dump) at exactly the moment memory consumption starts deviating markedly from the expected trend, which considerably eases the subsequent root-cause analysis compared to a retrospective analysis without a concrete point of deviation.

8. Sensibly scheduling soak tests within the CI/CD pipeline

A soak test running for twenty-four hours naturally doesn't fit into a regular pull request workflow designed for fast feedback within a few minutes. In practice, a time-decoupled approach establishes itself instead: a soak test starts automatically, say via a nightly or weekly scheduled CI job, against a dedicated, production-like staging environment, while regular development work on the main branch continues independently.

What matters is that a soak test's results don't vanish into an isolated log but get actively communicated to the responsible team, say via an automated summary posted to a team chat channel once the test run completes, since a soak test whose result nobody actually looks at completely misses its purpose, regardless of how cleanly it was configured technically.

9. Soak test findings at a glance

The table below summarizes the most important findings a soak test can surface.

10. Summary

11. FAQ: Soak Tests: The Essentials at a Glance