Systematically Tracking Down Memory Leaks in Linux Processes

1. What a memory leak on Linux really is

A memory leak occurs when a process keeps claiming more and more memory over time without ever releasing it, even though the actual workload stays constant. On Linux this means concretely: the process keeps requesting new memory pages through malloc() or a language runtime such as the PHP Zend allocator, but never returns them to the operating system due to a programming bug, a circular reference, or a poorly configured caching layer. The kernel itself has no concept of a memory leak, it only sees that a process keeps claiming more and more physical memory (RSS).

It is important to distinguish this from normal memory behavior: many applications grow quickly at startup because they fill caches, build connection pools, or initialize opcode caches such as OPcache. That is not a memory leak, it is a one-time rise to a stable plateau. A real leak is recognized by the curve not flattening out after the plateau but continuing to rise linearly or in steps over hours or days, while the number of processed requests stays constant. This distinction between normal growth and a real leak is the first and most important step in any diagnosis.

2. Recognizing symptoms: RSS growth, swap pressure, OOM kills

The earliest visible symptom of a memory leak is a steadily rising RES value in top or htop for a single process or a group of similar workers, for example all PHP-FPM child processes at once. At the same time, the memory reported as available in free -h keeps shrinking as the page cache gets squeezed out in favor of the growing processes. On servers with active swap, the leak often shows up first as rising swap usage and growing si/so values in vmstat, as the kernel tries to swap out rarely used pages to make room.

In the final stage the OOM killer steps in and terminates the process with the highest memory usage, visible in dmesg as "Out of memory: Killed process". The problem here: the OOM killer often does not hit the actual culprit but the process with the worst oom_score, which causes confusion when, for example, the MySQL server gets killed instead of the actually leaking PHP worker. A systematic look at /proc/[pid]/oom_score before the next kill helps identify the real candidate instead of being led astray by the symptoms.

3. Measuring memory precisely with /proc/[pid]

Before installing tools like smem, it is worth taking a direct look at /proc/[pid]/status and /proc/[pid]/smaps_rollup, since both are available on every Linux system without extra software. VmRSS in status shows the total resident memory, VmSwap the portion already swapped out. What matters most for a memory leak, though, is the breakdown in smaps_rollup, which distinguishes between Rss, Pss (proportionally shared memory), Private_Dirty, and Shared_Clean. A growing Private_Dirty value while Shared_Clean stays stable is a strong indicator that the process itself, and not a shared library, is responsible for the growth.

A simple repeating script turns this manual check into a reliable time series. Instead of glancing at the numbers once, you log the values at fixed intervals and can later present a clear growth curve instead of a subjective impression. That curve is exactly what turns a conversation with developers from "feels like a leak" into "provably 40 MB growth per hour".

#!/usr/bin/env bash
# track-rss.sh — sample RSS and private-dirty memory for a PID over time
set -euo pipefail

PID="${1:?Usage: track-rss.sh <pid> <log-file>}"
LOG_FILE="${2:?Usage: track-rss.sh <pid> <log-file>}"
INTERVAL=60

echo "timestamp,rss_kb,pss_kb,private_dirty_kb" > "$LOG_FILE"

while kill -0 "$PID" 2>/dev/null; do
  ts="$(date -Iseconds)"
  # Read the aggregated smaps rollup — no external tools required
  rss=$(awk '/^Rss:/ {print $2}' "/proc/$PID/smaps_rollup")
  pss=$(awk '/^Pss:/ {print $2}' "/proc/$PID/smaps_rollup")
  dirty=$(awk '/^Private_Dirty:/ {print $2}' "/proc/$PID/smaps_rollup")
  echo "$ts,$rss,$pss,$dirty" >> "$LOG_FILE"
  sleep "$INTERVAL"
done

echo "Process $PID exited — log written to $LOG_FILE"

4. smem and pmap: separating shared from private memory

smem builds on the same /proc data but presents it aggregated and with the so-called PSS (Proportional Set Size), which fairly splits shared memory across all involved processes. This matters because a plain RSS sum across several PHP-FPM workers would count the shared OPcache memory multiple times and thus fake a memory leak where only normal library and opcode cache usage is actually happening. With smem -tk -P php-fpm you get a sorted overview of all PHP-FPM processes including USS (purely private memory), which is the most reliable figure for actual, non-shared consumption.

pmap -x complements this view by listing the memory mappings of a single process: heap, stack, loaded libraries, and anonymous mappings, each with size and dirty portion. A memory leak in application code typically shows up here as a continuously growing heap entry ([heap]) or a growing number of anonymous mappings, while library mappings stay stable. Anyone who regularly logs pmap -x $PID | tail -1 for the total sum can immediately see whether the growth happens in the heap or in dynamically loaded libraries.

# Install smem if not already present (Debian/Ubuntu)
sudo apt-get install -y smem

# Sorted overview of all php-fpm workers by private memory (USS)
smem -tk -P php-fpm

# Example output (trimmed):
#   PID User     Command                         Swap      USS      PSS      RSS
#  4821 www-data php-fpm: pool www              0        48.2M    52.1M    61.4M
#  4822 www-data php-fpm: pool www              0        49.8M    53.6M    62.9M
#  4823 www-data php-fpm: pool www              0        91.3M    95.7M   104.2M   <- outlier
#  ----------------------------------------------------------------------
#                 3                              0       189.3M   201.4M   228.5M

# Detailed memory map for the suspicious worker (PID 4823)
pmap -x 4823 | sort -k3 -n -r | head -20

# Watch heap growth of a specific process every 30 seconds
watch -n 30 'pmap -x 4823 | grep "\[ heap \]"'

5. Logging growth over time instead of guessing

A single snapshot never proves a memory leak, since normal load fluctuations can produce similar values. Only a time series spanning several hours or days shows whether memory usage drops again after a rise (normal behavior under load) or keeps growing monotonically (a leak). A simple cron job running smem or the track-rss.sh script shown above every five minutes and writing to a CSV file provides the raw data needed for a solid conclusion.

For the analysis, a simple plot with gnuplot or an import into a spreadsheet is often enough. More important than the visualization, though, is correlating the growth with external events: if memory usage climbs again after every deployment, that points to a leak in newly shipped code. If it grows steadily over days regardless of deployments, the cause is more likely a rarely triggered code path combination or a long-running cache without an eviction strategy. Building in this correlation is the difference between a chart and an actual diagnosis.

6. PHP-FPM workers: the most common suspect in the web stack

In PHP-based setups such as Magento, the PHP-FPM worker process is by far the most common source of observable memory leaks, because a single request handler process is reused across many requests and can accumulate state that should actually be discarded per request. Static class variables, object caches in application code that are never cleared, or extensions with their own C memory handling (such as buggy PHP extensions) are typical causes. The built-in countermeasure in PHP-FPM is pm.max_requests: after the configured number of requests, the worker process is terminated and restarted, guaranteeing that memory is released regardless of whether the actual cause in the code has been fixed.

This mechanism only fixes the symptom, not the cause, and can even come too late with very aggressive leaks (for example several hundred megabytes within minutes). It makes more sense to keep pm.max_requests as a safeguard while simultaneously observing actual memory development per worker through request_slowlog_timeout and the PHP-FPM status endpoint (pm.status_path). A worker that occupies significantly more memory than its siblings shortly before the configured limit is a strong hint of a concrete, reproducible trigger request, which can be traced back through the access logs.

; /etc/php/8.4/fpm/pool.d/www.conf — leak mitigation and observability
[www]
; Recycle workers after N requests — bounds the blast radius of any leak
pm.max_requests = 500

; Expose per-pool status for external monitoring
pm.status_path = /fpm-status
ping.path = /fpm-ping

; Log requests slower than 5 seconds with a full backtrace
request_slowlog_timeout = 5s
slowlog = /var/log/php-fpm/slow.log

; Emergency restart if a worker exceeds this much memory (requires php.ini)
; php_admin_value[memory_limit] = 512M

7. Digging deeper: strace, massif and application metrics

When worker recycling and monitoring confirm the suspicion but the exact code location remains unclear, deeper tools are needed. strace -e trace=memory -p $PID shows every brk() and mmap() syscall in real time and reveals whether memory grows in large, rare jumps (typical of a single faulty request) or in many small steps. For PHP applications themselves, memory_get_usage(true) and memory_get_peak_usage(true) placed at strategic points in the code are often more informative than any system tool, because they show the perspective of the PHP runtime instead of the operating system view.

For native extensions or C libraries loaded by PHP, valgrind --tool=massif is the most thorough but also the slowest tool: it logs every memory allocation over the runtime and, with ms_print, produces a chart that shows exactly which call chain is responsible for the growth. Due to the substantial performance overhead of Valgrind, this approach is only suitable in a staging environment with a reproduced leak, not in live operation. The combination of application metrics for fast triage and Massif for deep root cause analysis covers almost every memory leak encountered in practice.

# Trace memory-related syscalls of a running process for 30 seconds
timeout 30 strace -tt -e trace=memory -p 4823 2>&1 | tee /tmp/memtrace.log

# Count how many brk() calls actually grew the heap
grep -c '^brk(' /tmp/memtrace.log

# Reproduce a suspected leak under Valgrind's massif profiler (staging only)
valgrind --tool=massif --massif-out-file=massif.out \
  php-fpm -F -y /etc/php/8.4/fpm/php-fpm.conf

# Turn the raw profile into a human-readable call-graph report
ms_print massif.out | less

8. Using cgroup v2 limits as a safety net

As long as the actual cause of a memory leak is not fixed, a hard cgroup v2 memory limit prevents a single leaking service from destabilizing the entire server. Through systemd, MemoryMax can be set directly for a service, which causes the kernel to trigger the cgroup OOM killer instead of the global OOM killer once the service exceeds its limit. The decisive advantage: only the affected service is terminated, not some arbitrary other process like the database, which considerably improves the predictability of outages.

In addition, systemd-cgtop provides a live overview of memory usage across all cgroups and immediately shows which service is approaching its limit, long before a kill becomes necessary. These limits do not replace root cause analysis, but they are the most pragmatic immediate measure to keep a known memory leak manageable in production while the actual fix in code or configuration proceeds in parallel.

# /etc/systemd/system/php8.4-fpm.service.d/memory-limit.conf
[Service]
# Hard cap — cgroup OOM killer targets only this service, not the whole host
MemoryMax=1536M
# Soft warning threshold logged to the journal before the hard limit hits
MemoryHigh=1280M

sudo systemctl daemon-reload
sudo systemctl restart php8.4-fpm

# Live view of memory usage per cgroup, refreshed every 2 seconds
systemd-cgtop -m

# Check whether the cgroup OOM killer already intervened
journalctl -u php8.4-fpm --since "1 hour ago" | grep -i "killed process\|oom"

9. Memory leak diagnostic tools compared

The tools presented cover different phases of investigating a memory leak, from first detection to exact root cause analysis. The overview below arranges them by purpose and effort.

The pragmatic path starts at the top of the table: smaps_rollup and smem for the first confirmation, pmap for rough localization, strace for correlating with requests over time, and only lastly valgrind massif in a staging environment when all other means fail to yield a clear cause. This order saves time, because most memory leaks in practice can already be located with the first three tools.

10. Summary

A memory leak on Linux is not recognized by a single high number, but by a curve that keeps monotonically rising over hours or days and does not drop even after load subsides. The systematic investigation starts with /proc/[pid]/smaps_rollup for the first confirmation, moves through smem and pmap to localize between heap and shared mappings, and uses strace as well as, in stubborn cases, valgrind massif for the exact cause. For PHP-FPM workers, pm.max_requests is an effective but symptomatic immediate measure.

Alongside the actual root cause investigation, a cgroup v2 memory limit ensures that a known, not-yet-fixed memory leak does not destabilize the entire server but only affects the service in question in a controlled way. Anyone who logs time series, correlates deployments, and applies the tools in the right order will find the cause of a memory leak in most cases within a few hours instead of living with repeated restarts for weeks.

11. FAQ: Tracking Down Memory Leaks in Linux Processes