Diagnosing High Load Average Systematically

1. What Load Average actually measures

Load Average does not count CPU utilization in percent, it counts the average number of processes that are either running on the CPU or waiting for a resource while sitting in the kernel's runqueue. That second part is exactly where the confusion starts: a process waiting on a slow disk counts toward Load Average in Linux just as much as a process actively computing. Anyone who reflexively interprets a high Load Average as a CPU problem misses half of the possible causes right away.

The three values from uptime show the moving average over one, five and fifteen minutes. A value of 12 on a four-core server means an average of twelve processes running or waiting against four available cores, a clear oversubscription. Whether that oversubscription comes from actual compute load, storage latency or blocked network calls is not revealed by the raw number alone. That is precisely why a systematic diagnosis needs more than a single command, and that is the core of this workflow for Load Average.

2. First look: uptime, /proc/loadavg and vmstat

The first step in any Load Average diagnosis is establishing the trend. uptime gives the three time windows at a glance, /proc/loadavg additionally provides the current runqueue size and total process count as a fourth and fifth value. If the one-minute value rises much faster than the fifteen-minute value, it is an acute spike. If all three values are similarly high, an ongoing problem has already existed for a while.

vmstat 2 10 shows ten samples at two-second intervals and splits CPU time into us (user), sy (system), id (idle) and wa (IO wait). A high wa value combined with a high Load Average is the first strong hint at IO-bound waiting rather than actual compute load. This step already decides which direction the further diagnosis takes, before even looking at a single process.

# Step 1: quick orientation before deep-diving into any single process
uptime
# 14:32:10 up 12 days,  3:11,  2 users,  load average: 11.84, 9.02, 6.55

cat /proc/loadavg
# 11.84 9.02 6.55 14/312 28841
# fields: 1min 5min 15min running/total-processes last-pid

# Sample CPU states every 2 seconds, 10 samples
vmstat 2 10
# procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
#  r  b   swpd   free  buff  cache   si   so    bi    bo   in   cs us sy id wa st
# 14  3      0 812340 98234 4213456    0    0   412  1820 3201 5920 18  6 12 64  0

In this example wa sits at 64 percent and id at only 12 percent, while us and sy combined make up barely a quarter of the time. That is an unambiguous signal: Load Average here is not driven by compute load but by waiting processes. The b (blocked) column confirms this further, showing processes waiting on non-interruptible resources that directly add to the runqueue count.

3. Distinguishing CPU-bound from IO-bound load

Distinguishing between CPU-bound and IO-bound Load Average is the most important fork in the entire diagnostic workflow, because both cases require completely different remedies. With CPU-bound load, additional cores, prioritization with nice or moving batch jobs to quieter time windows all help. With IO-bound load, additional cores help not at all, because the processes are not computing in the first place, they are waiting.

mpstat -P ALL 2 5 shows CPU utilization per core instead of aggregated, revealing whether only a single core is fully saturated by a single-threaded process while the others sit idle. This is a common pattern with PHP-FPM workers or poorly parallelized batch scripts. If, on the other hand, every core is evenly utilized while wa is also high, that points to a system-wide storage problem, for example an overloaded NFS share or a slow cloud disk with a limited IOPS budget.

# Per-core CPU breakdown — reveals single-core saturation
mpstat -P ALL 2 5
# 14:35:01     CPU    %usr   %sys  %iowait   %idle
# 14:35:03     all    18.20   6.10    64.30   11.40
# 14:35:03       0    72.00   4.00     2.00   22.00
# 14:35:03       1     3.10   6.80    88.10    2.00
# 14:35:03       2     2.90   7.20    87.60    2.30
# 14:35:03       3     3.30   6.20    89.50    1.00

# core 0 is CPU-bound (single-threaded process), cores 1-3 are IO-bound

This breakdown is exactly what makes the difference: core 0 is actually computing, the remaining three cores are waiting on IO almost exclusively. A blanket conclusion like just buy more CPU cores would only address a fraction of the actual problem here. The Load Average diagnosis needs to move on toward storage and process attribution at this point.

4. Identifying processes in D state

Processes in the so-called D state (uninterruptible sleep) are the most direct proof of IO-bound Load Average spikes. A process in this state is waiting on an internal kernel operation, usually a disk or network IO operation, and cannot even be interrupted by a signal such as SIGKILL during that time. This is exactly why many classic debugging approaches do not help here: the process simply does not respond, because it is stuck inside the kernel, not in userspace.

ps -eo pid,ppid,state,wchan:32,comm --sort=-state filters specifically by state and additionally shows the kernel function name (wchan) the process is waiting on. Values like jbd2_journal_commit point to filesystem journal activity, nfs_wait_bit_killable to a hanging NFS mount. This information is often more informative than any generic performance graph, because it points directly at the kernel component causing the stall.

# List processes stuck in uninterruptible sleep (D state)
ps -eo pid,ppid,state,wchan:32,comm --sort=-state | awk '$3=="D"'
#   PID  PPID S WCHAN                            COMMAND
# 28841 28102 D jbd2_journal_commit             mysqld
# 29103     1 D nfs_wait_bit_killable           rsync

# Count D-state processes over time — a repeated snapshot is more useful than one
for i in $(seq 1 5); do
  echo "$(date +%T): $(ps -eo state | grep -c ^D) processes in D state"
  sleep 2
done

If the repeated counting loop shows a stable number of D-state processes over several seconds, this is not a brief outlier but a persistent blocking problem. In practice this is frequently a single process with excessive fsync behavior, a full journal, or an unstable network mount that is dragging the Load Average up for the entire system.

5. pidstat and top for per-process attribution

Once it is clear whether CPU or IO is the driving force behind Load Average, the workflow needs to break down to individual processes. pidstat -d 2 5 shows per-process read and write rates in kilobytes per second along with IO delay percentages, while pidstat -u 2 5 provides classic CPU time per process. The advantage over top is the temporal resolution: pidstat delivers multiple consecutive measurement points and thereby reveals whether a process is under sustained or only brief load.

Interactive top with the 1 key for the per-core view and sorting by %CPU, or htop with its IO column, add a fast visual perspective during live operations. It is important to combine both tools during maintenance windows: pidstat for reliable, loggable numbers, and top/htop for quick live observation during an acute incident.

6. iostat: disk latency as the most common cause

In many cases, the actual root of a high Load Average is not a single process but the underlying storage itself. iostat -x 2 5 shows per block device the queue length (avgqu-sz), the average wait time (await) and utilization (%util). A %util near 100 percent combined with a high await shows that the storage device itself is the bottleneck, regardless of which process is currently accessing it.

# Extended disk statistics — the storage device itself may be the bottleneck
iostat -x 2 5
# Device            r/s     w/s   rkB/s   wkB/s  await  %util
# nvme0n1          12.50  420.30  512.00 8940.10  184.20  98.70

# await of 184ms on an NVMe device is far above the expected sub-millisecond range
# — this points to a saturated underlying volume, not a specific process

An await of 184 milliseconds on an NVMe device, which should normally deliver values under a millisecond, is a clear warning sign. On cloud instances with throttled IOPS budgets this is a very common pattern: the storage backend is limiting, not the process. The consequence for the Load Average diagnosis is that no application change but a storage upgrade or an IOPS adjustment is the actual fix here.

7. Understanding runqueue and context switches with vmstat

The r column in vmstat shows the number of processes currently waiting for a free CPU core, i.e. the actual runqueue length. If this value permanently exceeds the number of available cores by a wide margin, the system is genuinely CPU-bound and overloaded, independent of the IO share. The cs column shows context switches per second: a sudden increase points to many short, competing processes, for example due to a PHP-FPM configuration with too many workers for the available core count.

A frequently overlooked relationship in Load Average analysis: high context switch rates cost CPU time themselves, because every switch invalidates the processor cache. A system with many small, short-lived processes can therefore produce a higher effective load than a system with fewer, longer-running processes, even at identical nominal CPU utilization. In practice this often justifies reducing the worker count rather than increasing it.

8. Load Average inside containers and cgroups

Inside a Docker container, uptime traditionally shows the Load Average of the host system, not the container itself, because the kernel maintains the runqueue system-wide. Anyone seeing a high Load Average inside a container may therefore be measuring the load of other containers on the same host as well. In practice this regularly leads to misdiagnosis when a team debugs exclusively inside its own container and loses sight of the host.

For an accurate assessment, Load Average should always additionally be measured on the host, for example with docker stats for per-container CPU percentages or with the cgroup's own metrics under /sys/fs/cgroup/cpu.stat in cgroup v2. The nr_throttled value there shows how often a container was throttled against its CPU quota limit, which also manifests as increased wait time even though the host itself might still have free capacity.

9. Symptoms, tools and causes compared

The following overview summarizes the typical symptom combinations and maps them to the appropriate tool and the most likely cause. It serves as a quick entry point for the next incident involving high Load Average, but does not replace the detailed analysis from the previous sections.

The common denominator across all cases in the table: the raw Load Average number alone is never enough for a decision. Only the combination of CPU breakdown, process state and storage metrics turns an alert into a reliable diagnosis you can actually act on, instead of making expensive infrastructure changes based on guesswork.

10. Summary

A high Load Average is a symptom, not a finding. The workflow described here starts with uptime and vmstat to establish the broad direction between CPU-bound and IO-bound load, moves to mpstat for the per-core view, identifies specific blocked processes via ps and the D state, and confirms storage bottlenecks with iostat -x. Only this combination provides a reliable basis for a decision.

Anyone who jumps straight to infrastructure upgrades when facing a high Load Average without going through these steps risks expensive mistakes: additional CPU cores for a pure storage problem, or a storage upgrade for what is actually a misconfigured worker pool. The systematic look at CPU state, process state and storage latency saves not only time but also budget in practice.

11. FAQ: Diagnosing Load Average