Disk I/O Analysis with iostat and iotop

1. When Disk I/O Is Actually the Bottleneck

A classic misdiagnosis in practice: a shop feels sluggish, the team checks htop, sees moderate CPU load, and concludes that more CPU cores will fix it. In reality, the processes are stuck in state D (uninterruptible sleep), waiting for the disk to respond. CPU load alone says nothing about whether a system is disk I/O bound, because waiting processes barely consume compute time. The real signal is the %wa column in top or vmstat, which shows the share of CPU time the processor spends idle while waiting on outstanding I/O operations.

Once %wa is elevated over a longer period, say consistently above 20 percent, the next step is worthwhile: iostat and iotop reveal which disk is under how much load and which process is generating the requests. Without these two tools, any diagnosis remains speculation, because neither htop nor free shows the actual queue depth or the latency of individual I/O requests. The following sections show how to use concrete metrics to distinguish a genuine hardware bottleneck from an inefficient application, before expensive infrastructure decisions get made.

2. iostat Basics: Understanding %util, await, and svctm

iostat is part of the sysstat package and reads its data directly from /proc/diskstats. In extended mode with -x, it reports per-device metrics such as r/s and w/s (read and write operations per second), rkB/s and wkB/s (throughput), await (average wait time of an I/O request in milliseconds, including queueing), avgqu-sz (average queue length), and %util (the share of time the device had at least one outstanding request). The -z option hides inactive devices and keeps the output readable.

A common trap: the very first report from iostat shows the average since the last system boot, not the current state. For a reliable snapshot, ignore that first block and only evaluate the second or later reports. The svctm column is considered unreliable with modern kernel I/O schedulers and has been removed in recent sysstat versions, which is why await together with avgqu-sz are today's more meaningful metrics for latency and saturation.

# Install sysstat if iostat/iotop-related tooling is missing (Debian/Ubuntu)
apt-get install -y sysstat

# Extended stats, skip inactive devices, refresh every second
iostat -xz 1

# Example output (second report, first block already discarded)
# Device            r/s     w/s   rkB/s   wkB/s  await  avgqu-sz  %util
# nvme0n1           4.00  312.00   64.00  9840.00  38.42     6.10  97.50
# sda               0.00    2.00    0.00    24.00   1.80     0.01   0.40

3. Interval Sampling with iostat for Trend Detection

A single snapshot with iostat -x 1 1 is nearly worthless for a serious diagnosis, because it only captures one arbitrary second. Short bursts, for example from a single backup job or a cron run, look identical to a persistent structural problem in a single snapshot. Only continuous sampling over several minutes or hours reveals whether %util and await actually stay consistently elevated, or just spike briefly and normalize again afterward.

For production systems, iostat -xz 2 30 is a good starting point, 30 samples spaced two seconds apart, redirected into a log file for later analysis. Anyone who needs durable historical data should enable persistent sampling via sysstat's sa1, which automatically writes binary data into /var/log/sysstat and can later be evaluated for any time window with sar -d, for example to check whether an incident three days ago at 2 PM coincided with an I/O spike.

4. iotop: Identifying the Process Behind the Load

iostat shows which device is under load, but not which process is responsible. That is exactly the gap iotop fills: it reports actual read and write rates per process and thread via the kernel's taskstats interface. Because that interface communicates over netlink sockets and exposes privileged kernel information, iotop requires root privileges or the CAP_NET_ADMIN capability; a plain invocation as a normal user fails with an error.

The -o option (only show processes with actual I/O) trims the output to what matters, while -a shows accumulated rather than instantaneous rates, making short but frequent I/O bursts visible that would otherwise disappear in a single snapshot. For scripts and cron-based monitoring, batch mode with -b writes output to a log file without requiring an interactive terminal. The PID reported by iotop can then be mapped to a concrete command with ps -fp <pid> or cat /proc/<pid>/cmdline.

# Interactive view, only processes with actual I/O, accumulated totals
iotop -oa

# Batch mode for cron/logging: 5 samples, 2 seconds apart
iotop -b -n 5 -d 2 -o > /var/log/iotop-batch.log

# Resolve the PID iotop reported to the actual command line
ps -fp 18342
# UID   PID  PPID  C STIME TTY  TIME     CMD
# mysql 18342 1     6 09:14 ?   00:04:12 mysqld --datadir=/var/lib/mysql

5. Interpreting %util and await Correctly

%util at 100 percent does not automatically mean a bottleneck. On a single rotating disk with only one read/write head, 100 percent is genuinely a saturation signal. On modern NVMe SSDs with many parallel queues (up to 64,000 queues with 64,000 entries each per the NVMe specification), the device can sit at 100 percent %util and still comfortably serve additional parallel requests without a noticeable latency increase. What matters is always the combination of %util and await, never a single metric in isolation.

await also needs a reference value per hardware type: a good NVMe SSD typically sits at 1 to 3 milliseconds, a SATA SSD at 3 to 8 milliseconds, a rotating disk at 5 to 15 milliseconds, and network storage such as iSCSI or NFS can show significantly higher values than normal depending on network latency. Without that baseline for the specific setup, any alerting threshold is arbitrary. A rising avgqu-sz alongside rising await is the most reliable signal of real saturation, because it shows that requests are piling up instead of being processed promptly.

6. Telling a Slow-Disk Problem from a Slow-Application Problem

The central diagnostic question is: is the application waiting on a genuinely slow disk, or is the application itself generating unnecessarily many I/O requests that would bring even a fast disk to its knees? Both cases show similarly high %wa values in top, but they can be clearly separated: if await stays high even at a moderate request rate, that points to a real hardware or storage-layer problem, such as a degraded RAID array or overloaded network storage. If instead mainly the number of requests (r/s/w/s) climbs drastically while the individual request is still answered normally fast, the problem sits in the application code, for example due to missing indexes or too small a cache.

A simple practical test confirms the diagnosis: if the suspected cause is fixed in code, for example by adding a missing index or enabling a query cache, and %util as well as await then drop noticeably, the application was the cause. If the values stay high despite a reduced request rate, the bottleneck genuinely lies in the storage layer itself and needs to be addressed there, for instance with faster disks or a reviewed RAID configuration.

{
  "check": "disk_io_bottleneck_classification",
  "host": "shop-prod-03",
  "timestamp": "2026-07-12T14:02:00Z",
  "device": "nvme0n1",
  "await_ms": 4.20,
  "util_percent": 38.00,
  "iops_read": 210,
  "iops_write": 1840,
  "cpu_iowait_percent": 3.10,
  "classification": "application_bound",
  "note": "High IOPS but low await/util: check application query patterns, not the storage layer"
}

7. Practical Example: Diagnosing MySQL and Magento I/O

A typical scenario: a Magento shop becomes noticeably slower after a traffic spike, page loads take several seconds. iostat -xz 2 30 shows an await value of a constant 45 milliseconds for the volume backing /var/lib/mysql, with %util near 95 percent, well above the normal value for the SSD in use. iotop -oa confirms it: mysqld generates the vast majority of the write load, with an unusually large number of small, random writes instead of fewer, larger sequential blocks.

The next step is not swapping the disk, but checking MySQL itself: SHOW ENGINE INNODB STATUS and the InnoDB buffer pool hit ratio show whether the buffer pool (innodb_buffer_pool_size) is sized too small for the active dataset. If the hit ratio is low, MySQL has to hit the disk for nearly every read instead of serving it from memory, which produces exactly the observed pattern as traffic increases. Increasing the buffer pool to a realistic share of available RAM noticeably lowers await and %util in this case, without any new hardware needing to be purchased at all.

8. Complementary Tools: pidstat, /proc/diskstats, and blktrace

pidstat -d 1, from the same sysstat package, offers an alternative to iotop that is better suited to historical logging, because like iostat it outputs plain text lines that can easily be redirected to a file or processed further with awk. For custom monitoring scripts that should not depend on external tools, /proc/diskstats exposes the same raw data iostat calculates its metrics from, directly as numeric columns, though without any pre-processing such as percentages or moving averages.

If the cause remains unclear despite iostat and iotop, for example because the latency does not originate at the device itself but somewhere in the block layer stack, blktrace together with blkparse helps with a detailed recording of every single I/O request, including timestamps for queueing, dispatch, and completion. That is considerably more work to analyze than iostat, but it also surfaces effects like I/O merging by the scheduler or unusually long dwell times of individual requests in the queue, both of which stay hidden in aggregated metrics. For durable historical collection, the persistent configuration of sysstat is usually enough.

; /etc/default/sysstat - enable persistent historical sampling on Debian/Ubuntu
ENABLED="true"

; /etc/cron.d/sysstat - default sampling every 10 minutes into /var/log/sysstat
*/10 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1

; Query historical disk stats for a specific window later on
; sar -d -f /var/log/sysstat/sa12 -s 14:00:00 -e 14:30:00

# prometheus alert rule: fire when a device is both saturated and slow
groups:
  - name: disk-io
    rules:
      - alert: DiskIOBound
        expr: |
          rate(node_disk_io_time_seconds_total[5m]) > 0.9
          and
          rate(node_disk_write_time_seconds_total[5m])
            / rate(node_disk_writes_completed_total[5m]) > 0.03
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Disk {{ $labels.device }} saturated with high await"
          description: "util > 90% and await > 30ms for 5m - likely a real storage bottleneck"

9. iostat and iotop Compared

The following table summarizes the threshold values that have proven useful in practice for telling normal operation apart from a real problem. It is worth remembering: a single value outside the norm is not proof by itself, what matters is the pattern over time combined with several metrics together.

In practice, it is worth encoding these thresholds as Prometheus or Zabbix rules rather than only checking them manually with iostat. That way a gradual degradation over weeks gets noticed long before users feel it through load times, and the team is not stuck reactively hunting for the cause only after a complaint comes in.

10. Summary

Disk I/O analysis with iostat and iotop follows a clear sequence: first, %wa in top confirms that processes are waiting on I/O at all. Then iostat -xz with interval sampling over several minutes shows whether %util and await are actually consistently elevated or just spiking briefly. iotop -oa then identifies the specific process behind the load, while pidstat -d provides the same information for historical logging. A single snapshot is not enough for any of these questions, every solid diagnosis needs continuous observation over time.

The most common thinking error in practice is assuming that high I/O load automatically means a disk that is too slow. Often the opposite is true: the application itself generates too many or too inefficient requests, for example through an undersized InnoDB buffer pool or missing indexes. If await and %util remain high after an application optimization, the bottleneck genuinely lies in the storage layer and justifies a hardware investment. This order, check the application first, then question the hardware, saves unnecessary cost in most cases.

11. FAQ: Disk I/O Analysis with iostat and iotop