System Monitoring with top, htop, and Alternatives

1. Getting load average right: uptime, nproc, and the three-value rule

The uptime command returns three numbers that represent exponentially weighted moving averages of the run queue length over the last 1, 5, and 15 minutes. On Linux, the run queue counts not only processes waiting for the CPU, but also processes in state D (uninterruptible sleep) that are waiting on blocking I/O. This is different from classic BSD systems, where only CPU-waiting processes are counted. A high load average therefore does not necessarily mean a CPU bottleneck, it can just as easily point to an overloaded storage controller or hung NFS mounts.

To evaluate a load figure meaningfully, you need to put it in relation to the number of available CPU cores, which nproc returns. A load of 4.0 on a four core server means the CPUs were fully utilized on average, but no processes were yet queued. Once the load climbs noticeably above the core count, processes are actively waiting on resources. The trend across the three time windows is more informative than a single snapshot: if the 1 minute load is sharply above the 15 minute load, that is an acute spike, not a sustained state.

# prometheus-alerts.yml
# Alert when load average per core stays above 1.0 for a sustained period
groups:
  - name: system-load
    rules:
      - alert: HighLoadPerCore
        expr: node_load5 / count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1.0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Load average per core exceeds 1.0 on {{ $labels.instance }}"
          description: "5m load / core count has been above 1.0 for 10 minutes. Check top/htop for the offending process."

2. Reading the top header correctly: Tasks, %Cpu(s), Mem, Swap

The first five lines of top carry more actionable information than the entire process list below them, yet they are often ignored. The Tasks line lists the total number of processes along with their breakdown into running, sleeping, stopped, and zombie. A growing zombie count points to a parent process that is not calling wait() correctly. The %Cpu(s) line breaks CPU time down into us (user), sy (system/kernel), ni (nice-adjusted processes), id (idle), wa (I/O wait), hi/si (hardware/software interrupts), and st (steal time in virtualized environments).

A high wa value points to an I/O bottleneck, while a nonzero st on a cloud VM means the hypervisor is taking CPU time away from your VM, something you cannot fix from the inside. The Mem and Swap lines show total, free, used, and buff/cache. What matters is not free, but available in /proc/meminfo, since the Linux kernel aggressively uses memory for page cache and releases it instantly under pressure. A low free value alongside a high buff/cache is generally not a problem at all. A sample header might look like load average: 3.84, 2.91, 2.20 with 7.1 wa, which points to an I/O bottleneck even before you look at the process list, and 2 zombie, which indicates a parent process missing a proper wait() call.

3. Using top interactively: sorting, filtering, and key keys

In interactive mode, top offers a set of keyboard shortcuts that make the difference between a quick find and minutes of scrolling. Shift+P sorts by CPU usage, Shift+M by resident memory, Shift+N by PID. With u you filter the view down to a specific user, for example www-data for PHP-FPM workers, and with o you can define an arbitrary filter expression such as COMMAND=nginx. The 1 key toggles between an aggregated CPU display and a per-core breakdown, which immediately reveals uneven core utilization in multithreaded workloads.

With c the full command line including arguments is shown instead of just the program name, which is essential when several PHP-FPM pools share the same binary and you need to identify the right pool. k sends a signal to a PID, r adjusts the nice priority without terminating the process. For automation, batch mode with -b combined with -n for the number of iterations matters, since it produces no terminal control characters and can be used in cron jobs and log files.

4. htop versus top: color coding, tree view, mouse control

htop presents the same core information as top, but in a noticeably more accessible form. The color-coded meter bars at the top show per-core CPU usage, memory, and swap graphically, so a bottleneck is visible at a glance before you even read a single line. The F5 key switches to tree view and makes parent-child relationships immediately visible, which for nested process groups like a PHP-FPM master with many workers gets you to the answer noticeably faster than manually tracing PPID columns in top.

F6 opens a sort menu listing every available column, and F9 shows a list of selectable signals instead of defaulting to only SIGTERM. Mouse clicks on column headers sort directly, and clicking a process highlights it persistently even as its position in the sorted list changes. The configuration, including column selection and color scheme, is stored persistently in ~/.config/htop/htoprc, so every SSH session shows the same customized view. For quick interactive diagnosis on a single server, htop is the better choice in almost every case, while top remains the more reliable option in scripts because it ships preinstalled on practically every Linux system.

5. Reading per-process memory correctly: VIRT, RES, SHR

The three memory columns in top and htop are regularly misread. VIRT (virtual memory) covers the entire virtual address space of a process, including mapped libraries, reserved but unused heap regions, and memory-mapped files. A high VIRT value alone is rarely a problem, since virtual address space is cheap and only consumes physical memory once actually accessed. RES (resident set size) shows the portion currently actually resident in physical RAM, including shared libraries.

That is exactly where the trap lies: SHR (shared memory) is the part of RES that is shared with other processes, such as libc or shared memory segments used by a database. If you add up the RES values of every PHP-FPM worker in a pool, you count shared libraries multiple times and massively overestimate actual memory consumption. For a correct total, the relevant figure is PSS (proportional set size), which distributes shared memory proportionally across all processes using it. Tools like smem -tk or a direct look at /proc/<pid>/smaps_rollup provide this more precise number, which neither top nor htop shows by default.

6. Sorting and filtering: identifying the actual offender

During an acute load problem, sorting gets you to the answer faster than scrolling through the entire process list. In htop, F6 followed by selecting PERCENT_CPU or M_RESIDENT sorts immediately by the relevant criterion, while in top the keys Shift+P and Shift+M achieve the same effect. With several instances of the same service, for example a MySQL server with many connection threads or a PHP-FPM pool, a sorted view of the main process line is often not enough, because a single hot thread stays hidden behind the aggregated process line.

This is exactly where the thread view helps: in htop, Shift+H expands every thread individually, in top the H key enables the same mode. This makes it visible which single thread inside a multithreaded process such as mysqld is actually consuming 100 percent of a core, something that stays hidden in the aggregated view. Combined with a filter on the suspected process name via o, or the search function / in htop, you can narrow the field of candidates down to a single PID within seconds.

# Find the single hottest thread inside a multithreaded process (e.g. mysqld)
top -H -p "$(pgrep -o mysqld)"

# Cross-check via /proc: per-thread CPU time for a given PID
ps -L -o pid,lwp,pcpu,stat,comm -p 4211 --sort=-pcpu | head -n 5

# htop equivalent workflow (interactive, keys annotated):
#   F6            -> sort menu, choose PERCENT_CPU
#   Shift+H       -> toggle per-thread view
#   /             -> search/filter by process or command name
#   F9            -> send a specific signal (not just SIGTERM) to the found PID

7. Modern alternatives: btop and glances compared

btop is a C++ successor to the earlier bpytop project, and its main advantage over htop is graphical history charts for CPU, memory, network, and disk I/O that build up over time instead of showing just the current value. Full mouse support lets you expand and collapse process trees with a click, integrated themes and, where drivers are present, even GPU utilization for Nvidia cards make it the preferred choice for longer interactive diagnosis sessions on a single host.

glances takes a different approach: it is written in Python, works cross-platform, and offers a web interface plus a REST API alongside its TUI. In client-server mode (glances -s on the target server, glances -c <host> from your workstation) you can monitor a remote system without logging in via SSH. Exports to InfluxDB, Prometheus, or CSV turn glances into a bridge between interactive debugging and centralized monitoring, while top and htop remain purely local tools.

; /etc/glances/glances.conf
; Enable web server mode and export metrics for central monitoring

[global]
refresh = 3
history_size = 1200

[cpu]
disable = False
career_max_history = 200

[outputs]
max_processes_display = 20

[influxdb]
host = influxdb.internal.mironsoft.de
port = 8086
db = glances
prefix = webserver01

8. When the extra install of btop or glances pays off

For a quick, one-off SSH diagnosis on a single server, top and htop are usually completely sufficient, especially since both come preinstalled on almost every distribution or are instantly available from the standard package sources. Installing btop pays off once you run longer diagnosis sessions and need historical trends, for example to see whether a CPU spike already started ten minutes ago, something a single snapshot cannot show. glances justifies the extra effort mainly when several servers need to be monitored centrally, without opening an interactive SSH session for each one.

In minimal container images without a package manager, such as distroless or scratch images, even htop is often unavailable. Here you either copy a statically linked binary into the image ahead of time, or fall back on the built-in BusyBox top, which offers noticeably less functionality than the GNU equivalent. For teams that want to standardize monitoring, the REST API of glances is also the most pragmatic way to poll metrics programmatically, without setting up a full Prometheus exporter pipeline.

// GET http://webserver01:61208/api/4/cpu
// glances REST API response, polled by an external dashboard or script
{
  "total": 42.7,
  "user": 31.2,
  "system": 9.8,
  "idle": 57.3,
  "iowait": 1.5,
  "steal": 0.2,
  "cpucore": 4,
  "ctx_switches": 184213,
  "interrupts": 92104,
  "time_since_update": 3.01
}

9. A practical live incident workflow: from alert to root cause

A structured workflow keeps you from getting lost in individual process lines during an incident. The first step after logging in is always uptime followed by nproc, to immediately put the load in relation to the core count. Right after that, a non-interactive snapshot with top -b -n 1 -o %CPU gives you a documentable, script-friendly capture that can also be redirected into a log file, before you go interactive at all. If this snapshot shows a clear CPU outlier, you switch to htop, enable tree view with F5, and if you suspect multithreading, additionally enable the thread view with Shift+H.

If the suspicion instead points to I/O wait, that is a high wa value in the header, you filter specifically for processes in state D and check iostat -x 1 in parallel for saturated block devices. Only once the process list and resource figures produce a concrete candidate do you move to deep analysis with strace -p or lsof -p, to identify the exact system call or blocked file. On distributed systems, glances --export csv automatically documents a history during the incident that can later be used for root cause analysis and the incident report, instead of relying on the memory of everyone involved.

# Step-by-step live incident workflow
uptime && nproc

# 1) Non-interactive, loggable snapshot first
top -b -n 1 -o %CPU | tee -a /var/log/incident-2026-07-12.log

# 2) Drill into the tree view interactively if a clear offender is visible
htop   # then F5 for tree view, Shift+H for per-thread view

# 3) If I/O-wait dominates, check block device saturation
iostat -x 1 5

# 4) Deep dive once a candidate PID is identified
strace -p 18422 -f -tt
lsof -p 18422 | head -n 20

# 5) Keep a running export for the post-incident report
glances --export csv --export-csv-file /var/log/glances-incident.csv

The table below sets the imprecise or risky approach against the recommended one for the most common monitoring tasks.

10. Summary

Reliable system monitoring starts with correctly reading the fundamentals: load average must always be evaluated in relation to the core count and, on Linux, also counts I/O-waiting processes. The header lines of top and htop, with %Cpu(s), Mem, and Swap, already carry most of the diagnosis before you look at a single process line. For memory figures, the rule is: do not add up RES values across processes, instead determine actual usage with PSS via smem or smaps_rollup.

Targeted sorting by CPU or memory, and the thread view via H or Shift+H, reliably lead to the actual offender instead of wasting time scrolling manually. Modern alternatives like btop and glances pay off once historical trends or centralized monitoring across several servers are needed, while top and htop remain the most reliable baseline for quick single-server diagnosis. A structured workflow from snapshot, through tree view, to deep analysis with strace saves decisive minutes during an acute incident.

11. FAQ: System Monitoring with top, htop, and Alternatives