Tracking Down Mysterious CPU Usage

1. The first suspicion: top and the right questions

The entry point for any diagnosis of mysterious CPU usage is top or htop, sorted by %CPU using the P key. But the mere observation that a process consumes a lot of CPU is not yet a diagnosis, it is the starting point for a series of targeted questions: is it always the same process, or does the PID keep changing? Is the load sustained or does it come in waves? Did it start after a specific event, such as a deploy, a cron job, or an incoming request?

These preliminary questions determine the further direction. CPU usage permanently stuck on a fixed PID suggests an infinite loop or a hanging syscall. Changing PIDs with a constant process name, for example ever new php-fpm workers, points more toward a recurring pattern in incoming requests that burdens every worker equally. Without this classification, further searching often leads nowhere.

Also helpful is a quick look at uptime, to check whether the system's overall Load Average has risen as well, or whether the CPU usage is concentrated exclusively on a single process while the rest of the system remains unaffected.

Another useful early indicator is the absolute number of affected processes. If CPU usage affects only a single worker out of twenty, that points toward a specific, reproducible trigger, for example a concrete request. If, on the other hand, all workers are affected simultaneously, the cause is more likely a global configuration change or a system-wide event such as a failed deploy.

2. Process or single thread? Using -H in top

Many modern applications, including Java processes, Node.js with worker threads, and some PHP extensions, are multi-thread capable. A process showing 100 percent CPU overall might spread that load across ten threads at ten percent each, or concentrate it in a single fully saturated thread. This distinction is crucial for further diagnosis of mysterious CPU usage, because tools like strace and perf can be targeted at individual thread IDs.

top -H -p <PID> switches on the thread view and shows every thread of the target process with its own thread ID (TID) and individual CPU load. If exactly one thread is responsible for the bulk of the load, further analysis can be focused specifically on that one TID instead of considering the entire process with all its threads.

An equivalence rule helps with classification: the sum of individual thread percentages in the TID view should roughly match the total value of the process in the normal process view. If this sum deviates significantly, that points to short-lived threads that were created and terminated between two measurements, which for diagnosing mysterious CPU usage is an additional hint at a thread-pool pattern.

# Show per-thread CPU usage for a specific process
top -H -p 28841
#   PID   TID USER      %CPU  COMMAND
# 28841 28855 www-data   98.7  php-fpm
# 28841 28856 www-data    0.3  php-fpm
# 28841 28857 www-data    0.2  php-fpm

# thread 28855 is clearly the culprit, not the process as a whole

# Cross-check with pidstat for a time-resolved view of the same thread
pidstat -t -p 28841 2 5
# Time   UID      TGID       TID    %usr %system  %CPU  Command
# 14:40  33      28841         -    0.10    0.20  0.30  php-fpm
# 14:40  33         -     28855   97.80    0.90 98.70    |__php-fpm

3. When did it start? Checking start time and process tree

ps -o pid,ppid,lstart,etime,cmd -p <PID> shows the exact start time and elapsed runtime of the affected process. A process that has been running for days and has only shown high CPU usage for a few minutes likely underwent an internal state change, for example an infinite loop triggered after reaching a particular record. A process that was started exactly when the problem began, on the other hand, points directly to its caller.

pstree -p <PID> shows the process tree with all parent and child processes, revealing whether the suspicious process was created by cron, a supervisor such as systemd, or a web server instance. For PHP-FPM workers, the parent is always the FPM master, so instead the FPM status endpoint (pm.status_path) is more useful here to determine the currently processed request, revealing the triggering URL or script.

Additionally, systemctl status <service> for systemd-managed services provides the time of the last start along with the number of restarts so far. A service that has restarted unexpectedly several times since the last deploy points to an already known but still unresolved problem that may be related to the current CPU usage.

4. strace: capturing syscalls live

strace is the central tool for understanding mysterious CPU usage at the system call level. strace -c -p <PID> collects all syscalls of the target process over a period of time and summarizes them at the end by frequency and time consumed. If the summary shows predominantly futex calls, that points to lock contention between threads. If read or recvfrom calls dominate in a tight loop instead, a busy loop waiting for data is likely, rather than a blocking wait call.

# Summarize syscalls over 5 seconds for a suspicious PID
strace -c -p 28855
# ^C (after ~5 seconds)
# % time     seconds  usecs/call     calls    syscall
# ------ ----------- ----------- --------- ------------------
#  94.20    2.981022         120     24801 futex
#   3.10    0.098213           8     12277 read
#   1.80    0.056904          45      1264 mmap

# Heavy futex activity suggests lock contention between threads

This example shows 94 percent of the time spent in futex calls, a strong indicator of contending locking, for example caused by a poorly implemented cache with overly granular locks or a library that synchronizes internally more than necessary. Without strace, this relationship would never have become visible from a plain CPU percentage display, because top only shows the time consumed, not its composition.

For an even finer resolution, strace -T -tt -p <PID> is well suited, logging every single syscall with an exact timestamp and individual duration instead of just an aggregated summary. This makes it possible to identify individual, unusually long calls within an otherwise unremarkable sequence, which is particularly valuable for sporadic, hard-to-reproduce CPU usage.

5. perf top: seeing the hottest functions live

While strace operates at the syscall level, perf top -p <PID> shows the actual hottest functions in user and kernel space live, based on sampling the instruction pointer register via hardware performance counters. This makes it possible, for compiled code or even for PHP with opcache enabled, to identify the specific function causing the CPU usage, instead of only knowing that something in this process is computing heavily.

For interpreted languages like PHP, perf top primarily shows functions of the Zend Engine itself, such as execute_ex or regex routines from PCRE, which does not directly reveal the PHP code line but already provides strong clues. Combined with debug symbols (perf top --call-graph dwarf), the call path can additionally be reconstructed, often narrowing down the triggering PHP function, particularly for compute-heavy operations like complex regular expressions or image processing.

For a persistent recording instead of a live view, perf record -p <PID> -g -- sleep 10 followed by perf report is well suited. This approach captures exactly ten seconds of activity into a file that can be analyzed later at leisure, which is considerably more practical than continuous live observation with perf top for sporadically occurring CPU usage.

6. PHP-FPM special case: Xdebug, infinite loops, regex

In PHP-heavy environments like Magento installations, there are recurring patterns of mysterious CPU usage. An xdebug module accidentally left enabled in production with its profiler active can massively slow down every request and produce CPU load that was never noticed in the development environment because fewer parallel requests ran there. php -m | grep xdebug and a look into php.ini are therefore among the first checks for a sudden spike in PHP-FPM load.

Another often-forgotten check is the PHP opcache configuration itself: if opcache.validate_timestamps is enabled, PHP checks the timestamp of every included file on each request, which on very large codebases with thousands of files can produce noticeable additional CPU usage. In production environments this option should be disabled, with the opcache instead cleared explicitly during deploys.

A second common pattern is catastrophic backtracking in regular expressions, so-called ReDoS. A seemingly harmless regex with nested quantifiers can produce exponentially long runtimes for certain input strings, which shows up as a sudden spike in CPU usage on exactly one worker as soon as a user sends a matching input. strace typically shows very few syscalls combined with high CPU load in this case, a strong indicator of pure compute loops without IO.

A third, less commonly discussed case is inefficient autoloading or repeated re-parsing of large configuration files on every single request, when an opcache entry is constantly evicted due to a faulty invalidation rule. The resulting CPU usage spreads evenly across all workers here, instead of concentrating on a single one as with ReDoS, which makes these two causes readily distinguishable by how the load is distributed across workers.

7. Phantom processes from cron and zombie children

An often overlooked case of mysterious CPU usage arises when a cron job is started multiple times in parallel due to a bug in the execution logic, because the previous run had not yet finished and no locking mechanism prevents this. ps aux | grep <scriptname> | wc -l quickly shows whether, instead of an expected single run, ten or twenty parallel instances of the same script are suddenly active and jointly saturating the CPU.

A related pattern: a parent process spawns child processes for subtasks but, due to a bug, does not correctly wait for their termination, causing zombie or orphaned processes to accumulate. While true zombies do not consume CPU themselves, the faulty parent process may poll for them in a loop and thereby produce unnecessary CPU load itself. A look with ps --ppid <PID> at all children of a suspect reliably reveals such patterns.

A systemd timer that accidentally triggers the same job in parallel with a classic cron entry is another common source of doubled CPU usage after a migration from cron to systemd timers. systemctl list-timers together with crontab -l on the same system quickly reveals such a duplicate configuration, before it has to be tracked down through elaborate process analysis.

8. Immediate measures: cgroups, nice, cpulimit

Before the actual root cause is fully understood, a production system often needs immediate relief. renice with a higher nice value reduces the priority of a suspicious process relative to others without terminating it, which for non-critical batch processes immediately softens the impact on the rest of the server. cpulimit -p <PID> -l 50 caps a process to a fixed percentage of one core, ideal for a single process that has run out of control while the actual cause is still being investigated.

For PHP-FPM workers, a targeted restart of the affected worker is often the fastest immediate measure, without restarting the entire pool. kill -QUIT <worker-PID> gracefully terminates a single FPM worker, the master process automatically spawns a replacement worker, which limits the impact of the CPU usage to a brief moment instead of interrupting the entire service for all users.

For more lasting containment, cgroups are well suited: systemctl set-property <service>.service CPUQuota=50% caps a systemd-managed service after the fact, without changing the application itself. These measures do not fix the cause of the CPU usage, but they prevent a single faulty process from affecting the entire system while the diagnosis from the previous sections continues in parallel.

A documented escalation threshold, for example a fixed nice value or a CPU quota percentage automatically applied to any suspicious process via a monitoring script, considerably shortens response time in an emergency. This way nobody on the team has to think under time pressure about which command to type, but instead follows an already-tested procedure for handling unexplained CPU usage.

9. Tools compared: when to use what

The following overview maps the tools presented to their area of use, so that in an emergency the right tool for the given line of suspicion is chosen immediately.

Combining these tools turns a vague observation like a process is at 100 percent into a concrete, actionable diagnosis. In practice, a single tool is rarely enough: top -H locates the thread, strace reveals the type of activity, and perf pins down the actual code path, while cpulimit buys the time needed to run this analysis in the first place.

10. Summary

Tracking down mysterious CPU usage rarely succeeds through a single observation, but through a chain of tools building on one another. top -H narrows the suspicion to a single thread, ps and pstree clarify origin and timing, strace -c reveals the type of activity at the syscall level, and perf top delivers the actual code path. PHP-specific pitfalls such as leftover Xdebug and regex backtracking are among the most common causes in PHP-heavy environments.

Anyone who has documented this diagnostic path once does not need to reinvent it at the next incident, but can start directly with thread analysis instead of wasting valuable time randomly trying out different tools.

While the actual diagnosis runs, cpulimit or cgroup-based CPU quotas prevent a single faulty process from affecting the entire server. Anyone who masters this toolbox once turns vague alerts into concrete, understandable causes that can be fixed permanently, instead of repeatedly restarting the affected process.

11. FAQ: Tracking down mysterious CPU usage