strace: Debugging a Process's System Calls Live

1. What strace actually exposes

strace hooks into a process through the kernel's ptrace interface and logs every system call the process makes, together with the arguments passed, the return value, and, if the call fails, the matching errno. That surfaces a layer that is normally invisible to the application itself: every file that gets opened, every socket that gets connected, every signal that arrives.

In a Magento hosting context this is especially valuable, because a large share of real world problems, hanging PHP-FPM workers, denied file access due to wrong permissions, blocking network connections to Redis or Elasticsearch, manifest exactly at the syscall level, long before they show up in an application log, if they ever show up there at all.

2. Attaching to an already running process

The most common production use case is not launching a brand new process under strace, but attaching to one that is already running, using the -p flag with its PID. That matters when a PHP-FPM worker has already been hanging for minutes and restarting the entire pool would only hide the problem instead of diagnosing it.

When attaching, strace briefly stops the process to hook into it, then the process continues running normally, just observed. To follow several workers at once, combine -f with a process group so child processes are traced automatically as well, which is almost always necessary given the master worker model of PHP-FPM.

# Find the PID of the hanging PHP-FPM worker
ps aux | grep php-fpm

# Attach to the running process, write timestamped output to a file
strace -p 28417 -tt -o /var/log/strace-fpm-worker.log

# Follow multiple threads/child processes of the same process at once
strace -f -p 28417 -tt

3. Case study: diagnosing a hanging PHP-FPM worker

A classic symptom: a FPM worker keeps accumulating CPU time in ps but no longer responds to new requests. strace almost always reveals the last syscall the process is blocked in, such as a read on a socket that never returns a response, or a flock waiting on a lock already held elsewhere.

If no output appears at all after attaching, with no new syscall being logged, the process was already stuck in that exact state before strace even started. That alone is a valuable diagnosis: the process is not waiting for CPU time, it is blocked inside a single system call, usually network I/O or a filesystem locking mechanism.

# Attach and watch whether any new syscalls show up at all
strace -p 28417 -tt

# Typical output for a blocking Redis call:
# 14:22:01.120441 read(11, ...) = ... (never returns)
#
# If no further syscall is logged after several seconds,
# the process is stuck exactly inside that read() call

4. Launching a new process directly under strace

Besides attaching to running processes, strace can also be placed directly in front of a command to observe a new process from the very start. That works especially well for short lived command line scripts, cron jobs, or Magento CLI calls where the failure is reproducible on demand.

For PHP CLI scripts that fail inside a cron job, this approach often provides the full explanation: a missing directory, a wrong environment variable, or an unreachable configuration file shows up immediately as a failed openat call returning ENOENT.

# Launch a new process directly under strace
strace -f -o /tmp/cron-debug.log php bin/magento indexer:reindex

# Only show file related syscalls to find missing paths
strace -f -e trace=open,openat,stat,access php bin/magento cache:flush

5. Tracking down failing file access precisely

Permission problems are among the most common yet hardest to pin down errors in grown Magento installations. Instead of manually checking directory after directory, filter with -e trace= for file related syscalls and read directly from the output which path fails with which error code.

Particularly useful is combining -e trace=file, which groups all file related calls such as openat, stat, unlink, and chmod, with -Z, which shows only failing calls. That removes the noise of successful calls entirely and leaves only the actual failures visible.

# Show only failing, file related system calls
strace -f -e trace=file -Z -p 28417

# Example output for a missing write permission:
# openat(AT_FDCWD, "/var/www/html/var/cache/lock", O_CREAT|O_WRONLY) = -1 EACCES (Permission denied)

6. Filtering by syscall categories for targeted analysis

Beyond plain file filtering, strace ships predefined syscall groups such as network, signal, ipc, and process, selectable through -e trace=. For network problems with external services like Elasticsearch or Redis, -e trace=network shows only connection setup, data transfer, and teardown, without the volume of file syscalls a PHP process also generates in parallel.

The output can additionally be turned into a statistical summary with -c, listing frequency and cumulative time per syscall type. That is the fastest way to determine whether a process spends its time mostly on file I/O, network I/O, or memory management, before digging into individual calls.

# Trace only network syscalls, e.g. for Redis/Elasticsearch connections
strace -f -e trace=network -p 28417

# Statistical summary over a 5 second window
timeout 5 strace -c -p 28417

7. Performance overhead and use in production

strace is not a tool for continuous use: since every system call stops the observed process, hands it off to strace, then resumes it, the overhead can slow the process down by a factor of two to several hundred depending on the syscall rate. For a single hanging worker during an incident that is usually acceptable, for an entire production FPM pool under load it is not.

In practice this means using strace on individual, already suspicious processes, keeping the observation window as short as possible, and bounding it with timeout when in doubt. For broader, continuous observation of an entire system, strace is fundamentally the wrong tool, kernel native tracing mechanisms are a much better fit.

# Deliberately bound the observation window to minimize overhead
timeout 10 strace -p 28417 -tt -o /tmp/quickcheck.log

# Roughly estimate syscalls per second to gauge the overhead
strace -c -f -p 28417 &
sleep 5
kill %1

8. When ltrace or perf are the better choice

strace exclusively shows kernel boundary crossings, meaning system calls. If the suspicion lies with a library function instead, say a particular OpenSSL or libc routine that behaves oddly but does not trigger its own syscall, ltrace provides the more relevant view, since it logs calls into dynamically linked library functions rather than kernel calls.

For performance questions beyond a single blocking call, such as why a process overall consumes a lot of CPU time, perf is the far more suitable tool, because it relies on statistical sampling and therefore causes a fraction of the overhead strace does. strace answers 'what is this one process stuck on right now', perf answers more the question of 'where does this process spend its time overall'.

For deeper, system wide tracing without the overhead of ptrace, eBPF based tools such as bpftrace are also worth considering, since they evaluate syscall events directly inside the kernel instead of handing every single call to a userspace program through a context switch.

9. Best practices for production use

Always work with as narrow a syscall filter as possible instead of logging everything indiscriminately: that reduces both the overhead and the amount of data you need to sift through afterwards. Timestamps via -tt are mandatory once timing matters, for instance to determine exactly how long a particular call actually blocked.

Always redirect output to a file with -o instead of reading it live on the terminal, especially with -f and many child processes, since the output volume otherwise becomes unmanageable quickly. After diagnosing the issue, strace must be switched off again consistently, a forgotten background strace process remains an unnecessary, ongoing performance overhead on the server.

10. Summary

11. FAQ: strace