Analyzing Core Dumps and Segfaults with gdb

1. What a segfault and a core dump really are

A segfault (segmentation fault) occurs when a process accesses memory it is not authorized to, for example an already freed memory region, a null pointer, or an address outside its assigned virtual address space. The kernel detects this invalid access via the CPU's memory management unit and terminates the process with the SIGSEGV signal. Unlike a PHP exception or a cleanly handled error, a segfault is always a sign of a bug at a deeper level, usually in native C code, an extension, or rarely the kernel itself, since plain PHP normally cannot cause direct memory access errors of this kind.

A core dump is a snapshot of the entire process memory at the moment of the crash, including all registers, the stack, and the heap. Without this core dump, only the error message itself remains, which provides no usable information about the cause beyond the timestamp and the process name involved. With a correctly preserved core dump, however, it is possible to reconstruct exactly which function with which arguments was active when the memory access failed, which is the decisive difference between days of guessing and a diagnosis within minutes.

2. Enabling core dumps: ulimit, core_pattern and systemd

The most common reason no core dump exists after a segfault is ulimit -c set to 0. Many distributions disable core dumps by default to prevent accidentally filling the disk with large memory snapshots. For a service started via systemd, such as PHP-FPM, setting ulimit -c unlimited in an interactive shell is not enough; here LimitCORE=infinity must be set in the respective unit file, analogous to the already familiar LimitNOFILE pattern for file descriptor limits.

In addition, /proc/sys/kernel/core_pattern determines where the core dump is written. On modern systemd systems, this value typically points to a piping command to systemd-coredump, which stores the dumps centrally in /var/lib/systemd/coredump/, compressed and annotated with metadata. A manually set, deviating core_pattern value, for example a simple relative path like core, often causes dumps to end up in a service's current working directory, where they never get created in the first place due to missing write permissions, which is incorrectly interpreted as "there are no core dumps".

# Check current core dump size limit for the shell
ulimit -c

# Where does the kernel actually write core dumps?
cat /proc/sys/kernel/core_pattern

# Typical systemd-coredump integration (Debian/Ubuntu default)
# |/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e

# Enable unlimited core dumps for a systemd-managed service
sudo mkdir -p /etc/systemd/system/php8.4-fpm.service.d
cat <<'EOF' | sudo tee /etc/systemd/system/php8.4-fpm.service.d/coredump.conf
[Service]
LimitCORE=infinity
EOF
sudo systemctl daemon-reload
sudo systemctl restart php8.4-fpm

3. coredumpctl: finding and managing core dumps

coredumpctl is the central tool on systemd systems for finding existing core dumps without manually searching the file system. coredumpctl list shows a chronological overview of all recorded crashes with process name, PID, signal, and timestamp, while coredumpctl info php-fpm provides details on the most recent crashes of a specific program, including the command line and, partly, an already rudimentary backtrace. This saves the first step of manually searching for the right file in /var/lib/systemd/coredump/, whose filenames would otherwise have to be tediously searched by PID and timestamp.

Particularly practical is the direct integration with gdb via coredumpctl debug php-fpm, which automatically loads the matching core dump and starts a gdb session with the correct binary path set. This combination eliminates a common mistake in manual gdb analysis: loading a core dump against the wrong or a meanwhile updated program version, which leads to misleading or incomplete backtraces.

# List all recorded crashes, newest first
coredumpctl list

# Show details for the most recent crash of a specific binary
coredumpctl info php-fpm8.4

# Filter by PID if multiple crashes of the same binary exist
coredumpctl list php-fpm8.4 | tail -5

# Jump straight into a gdb session with the correct core and binary loaded
coredumpctl debug php-fpm8.4

4. Creating and reading a backtrace with gdb

Inside a gdb session, the bt (backtrace) command provides the call chain from the crash point back to the program start, with each line showing a so-called stack frame with function name, arguments, and, when debug symbols are available, file name and line number. The topmost frame (#0) shows exactly the function in which the invalid memory access took place, while the frames below show the calling context, i.e., which higher-level function reached this code path in the first place. For deeper analysis, bt full additionally provides all local variables of each frame, which often gives the decisive clue, for example a recognizably invalid pointer value like 0x0 or an obviously wrong array size.

The info registers command shows the state of all CPU registers at the moment of the crash, which is particularly helpful for analysis at the assembly level when the error occurs in a library function with no available source code. With frame N you switch specifically to a particular stack frame to inspect the value of a specific variable at the moment of the crash with print variable_name. This combination of backtrace, local variables, and register state covers the vast majority of all crash causes without having to reproduce the error first.

# Start gdb against a binary and its core dump
gdb /usr/sbin/php-fpm8.4 /var/lib/systemd/coredump/core.php-fpm8.4.0.abc123.12345.1234567890000.zst

# Inside the gdb prompt:
(gdb) bt
#0  0x00007f8a1c2b3456 in zend_hash_find () from /usr/lib/php/8.4/extensions/opcache.so
#1  0x00007f8a1c2a1234 in zend_execute_ex () from /usr/lib/x86_64-linux-gnu/libphp8.4.so
#2  0x00007f8a1c1a5678 in execute_ex () from /usr/lib/x86_64-linux-gnu/libphp8.4.so
#3  0x0000560a1b2c3456 in main ()

(gdb) bt full        # backtrace including all local variables per frame
(gdb) frame 1         # switch to a specific stack frame
(gdb) print *some_ptr  # inspect the dereferenced value of a suspect pointer
(gdb) info registers   # CPU register state at the moment of the crash

5. Debug symbols: why a backtrace without them is worthless

A backtrace without debug symbols shows only memory addresses and, at best, the name of the library the address is located in, but neither function names nor line numbers. On Debian-based systems, the dbgsym repository provides the matching debug symbol packages, recognizable by the -dbgsym suffix, for example php8.4-fpm-dbgsym. On RHEL-based systems, debuginfo-install handles the same task. Without these packages, a backtrace remains reduced, in the worst case, to plain hexadecimal addresses, from which no usable statement about the cause can be derived without extensive manual disassembly.

A common pitfall: debug symbols must match the installed binary version exactly. An update of PHP or an extension without a matching update of the debug symbol packages leads to a backtrace that shows function names but wrong or no line numbers at all, because the addresses have shifted slightly between versions. Anyone who regularly analyzes core dumps should therefore include debug symbol packages in the same update pipeline as the actual binaries, rather than installing them afterward as needed.

# Debian/Ubuntu: enable the debug symbol repository once
echo "deb http://deb.debian.org/debian-debug bookworm-debug main" | \
  sudo tee /etc/apt/sources.list.d/debug.list
sudo apt-get update

# Install debug symbols matching the exact installed package version
sudo apt-get install php8.4-fpm-dbgsym

# RHEL/CentOS/Alma equivalent
sudo debuginfo-install php-fpm

# Verify gdb now resolves symbols instead of showing raw addresses
gdb -batch -ex "bt" /usr/sbin/php-fpm8.4 core.dump

6. Systematically narrowing down segfaults in PHP extensions

In PHP environments such as Magento, faulty C extensions are by far the most common cause of segfaults, because PHP itself, as an interpreted language, guards memory access, but native extensions can bypass this safeguard. A backtrace showing a function within a .so file such as opcache.so, redis.so, or an Xdebug library as the topmost frame points directly to the responsible extension. The combination of PHP version, extension version, and operating system architecture is decisive here: many such segfaults are known, already reported compatibility issues between a specific PHP minor version and an extension developed in an older release cycle.

A systematic approach for narrowing this down is progressively disabling individual extensions via php.ini, combined with observing whether the segfault reproducibly disappears. For OPcache-related crashes, disabling opcache.jit additionally helps, since the PHP JIT compiler repeatedly failed on edge cases in early 8.x versions, while classic, interpreted execution remained stable. This targeted isolation by extension is often faster than a full gdb analysis and should run in parallel with the backtrace investigation.

; Systematic extension isolation — disable one at a time and observe
; /etc/php/8.4/fpm/conf.d/99-debug-isolation.ini
; extension=redis.so       ; commented out during isolation testing
; zend_extension=xdebug.so ; commented out during isolation testing

; If OPcache/JIT is suspected, disable JIT while keeping the cache itself
opcache.jit=disable
opcache.jit_buffer_size=0

7. Safely handling core dumps in production environments

Leaving core dumps active carries two practical risks that must be considered before production use. First, core dumps can contain sensitive data, such as database passwords or session tokens that resided in process memory at the moment of the crash, which is why access permissions on /var/lib/systemd/coredump/ should be kept restrictive. Second, a service that crashes repeatedly and in quick succession can, through the size of the core dumps themselves, lead to a full disk, especially with applications with high memory usage whose core dumps are correspondingly large.

systemd-coredump limits the size and number of stored dumps via Storage=, Compress=, and MaxUse= in /etc/systemd/coredump.conf, which pragmatically addresses both risks: compression significantly reduces storage needs, while MaxUse prevents a series of crashes from completely consuming available disk space. For production Magento servers, it is recommended to leave core dumps enabled by default but with clear upper limits, rather than disabling them preemptively and standing there with no diagnostic options at all at the next incident.

# /etc/systemd/coredump.conf
[Coredump]
Storage=external
Compress=yes
ProcessSizeMax=2G
ExternalSizeMax=2G
MaxUse=10G
KeepFree=5G

8. Forcing reproduction instead of waiting for the next crash

If a segfault occurs too rarely to wait for another incident, it helps to specifically reproduce the suspected trigger identified from the backtrace. If, for example, a specific Redis operation was identified as the trigger, a minimal PHP script can be written that isolates exactly this operation and executes it repeatedly under load, ideally under gdb --args php script.php, so that a renewed crash immediately lands in a running gdb session instead of having to analyze a core dump afterward.

For hard-to-reproduce cases with suspected race conditions between threads or processes, gdb with catch signal SIGSEGV and automated bt on every occurrence provides a way to collect multiple crashes over time and check them for commonalities in the backtrace. If several independent crashes agree in the topmost frame, that is strong evidence that an actually reproducible bug is present, rather than hardware failures or cosmic radiation, as is occasionally wrongly suspected in rare cases.

9. Crash analysis tools compared

Depending on the situation and available preparation, different tools are suited for analyzing a core dump. The overview below arranges the approaches presented by prerequisite and informational value.

In practice, investigating a segfault always starts with coredumpctl list to confirm that a usable dump even exists, followed by installing matching debug symbols, without which every further step loses informational value. Only after that does gdb bt full provide the decisive details, and a targeted reproduction confirms the suspected cause before a fix is considered production-ready.

10. Summary

A segfault is not a puzzle with no solution, but a technically precisely analyzable event, provided a core dump was correctly enabled and preserved. The foundation consists of ulimit -c or LimitCORE in the systemd unit, along with a sensibly configured core_pattern, usually via systemd-coredump. coredumpctl greatly simplifies finding existing dumps, while gdb with installed debug symbols provides a complete backtrace down to the concrete line of code.

In PHP environments, faulty native extensions are the most common cause of segfaults, recognizable by the topmost frame in the backtrace being within a .so file. Systematically disabling individual extensions and targeted reproduction of the suspected trigger confirm the cause before a fix is considered complete. Anyone who leaves core dumps enabled by default but limits size and count via systemd-coredump has the necessary data for a fast diagnosis immediately at hand for every future crash.

11. FAQ: Analyzing Core Dumps and Segfaults with gdb