Analyzing Kernel Panics and Crash Dumps with kdump

1. What a kernel panic is and why crash dumps matter

A kernel panic is the moment the Linux kernel reaches a state it cannot recover from safely, for example an invalid memory reference in kernel space or a violated internal invariant. Unlike a crashing userspace process, the kernel cannot simply handle this failure with a process kill, because it is itself the instance responsible for memory management and scheduling. The consequence is an immediate, controlled halt or restart of the entire system, often without any warning to running services.

Without a crash dump, all that remains after a kernel panic is a single line in the serial log or a black screen, from which no cause can be derived. That is exactly where kdump comes in: it catches the crash, saves the complete memory contents into a file called vmcore, and enables a forensic analysis afterwards. For production servers running databases, PHP-FPM pools or virtualized workloads, a configured crash dump mechanism is the difference between guesswork and a solid root cause analysis.

2. Understanding kdump: kexec, reserved memory and the crash kernel

kdump relies on two building blocks that together enable a clean crash workflow. The first is kexec, a kernel feature that boots a new kernel directly from a running kernel, without going through the full firmware and bootloader path via BIOS or UEFI. The second is a reserved memory region that is set aside at boot time through a kernel parameter and stays invisible during regular operation.

When a kernel panic occurs, kexec immediately boots a minimal crash kernel into this reserved memory region. This crash kernel has access to the entire memory content of the crashed system and writes it to disk or over the network as a vmcore file before the server reboots normally. The decisive advantage over a classic reboot through firmware: the memory state at the exact moment of the kernel panic remains fully preserved, including every process stack, lock and kernel data structure.

3. Installing and configuring kdump

Setting up kdump requires two related steps: reserving memory through a kernel parameter and configuring the kdump service. The kernel parameter crashkernel determines how much RAM is set aside for the crash kernel, typically between 128 and 512 megabytes depending on server memory and loaded modules. Without enough reserved memory, the crash kernel itself cannot start cleanly, which renders the entire mechanism useless.

# Debian/Ubuntu: install kdump tools
sudo apt install linux-crashdump kdump-tools

# RHEL/Rocky family: install kdump
sudo dnf install kexec-tools

# Reserve memory for the crash kernel via GRUB kernel parameter
sudo sed -i 's/GRUB_CMDLINE_LINUX="/GRUB_CMDLINE_LINUX="crashkernel=256M /' \
  /etc/default/grub
sudo update-grub
sudo systemctl reboot

# Verify the crash kernel memory was actually reserved
cat /proc/cmdline | grep crashkernel
grep -i crashk /var/log/dmesg

# Configure the vmcore storage location (Debian/Ubuntu)
sudo sed -i 's|^KDUMP_KERNEL_PARAMS.*|KDUMP_KERNEL_PARAMS=""|' \
  /etc/default/kdump-tools
echo 'path /var/crash' | sudo tee -a /etc/kdump-tools.conf

# Enable and start the kdump service
sudo systemctl enable --now kdump-tools    # Debian/Ubuntu
sudo systemctl enable --now kdump          # RHEL family
sudo kdumpctl status                       # RHEL family status check

After installation, cat /sys/kernel/kexec_crash_loaded returning 1 indicates that a crash kernel has been successfully loaded and stands ready in an emergency. If this value never becomes 1, kdump will not work when it matters, no matter how carefully the rest of the configuration was done. This check belongs in the baseline setup of every server where a kernel panic is expected to be answered with a solid diagnosis.

4. Deliberately triggering and testing a kernel panic

A kdump configuration that has never been tested is not a working safety net, it is an assumption. The Linux kernel provides SysRq, a built in mechanism to deliberately trigger a kernel panic without having to wait for a real hardware or software fault. This test should only run on a staging system or outside business hours, since it actually crashes the server.

# Enable the SysRq trigger mechanism (root required)
echo 1 | sudo tee /proc/sys/kernel/sysrq

# Deliberately crash the kernel on a test system to validate kdump
echo c | sudo tee /proc/sysrq-trigger

# After reboot, confirm the crash kernel captured a vmcore
ls -lh /var/crash/*/vmcore

# If no vmcore appears, check the kdump service logs for the failure
journalctl -u kdump-tools --no-pager | tail -50   # Debian/Ubuntu
journalctl -u kdump --no-pager | tail -50         # RHEL family

A successful test confirms two things at once: that the reserved memory is sufficient, and that the crash kernel actually boots under real conditions. If the vmcore file is still missing afterwards, the problem is usually either insufficient reserved crashkernel memory or missing storage drivers in the crash kernel, a common issue on unusual RAID or NVMe configurations.

5. Analyzing the vmcore file with crash

The vmcore file alone is a raw binary dump and not readable on its own. The crash utility combines this dump with the debug symbols of the exact matching kernel and turns it into an interactive analysis environment strongly reminiscent of gdb. This makes it possible to inspect kernel data structures, process lists and memory regions at the moment of the kernel panic directly, instead of relying on plain log lines.

# Install the crash utility and matching debug symbols
sudo apt install crash linux-image-$(uname -r)-dbgsym   # Debian/Ubuntu
sudo dnf install crash kernel-debuginfo                 # RHEL family

# Open the vmcore with the matching vmlinux debug image
sudo crash /usr/lib/debug/boot/vmlinux-$(uname -r) \
           /var/crash/202607301400/vmcore

# Inside the crash shell, common first commands:
crash> sys                 # basic system information at crash time
crash> log                 # kernel ring buffer as captured in the dump
crash> bt                  # backtrace of the crashing CPU/task
crash> ps | head -20       # process list at the moment of the panic

The log command inside the crash session often provides the decisive clue, because it shows the kernel ring buffer exactly as it looked at the moment of the crash, including the last message before the kernel panic. Combined with bt for the backtrace, it is usually possible within minutes to narrow down whether a driver, a filesystem or a hardware fault was the cause.

6. Backtrace and process state at the moment of the crash

The backtrace shows the chain of kernel function calls that were active immediately before the kernel panic, from the triggering function back to the system call that originally set it off. If a driver name shows up in that chain, for example a network or storage module, that is a strong hint towards a faulty third party component rather than a generic kernel bug.

In addition to the backtrace, crash offers commands such as foreach bt to produce backtraces for every running process at once, and files to view a specific process's open file descriptors at the moment of the crash. For a kernel panic that occurred alongside high system load, this combination often shows which process was holding how much memory or how many file descriptors right before the kernel gave up.

7. Automated evaluation and alerting

In production environments, a kernel panic should not first be noticed because a customer complains. A simple script that checks at boot time whether a new vmcore file exists can automatically trigger a notification and extract the most important crash metadata before anyone manually enters the crash shell.

#!/usr/bin/env bash
# check-new-crash.sh — run via a systemd unit at boot
set -euo pipefail

CRASH_DIR="/var/crash"
STATE_FILE="/var/lib/kdump-alert/last-seen"
mkdir -p "$(dirname "$STATE_FILE")"

latest=$(find "$CRASH_DIR" -maxdepth 1 -type d -name '2*' | sort | tail -1)
[[ -z "$latest" ]] && exit 0

last_seen=$(cat "$STATE_FILE" 2>/dev/null || echo "")
if [[ "$latest" != "$last_seen" ]]; then
  summary=$(crash -s "$latest/vmcore" 2>/dev/null <<< "sys; bt" || true)
  echo "New kernel panic detected: $latest" | \
    mail -s "[ALERT] Kernel panic on $(hostname)" ops@example.com
  echo "$summary" >> "$latest/summary.txt"
  echo "$latest" > "$STATE_FILE"
fi

This kind of automation ensures that every kernel panic is documented and reported, regardless of whether an administrator happens to be awake or not. Combined with a monitoring system, the check can also be wired in as its own probe that raises a high priority alert for every new crash directory.

8. Common causes of kernel panics in production

In practice, most kernel panics trace back to a handful of recurring causes. Faulty or incompatible kernel modules, often after a kernel version upgrade without a prior compatibility check, top the list. Memory hardware defects also frequently show up first as a sporadic kernel panic, before they are noticed through other symptoms.

# Search past crash summaries for the responsible module or function
grep -A5 '^PID:' /var/crash/*/summary.txt

# Cross-check crash timestamps against recent module or driver changes
for d in /var/crash/*/; do
  echo "=== $d ==="
  crash -s "$d/vmcore" <<< "sys" 2>/dev/null | grep -E 'PANIC|DATE'
done

# Rule out memory errors as a root cause on suspicious hardware
sudo apt install memtester
sudo memtester 1024 1     # test 1 GB of RAM for one pass, offline test preferred

Another frequent trigger is an out of memory situation combined with a misconfigured swap or cgroup setup, where the kernel itself ends up in an inconsistent state instead of only the OOM killer terminating a single process. Firmware bugs in storage controllers and outdated network drivers also show up disproportionately often in the backtraces of kernel panics, especially on older server platforms without current firmware updates.

9. kdump compared to other diagnostic methods

kdump is not the only method for investigating system crashes, but it is the only one that preserves the complete memory state at the moment of the kernel panic. Other tools provide complementary but less complete information.

In practice, these methods complement each other. A kernel panic is first noticed through dmesg or the serial console, then kdump provides the full forensic foundation, and hardware logs finally rule out or confirm a physical defect as the cause. Anyone relying on only one of these sources risks an incomplete or wrong diagnosis.

10. Summary

A kernel panic without a crash dump remains an unsolved mystery that can recur at any time. kdump solves this problem by immediately starting a minimal crash kernel via kexec at the moment of the crash, saving the complete memory contents as vmcore, and thereby enabling a real forensic analysis. The crash utility makes this dump readable and delivers backtrace, process list and kernel log exactly as they stood at the moment of the crash.

It matters to set up kdump not only after the first kernel panic, but preventively on every production server, including a real test via SysRq on a staging system. Automated alerting on new vmcore files and a systematic look at recurring causes such as faulty drivers or hardware defects turn a one off incident into a repeatable, well documented diagnostic process.

11. FAQ: Kernel Panics and Crash Dumps