Emergency Runbook for an Unresponsive Server

1. First classification: network, service, or kernel?

Before restarting an unresponsive server in a panic, a quick classification of the problem is worthwhile, because the correct reaction differs completely depending on the cause. A server can fail to respond for three fundamentally different reasons: a pure network problem, where the server itself is healthy but unreachable; a hanging application service with an otherwise functioning operating system; or a genuinely hung kernel that no longer responds to any input at all.

This distinction determines whether a gentle or a hard intervention is appropriate. An unresponsive server with a healthy kernel but a hanging SSH daemon can often be saved via alternative access paths without losing data. A true kernel hang, on the other hand, eventually requires a reset, but even then it is worth trying to preserve as much diagnostic data as possible beforehand.

In practice it helps to establish this classification as a deliberate first question within the team, instead of immediately falling into hectic individual actions. A brief moment of classification often saves more time than it costs with an unresponsive server, because it rules out unnecessary escalation steps from the start.

This classification should ideally not be made by a single person alone, but briefly coordinated within the team, to avoid blind spots.

2. Step 1: check reachability from the outside

The first step with any unresponsive server is to test basic network reachability from a second, independent location, to rule out a purely local problem with your own workstation. ping against the server IP and a traceroute from a different network, for example via a mobile hotspot, quickly show whether the problem actually lies with the server or only with your own internet connection or a local corporate network outage.

If the server responds to ICMP but SSH on port 22 yields no connection, that points to a hanging or crashed SSH daemon while the kernel itself is still functioning. If the server does not respond to ICMP either, but the upstream cloud provider shows the instance as running, either the kernel's network stack is blocked or there is genuinely a deeper kernel problem, which justifies moving on to the next step of this runbook for the unresponsive server.

A look at the cloud provider dashboard or the hypervisor console additionally provides metrics like CPU utilization and network throughput for the last few minutes, independent of the guest system itself. If this outside view shows CPU load stuck at zero, a kernel hang is more likely than a pure application problem with the unresponsive server.

These metrics should ideally already be known before the incident, so a comparison with the normal state is possible. A team that knows a server's typical baseline recognizes a deviation with an unresponsive server considerably faster than one seeing the values for the first time during the incident.

3. Step 2: alternative access paths besides standard SSH

Before an unresponsive server is considered completely lost, it is worth trying alternative access paths. A second SSH attempt with an explicit timeout, for example ssh -o ConnectTimeout=5 user@server, distinguishes a hanging connection setup from an immediate rejection. If SSH does not respond at all but the server is reachable via ping, a look at other open services can help: if, for example, a monitoring agent or another TCP port still responds, the kernel is likely functional and only the SSH daemon itself is hanging or overloaded.

# Step 2: distinguish a hanging connection from an outright refusal
ssh -o ConnectTimeout=5 -o BatchMode=yes user@server "echo alive"
# ssh: connect to host server port 22: Connection timed out
# — server does not respond on port 22 at all, different from "Connection refused"

# Check if any other port is still responsive (e.g. monitoring agent)
nc -zv -w 3 server 9100

A particularly valuable, often forgotten access path is an already existing but idle SSH session on the same server. If an old terminal session unexpectedly remains reachable even though new connections fail, it may still allow enough diagnosis to identify the cause before the next, more invasive step of this runbook for the unresponsive server becomes necessary.

In such a remaining session, a quick look with uptime and free -h is worthwhile first, to capture Load Average and memory usage before the session possibly freezes itself. These few seconds often deliver the decisive data points for the later root cause analysis of the unresponsive server.

A second, parallel SSH login attempt from another team member can also be worthwhile, because some connection problems only affect individual client IPs, for example due to an overly aggressive fail2ban rule set that wrongly blocks a legitimate IP after several failed attempts instead of actually indicating an unresponsive server.

4. Step 3: out-of-band access via IPMI or KVM

If all network-based access paths fail, out-of-band management is the next step for an unresponsive server. Dedicated servers usually have IPMI or iDRAC/iLO with their own network interface that runs independently of the main operating system and keeps responding even when the kernel is completely hung. Via a serial console (ipmitool -I lanplus -H bmc-ip -U user sol activate) the screen contents can be read live, which often immediately shows the cause, for example a kernel panic message with a stack trace that would never have become visible over SSH.

On cloud instances, the provider's own serial console (for example the AWS EC2 Serial Console or the Hetzner Cloud Console feature) replaces classic IPMI and delivers the same value: a look at the server's screen, independent of the network stack. This step is often the decisive moment with an unresponsive server where confusion turns into a concrete diagnosis, because it is the only way to actually see what the server is currently doing.

It is important to regularly test access to this console outside of incidents. An IPMI password never used again since the initial setup, or a forgotten firmware update, can become a problem at exactly the moment access to an unresponsive server is most urgently needed.

With rented servers from large hosting providers, IPMI access is often secured via a separate VPN or an IP whitelist, which must be set up and tested in advance. An unresponsive server whose IPMI access has to be requested only during the incident loses valuable minutes waiting for the provider's support.

5. Step 4: Magic SysRq for a controlled intervention

Magic SysRq keys allow direct commands to the Linux kernel, independent of userspace processes, and are therefore often the last rescue before an unresponsive server faces a full reboot. Provided /proc/sys/kernel/sysrq was enabled beforehand and out-of-band access to the console exists, the REISUB sequence delivers a controlled shutdown instead of an abrupt power cut: R (keyboard back to raw mode), E (terminate all processes), I (hard-kill all processes), S (sync filesystems), U (remount read-only), B (reboot).

# Enable SysRq beforehand (must be done in advance, not during an incident)
echo 1 > /proc/sys/kernel/sysrq

# During an incident, via serial console, trigger REISUB sequence:
# echo r > /proc/sysrq-trigger   # raw keyboard mode
# echo e > /proc/sysrq-trigger   # SIGTERM to all processes except init
# echo i > /proc/sysrq-trigger   # SIGKILL to all processes except init
# echo s > /proc/sysrq-trigger   # sync all filesystems
# echo u > /proc/sysrq-trigger   # remount all filesystems read-only
# echo b > /proc/sysrq-trigger   # immediate reboot

# Before REISUB, if kernel still responds: dump a call trace for later analysis
echo t > /proc/sysrq-trigger    # dump all task states to dmesg/console

# Optional: check current memory pressure before deciding on REISUB
echo m > /proc/sysrq-trigger    # dump current memory info to dmesg/console

The decisive advantage of REISUB over a hard power reset: filesystems are synced and remounted read-only before the actual reboot happens, considerably reducing the risk of filesystem corruption. If the kernel does not even respond to SysRq keys, the kernel itself is completely blocked, and an unresponsive server in this state can only be ended through a reset via the out-of-band interface.

6. Step 5: kernel panic and OOM traces after the reboot

After rebooting a previously unresponsive server, root cause analysis is just as important as the recovery itself, to prevent a recurrence. journalctl -k -b -1 shows the kernel logs from the previous boot, that is, exactly the session that led to the failure. A kernel panic entry with a full stack trace often directly reveals the causing kernel component, for example a faulty driver or a filesystem problem.

If there is no explicit panic entry, but messages from the OOM killer are found (Out of memory: Killed process), memory exhaustion was likely the cause: the kernel then kills individual processes, but in doing so can end up in a state that also affects critical system services like sshd. dmesg -T | grep -i "killed process" after the reboot shows exactly which process fell victim to the OOM killer, providing a concrete starting point for further investigation.

7. Step 6: the controlled reset as the last resort

If neither SSH nor the out-of-band console nor Magic SysRq responds, the only option left for an unresponsive server is a hard reset via the power management interface of IPMI, the cloud provider, or physically on site. Important here: before the reset, if any access path still works, preserve as much diagnostic data as possible, for example a screenshot of the serial console or the last visible kernel messages, because this information is often irrecoverably lost after the reboot.

After a hard reset without a prior REISUB, there is an increased risk of filesystem inconsistencies, which is why an fsck run is frequently triggered automatically on the next boot and can take considerable time on large filesystems. This time should be factored into the incident communication plan in advance, so stakeholders are not surprised when the unresponsive server needs several more minutes for filesystem checking after the reset before services start again.

8. After the incident: documentation and follow-up

Every incident involving an unresponsive server should be recorded in a structured post-mortem: the time of the first alert, the escalation steps taken, the actual cause according to the kernel logs, and the time to recovery. This documentation is not only important for compliance purposes, it also provides the basis for reacting faster next time, because it is already known which step in this runbook typically leads to success.

Particularly valuable is the question of whether an earlier intervention could have prevented the problem, for example through a memory limit for a specific service, an enabled watchdog module, or earlier monitoring alerting on rising memory pressure. An unresponsive server whose cause has been identified and fixed is a significantly smaller risk than the same incident that could recur under identical conditions at any time.

9. Prevention: what every server should have beforehand

The most effective measure against an unresponsive server is preparation, long before the incident occurs. echo 1 > /proc/sys/kernel/sysrq configured permanently via /etc/sysctl.d/, working and regularly tested access to IPMI or the cloud provider console, and a documented password and access concept for emergencies belong to the baseline equipment of every production server.

A kernel watchdog (the softdog module or a hardware watchdog) can automatically trigger a reboot in the event of a genuine kernel hang, without waiting for human intervention, which drastically reduces downtime for an unresponsive server. Combined with alerting that monitors not only application metrics but also the server's basic reachability itself, the time from the start of an outage to the first reaction often shrinks from hours to minutes.

These five escalation levels form the backbone of the runbook for an unresponsive server: each level is only abandoned once the previous one has demonstrably failed, keeping the risk of filesystem corruption or data loss minimized for as long as possible.

10. Summary

An unresponsive server demands a clear, pre-defined escalation order instead of a panic reaction: first check network reachability from a second location, then test alternative access paths, then use out-of-band access via IPMI or the cloud provider console, then try Magic SysRq with REISUB for a controlled shutdown, and only as an absolute last resort trigger a hard reset.

After recovery, analysis of the kernel logs from the previous boot determines whether a kernel panic, memory exhaustion, or another cause was responsible. Anyone who documents this order in advance and sets up the necessary access paths like IPMI and SysRq beforehand turns a potentially hours-long outage of an unresponsive server into an incident resolved in a controlled manner within minutes.

11. FAQ: Unresponsive server