Linux Watchdog Timers: Using Hardware and Software Watchdogs Correctly

1. What a watchdog timer actually does

The basic principle of a watchdog timer is deliberately simple: a timer continuously counts down, and a monitoring instance, usually the kernel or a userspace daemon, must reset the timer at regular intervals, colloquially called feeding it. If that confirmation stops arriving, because the monitored process hangs, crashes, or gets stuck in an infinite loop, the timer triggers a hard reset of the system once it reaches zero.

This simplicity is also the strength of the concept: a watchdog makes no complex diagnosis of why a system stopped responding, it merely reacts to the absence of the expected confirmation. That reduced feature set is exactly what makes it robust enough to keep working even when practically every other component of the system has already failed.

2. The hardware watchdog under /dev/watchdog

Server mainboards frequently ship their own hardware watchdog chip, for example as part of the iTCO chipset on Intel platforms, or as part of an IPMI controller on dedicated server boards. The kernel driver for that chip exposes the device /dev/watchdog, through which a userspace process resets the timer regularly via a write() call.

The decisive advantage of a hardware watchdog is its independence from the operating system itself: even if the kernel is completely frozen and no longer executing any code, the timer on the mainboard keeps running independently, and once it expires it triggers a genuine hardware reset, regardless of whether the operating system would even still be able to initiate a reboot on its own.

# Identify the available hardware watchdog driver
ls -la /dev/watchdog*
cat /sys/class/watchdog/watchdog0/identity

# Read the currently configured hardware watchdog timeout
cat /sys/class/watchdog/watchdog0/timeout

3. Software watchdog versus hardware watchdog

Where dedicated watchdog hardware is missing, for example on virtual machines without a passed through chipset, the kernel module softdog steps in and emulates a watchdog purely in software. Since that timer is itself part of the kernel, it reliably protects against hanging userspace processes, but not against an actual kernel panic or a genuine kernel level lockup, because the emulated timer cannot keep running in that case either.

In cloud environments with virtual machines, the situation is mixed: some hypervisors, such as KVM with the i6300esb emulator, pass a virtual hardware watchdog through to the guest that behaves almost identically to real hardware. Other cloud providers instead offer their own health checks running outside the VM, which fill a similar role but need to be configured differently from a classic Linux watchdog.

4. The systemd integrated watchdog

systemd itself can act as a watchdog client, taking over the regular feeding of /dev/watchdog once the option RuntimeWatchdogSec is set in /etc/systemd/system.conf. That removes the need to run a separate classic watchdog daemon, since PID 1 already handles this task centrally.

In addition, systemd has a second, independent watchdog mechanism called ShutdownWatchdogSec, which specifically safeguards the shutdown process itself. If a server hangs during shutdown, for instance because a service refuses to terminate cleanly, this separate timer triggers a forced reset once it expires, instead of leaving the server stuck indefinitely in a shutdown state.

# /etc/systemd/system.conf
[Manager]
RuntimeWatchdogSec=30s
ShutdownWatchdogSec=10min

5. Configuring WatchdogSec in individual systemd units

Besides the system wide hardware watchdog, systemd also supports an application specific watchdog mechanism through the WatchdogSec directive in a single service unit. The monitored service must regularly report WATCHDOG=1 to systemd through the sd_notify interface, usually via the libsystemd library or a corresponding language binding.

If that confirmation stops arriving for longer than the configured interval, systemd treats the service as hung and applies the reaction configured through WatchdogSignal and FailureAction, usually restarting the service through the regular Restart= logic. This mechanism also catches application level hangs that would not trigger a kernel watchdog, since the process itself is technically still running, just no longer responding to requests.

# /etc/systemd/system/myapp.service
[Service]
ExecStart=/usr/local/bin/myapp
WatchdogSec=15s
Restart=on-watchdog

6. The classic watchdog daemon and /etc/watchdog.conf

Before the systemd integrated watchdog became widely available, the standalone watchdog package handled this task, configured via /etc/watchdog.conf. That daemon offers additional checks beyond plain timer confirmation, such as load average thresholds, free memory limits, or the reachability of configured test files, and triggers a reboot once those thresholds are exceeded, independent of the actual hardware timeout.

For most modern setups with a current systemd, the integrated watchdog mechanism is the simpler choice, since it requires no extra package and integrates cleanly with the rest of the unit configuration. The classic daemon remains relevant mainly where very old systems are in play, or where extra checks beyond a plain watchdog are needed.

7. Practical case: automatically rebooting an unattended server

For a Magento hosting server without physical on site coverage outside business hours, combining a hardware watchdog with systemd integration is the most robust solution. The hardware watchdog covers a completely frozen kernel, while WatchdogSec on critical service units, such as for PHP-FPM or Nginx, detects application level hangs that never touch the kernel itself.

A realistic timeout matters: too short a RuntimeWatchdogSec triggers unnecessary reboots during brief but harmless load spikes, while too long a timeout unnecessarily extends downtime in a real incident. A starting value of 30 to 60 seconds for the hardware watchdog and 15 to 30 seconds for critical application watchdogs has proven reliable in practice for most web server workloads.

# Check watchdog status and the currently configured timeout
systemctl show -p RuntimeWatchdogUSec
wdctl /dev/watchdog

8. Testing watchdog configuration without risking a production outage

A watchdog that was never tested is an unknown risk in a real incident. On a separate test machine or VM, an actual hang can be simulated, for example through a deliberately blocking kernel module call, or by stopping the systemd process itself with SIGSTOP, to verify that the configured reboot actually fires.

At the application level, WatchdogSec can be tested just as specifically, by briefly blocking the monitored service, for instance with an artificial infinite loop in a test build. Only once the observed reboot actually occurs within the expected time frame should the configuration be considered production ready, since a theoretically correct configuration and actual behavior can genuinely diverge.

9. Best practices and common pitfalls

A common mistake is assuming that a software watchdog also protects against a real kernel panic. That is only partly true: a kernel panic already triggers a reboot on its own in most configurations via kernel.panic in sysctl, while a watchdog primarily protects against silent hangs, where the kernel is technically running but no longer responding.

Cloud providers vary widely in their watchdog support: some pass a virtual hardware watchdog through completely, others block access to /dev/watchdog entirely in their default images. Before going into production it is therefore worth an explicit test in the target environment, rather than relying on the configuration of an on premises server carrying over one to one.

10. Summary

11. FAQ: Watchdog Timers