Testing Failover Strategies Properly: Chaos Engineering for Linux Clusters

1. Why untested failover is not real protection

Testing failover is the step frequently skipped in many infrastructure projects once keepalived, Pacemaker or HAProxy have been successfully configured once. The fallacy here: a configuration that looks correct in theory and works in the ideal case says nothing about how the system reacts under realistic failure conditions, for example a half functioning network, a hung rather than crashed application, or a node that is only intermittently reachable.

The difference between a clean process crash and a real production outage is substantial. A kill -9 on a process produces an unambiguous signal that every health check detects immediately. A network switch with packet loss, a disk that only responds with heavily increased latency, or a database that accepts requests but never answers, on the other hand, produce ambiguous symptoms that many health checks do not cover at all. Anyone testing failover only with the simple case is not testing the case that most often causes long outages in practice.

Systematic failover testing means conducting controlled failure simulations as close as possible to realistic failure modes, measuring the results, and deriving concrete improvements to health checks, timeout values and constraints from them. The following sections show how such a test process is built, from the first metric to a recurring game day.

2. RTO and RPO as measurable targets

Before any test even takes place, two target values must be defined: Recovery Time Objective, or RTO, describes the maximum acceptable time until service is restored after a failure. Recovery Point Objective, or RPO, describes the maximum acceptable data loss, measured in time between the last confirmed state and the failure. Without these two numbers, a failover test is meaningless, because there is no benchmark against which a measured result can be evaluated.

A typical example: a keepalived setup with a one second advert_int and three required consecutive failures theoretically achieves an RTO of about four to six seconds. Whether that value is acceptable for the application depends on business context, not technical possibility. A database cluster with synchronous DRBD replication theoretically achieves an RPO of zero, but only if the test actually confirms that not a single acknowledged write is lost during failover.

3. Chaos engineering principles for Linux clusters

Chaos engineering applies a simple idea to infrastructure: instead of hoping a system is resilient, it is intentionally and controlledly disturbed, to find weaknesses before a real outage exposes them. The four core principles are: define a measurable steady state, formulate a hypothesis about behavior under disturbance, simulate real world failure events, and limit the blast radius of the experiment.

For Linux HA clusters, this concretely means: before a failover test runs, the current steady state is documented via monitoring metrics such as response times, error rates and active connections. The hypothesis is usually that a failure of the primary node leads to a functioning backup node within the defined RTO, without measurable data loss beyond the RPO. The limited blast radius in practice means testing first in a staging environment before running controlled tests in production with a limited share of traffic.

4. Systematically running through concrete failure scenarios

A complete test plan for failover covers several different failure modes, not just the simplest case of a crashed process. The following list shows scenarios that should be tested in ascending order of realism, with each scenario checking a different part of the failover chain.

A clean process stop with systemctl stop checks whether health checks and notify scripts fundamentally work. A hard process kill with kill -9 checks whether the system still reacts reliably without the process getting a cleanup opportunity. A simulated network failure between nodes checks split brain protection mechanisms such as fencing and secondary communication paths. Artificially increased latency instead of a total outage checks whether health check timeouts are sensibly calibrated, or whether a merely slow but functioning node gets falsely marked as failed.

# Scenario 1: clean process stop (baseline test)
sudo systemctl stop haproxy

# Scenario 2: hard kill without cleanup opportunity
sudo pkill -9 -f keepalived

# Scenario 3: simulate full network partition between nodes
sudo iptables -A INPUT -s 10.0.1.12 -j DROP
sudo iptables -A OUTPUT -d 10.0.1.12 -j DROP
# ... observe failover behavior, then restore:
sudo iptables -D INPUT -s 10.0.1.12 -j DROP
sudo iptables -D OUTPUT -d 10.0.1.12 -j DROP

# Scenario 4: artificial latency instead of a full outage (requires tc/netem)
sudo tc qdisc add dev eth0 root netem delay 800ms
# ... observe whether health checks misclassify a slow-but-alive node
sudo tc qdisc del dev eth0 root netem

5. Tools for controlled failure simulation

Besides simple built in tools like iptables and tc netem, dedicated chaos engineering tools exist that enable more realistic and repeatable fault injection. The netem module in the Linux kernel simulates not only latency but also packet loss, duplication and reordering, all conditions that reflect real network problems much better than a simple connection drop.

For more complex, repeatable test scenarios with automated evaluation, tools like Chaos Mesh or Litmus, originally built for Kubernetes but transferable to classic Linux clusters, combine fault injection and observation in a scripted way. It is important with all tools to first establish tests in a staging environment before repeating them in production with a limited blast radius.

6. Measurement during the test: what really counts

A failover test without measurement is just an observation, not solid proof. During every test, at least four values should be continuously captured: the actual time from start of failure until service restoration, the number of failed client requests during the transition, whether data loss occurred compared to the last confirmed write, and whether after the failover all resources actually run in the expected state, not only partially.

A simple but effective pattern is a continuous synthetic client that sends requests to the service throughout the entire test and logs every successful and failed response with a timestamp. From this log, the actual downtime can be calculated exactly, instead of relying solely on estimated values from cluster software logs.

#!/usr/bin/env bash
# synthetic-client.sh — continuous probe during a failover test
set -euo pipefail

TARGET="http://192.168.10.100/healthz"
LOG="/tmp/failover-test-$(date +%s).log"

while true; do
  start=$(date +%s.%N)
  if curl -sf -o /dev/null -m 2 "$TARGET"; then
    status="OK"
  else
    status="FAIL"
  fi
  end=$(date +%s.%N)
  elapsed=$(echo "$end - $start" | bc)
  echo "$(date -Iseconds) status=$status latency=${elapsed}s" >> "$LOG"
  sleep 0.5
done

7. Game days: failover tests as a recurring process

A one off failover test during the initial setup only proves the configuration worked at that point in time, not that it still does after the next software update, network change or configuration adjustment. Game days are planned, recurring sessions in which a team jointly runs controlled failure scenarios in a production like environment, documents them, and compares results with the previous test.

A good rhythm for most infrastructures is one game day per quarter, supplemented by an additional test after every major change to the cluster configuration or the underlying cluster software version. The results of every game day are documented, including measured RTO, observed error rates and all identified weaknesses, so a long term improvement history builds up instead of starting from zero every time.

A simple runner script that executes multiple scenarios in sequence and stores the results in a shared summary significantly simplifies running a game day, because the team can focus on observation instead of manually executing every single step.

#!/usr/bin/env bash
# game-day-runner.sh — orchestrates multiple failover scenarios in sequence
set -euo pipefail

REPORT="/tmp/game-day-$(date +%Y%m%d).md"
echo "# Game Day Report $(date -Iseconds)" > "$REPORT"

run_scenario() {
  local name="$1"
  local action="$2"
  local restore="$3"

  echo "## Scenario: $name" >> "$REPORT"
  start=$(date +%s)
  eval "$action"
  sleep 30   # observation window before restoring
  eval "$restore"
  end=$(date +%s)
  echo "- Wall clock duration: $((end - start))s" >> "$REPORT"
  echo "- Manual RTO/RPO notes: _fill in after reviewing synthetic-client log_" >> "$REPORT"
}

run_scenario "clean-stop-haproxy" \
  "sudo systemctl stop haproxy" \
  "sudo systemctl start haproxy"

run_scenario "network-partition-node2" \
  "sudo iptables -A INPUT -s 10.0.1.12 -j DROP" \
  "sudo iptables -D INPUT -s 10.0.1.12 -j DROP"

echo "Report written to $REPORT"

8. Common pitfalls in failover testing

The most common mistake is testing exclusively the simplest failure mode, a clean process stop, and falsely concluding from that the entire failover system is resilient. Real failures are rarely that unambiguous, and it is exactly the ambiguous cases like network partitioning or hung processes that cause the longest outages in practice.

A second widespread mistake is running tests only in an isolated staging environment that differs significantly from the production environment in network topology, hardware or load. Results from a heavily diverging environment do not reliably transfer to production behavior. A third pitfall is running tests only once during the initial setup and never repeating them, even though configuration, software versions and load profiles change over time.

9. Test maturity compared: from ad hoc to game day

Organizations differ substantially in how systematically failover is tested. The following table classifies different maturity levels and shows which effort corresponds to which confidence level in actual failure resistance.

The jump from no testing to an ad hoc test costs little effort but already brings a noticeable confidence gain, since at least the basic mechanism has been observed once. The bigger jump, however, lies between a regular test plan and real game days with chaos engineering principles, because only there are realistic, ambiguous failure modes systematically covered, instead of continuing to rely on the simplest case. Teams that want to take failover seriously should treat this maturity level as a medium term goal, not an optional extra for later.

An often underestimated side effect of regular failover tests is the knowledge build up within the team itself. Someone who only touched the cluster configuration once during initial setup will struggle more to react quickly and correctly in a real emergency than a team already familiar with the procedures from repeated game days. This routine is itself part of actual resilience, independent of the pure technology.

10. Summary

Testing failover is the difference between a theoretical assumption about resilience and actual proof. Without clearly defined RTO and RPO targets, a test is meaningless, because no benchmark exists against which the measured result can be evaluated. Chaos engineering principles with a limited blast radius help simulate realistic failure modes such as network partitioning or increased latency in a controlled way, instead of limiting testing to the simplest case of a clean process crash.

Continuous measurement during the test, for example via a synthetic client with a timestamped log, delivers solid numbers instead of estimated values. The biggest leverage, however, lies in repetition: a one off test during initial setup only proves the state at that time. Game days as a recurring, documented process ensure that failover still works reliably after the next configuration change.

11. FAQ: Testing Failover Strategies Properly