Avoiding Split-Brain Scenarios in Redis Clusters

1. What split-brain in Redis clusters actually means

A split-brain occurs when a replicated Redis setup is torn apart into two or more pieces by a network partition, and several nodes simultaneously believe they are the valid master. Both sides of the partition then independently accept writes, unaware of each other. Once the network reconnects, two diverging data states exist, and Redis must pick one, discarding the other.

What makes a split-brain dangerous is not the outage itself, but the time window during which both sides appear perfectly functional on their own. Applications on either side of the partition keep writing without seeing any errors, and only when the partition is merged back together does it become visible that some of those writes are gone. For stateful systems like shopping carts, counters, or sessions, this can lead to inconsistent or vanished data that is nearly impossible to reconstruct afterward.

2. How network partitions arise and behave

Network partitions rarely result from a complete total outage. More often they stem from partial disruptions: a switch firmware update, a misconfigured firewall rule, an overloaded cross-AZ link in the cloud, or a routing error after a deployment. From the perspective of a single Redis node, a network fault that triggers split-brain is indistinguishable from a simple crash of its peer. Both symptoms look identical to the observing node: the other side just stops responding.

Particularly tricky are asymmetric partitions, where node A can no longer reach node B, but node B can still reach node A. In such a scenario, different observers in the cluster can reach contradictory conclusions about which node has actually failed. This exact scenario is why Redis high availability must never rely on the opinion of a single observer, but always on a majority decision made by several independent instances.

3. Sentinel architecture and the quorum mechanism

Redis Sentinel solves the problem of a single, failure-prone observer by using several independent Sentinel processes, typically running on separate hosts, that monitor each other as well as the master. Only when a configured minimum number of Sentinels, the so-called quorum, independently determine that the master is unreachable does it get officially marked as subjectively down (SDOWN) and subsequently as objectively down (ODOWN). Only then does the failover process begin.

The quorum is set in the Sentinel configuration with sentinel monitor mymaster 10.0.0.1 6379 2, where the last number specifies the minimum number of agreeing Sentinels. With three Sentinels and quorum 2, at least two independent Sentinels must confirm the outage before a failover is triggered. This majority requirement is the central safeguard against split-brain: a single Sentinel isolated by its own network fault cannot force a failover on its own.

An odd number of Sentinel instances, usually three or five, spread across different failure zones, is essential. An even number of Sentinels risks a deadlock in the case of an exact network split, where neither side reaches a majority and the failover stays blocked, which does not create a split-brain but does unnecessarily limit availability.

# sentinel.conf , quorum-based monitoring across 3 independent hosts
port 26379
sentinel monitor mymaster 10.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1

# Deploy one sentinel per availability zone, odd total count (3 or 5)
# so a network split can never produce two equal-sized quorum groups

4. Configuring min-replicas-to-write and min-replicas-max-lag

A master that gets isolated from its replicas can keep accepting writes even though nobody is replicating those changes anymore. This is exactly where min-replicas-to-write comes in: this setting refuses writes as soon as fewer than the configured number of replicas is reachable within an acceptable delay. An isolated master that no longer receives replica acknowledgments effectively becomes read-only, instead of continuing to write uncontrolled and later producing conflicting data states.

The companion setting min-replicas-max-lag defines how many seconds of delay a replica may have at most to still count as available. A replica that has not sent an ACK in 15 seconds no longer counts with min-replicas-max-lag 10. This combination reduces how many writes can accumulate on an isolated master in the worst case before it stops writing itself.

It is important to understand that min-replicas-to-write does not fully prevent split-brain, it limits the time window and the amount of potentially lost writes. Without active replicas, a master configured this way simply stops writing altogether, which for many applications is an acceptable tradeoff between availability and consistency, while others consider it too restrictive.

# redis.conf on the master node
min-replicas-to-write 1
min-replicas-max-lag 10

# Verify current replica ACK status
redis-cli -h master.internal INFO replication
# role:master
# connected_slaves:2
# slave0:ip=10.0.0.2,port=6379,state=online,offset=910234,lag=0
# slave1:ip=10.0.0.3,port=6379,state=online,offset=910234,lag=1

# Simulate isolation: after losing both replicas, writes are refused
redis-cli -h master.internal SET orders:1042 "paid"
# (error) NOREPLICAS Not enough good replicas to write.

5. Majority rule in Redis Cluster: cluster-require-full-coverage

Redis Cluster does not use Sentinel instances at all, it uses the master nodes themselves as a distributed voting body. If a majority of master nodes detects that another master has been unreachable for a certain time, that master's best replica is automatically promoted to become the new master. This majority requirement, at least half plus one of all master nodes, prevents a minority partition from unilaterally triggering a failover and thereby creating a second, competing master for the same slot range.

In a classic network partition that splits a cluster into a majority half and a minority half, only the majority side remains writable. The minority side recognizes that it can no longer form a cluster majority, and with cluster-require-full-coverage enabled, it also stops accepting writes. That is the decisive structural difference from a poorly configured Sentinel environment: Redis Cluster actively refuses to let the minority keep writing, instead of allowing two equally valid masters.

6. The failover sequence in detail: timeout, voting, promotion

The failover sequence follows a fixed pattern. First, every observing Sentinel locally marks an unreachable master as SDOWN once down-after-milliseconds is exceeded. Then the Sentinels query each other to see whether they share the same assessment. Once the number of agreeing Sentinels reaches the configured quorum, the state escalates to ODOWN and a Sentinel is elected leader for the failover, through its own Raft-like election process using epoch numbers.

The elected leader Sentinel then identifies the replica with the most up-to-date replication offset, promotes it to the new master with REPLICAOF NO ONE, and reconfigures the remaining replicas, controlled by parallel-syncs to limit network load during resynchronization. The entire process typically takes a few seconds, up to failover-timeout, depending on network latency and the amount of data that needs to be resynchronized.

# Check cluster-wide majority state on any node
redis-cli -c CLUSTER INFO
# cluster_state:ok
# cluster_slots_assigned:16384
# cluster_slots_ok:16384
# cluster_known_nodes:6
# cluster_size:3

# During a minority-side partition, the isolated nodes report:
# cluster_state:fail
# cluster_slots_assigned:5461   <- only the locally known range
redis-cli -c SET orders:9001 "paid"
# (error) CLUSTERDOWN Hash slot not served

# Sentinel log excerpt during a real failover sequence
+sdown master mymaster 10.0.0.1 6379
+odown master mymaster 10.0.0.1 6379 #quorum 2/2
+new-epoch 1
+try-failover master mymaster 10.0.0.1 6379
+vote-for-leader 7d3e2f1a sentinel-2 1
+elected-leader master mymaster 10.0.0.1 6379
+selected-slave slave 10.0.0.3 6379
+failover-state-send-slaveof-noone slave 10.0.0.3 6379
+failover-state-reconf-slaves master mymaster 10.0.0.1 6379
+switch-master mymaster 10.0.0.1 6379 10.0.0.3 6379

7. Data loss from asynchronous replication despite safeguards

Even with correctly configured quorum, min-replicas-to-write, and cluster majority checks, a structural residual risk remains: Redis replicates asynchronously by default. A write is considered successful as soon as the master has processed it locally, regardless of whether any replica has received it yet. If the master fails in exactly that short window, the last writes that were not yet replicated are lost during failover, even if no classic split-brain moment ever occurred.

This behavior is often confused with split-brain, but it is a distinct problem: it is not two competing masters, it is ordinary data loss caused by asynchronous replication at the moment of failover. min-replicas-to-write shrinks this window by refusing writes without replica acknowledgment upfront, but it can only be fully eliminated with a synchronous WAIT before critical writes, which costs latency.

8. Deliberately simulating and testing split-brain

Anyone who wants to rely on failover behavior should have tested it under controlled conditions, rather than verifying the configuration for the first time during a real outage. With iptables you can deliberately block packets between two nodes to simulate a genuine network partition without terminating the process itself. With tc (Traffic Control) you can additionally introduce increased latency or packet loss to model more realistic, partial network disruptions instead of a hard cutoff.

A meaningful test scenario fully isolates the current master from its replicas and all Sentinels, lets the application keep writing during the isolation, and then, after the connection is restored, checks how many writes were actually lost. This lets you empirically verify whether the configured values for min-replicas-to-write, quorum, and timeouts actually meet your own consistency requirements.

# Simulate a network partition between master (10.0.0.1) and one replica (10.0.0.2)
iptables -A INPUT -s 10.0.0.2 -j DROP
iptables -A OUTPUT -d 10.0.0.2 -j DROP

# Add latency and packet loss for a softer partition instead of a hard cut
tc qdisc add dev eth0 root netem loss 30% delay 200ms 50ms

# Watch Sentinel react on the isolated side
redis-cli -p 26379 SENTINEL master mymaster
# ... check flags field for s_down / o_down / failover-in-progress

# After the test: always remove the simulated failure
iptables -D INPUT -s 10.0.0.2 -j DROP
iptables -D OUTPUT -d 10.0.0.2 -j DROP
tc qdisc del dev eth0 root netem

9. Monitoring and alerting for partitioning

A split-brain that goes unnoticed for hours causes significantly more damage than one detected within seconds. Sentinel itself provides pub/sub channels like +sdown, +odown, and +switch-master that can be fed directly into a monitoring system without parsing log files. In addition, INFO replication on every node returns the current role value, so an alert can trigger as soon as more than one node reports role:master for the same logical dataset, which is the clearest signal of an actual split-brain.

It is also worth alerting on the replication lag values from INFO replication, since a suddenly rising lag is often the first visible sign of a developing network partition, long before Sentinel reports a complete outage. Continuously watching connected_slaves on the master and treating a drop to zero as a critical alert often detects master isolation faster than the Sentinel failover timeout would otherwise allow.

10. Summary

A split-brain arises when a network partition leads several Redis nodes to make write decisions simultaneously without knowing about each other. Sentinel counters this risk with a quorum mechanism that ties failover decisions to a majority of independent observers, instead of leaving them to a single node. min-replicas-to-write and min-replicas-max-lag further limit how long an isolated master can keep writing uncontrolled, by switching it to read-only once replica acknowledgment is missing.

Redis Cluster solves the same problem structurally through majority decisions among master nodes and cluster-require-full-coverage, which actively blocks writes on a minority partition. A residual risk remains due to asynchronous replication, which is why regular failover tests with iptables and tc, along with continuous monitoring of Sentinel events and replication lag, are essential, so a split-brain scenario is understood well before it happens for real.

11. FAQ: split-brain scenarios in Redis clusters