Setting Up Redis Sentinel for Automatic Failover

1. Why Redis Sentinel? Positioning and architecture

Redis Sentinel is the built in high availability solution for classic master replica setups that have no sharding requirement. Without Sentinel, detecting a master outage and promoting a replica rests entirely on a human: someone has to notice the outage, pick the most suitable replica, promote it with REPLICAOF NO ONE, and repoint every client at the new address. During a nighttime outage that often means minutes to hours of downtime, while Redis Sentinel typically performs the same operation automatically within a few seconds.

It is important to understand that Redis Sentinel is not a proxy that traffic flows through. It consists of separate Redis processes running in Sentinel mode that monitor the configured master and its replicas via PING and INFO, and that know the current state of the system. Clients still connect directly to the Redis instance, but first ask a Sentinel instance for the current address of the master. This indirection is the core of the architecture: Sentinel knows the truth about the system's state, and clients query that truth instead of guessing it.

A single Redis Sentinel instance would itself become a single point of failure, so Sentinel always runs as a group of at least three independent processes, ideally on separate hosts. This group makes decisions collectively, not through a single instance. This distributed decision model with quorum and leader election is exactly the topic of the next section, and the real reason why Redis Sentinel is more reliable than a simple health check script with failover logic bolted on.

2. Sentinel architecture: quorum, voting, and leader election

The central concept of Redis Sentinel is the distinction between subjective and objective failure. When a single Sentinel instance detects that the master no longer responds to PING, it internally marks it as Subjectively Down (SDOWN). That alone does not trigger a failover, because it could be a local network issue affecting just that one Sentinel instance. Only once a sufficient number of other Sentinels, at least the configured quorum, confirms the same outage does the master get classified as Objectively Down (ODOWN), and only from that point does the actual failover process begin.

The quorum is set per monitored master in the configuration and defines how many Sentinels must agree on the outage for it to count as real. A quorum of 2 across three Sentinel instances means two out of three Sentinels must independently confirm the outage. The quorum alone, however, does not decide who actually performs the failover. That takes an additional step: once ODOWN is reached, the Sentinels start a leader election using a Raft inspired procedure, in which one Sentinel instance must win to be allowed to execute the failover. Electing the leader requires a true majority of all known Sentinels, not just the configured quorum, a distinction that is frequently overlooked.

This two stage model of quorum based state detection plus majority based leader election makes Redis Sentinel robust against isolated faulty observations. A single Sentinel that mistakenly believes the master is dead because of its own network problem cannot trigger a failover on its own. Only when the majority of the Sentinel group independently reaches the same conclusion does the system actually act. This architecture is the reason you should never run Redis Sentinel with only two instances, more on that in the section on split brain pitfalls.

3. sentinel.conf: monitoring configuration in detail

Configuring Redis Sentinel happens in its own sentinel.conf, separate from the regular redis.conf. The central directive is sentinel monitor, which sets the name of the monitored master group, its IP and port, and the quorum. All other parameters, such as down-after-milliseconds, which determines the time until an SDOWN marking, or failover-timeout, which bounds the time window for a single failover attempt, are configured relative to this master name.

An often underestimated parameter is parallel-syncs, which sets how many replicas may resync against the new master at the same time. A value that is too high can overload the new master's network bandwidth during the transition when there are many replicas, while a value that is too low extends the time until all replicas are consistent again. In practice a value of 1 has proven effective for setups with many replicas, so the new master is not put under extra load during the critical phase.

# sentinel.conf: monitoring configuration for one master group
port 26379
sentinel resolve-hostnames yes
sentinel announce-ip 10.0.1.11
sentinel announce-port 26379

# Monitor master "orders-cache" at 10.0.1.10:6379, quorum of 2
sentinel monitor orders-cache 10.0.1.10 6379 2

# Mark SDOWN after 5s of no valid PING reply
sentinel down-after-milliseconds orders-cache 5000

# Only 1 replica resyncs against the new master at a time
sentinel parallel-syncs orders-cache 1

# Abort a stuck failover attempt after 3 minutes
sentinel failover-timeout orders-cache 180000

# Optional: require replica auth
sentinel auth-pass orders-cache s3cur3-replica-pw

# Run a script on every failover event (paging, DNS update, etc.)
sentinel notification-script orders-cache /usr/local/bin/notify-failover.sh
sentinel client-reconfig-script orders-cache /usr/local/bin/update-dns.sh

It has also proven effective to give all three sentinel.conf files in a group exactly the same down-after-milliseconds and failover-timeout values. Different values across individual Sentinel instances lead to inconsistent timing during SDOWN detection and make troubleshooting significantly harder if a later failover does not behave as expected.

4. Failover flow: from detection to promotion

The sequence of a failover in Redis Sentinel always follows the same pattern. First, a Sentinel instance detects SDOWN, queries the other configured Sentinels via SENTINEL is-master-down-by-addr, and waits for their confirmation. Once the quorum is reached, the state moves to ODOWN and the leader election begins. The elected leader then picks the most suitable replica out of all known replicas, based on replica-priority, the amount of already replicated data, and the replication offset position, and runs REPLICAOF NO ONE on that replica.

After the promotion, the Sentinel leader reconfigures all remaining replicas to replicate against the new master, and updates its internal view of the master group. All other Sentinel instances adopt this new configuration within a short time. The whole process, from the first SDOWN detection to the complete switchover, typically takes five to fifteen seconds in a well configured setup, depending on down-after-milliseconds and the number of replicas.

# Inspect Sentinel's live view of the master group
redis-cli -p 26379 sentinel master orders-cache

# List replicas Sentinel currently knows about
redis-cli -p 26379 sentinel replicas orders-cache

# Force a manual failover for testing (do this in staging first)
redis-cli -p 26379 sentinel failover orders-cache

# Typical log sequence during an automatic failover
# +sdown master orders-cache 10.0.1.10 6379
# +odown master orders-cache 10.0.1.10 6379 #quorum 2/2
# +new-epoch 1
# +try-failover master orders-cache 10.0.1.10 6379
# +vote-for-leader <sentinel-id> 1
# +elected-leader master orders-cache 10.0.1.10 6379
# +failover-state-select-slave master orders-cache 10.0.1.10 6379
# +selected-slave slave 10.0.1.12:6379 orders-cache 10.0.1.10 6379
# +failover-state-send-slaveof-noone slave 10.0.1.12:6379 ...
# +failover-state-reconf-slaves master orders-cache 10.0.1.10 6379
# +failover-end master orders-cache 10.0.1.10 6379

A detail that is often overlooked in production incidents: if the old master becomes reachable again after its outage, it does not automatically become master again. Redis Sentinel instead reconfigures it as a replica of the newly promoted master as soon as it checks back in. That prevents split brain situations with two active masters at once, but can cause data loss if the old master had still accepted writes between the last replication sync and the outage that were never replicated.

5. Client discovery: how applications use Sentinel

Applications must never hardcode the address of the Redis master when Redis Sentinel is in use, because that address changes on every failover. Instead, the client knows the addresses of all Sentinel instances and, on connection setup, asks SENTINEL get-master-addr-by-name for the current master address. Most modern Redis client libraries offer built in Sentinel support that performs this lookup automatically and reconnects transparently on a +switch-master notification.

It is important to configure the client with several Sentinel addresses, not just one, so that the failure of a single Sentinel instance does not block discovery. Clients should also be able to subscribe to +switch-master pub/sub notifications from Sentinel, to detect connection changes proactively instead of only on the next failed request. That reduces the number of requests that hit the old, no longer existing master address during a failover window.

# Query current master address directly via redis-cli
redis-cli -p 26379 sentinel get-master-addr-by-name orders-cache
# -> 10.0.1.12
# -> 6379

# Subscribe to failover notifications for proactive reconnects
redis-cli -p 26379 subscribe +switch-master
# Example message payload:
# orders-cache 10.0.1.10 6379 10.0.1.12 6379

# Generic client connection pattern (pseudocode, applies to
# most Sentinel-aware client libraries regardless of language)
sentinels = [
  {"host": "10.0.1.11", "port": 26379},
  {"host": "10.0.1.12", "port": 26379},
  {"host": "10.0.1.13", "port": 26379}
]
client = RedisSentinelClient(
  sentinels=sentinels,
  service_name="orders-cache",
  socket_timeout=0.5
)
master = client.master_for("orders-cache")

Another point regarding client discovery: read traffic can be routed specifically to replicas by having the client query SENTINEL slaves orders-cache instead of concentrating all load on the master. That relieves the master and is particularly relevant for read heavy workloads. Doing so requires the application to accept that replica reads can return slightly stale data, since replication is asynchronous.

6. Running a Sentinel cluster: count, placement, network

The number of Redis Sentinel instances should always be odd, commonly three or five. An odd number prevents tie situations during the majority based leader election. Three Sentinels are enough for most setups and tolerate the loss of one instance without losing failover capability. Five instances further increase fault tolerance and make sense once the infrastructure spans three or more availability zones.

Physical placement matters just as much as the count. Each Sentinel instance should run on a different host than the other Sentinels, and ideally separate from the Redis data nodes themselves, so that a host failure does not take down a Sentinel instance and a Redis node at the same time. In cloud environments that means spreading Sentinels across different availability zones. A common anti pattern is running all three Sentinels on the same three hosts as master and replicas without mixing the assignment, which significantly reduces the actual fault tolerance.

# Recommended topology: 3 Sentinels spread across 3 AZs,
# each AZ also hosting one Redis data node (master or replica)

# AZ-a: redis-master + sentinel-1
# AZ-b: redis-replica-1 + sentinel-2
# AZ-c: redis-replica-2 + sentinel-3

# Start Sentinel with its own dedicated config file
redis-sentinel /etc/redis/sentinel.conf --daemonize no

# systemd unit snippet for production deployment
# [Unit]
# Description=Redis Sentinel
# After=network.target
# [Service]
# ExecStart=/usr/bin/redis-sentinel /etc/redis/sentinel.conf
# Restart=always
# User=redis
# [Install]
# WantedBy=multi-user.target

An often underestimated detail: Redis Sentinel writes its detected configuration, for example after a failover, back into its own sentinel.conf. This file therefore must be writable and must not be overwritten by configuration management tools on every deploy, otherwise Sentinel loses its last known correct view of the current master after a restart.

7. Avoiding split brain and quorum pitfalls

A split brain scenario arises when a network partition event splits the Sentinel group into two isolated halves, and both halves independently believe they see the master correctly. With a quorum of 2 out of three Sentinels, a partition containing two Sentinels can trigger a failover, while the isolated third instance still considers the old master healthy. As long as clients on the isolated side keep writing against the old master, that data gets lost once the partition ends, because the old master turns into a replica of the new one.

To minimize this risk, the quorum should never be set lower than half plus one of the Sentinel instances, even though Redis technically allows smaller values. In addition, the min-replicas-to-write parameter on the Redis master itself protects against an isolated master that no longer has a connection to enough replicas continuing to accept writes. Combined with min-replicas-max-lag, the master refuses writes once too few replicas are sufficiently up to date, which considerably shortens the window for data loss during a split brain.

# redis.conf on the master: reject writes if fewer than 1
# replica is connected with an ack lag under 10 seconds
min-replicas-to-write 1
min-replicas-max-lag 10

# Check current replica ack status live
redis-cli -p 6379 info replication
# connected_slaves:2
# slave0:ip=10.0.1.12,port=6379,state=online,offset=88213,lag=0
# slave1:ip=10.0.1.13,port=6379,state=online,offset=88213,lag=1

A third safeguard is a cleanly configured failover-timeout. If a failover attempt does not conclude within this time window, Redis Sentinel automatically starts a new attempt with a new epoch, which prevents repeated, inconsistent states. Combining all three mechanisms, a quorum that is a true majority, min-replicas-to-write, and a realistic failover-timeout, reduces the split brain risk to a level that is negligible in practice.

8. Monitoring and health checks for Sentinel itself

Sentinel monitors Redis, but Sentinel itself must also be monitored, otherwise nobody notices when the high availability layer fails while Redis keeps running normally. The command SENTINEL info-cache and the standard INFO output of every Sentinel instance provide metrics such as the number of known Sentinels, the number of known replicas, and the current state of the master as seen by that instance. If these values diverge between the three Sentinel instances, it points to a network problem or a misconfiguration.

In practice it has proven useful to have an external monitoring system periodically query SENTINEL master orders-cache on all three instances and compare the returned num-other-sentinels and num-slaves values. A persistently lower value on one instance shows that this Sentinel no longer sees the others fully, long before an actual failover makes the problem visible. The alert script from notification-script should also fire not only on real failovers but on SDOWN events without a following ODOWN, because such flapping patterns indicate unstable network connections.

# Cross-check Sentinel state consistency across all instances
for port in 26379 26380 26381; do
  echo "=== sentinel on port $port ==="
  redis-cli -p "$port" sentinel master orders-cache | \
    grep -E "num-other-sentinels|num-slaves|flags"
done

# Watch for flapping SDOWN events in the log (instability signal)
grep -c "+sdown" /var/log/redis/sentinel.log

# Confirm all three Sentinels agree on the current master
redis-cli -p 26379 sentinel get-master-addr-by-name orders-cache
redis-cli -p 26380 sentinel get-master-addr-by-name orders-cache
redis-cli -p 26381 sentinel get-master-addr-by-name orders-cache

It also pays off to add a simple end to end check that periodically writes and reads an actual value through the master address returned by Sentinel. That catches scenarios where Sentinel itself looks healthy but the Redis master, for some other reason such as a full persistence disk, no longer accepts writes. Pure Sentinel monitoring alone never replaces functional monitoring of the actual database.

9. Redis Sentinel compared to other approaches

Besides Redis Sentinel there are other approaches to high availability, and the choice depends heavily on the existing ecosystem. Manual failover scripts with health checks via cron or external monitoring tools offer full control, but are error prone because they usually do not cleanly solve the quorum and leader election problem at all. Redis Cluster also solves high availability, but ties it inseparably to sharding, which means unnecessary operational complexity for setups that do not need sharding.

For setups that primarily need resilience rather than additional capacity, Redis Sentinel remains the leanest built in solution. Moving from Sentinel to Cluster only pays off once data volume or write throughput exceeds the capacity of a single master, not simply because of a wish for more automation.

10. Summary

Redis Sentinel for automatic failover solves a concrete operational problem: without automation, a human's reaction time determines the length of the outage. The two stage architecture of subjective failure detection, quorum based confirmation, and majority based leader election makes Sentinel robust against false alarms from individual instances. A clean sentinel.conf with suitable values for down-after-milliseconds, parallel-syncs, and failover-timeout is the foundation of every stable setup.

Just as important as configuring Sentinel itself is correct client discovery: applications must never hardcode the master address, but must query it through Sentinel and react to +switch-master events. With at least three Sentinel instances spread across separate availability zones, a realistic quorum, and additional protection through min-replicas-to-write, the split brain risk can be reduced to a negligible level without giving up the operational simplicity of a classic master replica setup.

11. FAQ: Redis Sentinel for Automatic Failover