Redis Cluster: Understanding Sharding Fundamentals

1. Why sharding becomes necessary in Redis at all

Redis Cluster solves a problem that plain master replica replication cannot: a single Redis process is always bound by the memory and network bandwidth of one host. Replicas increase read capacity and resilience, but every replica holds a full copy of the same data, not extra capacity for the overall dataset. Once a dataset outgrows the largest available host, or write throughput exceeds the capacity of a single master, only one option remains: splitting data across several independent nodes. That is exactly what sharding means, and Redis Cluster is the built in implementation of it.

The fundamental difference from manual application level sharding, where an application itself decides which key lives on which Redis node, is that Redis Cluster manages this assignment transparently and automatically. The application connects to any node of the cluster, and the cluster itself either forwards requests internally to the responsible node or informs the client of the correct address via a redirect. That drastically reduces complexity in the application, but shifts the complexity into the cluster infrastructure itself, which must be understood to avoid mistakes.

The core concept that sets Redis Cluster sharding apart from naive approaches like modulo based hashing is the 16384 fixed hash slots. Instead of mapping keys directly to nodes, Redis first maps every key to one of 16384 slots, and each slot is then assigned to exactly one node. This layer of indirection is the reason nodes can later be added or removed without changing the hash function itself, more on that in the following sections.

2. Hash slots: the 16384 slots in detail

Redis Cluster divides the entire keyspace into exactly 16384 hash slots, numbered from 0 to 16383. That number is a deliberate choice: it is large enough to allow fine grained distribution across hundreds of nodes, but small enough that the bitmap representing which node owns which slots stays compact (16384 bits equal exactly 2 KB), which keeps network load low during the regular gossip messages exchanged between nodes. A larger slot count, say 65536, would degrade that compactness without offering a practical benefit at realistic cluster sizes.

Every node in the cluster is responsible for a contiguous or non contiguous subset of these 16384 slots. With three master nodes, an even split might assign slots 0 through 5460 to node A, 5461 through 10922 to node B, and 10923 through 16383 to node C. This assignment is not static in the sense of a fixed formula but an explicit configuration stored in each node's cluster state, and it can change through resharding operations without ever changing the total count of 16384 slots.

# Inspect current slot ownership across the cluster
redis-cli -c -p 7000 cluster slots

# Example output structure (simplified)
# 1) 1) (integer) 0        # slot range start
#    2) (integer) 5460     # slot range end
#    3) 1) "10.0.2.10"     # master node
#       2) (integer) 7000
#    4) 1) "10.0.2.13"     # replica node
#       2) (integer) 7000

# Check how many slots a specific node currently owns
redis-cli -c -p 7000 cluster nodes | grep myself

# Ask which slot a given key would map to
redis-cli -c -p 7000 cluster keyslot "order:48213"
# -> (integer) 9842

One important detail: every slot in the cluster must be assigned to exactly one master, with no gaps and no overlap, otherwise the cluster is considered not fully covered and refuses writes by default. The cluster-require-full-coverage parameter can change that behavior, but it should be used with care, since it allows the cluster to keep serving requests even though part of the slots, and therefore part of the data space, is unreachable.

3. CRC16: how keys map to slots

Mapping a key to a hash slot in Redis Cluster follows a deterministic formula: HASH_SLOT = CRC16(key) mod 16384. CRC16 is a fast, well distributing checksum algorithm that computes a 16 bit value for any given key. The modulo operation with 16384 then projects that value onto the valid slot range. Importantly, this computation is fully reproducible on the client side and deterministic, any client can precompute which slot a given key will land in without ever asking the cluster.

This property makes Redis Cluster clients efficient: instead of blindly querying a random node on every access and waiting for a redirect, modern cluster aware client libraries compute the slot locally, keep a slot to node mapping table in memory, and connect directly to the correct node. That table is only updated on a MOVED redirect, which signals that the cluster topology has changed, for example due to a resharding operation or a failover.

# CRC16 slot calculation, conceptually (Redis uses CRC16-CCITT
# with a specific 256-entry lookup table, simplified here)
def key_hash_slot(key: str) -> int:
    # Only the substring inside {} matters if hash tags are used
    key_for_hashing = extract_hash_tag(key) or key
    return crc16(key_for_hashing.encode()) % 16384

# Real-world verification against a running cluster
redis-cli -c -p 7000 cluster keyslot "session:user:9931"
# -> (integer) 3300

redis-cli -c -p 7000 cluster keyslot "session:user:9932"
# -> (integer) 11821   # different key, likely different slot,
#                       # likely a different node

A common misconception: CRC16 does not make similar keys land on the same slot, quite the opposite, the whole point is that even very similar keys such as user:1 and user:2 get mapped to different, randomly distributed slots in order to achieve an even load distribution across all nodes. If you deliberately want to force several keys onto the same node instead, you need hash tags, the topic of the next section.

4. Hash tags: controlling multi-key operations on purpose

Redis Cluster does not support multi-key operations such as MGET or transactions via MULTI/EXEC by default when the involved keys sit on different slots, because such an operation would need to involve several nodes at once, something Redis Cluster does not support architecturally. The solution is hash tags: if a key contains curly braces {...}, Redis Cluster uses only the part inside the braces for the slot computation, not the whole key.

With keys like {customer:4471}:orders and {customer:4471}:profile, Redis Cluster computes the hash slot of both keys purely from customer:4471, guaranteeing that both land on the same slot and therefore the same node. That makes MGET {customer:4471}:orders {customer:4471}:profile possible in a single call, as well as transactions and Lua scripts that operate on several related keys together, which would otherwise fail against the cluster constraint without hash tags.

# Without hash tags: keys likely land on different slots
redis-cli -c -p 7000 cluster keyslot "customer:4471:orders"
# -> (integer) 2145
redis-cli -c -p 7000 cluster keyslot "customer:4471:profile"
# -> (integer) 9903   # different slot, different node

# With hash tags: only the part inside {} is hashed
redis-cli -c -p 7000 cluster keyslot "{customer:4471}:orders"
# -> (integer) 6672
redis-cli -c -p 7000 cluster keyslot "{customer:4471}:profile"
# -> (integer) 6672   # identical slot, guaranteed same node

# Now multi-key operations work in a single call
redis-cli -c -p 7000 mget "{customer:4471}:orders" "{customer:4471}:profile"

# Transactions across related keys become possible
redis-cli -c -p 7000 <<'EOF'
MULTI
HSET {customer:4471}:profile status active
LPUSH {customer:4471}:orders order-88213
EXEC
EOF

A risk of overusing hash tags is uneven distribution: if too many keys share the same hash tag, load concentrates on a single slot and therefore a single node, undermining the whole purpose of sharding. Hash tags should therefore be applied deliberately to small, logically related groups of keys, such as all the data belonging to a single customer, not to entire entity types with millions of records.

5. Cluster topology: masters, replicas, and gossip

A Redis Cluster topology consists of several master nodes, each owning a subset of the 16384 slots, plus optional replica nodes, each replicating a specific master. If a master fails, the cluster can automatically promote one of its replicas, similar to Sentinel, but without separate Sentinel processes, since that logic is built directly into every cluster node. For production setups the rule of thumb is: at least three master nodes, each with at least one replica, spread across different hosts or availability zones.

Nodes communicate with each other over the cluster bus protocol, which is based on gossip. Every node periodically exchanges information about the known cluster state with a random subset of other nodes, which slots belong to whom, which nodes are reachable, which are considered failed. This cluster bus runs on a separate port, by default the client port plus 10000, so it must be reachable between all nodes in addition to the normal Redis port.

# redis.conf: cluster-mode node configuration
port 7000
cluster-enabled yes
cluster-config-file nodes-7000.conf
cluster-node-timeout 15000
cluster-require-full-coverage yes

# The cluster bus uses port 7000+10000 = 17000, must be open
# between all nodes in addition to the client port itself

# Inspect the gossip-derived view of the whole cluster
redis-cli -c -p 7000 cluster nodes
# <id> <ip:port@bus-port> <flags> <master> <ping-sent> <pong-recv>
# <config-epoch> <link-state> <slot> <slot> ...

The cluster-node-timeout parameter determines how long a node is tolerated as unreachable before it is considered failed and a failover is triggered. Too short a value causes unnecessary failovers during brief network blips, too long a value extends real downtime. In most production environments a value between 10 and 15 seconds has proven effective, depending on the stability of the underlying network.

6. Setting up a cluster: redis-cli --cluster create

Setting up a Redis Cluster starts by launching individual Redis instances in cluster mode, followed by joining these instances into a cluster. The redis-cli --cluster create command automatically handles the initial distribution of the 16384 slots across all specified master nodes and assigns replicas to masters so that no replica sits on the same host as its master, provided enough hosts are available.

For a setup with three masters and three replicas, six nodes in total, redis-cli --cluster create automatically splits the slots as evenly as possible, so each master owns around 5461 slots. This initial distribution can be adjusted at any time later through resharding, for example when nodes with different hardware capacity should carry an unequal number of slots.

# Start six Redis instances in cluster mode (ports 7000-7005)
for port in 7000 7001 7002 7003 7004 7005; do
  redis-server --port "$port" \
    --cluster-enabled yes \
    --cluster-config-file "nodes-${port}.conf" \
    --cluster-node-timeout 15000 \
    --daemonize yes
done

# Form the cluster: 3 masters, each with 1 replica
redis-cli --cluster create \
  10.0.2.10:7000 10.0.2.11:7001 10.0.2.12:7002 \
  10.0.2.13:7003 10.0.2.14:7004 10.0.2.15:7005 \
  --cluster-replicas 1

# Verify the cluster is healthy and fully covered
redis-cli -c -p 7000 cluster info
# cluster_state:ok
# cluster_slots_assigned:16384
# cluster_slots_ok:16384
# cluster_known_nodes:6
# cluster_size:3

After setup it is important to check with cluster info that cluster_state reads ok and cluster_slots_assigned shows exactly 16384. If the state reads fail, usually not all slots have been assigned, for example because a node was unreachable during setup. In that case the missing slots must be assigned manually via redis-cli --cluster fix before the cluster can be used in production.

7. Slot distribution and rebalancing basics

The initial slot distribution of Redis Cluster is rarely the final one. If part of the dataset grows faster than the rest, or new nodes with more capacity are added, the distribution of the 16384 slots needs to be adjusted. The redis-cli --cluster rebalance command automatically computes a more even distribution based on the current number of master nodes and moves slots accordingly, while redis-cli --cluster reshard is meant for targeted, manually defined moves of individual slot ranges.

Important to understand: the slot distribution does not have to be even. Redis Cluster explicitly allows assigning more slots to a more powerful node than to a weaker one, via the --cluster-weight parameter during rebalancing. That is particularly relevant with heterogeneous hardware, for example when a cluster has grown over years and newer nodes have more memory than the original ones.

# Automatic rebalance across all master nodes, weighted equally
redis-cli --cluster rebalance 10.0.2.10:7000

# Rebalance with custom weights (node with more RAM gets more slots)
redis-cli --cluster rebalance 10.0.2.10:7000 \
  --cluster-weight abc123def=2 \
  --cluster-weight 456ghi789=1

# Simulate the rebalance first without moving anything
redis-cli --cluster rebalance 10.0.2.10:7000 --cluster-simulate

# Check current slot distribution per node
redis-cli --cluster check 10.0.2.10:7000

Rebalancing is a live operation that can run while the cluster is serving traffic, but it causes extra network and CPU load on the affected nodes while data is being migrated. In production environments it is advisable to schedule rebalancing outside of peak load windows and to actively watch progress via cluster nodes, rather than letting the operation run unattended in the background.

8. Redirects: understanding MOVED and ASK

If a client asks a node for a key whose slot that node does not own, Redis Cluster responds with a MOVED redirect containing the correct node address for that slot. Cluster aware clients follow this redirect automatically and update their local slot table, so future requests for the same slot go directly to the right node without another detour.

During an in progress slot migration, for example during a resharding operation, a slot can temporarily sit partly on the old node and partly on the new one. During this transition period, the old node responds with an ASK redirect instead of MOVED if the requested key has already been migrated. The crucial difference: ASK does not permanently update the client's local slot table, it only signals where this single request should be redirected, because the migration is not yet fully complete.

# MOVED: permanent redirect, client should update its slot table
redis-cli -p 7000 get "order:99120"
# (error) MOVED 12539 10.0.2.12:7002

# Cluster-mode redis-cli follows redirects automatically with -c
redis-cli -c -p 7000 get "order:99120"
# -> "shipped"   # transparently redirected to 10.0.2.12:7002

# ASK: temporary redirect during an in-progress slot migration
# 1. Client must first send ASKING to the target node
redis-cli -p 7002 asking
redis-cli -p 7002 get "order:99120"
# (error) ASK 12539 10.0.2.13:7003   # if still mid-migration

Applications using their own, non cluster aware client library must implement this redirect logic themselves, otherwise access attempts fail during a resharding operation. That is why choosing a demonstrably cluster aware client library for Redis Cluster is not a minor detail, but one of the most important decisions when moving into sharded Redis.

9. Limits and pitfalls of Redis Cluster sharding

Despite all its advantages, Redis Cluster has clear architectural limits. Multi-key operations across keys that are not assigned to the same slot via a hash tag simply do not work, and that also applies to database selection via SELECT, which is completely disabled in cluster mode, all data effectively lives in database 0. Likewise, KEYS * and FLUSHALL must be run individually on every node, there is no central command that operates across the whole cluster.

Another pitfall concerns Lua scripts and transactions: both only work in Redis Cluster if all referenced keys sit on the same slot, which requires consistent use of hash tags in the application design from the outset. Anyone migrating an existing standalone system to Redis Cluster later on therefore often has to rework the key schema, not just swap out the infrastructure. Those migration costs should be factored into any sharding decision from the beginning.

10. Summary

Redis Cluster sharding fundamentals boil down to one core principle: 16384 fixed hash slots form the layer of indirection between keys and physical nodes. CRC16 deterministically computes the responsible slot for every key, while hash tags deliberately let you force several related keys onto the same slot, and therefore the same node, whenever multi-key operations or transactions are needed. Together these three mechanisms enable horizontal scaling without applications having to manage the mapping of keys to nodes themselves.

Anyone adopting Redis Cluster must plan from day one around its architectural constraints: no SELECT, multi-key operations only with matching hash tags, and a cluster-aware client as a prerequisite. The initial slot distribution from redis-cli --cluster create is rarely final, rebalancing and resharding are part of the normal operation of a growing cluster, not exceptional situations.

11. FAQ: Redis Cluster Sharding Fundamentals