Understanding and Fixing Memory Fragmentation

1. What Memory Fragmentation Actually Means

Memory fragmentation describes the difference between the physical memory the operating system allocates to a Redis process and the memory Redis actually needs for the stored data. This difference does not arise from a bug in Redis itself, but from how the underlying memory allocator works: when objects of varying sizes are repeatedly created and freed, gaps appear in memory that are too small for new, larger objects but do not get merged unless they lie free contiguously.

The effect is real and measurable: a Redis process with 10 GB of actual data can easily hold 15 GB or more of physical memory from the operating system, without any classic memory leak being present. Those extra 5 GB are not lost, they exist as fragmented blocks that are hard for the allocator to reuse. In operational terms, that means higher RAM requirements than the raw dataset would suggest, which is frequently overlooked in capacity planning.

It is important to distinguish this from an actual memory leak: with fragmentation, the used_memory value reported by Redis itself stays stable or grows proportionally to the dataset, while the used_memory_rss value actually held by the operating system grows disproportionately. A real leak, in contrast, shows continuously growing used_memory, independent of the actual data volume. This distinction is the first step of any diagnosis.

2. Reading mem_fragmentation_ratio Correctly

The central metric for memory fragmentation is mem_fragmentation_ratio, available via redis-cli INFO memory. It is calculated as the quotient of used_memory_rss, the physical memory actually assigned by the operating system, and used_memory, the memory Redis itself reports as needed. A value of 1.0 would mean both values match exactly, which practically never happens.

Values between 1.0 and about 1.5 are generally considered normal and unproblematic, because moderate overhead from allocator metadata and usual fragmentation effects is unavoidable. Values noticeably above 1.5, especially above 2.0, indicate significant fragmentation deserving attention. What matters most is not the absolute value at a single point in time, but the trend over time: a steadily rising mem_fragmentation_ratio, correlated with certain workload patterns such as frequent resizing of values, is a much more reliable signal than a one-off snapshot.

An often overlooked special case is a mem_fragmentation_ratio below 1.0. That means Redis is requesting more memory than needed, but parts of it have been swapped out by the operating system, a considerably more serious problem than regular fragmentation, because swap access can increase Redis operation latency by orders of magnitude. A value below 1.0 should therefore be treated immediately as an alarm signal for insufficient physical memory.

# Read the core fragmentation metrics
redis-cli INFO memory | grep -E "used_memory:|used_memory_human|used_memory_rss|mem_fragmentation_ratio|mem_allocator"

# Example output:
# used_memory:10737418240        (10.0 GB actually needed by Redis)
# used_memory_human:10.00G
# used_memory_rss:16106127360    (15.0 GB physically resident)
# mem_fragmentation_ratio:1.50
# mem_allocator:jemalloc-5.3.0

# A ratio below 1.0 signals swapping, not fragmentation - check immediately
redis-cli INFO memory | grep -E "mem_fragmentation_ratio"
free -h

3. How jemalloc Actually Allocates Memory

Redis uses jemalloc as its default memory allocator instead of the standard glibc allocator, precisely because jemalloc was designed to address this very fragmentation problem. jemalloc organizes memory into so-called size classes: fixed, predefined size buckets that every allocation gets rounded up to. A request for 130 bytes, for example, lands in a size class of 144 bytes instead of allocating exactly 130 bytes, which creates internal overhead but massively simplifies reusing freed blocks within the same size class.

This size-class strategy substantially reduces external fragmentation, because freed blocks of one size class can be reused directly for new allocations of the same class without the allocator having to merge complex memory regions. The problem thereby shifts from external to internal fragmentation: memory reserved within an oversized size class for a smaller actual value. For workloads with strongly varying value sizes, particularly with frequent overwriting of strings of differing lengths, this internal fragmentation can still become noticeable.

jemalloc additionally organizes memory into arenas, independent memory regions that allow parallel allocations from multiple threads without mutual locking. For Redis, which operates primarily single-threaded, the multi-arena property is less relevant than for heavily parallel applications, but it still influences how memory gets organized internally and actually returned to the operating system when freed, which in turn affects the measured mem_fragmentation_ratio.

# Ask Redis for a human-readable memory diagnosis
redis-cli MEMORY DOCTOR
# "High allocator fragmentation" or "Peak memory: OK" etc.

# Inspect allocator-level stats directly (jemalloc-specific)
redis-cli MEMORY STATS | grep -E "allocator.allocated|allocator.active|allocator.resident"
# allocator.allocated  -> bytes jemalloc has handed out to Redis
# allocator.active     -> bytes in active jemalloc pages (includes internal frag)
# allocator.resident   -> bytes physically resident (includes unreturned pages)

4. Common Causes of Rising Fragmentation

The most common trigger for rising memory fragmentation is a workload with frequent resizing of existing values, for example when a string value gets repeatedly overwritten with content of differing length. Every resize can require a new allocation in a different size class, while the old memory block gets freed but potentially cannot be immediately reused for other purposes unless a matching new allocation of the same size follows.

A second common trigger is a high TTL churn rate: many short-lived keys that get continuously created and then deleted via expiration. At a very high rate, the allocator cannot keep up quickly enough to identify and merge contiguous free regions, which causes fragmentation to rise over time, even when the net dataset stays stable.

A third, less commonly discussed factor is a generously sized maxmemory combined with an aggressive eviction policy. When Redis frequently operates near the memory limit and constantly evicts keys to make room for new ones, a permanent pattern of freeing and reallocating develops, which structurally favors fragmentation compared to a setup with sufficient headroom between the actual dataset and the configured limit.

5. Active Defrag: Mechanics and Configuration

Since Redis 4, Active Defrag is a built-in mechanism that reorganizes fragmented memory blocks while the server is running, without requiring a restart. Enabled via activedefrag yes, Active Defrag runs as a background process that cyclically checks keys, compares their current memory position with a more ideal position, and moves the value to the new position when there is significant improvement, while freeing the old, fragmented memory.

The decisive design aspect of Active Defrag is its incremental, resource-conscious approach: instead of reorganizing the entire dataset in one pass, which would block the main process, Active Defrag works in small time slices between processing regular client commands. That keeps latency for regular operations largely unaffected, while fragmentation gradually decreases over a longer period, instead of during a single, potentially disruptive operation.

# redis.conf - Enable and configure Active Defrag

# Enable the background defragmentation process
activedefrag yes

# Minimum fragmentation ratio before defrag starts working
active-defrag-ignore-bytes 100mb
active-defrag-threshold-lower 10

# Maximum fragmentation ratio: defrag runs at full configured effort
active-defrag-threshold-upper 100

# CPU usage bounds for the defrag cycle (percent of one core)
active-defrag-cycle-min 5
active-defrag-cycle-max 75

6. Tuning Active Defrag Parameters in Detail

active-defrag-threshold-lower defines the fragmentation percentage at which Active Defrag starts working at all, 10 percent above the theoretical ideal by default. Below this threshold, Active Defrag stays idle, because the effort of reorganizing does not justify the benefit at low fragmentation. active-defrag-threshold-upper defines the point at which Active Defrag works at maximum configured intensity, because fragmentation is then considered critical enough to justify more resources.

The two CPU parameters active-defrag-cycle-min and active-defrag-cycle-max bound the share of one CPU core's time Active Defrag may consume, as lower and upper percentages. A too-low active-defrag-cycle-max value makes defragmentation progress unnecessarily slowly under heavily fragmented memory, while a too-high value can cause noticeable latency impact on regular client requests, because defrag and regular commands compete for the same CPU time.

active-defrag-ignore-bytes prevents Active Defrag from becoming active for very small absolute amounts of memory, even if the percentage fragmentation value appears high. With a tiny dataset of a few megabytes, a high percentage can still mean a negligible absolute memory amount, which is why this threshold avoids unnecessary defrag cycles for irrelevant data volumes.

7. Monitoring Fragmentation and Setting Alerts

Continuous monitoring of mem_fragmentation_ratio belongs in every production Redis setup, ideally with alerting on sustained threshold breaches over a period of time, not on a single snapshot that could be skewed by short-lived load spikes. An alert firing for a single, seconds-long breach of 1.5 creates unnecessary noise, while a sustained high value over several hours represents a genuine operational signal.

In addition to mem_fragmentation_ratio, it is worth watching allocator_frag_ratio, which since newer Redis versions distinguishes between jemalloc-internal fragmentation and the overall RSS difference measured through process overhead. This finer breakdown helps distinguish between an actual jemalloc fragmentation problem and other causes of elevated RSS usage, such as fork-related copy-on-write during a BGSAVE, which might otherwise be misinterpreted as regular fragmentation.

#!/usr/bin/env bash
# check_fragmentation.sh - Alert on sustained high fragmentation
set -euo pipefail

RATIO=$(redis-cli INFO memory | grep mem_fragmentation_ratio | cut -d: -f2 | tr -d '\r')
THRESHOLD=1.5

if (( $(echo "$RATIO < 1.0" | bc -l) )); then
  echo "[CRITICAL] Fragmentation ratio ${RATIO} below 1.0, possible swapping" >&2
  exit 2
fi

if (( $(echo "$RATIO > $THRESHOLD" | bc -l) )); then
  echo "[WARN] Fragmentation ratio ${RATIO} exceeds ${THRESHOLD}" >&2
  exit 1
fi

echo "[OK] Fragmentation ratio: ${RATIO}"

8. Defrag vs. Restart: Making the Right Call

Active Defrag does not fully solve every fragmentation problem. With extreme, long-accumulated fragmentation, for example after months of varying workloads without ever restarting, Active Defrag can still achieve improvements but may not always reduce fragmentation to an ideal level, because certain memory structures can only be reorganized to a limited extent with the incremental approach. In such cases, a controlled restart, where Redis reloads the entire dataset fresh from an RDB or AOF file and allocates fresh, unfragmented memory, remains the more reliable solution.

The choice between Active Defrag and a restart depends on the acceptable maintenance window: Active Defrag works without downtime, but gradually over hours to days, depending on the configured CPU intensity. A restart takes effect immediately but requires a maintenance window or a failover to a replica to avoid downtime. For setups with replication, a rolling restart, first the replica, then failover, then restarting the former primary, is often the most pragmatic way to fully eliminate fragmentation without noticeable downtime.

#!/usr/bin/env bash
# rolling_restart_defrag.sh - Clear fragmentation without downtime
set -euo pipefail

REPLICA_HOST="redis-replica-01"
PRIMARY_HOST="redis-primary-01"

echo "Step 1: restart the replica (loads a fresh, defragmented copy)"
ssh "$REPLICA_HOST" "systemctl restart redis"
sleep 10

echo "Step 2: promote the replica to primary"
redis-cli -h "$REPLICA_HOST" REPLICAOF NO ONE

echo "Step 3: point application traffic to the new primary (external step)"

echo "Step 4: restart the old primary, now safe to reload as fresh replica"
ssh "$PRIMARY_HOST" "systemctl restart redis"
redis-cli -h "$PRIMARY_HOST" REPLICAOF "$REPLICA_HOST" 6379

echo "[OK] Rolling restart complete, fragmentation cleared on both nodes"

9. Common Mistakes in Fixing Fragmentation

A common mistake is enabling activedefrag with maximum CPU values without first measuring the effect on regular request latency. On instances with already high CPU utilization, an overly aggressively configured Active Defrag can produce noticeable latency spikes that undermine the original goal of stable performance. Gradually raising active-defrag-cycle-max while watching latency is the safer path than jumping straight to high values.

A second mistake is prematurely interpreting a high mem_fragmentation_ratio as a memory leak and panic-restarting without first distinguishing between fragmentation and an actual leak. This confusion leads to unnecessary downtime, while the actual problem, if it really is a leak, gets only temporarily masked by the restart rather than fixed, and typically recurs.

A third mistake concerns capacity planning: setting maxmemory exactly to the size of the expected dataset without planning headroom for fragmentation overhead can put Redis under memory pressure even though the raw dataset stays within the limit, because actual RSS usage from fragmentation runs noticeably higher than used_memory alone.

10. Summary

Memory fragmentation is the difference between physically held and actually needed memory, measured via mem_fragmentation_ratio. Values between 1.0 and 1.5 are normal, values below 1.0 signal critical swapping, values above 2.0 justify active intervention. The jemalloc allocator reduces external fragmentation through size classes, but partially shifts the problem to internal fragmentation with strongly varying value sizes.

Active Defrag, enabled via activedefrag yes, reorganizes fragmented memory during live operation in a resource-conscious way, without downtime, but gradually over time. With extreme, long-accumulated fragmentation, a controlled, ideally rolling restart via a replica remains the more reliable solution. Continuous monitoring with alerting on sustained high values, rather than single snapshots, is the foundation of any sustainable memory fragmentation strategy.

11. FAQ: Understanding and Fixing Memory Fragmentation