Active Defrag in Redis: Automatically Reducing Memory Fragmentation in the Background

1. How memory fragments despite deleted keys

Redis does not manage its memory byte by byte on its own, but delegates the actual memory allocation to an allocator such as jemalloc, which hands out memory in blocks of fixed size classes. When a small object is requested, it gets placed into a matching size class, for example 64 bytes or 128 bytes, even if the actual object is smaller. That significantly speeds up allocation and deallocation, but means memory is not managed as one arbitrarily fine-grained contiguous space.

Once many objects of varying sizes are created and deleted in changing order, for example through constantly rotating session data or a cache with high turnover, gaps appear within the memory pages reserved by the allocator. Those gaps are marked free and available for new allocations of the same size class, but cannot be reclaimed by the operating system as long as any other still-alive object sits within the same memory page. The result is a used_memory value that barely drops, or does not drop at all, despite the actual data volume shrinking.

2. Measuring fragmentation: mem_fragmentation_ratio

The key indicator for memory fragmentation is the mem_fragmentation_ratio field in the output of INFO Memory. It describes the ratio between the memory actually assigned to the Redis process by the operating system (resident set size) and the memory Redis itself reports as logically needed. A value of 1.0 means no measurable fragmentation, a value of 1.5 shows that Redis occupies fifty percent more physical memory than would actually be needed for the data.

Values well above 1.5 count as a warning sign and should prompt enabling active defrag or reviewing its tuning. The reverse case is also worth noting: a ratio well below 1.0 does not indicate efficient memory use, it usually means Redis needs more memory than the operating system is currently providing, for example because part of the memory has been swapped out, which is a separate and more serious problem.

redis-cli INFO memory | grep -E "used_memory:|used_memory_rss:|mem_fragmentation_ratio"
# used_memory:2147483648
# used_memory_rss:3221225472
# mem_fragmentation_ratio:1.50

3. How the active defrag mechanism copies data incrementally

Active defrag runs as a periodic background cycle inside Redis's main thread, deliberately designed to split its work into small, interruptible steps rather than blocking the server for one continuous stretch of time. In each cycle, Redis examines a limited number of keys, checks based on internal allocator statistics whether the underlying memory object is heavily fragmented, and, if needed, copies it into a freshly allocated, more compact memory location.

After copying, the old, fragmented memory region is freed, which lets the allocator fully reclaim contiguous memory pages and eventually return them to the operating system. Because this process runs in many small steps and repeatedly hands control back to regular command processing between steps, latency for normal client requests stays practically unchanged during defragmentation, in stark contrast to a hypothetical one-off, full compaction pass.

4. Enabling active defrag and its prerequisites

Active defrag is disabled by default and must be explicitly enabled via activedefrag yes, either in redis.conf or at runtime through CONFIG SET. One important technical prerequisite: the mechanism only works reliably with jemalloc as the memory allocator, because it relies on its internal per-page fragmentation statistics. If Redis is compiled with libc as the allocator, which happens in some minimal container images, active defrag is not available.

Anyone unsure which allocator is active can find that information directly in the output of INFO Server under the mem_allocator field. Official Redis Docker images use jemalloc by default, so active defrag is ready to use there without any extra steps once it is configured.

# Check the allocator
redis-cli INFO server | grep mem_allocator
# mem_allocator:jemalloc-5.3.0

# Enable active defrag at runtime
redis-cli CONFIG SET activedefrag yes

# Permanently in redis.conf
activedefrag yes

5. Thresholds: when active defrag kicks in

Active defrag does not run continuously, it only starts once measured fragmentation crosses certain configurable thresholds. active-defrag-ignore-bytes sets an absolute minimum amount of fragmented memory below which a defrag pass is not yet worthwhile, 100 megabytes by default. active-defrag-threshold-lower defines the percentage fragmentation level at which active defrag starts working at all, ten percent by default.

active-defrag-threshold-upper defines the fragmentation level at which Redis works at its maximum configured intensity, 100 percent by default. Between these two thresholds, the aggressiveness of the defrag process scales linearly: the higher the measured fragmentation, the more CPU time Redis invests per cycle in copying, until the upper limit is reached and the configured maximum intensity applies.

# Adjusting the thresholds
redis-cli CONFIG SET active-defrag-ignore-bytes 50mb
redis-cli CONFIG SET active-defrag-threshold-lower 5
redis-cli CONFIG SET active-defrag-threshold-upper 80

6. Tuning the CPU limits of the defrag process

Two further parameters directly control how much CPU time active defrag is allowed to consume per second: active-defrag-cycle-min sets the minimum CPU usage the defrag process applies even at low fragmentation, one percent by default. active-defrag-cycle-max caps the maximum CPU usage at very high fragmentation, 25 percent by default. These limits determine the trade-off between fast cleanup and the CPU capacity left over for actual command processing.

In environments with tight CPU capacity, for example smaller cloud instances that already operate close to their CPU limit, a lower active-defrag-cycle-max value is advisable, to avoid defrag activity competing with actual application traffic for CPU time. On generously sized hardware with plenty of spare CPU headroom, a higher value can reduce fragmentation considerably faster without production requests noticeably suffering.

# Conservative tuning for CPU-constrained environments
redis-cli CONFIG SET active-defrag-cycle-min 1
redis-cli CONFIG SET active-defrag-cycle-max 15

# More aggressive tuning with ample spare CPU capacity
redis-cli CONFIG SET active-defrag-cycle-min 5
redis-cli CONFIG SET active-defrag-cycle-max 50

7. When active defrag pays off and when it does not

Active defrag pays off especially for workloads with a high write and delete frequency on keys of varying sizes, for example session storage with constantly rotating users, short-lived rate-limiting counters, or cache layers with frequent invalidation. For mostly static datasets that rarely change, hardly any relevant fragmentation accumulates, so the extra CPU consumption from active defrag brings no measurable benefit.

In a Magento setup using Redis as a session and full page cache backend, the write and delete frequency is typically high enough to benefit from active defrag, especially during peak sales periods with many concurrent sessions and frequent cache invalidation after price or stock changes. Enabled active defrag with moderate tuning prevents memory usage from continuously growing over weeks there, without the actual data volume increasing to the same degree.

8. Limits of active defrag and alternative measures

Active defrag reduces fragmentation but does not fundamentally prevent it, and under extremely high write load, fragmentation can accumulate faster than the process can reduce it with the configured CPU limits. In such cases, mem_fragmentation_ratio stays persistently elevated even with active defrag enabled, and a higher active-defrag-cycle-max is often the most effective next step, provided available CPU capacity allows for it.

For cases where active defrag is insufficient or unavailable for compatibility reasons, for example with an allocator other than jemalloc, a manual, controlled restart of the Redis instance followed by loading from an RDB snapshot or an AOF log remains the last resort, which rebuilds the entire memory space fresh and without any fragmentation. That does come with brief downtime, though, and should only be used as a planned maintenance measure, not as a substitute for ongoing defrag tuning.

9. Practical conclusion: moderate tuning as a good starting point

For most production Redis instances running jemalloc as the allocator, enabling active defrag with default values is already a sensible first step that continuously reduces memory fragmentation without noticeable latency side effects. Regularly monitoring mem_fragmentation_ratio through INFO Memory reliably shows whether the default values are sufficient or whether the CPU limits need further adjustment.

Anyone who continues to observe persistently high fragmentation values after enabling the mechanism should first gradually raise active-defrag-cycle-max before considering more drastic measures such as a planned restart. The combination of correctly configured jemalloc, active defrag, and regular monitoring reliably covers the large majority of fragmentation problems in production Redis environments.

10. Summary

11. FAQ: Active Defrag