Understanding OBJECT ENCODING: Listpack vs. Hashtable and the Automatic Switchover Point

1. Why Redis has multiple internal encodings for the same data structure

Redis data types like hash, list, set, and sorted set are uniform at the command level, but internally Redis stores them in different physical representations depending on their size and content. The reason lies in memory usage: a full hashtable with hash buckets, pointers, and collision chains is efficient for very many entries, since it offers constant access time independent of element count, but it incurs noticeable per-element memory overhead from pointers and internal bookkeeping structures.

For small collections with only a few elements, this overhead often clearly outweighs the actual payload. A hash with three short fields would, in classic hashtable encoding, consume more memory for pointers and bucket structures than for the actual data. Redis addresses this with a more compact alternative encoding that minimizes overhead for small collections, at the cost of linear rather than constant access time.

2. The basic principle of listpack: compact, sequential storage

A listpack stores all elements of a collection sequentially, one after another, in a single contiguous memory block, without separate pointers between elements. Each element consists of a compact encoding of its length, followed by the actual value and a backward-length field that allows the listpack to be traversed in reverse as well, without needing separate index structures. For a hash, key and value are stored directly adjacent within the same listpack; for a sorted set, member and score correspondingly.

Accessing a specific element in a listpack requires a linear scan from the start of the block, since there are no random-access structures. For a small number of elements, this linear scan barely matters in practice, because modern CPUs handle sequential memory access very efficiently through cache locality, often faster than chasing several scattered pointers in a hashtable with a comparably small element count.

# Create a small hash and check its encoding
HSET user:100 name Alice age 30 city Berlin
OBJECT ENCODING user:100
# -> "listpack"

# Create a small sorted set and check its encoding
ZADD leaderboard:daily 100 user:1 95 user:2 80 user:3
OBJECT ENCODING leaderboard:daily
# -> "listpack"

3. The switchover point to classic hashtable encoding

As soon as a collection exceeds one of the configured thresholds, either the maximum element count or the maximum size of a single element, Redis automatically and permanently converts its encoding to classic hashtable encoding. This conversion is one-directional: a hash that has switched to hashtable encoding never switches back to listpack, even if elements are subsequently deleted and the collection shrinks back below the thresholds. Redis deliberately does not optimize for this reverse case, in order to keep the encoding logic simple and avoid unnecessary conversion operations when element counts fluctuate.

For hashes, hash-max-listpack-entries and hash-max-listpack-value control this threshold, for sets set-max-listpack-entries and set-max-listpack-value, for sorted sets zset-max-listpack-entries and zset-max-listpack-value, and for lists list-max-listpack-size. As soon as a single element exceeds the configured maximum element size, for instance a single field value of a hash, Redis converts the encoding even if the overall element count is still small.

# Show current thresholds for hash encoding
CONFIG GET hash-max-listpack-entries
CONFIG GET hash-max-listpack-value

# Adjust thresholds (in redis.conf or at runtime)
CONFIG SET hash-max-listpack-entries 128
CONFIG SET hash-max-listpack-value 64

4. Default values in practice: 128 elements, 64 bytes

In the default configuration, the threshold for element count on hashes, sets, and sorted sets is usually 128 entries, and the threshold for the maximum size of a single element is 64 bytes. These values are deliberately conservative, providing a good compromise between memory savings and access speed for typical use cases without users needing to touch the configuration at all.

For use cases with very many small but not tiny collections, it is worth deliberately revisiting these default values. If, for instance, a hash typically has around 150 short fields, the default of 128 narrowly misses listpack encoding, even though the collection would still be small enough content-wise to benefit from the memory advantages. A moderate increase to 200 could bring noticeable memory savings in this case without meaningfully degrading access speed in practice.

5. Measuring memory savings concretely: MEMORY USAGE compared

The actual memory difference between listpack and hashtable encoding can be measured directly with the MEMORY USAGE command by creating the same logical data structure once just under and once just over the configured threshold and comparing the reported memory usage. In practice, small hashes often show savings of well over half compared to classic hashtable encoding, depending on the number of fields and the length of the stored values.

This measurement should always be done with realistic sample data taken from the actual application before adjusting the thresholds, since the actual savings depend heavily on field name length, value length, and the element count of the specific collection, and cannot be assumed to generalize across every application.

# Measure the memory usage of a hash directly
HSET session:abc token xyz789 user_id 42 role admin
MEMORY USAGE session:abc
# -> (integer) 96  (listpack encoding)

# Artificially grow the same hash past the threshold
CONFIG SET hash-max-listpack-entries 2
HSET session:abc extra_field value
OBJECT ENCODING session:abc
# -> "hashtable"
MEMORY USAGE session:abc
# -> noticeably higher value due to hashtable overhead

6. Practical consequence for data modeling

The central practical consequence of the listpack mechanism is that many small collections are more memory-efficient than few large ones, as long as the access pattern allows it. Instead of a single huge hash with hundreds of thousands of fields, which inevitably ends up hashtable-encoded, splitting the data into many smaller, topically grouped hashes that each stay below the threshold can consume noticeably less memory overall, for instance through sharding by category or time window instead of a single global collection hash.

For a Magento setup, this applies to things like session data, product attribute caches, or per-store configuration values: if each logical unit, such as each session or each product, uses its own small hash, it will generally stay well below the default thresholds and permanently benefit from listpack encoding, while a single global hash spanning all sessions would inevitably switch to the more expensive hashtable encoding.

7. Special case lists: quicklist as a chain of listpacks

For Redis lists, the encoding logic works somewhat differently than for hashes, sets, and sorted sets. A small list is also stored as a single listpack, but once the list grows past the configured threshold list-max-listpack-size, Redis does not switch directly to a classic linked list, but to a quicklist, a linked structure whose individual nodes are themselves listpacks.

This intermediate step combines the advantages of both approaches: within a single quicklist node, memory overhead stays low thanks to compact listpack encoding, while chaining multiple nodes allows the list to grow across arbitrarily many elements without any single memory block having to grow unboundedly. For very long lists, such as queues with thousands of entries, this behavior is largely transparent and requires no special handling in application code.

8. Monitoring in production: keeping an eye on encoding distribution

For a running system, it is worth occasionally monitoring what share of keys under a given prefix actually still uses listpack encoding. Since OBJECT ENCODING only works for a single key at a time, an aggregated overview calls for a small script that iterates over all keys of a prefix via SCAN and collects each key's encoding, rather than using KEYS, which can block the server on large data sets.

A sudden increase in the share of hashtable-encoded keys within a data class that should normally stay small is often an early indicator of a changing usage pattern in the application, such as growing product attribute lists or an unexpectedly high number of session fields, and should prompt a fresh review of both data modeling and the configured thresholds.

#!/bin/bash
# Determine encoding distribution for all keys with prefix "session:"
redis-cli --scan --pattern 'session:*' | while read -r key; do
    redis-cli OBJECT ENCODING "$key"
done | sort | uniq -c

9. Limits of the mechanism: no substitute for sound data modeling

As useful as automatic listpack encoding is for memory planning, it is no substitute for thoughtful data modeling. A data model that inherently requires very many elements per key, such as a global product database with hundreds of thousands of entries in a single sorted set, does not benefit from the encoding threshold, because hashtable encoding is unavoidable there anyway and content-wise also the right choice, since it offers constant instead of linear access time for large element counts.

The encoding threshold should therefore mainly be understood as an optimization lever for collections whose size is deliberately kept small or can reasonably be kept small, not as a reason to artificially force small collections where a larger, cohesive structure would be the better content-wise choice. The decision between many small and few large collections should primarily be guided by the application's access pattern, with storage encoding being a secondary, but measurably effective, optimization factor.

10. Summary

11. FAQ: Understanding OBJECT ENCODING: The Essentials at a Glance