Redis HyperLogLog for Approximate Counting

1. The problem: counting unique values in massive sets

The question "how many unique visitors did this page have today?" sounds trivial, but at high traffic it becomes a real memory problem. The naive solution, storing every visitor ID in a Redis set and calling SCARD, works correctly, but memory usage grows linearly with the number of unique visitors. At ten million unique IDs per day, each maybe 16 bytes, hundreds of megabytes per day add up quickly, multiplied by however many days you want to retain.

This is exactly where HyperLogLog comes in: a probabilistic data structure that estimates the cardinality, meaning the number of unique elements, of a set without storing the elements themselves. Instead of linearly growing memory, a HyperLogLog in Redis needs a constant roughly 12 kilobytes, regardless of whether the estimated set contains a thousand or a billion unique elements. The price for this memory saving is a controlled, mathematically known error margin, typically under one percent.

For use cases where the exact number is not critical but a reliable order of magnitude suffices, such as dashboards, analytics or A/B test evaluations, HyperLogLog is one of the most elegant solutions in the Redis toolbox. This post explains the commands, the math behind it at a conceptual level, and the concrete application for unique visitor counting.

An important distinction upfront: HyperLogLog does not replace a set for use cases where membership, removal of individual elements or absolute exactness are required. It solely solves the counting problem, and does so efficiently enough that systems with millions of parallel counts only become realistic on a single Redis instance because of HyperLogLog.

The name HyperLogLog comes from theoretical computer science and describes that memory needs grow with the double logarithm of the set being counted, practically not increasing measurably even as the actual set grows by orders of magnitude. This asymptotic property is the theoretical core that explains the constant memory size in practice.

2. PFADD and PFCOUNT: the basic commands

All Redis commands for HyperLogLog carry the prefix PF, a nod to Philippe Flajolet, the mathematician whose research founded the algorithm. PFADD key element adds an element to the estimated set. Internally the element gets hashed, and the hash affects one of 16384 registers that together form the internal state of the HyperLogLog. Multiple elements can be passed in a single PFADD call.

Calling PFADD key without any element simply ensures the key exists as a HyperLogLog, without changing the estimate. That is handy for creating a daily counter right at the day change, before the first actual visitor arrives, so downstream processes are guaranteed to find the key.

An empty HyperLogLog key reliably returns 0 for PFCOUNT, just like an empty set does for SCARD. This consistent behavior at the edges of the value range simplifies integration into existing counting pipelines, because no special cases need to be handled for "key does not exist" versus "key is empty".

PFCOUNT key returns the estimated number of unique elements that were ever added via PFADD. Important to understand: there is no command to remove a single element again, HyperLogLog is a pure add only structure. That is a deliberate tradeoff of the algorithm: compactness is bought by making individual elements no longer identifiable or removable after being added, only the overall estimate persists.

PFADD returns 1 if the internal register representation actually changed due to the new element, and 0 if not. This return value is a useful side effect: a client can tell whether an element was very likely new without calling a separate command, even though this behavior is no reliable substitute for a true membership check.

# HyperLogLog: approximate cardinality with constant memory
redis-cli PFADD visitors:2026-07-23 "user:1001" "user:1002" "user:1003"
redis-cli PFADD visitors:2026-07-23 "user:1001"
redis-cli PFCOUNT visitors:2026-07-23
redis-cli STRLEN visitors:2026-07-23
redis-cli PFADD visitors:2026-07-24 "user:2001" "user:2002"
redis-cli PFCOUNT visitors:2026-07-23 visitors:2026-07-24
redis-cli PFADD visitors:2026-07-24
redis-cli TYPE visitors:2026-07-24
redis-cli OBJECT ENCODING visitors:2026-07-24

3. The algorithm behind HyperLogLog explained simply

The core idea of HyperLogLog can be understood without deep math: every added element gets turned by a hash function into a seemingly random bit sequence. A central observation of probabilistic counting algorithms is that the probability of observing a long run of leading zeros in a random bit sequence drops exponentially with the length of that run. If a set of hash values shows an especially long run of leading zeros, that statistically suggests a large number of distinct elements, because the more distinct random values were generated, the more likely a long zero run eventually shows up.

HyperLogLog improves on this basic principle by not keeping just a single counter, but distributing incoming hash values across 16384 independent registers, based on part of the hash. Each register remembers the longest observed run of leading zeros within its share of the elements. In the end, a harmonic mean gets computed across all registers, which strongly dampens the variance of individual outlier registers and delivers a noticeably more stable overall estimate than a single counter could.

This register based averaging is why HyperLogLog, with only 16384 registers of 6 bits each, roughly 12 kilobytes total, delivers an estimate with a standard error of about 0.81 percent, entirely independent of the actual number of elements. Whether a million or a billion elements were added, the HyperLogLog's memory footprint stays exactly the same, only the statistical estimate adjusts.

The original Flajolet-Martin algorithm from the 1980s already used the idea of leading zeros, but suffered from high variance with a single counter. The HyperLogLog refinement by Flajolet and colleagues in 2007 added the split across many registers and the harmonic averaging, which brought the error margin down to a practical, predictable level and made the structure production ready.

Redis implements this algorithm in plain C directly inside the server process, so every PFADD operation needs no extra network round trips or external libraries. This native integration is a major reason HyperLogLog is so widespread in Redis, compared to similar implementations that would have to be realized as separate application logic on the client side.

Redis additionally uses a so called sparse encoding variant for small sets, where only the actually populated registers get stored compactly, instead of all 16384 registers at full size. Only once a set grows large enough that the sparse representation would be less efficient than the fixed dense representation does Redis convert automatically and transparently, similar to the listpack encoding of other data types.

4. Error margin and accuracy in practice

The standard error of about 0.81 percent in the Redis implementation of HyperLogLog concretely means: for an actual cardinality of one million unique elements, the estimate typically falls between 991,900 and 1,008,100. For dashboards, trend analysis or capacity planning, this deviation is generally irrelevant, because decisions rarely hinge on the last digit of a number.

It is important to evaluate the error margin in the context of the application: for billing systems, where every additional unique usage actually gets billed, an estimate with 0.81 percent error is unsuitable, an exact set must be used there. For analytics dashboards that already display rounded or aggregated numbers, for example "about 1.2 million visitors this week", the error margin is completely unproblematic and does not stand out visually.

A practical rule of thumb from operational experience: for metrics that are already presented rounded to full thousands or ten thousands, a deviation of 0.81 percent lies clearly below the rounding threshold and is never visible to end users. Only for metrics communicated down to the last digit, for example in a contract document, does the approximation become a real problem.

An interesting aspect: the error margin is relatively constant, regardless of the absolute size of the set. That distinguishes HyperLogLog from naive sampling methods, where the relative error tends to grow with an increasing population. HyperLogLog delivers similarly reliable percentage accuracy for a million as for a billion elements, as long as enough distinct hash values have fed into the registers.

For very small sets, in the low double or triple digit range, Redis uses a correction that falls back to linear counting, to improve the otherwise less accurate default estimate for small sets. This built in correction ensures HyperLogLog delivers reliable results even at the lower end of the value range, without application developers having to worry about this edge case.

5. Practical example: daily unique visitor tracking

A classic use case is daily tracking of unique visitors to a website or API. On every incoming request, the visitor ID, for example a cookie ID or IP hash, gets added via PFADD visitors: . At the end of the day, or in real time, PFCOUNT visitors: returns the estimated number of unique visitors for that day, without a single visitor ID ever having to be stored permanently.

For weekly or monthly views, the application does not need to recount: PFCOUNT accepts multiple keys at once and returns the combined cardinality across all supplied HyperLogLogs, without changing the individual daily structures for it. A call such as PFCOUNT visitors:2026-07-21 visitors:2026-07-22 visitors:2026-07-23 immediately returns the unique visitors combined across all three days, including correct deduplication of visitors active on multiple days.

This combinability makes HyperLogLog especially attractive for multidimensional analytics: a separate HyperLogLog per day, per country or per marketing channel enables flexible retrospective analysis, without having to decide in advance which aggregation levels will be needed later. This flexibility would hardly be practical with exact sets at a comparable data volume.

For retention, it is recommended to set each daily key with EXPIRE to a sensible retention window, for example 90 days, so old HyperLogLogs disappear automatically without needing a separate cleanup job. Since every key occupies a constant 12 kilobytes only, even retaining several years of daily counts is unproblematic for a Redis instance's memory footprint.

For production systems with high traffic, it is also worth batching PFADD asynchronously through a small queue or batching in the application code, instead of blocking synchronously on every single request. Since HyperLogLog operations are already very cheap, a direct, synchronous call is entirely sufficient in most cases though, without needing any additional infrastructure.

# Daily unique visitor tracking, combined across days
redis-cli PFADD visitors:2026-07-21 "cookie:a1" "cookie:a2" "cookie:a3"
redis-cli PFADD visitors:2026-07-22 "cookie:a2" "cookie:a4"
redis-cli PFADD visitors:2026-07-23 "cookie:a1" "cookie:a5"
redis-cli PFCOUNT visitors:2026-07-21
redis-cli PFCOUNT visitors:2026-07-21 visitors:2026-07-22 visitors:2026-07-23
redis-cli EXPIRE visitors:2026-07-21 2592000

6. PFMERGE: combining sets without losing precision

While PFCOUNT with multiple keys only returns a combined estimate without modifying the underlying HyperLogLogs, PFMERGE destkey sourcekey1 sourcekey2 creates a new, persistent HyperLogLog that represents the union of multiple source structures. That is useful when a combined view needs to be stored persistently and reused, for example a rolling 30 day window of unique users that gets extended by a new day and reduced by the oldest day daily.

An important mathematical advantage of PFMERGE over a naive summation of individual counts: if a user was active both on Monday and Tuesday, a simple addition of the daily PFCOUNT values would count that user twice. PFMERGE followed by PFCOUNT on the result deduplicates correctly, because the register based structure takes the maximum of the two source registers per register during the merge, which corresponds to the correct mathematical union operation.

A rolling window can be maintained efficiently by generating a new PFMERGE from the last seven daily keys every day and overwriting the old rolling key. This approach is significantly cheaper than rebuilding a set from seven days of raw data and delivers the same correct deduplication over the entire period.

Since PFMERGE overwrites the target object, production systems should use a temporary intermediate key and activate it only after a successful merge via RENAME. That way the previous rolling value stays retrievable until the last moment, in case the merge operation fails or gets interrupted for any reason.

# PFMERGE: build a persistent rolling 7-day unique count
redis-cli PFMERGE visitors:rolling7d:tmp visitors:2026-07-17 visitors:2026-07-18 visitors:2026-07-19 visitors:2026-07-20 visitors:2026-07-21 visitors:2026-07-22 visitors:2026-07-23
redis-cli RENAME visitors:rolling7d:tmp visitors:rolling7d
redis-cli PFCOUNT visitors:rolling7d
redis-cli EXPIRE visitors:rolling7d 604800

7. Memory efficiency compared directly to sets

The quantitative memory advantage of HyperLogLog becomes clear in a direct comparison with a set. A Redis set with a million unique string IDs of 20 bytes each needs, including the hash table's internal management overhead, typically several tens of megabytes of memory. A HyperLogLog for the same data volume occupies a constant roughly 12 kilobytes, regardless of the length or number of the original elements. That corresponds to a memory saving of a factor of a thousand or more.

This difference becomes especially relevant when many parallel counts are being maintained, for example one HyperLogLog per combination of day, country and device type. Where thousands of sets with millions of entries would blow the available memory of a Redis instance, thousands of HyperLogLogs with their constant 12 kilobytes fit easily into a fraction of the memory a single large set would need.

The compactness also pays off for replication and persistence snapshots: an RDB snapshot with thousands of HyperLogLog keys stays manageably small, while the same number of full sets would noticeably bloat the snapshot and significantly extend the time for BGSAVE as well as the initial replication sync.

Network bandwidth benefits too: while SMEMBERS on a large set potentially transfers megabytes of data over the network, PFCOUNT always returns just a single number, regardless of the estimated cardinality. That makes HyperLogLog resource friendly even for very frequent live queries, for example for a real time dashboard.

8. Limits: when HyperLogLog is the wrong choice

HyperLogLog is not a universal solution for every counting problem. If an exact number is required, for example for invoicing, legal evidence or hard capacity limits, a set or another exact counting mechanism is mandatory. HyperLogLog is equally unsuitable when individual elements need to be identified or removed again later, because the structure stores no elements, only a statistical summary.

For very small sets, say under a hundred elements, HyperLogLog is hardly worth it, because the fixed 12 kilobyte memory footprint is in that case larger than a simple set with the same few elements. HyperLogLog plays to its strength only at large to very large set sizes, where the constant memory footprint clearly beats the linearly growing footprint of a set.

Security critical counts too, for example detecting brute force login attempts with a hard lockout threshold, make an approximation risky, because an error of one percent in the wrong direction could, in doubt, make the difference between a correctly blocked attack and a request let through. In such cases, an exact counter, for example via INCR on a string, is the right choice.

Another edge case involves very small but very numerous parallel counts: thousands of HyperLogLogs with just a few elements each add up to a relevant total memory footprint despite compact sparse encoding. In such cases it can be cheaper to bundle multiple small counting contexts into a shared hash with a bitmap solution, instead of creating a dedicated HyperLogLog for every single context.

9. HyperLogLog compared to set and bitmap

The table below compares HyperLogLog to the two related approaches, set and bitmap, for counting and membership problems.

Choosing between the three structures depends on the accuracy requirement and the expected order of magnitude. HyperLogLog wins whenever an approximation is sufficient and the set is large enough that the constant memory footprint pays off against linearly growing alternatives. In practice, all three structures are frequently used side by side, depending on which requirement takes priority for a given counter.

10. Summary

Redis HyperLogLog solves a very specific problem: estimating the cardinality of massive sets with a constant, minimal memory footprint. PFADD adds elements, PFCOUNT queries the estimate, PFMERGE combines multiple HyperLogLogs persistently and correctly deduplicated. Memory usage stays around 12 kilobytes per structure, entirely independent of whether a thousand or a billion elements were added, with a typical standard error of about 0.81 percent.

For exact requirements such as billing or legal evidence, a set remains the right choice. For analytics, dashboards and capacity planning, where a reliable order of magnitude suffices, HyperLogLog is one of the most efficient data structures Redis offers, typically saving several orders of magnitude of memory at scale compared to exact set based counting.

Anyone introducing HyperLogLog into an existing analytics system should start with non-critical metrics, such as page views or campaign reach, and check the error margin there against the real precision requirement. Once it turns out that 0.81 percent deviation is not noticeable in practice, the migration can be extended step by step to further counters, without risking accidentally losing exactness at a critical point.

11. FAQ: Redis HyperLogLog