Cardinality Aggregation: Interpreting Approximate Counts Correctly

1. Why Exact Unique Counts Are Expensive

An exact count of unique values, for example "how many different users visited this page", naively requires storing every single value seen so far in full in order to detect duplicates. For a field with a few thousand unique values that is trivial; for a field with hundreds of millions of unique values, such as session IDs on a high-traffic website, an exact count would need gigabytes of intermediate storage per aggregation request, spread across all involved shards.

Elasticsearch solves this problem with the cardinality aggregation, which returns a probabilistic estimate instead of an exact count. Instead of storing every value in full, the aggregation uses an algorithm that computes a hash from each value and draws statistical conclusions about the total number of unique values from it. Memory usage stays nearly constant regardless of whether there are a thousand or a billion unique values, a fundamental difference from any exact counting method.

2. HyperLogLog++ Overview

The algorithm underlying the cardinality aggregation is called HyperLogLog++, a variant of the original HyperLogLog algorithm improved by Google. The basic principle: for every aggregated value, a hash is computed whose bit pattern provides statistical information about the distribution of unique values. Specifically, the algorithm observes how many leading zeros the hash values tend to have on average; the more unique values there are, the more likely a hash with many leading zeros is observed. From this statistical observation, the total number of unique values can be reconstructed with remarkable accuracy, without ever fully storing a single value.

HyperLogLog++ improves on the original algorithm particularly for small and medium cardinalities, where plain HyperLogLog estimation becomes inaccurate. For small value sets below an internal threshold, HyperLogLog++ internally switches to a linear count with exact values and only transitions to probabilistic estimation for larger sets. This hybrid approach ensures that the cardinality aggregation produces fewer outliers in the practically relevant range than the purely academic HyperLogLog variant.

POST /web_logs/_search
{
  "size": 0,
  "aggs": {
    "unique_visitors": {
      "cardinality": {
        "field": "session_id.keyword"
      }
    }
  }
}
// Response excerpt:
// "aggregations": {
//   "unique_visitors": { "value": 4837291 }
// }
// Approximate count via HyperLogLog++, not an exact distinct count

3. Understanding and Setting precision_threshold

The precision_threshold parameter directly controls how much memory the HyperLogLog++ algorithm uses per aggregation, and thereby how accurate the result is. Specifically, precision_threshold specifies the number of unique values up to which the cardinality aggregation returns practically exact results. Above this threshold, the relative error rises slowly but stays within a predictable range even at very high cardinalities. The default value is 3000, which already delivers sufficiently precise results for most use cases with moderate cardinality.

Increasing precision_threshold improves accuracy but simultaneously increases memory usage per aggregation bucket linearly, up to an internal maximum of 40000. This trade-off becomes particularly relevant when a cardinality aggregation runs as a sub-aggregation inside a terms aggregation with many buckets: every single bucket gets its own HyperLogLog++ structure, so memory usage multiplies with the number of buckets, not just with the chosen precision_threshold.

POST /web_logs/_search
{
  "size": 0,
  "aggs": {
    "unique_visitors": {
      "cardinality": {
        "field": "session_id.keyword",
        "precision_threshold": 10000
      }
    }
  }
}
// precision_threshold trades memory for accuracy
// Values up to 10000 unique entries are near-exact

4. Error Rate and Memory Usage in Proportion

The practical error rate of the cardinality aggregation at a default precision_threshold of 3000 is typically below 5 percent relative deviation from the exact value, even at cardinalities in the tens of millions. This error rate is not a fixed constant but fluctuates statistically around an expected value, meaning repeated queries on identical data can return slightly different but consistently close values, provided the underlying data does not change in between.

Memory usage per HyperLogLog++ structure grows with increasing precision_threshold, but even at very high settings stays in the low single-digit kilobyte range per bucket, far below what an exact count with the same cardinality would require. This ratio of minimal memory usage to controllable error rate makes cardinality aggregation the only practical solution for unique count analysis over very large data volumes, where exact counting is simply no longer feasible.

5. When Approximate Counts Are Sufficient

Approximate counts from the cardinality aggregation are completely sufficient for the vast majority of analytics and monitoring use cases. A dashboard tile showing "approximately 4.8 million unique visitors" does not lose its business value just because the actual number could be 4.75 or 4.85 million. Even for relative comparisons over time, such as "unique users this week versus last week", systematic deviations of the estimation method largely cancel out, because both values are computed with the same algorithm and the same setting.

It becomes critical, however, for use cases that require an exact, legally or financially binding number, such as billing systems charging per unique user, or compliance reports with legally mandated precision. In such cases, cardinality aggregation is unsuitable as the sole data source, but can still serve as a fast preliminary estimate while the binding number is determined via an exact count, for example through terms or composite aggregation with complete iteration.

6. Cardinality Combined with Bucket Aggregations

The cardinality aggregation shows its greatest benefit as a sub-aggregation inside a bucket aggregation, for example to determine the number of unique visitors per landing page or per marketing channel. Combined with a terms aggregation on channel.keyword, a nested cardinality aggregation on session_id.keyword returns the unique visitor count for every channel in a single request, without the application having to send a separate filtered request per channel.

With this combination, it is especially worthwhile to keep memory usage in mind: with twenty channels and a precision_threshold of 10000, twenty independent HyperLogLog++ structures exist in memory simultaneously. With a high cardinality bucket aggregation with hundreds of buckets, this multiplication can become noticeable, which is why precision_threshold should be deliberately set lower for heavily nested cardinality aggregations than for a single, global cardinality query.

POST /web_logs/_search
{
  "size": 0,
  "aggs": {
    "by_channel": {
      "terms": { "field": "channel.keyword", "size": 20 },
      "aggs": {
        "unique_visitors": {
          "cardinality": {
            "field": "session_id.keyword",
            "precision_threshold": 3000
          }
        }
      }
    }
  }
}
// One HyperLogLog++ structure per bucket
// Lower precision_threshold keeps total memory bounded

7. Cardinality vs. Exact Distinct Count Alternatives

For use cases that genuinely need an exact number of unique values, Elasticsearch has no direct, efficient alternative to cardinality aggregation: a terms aggregation with size equal to the expected cardinality does return an exact count via buckets.length, but materializes every single value in memory, which hits the same limits at high cardinality that led to the development of cardinality aggregation in the first place. Complete iteration via composite aggregation also delivers an exact number, but requires significantly more requests and time.

Outside Elasticsearch, exact counting often only remains possible via a relational database with COUNT(DISTINCT ...) on a separate, aggregated table, or a dedicated streaming system that combines exact counting with a bounded time window. In practice, teams therefore usually pick a hybrid approach: cardinality aggregation for real-time dashboards and exploratory analysis, exact batch counting for the few metrics where absolute precision is genuinely required for business reasons.

// Approximate: cardinality aggregation, constant memory
POST /web_logs/_search
{
  "size": 0,
  "aggs": {
    "unique_visitors": { "cardinality": { "field": "session_id.keyword" } }
  }
}

// Exact: composite aggregation, full iteration required
POST /web_logs/_search
{
  "size": 0,
  "aggs": {
    "visitors": {
      "composite": {
        "size": 1000,
        "sources": [
          { "session": { "terms": { "field": "session_id.keyword" } } }
        ]
      }
    }
  }
}
// Count buckets across all pages for the exact number

8. Monitoring and Validating Accuracy

To build confidence in the accuracy of cardinality aggregation for a specific use case, a one-time validation run is recommended: for a manageable subset of the data, both an exact count, for example via a complete composite aggregation iteration, and the cardinality aggregation with the production-planned precision_threshold are executed, and the deviation is measured. This one-time comparison shows concretely how large the actual error is within your own data landscape, instead of relying on generic vendor claims.

For continuous production operation, a rough plausibility check is usually enough: if the estimated cardinality jumps up or down abruptly and unexpectedly between two consecutive time windows, that points more toward a data problem than a weakness of the estimation method, since HyperLogLog++ naturally delivers stable estimates for stable data volumes. A monitoring alert on unusual jumps in the cardinality metric is therefore a useful side effect of the aggregation, independent of its primary business use.

// Compare estimated vs. exact on a bounded validation subset
POST /web_logs/_search
{
  "size": 0,
  "query": {
    "range": { "timestamp": { "gte": "2026-07-01", "lt": "2026-07-02" } }
  },
  "aggs": {
    "estimated": {
      "cardinality": {
        "field": "session_id.keyword",
        "precision_threshold": 3000
      }
    }
  }
}
// Run the same range with a full composite export,
// then diff the two counts to know your real error rate

9. precision_threshold Settings Compared

The table below shows typical settings and their effects.

Choosing the right precision_threshold is therefore a deliberate decision between memory usage, especially with nested aggregations across many buckets, and the accuracy actually required for the business case. For most use cases, the default value of 3000 is a good starting point that should only be adjusted when there is a concrete need.

10. Summary

The cardinality aggregation solves the problem of unique count calculation over very large data volumes with the HyperLogLog++ algorithm, which approximates unique values through hash-based statistical estimation instead of storing every value individually. Memory usage stays nearly constant regardless of the actual cardinality, a fundamental advantage over any exact counting method at millions or billions of unique values.

The precision_threshold parameter controls the ratio between accuracy and memory usage, with a sensible default of 3000 that delivers under 5 percent relative deviation for most use cases. For dashboards, reports and exploratory analysis, these approximate counts are generally entirely sufficient. Only for legally or financially binding numbers should you fall back on an exact count via terms or composite aggregation, optionally supplemented by a one-time validation run measuring the actual error rate within your own data landscape.

11. FAQ: Cardinality Aggregation and Approximate Counts