Percentile Aggregation for Reliable Performance Metrics

1. Why an average hides latency spikes

An arithmetic average condenses a distribution of values into a single number, but in doing so loses exactly the information that matters most for performance monitoring: how the outliers behave. With 999 requests at 50 milliseconds and a single request at 5 seconds, the average sits around 55 milliseconds, a number that completely swallows the catastrophic experience of the one affected user. A percentile aggregation makes that outlier visible, because instead of a single mean it computes several points of the actual distribution.

The reason this matters in practice for web shops and APIs: users remember their slowest experiences, not the statistical average. If 5 percent of checkout requests take longer than 3 seconds, that means five hundred frustrated customers out of ten thousand daily orders, even if the average comfortably sits at 200 milliseconds. A percentile aggregation with p95 or p99 makes exactly this problem quantifiable, while a plain avg value systematically hides it.

The following sections explain the percentiles aggregation from basic syntax through the TDigest algorithm to complete latency dashboards with multiple percentile tiers. Every example uses real aggregation syntax as it runs against logging or APM indices in Elasticsearch.

2. Basics: the percentiles aggregation

The percentiles aggregation computes seven percentiles of a numeric field by default: 1, 5, 25, 50, 75, 95 and 99. Each percentile states the value below which a given proportion of all data points falls. p50, the median percentile, corresponds to the classic median: half of all requests are faster, half slower than this value. For most performance analyses, p50, p95 and p99 are the most relevant metrics from a percentile aggregation, which is why they are usually requested explicitly via the percents parameter, rather than relying on the default selection.

The query syntax resembles other metric aggregations: field specifies the numeric field whose distribution should be analyzed, percents a list of the desired percentile values. For latency data, response_time_ms in milliseconds is a typical target field, coming from access logs or APM traces.

GET /api-logs/_search
{
  "size": 0,
  "query": {
    "range": { "@timestamp": { "gte": "now-1h" } }
  },
  "aggs": {
    "response_time_percentiles": {
      "percentiles": {
        "field": "response_time_ms",
        "percents": [50, 75, 90, 95, 99, 99.9]
      }
    }
  }
}

3. How TDigest approximates percentiles

An exact percentile calculation would require sorting all values and reading off the value at the corresponding position, which is neither memory- nor time-efficient with millions or billions of documents. The percentiles aggregation in Elasticsearch therefore uses the TDigest algorithm by default, a probabilistic data structure that keeps a compact summary of the distribution in memory instead of retaining every individual value. This percentile aggregation structure therefore delivers approximated rather than exact values, with accuracy that is sufficient for practically all monitoring purposes.

The special property of TDigest is that accuracy is higher at the edges of the distribution, i.e. at very low and very high percentiles like p1 or p99.9, than in the middle around p50. That is not accidental, it is deliberate design: for latency monitoring, the extreme percentiles matter most, because they represent the users with the worst experience, while a less precise approximation in the middle range barely matters in practice.

4. Interpreting p50, p95 and p99 correctly

p50 corresponds to the median and describes the typical experience of an average user. For most performance goals, p50 alone is insufficient, though, because by definition it ignores half of all requests. p95 states that 95 percent of all requests are faster than this value, while the remaining 5 percent are slower. For a high-traffic shop, a p95 of 2 seconds means one in twenty requests exceeds that threshold, a share that, at high volume, translates into a large absolute number of affected users.

p99 sharpens this view further and is especially relevant for systems with strict latency requirements, such as payment processing or real-time search. A percentile aggregation with p99 shows the response time exceeded by the slowest 1 percent of all requests. The gap between p50 and p99 is often more telling than either value on its own: a system with p50 at 100 milliseconds and p99 at 150 milliseconds behaves fundamentally differently from one with p50 at 100 milliseconds and p99 at 8 seconds, even though the median looks identical in both cases.

GET /api-logs/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        { "term": { "endpoint": "/checkout" } },
        { "range": { "@timestamp": { "gte": "now-24h" } } }
      ]
    }
  },
  "aggs": {
    "checkout_latency": {
      "percentiles": {
        "field": "response_time_ms",
        "percents": [50, 95, 99]
      }
    }
  }
}

5. percentile_ranks: measuring SLA compliance

While percentiles translates a percentile spec into a concrete value, percentile_ranks works exactly the other way around: it translates a concrete threshold into the percentage of requests falling below that value. For an SLA promising a response time under 500 milliseconds, this percentile aggregation variant directly answers the question "what percentage of all requests meet this target", without having to try a series of percentiles.

percentile_ranks is therefore ideal for SLA dashboards and alerting rules that need to monitor a fixed compliance rate, for example "99 percent of all requests under 1 second". An alert that fires as soon as the value returned by percentile_ranks drops below 99 is more direct and more informative than an alert on a single percentile value, because it expresses the actual target miss in percentage points.

GET /api-logs/_search
{
  "size": 0,
  "aggs": {
    "sla_compliance": {
      "percentile_ranks": {
        "field": "response_time_ms",
        "values": [500, 1000, 3000]
      }
    }
  }
}

6. Percentile aggregation over time

For monitoring dashboards, a percentile aggregation is usually attached as a sub-aggregation under a date_histogram, to see how p95 and p99 change over time. This combination reveals latency trends that a single overall value would never make visible, such as a gradual degradation of response times over several days or sudden latency spikes at particular high-load hours.

An important caveat: TDigest states cannot be meaningfully averaged or directly compared across buckets of a date_histogram, because every bucket calculation builds its own, independent TDigest structure. A global p99 over a month is therefore not the same as the average of the daily p99 values, since both computations operate on different data volumes and distributions. For reliable monthly or quarterly reports, always run a separate percentile aggregation over the entire period, rather than retroactively averaging daily values.

GET /api-logs/_search
{
  "size": 0,
  "query": {
    "range": { "@timestamp": { "gte": "now-7d" } }
  },
  "aggs": {
    "latency_per_hour": {
      "date_histogram": {
        "field": "@timestamp",
        "calendar_interval": "hour",
        "min_doc_count": 0
      },
      "aggs": {
        "p95_p99": {
          "percentiles": {
            "field": "response_time_ms",
            "percents": [95, 99]
          }
        }
      }
    }
  }
}

7. Controlling accuracy: compression and hdr

The compression parameter controls the accuracy of the TDigest-based percentile aggregation by setting how many internal nodes the data structure uses. A higher compression value increases accuracy at the cost of more memory usage, a lower value saves memory at the cost of precision. The default of 100 is accurate enough for most use cases, and monitoring systems with strict requirements on extreme percentiles can benefit from raising it.

As an alternative to TDigest, Elasticsearch offers the HDR histogram algorithm, enabled via the method parameter. HDR works with fixed rather than adaptive precision and fits particularly well with values that have a known, bounded range, such as latencies between 0 and 60000 milliseconds. For most performance metric use cases, TDigest remains the more practical choice, though, because it works without prior knowledge of the value range and still delivers reliable results.

8. Latency dashboards with multiple percentiles

An informative latency dashboard rarely shows just a single percentile value, but several at once, for example p50, p95 and p99 as overlapping lines in a time series chart. This view reveals at a glance whether the entire distribution is shifting, for example due to a slower database server, or whether only the extreme percentiles are misbehaving, for example due to occasional garbage collection pauses or a few overloaded nodes.

For a breakdown by endpoint or service name, the percentile aggregation is additionally combined with a terms aggregation, so every API endpoint gets its own p95 and p99 values. This combination frequently reveals that a single slow endpoint dominates a system's overall metric, while the vast majority of endpoints remain quietly performant, an insight a single global average value would completely obscure.

9. Common mistakes and debugging

The most common mistake when working with a percentile aggregation is naively averaging TDigest-based percentiles from several periods or buckets to approximate an overall value. That is not statistically correct, because percentiles do not combine linearly. The average of 24 hourly p99 values is not the same as the p99 over the full day, because the hourly distributions can have different shapes and data volumes.

// WRONG: averaging hourly p99 values to approximate a daily p99
// This produces a statistically meaningless number
avg(p99_hour1, p99_hour2, ..., p99_hour24)

// RIGHT: run a single percentiles aggregation over the full day
{
  "query": {
    "range": { "@timestamp": { "gte": "now-24h" } }
  },
  "aggs": {
    "daily_p99": {
      "percentiles": {
        "field": "response_time_ms",
        "percents": [99]
      }
    }
  }
}

A second common mistake is running a percentile aggregation on a far too small sample. With only a few hundred requests per bucket, p99 fluctuates heavily and provides no reliable insight, because a single outlier dominates the result at that low data volume. For stable p99 values, the underlying data volume per bucket should be at least in the low thousands, otherwise a coarser time granularity such as daily instead of hourly buckets is the better choice.

10. Summary

The percentile aggregation replaces the misleading average with real distribution points, revealing what an avg value systematically hides: the experience of the slowest users. p50 shows the typical experience, p95 and p99 the experience of the slower outliers, with the TDigest algorithm efficiently approximating these values even across billions of documents.

percentile_ranks reverses the calculation and directly answers the SLA compliance question. Combined with date_histogram, latency dashboards emerge that show trends over time, where every time period needs its own, independent percentile calculation instead of naively averaging percentiles across buckets. Combine these building blocks correctly, and you get reliable performance metrics instead of smoothed-over, misleading averages.

11. FAQ: Percentile Aggregation for Performance Metrics