Pipeline Aggregations Explained in Detail

1. Why pipeline aggregations are a concept of their own

Regular aggregations in Elasticsearch, so-called bucket and metric aggregations, compute directly over the documents of an index. A pipeline aggregation, on the other hand, does not take documents as input but the output of an already computed aggregation. Instead of "sum of orders per day", a pipeline aggregation computes, for example, "change in the daily sum compared to the previous day" by building on the results of a date_histogram aggregation. This two-stage model makes it possible to derive complex trend and progression calculations from simple bucket values, without querying the raw data a second time.

The advantage over client-side post-processing is twofold. First, computing a pipeline aggregation server-side saves network traffic, because only the finished result gets transferred. Second, it guarantees consistency, because the derivative, moving average or cumulative sum is computed on exactly the same buckets that are shown in the chart. Anyone who computes differences retroactively in the frontend instead risks rounding errors or inconsistencies from missing buckets.

The following sections explain the most important pipeline aggregations in detail: derivative for rates of change, moving_fn for sliding windows, cumulative_sum for running totals, and sibling pipelines such as max_bucket and stats_bucket. Every pipeline aggregation is shown with real query syntax as it runs against a production cluster.

2. Sibling vs parent pipeline aggregations

Elasticsearch distinguishes two categories of pipeline aggregations that differ in their position within the aggregation hierarchy and in their output structure. A parent pipeline aggregation is defined as a sibling aggregation inside the buckets of a bucket aggregation and adds a newly computed value to every single bucket. derivative, cumulative_sum and moving_fn are parent pipelines: they extend every bucket of a date_histogram with an additional field, without changing the number of buckets.

A sibling pipeline aggregation, on the other hand, is defined on the same level as the bucket aggregation, not inside its buckets, and condenses all buckets into a single result. max_bucket, for example, finds the bucket with the highest value across an entire time series and returns its key and value as a single result, regardless of how many buckets the underlying aggregation produced. stats_bucket returns min, max, avg, sum and count across all buckets in a single compact object.

The practical difference shows up in the JSON structure of the response: with a parent pipeline aggregation, the new value appears inside every single bucket object of the original aggregation. With a sibling pipeline aggregation, the result appears as a standalone aggregation object at the same nesting level as the source aggregation. That distinction is not a formality, it directly determines where to reference the buckets_path and where to parse the response.

3. derivative: computing change between buckets

The derivative pipeline aggregation computes the difference between a bucket's value and the value of the preceding bucket. That is the direct way to derive a second time series, "change vs. previous day", from a time series like "revenue per day", without duplicating the computation in the client. The buckets_path parameter points via path notation at the metric aggregation whose values serve as input, for example "daily_revenue" inside the same date_histogram bucket.

For growth rates rather than absolute differences, this pipeline aggregation is often combined with a second derivative stage that builds on the result of the first derivative, computing the second derivative, i.e. the acceleration of growth. This chaining of several pipeline aggregations is a central feature of the model: every pipeline aggregation can itself serve as input to another pipeline aggregation, as long as the buckets_path correctly references the name of the previous stage.

GET /orders/_search
{
  "size": 0,
  "aggs": {
    "sales_per_day": {
      "date_histogram": {
        "field": "order_date",
        "calendar_interval": "day",
        "min_doc_count": 0
      },
      "aggs": {
        "daily_revenue": { "sum": { "field": "grand_total" } },
        "revenue_change": {
          "derivative": {
            "buckets_path": "daily_revenue"
          }
        }
      }
    }
  }
}

4. moving_fn: sliding windows and custom scripts

moving_fn is the most flexible pipeline aggregation for sliding windows, because it accepts a Painless script that can compute freely over the values of a configurable window of preceding buckets. The window parameter sets how many buckets feed into the calculation, shift moves the window relative to the current bucket. For a classic 7-day moving average, Elasticsearch provides the helper function MovingFunctions.unweightedAvg, which averages over the values available in the window.

The advantage over a simple avg metric on a larger bucket is smoothing without loss of information: a moving average still shows daily data points but smooths out short-term noise such as weekend dips in e-commerce, while the daily resolution of the underlying date_histogram aggregation is preserved. moving_fn also fits more complex calculations such as a rolling standard deviation or a rolling maximum, because the Painless script can run arbitrary logic over the values array it receives.

GET /orders/_search
{
  "size": 0,
  "aggs": {
    "sales_per_day": {
      "date_histogram": {
        "field": "order_date",
        "calendar_interval": "day",
        "min_doc_count": 0
      },
      "aggs": {
        "daily_revenue": { "sum": { "field": "grand_total" } },
        "revenue_7d_moving_avg": {
          "moving_fn": {
            "buckets_path": "daily_revenue",
            "window": 7,
            "script": "MovingFunctions.unweightedAvg(values)"
          }
        }
      }
    }
  }
}

5. cumulative_sum: running totals over time

cumulative_sum adds the value of each bucket to the sum of all preceding buckets, producing a running total typical for progress indicators and annual target tracking. Unlike sum, which only returns the value of a single bucket, the value of this pipeline aggregation grows monotonically with every additional period. For a sales dashboard with an annual target, that means: cumulative_sum directly shows how cumulative revenue approaches a target line over the course of the year, without rebuilding that calculation in the frontend.

A quirk of cumulative_sum is that the accumulation runs over all returned buckets, not just the visible slice of a dashboard. If the range of the underlying date_histogram aggregation is restricted, the cumulative sum restarts at zero at the first bucket of that restricted range. For a correct annual-target chart, the queried range must therefore always start at the desired accumulation starting point, such as January 1st, and not arbitrarily later.

GET /orders/_search
{
  "size": 0,
  "query": {
    "range": { "order_date": { "gte": "2026-01-01", "lte": "2026-12-31" } }
  },
  "aggs": {
    "sales_per_month": {
      "date_histogram": {
        "field": "order_date",
        "calendar_interval": "month",
        "min_doc_count": 0
      },
      "aggs": {
        "monthly_revenue": { "sum": { "field": "grand_total" } },
        "cumulative_revenue": {
          "cumulative_sum": {
            "buckets_path": "monthly_revenue"
          }
        }
      }
    }
  }
}

6. Sibling pipelines: max_bucket, min_bucket, stats_bucket

While derivative, moving_fn and cumulative_sum each extend every bucket individually, sibling pipeline aggregations condense all buckets into a single metric. max_bucket identifies the highest-revenue day in a time series and returns both its bucket key and value, without needing separate sorting or an extra query pass. min_bucket works the same way for the weakest period.

stats_bucket is the most compact of these pipeline aggregations, because it computes count, min, max, avg and sum in a single call across all buckets. That is especially useful for overview cards in dashboards that need, alongside the detailed time series, additional metrics like "best day", "weakest day" and "average over the range", without running a separate aggregation over the same raw data. percentiles_bucket extends this family with percentile values across bucket results, for example the median of daily revenue.

GET /orders/_search
{
  "size": 0,
  "aggs": {
    "sales_per_day": {
      "date_histogram": {
        "field": "order_date",
        "calendar_interval": "day",
        "min_doc_count": 0
      },
      "aggs": {
        "daily_revenue": { "sum": { "field": "grand_total" } }
      }
    },
    "best_day": {
      "max_bucket": { "buckets_path": "sales_per_day>daily_revenue" }
    },
    "revenue_stats": {
      "stats_bucket": { "buckets_path": "sales_per_day>daily_revenue" }
    }
  }
}

7. bucket_selector and bucket_sort for post-aggregation filtering

bucket_selector is a special pipeline aggregation that does not produce a new value but removes buckets after they were computed, if a Painless script returns false. That lets you, for example, restrict a time series to days with revenue above a threshold, without rebuilding the filtering in a separate request or in the client. That is an advantage over a plain query filter, because bucket_selector filters on the already aggregated value, not on fields of individual documents.

bucket_sort adds sorting and pagination for buckets on top of bucket_selector. While a normal bucket aggregation sorts by its own criterion by default, for example chronologically for date_histogram, bucket_sort allows a retroactive sort by the value of a sub-aggregation, combined with from and size for top-N reports like "the ten best-selling weeks of the year". Both pipeline aggregations together replace many cases that previously needed a second query or client-side post-processing.

8. Performance and limits of pipeline aggregations

Pipeline aggregations are computationally cheap, because they run exclusively over already computed bucket values and scan no additional documents. The compute cost of a pipeline aggregation depends almost entirely on the number of buckets, not on the number of underlying documents. A derivative over a thousand daily buckets is trivially fast, even if every single bucket summarizes billions of documents.

One important limit concerns nested bucket aggregations: pipeline aggregations like derivative or moving_fn only work within a single bucket level, for example along a date_histogram. If the buckets are additionally split by category in a terms aggregation, the pipeline aggregation must be defined separately inside each category group, because buckets_path is resolved relative to the current nesting level. A direct comparison of trends across different category branches is not possible with standard pipeline aggregations and requires either several sub-requests or client-side post-processing.

9. Common mistakes and debugging

The most common mistake with pipeline aggregations is an incorrect buckets_path. The path must exactly match the name of the target aggregation and, for sibling pipelines, use the "aggregationname>metricname" syntax to navigate through a nesting level. A typo in the buckets_path does not fail silently but raises an explicit error while executing the query, which makes debugging easier in practice.

// WRONG: buckets_path points to a bucket aggregation, not a metric
{
  "derivative": { "buckets_path": "sales_per_day" }
}

// RIGHT: buckets_path points to the metric inside each bucket
{
  "derivative": { "buckets_path": "sales_per_day.daily_revenue" }
}

// WRONG: sibling pipeline path missing the ">" navigation
{
  "max_bucket": { "buckets_path": "sales_per_day.daily_revenue" }
}

// RIGHT: sibling pipelines navigate with ">" through nesting levels
{
  "max_bucket": { "buckets_path": "sales_per_day>daily_revenue" }
}

A second common mistake concerns the first buckets of a time series: derivative returns no value for the very first bucket, because there is no preceding bucket to compare against. Clients that do not handle this missing field fail with a null pointer error. moving_fn behaves similarly at the start of a time series, as long as the configured window is not yet fully populated with values. Both cases are not a bug but expected behavior of a pipeline aggregation that must be explicitly handled in the frontend.

10. Summary

Pipeline aggregations extend the Elasticsearch aggregation model with a second computation layer that runs not over documents but over the results of other aggregations. Parent pipelines like derivative, moving_fn and cumulative_sum extend every single bucket with an additional computed value. Sibling pipelines like max_bucket and stats_bucket condense all buckets into a single overall result. bucket_selector and bucket_sort enable filtering and top-N reporting on already aggregated values.

Combine these building blocks correctly and you move trend calculation, smoothing and target tracking entirely into the query, avoiding error-prone client-side post-processing. buckets_path is the central lever here: for parent pipelines it points with dot notation into the same nesting level, for sibling pipelines it navigates with the arrow operator through an additional level. Internalize that difference and you avoid the most common configuration mistakes with pipeline aggregations.

11. FAQ: Pipeline Aggregations Explained in Detail