Composite Aggregation for Pagination of Large Result Sets

1. Why Terms Aggregation Pagination Does Not Scale

Terms aggregation has no real pagination mechanism. The only way to get more than the top size buckets is to raise size itself and receive all buckets in a single response. For a field with a thousand unique values, that still works acceptably; for a field with a million unique values, the coordinating node has to hold a million buckets in memory simultaneously before the response can even be serialized. This leads to long response times, high garbage collection pressure and, in the worst case, the circuit breaker already mentioned, which aborts the request entirely.

A second, subtler problem: terms aggregation with a very high size still gives no guarantee of completeness or stable ordering between two consecutive requests, especially if the underlying data set changes in between. For use cases like a complete CSV export of all brand values or a data pipeline that has to systematically process every bucket, terms aggregation is therefore structurally the wrong choice. This is exactly the use case the composite aggregation was built for.

2. Composite Aggregation: Core Concept

The composite aggregation returns only a bounded, configurable number of buckets per request via the size parameter, so it initially behaves similar to a normal bucket aggregation. The crucial difference lies in the response: every composite response additionally contains an after_key field that uniquely identifies the last bucket returned. If this after_key is passed as the after parameter in the next request, Elasticsearch returns exactly the next page of buckets, directly following the last value seen.

This cursor-based pagination model is fundamentally different in memory terms from the size-based model of terms aggregation: the coordinating node never has to hold more than one page of buckets in memory at any time. The composite aggregation combines one or more sources for this, each defining a bucketing criterion such as terms, histogram or date_histogram, and produces a composite, uniquely sorted bucket sequence from them.

POST /products/_search
{
  "size": 0,
  "aggs": {
    "all_brands": {
      "composite": {
        "size": 100,
        "sources": [
          { "brand": { "terms": { "field": "brand.keyword" } } }
        ]
      }
    }
  }
}
// Response excerpt:
// "all_brands": {
//   "after_key": { "brand": "Lenovo" },
//   "buckets": [
//     { "key": { "brand": "Dell" }, "doc_count": 340 },
//     { "key": { "brand": "HP" }, "doc_count": 251 },
//     { "key": { "brand": "Lenovo" }, "doc_count": 298 }
//   ]
// }

3. The after_key Cursor in Detail

To fetch the next page of buckets, the after_key value from the previous response is carried over unchanged as the after parameter in the next request. Elasticsearch internally sorts the composite bucket keys uniquely and deterministically, so every bucket is returned exactly once across the entire iteration, even if the underlying data set changes slightly between two requests. This property fundamentally distinguishes the composite aggregation from naive offset-based paging, where shifts in the data set directly affect page consistency.

The end of the iteration is reached once a response returns an empty buckets array, or the number of returned buckets is smaller than the requested size value. Application code using composite aggregation therefore typically implements a simple loop: send request, process buckets, carry over after_key for the next request, repeat until no more buckets come back.

POST /products/_search
{
  "size": 0,
  "aggs": {
    "all_brands": {
      "composite": {
        "size": 100,
        "after": { "brand": "Lenovo" },
        "sources": [
          { "brand": { "terms": { "field": "brand.keyword" } } }
        ]
      }
    }
  }
}
// Continues exactly after the last bucket from the previous page
// Loop until "buckets" comes back empty or shorter than size

4. Combining Sources: terms, histogram, date_histogram

The real value of the composite aggregation shows up when several sources are combined at the same time. Every additional source refines the composite bucket key by one more dimension, for example brand and category together, or brand combined with a monthly date_histogram for time series reports. The order of sources in the array determines both the sort order of the composite keys and their priority in the internal sorting.

Unlike nested bucket aggregations, the composite aggregation does not produce a hierarchical tree structure but a flat list of composite keys, where each bucket key contains all source values simultaneously. That makes processing in application code simpler, because no recursive traversal of nested buckets is needed, just iteration over a simple flat list.

POST /orders/_search
{
  "size": 0,
  "aggs": {
    "sales_by_brand_month": {
      "composite": {
        "size": 500,
        "sources": [
          { "brand": { "terms": { "field": "brand.keyword" } } },
          { "month": { "date_histogram": { "field": "order_date", "calendar_interval": "month" } } }
        ]
      },
      "aggs": {
        "revenue": { "sum": { "field": "total_price" } }
      }
    }
  }
}
// Each bucket key combines brand AND month, e.g.
// { "brand": "Dell", "month": "2026-06-01T00:00:00.000Z" }

5. Complete Iteration: Exporting All Buckets

For a complete export of all bucket values, for example every SKU value ever used in an order index, you implement a loop in application code: the first request contains no after parameter, every further iteration carries over the after_key of the previous response. This loop runs until an empty bucket array signals that no further values exist. Unlike scroll-based approaches, the composite aggregation requires no server-side context that has to stay open, making it robust against connection drops between individual requests.

A practical advantage: since every request is independent and stateless, an interrupted iteration can be resumed at any time with the last saved after_key, without having to start over. This is particularly relevant for long-running export jobs that iterate over very large amounts of data for hours or days and should not require an expensive restart on a crash.

6. Sorting and Performance with Composite

The composite aggregation sorts bucket keys ascending by default, but this can be individually reversed per source with the order: "desc" parameter. Important: sorting happens exclusively over the composite keys themselves, not over doc_count or a sub-aggregation as is possible with terms aggregation. Anyone who needs buckets sorted by a computed metric has to perform that sorting themselves in application code after completing the full iteration.

Performance-wise, composite aggregation scales noticeably better than a terms aggregation with an artificially high size, because only a bounded number of buckets need to be materialized per request. The price is several sequential requests instead of a single one: for a million unique values and a page size of 1000, a thousand requests are needed to iterate completely, which should be taken into account for time-critical use cases.

POST /products/_search
{
  "size": 0,
  "aggs": {
    "all_brands": {
      "composite": {
        "size": 100,
        "sources": [
          {
            "brand": {
              "terms": { "field": "brand.keyword", "order": "desc" }
            }
          }
        ]
      }
    }
  }
}
// Sorting applies only to the composite key itself,
// not to doc_count or any sub-aggregation value

7. Combining Composite with Sub-Aggregations

Just like other bucket aggregations, the composite aggregation accepts its own aggs block for metric aggregations per bucket. That allows returning, during the complete iteration, not only bucket keys but also computed metrics like revenue or average price per combination of brand and month, without having to issue a separate request for each bucket. This combination is the core of most report generation pipelines built on top of composite aggregation.

One limitation is that nested bucket aggregations inside a composite aggregation are technically possible, but they correspondingly increase the response size per page and can partially undo the benefits of bounded memory usage. In practice you therefore mostly stick to a combination of composite aggregation and simple metric sub-aggregations, rather than building deeply nested bucket structures inside every composite page.

8. Use Cases: Exports, Reports, Data Pipelines

The classic use case for composite aggregation is a complete export of all unique values of a field, for example to synchronize an external recommendation system with every product category of a catalog. A second common use case is generating periodic reports that need to iterate over every combination of several dimensions, for example revenue per brand and month over the last three years, where a single terms aggregation with a nested sub-aggregation would fail on the sheer number of combinations.

A third, increasingly important use case is using it as a data source for ETL pipelines and machine learning feature extraction, where every combination of user segment and time window needs to be processed systematically. Since composite aggregation is stateless and resumable, it suits such long-running, batch-oriented processes well, run regularly in production data pipelines.

// Pseudocode for a resumable full export loop
let afterKey = null;
do {
  const body = {
    size: 0,
    aggs: {
      export: {
        composite: {
          size: 500,
          after: afterKey,
          sources: [
            { sku: { terms: { field: "sku.keyword" } } }
          ]
        }
      }
    }
  };
  const response = search("/products/_search", body);
  processBuckets(response.aggregations.export.buckets);
  afterKey = response.aggregations.export.after_key;
} while (response.aggregations.export.buckets.length > 0);
// Save afterKey after each page to resume on failure

9. Composite vs. Terms Pagination Compared

The table below compares the two approaches for iterating over many buckets.

The choice between the two approaches is a question of use case, not general superiority: for top-N displays in the user interface, terms aggregation remains the right, simpler choice. But once every single bucket genuinely needs to be processed, composite aggregation is the only robust solution that stays stable even at millions of unique values.

10. Summary

The composite aggregation solves a structural problem that terms aggregation cannot solve at high cardinality: complete, memory-efficient iteration over any number of buckets. The after_key cursor returns exactly the next page of buckets after every request, without the coordinating node ever having to hold more than one page in memory at a time. Several sources can be combined into a composite, uniquely sorted bucket key, for example brand and month together.

For top-N facets in the user interface, terms aggregation remains the simpler, right choice. But once a complete export, a multi-dimensional report or a data pipeline needs to systematically process every bucket, composite aggregation is the more robust solution: stateless, resumable and without the memory problems an artificially high size causes in terms aggregation.

11. FAQ: Composite Aggregation for Pagination