The Point-in-Time API for Consistent Pagination in Elasticsearch

1. The problem: deep pagination with from and size

Simple pagination through from and size requires Elasticsearch to fully sort and materialize every hit up to the requested position on every single request, even when only a small subset of it actually gets returned. On page one this adds hardly any overhead, on page two hundred with fifty hits per page, ten thousand documents already need to be sorted before the fifty actually wanted are delivered. Past a certain depth, Elasticsearch even refuses such requests outright by default, controlled through the index.max_result_window setting.

A second, subtler problem concerns consistency: if a new document is inserted between fetching page one and page two, and it sorts ahead of the current position, every subsequent hit shifts by one position. The result is documents delivered twice or skipped entirely, a problem that is practically unavoidable during a stable export run over an actively changing index.

2. Creating a point-in-time snapshot

A point in time is opened through a dedicated endpoint and returns a pit_id that gets used in subsequent search requests instead of the index name. Internally, Elasticsearch retains the relevant segments across the involved shards, so later search requests get evaluated against exactly the data state at the moment the snapshot was created, regardless of any write operations happening on the live index in the meantime.

The keep_alive parameter determines how long Elasticsearch retains the segments needed for the snapshot before automatically releasing them. Every search request that uses the point in time automatically extends this window by the specified value, so an active export run does not get cut off by a fixed time limit as long as it keeps issuing further requests regularly.

POST /products/_pit?keep_alive=2m

// response:
{
  "id": "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYQ..."
}

3. Combining search_after with point in time

For actual pagination, the point in time gets embedded in the pit object of the search request, while search_after passes the sort value of the last document from the previous page. This lets Elasticsearch skip the expensive full sort up to the requested position and jump efficiently straight to the next matching range, which delivers roughly constant response time even at very deep pagination.

For search_after to yield unambiguous results, the sort needs to include a unique tie-breaker field, usually _shard_doc or the document ID, since otherwise the order among documents with identical sort values would not be deterministic, and documents could appear twice or go missing.

GET /_search
{
  "size": 100,
  "query": { "term": { "category": "power_tools" } },
  "pit": { "id": "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMA...", "keep_alive": "2m" },
  "sort": [
    { "price": "asc" },
    { "_shard_doc": "asc" }
  ],
  "search_after": [39.90, 184023]
}

4. Difference from the classic scroll mechanism

The older scroll mechanism also produces a consistent snapshot, but it retains additional internal state per active scroll context on every involved shard, which adds up with the number of concurrently open scrolls and can create noticeable memory pressure. Scroll was originally designed for long-running but essentially sequential export operations, and it is explicitly not meant for interactive pagination with changing sort criteria.

Point in time combined with search_after is considerably more resource-friendly, since the snapshot state is lighter and can also be flexibly combined with different search requests against the same snapshot, for instance to apply different filters or sorts against the same consistent data state. Elastic now recommends point in time as the preferred replacement for scroll for the vast majority of use cases that require a stable snapshot.

5. Keep-alive behavior and resource usage in detail

Every open point in time pins segments on the involved shards that Elasticsearch would normally release and delete as part of merges. As long as the snapshot stays active, these segments remain, which, with very long-lived point-in-time contexts under simultaneous heavy write load, can lead to growing disk space usage, since deleted or overwritten documents in the old segment still need to physically persist.

A sensible keep_alive value should be based on the expected time between two consecutive requests plus a safety margin, not on the total duration of the entire export run. Since every request automatically extends the window, a value far shorter than the expected total runtime is usually sufficient, what matters is only that no more time passes between two requests than the configured value.

6. Practical example: a robust export function for large result sets

A typical export function opens a point in time at the start, then repeatedly runs search requests in a loop with an increasing search_after value, writing every result page directly to an output file or stream instead of collecting all results in application memory. As soon as a response contains fewer hits than the requested page size, the end of the result set has been reached and the loop can terminate.

After completing the export, the point in time should be explicitly closed through a DELETE call, rather than relying solely on the automatic expiry of the keep_alive window. This immediately frees the pinned segments instead of waiting for the timeout, and it matters especially for export functions that run frequently with many concurrent runs.

// After a successful export: close the point in time
DELETE /_pit
{
  "id": "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMA..."
}

7. Error handling: expired or invalid point-in-time IDs

If the keep_alive window expires, for instance because an export process paused unexpectedly long or crashed, every further search request with that point-in-time ID fails. A robust implementation should detect this failure case and then decide whether to restart the export completely with a fresh point in time, or whether resuming from the last successfully processed position makes sense, which however gives up the consistency guarantee of the original snapshot.

For distributed export processes that hold multiple point-in-time contexts open in parallel, it also pays off to monitor cluster statistics that report the number of currently open point-in-time contexts, in order to catch forgotten or improperly closed snapshots early, before they unnecessarily pin resources.

8. When from/size, scroll, or point in time each fit best

For shallow, interactive pagination with just a few pages, as needed by a typical result list in a web interface, simple from/size pagination remains the most pragmatic choice, since it requires no additional state and performs well enough for the first pages. Once deep pagination, full data exports, or batch processing across the entire index are required, however, point in time with search_after is the technically superior and Elastic-recommended solution.

The classic scroll mechanism remains functional but is now considered a legacy approach, worth using only in existing systems already built on it. For new implementations, there is hardly any way around point in time combined with search_after once consistent deep pagination is required.

9. Best practices for production use

A tightly sized but sufficient keep_alive value, explicitly closing every point in time after processing completes, and correct, unambiguous sorting with a tie-breaker field together form the foundation for reliable use. It also matters that application code is prepared for expired point-in-time IDs and does not blindly assume that a once-opened snapshot stays available indefinitely.

Regular monitoring of open point-in-time contexts through cluster statistics helps catch forgotten snapshots early, before they unnecessarily pin disk space on the involved shards. With these measures, point in time provides a robust, resource-friendly foundation for any application that needs to process large result sets consistently and completely.

10. Summary

11. FAQ: Point-in-Time API: The Essentials at a Glance