The Percolator Query: Implementing Reverse Search in Elasticsearch

1. Core principle: queries as documents, documents as queries

In a classic search, a fixed set of documents lives in the index, and a search request is evaluated against that set to find the most relevant hits. The percolator flips this relationship around: instead of checking documents against a request, search queries themselves are stored as documents in an index, and a newly arriving document is checked against all stored queries. The result is not a ranked list of products but a list of stored queries that would match that one new document.

This inversion makes sense whenever the number of search criteria clearly exceeds the number of newly arriving documents per time unit, for instance with price alerts, stock saturation notifications, or content moderation rules that must be applied to every new post. A naive implementation would loop through every stored criterion individually in application code for each new document, which quickly becomes a bottleneck with many thousands of criteria.

2. The percolator field type and how stored queries get indexed

To use percolate queries, the index needs a field of type percolator that stores the actual query DSL definition as its value. Elasticsearch analyzes this stored query at index time and efficiently extracts searchable features from it, so that percolating later does not naively evaluate every stored query one by one, but first performs a candidate preselection through an internal index.

Besides the percolator field itself, every field that stored queries might reference needs to exist in the same index, or in an index with a compatible mapping, so that Elasticsearch can correctly evaluate the query structure against incoming documents. In practice, this often means setting up a dedicated index with two logical document types: stored customer criteria on one side, and a sample mapping of the product fields on the other.

PUT /price_alerts
{
  "mappings": {
    "properties": {
      "query": { "type": "percolator" },
      "category": { "type": "keyword" },
      "price": { "type": "float" },
      "brand": { "type": "keyword" }
    }
  }
}

PUT /price_alerts/_doc/customer_4711
{
  "query": {
    "bool": {
      "filter": [
        { "term": { "category": "power_tools" } },
        { "range": { "price": { "lte": 49.90 } } }
      ]
    }
  }
}

3. Running a percolate query against a new document

When a new product arrives, it is not written into the index of stored queries, instead it is passed as part of a percolate query to Elasticsearch. The response contains the IDs of every stored query whose criteria the passed document satisfies. The application can then use these IDs, for example to look up the associated customer ID and trigger a notification.

It matters that the percolate query never persists the new document, it only holds it temporarily for the duration of the request while checking it against the stored queries. Actually storing the new product in the regular product index remains a separate, independent step.

GET /price_alerts/_search
{
  "query": {
    "percolate": {
      "field": "query",
      "document": {
        "category": "power_tools",
        "price": 39.90,
        "brand": "ProTools"
      }
    }
  }
}
// response returns customer_4711 as a hit, since category and price match

4. Use case: implementing price alerts and stock saturation notifications

For price alerts, every customer creates a document in the percolator index when setting up a notification, encoding their individual criteria as query DSL, for instance the desired category, a maximum price, and optionally preferred brands. When new products are imported or existing product prices change, every affected product document gets checked once against the percolator index. Every hit corresponds to a customer who should be informed about the new or changed offer.

The same pattern works for stock saturation notifications: a sales representative stores a rule like stock in a certain category falling below a threshold, and every incoming stock change gets percolated as a document against these stored rules. This produces a flexible alerting system without writing custom application code for every rule, since the entire condition logic is expressed in query DSL and managed centrally.

5. Percolating multiple documents at once

Instead of issuing a separate percolate query for every new product, Elasticsearch supports percolating multiple documents in a single request through the documents array. This significantly reduces network overhead when, for example, an import batch of several hundred new products needs to be checked against every stored price alert at once, rather than sending a separate request over the network for each individual product.

The response maps every hit back to its originating document through an internal field, so the application knows, after a single batch request, exactly which product satisfied which stored criteria. For regular batch imports, as is common with product feeds, this approach is practically always preferable to a loop of individual requests.

6. Performance limits with very many stored queries

The percolator scales well with the number of newly arriving documents, since the internal preselection through extracted query features significantly reduces the number of candidate queries that actually get fully evaluated. With a very large number of stored queries, for instance several million individual customer criteria, the effort for this preselection itself grows, and the latency of a single percolate request rises noticeably.

Especially critical are very generic stored queries that yield hardly any extractable features, for instance a query without concrete term filters, only range conditions on a highly variable field. Such queries can barely be preselected internally and end up fully evaluated more often. In practice it is therefore advisable to partition the percolator index by category or region and only percolate the relevant subindex, rather than checking every stored query for every new document by default.

7. Maintenance: updating and cleaning up stored search queries

Stored percolate queries behave like regular documents and can be updated or deleted through the standard update API accordingly, for instance when a customer adjusts or deactivates their price alert. Since every stored query takes up space in the internal preselection index, regular cleanup of expired or inactive criteria pays off, especially for time-limited campaigns like a seasonal price alert.

For alerts with a fixed validity period, combining this with index lifecycle management or a simple TTL field, removed by a scheduled job, works well to purge expired documents from the percolator index. Without this upkeep the index keeps growing indefinitely, which directly affects the latency of every single percolate request.

8. Boundaries versus alternatives: watcher and external rule engines

The percolator is well suited for the concrete match of a single new document against many stored criteria, but it is not a complete alerting framework with scheduling, notification channels, or escalation logic. For periodic checks across an entire data set, for instance a daily summary of all price changes, watcher or an external scheduler solution is more appropriate, using the percolator merely as one building block inside a larger pipeline.

Compared to a fully custom rule engine in the application, the percolator has the advantage that the entire matching logic is expressed in the proven Elasticsearch query DSL and benefits from the same infrastructure already running for regular product search. For very complex, multi-stage rule sets that require state management across multiple events, the percolator hits conceptual limits, since it is fundamentally stateless, checking one single document against stored queries at a time.

9. Rolling it out in practice: a short checklist

Before going into production, a test run with a realistic number of stored queries and realistic document volume is worthwhile, to measure actual latency under load instead of relying on assumptions. Equally important is clear monitoring of the percolator index size, so uncontrolled growth of stored but long-irrelevant queries gets noticed early.

Anyone who applies the percolator deliberately to a clearly scoped use case like price alerts, regularly cleans up expired queries, and partitions by category or region as needed ends up with a robust and maintainable alerting system that requires no additional external components and builds directly on the already existing Elasticsearch infrastructure.

10. Summary

11. FAQ: Percolator Query: The Essentials at a Glance