Runtime Fields for Flexible Analytics Without a Reindex

1. Why runtime fields shorten the reindex cycle

In a classic Elasticsearch workflow, a new field is created through a mapping change followed by a reindex that reprocesses every document. On indices with billions of documents, that cycle takes hours to days and ties up cluster resources, just to find out whether the new analysis is even useful in practice. Runtime fields solve exactly that problem by computing a field not at index time, but at query time from a Painless script.

The central advantage is decoupling data modeling from iteration speed. An analyst can use a runtime field to immediately test whether a derived metric like "order value per item" or "categorization by price tier" produces useful results in practice, without scheduling a reindex job first. Only once a runtime field proves permanently useful does it become worth the effort to turn it into a real, indexed field.

The following sections walk through runtime fields from basic syntax through Painless scripts to migrating into an indexed field. Every example uses real query and mapping syntax as it runs against a production Elasticsearch or OpenSearch cluster.

2. Basic syntax: runtime_mappings in the query

The simplest way to use a runtime field is the runtime_mappings parameter directly in the search query. This changes no existing mapping, the field only exists for the duration of that one request. Every runtime_mappings field needs at least a type, such as keyword, long or double, and a script that computes the value for every document and passes it to the result via emit().

This approach fits exploratory analysis in Kibana or directly via the REST API particularly well, because no permission for mapping changes on the index is needed and nothing is permanently changed on the cluster. A runtime field defined only in a query's runtime_mappings automatically disappears once the request finishes, leaving no trace in the persistent mapping.

GET /orders/_search
{
  "runtime_mappings": {
    "order_value_bucket": {
      "type": "keyword",
      "script": {
        "source": """
          double total = doc['grand_total'].value;
          if (total < 50) emit('low');
          else if (total < 200) emit('medium');
          else emit('high');
        """
      }
    }
  },
  "query": { "match_all": {} },
  "aggs": {
    "orders_by_value_bucket": {
      "terms": { "field": "order_value_bucket" }
    }
  }
}

3. Painless scripts for computed fields

The Painless script at the core of a runtime field has access to the doc object, which exposes the current document's indexed field values via doc_values, as well as to params._source for the complete original JSON document. Access via doc is noticeably faster than via _source, because doc_values already sit column-oriented and optimized for fast access in the index, while _source has to be re-parsed from the stored raw document on every call.

Inside the script, the usual Painless language constructs are available: conditionals, loops, arithmetic operations, and a restricted set of standard library methods for strings, dates and math. For a runtime field that combines several source fields, for example a net margin from sale price and cost price, a script of just a few lines usually suffices, reading both values, computing the difference and returning the result via emit().

GET /products/_search
{
  "runtime_mappings": {
    "margin_percent": {
      "type": "double",
      "script": {
        "source": """
          if (doc['cost_price'].size() == 0 || doc['sale_price'].size() == 0) {
            return;
          }
          double cost = doc['cost_price'].value;
          double sale = doc['sale_price'].value;
          if (cost == 0) return;
          emit(((sale - cost) / cost) * 100);
        """
      }
    }
  },
  "query": {
    "range": { "margin_percent": { "lt": 10 } }
  }
}

4. Defining runtime fields in the index mapping

Instead of redefining a runtime field in every single query, it can also be stored permanently in the index mapping under the runtime key. The field then behaves like a regular field name for all subsequent requests, without needing to repeat runtime_mappings in every single query. The crucial difference from a normal indexed field remains, though: the value is still recomputed on every query, no precomputation happens at index time.

This mode fits runtime fields that have proven useful and are meant to be used more frequently, but do not yet justify the effort of a full reindex. A runtime field defined in the mapping can also be removed or adjusted at any time without a reindex, because the change only affects the mapping and produces no stored values in the inverted index or in doc_values.

PUT /products/_mapping
{
  "runtime": {
    "margin_percent": {
      "type": "double",
      "script": {
        "source": """
          if (doc['cost_price'].size() == 0 || doc['sale_price'].size() == 0) {
            return;
          }
          double cost = doc['cost_price'].value;
          double sale = doc['sale_price'].value;
          if (cost == 0) return;
          emit(((sale - cost) / cost) * 100);
        """
      }
    }
  }
}

5. Using runtime fields in aggregations

A key advantage of runtime fields is that they can be used in aggregations exactly like indexed fields. A terms aggregation on a runtime field like order_value_bucket groups documents by the computed bucket value, even though that value was never permanently stored. Metric aggregations like avg or sum also work on numeric runtime fields like margin_percent, enabling complex ad hoc analyses that would previously have required adjusting the index schema.

It is important to set the right performance expectations: since Elasticsearch has to run the script for every single document in the aggregation candidate set, the compute cost of an aggregation on a runtime field scales linearly with the number of affected documents, while an aggregation on an indexed keyword field benefits from global ordinals and pre-built structures. For a quick feasibility check that is usually not an issue, but for heavily trafficked production dashboards with millions of documents per request, the difference can become noticeable.

6. Exploration phase: testing new metrics

The typical practical use case for runtime fields is an exploration phase where a team wants to define a new business metric whose exact calculation logic is not yet finalized. Instead of immediately designing a mapping and scheduling a reindex, the calculation logic is first formulated as a runtime field and tested against real production data. Stakeholders can review the results in Kibana, identify edge cases and iteratively adjust the script before a single line of infrastructure code is written.

This iterative approach significantly reduces the risk of faulty reindex cycles. A bug in the calculation logic of a runtime field can be fixed immediately in the script and tested on the next query call, without billions of documents having already been reindexed with a faulty calculation. Only once the logic is stable and the metric is regularly needed in production dashboards does the migration step to a real indexed field follow.

7. The performance tradeoff in detail

The performance difference between a runtime field and an indexed field is not a blanket statement, it depends heavily on the use case. For a query that only returns a few hundred documents through a restrictive filter anyway, the per-document script execution overhead is negligible, because the absolute number of script calls stays small. For an aggregation that potentially runs over millions of documents, for example a terms aggregation without a restricting filter, the same overhead can become noticeable and multiply the response time.

An important optimization lever is keeping the Painless script of a runtime field as simple as possible and avoiding complex calculations with loops over nested structures or expensive string operations. Elasticsearch compiles Painless scripts into bytecode and caches these compiled artifacts, but the actual per-document execution still remains more expensive than a plain doc_values read on a precomputed field. Anyone regularly using a runtime field in aggregations across the entire index should seriously consider migrating it to an indexed field.

8. Migrating from a runtime field to an indexed field

Once a runtime field has proven permanently useful, the next step is to carry the same calculation logic over into an ingest pipeline processor or directly into the application's indexing logic, so the value is computed once when the document is written and stored as a real indexed field. This migration benefits from the fact that the Painless script from the runtime field definition can be carried over practically unchanged into a script processor of an ingest pipeline, because both use the same Painless environment.

After the migration, a full reindex runs that enriches every existing document with the new, now indexed field. From this point on, the field benefits from global ordinals, terms aggregations with eager_global_ordinals, and all the other optimizations of regular fields. The original runtime field mapping can then be removed, as long as no remaining use case still needs the dynamic, unstored calculation.

PUT /_ingest/pipeline/compute_margin_percent
{
  "processors": [
    {
      "script": {
        "source": """
          if (ctx.cost_price == null || ctx.sale_price == null) {
            return;
          }
          double cost = ctx.cost_price;
          double sale = ctx.sale_price;
          if (cost == 0) return;
          ctx.margin_percent = ((sale - cost) / cost) * 100;
        """
      }
    }
  ]
}

POST /_reindex
{
  "source": { "index": "products" },
  "dest": { "index": "products_v2", "pipeline": "compute_margin_percent" }
}

9. Common mistakes and debugging

The most common mistake with runtime fields is failing to handle a missing source field in the Painless script. If a document does not contain the referenced field, doc['fieldname'].value without a prior size() check throws an exception that fails the entire query, instead of simply skipping the document. The correct approach is always an explicit size() check before accessing value, combined with an early return when the field is missing.

// WRONG: no null-check, throws on documents missing the field
{
  "script": {
    "source": "emit(doc['discount_percent'].value)"
  }
}

// RIGHT: explicit size check before accessing value
{
  "script": {
    "source": """
      if (doc['discount_percent'].size() == 0) {
        return;
      }
      emit(doc['discount_percent'].value);
    """
  }
}

A second common mistake is confusing doc access with _source access. doc always returns the first value for multivalue fields or requires explicit iteration over all values, while _source returns the full original array. Anyone defining a runtime field over an array attribute like tags while using doc instead of _source silently loses every value except the first, which leads to incomplete and misleading results in aggregations.

10. Summary

Runtime fields move field computation from index time to query time, solving a central problem of classic Elasticsearch workflows: the long latency between a new analytics idea and the first chance to test it. Via runtime_mappings in a single query or permanently in the index mapping, Painless scripts can be defined that behave like normal fields in queries, filters and aggregations, without triggering a reindex.

The price for this flexibility is the performance tradeoff: a runtime field re-runs its script on every query, while an indexed field benefits from precomputed doc_values and global ordinals. For exploration and rarely used analyses, that tradeoff is usually unproblematic; for heavily trafficked production dashboards, migrating to a real indexed field with a subsequent reindex pays off once the field has proven itself.

11. FAQ: Runtime Fields for Flexible Analytics