Data Streams for Time-Based Data Management

1. The classic pattern: manually managed time-series indices

Before data streams, time-series management typically consisted of several parts: an index template that automatically applies the matching mapping for every new index following a naming pattern like logs-2026.08.08, a write alias like logs-write that always points at the current, writable index, and an ILM policy that triggers a rollover based on size or age conditions and redirects the alias to the newly created index.

This construction works, but requires every one of these parts to interact correctly: the template must set is_write_index correctly, the ILM policy must reference the right alias, and new indices must exactly follow the expected naming pattern so that search queries through a read alias keep covering all relevant indices. A configuration mistake in any of these places often only surfaces weeks later, when a rollover fails to work as expected.

2. The data stream concept: one logical name, several backing indices

A data stream presents itself externally as a single, logical name, behind which sits an ordered sequence of hidden backing indices, named following the pattern .ds-<stream-name>-<generation>. Writes always go to the currently active, most recent backing index, while reads through the data stream name search transparently across all its backing indices, without the application needing to know their exact names.

These backing indices are marked as hidden indices and therefore do not show up by default in _cat/indices, which noticeably cleans up the overview in a cluster with many time-series data streams. Rotation to a new backing index happens automatically based on the rollover conditions attached to the data stream, without an application ever having to create a new index or redirect an alias itself.

3. A data stream is born from an index template

A data stream is not created directly, but implicitly defined through a composable index template that contains an empty configuration in the data_stream field and matches the right data streams through a naming pattern like logs-search-queries*. As soon as the first document gets written under that name, Elasticsearch automatically creates the data stream along with its first backing index, with no explicit creation command needed.

Every document in a data stream requires a timestamp field, @timestamp by default, which must be declared as such in the mapping. If this field is missing from an incoming document, Elasticsearch rejects the write, since temporal ordering is a fixed part of the data stream concept.

PUT _index_template/logs-search-queries-template
{
  "index_patterns": ["logs-search-queries*"],
  "data_stream": {},
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "query": { "type": "keyword" },
        "hits": { "type": "integer" },
        "response_time_ms": { "type": "integer" }
      }
    }
  }
}

4. ILM policy integration for automatic rollover

The actual rollover logic of a data stream comes from a linked ILM policy referenced in the same index template. The policy defines conditions such as a maximum shard size or a maximum age, and once those are met, a rollover to a new backing index triggers automatically, exactly as with classic, alias-based time series, just without needing to manually manage the alias itself.

Beyond the classic ILM policy, newer versions also offer a lighter-weight data retention rule configurable directly on the data stream, for simple use cases where only a maximum retention period matters, without needing the full flexibility of multiple ILM phases like warm or cold.

5. Writing and reading: append-only instead of arbitrary changes

Data streams are fundamentally designed for append-only writes: new documents get added through the create action, while direct updates or deletions of individual documents through the data stream name itself are only possible in a limited way, since it is not immediately clear which of the several backing indices a given document lives in.

Changes to individual documents instead require addressing the specific backing index, discoverable through a prior search that returns the relevant backing index's metadata field name. For classic log or event data, where every document stays unchanged after being written, this restriction rarely matters in practice.

6. Practical example: a search query log for later analysis

An obvious use case for data streams is logging search queries from a Magento search: every incoming search query gets written as a document with fields such as search term, hit count, applied filters, response time, and the mandatory @timestamp field, into a data stream named logs-search-queries, without the application needing to worry about index rotation or retention.

For later analysis, for example which search terms frequently return zero hits or how response time trends over time, aggregations can run directly against the data stream name, as if it were a single index. The ILM policy deletes older backing indices automatically once the configured retention period expires, so the data volume stays permanently bounded without manual cleanup.

POST logs-search-queries/_doc
{
  "@timestamp": "2026-08-08T10:15:30Z",
  "query": "running shoes men",
  "hits": 0,
  "filters": ["category:shoes"],
  "response_time_ms": 42
}

GET logs-search-queries/_search
{
  "size": 0,
  "query": { "term": { "hits": 0 } },
  "aggs": { "top_zero_hit_terms": { "terms": { "field": "query" } } }
}

7. The practical difference from the manually managed alias pattern

The biggest practical difference lies in error-proneness: with the classic alias pattern, every single component, the naming pattern of new indices, correctly setting is_write_index, and the alias reference inside the ILM policy, has to be configured correctly by hand and kept consistent across changes. Data streams fully encapsulate this logic inside the template and structurally guarantee that backing indices are named and rotated correctly.

Another difference is visibility: while classic time-series indices show up as regular, visible indices in every cluster overview and can potentially be written to directly by mistake, backing indices are hidden and not meant for direct writes, which effectively rules out the mistake of accidentally writing to the wrong, stale index.

8. Downsampling and data retention for older backing indices

For time-series data streams with very high data volume, Elasticsearch also supports downsampling, where older backing indices get aggregated into coarser time intervals, for example from minute-level to hour-level values, which noticeably reduces storage needs for historical data without deleting it entirely.

For a search query log, the exact individual query usually matters less after a few weeks than aggregated trends, which makes an ILM phase with downsampling, or a plain deletion rule once the retention period expires, a good fit, depending on whether long-term historical trend analysis is actually needed.

9. Migrating from an existing alias pattern to data streams

An existing, manually managed time-series setup cannot simply be renamed into a data stream, since the internal structure differs fundamentally. A workable path is to create a new index template with data stream configuration under a new or the same naming pattern, so that from a defined point onward, new writes flow into the data stream while the old, classic indices remain until their own retention period expires.

For a complete historical migration, the Reindex API can additionally be used to move existing legacy indices into the new data stream, keeping in mind that every migrated document needs a valid @timestamp field and that the target index fits correctly into the data stream's backing index order.

10. Summary

11. FAQ: Data Streams: The Essentials at a Glance