Hot-Warm-Cold Architecture for Growing Data Volumes

1. Why a hot-warm-cold architecture is needed at all

Log and time-series data have a typical access pattern: the last hours or days get queried intensively, while older data is queried rarely if at all, yet still needs to be retained for compliance or analysis reasons. A cluster that treats all data equally and stores it on identical, uniformly expensive SSD hardware wastes budget on performance nobody needs anymore for older data. This is exactly the problem a hot-warm-cold architecture solves.

The basic principle of a hot-warm-cold architecture: data nodes are split into multiple tiers, each with different hardware specifications, and indices automatically move through their lifecycle from the expensive, fast hot tier through cheaper intermediate stages down to very cost-efficient cold or frozen storage. This migration does not happen manually, it runs through index lifecycle management policies that translate time or size criteria into concrete actions.

For organizations with growing log volumes, for example from application monitoring, security auditing or IoT sensor data, a hot-warm-cold architecture is usually not an optional nice-to-have but an economic necessity: without tiering, the infrastructure bill grows linearly with data volume, with tiering it grows far more slowly, because only a small, current share of data sits on expensive hardware.

2. The hot tier: fast hardware for fresh data

The hot tier hosts the currently written indices and carries the entire write load of a hot-warm-cold architecture. These nodes need the fastest available hardware: NVMe SSDs for high indexing throughput, enough CPU cores for parallel segment merges, and enough RAM for both heap and file cache. Since this tier both writes and reads most frequently, the hot tier is usually the most expensive component per gigabyte in the entire cluster.

How long an index stays in the hot tier depends on the use case, but for log data it typically ranges from a few hours to a few days. During this phase, an index benefits maximally from the fast hardware, both for incoming write load and for the dashboards and alerts that usually operate on the most recent data. A well-sized hot-warm-cold architecture deliberately keeps the hot tier small, because every additional day on expensive hardware directly increases cost.

3. The warm tier: cheaper hardware for less frequent access

The warm tier holds indices that are no longer written but still needed regularly for search and analysis, for instance for weekly reports or trend analysis over the last thirty days. These nodes can run on slower but significantly cheaper SSDs or even capable HDDs, since no write load remains and read latency is less critical than in the hot tier.

An important part of the transition into the warm tier within a hot-warm-cold architecture is force merging: closed, no-longer-written indices get compacted down to a single Lucene segment per shard, which saves storage space and speeds up read operations, since fewer segments need to be searched. Replica count is also frequently reduced in the warm tier, because the failure risk for historical, immutable data can be assessed differently than for actively written hot data.

4. Cold and frozen tier: long-term archiving

The cold tier stores data that is rarely but occasionally still queried, for example for compliance audits or legal requests with a long retention period. Highly compressed, cheap storage is often used here, sometimes with reduced replica count or even searchable snapshots, where the actual data resides in an object store like S3 and only gets temporarily loaded into the cluster when needed.

The frozen tier is the final stage of a complete hot-warm-cold architecture: data lives almost entirely in object storage, local node resources are used only minimally for caching. Search requests against frozen data are noticeably slower because data has to be fetched from the object store first, but storage cost per gigabyte is a fraction of what the same dataset would cost in the hot tier. For data that is practically never queried but theoretically might need to be someday, that is a sensible trade-off.

5. Configuring node attributes and data tier roles

Technically, a hot-warm-cold architecture is implemented through node roles that, since more recent Elasticsearch versions, can be set directly as data_hot, data_warm, data_cold and data_frozen in node.roles. In older setups, or for finer control, custom node attributes can additionally be assigned, for example node.attr.data_tier: hot, combined with shard allocation filtering to steer indices deliberately onto specific node groups.

What matters during configuration: every tier needs at least one, better several nodes for redundancy, and network topology should account for the fact that data gets physically copied between nodes during a tier change, which takes time and bandwidth for large indices. A hot-warm-cold architecture with too few warm nodes for the incoming data volume leads to migration bottlenecks even if hot and cold tiers are sized adequately.

# elasticsearch.yml - warm tier data node
node.name: warm-node-01
node.roles: [ data_warm, data_cold ]

# Slower, cheaper storage class is expected here, not NVMe
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

discovery.seed_hosts:
  - master-node-01:9300
  - master-node-02:9300
  - master-node-03:9300

6. ILM policies: automating the lifecycle

Index lifecycle management is the tool that makes a hot-warm-cold architecture practical in the first place. An ILM policy defines phases, hot, warm, cold, frozen and delete, each with conditions for when an index transitions to the next phase, and actions describing what should happen in that phase, such as rollover, force merge, shrink, or moving to different node attributes. Once defined, the entire migration runs automatically in the background, with no manual intervention.

The policy gets attached to an index template, so every newly created index of a time series automatically follows the same hot-warm-cold architecture. This is especially important for log pipelines with daily or hourly new indices: without a central policy, every single index would need to be managed manually, which is practically impossible with hundreds or thousands of indices.

PUT _ilm/policy/logs-hot-warm-cold-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": { "max_primary_shard_size": "40gb", "max_age": "1d" },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "3d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "allocate": { "number_of_replicas": 1, "require": { "data_tier": "warm" } },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "allocate": { "number_of_replicas": 0, "require": { "data_tier": "cold" } },
          "set_priority": { "priority": 0 }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": { "delete": {} }
      }
    }
  }
}

7. Choosing rollover criteria correctly

The rollover action in the hot phase is the foundation of any time-based hot-warm-cold architecture: instead of writing to a single, indefinitely growing index, rollover automatically creates a new index once defined criteria are met. The most common criteria are max_primary_shard_size for shard sizing and max_age as a time ceiling, whichever criterion is reached first triggers the rollover.

A common mistake is relying only on max_age, for example a fixed daily rollover regardless of actual data volume. On days with unusually high traffic this produces oversized shards, on quiet days unnecessarily small ones. Combining size and time criteria makes a hot-warm-cold architecture resilient against fluctuating data volume, without shard sizes needing manual readjustment.

GET logs-2026.07.20-000003/_ilm/explain

// Shows exactly which phase and action an index is currently in,
// invaluable for debugging why a migration has not happened yet
{
  "indices": {
    "logs-2026.07.20-000003": {
      "index": "logs-2026.07.20-000003",
      "phase": "warm",
      "action": "forcemerge",
      "step": "wait_for_forcemerge_step",
      "age": "3.2d"
    }
  }
}

8. Calculating actual cost savings

The economic benefit of a hot-warm-cold architecture can be roughly estimated by calculating storage cost per gigabyte on each tier against the distribution of data volume over time. A typical log setup with seven days in the hot tier, twenty three days in the warm tier and a further three hundred sixty five days in the cold tier shifts the bulk of total volume onto the cheapest stage, while only a small, current share occupies the expensive hot hardware.

On cloud infrastructure, this effect is especially easy to measure: NVMe-optimized instance types cost significantly more per gigabyte than HDD-based or object-storage-backed alternatives. For a data volume that keeps growing continuously over a year, a consistently implemented hot-warm-cold architecture can cut total storage cost by a substantial share compared to a uniform cluster running entirely on expensive hardware.

GET _cat/allocation?v&h=node,disk.indices,disk.used,disk.total,disk.percent

// Compare disk usage across tiers to verify the cost distribution
// matches expectations, most volume should sit on warm and cold nodes
node           disk.indices disk.used disk.total disk.percent
hot-node-01    180gb        210gb     500gb      42
warm-node-01   1.2tb        1.3tb     2tb        65
cold-node-01   4.8tb        4.9tb     8tb        61

9. Common pitfalls in implementation

A common mistake when introducing a hot-warm-cold architecture is underestimating tier migration: copying large shards from the hot to the warm tier needs network bandwidth and time, which causes backlogs at too tightly scheduled rollover intervals if multiple indices need to migrate simultaneously. Realistic capacity planning for migration bandwidth therefore belongs in the design from the start.

A second pitfall is overly aggressive reduction of replica count in later tiers: zero replicas in the cold tier mean a single node failure causes genuine data loss, not just temporary unavailability. For data with compliance requirements, this decision should be made deliberately and ideally backed by snapshots in a separate object store, instead of relying solely on the replicas held inside the cluster.

// Register a snapshot repository as a safety net for zero-replica cold data
PUT _snapshot/cold-tier-backup
{
  "type": "s3",
  "settings": {
    "bucket": "es-cold-tier-snapshots",
    "region": "eu-central-1"
  }
}

// Automate daily snapshots so zero replicas do not equal zero durability
PUT _slm/policy/cold-daily-snapshots
{
  "schedule": "0 30 1 * * ?",
  "name": "<cold-snap-{now/d}>",
  "repository": "cold-tier-backup",
  "config": { "indices": ["*-cold-*"] },
  "retention": { "expire_after": "90d", "min_count": 5, "max_count": 30 }
}

10. Summary

A hot-warm-cold architecture matches hardware cost to the actual access pattern of time-series and log data: fast, expensive nodes for fresh, actively written data, cheaper nodes for infrequent access, and minimal object storage cost for data that is practically never but theoretically still needed. ILM policies automate the entire migration through rollover, shrink, force merge and allocation moves, without manual intervention for every single index.

The biggest lever lies in combining realistic rollover criteria, adequately sized intermediate tiers for migration, and a deliberate decision about replica reduction in later phases. Anyone who plans these three aspects from the start achieves noticeable cost savings with a hot-warm-cold architecture, without giving up searchability of historical data.

11. FAQ: Hot-Warm-Cold Architecture in Elasticsearch