Index Lifecycle Management: Configuring ILM Policies in Detail

1. Why a single rollover rule is not enough for growing data

Anyone managing time series or log data in Elasticsearch runs into the same fundamental problem: a single, unbounded index eventually becomes too large for fast queries and too unwieldy for maintenance tasks such as snapshots or mapping changes. The obvious fix, switching indices regularly through rollover, only solves half the problem, because fresh and old data have completely different requirements for hardware, replica count and query speed.

This is exactly where Index Lifecycle Management comes in: it combines plain rollover mechanics with a full policy that defines what should happen to an index at every stage of its life. Instead of a rigid schedule, the result is a rule set that reacts automatically once an index reaches a certain size, age or document count, applying to each phase exactly the actions that make sense for the corresponding access frequency.

2. The four classic phases: hot, warm, cold and delete

An ILM policy splits an index's lifetime into consecutive phases. The hot phase holds the actively written index, where write load and current queries concentrate, usually on performant hardware with SSD storage. The warm phase takes over indices that are still queried but no longer written to, where optimizations such as a reduced replica count and segment compaction pay off. The cold phase is meant for rarely queried data where slightly higher latency is acceptable, while the delete phase at the end simply marks final deletion.

Each phase can define a minimum age after which the transition to the next phase is evaluated, along with its own list of actions. Importantly, a phase does not have to be traversed at all: a policy can jump straight from hot to cold if a separate warm phase offers no benefit for the use case at hand, for example because warm and cold requirements are identical anyway.

3. Rollover conditions as the trigger for phase transitions

Rollover is the central action of the hot phase and decides when a new physical index gets created and the write alias switches over to it. Its conditions can be freely combined: a maximum index size, a maximum primary shard size, a maximum age or a maximum document count. As soon as one of these conditions is met, ILM triggers the rollover, regardless of whether the other conditions have been reached yet.

This combinability matters a great deal for irregular data volumes: an index with uneven traffic, say due to seasonal spikes, would end up either oversized or undersized under a pure time condition. A size condition of roughly fifty gigabytes combined with an age condition of thirty days ensures that neither an oversized index under heavy load nor a tiny index under light load results, because whichever condition is met first takes effect.

4. Warm phase actions: shrink, force merge and allocate

Once an index is no longer written to, several optimizations become worthwhile that would be risky during the hot phase. The shrink action reduces the number of primary shards to a smaller value, which is useful for indices originally created with many shards for high write load but whose data volume after rollover fits comfortably into fewer shards. The force merge action compacts Lucene segments down to a lower count, noticeably speeding up subsequent read queries.

The allocate action, in turn, controls which nodes an index should live on during this phase, for instance cheaper hardware with more disk space instead of fast SSD storage, and frequently also reduces the replica count from two to one, since slightly lower fault tolerance is usually acceptable for data that is no longer written to. All three actions together ensure that finished indices tie up considerably fewer resources than freshly written ones.

PUT _ilm/policy/log-lifecycle-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "40gb",
            "max_age": "7d"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "allocate": {
            "number_of_replicas": 1,
            "require": { "data": "warm" }
          },
          "set_priority": { "priority": 50 }
        }
      }
    }
  }
}

5. The cold phase: cost savings without further structural work

The cold phase is primarily about cost savings rather than further structural optimization of segments. Typical actions include a further reduction of the replica count, often down to zero, as well as moving the index to even cheaper nodes with slower but less expensive storage. It is worth stressing that dropping the replica in this phase is a deliberate trade-off, since an index without a replica is lost on node failure unless a recent snapshot exists. That trade-off should therefore always be planned together with a working snapshot strategy.

Alternatively, the cold phase can mount an index as a searchable snapshot instead of keeping a full local copy. Segment data then stays in object storage while a shared cache on the cold nodes holds frequently queried blocks. This variant lowers local storage needs even further, but blurs the line with the frozen tier described later, and should be weighed deliberately against the actual query frequency of the data in question.

6. The delete phase: irreversible deletion after a fixed retention period

The delete phase, finally, needs no further configuration options beyond the minimum age after which deletion should occur, for example ninety days after rollover for log data with limited regulatory retention needs. It is important to choose that minimum age deliberately so it matches the actual retention obligation, because ILM deletes without exception and without confirmation once the condition is met.

Anyone running an SLM policy for backups alongside ILM should deliberately align both schedules: if ILM deletes an index before a matching snapshot exists, the affected data is lost for good the moment no current snapshot remains either. A sensible ordering ensures the last snapshot completes successfully before the source index gets permanently deleted, instead of letting both processes run completely independently of each other.

7. Practical example: a complete policy for log and analytics data

For a typical logging scenario with a ninety day retention period, a complete policy emerges that combines all four phases. Rollover happens after forty gigabytes or one day, whichever comes first, which is realistic for log data with strongly fluctuating volume. After seven days the index moves into the warm phase, where shrink and force merge apply, after thirty days the cold phase follows with a full replica drop, and after ninety days the delete phase kicks in.

This combination substantially reduces resource consumption over the lifetime of the data: an index that occupies several shards with two replicas on performant nodes while freshly written needs only a single, compacted shard without a replica on cheap hardware by the time it reaches the cold phase. For analytics workloads with dashboards that mostly query the last few days but occasionally need historical comparisons across months, exactly this tiered model is the key lever for sustainable cluster costs.

"cold": {
  "min_age": "30d",
  "actions": {
    "allocate": { "number_of_replicas": 0 },
    "set_priority": { "priority": 0 }
  }
},
"delete": {
  "min_age": "90d",
  "actions": {
    "delete": {}
  }
}

8. Monitoring and error handling with the ILM explain API

ILM operates asynchronously in the background, which makes visibility into the current state of every index essential. The explain API returns, for every index managed by a policy, the current phase, the remaining time until the next transition and, in case of failure, a concrete error message. Common failure causes include missing target nodes for an allocate action, for example because no node carries the required attribute, or a shrink attempt on an index whose shard count is already smaller than the target size.

An index that enters an error state stays stuck in its current phase until the error is resolved and the action is retried through the retry API. Anyone running ILM in production should therefore build regular monitoring around this error state, because an unnoticed stuck index can mean that shrink or force merge never actually run and the expected resource savings never materialize, without this being obvious at first glance.

9. Common pitfalls when configuring ILM

A frequent mistake is confusing min_age with an absolute clock time: phase duration is measured from the rollover point of the specific index, not from the creation of the very first index in the sequence. Another typical stumbling block is forgetting the set_priority action, which determines the order in which indices become available again after a cluster restart. Without it, hot and delete candidates rank equally, unnecessarily prolonging recovery time.

Also commonly underestimated is the need to attach an ILM policy to an index template with the matching rollover alias before the very first index is created, since assigning a policy retroactively to an already running index requires extra manual steps. Finally, the force merge action should never be applied to an index that is still being written to, because force merging an active index generates significant I/O load and can noticeably degrade write performance while it runs.

10. Summary

11. FAQ: ILM policies in Elasticsearch: the essentials at a glance