Snapshot and Restore Strategies for Production Elasticsearch Clusters

1. Why a single snapshot is not yet a strategy

A snapshot in Elasticsearch is a point-in-time image of one or more indices, written into an external repository. Technically, a single snapshot is created quickly, but the real challenge starts afterwards: how often is a snapshot triggered, how long is it retained, where does the repository physically live, and does a restore actually work when it is needed. A single manually created snapshot is a lucky shot, not a strategy.

In practice, many teams treat snapshots as an afterthought because the cluster already has replicated shards in normal operation and therefore feels protected against a single node failure. Replication, however, does not protect against deleted indices, faulty bulk updates, corrupted segments, or a complete data center outage. Exactly for these cases a well thought out snapshot strategy is needed, with an external repository, a regular schedule, and a documented restore path that is not tried for the first time during an actual incident.

This article covers the full chain: setting up a repository, automating it with Snapshot Lifecycle Management, and performing an actual restore plus its regular verification. Every section shows concrete API calls that can be run directly against a production or test cluster.

2. Snapshot repositories: filesystem versus S3

Before a snapshot can be created at all, a repository must be registered as the target. Elasticsearch supports several repository types as plugins, the most common being a shared filesystem (fs) and Amazon S3 (s3). A filesystem repository requires that every node in the cluster sees the same path under the same mount address, typically mounted via NFS, and that this path is explicitly whitelisted in elasticsearch.yml under path.repo. Without this entry Elasticsearch refuses, for security reasons, to register an arbitrary path as a repository target.

An S3 repository is usually the more practical choice for cloud deployments because no shared mounted storage is needed, and capacity scales practically without limit. The repository-s3 plugin must be installed on every node, and credentials are stored encrypted in the Elasticsearch keystore instead of in plain text in the configuration file. It is important that the IAM user or IAM role used only receives the minimally required permissions on the target bucket, in particular s3:PutObject, s3:GetObject, s3:ListBucket and s3:DeleteObject for cleaning up old snapshots.

Regardless of repository type, the rule holds: a repository should never sit on the same physical storage as the data nodes themselves. A repository on the same server that also holds the index data offers no protection against hardware failures of that server and undermines the whole point of an external snapshot target. Separating compute and backup storage is one of the first decisions in any solid snapshot architecture.

PUT /_snapshot/s3_backup_repo
{
  "type": "s3",
  "settings": {
    "bucket": "mironsoft-es-snapshots",
    "region": "eu-central-1",
    "base_path": "cluster-prod/snapshots",
    "compress": true,
    "server_side_encryption": true,
    "chunk_size": "1gb"
  }
}

// Filesystem alternative, requires shared mount and path.repo entry
PUT /_snapshot/fs_backup_repo
{
  "type": "fs",
  "settings": {
    "location": "/mnt/es-backups/cluster-prod",
    "compress": true
  }
}

// Verify the repository is reachable from every node
POST /_snapshot/s3_backup_repo/_verify

3. How incremental snapshots save storage and time

A common misconception is that every snapshot creates a full copy of all index data. In reality, snapshots in Elasticsearch are incremental at the segment level: on the first snapshot of an index, all Lucene segments are copied into the repository. On every subsequent snapshot of the same index, Elasticsearch compares the existing segments against those of the previous snapshot and transfers only the new or changed segments into the repository. Unchanged segments are reused by reference from the previous snapshot, not copied again.

This mechanism works because Lucene segments themselves are immutable: an existing segment is never modified, instead new segments are created on changes, while old ones are eventually merged together and deleted. A snapshot therefore only needs to transfer the delta between two points in time, which drastically reduces both the required network bandwidth and the storage consumed in the repository compared to a naive full copy on every run.

The practical consequence: a daily snapshot of a multi-terabyte index often costs only a few percent of additional storage and runtime after the first full run, as long as the change rate stays moderate. For very write-heavy indices with high merge activity, the delta per snapshot is correspondingly larger, because more segments are rewritten. This is an important factor when choosing the snapshot frequency for heavily writing workloads such as log indices.

4. Snapshot Lifecycle Management: policies instead of cron jobs

Before the introduction of Snapshot Lifecycle Management, known as SLM, teams had to build external cron jobs or orchestration tools to trigger snapshots regularly and delete old ones again. SLM moves this logic directly into the cluster: an SLM policy defines a schedule in cron format, a naming scheme for the created snapshots, the target repository, and a retention rule that specifies the minimum and maximum number of snapshots to keep before older ones are deleted automatically.

The big advantage of SLM over external scripts is that the policy is part of the cluster configuration and can be inspected, adjusted and monitored through the normal Elasticsearch API. The status of every SLM run, including success and failure timestamps, is stored directly against the repository and available via GET _slm/policy. This reduces the number of external dependencies that would otherwise need separate monitoring and makes snapshot failures visible directly in cluster monitoring.

PUT /_slm/policy/nightly-snapshots
{
  "schedule": "0 30 1 * * ?",
  "name": "<nightly-snap-{now/d}>",
  "repository": "s3_backup_repo",
  "config": {
    "indices": ["*"],
    "ignore_unavailable": true,
    "include_global_state": true
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 5,
    "max_count": 60
  }
}

// Trigger the policy manually for testing purposes
POST /_slm/policy/nightly-snapshots/_execute

// Inspect run history and last error, if any
GET /_slm/policy/nightly-snapshots?human

5. Creating snapshots and monitoring progress

Alongside the automated SLM policy, creating a snapshot manually via the API remains useful, for example before a major migration step or right before a rolling upgrade. A manually triggered snapshot follows the same API that SLM uses internally, just without an automatic schedule. By default, a snapshot request runs asynchronously in the background, the calling client gets an immediate response while the cluster keeps writing in the background.

Progress of a running snapshot can be queried through the status endpoint, which shows per shard whether it has already completed, is still in progress, or has failed. On very large clusters with thousands of shards, it is advisable not to poll status in short intervals, since determining the status itself consumes cluster resources, but rather at intervals of several minutes. A snapshot can be aborted at any time via the delete API if it is no longer needed or was started by mistake.

# Trigger a manual snapshot before a risky operation
curl -s -X PUT "https://localhost:9200/_snapshot/s3_backup_repo/pre-migration-2026-07-24?wait_for_completion=false" \
  -H "Content-Type: application/json" \
  -u elastic:changeme \
  -d '{"indices": "products,orders", "include_global_state": false}'

# Poll snapshot status without blocking on wait_for_completion
curl -s "https://localhost:9200/_snapshot/s3_backup_repo/pre-migration-2026-07-24/_status" \
  -u elastic:changeme | jq '.snapshots[0].state'

# List all snapshots in a repository, sorted by start time
curl -s "https://localhost:9200/_snapshot/s3_backup_repo/_all?sort=start_time&order=desc" \
  -u elastic:changeme

6. The restore process: from selection to shard allocation

A restore recreates one or more indices from a snapshot, either under their original name or, often more sensible in practice, under a renamed name alongside the existing index. The rename_pattern and rename_replacement parameters in the restore request allow restoring an index as products_restored instead of products, so the original index stays untouched and the restored data can be validated before it is switched live.

During the restore, the shards of the restored index are distributed across the cluster, exactly as when creating a new index. For large indices with many shards, it makes sense to control shard allocation deliberately during a particularly sensitive time window, for example through index.routing.allocation settings, to limit network load on certain nodes. A restore does not block the entire cluster, but runs as a background process just like a snapshot itself, and its progress can likewise be observed through the recovery API.

A frequently overlooked detail: a restore from a snapshot can only be written into an index that does not yet exist at the time of the restore, unless the existing index is explicitly closed beforehand. Anyone trying to restore directly into an open, already existing index of the same name gets a clear error. This behavior is intentional and prevents a restore from accidentally overwriting production data unnoticed.

POST /_snapshot/s3_backup_repo/pre-migration-2026-07-24/_restore
{
  "indices": "products",
  "rename_pattern": "(.+)",
  "rename_replacement": "$1_restored",
  "index_settings": {
    "index.number_of_replicas": 1
  },
  "ignore_index_settings": ["index.refresh_interval"]
}

// Watch recovery progress per shard until it reaches DONE
GET /_cat/recovery/products_restored?v&h=index,shard,stage,bytes_percent

7. Restore testing: why an untested snapshot is not a backup

The core rule of any backup discipline applies to Elasticsearch just as it does to any other database: a snapshot whose restore has never been tested is not a verified backup, it is an assumption. A broken repository, missing IAM permissions, an expired certificate, or an inconsistent snapshot typically only surfaces during an actual restore attempt, and that is exactly when the team is under the most time pressure.

A sensible cadence is to perform a full restore test in an isolated test environment at least once per quarter: restore a randomly chosen production snapshot into a fresh test cluster, verify document counts and sample queries against the original values, and document the required restore time. This time is an important metric for the recovery time objective, or RTO, since restoring a multi-terabyte index can take several hours depending on network bandwidth.

Automated restore tests integrate well into a CI pipeline: a weekly job restores the most recent snapshot into an ephemeral cluster, runs a set of defined validation queries, and reports the result to a monitoring system. If this job fails, it is just as critical an alert as a failed snapshot itself, because both cases carry the same consequence: in a real incident there is no working backup.

# Force-delete a snapshot stuck in state IN_PROGRESS after a node crash
curl -s -X DELETE "https://localhost:9200/_snapshot/s3_backup_repo/stuck-snapshot-2026-07-20" \
  -u elastic:changeme

# Verify no snapshot is currently blocking the repository afterwards
curl -s "https://localhost:9200/_snapshot/s3_backup_repo/_status" -u elastic:changeme

8. Snapshots versus cross-cluster replication

Snapshots are occasionally confused with cross-cluster replication, or CCR, even though both solve different problems. CCR replicates changes near real time from a leader cluster into a follower cluster and primarily serves availability across multiple data centers or regions, as well as read locality for geographically distributed users. A CCR follower cluster, however, does not protect against logical errors: if an index is accidentally deleted or overwritten with faulty data, CCR replicates that mistake into the follower cluster in near real time as well.

A snapshot, on the other hand, is a point-in-time image that stays in the repository independent of ongoing changes, until it is explicitly deleted. That is exactly what makes snapshots the right tool against logical errors and accidental deletion, while CCR is the right tool for high availability and geographic distribution. A resilient overall architecture typically combines both mechanisms: CCR for fast failover capability, snapshots as an additional, time-delayed safety net against errors that would otherwise be replicated without a brake.

9. Troubleshooting: stuck and failed snapshots

The most common failure case is a snapshot stuck in the IN_PROGRESS state, even though it is obvious that no progress is being made anymore. The cause is usually a node that crashed or restarted during the snapshot operation without the cluster being able to cleanly finish the process. In this case, calling the delete API on the stuck snapshot helps, which cleans up the internal state even if the snapshot itself never completed successfully.

A second common error is SnapshotInProgressException, which occurs when trying to start a second snapshot in the same repository while one is already running. By default, Elasticsearch only allows a single running snapshot per repository at a time. Anyone needing multiple parallel snapshots must either create multiple repositories or cleanly sequence snapshot execution instead of triggering it in parallel from multiple sources.

Repository locks caused by read_only flags, expired cloud credentials, or network timeouts on very slow storage backends are further classic causes. The master node logs usually contain the concrete error message from the repository plugin, while the plain snapshot status API often only shows a generic failure state. A systematic look into the master node logs is therefore always the first step for any failed snapshot.

10. Summary

A resilient snapshot strategy for Elasticsearch and OpenSearch consists of several interlocking pieces: a cleanly separated repository on S3 or a shared filesystem, incremental snapshots that, thanks to immutable Lucene segments, transfer only real deltas, an SLM policy that controls schedule and retention without external cron jobs, and a documented restore process with renaming to avoid endangering production data.

The most important point remains the restore test: a snapshot that has never been restored is an unproven assumption. Anyone who restores regularly, ideally in an automated way, against a test cluster knows both the actual restore duration and the integrity of the data, before a real incident answers these questions under time pressure. Combined with CCR for high availability, this creates a two-layered safety net that covers both hardware failures and logical errors.

11. FAQ: Snapshot and Restore Strategies