Rolling Upgrades Without Downtime

1. Why a rolling upgrade is the default path

A rolling upgrade updates an Elasticsearch cluster node by node, so that at every point in time enough nodes stay online to keep serving read requests and write operations. The alternative, a full cluster restart, where all nodes are stopped, updated, and started again at the same time, requires planned downtime and is only needed in rare cases today, for example for certain large version jumps with incompatible data formats. For the vast majority of version updates, a rolling upgrade is the default path, because it combines availability with staying current.

The core of the procedure sounds simple but is demanding in the details: a node is taken out of serving new requests, shut down cleanly, restarted with the new version, and must successfully rejoin the cluster before the next node takes its turn. During this process, the remaining replica shards on the other nodes take over the requests that would otherwise have been served by the node currently being updated. Without sufficient replication, meaning number_of_replicas: 0, a true zero-downtime rolling upgrade does not work, because a primary shard on the stopped node is simply unreachable for that period.

This article describes the full process of a rolling upgrade, from version checking through shard allocation to automation, including the special cases around master nodes and the most common failure patterns seen in practice.

2. Version compatibility: what is allowed between which versions

Elasticsearch only permits a rolling upgrade within clearly defined compatibility boundaries. Within the same major version, for example from 8.10 to 8.15, a rolling upgrade is always possible, as long as no version marked as not directly compatible is skipped. When moving between two major versions, for example from 7.x to 8.x, an additional rule applies: the cluster must first be updated to the latest minor version of the old major version before the jump to the new major version begins. This last minor version usually contains the necessary compatibility bridges and deprecation warnings for the next big step.

A direct jump across more than one major version, for example from 6.x straight to 8.x, is not supported by Elasticsearch and requires either several consecutive upgrade cycles or a full reindex into a new cluster. Before any larger rolling upgrade, it is therefore worth checking the official compatibility matrix of the respective version, since the exact rules can differ slightly from version to version, particularly for deep internal changes such as the removal of mapping types between 6.x and 7.x.

Similar principles apply to OpenSearch within its own version line, though the divergence since the fork from Elasticsearch means that a direct rolling upgrade path between the two systems fundamentally does not exist. A switch from Elasticsearch to OpenSearch or vice versa is always a migration with a reindex, not a rolling upgrade in the strict sense.

# Check current cluster version and node versions before starting
curl -s "https://localhost:9200/" -u elastic:changeme | jq '.version.number'
curl -s "https://localhost:9200/_cat/nodes?v&h=name,version,node.role" -u elastic:changeme

# Elasticsearch upgrade path rule for major version jumps:
# 7.x -> must first reach the latest 7.17.x minor release
# 7.17.x -> then rolling upgrade to 8.x is supported
# Direct 6.x -> 8.x is NOT supported, requires intermediate steps

3. Preparation: backup, health check and deprecation log

Before every rolling upgrade, a full snapshot of the cluster comes first. Even though a rolling upgrade normally does not change any data, a fresh snapshot protects against unexpected problems, for example a plugin that turns out to be incompatible after the upgrade. In addition, the cluster health status before starting must absolutely be green, because a rolling upgrade that begins on an already unstable cluster with yellow or red status significantly increases the risk of data loss if a node is missing on top of that.

The deprecation log, available through the deprecation API, shows every feature, setting, and query syntax form that has been removed or changed in the target version and is still being used in the current cluster. Anyone who ignores this log before a rolling upgrade risks applications suddenly throwing errors after the upgrade, because a used API form no longer exists. A clean preparation step checks every reported deprecation, adapts affected queries and mappings, and only then performs the rolling upgrade.

# Full cluster health check before starting
curl -s "https://localhost:9200/_cluster/health?pretty" -u elastic:changeme

# Deprecation warnings relevant for the target version
curl -s "https://localhost:9200/_migration/deprecations?pretty" -u elastic:changeme

# Manual snapshot right before the upgrade window
curl -s -X PUT "https://localhost:9200/_snapshot/s3_backup_repo/pre-upgrade-2026-07-24?wait_for_completion=true" \
  -u elastic:changeme

4. Disabling shard allocation before stopping a node

The single most important step in every rolling upgrade is to disable shard allocation shortly before stopping a node. Without this step, the cluster immediately starts, as soon as a node leaves the cluster, redistributing and re-replicating the missing shards across the remaining nodes. For a short, planned restart as part of a rolling upgrade, this redistribution is unnecessary, it costs massive network bandwidth and I/O, only to be undone minutes later once the node comes back.

Setting cluster.routing.allocation.enable to the value primaries still allows allocation of primary shards, but blocks redistribution of replica shards during the short downtime of the single node. That is the decisive trade-off: the cluster stays fully functional but does not waste resources on a redistribution that will be reversed again anyway. It is also advisable to issue a POST _flush/synced, or on newer versions a regular flush, before stopping, so the node has to replay as few translog operations as possible when it rejoins.

// Step 1: disable replica shard reallocation before stopping a node
PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}

// Step 2: flush to speed up recovery after restart
POST /_flush

// Step 3: stop Elasticsearch on the target node via the service manager
// systemctl stop elasticsearch

5. The single node upgrade step in detail

Once the node has been cleanly stopped, the actual update follows: package update through the system package manager, container image swap, or binary replacement, depending on the deployment model. It is important not to accidentally overwrite configuration files in the process, and to update plugins, if used, to the same version level as the Elasticsearch core, since incompatible plugin versions can prevent the node from starting. After the update, the node is restarted and must automatically rejoin the cluster.

The critical waiting point in this step is to only consider the node successfully upgraded once it reappears as an active node in the cluster state and all shards assigned to it have reached the STARTED status. Moving on to the next node too early, before the current one is truly fully synchronized, can create a data-loss risk if a further failure happens at the same time, because there would temporarily not be enough copies of a shard in the cluster.

# After restarting the upgraded node, wait until it rejoins successfully
curl -s "https://localhost:9200/_cat/nodes?v&h=name,version,node.role" -u elastic:changeme

# Confirm all shards on the node are STARTED, not INITIALIZING or RELOCATING
curl -s "https://localhost:9200/_cat/shards?v" -u elastic:changeme | grep <node-name>

# Wait for cluster health to return to green before touching the next node
curl -s "https://localhost:9200/_cluster/health?wait_for_status=green&timeout=5m" -u elastic:changeme

6. Re-enabling allocation and waiting for green

Once a node is fully back in the cluster, cluster.routing.allocation.enable is reset to all, so the normal allocation logic takes over again and the cluster can pick up any remaining repair work on shard distribution on its own. Only after that does the health status start to realistically converge toward green, because as long as allocation is restricted to primaries, any missing replica shards remain permanently stuck in the UNASSIGNED state.

A common mistake at this point is moving straight on to the next node without re-enabling allocation. The cluster then appears deceptively stable, because primaries keeps working, but replica coverage drops with every further node in the rolling upgrade, without any obvious error, until at some point too few copies of a shard exist. The consistent sequence of disable, upgrade node, re-enable, wait for green, must therefore be followed for every single node without exception.

7. Special considerations for master and coordinating nodes

Master-eligible nodes deserve special attention during a rolling upgrade, because the cluster always needs a quorum of master nodes to agree on cluster state changes. In a typical configuration with three dedicated master nodes, the cluster tolerates the loss of a single master node during its upgrade without losing the ability to elect a new active master. If master nodes are upgraded one after another without waiting for a stable quorum in between, the cluster risks temporarily being unable to elect an active master at all, which blocks write operations.

The currently active master node should deliberately be treated as the last of the master-eligible nodes in the upgrade order, so a new master does not have to be elected unnecessarily more than once. Coordinating-only nodes without a data role, on the other hand, are uncritical, because they hold no shards and a restart only briefly interrupts incoming client connections, which load balancers usually redirect automatically to the remaining coordinating nodes.

8. Automating the process in CI/CD

A manually performed rolling upgrade is still manageable with three nodes, but quickly becomes error-prone with twenty or more nodes if every step is executed by hand. Configuration management tools like Ansible, or custom orchestration scripts, automate the entire process: check health, disable allocation, update node, wait for successful rejoin, re-enable allocation, wait for green, only then move to the next node. This sequence can be expressed as an idempotent playbook that automatically aborts on any failure instead of continuing incorrectly.

In Kubernetes environments with the Elastic Cloud on Kubernetes operator, or similar operators for OpenSearch, the operator automatically handles a large part of this logic, including controlling pod disruption budgets that prevent too many nodes from restarting at the same time. Even here, the rule holds: version compatibility should be checked manually before an auto-upgrade, because the operator orchestrates the process but cannot automatically verify whether the target version is supported for the current cluster state.

# Simplified Ansible task sequence for one node in a rolling upgrade
- name: Disable shard allocation before stopping the node
  uri:
    url: "https://{{ es_host }}:9200/_cluster/settings"
    method: PUT
    body_format: json
    body:
      persistent:
        cluster.routing.allocation.enable: primaries

- name: Stop Elasticsearch service on the target node
  systemd:
    name: elasticsearch
    state: stopped

- name: Upgrade Elasticsearch package to the target version
  apt:
    name: "elasticsearch={{ target_version }}"
    state: present

- name: Start Elasticsearch service again
  systemd:
    name: elasticsearch
    state: started

- name: Wait until cluster health returns to green
  uri:
    url: "https://{{ es_host }}:9200/_cluster/health?wait_for_status=green&timeout=5m"
  register: health_check
  until: health_check.json.status == "green"
  retries: 30
  delay: 10

- name: Re-enable shard allocation for the whole cluster
  uri:
    url: "https://{{ es_host }}:9200/_cluster/settings"
    method: PUT
    body_format: json
    body:
      persistent:
        cluster.routing.allocation.enable: all

9. Common failure patterns during a rolling upgrade

The most common problem is a node that fails to rejoin the cluster after restarting, usually because of an incompatible plugin version, a certificate error following a security configuration change made in parallel, or because discovery.seed_hosts no longer resolves correctly after a network change. In this case, the node stays isolated while the rest of the cluster keeps running with reduced capacity. The logs of the affected node almost always show the concrete cause right at startup.

A second classic problem is a cluster that gets stuck in yellow status after several node upgrades, because re-enabling shard allocation was forgotten after one step. Diagnosis is simple: GET _cluster/settings immediately shows whether cluster.routing.allocation.enable is still set to primaries. A third failure pattern involves split-brain-like situations around master nodes, when too many master-eligible nodes are upgraded at once and the quorum briefly drops below threshold, which is reliably avoided by the sequential approach described in the previous section.

10. Summary

A rolling upgrade moves a production Elasticsearch or OpenSearch cluster to a new version without planned downtime, as long as the sequence is followed consistently: preparation with a snapshot and health check, setting shard allocation to primaries before every node stop, updating the node, waiting for a full rejoin, re-enabling allocation, and only moving to the next node once green. Master-eligible nodes need extra attention for the quorum, while coordinating nodes are uncritical.

The version compatibility rules determine how many intermediate steps a rolling upgrade needs, especially for major version jumps, which always have to pass through the latest minor version of the old series. Anyone who automates this process and performs it regularly, instead of postponing upgrades for years, reduces the risk of large, risky version jumps and keeps the cluster continuously on a supported, security-patched release.

11. FAQ: Rolling Upgrades Without Downtime