Cross-Cluster Replication for Disaster Recovery: Keeping Leader and Follower Indices in Sync

1. What cross-cluster replication is and which problem it solves

Cross-cluster replication continuously replicates an index from a leader cluster to one or more follower clusters, usually in a spatially separate region or a different data center. Unlike an ordinary, in-cluster replica, this replication runs asynchronously across the cluster boundary and therefore tolerates noticeable network latency between the involved locations.

The problem it solves is clearly defined: if the leader cluster fails entirely, for instance due to an outage of the underlying cloud region, the follower cluster provides a near current copy of the data at an independent location, ready to be switched into production without waiting for the significantly slower restore from a snapshot.

2. Leader and follower indices: architecture in detail

A follower index is technically a special, read-only index that internally reads the operations of its associated leader index from that leader's transaction log and applies them to itself in the same order. That preserves document ordering and thereby the internal consistency of the index, instead of simply re-indexing the data as an independent copy.

Because a follower index internally reads from the leader's transaction log, CCR only supports indices with soft deletes, which is the default in current Elasticsearch versions anyway. The follower index itself doesn't accept direct write requests from clients as long as the replication relationship is active, and in that state serves purely as a read copy.

PUT eu_follower_products/_ccr/follow
{
  "remote_cluster": "eu_leader_cluster",
  "leader_index": "products",
  "max_read_request_operation_count": 5120,
  "max_outstanding_read_requests": 12
}

3. Setup: remote cluster and replication agreements

CCR requires, similar to cross-cluster search, a remote cluster connection from the follower cluster to the leader cluster. Through that connection, the follower cluster actively reads changes from the leader, meaning replication is initiated and driven by the follower, not actively pushed from the leader to the follower.

For recurring patterns, for instance daily-rotating indices of a time-series based logging or order index, an auto-follow pattern can additionally be set up, which automatically replicates newly created leader indices matching the pattern as followers, without a replication relationship having to be set up manually for every new index.

PUT _ccr/auto_follow/orders-pattern
{
  "remote_cluster": "eu_leader_cluster",
  "leader_index_patterns": [ "orders-*" ],
  "follow_index_pattern": "{{leader_index}}"
}

4. How close to real time is the synchronization really

CCR replicates changes in short, continuous cycles, typically within seconds, as soon as new operations become available in the leader index's transaction log. Under normal load, the so-called replication lag usually stays well under a minute, but it can rise noticeably under very high write load on the leader or with limited network bandwidth between regions.

It's important to note that, despite the small delay, CCR is not a synchronous replication model: a write on the leader is already considered successful before it has arrived on the follower. In the worst case, for instance a sudden total outage of the leader exactly while a replication cycle is in progress, the most recent, not yet replicated operations can be lost as a result.

5. Difference from snapshot-based backups: RPO and RTO

A classic snapshot backup gets created periodically, for instance hourly or daily, which creates a potential data loss equal to the interval between two snapshots, the so-called recovery point objective, RPO for short. In an outage shortly before the next scheduled snapshot, all changes since the last snapshot are correspondingly lost.

CCR reduces the RPO to the actual replication lag, typically a few seconds to minutes. The recovery time objective, the time needed to restore a functioning system, also turns out significantly lower with CCR, because the follower cluster is already fully running and merely needs to switch from follower to leader operation, instead of first fully restoring a snapshot from the repository.

6. Failover scenario: switching to the follower cluster

If the leader cluster actually fails, the affected follower index gets detached from its replication relationship via the pause and subsequent unfollow commands and converted into a regular, writable index. Only after this step can the follower cluster accept write requests for the affected index at all.

After that, the application layer, in the case of a Magento store the search configuration, needs to be switched over to the endpoints of the former follower cluster, for instance via a DNS switch or a changed environment configuration. This entire sequence should exist beforehand as a concrete, documented runbook and be tested regularly, so it never has to be improvised in a real emergency.

# Detach the follower index from replication and make it writable
curl -X POST "follower-cluster:9200/eu_follower_products/_ccr/pause_follow"
curl -X POST "follower-cluster:9200/eu_follower_products/_close"
curl -X POST "follower-cluster:9200/eu_follower_products/_ccr/unfollow"
curl -X POST "follower-cluster:9200/eu_follower_products/_open"

7. Failback: returning to the original leader

Once the original leader cluster is available again, it typically holds stale data compared to the former follower that has since become production. Directly bringing the old leader back as a write target would lead to conflicting, diverging data states and is therefore not a safe option.

The usual path is to reverse the replication direction: the restored, former leader gets set up as a new follower of the now active cluster, catches up on the changes that occurred during the outage, and only once both clusters are fully in sync again can a planned switch back to the original location happen, if desired.

8. Monitoring replication lag and auto-follow patterns

The follow stats API returns detailed metrics per follower index, among them the number of not yet processed operations and the time distance between leader and follower state. A continuously growing lag is an early warning sign that either the network connection between regions is saturated, or the follower cluster can no longer keep up with the leader's write rate.

The status of all auto-follow patterns should additionally be checked regularly, so newly created indices matching the pattern actually get replicated reliably. A silently failed auto-follow pattern often goes unnoticed during normal operation, but becomes a real problem in an emergency, when precisely the newest, most important index turns out never to have been replicated.

# Query replication lag for a specific follower index
curl "follower-cluster:9200/eu_follower_products/_ccr/stats?pretty"

# Check the status of all auto-follow patterns
curl "follower-cluster:9200/_ccr/auto_follow/stats?pretty"

9. License and cost aspects, plus limits of CCR

Cross-cluster replication is a commercial feature and requires at least a Platinum license, or the corresponding OpenSearch alternative, if a pure open source distribution without commercial support is in use. Those license costs need to be weighed against the RPO and RTO benefits gained over a pure snapshot strategy.

On top of that, a fully replicated follower cluster doubles the required infrastructure, since it runs continuously in production instead of only being activated when needed. For smaller Magento operations without strict availability requirements, a combination of regular snapshots and a documented, manual restore procedure therefore often remains the more economical choice.

10. Summary

11. FAQ: Cross-Cluster Replication: The Essentials at a Glance