Cluster Architecture: Distributing Master and Data Nodes Correctly

1. Why node roles are the foundation of every cluster architecture

Every Elasticsearch node starts with all roles enabled by default: it is master-eligible, stores data, processes ingest pipelines and accepts search requests. For a local test or a two-node cluster that is practical. Once a cluster runs in production under real load, though, exactly this mixing becomes the biggest risk to the cluster architecture: a node that simultaneously computes large aggregations, hosts shards and participates in master elections can run into memory pressure and destabilize the control plane, even though the actual problem was just a single expensive search request.

A well thought out cluster architecture therefore separates roles along clear responsibilities: master-eligible nodes manage metadata and cluster state, data nodes carry the actual indexing and search load, ingest nodes transform documents before writing, and coordinating nodes aggregate requests across many shards. This separation is not over-engineering for small setups, it is a precondition for a cluster to scale predictably as data volume grows, instead of degrading uncontrollably under load spikes.

Role configuration since Elasticsearch 7.9 happens through node.roles in elasticsearch.yml, as an explicit list instead of the old boolean flags. This list is the central lever of every cluster architecture decision: it defines which task a physical or virtual host takes on in the cluster, and therefore also how it needs to be sized.

2. Master-eligible nodes: cluster state and elections

Master-eligible nodes (node.roles: [master]) are responsible for the cluster state: index metadata, mappings, settings, shard allocation and the list of all cluster members. Exactly one node in the cluster holds the active master role at any point in time, all other master-eligible nodes stand by as candidates. If the active master fails, the remaining master-eligible nodes elect a successor through a Raft-like consensus algorithm, without losing data.

What matters for any cluster architecture is that master work is CPU-light but latency-sensitive: cluster state publication needs to propagate quickly to all nodes so that shard allocations stay consistent. A master node that also handles heavy aggregation queries can, through garbage collection pauses, delay state publication and in the worst case trigger unwanted master switches. That is the core advantage dedicated master nodes offer over mixed roles.

3. Data nodes: storage load and shard distribution

Data nodes carry the actual workload of an Elasticsearch installation: they store shards on disk, execute indexing operations and answer search requests using the Lucene layer underneath. In modern versions, data nodes are further subdivided into data_content, data_hot, data_warm, data_cold and data_frozen, enabling a tiered storage architecture across hardware of varying performance, covered in more detail in the dedicated hot-warm-cold article of this series.

For the basic cluster architecture, what matters first is: data nodes need enough RAM for the JVM heap and the operating system file cache, fast SSDs for indexing throughput, and enough CPU cores for parallel segment merges. Every role activated additionally on a data node, such as master eligibility, competes for the same resources as the actual indexing and search load, which is why in production clusters from around six nodes onward, role separation brings noticeable stability gains.

# elasticsearch.yml - dedicated data node
node.name: data-node-01
node.roles: [ data_hot, data_content ]

# Storage and memory tuning for a data-heavy node
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

# Discovery: point to the master-eligible nodes, not to other data nodes
discovery.seed_hosts:
  - master-node-01:9300
  - master-node-02:9300
  - master-node-03:9300

# Shard allocation awareness by availability zone
node.attr.zone: eu-central-1a
cluster.routing.allocation.awareness.attributes: zone

4. Ingest nodes: pipelines before indexing

Ingest nodes (node.roles: [ingest]) run ingest pipelines before a document is actually indexed: grok parsing of log lines, GeoIP enrichment, field renaming, script processors, or splitting documents. In small clusters the ingest role usually runs combined with the data role, because the additional CPU load from pipelines stays moderate.

For log and metrics pipelines with high document throughput and compute-heavy processors, such as complex grok patterns or Painless script processors, a dedicated ingest layer pays off. It decouples the CPU load of preprocessing from the indexing load on the data nodes and prevents an expensive pipeline update from suddenly raising write latency for the entire cluster. This decoupling is an often overlooked building block of a resilient cluster architecture.

5. Coordinating nodes: load distribution for search requests

A coordinating node has none of the other roles enabled (node.roles: []) and serves purely as the entry point for client requests. It accepts a search request, distributes it to all relevant shards on the data nodes using the scatter-gather approach, and merges the partial results before sending the final response to the client. This merging, especially for large aggregations across many shards, costs heap memory and CPU.

In clusters with many shards or complex dashboards, such as Kibana visualizations with deep aggregations, a dedicated coordinating layer prevents this merging work from adding extra load to the data nodes. For smaller clusters this role is optional: any data node can implicitly act as coordinator for requests it accepts itself. A dedicated cluster architecture decision for coordinating-only nodes usually pays off starting around ten data nodes or with very heterogeneous search traffic.

6. Dedicated master nodes: why they secure cluster stability

The most important principle for a production-grade cluster architecture is: master-eligible nodes should run dedicated in any cluster from medium size upward, meaning without data, ingest or coordinating role. A dedicated master node needs comparatively little hardware for this, typically four virtual CPUs and eight to sixteen gigabytes of RAM are enough for most clusters, because it hosts no shards and processes no search load.

The stability gain comes from physically decoupling the control plane from the data plane: a full heap on a data node caused by an unexpectedly expensive aggregation can no longer affect the master election, because the master processes run on completely separate hardware. In practice this effect shows up especially during reindexing or recovery phases, when data nodes are already under increased load: with dedicated master nodes, cluster state management stays unaffected, with mixed roles such phases risk an unstable cluster with frequent master switches.

GET _cat/master?v

// Confirms which node currently holds the active master role,
// useful to verify stability after a rolling restart of the cluster
id             host            ip           node
a1B2c3D4E5f    10.0.1.11       10.0.1.11    master-node-01

7. Minimum sizing for production clusters

For production clusters, the minimum sizing rule is: three dedicated master-eligible nodes, so that even if one node fails, a quorum of the remaining two nodes is still available for a master election. The number of data nodes depends on data volume, replication factor and desired throughput, but should never drop below two for redundancy reasons, three or more spread across different availability zones is better.

For small clusters with limited budget, a compromise is common: three nodes take on both master and data roles together. That is acceptable for test environments and small internal applications, but should be avoided for business-critical production workloads. Once data volume or request rate grows, migrating to dedicated roles without downtime is possible by adding new master-only nodes to the cluster and gradually converting the existing nodes to pure data roles.

8. Avoiding split brain: quorum and discovery settings

A split brain occurs when a network partition causes two parts of a cluster to simultaneously believe they hold the active master, resulting in conflicting cluster states. Since the Zen2 discovery protocol from Elasticsearch 7 onward, this is structurally prevented: a master election is only valid if a real majority, meaning a quorum, of master-eligible nodes agrees. With three master-eligible nodes, a valid election needs at least two votes, a single isolated node can never declare itself master.

This is exactly why an even number of master nodes is an antipattern for any cluster architecture: with four nodes and a two-to-two network partition, no side has a real majority, the cluster becomes unable to act instead of continuing to operate. Initial bootstrap configuration happens once through cluster.initial_master_nodes on the first startup of a new cluster, after that internal cluster state management automatically tracks the membership list.

# elasticsearch.yml - master-eligible node, bootstrap of a new cluster
node.name: master-node-01
node.roles: [ master ]

# Bootstrap only on first cluster startup, remove afterwards
cluster.initial_master_nodes:
  - master-node-01
  - master-node-02
  - master-node-03

discovery.seed_hosts:
  - master-node-01:9300
  - master-node-02:9300
  - master-node-03:9300

# Keep master nodes lean, no heavy JVM heap needed for control plane work

GET _cat/nodes?v&h=name,node.role,master,heap.percent,ram.percent,cpu

// Example response, "m" marks master-eligible, "d" marks data role
name             node.role   master heap.percent ram.percent cpu
master-node-01   m           *      12           35          2
master-node-02   m           -      9            33          1
master-node-03   m           -      10           34          1
data-node-01     d           -      68           81          45
data-node-02     d           -      71           79          52

9. Common mistakes in node role distribution

The most frequent mistake in grown clusters is an even number of master-eligible nodes, usually the result of gradually adding hardware without checking the quorum math. A second common mistake: master nodes are run combined with the data role because it saves hardware initially, but that directly translates into cluster instability as search load grows.

A third mistake concerns network separation: master-eligible nodes should sit in different availability zones or at least different physical racks, so a single hardware or network failure does not immediately threaten the quorum. Anyone running all three master nodes in the same zone formally has the correct cluster architecture, but still loses the entire cluster's ability to act if that zone fails.

A fourth, often overlooked mistake is missing monitoring of the quorum situation itself: teams watch heap usage and shard allocation but forget alerting for the loss of a master-eligible node. If, in a three-node setup, a second master node fails before the first one has been replaced, the cluster loses its quorum entirely and can no longer make new master decisions, even though all data nodes remain online. A simple alert on the count of active master-eligible nodes in _cluster/health closes this gap reliably.

// Verify zone awareness attributes are set consistently across nodes
GET _nodes?filter_path=nodes.*.attributes

{
  "nodes": {
    "abc123": { "attributes": { "zone": "eu-central-1a" } },
    "def456": { "attributes": { "zone": "eu-central-1b" } },
    "ghi789": { "attributes": { "zone": "eu-central-1c" } }
  }
}

10. Summary

A resilient cluster architecture relies on clean role separation: master-eligible nodes manage cluster state, data nodes carry indexing and search, ingest nodes process pipelines, coordinating nodes aggregate large search requests. Dedicated master nodes physically decouple the control plane from the data plane and prevent heap pressure on data nodes from affecting the master election. Three master-eligible nodes are the minimum for a functioning quorum, an even number is always a mistake.

The biggest stability gain does not come from more hardware, but from correctly distributing existing resources along clear roles. Planning the cluster architecture with dedicated master nodes from the start saves a risky later migration under production load and structurally reduces the likelihood of split brain incidents, instead of only reacting to symptoms.

11. FAQ: Cluster Architecture, Master and Data Nodes