Node Roles Beyond Master and Data: Ingest, ML, Coordinating, and Transform Explained

1. Why the simple master-plus-data split runs into limits

In a small cluster of three to five nodes, practically every node takes on every role at once: master-eligible, data, and implicitly ingest and coordinating as well. That works fine as long as the overall load stays modest and a single node has enough capacity for all of it. Once extra workloads such as heavy ingest pipelines with many processors, or compute-intensive machine learning jobs, enter the picture, they compete directly with actual search traffic for CPU, heap, and network bandwidth on the very same nodes.

The result is often a hard-to-diagnose performance degradation: search requests get slower even though nothing changed about the search index itself, because an ingest pipeline run or a machine learning job is loading the same node in the background. Deliberate role separation solves this by spreading different workloads across different, dedicated nodes, so an overloaded ingest node never affects the response time of a product search.

2. Ingest nodes: processing pipelines directly inside the cluster

Ingest nodes run ingest pipelines before a document is actually indexed: they can rename fields, extract values from unstructured text via grok patterns, enrich geo data, or drop entire documents based on conditions. This processing happens entirely inside the cluster, which often removes the need for an external Logstash or ETL layer that would otherwise sit in front of indexing.

For simple pipelines with only a few processors, it's usually enough for every data node to implicitly carry the ingest role as well. Once pipelines involve many processors, expensive scripted processors, or a high document rate, it pays off to offload them onto dedicated ingest nodes, so that indexing and search on the data nodes stay free of that extra CPU load.

# elasticsearch.yml of a dedicated ingest node
node.name: es-ingest-01
node.roles: [ ingest ]

# Example pipeline that normalizes product attributes before indexing
PUT _ingest/pipeline/normalize-product-attributes
{
  "processors": [
    { "lowercase": { "field": "color" } },
    { "trim": { "field": "manufacturer" } },
    {
      "remove": {
        "field": "temp_import_flag",
        "ignore_missing": true
      }
    }
  ]
}

3. Coordinating-only nodes: bundling requests without holding data

A coordinating-only node has none of the other roles enabled: no master, no data, no ingest. Its sole job is to accept incoming search requests, forward them to the responsible data nodes, merge the partial results from individual shards, and hand the finished result back to the client. Because it holds no shards of its own, it doesn't need a large heap for data storage, mainly enough memory for merging and sorting many partial results.

In practice, data nodes in small clusters take on this coordination task as an extra duty. For very wide fan-out queries, for instance searches spanning a large number of shards or indices at once, a dedicated coordinating-only node can serve as a stable, predictable entry point that hides the actual cluster topology behind it, similar to a load balancer sitting directly inside the Elasticsearch cluster.

4. Machine learning nodes: resource isolation for ML jobs

Machine learning jobs, for instance anomaly detection on order volumes or automatic classification of search queries, are memory and CPU intensive and run independently of the timing of normal search traffic. Without dedicated machine learning nodes, these jobs compete with regular search load for the same heap, which can produce noticeable delays for perfectly ordinary product searches whenever job activity suddenly spikes.

The machine learning role ensures that such jobs are scheduled exclusively on nodes carrying that role, leaving data and coordinating nodes completely unaffected. For a Magento operation running classic product search without its own machine learning evaluations, this role usually isn't relevant, but it gains importance as soon as search behavior or traffic anomalies need to be evaluated automatically.

5. Transform nodes: continuous aggregations as their own role

Transforms continuously build a second, already aggregated destination index out of an existing source index, for instance a daily summary of orders per category or a continuously updated metric on the most searched terms. Computing these aggregations runs periodically in the background and can be genuinely compute-intensive against large source indices.

The transform role bundles that background load onto its own nodes, so an expensive transform run never affects the latency of a regular search request. Especially for reporting or analytics use cases around a Magento store, for instance an internal evaluation of search behavior, this separation pays off once transform jobs run regularly and with noticeable compute load.

6. At what cluster size dedicated role separation pays off

For clusters up to roughly five nodes with moderate load, the operational overhead of strict role separation usually outweighs the actual benefit. All nodes share master, data, and implicit ingest duties, and the extra operational complexity of separate node types has no meaningful payoff relative to the isolation gained.

Around six to eight nodes, or as soon as noticeable resource conflicts appear between different workloads, for instance visible latency spikes during an ingest run, dedicated separation starts to make sense. As a rough rule of thumb: introduce dedicated master nodes first, once cluster stability becomes critical, then coordinating-only or ingest nodes as needed, and machine learning or transform nodes only once those features are actually active and used regularly.

7. Practical topology examples for stores of different sizes

A small Magento store with a modest catalog often gets by with three nodes that each combine every role, complemented by a simple ingest pipeline for attribute normalization running directly on the data nodes. This topology is simple to operate and offers plenty of headroom for the expected traffic.

A medium to large store with high search traffic and additional evaluations benefits instead from a clearly separated topology: three dedicated master nodes for stability, several pure data nodes for indexing and storage, two coordinating-only nodes as a stable search entry point, and a separate node carrying the ingest and transform roles for attribute enrichment and periodic reports, all without that background load ever affecting the actual product search.

# Example topology for a medium to large store

# node.roles: [ master ]         -> three dedicated master nodes
# node.roles: [ data, data_hot ]  -> several pure data nodes
# node.roles: []                  -> two coordinating-only nodes
# node.roles: [ ingest, transform ] -> one node for enrichment and reports

node.name: es-data-03
node.roles: [ data, data_hot ]
node.attr.zone: eu-central-1a

8. Role configuration in elasticsearch.yml and node attributes

Role assignment happens via the node.roles setting in each individual node's elasticsearch.yml, as a list of enabled roles such as master, data, ingest, ml, or transform. A coordinating-only node deliberately gets an empty list, which leaves none of the specialized roles active.

On top of that, custom node attributes, for instance a zone or hardware label, allow targeted shard allocation and exclusion rules, so role separation can be combined with physical or virtual distribution across racks, availability zones, or hardware classes. This combination matters most when ingest or machine learning nodes are meant to run on instances with a different CPU-to-memory ratio than the actual data nodes.

9. Monitoring and pitfalls with mixed roles

A common pitfall is accidentally mixing roles on nodes with a limited heap, for instance when a small data node also picks up the machine learning role and regularly hits its memory ceiling as a result. The node stats API shows, per node, how heavily heap, CPU, and thread pools are loaded by each role, providing the basis for well-founded separation decisions.

Equally important is watching the write, search, and ingest-specific thread pools separately from one another: if the queue of a particular pool keeps growing while other pools stay calm, that's a clear signal that this specific workload should be isolated from the other roles, rather than simply increasing overall cluster capacity across the board.

10. Summary

11. FAQ: Node Roles in Elasticsearch: The Essentials at a Glance