Page 10 - Elasticsearch

As a company grows across multiple regions or brands, several independent Elasticsearch clusters often end up running side by side, each operated, scaled, and upgraded on its own. Cross-cluster search lets a single query run across the boundaries of those separate clusters without first merging the data into one shared cluster. For operators of several regionally separate Magento instances, each with its own search index, that means being able to offer a central, company-wide search without giving up the existing, deliberately separated cluster topology. How remote cluster configuration works in detail, what latency and consistency trade-offs come with it, and where the practical limits sit determine whether cross-cluster search is genuinely the right fit for a given use case.

Classic page-by-page pagination with from and size works reliably for the first few result pages, but becomes expensive with deep pagination and returns inconsistent results the moment the underlying data set changes between two page fetches. For export functions that need to reliably and completely walk through thousands or millions of documents, that is not good enough. The point-in-time API solves this by capturing a consistent snapshot of the index at a fixed moment, against which search_after can then paginate efficiently and without duplicates or missing documents. How this combination works technically, how it differs from the older scroll mechanism, and how a robust export function can be built on top of it is what this article covers.

The Bool Query is the central tool of the Elasticsearch Query DSL for combining multiple conditions logically. Anyone who understands must, should, must_not, and filter, and knows when scoring context applies versus filter context, builds search queries that are both precise and fast, instead of throwing all clauses into one bucket and hoping for good results.

An ILM policy does not just describe how long an index should live, it describes exactly which action runs in which phase: rollover in the hot phase, shrink and force merge on the transition to warm, replica reduction in the cold phase, and finally deletion. Anyone who understands these building blocks and wires them together correctly ends up with data management that runs without manual intervention while still matching the real access pattern of the data.

Most introductions describe an Elasticsearch cluster as having just two roles: master nodes, which manage cluster state, and data nodes, which hold shards and answer queries. As soon as a cluster grows or extra workloads such as ingest pipelines, machine learning jobs, or continuous transforms enter the picture, that simple split stops being enough. Elasticsearch defines a whole set of additional, specialized node roles, each handling one particular task and each capable of being isolated from the rest of the cluster's load. Understanding what ingest, coordinating-only, machine learning, and transform nodes actually do, and at what cluster size dedicated separation starts to pay off, lets an operator assign resources more deliberately, contain failures better, and keep search performance stable for a Magento store even as traffic grows.

A sound Elasticsearch cluster architecture cleanly separates the control plane from the data plane: dedicated master nodes manage cluster state and shard allocation, while data nodes handle indexing and search under load. Ignoring this role separation risks split brain scenarios and cluster instability exactly when load is highest.

A single Elasticsearch cluster remains a single point of failure, no matter how much replication happens inside it, the moment an entire data center or cloud region goes down. Cross-cluster replication, CCR for short, synchronizes indices in near real time from a leader cluster to a spatially separate follower cluster, building a resilient foundation for disaster recovery that goes well beyond classic, periodic snapshots. How CCR works in detail, how the approach differs from snapshot-based backups, and how a realistic failover scenario unfolds determine how quickly a Magento store becomes searchable again after a regional outage.

A new product attribute does not automatically end up where it is needed for search and filtering in the Elasticsearch index. Only a deliberate custom attribute mapping with dedicated FieldMapper classes ensures that searchable, filterable and the right field type fit together, instead of relying on Magento's default derivation.