Elasticsearch Cluster Sizing for Magento Stores With Very Large Catalogs

1. Why default sizing recommendations often do not fit Magento

General sizing guides usually assume fairly lean documents with few fields, such as log entries or simple content records. A Magento product document, in contrast, frequently holds several dozen indexed attributes, plus price data per customer group and website as well as category assignments, making a single document noticeably larger than most generic sizing examples assume.

Adding to that, Magento creates a separate index for every store view with its own language or price context, so the effective total document count does not match the product count in the catalog, it roughly equals the product count multiplied by the number of relevant store views. A catalog with 500,000 products and six store views can therefore generate up to three million indexed documents across all indices combined.

2. Capacity planning metrics: determining document count and document size

The first planning step is a realistic estimate of the total document count across all store views, not just the raw product count in the Magento admin. That includes checking whether configurable products get indexed in addition to their simple variants, and whether disabled or invisible products remain in the index, which can easily understate the real figure if only active storefront products are counted.

Average document size is measured most reliably directly on the existing index, by dividing total index size by document count, rather than deriving it theoretically from attribute count. For catalogs with many multilingual text attributes and long descriptions, average document size in practice usually falls between two and eight kilobytes, though it can run noticeably higher for very attribute-heavy B2B catalogs.

# Measure average document size on the existing index
curl -s "https://search.example.com/catalogsearch_fulltext_en/_stats/store" \
  | jq '.indices[]._all.total.store.size_in_bytes as $size
        | .indices[]._all.total.docs.count as $docs
        | ($size / $docs)'

3. Shard sizing: target size and number of primary shards

A general rule of thumb is that a single shard should be between ten and fifty gigabytes, since shards that are too small cause unnecessary management overhead, while shards that are too large unnecessarily extend recovery time after a node failure and slow down individual search requests, because every request within a shard still has to run sequentially through its segments.

The number of primary shards for a Magento catalog index therefore follows directly from the expected index size divided by the target shard size, with some growth buffer planned in, since the shard count of an existing index cannot be changed afterward without rebuilding the index via reindex or a split operation.

PUT /_index_template/magento_catalog_shards
{
  "index_patterns": ["catalogsearch_fulltext_*"],
  "template": {
    "settings": {
      "number_of_shards": 4,
      "number_of_replicas": 1
    }
  }
}

4. Replica configuration: availability versus search throughput

Every replica increases both fault tolerance and possible read throughput, since search requests can be distributed across all available copies of a shard, primary and replicas alike, but it costs additional storage and additional indexing load, since every write operation has to run on every copy. For Magento storefront traffic with high read volume and comparatively infrequent reindex runs, a replica count of one to two is usually a good starting point.

For stores with very high search volume during peak periods, such as seasonal sales campaigns, temporarily raising the replica count ahead of the expected traffic increase pays off, since additional replicas can be adjusted as a dynamic index setting without rebuilding the index, and they increase read throughput directly proportionally, as long as enough data nodes exist to distribute the extra copies.

5. Node count and hardware sizing

The heap size of every data node should be at most fifty percent of available RAM and should generally never exceed thirty to thirty-two gigabytes, since beyond that threshold Java's pointer compression gets disabled and the effectively usable heap paradoxically shrinks despite more assigned memory. The remaining RAM stays available to the operating system for filesystem caching, which matters at least as much for segment read speed as the heap itself.

For production Magento clusters past a certain size, separating master nodes from data nodes also pays off: dedicated master nodes without data storage react more reliably to cluster state changes, since they are not simultaneously loaded by indexing or search traffic, which can be the difference between a stable and an unstable cluster state, especially during a full Magento reindex with high bulk indexing load.

6. Magento-specific considerations for sizing

A full Magento reindex briefly generates a noticeably higher write load than normal operation, since every product document gets reindexed, while the old index simultaneously keeps serving storefront search, provided Magento's blue-green reindex strategy is in use. This load spike has to be planned for explicitly in sizing, since a cluster dimensioned only for normal operation can hit its limits during a reindex.

With many store views, the total number of simultaneously active indices adds up quickly, which is why the absolute number of indices and shards in the cluster needs attention alongside raw data volume, since every shard, regardless of its size, causes a certain baseline overhead in cluster state management that can noticeably affect cluster stability once it reaches several hundred shards.

7. Scaling steps as the catalog grows

At the first stage, a single node with combined master and data roles is often enough for smaller catalogs, as long as total index size stays well below available storage capacity and a brief outage is business tolerable. Once availability becomes production relevant, the second stage follows with at least three nodes, so a stable quorum for master election survives a single node failure.

With further growth, dedicated master nodes get separated from data nodes, followed by horizontal scaling of data nodes matching the growing index size. At the final stage, relevant for very large Magento installations, a hot-warm architecture pays off, where newly written or frequently searched indices sit on performant hot hardware, while rarely used, historical data stays on cheaper warm hardware.

8. Capacity testing and benchmark methodology

Theoretical sizing calculations should always be validated through a realistic load test before a production rollout, since the real query patterns of a Magento storefront, a mix of full-text search, layered navigation aggregations, and sorting, can only partly be derived from abstract metrics. Tools like Rally let you replicate a realistic request mix that reflects the actual query distribution of a production Magento store rather than a synthetic uniform distribution of simple search terms.

A meaningful benchmark simulates not just normal traffic but also the load spike of a running reindex in parallel with active storefront traffic, since exactly that combination causes performance problems most often in practice. The result of the benchmark should be a defensible statement about which node and shard configuration keeps the response time distribution within business acceptable bounds.

9. Monitoring metrics for ongoing sizing validation

After a production rollout, continuous monitoring does not replace the one-time sizing decision, it validates it on an ongoing basis using real metrics: JVM heap usage staying close to the configured ceiling points to memory dimensioned too tightly, while a persistently high number of rejected threads in the search or indexing thread pool points to an insufficient node count for the current load.

Latency percentiles matter just as much, in particular the 95th and 99th percentile rather than plain averages, since individual slow outliers on Magento storefront requests directly affect perceived user experience even while the average still looks unremarkable. Regularly comparing these metrics against the original sizing assumptions shows early on when the next scaling stage is due.

10. Summary

11. FAQ: Sizing for Large Catalogs