Indexer Performance for Very Large Catalogs

1. Where indexer performance becomes a problem on large catalogs

As long as a Magento store carries a few thousand products, poor indexer performance barely stands out. A full reindex finishes in a few minutes, the cron job disturbs nobody, and the default batch size value from the factory configuration is enough. On catalogs with several hundred thousand SKUs, common in B2B marketplaces or spare parts catalogs, this picture flips completely: a full reindex run of the catalogsearch_fulltext indexer can take several hours, tying up database resources and competing with other cron jobs for CPU and network bandwidth.

Indexer performance does not depend on a single lever, but on the interplay between PHP-side batch processing, database queries for product data, and the efficiency with which documents are transferred to Elasticsearch. Anyone who optimizes only one spot, for example raising PHP memory limits without adjusting batch size, merely shifts the bottleneck instead of removing it. This article systematically walks through every relevant lever, from configuration to ongoing monitoring.

A practical starting point for any optimization is measurement: how long does a full reindex currently take, how many products are processed per second, and where in the process does the biggest wait occur. Without this baseline, the effect of any change to indexer performance cannot be judged objectively.

2. Batch size tuning for catalogsearch_fulltext

The central lever for indexer performance is batch size, configured in app/etc/env.php under the key indexer.batch_size. Magento does not process products for the fulltext index one at a time, but in batches that are loaded from the database, enriched, and then sent to Elasticsearch in bulk. A batch size that is too small generates many small database queries and many small bulk requests, driving up overhead from network latency and connection setup. A batch size that is too large, on the other hand, pushes PHP processes toward their memory limit and can overwhelm Elasticsearch with oversized bulk payloads.

The default value for catalogsearch_fulltext is 100 documents per batch, which is adequate for small to medium catalogs but often too conservative for very large ones. In practice, values between 300 and 1000 frequently show better throughput, depending on average document size. Products with many attributes, long descriptions, and numerous images produce far larger Elasticsearch documents than simple SKUs, which is why the optimal batch size depends heavily on the product data model and should not be copied blindly from another project.

// app/etc/env.php - indexer batch size configuration
'indexer' => [
    'batch_size' => [
        'catalogsearch_fulltext' => [
            'partial' => 500,
            'full' => 500,
        ],
        'catalog_product_price' => [
            'partial' => 500,
            'full' => 500,
        ],
    ],
],

# After changing env.php, run a full reindex once to validate memory usage
bin/magento indexer:reindex catalogsearch_fulltext

3. Update on Save vs. Update by Schedule

Besides batch size, the indexer mode massively affects perceived indexer performance. In "Update on Save" mode, reindexing runs synchronously on every product change, which is fine for individual edits but catastrophic for bulk imports: a CSV import of ten thousand products would trigger ten thousand individual, synchronous reindex operations. "Update by Schedule" mode (mview) decouples changes from reindexing: product changes mark affected records in a changelog table, and a cron job processes them in controlled batches in the background.

For catalogs with frequent bulk updates, "Update by Schedule" is almost always the right choice. The relevant cron group is called indexer, and the frequency of the associated indexer_update_all_views job can be adjusted through the crontab configuration. It is important not to let the cron job run too infrequently, because otherwise the changelog piles up and a single run suddenly has to process tens of thousands of records, which drastically degrades indexer performance for that one run.

# Switch catalogsearch_fulltext to scheduled (mview) mode
bin/magento indexer:set-mode schedule catalogsearch_fulltext

# Verify current mode for all indexers
bin/magento indexer:show-mode

# Example output
# Title                  Mode
# Catalog Search          Schedule
# Product Price           Schedule
# Product EAV              Update on Save

4. Partial reindex strategies in detail

A full reindex reprocesses every product in the catalog, regardless of whether it changed. On a catalog with 500,000 SKUs, of which only a few thousand are updated daily, that is enormously inefficient. Partial reindex in scheduled mode solves exactly this problem: through the mview system (materialized view), only products whose underlying data changed since the last run are re-indexed. The central table for this is catalogsearch_fulltext_cl (change log), where every relevant change lands as a row with the affected entity ID.

For strong indexer performance during partial reindexes, what matters most is which changes actually generate a changelog entry. Price changes, stock changes, and attribute changes flagged as "used in search" trigger an entry. Attributes that are not configured as searchable or filterable, on the other hand, generate no reindex trigger at all, which makes unused attribute flags a simple but often overlooked lever for better indexer performance: consistently marking attributes as searchable only where actually needed noticeably reduces the number of changelog entries.

5. What happens under the hood: the Elasticsearch Bulk API

Whether full or partial, Magento never transfers documents to Elasticsearch one at a time, always through the _bulk API. Each batch from the PHP indexer is assembled into a single HTTP request with multiple lines in NDJSON format, where every action (typically index) is directly followed by the document data. This bundling is the main reason batch size has such a large effect on indexer performance in the first place: it directly determines the size of each individual bulk request.

Elasticsearch processes incoming bulk requests through a dedicated thread pool with a bounded queue size. If too many bulk requests are sent concurrently, for example because multiple cron consumers run in parallel, the queue can fill up and Elasticsearch starts rejecting requests with TOO_MANY_REQUESTS (HTTP 429). Magento does not automatically handle this with retry logic in every version, so overly aggressive parallelization can lead to failed, incomplete index runs that are often hard to distinguish from genuine data errors in the log.

POST /_bulk
{ "index": { "_index": "magento2_default_catalogsearch_fulltext_1", "_id": "10231" } }
{ "sku": "SHOE-BLK-42", "name": "Running Shoe Black", "price": 89.90, "visibility": 4 }
{ "index": { "_index": "magento2_default_catalogsearch_fulltext_1", "_id": "10232" } }
{ "sku": "SHOE-BLK-43", "name": "Running Shoe Black", "price": 89.90, "visibility": 4 }

// Response contains per-item status; a single failed item does not
// abort the whole batch, so failures must be checked individually
{
  "took": 42,
  "errors": false,
  "items": [
    { "index": { "_id": "10231", "status": 201 } },
    { "index": { "_id": "10232", "status": 201 } }
  ]
}

6. Cluster-side tuning during bulk runs

On the Elasticsearch side, there are two levers that meaningfully improve indexer performance during large reindex runs. The first is temporarily raising refresh_interval. By default, Elasticsearch refreshes the search index every second, which under a high rate of incoming bulk requests creates significant internal overhead because every refresh creates a new Lucene segment. During a full reindex run, the interval can be set to 30 seconds or even -1 (disabled), then reset to the default afterward.

The second lever is temporarily reducing the replica count to 0 during an initial bulk load, for example when fully repopulating an index after a migration. Without replicas, Elasticsearch does not need to additionally replicate every document to other nodes, which significantly increases write throughput. After the bulk load finishes, the replica count is set back to the production value, and Elasticsearch fills the replicas in the background without blocking ongoing operations.

# Temporarily relax refresh interval and disable replicas during a bulk load
curl -X PUT "localhost:9200/magento2_default_catalogsearch_fulltext_1/_settings" \
  -H "Content-Type: application/json" -d '{
    "index": {
      "refresh_interval": "30s",
      "number_of_replicas": 0
    }
  }'

# Restore production settings after the bulk load finishes
curl -X PUT "localhost:9200/magento2_default_catalogsearch_fulltext_1/_settings" \
  -H "Content-Type: application/json" -d '{
    "index": {
      "refresh_interval": "1s",
      "number_of_replicas": 1
    }
  }'

7. Monitoring: how to spot a slow indexer

Without monitoring, any statement about indexer performance remains speculative. The simplest entry point is the cron_schedule table, which lets you track the actual runtime of the indexer_update_all_views job over time. A steadily growing runtime trend is an early warning sign that the catalog is growing faster than the current configuration can handle. In addition, bin/magento indexer:status provides the current status, but no historical runtime data, which is why an external monitoring system is needed for reliable trend statements.

On the Elasticsearch side, it is worth checking the thread pool statistics for write and bulk via the _nodes/stats/thread_pool API. A high number of rejected entries shows directly that bulk requests were rejected, often because too many arrived at once. This metric is more reliable than raw CPU usage, because it measures the actual symptom of poor indexer performance rather than a resource utilization figure that can be high for other reasons too.

# Track catalogsearch_fulltext cron runtime over time
bin/mysql magento -e "
  SELECT job_code, scheduled_at, executed_at, finished_at,
         TIMESTAMPDIFF(SECOND, executed_at, finished_at) AS duration_seconds
  FROM cron_schedule
  WHERE job_code = 'indexer_update_all_views'
    AND status = 'success'
  ORDER BY scheduled_at DESC
  LIMIT 20;
"

# Check Elasticsearch bulk thread pool for rejections
curl -s "localhost:9200/_nodes/stats/thread_pool/write,bulk?pretty" \
  | grep -A 3 '"rejected"'

8. Common pitfalls during optimization

The most common mistake is drastically increasing batch size without adjusting PHP memory limits accordingly. A PHP process holding product data for 1000 documents in memory at once needs significantly more RAM than one holding 100 documents, especially for products with many attributes. The result is a fatal error from exhausted memory limits in the middle of a reindex run, which invalidates the entire batch and forces the process to start over.

A second pitfall is running multiple indexer:reindex calls in parallel without coordination, assuming that speeds up the process proportionally. In reality, parallel processes compete for the same database connections and the same Elasticsearch bulk thread pool, and beyond a certain level of parallelism, overall throughput drops instead of rising. A third, less frequently noticed pitfall concerns the MySQL side: missing or outdated indexes on the flat tables used by the indexer can slow down data retrieval for every batch, long before Elasticsearch is even involved.

9. Configuration comparison for different catalog types

The concrete configuration for optimal indexer performance differs significantly by catalog type. A B2B spare parts catalog with very many but data-sparse SKUs behaves differently from a fashion catalog with a few thousand products but extensive descriptions, images, and variant data. The table in section 4 provides rough guideline values, but the actually optimal batch size should always be determined empirically, by testing several values under realistic load and measuring the resulting runtime.

A proven approach is to start with the default value, increase batch size in increments of 100, and log runtime as well as PHP worker memory usage after every step. The point at which runtime improvement clearly flattens or memory errors appear marks the practical ceiling for indexer performance optimization through batch size alone. Beyond that point, only parallelization or additional hardware bring further improvements.

10. Summary

Strong indexer performance on large Magento catalogs does not come from a single setting, but from the interplay of the right batch size, the right indexer mode, efficient partial reindex use through the mview system, and an Elasticsearch configuration that is temporarily adjusted during bulk runs. "Update by Schedule" mode is almost mandatory for catalogs of medium size and above, because it decouples bulk changes instead of processing them synchronously and individually.

The right batch size cannot be recommended as a blanket rule; it must be determined empirically for the specific catalog, depending on document size and available PHP memory. Monitoring cron runtimes and Elasticsearch thread pool statistics makes degradations in indexer performance visible before they become an acute operational problem. Consistently applying these building blocks keeps reindex runtimes within a predictable range even as the catalog grows.

11. FAQ: Indexer Performance for Large Catalogs