Magento Full Page Cache Warmup Strategies Against the Thundering Herd Problem

1. Why an empty cache after deployment becomes a problem

A full page cache flush is a routine event in Magento, it happens on every deployment, after large-scale price changes, after indexer runs with a full reindex, or simply through a manual bin/magento cache:flush. Right after that, the full page cache is completely empty, and every incoming request has to run the entire rendering pipeline: load the layout, build the blocks, pull product data from the database, calculate prices, and finally output HTML.

At low traffic this barely registers, because requests arrive spread out over time and the cache fills itself page by page. For a store with meaningful concurrent traffic, though, the classic thundering herd problem shows up: dozens or hundreds of visitors request the same, still uncached homepage or category page within seconds, and the server has to process this expensive request multiple times in parallel and redundantly, instead of computing it once and serving it from cache afterward. The result is noticeably slower response times at exactly the moment the store comes back online, and in the worst case an overloaded PHP-FPM or database layer.

2. A brief technical background: Magento full page cache

Magento supports two operating modes for the full page cache: the built-in, file-based or Redis-backed cache for smaller setups, and Varnish as an upstream HTTP reverse proxy for production environments with higher traffic. In both cases the same basic principle applies: a fully rendered HTML response gets stored under a cache key composed partly of the URL, the store view, and relevant cookies such as customer group, and on a repeat request with the same key, the stored response gets served directly without PHP being invoked at all.

That very property, a cache hit bypassing PHP entirely, is what makes the full page cache so effective for performance, but also so painfully empty right after a flush. A Varnish cache with several gigabytes of RAM can theoretically hold tens of thousands of pages, but only does so once each individual page has been requested at least once and rendered from PHP in the process. Without active warmup, the fill level of the cache depends entirely on organic visitor behavior, which leads to a very uneven and potentially overloaded starting position in the first minutes after a flush.

3. The difference between reactive and proactive cache warmup

Reactive warmup means the cache fills itself purely through incoming real visitor traffic, without any deliberate intervention. That is the default in essentially every Magento setup and works well enough at low to medium traffic, because the cache fills itself within a few minutes. The downside is that exactly the visitors who arrive right after the flush feel the full, uncached rendering time and, in the worst case, even contribute to overload by collectively triggering the thundering herd problem.

Proactive warmup reverses that order: an automated script deliberately requests the most important pages before real visitors arrive, filling the cache in a controlled way, at a defined pace and in a defined order. The crucial difference is that proactive warmup is entirely under the operator's control, while reactive warmup depends on actual, unpredictable visitor behavior. For stores with plannable deployment windows or after large price and catalog changes, proactive warmup is therefore the clearly more reliable strategy.

4. Prioritization: homepage, top categories, and bestsellers first

Not every page in the catalog deserves the same warmup priority, because a full catalog with tens of thousands of product pages simply takes too long to fully pre-heat before the first real visitor arrives. A clear prioritization by actual traffic share makes sense: the homepage comes first, because almost every visitor touches it at least indirectly regardless of entry point, and because it is often the most complex, most heavily aggregated page in the entire store.

Next come the top category pages, typically the 10 to 30 most visited categories according to analytics data, because these pages capture the bulk of both organic and paid traffic. Third, bestseller product pages get pre-heated, identified either through Magento's own bestseller report or through the most visited product URLs from the analytics tool. All remaining product pages are deliberately left to reactive warmup, because the effort of fully pre-heating them rarely pays off.

5. Systematic warmup via script

A warmup script needs a list of prioritized URLs, typically drawn from the Magento sitemap combined with a manually maintained list of the most important categories and a product list generated dynamically from the bestseller report. The example below shows a simple but practical Bash script that reads a URL list and deliberately requests each page via curl to place it in the full page cache before traffic sets in.

It matters that the script sends the same user agent and the same relevant cookies as a regular visitor, since Magento builds the cache key partly from customer group and store view. A warmup request without the matching cookies might otherwise only fill the cache for one customer group, while the actually most common customer group stays cold.

#!/usr/bin/env bash
set -euo pipefail

URL_LIST="var/warmup/priority-urls.txt"
CONCURRENCY=8
BASE_URL="https://www.example-store.com"

echo "Starting cache warmup with ${CONCURRENCY} parallel requests..."

cat "$URL_LIST" | xargs -P "$CONCURRENCY" -I{} curl \
    --silent \
    --output /dev/null \
    --write-out "%{http_code} %{time_total}s {}\n" \
    --header "User-Agent: MironsoftCacheWarmup/1.0" \
    --max-time 20 \
    "${BASE_URL}{}"

echo "Cache warmup complete."

6. Parallelism and rate limiting during warmup

A warmup script that requests every page at maximum parallelism can load the server more heavily than the very thundering herd problem it is meant to prevent, because every still-uncached page triggers a full PHP rendering request. A moderate parallelism of around 5 to 10 concurrent requests has proven to be a good compromise in practice, filling the cache within a few minutes without fully saturating PHP-FPM workers or database connections.

It is also worth structuring the warmup script into waves, where the absolutely critical pages like the homepage and top categories are processed first at high priority, followed by the second priority tier, such as bestseller products, at somewhat lower parallelism. This way the store becomes ready for the bulk of expected traffic within a short time, while less critical pages catch up in the background without slowing down the critical phase.

7. Integration into the deployment pipeline

For cache warmup not to depend on manual intervention, it should be anchored as a fixed, automated step right after cache:flush in the deployment pipeline. A typical flow has the warmup script start automatically after deploying the new code, running database migrations, and the final cache flush, before the load balancer marks the server as active again and lets real traffic in.

With multiple application servers behind a load balancer, this concretely means a server only gets removed from maintenance mode or health check exclusion once the warmup script has run successfully for that server. This pattern, often called a rolling deployment with a warmup gate, ensures that no fully cold server ever goes live and receives real traffic without any pre-heating at all.

8. Monitoring warmup success

A warmup script that simply runs to completion without checking the actual results offers no guarantee that the cache is really filled afterward. It is therefore worth checking the X-Magento-Cache-Debug header, or the corresponding Varnish header, on a second request against the same URLs after warmup, to confirm the response actually comes from cache rather than being rendered again from PHP.

It is also worth watching the cache hit rate right after deployment in the monitoring system, for example via varnishstat or a corresponding Grafana dashboard, to see whether the hit rate climbs to a normal level within the expected time. If the hit rate stays unusually low, that points either to a faulty URL list in the warmup script or to a problem with the cache key itself, for example a newly introduced cookie or session variable that is unintentionally cache-relevant.

9. Limits of warmup and a comparison of strategies

Cache warmup does not solve every performance problem: personalized content, logged-in customer areas, and the checkout process are excluded from the full page cache anyway and do not benefit from pre-heating. A very large, long-tail-heavy catalog with tens of thousands of rarely visited product pages also cannot realistically be fully pre-heated, here reactive warmup through real traffic remains the only practical solution. The table below compares the strategies discussed side by side.

10. Summary

11. FAQ: FPC Warmup Strategies: The Essentials at a Glance