Cache Warmup Strategies After Deployments in Magento

1. The problem: a cold cache after deployment

Every Magento deployment that runs bin/magento cache:flush or regenerates the configuration cache empties the Redis object cache and often the full page cache too. The first request to any category or product page after deployment then no longer hits a filled cache, but instead has to render the entire page from scratch: loading product data from MySQL, calculating prices, assembling the layout, and only at the end storing the result in Redis. These cold start requests often take ten to twenty times longer than a cache hit.

At low traffic this effect barely shows, because the cache fills up slowly over the first minutes after deployment. For stores with consistently high traffic or during a campaign, the picture is different: within seconds, hundreds or thousands of requests hit the same empty Redis cache simultaneously, and every single one triggers its own full database query. The result is a brief but massive load spike on the database, which in the worst case leads to timeouts or even a complete outage, exactly at the moment a deployment was supposed to go unnoticed.

The core of the problem is that an empty Redis cache and real user traffic collide at the same time. Cache warmup solves this by proactively filling the cache before real users load the page, so the database receives the load in a controlled, sequential way instead of all at once and in parallel.

2. Thundering herd: when many requests load at once

The technical term for this phenomenon is thundering herd: many simultaneous requests all discover the same missing cache entry and start the same expensive computation in parallel, even though only one computation was actually needed. Without a protection mechanism, a popular product page after deployment causes ten or more parallel PHP processes to load the same product data from the database simultaneously, run the same price calculation, and ultimately all write the same value into Redis.

Magento's full page cache layer with Redis as the backend does not provide complete protection against this effect out of the box when many requests arrive within the same millisecond window. An effective countermeasure is an application level lock mechanism: the first request that detects a cache miss sets a short lived lock key in Redis with SET key value NX EX 5, computes the value, and releases the lock afterward. Further requests that find the same lock either wait briefly or serve a slightly stale version instead of re-running the expensive computation.

Warmup strategies solve the thundering herd problem indirectly by filling the cache before real parallel traffic even occurs. A systematic crawler that fetches pages sequentially instead of in parallel generates exactly one database query per page and avoids the duplication that would occur with real users combined with an empty cache.

3. Warmup crawlers: filling the full page cache deliberately

A warmup crawler is a script that automatically visits a list of important URLs after deployment, filling the Redis full page cache without requiring real users to trigger the first, slow request. There are several approaches for Magento: the official bin/magento cache:enable combined with an external HTTP crawler, specialized warmer modules, or a simple bash script using curl that iterates sequentially or with bounded parallelism through a sitemap or URL list.

It is important that the crawler sends real requests with the same headers a browser would, in particular the Accept-Encoding header and a realistic User-Agent, because Magento's Varnish or full page cache layer can create different cache entries for different contexts. A crawler that ignores these headers might end up filling a cache entry that real browsers never actually use, because their request looks slightly different.

#!/usr/bin/env bash
# warmup.sh: sequential Redis cache warmup crawler for Magento
set -euo pipefail

BASE_URL="https://shop.example.com"
URL_LIST="/var/www/deploy/warmup-urls.txt"
CONCURRENCY=4
LOG_FILE="/var/log/magento/warmup-$(date +%Y%m%d-%H%M%S).log"

echo "[INFO] Starting cache warmup with concurrency=${CONCURRENCY}" | tee "$LOG_FILE"

# xargs runs requests with bounded parallelism, avoiding a new thundering herd
cat "$URL_LIST" | xargs -P "$CONCURRENCY" -I {} \
  curl -s -o /dev/null -w "%{http_code} %{time_total}s {}\n" \
    -H "Accept-Encoding: gzip, deflate" \
    -H "User-Agent: Mozilla/5.0 (compatible; MironsoftWarmupBot/1.0)" \
    "${BASE_URL}{}" >> "$LOG_FILE"

echo "[INFO] Warmup complete. Requests logged to ${LOG_FILE}"

4. Choosing the right URLs for warmup

A warmup crawler that tries to walk the entire catalog after every deployment wastes time and resources. A prioritized URL list based on actual traffic data works better: the homepage, the most important category pages, the most visited product pages, and static pages such as terms of service or checkout entry pages. This list can be derived automatically from Google Analytics numbers or from the web server access logs of the last 30 days.

For very large catalogs with hundreds of thousands of products, warming every single page is neither practical nor necessary. The Pareto principle reliably applies here: usually 20 percent of pages generate 80 percent of traffic, and a warmup covering exactly these top URLs already reduces database load after deployment by the majority, without unnecessarily prolonging the crawler run. The remaining, less frequently visited pages fill up organically through real user traffic, but with a lower probability of simultaneous parallel access and therefore a lower thundering herd risk.

# Generate a prioritized warmup URL list from the last 30 days of Nginx access logs
awk '{print $7}' /var/log/nginx/shop.example.com-access.log \
  | grep -E '^/(catalog|category|checkout|customer)?' \
  | sort | uniq -c | sort -rn \
  | head -n 500 \
  | awk '{print $2}' > /var/www/deploy/warmup-urls.txt

wc -l /var/www/deploy/warmup-urls.txt

5. Object cache preload for categories and configuration

Besides the full page cache, a deliberate preload of the Magento object cache in Redis pays off, especially for data needed on almost every page: the category tree structure, store configuration, active layered navigation attributes and layout XML merges. This data rarely changes but is read on every request, which is why a missing entry here is particularly expensive, since it affects practically every page type simultaneously.

A simple PHP script running through the Magento bootstrap class can deliberately pre-fill these critical object cache entries by calling the corresponding repository methods once, before the crawler even starts. This ensures that even the very first full page cache warmup request already hits a partially filled object cache instead of also having to reload the base data.

<?php
// bin/warmup-object-cache.php: preload frequently used object cache entries
require __DIR__ . '/../app/bootstrap.php';

$bootstrap = \Magento\Framework\App\Bootstrap::create(BP, $_SERVER);
$objectManager = $bootstrap->getObjectManager();
$state = $objectManager->get(\Magento\Framework\App\State::class);
$state->setAreaCode('frontend');

$storeManager = $objectManager->get(\Magento\Store\Model\StoreManagerInterface::class);
$categoryRepository = $objectManager->get(\Magento\Catalog\Api\CategoryRepositoryInterface::class);

foreach ($storeManager->getStores() as $store) {
    echo "[INFO] Warming category tree for store: {$store->getCode()}\n";
    // Touching the root category forces Magento to rebuild and cache the tree
    $categoryRepository->get((int) $store->getRootCategoryId(), $store->getId());
}

echo "[INFO] Object cache preload complete\n";

6. Integrating warmup into the deployment pipeline

Warmup should never be a manual step, but a fixed part of the deployment pipeline, immediately after the cache flush and before releasing traffic to the new version. In a typical sequence, warmup sits directly after bin/magento cache:flush and before removing an active maintenance page or switching the load balancer to the new deployment slot.

A common mistake is timing the warmup step too tightly or letting it run in the background while traffic is already being released. This defeats the entire purpose of the measure, because real users then hit the same empty cache in parallel with the crawler. It is cleaner to only disable maintenance mode after the warmup run has completed successfully, so that a significant portion of the cache is already filled by the time traffic is released.

#!/usr/bin/env bash
# deploy-with-warmup.sh: deployment sequence with cache warmup before traffic release
set -euo pipefail

echo "[STEP 1] Enabling maintenance mode"
bin/magento maintenance:enable

echo "[STEP 2] Running setup upgrade and flushing cache"
bin/magento setup:upgrade
bin/magento cache:flush

echo "[STEP 3] Preloading object cache"
php bin/warmup-object-cache.php

echo "[STEP 4] Running full-page-cache warmup crawler"
./warmup.sh

echo "[STEP 5] Verifying warmup coverage before releasing traffic"
HIT_RATIO=$(redis-cli -n 1 info stats | grep keyspace_hits | cut -d: -f2)
echo "[INFO] Current keyspace_hits: ${HIT_RATIO}"

echo "[STEP 6] Disabling maintenance mode, traffic is released"
bin/magento maintenance:disable

7. Blue-green deployments and a pre-warmed cache

With blue-green deployments, the cold start problem can be solved even more elegantly: the new version runs in parallel to the old one on separate infrastructure, with its own, initially empty Redis cache. The warmup crawler runs against this new environment while the old version continues serving all production traffic. Only once the new environment's cache is sufficiently filled does the load balancer switch traffic over.

This approach nearly eliminates the thundering herd problem, because real user traffic never hits an empty cache at any point. The cost is additional infrastructure that must run in parallel, and a somewhat longer deployment window, since warmup must finish before the switchover. For stores with high revenue per minute, this extra effort is usually justified, because a single failed deployment traffic spike can be significantly more expensive.

8. Measuring success: monitoring hit rate and load times

Whether a warmup run actually works can be read directly from the Redis hit rate. The INFO stats command returns keyspace_hits and keyspace_misses, from which the hit ratio can be calculated. Immediately after a successful warmup, the miss rate for the pre-warmed URLs should be close to zero, whereas without warmup it can exceed 50 percent in the first minutes after deployment.

Beyond the raw hit rate, it is worth monitoring response times in the first five minutes after every deployment, for example through application performance monitoring tools. An effective warmup shows itself in the p95 response time staying close to normal right after release, instead of briefly spiking. This metric works well as an automated check in the pipeline that marks a deployment as failed if response times after warmup do not stay within a defined threshold.

# Compute Redis hit ratio before and after a warmup run
before_hits=$(redis-cli -p 6381 info stats | grep -oP 'keyspace_hits:\K\d+')
before_miss=$(redis-cli -p 6381 info stats | grep -oP 'keyspace_misses:\K\d+')

./warmup.sh

after_hits=$(redis-cli -p 6381 info stats | grep -oP 'keyspace_hits:\K\d+')
after_miss=$(redis-cli -p 6381 info stats | grep -oP 'keyspace_misses:\K\d+')

echo "Hits before: ${before_hits}, misses before: ${before_miss}"
echo "Hits after:  ${after_hits}, misses after:  ${after_miss}"

# Fail the pipeline if miss rate stays too high after warmup
delta_miss=$(( after_miss - before_miss ))
if (( delta_miss > 500 )); then
  echo "[ERROR] Warmup coverage insufficient: ${delta_miss} new misses" >&2
  exit 1
fi

9. Warmup strategies compared

Depending on store size, traffic pattern and infrastructure, different warmup approaches make sense. The following table compares the most common strategies for Magento stores using Redis as the cache backend.

Most Magento stores already get very far with a prioritized top URL crawler combined with an object cache preload script. Blue-green deployments with full pre-warmup pay off especially when every minute of elevated load time translates directly into measurable revenue loss.

10. Summary

A cold Redis cache after deployment is not a minor edge case, but one of the most common causes of database load spikes and performance dips right after a release. Warmup crawlers that fetch prioritized URL lists sequentially or with bounded parallelism proactively fill the full page cache before real user traffic hits the new version. In addition, a targeted object cache preload script ensures that base data such as the category tree and store configuration is already available before the first request.

The decisive rule is: warmup belongs before traffic release, not after. Whether through a classic maintenance mode window or a blue-green deployment with a separate warmup environment, in both cases no real user should ever hit an empty Redis cache. Monitoring hit rate and response times in the first minutes after every deployment makes it visible whether the warmup strategy is actually working.

11. FAQ: Cache Warmup Strategies for Magento