Automating Cache Warmup for Priority Pages After a Release

1. Why cache warmup is necessary after deployment

A Magento full page cache (FPC) is completely cleared by cache:clean full_page or cache:flush. After deployment, the cache is empty, and every page has to be fully rendered from PHP and the database on first request. On a well-optimized Magento shop, a cached page load can take 50 to 200 ms, while an uncached one takes 2 to 10 seconds. Visitors feel that difference directly, and it has measurable effects on conversion rate and bounce rate.

The problem gets worse with deployments during peak traffic hours or after large cache-clearing actions: when many users hit empty cache entries at the same time, a cache stampede occurs, where all PHP processes render the same pages in parallel, causing heavy load and longer response times for everyone. A warmup job that calls the most important pages before the first real visitor arrives prevents this scenario entirely.

2. Which pages should be warmed up first

Not all pages carry equal weight. Prioritization should follow traffic volume, conversion relevance, and cache efficiency. First in line are the homepage and the most important category pages, since they get the highest traffic and have the biggest effect on perceived shop performance. Second come the top product pages ranked by sales volume or page views from the analytics system. Third are standard pages such as the search results page, contact page, imprint, and privacy policy.

What is not a priority: checkout pages (personalized content, almost never cached), customer account pages (also not cacheable), and pages with very little traffic. The warmup focuses on publicly accessible, cacheable pages, exactly the ones Magento includes in the FPC by default. A solid first strategy: use the top 20 pages from Google Analytics or Magento's own visitor log as the warmup URL list.

3. Building the URL list: static vs. dynamic

There are two approaches to building a warmup URL list: a statically maintained file and a list dynamically generated from Magento. The static file is simple, a text file in the repository with one URL per line, maintained by hand. This is fine for small shops with a stable page structure, but has the downside that new products and categories are not added automatically.

The dynamic variant reads categories and top products directly from Magento: bin/magento sitemap:generate produces an XML sitemap that contains all public pages. This sitemap can be parsed to extract the most important URLs. A pragmatic combination: use the static list for critical pages (homepage, top categories) supplemented by sitemap-based top products sorted alphabetically or by ID. That keeps the warmup deterministic and reproducible without requiring fully manual upkeep.

warmup:cache:
  stage: warmup
  script:
    # Download priority URL list from repository
    - |
      cat > /tmp/priority-urls.txt << 'URLS'
      https://${SHOP_HOST}/
      https://${SHOP_HOST}/sale.html
      https://${SHOP_HOST}/new-arrivals.html
      https://${SHOP_HOST}/men.html
      https://${SHOP_HOST}/women.html
      https://${SHOP_HOST}/accessories.html
      URLS

    # Add top product URLs from sitemap (first 50 product URLs)
    - |
      curl -sf "https://${SHOP_HOST}/sitemap.xml" \
        | grep -oP '(?<=<loc>)[^<]+(?=</loc>)' \
        | grep '/p/' \
        | head -50 >> /tmp/priority-urls.txt

    # Execute warmup with rate limiting (max 4 parallel requests)
    - |
      xargs -P 4 -I {} \
        curl -sf -o /dev/null -w "%{http_code} %{time_total}s %{url_effective}\n" \
        -H "User-Agent: GitLab-Warmup/1.0" \
        {} < /tmp/priority-urls.txt

  when: on_success
  dependencies:
    - deploy:production
  allow_failure: true

4. curl-based warmup: simple and reliable

A simple, effective warmup mechanism is curl with the right options. The basic principle: each URL in the list is called with an HTTP GET request. Magento renders the page, stores it in the FPC, and serves it straight from the cache on the next request. The warmup request itself never gets a cached response, it is the trigger that generates the page and puts it in the cache.

Useful curl options for warmup: -sf for silent mode with an error exit code, -o /dev/null to discard the response body, -w for a structured output format with HTTP status code and load time, and --max-time 30 for a request timeout so slow pages don't block the warmup. A User-Agent header identifies warmup requests in the logs and makes it possible to monitor warmup activity.

5. Parallelization for a faster warmup

Sequential warmup calls are fine for small URL lists, but with 50 to 200 URLs it can take several minutes. Parallelization brings the warmup down to seconds. The xargs -P N tool starts N parallel processes, each fetching one URL. The optimal degree of parallelism depends on the server: sending more parallel requests than there are PHP-FPM workers can push the server into a backlog and becomes counterproductive.

For shops running Varnish or a CDN cache, parallelization should be moderate, because Varnish already has request coalescing built in: multiple simultaneous requests for the same URL are merged into a single backend request. However, too many parallel warmup requests for different URLs can exhaust the PHP-FPM worker pool. A value of 4 to 8 parallel requests is a good starting point for a single Magento server.

6. Integrating the warmup job into GitLab CI

The warmup job belongs in a warmup stage that runs after deploy and either before or alongside verify. Stage order: build → test → package → deploy → warmup → verify. The warmup job explicitly depends on the deploy job and only runs when the deploy succeeded. Setting allow_failure: true prevents a failed warmup from marking the entire pipeline as failed, a warmup failure is annoying, but it is not a deployment failure.

One important detail: the warmup job should combine when: on_success with allow_failure: true. That way it only runs after a successful deploy, but a warmup failure never blocks the verify job. The warmup results, load times and cache hit rates, are stored as an artifact and serve as a post-deploy monitoring data point. Across multiple deployments, this record shows whether performance stays stable after releases or degrades over time.

7. Warmup with Varnish and ESI blocks

For Magento running Varnish as the full page cache, the warmup principle is the same, but the implementation differs slightly. Varnish responds to an empty cache with a backend request to Magento that triggers rendering. So the warmup request first hits Varnish, which then forwards it to Magento. Varnish's X-Cache response header shows whether the request was a cache miss (MISS) or a cache hit (HIT).

ESI blocks (Edge Side Includes) in Magento with Varnish are a special case: pages with ESI blocks are split into multiple cache units. Warming up a page with ESI triggers several backend requests, one for the main page and one for each ESI block. That means the warmup effect of a single URL is larger with ESI than with a simple cached page, because multiple cache entries get filled. Anyone using Varnish with ESI should start the warmup with slightly lower parallelism to avoid overloading the backend.

warmup:varnish:
  stage: warmup
  script:
    # Warm up pages and verify Varnish cache status
    - |
      while IFS= read -r url; do
        # First request: cache MISS, triggers Magento rendering
        response=$(curl -sf -o /dev/null \
          -w "%{http_code}|%{time_total}|%{size_download}" \
          -H "User-Agent: Warmup-Bot/1.0" \
          --max-time 30 "${url}")

        http_code="${response%%|*}"
        rest="${response#*|}"
        load_time="${rest%%|*}"

        # Second request: should be a cache HIT from Varnish
        cache_header=$(curl -sf -I \
          -H "User-Agent: Warmup-Bot/1.0" \
          --max-time 5 "${url}" 2>/dev/null \
          | grep -i "x-cache:" || echo "x-cache: UNKNOWN")

        echo "[${http_code}] ${load_time}s ${cache_header##*: } ${url}"
      done < /tmp/warmup-urls.txt | tee warmup-results.log

  artifacts:
    paths:
      - warmup-results.log
    expire_in: 7 days
  when: on_success
  allow_failure: true

8. Validating results: checking for cache hits

A warmup job that never validates its results is barely better than no warmup at all. Validation happens in two steps: first, the HTTP status code of the warmup request is checked, 200 means success, 4xx or 5xx means a problem that needs investigating. Second, after the warmup request a second request is sent to the same URL to read the X-Cache header. A HIT confirms the page is now sitting in the cache.

The warmup results are stored as a CSV or log file as a GitLab artifact. Through the GitLab interface, the team can review the warmup results after each deployment: which pages were successfully cached, which had unexpected load times, and which came back with errors. A warmup failure on a specific URL points to a problem with that page that should be investigated independently of the warmup itself.

9. Comparison: no warmup vs. prioritized warmup

The table makes it clear: a prioritized warmup job requires minimal effort (20 to 30 lines of YAML and a URL list) and delivers substantial improvements across several dimensions. First load time for real visitors drops from several seconds to milliseconds, the cache stampede is prevented, and the warmup job doubles as an early error indicator for pages that don't respond correctly after deployment.

10. Summary

An automated cache warmup job in the GitLab pipeline is one of the simplest and most effective measures you can take after a Magento deployment. Implementation requires a prioritized URL list, a curl-based warmup mechanism with controlled parallelization, and a verification step that confirms cache hits. The result: visitors experience the shop at full cache performance right after deployment, no cache stampede, no post-deploy performance dip.

The warmup job runs as an allow_failure: true job after deploy, so warmup issues never affect the pipeline status. The warmup results are stored as an artifact and enable post-deploy monitoring across multiple deployments. Once you've implemented this job, you'll wonder how you ever deployed without it.

11. FAQ: Cache warmup after a Magento release