Automating Magento Smoke Tests After Deployment

1. Why smoke tests are essential

A deployment process that includes no verification after the symlink switch ends technically, but not functionally. The pipeline job reported exit code 0, the symlink points to the new release directory, the cache was cleared, but nobody checked whether Magento actually responds correctly. In practice, some of the most common post-deploy problems are not caused by the deployment process itself, but by factors that only become visible after the switch: a class that triggers an autoload error in the new version, a cache that was not fully cleared, or a shared file that is missing or has incorrect permissions.

Smoke tests are simple, fast checks that confirm the application's most important functions still work at a basic level after the deploy. They are not a replacement for comprehensive integration or end-to-end tests, which take minutes or hours to run: they are the 60-second safety net that catches obvious failures immediately and, in an automated deployment, can trigger a rollback before the first customer ever sees the error.

In GitLab CI/CD, smoke tests belong in the verify stage, which runs after the deploy stage. If a smoke test job fails, the entire pipeline fails. That is the signal that either triggers a manual rollback job or, in a fully automated setup, directly triggers the rollback step. With this structure, the deployment process is not complete at deployed, but only at deployed and verified.

2. Which smoke tests make sense for Magento

Smoke tests for Magento can be divided into three categories: HTTP checks test whether specific shop URLs return the correct HTTP status codes and contain specific content. CLI health checks use SSH to check the state of the Magento installation on the server: cache status, maintenance mode status, module status. Service checks verify that external services such as Redis, RabbitMQ, or OpenSearch are reachable and functioning.

For most Magento setups, ten to fifteen targeted checks are enough to cover the most critical paths. The most important ones are: HTTP 200 for the homepage, the category and product pages, checkout, and the customer login page. HTTP 200 for the health check endpoint, if one exists. Cache status via CLI: no single cache type should show the status disabled unless that is explicitly intended. Maintenance mode status: must be 0 (disabled). Checking Magento deployment logs for errors from the last five minutes. Together, these checks take less than 30 seconds and cover the most common post-deploy failure patterns.

3. HTTP checks: verifying status codes and response content

HTTP checks with curl are the simplest and most reliable tool for external smoke tests. The -f flag makes curl exit with an error code when the HTTP status code is 4xx or 5xx. The --max-time flag sets a timeout so hanging requests do not block the pipeline job indefinitely. For more complex checks, grep or jq can be used to check the response body for expected content, for example whether the homepage actually contains the shop name instead of an error message or an empty body.

One important aspect of HTTP checks in CI pipelines: the GitLab runner needs network access to the target URL. If the production server is not reachable from the internet, the runner must run in the same network segment as the server or be addressed via an internal DNS name. For staging environments this is usually straightforward. For production servers behind firewalls or in private networks, the self-hosted runner needs network access configured explicitly.

4. CLI health checks over SSH

CLI health checks run over SSH on the target server and invoke Magento commands there. They complement the HTTP checks with internal states that are not visible from the outside: cache status, maintenance mode, module activation, and deployment log errors. The SSH key for this connection is already stored as a protected variable in GitLab and used in the deploy job, so the same key can be reused in the verify job.

The most important CLI check for Magento after a deploy is bin/magento cache:status. It outputs the status of all cache types: if one of the cache types shows as 0 (disabled) when it should be enabled, that points to a problem with the deploy. Equally critical: bin/magento maintenance:status must return 0. A forgotten maintenance:disable at the end of the deploy script is a classic mistake that makes the shop unreachable for end customers, so a check that catches exactly that is essential.

5. A complete verify job in GitLab CI/CD

The following verify job combines HTTP checks and CLI health checks into a single GitLab job that runs after the deploy job succeeds. On failure, the entire job fails, which marks the pipeline as failed and triggers notifications to the team.

# .gitlab-ci.yml: verify stage with smoke tests for Magento
verify:production:
  stage: verify
  image: alpine:latest
  before_script:
    # Install curl and openssh for HTTP and CLI checks
    - apk add --no-cache curl openssh-client bash
    # Configure SSH for deployment server access
    - mkdir -p ~/.ssh
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts
  script:
    # --- HTTP Smoke Tests ---
    # Check homepage: must return HTTP 200
    - |
      curl -sf --max-time 15 --retry 3 --retry-delay 5 \
        -o /dev/null -w "HTTP %{http_code} homepage\n" \
        "https://${SHOP_DOMAIN}/" || (echo "FAIL: Homepage did not return 200" && exit 1)
    # Check category page: must return HTTP 200
    - |
      curl -sf --max-time 15 \
        -o /dev/null -w "HTTP %{http_code} category\n" \
        "https://${SHOP_DOMAIN}/catalogsearch/result/?q=test" || exit 1
    # Check checkout: must return HTTP 200 (not redirect loop or 500)
    - |
      curl -sf --max-time 15 -L \
        -o /dev/null -w "HTTP %{http_code} checkout\n" \
        "https://${SHOP_DOMAIN}/checkout/" || exit 1
    # Check customer login page
    - |
      curl -sf --max-time 15 \
        -o /dev/null -w "HTTP %{http_code} customer-login\n" \
        "https://${SHOP_DOMAIN}/customer/account/login/" || exit 1
    # --- CLI Health Checks via SSH ---
    # Check maintenance mode: must be disabled (output: Status: 0)
    - |
      ssh -o StrictHostKeyChecking=no \
        "${DEPLOY_USER}@${DEPLOY_HOST}" \
        "cd ${DEPLOY_PATH}/current && php bin/magento maintenance:status" \
        | grep -q "Status: 0" || (echo "FAIL: Maintenance mode is still enabled" && exit 1)
    # Check cache status: no cache type should be disabled
    - |
      ssh "${DEPLOY_USER}@${DEPLOY_HOST}" \
        "cd ${DEPLOY_PATH}/current && php bin/magento cache:status" \
        | grep -v "^$" | grep -v "Current status" \
        | awk '{print $NF}' | grep -q "^0$" \
        && echo "FAIL: One or more cache types are disabled" && exit 1 || true
    # Check Magento deployment log for recent errors
    - |
      ssh "${DEPLOY_USER}@${DEPLOY_HOST}" \
        "find ${DEPLOY_PATH}/current/var/log -name 'support_report*.log' \
          -newer ${DEPLOY_PATH}/current/var/.deploy_timestamp \
          -exec grep -l 'ERROR\|CRITICAL' {} \;" \
        | grep -q "." \
        && echo "WARN: Error logs found after deployment" || true
  environment:
    name: production
  when: on_success
  allow_failure: false
  tags:
    - deploy
    - shell
  needs:
    - job: deploy:production

Three details in this verify job matter in particular: first, curl uses the flags --retry 3 and --retry-delay 5 so that temporary network issues do not immediately cause a test failure. Magento sometimes needs a few seconds after the cache flush before the first request is fully rendered, and retries bridge this short warmup phase. Second, allow_failure: false is set explicitly: the job must not be allowed to pass as a warning. Third, needs: [deploy:production] is set so the verify job starts immediately after the deploy job, without waiting for other parallel jobs.

6. Automatic rollback on failed tests

When a smoke test job fails, that is the moment a rollback is initiated, either manually or automatically. GitLab supports both models. In the manual model, there is a rollback job with when: manual in the rollback stage, which is only offered after a failed verify job. A developer then clicks the rollback button manually in the GitLab pipeline view. This gives the team control before the rollback runs, which is useful if the verify job occasionally produces false positive failures.

In the automatic rollback model, the rollback job is configured with when: on_failure and runs automatically when the verify job fails. This minimizes the time a broken deployment stays active. The risk: a false positive smoke test failure, for example caused by a brief network outage during the test, triggers an unnecessary rollback. That is why it is important to make the smoke tests robust, with retry mechanisms, realistic timeouts, and checks that genuinely detect Magento problems rather than infrastructure fluctuations.

7. Timing: when smoke tests should run

Smoke tests must run after the symlink switch and after the cache flush, not before. If they ran before the cache flush, they could still be testing responses from the old cache and would fail to detect problems with the new release. The correct timing in the pipeline: deploy job (symlink, env.php, cache flush, maintenance:disable), then verify job (smoke tests).

Another timing issue occurs when Magento still needs a few seconds to warm up the new cache after the cache flush and maintenance:disable. The first request after a cache flush is always slower than subsequent ones, because Magento recalculates configuration, layout, and block output at that point. If the smoke test is too aggressive, with a very short timeout and no retries, it can fail on exactly this slow first request. The solution: build in a short wait (10 to 15 seconds) after the deploy job, or use retries with delay in the curl command.

8. Comparison: different testing strategies

There are several approaches to implementing smoke tests after deployments. The difference lies in the effort involved, the reliability, and the depth of the check.

The recommended strategy for most Magento teams is the combination of curl HTTP checks and SSH CLI checks: it completes in under 60 seconds, requires no additional test frameworks, and covers the most common post-deploy failure patterns. Playwright or Cypress make sense as separate quality assurance in the test stage before the deploy, not as smoke tests after the deploy, because they are too time consuming for the verify stage.

9. Common mistakes with smoke tests

The most common mistake with smoke tests is testing against a URL that runs caching at the infrastructure level (CDN, Varnish) and still serves the old cache even though the Magento application cache has already been cleared. A curl check against https://shop.example.com/ can return HTTP 200, but the content still comes from the Varnish cache of the old release. Solution: send smoke tests with a Cache-Control: no-cache header, or test directly against the web server without Varnish where possible.

A second mistake is a --max-time that is too short without retries. The first request after a cache flush can take 5 to 10 seconds if Magento needs to rebuild configuration and layout. A 5-second timeout without retry fails on this first request even though Magento is working correctly. Always use --retry 2 or --retry 3 combined with a reasonable timeout of 15 to 20 seconds. A third mistake is checking too many URLs in the smoke test job, which stretches the runtime to several minutes. Smoke tests should be fast: at most ten to fifteen checks that together stay under 60 seconds.

10. Summary

Automated smoke tests after Magento deployments are the functional conclusion of a deployment process. A deployment without a verify stage ends technically, but not with the certainty that Magento actually works correctly. The combination of curl HTTP checks (homepage, category, checkout, login) and SSH CLI checks (maintenance status, cache status) covers the most common post-deploy failure patterns in under 60 seconds.

In GitLab CI/CD, smoke tests belong in the verify stage with when: on_success and allow_failure: false. When tests fail, the pipeline fails and the team gets notified. A rollback job in the rollback stage with when: on_failure or when: manual gives the team the ability to quickly return to the previous release. With this structure, the deployment process is complete: build, test, deploy, and verified.

11. FAQ: Smoke Tests for Magento After Deployment