Systematically Narrowing Down Production Errors After Deployments

1. Why Systematic Beats Intuitive

When a production error appears after a deployment, the natural impulse is to look for a solution right away. The problem: without a system, several changes often get made at once, which makes diagnosis harder and in some cases creates new errors. A structured diagnostic approach separates observation from hypothesis and hypothesis from action, in that order, not mixed together.

For Magento deployments a system is especially important because a single release typically contains many changes: PHP code, Composer packages, DI compilation, static content, database migrations, and possibly configuration changes. The error can be in any of these layers. Without a structured approach the search is guesswork, with a system it becomes narrowing down.

The framework is simple: first observe (logs, HTTP status, monitoring), then classify (which layer is affected?), then test specifically (check individual hypotheses), then decide (roll back or fix). This sequence applies regardless of how urgent the error is, urgency affects the speed, not the system.

2. The First 60 Seconds: Triage Without Panic

In the first 60 seconds after a production error is reported, it is about triage, not diagnosis. Three questions determine the first step: Has the application failed completely, or is it a partial error? Is the error behavior reproducible? Are other users having the same experience? The answers to these questions determine whether maintenance mode should be switched on immediately and a rollback started, or whether a calmer diagnosis is possible.

A complete outage (500 errors on every page, the health endpoint not responding) requires an immediate rollback without further diagnosis. A partial error (specific pages or features) allows for a more targeted analysis. The rule of thumb: if the error affects more than 10% of users or touches revenue critical paths (checkout, login), rollback is the first response, diagnosis comes afterward in the old state.

# GitLab pipeline as diagnostic tool: rerun verify stage manually
verify:manual-diagnostic:
  stage: verify
  rules:
    - if: '$CI_PIPELINE_SOURCE == "web"'
      when: manual
  script:
    # Check current symlink target
    - ssh "${DEPLOY_USER}@${DEPLOY_HOST}"
        "readlink -f ${DEPLOY_PATH}/current"

    # Show last 50 lines of exception log
    - ssh "${DEPLOY_USER}@${DEPLOY_HOST}"
        "tail -50 ${DEPLOY_PATH}/current/var/log/exception.log 2>/dev/null || echo 'Log empty'"

    # Check PHP-FPM status
    - ssh "${DEPLOY_USER}@${DEPLOY_HOST}"
        "systemctl status php8.4-fpm --no-pager -l | tail -20"

    # HTTP status of main pages
    - for URL in / /checkout/cart /customer/account/login/; do
        STATUS=$(curl -s -o /dev/null -w "%{http_code}" "https://shop.example.com${URL}");
        echo "  $URL → HTTP $STATUS";
      done

3. Reading and Interpreting Magento Logs

The Magento log system has several files with different purposes: var/log/exception.log contains stack traces for unhandled exceptions, var/log/system.log contains general system events, and var/log/debug.log contains detailed debug output when debug mode is enabled. The most important file for deployment errors is exception.log, this is where PHP exceptions that Magento could not handle itself end up.

When reading the exception log after a deployment, the timestamp is the most important filter: entries that appear exactly at the time of the symlink switch or shortly after are directly associated with the deployment. Older entries may not be relevant. A quick grep for the release timestamp: grep "2026-05-09 14:30" exception.log, the deployment time is logged in the GitLab pipeline output.

4. The GitLab Pipeline as a Diagnostic Tool

The GitLab pipeline contains more diagnostic information than is visible at first glance. The deploy job's output shows the exact time of the symlink switch, any error messages from the Magento CLI commands, and the output of setup:upgrade. If setup:upgrade ended with a warning or a non standard exit code but was still marked as successful, that is the first clue.

The GitLab pipeline view also shows which jobs ran in which order, which artifacts were produced, and whether there was a cache hit. A build job that loaded Composer packages from cache could have brought along an outdated dependency. The pipeline trace under "Download artifacts" and "Job artifacts" is often the fastest way to reconstruct the exact build state.

5. Narrowing Down Changes: git bisect and diff

If the error appears after several commits and can't be explained directly from the logs, git bisect is the most precise tool for narrowing it down. It enables a binary search through the commit history: you mark a known good state and the current bad state, and Git automatically picks the next commit to test. With 20 commits, narrowing it down takes at most 5 steps.

For Magento deployments an alternative to git bisect is a direct diff between the current release and the previous one: git diff PREVIOUS_TAG HEAD -- app/code/ app/design/. This diff shows all PHP, template, and layout changes included in the current release. Combined with the stack trace from the exception log, this often identifies the affected file directly, without having to run a full bisect.

# Diagnostic job to compare current and previous release on server
diagnose:diff-releases:
  stage: verify
  rules:
    - if: '$CI_PIPELINE_SOURCE == "web"'
      when: manual
  script:
    - |
      ssh "${DEPLOY_USER}@${DEPLOY_HOST}" bash -s << 'REMOTE'
        set -euo pipefail
        CURRENT=$(readlink -f ${DEPLOY_PATH}/current)
        RELEASES=$(ls -1t ${DEPLOY_PATH}/releases/)
        PREVIOUS=$(echo "$RELEASES" | sed -n '2p')

        echo "=== Current release: $(basename $CURRENT) ==="
        echo "=== Previous release: $PREVIOUS ==="

        # Show what changed in app/code between releases
        diff -rq --brief \
          "${DEPLOY_PATH}/releases/${PREVIOUS}/app/code/" \
          "${CURRENT}/app/code/" 2>/dev/null \
          || echo "Diff completed above"

        # Show exception log entries since current deploy time
        DEPLOY_TIME=$(stat -c %Y "$CURRENT")
        find ${CURRENT}/var/log/ -name "exception.log" -newer "$CURRENT" \
          -exec tail -30 {} \; 2>/dev/null || echo "No recent exceptions"
      REMOTE

6. When to Roll Back, When to Fix?

The rollback decision depends on three factors: the severity of the error, the availability of a known good state, and the complexity of the fix. A complete store outage always requires an immediate rollback, diagnosis can happen afterward in the stable state. A partial error in a non critical area (for example a broken widget on an information page) can be fixed directly if the cause is clear and the fix is simple.

The most dangerous situation is an error whose cause is unclear but for which a fix is attempted directly anyway. This often leads to a chain of changes where, in the end, it is unclear what caused the original error and what the fixes introduced. In this case rollback is always the better choice: a stable starting state, a clean diagnosis, a controlled fix.

7. Environment Differences as a Source of Errors

A significant portion of production errors after deployments doesn't lie in the code itself but in differences between the build environment and the production environment. PHP version, installed PHP extensions, file permissions, environment variables, database version, and Redis configuration can differ between the runner and the production server. These differences aren't visible at build time but show up at runtime on the server.

The systematic checklist for environment differences: compare the PHP version on the runner and the server (php -v), check the installed extensions (php -m), validate the file permissions of the release directory, check the env.php configuration for correct values, and compare the output of bin/magento config:show against the expected configuration state. This checklist should be run through for every unexplained production error.

8. Follow Up: Documenting the Error

Following up on a production error is at least as important as fixing it. A documented error with cause, symptom, diagnostic path, and fix becomes part of the team's knowledge base, an undocumented error has a high probability of recurring. The documentation doesn't have to be extensive: date, deployment ID from GitLab, symptom, cause, fix, and preventive measure in four to five sentences is enough.

More important than the form is the consequence: every production error after a deployment should lead to at least one change in the process. That can be a new post deploy check, an expanded test stage, additional validation in the build stage, or a change to the environment configuration. Without that consequence, the follow up is just a record without any learning effect.

9. Error Classes and First Points of Contact

Production errors after Magento deployments typically fall into one of a few classes. The following table shows the most common classes, their typical symptoms, and the first point of contact for diagnosis.

The table shows that every error class has a clearly defined first point of contact. The goal of triage is to determine the error class as quickly as possible and then check the right point of contact, instead of searching every possible log at once. A targeted first check often saves several minutes during an incident, minutes that make the difference between a short outage and a long one.

10. Summary

Systematically narrowing down production errors after deployments is a skill teams build through repetition, not through intuition. The three core principles: observe before acting, test hypotheses one at a time, and when in doubt prefer rollback over diagnosis. In this context GitLab offers more than just a CI/CD platform: pipeline output, job traces, and artifact metadata are valuable diagnostic data available right after the deployment.

The investment in structured diagnosis doesn't only pay off during an incident. Teams that diagnose systematically learn from every error and build processes that prevent errors of the same class at the next deployment. That is the real value of following up: no error should ever catch the team unprepared twice.

11. FAQ: Production Errors After Deployments