Debugging Production-Only Bugs in Magento 2

1. Why some bugs only occur in production

Production-only bugs almost always arise from a combination of factors that simply do not exist locally: real load patterns, production-sized data volumes, race conditions under concurrent usage, or configuration values that differ between environments. A bug that only occurs with a thousand concurrent sessions naturally cannot be triggered on a local single-user instance, no matter how many times the code gets read.

The first step in dealing with production-only bugs is accepting that classic local debugging with breakpoints reaches its limits here. Instead, it takes a combination of observability tools that collect data during live operation without disrupting it, plus a systematic method for narrowing down environment differences instead of guessing wildly.

This article shows a structured approach to production-only bugs in Magento 2, from systematic scoping through safe remote debugging to techniques like canary deployments that deliberately isolate a bug to a small group of users before it affects the entire system.

2. Systematically narrowing down environment differences

Before even attaching a debugger, a systematic comparison of production and local environment is worthwhile: PHP version, installed extensions, php.ini values like memory_limit and max_execution_time, Redis and database version, and enabled modules with their versions. Production-only bugs surprisingly often arise from a single deviating PHP extension or a different OPcache configuration that never shows up locally.

A simple but effective tool: bin/magento sys:info, or a custom diagnostic script that outputs the relevant environment data in structured form, both locally and via SSH in production, and automatically diffs the two outputs. Anyone who checks this diff first for every production-only bug surprisingly often finds the decisive clue right here, before code-level debugging even begins.

#!/usr/bin/env bash
# env-diff.sh — compare local and production environment fingerprints
set -euo pipefail

collect_fingerprint() {
  echo "PHP: $(php -v | head -n1)"
  echo "Extensions: $(php -m | sort | tr '\n' ',')"
  echo "memory_limit: $(php -i | grep '^memory_limit')"
  echo "opcache.enable: $(php -i | grep '^opcache.enable ')"
  bin/magento module:status | sort
}

echo "=== Local fingerprint ==="
collect_fingerprint > /tmp/local_fingerprint.txt
cat /tmp/local_fingerprint.txt

echo "=== Production fingerprint (via SSH) ==="
ssh deploy@production "cd /var/www/html && $(declare -f collect_fingerprint); collect_fingerprint" \
  > /tmp/prod_fingerprint.txt

diff /tmp/local_fingerprint.txt /tmp/prod_fingerprint.txt || echo "[INFO] Differences found above"

3. Remote debugging with Xdebug without production downtime

Permanently enabling Xdebug in production is not an option because of the performance overhead, but a deliberately triggered remote debugging session for a single request certainly is. Via the XDEBUG_TRIGGER cookie or query parameter, Xdebug can be configured to activate only for requests carrying this specific trigger, while every other request runs unaffected at full performance.

For production-only bugs that can be traced to a specific combination of user, session and request parameters, this targeted trigger approach is the safest way to get real step debugging access without burdening the rest of the production environment. Important: the trigger value should never be a simple, guessable secret, but a randomly generated value set only temporarily for the duration of the investigation and removed immediately afterward.

#!/usr/bin/env bash
# Trigger a single, targeted Xdebug session on production
# without enabling Xdebug globally (avoids performance overhead)
set -euo pipefail

XDEBUG_SECRET="$(openssl rand -hex 16)"

echo "[INFO] Set this cookie in your browser DevTools for the debugging session:"
echo "XDEBUG_TRIGGER=${XDEBUG_SECRET}"
echo "XDEBUG_SESSION_START=${XDEBUG_SECRET}"

echo "[INFO] Ensure xdebug.mode=debug and xdebug.discover_client_host=1"
echo "[INFO] Point xdebug.client_host to your local IDE listener"
echo "[INFO] Remove the cookie and rotate the secret once debugging is complete"

4. Structured logging and correlation IDs

Without a correlation ID, a single request in production can practically not be traced through all involved log files, especially when asynchronous message queue consumers or multiple PHP-FPM workers are involved at once. A unique request ID, generated when each request comes in and embedded into every log entry, every exception and every message sent to message queues, makes production-only bugs traceable across distributed system boundaries.

Magento's Monolog integration allows exactly this enrichment via a custom logger processor that automatically adds the correlation ID to every log entry, without needing to manually adjust every single $this->logger->info() call in the code. Once a production-only bug gets reported, filtering by the correlation ID from the support ticket is enough to reconstruct the full chain of all involved log lines.

<?php
declare(strict_types=1);

namespace Vendor\Module\Logger;

use Monolog\Processor\ProcessorInterface;
use Monolog\LogRecord;

/**
 * Adds a request-scoped correlation ID to every log record,
 * so a single request can be traced across all log files and queues.
 */
class CorrelationIdProcessor implements ProcessorInterface
{
    private string $correlationId;

    /**
     * Generates the correlation ID once per request lifecycle.
     */
    public function __construct()
    {
        $this->correlationId = bin2hex(random_bytes(8));
    }

    /**
     * Injects the correlation ID into the log record's extra context.
     *
     * @param LogRecord $record Log record being processed
     * @return LogRecord
     */
    public function __invoke(LogRecord $record): LogRecord
    {
        $record->extra['correlation_id'] = $this->correlationId;
        return $record;
    }
}

5. Feature flags for targeted reproduction

Some production-only bugs can be narrowed down most efficiently by disabling suspicious code behind a feature flag for specific customer groups, store views or even individual users, while the rest of the system keeps running unchanged. If the bug stops occurring with the flag disabled, the cause is narrowed down without needing a full deployment rollback.

Feature flags are also excellent for gradually rolling out a suspected fix while watching live to see whether the production-only bug actually disappears. Instead of switching a fix on for all users at once, you enable it first for a small percentage segment and compare the error rate between the flag-active and flag-inactive group before the full rollout happens.

# feature-flags.yaml — scoped flag for isolating a suspected bug source
flags:
  new_checkout_totals_calculation:
    enabled: false
    rollout:
      strategy: "percentage"
      percentage: 5
      sticky_by: "customer_id"
    scopes:
      - store_view: "store_de"
      - customer_group: "wholesale"
    description: >
      Suspected source of incorrect tax totals reported only in production.
      Enabled for 5 percent of German store customers to compare error rates.

6. Anonymized production data for staging

A common reason why production-only bugs are not reproducible in staging: the test data is too small, too clean or too homogeneous compared to the real production database. A customer with an unusually long order history, a product with an unusually large number of variants, or an address with unusual special characters can trigger edge cases that simply do not appear in synthetic test data.

A regularly refreshed, anonymized snapshot of the production database in staging, combined with a read replica for load-intensive analysis without production risk, closes this gap. What matters here is consistently anonymizing personal data before importing it into staging, so realistic data volumes are available for reproduction purposes without incurring GDPR risks.

7. Profiling with Blackfire in production

When a production-only bug turns out to be a performance problem, for example a request that occasionally times out under certain load, Blackfire's production-grade profiling delivers detailed call graphs without noticeably slowing down the request. Unlike Xdebug, Blackfire's profiler is designed for continuous operation in production and can be activated for individual requests via an HTTP header.

For production-only bugs that only occur under real load, Blackfire can additionally be combined with synthetic load testing: a load test that mimics production load while simultaneously capturing a Blackfire profile of the affected request often reveals exactly the spot where execution time increases exponentially instead of linearly under load.

#!/usr/bin/env bash
# Trigger a single-request Blackfire profile in production
set -euo pipefail

curl -sS \
  -H "X-Blackfire-Query: signature=abc123&flags=samples=10" \
  -H "Cookie: PHPSESSID=${SESSION_ID}" \
  "https://shop.example.com/checkout/cart/index" \
  -o /dev/null

echo "[INFO] Profile submitted — review the call graph in the Blackfire dashboard"

8. Canary deployments for bug isolation

A canary deployment rolls out a new version only to a small percentage of production traffic, while the majority of users stay on the stable version. For hunting down production-only bugs, this pattern can be inverted: a suspicious code path gets instrumented with extra, verbose debug logging only for canary traffic, while the rest of the traffic runs unchanged at a normal log level.

This approach limits both risk and log volume: instead of enabling verbose debug logging for all production traffic, which quickly leads to log flooding and extra I/O load, detailed logging concentrates on the small canary share where the production-only bug is suspected. Once enough data has been collected, the canary traffic share gets set back to zero.

9. Debugging techniques compared

The following table maps the techniques presented to their use case for production-only bugs.

In practice, several of these techniques get combined: an environment diff for initial scoping, correlation IDs for tracing the affected request, and depending on the finding, either triggered Xdebug for logic errors or Blackfire for performance issues. No single tool solves production-only bugs alone, but the right combination makes them systematically, rather than randomly, discoverable.

10. Summary

Production-only bugs arise from factors that are missing locally: real load, production-sized data, race conditions and environment differences. The systematic solution begins with a diff between the local and production environment, followed by deliberately triggered Xdebug for individual requests, correlation IDs for distributed tracing, and feature flags for controlled isolation of suspicious code.

For performance-related production-only bugs, Blackfire delivers production-grade profiling without noticeable overhead, while canary deployments controllably limit new code changes to a small traffic share. No single technique solves every case, but the combination of observability, targeted debugging and controlled traffic isolation makes even the most stubborn bugs systematically discoverable instead of left to chance.

11. FAQ: Debugging Production-Only Bugs