Log Management and Monitoring in Magento 2: from exception.log to Real Alerting

1. Why exception.log alone is not monitoring: the gap between logging and alerting

Many Magento teams confuse logging with monitoring because both operate on the same raw data. Log management and monitoring are actually two distinct disciplines: logging writes events to a file, monitoring actively evaluates those events and triggers a response when needed. An exception.log that quietly grows to several gigabytes inside var/log is, strictly speaking, just an archive. Nobody reads it proactively before a customer complains about a failed checkout. That exact gap between merely writing errors to disk and actually detecting a problem in real time is the starting point for any serious log management approach.

The difference shows up most clearly in response time. Without monitoring, a team often only learns about a broken payment module through a support ticket or a revenue dip that surfaces hours later in reporting. With working monitoring, the same error triggers a Slack message or a PagerDuty incident within seconds, because a handler classifies the log entry and the threshold for a critical notification has been crossed. This shift from reactive debugging to proactive alerting is the real value of a well thought out log management setup, not the mere existence of log files.

If you already know your way around Monolog channels and handlers in Magento, there is no need to repeat that groundwork here. The next layer is what matters: structured, machine-readable logs, central aggregation across every container, defined alerting rules, and an operational process that actually responds to an alert. That operational layer is the focus of the sections that follow.

2. Structured logging: a JSON formatter for machine-readable Magento logs

Classic Magento log lines are readable for humans but tedious for machines to parse. A log aggregator like Grafana Loki or Elasticsearch has to break free-text lines apart with regular expressions, which breaks every time the format changes even slightly. Production-grade log management therefore relies on structured JSON logging: every line is a complete, self-contained JSON object with fixed fields such as timestamp, level, channel, message, trace_id, and a freely extensible context object. This format can be indexed, filtered, and aggregated directly, with no parsing heuristics involved.

Monolog already ships a basic implementation via Monolog\Formatter\JsonFormatter, but it is usually too generic for real Magento operations. A custom formatter enriches every entry with fields that matter for monitoring: the active store view, the Magento version, the container hostname, and, if present, a trace ID from the running request. This enrichment happens once, inside the formatter, and does not need to be repeated in every single log call throughout the codebase, which keeps things consistent across all modules.

The formatter below shows a PHP 8.4 implementation with readonly properties for the static metadata that never changes within a single process:

declare(strict_types=1);

namespace Mironsoft\LogMonitoring\Logger\Formatter;

use Monolog\Formatter\NormalizerFormatter;
use Monolog\LogRecord;

/**
 * Structured JSON formatter enriching every log record with
 * static process metadata and the active trace identifier.
 */
final class StructuredJsonFormatter extends NormalizerFormatter
{
    public function __construct(
        private readonly string $hostname,
        private readonly string $magentoVersion,
        private readonly string $environment,
    ) {
        parent::__construct(self::SIMPLE_DATE);
    }

    /**
     * Formats a single log record as a single-line JSON document.
     *
     * @param LogRecord $record The record produced by the Monolog logger
     * @return string One JSON object terminated by a newline
     */
    public function format(LogRecord $record): string
    {
        $payload = [
            'timestamp' => $record->datetime->format(self::SIMPLE_DATE),
            'level' => $record->level->getName(),
            'channel' => $record->channel,
            'message' => $record->message,
            'context' => $this->normalize($record->context),
            'extra' => $this->normalize($record->extra),
            'trace_id' => $record->extra['trace_id'] ?? null,
            'host' => $this->hostname,
            'magento_version' => $this->magentoVersion,
            'environment' => $this->environment,
        ];

        return json_encode($payload, JSON_UNESCAPED_SLASHES) . "\n";
    }
}

The real advantage of a formatter like this only becomes clear once it meets the aggregation layer: fields such as environment or trace_id turn into searchable labels rather than text fragments that still need to be extracted with a regex. That cuts query time in Loki or Elasticsearch dramatically, and it is what makes alerting rules based on individual fields practical in the first place.

3. A custom Monolog handler for critical errors: push notifications to Slack or PagerDuty

Structured logs alone will not solve the alerting problem as long as nobody is watching them in real time. The next building block in the log management and monitoring stack is a custom Monolog handler that actively forwards critical log entries to an external channel instead of just writing them. The handler filters by log level (typically starting at CRITICAL or ERROR with an additional context check) and sends an HTTP POST to a Slack webhook or the PagerDuty Events API v2.

It is important that this handler never blocks the request cycle. A synchronous HTTP call inside a PHP-FPM worker that waits on a slow Slack API will slow down any checkout request that happens to trigger a critical error. In practice, teams either use a short timeout with fire-and-forget semantics, or dispatch asynchronously through a Magento queue (async.operations.all) that performs the actual push in a separate consumer process. For most Magento setups, a one- to two-second timeout combined with a try/catch that logs a failed push only internally, rather than triggering yet another alert storm, is enough.

declare(strict_types=1);

namespace Mironsoft\LogMonitoring\Logger\Handler;

use Monolog\Handler\AbstractProcessingHandler;
use Monolog\Level;
use Monolog\LogRecord;

/**
 * Custom Monolog handler pushing critical Magento log records
 * to a Slack incoming webhook or the PagerDuty Events API v2.
 */
final class CriticalAlertHandler extends AbstractProcessingHandler
{
    public function __construct(
        private readonly string $webhookUrl,
        private readonly string $serviceKey,
        private readonly string $alertProvider = 'slack',
        Level $level = Level::Critical,
    ) {
        parent::__construct($level, true);
    }

    /**
     * Sends a single log record to the configured alerting provider.
     *
     * @param LogRecord $record The record already filtered by minimum level
     * @return void
     */
    protected function write(LogRecord $record): void
    {
        $payload = $this->alertProvider === 'pagerduty'
            ? $this->buildPagerDutyPayload($record)
            : $this->buildSlackPayload($record);

        $context = stream_context_create([
            'http' => [
                'method' => 'POST',
                'header' => "Content-Type: application/json\r\n",
                'content' => json_encode($payload),
                'timeout' => 2.0,
                'ignore_errors' => true,
            ],
        ]);

        try {
            @file_get_contents($this->webhookUrl, false, $context);
        } catch (\Throwable) {
            // Alert delivery failure must never break the request cycle
            // and must never trigger another alert (avoids alert loops).
        }
    }

    /**
     * @param LogRecord $record The log record to convert
     * @return array<string, mixed> Slack-compatible message payload
     */
    private function buildSlackPayload(LogRecord $record): array
    {
        return [
            'text' => sprintf(
                ':rotating_light: *%s* on `%s`: %s',
                $record->level->getName(),
                $record->channel,
                $record->message
            ),
        ];
    }

    /**
     * @param LogRecord $record The log record to convert
     * @return array<string, mixed> PagerDuty Events API v2 payload
     */
    private function buildPagerDutyPayload(LogRecord $record): array
    {
        return [
            'routing_key' => $this->serviceKey,
            'event_action' => 'trigger',
            'payload' => [
                'summary' => $record->message,
                'source' => $record->channel,
                'severity' => 'critical',
            ],
        ];
    }
}

This handler is bound to a specific logger channel via di.xml, rather than being enabled globally for every log call. That keeps alerting scoped to channels that actually carry operationally relevant events, such as payment processing or inventory sync, while debug output from development modules never triggers a push.

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:ObjectManager/etc/config.xsd">
    <!-- Handler instance configured with alerting endpoint and provider -->
    <type name="Mironsoft\LogMonitoring\Logger\Handler\CriticalAlertHandler">
        <arguments>
            <argument name="webhookUrl" xsi:type="string">https://hooks.slack.com/services/REPLACE/WITH/TOKEN</argument>
            <argument name="serviceKey" xsi:type="string">REPLACE_WITH_PAGERDUTY_ROUTING_KEY</argument>
            <argument name="alertProvider" xsi:type="string">slack</argument>
        </arguments>
    </type>

    <!-- Dedicated logger channel for payment processing -->
    <type name="Mironsoft\LogMonitoring\Logger\PaymentLogger">
        <arguments>
            <argument name="name" xsi:type="string">payment</argument>
            <argument name="handlers" xsi:type="array">
                <item name="alert" xsi:type="object">Mironsoft\LogMonitoring\Logger\Handler\CriticalAlertHandler</item>
                <item name="system" xsi:type="object">Magento\Framework\Logger\Handler\System</item>
            </argument>
        </arguments>
    </type>
</config>

4. Central log aggregation: shipping logs from Magento containers to Grafana Loki or the ELK stack

A custom handler solves the problem for individual critical events, but it is no substitute for a central view across every container. Once a Magento setup consists of several web containers, cron containers, and consumer processes, log files end up scattered across different hosts and volumes. Operational log management therefore means shipping every log source to a central aggregator, instead of hopping from container to container with bin/log exception.log.

Two architectures dominate in practice: the ELK stack (Elasticsearch, Logstash, Kibana) with Filebeat as a lightweight log shipper, or Grafana Loki with Promtail or the Filebeat Loki output. Unlike Elasticsearch, Loki only indexes labels rather than the full log content, which reduces storage footprint and operating cost considerably, but is slower for complex full-text searches than Elasticsearch. For most Magento operators, who primarily filter by channel, level, and trace_id, Loki is the more cost-effective choice, while teams with a strong need for free-text search across error messages tend to lean toward Elasticsearch.

The Filebeat configuration below reads the structured JSON logs from var/log and ships them to a Loki endpoint. Because the logs are already in JSON format, there is no need for any Grok pattern definitions, which free-text logs would otherwise require.

# filebeat.yml: ships structured Magento JSON logs to Grafana Loki
filebeat.inputs:
  - type: filestream
    id: magento-json-logs
    paths:
      - /var/www/html/var/log/*.log
    parsers:
      - ndjson:
          target: ""
          add_error_key: true
          message_key: message

processors:
  - add_fields:
      target: ""
      fields:
        service: magento
        stack: production

output.loki:
  url: "http://loki:3100/loki/api/v1/push"
  batch_wait: 1s
  batch_size: 1024
  labels:
    keys: ["level", "channel", "environment", "host"]

If you run the ELK stack instead, you simply swap out the output block for Logstash or Elasticsearch directly, while the input and processor configuration stays identical, because both systems consume the same structured JSON lines. This decoupling of formatting in application code from transport in the log shipper is exactly why the JSON formatter from section two is the real foundation for any central log management setup.

5. Defining alerting rules: thresholds, rate-based alerts, and avoiding alert fatigue

A central log aggregator on its own still does not amount to monitoring, as long as nobody defines when a pattern in the logs actually justifies an alarm. Naive alerting rules that fire a Slack push for every single ERROR entry lead to alert fatigue within days: teams start ignoring notifications because most of them require no actual action. Good monitoring therefore distinguishes between individual events that are merely informative and patterns that demand a response.

Rate-based alerts are the most important lever here: instead of alerting on every single failed payment attempt, you define a rule such as "more than 20 failed payments per minute across all store views," which becomes a Grafana alert rule based on a Loki query, or an Elasticsearch watcher. These thresholds need to be tuned to the actual background noise of the shop: a high-traffic store has a different baseline of occasional payment failures than a small B2B store, where a single failure is already noteworthy.

Beyond raw rate, alerting rules should be tiered by severity: a WARNING alert lands in a team channel for awareness, while a CRITICAL alert with a sustained error rate triggers a PagerDuty incident with escalation. Deduplication matters just as much: the same underlying error occurring a hundred times a minute should create a single active incident, not a hundred separate notifications. PagerDuty and most modern alerting systems group events automatically via a dedup_key, provided the custom handler derives it consistently from error type and channel.

6. Log rotation and retention in production: storage costs and compliance requirements

Central aggregation does not automatically solve the problem of growing local log files inside the containers themselves. Without rotation, an active exception.log on a high-traffic store will fill the container volume within a few weeks, which in the worst case leads to a full filesystem and a broken checkout. Production-grade log management therefore always includes a clear rotation and retention strategy, regardless of whether the logs are also aggregated centrally.

For the Magento container, logrotate is the standard approach: log files are rotated once they reach a defined size or after a fixed time interval, compressed, and deleted after a defined number of generations. The copytruncate option matters here, because Magento and PHP-FPM hold file handles open on the log file for as long as the process is running, and simply moving the file without signaling the process would cause new log entries to be written into the void.

# /etc/logrotate.d/magento: rotate Magento log files in production
/var/www/html/var/log/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
    dateext
    dateformat -%Y%m%d
    size 500M
}

/var/www/html/var/log/exception.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

Retention is not just a matter of storage cost, it is increasingly a compliance question as well. Logs that contain personal data, such as email addresses in order errors, fall under GDPR and cannot be kept indefinitely. A sensible practice is tiered retention: raw logs locally for 14 to 30 days, aggregated and anonymized metrics in Grafana Loki for 90 days, and critical security events kept longer if needed in a separate, access-restricted store. This tiering reduces both storage costs and the risk of holding personal data longer than necessary.

7. Correlating logs with application performance monitoring: trace IDs across request boundaries

A single log entry shows that an error happened, but rarely why. To fully reconstruct a failed checkout, you need to see the associated HTTP request, the database query it triggered, the external API call to a payment provider, and the resulting log entry as one connected chain. That is exactly what correlating log management and monitoring with application performance monitoring, through a shared trace ID, delivers.

In practice, this means a front controller or an early plugin point generates a unique trace_id at the start of every request (or picks up one already set by the load balancer, for example via the traceparent header per W3C Trace Context), and that ID is then carried through the entire request lifecycle. The JSON formatter from section two writes this trace ID into every log entry, while APM tools such as New Relic, Datadog APM, or a self-hosted OpenTelemetry collector adopt the same ID in their spans.

The real operational payoff appears once you have a Grafana dashboard that links directly from an APM trace to the corresponding Loki log line, and conversely from a Slack alert straight to the full trace. This bidirectional linking shrinks root cause analysis from manually cross-referencing several systems down to a handful of clicks. Without consistent trace ID propagation across queue consumers, cron jobs, and asynchronous API calls, however, this chain breaks quickly, which is why the trace ID must be passed explicitly in message queue payloads and cron context, not just in the HTTP request header.

8. Operational process around alerts: on-call rotation, escalation levels, runbooks

Technical alerting without an organizational process behind it fizzles out. A PagerDuty incident that reaches nobody because no on-call rotation is defined has, in practice, almost the same effect as an alert that was never triggered. A working on-call rotation assigns a clearly responsible person at every point in time, who acknowledges first-tier alerts within a defined response time, typically five to fifteen minutes for critical production errors.

Escalation levels cover the case where the primary on-call person does not respond: once the response time elapses, PagerDuty automatically escalates to a second person or a team lead, and after that potentially to the entire development team. This escalation logic belongs firmly in the alerting tool's configuration, not in informal arrangements, because informal processes tend to fail under stress and outside core working hours.

Runbooks close the gap between "an alert came in" and "the problem is fixed." A runbook for a specific alert, say "payment error rate above threshold," describes concrete first steps: which dashboard to open, which log query in Grafana Loki narrows down the error, which external status pages (such as the payment provider's) to check, and when rolling back the last deployment is the right response. Without a runbook, even an experienced developer spends the first critical minutes of an incident orienting rather than fixing, which weighs especially heavily during nighttime alerts.

9. Dashboards for everyday use: which metrics from Magento logs are operationally relevant

Beyond reactive alerting, a team needs a dashboard that shows the shop's health at a glance, without an alarm having to fire first. For log management and monitoring in Magento, only a handful of metrics are genuinely operationally relevant, most everything else is just noise. Error rate per minute, broken down by log channel, reveals trend changes before they turn into a full-blown incident.

Just as important is the distribution of log levels over time: a sudden spike in WARNING entries often announces a CRITICAL problem before customers even notice, for example when an external payment endpoint gets progressively slower before it fails outright. A dashboard that makes these early warning signals visible shifts monitoring from pure error detection to genuine prevention.

For Grafana dashboards built on Loki queries, a fixed layout has proven itself: a time series of the overall error rate at the top, a breakdown by channel and store view below it, followed by a table of the ten most frequent error messages from the last hour, and a panel showing the current count of open PagerDuty incidents. This structure gives every team member, regardless of active on-call status, a reliable picture of the current system state in under a minute.

The table below compares the most common targets for central log management in terms of setup effort, searchability, cost, and alerting capability.

10. Summary

Log management and monitoring in Magento 2 starts exactly where plain logging stops: actively evaluating and forwarding critical events. A structured JSON formatter makes logs machine-readable, a custom Monolog handler sends critical errors straight to Slack or PagerDuty, and central aggregation in Grafana Loki or the ELK stack makes every container's logs searchable in one place. Alerting rules with sensible thresholds prevent alert fatigue, while log rotation and retention keep storage cost and compliance requirements under control.

The last, often underestimated piece is connecting monitoring with a working operational process: trace IDs link logs to APM traces, a clear on-call rotation with escalation levels ensures someone actually responds, and runbooks shorten the time from alarm to resolution. A dashboard with a small set of genuinely relevant metrics rounds off the setup and keeps system health visible to the whole team at all times, not just during an emergency.

11. FAQ: Log Management and Monitoring in Magento 2