Test Observability: Using Traces and Logs for Test Runs

1. The context problem of classic test failure reports

A classic test failure report consists of an error message, a screenshot, and at best a short video of the final seconds before the failure, showing exclusively the frontend symptom, not the actual cause, which often sits several layers deeper in the system. When a checkout test fails because an expected success message never appears, it stays completely unclear whether the underlying REST request returned an error, whether a payment gateway call took too long, or whether an internal caching issue in the Magento backend was the actual cause.

This lack of context forces developers to manually navigate several separate systems after every failure, say the CI log of the test runner, the application logs of the web server, and possibly a separate APM dashboard, mentally aligning timestamps to reconstruct a coherent sequence of events in the first place. In a distributed architecture with several microservices or external payment providers, this manual reconstruction quickly becomes so laborious that a single, sporadically occurring test failure needs more time for root cause analysis than for the actual fix.

2. OpenTelemetry as a shared foundation

OpenTelemetry has established itself as a vendor-neutral standard for generating and exporting traces, metrics, and logs, making it particularly well suited as a shared foundation for test observability, since both the test framework and the backend can propagate the same trace ID across system boundaries. A trace consists of a chain of spans, where each span represents a single, clearly bounded unit of work, say an individual HTTP request, a database query, or an internal function call, linked into a complete, coherent flow via a shared trace ID.

For integration into an E2E test, it's usually enough for the test to generate its own trace ID at the start and attach it as an HTTP header to every request the browser issues, letting the backend automatically continue the same trace, provided it's also OpenTelemetry-instrumented. This produces a single, continuous trace spanning from the click in the browser all the way to the final database query in the backend, which can then be viewed as a whole in a tracing backend like Jaeger, Tempo, or a commercial APM solution.

import { test, expect } from '@playwright/test';
import { trace, context } from '@opentelemetry/api';

test('checkout completes with trace correlation', async ({ page }) => {
  const tracer = trace.getTracer('e2e-checkout-tests');
  const span = tracer.startSpan('checkout-flow');
  const traceId = span.spanContext().traceId;

  await page.setExtraHTTPHeaders({
    'traceparent': `00-${traceId}-${span.spanContext().spanId}-01`,
  });

  await page.goto('/checkout');
  await page.click('[data-testid="place-order"]');
  await expect(page.locator('[data-testid="order-success"]')).toBeVisible();

  span.end();
  // On failure: log the traceId in the test report
  console.log(`Trace ID for this test run: ${traceId}`);
});

3. Making the trace ID visible in the test report

A generated trace ID does little good if it disappears exclusively into the console output of the test run and nobody actually has it at hand when looking at a failed test. That's why the trace ID should be stored as a fixed part of the test report, say as an additional attribute in Allure or as a direct, clickable link to the corresponding trace in the tracing backend, openable directly from the test report without manual copying and pasting.

In practice, a proven pattern is having a test hook automatically attach the associated trace ID to the test report after every failed test, so a developer glancing at CI results in the morning lands directly in the complete, distributed trace with a single click, instead of having to laboriously search for the matching request based on timestamps.

4. Root cause analysis using a concrete example

Suppose an E2E test for the Magento checkout fails sporadically because the order confirmation occasionally only appears after several seconds and a too-tightly-set timeout gets exceeded. Without tracing, it stays unclear whether the delay originates in the frontend, in the checkout controller, in an external tax calculation service, or in the database, making a targeted fix practically impossible without searching randomly in several places at once.

With a continuous trace, on the other hand, it immediately shows which single span consumes the bulk of the total time: if the trace shows that 90 percent of the time was spent in a span named `tax-service.calculate`, the cause is clearly identified, and the team can address that external service directly instead of a vague guess. This precision often reduces the average time to identify a problem, commonly called mean-time-to-detect or mean-time-to-diagnose in observability terminology, from several hours to a few minutes.

5. Correlating structured logs with the trace ID

Traces show the temporal structure of a flow, but don't always provide enough content detail, which is why structured log entries containing the same trace ID as a field are an important complement. A backend log entry, say a detailed error message from a payment provider, can then be directly attributed to the triggering test run simply by filtering for the known trace ID, instead of searching through the entire log volume based on approximate time windows.

For PHP applications like Magento, this pattern can be implemented via a simple Monolog processor that extracts the current trace ID from the incoming traceparent header and automatically adds it to every log entry as an extra context field, so all log lines of a request consistently carry the same trace ID and can be filtered specifically for that one ID in a central log aggregator like Elasticsearch.

<?php
declare(strict_types=1);

namespace Mironsoft\Observability\Logger;

use Monolog\LogRecord;

/**
 * Adds the current trace ID from the traceparent header to every log entry.
 */
final class TraceIdProcessor
{
    /**
     * Reads the traceparent header and extracts the trace ID.
     *
     * @param LogRecord $record The log record to process.
     * @return LogRecord The enriched log record.
     */
    public function __invoke(LogRecord $record): LogRecord
    {
        $header = $_SERVER['HTTP_TRACEPARENT'] ?? '';
        if ($header !== '') {
            $parts = explode('-', $header);
            $record->extra['trace_id'] = $parts[1] ?? 'unknown';
        }
        return $record;
    }
}

6. Integration into the CI pipeline

For test observability to remain useful beyond the ephemeral storage of the CI runner, the trace ID needs to be persisted permanently, usually as part of the test report artifacts the pipeline archives. A proven pattern is generating a compact JSON file at the end of every test run containing the test name, result, and associated trace ID for each test case, and storing this file as a build artifact alongside the actual test report.

It's also worth adding an automatic link in the pull request comment, so the code review view already shows directly which failed test belongs to which trace, without a reviewer having to manually switch to the CI interface and search for the matching test run there.

7. Sampling strategy: not every trace needs permanent storage

If every single span of every single test run were stored permanently and completely, an unaffordably large volume of data would quickly accumulate in the tracing backend with several thousand test runs per day, which is why a thoughtful sampling strategy is necessary. A pragmatic approach is tail-based sampling, where all spans of a trace are first collected, and only at the end of execution is it decided whether the complete trace gets stored permanently, say only if the associated test actually failed or had an unusually long total runtime.

Successful, unremarkable test runs, on the other hand, can usually be discarded after a short time or with a significantly reduced retention period, since their value for later analysis is low, while a single, well-preserved trace of a failed test represents substantial time savings during troubleshooting and generally justifies the additional storage cost.

8. Limits and the actual effort involved

Test observability with full distributed tracing pays off especially for teams with several interacting services or external integrations, where the root cause is regularly unclear, while for a single, monolithic application with manageable complexity, well-structured, correlated logs often already provide enough insight, without justifying the additional instrumentation effort of full tracing.

The initial effort lies mainly in consistently instrumenting all involved services, since a single non-instrumented service in the flow leaves a gap in the trace where the chain breaks, forcing root cause analysis to continue manually at exactly that point. A gradual rollout, starting with the most critical and most frequently failing test paths, usually delivers visible value faster than trying to instrument the entire system landscape completely from the start.

9. Test observability building blocks at a glance

The table below summarizes the building blocks for test observability presented.

10. Summary

11. FAQ: Test Observability: The Essentials at a Glance