Distributed Tracing for Containerized Applications with OpenTelemetry

1. Why metrics and logs do not replace distributed tracing

Metrics show that something is slow, logs show what happened inside a single container, but neither reliably shows where along a request through multiple containers the time is actually lost. Exactly this gap is closed by distributed tracing: instead of isolated data points per container, a connected chain emerges that makes a single request traceable from its entry at the reverse proxy to the last database query.

In a typical containerized architecture with a web container, an API container, a cache and a database, a slow request without distributed tracing is a guessing game: is the latency caused by the database, a cache miss, an external API call, or the network connection between containers? OpenTelemetry, a vendor neutral standard for instrumentation, answers exactly this question by capturing every step of the request as a span with timestamp and duration and merging them across all involved containers into a single trace. The following sections walk through the complete path from instrumentation to a finished trace view in Jaeger.

2. Basic concepts: trace, span and context

A trace represents the entire path of a single request through a system, from first contact to the response to the client. A span is a single named, time bounded unit of work within this trace, for example an HTTP request handler, a database query or a call to another service. Every span carries a unique span ID, a shared trace ID for all spans of the same trace, and optionally a parent span ID that maps the hierarchical relationship between spans.

The trace context is the mechanism that passes trace ID, span ID and further metadata from one container to the next, usually via the standardized traceparent HTTP header per the W3C Trace Context specification. Without consistent propagation of this context, isolated traces per container arise instead of a coherent picture across the entire request. Exactly this context propagation between containers is the technically most demanding part of building distributed tracing in containerized environments.

3. Instrumenting applications with OpenTelemetry

OpenTelemetry offers automatic instrumentation for most common languages, which attaches spans to HTTP frameworks, database clients and messaging libraries without manual code changes. For PHP applications, for example, installing the OpenTelemetry extension and the auto instrumentation package is enough to automatically capture incoming HTTP requests, outgoing cURL calls and PDO database queries, without adjusting every single line of code manually.

For business logic that goes beyond automatic instrumentation, for example a critical calculation step or a cart validation, manual spans are added via the OpenTelemetry API. This combination of automatic and manual instrumentation delivers the most complete picture: infrastructure calls are captured without extra effort, while business critical steps are given meaningful names and attributes deliberately.

# docker-compose.yml — application container with OpenTelemetry auto-instrumentation
services:
  checkout-api:
    build: ./checkout-api
    environment:
      OTEL_SERVICE_NAME: "checkout-api"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4318"
      OTEL_TRACES_EXPORTER: "otlp"
      OTEL_PHP_AUTOLOAD_ENABLED: "true"
    depends_on:
      - otel-collector

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.104.0
    volumes:
      - ./otel-collector-config.yml:/etc/otelcol-contrib/config.yaml:ro
    ports:
      - "4318:4318"   # OTLP HTTP receiver
      - "4317:4317"   # OTLP gRPC receiver

4. Propagating trace context between containers

Automatic instrumentation captures a span for every incoming and outgoing HTTP request of a container, but the connection between these spans across container boundaries only works if the traceparent header is consistently passed from one container to the next. For HTTP based communication, most OpenTelemetry instrumentations handle this propagation automatically, as long as both sides use the same instrumentation library and the same propagation format.

It becomes critical with asynchronous communication via message queues like RabbitMQ or Redis Streams, where no HTTP header is automatically carried along. Here, the trace context must be manually embedded into the message, usually as an additional metadata field, and explicitly extracted again on receipt to correctly continue the span tree. If this step is forgotten, two separate traces arise instead of one continuous picture, which undermines the actual strength of distributed tracing.

<?php
// Manually propagating trace context through a message queue payload
use OpenTelemetry\API\Trace\Propagation\TraceContextPropagator;
use OpenTelemetry\Context\Context;

// Producer side: inject the current trace context into the message
$carrier = [];
TraceContextPropagator::getInstance()->inject($carrier, null, Context::getCurrent());

$message = [
    'payload' => $orderData,
    'trace_context' => $carrier, // e.g. ["traceparent" => "00-abc123...-def456...-01"]
];
$queue->publish(json_encode($message));

// Consumer side: extract the context and continue the same trace
$decoded = json_decode($rawMessage, true);
$extractedContext = TraceContextPropagator::getInstance()->extract(
    $decoded['trace_context'],
);
$scope = $extractedContext->activate();
// ... process the order within the same trace, then detach the scope
$scope->detach();

5. Running the OpenTelemetry Collector in Docker

The OpenTelemetry Collector is an independent component that mediates between instrumented applications and the backend, for example Jaeger. Instead of every application sending traces directly to Jaeger, all containers send their spans to a central collector, which receives, batches, optionally filters and forwards them to one or more backends. This decoupling makes it possible to swap out the tracing backend without having to reconfigure every single application.

In a Docker environment, the collector typically runs as its own container, reachable via the internal compose network name, to which all other containers send their traces via OTLP, the OpenTelemetry Protocol. The collector configuration defines receivers for incoming data, optional processors for batching and filtering, and exporters for forwarding to Jaeger, Prometheus or other backends, all through a single YAML file.

# otel-collector-config.yml — receive OTLP, batch, export to Jaeger
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  otlp/jaeger:
    endpoint: "jaeger:4317"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger]

6. Visualizing and analyzing traces in Jaeger

Jaeger displays every received trace as a waterfall diagram, in which every span appears as a horizontal bar, positioned by start time and duration, nested according to the parent child relationship between spans. This representation shows at a glance which segment of a request causes the largest share of the total latency, without having to manually correlate individual log lines.

Jaeger's search function allows filtering by service name, operation, duration and custom span attributes, for example an order number or customer ID, provided these were set as attributes on the span. For troubleshooting a specific customer incident, the exact trace belonging to a particular order can thus be found precisely, instead of manually searching logs of multiple containers for an order number. This direct link between business context and technical trace is one of the biggest practical advantages of distributed tracing over pure metrics and log based monitoring.

7. Sampling strategies for production environments

Tracing every request completely generates considerable data volume and corresponding storage needs in the tracing backend in high load environments. Sampling reduces this volume by fully recording only a portion of the traces. Head based sampling decides already at the start of a request, for example with a fixed probability of 10 percent, whether the entire trace is recorded, which is simple to implement but risks missing rare but important error cases.

Tail based sampling instead makes the decision only after the complete trace is available, and can thus deliberately keep all traces with errors or unusually high latency in full, while normal, fast requests are only stored at a small percentage. This strategy requires a collector that buffers the entire trace before the sampling decision is made, but for production environments with high traffic it is the significantly more meaningful choice, because exactly the traces most valuable for troubleshooting are retained.

8. Common mistakes when introducing tracing

The most common mistake is incomplete instrumentation, where individual containers, often older or considered non critical services, are left out. The result is traces with gaps, where the request briefly disappears before reappearing in an instrumented container, which obscures the actual cause of latency rather than revealing it.

# Common mistakes when introducing distributed tracing

# WRONG: trace context not propagated through async messaging
# $queue->publish(json_encode($orderData));  # no trace_context field

# RIGHT: inject and extract trace context around the message boundary
# see PHP example in section 4 — inject on publish, extract on consume

# WRONG: 100% sampling in a high-traffic production service
# OTEL_TRACES_SAMPLER=always_on   # huge data volume, high backend cost

# RIGHT: tail-based sampling that always keeps errors and slow requests
# OTEL_TRACES_SAMPLER=parentbased_traceidratio
# OTEL_TRACES_SAMPLER_ARG=0.1     # 10% baseline, errors kept separately

# WRONG: missing service.name — all containers show up as "unknown_service"
# (no OTEL_SERVICE_NAME set)

# RIGHT: explicit, unique service name per container
# OTEL_SERVICE_NAME=checkout-api

A second widespread mistake is a missing or generic service.name, causing all containers to appear under the same name in Jaeger and losing the actual strength of distributed tracing, the clear attribution of latency to a specific service. A third mistake is overly aggressive sampling without special handling for errors, discarding exactly the traces that would be most important for troubleshooting, while unremarkable, fast requests are retained disproportionately often.

9. Observability pillars in direct comparison

Distributed tracing is one of three pillars of observability and complements metrics and logs, rather than replacing them.

In practice, these three pillars complement each other: an Alertmanager alert based on metrics signals increased latency, logs provide the error text of the affected container, and distributed tracing shows which step in the chain actually consumed the time. None of these three pillars fully replaces the others, only their combination delivers the complete picture for troubleshooting in containerized microservice architectures.

10. Summary

Distributed tracing for containerized applications with OpenTelemetry closes the gap that metrics and logs leave open for requests spanning multiple containers. Spans with a shared trace ID map the path of a request, the trace context is propagated via HTTP headers or manually across message queue boundaries, the OpenTelemetry Collector gathers and exports the data, and Jaeger visualizes it as a searchable waterfall diagram.

The decisive success factor is complete, consistent instrumentation across all involved containers, combined with a well thought out sampling strategy that preferentially retains errors and slow requests. Anyone who implements these fundamentals gets a tool with distributed tracing that localizes latency problems in complex, containerized architectures in minutes instead of hours.

For teams already running Prometheus and Loki, distributed tracing fits seamlessly as a third pillar into the same Grafana interface, so metrics, logs and traces can be evaluated together without switching tools.

11. FAQ: Distributed Tracing for Containerized Applications with OpenTelemetry