Observability for REST APIs: Logging, Tracing and Metrics

1. What observability means for REST APIs

The term observability comes from control theory and describes the ability to understand the internal state of a system purely from its outputs. For REST APIs this means concretely: you should be able to tell, without manual debugging, why a request failed, where the latency originated, which downstream services were affected, and whether it is an isolated incident or a pattern. The three pillars, logs, traces and metrics, complement one another: metrics show that something is wrong, traces show where the problem lies, and logs explain why.

In practice you often see API setups where logs are written as plaintext to files, troubleshooting happens via SSH into the server and grep through log files, and the term "tracing" is treated as optional, something to implement eventually. This approach works for small teams and low traffic volumes, but breaks down as soon as multiple services, multiple instances or complex failure patterns appear. Investing in a complete observability stack pays for itself the first time a hard to reproduce production incident would otherwise have meant hours instead of minutes of debugging without traces and structured logs.

Another frequently overlooked aspect: observability is not an infrastructure task, it is an API design task. The instrumentation has to be built into the API code itself: meaningful span names, relevant attributes, clear log fields. Anyone who bolts observability on afterward as a layer gets data, but rarely the data that actually helps during an incident. The following guide shows how to instrument REST APIs for full observability from the start.

2. Structured logging: JSON instead of plaintext

Structured logging is the foundation of every observability stack. Instead of human readable plaintext lines, logs are written as machine readable JSON objects that can be indexed and queried directly. Every log entry has a defined schema: timestamp, log level, message, service name, request ID, user ID where available, HTTP method, path, status code and response duration. With this schema a log aggregator such as Loki, Elasticsearch or Datadog can index the fields directly, so queries like "all failed POST requests to /orders in the last 5 minutes" are answered in milliseconds, not via full text search through gigabytes of text.

In Symfony, structured logging is configured via Monolog. The JsonFormatter turns every log entry into JSON. The critical concept is processors: each processor automatically adds fields to every log entry without the application code having to explicitly supply them. A RequestIdProcessor adds the current request ID, a UserProcessor adds the currently authenticated user ID. This way every log entry, regardless of where in the code it was created, automatically carries the full context of the current request.

# config/packages/monolog.yaml: structured JSON logging for REST API
monolog:
  handlers:
    app:
      type: stream
      path: php://stdout
      level: info
      formatter: monolog.formatter.json
      channels: ['!event']
      processors:
        - Monolog\Processor\UidProcessor
        - App\Log\RequestIdProcessor
        - App\Log\UserContextProcessor

    error:
      type: fingers_crossed
      action_level: error
      handler: error_stream
      channels: ['!event']

    error_stream:
      type: stream
      path: php://stderr
      level: debug
      formatter: monolog.formatter.json

services:
  monolog.formatter.json:
    class: Monolog\Formatter\JsonFormatter
    calls:
      - [includeStacktraces, [true]]

The logging schema should be consistent across all endpoints from the start. An API request log contains at minimum: request_id, method, path, status_code, duration_ms, ip and user_agent. An error log additionally contains exception_class, exception_message and, only in non production environments, stack_trace. Sensitive data such as passwords, API keys or full request bodies should never end up in logs. A log sanitizer processor that strips known sensitive fields from the context is mandatory in every production system.

3. Correlation IDs and request context

A correlation ID (also called a trace ID or request ID) is a unique identifier assigned to a single incoming request and propagated through all involved services and log entries. With a correlation ID you can instantly filter out all log entries belonging to a single request from millions of entries, even if the request passed through five different microservices. That is the crucial difference between a logging system and an observability system.

The implementation follows a clear pattern: when a request comes in, a middleware layer checks whether an X-Request-ID header is present. If so, that value is adopted (which enables client side correlation). If not, a new UUID is generated. The ID is stored in a request scoped container, retrieved by all log processors, and forwarded on all outgoing HTTP requests via header. The response includes the header too, so frontend teams or external consumers can supply their request ID with support inquiries.

# App\EventListener\RequestIdListener: assigns and propagates Correlation IDs

namespace App\EventListener;

use Symfony\Component\EventDispatcher\Attribute\AsEventListener;
use Symfony\Component\HttpKernel\Event\RequestEvent;
use Symfony\Component\HttpKernel\Event\ResponseEvent;

#[AsEventListener(event: RequestEvent::class, priority: 200)]
#[AsEventListener(event: ResponseEvent::class, priority: -200)]
final class RequestIdListener
{
    private string $requestId = '';

    public function __construct(
        private readonly RequestIdStorage $storage
    ) {}

    public function onRequestEvent(RequestEvent $event): void
    {
        $request = $event->getRequest();
        $this->requestId = $request->headers->get('X-Request-ID')
            ?? $this->generateId();
        $this->storage->set($this->requestId);
        $request->headers->set('X-Request-ID', $this->requestId);
    }

    public function onResponseEvent(ResponseEvent $event): void
    {
        $event->getResponse()->headers->set(
            'X-Request-ID',
            $this->requestId
        );
    }

    private function generateId(): string
    {
        return sprintf('%s-%s', date('Ymd'), bin2hex(random_bytes(8)));
    }
}

4. Distributed tracing with OpenTelemetry

Distributed tracing goes beyond correlation IDs: instead of merely linking log lines together, the entire execution tree of a request is recorded as a hierarchical tree structure, a trace. Every trace consists of spans: a root span for the incoming HTTP request and child spans for every database query, outgoing HTTP call, cache lookup or message queue publish. In a trace viewer such as Jaeger or Zipkin you immediately see which span consumed how much time and where in the execution tree an error occurred.

OpenTelemetry is the open standard that unifies tracing, logging and metrics under one roof and prevents vendor lock in. The PHP library open-telemetry/opentelemetry-php offers auto instrumentation for common frameworks. For Symfony this means: HTTP requests, Doctrine queries and outgoing HTTP calls are instrumented automatically, without every single function having to be annotated manually. You add custom spans for critical business logic with a few lines of code.

# docker-compose.yml: OpenTelemetry Collector + Jaeger for local development
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    volumes:
      - ./otel-collector-config.yaml:/etc/otel/config.yaml
    command: ["--config=/etc/otel/config.yaml"]
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
    depends_on:
      - jaeger

  jaeger:
    image: jaegertracing/all-in-one:latest
    environment:
      COLLECTOR_OTLP_ENABLED: "true"
    ports:
      - "16686:16686"  # Jaeger UI
      - "14250:14250"  # gRPC collector

# otel-collector-config.yaml
# receivers:
#   otlp:
#     protocols:
#       grpc:
#       http:
# exporters:
#   jaeger:
#     endpoint: jaeger:14250
#     tls:
#       insecure: true
#   prometheus:
#     endpoint: "0.0.0.0:8889"
# service:
#   pipelines:
#     traces:
#       receivers: [otlp]
#       exporters: [jaeger]
#     metrics:
#       receivers: [otlp]
#       exporters: [prometheus]

Adding custom spans for business operations is straightforward in OpenTelemetry and pays off during troubleshooting. A span for the entire order checkout process, with attributes such as order.total, order.item_count and user.id, makes it possible to see instantly in the trace viewer which order was affected and whether it is a recurring pattern for certain order sizes. Span attributes are the structured fields of the tracing system, they should follow the same standards as the structured fields in logging: consistent, documented and never containing sensitive data.

5. Metrics with Prometheus and Grafana

Metrics are numeric time series data that represent a system's state over time. While logs describe individual events and traces describe individual requests, metrics show trends and aggregates: how many requests per second does the service process? What share returns 5xx errors? What is the latency distribution across all requests over the last 24 hours? These questions can technically be answered with logs and traces, but not efficiently, metrics are the right channel for this.

For REST APIs, four metric types are especially relevant: counters for totals (total requests, total errors), gauges for current states (active connections, queue length), histograms for latency distributions (response time buckets), and summaries for precomputed quantiles. The RED method framework recommends tracking three metrics for every service: rate (requests per second), errors (error rate) and duration (latency distribution). These three figures are enough to initially classify 90% of all production incidents.

# Prometheus metrics endpoint: custom REST API metrics in PHP/Symfony

# Install: composer require promphp/prometheus_client_php

namespace App\Metrics;

use Prometheus\CollectorRegistry;
use Prometheus\Counter;
use Prometheus\Histogram;
use Symfony\Component\HttpKernel\Event\TerminateEvent;

final class ApiMetricsCollector
{
    private Counter $requestsTotal;
    private Counter $errorsTotal;
    private Histogram $requestDuration;

    public function __construct(
        private readonly CollectorRegistry $registry
    ) {
        $this->requestsTotal = $registry->getOrRegisterCounter(
            'api', 'requests_total',
            'Total number of API requests',
            ['method', 'endpoint', 'status_code']
        );

        $this->errorsTotal = $registry->getOrRegisterCounter(
            'api', 'errors_total',
            'Total number of API errors',
            ['method', 'endpoint', 'error_type']
        );

        // Latency buckets optimized for REST APIs (ms)
        $this->requestDuration = $registry->getOrRegisterHistogram(
            'api', 'request_duration_seconds',
            'API request duration in seconds',
            ['method', 'endpoint'],
            [0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
        );
    }

    public function recordRequest(
        string $method,
        string $endpoint,
        int $statusCode,
        float $durationSeconds
    ): void {
        $this->requestsTotal->inc([$method, $endpoint, (string)$statusCode]);
        $this->requestDuration->observe($durationSeconds, [$method, $endpoint]);

        if ($statusCode >= 500) {
            $this->errorsTotal->inc([$method, $endpoint, 'server_error']);
        } elseif ($statusCode >= 400) {
            $this->errorsTotal->inc([$method, $endpoint, 'client_error']);
        }
    }
}

6. Alerting and SLO definitions

Service Level Objectives (SLOs) are the bridge between observability data and business requirements. An SLO defines what quality a service has to deliver, for example "99.9% of all requests must be answered within 500ms" or "the error rate must not exceed 0.1%". Alerting rules in Prometheus trigger a notification when the SLO budget is at risk of being exhausted, not only once the service has fully failed. This concept is called error budget: a 99.9% SLO allows 43.8 minutes of downtime per month. Alerting kicks in when the budget is being consumed too quickly.

Alerting rules in Prometheus are YAML files based on PromQL expressions. A good alerting strategy for REST APIs distinguishes three levels: symptom based alerts (high error rate, high latency, what users directly feel), saturation alerts (database connections exhausted, memory critical), and cause based alerts only sparingly, because they generate too much noise. The most common mistake with alerting: too many alerts, thresholds set too low, alert fatigue. The rule: every alert must have a clear action associated with it and must be actionable by a human.

7. Typical failure scenarios and how to find them

Observability has no value if the team does not know how to use the tools during an actual troubleshooting session. The typical incident flow: the Prometheus dashboard shows an elevated error rate, a Grafana alert fires, the on call engineer opens Grafana, sees the affected endpoint and time window, searches Loki for logs with that endpoint and status code 500, finds the correlation ID of the first failed request, opens Jaeger with that trace ID, sees in the trace tree that a database query suddenly takes 8 seconds instead of 50ms, checks the database metrics, and finds a missing index after a recent schema migration. With full observability this process takes 5 to 10 minutes instead of hours.

# Prometheus alert rules: REST API SLO-based alerting
# prometheus/alerts/api.yml

groups:
  - name: api_slo
    rules:
      # Alert when error rate exceeds 1% over 5 minutes
      - alert: ApiHighErrorRate
        expr: |
          (
            sum(rate(api_errors_total[5m]))
            /
            sum(rate(api_requests_total[5m]))
          ) > 0.01
        for: 2m
        labels:
          severity: warning
          team: backend
        annotations:
          summary: "API error rate above 1%"
          description: >
            Error rate is {{ printf "%.2f" $value | humanizePercentage }}
            over the last 5 minutes. SLO budget at risk.
          runbook_url: "https://wiki.mironsoft.de/runbooks/api-high-error-rate"

      # Alert when p95 latency exceeds 500ms
      - alert: ApiHighLatency
        expr: |
          histogram_quantile(0.95,
            sum(rate(api_request_duration_seconds_bucket[5m])) by (le, endpoint)
          ) > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "API p95 latency above 500ms"
          description: "Endpoint {{ $labels.endpoint }} p95 = {{ $value | humanizeDuration }}"

8. Observability tools compared

The market for observability tools is large and confusing. The choice between a fully managed SaaS stack and a self hosted open source stack has a direct impact on cost, data protection and operational effort. The following table gives a structured overview of the most common tool combinations for REST API observability.

9. Summary

Observability for REST APIs is not an add-on you attach after launch. Structured logging with a consistent schema, correlation IDs propagated through all services, distributed tracing with OpenTelemetry, and Prometheus metrics following the RED method framework, these four components form a stack that reduces the average time to root cause during an incident from hours to minutes. The crucial point: instrumentation has to be anchored in the code, not bolted on as an external layer. Meaningful span names, relevant attributes and a clear log schema are architecture decisions, not operations tasks.

The recommended stack for new projects: Symfony with Monolog and JsonFormatter for logs, the OpenTelemetry PHP SDK for traces with Jaeger or Grafana Tempo as the backend, Prometheus with the php-prometheus-client for metrics, and Grafana as the unified dashboard frontend. All three data sources can be correlated in Grafana, a single click on a latency spike in the metric opens the associated logs and traces in the same time window.

10. FAQ: Observability for REST APIs