Observability in Container Stacks: Metrics, Logs, and Health

1. Why Observability Works Differently in Container Stacks

Observability in container stacks is more complex than in classic VM setups because containers are ephemeral. A container can be restarted, and with it all local logs disappear. A service might be scaled to three replicas, meaning you need to look at logs from all three instances in aggregate. A container's IP address changes with every restart. These characteristics make naive monitoring approaches, such as file path based log monitoring or hardcoded IP addresses in monitoring configurations, unusable.

True observability in container stacks requires three things: logs are emitted in structured form (JSON) and sent to a central aggregation system instead of being written to files. Metrics are exposed via HTTP endpoints in Prometheus format and discovered automatically through service discovery. Health checks verify the actual functionality of a service, not just whether the process is running. Anyone who implements these three pillars turns an opaque container stack into a system that communicates its own state.

2. The Three Pillars: Metrics, Logs, and Traces

The three pillars of observability, metrics, logs, and traces, cover different aspects and complement each other. Metrics are aggregated numerical values over time: requests per second, error rate, P99 latency. They are compact, efficient, and well suited for alerting and dashboards. Logs are event based and contain the full context of a single operation: which request, which error, which stack trace. Traces connect multiple services into a single request flow and reveal which service in a chain consumes the most time.

In a container stack, the three pillars work together: a metric raises the alarm (observability at the system level), the logs provide the concrete error context (observability at the event level), and the trace shows which upstream service caused the problem (observability at the request level). Anyone who implements only one of the three pillars has blind spots. Anyone who has all three can trace any error in a container stack back to its origin within minutes, regardless of how many services are involved.

# docker-compose.yaml: Observability stack (Prometheus + Grafana + Loki)
# This is the foundation for full observability in a container stack

services:
  prometheus:
    image: prom/prometheus:v2.51.0
    volumes:
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=15d'
      - '--web.enable-lifecycle'  # Allow config reload via HTTP POST /-/reload
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:10.4.0
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
      - ./config/grafana/dashboards:/etc/grafana/provisioning/dashboards:ro
      - ./config/grafana/datasources:/etc/grafana/provisioning/datasources:ro
    depends_on:
      prometheus:
        condition: service_healthy

  loki:
    image: grafana/loki:2.9.5
    volumes:
      - ./config/loki.yaml:/etc/loki/loki.yaml:ro
      - loki_data:/loki
    command: -config.file=/etc/loki/loki.yaml

volumes:
  prometheus_data:
  grafana_data:
  loki_data:

3. Health Checks: More Than a Simple Ping

Docker health checks are the mechanism by which a container communicates its own state. A simple health check using CMD curl -f http://localhost/health verifies that the HTTP server responds, but not that it returns correct responses. A health check that is truly fit for observability verifies that the database connection is active, that the cache is reachable, and that the queue is still processing. Only then can Docker Compose correctly resolve the depends_on: condition: service_healthy dependency.

The parameters --interval, --timeout, --start-period, and --retries define the timing of the health check. --start-period gives the container startup time before failures count, which is important for applications that run a database migration on startup. Running docker inspect --format '{{json .State.Health}}' CONTAINER shows the full health status, including the last log output of the health check command. That is the first place to look when a container switches to the unhealthy status without leaving any obvious error messages in the regular logs.

4. Structured Logs in Container Environments

Structured logs in JSON format are the foundation for effective observability in container stacks. Instead of human readable lines like [2026-05-09 10:00:00] ERROR: Database connection failed, a structured log entry contains machine readable fields: timestamp, level, message, service, request_id, duration_ms. These fields can be filtered, aggregated, and used for alerting rules directly in Loki, Elasticsearch, or CloudWatch Logs, without regex parsing on fragile text formats.

In Docker containers there are two paths for logs: stdout/stderr (which Docker collects through the configured log driver) and files in a volume (which require a sidecar or separate aggregation). For observability in container stacks, stdout is the recommended path: docker logs shows them immediately, and a log driver like fluentd or loki forwards them without modifying the container. With a Promtail sidecar container in Docker Compose, all container logs can be shipped to Loki automatically without adjusting a single application.

# Health check for a PHP-FPM web application: checks HTTP + DB + cache
# Dockerfile excerpt: comprehensive health check
HEALTHCHECK \
  --interval=30s \
  --timeout=10s \
  --start-period=60s \
  --retries=3 \
  CMD php -r "
    // Check HTTP endpoint
    \$ch = curl_init('http://localhost/health');
    curl_setopt(\$ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt(\$ch, CURLOPT_TIMEOUT, 5);
    \$body = curl_exec(\$ch);
    \$code = curl_getinfo(\$ch, CURLINFO_HTTP_CODE);
    if (\$code !== 200) exit(1);

    // Check JSON response has expected keys
    \$data = json_decode(\$body, true);
    if (!\$data['db'] || !\$data['cache']) exit(1);

    exit(0);
  "

# Check health status with last output (useful when container is unhealthy)
docker inspect \
  --format '{{json .State.Health}}' \
  myapp_container | jq '{status: .Status, last: .Log[-1]}'

# Promtail config to ship all container logs to Loki
# config/promtail.yaml
scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 15s
    pipeline_stages:
      - json:
          expressions:
            level: level
            message: message
      - labels:
          level:

5. Exporting Prometheus Metrics from Docker Containers

Prometheus collects metrics using a pull model: each service exposes an HTTP endpoint (/metrics) that Prometheus scrapes at regular intervals. For observability in a Docker stack, this means that every service needs its own metrics endpoint, and Prometheus discovers the running containers through service discovery. With the Docker provider in Prometheus (docker_sd_configs), new containers are detected automatically as soon as they start, with no more manual editing of the Prometheus configuration.

For the container infrastructure itself there is cAdvisor (Container Advisor), which exposes CPU, memory, network, and I/O of all running containers as Prometheus metrics. Node Exporter adds host level metrics: disk usage, CPU utilization, network interfaces. At the application level, four Prometheus metrics are central to observability: request rate (how many requests per second), error rate (how many of them fail), duration histogram (P50, P95, P99 latency), and saturation (how full the queues are). These four metrics, known as the RED method, are enough to fully assess the state of a service.

6. Configuring Health Checks in Docker Compose

Docker Compose enables detailed health check configurations that go far beyond the Dockerfile. The decisive difference for observability: depends_on: condition: service_healthy only starts a service once the dependent service passes its health check. This prevents race conditions during stack startup, where a service starts before the database is ready. This dependency chain with health checks makes the stack's startup state observable and reproducible.

For persistent services like PostgreSQL or Redis, there are standardized health check commands: pg_isready -U postgres for PostgreSQL, redis-cli ping for Redis. These commands check not only whether the process is running, but whether the service actually accepts connections. Combined with Prometheus alerting (observability at the metric level) and structured logs, this creates three layer monitoring: Docker reports the service as unhealthy, Prometheus shows the error rate rising, and the structured logs provide the concrete connection error with a stack trace.

# compose.yaml: complete health check configuration for all services
services:
  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres -d ${DB_NAME}"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s  # Give Postgres time to initialize

  redis:
    image: redis:7-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 3s
      retries: 3

  app:
    image: myapp:latest
    depends_on:
      db:
        condition: service_healthy  # Wait for db health check to pass
      redis:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s  # PHP-FPM needs time to warm up

  # cAdvisor exports Docker container metrics to Prometheus
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.49.1
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    ports:
      - "8080:8080"

7. Distributed Tracing with OpenTelemetry

Distributed tracing is the third pillar of observability and is essential in microservices architectures, where a request passes through multiple services. OpenTelemetry has established itself as the standard: it is vendor neutral, supports all common languages, and can send traces to Jaeger, Tempo, Zipkin, or commercial APM systems. In a Docker stack, the OpenTelemetry Collector is deployed as its own container that receives, processes, and forwards traces from all services.

Integrating tracing into an application requires minimal effort thanks to auto instrumentation in many languages. For PHP there are OpenTelemetry extensions that automatically trace HTTP requests, database queries, and cache operations without manually creating spans. For everyday development observability, Jaeger All in One is the fastest tool: a single Docker container that stores all traces and provides a web UI. In production, Grafana Tempo with Grafana as the frontend is preferred, since it can be correlated directly with Prometheus metrics and Loki logs.

8. Alerting: From Metric to Notification

Alerting closes the observability loop: metrics and logs only become actionable once critical states are reported automatically. In a Prometheus based stack, you define alerting rules in YAML that connect Prometheus queries with thresholds. Alertmanager receives the alerts and routes them to Slack, PagerDuty, email, or webhooks depending on severity, time of day, and label routing. With for: 5m, an alert only fires once the condition has persisted for 5 minutes, which significantly reduces false alarms caused by brief spikes.

Three alerting rules are essential in every Docker stack for real observability: an alert when a container switches to the unhealthy status (health check failure), an alert when a service's error rate rises above 5% (error rate), and an alert when disk space on the host falls below 20% (disk full). These three rules cover the most common production problems. The health check failure alert adds active notification to passive Docker monitoring, so nobody has to manually run docker ps.

9. Comparing Observability Strategies

There are approaches of varying effort for observability in container stacks, ranging from minimal to complete. The right choice depends on the size of the stack and the operational requirements.

The minimal variant comes for free: Docker's built in health checks and docker logs provide a basic level of observability without additional infrastructure. However, the logs are ephemeral (lost on container restart), and there is no alerting. The standard variant with Prometheus, Loki, and Grafana is the recommended starting point for any production container stack: the entire configuration consists of a compose.yaml and a handful of config files, and can be set up in a single day.

10. Summary

Observability in container stacks is not an optional luxury but an operational requirement for production Docker deployments. Without structured logs, error details are lost on container restart. Without health checks, Docker Compose reports a service as running even though it has no database connection. Without metrics, performance problems only become visible once users complain. The three pillars, metrics (Prometheus + cAdvisor), logs (JSON + Loki), and health checks (deep inspection), build on each other and together cover every aspect of a container stack's state.

The implementation does not have to be complex. A Prometheus + Grafana + Loki + cAdvisor stack in Docker Compose can be set up in a single day and requires no changes to existing applications. Health checks in the Dockerfile and in compose.yaml cost only a few lines. Structured logs are a configuration matter in most modern frameworks. The effort is small, but the benefit for observability is enormous: problems are diagnosed in minutes instead of being hunted for hours.

11. FAQ: Observability in Container Stacks