Collecting Container Metrics with Prometheus and cAdvisor

1. Why docker stats is not enough for container metrics

The docker stats command displays CPU, memory, network and block I/O in real time, but it stores nothing. As soon as the terminal is closed, the values are gone. For troubleshooting a memory leak that happened overnight, or for capacity planning across several weeks, container metrics without history are worthless. That is exactly where the combination of cAdvisor and Prometheus comes in: cAdvisor reads the same kernel counters as docker stats, but exposes them permanently as an HTTP endpoint, and Prometheus writes every measurement with a timestamp into a time series database.

The second weakness of docker stats is the lack of aggregation. Anyone who wants to know how the sum of all PHP FPM containers has developed over a week, or which container caused the most network traffic in the last 24 hours, needs a query language. PromQL provides exactly that: functions like rate(), sum by() and quantile_over_time() turn raw container metrics into figures that capacity decisions and alerts can be built on. The following sections walk through the complete path from a first cAdvisor instance to a finished dashboard.

2. Setting up cAdvisor as a metrics source

cAdvisor, short for Container Advisor, is a daemon developed by Google that runs as its own container on every Docker host and reads the kernel cgroups. Every second, cAdvisor collects CPU time, memory usage, network bytes and block I/O for every running container and exposes the values in Prometheus text format under /metrics. The big advantage over custom scripts: cAdvisor needs no agents inside the containers, it only reads at the host level, which keeps overhead and attack surface low.

For operation, a single container with read only access to /var/run, /sys and /var/lib/docker is enough. On multi host setups, cAdvisor runs as a DaemonSet equivalent, meaning one container per host, so local container metrics are also captured locally and no central single point of failure occurs. The following compose definition shows the minimal, production ready setup along with the necessary permissions.

# docker-compose.yml — cAdvisor as a metrics source per host
services:
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.49.1
    container_name: cadvisor
    restart: unless-stopped
    ports:
      - "127.0.0.1:8080:8080"   # bind to localhost, Prometheus scrapes internally
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    devices:
      - /dev/kmsg
    privileged: false
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:8080/healthz"]
      interval: 30s
      timeout: 5s
      retries: 3

# Quick verification: list all container-level metrics
# curl -s http://localhost:8080/metrics | grep container_cpu_usage_seconds_total

3. Configuring Prometheus as a scraper

Prometheus works as a pull based monitoring system: it periodically queries the configured targets instead of the targets pushing data themselves. For container metrics, that means entering cAdvisor as a scrape target in prometheus.yml. A scrape interval of 15 seconds is a good compromise between resolution and storage needs for most Docker environments, shorter intervals increase accuracy for brief load spikes but consume noticeably more storage.

Correct job naming matters, because it later serves as a filter in every PromQL query. In multi host setups, each host is listed individually under static_configs, or service discovery is used, for example via DNS or a file with file_sd_configs, so new hosts are detected automatically without manually adjusting the Prometheus configuration. The scrape timeout should stay smaller than the scrape interval so a hanging cAdvisor endpoint does not block the entire scrape cycle.

# prometheus.yml — scrape configuration for container metrics
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "cadvisor"
    scrape_interval: 15s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - "host1:8080"
          - "host2:8080"
        labels:
          environment: "production"

  - job_name: "node-exporter"
    static_configs:
      - targets:
          - "host1:9100"
          - "host2:9100"
        labels:
          environment: "production"

rule_files:
  - "alerts/*.yml"

4. PromQL: querying the most important container metrics

Raw counters like container_cpu_usage_seconds_total are cumulative and not very meaningful on their own. Only the rate() function, applied over a time window, produces a meaningful CPU usage in cores per second. For memory usage, instead of a counter, the gauge container_memory_working_set_bytes is queried directly, because memory represents an absolute value at any point in time and needs no rate. This distinction between counters and gauges is the foundation of every correct PromQL query for container metrics.

For daily practice, a handful of query patterns are enough: CPU usage per container via rate(...[5m]), memory usage relative to the configured limit via a division of two metrics, and network throughput via the derivative of container_network_transmit_bytes_total. The sum by (name) clause groups results by container name, so a dashboard does not show hundreds of individual time series but aggregated, readable curves per service.

# PromQL — common container metrics queries

# CPU usage per container, in cores, averaged over 5 minutes
sum by (name) (rate(container_cpu_usage_seconds_total{name!=""}[5m]))

# Memory usage relative to the configured limit, as a percentage
100 * (
  container_memory_working_set_bytes{name!=""}
  /
  container_spec_memory_limit_bytes{name!=""} > 0
)

# Network transmit rate in bytes per second, per container
sum by (name) (rate(container_network_transmit_bytes_total{name!=""}[5m]))

# Top 5 containers by CPU usage right now
topk(5, sum by (name) (rate(container_cpu_usage_seconds_total[1m])))

# Containers restarting more than twice in the last hour (needs cadvisor >= 0.47)
increase(container_last_seen{name!=""}[1h]) > 2

5. Labels and relabeling for clean attribution

cAdvisor provides many useful labels out of the box, including name, image and id, but this raw data often contains technical container IDs instead of speaking service names. With metric_relabel_configs in Prometheus, labels can be renamed, merged or filtered before storage, so dashboards and alerts are based on speaking names like service="magento-web" instead of a random container ID.

A common pattern is filtering out pause containers and infrastructure containers that are irrelevant for actual application observability. Without this filter, container metrics dashboards bloat with noise and the cardinality of the time series rises unnecessarily. A relabeling ruleset that consistently reads compose project labels such as com.docker.compose.service makes the mapping between metric and application immediately traceable, even for colleagues who did not build the infrastructure themselves.

6. Building dashboards in Grafana

Prometheus itself only offers a rudimentary interface for ad hoc queries, for durable dashboards Grafana is the obvious choice. After adding Prometheus as a data source, the PromQL queries from section 4 can be pasted directly into panels. A sensible baseline dashboard for container metrics shows at least four panels: CPU usage per container as a time series, memory relative to the limit as a bar chart, network throughput as a stacked area, and a table of the top 5 containers by resource usage.

Grafana variables like $container or $environment, fed from label_values() queries, make a single dashboard reusable across all containers, instead of maintaining a separate dashboard per service. Ready made community dashboards for cAdvisor, for instance ID 893 in the Grafana dashboard catalog, provide a good starting point but should be adapted to your own label names, because differing relabeling rules otherwise lead to empty panels.

7. Keeping storage, retention and cardinality under control

Every additional label combination creates a new time series in Prometheus, and every time series costs storage and query time. With container metrics for short lived containers, for example with frequent deployments and changing container IDs, cardinality can explode unnoticed if labels like id are used instead of stable service names. A look at prometheus_tsdb_symbol_table_size_bytes and the number of active series via count({__name__=~".+"}) shows early whether cardinality is getting out of hand.

The default retention of 15 days is enough for operational troubleshooting, for capacity trends across months it takes either a higher --storage.tsdb.retention.time, which increases local storage needs linearly, or a remote write into a long term storage system like Thanos or Mimir. For most medium sized Docker environments, a combination of 30 days of local retention plus weekly snapshots as a backup is a practical middle ground between effort and benefit.

8. Common mistakes when collecting metrics

The most common mistake is a scrape interval that is chosen too coarse combined with short rate() windows. A scrape interval of 60 seconds with rate(...[1m]) produces unstable, noisy values, because the window often contains only one or two data points. The rule of thumb: the rate() window should be at least four times the scrape interval, so enough data points are available for a stable derivative.

# Common mistakes when collecting container metrics

# WRONG: scrape interval too coarse combined with a narrow rate window
# scrape_interval: 60s
# rate(container_cpu_usage_seconds_total[1m])   # unstable, too few samples

# RIGHT: rate window at least 4x the scrape interval
# scrape_interval: 15s
# rate(container_cpu_usage_seconds_total[5m])   # smooth, reliable

# WRONG: querying memory as if it were a counter
# rate(container_memory_working_set_bytes[5m])  # meaningless for a gauge

# RIGHT: gauges are read directly, no rate needed
# container_memory_working_set_bytes{name="magento-web"}

# WRONG: relying on unstable container IDs as the grouping label
# sum by (id) (rate(container_cpu_usage_seconds_total[5m]))

# RIGHT: group by a stable label added via relabeling
# sum by (compose_service) (rate(container_cpu_usage_seconds_total[5m]))

A second common mistake is the lack of a limit as a reference value. Without container_spec_memory_limit_bytes as a denominator, the absolute memory usage says little about how close a container is to an OOM kill. A third mistake concerns network metrics: container_network_receive_bytes_total is tracked per network interface, so with multiple networks per container it must be aggregated across the interface label, otherwise the dashboard shows only a fraction of the actual traffic.

9. Metrics sources in direct comparison

Besides cAdvisor, there are other ways to obtain container metrics, each with different trade offs regarding overhead, granularity and integration effort.

In practice these sources do not exclude each other. cAdvisor delivers the host and container perspective, while in app metrics through their own Prometheus client libraries add application specific figures such as request latencies or queue lengths. Combining both levels produces the complete picture of container metrics in a production environment.

10. Summary

Container metrics with Prometheus and cAdvisor solve the core problem of docker stats: missing history and missing aggregation. cAdvisor reads CPU, memory, network and I/O directly from the kernel cgroups and exposes them in Prometheus format. Prometheus periodically scrapes these values and stores them as time series. PromQL turns them into meaningful figures, from CPU rate to memory usage relative to the limit to top lists of the most resource hungry containers.

The decisive lever for reliable dashboards lies in clean labels: stable service names instead of random container IDs, a scrape interval that matches the chosen rate() window, and a deliberate retention strategy that balances storage needs and history. Anyone who follows these fundamentals has a monitoring foundation with cAdvisor and Prometheus that can be seamlessly extended with alerting and distributed tracing.

11. FAQ: Container Metrics with Prometheus and cAdvisor