Setting Up Resource Alerting for Containers with Alertmanager

1. Why dashboards alone do not replace alerting

A Grafana dashboard with cleanly visualized container resources is valuable for troubleshooting, but useless if nobody is looking at it right now. Alerting closes exactly this gap: instead of waiting for a human to happen to glance at a chart, an automated system continuously evaluates metrics and actively reports as soon as a defined threshold is exceeded. For container environments, this concretely means memory leaks, CPU saturation or repeated restarts are detected before customers notice the outage.

Prometheus itself evaluates alert rules and generates alerts, but does not send them. That task falls to Alertmanager, an independent component that receives alerts, deduplicates, groups and forwards them via configurable routes to email, Slack, PagerDuty or other systems. This separation between detection and delivery is deliberate and allows defining complex routing behavior without complicating the Prometheus configuration itself. The following sections walk through the complete path from a first alert rule to an alerting setup that works reliably at night without being annoying.

2. Architecture: Prometheus, rules and Alertmanager

The flow in the alerting stack follows a clear pipeline. Prometheus periodically evaluates the alert rules defined in rule_files against the collected time series. If a condition holds true for the configured for duration, the alert transitions from pending to firing and is forwarded to Alertmanager. Alertmanager itself knows nothing about metrics, it exclusively processes the alerts sent by Prometheus.

This architecture scales well across multiple Prometheus instances: in larger environments, several Prometheus servers, for example per region or cluster, can address the same central Alertmanager. Alertmanager then takes care of deduplication if the same alert is reported redundantly from multiple sources, as well as grouping related alerts into a single notification. For Docker environments with cAdvisor as a metrics source, this setup is the standard path for reliable alerting.

# docker-compose.yml — Prometheus + Alertmanager stack
services:
  prometheus:
    image: prom/prometheus:v2.53.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./alerts:/etc/prometheus/alerts:ro
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--web.enable-lifecycle"
    ports:
      - "9090:9090"

  alertmanager:
    image: prom/alertmanager:v0.27.0
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    ports:
      - "9093:9093"
    command:
      - "--config.file=/etc/alertmanager/alertmanager.yml"

3. Writing alert rules for CPU, memory and restarts

A good alert rule for container resources consists of three parts: a PromQL condition, a for duration that filters out brief outliers, and meaningful labels and annotations for the later notification. For CPU saturation, a rule that checks a container's CPU rate over several minutes against a threshold works well. For memory, a rule relative to the configured limit is more appropriate, because an absolute memory value has completely different meaning depending on the container.

Restart alerts deserve special attention, because a single restart is often normal, for example after a deployment, while repeated restarts in a short time indicate a crash loop. The increase() function over a one hour window is well suited to represent exactly this difference between a one time and a repeated restart. The severity label in every rule later drives routing in Alertmanager and should consistently distinguish between warning and critical.

# alerts/container-resources.yml — Prometheus alerting rules
groups:
  - name: container-resources
    rules:
      - alert: ContainerHighCpuUsage
        expr: |
          sum by (name) (rate(container_cpu_usage_seconds_total{name!=""}[5m])) > 1.5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.name }} exceeds 1.5 CPU cores"
          description: "Sustained CPU usage above the expected threshold for 10 minutes."

      - alert: ContainerMemoryNearLimit
        expr: |
          (container_memory_working_set_bytes{name!=""}
            / container_spec_memory_limit_bytes{name!=""}) > 0.90
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Container {{ $labels.name }} is at 90 percent of its memory limit"
          description: "OOM kill is likely if usage keeps rising."

      - alert: ContainerRestartingFrequently
        expr: |
          increase(container_last_seen{name!=""}[1h]) > 3
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Container {{ $labels.name }} restarted more than 3 times in one hour"
          description: "Possible crash loop, check application logs immediately."

4. Configuring Alertmanager: routing and receivers

The Alertmanager configuration consists of a route tree that forwards incoming alerts to different receivers based on their labels, and a list of receivers with the technical details for email, Slack, PagerDuty or webhooks. The top level routing node defines default values for grouping and wait times, subordinate routes can override these based on label matches like severity: critical.

A proven practice is separation by severity: warning alerts land in a Slack channel the team monitors during the day, while critical alerts additionally alert an on-call duty via PagerDuty. This differentiation prevents every minor anomaly from triggering the same alert level as a real production outage, and forms the foundation of an alerting system the team trusts long term instead of ignoring out of frustration.

# alertmanager.yml — routing by severity
route:
  receiver: "slack-warnings"
  group_by: ["alertname", "compose_project"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = "critical"
      receiver: "pagerduty-oncall"
      repeat_interval: 1h

receivers:
  - name: "slack-warnings"
    slack_configs:
      - api_url: "https://hooks.slack.com/services/REPLACE/WITH/WEBHOOK"
        channel: "#container-alerts"
        send_resolved: true

  - name: "pagerduty-oncall"
    pagerduty_configs:
      - service_key: "REPLACE_WITH_PAGERDUTY_KEY"
        send_resolved: true

5. Using grouping, silencing and inhibition correctly

Without grouping, every affected container would trigger its own alert, so an outage of the underlying Docker host would lead to dozens of individual notifications. group_by bundles related alerts, for example all with the same alertname and compose_project, into a single message. This significantly reduces alert fatigue without losing relevant information, because the individually affected containers are still listed in the bundled message.

Silencing pauses notifications for a defined period, for example during planned maintenance, without disabling the underlying alert rule. Inhibition goes a step further and automatically suppresses downstream alerts when a parent alert is already active, for example a single container alert is suppressed as soon as a host outage alert for the same host is already firing. Together, these mechanisms prevent a single incident from ending in a flood of contradictory or redundant notifications.

6. Setting thresholds from data instead of guessing

The most common reason for ignored alerting is thresholds guessed without reference to actual load distribution. A CPU threshold of 80 percent sounds intuitively reasonable, but is completely unsuitable for a batch job that regularly runs briefly at 100 percent and produces constant false alarms. The better way: evaluate historical metrics from Prometheus via quantile_over_time() and set thresholds based on the 95th or 99th percentile of actual usage over the past weeks.

For new containers without historical data, a conservative starting value makes sense, which is then adjusted after an observation period of two to four weeks based on the collected metrics. This data driven approach to thresholds is the decisive difference between an alerting system the team trusts and one that gets muted after a short time because it constantly raises alarms for no reason.

7. Mapping escalation levels and on-call duty

Not every alert needs an immediate human reaction at three in the morning. A well thought out escalation strategy distinguishes between alerts that can be handled during business hours and those that must wake an on-call duty. Alertmanager supports this via time based routing rules combined with severity labels, so a low priority memory alert lands in the team chat during the day, while the same alert type with high priority is escalated immediately.

For teams with multi tier on-call duty, repeat_interval can be used to resend unacknowledged critical alerts at shorter intervals until someone responds. Mapping this escalation logic directly in Alertmanager, instead of coordinating it manually, ensures that critical container resource problems get attention even when the first notified person does not respond in time.

8. Common mistakes in container alerting

The most common mistake is a too short or missing for duration, causing every brief outlier to immediately trigger an alert instead of only reporting sustained problems. A second widespread mistake is missing grouping, which leads to a flood of individual notifications during a larger incident that obscures rather than clarifies the actual problem.

# Common mistakes in container resource alerting

# WRONG: no "for" duration — fires on every brief spike
# - alert: ContainerHighCpuUsage
#   expr: rate(container_cpu_usage_seconds_total[1m]) > 1.5
#   # missing: for: 10m

# RIGHT: sustained condition required before firing
# - alert: ContainerHighCpuUsage
#   expr: rate(container_cpu_usage_seconds_total[5m]) > 1.5
#   for: 10m

# WRONG: no group_by — one notification per affected container
# route:
#   receiver: "slack"
#   # missing: group_by

# RIGHT: related alerts bundled into a single notification
# route:
#   receiver: "slack"
#   group_by: ["alertname", "compose_project"]

# WRONG: identical severity for everything — no escalation possible
# labels:
#   severity: warning   # used for both minor and critical issues

# RIGHT: severity reflects actual urgency, drives routing
# labels:
#   severity: critical   # routes to on-call, shorter repeat_interval

A third mistake is missing send_resolved: true in the receiver configurations. Without this setting, the team never learns when an alert resolved itself, leading to unnecessary confusion about whether a reported problem still exists. A fourth mistake is applying static thresholds equally to all containers, even though database containers, worker containers and web containers have completely different normal load profiles and correspondingly need different thresholds.

9. Alerting strategies in direct comparison

There are several approaches to alerting in Docker environments, with different levels of maturity and effort.

Prometheus with Alertmanager remains the standard for Docker based environments, because it integrates seamlessly into an already existing metrics pipeline with cAdvisor and does not incur ongoing license costs. Managed platforms make sense when the team has no capacity to operate its own alerting infrastructure, but they merely shift the effort into ongoing cost instead of operational work.

10. Summary

Resource alerting for containers with Alertmanager closes the gap between passive dashboards and active notification. Prometheus evaluates alert rules for CPU, memory and restarts against the collected time series, Alertmanager handles grouping, routing and delivery to the right receivers. Silencing and inhibition prevent a single incident from ending in a flood of redundant notifications.

The decisive success factor is setting thresholds from data based on historical percentiles instead of guessed percentages, combined with consistent use of severity labels for escalation. Anyone who follows these fundamentals gets an alerting system the team trusts long term, because it reports exactly when action is actually needed and stays quiet when everything runs within normal bounds.

11. FAQ: Resource Alerting for Containers with Alertmanager