Building Centralized Container Logging with Loki and Promtail

1. Why centralized logging becomes essential with containers

As long as a single Docker host carries all containers of an application, docker logs is usually enough for troubleshooting. As soon as multiple hosts, multiple environments or a growing number of services come into play, this method becomes impractical. Centralized logging solves exactly this scaling problem: instead of connecting via SSH to every single host and searching logs manually, all output ends up in one central place, searchable through a single interface.

The second reason for centralized logging is traceability across container restarts. Docker rotates and discards logs by default according to the log driver settings, and a newly created container starts with an empty log. Without central collection, older errors are irrevocably lost after a deployment or restart. Loki, developed by Grafana Labs, was designed specifically for this use case: cost effective, horizontally scalable storage of logs with minimal indexing overhead.

2. Architecture: Promtail, Loki and Grafana working together

The Loki stack consists of three components with clearly separated responsibilities. Promtail runs as an agent on every Docker host, automatically discovers running containers and reads their log files or the Docker socket. Loki itself is the storage and query engine that accepts incoming logs, attaches labels and persists them. Grafana finally serves as the interface for queries and visualization, the same interface already used for Prometheus metrics.

The decisive architectural difference from classic log systems like Elasticsearch: Loki does not index the full log content, only the labels, such as container, compose_service or level. The actual text is stored compressed in chunks and only searched at query time. This approach significantly reduces storage needs and operating costs, but in return requires a well thought out label strategy so queries do not become too slow.

# docker-compose.yml — minimal Loki + Promtail + Grafana stack
services:
  loki:
    image: grafana/loki:2.9.6
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml:ro
      - loki-data:/loki
    command: -config.file=/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:2.9.6
    volumes:
      - ./promtail-config.yml:/etc/promtail/config.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    command: -config.file=/etc/promtail/config.yml
    depends_on:
      - loki

  grafana:
    image: grafana/grafana:10.4.2
    ports:
      - "3000:3000"
    depends_on:
      - loki

volumes:
  loki-data:

3. Setting up Promtail and capturing Docker logs

Promtail uses Docker service discovery to automatically detect all running containers, without maintaining a separate configuration for every service. Through docker_sd_configs, Promtail regularly queries the Docker socket and reads the associated log files under /var/lib/docker/containers. Relabeling rules, syntactically identical to Prometheus, extract speaking values like the compose service name or the project name from the Docker labels.

An important detail: by default, Docker logs JSON formatted lines with an embedded log field. The json pipeline stage in Promtail parses this structure, extracts the actual log text and separates stdout from stderr. Anyone writing structured application logs in JSON format, for example from PHP or Node.js, can nest a second json stage to extract fields like level or request_id directly as a Loki label.

# promtail-config.yml — Docker service discovery + relabeling
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ["__meta_docker_container_label_com_docker_compose_service"]
        target_label: "compose_service"
      - source_labels: ["__meta_docker_container_label_com_docker_compose_project"]
        target_label: "compose_project"
      - source_labels: ["__meta_docker_container_name"]
        regex: "/(.*)"
        target_label: "container"
    pipeline_stages:
      - docker: {}
      - json:
          expressions:
            level: level
            request_id: request_id
      - labels:
          level:

4. Configuring Loki: storage, retention and limits

Loki supports several storage backends, from local filesystem for small setups to S3 compatible object storage for production, scaling environments. For individual Docker hosts, the filesystem store is entirely sufficient, for multi host environments with several Loki instances, object storage is recommended, so all instances can access the same dataset. Retention is controlled via limits_config and compactor, a value of 30 days is a sensible starting point for most centralized logging setups.

Ingestion limits deserve special attention. By default, Loki limits the rate of incoming logs per stream and tenant, to protect a single misbehaving container from overloading the entire Loki instance with an endless loop of error messages. These limits should be deliberately adjusted to the expected log volume of your own container fleet, instead of raising them reflexively as soon as the first rate limit errors appear.

5. LogQL: filtering and searching logs precisely

LogQL, the query language of Loki, combines label selectors with optional text and parsing filters, similar to the syntax of PromQL. A typical query starts with a label selector like {compose_service="magento-web"}, which narrows the search to a specific service before the log text is even searched. Only after that come text filters like |= "error" or pattern filters using regular expressions.

For alerts and dashboards, LogQL also supports metric queries directly on logs, for example count_over_time(), to count the number of error lines per time window without needing a separate metrics pipeline. This capability makes Loki more than a pure log store: figures for Alertmanager or Grafana alerting can be derived directly from logs, without any additional instrumentation of the application.

# LogQL — common queries for centralized container logging

# All logs from one service in the last hour
{compose_service="magento-web"}

# Only error-level lines, across all containers of a project
{compose_project="shop"} | json | level="error"

# Text search combined with a label filter
{compose_service="magento-web"} |= "MySQL server has gone away"

# Rate of error lines per minute, usable as a Grafana panel or alert
sum(count_over_time({compose_project="shop"} |= "error" [1m]))

# Extract and filter on a JSON field without pre-defining it as a label
{compose_service="checkout-api"} | json | duration_ms > 2000

6. Label strategy: low cardinality, high value

The biggest conceptual difference between Loki and classic full text search systems lies in the deliberate restriction of labels. Every additional high cardinality label, for example a unique request ID or a timestamp used as a label instead of a field, creates a new stream in Loki and multiplies the indexing overhead. The recommendation is: labels only for values with a limited, stable value set, such as service name, environment or log level, while variable values like request IDs remain as searchable text or a JSON field within the log itself.

This deliberate separation between labels and content is the core of every successful centralized logging strategy with Loki. Anyone who instead tries to treat Loki like Elasticsearch and extract every field as a label creates thousands of streams and experiences noticeable performance degradation on queries as well as an unnecessarily bloated index. The rule of thumb: fewer, more stable labels, and powerful text and JSON filters within the query itself.

7. Evaluating logs and metrics together in Grafana

The real value of centralized logging with Loki emerges when logs and metrics are combined in the same Grafana interface. A dashboard panel showing a container's CPU usage from Prometheus can be placed directly next to a log panel with the associated error messages from Loki, both filtered to the same time range and the same service name. This correlation saves considerable time in practice during troubleshooting, because the connection between a load spike and the associated error messages becomes immediately visible.

Grafana's Explore view additionally supports jumping directly from a metric to correlating logs via so called derived fields, provided trace or request IDs are consistently present in both logs and metrics. For teams that do not yet use distributed tracing, this log to metric correlation is often the most pragmatic first step toward complete observability, without having to build a full tracing infrastructure right away.

8. Common mistakes when building centralized logging

The most common mistake is unreflectively extracting every JSON field as a Loki label, which, as described in section 6, causes cardinality to explode. A second widespread mistake concerns Promtail's positions file: if it is accidentally deleted when the Promtail container restarts, instead of living on a persistent volume, Promtail reads all logs again from the beginning and creates massive duplicates in Loki.

# Common mistakes when building centralized logging with Loki

# WRONG: every JSON field becomes a label — cardinality explosion
# pipeline_stages:
#   - json:
#       expressions:
#         request_id: request_id
#   - labels:
#       request_id:            # thousands of unique streams

# RIGHT: high-cardinality fields stay in the log line, not as labels
# pipeline_stages:
#   - json:
#       expressions:
#         request_id: request_id
#   # no labels stage for request_id — filter with | json | request_id="..."

# WRONG: positions file lost on container restart — full log re-read
# volumes:
#   - promtail-tmp:/tmp   # ephemeral, resets on recreate

# RIGHT: persistent volume for the positions file
# volumes:
#   - promtail-positions:/data
# positions:
#   filename: /data/positions.yaml

A third mistake is the absence of ingestion limits in Loki, combined with a faulty application that writes thousands of identical error messages per second in a loop. Without configured rate limits, a single container can overload the entire Loki instance and thereby cripple centralized logging for all other services. Sensible limits per tenant and stream prevent exactly this scenario without restricting normal log volumes.

9. Centralized logging approaches in direct comparison

Loki is not the only option for centralized logging of containers. The choice between the most common approaches depends on data volume, search requirements and existing infrastructure.

For most Docker based projects, Loki is the best compromise between operating costs and functionality, because it integrates seamlessly into an existing Prometheus and Grafana landscape. Elasticsearch remains the right choice when complex full text search over huge, unstructured log volumes is actually needed, for example in security audits with highly variable search patterns.

10. Summary

Centralized logging with Loki and Promtail solves the problem of scattered, ephemeral container logs across multiple hosts. Promtail automatically discovers containers via Docker service discovery, reads their logs and attaches speaking labels. Loki stores the logs cost effectively, because only labels are indexed while the actual text sits compressed in chunks. LogQL combines label selectors with text and JSON filters and even allows metric queries directly on logs.

The decisive success factor is a disciplined label strategy: stable, low cardinality values as labels, variable values like request IDs in the searchable log text. Anyone who consistently implements this separation gets a log platform with Loki that integrates smoothly alongside Prometheus metrics in Grafana and forms the foundation for later extensions like alerting or distributed tracing.

11. FAQ: Centralized Container Logging with Loki and Promtail