Structuring Docker Compose Cleanly: Services, Profiles, Overrides, and .env Files

1. The Problem: One File for All Environments

The most common cause of high maintenance Docker Compose projects is a single docker-compose.yml that tries to serve local development, CI tests, and production deployments all at once. The result: volumes for hot reload sit next to health check settings meant for production, debugger ports sit next to performance tuning flags, and somewhere a password is hardcoded because the variable was always available in one particular context. Docker Compose offers all the mechanisms needed to separate these configurations cleanly, they are simply rarely used consistently.

The core problem is not the complexity of Docker Compose, it is the lack of a structural convention within the team. Once profiles, overrides, and separate .env files are introduced sensibly, the payoff is a Compose configuration that behaves predictably in every environment, keeps no credentials in the repository, and clearly communicates to new developers which services exist for which purpose. The effort is a one time investment, the benefit lasts.

2. Base Structure: What Belongs in the Core Compose File

The docker-compose.yml is the base layer: it defines every service with its image, networks, and volumes in an environment neutral form. No ports that only make sense locally. No stdin_open: true just for debugging. No volume mounts that only exist on a developer's laptop. What belongs in the base: service name, image, dependencies, network membership, and only the environment variables that are strictly required across every environment. Everything else belongs in specific override files.

The base structure of a Docker Compose project should always include a docker-compose.yml (base), a docker-compose.override.yml (loaded automatically for local development), and optional docker-compose.ci.yml, docker-compose.staging.yml, and docker-compose.prod.yml files. This separation lets CI explicitly call docker compose -f docker-compose.yml -f docker-compose.ci.yml up without accidentally mounting development volumes or opening debug ports.

# docker-compose.yml: Base configuration, environment-neutral
# All environments share this file; no dev-only or prod-only settings here

services:
  app:
    image: registry.mironsoft.de/shop/app:${APP_VERSION:-latest}
    networks: [backend, frontend]
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      APP_ENV: ${APP_ENV}
      DB_HOST: db
      REDIS_HOST: redis
    restart: unless-stopped

  db:
    image: mariadb:11.4
    networks: [backend]
    volumes:
      - db_data:/var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
      MYSQL_DATABASE: ${DB_NAME}
      MYSQL_USER: ${DB_USER}
      MYSQL_PASSWORD: ${DB_PASSWORD}
    healthcheck:
      test: ["CMD", "healthcheck.sh", "--connect"]
      interval: 10s
      timeout: 5s
      retries: 5

  redis:
    image: redis:7-alpine
    networks: [backend]
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 3s
      retries: 3

networks:
  backend:
  frontend:

volumes:
  db_data:

A common mistake in the base configuration is setting restart: always instead of restart: unless-stopped. The difference: always restarts the container on the next Docker daemon start even after a manual docker compose stop. unless-stopped respects manual stops. In development environments, unless-stopped is almost always the right choice. In production, on the other hand, check whether the host's init system (systemd) already controls the daemon start.

3. Separating .env Files Correctly

Docker Compose automatically loads a .env file from the project directory and interpolates its values into the Compose configuration. That is convenient, but it quickly turns into chaos if every environment shares the same file. The clean solution is a .env.example file in the repository (committed, without real credentials) that serves as a template for environment specific .env files. The actual files live in .gitignore and are never committed.

Teams working with multiple environments benefit from an explicit structure: env/local.env, env/ci.env, env/staging.env, each holding the values that match its environment. At invocation time, the matching file is passed with the --env-file flag: docker compose --env-file env/ci.env -f docker-compose.yml -f docker-compose.ci.yml up -d. That makes every call fully deterministic and traceable: no implicit loading from the working directory, no surprises from locally set shell variables that override Compose variables.

# env/local.env: Local development values (never committed with real secrets)
# Copy from .env.example and fill in your local values

APP_ENV=development
APP_VERSION=local
APP_DEBUG=true

DB_NAME=shop_dev
DB_USER=shop_user
DB_PASSWORD=dev_password_only_local
DB_ROOT_PASSWORD=root_dev_only

REDIS_PASSWORD=

# Enable Xdebug for local PHP debugging
PHP_XDEBUG_MODE=develop,debug
PHP_XDEBUG_CLIENT_HOST=host-docker-internal

---

# env/ci.env: CI-specific values (stored in CI secrets, not in repo)
APP_ENV=testing
APP_VERSION=${CI_COMMIT_SHORT_SHA}
APP_DEBUG=false

DB_NAME=shop_test
DB_USER=shop_test
DB_PASSWORD=${CI_DB_PASSWORD}    # injected by GitLab CI / GitHub Actions
DB_ROOT_PASSWORD=${CI_DB_ROOT}

PHP_XDEBUG_MODE=off

---

# .env.example: Template committed to the repo
APP_ENV=
APP_VERSION=
APP_DEBUG=
DB_NAME=
DB_USER=
DB_PASSWORD=
DB_ROOT_PASSWORD=
REDIS_PASSWORD=
PHP_XDEBUG_MODE=off

4. Overrides: Environment Specific Adjustments

Docker Compose automatically loads docker-compose.override.yml in addition to the base Compose file. That makes this file ideal for development specific extensions: volume mounts for hot reload, open debug ports, simplified health checks, and enabled stdin_open. Since this file never needs to be specified explicitly, developers can simply run docker compose up and immediately get the right development environment. In CI and production it is deliberately excluded through explicit -f flags.

Override files overwrite individual fields of the base configuration, but they do not replace it entirely. Lists like volumes, ports, and environment are merged. That means an override file can open new ports without losing the ones defined in the base. Scalars like image and restart, on the other hand, are fully replaced. This merge behavior has to be kept in mind when designing overrides, particularly for command and entrypoint, which are replaced wholesale.

# docker-compose.override.yml: Loaded automatically for local development

services:
  app:
    # Mount source code for hot reload, local only
    volumes:
      - ./src:/var/www/html:cached
      - ./var/cache:/var/www/html/var/cache
    # Expose Xdebug port to host
    ports:
      - "9003:9003"
    environment:
      PHP_XDEBUG_MODE: develop,debug
      PHP_XDEBUG_START_WITH_REQUEST: "yes"
    # Faster restart, less strict healthcheck for dev
    restart: "no"

  db:
    # Expose MariaDB to host for local DB tools (e.g. TablePlus, DBeaver)
    ports:
      - "3306:3306"
    # Shorter healthcheck interval for dev convenience
    healthcheck:
      interval: 5s
      retries: 10

  # Mail catcher, only needed locally
  mailhog:
    image: mailhog/mailhog:latest
    networks: [backend, frontend]
    ports:
      - "1025:1025"   # SMTP
      - "8025:8025"   # Web UI
    profiles: []      # no profile, always starts in dev override

  # Adminer: DB admin UI for local development
  adminer:
    image: adminer:4
    networks: [backend, frontend]
    ports:
      - "8080:8080"

5. Profiles: Activating Optional Services on Demand

Profiles are the feature in Docker Compose that gets overlooked most often. A profile is a named label assigned to one or more services. Services with a profile do not start when docker compose up is run plainly, they have to be activated explicitly with --profile profilename. That makes it possible to define rarely needed services such as a mail catcher, database admin UIs, monitoring stacks, or import tools in the same Compose file without having them spin up on every start.

A typical example of profiles used sensibly in a Docker Compose project: the tools profile contains Adminer and a Redis commander, the monitoring profile contains Prometheus and Grafana, and the import profile contains a one shot container that imports test data. Anyone who only wants to start the core application runs docker compose up. Anyone who needs database tools uses docker compose --profile tools up. Multiple profiles can be combined: --profile tools --profile monitoring. That saves considerable resources on developer laptops and in CI pipelines.

6. YAML Anchors and Extends: Avoiding Duplication

In larger Docker Compose projects, the same configuration blocks tend to repeat: logging settings, shared environment variables, restart policies, and health check templates. YAML anchors (&anchorname) and aliases (*anchorname) make it possible to define these blocks once and reference them multiple times. Changing the anchor updates every alias at once. The feature is defined in the YAML specification and is fully supported by Docker Compose v2.

The extends keyword in Docker Compose enables a different kind of reuse: a service can inherit the configuration of another service in the same file or a different one and override individual fields. This is especially useful when several services are based on the same image but use different commands or environment variables, for example a web service and a worker service that both run the same application but use different entry points.

# docker-compose.yml: YAML anchors eliminate configuration duplication

# Reusable logging configuration block
x-logging: &default-logging
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "3"

# Reusable restart and resource limits
x-defaults: &service-defaults
  restart: unless-stopped
  logging: *default-logging

# Common environment for all PHP services
x-php-env: &php-env
  APP_ENV: ${APP_ENV}
  DB_HOST: db
  REDIS_HOST: redis
  REDIS_PORT: 6379

services:
  app:
    <<: *service-defaults        # merge defaults
    image: registry.mironsoft.de/shop/app:${APP_VERSION}
    environment:
      <<: *php-env               # merge PHP env block
      SERVER_ROLE: web

  worker:
    <<: *service-defaults
    image: registry.mironsoft.de/shop/app:${APP_VERSION}
    environment:
      <<: *php-env
      SERVER_ROLE: worker
    command: ["php", "bin/magento", "queue:consumers:start", "async.operations.all"]

  scheduler:
    <<: *service-defaults
    image: registry.mironsoft.de/shop/app:${APP_VERSION}
    environment:
      <<: *php-env
      SERVER_ROLE: scheduler
    command: ["php", "bin/magento", "cron:run"]

7. Passing Secrets and Sensitive Values Securely

The most common security mistake in Docker Compose projects is hardcoding credentials directly in the Compose file or in committed .env files. The safer alternative is Docker Secrets for Swarm deployments, or a combination of external secret stores (HashiCorp Vault, AWS Secrets Manager) and environment variables injected at runtime. For local development without Swarm, the .env pattern is acceptable as long as the file sits in .gitignore and never contains real production credentials.

Docker Compose v2 also offers the top level secrets configuration, which can mount files from the host filesystem into containers as read only. That is more practical than Swarm secrets because it requires no Swarm initialization. A secret file on the host (for example ./secrets/db_password.txt, listed in .gitignore) gets mounted into the container as /run/secrets/db_password. The application reads the value from the file instead of from an environment variable, which prevents secrets from showing up in docker inspect or process logs.

8. Comparing Structure Variants

There are several approaches to structuring a Docker Compose project. Choosing the right one depends on team size, the number of environments, and deployment requirements.

The recommendation for professional Docker Compose projects is: a base file plus override.yml for local development, a separate ci.yml for tests, and profiles for optional services. This combination scales for any team size and makes the difference between environments instantly readable, without anyone having to mentally trace the merge logic.

9. Typical Workflow Patterns for Teams

A well structured Docker Compose project also needs clear workflow conventions. The most important pattern: developers never run docker compose up without explicit flags in CI, always with --env-file and explicit -f flags. That prevents .override.yml from being active in CI builds and opening mounts or debug ports that are not wanted there. A Makefile or shell script that encapsulates the correct flags for each environment reduces mistakes significantly.

A second important pattern is explicit versioning of Docker Compose images. Instead of image: mariadb:latest, always use image: mariadb:11.4. latest means different images across environments and at different points in time, a subtle drift that is hard to debug. The same applies to your own images: ${APP_VERSION} should always carry a concrete tag or commit SHA, never latest in non local environments. The docker compose config command renders the final, merged configuration, a useful debugging tool before starting any services.

10. Summary

Cleanly structured Docker Compose projects make use of the separation mechanisms the tool provides: an environment neutral base file, an automatically loaded override.yml for local development, explicit override files for CI and staging, profiles for optional services, and separate .env files for each context. YAML anchors avoid configuration duplication across similar services. Secrets never belong in committed files, neither in the Compose file itself nor in .env files.

A practical way to start: refactor an existing Docker Compose project in three steps. First, extract every environment specific value into a variable and move it into separate .env files. Second, move development specific configuration (ports, volumes, debug settings) into docker-compose.override.yml. Third, mark rarely used services with profiles. The result is a Compose project that behaves predictably in every environment and does not overwhelm new team members.

11. FAQ: Structuring Docker Compose Cleanly