Building a Container Based Test Matrix for Multiple Runtime Versions

1. Why a test matrix gives more confidence than a single test run

A test matrix checks an application not just once, but in several combinations of relevant conditions, usually different language versions such as PHP 8.2, 8.3 and 8.4, or Node 18, 20 and 22. Without such a matrix, a team effectively only tests the one combination that happens to be installed on its own CI runner or developer machine, and only learns about incompatibilities with other versions once a customer or another team hits them in production.

Docker makes a test matrix practical because every version exists as its own, isolated image, without needing to install multiple interpreter versions in parallel on the same host. Instead of maintaining PHP 8.2, 8.3 and 8.4 side by side on a CI machine, with the well known conflicts between extensions and configuration files, every matrix branch simply starts a different base container with the appropriate version.

The value of a well built test matrix shows up especially for libraries, Composer packages or Symfony bundles used by multiple projects with different PHP versions. Without matrix tests, a new PHP version would in the worst case only be noticed by the first customer to upgrade to it. With matrix tests, compatibility is known and documented before the release even happens.

2. Parametrized Dockerfiles for multiple runtime versions

The simplest way to a test matrix with Docker is a single, parametrized Dockerfile that selects the base version through a build argument. Instead of maintaining five nearly identical Dockerfiles, one per PHP version, you use ARG PHP_VERSION directly in the FROM line and pass the concrete version at build time. That drastically reduces maintenance effort, because changes to the test environment only need to be made in one place.

What matters with this kind of test matrix is that the build argument only affects the base version, not application logic or configuration, otherwise you violate the same separation of build and configuration that applies with build once, deploy many. The test matrix tests compatibility with different runtimes, not different application variants.

# Dockerfile.test -- parametrized for a compatibility test matrix
ARG PHP_VERSION=8.4
FROM php:${PHP_VERSION}-cli AS test

WORKDIR /app
COPY composer.json composer.lock ./
RUN docker-php-ext-install pdo_mysql \
  && curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer \
  && composer install --no-scripts --prefer-dist

COPY . .
CMD ["vendor/bin/phpunit", "--colors=never"]

3. Defining matrix jobs in GitLab CI

GitLab CI natively supports matrix jobs through the parallel: matrix key, which describes a test matrix declaratively instead of imperatively. A single job is automatically duplicated for every combination of the specified variables, each with its own, unique job name in the pipeline overview. That saves manually defining an almost identical job for every PHP version and maintaining it multiple times on every change.

For larger test matrix configurations with several dimensions, for example a PHP version combined with a database version, the number of combinations grows quickly. GitLab allows combining several variable lists in the same matrix block, which produces a complete cartesian product. It is worth deliberately deciding which combinations actually need testing, instead of running every theoretically possible combination.

# .gitlab-ci.yml -- test matrix across PHP versions and database versions
test-matrix:
  stage: test
  parallel:
    matrix:
      - PHP_VERSION: ["8.2", "8.3", "8.4"]
        DB_VERSION: ["8.0", "8.4"]
  image: docker:24
  services:
    - docker:24-dind
  script:
    - docker build --build-arg PHP_VERSION=$PHP_VERSION -f Dockerfile.test -t test-$PHP_VERSION .
    - docker run --rm test-$PHP_VERSION
  allow_failure:
    exit_codes: 2  # deprecation-only failures don't block the pipeline

4. Docker Compose profiles for local matrix runs

Not every test matrix has to run exclusively in CI. For local development, it helps to map the same matrix through Docker Compose profiles, so a developer can test against a single version or several versions in a targeted way, without triggering the entire CI pipeline. Each profile represents one version, and docker compose --profile php83 up starts only the corresponding service.

This approach considerably speeds up test matrix development, because a developer can immediately reproduce a suspected incompatibility locally against the affected version, instead of waiting for a full CI run. Compose profiles and GitLab matrix jobs should reference the same version list, ideally from a shared variable file, so the two environments never drift apart.

5. Keeping parallelization and runtime under control

A test matrix with many combinations can significantly extend the overall pipeline runtime if all matrix branches run serially instead of in parallel. GitLab CI runs matrix jobs in parallel across available runners by default, which drastically reduces wall clock time compared to serial execution, at the cost of needing more simultaneous runner capacity.

For teams with limited runner capacity, it is worth staggering the test matrix by criticality: the currently supported version runs on every commit, older or upcoming versions only nightly or on tags. The rules construct in GitLab CI allows exactly this staggering, by tying individual matrix combinations to different trigger conditions.

# .gitlab-ci.yml -- staggered matrix: current version always, others nightly
test-matrix-staggered:
  stage: test
  parallel:
    matrix:
      - PHP_VERSION: "8.4"
        WHEN: "always"
      - PHP_VERSION: ["8.2", "8.3"]
        WHEN: "nightly"
  rules:
    - if: '$WHEN == "always"'
    - if: '$WHEN == "nightly" && $CI_PIPELINE_SOURCE == "schedule"'
  script:
    - docker build --build-arg PHP_VERSION=$PHP_VERSION -f Dockerfile.test -t test-$PHP_VERSION .
    - docker run --rm test-$PHP_VERSION

6. Consolidating results: one compatibility report per version

Scattered green and red job icons in the pipeline overview are hard to read for a test matrix with many combinations. A consolidated compatibility report is more useful, summarizing per version how many tests passed, failed, or were skipped. JUnit XML reports from each matrix branch can be collected in GitLab via artifacts.reports.junit and displayed together in the merge request view.

For a project that wants to publicly document its test matrix, for example a Composer library, an automatically generated compatibility table in the README is also worthwhile, updated on every successful matrix run. Users then see at a glance which PHP versions are officially supported and tested, instead of relying on an outdated, manually maintained list.

7. Build caching in the matrix without redundant downloads

Without caching, every branch of a test matrix downloads the same Composer or npm dependencies again, which unnecessarily inflates total runtime with five or more versions. BuildKit cache mounts (RUN --mount=type=cache) only help partially here, because different PHP versions sometimes require different dependency resolutions. A shared, registry backed cache for Composer, configured via COMPOSER_CACHE_DIR and shared as a Docker volume between matrix runs, noticeably reduces redundant downloads.

With a test matrix in GitLab CI, the cache key can be deliberately extended with the PHP version, so every version gets its own cache area without different dependency resolutions overwriting each other. cache: key: "composer-$PHP_VERSION" is the simplest, most robust solution for this.

8. Common mistakes when building a test matrix

The most common mistake with a test matrix is testing too many combinations without questioning the actual benefit. A project that no longer supports PHP 7.4 should not keep that version in the matrix just because it historically was there once. Every additional combination costs runtime and runner minutes that are missing elsewhere.

# WRONG: testing an unsupported version out of habit
test-matrix:
  parallel:
    matrix:
      - PHP_VERSION: ["7.4", "8.0", "8.1", "8.2", "8.3", "8.4"]
      # 7.4 and 8.0 are long past end of life -- wasted CI minutes

# RIGHT: matrix limited to officially supported versions
test-matrix:
  parallel:
    matrix:
      - PHP_VERSION: ["8.2", "8.3", "8.4"]
      # matches composer.json's "require": {"php": "^8.2"}

A second mistake is ignoring matrix results when only an older version fails. If a team habitually dismisses a failed matrix branch as a known, unimportant problem, the entire test matrix loses its purpose. Either the version is officially supported and a failure blocks the merge, or the version is removed from the matrix; a third, ignored state undermines trust in all matrix results.

9. Test matrix strategies compared side by side

The following table compares a single test run against a full test matrix across multiple runtime versions.

The extra effort of a test matrix pays off especially for projects with external users who decide the supported version range themselves. Internal projects with a single, fixed production version benefit less strongly, but should at least test the current and the next upcoming version, to prepare for migrations early.

10. Summary

A container based test matrix solves a problem many teams silently accept: applications are effectively only checked against a single, randomly installed runtime version. Docker makes it practical to test multiple PHP or Node versions in parallel and in isolation, without conflicts between interpreter installations on the same host. GitLab matrix jobs and Compose profiles map the same version list both in CI and locally.

The decisive factor for the long term benefit of a test matrix is discipline: only officially supported versions belong in the matrix, failures must be handled consistently, and consolidated reports replace scattered job icons. Anyone who observes these points gains a reliable, automated statement about compatibility, instead of relying on accidental discoveries by customers or other teams.

11. FAQ: Container Based Test Matrix