GitLab CI Troubleshooting: Green Locally, Red in the Pipeline

1. The classic symptom and why it erodes trust

When a test, a build step, or a deploy job runs cleanly locally but fails in GitLab CI, the first reaction is usually confusion: after all, the code has not changed, only the environment it runs in. That exact contrast is the most important clue, because it immediately shifts the troubleshooting away from the code and toward the environment in which the code runs locally versus in the pipeline.

If this pattern repeats often enough, trust in the CI system overall erodes, developers start dismissing red pipelines as an environment issue prematurely, and eventually miss real bugs. A systematic, repeatable troubleshooting approach prevents exactly this pattern, because it finds the root cause reliably rather than relying on lucky guesses during trial and error.

2. Systematically checking environment differences: PHP version and extensions

The most common single reason for a green-locally-red-in-CI pattern is a differing PHP version or a PHP extension that is installed locally but missing from the CI image. A PHP 8.4 installed locally via Homebrew or the system package manager, with all the usual extensions, often differs substantially from a slim CI image deliberately kept minimal and, for instance, not shipping ext-intl or ext-bcmath at all.

The most reliable first step is therefore printing php -v and php -m both locally and as the first step in the CI job, and comparing the two outputs directly instead of guessing. If the extension list differs, that can be fixed either by adjusting the CI image or through an explicit docker-php-ext-install step before the actual job, depending on whether a custom image is maintained or a standard image is used.

debug_environment:
  stage: .pre
  script:
    - echo "PHP version:"
    - php -v
    - echo "Loaded extensions:"
    - php -m
    - echo "Timezone:"
    - php -r 'echo date_default_timezone_get() . PHP_EOL;'
    - echo "Locale:"
    - locale

3. Runner tags and executor differences as a source of failures

GitLab runners can be targeted at specific jobs through tags, which in mature projects often leads to different jobs running on different runners with different base images or even different executor types, for instance a Docker executor on one runner and a shell executor on another. A job that happens to get routed to a runner with a stale cache layer or an older Docker image version can fail, while the same job runs fine on a different runner.

A targeted look at the Runner field in GitLab's job detail view immediately shows which specific runner executed the failed job, including its tags and executor type. If runner-specific differences are suspected, an explicit comparison of two runs on different runners helps, most easily achieved by forcing a rerun of the job with an adjusted tag in the job definition.

4. Timezone, locale, and silent environment variable differences

Container images frequently default to the UTC timezone, while a local development machine is configured for a timezone such as Europe/Berlin. Tests that perform date comparisons or check timestamps against an expected value therefore pass quietly locally but fail in the pipeline over a one- or two-hour difference that was never explicitly accounted for in the test code.

The locale behaves similarly: a German-language local system formats decimal numbers with a comma instead of a period, which can produce different results in tests relying on number_format or similar functions without an explicit locale setting. The robust fix is to never let timezone and locale implicitly depend on the system environment in tests, but instead set them explicitly in the test code or in the PHPUnit configuration.

# Set timezone and locale explicitly in the CI job,
# instead of relying on the container image
export TZ="Europe/Berlin"
export LC_ALL="de_DE.UTF-8"

php -r 'echo date_default_timezone_get() . PHP_EOL;'

5. Filesystem differences: case sensitivity, line endings, and local-only files

A macOS filesystem is case-insensitive by default, while the Linux container the CI job runs in strictly distinguishes between uppercase and lowercase. A require 'Vendor/Module/Model/MyClass.php' that works fine locally despite incorrect casing in the path, because macOS treats both spellings as identical, fails in the CI container with an unambiguous file-not-found error, even though the code itself is unchanged.

Line endings can also become a problem when Git locally converts CRLF to LF or vice versa depending on the core.autocrlf setting, causing a shell script with Windows line endings to abort in the CI container with a cryptic bad interpreter error. Local configuration files excluded via .gitignore are similarly deceptive, present locally but entirely absent from the freshly cloned CI workspace, which is why comparing the files actually tracked in the repository via git ls-files is worth doing before troubleshooting elsewhere.

# Check whether a file is actually tracked in the repository,
# instead of only existing locally untracked
git ls-files | grep -i "config/local"

# Check the line endings of a file
file scripts/deploy.sh

6. Reading job logs with intent using CI_DEBUG_TRACE

A GitLab job's standard output normally only shows the script: section and its output, not which environment variables are actually set or in what exact order shell commands are executed. Setting the project or pipeline variable CI_DEBUG_TRACE to true enables a considerably more detailed trace mode that logs every single executed shell command, including resolved variable values.

Because CI_DEBUG_TRACE can potentially write sensitive values such as secrets into the log in plain text, it should only ever be enabled temporarily for troubleshooting and never left on permanently in a production pipeline. It additionally helps to selectively preserve artifacts such as log files or generated configuration files through artifacts: when: on_failure, to examine them more closely locally after a failed run.

variables:
  CI_DEBUG_TRACE: "true"  # enable only temporarily for troubleshooting

test_job:
  stage: test
  script:
    - vendor/bin/phpunit
  artifacts:
    when: on_failure
    paths:
      - var/log/
      - var/report/
    expire_in: 3 days

7. Local reproduction with gitlab-runner exec and an identical Docker image

Instead of testing every hypothesis through another commit and another pipeline run, which costs minutes and ties up runner capacity, the exact CI image can be started locally with docker run -it bash and the step in question run manually and interactively. This immediately reveals whether the problem actually lies with the image or only surfaces through the pipeline configuration.

For shell-executor-based runners, GitLab Runner additionally offers the gitlab-runner exec docker command, which runs a single job largely the way it would run in an actual pipeline, including the same before_script and script sections, though without the full set of GitLab CI variables from the project context, which have to be set separately.

# Start the exact CI image locally and inspect it interactively
docker run -it --rm registry.gitlab.com/mironsoft/php:8.4-ci bash

# Inside the container:
php -v
composer install --no-dev
vendor/bin/phpunit --filter TestThatFailsInCi

8. Caching traps: the CI cache holds something different than locally

GitLab CI caches for Composer or npm dependencies are reused between pipeline runs through a configured cache key, but they can hold stale or incompatible versions if the composer.lock changed while the cache key itself was not updated to match. Locally a fresh composer install usually runs, which means the same inconsistency never occurs there.

A proven first diagnostic step is disabling the cache entirely for a single test run, for instance through a manually triggered pipeline run with a cleared cache via the GitLab interface, and checking whether the error then disappears. If it does, the root cause most likely lies in the cache key scheme, which ideally should incorporate the hash of composer.lock rather than a static key that never changes.

9. A systematic checklist for the next red pipeline

Instead of trying hypotheses at random, a fixed order pays off: first compare PHP version and extensions, then check timezone and locale, then identify the executing runner and its tags, then question the cache state, and only after that, if everything else looks unremarkable, actually consider code changes as the root cause.

This order is deliberately sorted by frequency: environment differences are statistically by far the most common cause of a green-locally-red-in-CI pattern, while an actual, environment-independent bug in the code that happens not to surface locally occurs considerably more rarely. The table below summarizes the key diagnostic tools and their respective use cases.

10. Summary

11. FAQ: Local vs. CI Troubleshooting: The Essentials at a Glance