GitLab CI needs: DAG pipelines instead of strict stages

1. The problem with the classic stage model

In GitLab CI's standard execution model a pipeline is organized into stages, and all jobs of a stage run in parallel, but the next stage only starts once truly every job of the previous stage has completed successfully. This model is easy to understand and perfectly adequate for small pipelines, but it becomes a bottleneck as soon as a stage contains a single slow job that has no real bearing on several fast jobs in the next stage. A typical example is a test stage with a fast unit test job and a much slower end-to-end test job, followed by a build stage that really only depends on the unit test result but still has to wait until the slow end-to-end job finishes too, because the stage model has no finer-grained dependency control.

This waiting adds up considerably in practice, especially in pipelines with many stages and heterogeneous job runtimes. A build job that could start within seconds may end up waiting several minutes just because a completely unrelated job in the same stage is still running. Across several consecutive stages with such imbalances, the wait time accumulates into a total runtime that is far longer than the actual dependency chain of the jobs would require. This exact structural problem is what needs solves, by decoupling ordering from stage boundaries and tying it instead to explicitly declared job-to-job dependencies.

2. How needs changes the execution graph

As soon as a job gets the needs keyword with a list of job names, GitLab ignores the strict stage order for that job and starts it as soon as all jobs listed in needs have completed, regardless of whether other jobs in the same or earlier stages are still running. The stage assignment remains for display in the pipeline view and for logical grouping, but loses its role as an execution barrier. This effectively turns the pipeline into a directed acyclic graph, or DAG, where every job is a node and every needs relationship is a directed edge that GitLab translates into an optimal execution order when the pipeline starts.

It matters that needs controls not only order but also artifact flow: by default a job with needs automatically downloads the artifacts of the listed dependencies, even if they come from an earlier stage, and ignores artifacts from jobs not listed in needs, even if those ran in an earlier stage. This is a deliberate difference from the classic behavior, where all artifacts from all previous stages are available by default. Anyone switching from a classic stage model to needs must therefore check whether a job actually receives every artifact it needs, because the artifact scope changes with the migration and otherwise hard-to-trace failures can creep into downstream jobs.

stages: [build, test, package, deploy]

build_frontend:
  stage: build
  script: [npm run build]

build_backend:
  stage: build
  script: [composer install --no-dev]

unit_tests:
  stage: test
  needs: [build_backend]
  script: [vendor/bin/phpunit]

package_app:
  stage: package
  needs: [build_frontend, unit_tests]
  script: [./package.sh]

3. Parallel branches and shorter overall runtime

The big practical benefit of needs shows up as soon as a pipeline contains several independent branches that only converge later. In the classic model, frontend build and backend build might run in parallel within the same stage, but the next stage still has to wait for both, even if the tests depending on them only concern one of the two branches. With needs, each test job can specifically wait only for the build job relevant to it, so frontend tests can start as soon as the frontend build finishes, entirely independent of whether the backend build is still running. In practice this decoupling means the overall pipeline runtime moves closer to the length of the longest actual dependency path instead of the sum of all stage runtimes.

For pipelines with clearly separable components, such as a frontend, a backend and an infrastructure definition, needs can cut overall runtime by 30 to 50 percent, depending on how unevenly job runtimes were originally distributed within the individual stages. The effect is especially large when a pipeline has many stages with only a few jobs each, because that is where the wait times between stages accumulate the most. Teams looking to reduce their pipeline runtime should therefore first check which jobs actually depend on each other and which only happen to sit in the same stage order, because that is exactly where the biggest unused optimization potential lies.

4. needs without stages and the fully DAG pipeline

Since more recent GitLab versions, the stages keyword is no longer strictly required for needs-based pipelines, because GitLab can derive the execution order entirely from the needs graph without every job still having to be assigned to a stage. In this mode the stage column disappears from the pipeline view and instead only the dependency graph is visualized, which can be much clearer for very complex pipelines with many independent branches than a long list of artificially named stages that really only served as grouping.

In practice this fully DAG variant is mainly recommended for newly built pipelines, while a hybrid approach is often more sensible when migrating an existing pipeline: stages remain as a rough logical grouping, but needs determines the actual execution order within and across those stages. This hybrid approach is easier to communicate, because team members can still orient themselves around familiar terms like build, test and deploy, while the performance benefits of needs are still fully realized, without having to rethink the entire pipeline structure.

# Fully DAG pipeline without explicit stage order
lint:
  stage: .pre
  script: [composer run lint]

build:
  needs: []
  script: [composer install]

test:
  needs: [build]
  script: [vendor/bin/phpunit]

deploy:
  needs: [test, lint]
  script: [./deploy.sh]

5. Artifact control and needs options in detail

needs accepts not just a simple list of job names but also an extended object syntax with the fields job, artifacts and optional. Setting artifacts: false lets you specifically prevent downloading the artifacts of a dependency even though execution order is still determined by that dependency, which is useful for a job that only waits for a previous job to complete successfully but does not need its build artifacts. The field optional: true allows a dependency to not exist at all under certain rules conditions without the pipeline failing because of it, which is often needed for conditionally generated jobs combined with rules.

Additionally, needs also supports cross-pipeline and cross-project dependencies through the pipeline and project fields, allowing a job in one pipeline to wait for artifacts or the status of a job from another, already completed pipeline. This capability is mostly used in larger organizations where several projects are loosely coupled, for example when a deployment repository waits for a successful build of an artifact in a separate build repository without both repositories having to be merged into a single monorepo pipeline.

deploy:
  stage: deploy
  needs:
    - job: build
      artifacts: true
    - job: security_scan
      optional: true
    - job: lint
      artifacts: false
  script:
    - ./deploy.sh

6. Limits: maximum number of needs entries and cycle detection

By default GitLab limits the number of needs entries per job to 50, which is more than enough for the vast majority of pipelines but can theoretically become relevant in extremely branched monorepo pipelines with many sub-projects. If this limit is exceeded, GitLab reports an error already when parsing the .gitlab-ci.yml, so the problem does not only surface at runtime but already at commit time or in the CI lint check. In such cases it usually helps to bundle dependencies through intermediate jobs that themselves aggregate several needs entries and are then referenced by downstream jobs as a single dependency.

GitLab also automatically checks whether the graph resulting from needs is free of cycles, meaning there is no chain of dependencies that ultimately loops back on itself. Such a cycle would be logically impossible to execute, because two jobs would be waiting on each other, and GitLab rejects a pipeline with a detected cycle already during validation, before a single job is started. This built-in check is especially valuable in large pipelines maintained by multiple teams, where an accidentally introduced cycle would otherwise only surface after a failed pipeline start.

7. Migrating an existing stage pipeline to needs

The pragmatic starting point for an existing pipeline is to first map out the actual content dependencies between jobs, regardless of which stage they currently sit in. This often reveals that the stage order was historically more or less arbitrary and that many jobs do not actually depend on every job in the previous stage, only on one or two specific ones. This mapping is the most important step, because an incorrectly set needs list either leads to missing artifacts, if a real dependency is forgotten, or to pipelines that remain just as slow, if too many jobs are still being waited on out of excess caution.

After mapping, a gradual rollout is recommended, starting with the jobs that promise the biggest time savings, typically jobs at the end of a long stage chain that really only depend on a single early job. GitLab's CI/CD pipeline view visualizes the resulting graph directly and makes it visible whether the new needs structure actually leads to the expected parallelization. Comparing pipeline duration before and after the switch, visible in the project's pipeline analytics view, provides concrete evidence of the benefit and helps justify the investment to the team.

8. When needs brings little benefit

needs delivers its value mainly with unevenly distributed job runtimes and several parallel dependency branches. In very small pipelines with only two or three jobs that already depend on each other almost entirely sequentially, the switch brings barely any measurable time gain but still adds extra complexity to the pipeline file. In such cases it is usually better to keep the simple stage order, because the maintenance overhead of managing needs does not justify the small time gain.

In pipelines where runner capacity is the actual bottleneck, for example because only a few shared runners are available, needs also delivers less than expected, because jobs that become ready to start in parallel still have to wait for a free runner. In such environments, runner capacity should be checked before optimizing with needs, for example through autoscaling or additional runner tags for different job types, because otherwise the theoretical parallelization from needs fails in practice due to the limited number of simultaneously available runners and the hoped-for time savings never materialize.

9. Conclusion: needs as a targeted lever for pipeline speed

needs turns a rigid stage pipeline into a directed acyclic graph where jobs start as early as possible instead of waiting on artificial stage boundaries, which brings noticeable time savings in pipelines with uneven job runtimes and several parallel branches. The switch requires careful mapping of actual dependencies and awareness that the artifact scope changes with needs, but is achievable with low risk using the hybrid approach of existing stages plus targeted needs entries.

The table below contrasts the classic stage model and the needs-based DAG model along the most important criteria, as a decision aid for your own pipeline.

10. Summary

11. FAQ: needs and DAG pipelines: The essentials at a glance