Controlling GitLab Runner Capacity, Concurrency and Queue Times in Large Teams

1. The queue problem in growing teams

When a team grows and more developers open merge requests at the same time, pipeline starts collide with each other. GitLab distributes jobs to available runners, and once all runners are fully occupied, new jobs land in the queue. The queue time is the span between a job being triggered and its actual start. In small teams with few simultaneous pipelines this time stays short. In teams with ten or more active developers, queue time can exceed a job's actual execution time, which noticeably stretches out feedback cycles.

The problem is especially pronounced in Magento projects: build jobs running composer install, setup:di:compile and a Tailwind build often take four to eight minutes. If three teams depend on the same runner at once, the last jobs in line do not wait seconds, they wait minutes before they even start. The result is an artificially extended deployment cycle, even though the server hardware would theoretically be sufficient. The fix is not always more hardware, but first the right capacity configuration of the existing runner.

The two most important levers are concurrent at the runner level and limit at the executor level in config.toml. Beyond that, tags, fair scheduling and the structure of the pipelines themselves also play a role. This article shows how to use these levers deliberately to measurably reduce queue times for a growing Magento team.

2. concurrent and limit: the core of capacity control

The global concurrent setting in /etc/gitlab-runner/config.toml specifies how many jobs the runner process may execute simultaneously, across all registered runner instances. The default after installation is usually 1, meaning the runner processes every job sequentially. For a server with eight CPU cores and 32 GB of RAM, this value is drastically too low. A realistic starting point for a dedicated build server is concurrent = 4, i.e. four jobs running in parallel, since Magento builds are memory intensive due to composer install and setup:di:compile.

The per-runner value limit inside the [[runners]] block caps how many jobs a single runner instance may take on at once. If concurrent = 4 and two runners are registered with limit = 2, a total of four jobs can run, two per runner instance. This model lets you configure different runner instances for different tasks, for example a runner with a high limit for fast test jobs and a runner with a low limit for resource-heavy build jobs, without excessively raising the global concurrent value.

# /etc/gitlab-runner/config.toml
# Global concurrent: maximum simultaneous jobs across all runners on this host
concurrent = 6
check_interval = 3
shutdown_timeout = 30

[[runners]]
  name = "magento-build-runner"
  url = "https://gitlab.example.com"
  token = "RUNNER_TOKEN_BUILD"
  executor = "docker"
  # Limit for this runner instance only
  limit = 3
  [runners.docker]
    image = "php:8.4-cli"
    volumes = ["/cache:/cache", "/composer-cache:/root/.composer"]
    memory = "4g"
    cpus = "2"

[[runners]]
  name = "magento-deploy-runner"
  url = "https://gitlab.example.com"
  token = "RUNNER_TOKEN_DEPLOY"
  executor = "shell"
  # Deploy runner: only one job at a time to avoid concurrent symlink switches
  limit = 1
  [runners.custom]
    run_exec = ""

A common mistake is raising concurrent without accounting for available system resources. If five parallel composer install processes run at once and each needs 2 GB of RAM, a server with 8 GB of RAM slides into swap, which slows down every job instead of speeding them up. The correct formula: concurrent equals available RAM divided by the average RAM demand per job, minus a buffer for the operating system.

3. Runner tags as a routing mechanism

Runner tags are the primary mechanism in GitLab CI/CD for routing jobs to specific runners. Without tags, any untagged runner picks up any untagged job, which in a mixed environment of build, test and deploy runners causes unwanted cross contamination. A deploy runner that accidentally picks up a resource-heavy build job ties up its limit capacity and delays deploy jobs that depend on that same runner.

The recommended pattern for large teams is a clear tag hierarchy: build for compiler and asset build jobs, test for PHPUnit and PHPStan, deploy for SSH-based deployments, docker for jobs that need a Docker executor, and shell for jobs with direct server access. Every runner is registered with one or more of these tags, and every job in .gitlab-ci.yml declares its tags block. That way every job reliably lands on the right runner type, and queue times for critical deploy jobs are decoupled from the load of build jobs.

4. job_policy and fair scheduling between projects

In environments where multiple Magento projects share the same GitLab runner, a job fairness problem emerges: a project that triggers many parallel pipelines can permanently block the runner for other projects. GitLab offers automatic balancing through its instance-level fair scheduling algorithm: the GitLab server does not distribute jobs in strict FIFO order but favors projects with fewer jobs currently running. This reduces starvation, where one project waits a long time because another project dominates runner capacity.

At the runner level, you can additionally use the maximum_timeout value in the [[runners]] block to make sure stuck jobs are automatically aborted and free up capacity. A typical value for Magento build jobs is 1800 seconds (30 minutes). Jobs that run longer are almost always a symptom of a problem, such as a stalled composer install or a failed frontend build waiting on user input that never times out on its own. With a defined timeout, the runner releases the resource in a controlled way instead of holding it indefinitely.

5. Horizontal scaling: when a second runner makes sense

Horizontal scaling, meaning adding further runner hosts, is the right move once the vertical capacity of the existing server is exhausted. The indicator for this is not high CPU or RAM utilization alone, but a stubbornly high queue time despite an already optimized concurrent value. If jobs wait an average of two minutes in the queue before starting and all available runner resources are already saturated, a second runner host makes more sense than further tuning.

GitLab supports several patterns here: multiple physical or virtual hosts, each running a gitlab-runner process registered with the same tags. GitLab automatically distributes jobs among these runners. Alternatively, cloud autoscalers (AWS EC2 Autoscaler, Google Cloud Autoscaler) let you start and stop runner instances dynamically based on demand. For Magento teams with predictable workloads, for example with defined sprint cycles, a static second runner machine is often the simpler and more cost-effective solution.

6. A complete config.toml for a Magento build runner

The following configuration shows a production-ready config.toml for a dedicated Magento build runner with a Docker executor. It accounts for concurrency, resource limits, cache volumes, network configuration and a separate deploy runner with no concurrency issue. The comments explain the reasoning behind each parameter.

# /etc/gitlab-runner/config.toml
# Global limit: maximum jobs across all [[runners]] on this host
concurrent = 5
check_interval = 5
shutdown_timeout = 60
log_level = "info"
log_format = "json"

[[runners]]
  name = "magento-build-php84"
  url = "https://gitlab.example.com"
  token = "${GITLAB_RUNNER_TOKEN_BUILD}"
  executor = "docker"
  # Max 3 concurrent build jobs on this runner instance
  limit = 3
  # Job timeout: abort after 25 minutes
  maximum_timeout = 1500
  # Route only jobs tagged with 'build' and 'php84'
  tag_list = ["build", "php84", "docker"]

  [runners.cache]
    Type = "local"
    Path = "/runner-cache"
    Shared = false

  [runners.docker]
    tls_verify = false
    image = "php:8.4-cli"
    privileged = false
    # Bind-mount Composer cache to speed up repeated builds
    volumes = [
      "/srv/composer-cache:/root/.composer:rw",
      "/srv/npm-cache:/root/.npm:rw",
      "/runner-cache:/cache"
    ]
    shm_size = 0
    memory = "4096m"
    memory_swap = "4096m"
    cpus = "2.0"
    network_mode = "bridge"

[[runners]]
  name = "magento-deploy-ssh"
  url = "https://gitlab.example.com"
  token = "${GITLAB_RUNNER_TOKEN_DEPLOY}"
  executor = "shell"
  # Deploy: only one job at a time to prevent concurrent symlink switches
  limit = 1
  maximum_timeout = 900
  tag_list = ["deploy", "production", "shell"]
  # Shell executor uses the gitlab-runner user's environment
  shell = "bash"

An important detail in this configuration: the build runner uses the Docker executor with per-container resource limits, ensuring that individual jobs cannot claim the entire host's capacity. The deploy runner runs on the shell executor with a strict limit = 1, because parallel deployments would cause race conditions during the symlink switch. The global log_format = "json" setting enables structured parsing of runner logs in monitoring systems such as Grafana Loki or Elasticsearch.

7. Observing and evaluating queue times

GitLab itself provides an overview of actively running and waiting jobs in instance administration under Admin Area → Monitoring → GitLab Runner. This view is sufficient for troubleshooting, but not suited for continuous monitoring since it stores no historical data. For lasting capacity monitoring, there are two proven approaches: the GitLab Prometheus endpoint and the GitLab API.

GitLab exports runner metrics under /metrics, including gitlab_runner_jobs_total, gitlab_runner_jobs{state="running"} and gitlab_runner_jobs{state="waiting"}. These metrics can be scraped by Prometheus and visualized in Grafana. A simple alerting rule could look like this: if gitlab_runner_jobs{state="waiting"} > 5 for more than five minutes, Alertmanager sends a notification. That is the early warning sign that capacity is not keeping pace with load and that adjusting the concurrent value or adding another runner host is needed.

8. Comparing strategies for reducing queue time

There are several levers for shorter queue times, and the right choice depends on the actual bottleneck. Blanket increases to concurrent only help if system resources are still available. Tag routing only helps if jobs are misdistributed. Horizontal scaling only helps if the bottleneck truly lies in capacity and not in the pipeline structure itself.

In practice, the first step is always measurement: what is the average queue time per job type? Where is the bottleneck, at the build runner, the deploy runner or the test runner? Only once these questions are answered with data does it make sense to apply one of the strategies above. Blindly raising concurrent without monitoring regularly leads to OOM situations that abort every running job and stretch the queue time for the entire evening.

9. Common mistakes in capacity management

The most common mistake is a concurrent value set too high without matching Docker resource limits. If ten parallel jobs run without a memory limit and one of them has a memory leak, memory demand grows uncontrolled until the kernel's OOM killer terminates processes. The fix: always set memory and memory_swap in the [runners.docker] block, so a single job can only use the RAM it was assigned.

A second mistake is forgetting tags when registering a new runner. A runner without tags picks up all untagged jobs, including jobs from projects this runner was never meant to serve. In an environment with multiple projects, this results in a deploy runner for project A accidentally taking on build jobs from project B. A consistent tag strategy combined with the run_untagged = false setting in the [[runners]] block rules this problem out structurally. A third mistake is ignoring maximum_timeout: without a timeout, a stuck job can permanently block a runner's entire limit capacity until someone intervenes manually.

10. Summary

Queue times in large teams are not an unavoidable fate, they are a configuration problem with well-known solutions. The first step is always measuring: GitLab metrics and Prometheus show which runner type is the bottleneck. The second step is adjusting concurrent and limit based on real resource availability, not wishful thinking. Tag routing decouples build, test and deploy runners from each other, so a full build queue does not block the deploy runner. maximum_timeout prevents individual stuck jobs from permanently holding capacity hostage.

For Magento projects the recommendation is clear: a dedicated build runner with a Docker executor and limit = 2 to 4, a separate deploy runner with a shell executor and limit = 1, and Prometheus-based monitoring for queue times and job throughput. Only once this baseline configuration is fully exploited does the effort of a second runner host or a cloud autoscaler pay off.

11. FAQ: Runner capacity and queue times in large teams