GitLab Runner Autoscaling with Docker Machine and Fleeting

1. Why fixed runner capacity almost always has the wrong size

A fixed-size pool of runner machines inevitably has to be sized for peak load, otherwise developers wait in the job queue for a long time during busy hours. That peak load, however, typically occurs for only a few hours a day, for example when many teams trigger merge request pipelines at the same time. Outside those windows, at night or on weekends, the same machines sit largely idle while still incurring the full infrastructure cost.

Autoscaling breaks this dilemma by spinning up runner machines only when actually needed and tearing them down automatically after an idle period. Instead of permanently keeping a fixed number of machines around, only a small base capacity stays up at all times, while additional capacity is added elastically according to pipeline volume. For cloud-based runners this translates directly into cost savings, since only actually-used compute time gets billed.

2. The Docker Machine model: how it historically worked

The Docker Machine executor was, for many years, the standard way to autoscale GitLab runners. It used the Docker Machine tool, since discontinued by Docker itself, to provision new virtual machines with a cloud provider like AWS, GCP, or Azure on demand, then installed Docker on them and started a job container. The runner itself ran as a kind of manager process, deciding based on the job queue when to spin up new machines and when to tear down existing ones no longer needed.

Configuration happened under the runners.machine section in config.toml, where among other things IdleCount defined the number of permanently kept idle machines, and IdleTime determined how long an unused machine would remain available before being torn down. MaxBuilds additionally limited how many jobs a single machine could run before it was replaced with a fresh instance for security and cleanliness reasons.

# Excerpt from /etc/gitlab-runner/config.toml (Docker Machine, historical)
[[runners]]
  name = "autoscale-runner"
  executor = "docker+machine"
  [runners.machine]
    IdleCount = 2
    IdleTime = 600
    MaxBuilds = 20
    MachineDriver = "amazonec2"
    MachineName = "gitlab-runner-%s"
    MachineOptions = [
      "amazonec2-instance-type=t3.medium",
      "amazonec2-region=eu-central-1"
    ]

3. Why Docker Machine is no longer the future

Docker itself stopped further development of Docker Machine years ago, and the tool has since received no new cloud provider drivers and no security-related updates. GitLab kept relying on it for a long time regardless, since no direct replacement existed, but the lack of active maintenance gradually became a real operational risk, particularly against older, unmaintained cloud API versions that the existing drivers were still written against.

In response, GitLab built Fleeting, its own actively maintained plugin system intended to replace Docker Machine as the recommended autoscaling mechanism. For existing setups this does not mean Docker Machine suddenly stops working overnight, but it does mean new projects and migrations should be built on Fleeting rather than the foreseeably sunsetting Docker Machine path.

4. Fleeting: the new plugin-based autoscaling model

Fleeting abstracts provisioning and tearing down machines behind a clearly defined plugin interface, with separate plugins per cloud provider, such as fleeting-plugin-aws for AWS Auto Scaling Groups or fleeting-plugin-googlecompute for Google Cloud. Rather than attempting a monolithic abstraction across many providers the way Docker Machine did, Fleeting delegates the actual provider communication to dedicated, independently maintained plugins that stay closer to each cloud provider's own native autoscaling mechanisms.

Concretely, the AWS Fleeting plugin, for example, uses an actual AWS Auto Scaling Group as its backend, instead of creating every instance individually via API call the way Docker Machine did. This means many proven cloud-native mechanisms, such as spot instance support, lifecycle hooks, and health checks, come directly from the cloud provider itself instead of having to be reimplemented by the runner, which noticeably improves reliability over the older model.

# Excerpt from /etc/gitlab-runner/config.toml (Fleeting, current)
[[runners]]
  name = "fleeting-runner"
  executor = "docker-autoscaler"
  [runners.autoscaler]
    plugin = "fleeting-plugin-aws"
    capacity_per_instance = 1
    max_use_count = 20
    max_instances = 10
    [runners.autoscaler.plugin_config]
      name = "gitlab-runner-asg"
      profile = "default"
    [runners.autoscaler.connector_config]
      username = "ec2-user"
      use_external_addr = true
    [[runners.autoscaler.policy]]
      idle_count = 2
      idle_time = "10m"

5. From Docker Machine to Fleeting: what actually changes

The executor name changes from docker+machine to docker-autoscaler, and the previously central configuration in the runners.machine block moves into a new runners.autoscaler block with plugin-specific subsections. Anyone already running a working Docker Machine configuration cannot simply copy the old values one to one, since Fleeting plugins sometimes use different concepts, such as Auto Scaling Groups instead of individual instance creation, which also requires adjusting the underlying cloud infrastructure.

In practice a gradual migration is recommended: a new Fleeting-based runner is set up alongside the existing Docker Machine runner and initially targeted only at a subset of projects or via tags. After an observation period confirming reliability and cost behavior with the new model, the old Docker Machine runner can be phased out gradually, without requiring a hard cutover on a single day.

6. Cost control through the right autoscaling parameters

The most important cost levers are idle_count, idle_time, and max_instances. Too high an idle_count keeps more machines available at all times than necessary, producing unnecessary baseline cost, while too low a value means almost every job has to boot a fresh machine, which injects noticeable delay into every pipeline through instance boot time. The right value depends heavily on the typical pipeline pattern and should be tuned against real usage data, not gut feeling.

max_instances caps the absolute upper limit of simultaneously running machines and is the most important safeguard against cost blowouts, for example when a faulty trigger suddenly fires hundreds of pipelines at once. Without this cap, an autoscaler would in theory try to spin up a new machine for every waiting job, which with usage-based cloud billing can quickly turn into an unpleasant surprise on the next invoice.

# Cost-conscious autoscaling policy in config.toml
[runners.autoscaler]
  max_instances = 15
  max_use_count = 30
  [[runners.autoscaler.policy]]
    idle_count = 1
    idle_time = "5m"
    scale_factor = 0.0
    scale_factor_limit = 5

7. Spot instances and further cost optimizations

For non-time-critical jobs like lint checks or non-production test suites, spot instances are worth using: cloud providers offer them at significantly reduced prices, but can reclaim them at any time with short notice. Fleeting plugins like fleeting-plugin-aws support spot instances natively through the underlying Auto Scaling Group, which makes integration considerably simpler than manually managing spot instances on a Docker Machine basis.

It is also worth deliberately separating runners by job criticality: production deploy jobs run on regular, guaranteed on-demand instances via a dedicated runner with corresponding tags, while the bulk of build and test jobs are handled by a separate, spot-based runner. If a spot instance disappears mid-job, GitLab generally retries the job automatically on a new machine, which is an acceptable trade-off between cost and reliability for non-critical jobs.

8. Monitoring: making autoscaling behavior visible

Without monitoring, autoscaling remains a black-box mechanism where it is unclear whether the chosen idle_count and idle_time values actually fit, and whether unexpected cost spikes are occurring. GitLab Runner exposes Prometheus metrics, including counts of active, provisioning, and terminating machines, which can be visualized in a Grafana dashboard alongside the cloud provider's own cost metrics.

Sensible alerting reacts to two extreme situations: a persistently high job queue length, which points to overly restrictive max_instances values, and an unexpectedly high number of simultaneously running instances, which could indicate a faulty trigger or misuse. Together, both signals allow the autoscaling parameters to be iteratively refined instead of being set once and never touched again.

9. Docker Machine or Fleeting: the decision at a glance

For new runner setups there is hardly any good reason left to start with Docker Machine, since the tool is no longer actively developed and Fleeting now provides mature plugins for the common cloud providers. Existing Docker Machine runners do not necessarily need to be replaced immediately, but should be prepared for migration in the medium term before compatibility issues arise with newer cloud APIs or GitLab Runner versions.

The table below summarizes the key differences between both models and shows what to watch for when planning a migration.

10. Summary

11. FAQ: Runner Autoscaling: Key Takeaways