CPU Throttling in Containers: Detect It and Set CPU Limits Correctly

1. What CPU throttling technically means

CPU throttling happens when a process requests more CPU time within a billing period than its configured quota allows, and the Linux scheduler then pauses the process entirely for the rest of that period, regardless of whether free CPU capacity is currently available on the host. That is the crucial difference from a normally overloaded CPU: even if eight out of twelve host cores sit completely idle, a container with a tight limit still gets throttled, because throttling depends exclusively on its own cgroup's quota, not on actual host utilization.

The underlying mechanism is called the Completely Fair Scheduler, CFS for short, and it works in fixed time windows, 100 milliseconds long by default. Within each window a container may consume as much CPU time as its quota allows. If the quota is already exhausted within the first 20 milliseconds of a 100-millisecond window, for example because several threads were briefly active at once, the kernel pauses the container for the remaining 80 milliseconds, entirely regardless of how urgent the request actually was.

2. cpu.stat: the file that makes throttling visible

The kernel file cpu.stat, which exists for every cgroup under /sys/fs/cgroup/.../cpu.stat, contains exactly the counters that prove throttling. The field nr_periods counts how many CFS time windows have been observed in total since the cgroup was created. nr_throttled counts in how many of those windows the container was actually throttled because its quota ran out early. throttled_usec sums the total time in microseconds the container has spent in a throttled state since the cgroup was created.

The ratio of nr_throttled to nr_periods is the single most informative value for deciding whether a CPU limit is too tight: if that ratio consistently sits above a few percent, the container is throttling regularly, even if its average CPU usage looks harmless in docker stats, because averages taken over several seconds smooth out short, hard throttling spikes and make them practically invisible.

# Read cpu.stat of a running container
CID=$(docker inspect --format '{{.Id}}' my-container)
cat /sys/fs/cgroup/system.slice/docker-${CID}.scope/cpu.stat

# Example output:
# usage_usec 48213942
# user_usec 41022103
# system_usec 7191839
# nr_periods 128841
# nr_throttled 9532
# throttled_usec 4821394

# Rough throttling ratio: 9532 / 128841 = ~7.4 percent of all periods

3. CPU requests and CPU limits are two different things

A common misunderstanding treats CPU requests and CPU limits as the same concept under different names. A request, mapped in Docker via --cpu-shares, only defines a relative priority when multiple containers compete for the same CPU time, but it sets no absolute upper bound. A container with a high share value gets allocated more CPU time than one with a lower value whenever there is competition, but as long as there is no competition, it can freely use the full available host CPU.

A limit, on the other hand, implemented through the quota in cpu.max and the Docker flag --cpus, sets a hard, absolute ceiling that applies regardless of the rest of the host's utilization. That is exactly where the most common mistake happens: a team sets a limit to make resource usage predictable, but picks a value below the application's actual peak load, and thereby produces artificial throttling in exactly the moments the application needs the most performance.

4. Typical symptoms in production

CPU throttling rarely shows up as an obvious error, it shows up as a diffuse performance problem: latency spikes that appear seemingly at random, requests that occasionally take far longer than the median with no visible relationship to overall load, and health checks that fail sporadically even though the application, according to its logs, keeps running normally. What makes it especially tricky is that classic CPU usage dashboards often hide these spikes completely, because they are averaged over seconds or minutes, while a single throttling event lasts only a few milliseconds.

In environments with synchronous request handling, for example classic web frameworks without asynchronous I/O, the problem intensifies further: a throttled thread does not just block itself, it potentially blocks the entire worker, turning a brief CFS throttling event of a few milliseconds into a noticeable double-digit millisecond latency spike, as soon as multiple requests within the same period compete for the same tight quota.

5. Measuring throttling systematically instead of guessing

Instead of immediately suspecting application code for vague latency issues, a systematic first step pays off: reading cpu.stat before and after a load spike and computing the difference in nr_throttled and throttled_usec. If both values rise noticeably during the load spike, CPU throttling is a very likely contributing factor, regardless of what application metrics say about CPU usage in percent.

For ongoing monitoring, cAdvisor is a good fit, extracting the metric container_cpu_cfs_throttled_periods_total directly from the cgroup statistics and exposing it through Prometheus as a time series. A Grafana panel that shows this metric alongside classic CPU usage reliably surfaces throttling problems that pure usage dashboards left completely invisible, because high throttling rates can well occur at low average usage.

# Compare throttling before and after a load spike
CG=/sys/fs/cgroup/system.slice/docker-${CID}.scope

echo "Before:"; grep -E 'nr_throttled|throttled_usec' $CG/cpu.stat
# ... trigger a load spike, e.g. via a load test ...
echo "After:"; grep -E 'nr_throttled|throttled_usec' $CG/cpu.stat

# Prometheus query for ongoing monitoring
# rate(container_cpu_cfs_throttled_periods_total[5m])
#   / rate(container_cpu_cfs_periods_total[5m])

6. Sizing CPU limits correctly

The basic rule for sensible CPU limits is the same as for memory limits: measure first, then limit, not the other way around. A container should first be observed without a limit, or with a very generous one, under realistic peak load, ideally with a load test that simulates short bursts rather than constant steady load, because short, intense load spikes are especially prone to throttling. Only with these measurements in hand does it make sense to set a limit that covers the actual peak load with sufficient headroom.

A concrete calculation illustrates the quota mechanics: a limit of --cpus=2 produces a quota of 200000 microseconds against a default period of 100000 microseconds. If the application runs with four parallel threads, each briefly active on different cores, the combined CPU time of all four threads can exhaust the quota within a fraction of the period, even if no single thread computes continuously. Multi-threaded applications therefore tend to need more generous limits than pure average consumption would suggest.

# Calculation example: --cpus=2 translated into cpu.max
docker run -d --name calc-test --cpus=2 nginx:alpine
CID=$(docker inspect --format '{{.Id}}' calc-test)
cat /sys/fs/cgroup/system.slice/docker-${CID}.scope/cpu.max
# Output: 200000 100000  --> 2 full CPU cores per 100ms window

# The period itself can theoretically be adjusted (rarely useful),
# it usually stays at the default of 100000 microseconds

7. Alternative strategy: shares instead of hard limits

For latency-critical applications where even occasional throttling is unacceptable, one alternative is to skip hard CPU limits entirely and rely solely on CPU shares. Without a limit, a container can use the full available host CPU at any time, while shares only kick in during actual resource competition and then ensure a fair, proportional distribution, instead of applying a hard, time-window-bound brake.

This strategy works best on dedicated or clearly overprovisioned hosts where enough combined CPU capacity exists for all containers, so that a single container could theoretically claim more CPU during a malfunction, but in practice this rarely becomes a problem for other workloads. On densely packed, heavily utilized multi-tenant hosts, on the other hand, a hard limit is usually indispensable, to prevent a single misbehaving container from crowding out every other container.

8. Common mistakes in limit configuration

A particularly widespread mistake is copying CPU limits wholesale from a Kubernetes or Docker Compose template without adapting them to the actual application, for example a generic cpus: 0.5 applied to every microservice regardless of whether that service performs compute-heavy work or just lightweight I/O forwarding. A second common mistake is setting limits once at initial deployment and never revisiting them, even though an application's actual resource needs change over months as new features and growing data volumes are added.

A third, subtler mistake involves languages and frameworks that query the number of available CPU cores themselves for sizing internal thread pools: if that query returns the host's physical core count instead of the effective core count set through cpu.max, the application sizes its own thread pools too large, which further increases the likelihood of throttling, because more threads end up competing simultaneously for the same, much smaller quota than originally intended.

9. A practical workflow from symptom to fix

A proven workflow starts with the symptom, usually unexplained latency spikes despite low average CPU usage, and moves through cpu.stat to confirm whether throttling is actually occurring. Once confirmed, the next step is measuring the real peak load without a limit, followed by resizing the limit with sufficient headroom, and finally setting up ongoing monitoring of the throttling rate to catch future regressions early, before they show up again as diffuse latency problems.

The table below compares the key metrics from cpu.stat with their respective meaning, as a quick reference for the next performance analysis of a container whose CPU usage looks harmless on the dashboard, while its users still complain about delays.

10. Summary

11. FAQ: Detecting CPU Throttling: The Essentials at a Glance