cgroups v2 Controllers in Detail: Limiting CPU, Memory, and I/O Precisely

1. The unified hierarchy as the foundation of cgroups v2

cgroups v1 organized every controller into its own, independent hierarchy, which meant a process could sit in several unrelated tree structures for CPU, memory, and blkio at the same time, with correspondingly complex and error prone bookkeeping. cgroups v2 does away with that fragmentation and introduces a single, unified hierarchy in which every process is assigned to exactly one node, with all relevant controllers activated inside that same tree.

This unified hierarchy is typically mounted under /sys/fs/cgroup/ and managed automatically by systemd as the init system, which creates its own cgroup for every service, scope, and slice unit. For administrators this means resource limits can largely be configured today directly through systemd unit directives, without touching the cgroup filesystem structure under /sys/fs/cgroup/ by hand, even though the underlying model still exposes plain virtual files for every controller.

# Check whether cgroups v2 is active as the unified hierarchy
mount | grep cgroup2

# List the controllers available for the root cgroup
cat /sys/fs/cgroup/cgroup.controllers

2. The CPU controller: quota, weight, and bursting

The cpu controller governs CPU allocation through two complementary mechanisms: cpu.max sets a hard ceiling as a quota and period pair, typically expressed as a number of microseconds of CPU time per defined period, while cpu.weight controls relative priority when several cgroups at the same level compete for available CPU time, ranging from 1 to 10000 by default with a default value of 100.

These two mechanisms complement each other: a quota limit absolutely caps how much CPU time a cgroup may claim at most, regardless of whether the CPU would otherwise sit idle, while the weight only kicks in when actual competition for CPU time exists. Since kernel 5.14, cpu.max.burst additionally supports briefly exceeding the quota within a predefined burst budget, which is more realistic than a hard limit with zero tolerance for workloads with occasional, short load spikes, such as PHP FPM processes hitting a cache miss.

# Limit CPU quota to 50 percent of one CPU (50000 out of 100000 microseconds)
echo "50000 100000" > /sys/fs/cgroup/shop-php.slice/cpu.max

# Set relative priority against other cgroups at the same level
echo 200 > /sys/fs/cgroup/shop-php.slice/cpu.weight

3. The memory controller: from low to OOM

The memory controller offers four staged thresholds instead of a single limit: memory.low defines a protection floor below which memory only gets reclaimed under system wide pressure, memory.high sets a soft ceiling beyond which the kernel actively throttles the process and forces reclaim, memory.max is the hard ceiling whose breach triggers an out of memory kill inside the cgroup, and memory.min guarantees a minimum amount that never gets reclaimed even under system wide memory pressure.

This staging allows considerably finer control than the binary limit cgroups v1 offered: a memory.high value set just below memory.max gives a process the chance to actively counteract via reclaim and throttling before the hard OOM killer strikes, which makes the decisive difference between a controlled slowdown and an abrupt crash for database or PHP processes with internal cache management.

# Set a hard memory ceiling and a soft throttling threshold
echo "900M" > /sys/fs/cgroup/shop-php.slice/memory.max
echo "750M" > /sys/fs/cgroup/shop-php.slice/memory.high

# Check current memory usage and OOM events
cat /sys/fs/cgroup/shop-php.slice/memory.current
cat /sys/fs/cgroup/shop-php.slice/memory.events

4. The I/O controller: bandwidth and IOPS per device

The io controller limits read and write throughput as well as IOPS per block device via io.max, with every device addressed individually through its major and minor number, since different physical or virtual storage devices carry different performance profiles and a blanket limit across all devices would be nearly meaningless. Separate limits are configurable for rbps and wbps (read and write bandwidth in bytes per second) as well as riops and wiops.

For relative prioritization between several cgroups under I/O contention, io.weight exists as well, analogous to the CPU weight, kicking in only when actual I/O saturation occurs. In practice, the I/O controller matters especially for database workloads on shared storage, where a single container with excessive write behavior could otherwise drive up I/O latency for every other cgroup sharing the same physical storage device.

# Determine the device major:minor number
lsblk -d -o NAME,MAJ:MIN

# Limit read and write bandwidth for device 8:0
echo "8:0 rbps=50000000 wbps=20000000" > /sys/fs/cgroup/shop-db.slice/io.max

5. Setting limits on the fly with systemd-run

Rather than editing the virtual cgroup files directly, which is error prone and gets lost on process restart, systemd-run offers a much more convenient way to launch a command directly as a transient systemd unit with defined resource limits. The relevant properties like MemoryMax, CPUQuota, and IOWriteBandwidthMax map one to one onto the underlying cgroup v2 controller files, but systemd expresses them in human readable units instead of raw byte or microsecond values.

This approach is excellent for quickly testing how a process behaves under a given limit, for example a database import under a reduced memory budget, without having to create a permanent systemd unit file right away. For production, long lived limits, the same properties belong in a regular unit file under /etc/systemd/system/ instead.

# Run a command transiently with a CPU and memory limit
systemd-run --scope -p CPUQuota=50% -p MemoryMax=512M \
  mysqldump shop_db > /srv/backup/shop_db.sql

# Limit I/O bandwidth for a single command
systemd-run --scope -p IOWriteBandwidthMax="/dev/sda 10M" \
  rsync -a /srv/data/ /srv/backup/

6. Permanent limits through systemd unit directives

For long lived services, resource limits belong directly in the systemd unit file, either in the [Service] section of a regular service unit or centrally in a slice unit that several related services attach to. Slice units form a hierarchical grouping: a limit set at the slice level automatically caps the sum of all services running inside it, which works great for assigning a shared total budget to, say, all of one customer's PHP FPM workers together, regardless of how many individual worker processes happen to be active.

Changes to unit directives require a systemctl daemon-reload followed by a restart after editing, or, for already running services, many properties can also be adjusted live via systemctl set-property without restarting the service itself, since systemd passes the change straight through to the corresponding cgroup file.

# /etc/systemd/system/shop-php.slice
[Unit]
Description=Resource slice for all PHP FPM workers of the shop

[Slice]
CPUQuota=200%
MemoryMax=2G
IOWeight=100

7. Delegation: nested cgroups for container runtimes

Container runtimes like Docker or Podman create their own nested cgroup for every container underneath the systemd managed hierarchy, which requires systemd to explicitly hand control over certain controllers to that subtree, a process systemd calls delegation. Without delegation enabled, the container runtime cannot set further limits for individual containers inside its own cgroup branch, since systemd keeps controller control at the top level by default.

On modern systemd versions, delegation for the cpu, memory, io, and pids controllers inside a unit marked Delegate=yes is already correctly configured for Docker and Podman out of the box, as long as the runtime uses systemd as its cgroup driver. Anyone creating custom slice units for container groups needs to set this delegation explicitly, otherwise resource limits the runtime itself tries to set will fail with a permission denied error.

# /etc/systemd/system/shop-containers.slice
[Slice]
CPUQuota=400%
MemoryMax=4G

[Unit]
# Allows the container runtime to set its own limits inside this slice

8. How this maps to Docker and Kubernetes resource limits

Docker CLI flags such as --cpus=1.5 and --memory=512m translate internally one to one into the cgroups v2 properties cpu.max and memory.max for the cgroup the container runtime creates, with the difference being that Docker abstracts away the human readable syntax and Kubernetes layers yet another abstraction on top with resources.requests and resources.limits. A Kubernetes limits.memory: 512Mi ultimately lands as exactly the same memory.max value in the underlying pod cgroup that you could also set manually via systemd.

The practical value of understanding the underlying cgroup layer shows up during debugging: when a Kubernetes pod gets OOM killed unexpectedly despite a seemingly generous memory limit, a direct look at memory.current and memory.events for the corresponding cgroup under /sys/fs/cgroup/kubepods.slice/ often gets you an answer faster than any Kubernetes native debugging tool, because it shows the actual kernel numbers with no abstraction layer in between.

# Determine the cgroup path of a running Docker container
docker inspect --format '{{.Id}}' shop-php
cat /sys/fs/cgroup/system.slice/docker-*.scope/memory.max

# Check current memory usage and OOM counters directly at the kernel
cat /sys/fs/cgroup/system.slice/docker-*.scope/memory.events

9. Troubleshooting: verifying limits and finding bottlenecks

When behavior looks unexpected, start with cpu.stat, which among other things reports cumulative throttled time, meaning how long a process has actually been throttled by its CPU quota. A high value in nr_throttled is a clear sign the configured CPU limit is too tight and the application regularly bumps against it, even if average CPU utilization looks unremarkable at first glance.

For memory issues, memory.events provides the decisive counters: high shows how often the soft ceiling was exceeded, max how often the hard ceiling was hit, and oom_kill the actual number of out of memory kills triggered inside the cgroup. These counters are cumulative since the cgroup was created and therefore give a more reliable historical picture than a single snapshot from free or top, which only captures the current moment.

10. Summary

11. FAQ: cgroups v2 Controllers