Traffic Shaping With tc and qdisc: Steering Bandwidth With Purpose

1. Why unthrottled backup traffic slows down production traffic

A nightly backup job transferring several hundred gigabytes over a server's network connection to a remote storage server directly competes with production HTTP and database traffic for the same physical bandwidth. Without prioritization, the backup process fills the network card's send queue to the point where time critical packets, such as responses to incoming shop requests, get stuck in the same queue and are delivered noticeably later.

This phenomenon is known as bufferbloat and happens because many network cards and switches use generously sized buffers that avoid packet loss during short lived load spikes, but cause growing latency on continuously saturated links, since packets sit in the queue for a long time before they even get sent. Linux Traffic Control addresses exactly this problem by actively controlling the order and rate at which packets leave the system, instead of leaving it to the network card's uncontrolled default behavior.

2. Fundamentals: what tc and a qdisc actually are

The tc command, Traffic Control, is the user interface to the queueing disciplines in the Linux kernel, known as qdiscs. Every network interface has a root qdisc by default that determines the order and priority in which outgoing packets are actually sent before they physically leave the network card. Without explicit configuration, the kernel uses pfifo_fast or, more commonly on newer versions, fq_codel as a sensible default.

Qdiscs split into classless and classful. Classless qdiscs such as fq_codel or tbf treat all traffic on an interface under a single strategy, while classful qdiscs such as HTB build a tree of classes, each of which can be assigned its own bandwidth limits and priorities. For targeted shaping, where different services should be treated differently, a classful qdisc like HTB combined with filters is the right choice.

# Show the currently configured qdisc on an interface
tc qdisc show dev eth0

# Show statistics including dropped packets per qdisc
tc -s qdisc show dev eth0

3. HTB: hierarchical bandwidth management with classes

The Hierarchical Token Bucket, HTB for short, is the most common qdisc for differentiated bandwidth management, since it allows a tree of classes each with their own rate and ceil value. The rate value defines a class's guaranteed minimum bandwidth, while ceil sets the maximum bandwidth a class may use as long as no other class currently needs its guaranteed rate. This combination lets unused bandwidth be dynamically redistributed to classes with high demand instead of reserving it rigidly.

Within the HTB hierarchy, child classes are created for individual services, for example a class for production HTTP traffic with a high guaranteed rate and a separate class for backup traffic with a low guaranteed but flexible maximum rate. The actual assignment of a packet to a class is not handled by HTB itself, but by a downstream filter, usually based on u32 rules or, more commonly today, firewall marks previously set with iptables or nftables.

# HTB root qdisc with default class 30 for unclassified traffic
tc qdisc add dev eth0 root handle 1: htb default 30

# Parent class for the interface's total bandwidth
tc class add dev eth0 parent 1: classid 1:1 htb rate 1000mbit ceil 1000mbit

# Class for production HTTP traffic, high guaranteed rate
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 800mbit ceil 1000mbit prio 1

# Class for backup traffic, low guaranteed but capped maximum rate
tc class add dev eth0 parent 1:1 classid 1:20 htb rate 100mbit ceil 200mbit prio 3

4. fq_codel: the modern default against bufferbloat

fq_codel combines fair queueing, which spreads traffic from different flows evenly across several internal queues, with the CoDel algorithm, which deliberately drops packets once their time spent in the queue crosses a certain threshold. This approach noticeably reduces latency under load without significantly hurting maximum throughput, which is why fq_codel has served as the default qdisc for many distributions for several kernel versions now.

Unlike HTB, fq_codel offers no explicit bandwidth cap, instead primarily optimizing latency and fairness among competing connections. In practice, fq_codel is therefore often used as a child qdisc within an HTB class, so the class itself sets the hard bandwidth limit for, say, backup traffic, while fq_codel ensures fair distribution among individual TCP connections and low latency within that limit.

# Use fq_codel as a child qdisc of the backup class for fair distribution
tc qdisc add dev eth0 parent 1:20 handle 20: fq_codel

5. tbf: simple rate limiting without a class hierarchy

The Token Bucket Filter, tbf for short, suits scenarios that do not need a differentiated class hierarchy, but simply require the total rate of an interface or a single process to be capped at a fixed value. The principle is based on a virtual bucket continuously filled with tokens, where every byte sent consumes a token, so the effective send rate matches the configured rate exactly as long as enough tokens are available.

The burst parameter determines how many tokens can accumulate in the bucket before excess capacity is lost, and thereby influences how tolerant tbf is toward short load spikes above the configured rate. For a simple, isolated backup interface or a dedicated VPN interface used exclusively by a single backup process, tbf is often the simpler alternative to a full HTB hierarchy.

# Cap outgoing rate on a dedicated backup interface
tc qdisc add dev eth1 root tbf rate 100mbit burst 32kbit latency 400ms

6. Classifying traffic deliberately: firewall marks instead of rigid u32 filters

Classic tc filters using u32 syntax allow direct matching by IP address or port, but are considered error prone and hard to read once several criteria need to be combined. In practice, a two stage approach has become the standard: first, an iptables or nftables rule marks packets based on port, process, or cgroup with a firewall mark, then a simple tc filter of type fw picks up that mark and assigns the packet to the matching HTB class.

This approach makes classification logic considerably more maintainable, since it lives in one central place inside the firewall rule set instead of scattered tc filter definitions, and it integrates cleanly with firewall rules that already exist for other purposes. For a backup process communicating over a fixed port or a dedicated source IP, a single marking rule is enough to reliably assign all its traffic to the backup class.

# Tag backup traffic with a firewall mark based on the destination port
iptables -t mangle -A OUTPUT -p tcp --dport 873 -j MARK --set-mark 20

# tc filter that maps the mark to the matching HTB class
tc filter add dev eth0 parent 1: protocol ip prio 1 handle 20 fw classid 1:20

7. Throttling inbound traffic: ingress policing instead of egress shaping

All the qdisc types shown so far only affect outgoing traffic, since tc can conceptually only control an interface's send queue, not packets already received over the physical medium. For inbound traffic, for example to cap a single client's large file download before it saturates a server's internal network capacity, the special ingress qdisc combined with a police action is used instead.

Policing differs fundamentally from shaping: instead of delaying excess packets in a queue, policing drops packets above the configured rate immediately, which can increase TCP retransmission rates but requires no additional memory. For even finer control over inbound traffic, including real queueing instead of hard dropping, an IFB device, Intermediate Functional Block, can be used, which virtually redirects inbound traffic so it can be treated like outbound traffic with full featured qdiscs like HTB.

# Simple policing capping inbound traffic at 500mbit
tc qdisc add dev eth0 handle ffff: ingress
tc filter add dev eth0 parent ffff: protocol ip u32 match u32 0 0 \
  police rate 500mbit burst 64k drop flowid :1

# Alternative using IFB for full featured shaping of inbound traffic
modprobe ifb numifbs=1
ip link set dev ifb0 up
tc qdisc add dev eth0 handle ffff: ingress
tc filter add dev eth0 parent ffff: matchall action mirred egress redirect dev ifb0
tc qdisc add dev ifb0 root handle 1: htb default 10

8. Practical example: throttling backup traffic without hurting production

A complete setup for a Magento hosting server combines the previous building blocks into a consistent configuration: HTB as the root qdisc with a high priority class for web and database traffic, plus a separately capped class for rsync or Borg backup traffic, paired with fq_codel inside the backup class for fair distribution across multiple parallel backup streams. Firewall marks based on the backup destination port handle classification reliably without touching the production class.

It matters that the default class in the HTB hierarchy, class 30 in the example, receives a sensible rate, so unclassified traffic is neither blocked entirely nor prioritized without limit. In practice it works well to set the default class somewhere between production traffic and backup traffic, so newly appearing services that are not yet explicitly classified neither crowd out backup traffic nor get crowded out by it themselves.

9. Monitoring and fine tuning the shaping configuration

The command tc -s class show dev eth0 provides detailed per class statistics on transferred bytes, dropped packets, and the time a class has spent above its guaranteed rate. A consistently high number of dropped packets in the production class suggests the configured rates no longer match actual load distribution and should be adjusted before users experience it as page load problems.

For ongoing supervision, it is worth regularly evaluating these statistics through a monitoring system, combined with active latency measurements during known backup windows to verify the shaping configuration actually works as intended. Changes to rate and ceil values should be applied gradually, since throttling backup traffic too aggressively can unnecessarily extend backup windows and, in the worst case, collide with the next scheduled run.

# Show detailed class statistics including drop counters
tc -s class show dev eth0

# List current filter assignments for verification
tc filter show dev eth0

10. Summary

11. FAQ: Traffic Shaping