IRQ Affinity and Network Interrupt Tuning for Multi-Core Servers

1. Why IRQ affinity matters on high-traffic servers

Every incoming network packet has to notify the CPU via a hardware interrupt (IRQ) that data is ready. On a server with a single network card and a single interrupt queue, all these interrupts land by default on exactly one CPU core, regardless of how many cores the server has in total. At low packet rates this is not a problem, but on a Magento server with many concurrent users and correspondingly many TCP connections, that one core can quickly reach 100 percent utilization handling interrupts, while the remaining cores stay nearly idle.

This is exactly where IRQ affinity comes in: the deliberate assignment of which CPU core handles which interrupt. Modern network cards offer multiple hardware queues for this purpose, so interrupts can be spread across several cores from the start instead of being bundled onto a single one. Without deliberate IRQ affinity configuration, this potential often goes unused, because the default setting does not always match the server's actual core count and workload distribution.

This article shows how to analyze interrupt distribution, how to configure IRQ affinity manually and automatically, and how receive side scaling together with interrupt coalescing spread network load evenly across all cores of a Magento web server.

2. How interrupts and kernel interrupt handling work

A hardware interrupt briefly suspends a CPU core's normal execution so it can react immediately to an event, for example an incoming network packet. The kernel processes an interrupt in two phases: the so-called top half, which is minimal and extremely fast and only handles the bare essentials, and the bottom half, or softirq, which takes on the actual, more involved processing, such as handing the packet off to the network stack.

When all network interrupts land on a single core without IRQ affinity configuration, that core has to handle both the top half and a substantial share of softirq processing. Under high packet rates this shows up in the kernel metric si (software interrupt) in tools like top, where a single core displays an unusually high softirq share while other cores are barely affected. This imbalance is a direct symptom of missing or misconfigured IRQ affinity.

3. Analyzing interrupt distribution with /proc/interrupts

The file /proc/interrupts shows, for every interrupt, how many events each CPU core has processed so far. For a network card with multiple queues, one line typically appears per queue, named after the interface and queue number. If the counters in only one or two columns rise sharply and continuously while the remaining columns stay practically flat, that is a clear sign of unfavorable IRQ affinity.

Repeated snapshots a few seconds apart, combined with a diff of the counter values, reveal the actual distribution during a specific load window far more precisely than a single point-in-time reading. For servers with multiple network cards or bonded interfaces, this analysis should be done separately per physical card, because imbalances can affect each card differently.

# Show interrupt counters per CPU core for a specific interface
grep eth0 /proc/interrupts

# Take two snapshots 5 seconds apart to see live distribution
cat /proc/interrupts > /tmp/irq_before
sleep 5
cat /proc/interrupts > /tmp/irq_after
diff /tmp/irq_before /tmp/irq_after | grep eth0

# List all IRQ numbers assigned to a given network interface
cat /sys/class/net/eth0/device/msi_irqs 2>/dev/null

4. Understanding irqbalance and when to disable it

Most Linux distributions run the irqbalance service by default, which tries to distribute interrupts automatically and dynamically across available CPU cores. For general server workloads with fluctuating load, that is a sensible compromise without manual configuration. On dedicated high-traffic web servers with predictable traffic patterns, however, irqbalance can itself become a problem, because its automatic rebalancing periodically destroys CPU cache warmth, briefly causing more latency than a static, carefully planned IRQ affinity configuration would.

On servers where certain CPU cores should be deliberately reserved for network interrupts while other cores are available exclusively for PHP-FPM workers or MySQL, it is therefore common to disable irqbalance and instead set IRQ affinity statically and explicitly via smp_affinity. This decision should only be made after analyzing the actual interrupt distribution, not applied blanket-style to every server.

# Check whether irqbalance is currently running
systemctl status irqbalance --no-pager

# Disable irqbalance for static, manual IRQ affinity control
systemctl stop irqbalance
systemctl disable irqbalance

5. Setting IRQ affinity manually with smp_affinity

Every interrupt has, under /proc/irq/<number>/smp_affinity, a bitmask that defines which CPU cores are allowed to handle it. This file allows precise, manual configuration of IRQ affinity, for example to deliberately restrict network interrupts to the first four cores of an eight-core server, while the remaining four cores process only application workloads.

A commonly used strategy is to pin each interrupt queue of a network card to exactly one dedicated core, instead of allowing multiple cores per queue. This prevents a single packet from migrating between cores and losing CPU cache locality in the process, an effect that measurably contributes to latency, particularly at very high packet rates.

#!/usr/bin/env bash
# bind-irqs.sh — pin each NIC queue's IRQ to one dedicated CPU core
set -euo pipefail

IFACE="eth0"
CORES=(0 1 2 3)

irqs=($(grep "$IFACE" /proc/interrupts | awk -F: '{print $1}'))

for i in "${!irqs[@]}"; do
  irq="${irqs[$i]}"
  core="${CORES[$((i % ${#CORES[@]}))]}"
  mask=$(printf "%x" $((1 << core)))
  echo "$mask" > "/proc/irq/${irq}/smp_affinity"
  echo "IRQ $irq -> core $core (mask $mask)"
done

6. RSS: receive side scaling for multi-core distribution

Receive Side Scaling (RSS) is a hardware feature of modern network cards that spreads incoming packets across multiple hardware queues based on a hash of source and destination IP plus port, already on the card itself, before any interrupt is even raised. Combined with matching per-queue IRQ affinity, this creates end-to-end parallelism from the network card all the way to the processing CPU core, without packets of a single TCP connection landing on different cores and thereby risking packet ordering or cache locality.

The number of RSS queues a card supports can be queried with ethtool -l and, if the hardware allows it, adjusted with ethtool -L to match the server's actual core count. On servers with significantly more cores than the card's supported queue count, some cores will necessarily be excluded from direct network processing, which should be factored into IRQ affinity planning.

# Show current and maximum RSS queue counts
ethtool -l eth0
# Channel parameters for eth0:
# Pre-set maximums:
# Combined: 8
# Current hardware settings:
# Combined: 4

# Increase to match the server's core count (if hardware supports it)
ethtool -L eth0 combined 8

7. Fine-tuning interrupt coalescing with ethtool

Without any optimization, every single network packet would trigger its own interrupt, which under very high packet rates leads to an interrupt storm that keeps the CPU busier with interrupt handling than with actual application work. Interrupt coalescing bundles several packets into a single interrupt, either after a fixed time span or a fixed packet count, which drastically reduces the interrupt rate, at the cost of slightly higher latency per packet.

These values can be read with ethtool -c and adjusted with ethtool -C. For latency-sensitive applications a lower coalescing value is preferred, for throughput-oriented batch workloads a higher value. On a Magento web server processing many short HTTP requests, a moderate middle ground is usually appropriate, one that neither generates unnecessarily many interrupts nor introduces noticeable extra latency per request.

# Show current interrupt coalescing settings
ethtool -c eth0

# Lower coalescing values reduce added latency per packet,
# at the cost of a higher interrupt rate under heavy load
ethtool -C eth0 rx-usecs 20 rx-frames 8

8. Common mistakes in IRQ tuning

The most common mistake is configuring IRQ affinity manually without first disabling irqbalance. Both mechanisms then compete with each other, and irqbalance periodically overwrites the manually set smp_affinity values again, without any obvious error message pointing to this issue. A second common mistake is binding network interrupts to the same cores that also run the most compute-intensive PHP-FPM workers, causing both workloads to compete for the same cores instead of relieving each other.

A third mistake concerns NUMA systems: if network interrupts are bound to cores on a different NUMA node than the one the network card is physically attached to, additional, avoidable remote memory accesses occur for every single packet. The physical node assignment of a network card can be determined with cat /sys/class/net/eth0/device/numa_node and should always be factored into IRQ planning.

9. IRQ strategies compared

The following table compares the main approaches to IRQ affinity on multi-core servers.

For most Magento web servers with moderate to high packet rates, static smp_affinity combined with enough RSS queues is the best compromise between control and maintenance overhead. Only at very high, consistently constant packet rates does dedicating entire cores exclusively to network interrupts pay off.

10. Summary

IRQ affinity determines whether network interrupts are spread evenly across all CPU cores of a server or bundled onto a single core that becomes a bottleneck long before the rest of the CPU. /proc/interrupts reveals the actual distribution, irqbalance offers a dynamic default solution, and smp_affinity enables a precise, static configuration for dedicated high-traffic servers.

Receive side scaling distributes packets across multiple queues already at the hardware level, while interrupt coalescing reduces the overall interrupt rate. Both mechanisms complement a well-thought-out IRQ affinity configuration, but must always be planned together with the server's NUMA topology to avoid additional remote memory accesses.

11. FAQ: IRQ Affinity and Network Interrupt Tuning