NUMA Considerations for Multi-CPU Servers Running Magento and MySQL

1. Why NUMA matters on multi-CPU servers

As soon as a server carries more than one physical CPU, the physics of memory access change fundamentally. Each CPU socket brings its own memory controller and is directly wired to a portion of the total RAM. When a process accesses memory attached to its own socket, the access is fast. When the CPU instead has to reach across to the memory of the other socket, noticeably higher latency appears. This exact behavior is what NUMA, short for Non-Uniform Memory Access, describes.

On a Magento server with a single CPU, NUMA plays no role at all, because there is only one memory region. But as soon as a database server runs with two or more sockets, for example to host a large InnoDB buffer pool or several highly parallel PHP-FPM pools, the NUMA placement of processes and memory directly determines whether the extra hardware is actually being used, or whether a substantial share of accesses travels over the slower remote path.

This article shows how to determine a server's NUMA topology, how MySQL and PHP-FPM react to unsuitable NUMA placement, and which concrete tools bind processes and memory to the same node on purpose.

2. Understanding NUMA architecture: nodes and memory access

A NUMA node consists of a group of CPU cores plus the RAM physically wired directly to their memory controller. A typical two-socket server forms two NUMA nodes, a four-socket server correspondingly four. Within one node, memory access is local and fast. When a process running on a core of node 0 accesses memory physically attached to node 1, the request has to travel across an interconnect link, for example Intel UPI or AMD Infinity Fabric, which costs extra cycles.

The Linux kernel is aware of this topology and by default tries to place processes and the memory they request on the same NUMA node, a behavior known as the first-touch policy: memory is allocated on the node where the requesting thread happens to be running. Problems appear when a process gets migrated between nodes, for example by the scheduler, or when a process needs more memory from the outset than a single node can provide. In both cases, permanently remote memory accesses show up in benchmarks as elevated latency without any corresponding rise in CPU utilization.

For database workloads with a large buffer pool, this second case is the norm: a 64 gigabyte buffer pool rarely fits entirely into the memory of a single NUMA node when the server distributes a total of 128 gigabytes across two nodes. Without deliberate configuration, the buffer pool then ends up spread unfavorably, with a measurable effect on the latency of individual queries.

3. Mapping your server's NUMA topology with numactl

Before any NUMA optimization makes sense, the server's actual topology must be known. The numactl package provides the necessary tools for this. The command numactl --hardware shows how many NUMA nodes exist, how much memory is assigned to each node, and how expensive access between nodes is relative to each other, expressed in the so-called distance matrix.

lscpu complements this with an overview of which CPU cores belong to which NUMA node. This mapping is required later to bind processes to the correct cores deliberately. On virtualized servers it is worth noting that the NUMA topology reported by the hypervisor does not always match the physical hardware, a vServer can report a single node even though the host has multiple sockets.

# Show NUMA nodes, memory per node, and inter-node distance
numactl --hardware
# available: 2 nodes (0-1)
# node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11
# node 0 size: 64729 MB
# node 1 cpus: 12 13 14 15 16 17 18 19 20 21 22 23
# node 1 size: 64768 MB
# node distances:
# node   0   1
#   0:  10  21
#   1:  21  10

# Show which CPU cores belong to which NUMA node
lscpu | grep -i numa
# NUMA node(s):        2
# NUMA node0 CPU(s):   0-11
# NUMA node1 CPU(s):   12-23

# Confirm actual current memory usage per node
cat /sys/devices/system/node/node*/meminfo | grep MemFree

4. How NUMA concretely affects MySQL and PHP-FPM

With MySQL, the NUMA effect shows up most clearly with the InnoDB buffer pool. If MySQL starts without any NUMA awareness on a two-node system, the kernel usually allocates the buffer pool on a single node under the first-touch policy, because the main thread happens to run there at startup. Subsequent query threads scheduled on cores of the other node then constantly access the buffer pool remotely, raising effective memory latency for those threads even though the server as a whole has plenty of free memory.

With PHP-FPM the effect is more subtle but still measurable: many parallel PHP-FPM worker processes, spread by the scheduler across both NUMA nodes, partly compete for the same interconnect bandwidth path when they simultaneously access shared memory, for example OPcache shared memory segments. On servers with very high PHP-FPM parallelism and tight CPU allocation per request, this interconnect load can effectively make part of the theoretically available compute power unreachable.

In both cases the symptom looks similar: CPU utilization looks unremarkable, but response times fluctuate more than expected, and a glance at classic metrics like load average or CPU percentage offers no explanation. Only a NUMA-specific analysis reveals how many memory accesses actually happen remotely.

5. numactl: binding processes to specific NUMA nodes

The most direct measure is to bind a process to a specific NUMA node from the very start, both for CPU allocation and memory allocation. The command numactl --cpunodebind=0 --membind=0 starts a process so it runs exclusively on the CPU cores of node 0 and requests its memory exclusively from node 0. This eliminates remote accesses entirely, as long as the process does not need more memory than the node provides.

For MySQL this means in practice: if the buffer pool is smaller than the memory of a single NUMA node, strict node binding is usually the best choice, because the entire database then consistently works locally, at the cost of the second socket's compute power remaining unused by MySQL.

# /etc/systemd/system/mysql.service.d/numa.conf
# Bind MySQL to NUMA node 0 for both CPU and memory
[Service]
ExecStart=
ExecStart=/usr/bin/numactl --cpunodebind=0 --membind=0 /usr/sbin/mysqld

systemctl daemon-reload
systemctl restart mysql

# Verify the running process is actually bound to node 0
numastat -p $(pgrep mysqld)

6. NUMA interleaving for the InnoDB buffer pool

Once the buffer pool no longer fits into a single NUMA node, strict binding is no longer an option. For this case, MySQL has offered the option innodb_numa_interleave since version 5.7, which deliberately spreads the buffer pool evenly across all available NUMA nodes instead of concentrating it on a single one. Interleaving ensures every thread experiences, on average, the same mix of local and remote accesses, instead of one subset of threads accessing remotely throughout while another subset works locally throughout.

This even distribution is no substitute for local memory access, but it prevents the worst-case scenario: a buffer pool sitting entirely on the wrong node while most requests arrive on the other node. For buffer pools significantly larger than a single node, interleaving is in practice usually the better choice over strict binding.

# /etc/mysql/mysql.conf.d/mysqld.cnf
[mysqld]
# Spread the buffer pool evenly across all NUMA nodes
# instead of concentrating it on a single node
innodb_numa_interleave = 1
innodb_buffer_pool_size = 96G
innodb_buffer_pool_instances = 12

7. Kernel parameters and BIOS settings for NUMA

The kernel parameter numa_balancing enables Automatic NUMA Balancing, a feature that tries to bring running processes and their memory pages closer together over time by periodically moving pages between nodes. For many workloads this is helpful, but for a long-running database process with a huge buffer pool, periodically moving large memory regions itself causes noticeable latency spikes, similar to transparent huge pages. On dedicated database servers with explicit NUMA configuration, automatic balancing is therefore usually disabled.

At the hardware level, many server mainboards offer a BIOS option called Node Interleaving or similar, which hides the NUMA topology from the operating system entirely and presents memory as a single, uniform zone. That sounds convenient, but it removes any possibility of targeted optimization and should stay disabled on database servers, so that Linux and MySQL can see the actual topology and respond to it.

# Check current Automatic NUMA Balancing state
cat /proc/sys/kernel/numa_balancing
# 1 = enabled

# Disable for dedicated database servers with explicit NUMA config
echo 0 > /proc/sys/kernel/numa_balancing

# Persist across reboots
echo 'kernel.numa_balancing = 0' > /etc/sysctl.d/61-numa.conf
sysctl -p /etc/sysctl.d/61-numa.conf

8. Diagnosing NUMA-related performance problems

The numastat tool shows, per process, how many memory accesses were local (numa_hit) versus how many crossed to another node (numa_foreign, other_node). A high share of other_node accesses relative to numa_hit is a clear sign that a process is placed unfavorably. For MySQL this value can be queried directly using the process ID of the mysqld process.

numastat -m complements this by showing memory distribution across all nodes system-wide, helping to reveal whether one node is disproportionately loaded while another node has large amounts of free memory the application never reaches. Combined with perf stat -e and the matching hardware counters for remote memory accesses, the problem can be narrowed down quantitatively to specific time windows, instead of relying on mere suspicion.

# Per-process NUMA hit/miss statistics for the mysqld process
numastat -p $(pgrep mysqld)
# Per-node memory statistics                           Node 0          Node 1
# numa_hit                                          48213221        41029013
# numa_miss                                                0               0
# numa_foreign                                            0               0
# other_node                                         9021113        11238042

# System-wide memory distribution across NUMA nodes
numastat -m | head -5

9. NUMA strategies compared

Which NUMA strategy makes sense for a server depends above all on how large the buffer pool is relative to the memory of a single node. The following table compares the three practically relevant options.

For the vast majority of production Magento database servers with a buffer pool that occupies a good share of total memory, interleaving combined with disabled automatic balancing is the most robust choice. Strict binding only pays off when the buffer pool is deliberately kept small enough to fit into one node, and the server runs several independent services on separate sockets anyway.

10. Summary

NUMA describes the asymmetry between local and remote memory access on multi-socket servers, and becomes entirely irrelevant on single-socket systems. As soon as a server has multiple NUMA nodes, the placement of MySQL and PHP-FPM determines how many memory accesses travel expensively across the interconnect. numactl --hardware and lscpu reveal the topology, while numastat measures the actual distribution of accesses.

For small buffer pools, strict node binding with numactl --cpunodebind --membind is the simplest and most effective solution. For large buffer pools that must span multiple nodes, innodb_numa_interleave delivers an even and predictable distribution. Automatic NUMA balancing should be disabled on dedicated database servers with explicit configuration, because it can itself cause latency spikes.

11. FAQ: NUMA Considerations for Multi-CPU Servers