How NUMA Tripled VPP’s Packet Rate on AMD EPYC

I run an FD.io VPP data plane with an ACL filter plugin on a 64-core AMD EPYC 9534. Six 100G and 200G links load it with 64-byte packets. With 62 worker threads VPP dropped only 148 Mpps; with 32 workers it dropped about 350. Adding cores made the machine slower. The cause was a single lock in VPP’s packet-buffer pool, and the fix was one BIOS setting: NUMA Nodes Per Socket from NPS1 to NPS4. After the change the same 62 workers drop 450 Mpps.

The Test Setup

	Device under test
CPU	1 × AMD EPYC 9534, single socket (Supermicro H13SSL-N board): 64 cores / 128 threads (Zen 4), 2.45 GHz base, 256 MB L3
Memory	12 × 16 GB DDR5-4800, one DIMM per channel
NICs	2 × NVIDIA ConnectX-8 (PCIe Gen5 x16) and 1 × BlueField-3 in NIC mode, six ports in all
OS	Ubuntu 24.04, kernel 6.8, isolcpus=1-63
Data plane	VPP 25.10, DPDK mlx5 driver, one receive queue per worker
Rules	five drop rules on UDP destination port ranges; every packet matches
Generator	TRex on a second host with 2 × ConnectX-7 (200G) and 3 × ConnectX-5 Ex (100G), one cable per port

The generator sends 64-byte UDP frames (64 bytes including the FCS) on all six ports at once. The source address changes with every packet, so RSS spreads the load over all receive queues. Every number below is the rate at which the filter dropped packets, read from its counters over a 10 to 20 second window.

More Workers, Fewer Packets

With the BIOS at its default of one NUMA node per socket (NPS1), I gave VPP more and more worker threads, one receive queue each, spread over the six ports. Up to 32 workers the rate grew with the core count, as a packet-processing workload should: 212 Mpps at 16, 296 at 24, about 350 at 32. Past 32 it fell, to 191 at 48 and 148 at 62, lower than with 16. The traffic was the same in every run; only the number of workers changed.

Where the Time Went

perf record on two busy workers at 62 workers showed where they spent their cycles:

Three quarters of each worker went into taking packet buffers from VPP and giving them back.

Both functions end in the same place. When the mlx5 driver refills a receive ring it asks its DPDK mempool for buffers, and VPP backs that mempool with its own buffer pool (dpdk_ops_vpp_dequeue in src/plugins/dpdk/buffer.c). vlib_buffer_alloc_from_pool first serves the request from a per-thread cache of up to 512 buffers. When the cache runs dry it refills from the shared pool, and that takes a spinlock:

clib_spinlock_lock (&bp->lock);
len = bp->n_avail;
...
clib_spinlock_unlock (&bp->lock);

Freeing works the same way in the other direction: buffers go back to the per-thread cache, and a full cache spills into the shared pool under the same lock. At 64 bytes a worker handles millions of packets a second, so it goes through its 512-buffer cache thousands of times a second. With one pool, 62 workers take turns on one lock.

The Chip

Where the cores that fight over that lock sit matters.

The EPYC 9534 is built from eight CPU dies (CCDs) of eight Zen 4 cores, each with its own 32 MB L3 cache, around a central I/O die. The I/O die holds the twelve DDR5 memory controllers and the 128 PCIe 5.0 lanes, and each CCD reaches it over its own Infinity Fabric (GMI) link. Two cores on the same CCD share an L3; two cores on different CCDs can only exchange a cache line by going through the I/O die.

The NPS setting decides how that I/O die is presented to the operating system:

• NPS1 interleaves memory over all twelve channels and reports the whole socket as one NUMA node.
• NPS4 splits the I/O die into four quadrants. Each quadrant interleaves memory over its own three channels, has its own PCIe root ports and gets the CCDs closest to it; on the EPYC 9534 that is two CCDs, 16 cores. Linux sees four NUMA nodes.

AMD’s architecture white paper says NPS1 gives excellent performance for most applications and NPS4 gives small improvements for some. On this machine it was not small.

Why NPS4 Wins

VPP creates one buffer pool per NUMA node. With NPS1 that is one pool for the whole machine, so every worker on all eight CCDs allocates from and frees into the same pool under the same lock. The more workers there are, the longer each waits, and past 32 workers the waiting costs more than the extra cores add.

With NPS4 there are four pools. The DPDK plugin picks the pool for each receive queue from the NUMA node of the NIC, not of the worker (src/plugins/dpdk/device/common.c):

u8 bpidx = vlib_buffer_pool_get_default_for_numa (
  vm, vnet_hw_if_get_rx_queue_numa_node (vnm, rxq->queue_index));

In this machine the first ConnectX-8 sits in the slot on node 2, and the second ConnectX-8 and the BlueField-3 on node 3. Their receive queues now refill from two pools instead of one: 24 workers on the node 2 pool and 38 on the node 3 pool.

What the workers actually fight over is a few cache lines: the spinlock itself, and the pool header line with n_avail, which every refill and spill writes under the lock, plus the top of the free list. A core can only write a line it owns, so each time another worker takes the lock these lines move into that worker’s CCD and every other copy is invalidated. With the default queue placement the lines still visit five or six CCDs at NPS4, because most workers poll a NIC on another node. What NPS4 changes is how many workers queue for each line.

Two things change at once: fewer workers share each lock, and the buffers the NIC writes and the workers read sit in memory attached to the same quadrant as the NIC. I did not separate the two effects. The NPS1 numbers point to the first one doing most of the work: halving the number of workers on the single lock (62 to 32) already took NPS1 from 148 to 350 Mpps.

Results

The NPS4 runs give VPP 524,288 buffers per node, 2 million in all; the NPS1 runs had one pool of 2,097,152. At NPS4 the rate grows with the worker count again, and all 62 workers drop 450 Mpps (median of three runs, 439 to 464). That is three times the NPS1 rate with the same workers and 29 % more than the best NPS1 configuration.

Per packet, the workers now spend 291 cycles in these nodes instead of 937. dpdk-input and the drop path get six to eleven times cheaper; the filter and ethernet-input get somewhat more expensive, because three times as many packets now go through the memory system. In the NPS4 profile the buffer allocation is down to 9 % of a worker’s time.

How the Load Spreads

All three cards carry the same load. What VPP drops on each depends on how many workers poll it and which buffer pool they share. The 24 workers on ConnectX-8 A have pool 2 to themselves and each drops about 8.7 Mpps. The 38 workers on ConnectX-8 B and the BlueField-3 share pool 3 and each drops about 6.3 Mpps: the busier lock again costs throughput.

The NICs sit on two of the four nodes, and VPP places the receive queues on workers in port order, so only 21 of the 62 workers run on the same node as the NIC they poll: the seven serving ConnectX-8 A port 0 on node 2 and the fourteen serving the BlueField-3 on node 3. The others reach their NIC and its buffer pool across the I/O die. Nodes 2 and 3 have only 32 cores between them, so not every queue can stay local; a placement that keeps as many as possible on their NIC’s node is the next thing I want to measure.

Why Not One Pool per Worker?

VPP has no option for it, and buffers freed on another thread would still go back to the receiving worker’s pool under its lock. Splitting each node’s pool into four in a patched VPP gained nothing measurable at NPS4.

Changing the Setting

On this Supermicro H13SSL-N board:

1. Reboot and press Del during POST to enter BIOS setup.
2. Go to Advanced > ACPI Settings and set NUMA Nodes Per Socket to NPS4. The choices are NPS0, NPS1, NPS2, NPS4 and Auto; Auto means NPS1. Leave ACPI SRAT L3 Cache As NUMA Domain on Auto.
3. Press F4 to save and exit.
4. After the reboot, numactl -H should report available: 4 nodes (0-3), and /sys/bus/pci/devices/<NIC>/numa_node shows which node each NIC belongs to.

With NPS4, give VPP one buffer pool per node that is large enough for its receive rings (buffers { buffers-per-numa ... }), and keep 1 GB hugepages on every node: the kernel spreads boot-time hugepages evenly over the nodes.

Summary

• At NPS1 VPP has one packet-buffer pool, and every worker refills its receive rings and frees dropped packets through one spinlock.
• On a 64-core EPYC 9534 with eight CPU dies, more than 32 workers on that lock made the machine slower: 350 Mpps at 32 workers, 148 at 62.
• NPS4 gives four NUMA nodes and four pools; VPP takes each receive queue’s buffers from the pool of the NIC’s node, so fewer workers share each lock and the buffers stay in nearby memory.
• With NPS4 the same 62 workers drop 450 Mpps of 64-byte packets, three times as much; buffer allocation drops from 55 % to 9 % of a worker’s time.

References

• AMD: 4th Gen AMD EPYC Processor Architecture white paper
• AMD: EPYC 9004 Series Architecture Overview (publication 58015)
• AMD: 4th Gen EPYC architecture overview page
• AMD EPYC 9534 specifications
• VPP source: buffer pool allocation (src/vlib/buffer_funcs.h)
• VPP source: DPDK mempool backed by VPP buffers (src/plugins/dpdk/buffer.c)
• VPP source: receive queue buffer pool selection (src/plugins/dpdk/device/common.c)
• DPDK mlx5 driver guide
• numactl(8)
• TRex traffic generator