This post is a repost of a technical blog originally published by Denys Haryachyy, shared here with permission as part of ongoing research and engineering work around FastNetMon’s inline traffic processing capabilities.
In the first post on the BlueField-3 I ran an FD.io VPP data plane with an ACL filter plugin on the DPU’s sixteen Arm cores, with DPDK driving its two 100G uplinks. VPP has a second way to drive the same mlx5 NIC: its native RDMA driver, which uses mlx5 Direct Verbs and no DPDK at all. This post puts the two side by side on the same cores, with the same filter and the same traffic. On 64-byte packets the RDMA driver drops 86.0 Mpps against 58.7 Mpps with DPDK (+46.5%), and it filters at 100G line rate from 128-byte frames instead of from 256 bytes.
What Changed Since the First Post
| First post | This post | |
|---|---|---|
| DOCA / BF-bundle | 2.9.1 (DOCA 24.11) | 3.4.0 (bf-release 4.15.0) |
| Arm kernel | 5.15.0-1057-bluefield | 5.15.0-1093-bluefield |
| mlx5 driver (OFED) | 24.10 | 26.04 |
| NIC firmware | 32.43.2408 | 32.49.1014 |
| VPP | 25.10 | 26.10-rc1 |
| NIC driver in VPP | DPDK | RDMA (Direct Verbs) |
The cores, the filter, the traffic and the twelve VPP workers are the same as in the first post. For a fair comparison I re-ran the DPDK driver on exactly this setup, so the two columns in the results below differ only in the NIC driver. Every number is the median of three 15-second trials on one 100G ingress port with a drop rule matching the flood; the trials agreed within 1%.
Two Receive Paths into VPP
Both drivers sit on the same bifurcated mlx5 device: the uplinks stay on the kernel’s mlx5_core and user space gets its own receive and transmit queues (DPDK mlx5 guide: bifurcated driver). With DPDK, the mlx5 poll-mode driver fills DPDK rte_mbuf descriptors and dpdk-input turns them into VPP buffers. VPP’s RDMA plugin uses mlx5 Direct Verbs instead: the queue rings are mapped into VPP, and rdma-input reads the NIC’s completion entries and builds VPP buffers from them directly, with no DPDK layer in between (VPP docs: RDMA (ibverb) driver, mlx5dv(7)).
The DMA-Ordering Fix on Arm
In Direct Verbs mode the CPU and the NIC hand ring entries to each other through completion-entry owner bits, doorbell records in host memory and a write to the NIC’s doorbell register. On x86, compiler barriers are enough for the DMA memory, plus a store fence before the doorbell write; Arm’s weaker memory model can reorder these accesses, so compiler barriers are not enough there (Linux kernel: memory barriers). The change “rdma: enforce mlx5 DV DMA ordering” adds dmb oshld after each ownership check and dmb oshst before each publication to the device on AArch64, and leaves x86 with compiler barriers only. It first ships in VPP 26.10-rc1, which is why this test runs on that release.
On Arm, run VPP’s RDMA driver in Direct Verbs mode only on a release that contains the DMA-ordering fix: VPP 26.10-rc1 or later.
Starting VPP with the RDMA Driver
The start-up configuration loses its dpdk section; the DPDK plugin is disabled and the RDMA plugin enabled, and everything else from the first post stays as it was:
plugins {
plugin dpdk_plugin.so { disable }
plugin rdma_plugin.so { enable }
plugin acl_filter_plugin.so { enable }
}
The interfaces are created in the start-up script, before the same cross-connect and filter set-up as in the first post. Giving them the Linux names keeps the rest of the configuration unchanged:
create interface rdma host-if p0 name p0 num-rx-queues 12 rx-queue-size 4096 tx-queue-size 4096 mode dv create interface rdma host-if p1 name p1 num-rx-queues 12 rx-queue-size 4096 tx-queue-size 4096 mode dv
host-if is the kernel network device, which stays up after the uplinks leave the OVS bridges. One RX queue per worker spreads the flood across all twelve cores, as with DPDK. The rate comes from VPP’s interface counter, which matched the filter node’s and rdma-input’s packet counts within timing slack, and the NIC-side loss from the port’s rx_packets_phy counter. VPP 26.10-rc1 comes from FD.io’s arm64 packages and the plugin is built out of tree against them, as described in the first post (FD.io 26.10 package repository).
Results
| Frame | DPDK | RDMA | Change |
|---|---|---|---|
| 64 B | 58.7 Mpps · 41 Gbps · 58.6% dropped in the NIC | 86.0 Mpps · 61 Gbps · 39.4% dropped in the NIC | +46.5% |
| 128 B | 50.2 Mpps · 61 Gbps · 39.0% dropped in the NIC | 82.2 Mpps · 100 Gbps (line rate) | +63.7% |
| 256 B | 44.2 Mpps · 99 Gbps (line rate) | 44.3 Mpps · 99 Gbps (line rate) | — |
| 512 B | 23.3 Mpps (line rate) | 23.3 Mpps (line rate) | — |
| 1024 B | 11.9 Mpps (line rate) | 11.9 Mpps (line rate) | — |
| 1500 B | 8.2 Mpps (line rate) | 8.2 Mpps (line rate) | — |
| IMIX | 32.9 Mpps (line rate) | 32.9 Mpps (line rate) | — |
Where the cores have headroom, both drivers run at line rate. Where they do not, the RDMA driver moves the limit: 64-byte traffic gains 27 Mpps, and 128-byte traffic goes from 61% to 100% of line rate.
Where the Cycles Go
At 64 bytes all twelve workers are saturated, so the measured rate fixes the budget per packet: 12 cores × 2.13 GHz divided by the rate gives 435 core cycles per packet with DPDK and 297 with RDMA. The filter node costs the same in both, about 100 cycles (15.4 and 16.3 ticks of the 330 MHz Arm timer). The difference is all in receiving, buffer handling and graph dispatch: 336 cycles with DPDK, 192 with RDMA. On frames of 256 bytes and more, the filter node’s own figure rises to 19–24 ticks with the RDMA driver against 15–17 with DPDK, but at those sizes both drivers run at line rate with cycles to spare.
Summary
On the BlueField-3 Arm cores the NIC driver underneath VPP decides the small-packet rate. With VPP 26.10-rc1, DOCA 3.4.0 and twelve workers, VPP’s native RDMA driver in Direct Verbs mode drops 86.0 Mpps of 64-byte packets, 46.5% more than DPDK’s 58.7 Mpps, and filters at 100G line rate from 128-byte frames. Both drivers run at line rate from 256 bytes and on IMIX. On Arm the RDMA driver needs the DMA-ordering fix that first ships in VPP 26.10.
References
- First post: Packet Filtering with VPP on BlueField-3 Arm Cores and 100G NICs
- VPP commit 34b80e5e0: rdma: enforce mlx5 DV DMA ordering
- VPP docs: RDMA (ibverb) device driver
- VPP source: rdma-input node (src/plugins/rdma/input.c)
- VPP source: dpdk-input node (src/plugins/dpdk/device/node.c)
- mlx5dv(7): mlx5 Direct Verbs
- DPDK mlx5 guide: bifurcated driver
- Linux kernel: memory barriers
- FD.io VPP 26.10 packages, including arm64
