Learning Mellanox ConnectX-5: CQE Compression Tuning

TL;DR. On a Mellanox ConnectX-5, the firmware setting CQE_COMPRESSION=AGGRESSIVE is the biggest receive-side win — +17–21 Mpps. Instead of one 64-byte completion (CQE) per packet over PCIe, the NIC packs many into a title CQE + 8-byte mini-CQEs, so rte_eth_rx_burst() returns ~4× more packets per call at −28 % cyc/pkt. That headroom let our EPYC 7742 — using just 32 of its 64 cores (one poll worker per core), ConnectX-5 Ex, DPDK 25.07 — absorb the full 142 Mpps 100GbE line rate. Enable it with mlxconfig + mlxfwreset.

A traffic generator’s job is to emit line rate; a device under test’s job is often to absorb it. Absorbing 142 Mpps is a different problem, and on the ConnectX-5 the dominant lever isn’t CPU tuning — it’s how the NIC reports completions over PCIe.

Our Test Setup

Every number below comes from one machine absorbing a 100GbE flood:

Component	Spec
CPU	AMD EPYC 7742 — 64 cores / 128 threads
Poll workers	32 — one RX queue pinned per worker, one worker per core (only 32 of the 64 cores used)
NIC	Mellanox ConnectX-5 Ex, 100GbE, FW 16.35.4030
Driver / DPDK	mlx5 PMD, DPDK 25.07
Data plane	FD.io VPP 25.10
OS	Ubuntu 24.04.4 LTS, kernel 6.8.0
Offered load	TRex at 142 Mpps (100GbE, 64 B frames — line rate)

With CQE_COMPRESSION=AGGRESSIVE and one queue per core, this DUT receives and forwards the full 142 Mpps for cache-friendly traffic — up to 142.2 Mpps received — at or above what TRex can emit. Compression is the receive-side lever that makes line-rate absorption possible; the rest of this post is how it works.

How CQE Compression Works

Every time the NIC finishes a packet — received or transmitted — it writes a Completion Queue Entry (CQE) into host memory over PCIe, and the DPDK 25.07 mlx5 poll-mode driver reads it to learn what happened. A standard mlx5 CQE is 64 bytes. At 142 Mpps that is 142e6 × 64 = ~9 GB/s of PCIe writes spent purely on completions, before a single byte of packet data moves — on a Gen4 link that competes directly with the DMA of the packets themselves.

CQE_COMPRESSION=AGGRESSIVE attacks exactly this. Instead of one full CQE per packet, the NIC emits one title CQE carrying the fields common to a run of packets, followed by an array of tiny mini-CQEs (8 bytes each) for the per-packet fields that vary. Many completions now ride in a single PCIe transaction. AGGRESSIVE compresses unconditionally — it assumes a homogeneous packet stream, which is exactly the case under a flood.

Enabling it — three ways. Firmware-persistent with the NVIDIA MFT utility, or per-application via the DPDK 25.07 mlx5 rxq_cqe_comp_en devarg in testpmd or VPP.

Firmware utility (mlxconfig) — persistent across reboots:

mlxconfig -d 0000:01:00.0 set CQE_COMPRESSION=1   # 1 = AGGRESSIVEmlxfwreset -d 0000:01:00.0 --level 3 --yes reset  # apply nowmlxconfig -d 0000:01:00.0 query CQE_COMPRESSION   # verify

testpmd (DPDK 25.07 devarg, no firmware change):

dpdk-testpmd -a 0000:01:00.0,rxq_cqe_comp_en=1 -- -i

VPP (dpdk plugin startup.conf):

dpdk {  dev 0000:01:00.0 { devargs rxq_cqe_comp_en=1 }}

On RX: The Burst Size Explodes

The receiver’s DPDK 25.07 mlx5 PMD calls rte_eth_rx_burst() in a tight loop. With uncompressed CQEs it reads one 64-byte completion per packet, so each PCIe read yields one descriptor and the burst stays small. With compression, a single read of one compressed block decodes into many completed descriptors at once:

	Compression OFF	CQE_COMPRESSION=AGGRESSIVE
RX descriptors per rx_burst()	~63	~240 (≈ 4×)
Cycles per packet	122	88 (−28 %)

Bigger RX bursts compound: the per-call overhead amortizes over 4× more packets, instruction-cache and prefetch locality improve, empty polls drop, and the receiver drains its RX ring fast enough to keep rx_missed near zero. That chain — fewer PCIe completion reads → larger RX bursts → lower cyc/pkt — is the entire +17–21 Mpps. (Those figures isolate compression alone; raising the per-worker drain rate this way is exactly what lets the full 32-worker DUT take the 142 Mpps line rate without the NIC’s RX ring overflowing — the rx-miss counter stays near zero instead of silently discarding packets before rte_eth_rx_burst.)

Summary

1. CQE compression is the dominant receive-side lever on the ConnectX-5: +17–21 Mpps.
2. It works by packing many completions into one PCIe read (title + 8 B mini-CQEs).
3. On RX it ~4× the rx_burst() size (63→240 pkt/call) and cuts cyc/pkt 28 %.
4. Set it with mlxconfig + mlxfwreset; it’s firmware-persistent.
5. On our DUT — AMD EPYC 7742 using only 32 of its 64 cores (32 poll workers), ConnectX-5 Ex, VPP 25.10 on Ubuntu 24.04 — compression is what lets VPP absorb the full 142 Mpps 100GbE line rate (142.2 Mpps received).

References

• DPDK mlx5 driver — rxq_cqe_comp_en / CQE compression — the DPDK-on-Mellanox documentation of the CQE-compression device argument and its formats, the runtime counterpart to the mlxconfig firmware setting used here.
• NVIDIA DPDK NIC Performance Report (DPDK 25.03) — NVIDIA’s official ConnectX/BlueField DPDK benchmark methodology and results; the closest published release to the DPDK 25.07 used here, and it lists CQE compression among the tuned settings.