This post is a repost of a technical blog originally published by Denys Haryachyy, shared here with permission as part of ongoing research and engineering work around FastNetMon’s inline traffic processing capabilities.
TL;DR. Cisco TRex 3.06 pushes full 100GbE line rate — 142 Mpps, 100.01 Gbps L1,
tx_util100 % — from one desktop Ryzen 7 5800X and a single ConnectX-5 Ex (PCIe Gen4). The rate is set by hardware, not tuning: 14 worker cores, 800×2 MB hugepages, and stateless streams with randomized source IPs. Why 142 and not 148.8 Mpps? A 64-byte Scapy frame is 68 bytes on the wire, so100e9 ÷ (88×8) = 142 Mpps.
A traffic generator earns its keep by saturating the wire, so your measurements blame the device under test, not the test rig. Here’s how to get there with TRex 3.06 on a ConnectX-5.
The Hardware
| Component | Value |
|---|---|
| CPU | AMD Ryzen 7 5800X — 8 cores / 16 threads |
| NIC | Mellanox ConnectX-5 Ex, 100 GbE, PCIe Gen4 x16 |
| OS | Ubuntu 24.04, TRex in a privileged container |
| TRex | v3.06 |
TRex is CPU-bound, so cores are the ceiling. Core 0 runs the master and latency threads; the remaining 14 threads are data cores that fill the NIC queues. A PCIe Gen4 NIC matters — on Gen3 the bus caps you before the cores do.
Why 142 Mpps Is Line Rate (Not 148.8)
The textbook 100 GbE figure is 148.8 Mpps (a 64-byte wire frame: 64 + 20 bytes preamble/IFG = 84 = 672 bits). But Scapy’s Ether excludes the 4-byte FCS the NIC appends, so a 64-byte Scapy frame is 68 bytes on the wire:
68 B frame (64 Scapy + 4 FCS) + 8 preamble + 12 IFG = 88 B = 704 bits100e9 / 704 = 142.0 Mpps <- line rate for these frames
TRex confirms it: 142.06 Mpps, 100.01 Gbps L1, tx_util 100 %. To hit the literal 148.8 Mpps, build a 60-byte Scapy frame so the wire frame is exactly 64 bytes.
TRex Configuration
/etc/trex_cfg.yaml — two ports and the 14-core layout:
- port_limit : 2 version : 2 interfaces : ["0000:26:00.0", "0000:26:00.1"] port_info : - ip : "10.0.1.1" - ip : "10.0.2.2" platform : master_thread_id : 0 latency_thread_id : 8 dual_if : - socket : 0 threads : [1,2,3,4,5,6,7,9,10,11,12,13,14,15]
The ConnectX-5 uses the mlx5 bifurcated driver — no vfio unbind, the kernel and TRex share the port. Flush the kernel IP off the data-plane port, then launch:
ip addr flush dev enp38s0f0np0 # port 0echo 800 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages./t-rex-64 -i --iom 0 --no-ofed-check -c 14
Streams: Where Line Rate Comes From
A single fixed 5-tuple pins to one queue (~10–15 Mpps). To spread across all 14 cores, vary the source IP randomly per packet:
vm = STLScVmRaw([ # random source IP - spreads RSS across all RX queues STLVmFlowVar(name="src", min_value="10.1.0.1", max_value="10.1.15.254", size=4, op="random"), STLVmWrFlowVar(fv_name="src", pkt_offset="IP.src"), STLVmFixIpv4(offset="IP"), # fix checksum after the rewrite])stream = STLStream(packet=STLPktBuilder(pkt=pkt, vm=vm), mode=STLTXCont(pps=25_000_000))
Two gotchas. First, use op="random", not op="inc": incrementing every FlowVar in lockstep walks the values down a diagonal, so the unique-flow count collapses to the largest single range instead of the product of the ranges.
op="inc" collapses flows onto a diagonal; op="random" fills the grid.Second, rewriting IP.src invalidates the IPv4 header checksum, so STLVmFixIpv4 must recompute it — otherwise a checksum-validating receiver drops every frame. (UDP checksums are optional in IPv4, so UDP(chksum=0) saves a FlowVar.)
Spreading Across the RX Queues
Random source IPs aren’t just convenient — they spread the load evenly across every RX queue. The receiver steers each packet by hashing its source IP, so a stream of random sources lands on all 32 queues to within 0.1 % of each other. Every poll worker stays busy, and no single core caps the receiver.
The Result
Port 0 tx_pps 142.06 Mpps tx_bps_L1 100.01 Gbps tx_util 100.06 %
Full 100 GbE line rate, sustained, from one desktop Ryzen and one ConnectX-5. tx_util 100 % means there are no idle bit-times left on the wire.
Summary
- Hardware sets the ceiling — Gen4 ConnectX-5 + 14 cores dedicated to TRex.
- Target 142 Mpps, not 148.8 (68-byte wire frames). Trust
tx_util. - TRex:
trex_cfg.yamlwith 14 threads,-c 14, 800×2 MB hugepages. - Streams: randomized source IPs +
STLVmFixIpv4— they spread RSS evenly across every RX queue (measured 0.1% CoV).
References
- Cisco TRex documentation — full manual, including the stateless mode used here.
- TRex Stateless Python API —
STLClient,STLStream, and the Field Engine (STLVm*). - DPDK mlx5 poll-mode driver — the ConnectX-5 PMD and its devargs.
- Microsoft — RSS Hashing Functions — how receive-side scaling maps flows to queues.
