This post is a repost of technical blog originally published by Denys Haryachyy, shared here with permission as part of ongoing research and engineering work around FastNetMon’s inline traffic processing capabilities.
TL;DR. A VPP software data plane drops packets at full 100GbE line rate (~140 Mpps) on one server — but staying there is a systems problem, not an algorithm one. Two things decide it, neither about the classifier: the per-packet cycle budget (~346 cyc/pkt spent of a 597 budget on a 3.26 GHz core, dominated by VPP node-graph overhead, not rule-matching); and the NIC RX ring — the real ceiling. More workers help only by adding queues, so RSS spreads the load and no single ring overflows. The drop path reaches line rate once enough queues spread the load — 26 workers already hold 142 Mpps.
Dropping at 100GbE line rate is a systems problem as much as an algorithm one. A previously published article — Learning VPP: Filtering Packets at 100GbE Line Rate — covered the classifier, the tuple-space search that makes rule-matching effectively free no matter how many rules you load. This is its companion: once matching is cheap, where do the cycles actually go, and why does the NIC — not the CPU — set the ceiling.
The Test Platform
Every number below comes from a two-machine 100GbE lab wired back-to-back: a VPP/FastACL device under test, and a TRex traffic generator that both sources and sinks the load.
| Role | CPU | NIC |
|---|---|---|
| Device under test (VPP + FastACL) | AMD EPYC 7742 (64C/128T, ~3.26 GHz) | Mellanox ConnectX-5 Ex 100GbE, dual-port |
| Traffic generator + sink (TRex) | AMD Ryzen 5800X (8C/16T, 14 data cores) | Mellanox ConnectX-5 Ex 100GbE, dual-port |
The DUT runs a release VPP with 26 workers, 8192-entry RX rings, and the ConnectX-5 in CQE_COMPRESSION=AGGRESSIVE firmware mode. The card negotiates a full PCIe 4.0 x16 link (252 Gb/s) — far above the ~95 Gb/s of L1 bandwidth that 64-byte frames at 142 Mpps actually carry — so the bus is never the limit; the NIC RX ring is. TRex sources up to 142 Mpps of 64-byte frames, full 100GbE line rate.
The Budget: 597 Cycles per Packet
Start from the wire. At 141.9 Mpps spread over 26 workers, each worker sees ~5.5 Mpps. On a 3.26 GHz core that is a hard budget of 597 cycles per packet — receive, classify, drop, and recycle the buffer, all of it. Everything below is about staying under that line.
Where the Cycles Go
If rule-matching is fixed and small (the classifier is O(number of distinct masks), not O(rules)), what spends the 346 cycles? VPP’s node graph. Profiling the drop path on the low-cardinality (frag-flood) profile:
| Node | cyc/pkt | Share |
|---|---|---|
dpdk-input | 193 | 56 % |
| filter node (TSS bihash lookup) | 74 | 21 % |
ethernet-input | 39 | 11 % |
ip4-input | 23 | 7 % |
error-drop + drop | 17 | 5 % |
| Total | 346 | of a 597 budget |
The filter node is the classifier lookup — a small, fixed number of per-mask bihash probes regardless of how many rules are loaded. The dominant cost is VPP graph overhead — dpdk-input + ethernet-input + ip4-input are ~74% of the pipeline. Further wins come from bypassing nodes (a combined L3-drop node), not from faster matching. The lookup itself rises only modestly on high-cardinality floods — every packet carries a fresh 5-tuple, so each pays a full bihash probe (~80 cyc) instead of hitting a warm bucket — but it stays bounded by the mask count, never the rule count.
Always Build Release VPP
The single biggest performance cliff has nothing to do with the algorithm. A debug VPP build enables vlib_buffer_validate_alloc_free on every buffer:
| Build | Drop throughput |
|---|---|
| debug | 0.56 Mpps |
| release | 35+ Mpps |
Same code, ~60× difference. If your numbers look impossibly bad, check the build first (show version; a giveaway is vlib_buffer_validate in show errors).
Why More Workers Doesn’t Mean More Throughput
The per-worker ceiling is 3.26 GHz ÷ 346 cyc = 9.4 Mpps, so 26 workers is 245 Mpps theoretical — comfortably above the NIC’s 142. CPU is not the limit. The limit is how fast workers drain the NIC RX ring, which depends on RSS spread and queue count:
| Workers | Mpps (random-src) | Why |
|---|---|---|
| 8 | 34 | RSS/RETA imbalance — flows clump on a few queues |
| 16 | 88 | each queue gets 8.9 Mpps > drain rate → RX ring overflow |
| 26 | 142 | each queue 5.4 Mpps < drain rate → line rate |
| 32 | 142 | each queue 4.4 Mpps; half the cores on this 64-core box — same line rate, extra headroom |
Adding workers helps only because it adds queues: more queues means less traffic per queue, so no single RX ring overflows. With too few queues — say 8 — the NIC’s 128-entry redirection table and a Toeplitz hash over random source IPs clump flows onto a few queues while others sit idle. The number that matters isn’t 32 but “enough queues that per-queue load stays under the drain rate”: 26 workers already hold full line-rate drop (142 Mpps), and 32 is just margin.
Summary
- At 100 GbE the budget is 597 cyc/pkt (26 workers, 3.26 GHz); ~346 are spent, dominated by VPP node-graph overhead, not rule-matching.
- Build release. A debug VPP build collapses drop throughput ~60× (0.56 vs 35+ Mpps) — check this before anything else.
- The ceiling is the NIC RX ring, not the CPU. More workers help only by adding queues so RSS spreads the load; the drop path reaches line rate once per-queue load drops under the drain rate — 26 workers already hold 142 Mpps, and 32 (half the cores on this 64-core box) leaves headroom.
- Per-worker headroom is 9.4 Mpps, so the CPU could theoretically do 245 Mpps — the NIC’s 142 gets there first.
References
- Learning VPP: Filtering Packets at 100GbE Line Rate — the companion article on the TSS classifier that keeps rule-matching off the hot path.
- FD.io VPP documentation — the Vector Packet Processor and its node-graph data plane.
- DPDK mlx5 poll-mode driver — RSS, the redirection (RETA) table, and RX descriptors on the ConnectX-5.
