This post is a repost of a technical blog originally published by Denys Haryachyy, shared here with permission as part of ongoing research and engineering work around FastNetMon’s inline traffic processing capabilities.
The VPP ACL plugin matches packets with a hash-based TupleMerge lookup, so a lookup does not slow down much as an ACL grows. Changing the rules is a different story. I wanted to know how long VPP takes to absorb a rule change when there are 100,000 rules, and whether it matters if those rules live as 100,000 entries in one ACL or as 100,000 ACLs with one entry each. I measured both on an AMD EPYC server running stock VPP 25.10, driving the changes through the ACL plugin’s binary API and measuring the plugin’s own work with perf.
The two layouts behave very differently: one grows linearly with the number of entries, the other grows with the square of the number of ACLs. This post shows the numbers and the code paths behind them.
How the ACL Plugin Stores Rules
Three objects matter for both lookup and updates:
- ACL: an ordered list of entries (ACEs) created or replaced as a whole with
acl_add_replace. There is no call that inserts or removes a single entry. - Lookup context: the list of ACLs one user evaluates, for example the input ACLs of an interface or the policies attached by ABF. The order of ACLs in the list is part of the precedence.
- Applied entries: when ACLs are applied to a context, every entry of every ACL is copied into one per-context vector. An entry’s index in that vector is its rank: the lookup returns the lowest index that matches. Entries are grouped into mask partitions (TupleMerge tables) backed by one shared bihash.
The lookup walks the mask partitions and keeps the lowest matching index. Each partition records the first index it holds, so the walk stops as soon as no remaining partition can beat the best match. That early stop is why the rank has to be a position in one vector.
What Happens When an ACL Changes
Replacing the entries of an ACL that is already applied runs hash_acl_reapply() in every lookup context that uses it:
- Every ACL after the changed one in the context is un-applied, last to first.
- The changed ACL and every ACL after it are applied again, so their entries get new positions.
- Each apply and un-apply ends with
remake_hash_applied_mask_info_vec(), which rebuilds the mask-partition summary by walking every applied entry of the context, not only the ones that changed.
Adding an ACL to a context or removing one goes through acl_plugin_set_acl_vec_for_context(), which un-applies the whole old ACL list and applies the whole new one.
So the cost of a change has two parts. One is proportional to the entries that are re-applied. The other is paid once per ACL that is re-applied, and each of those payments walks the whole context. With one big ACL there is one such walk per change, so the cost stays linear in the number of entries. With N ACLs a change can re-apply all of them: N walks over N entries, which is quadratic.
Test Setup
| Item | Value |
|---|---|
| Server | AMD EPYC 9534 (Genoa), 64 cores |
| VPP | 25.10 release packages, ACL and ABF plugins only |
| Rule shape | UDP to a distinct /32 destination, destination ports 8000–8255, deny |
| One big ACL | 1 ACL with N entries, applied as the input ACL of a loopback interface |
| Many ACLs | N ACLs with 1 entry each, each referenced by one ABF policy attached to the same interface |
| Measured | The ACL plugin’s own work per change: CPU time in the ACL and ABF plugins and their hash tables, sampled with perf on the VPP main thread; the API request itself is excluded |
Many ACLs are built through ABF because the interface ACL message carries the list length in a single byte: acl_interface_set_acl_list accepts at most 255 ACLs per interface. ABF has no such limit; each attached policy adds its ACL to the interface’s ABF lookup context.
Results: One ACL With Many Entries
Each entry in the ACL costs about 0.12 µs of plugin work per change at every size, so the time grows in step with the ACL.
Results: Many One-Entry ACLs
Every 3.3× increase in ACLs costs 7–11× more per attach. Building the 10K-ACL context one policy at a time took 931 seconds; at that rate a 100K-ACL context is not something one builds incrementally.
At the same rule counts one big ACL needs 0.15 ms at 1K and 1.1 ms at 10K (figure 4): about 21× less at 1K and 240× less at 10K, and the gap keeps widening.
100K Entries vs 100K ACLs
At 100K rules the two layouts are far apart. One ACL absorbs an added entry in 11.2 ms (figure 4). With 100K one-entry ACLs, the quadratic trend puts one more attach at about 26 s (figure 5), extrapolated from the measured 10K point.
Update cost in the VPP ACL plugin scales with the number of ACLs in a lookup context far more than with the number of entries. Keep large rule sets in a few big ACLs.
Summary
- The VPP ACL plugin applies changes per ACL, and every per-ACL step touches the whole lookup context.
- One ACL with 100K entries absorbs a change with 11.2 ms of plugin work; 100K one-entry ACLs would need about 26 s per added ACL.
- The interface ACL API caps a list at 255 ACLs, so large rule sets belong in large ACLs anyway.
References
- hash_lookup.c: hash_acl_apply, hash_acl_reapply, remake_hash_applied_mask_info_vec
- lookup_context.c: acl_plugin_set_acl_vec_for_context
- acl.api: acl_interface_set_acl_list (u8 count)
- abf_itf_attach.c: ABF lookup context per interface
- FD.io: ACL plugin constant-time lookup design
- FD.io CSIT: ACL test methodology
- VPP-897: applying of large number of ACEs is slow
