This post is a repost of a technical blog originally published by Denys Haryachyy, shared here with permission as part of ongoing research and engineering work around FastNetMon’s inline traffic processing capabilities.
Every firewall, NAT or load balancer built on VPP needs the same plumbing: a connection table, timers, and handoff between workers. FD.io VPP now ships it as SFDP, the StateFul Data Plane, so you write only what makes your function different. The cost, measured on a 400G NIC: about 94 CPU cycles per packet, scaling linearly with cores.
SFDP is the work of Mohammed Hawari, Jérôme Tollet and Hadi Sandid from Cisco.
SFDP Terms at a Glance
SFDP introduces a small vocabulary, and every later section builds on it. The map places each term: an entry point picks a tenant, the tenant’s template shapes every session, each session is a forward and a reverse flow, and each flow runs a chain of services from one scope, ending in a terminal service.
Data Plane Overview
A packet entering SFDP is first matched against the session table, with three outcomes:
- hit, this worker owns the session — the packet enters the session’s service chain here;
- hit, another worker owns it — the packet is handed to the owner (see Ownership and Handoff below);
- miss — SFDP creates the session on the fly, on this worker, and the packet becomes its first forward packet.
In the VPP graph, SFDP is a handful of nodes between IPv4 input and the FIB. This is the graph as it ran in my benchmark below, with two workers:
Tenants and Contexts
Before a packet enters SFDP, something outside SFDP decides which tenant it belongs to: an interface, a VLAN, a VNI, a FIB lookup — SFDP does not care how. A tenant is the template for every session its packets create:
- the context id, an opaque 32-bit number that joins the 5-tuple in the lookup key; several tenants may share one context, and two tenants with different contexts never see each other’s sessions even when their addresses overlap;
- the forward and reverse service chains copied into each new session;
- service options such as the timeouts (embryonic, established, TCP established, security);
- the slow-path nodes for fragments, unknown protocols, ICMP errors and table overflow.
The Session Table
Sessions live in one fixed pool, sized at start-up and allocated when the first tenant is added; each session is 256 bytes, four cache lines. Two hash tables, one for IPv4 and one for IPv6 keys, map a key to an entry that names the owner worker, the slot, and a version that changes every time the slot is reused.
Allocation is two-level: each worker keeps a cache of free slots, and only cache misses touch the shared pool under a lock. Session ids are 64 bits made of a start-up epoch, the thread index and a per-thread counter, so workers allocate ids without coordinating.
Normalized Lookup and Pseudo-Direction
For both directions of a connection to find the same entry, the lookup normalises the 5-tuple: it orders the two endpoints so that the higher address always comes first, swapping source and destination when needed. Whether it swapped is the packet’s pseudo-direction. The session remembers the pseudo-direction of its first packet, and the real direction of any later packet falls out of one XOR:
The same XOR applies to the flow index, so the lookup turns a hash-table hit into a flow id without branching on direction.
Ownership and Handoff
VPP gives every worker its own receive queues. Each queue of each interface is polled by exactly one worker, assigned round-robin at start-up or pinned by the operator with the rx-placement command, and the NIC picks the queue for each packet by hashing its addresses and ports (RSS).
Nothing on that path keeps the two directions of a connection on one worker:
- the default hash is not symmetric, so a reply with swapped addresses and ports usually lands in a different queue;
- replies often arrive on another interface, whose queues are polled by other workers;
- with NAT the reply carries the translated addresses, so even a symmetric hash cannot pair it with the original packet.
SFDP therefore makes a session belong to the worker that created it, and only that worker touches it. A packet that arrives on another worker is enqueued to the owner through a VPP frame queue and resumes there in a handoff node. A TCP connection shows how it plays out:
Between the two workers the session may have expired and its slot been reused by a new connection, so the handoff node runs the packet only if the session version is still the one the other worker saw, and drops it otherwise.
Two workers racing to create the same session resolve through the hash table itself: the insert refuses to overwrite, and the loser frees its slot and retries the lookup.
Buffer Metadata
SFDP carries its per-packet state in the VPP buffer. Before lookup, the entry node writes the tenant index and puts the context id into the buffer’s flow id. The lookup replaces the flow id with the session index and direction, and fills in the rest:
The service bitmap in the buffer is a working copy: services consume it bit by bit, while the session keeps the original. When a packet leaves SFDP temporarily, for IP reassembly for example, the metadata is saved in a second area of the buffer and restored when it comes back, so processing resumes with the same bitmap and tenant.
Services, Scopes and Execution Order
A service is a graph node plus a registration. The registration names the node, the services it must run before or after, whether it is terminal (the last SFDP node a packet visits), and optionally a scope. This is the registration of the TCP state checker that ships with VPP:
SFDP_SERVICE_DEFINE (tcp_check) = {
.node_name = "sfdp-tcp-check",
.runs_before = SFDP_SERVICES (0),
.runs_after = SFDP_SERVICES ("sfdp-drop", "sfdp-l4-lifecycle"),
.is_terminal = 0
};
At start-up SFDP collects every registration in a scope, resolves the before/after constraints into one total order, and gives each service its position in that order as a bit number, 0 to 63. A chain is then simply a 64-bit bitmap, and dispatch is a find-first-set: each service ends its per-packet work by taking the lowest set bit as the next node and clearing it. Lower bits always run first, so the order is fixed by the registrations, not by the order in which an operator lists the services.
Because each session holds its own bitmaps, a service can change the chain of one connection while it runs. The stock l4-lifecycle service does exactly that: on the first TCP packet it replaces itself with the TCP state checker in both directions; for UDP and other protocols it stays, marks the session established on the first reverse packet and refreshes the timer.
The services that ship with VPP:
| Service | What it does |
|---|---|
sfdp-drop | terminal drop |
sfdp-l4-lifecycle | first service of every session; hands TCP to tcp-check |
sfdp-tcp-check | tracks SYN/FIN/ACK per direction and sets the embryonic, TCP-established or security timeout |
sfdp-nat-output, sfdp-nat-early-rewrite, sfdp-nat-late-rewrite | source NAT from an address pool; late rewrite is terminal |
sfdp-geneve-output | terminal Geneve encapsulation |
ip4-lookup, ip6-lookup | terminal: hand the packet back to FIB forwarding |
sfdp-snort-input, sfdp-snort-output | hand sessions to Snort and back |
sfdp-acl-sample and other samples | examples to start a service from |
Scopes are independent service pipelines over one session table. A service joins a scope by naming it in its registration, and the first registration that names a new scope creates it at start-up. Each scope gets its own 64 service slots and its own ordering, plus its own lookup node, handoff node and handoff queue, named with a -<scope> suffix; every session remembers the scope it was created in. A feature chooses the pipeline simply by sending packets to that scope’s lookup node — one interface could feed a firewall pipeline and another a NAT gateway pipeline in the same VPP, each with up to 64 services, while both share the session pool, the owner model and expiry.
Where scopes are used today: every service in the table above registers in the default scope, so a stock build has exactly one. Scopes are the extension point for plugins: a plugin that needs more than 64 services, or an ordering that conflicts with the stock services, registers its own scope instead of competing for bits in the default one.
Getting Packets Into SFDP and Back Out
SFDP does not care how packets reach it. To inject a packet, any node sets two fields in the buffer — the tenant index and the context id — and sends the packet to the lookup node of the scope it wants. Typical entry points are an interface feature, a FIB DPO that decides per prefix whether SFDP applies, or a plugin’s own node.
The stock entry point is sfdp-interface-input, an IPv4 and IPv6 feature that maps each ingress interface to a tenant. Leaving SFDP is the job of the terminal service at the end of the chain, which picks where the packet rejoins the graph.
Parsers extend lookup to keys that are not an IP 5-tuple. A parser supplies functions that extract and normalise its key and the key size; SFDP gives each parser its own hash table and a parser node per scope, which can be used as an entry point exactly like the standard lookup nodes. The sessions it finds are ordinary SFDP sessions with the same services, timers and owner rules.
Slow-Path Nodes
The lookup can decide that a packet needs special handling and send it to a slow-path node instead of the service chain. These are ordinary graph nodes, chosen per tenant and separately for IPv4 and IPv6, and each defaults to a drop. Keeping these cases out of the lookup keeps the hot path short:
Primary and Secondary Session Keys
Some services need one session to answer to two different 5-tuples. With NAT, the reply arrives addressed to the translated address and port, which is not the reverse of the original key. The NAT service therefore adds a secondary key to the existing session — already normalised, together with the pseudo-direction it implies — and the translated side of the connection lands on the same session, the same owner and the same chain.
The NAT service picks a new outside port and retries when the secondary key it wants is already taken.
Session Lifetime, Timers and Callbacks
A session is created by a lookup miss, or explicitly by a service, and removed only on its owner worker. In between, the services alone move it from embryonic to established or time-wait, each state with the tenant’s timeout. Plugins that keep their own per-session state follow along through two callback lists: new session callbacks run after creation and before the first packet runs the chain, so a service can initialise its state or adjust the chain; deleted session callbacks run in the expiry node before packet processing, so no packet of the session is in flight. A plugin can disable a stock callback at start-up to substitute its own.
Services move the timer with two functions: one for a deadline known to be later than the current one, and one for a deadline that may be earlier — called with a timeout of zero, the second forces a session to expire. The default timer module makes a live connection pay almost nothing:
- A refresh is one store. The session keeps its absolute deadline next to the timer handle; extending the deadline only rewrites that field, in cache lines the services read anyway.
- The wheel catches up once per timeout. When a slot fires, the expiry node re-arms a refreshed session with its remaining time, so an active connection costs one wheel operation per timeout period, not one per packet.
- No sharing. Each worker has its own wheel and only touches its own sessions, and removes every expired session in one pass per tick, with no locks or atomics.
Expiry is a pluggable module. Besides removing sessions whose timer is due, it is asked to evict sessions early whenever a worker’s free sessions fall below the eviction margin, which should cover the sessions one VPP loop can create plus the delay of the expiry module.
Writing a Service
A service node receives buffers whose flow id already names the session and the direction; it moves the timer and may rewrite its session’s chain, as shown above. Two more patterns the stock services follow:
- State beside the session. The session has no per-service area, so a service keeps its own vector indexed by session index and stores the session version in each record; a record whose version differs belongs to a reused slot and is reset.
- Create sessions elsewhere. A node that is not the lookup node can create a session on its own worker; creation reports success, a full pool, or a key already in the table.
Performance
I measured SFDP on stock VPP from the FD.io packages, on an AMD EPYC 9534 with one 400G ConnectX-8 port whose two PCIe functions serve as the two VPP interfaces. A TRex generator sends 64-byte UDP packets; throughput is measured with the workers saturated, and cycles per packet come from VPP’s own per-node counters.
Cost per Packet
On one worker with 65,536 established sessions, plain forwarding runs at 24.7 Mpps and SFDP at 12.6 Mpps. The breakdown shows where the extra 94 cycles go: the session lookup takes 74 of them, the interface-input entry node 8, and the l4-lifecycle service 6. The stock VPP nodes cost 105 cycles instead of 99, because IPv4 input now runs a feature arc. A chain with only the terminal ip4-lookup costs 185 cycles, so the session table itself, not the services, is what SFDP adds.
New Sessions
When every packet carries a new 5-tuple, the same worker creates and later expires one session per packet. It sustains 2.5 Mpps of new sessions. The lookup cost rises from 73 to 480 cycles, because a miss allocates a slot, inserts the key and initialises the session, and the expiry node spends another 376 cycles per packet removing the sessions again. In total a session that lives for a single packet costs 978 cycles, five times an established one.
Scaling With Workers
With both ports loaded and 8,192 sessions, SFDP forwards 14.0 Mpps on one worker, 27.4 on two, 53.8 on four and 107.8 on eight: 13.5 Mpps per worker, linear, because each worker only touches its own sessions. Plain forwarding stops scaling linearly beyond eight workers on this port, and at sixteen workers SFDP forwards 180.8 Mpps against 191.9 Mpps without it.
Handoff Between Workers
Two workers carry both directions of 65,536 sessions. When the worker that owns the sessions also polls the replies, it forwards 12.2 Mpps at 200 cycles per packet. When the replies arrive on the other worker, that worker spends 139 cycles per reply to receive it, look it up and hand it over, and the owner 132 to resume and forward it: 271 cycles in total, spread over two cores, for 14.4 Mpps. The owner is the limit in this case, and replies beyond what it can take are dropped at the handoff queue.
Session Table Size
The cost of the lookup depends on how much session memory the traffic touches. From 1,024 to 65,536 live sessions, sfdp-lookup-ip4 goes from 48 to 72 cycles per packet. At 262,144 sessions it takes 161 cycles: 64 MB of sessions no longer fit in the 32 MB L3 cache of one CCD of this processor, and the worker drops from 12.9 to 8.8 Mpps. At a million sessions the lookup costs 178 cycles and the worker forwards 8.2 Mpps.
Summary
- SFDP is a session table keyed by a normalised 5-tuple plus a context id, over a fixed pool of 256-byte sessions, with a per-session chain of services.
- Each session has one owner worker; packets that land elsewhere are handed over, and a version check drops those whose session was replaced meanwhile.
- Services are graph nodes ordered into a 64-bit bitmap per scope; each session carries, and may rewrite, its own chain.
- Timers live on per-worker wheels where a refresh costs one store; expiry and eviction run once per tick, with callbacks for plugins.
- On an EPYC 9534 with a 400G ConnectX-8, SFDP costs about 94 cycles per packet on an established session (12.6 Mpps per worker against 24.7 without it), scales linearly to 107.8 Mpps on eight workers and 180.8 Mpps on sixteen, creates and expires 2.5 million sessions per second per worker, and its lookup cost more than doubles once live sessions outgrow the L3 cache.
