This post is a repost of a technical blog originally published by Denys Haryachyy, shared here with permission as part of ongoing research and engineering work around FastNetMon’s inline traffic processing capabilities.
TL;DR. Testing a data plane at 100GbE line rate needs real NICs running VPP — which no cloud CI runner has. So an ephemeral GitHub Actions runner joins a private two-machine lab over Tailscale, deploys its code to a scratch tree, pulls a fresh base image, brings up a VPP device under test (auto-recovering a wedged NIC via IPMI), drives a TRex generator through a profile sweep, and gates the merge on per-profile line-rate floors — verified on both sides of the wire. Jobs queue on the single lab, and every run tears itself down.
Unit tests don’t tell you whether your VPP forwarding plane still does 100GbE line rate after a refactor. For that you need real hardware in the loop — reached from CI, benchmarked on every push, and cleaned up after.
The Problem
A cloud runner is a VM with a virtual NIC. It can’t run VPP on a physical 100GbE adapter, generate ~142 Mpps, or measure what a real DUT forwards. That hardware lives in a private lab with no public address — so the question is how a cloud runner safely drives hardware it can’t see.
One Lab, One Job at a Time
A physical lab serves one test at a time — two jobs can’t share one NIC pair. A GitHub Actions concurrency group with cancel-in-progress: false serializes them: a second push waits its turn instead of colliding on the wire, and is never dropped.
The Run, End to End
Every run follows the same arc — and every stage cleans up after itself:
Tailscale Auth: Keys and ACLs
The lab has no public IP. The runner joins its Tailscale mesh per job, and the control is all in the auth:
- Ephemeral, tagged, single-use key (a CI secret): the node is pre-authorized, tagged
tag:ci, and removed automatically when the job ends. - An ACL that scopes it:
tag:cimay reachtag:labon SSH and nothing else — a leaked key still can’t roam. - No inbound exposure: the runner dials out; the lab stays dark to the public internet.
"acls": [ // the ephemeral CI node may only SSH the lab machines { "action": "accept", "src": ["tag:ci"], "dst": ["tag:lab:22"] }]
Isolation
A shared lab is also a development lab, so CI must touch nothing it didn’t bring:
- Scratch checkout: the runner rsyncs to a dedicated scratch tree, not the dev checkout, and the container mounts that — so it runs exactly the code under test.
- Fresh base image every run, pulled from the registry — never a stale local cache (the bug that once slipped a debug build into a perf test).
Guarded Bring-Up
Bringing a 100GbE NIC up under VPP can hard-lock the host. A watcher tells a transient miss from a real wedge; on a wedge it power-cycles the box over IPMI — out of band, independent of the hung OS — and retries, instead of hanging the job.
Verifying Line Rate
Line rate is verified on both sides of the wire, not assumed:
- Generator:
tx_util = 100 %means no idle bit-times — it really is offering line rate (~142 Mpps at 64 B). - DUT: per-worker vectors ÷ time = Mpps processed;
rx_missed ≈ 0proves the NIC kept up. - Gate: each profile’s floor sits a few percent below its ceiling — catches regressions without tripping on jitter.
tx_util and the DUT rate with rx_missed ≈ 0 — against a floor.# generator (TRex) # DUT data plane, over the sample windowtx_pps 142.0 Mpps show runtime -> sum(Vectors)/seconds = Mppstx_util 100.0 % rx_missed_errors 0 # nothing dropped before software
Self-Cleaning Teardown
On success, failure, or cancel, the run stops the containers gracefully (a hard kill is what wedges the firmware), then removes the containers, every lab image, and the scratch checkout from both hosts. The lab is left exactly as CI found it — powered on, idle, holding none of CI’s code or images.
The Workflow
name: hw-line-rateon: { push: { branches: ['**'] } }concurrency: # one job on the lab at a time group: hw-lab cancel-in-progress: falsejobs: line-rate: runs-on: ubuntu-latest # an ordinary cloud runner steps: - uses: actions/checkout@v4 - uses: tailscale/github-action@v3 with: { authkey: ${{ secrets.TS_AUTHKEY }} } # ephemeral, tag:ci - name: Deploy + base + guarded bring-up run: | for h in "$GEN_HOST" "$DUT_HOST"; do rsync -az --delete ./ "ci@$h:~/ci-scratch/"; done ssh ci@$DUT_HOST 'ENSURE_BASE_FORCE=1 ./labs/hw/ensure-base-image.sh' ./labs/hw/bringup-guarded.sh --watch-secs 600 - name: Bench + assert line-rate floors run: ./labs/hw/bench.sh && ./labs/hw/assert-floors.sh - name: Tear down — leave no trace if: always() run: ./labs/hw/teardown.sh
Summary
- Real line-rate testing needs real hardware; a cloud runner orchestrates it, doesn’t host it.
- Tailscale admits the ephemeral runner with a tagged single-use key + an ACL — no public exposure.
- Serialize jobs on the single lab; isolate CI’s code; pull a fresh base every run.
- Guard the NIC bring-up; recover wedges out-of-band via IPMI.
- Verify line rate on both sides, gate on floors, and tear everything down.
References
- Tailscale — GitHub Action
- Tailscale — Access Controls (ACLs)
- GitHub Actions — Concurrency
- Cisco TRex
- FD.io VPP
- ipmitool — out-of-band BMC control
