Skip to content

xen/netfront: netfront persistent grants - #119

Merged
bleggett merged 5 commits into
edera/mainlinefrom
bleggett/netfront-static-grants
Nov 19, 2026
Merged

bleggett merged 5 commits into
edera/mainlinefrom
bleggett/netfront-static-grants

Conversation

@bleggett

@bleggett bleggett commented Nov 19, 2026 •

Copy link
Copy Markdown

This adds a new grant mechanism to the xen-netfront driver which reduces CPU usage and improves throughput by a measurable amount (see below)

This is logically similar to the persistent grant mechanism xen-blkfront already uses, just applied to xen-netfront.

Existing method used in netfront: grant copy for each request

  • For every packet buffer, the zone's netfront granted the backend access to that one page, then put the grant reference in a ring request.

  • Then the backend asks Xen to copy the data (GNTTABOP_copy). Xen checks the grant, takes locks and page references, maps the page, copies, and releases it. When the response comes back, netfront revokes the grant.

So each page costs 1 grant, 1 hypercall with fixed work of about 400 to 600 ns, and a revoke. The backend never has the zone's memory mapped. It reaches each page only through Xen, one request at a time.

New method for netfront: static/persistent grant

  • At connect, netfront allocates a fixed pool of pages for each TX/RX queue(s), grants them once up front, and registers them with the backend over the netif control ring (ADD_GREF_MAPPING). The backend maps those pages once and keeps them mapped.

  • For each packet, netfront puts a pool page's reference in the request. The backend copies to/from via a regular memory copy with no hypercall. On receive netfront copies the data out of the pool page into its own buffer, and on transmit it copies the data in first.

  • Requests that do not use a pool page (pool empty, an old kernel, or the flag off) still use grant copy as before.

Measurements

PV dom0, before and after static grants, 4-vCPU zones, median of 3

Path Streams Before (Gbit/s) After (Gbit/s) Change
Zone egress 1 9.32 18.66 +100%
Ingress 1 10.51 13.59 +29%
Zone to zone 1 10.61 12.80 +21%
Zone egress 4 39.06 48.42 +24%
Ingress 4 33.85 38.53 +14%
Zone to zone 4 21.90 25.36 +16%
runc to runc (control) 1 21.22 21.38 none
Path Connections Before, avg / p99 (µs) After, avg / p99 (µs)
Zone to runc 1 164 / 237 142 / 178
Zone to zone 1 271 / 347 223 / 284
Zone to runc 16 336 / 584 325 / 560
Zone to zone 16 478 / 699 393 / 640
Case Throughput before (Gbit/s) Throughput after (Gbit/s) CPU cost before (cores per Gbit/s) CPU cost after (cores per Gbit/s) Cost change
Egress, 1 stream 9.14 17.46 0.407 0.261 36% less
Ingress, 1 stream 11.02 13.07 0.286 0.265 7% less
Egress, 4 streams 24.04 33.38 0.323 0.282 13% less
Ingress, 4 streams 30.65 37.33 0.274 0.252 8% less

CPU cost is dom0 cores plus zone cores, divided by Gbit/s. The dom0 cores include the runc iperf3 endpoint, and
the zone cores include the zone's endpoint.

Note that the 4-stream throughput ranges are wide, because they depend on how the flows hash to queues, which is currently randomized in Xen, but this might not be hard to fix.

@alexandermerritt alexandermerritt left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the data.

@bleggett
bleggett merged commit cfdd691 into edera/mainline Nov 19, 2026
4 checks passed
@github-actions

Copy link
Copy Markdown

Backport to edera/6.18-lts: not needed

Nothing to backport: every commit of #119 is already on edera/6.18-lts (c20f0c9b497c), changes only automation, or is no longer an Edera commit on edera/mainline.

Run log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants