Skip to content

xen/dom0: refactor dom0 memory growth to avoid splitting based on max - #63

Merged
alexandermerritt merged 7 commits into
edera/mainlinefrom
alexm/xen-dom0-unbacked-runs-ranges2
Oct 29, 2026
Merged

alexandermerritt merged 7 commits into
edera/mainlinefrom
alexm/xen-dom0-unbacked-runs-ranges2

Conversation

@alexandermerritt

@alexandermerritt alexandermerritt commented Oct 27, 2026 •

Copy link
Copy Markdown

Description

Part of COR-141

Replacement of proposal in #59 where we use separate page list arrays for small and super pages, rather than a range allocator.

Prior work in ENG-99 split unbacked boot-time memory using the max: parameter in dom0_mem from Xen between the dom0 balloon driver and grant / foreign mappings. This makes it hard to remain flexible across varied deployments.

Remove the implied function of max: from splitting this memory, and add 2mb ranges to an additional list alongside the existing page list. Balloon and mappings share this unified memory range: the balloon consumes super pages found in the block list, and mappings take from the page list.

Testing

  • QEMU TCG boot PV
  • Dell x86 PV
  • Dell x86 PVH
  • NUMA (on PVH)

This reverts commit 0a2ac17.

Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
This reverts commit 8788269.

We no longer want to split the unpopulated boot-time space between
the dom0 balloon and grant and foreign mappings.

Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
This reverts commit 7498978.

Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
Consolidate existing duplicated code. No functional change.

Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
The boot-time unbacked address space reaches the unpopulated allocator
(page_list) one 4 KiB page at a time, with a lock round trip and a list
push for each page. On a large dom0 that is millions of round trips at
boot. page_list also is unaware of page contiguity, limiting dom0
balloon growth to 4 KiB updates.

Hand each extra memory region over whole, and keep its aligned 2 MiB
blocks on a list per node, linked through the first page of each
block as page_list links pages. The pages around the blocks, and a
block that a node boundary cuts, go on page_list.

Grants and foreign mappings still take pages from page_list only.
When page_list of a node is empty, one block moves onto it, so taking
a page stays a list pop. Pages given back go on page_list and do not
form a block again.
With no pages parked on the balloon, dom0 grows by hotplugging a
memory section at the first free physical address. That address space
can belong to a PCI device that has not claimed it yet, and Xen does
not stop dom0 from growing when dom0_mem sets no max:.

dom0's balloon needs room to grow into, and grows faster when Xen
populates a whole 2 MiB extent at a time. The unpopulated allocator
keeps that room, the boot-time unbacked address space. Grant and foreign
mappings and the balloon have to share it.

Feed the dom0 balloon from the allocator's 2 MiB blocks when it is
instructed to grow.
Add `info/room_kb` to the sysfs of the xen balloon driver, so user tools
can discover the amount of memory dom0 can grow by. Reported are only
complete 2MiB blocks in the unpopulated allocator.
@alexandermerritt alexandermerritt self-assigned this Oct 27, 2026
@alexandermerritt alexandermerritt changed the title alexm/xen dom0 unbacked runs ranges2 xen/dom0: refactor dom0 memory growth to avoid splitting based on max Oct 27, 2026
Comment on lines +488 to +490
for (node = 0; node < MAX_NUMNODES; node++)
while (nr < nr_blocks && spare_blocks(node))
heads[nr++] = pop_block(node);

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bleggett apropos #59 (comment)

AFAIK this fills all of NUMA node 0 before it touches node 1, so I don't think (could be wrong) that this plays particularly nicely with NUMA?

xen_alloc_unpopulated_blocks is not given any information from the caller to instruct it which node to grow from. We could allocate RR?

@alexandermerritt
alexandermerritt force-pushed the alexm/xen-dom0-unbacked-runs-ranges2 branch from a8bc870 to f3450aa Compare October 27, 2026 18:35
@alexandermerritt

Copy link
Copy Markdown
Author

@bleggett apropos #59 (comment)

There's an arch/arm/xen/enlighten.c - shouldn't it get the same/equiv change?

I haven't dug into this much, but on ARM what would be different is where the "available" memory would come from at boot. ARM uses a device tree to discover the memory, not an e820 map.

However, the xen changes that use the logarithm to decide on the amount of dom0 memory still apply. Those changes were meant to make it less likely to need to adjust memory size at startup.

@alexandermerritt

Copy link
Copy Markdown
Author

Also, PV has no NUMA support, so it remains unaffected.

@alexandermerritt
alexandermerritt marked this pull request as ready for review October 29, 2026 04:17
@alexandermerritt
alexandermerritt merged commit 453f94e into edera/mainline Oct 29, 2026
1 check passed
@alexandermerritt

alexandermerritt commented Oct 29, 2026 •

Copy link
Copy Markdown
Author

back port is building here: https://github.com/edera-dev/linux/actions/runs/40818328735

@github-actions

Copy link
Copy Markdown

Backport to edera/6.18-lts: proposed in #67

Checks and builds passed.

Run log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants