KVM: remove leftover conversion XML and UEFI NVRAM files by andrijapanicsb · Pull Request #14534 · apache/cloudstack

andrijapanicsb · 2026-10-30T07:27:52Z

Description

In short: KVM leaves two kinds of small files behind that nothing ever deletes. This PR deletes each one once it is no longer needed, and changes nothing else.

Who runs into this

Migrating VMs from VMware to KVM. If you convert a VM straight onto NFS primary storage (the "Force converting to storage pool directly" option), a small .xml file is left next to the VM's disks after every migration. It stays there even after the VM is deleted.

Deleting UEFI VMs. UEFI is the modern boot firmware. Every UEFI VM has a small file on its KVM host that stores its boot settings, called the NVRAM file. When the VM is permanently deleted (expunged), that file stays on the host.

Nothing uses these files afterwards, and nothing cleans them up. After many migrations or many deleted VMs, an operator has to find and remove them by hand.

What changes

Situation	Before	After
VMware VM migrated straight onto NFS, import succeeds	`.xml` file left next to the disks	`.xml` file removed, disks untouched
Same migration, but the import fails	Existing cleanup removes the disks and the `.xml`	Unchanged
VMware VM migrated through temporary storage	Nothing left behind	Unchanged
Warm (CBT) migration	Nothing left behind	Unchanged
UEFI VM deleted permanently, its host is up	NVRAM file left on the host	NVRAM file deleted
UEFI VM deleted permanently, its host is down	Delete succeeds, file left	Delete still succeeds, file left, warning logged
UEFI VM stopped, rebooted, migrated, or destroyed but recoverable	NVRAM file kept	Unchanged
Non-UEFI VMs, system VMs	Nothing to clean	Unchanged

What it never does

It never makes deleting a VM fail or hang. The NVRAM cleanup is best effort with a short timeout. If the host is unreachable, the VM is still deleted and only a warning is logged.
It never deletes the disks of an imported VM. Only the one .xml file the import just read is removed.
It never deletes the NVRAM file of a VM that is still defined on the host.
No API, database or UI changes.

Not covered

Only the host the VM last ran on is cleaned. If a UEFI VM ran on several hosts during its life, older copies of its NVRAM file on those other hosts stay.
Files left behind by VMs deleted before this change are not cleaned up.
Tested with unit tests only so far, not yet on a live environment (see below).

Technical details

Conversion XML

With forceconverttopool=true on file-based storage, LibvirtImportConvertedInstanceCommandWrapper adopts the converted disks in place and skipped the cleanup step that the normal path runs.
It now deletes exactly <conversion-uuid>.xml (the file it just parsed), only after the converted instance was read successfully.
On failure the XML is deliberately kept: the management server then sends CleanupConvertedInstanceDisksCommand, which reads that XML to find the converted disks to remove.
Warm (CBT) cutover was checked and does not leak an XML: its input XML is a /tmp file deleted in a finally block, in-place finalization writes no output XML, and the fallback output directory is deleted once the disks are moved out.
Origin: forceconverttopool, VMware to KVM Migrations improvements #11594 (b106d6e).

UEFI NVRAM

The per-VM NVRAM file is host-local: guest.nvram.path + <vm-uuid>.fd.
KVMGuru.finalizeExpunge now sends a new DeleteUefiNvramCommand to the VM's current or last host, only for user VMs with the UEFI boot detail.
It uses easySend with a 60 second timeout, catches all errors, and still returns no expunge commands. Returning the command from finalizeExpunge instead would have made expunge fail whenever the last host is down.
On the host, LibvirtDeleteUefiNvramCommandWrapper:
- rejects anything that is not a plain UUID, so no other path can be targeted;
- does nothing if no NVRAM path is configured;
- keeps the file if a libvirt domain with that UUID is still defined, or if that cannot be checked;
- otherwise deletes the file, and treats an already-missing file as success.
Mixed versions during an upgrade: an older agent answers that the command is unsupported; the management server logs a warning and the file stays, as today.
Origin: per-VM NVRAM files since UEFI Support on CloudStack #3638 (d4b537e), never removed on expunge.

Files

core/src/main/java/com/cloud/agent/api/DeleteUefiNvramCommand.java (new)
plugins/hypervisors/kvm/.../wrapper/LibvirtDeleteUefiNvramCommandWrapper.java (new)
plugins/hypervisors/kvm/.../wrapper/LibvirtImportConvertedInstanceCommandWrapper.java
server/src/main/java/com/cloud/hypervisor/KVMGuru.java
tests for all of the above

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

How Has This Been Tested?

In short: all affected unit tests pass, including 19 new ones. No live test yet.

What was checked	Result
Import after migration straight onto NFS: only the `.xml` goes, the disk and unrelated files stay	Pass
Failed import keeps the `.xml` for the existing cleanup	Pass
Host deletes only the expunged VM's own NVRAM file	Pass
Host keeps the file while the VM is still defined, or when it cannot check	Pass
Host rejects a malformed VM UUID such as `../...`	Pass
Management server only asks for cleanup for UEFI user VMs with a known host	Pass
Deleting a VM still succeeds when its host is down or an error occurs	Pass
Checkstyle	No violations
Live run on a lab	Not done yet

Test details

Run locally with JDK 17:

mvn -pl core,server,plugins/hypervisors/kvm -am test \
  -Dtest='KVMGuruTest,LibvirtDeleteUefiNvramCommandWrapperTest,LibvirtImportConvertedInstanceCommandWrapperTest' \
  -Dsurefire.failIfNoSpecifiedTests=false -Dexec.skip=true

Test class	Result	New tests
`LibvirtImportConvertedInstanceCommandWrapperTest`	16/16	3
`LibvirtDeleteUefiNvramCommandWrapperTest` (new)	10/10	10
`KVMGuruTest`	41/41	6

-Dexec.skip=true skips an unrelated schema shell test that cannot run on Windows (CRLF line endings).

Planned live checks:

cold VMware → KVM import into NFS with forceconverttopool=true, then confirm no <uuid>.xml is left in the pool;
expunge of a UEFI VM with its last host up (file gone) and down (expunge succeeds, warning logged).

When a VMware VM is converted straight into file-based primary storage (forceconverttopool), the converted disks are used where they are, but the <uuid>.xml file written by virt-v2v was never removed. It stayed in the storage pool even after the VM was expunged. Delete that one file once the converted instance has been read successfully. On failure it is kept, because the existing cleanup command reads it to find the converted disks.

Every UEFI VM has a <uuid>.fd NVRAM file on its KVM host. It was never removed, so it stayed on the host after the VM was expunged. On expunge, ask the host the VM last ran on to delete it. This is best effort: it never fails the expunge and waits at most 60 seconds. The host only deletes the file of a valid VM UUID, and keeps it if a domain with that UUID is still defined there.

codecov · 2026-10-30T07:42:08Z

Codecov Report

❌ Patch coverage is 77.57009% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.77%. Comparing base (b9b5436) to head (f30fdd1).

Files with missing lines	Patch %	Lines
...va/com/cloud/agent/api/DeleteUefiNvramCommand.java	37.50%	10 Missing ⚠️
.../wrapper/LibvirtDeleteUefiNvramCommandWrapper.java	85.45%	6 Missing and 2 partials ⚠️
.../LibvirtImportConvertedInstanceCommandWrapper.java	72.72%	2 Missing and 1 partial ⚠️
...er/src/main/java/com/cloud/hypervisor/KVMGuru.java	88.00%	0 Missing and 3 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #14534      +/-   ##
============================================
+ Coverage     20.75%   20.77%   +0.01%     
- Complexity    21901    21927      +26     
============================================
  Files          6496     6498       +2     
  Lines        591602   591709     +107     
  Branches      72583    72597      +14     
============================================
+ Hits         122814   122940     +126     
+ Misses       455456   455425      -31     
- Partials      13332    13344      +12

Flag	Coverage Δ
uitests	`3.88% <ø> (ø)`
unittests	`22.08% <77.57%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

andrijapanicsb · 2026-10-30T10:45:40Z

@blueorangutan package

blueorangutan · 2026-10-30T10:48:05Z

@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-10-30T12:45:25Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19855

nvazquez · 2026-11-01T22:52:52Z

@blueorangutan test

blueorangutan · 2026-11-01T22:54:04Z

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-11-02T18:01:22Z

[SF] Trillian test result (tid-17430)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 64101 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr14534-t17430-kvm-ol8.zip
Smoke tests completed. 157 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

andrijapanicsb added 2 commits October 30, 2026 08:27

boring-cyborg Bot added component:api component:kvm labels Oct 30, 2026

andrijapanicsb requested review from nvazquez and shwstppr October 30, 2026 08:45

andrijapanicsb requested a review from DaanHoogland October 30, 2026 12:13

Navigation Menu

Repository navigation

KVM: remove leftover conversion XML and UEFI NVRAM files - #14534

KVM: remove leftover conversion XML and UEFI NVRAM files#14534
andrijapanicsb wants to merge 2 commits into
apache:mainfrom
andrijapanicsb:kvm-cleanup-conversion-xml-uefi-nvram

andrijapanicsb commented Oct 30, 2026

Uh oh!

codecov Bot commented Oct 30, 2026 •

edited

Loading

Uh oh!

andrijapanicsb commented Oct 30, 2026

Uh oh!

blueorangutan commented Oct 30, 2026

Uh oh!

blueorangutan commented Oct 30, 2026

Uh oh!

nvazquez commented Nov 1, 2026

Uh oh!

blueorangutan commented Nov 1, 2026

Uh oh!

blueorangutan commented Nov 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

andrijapanicsb commented Oct 30, 2026

Description

Who runs into this

What changes

What it never does

Not covered

Types of changes

Feature/Enhancement Scale or Bug Severity

Bug Severity

How Has This Been Tested?

Uh oh!

codecov Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

andrijapanicsb commented Oct 30, 2026

Uh oh!

blueorangutan commented Oct 30, 2026

Uh oh!

blueorangutan commented Oct 30, 2026

Uh oh!

nvazquez commented Nov 1, 2026

Uh oh!

blueorangutan commented Nov 1, 2026

Uh oh!

blueorangutan commented Nov 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Oct 30, 2026 •

edited

Loading