Skip to content

Add PCI hotplug support#5786

Open
ilstam wants to merge 8 commits intofirecracker-microvm:mainfrom
ilstam:hotplug
Open

Add PCI hotplug support#5786
ilstam wants to merge 8 commits intofirecracker-microvm:mainfrom
ilstam:hotplug

Conversation

@ilstam
Copy link
Copy Markdown
Contributor

@ilstam ilstam commented Mar 23, 2026

Add PCI hotplug support. No hotplug notification mechanism is implemented yet, so the the guest needs to rescan the PCI "bus" manually in order to see new attachments.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

  • I have read and understand CONTRIBUTING.md.
  • I have run tools/devtool checkbuild --all to verify that the PR passes
    build checks on all supported architectures.
  • I have run tools/devtool checkstyle to verify that the PR passes the
    automated style checks.
  • I have described what is done in these changes, why they are needed, and
    how they are solving the problem in a clear and encompassing way.
  • I have updated any relevant documentation (both in code and in the docs)
    in the PR.
  • I have mentioned all user-facing changes in CHANGELOG.md.
  • If a specific issue led to this PR, this PR closes the issue.
  • When making API changes, I have followed the
    Runbook for Firecracker API changes.
  • I have tested all new and changed functionalities in unit tests and/or
    integration tests.
  • I have linked an issue to every new TODO.

  • This functionality cannot be added in rust-vmm.

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 23, 2026

Codecov Report

❌ Patch coverage is 61.70213% with 54 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.97%. Comparing base (86aac24) to head (cdcf610).

Files with missing lines Patch % Lines
src/firecracker/src/api_server_adapter.rs 0.00% 23 Missing ⚠️
src/vmm/src/rpc_interface.rs 20.83% 19 Missing ⚠️
src/vmm/src/lib.rs 0.00% 9 Missing ⚠️
src/vmm/src/device_manager/pci_mngr.rs 96.82% 2 Missing ⚠️
src/vmm/src/devices/virtio/pmem/device.rs 88.88% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5786      +/-   ##
==========================================
- Coverage   83.04%   82.97%   -0.08%     
==========================================
  Files         276      276              
  Lines       29466    29580     +114     
==========================================
+ Hits        24471    24544      +73     
- Misses       4995     5036      +41     
Flag Coverage Δ
5.10-m5n.metal 83.24% <61.70%> (-0.09%) ⬇️
5.10-m6a.metal 82.57% <61.70%> (-0.08%) ⬇️
5.10-m6g.metal 79.89% <61.70%> (-0.08%) ⬇️
5.10-m6i.metal 83.24% <61.70%> (-0.09%) ⬇️
5.10-m7a.metal-48xl 82.57% <61.70%> (-0.08%) ⬇️
5.10-m7g.metal 79.89% <61.70%> (-0.08%) ⬇️
5.10-m7i.metal-24xl 83.22% <61.70%> (-0.08%) ⬇️
5.10-m7i.metal-48xl 83.21% <61.70%> (-0.09%) ⬇️
5.10-m8g.metal-24xl 79.88% <61.70%> (-0.08%) ⬇️
5.10-m8g.metal-48xl 79.88% <61.70%> (-0.08%) ⬇️
5.10-m8i.metal-48xl 83.21% <61.70%> (-0.09%) ⬇️
5.10-m8i.metal-96xl 83.22% <61.70%> (-0.09%) ⬇️
6.1-m5n.metal 83.27% <61.70%> (-0.08%) ⬇️
6.1-m6a.metal 82.60% <61.70%> (-0.09%) ⬇️
6.1-m6g.metal 79.88% <61.70%> (-0.08%) ⬇️
6.1-m6i.metal 83.27% <61.70%> (-0.08%) ⬇️
6.1-m7a.metal-48xl 82.59% <61.70%> (-0.09%) ⬇️
6.1-m7g.metal 79.88% <61.70%> (-0.08%) ⬇️
6.1-m7i.metal-24xl 83.28% <61.70%> (-0.09%) ⬇️
6.1-m7i.metal-48xl 83.28% <61.70%> (-0.09%) ⬇️
6.1-m8g.metal-24xl 79.88% <61.70%> (-0.08%) ⬇️
6.1-m8g.metal-48xl 79.88% <61.70%> (-0.08%) ⬇️
6.1-m8i.metal-48xl 83.29% <61.70%> (-0.09%) ⬇️
6.1-m8i.metal-96xl 83.29% <61.70%> (-0.09%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ilstam added 7 commits March 31, 2026 18:10
ApiServerAdapter has a handle_request() method for dispatching requests
received from the API thread to the right handler. This method is called
from inside EventManager::run() which takes &mut self as an argument.
This can be problematic if we want to modify the EventManager object
from inside handle_request().

Work around that by having ApiServerAdapter::process() store the request
but not call handle_request(). EventManager::run() returns after
handling each event. Call handle_request() after EventManager::run() in
the event loop.

In a subsequent patch, handle_request() will take a &mut EventManager as
an argument.

Signed-off-by: Ilias Stamatis <ilstam@amazon.com>
The handle_request() function will soon need access to the EventManager
in order to add and remove new objects to support device hot-plugging.
Pass a mutable reference of EventManager to handle_request() in
preparation of that.

Signed-off-by: Ilias Stamatis <ilstam@amazon.com>
Pmem::alloc_region() panics if it fails to allocate the address range it
needs. That is especially problematic for hot-plugging support, since it
means failing to attach a device would kill the entire VM. Remove the
unwrap and introduce a new error code that is propagated to the caller
in case of failure.

Signed-off-by: Ilias Stamatis <ilstam@amazon.com>
The EntropyDevice and PmemDevice error names are inconsistent with the
error names used for all other devices all of which use the Config
suffix. Rename them to EntropyConfig and PmemConfig for consistency.

Signed-off-by: Ilias Stamatis <ilstam@amazon.com>
Firecracker has been rejecting device attach API requests after the VM
was started until now.

Add hot-plugging for Block, Pmem and Net PCIe devices. This enables the
relevant API calls and attaches the device to the PCIe "bus".

No notification is delivered to the guest at the moment to notify it
that a new device has been added. The guest has to manually rescan the
bus in order to detect new devices.

Signed-off-by: Ilias Stamatis <ilstam@amazon.com>
The device hotplug path creates devices after boot, when seccomp
filters are already active. This requires allowing syscalls that were
previously only called during boot before seccomp was installed.

Add the following to the vmm thread filter:

- timerfd_create: RateLimiter creates a TimerFd for each new device
- ioctl(KVM_IOEVENTFD): registering ioeventfds for virtqueue
  notification
- ioctl(TUNSETIFF): opening tap device for net hotplug
- ioctl(TUNSETOFFLOAD): configuring tap offload for net hotplug
- ioctl(TUNSETVNETHDRSZ): setting vnet header size for net hotplug
- mmap(MAP_SHARED|MAP_NORESERVE): pmem backing file mapping
- mmap(MAP_PRIVATE|MAP_NORESERVE|MAP_ANONYMOUS): pmem aligned region
- mmap(MAP_SHARED|MAP_NORESERVE|MAP_FIXED): pmem file overlay mapping
- mmap(MAP_SHARED|MAP_FIXED): IovDeque ring buffer for net device
- memfd_create: IovDeque ring buffer for net device
- fcntl(F_ADD_SEALS): IovDeque memfd sealing for net device

Signed-off-by: Ilias Stamatis <ilstam@amazon.com>
Add unit tests for testing the possible failure modes of hotplugging
different classes of devices.

Signed-off-by: Ilias Stamatis <ilstam@amazon.com>
Add integration tests for block, pmem and net hotplugging.

The tests require a manual PCI bus rescan at the moment since no hotplug
notification mechanism is implemented at the moment.

Signed-off-by: Ilias Stamatis <ilstam@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant