| Message ID | 20230802081449.2528-1-avihaih@nvidia.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | vfio/migration: Add P2P support for VFIO migration | expand |
Test result of multiple Mellanox CX-7 VFs migration: PASS [1] create two VFs and bind them to mlx5_vfio_pci driver [2] start a VM with two VFs [3] migrate the VM [4] check if the VM works well Tested-by: YangHang Liu <yanghliu@redhat.com> Best Regards, YangHang Liu On Wed, Aug 2, 2023 at 4:16 PM Avihai Horon <avihaih@nvidia.com> wrote: > > Hi all, > > The first patch in this series adds a small optimization to VFIO > migration by moving the STOP_COPY->STOP transition to > vfio_save_cleanup(). Testing with a ConnectX-7 VFIO device showed that > this can reduce downtime by up to 6%. > > The rest of the series adds P2P support for VFIO migration. > > VFIO migration uAPI defines an optional intermediate P2P quiescent > state. While in the P2P quiescent state, P2P DMA transactions cannot be > initiated by the device, but the device can respond to incoming ones. > Additionally, all outstanding P2P transactions are guaranteed to have > been completed by the time the device enters this state. > > The purpose of this state is to support migration of multiple devices > that might do P2P transactions between themselves. > > To implement P2P migration support, all the devices must be transitioned > to the P2P quiescent state before being stopped or started. > > This behavior is achieved by adding an optional prepare callback to > VMChangeStateEntry. These callbacks are invoked before the main VM state > change callbacks, transitioning all the VFIO devices to the P2P state, > and only then are the main callbacks invoked, which stop or start the > devices. > > This will allow migration of multiple VFIO devices if all of them > support P2P migration. > > Thanks. > > Changes from v2 [2]: > * Registered VFIO prepare callback only if P2P migration is supported by > the device. (Cedric) > * Added Cedric R-b tags to patches 2,3 and 6. > > Changes from v1 [1]: > * Rebased on latest master branch. > * Renamed pre_change_cb to prepare_cb and adjusted relevant code > accordingly. (Cedric) > * Split VFIO vmstate change handler to two separate handlers, one for > prepare callback and another for main callback. (Cedric) > * Renamed vfio_should_block_multiple_devices_migration() to > vfio_multiple_devices_migration_is_supported() and reverted its logic > accordingly. (Cedric) > * Rephrased "that are doing P2P" to "that might do P2P" in docs and > commit message. (Jason) > * Added Cedric R-b tag to patch #4. > > [1] > https://lore.kernel.org/qemu-devel/20230716081541.27900-1-avihaih@nvidia.com/ > > [2] > https://lore.kernel.org/qemu-devel/20230731102521.15335-1-avihaih@nvidia.com/ > > Avihai Horon (5): > vfio/migration: Move from STOP_COPY to STOP in vfio_save_cleanup() > sysemu: Add prepare callback to struct VMChangeStateEntry > qdev: Add qdev_add_vm_change_state_handler_full() > vfio/migration: Add P2P support for VFIO migration > vfio/migration: Allow migration of multiple P2P supporting devices > > Joao Martins (1): > vfio/migration: Refactor PRE_COPY and RUNNING state checks > > docs/devel/vfio-migration.rst | 93 +++++++++++++++++++------------ > include/hw/vfio/vfio-common.h | 2 + > include/sysemu/runstate.h | 7 +++ > hw/core/vm-change-state-handler.c | 14 ++++- > hw/vfio/common.c | 50 +++++++++++++---- > hw/vfio/migration.c | 75 ++++++++++++++++++++----- > softmmu/runstate.c | 40 +++++++++++++ > hw/vfio/trace-events | 1 + > 8 files changed, 218 insertions(+), 64 deletions(-) > > -- > 2.26.3 > >