[v1,0/2] vfio/pci: Allow MMIO regions to be exported through dma-buf

Message

Kasireddy, Vivek April 22, 2024, 6:30 a.m. UTC

This is an attempt to revive the patches posted by Jason Gunthorpe at:
https://patchwork.kernel.org/project/linux-media/cover/[e-mail]/

Here is the cover letter text from Jason's original series:
"dma-buf has become a way to safely acquire a handle to non-struct page
memory that can still have lifetime controlled by the exporter. Notably
RDMA can now import dma-buf FDs and build them into MRs which allows for
PCI P2P operations. Extend this to allow vfio-pci to export MMIO memory
from PCI device BARs.

This series supports a use case for SPDK where a NVMe device will be owned
by SPDK through VFIO but interacting with a RDMA device. The RDMA device
may directly access the NVMe CMB or directly manipulate the NVMe device's
doorbell using PCI P2P.

However, as a general mechanism, it can support many other scenarios with
VFIO. I imagine this dmabuf approach to be usable by iommufd as well for
generic and safe P2P mappings.

This series goes after the "Break up ioctl dispatch functions to one
function per ioctl" series."

In addition to the SPDK use-case mentioned above, the capability added
in this patch series can also be useful when a buffer (located in device
memory such as VRAM) needs to be shared between any two GPU devices or
instances (assuming one of them is bound to VFIO PCI) as long as they
are P2P DMA compatible.

The main difference between this series and the original one is the usage
of P2P DMA APIs to create struct pages (ZONE_DEVICE) to populate the
scatterlist instead of using DMA addresses. Other additions include a
mmap handler to provide CPU access to the dmabuf and support for
creating the dmabuf from multiple areas (or ranges).

This series is available at:
https://gitlab.freedesktop.org/Vivek/drm-tip/-/commits/vfio_dmabuf_v1

along with additional patches for Qemu and Spice here:
https://gitlab.freedesktop.org/Vivek/qemu/-/commits/vfio_dmabuf_1
https://gitlab.freedesktop.org/Vivek/spice/-/commits/encode_dmabuf_v4 

This series is tested using the following method:
- Run Qemu with the following relevant options:
  qemu-system-x86_64 -m 4096m ....
  -device vfio-pci,host=0000:03:00.0
  -device virtio-vga,max_outputs=1,blob=true,xres=1920,yres=1080
  -spice port=3001,gl=on,disable-ticketing=on,preferred-codec=gstreamer:h264
  -object memory-backend-memfd,id=mem1,size=4096M
  -machine memory-backend=mem1 ...
- Run upstream Weston with the following options in the Guest VM:
  ./weston --drm-device=card1 --additional-devices=card0

where card1 is a DG2 dGPU (assigned to vfio-pci) and card0 is
virtio-gpu.

Cc: Jason Gunthorpe <[e-mail]>
Cc: Christoph Hellwig <[e-mail]>
Cc: Robin Murphy <[e-mail]>
Cc: Christian König <[e-mail]>
Cc: Daniel Vetter <[e-mail]>
Cc: Oded Gabbay <[e-mail]>
Cc: Gerd Hoffmann <[e-mail]>
Cc: Alex Williamson <[e-mail]>
Cc: Kevin Tian <[e-mail]>

Vivek Kasireddy (2):
  vfio: Export vfio device get and put registration helpers
  vfio/pci: Allow MMIO regions to be exported through dma-buf

 drivers/vfio/pci/Makefile          |   1 +
 drivers/vfio/pci/dma_buf.c         | 348 +++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_config.c |   8 +-
 drivers/vfio/pci/vfio_pci_core.c   |  28 ++-
 drivers/vfio/pci/vfio_pci_priv.h   |  23 ++
 drivers/vfio/vfio_main.c           |   2 +
 include/linux/vfio.h               |   2 +
 include/linux/vfio_pci_core.h      |   1 +
 include/uapi/linux/vfio.h          |  25 +++
 9 files changed, 430 insertions(+), 8 deletions(-)
 create mode 100644 drivers/vfio/pci/dma_buf.c