[RFC,0/3] vfio/ims: Back guest interrupts from Interrupt Message Store (IMS)

Message

Reinette Chatre Aug. 24, 2023, 4:15 p.m. UTC

Hi Everybody,

With Interrupt Message Store (IMS) support introduced in
commit 0194425af0c8 ("PCI/MSI: Provide IMS (Interrupt Message Store)
support") a device can create a secondary interrupt domain that works
side by side with MSI-X on the same device. IMS allows for
implementation-specific interrupt storage that is managed by the
implementation specific interrupt chip associated with the IMS domain
at the time it (the IMS domain) is created for the device via
pci_create_ims_domain().

An example usage of IMS is for devices that can have their resources
assigned to guests with varying granularity. For example, an
accelerator device may support many workqueues and a single workqueue
can be composed into a virtual device for use by a guest. Using
IMS interrupts for the guest preserves MSI-X for host usage while
allowing a significantly larger number of interrupt vectors than
allowed by MSI-X. All while enabling usage of the same device driver
within the host and guest.

This series introduces a VFIO library (VFIO PCI IMS) for use by
virtual devices that support MSI-X interrupts that are backed by IMS
interrupts on the host. Specifically, that means that when the virtual
device's VFIO_DEVICE_SET_IRQS ioctl() receives a "trigger interrupt"
(VFIO_IRQ_SET_ACTION_TRIGGER) for a MSI-X index then VFIO PCI IMS
allocates/frees an IMS interrupt on the host.

An IMS interrupt is allocated via pci_ims_alloc_irq() that requires
an implementation specific cookie that is opaque to VFIO PCI IMS. This
can be a PASID, queue ID, pointer etc. During initialization
VFIO PCI IMS learns which PCI device to operate on and what the
default cookie should be for any new interrupt allocation. VFIO PCI
IMS can also associate a unique cookie with each vector and to maintain
this association the library maintains interrupt contexts for the virtual
device's lifetime.

Guests may access a virtual device via both 'direct-path', where the
guest interacts directly with the underlying hardware, and 'intercepted
path', where the virtual device emulates operations. VFIO PCI IMS
supports emulated interrupts (better naming suggestions are welcome) to
handle 'intercepted path' operations where completion interrupts are
signaled from the virtual device, not the underlying hardware.

This library has been tested with a yet to be published VFIO
driver for the Intel Data Accelerators (IDXD) present in Intel Xeon
CPUs.

While this series contains a working implementation it is presented
as an RFC with the goal to obtain feedback on whether VFIO PCI IMS
is appropriate for inclusion into VFIO and whether it is
(or could be adapted to be) appropriate for support of other
planned IMS usages you may be aware of.

Any feedback will be greatly appreciated.

Reinette

Reinette Chatre (3):
  vfio/pci: Introduce library allocating from Interrupt Message Store
    (IMS)
  vfio/ims: Support emulated interrupts
  vfio/ims: Add helper that returns IMS index

 drivers/vfio/pci/Kconfig        |  12 +
 drivers/vfio/pci/Makefile       |   2 +
 drivers/vfio/pci/vfio_pci_ims.c | 499 ++++++++++++++++++++++++++++++++
 include/linux/vfio.h            |  81 ++++++
 4 files changed, 594 insertions(+)
 create mode 100644 drivers/vfio/pci/vfio_pci_ims.c


base-commit: a5e505a99ca748583dbe558b691be1b26f05d678