[RFC,v4,00/21] vfio-user client

Message

John Johnson Jan. 12, 2022, 12:43 a.m. UTC

Hello,

This is the 4th revision of the vfio-user client implementation.

First of all, thank you for your time reviewing the previous versions.

The vfio-user framework consists of 3 parts:
 1) The VFIO user protocol specification.
 2) A client - the VFIO device in QEMU that encapsulates VFIO messages
    and sends them to the server.
 3) A server - a remote process that emulates a device.

This patchset implements parts 1 and 2.

The libvfio-user project (https://github.com/nutanix/libvfio-user)
can be used by a remote process to handle the protocol to implement the third part.
We also have sent a patch series that implement a server using QEMU.


Contributors:

John G Johnson <[e-mail]>
John Levon <[e-mail]>
Thanos Makatos <[e-mail]>
Elena Ufimtseva <[e-mail]>
Jagannathan Raman <[e-mail]>


Changes from v3->v4:

 vfio-user: introduce vfio-user protocol specification
   No v4 specific changess

 vfio-user: add VFIO base abstract class
   Put all properties except those specific to the ioctl() implementation in the base class 

 vfio-user: add container IO ops vector
   Move will_commit support to dma map/unmap patch below
   Use ternary return expression in IO ops vectors

 vfio-user: add region cache
   New patch with only region cache support
   Make vfio_get_region_info return region reference instead of a copy

 vfio-user: add device IO ops vector
   Move posted write support to region read/write patch below
   Move FD receiving code to get region info patch below
   Add VDEV_CONFIG_READ/WRITE macros to pci.c for convenient access to PCI config space
   Use ternary return expression in IO ops vectors

 vfio-user: Define type vfio_user_pci_dev_info
   Move secure DMA support to separate patch below
   Remove dummy function for vfio_hot_reset_multi ops vector
   Add vfio_user_instance_finalize code from connect proxy patch below

 vfio-user: connect vfio proxy to remote server
   Move vfio_user_instance_finalize code to define type patch above

 vfio-user: define socket receive functions
   Handle kernel splitting message from server into multiple read()s
   Fix incoming message queue handling in vfio_user_request()
   Move secure DMA support to separate patch below
   Move MAX_FDS and MAX_XFER defines to socket send patch below

 vfio-user: define socket send functions
   Free pending messages when the reply times out
   Add MAX_FDS and MAX_XFER defines from socket recv patch above
   Don't set error twice on a capabilities parsing error

 vfio-user: get device info
   Add vfio_get_all_regions() call
   Validate device info return values from server

 vfio-user: get region info
   Add FD receiving code from device IO ops patch above
   Add a generic FD to VFIORegion for mapping device regions
   Validate region info return values from server

 vfio-user: region read/write
   Add posted write support from device IO ops patch above
   Check region read/write count against max_xfer

 vfio-user: pci_user_realize PCI setup
    Refactor realize functions to use common setup functions

 vfio-user: get and set IRQs
   Validate irq return values from server

 vfio-user: proxy container connect/disconnect
   No v4 specific changes

 vfio-user: dma map/unmap operations
   Add wlll_commit support from container IO ops patch above
   Rename will_commit to async_ops to describe its operation better
   Pass memory region to dma_map op so only vfio-user needs to look up FD
   Free pending messages when the reply times out
   Move secure DMA support to separate patch below
   Set argz in dma_unmap message according to spec

 vfio-user: secure DMA support
   New patch consolidating all secure DMA support

 vfio-user: dma read/write operations
 vfio-user: pci reset
   No v4 specific changes

 vfio-user: migration support
   Move qemu file errors fix to its own patch below
   Set argz in get_dirty_bitmap message according to spec

 Only set qemu file error if saving state if the file exists
   New patch with just this fix found during vfio-user development

Removed from v4:

 Add validation ops vector
   Generic checking moved to the corresponding vfio-user function


Changes from v2->v3:

John Johnson (18):
  vfio-user: add VFIO base abstract class
    Moved common vfio pci cli options to base class

  Add container IO ops vector
    Added ops vectors to decide to use ioctl() or socket implementation

  Add device IO ops vector
    Added ops vectors to decide to use ioctl() or socket implementation

  Add validation ops vector
    Added validation vector to check user replies

  vfio-user: Define type vfio_user_pci_dev_info
    Added separate VFIO_USER_PCI config element to control whether vfio-user is compiled
    Fix scalar spelling

  vfio-user: connect vfio proxy to remote server
    Made socket IO non-blocking
    Use g_strdup_printf to save socket name

  vfio-user: define socket receive functions
    Made socket IO non-blocking
    Process inbound commands in main loop thread to avoid BQL interactions with recv
    Added comment describing inbound command callback usage
    Use true/false instead of 1/0 for booleans

  vfio-user: define socket send functions
    Made socket IO non-blocking
    Added version string NULL termination check

  vfio-user: get device info
    Added ops vectors to decide to use ioctl() or socket implementation
    Added validation vector to check user replies

  vfio-user: get region info
    Added ops vectors to decide to use ioctl() or socket implementation
    Added validation vector to check user replies
    Remove merge bug that filled region cache twice

  vfio-user: region read/write
    Added ops vectors to decide to use ioctl() or socket implementation
    Added validation vector to check user replies
    Made posted write conditional on region not mapped

  vfio-user: pci_user_realize PCI setup
    Moved common vfio pci cli options to base class

  vfio-user: get and set IRQs
    Added ops vectors to decide to use ioctl() or socket implementation
    Added validation vector to check user replies
    Fixed %m usage when not using syscall

  vfio-user: proxy container connect/disconnect
    Added separate VFIO_USER_PCI config element to control whether vfio-user is compiled
    Use true/false instead of 1/0 for booleans

  vfio-user: dma map/unmap operations
    Added ops vectors to decide to use ioctl() or socket implementation
    Use BQL instead of iolock in comments
    Fixed %m usage when not using syscall

  vfio-user: dma read/write operations
    Added header checking before loading DMA message content
    Added error handling if DMA fails

  vfio-user: pci reset
    no r3-specific changes

  vfio-user: migration support
    generic fix: only set qemu file error if there is a file

Thanos Makatos (1):
  vfio-user: introduce vfio-user protocol specification
    Spec specifies host endiannes instead of always LE
    Fixed grammar error




 docs/devel/index.rst          |    1 +
 docs/devel/vfio-user.rst      | 1810 +++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.h                 |   27 +-
 hw/vfio/user-protocol.h       |  221 +++++
 hw/vfio/user.h                |   96 +++
 include/hw/vfio/vfio-common.h |   73 ++
 hw/vfio/ccw.c                 |    5 -
 hw/vfio/common.c              |  477 +++++++++--
 hw/vfio/migration.c           |   34 +-
 hw/vfio/pci-quirks.c          |   19 +-
 hw/vfio/pci.c                 |  856 ++++++++++++++-----
 hw/vfio/user.c                | 1596 ++++++++++++++++++++++++++++++++++++
 MAINTAINERS                   |   11 +
 hw/vfio/Kconfig               |   10 +
 hw/vfio/meson.build           |    1 +
 15 files changed, 4919 insertions(+), 318 deletions(-)
 create mode 100644 docs/devel/vfio-user.rst
 create mode 100644 hw/vfio/user-protocol.h
 create mode 100644 hw/vfio/user.h
 create mode 100644 hw/vfio/user.c