| Message ID | cover.1667542066.git.john.g.johnson@oracle.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | vfio-user client | expand |
Cc'ing Mark & Edgar. On 9/11/22 00:13, John Johnson wrote: > Hello, > > This is the 6th revision of the vfio-user client implementation. > It is the first patch series (the previous revisions were RFCs) > > First of all, thank you for your time reviewing the RFC versions. > > The vfio-user framework consists of 3 parts: > 1) The VFIO user protocol specification. > 2) A client - the VFIO device in QEMU that encapsulates VFIO messages > and sends them to the server. > 3) A server - a remote process that emulates a device. > > This patchset implements parts 1 and 2. > > The libvfio-user project (https://github.com/nutanix/libvfio-user) > can be used by a remote process to handle the protocol to implement the third part. > We also have upstreamed a patch series that implement a server using QEMU. > > > Contributors: > > John G Johnson <john.g.johnson@oracle.com> > John Levon <john.levon@nutanix.com> > Thanos Makatos <thanos.makatos@nutanix.com> > Elena Ufimtseva <elena.ufimtseva@oracle.com> > Jagannathan Raman <jag.raman@oracle.com> > > John Johnson (23): > vfio-user: add VFIO base abstract class > vfio-user: add container IO ops vector > vfio-user: add region cache > vfio-user: add device IO ops vector > vfio-user: Define type vfio_user_pci_dev_info > vfio-user: connect vfio proxy to remote server > vfio-user: define socket receive functions > vfio-user: define socket send functions > vfio-user: get device info > vfio-user: get region info > vfio-user: region read/write > vfio-user: pci_user_realize PCI setup > vfio-user: get and set IRQs > vfio-user: forward msix BAR accesses to server > vfio-user: proxy container connect/disconnect > vfio-user: dma map/unmap operations > vfio-user: add dma_unmap_all > vfio-user: secure DMA support > vfio-user: dma read/write operations > vfio-user: pci reset > vfio-user: add 'x-msg-timeout' option that specifies msg wait times > vfio-user: add coalesced posted writes > vfio-user: add trace points
On 11/9/22 00:13, John Johnson wrote: > > Hello, > > This is the 6th revision of the vfio-user client implementation. > It is the first patch series (the previous revisions were RFCs) > > First of all, thank you for your time reviewing the RFC versions. > > The vfio-user framework consists of 3 parts: > 1) The VFIO user protocol specification. > 2) A client - the VFIO device in QEMU that encapsulates VFIO messages > and sends them to the server. > 3) A server - a remote process that emulates a device. > > This patchset implements parts 1 and 2. > > The libvfio-user project (https://github.com/nutanix/libvfio-user) > can be used by a remote process to handle the protocol to implement the third part. > We also have upstreamed a patch series that implement a server using QEMU. > Thanks for this contribution, This is a large and complex framework which looks quite mature. QEMU would need implementations of remote devices under contrib/, provided as examples for the QEMU crowd. These are important to show possible use cases and also, they could possibly be run for non regression tests done by maintainers and distros. e1000e is quite popular, it would be a good target. It could be a simpler device to start with, but we would need to cover the basic features, INTx, MSI, DMA, etc. when time permits. There are samples under libvfio-user and I wonder how we could leverage them. Unit tests under /tests would be (really) good to have. These would be run by CI. Yes, this is a lot of work :/ but very important for QEMU stability. The "socket" name property is not the preferred way in QEMU to define a remote peer to contact. Is there a possibility to use the chardev interface in some simple way ? I am thinking about the command line interface and the integration with libvirt. More code should be isolated under the *user.c file, even if that means exporting definitions of routines which are local today. I don't think the #ifdef CONFIG_VIO_USER in the vfio files are something we will want to merge. Some renaming to be done, like Kern -> Kernel, etc. nothing major. I am not convinced that the macros hiding the VFIO backend IO ops are useful. I even recall some places where the vfio-user implemented handler could be called directly without calling the top routine first. Anyhow, it would be better to be explicit, the macros don't add much. There is a lot of code duplication for the IOMMU support. Did you consider implementing a VFIOGroup class to share the common behaviour ? May be too early, but this is certainly something to keep in mind. The msg recycling feature looks nice but isn't it an optimization ? More globally speaking, for what kind of crazy setup this feature could help us with ? I was wondering if you had tried to implement a remote device or bridge in another QEMU process, for instance. Thanks, C. > Contributors: > > John G Johnson <john.g.johnson@oracle.com> > John Levon <john.levon@nutanix.com> > Thanos Makatos <thanos.makatos@nutanix.com> > Elena Ufimtseva <elena.ufimtseva@oracle.com> > Jagannathan Raman <jag.raman@oracle.com> > > John Johnson (23): > vfio-user: add VFIO base abstract class > vfio-user: add container IO ops vector > vfio-user: add region cache > vfio-user: add device IO ops vector > vfio-user: Define type vfio_user_pci_dev_info > vfio-user: connect vfio proxy to remote server > vfio-user: define socket receive functions > vfio-user: define socket send functions > vfio-user: get device info > vfio-user: get region info > vfio-user: region read/write > vfio-user: pci_user_realize PCI setup > vfio-user: get and set IRQs > vfio-user: forward msix BAR accesses to server > vfio-user: proxy container connect/disconnect > vfio-user: dma map/unmap operations > vfio-user: add dma_unmap_all > vfio-user: secure DMA support > vfio-user: dma read/write operations > vfio-user: pci reset > vfio-user: add 'x-msg-timeout' option that specifies msg wait times > vfio-user: add coalesced posted writes > vfio-user: add trace points > > Thanos Makatos (1): > vfio-user: introduce vfio-user protocol specification > > MAINTAINERS | 11 + > docs/devel/index-internals.rst | 1 + > docs/devel/vfio-user.rst | 1522 ++++++++++++++++++++++++++++++++ > hw/vfio/Kconfig | 10 + > hw/vfio/ap.c | 1 + > hw/vfio/ccw.c | 7 +- > hw/vfio/common.c | 648 ++++++++++++-- > hw/vfio/igd.c | 23 +- > hw/vfio/meson.build | 1 + > hw/vfio/migration.c | 2 - > hw/vfio/pci-quirks.c | 19 +- > hw/vfio/pci.c | 926 ++++++++++++++----- > hw/vfio/pci.h | 29 +- > hw/vfio/platform.c | 2 + > hw/vfio/trace-events | 15 + > hw/vfio/user-protocol.h | 244 +++++ > hw/vfio/user.c | 1904 ++++++++++++++++++++++++++++++++++++++++ > hw/vfio/user.h | 112 +++ > include/hw/vfio/vfio-common.h | 82 ++ > 19 files changed, 5230 insertions(+), 329 deletions(-) > create mode 100644 docs/devel/vfio-user.rst > create mode 100644 hw/vfio/user-protocol.h > create mode 100644 hw/vfio/user.c > create mode 100644 hw/vfio/user.h >
> On Dec 16, 2022, at 3:31 AM, Cédric Le Goater <clg@redhat.com> wrote: > > On 11/9/22 00:13, John Johnson wrote: >> Hello, >> This is the 6th revision of the vfio-user client implementation. >> It is the first patch series (the previous revisions were RFCs) >> First of all, thank you for your time reviewing the RFC versions. >> The vfio-user framework consists of 3 parts: >> 1) The VFIO user protocol specification. >> 2) A client - the VFIO device in QEMU that encapsulates VFIO messages >> and sends them to the server. >> 3) A server - a remote process that emulates a device. >> This patchset implements parts 1 and 2. >> The libvfio-user project (https://github.com/nutanix/libvfio-user) >> can be used by a remote process to handle the protocol to implement the third part. >> We also have upstreamed a patch series that implement a server using QEMU. > > Thanks for this contribution, > Thank you for taking the time to review it > > This is a large and complex framework which looks quite mature. QEMU > would need implementations of remote devices under contrib/, provided > as examples for the QEMU crowd. These are important to show possible > use cases and also, they could possibly be run for non regression tests > done by maintainers and distros. e1000e is quite popular, it would > be a good target. It could be a simpler device to start with, but we > would need to cover the basic features, INTx, MSI, DMA, etc. when time > permits. There are samples under libvfio-user and I wonder how we > could leverage them. > A QEMU server patch is already upstream, which allows QEMU to act as a vfio-user device server. We’ve tested with some disk devices (mpt, lsi895) > Unit tests under /tests would be (really) good to have. These would be > run by CI. Yes, this is a lot of work :/ but very important for QEMU > stability. > We have an avocado test using an mpt disk controller that didn’t get pushed with the server changes since the client isn’t upstream. I’ll add it to this patch series. > The "socket" name property is not the preferred way in QEMU to define > a remote peer to contact. Is there a possibility to use the chardev > interface in some simple way ? I am thinking about the command line > interface and the integration with libvirt. > I was asked to look at using the socket command line parser, but its visitor function doesn’t work with device property list options, so the compromise was to use a SocketAddress as the argument to vfio_user_connect_dev, so vfio-user can be changed without much work if the visitor changes. > More code should be isolated under the *user.c file, even if that > means exporting definitions of routines which are local today. I don't > think the #ifdef CONFIG_VIO_USER in the vfio files are something we > will want to merge. > > Some renaming to be done, like Kern -> Kernel, etc. nothing major. > > I am not convinced that the macros hiding the VFIO backend IO ops are > useful. I even recall some places where the vfio-user implemented > handler could be called directly without calling the top routine first. > Anyhow, it would be better to be explicit, the macros don't add much. > I have a patch series that renames the struct to your preference and removes the virtual function macros. It will be out shortly. > There is a lot of code duplication for the IOMMU support. Did you > consider implementing a VFIOGroup class to share the common behaviour ? > May be too early, but this is certainly something to keep in mind. > I did refactor the IOMMU group code so vfio-user and the kernel version can share common.c routines. I hope this helps. > The msg recycling feature looks nice but isn't it an optimization ? > It was done more as a place to hold bug catching code than as an optimization. e.g., If I misplace a msg, I can usually find it on free list. > More globally speaking, for what kind of crazy setup this feature could > help us with ? I was wondering if you had tried to implement a remote > device or bridge in another QEMU process, for instance. > \ We arrived at this setup from a KVM forum in 2019, where several groups wanted to do device emulation outside the VMM for security and perfomance reasons. We merged a couple projects together into vfio-user. Thanks again for the feedback. JJ
Hello, This is the 6th revision of the vfio-user client implementation. It is the first patch series (the previous revisions were RFCs) First of all, thank you for your time reviewing the RFC versions. The vfio-user framework consists of 3 parts: 1) The VFIO user protocol specification. 2) A client - the VFIO device in QEMU that encapsulates VFIO messages and sends them to the server. 3) A server - a remote process that emulates a device. This patchset implements parts 1 and 2. The libvfio-user project (https://github.com/nutanix/libvfio-user) can be used by a remote process to handle the protocol to implement the third part. We also have upstreamed a patch series that implement a server using QEMU. Contributors: John G Johnson <john.g.johnson@oracle.com> John Levon <john.levon@nutanix.com> Thanos Makatos <thanos.makatos@nutanix.com> Elena Ufimtseva <elena.ufimtseva@oracle.com> Jagannathan Raman <jag.raman@oracle.com> John Johnson (23): vfio-user: add VFIO base abstract class vfio-user: add container IO ops vector vfio-user: add region cache vfio-user: add device IO ops vector vfio-user: Define type vfio_user_pci_dev_info vfio-user: connect vfio proxy to remote server vfio-user: define socket receive functions vfio-user: define socket send functions vfio-user: get device info vfio-user: get region info vfio-user: region read/write vfio-user: pci_user_realize PCI setup vfio-user: get and set IRQs vfio-user: forward msix BAR accesses to server vfio-user: proxy container connect/disconnect vfio-user: dma map/unmap operations vfio-user: add dma_unmap_all vfio-user: secure DMA support vfio-user: dma read/write operations vfio-user: pci reset vfio-user: add 'x-msg-timeout' option that specifies msg wait times vfio-user: add coalesced posted writes vfio-user: add trace points Thanos Makatos (1): vfio-user: introduce vfio-user protocol specification MAINTAINERS | 11 + docs/devel/index-internals.rst | 1 + docs/devel/vfio-user.rst | 1522 ++++++++++++++++++++++++++++++++ hw/vfio/Kconfig | 10 + hw/vfio/ap.c | 1 + hw/vfio/ccw.c | 7 +- hw/vfio/common.c | 648 ++++++++++++-- hw/vfio/igd.c | 23 +- hw/vfio/meson.build | 1 + hw/vfio/migration.c | 2 - hw/vfio/pci-quirks.c | 19 +- hw/vfio/pci.c | 926 ++++++++++++++----- hw/vfio/pci.h | 29 +- hw/vfio/platform.c | 2 + hw/vfio/trace-events | 15 + hw/vfio/user-protocol.h | 244 +++++ hw/vfio/user.c | 1904 ++++++++++++++++++++++++++++++++++++++++ hw/vfio/user.h | 112 +++ include/hw/vfio/vfio-common.h | 82 ++ 19 files changed, 5230 insertions(+), 329 deletions(-) create mode 100644 docs/devel/vfio-user.rst create mode 100644 hw/vfio/user-protocol.h create mode 100644 hw/vfio/user.c create mode 100644 hw/vfio/user.h