| Message ID | 20210720081647.1980-1-yishaih@nvidia.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Introduce mlx5 user space driver over VFIO | expand |
On 7/20/2021 11:16 AM, Yishai Hadas wrote: > This series introduces mlx5 user space driver over VFIO. > > This enables an application to take full ownership on the opened device and run > any firmware command (e.g. port up/down) without any concern to hurt someone > else. > > The application look and feel is like regular RDMA application over DEVX, it > uses verbs API to open/close a device and then mostly uses DEVX APIs to > interact with the device. > > To achieve it, few mlx5 DV APIs were introduced, it includes: > - An API to get ibv_device for a given mlx5 PCI name. > - APIs to manage device specific events. > > Detailed man pages were added to describe the expected usage of those APIs. > > The mlx5 VFIO driver implemented the basic verbs APIs as of managing MR/PD and > the DEVX APIs which are required to write an RDMA application. > > The VFIO uAPIs were used to setup mlx5 vfio context by reading the device > initialization segment, setup DMA and enables the device command interface. > > In addition, the series includes pyverbs stuff which runs DEVX like testing > over RDMA and VFIO mlx5 devices. > > Some extra documentation of the benefits and the motivation to use VFIO can be > found here [1]. > > PR was sent [2]. > > [1] https://www.kernel.org/doc/html/latest/driver-api/vfio.html > [2] https://github.com/linux-rdma/rdma-core/pull/1034 > > Yishai > > Edward Srouji (10): > pyverbs: Support DevX UMEM registration > pyverbs/mlx5: Support EQN querying > pyverbs/mlx5: Support more DevX objects > pyverbs: Add auxiliary memory functions > pyverbs/mlx5: Add support to extract mlx5dv objects > pyverbs/mlx5: Wrap mlx5_cqe64 struct and add enums > tests: Add MAC address to the tests' args > tests: Add mlx5 DevX data path test > pyverbs/mlx5: Support mlx5 devices over VFIO > tests: Add a test for mlx5 over VFIO > > Maor Gottlieb (1): > mlx5: Enable debug functionality for vfio > > Mark Zhang (5): > util: Add interval_set support > mlx5: Support fast teardown over vfio > mlx5: VFIO poll_health support > mlx5: Set DV context ops > mlx5: Implement mlx5dv devx_obj APIs over vfio > > Yishai Hadas (11): > Update kernel headers > mlx5: Introduce mlx5dv_get_vfio_device_list() > verbs: Enable verbs_open_device() to work over non sysfs devices > mlx5: Setup mlx5 vfio context > mlx5: Add mlx5_vfio_cmd_exec() support > mlx5: vfio setup function support > mlx5: vfio setup basic caps > mlx5: Enable interrupt command mode over vfio > mlx5: Introduce vfio APIs to process events > mlx5: Implement basic verbs operation for PD and MR over vfio > mlx5: Support initial DEVX/DV APIs over vfio > > Documentation/pyverbs.md | 34 + > debian/ibverbs-providers.symbols | 4 + > kernel-headers/CMakeLists.txt | 4 + > kernel-headers/linux/vfio.h | 1374 ++++++++ > libibverbs/device.c | 39 +- > libibverbs/sysfs.c | 5 + > providers/mlx5/CMakeLists.txt | 3 +- > providers/mlx5/dr_rule.c | 10 +- > providers/mlx5/libmlx5.map | 7 + > providers/mlx5/man/CMakeLists.txt | 3 + > .../mlx5/man/mlx5dv_get_vfio_device_list.3.md | 64 + > providers/mlx5/man/mlx5dv_vfio_get_events_fd.3.md | 41 + > providers/mlx5/man/mlx5dv_vfio_process_events.3.md | 43 + > providers/mlx5/mlx5.c | 376 ++- > providers/mlx5/mlx5.h | 187 +- > providers/mlx5/mlx5_ifc.h | 1206 ++++++- > providers/mlx5/mlx5_vfio.c | 3379 ++++++++++++++++++++ > providers/mlx5/mlx5_vfio.h | 329 ++ > providers/mlx5/mlx5dv.h | 25 + > providers/mlx5/verbs.c | 966 +++++- > pyverbs/CMakeLists.txt | 7 + > pyverbs/dma_util.pyx | 25 + > pyverbs/mem_alloc.pyx | 46 +- > pyverbs/pd.pyx | 4 + > pyverbs/providers/mlx5/CMakeLists.txt | 4 +- > pyverbs/providers/mlx5/libmlx5.pxd | 103 +- > pyverbs/providers/mlx5/mlx5_vfio.pxd | 15 + > pyverbs/providers/mlx5/mlx5_vfio.pyx | 116 + > pyverbs/providers/mlx5/mlx5dv.pxd | 20 + > pyverbs/providers/mlx5/mlx5dv.pyx | 277 +- > pyverbs/providers/mlx5/mlx5dv_enums.pxd | 42 + > pyverbs/providers/mlx5/mlx5dv_objects.pxd | 28 + > pyverbs/providers/mlx5/mlx5dv_objects.pyx | 214 ++ > tests/CMakeLists.txt | 3 + > tests/args_parser.py | 5 + > tests/base.py | 14 +- > tests/mlx5_base.py | 460 ++- > tests/mlx5_prm_structs.py | 1046 ++++++ > tests/test_mlx5_devx.py | 23 + > tests/test_mlx5_vfio.py | 104 + > util/CMakeLists.txt | 2 + > util/interval_set.c | 208 ++ > util/interval_set.h | 77 + > util/util.h | 5 + > 44 files changed, 10650 insertions(+), 297 deletions(-) > create mode 100644 kernel-headers/linux/vfio.h > create mode 100644 providers/mlx5/man/mlx5dv_get_vfio_device_list.3.md > create mode 100644 providers/mlx5/man/mlx5dv_vfio_get_events_fd.3.md > create mode 100644 providers/mlx5/man/mlx5dv_vfio_process_events.3.md > create mode 100644 providers/mlx5/mlx5_vfio.c > create mode 100644 providers/mlx5/mlx5_vfio.h > create mode 100644 pyverbs/dma_util.pyx > create mode 100644 pyverbs/providers/mlx5/mlx5_vfio.pxd > create mode 100644 pyverbs/providers/mlx5/mlx5_vfio.pyx > create mode 100644 pyverbs/providers/mlx5/mlx5dv_objects.pxd > create mode 100644 pyverbs/providers/mlx5/mlx5dv_objects.pyx > create mode 100644 tests/mlx5_prm_structs.py > create mode 100644 tests/test_mlx5_devx.py > create mode 100644 tests/test_mlx5_vfio.py > create mode 100644 util/interval_set.c > create mode 100644 util/interval_set.h > The PR was merged. Yishai