mbox series

[GIT,PULL] Please pull mlx5 vfio changes

Message ID 20220907094344.381661-1-leon@kernel.org (mailing list archive)
State New, archived
Headers show
Series [GIT,PULL] Please pull mlx5 vfio changes | expand

Pull-request

https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git tags/mlx5-dma-logging

Message

Leon Romanovsky Sept. 7, 2022, 9:43 a.m. UTC
Hi Alex,

This series is based on clean 6.0-rc4 as such it causes to two small merge
conficts whis vfio-next. One is in thrird patch where you should take whole
chunk for include/uapi/linux/vfio.h as is. Another is in vfio_main.c around
header includes, which you should take too.

----------------------------------------------------------------
The following changes since commit 7e18e42e4b280c85b76967a9106a13ca61c16179:

  Linux 6.0-rc4 (2022-09-04 13:10:01 -0700)

are available in the Git repository at:

  https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git tags/mlx5-dma-logging

for you to fetch changes up to 7f5dc74155a9de8bf6adc86d84e26c5a1e762431:

  vfio/mlx5: Set the driver DMA logging callbacks (2022-09-07 12:11:22 +0300)

----------------------------------------------------------------
Add device DMA logging support for mlx5 driver

From Yishai:

This series adds device DMA logging uAPIs and their implementation as
part of mlx5 driver.

DMA logging allows a device to internally record what DMAs the device is
initiating and report them back to userspace. It is part of the VFIO
migration infrastructure that allows implementing dirty page tracking
during the pre copy phase of live migration. Only DMA WRITEs are logged,
and this API is not connected to VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE.

The uAPIs are based on the FEATURE ioctl as were introduced earlier by
the below RFC [1] and follows the notes that were discussed in the
mailing list.

It includes:
- A PROBE option to detect if the device supports DMA logging.
- A SET option to start device DMA logging in given IOVAs ranges.
- A GET option to read back and clear the device DMA log.
- A SET option to stop device DMA logging that was previously started.

Extra details exist as part of relevant patches in the series.

In addition, the series adds some infrastructure support for managing an
IOVA bitmap done by Joao Martins.

It abstracts how an IOVA range is represented in a bitmap that is
granulated by a given page_size. So it translates all the lifting of
dealing with user pointers into its corresponding kernel addresses.
This new functionality abstracts the complexity of user/kernel bitmap
pointer usage and finally enables an API to set some bits.

This functionality will be used as part of IOMMUFD series for the system
IOMMU tracking.

Finally, we come with mlx5 implementation based on its device
specification for the DMA logging APIs.

The matching qemu changes can be previewed here [2].
They come on top of the v2 migration protocol patches that were sent
already to the mailing list.

Note:
- As this series touched mlx5_core parts we may need to send the
  net/mlx5 patches as a pull request format to VFIO to avoid conflicts
  before acceptance.

[1] https://lore.kernel.org/all/20220501123301.127279-1-yishaih@nvidia.com/T/
[2] https://github.com/avihai1122/qemu/commits/device_dirty_tracking

Link: https://lore.kernel.org/all/20220905105852.26398-1-yishaih@nvidia.com/
Signed-of-by: Leon Romanovsky <leon@kernel.org>

----------------------------------------------------------------
Joao Martins (1):
      vfio: Add an IOVA bitmap support

Yishai Hadas (9):
      net/mlx5: Introduce ifc bits for page tracker
      net/mlx5: Query ADV_VIRTUALIZATION capabilities
      vfio: Introduce DMA logging uAPIs
      vfio: Introduce the DMA logging feature support
      vfio/mlx5: Init QP based resources for dirty tracking
      vfio/mlx5: Create and destroy page tracker object
      vfio/mlx5: Report dirty pages from tracker
      vfio/mlx5: Manage error scenarios on tracker
      vfio/mlx5: Set the driver DMA logging callbacks

 drivers/net/ethernet/mellanox/mlx5/core/fw.c   |   6 +
 drivers/net/ethernet/mellanox/mlx5/core/main.c |   1 +
 drivers/vfio/Kconfig                           |   1 +
 drivers/vfio/Makefile                          |   6 +-
 drivers/vfio/iova_bitmap.c                     | 422 +++++++++++
 drivers/vfio/pci/mlx5/cmd.c                    | 995 ++++++++++++++++++++++++-
 drivers/vfio/pci/mlx5/cmd.h                    |  63 +-
 drivers/vfio/pci/mlx5/main.c                   |   9 +-
 drivers/vfio/pci/vfio_pci_core.c               |   5 +
 drivers/vfio/vfio_main.c                       | 175 +++++
 include/linux/iova_bitmap.h                    |  26 +
 include/linux/mlx5/device.h                    |   9 +
 include/linux/mlx5/mlx5_ifc.h                  |  83 ++-
 include/linux/vfio.h                           |  28 +-
 include/uapi/linux/vfio.h                      |  86 +++
 15 files changed, 1895 insertions(+), 20 deletions(-)
 create mode 100644 drivers/vfio/iova_bitmap.c
 create mode 100644 include/linux/iova_bitmap.h

Comments

Alex Williamson Sept. 7, 2022, 7:21 p.m. UTC | #1
On Wed,  7 Sep 2022 12:43:44 +0300
Leon Romanovsky <leon@kernel.org> wrote:

> Hi Alex,
> 
> This series is based on clean 6.0-rc4 as such it causes to two small merge
> conficts whis vfio-next. One is in thrird patch where you should take whole
> chunk for include/uapi/linux/vfio.h as is. Another is in vfio_main.c around
> header includes, which you should take too.

Is there any reason you can't provide a topic branch for the two
net/mlx5 patches and the remainder are rebased and committed through
the vfio tree?  I don't see this as an exceptional case that requires
resolving conflicts in merge commits.  Thanks,

Alex


> ----------------------------------------------------------------
> The following changes since commit 7e18e42e4b280c85b76967a9106a13ca61c16179:
> 
>   Linux 6.0-rc4 (2022-09-04 13:10:01 -0700)
> 
> are available in the Git repository at:
> 
>   https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git tags/mlx5-dma-logging
> 
> for you to fetch changes up to 7f5dc74155a9de8bf6adc86d84e26c5a1e762431:
> 
>   vfio/mlx5: Set the driver DMA logging callbacks (2022-09-07 12:11:22 +0300)
> 
> ----------------------------------------------------------------
> Add device DMA logging support for mlx5 driver
> 
> From Yishai:
> 
> This series adds device DMA logging uAPIs and their implementation as
> part of mlx5 driver.
> 
> DMA logging allows a device to internally record what DMAs the device is
> initiating and report them back to userspace. It is part of the VFIO
> migration infrastructure that allows implementing dirty page tracking
> during the pre copy phase of live migration. Only DMA WRITEs are logged,
> and this API is not connected to VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE.
> 
> The uAPIs are based on the FEATURE ioctl as were introduced earlier by
> the below RFC [1] and follows the notes that were discussed in the
> mailing list.
> 
> It includes:
> - A PROBE option to detect if the device supports DMA logging.
> - A SET option to start device DMA logging in given IOVAs ranges.
> - A GET option to read back and clear the device DMA log.
> - A SET option to stop device DMA logging that was previously started.
> 
> Extra details exist as part of relevant patches in the series.
> 
> In addition, the series adds some infrastructure support for managing an
> IOVA bitmap done by Joao Martins.
> 
> It abstracts how an IOVA range is represented in a bitmap that is
> granulated by a given page_size. So it translates all the lifting of
> dealing with user pointers into its corresponding kernel addresses.
> This new functionality abstracts the complexity of user/kernel bitmap
> pointer usage and finally enables an API to set some bits.
> 
> This functionality will be used as part of IOMMUFD series for the system
> IOMMU tracking.
> 
> Finally, we come with mlx5 implementation based on its device
> specification for the DMA logging APIs.
> 
> The matching qemu changes can be previewed here [2].
> They come on top of the v2 migration protocol patches that were sent
> already to the mailing list.
> 
> Note:
> - As this series touched mlx5_core parts we may need to send the
>   net/mlx5 patches as a pull request format to VFIO to avoid conflicts
>   before acceptance.
> 
> [1] https://lore.kernel.org/all/20220501123301.127279-1-yishaih@nvidia.com/T/
> [2] https://github.com/avihai1122/qemu/commits/device_dirty_tracking
> 
> Link: https://lore.kernel.org/all/20220905105852.26398-1-yishaih@nvidia.com/
> Signed-of-by: Leon Romanovsky <leon@kernel.org>
> 
> ----------------------------------------------------------------
> Joao Martins (1):
>       vfio: Add an IOVA bitmap support
> 
> Yishai Hadas (9):
>       net/mlx5: Introduce ifc bits for page tracker
>       net/mlx5: Query ADV_VIRTUALIZATION capabilities
>       vfio: Introduce DMA logging uAPIs
>       vfio: Introduce the DMA logging feature support
>       vfio/mlx5: Init QP based resources for dirty tracking
>       vfio/mlx5: Create and destroy page tracker object
>       vfio/mlx5: Report dirty pages from tracker
>       vfio/mlx5: Manage error scenarios on tracker
>       vfio/mlx5: Set the driver DMA logging callbacks
> 
>  drivers/net/ethernet/mellanox/mlx5/core/fw.c   |   6 +
>  drivers/net/ethernet/mellanox/mlx5/core/main.c |   1 +
>  drivers/vfio/Kconfig                           |   1 +
>  drivers/vfio/Makefile                          |   6 +-
>  drivers/vfio/iova_bitmap.c                     | 422 +++++++++++
>  drivers/vfio/pci/mlx5/cmd.c                    | 995 ++++++++++++++++++++++++-
>  drivers/vfio/pci/mlx5/cmd.h                    |  63 +-
>  drivers/vfio/pci/mlx5/main.c                   |   9 +-
>  drivers/vfio/pci/vfio_pci_core.c               |   5 +
>  drivers/vfio/vfio_main.c                       | 175 +++++
>  include/linux/iova_bitmap.h                    |  26 +
>  include/linux/mlx5/device.h                    |   9 +
>  include/linux/mlx5/mlx5_ifc.h                  |  83 ++-
>  include/linux/vfio.h                           |  28 +-
>  include/uapi/linux/vfio.h                      |  86 +++
>  15 files changed, 1895 insertions(+), 20 deletions(-)
>  create mode 100644 drivers/vfio/iova_bitmap.c
>  create mode 100644 include/linux/iova_bitmap.h
>
Leon Romanovsky Sept. 8, 2022, 6:31 a.m. UTC | #2
On Wed, Sep 07, 2022 at 01:21:19PM -0600, Alex Williamson wrote:
> On Wed,  7 Sep 2022 12:43:44 +0300
> Leon Romanovsky <leon@kernel.org> wrote:
> 
> > Hi Alex,
> > 
> > This series is based on clean 6.0-rc4 as such it causes to two small merge
> > conficts whis vfio-next. One is in thrird patch where you should take whole
> > chunk for include/uapi/linux/vfio.h as is. Another is in vfio_main.c around
> > header includes, which you should take too.
> 
> Is there any reason you can't provide a topic branch for the two
> net/mlx5 patches and the remainder are rebased and committed through
> the vfio tree?  

You added your Acked-by to vfio/mlx5 patches and for me it is a sign to
prepare clean PR with whole series.

I reset mlx5-vfio topic to have only two net/mlx5 commits without
special tag.

https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git topic/mlx5-vfio
Everything else can go directly to your tree without my intervention.

Thanks
Alex Williamson Sept. 8, 2022, 4:53 p.m. UTC | #3
On Thu, 8 Sep 2022 09:31:13 +0300
Leon Romanovsky <leon@kernel.org> wrote:

> On Wed, Sep 07, 2022 at 01:21:19PM -0600, Alex Williamson wrote:
> > On Wed,  7 Sep 2022 12:43:44 +0300
> > Leon Romanovsky <leon@kernel.org> wrote:
> >   
> > > Hi Alex,
> > > 
> > > This series is based on clean 6.0-rc4 as such it causes to two small merge
> > > conficts whis vfio-next. One is in thrird patch where you should take whole
> > > chunk for include/uapi/linux/vfio.h as is. Another is in vfio_main.c around
> > > header includes, which you should take too.  
> > 
> > Is there any reason you can't provide a topic branch for the two
> > net/mlx5 patches and the remainder are rebased and committed through
> > the vfio tree?    
> 
> You added your Acked-by to vfio/mlx5 patches and for me it is a sign to
> prepare clean PR with whole series.
> 
> I reset mlx5-vfio topic to have only two net/mlx5 commits without
> special tag.
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git topic/mlx5-vfio
> Everything else can go directly to your tree without my intervention.

Sorry, I knew the intention initially was to send a PR and I didn't
think about the conflicts we'd have versus the base you'd use.  Thanks
for splitting this out, I think it'll make for a cleaner upstream path
given the clear code split.

Yishai, can you post a v7 rebased on the vfio next branch?  The comment
I requested is now ephemeral since it only existed in the commits Leon
dropped.  Also feel free to drop my Acks since I'll add new Sign-offs.
Thanks,

Alex
Yishai Hadas Sept. 8, 2022, 5:31 p.m. UTC | #4
On 08/09/2022 19:53, Alex Williamson wrote:
> On Thu, 8 Sep 2022 09:31:13 +0300
> Leon Romanovsky <leon@kernel.org> wrote:
>
>> On Wed, Sep 07, 2022 at 01:21:19PM -0600, Alex Williamson wrote:
>>> On Wed,  7 Sep 2022 12:43:44 +0300
>>> Leon Romanovsky <leon@kernel.org> wrote:
>>>    
>>>> Hi Alex,
>>>>
>>>> This series is based on clean 6.0-rc4 as such it causes to two small merge
>>>> conficts whis vfio-next. One is in thrird patch where you should take whole
>>>> chunk for include/uapi/linux/vfio.h as is. Another is in vfio_main.c around
>>>> header includes, which you should take too.
>>> Is there any reason you can't provide a topic branch for the two
>>> net/mlx5 patches and the remainder are rebased and committed through
>>> the vfio tree?
>> You added your Acked-by to vfio/mlx5 patches and for me it is a sign to
>> prepare clean PR with whole series.
>>
>> I reset mlx5-vfio topic to have only two net/mlx5 commits without
>> special tag.
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git topic/mlx5-vfio
>> Everything else can go directly to your tree without my intervention.
> Sorry, I knew the intention initially was to send a PR and I didn't
> think about the conflicts we'd have versus the base you'd use.  Thanks
> for splitting this out, I think it'll make for a cleaner upstream path
> given the clear code split.
>
> Yishai, can you post a v7 rebased on the vfio next branch?


Sure

Do you want me to include in V7 the two net/mlx5 patches that are part 
of the PR or that you'll take them first from the PR, publish your 
vfio/next tree and I'll drop them from V7 ?

> The comment
> I requested is now ephemeral since it only existed in the commits Leon
> dropped.  Also feel free to drop my Acks since I'll add new Sign-offs.
> Thanks,

OK, will drop them.

>
> Alex
>
Alex Williamson Sept. 8, 2022, 6 p.m. UTC | #5
On Thu, 8 Sep 2022 20:31:35 +0300
Yishai Hadas <yishaih@nvidia.com> wrote:

> On 08/09/2022 19:53, Alex Williamson wrote:
> > On Thu, 8 Sep 2022 09:31:13 +0300
> > Leon Romanovsky <leon@kernel.org> wrote:
> >  
> >> On Wed, Sep 07, 2022 at 01:21:19PM -0600, Alex Williamson wrote:  
> >>> On Wed,  7 Sep 2022 12:43:44 +0300
> >>> Leon Romanovsky <leon@kernel.org> wrote:
> >>>      
> >>>> Hi Alex,
> >>>>
> >>>> This series is based on clean 6.0-rc4 as such it causes to two small merge
> >>>> conficts whis vfio-next. One is in thrird patch where you should take whole
> >>>> chunk for include/uapi/linux/vfio.h as is. Another is in vfio_main.c around
> >>>> header includes, which you should take too.  
> >>> Is there any reason you can't provide a topic branch for the two
> >>> net/mlx5 patches and the remainder are rebased and committed through
> >>> the vfio tree?  
> >> You added your Acked-by to vfio/mlx5 patches and for me it is a sign to
> >> prepare clean PR with whole series.
> >>
> >> I reset mlx5-vfio topic to have only two net/mlx5 commits without
> >> special tag.
> >>
> >> https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git topic/mlx5-vfio
> >> Everything else can go directly to your tree without my intervention.  
> > Sorry, I knew the intention initially was to send a PR and I didn't
> > think about the conflicts we'd have versus the base you'd use.  Thanks
> > for splitting this out, I think it'll make for a cleaner upstream path
> > given the clear code split.
> >
> > Yishai, can you post a v7 rebased on the vfio next branch?  
> 
> 
> Sure
> 
> Do you want me to include in V7 the two net/mlx5 patches that are part 
> of the PR or that you'll take them first from the PR, publish your 
> vfio/next tree and I'll drop them from V7 ?

For the sake of having a series that compiles and doesn't confuse
anyone with buildbot errors, please include them, but I'll pull them
from Leon's topic branch rather than applying them directly.  Thanks,

Alex