mbox series

[RFC,v2,00/13] iommu/arm-smmu-v3: Add NVIDIA implementation

Message ID 20210831025923.15812-1-nicolinc@nvidia.com (mailing list archive)
Headers show
Series iommu/arm-smmu-v3: Add NVIDIA implementation | expand

Message

Nicolin Chen Aug. 31, 2021, 2:59 a.m. UTC
The SMMUv3 devices implemented in the Grace SoC support NVIDIA's custom
CMDQ-Virtualization (CMDQV) hardware. Like the new ECMDQ feature first
introduced in the ARM SMMUv3.3 specification, CMDQV adds multiple VCMDQ
interfaces to supplement the single architected SMMU_CMDQ in an effort
to reduce contention.

This series of patches add CMDQV support with its preparational changes:

* PATCH-1 to PATCH-8 are related to shared VMID feature: they are used
  first to improve TLB utilization, second to bind a shared VMID with a
  VCMDQ interface for hardware configuring requirement.

* PATCH-9 and PATCH-10 are to accommodate the NVIDIA implementation with
  the existing arm-smmu-v3 driver.

* PATCH-11 borrows the "implementation infrastructure" from the arm-smmu
  driver so later change can build upon it.

* PATCH-12 adds an initial NVIDIA implementation related to host feature,
  and also adds implementation specific ->device_reset() and ->get_cmdq()
  callback functions.

* PATCH-13 adds virtualization features using VFIO mdev interface, which
  allows user space hypervisor to map and get access to one of the VCMDQ
  interfaces of CMDQV module.

( Thinking that reviewers can get a better view of this implementation,
  I am attaching QEMU changes here for reference purpose:
      https://github.com/nicolinc/qemu/commits/dev/cmdqv_v6.0.0-rc2
  The branch has all preparational changes, while I'm still integrating
  device model and ARM-VIRT changes, and will push them these two days,
  although they might not be in a good shape of being sent to review yet )

Above all, I marked RFC for this series, as I feel that we may come up
some better solution. So please kindly share your reviews and insights.

Thank you!

Changelog
v1->v2:
 * Added mdev interface support for hypervisor and VMs.
 * Added preparational changes for mdev interface implementation.
 * PATCH-12 Changed ->issue_cmdlist() to ->get_cmdq() for a better
   integration with recently merged ECMDQ-related changes.

Nate Watterson (3):
  iommu/arm-smmu-v3: Add implementation infrastructure
  iommu/arm-smmu-v3: Add support for NVIDIA CMDQ-Virtualization hw
  iommu/nvidia-smmu-v3: Add mdev interface support

Nicolin Chen (10):
  iommu: Add set_nesting_vmid/get_nesting_vmid functions
  vfio: add VFIO_IOMMU_GET_VMID and VFIO_IOMMU_SET_VMID
  vfio: Document VMID control for IOMMU Virtualization
  vfio: add set_vmid and get_vmid for vfio_iommu_type1
  vfio/type1: Implement set_vmid and get_vmid
  vfio/type1: Set/get VMID to/from iommu driver
  iommu/arm-smmu-v3: Add shared VMID support for NESTING
  iommu/arm-smmu-v3: Add VMID alloc/free helpers
  iommu/arm-smmu-v3: Pass dev pointer to arm_smmu_detach_dev
  iommu/arm-smmu-v3: Pass cmdq pointer in arm_smmu_cmdq_issue_cmdlist()

 Documentation/driver-api/vfio.rst             |   34 +
 MAINTAINERS                                   |    2 +
 drivers/iommu/arm/arm-smmu-v3/Makefile        |    2 +-
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c  |   15 +
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |  121 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |   18 +
 .../iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c    | 1249 +++++++++++++++++
 drivers/iommu/iommu.c                         |   20 +
 drivers/vfio/vfio.c                           |   25 +
 drivers/vfio/vfio_iommu_type1.c               |   37 +
 include/linux/iommu.h                         |    5 +
 include/linux/vfio.h                          |    2 +
 include/uapi/linux/vfio.h                     |   26 +
 13 files changed, 1537 insertions(+), 19 deletions(-)
 create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c
 create mode 100644 drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c

Comments

Alex Williamson Aug. 31, 2021, 4:15 p.m. UTC | #1
On Mon, 30 Aug 2021 19:59:10 -0700
Nicolin Chen <nicolinc@nvidia.com> wrote:

> The SMMUv3 devices implemented in the Grace SoC support NVIDIA's custom
> CMDQ-Virtualization (CMDQV) hardware. Like the new ECMDQ feature first
> introduced in the ARM SMMUv3.3 specification, CMDQV adds multiple VCMDQ
> interfaces to supplement the single architected SMMU_CMDQ in an effort
> to reduce contention.
> 
> This series of patches add CMDQV support with its preparational changes:
> 
> * PATCH-1 to PATCH-8 are related to shared VMID feature: they are used
>   first to improve TLB utilization, second to bind a shared VMID with a
>   VCMDQ interface for hardware configuring requirement.

The vfio changes would need to be implemented in alignment with the
/dev/iommu proposals[1].  AIUI, the VMID is essentially binding
multiple containers together for TLB invalidation, which I expect in
the proposal below is largely already taken care of in that a single
iommu-fd can support multiple I/O address spaces and it's largely
expected that a hypervisor would use a single iommu-fd so this explicit
connection by userspace across containers wouldn't be necessary.

We're expecting to talk more about the /dev/iommu approach at Plumbers
in few weeks.  Thanks,

Alex

[1]https://lore.kernel.org/kvm/BN9PR11MB5433B1E4AE5B0480369F97178C189@BN9PR11MB5433.namprd11.prod.outlook.com/

 
> * PATCH-9 and PATCH-10 are to accommodate the NVIDIA implementation with
>   the existing arm-smmu-v3 driver.
> 
> * PATCH-11 borrows the "implementation infrastructure" from the arm-smmu
>   driver so later change can build upon it.
> 
> * PATCH-12 adds an initial NVIDIA implementation related to host feature,
>   and also adds implementation specific ->device_reset() and ->get_cmdq()
>   callback functions.
> 
> * PATCH-13 adds virtualization features using VFIO mdev interface, which
>   allows user space hypervisor to map and get access to one of the VCMDQ
>   interfaces of CMDQV module.
> 
> ( Thinking that reviewers can get a better view of this implementation,
>   I am attaching QEMU changes here for reference purpose:
>       https://github.com/nicolinc/qemu/commits/dev/cmdqv_v6.0.0-rc2
>   The branch has all preparational changes, while I'm still integrating
>   device model and ARM-VIRT changes, and will push them these two days,
>   although they might not be in a good shape of being sent to review yet )
> 
> Above all, I marked RFC for this series, as I feel that we may come up
> some better solution. So please kindly share your reviews and insights.
> 
> Thank you!
> 
> Changelog
> v1->v2:
>  * Added mdev interface support for hypervisor and VMs.
>  * Added preparational changes for mdev interface implementation.
>  * PATCH-12 Changed ->issue_cmdlist() to ->get_cmdq() for a better
>    integration with recently merged ECMDQ-related changes.
> 
> Nate Watterson (3):
>   iommu/arm-smmu-v3: Add implementation infrastructure
>   iommu/arm-smmu-v3: Add support for NVIDIA CMDQ-Virtualization hw
>   iommu/nvidia-smmu-v3: Add mdev interface support
> 
> Nicolin Chen (10):
>   iommu: Add set_nesting_vmid/get_nesting_vmid functions
>   vfio: add VFIO_IOMMU_GET_VMID and VFIO_IOMMU_SET_VMID
>   vfio: Document VMID control for IOMMU Virtualization
>   vfio: add set_vmid and get_vmid for vfio_iommu_type1
>   vfio/type1: Implement set_vmid and get_vmid
>   vfio/type1: Set/get VMID to/from iommu driver
>   iommu/arm-smmu-v3: Add shared VMID support for NESTING
>   iommu/arm-smmu-v3: Add VMID alloc/free helpers
>   iommu/arm-smmu-v3: Pass dev pointer to arm_smmu_detach_dev
>   iommu/arm-smmu-v3: Pass cmdq pointer in arm_smmu_cmdq_issue_cmdlist()
> 
>  Documentation/driver-api/vfio.rst             |   34 +
>  MAINTAINERS                                   |    2 +
>  drivers/iommu/arm/arm-smmu-v3/Makefile        |    2 +-
>  .../iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c  |   15 +
>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |  121 +-
>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |   18 +
>  .../iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c    | 1249 +++++++++++++++++
>  drivers/iommu/iommu.c                         |   20 +
>  drivers/vfio/vfio.c                           |   25 +
>  drivers/vfio/vfio_iommu_type1.c               |   37 +
>  include/linux/iommu.h                         |    5 +
>  include/linux/vfio.h                          |    2 +
>  include/uapi/linux/vfio.h                     |   26 +
>  13 files changed, 1537 insertions(+), 19 deletions(-)
>  create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c
>  create mode 100644 drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c
>
Tian, Kevin Sept. 1, 2021, 6:55 a.m. UTC | #2
> From: Alex Williamson
> Sent: Wednesday, September 1, 2021 12:16 AM
> 
> On Mon, 30 Aug 2021 19:59:10 -0700
> Nicolin Chen <nicolinc@nvidia.com> wrote:
> 
> > The SMMUv3 devices implemented in the Grace SoC support NVIDIA's
> custom
> > CMDQ-Virtualization (CMDQV) hardware. Like the new ECMDQ feature first
> > introduced in the ARM SMMUv3.3 specification, CMDQV adds multiple
> VCMDQ
> > interfaces to supplement the single architected SMMU_CMDQ in an effort
> > to reduce contention.
> >
> > This series of patches add CMDQV support with its preparational changes:
> >
> > * PATCH-1 to PATCH-8 are related to shared VMID feature: they are used
> >   first to improve TLB utilization, second to bind a shared VMID with a
> >   VCMDQ interface for hardware configuring requirement.
> 
> The vfio changes would need to be implemented in alignment with the
> /dev/iommu proposals[1].  AIUI, the VMID is essentially binding
> multiple containers together for TLB invalidation, which I expect in
> the proposal below is largely already taken care of in that a single
> iommu-fd can support multiple I/O address spaces and it's largely
> expected that a hypervisor would use a single iommu-fd so this explicit
> connection by userspace across containers wouldn't be necessary.

Agree. VMID is equivalent to DID (domain id) in other vendor iommus.
with /dev/iommu multiple I/O address spaces can share the same VMID
via nesting. No need of exposing VMID to userspace to build the 
connection.

Thanks,
Kevin

> 
> We're expecting to talk more about the /dev/iommu approach at Plumbers
> in few weeks.  Thanks,
> 
> Alex
> 
> [1]https://lore.kernel.org/kvm/BN9PR11MB5433B1E4AE5B0480369F97178C1
> 89@BN9PR11MB5433.namprd11.prod.outlook.com/
> 
> 
> > * PATCH-9 and PATCH-10 are to accommodate the NVIDIA implementation
> with
> >   the existing arm-smmu-v3 driver.
> >
> > * PATCH-11 borrows the "implementation infrastructure" from the arm-
> smmu
> >   driver so later change can build upon it.
> >
> > * PATCH-12 adds an initial NVIDIA implementation related to host feature,
> >   and also adds implementation specific ->device_reset() and ->get_cmdq()
> >   callback functions.
> >
> > * PATCH-13 adds virtualization features using VFIO mdev interface, which
> >   allows user space hypervisor to map and get access to one of the VCMDQ
> >   interfaces of CMDQV module.
> >
> > ( Thinking that reviewers can get a better view of this implementation,
> >   I am attaching QEMU changes here for reference purpose:
> >       https://github.com/nicolinc/qemu/commits/dev/cmdqv_v6.0.0-rc2
> >   The branch has all preparational changes, while I'm still integrating
> >   device model and ARM-VIRT changes, and will push them these two days,
> >   although they might not be in a good shape of being sent to review yet )
> >
> > Above all, I marked RFC for this series, as I feel that we may come up
> > some better solution. So please kindly share your reviews and insights.
> >
> > Thank you!
> >
> > Changelog
> > v1->v2:
> >  * Added mdev interface support for hypervisor and VMs.
> >  * Added preparational changes for mdev interface implementation.
> >  * PATCH-12 Changed ->issue_cmdlist() to ->get_cmdq() for a better
> >    integration with recently merged ECMDQ-related changes.
> >
> > Nate Watterson (3):
> >   iommu/arm-smmu-v3: Add implementation infrastructure
> >   iommu/arm-smmu-v3: Add support for NVIDIA CMDQ-Virtualization hw
> >   iommu/nvidia-smmu-v3: Add mdev interface support
> >
> > Nicolin Chen (10):
> >   iommu: Add set_nesting_vmid/get_nesting_vmid functions
> >   vfio: add VFIO_IOMMU_GET_VMID and VFIO_IOMMU_SET_VMID
> >   vfio: Document VMID control for IOMMU Virtualization
> >   vfio: add set_vmid and get_vmid for vfio_iommu_type1
> >   vfio/type1: Implement set_vmid and get_vmid
> >   vfio/type1: Set/get VMID to/from iommu driver
> >   iommu/arm-smmu-v3: Add shared VMID support for NESTING
> >   iommu/arm-smmu-v3: Add VMID alloc/free helpers
> >   iommu/arm-smmu-v3: Pass dev pointer to arm_smmu_detach_dev
> >   iommu/arm-smmu-v3: Pass cmdq pointer in
> arm_smmu_cmdq_issue_cmdlist()
> >
> >  Documentation/driver-api/vfio.rst             |   34 +
> >  MAINTAINERS                                   |    2 +
> >  drivers/iommu/arm/arm-smmu-v3/Makefile        |    2 +-
> >  .../iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c  |   15 +
> >  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |  121 +-
> >  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |   18 +
> >  .../iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c    | 1249
> +++++++++++++++++
> >  drivers/iommu/iommu.c                         |   20 +
> >  drivers/vfio/vfio.c                           |   25 +
> >  drivers/vfio/vfio_iommu_type1.c               |   37 +
> >  include/linux/iommu.h                         |    5 +
> >  include/linux/vfio.h                          |    2 +
> >  include/uapi/linux/vfio.h                     |   26 +
> >  13 files changed, 1537 insertions(+), 19 deletions(-)
> >  create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-
> impl.c
> >  create mode 100644 drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c
> >
> 
> _______________________________________________
> iommu mailing list
> iommu@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu
Jason Gunthorpe Sept. 2, 2021, 2:45 p.m. UTC | #3
On Wed, Sep 01, 2021 at 06:55:55AM +0000, Tian, Kevin wrote:
> > From: Alex Williamson
> > Sent: Wednesday, September 1, 2021 12:16 AM
> > 
> > On Mon, 30 Aug 2021 19:59:10 -0700
> > Nicolin Chen <nicolinc@nvidia.com> wrote:
> > 
> > > The SMMUv3 devices implemented in the Grace SoC support NVIDIA's
> > custom
> > > CMDQ-Virtualization (CMDQV) hardware. Like the new ECMDQ feature first
> > > introduced in the ARM SMMUv3.3 specification, CMDQV adds multiple
> > VCMDQ
> > > interfaces to supplement the single architected SMMU_CMDQ in an effort
> > > to reduce contention.
> > >
> > > This series of patches add CMDQV support with its preparational changes:
> > >
> > > * PATCH-1 to PATCH-8 are related to shared VMID feature: they are used
> > >   first to improve TLB utilization, second to bind a shared VMID with a
> > >   VCMDQ interface for hardware configuring requirement.
> > 
> > The vfio changes would need to be implemented in alignment with the
> > /dev/iommu proposals[1].  AIUI, the VMID is essentially binding
> > multiple containers together for TLB invalidation, which I expect in
> > the proposal below is largely already taken care of in that a single
> > iommu-fd can support multiple I/O address spaces and it's largely
> > expected that a hypervisor would use a single iommu-fd so this explicit
> > connection by userspace across containers wouldn't be necessary.
> 
> Agree. VMID is equivalent to DID (domain id) in other vendor iommus.
> with /dev/iommu multiple I/O address spaces can share the same VMID
> via nesting. No need of exposing VMID to userspace to build the 
> connection.

Indeed, this looks like a flavour of the accelerated invalidation
stuff we've talked about already.

I would see it probably exposed as some HW specific IOCTL on the iommu
fd to get access to the accelerated invalidation for IOASID's in the
FD.

Indeed, this seems like a further example of why /dev/iommu is looking
like a good idea as this RFC is very complicated to do something
fairly simple.

Where are thing on the /dev/iommu work these days?

Thanks,
Jason
Tian, Kevin Sept. 2, 2021, 10:27 p.m. UTC | #4
> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Thursday, September 2, 2021 10:45 PM
> 
> On Wed, Sep 01, 2021 at 06:55:55AM +0000, Tian, Kevin wrote:
> > > From: Alex Williamson
> > > Sent: Wednesday, September 1, 2021 12:16 AM
> > >
> > > On Mon, 30 Aug 2021 19:59:10 -0700
> > > Nicolin Chen <nicolinc@nvidia.com> wrote:
> > >
> > > > The SMMUv3 devices implemented in the Grace SoC support NVIDIA's
> > > custom
> > > > CMDQ-Virtualization (CMDQV) hardware. Like the new ECMDQ feature
> first
> > > > introduced in the ARM SMMUv3.3 specification, CMDQV adds multiple
> > > VCMDQ
> > > > interfaces to supplement the single architected SMMU_CMDQ in an
> effort
> > > > to reduce contention.
> > > >
> > > > This series of patches add CMDQV support with its preparational
> changes:
> > > >
> > > > * PATCH-1 to PATCH-8 are related to shared VMID feature: they are
> used
> > > >   first to improve TLB utilization, second to bind a shared VMID with a
> > > >   VCMDQ interface for hardware configuring requirement.
> > >
> > > The vfio changes would need to be implemented in alignment with the
> > > /dev/iommu proposals[1].  AIUI, the VMID is essentially binding
> > > multiple containers together for TLB invalidation, which I expect in
> > > the proposal below is largely already taken care of in that a single
> > > iommu-fd can support multiple I/O address spaces and it's largely
> > > expected that a hypervisor would use a single iommu-fd so this explicit
> > > connection by userspace across containers wouldn't be necessary.
> >
> > Agree. VMID is equivalent to DID (domain id) in other vendor iommus.
> > with /dev/iommu multiple I/O address spaces can share the same VMID
> > via nesting. No need of exposing VMID to userspace to build the
> > connection.
> 
> Indeed, this looks like a flavour of the accelerated invalidation
> stuff we've talked about already.
> 
> I would see it probably exposed as some HW specific IOCTL on the iommu
> fd to get access to the accelerated invalidation for IOASID's in the
> FD.
> 
> Indeed, this seems like a further example of why /dev/iommu is looking
> like a good idea as this RFC is very complicated to do something
> fairly simple.
> 
> Where are thing on the /dev/iommu work these days?
> 

We are actively working on the basic skeleton. Our original plan is to send
out the 1st draft before LPC, with support of vfio type1 semantics and
and pci dev only (single-device group). But later we realized that adding
multi-devices group support is also necessary even in the 1st draft to avoid
some dirty hacks and build the complete picture. This will add some time
though. If things go well, we'll still try hit the original plan. If not, it will be
soon after LPC.

Thanks
Kevin
Nicolin Chen Sept. 16, 2021, 6:21 p.m. UTC | #5
Hi Kevin,

On Thu, Sep 02, 2021 at 10:27:06PM +0000, Tian, Kevin wrote:

> > Indeed, this looks like a flavour of the accelerated invalidation
> > stuff we've talked about already.
> >
> > I would see it probably exposed as some HW specific IOCTL on the iommu
> > fd to get access to the accelerated invalidation for IOASID's in the
> > FD.
> >
> > Indeed, this seems like a further example of why /dev/iommu is looking
> > like a good idea as this RFC is very complicated to do something
> > fairly simple.
> >
> > Where are thing on the /dev/iommu work these days?
> 
> We are actively working on the basic skeleton. Our original plan is to send
> out the 1st draft before LPC, with support of vfio type1 semantics and
> and pci dev only (single-device group). But later we realized that adding
> multi-devices group support is also necessary even in the 1st draft to avoid
> some dirty hacks and build the complete picture. This will add some time
> though. If things go well, we'll still try hit the original plan. If not, it will be
> soon after LPC.

As I also want to take a look at your work for this implementation,
would you please CC me when you send out your patches? Thank you!