| Message ID | 20210831025923.15812-1-nicolinc@nvidia.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | iommu/arm-smmu-v3: Add NVIDIA implementation | expand |
On Mon, 30 Aug 2021 19:59:10 -0700 Nicolin Chen <nicolinc@nvidia.com> wrote: > The SMMUv3 devices implemented in the Grace SoC support NVIDIA's custom > CMDQ-Virtualization (CMDQV) hardware. Like the new ECMDQ feature first > introduced in the ARM SMMUv3.3 specification, CMDQV adds multiple VCMDQ > interfaces to supplement the single architected SMMU_CMDQ in an effort > to reduce contention. > > This series of patches add CMDQV support with its preparational changes: > > * PATCH-1 to PATCH-8 are related to shared VMID feature: they are used > first to improve TLB utilization, second to bind a shared VMID with a > VCMDQ interface for hardware configuring requirement. The vfio changes would need to be implemented in alignment with the /dev/iommu proposals[1]. AIUI, the VMID is essentially binding multiple containers together for TLB invalidation, which I expect in the proposal below is largely already taken care of in that a single iommu-fd can support multiple I/O address spaces and it's largely expected that a hypervisor would use a single iommu-fd so this explicit connection by userspace across containers wouldn't be necessary. We're expecting to talk more about the /dev/iommu approach at Plumbers in few weeks. Thanks, Alex [1]https://lore.kernel.org/kvm/BN9PR11MB5433B1E4AE5B0480369F97178C189@BN9PR11MB5433.namprd11.prod.outlook.com/ > * PATCH-9 and PATCH-10 are to accommodate the NVIDIA implementation with > the existing arm-smmu-v3 driver. > > * PATCH-11 borrows the "implementation infrastructure" from the arm-smmu > driver so later change can build upon it. > > * PATCH-12 adds an initial NVIDIA implementation related to host feature, > and also adds implementation specific ->device_reset() and ->get_cmdq() > callback functions. > > * PATCH-13 adds virtualization features using VFIO mdev interface, which > allows user space hypervisor to map and get access to one of the VCMDQ > interfaces of CMDQV module. > > ( Thinking that reviewers can get a better view of this implementation, > I am attaching QEMU changes here for reference purpose: > https://github.com/nicolinc/qemu/commits/dev/cmdqv_v6.0.0-rc2 > The branch has all preparational changes, while I'm still integrating > device model and ARM-VIRT changes, and will push them these two days, > although they might not be in a good shape of being sent to review yet ) > > Above all, I marked RFC for this series, as I feel that we may come up > some better solution. So please kindly share your reviews and insights. > > Thank you! > > Changelog > v1->v2: > * Added mdev interface support for hypervisor and VMs. > * Added preparational changes for mdev interface implementation. > * PATCH-12 Changed ->issue_cmdlist() to ->get_cmdq() for a better > integration with recently merged ECMDQ-related changes. > > Nate Watterson (3): > iommu/arm-smmu-v3: Add implementation infrastructure > iommu/arm-smmu-v3: Add support for NVIDIA CMDQ-Virtualization hw > iommu/nvidia-smmu-v3: Add mdev interface support > > Nicolin Chen (10): > iommu: Add set_nesting_vmid/get_nesting_vmid functions > vfio: add VFIO_IOMMU_GET_VMID and VFIO_IOMMU_SET_VMID > vfio: Document VMID control for IOMMU Virtualization > vfio: add set_vmid and get_vmid for vfio_iommu_type1 > vfio/type1: Implement set_vmid and get_vmid > vfio/type1: Set/get VMID to/from iommu driver > iommu/arm-smmu-v3: Add shared VMID support for NESTING > iommu/arm-smmu-v3: Add VMID alloc/free helpers > iommu/arm-smmu-v3: Pass dev pointer to arm_smmu_detach_dev > iommu/arm-smmu-v3: Pass cmdq pointer in arm_smmu_cmdq_issue_cmdlist() > > Documentation/driver-api/vfio.rst | 34 + > MAINTAINERS | 2 + > drivers/iommu/arm/arm-smmu-v3/Makefile | 2 +- > .../iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c | 15 + > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 121 +- > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 18 + > .../iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c | 1249 +++++++++++++++++ > drivers/iommu/iommu.c | 20 + > drivers/vfio/vfio.c | 25 + > drivers/vfio/vfio_iommu_type1.c | 37 + > include/linux/iommu.h | 5 + > include/linux/vfio.h | 2 + > include/uapi/linux/vfio.h | 26 + > 13 files changed, 1537 insertions(+), 19 deletions(-) > create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c > create mode 100644 drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c >
> From: Alex Williamson > Sent: Wednesday, September 1, 2021 12:16 AM > > On Mon, 30 Aug 2021 19:59:10 -0700 > Nicolin Chen <nicolinc@nvidia.com> wrote: > > > The SMMUv3 devices implemented in the Grace SoC support NVIDIA's > custom > > CMDQ-Virtualization (CMDQV) hardware. Like the new ECMDQ feature first > > introduced in the ARM SMMUv3.3 specification, CMDQV adds multiple > VCMDQ > > interfaces to supplement the single architected SMMU_CMDQ in an effort > > to reduce contention. > > > > This series of patches add CMDQV support with its preparational changes: > > > > * PATCH-1 to PATCH-8 are related to shared VMID feature: they are used > > first to improve TLB utilization, second to bind a shared VMID with a > > VCMDQ interface for hardware configuring requirement. > > The vfio changes would need to be implemented in alignment with the > /dev/iommu proposals[1]. AIUI, the VMID is essentially binding > multiple containers together for TLB invalidation, which I expect in > the proposal below is largely already taken care of in that a single > iommu-fd can support multiple I/O address spaces and it's largely > expected that a hypervisor would use a single iommu-fd so this explicit > connection by userspace across containers wouldn't be necessary. Agree. VMID is equivalent to DID (domain id) in other vendor iommus. with /dev/iommu multiple I/O address spaces can share the same VMID via nesting. No need of exposing VMID to userspace to build the connection. Thanks, Kevin > > We're expecting to talk more about the /dev/iommu approach at Plumbers > in few weeks. Thanks, > > Alex > > [1]https://lore.kernel.org/kvm/BN9PR11MB5433B1E4AE5B0480369F97178C1 > 89@BN9PR11MB5433.namprd11.prod.outlook.com/ > > > > * PATCH-9 and PATCH-10 are to accommodate the NVIDIA implementation > with > > the existing arm-smmu-v3 driver. > > > > * PATCH-11 borrows the "implementation infrastructure" from the arm- > smmu > > driver so later change can build upon it. > > > > * PATCH-12 adds an initial NVIDIA implementation related to host feature, > > and also adds implementation specific ->device_reset() and ->get_cmdq() > > callback functions. > > > > * PATCH-13 adds virtualization features using VFIO mdev interface, which > > allows user space hypervisor to map and get access to one of the VCMDQ > > interfaces of CMDQV module. > > > > ( Thinking that reviewers can get a better view of this implementation, > > I am attaching QEMU changes here for reference purpose: > > https://github.com/nicolinc/qemu/commits/dev/cmdqv_v6.0.0-rc2 > > The branch has all preparational changes, while I'm still integrating > > device model and ARM-VIRT changes, and will push them these two days, > > although they might not be in a good shape of being sent to review yet ) > > > > Above all, I marked RFC for this series, as I feel that we may come up > > some better solution. So please kindly share your reviews and insights. > > > > Thank you! > > > > Changelog > > v1->v2: > > * Added mdev interface support for hypervisor and VMs. > > * Added preparational changes for mdev interface implementation. > > * PATCH-12 Changed ->issue_cmdlist() to ->get_cmdq() for a better > > integration with recently merged ECMDQ-related changes. > > > > Nate Watterson (3): > > iommu/arm-smmu-v3: Add implementation infrastructure > > iommu/arm-smmu-v3: Add support for NVIDIA CMDQ-Virtualization hw > > iommu/nvidia-smmu-v3: Add mdev interface support > > > > Nicolin Chen (10): > > iommu: Add set_nesting_vmid/get_nesting_vmid functions > > vfio: add VFIO_IOMMU_GET_VMID and VFIO_IOMMU_SET_VMID > > vfio: Document VMID control for IOMMU Virtualization > > vfio: add set_vmid and get_vmid for vfio_iommu_type1 > > vfio/type1: Implement set_vmid and get_vmid > > vfio/type1: Set/get VMID to/from iommu driver > > iommu/arm-smmu-v3: Add shared VMID support for NESTING > > iommu/arm-smmu-v3: Add VMID alloc/free helpers > > iommu/arm-smmu-v3: Pass dev pointer to arm_smmu_detach_dev > > iommu/arm-smmu-v3: Pass cmdq pointer in > arm_smmu_cmdq_issue_cmdlist() > > > > Documentation/driver-api/vfio.rst | 34 + > > MAINTAINERS | 2 + > > drivers/iommu/arm/arm-smmu-v3/Makefile | 2 +- > > .../iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c | 15 + > > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 121 +- > > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 18 + > > .../iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c | 1249 > +++++++++++++++++ > > drivers/iommu/iommu.c | 20 + > > drivers/vfio/vfio.c | 25 + > > drivers/vfio/vfio_iommu_type1.c | 37 + > > include/linux/iommu.h | 5 + > > include/linux/vfio.h | 2 + > > include/uapi/linux/vfio.h | 26 + > > 13 files changed, 1537 insertions(+), 19 deletions(-) > > create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3- > impl.c > > create mode 100644 drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c > > > > _______________________________________________ > iommu mailing list > iommu@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/iommu
On Wed, Sep 01, 2021 at 06:55:55AM +0000, Tian, Kevin wrote: > > From: Alex Williamson > > Sent: Wednesday, September 1, 2021 12:16 AM > > > > On Mon, 30 Aug 2021 19:59:10 -0700 > > Nicolin Chen <nicolinc@nvidia.com> wrote: > > > > > The SMMUv3 devices implemented in the Grace SoC support NVIDIA's > > custom > > > CMDQ-Virtualization (CMDQV) hardware. Like the new ECMDQ feature first > > > introduced in the ARM SMMUv3.3 specification, CMDQV adds multiple > > VCMDQ > > > interfaces to supplement the single architected SMMU_CMDQ in an effort > > > to reduce contention. > > > > > > This series of patches add CMDQV support with its preparational changes: > > > > > > * PATCH-1 to PATCH-8 are related to shared VMID feature: they are used > > > first to improve TLB utilization, second to bind a shared VMID with a > > > VCMDQ interface for hardware configuring requirement. > > > > The vfio changes would need to be implemented in alignment with the > > /dev/iommu proposals[1]. AIUI, the VMID is essentially binding > > multiple containers together for TLB invalidation, which I expect in > > the proposal below is largely already taken care of in that a single > > iommu-fd can support multiple I/O address spaces and it's largely > > expected that a hypervisor would use a single iommu-fd so this explicit > > connection by userspace across containers wouldn't be necessary. > > Agree. VMID is equivalent to DID (domain id) in other vendor iommus. > with /dev/iommu multiple I/O address spaces can share the same VMID > via nesting. No need of exposing VMID to userspace to build the > connection. Indeed, this looks like a flavour of the accelerated invalidation stuff we've talked about already. I would see it probably exposed as some HW specific IOCTL on the iommu fd to get access to the accelerated invalidation for IOASID's in the FD. Indeed, this seems like a further example of why /dev/iommu is looking like a good idea as this RFC is very complicated to do something fairly simple. Where are thing on the /dev/iommu work these days? Thanks, Jason
> From: Jason Gunthorpe <jgg@nvidia.com> > Sent: Thursday, September 2, 2021 10:45 PM > > On Wed, Sep 01, 2021 at 06:55:55AM +0000, Tian, Kevin wrote: > > > From: Alex Williamson > > > Sent: Wednesday, September 1, 2021 12:16 AM > > > > > > On Mon, 30 Aug 2021 19:59:10 -0700 > > > Nicolin Chen <nicolinc@nvidia.com> wrote: > > > > > > > The SMMUv3 devices implemented in the Grace SoC support NVIDIA's > > > custom > > > > CMDQ-Virtualization (CMDQV) hardware. Like the new ECMDQ feature > first > > > > introduced in the ARM SMMUv3.3 specification, CMDQV adds multiple > > > VCMDQ > > > > interfaces to supplement the single architected SMMU_CMDQ in an > effort > > > > to reduce contention. > > > > > > > > This series of patches add CMDQV support with its preparational > changes: > > > > > > > > * PATCH-1 to PATCH-8 are related to shared VMID feature: they are > used > > > > first to improve TLB utilization, second to bind a shared VMID with a > > > > VCMDQ interface for hardware configuring requirement. > > > > > > The vfio changes would need to be implemented in alignment with the > > > /dev/iommu proposals[1]. AIUI, the VMID is essentially binding > > > multiple containers together for TLB invalidation, which I expect in > > > the proposal below is largely already taken care of in that a single > > > iommu-fd can support multiple I/O address spaces and it's largely > > > expected that a hypervisor would use a single iommu-fd so this explicit > > > connection by userspace across containers wouldn't be necessary. > > > > Agree. VMID is equivalent to DID (domain id) in other vendor iommus. > > with /dev/iommu multiple I/O address spaces can share the same VMID > > via nesting. No need of exposing VMID to userspace to build the > > connection. > > Indeed, this looks like a flavour of the accelerated invalidation > stuff we've talked about already. > > I would see it probably exposed as some HW specific IOCTL on the iommu > fd to get access to the accelerated invalidation for IOASID's in the > FD. > > Indeed, this seems like a further example of why /dev/iommu is looking > like a good idea as this RFC is very complicated to do something > fairly simple. > > Where are thing on the /dev/iommu work these days? > We are actively working on the basic skeleton. Our original plan is to send out the 1st draft before LPC, with support of vfio type1 semantics and and pci dev only (single-device group). But later we realized that adding multi-devices group support is also necessary even in the 1st draft to avoid some dirty hacks and build the complete picture. This will add some time though. If things go well, we'll still try hit the original plan. If not, it will be soon after LPC. Thanks Kevin
Hi Kevin, On Thu, Sep 02, 2021 at 10:27:06PM +0000, Tian, Kevin wrote: > > Indeed, this looks like a flavour of the accelerated invalidation > > stuff we've talked about already. > > > > I would see it probably exposed as some HW specific IOCTL on the iommu > > fd to get access to the accelerated invalidation for IOASID's in the > > FD. > > > > Indeed, this seems like a further example of why /dev/iommu is looking > > like a good idea as this RFC is very complicated to do something > > fairly simple. > > > > Where are thing on the /dev/iommu work these days? > > We are actively working on the basic skeleton. Our original plan is to send > out the 1st draft before LPC, with support of vfio type1 semantics and > and pci dev only (single-device group). But later we realized that adding > multi-devices group support is also necessary even in the 1st draft to avoid > some dirty hacks and build the complete picture. This will add some time > though. If things go well, we'll still try hit the original plan. If not, it will be > soon after LPC. As I also want to take a look at your work for this implementation, would you please CC me when you send out your patches? Thank you!
The SMMUv3 devices implemented in the Grace SoC support NVIDIA's custom CMDQ-Virtualization (CMDQV) hardware. Like the new ECMDQ feature first introduced in the ARM SMMUv3.3 specification, CMDQV adds multiple VCMDQ interfaces to supplement the single architected SMMU_CMDQ in an effort to reduce contention. This series of patches add CMDQV support with its preparational changes: * PATCH-1 to PATCH-8 are related to shared VMID feature: they are used first to improve TLB utilization, second to bind a shared VMID with a VCMDQ interface for hardware configuring requirement. * PATCH-9 and PATCH-10 are to accommodate the NVIDIA implementation with the existing arm-smmu-v3 driver. * PATCH-11 borrows the "implementation infrastructure" from the arm-smmu driver so later change can build upon it. * PATCH-12 adds an initial NVIDIA implementation related to host feature, and also adds implementation specific ->device_reset() and ->get_cmdq() callback functions. * PATCH-13 adds virtualization features using VFIO mdev interface, which allows user space hypervisor to map and get access to one of the VCMDQ interfaces of CMDQV module. ( Thinking that reviewers can get a better view of this implementation, I am attaching QEMU changes here for reference purpose: https://github.com/nicolinc/qemu/commits/dev/cmdqv_v6.0.0-rc2 The branch has all preparational changes, while I'm still integrating device model and ARM-VIRT changes, and will push them these two days, although they might not be in a good shape of being sent to review yet ) Above all, I marked RFC for this series, as I feel that we may come up some better solution. So please kindly share your reviews and insights. Thank you! Changelog v1->v2: * Added mdev interface support for hypervisor and VMs. * Added preparational changes for mdev interface implementation. * PATCH-12 Changed ->issue_cmdlist() to ->get_cmdq() for a better integration with recently merged ECMDQ-related changes. Nate Watterson (3): iommu/arm-smmu-v3: Add implementation infrastructure iommu/arm-smmu-v3: Add support for NVIDIA CMDQ-Virtualization hw iommu/nvidia-smmu-v3: Add mdev interface support Nicolin Chen (10): iommu: Add set_nesting_vmid/get_nesting_vmid functions vfio: add VFIO_IOMMU_GET_VMID and VFIO_IOMMU_SET_VMID vfio: Document VMID control for IOMMU Virtualization vfio: add set_vmid and get_vmid for vfio_iommu_type1 vfio/type1: Implement set_vmid and get_vmid vfio/type1: Set/get VMID to/from iommu driver iommu/arm-smmu-v3: Add shared VMID support for NESTING iommu/arm-smmu-v3: Add VMID alloc/free helpers iommu/arm-smmu-v3: Pass dev pointer to arm_smmu_detach_dev iommu/arm-smmu-v3: Pass cmdq pointer in arm_smmu_cmdq_issue_cmdlist() Documentation/driver-api/vfio.rst | 34 + MAINTAINERS | 2 + drivers/iommu/arm/arm-smmu-v3/Makefile | 2 +- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c | 15 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 121 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 18 + .../iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c | 1249 +++++++++++++++++ drivers/iommu/iommu.c | 20 + drivers/vfio/vfio.c | 25 + drivers/vfio/vfio_iommu_type1.c | 37 + include/linux/iommu.h | 5 + include/linux/vfio.h | 2 + include/uapi/linux/vfio.h | 26 + 13 files changed, 1537 insertions(+), 19 deletions(-) create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c create mode 100644 drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c