Message ID | 0-v4-9e99b76f3518+3a8-smmuv3_nesting_jgg@nvidia.com (mailing list archive) |
---|---|
Headers | show |
Series | Initial support for SMMUv3 nested translation | expand |
On Wed, Oct 30, 2024 at 09:20:44PM -0300, Jason Gunthorpe wrote: > The vIOMMU series is essential to allow the invalidations to be processed > for the CD as well. > > It is enough to allow qemu work to progress. > > This is on github: https://github.com/jgunthorpe/linux/commits/smmuv3_nesting Tested-by: Nicolin Chen <nicolinc@nvidia.com> With a branch adding RMR changes on top of this smmuv3_nesting: https://github.com/nicolinc/iommufd/commits/smmuv3_nesting-with-rmr and with an *updated* paring QEMU branch: https://github.com/nicolinc/qemu/commits/wip/for_smmuv3_nesting-v4 For folks who are interested in testing, please use this QEMU branch, as there are uAPI changes against previous verions. Thanks Nicolin
On Wed, Oct 30, 2024 at 09:20:44PM -0300, Jason Gunthorpe wrote: > [This is now based on Nicolin's iommufd patches for vIOMMU and will need > to go through the iommufd tree, please ack] Can't we separate out the SMMUv3 driver changes? They shouldn't depend on Nicolin's work afaict. Will
On Fri, Nov 01, 2024 at 12:18:50PM +0000, Will Deacon wrote: > On Wed, Oct 30, 2024 at 09:20:44PM -0300, Jason Gunthorpe wrote: > > [This is now based on Nicolin's iommufd patches for vIOMMU and will need > > to go through the iommufd tree, please ack] > > Can't we separate out the SMMUv3 driver changes? They shouldn't depend on > Nicolin's work afaict. We can put everything before "iommu/arm-smmu-v3: Support IOMMU_VIOMMU_ALLOC" directly on a rc and share a branch with your tree. That patch and following all depend on Nicolin's work, as they rely on the VIOMMU due to how different ARM is from Intel. How about you take these patches: [PATCH v4 01/12] vfio: Remove VFIO_TYPE1_NESTING_IOMMU [PATCH v4 02/12] ACPICA: IORT: Update for revision E.f [PATCH v4 03/12] ACPI/IORT: Support CANWBS memory access flag [PATCH v4 04/12] iommu/arm-smmu-v3: Report IOMMU_CAP_ENFORCE_CACHE_COHERENCY for CANWBS [PATCH v4 05/12] iommu/arm-smmu-v3: Support IOMMU_GET_HW_INFO via struct arm_smmu_hw_info [PATCH v4 06/12] iommu/arm-smmu-v3: Implement IOMMU_HWPT_ALLOC_NEST_PARENT [PATCH v4 07/12] iommu/arm-smmu-v3: Expose the arm_smmu_attach interface Onto a branch. I'll take these patches after merging your branch and Nicolin's: [PATCH v4 08/12] iommu/arm-smmu-v3: Support IOMMU_VIOMMU_ALLOC [PATCH v4 09/12] iommu/arm-smmu-v3: Support IOMMU_DOMAIN_NESTED [PATCH v4 10/12] iommu/arm-smmu-v3: Use S2FWB for NESTED domains [PATCH v4 11/12] iommu/arm-smmu-v3: Allow ATS for IOMMU_DOMAIN_NESTED [PATCH v4 12/12] iommu/arm-smmu-v3: Support IOMMU_HWPT_INVALIDATE using a VIOMMU object ? I can also probably push most of S2FWB and ATS into the first batch. Please let me know, I would like this to be done this cycle, Nicolin's vIOMMU series are all reviewed now. Jason
Hi Jason, On Fri, Nov 01, 2024 at 10:25:03AM -0300, Jason Gunthorpe wrote: > On Fri, Nov 01, 2024 at 12:18:50PM +0000, Will Deacon wrote: > > On Wed, Oct 30, 2024 at 09:20:44PM -0300, Jason Gunthorpe wrote: > > > [This is now based on Nicolin's iommufd patches for vIOMMU and will need > > > to go through the iommufd tree, please ack] > > > > Can't we separate out the SMMUv3 driver changes? They shouldn't depend on > > Nicolin's work afaict. > > We can put everything before "iommu/arm-smmu-v3: Support > IOMMU_VIOMMU_ALLOC" directly on a rc and share a branch with your tree. > > That patch and following all depend on Nicolin's work, as they rely on > the VIOMMU due to how different ARM is from Intel. > > How about you take these patches: > > [PATCH v4 01/12] vfio: Remove VFIO_TYPE1_NESTING_IOMMU > [PATCH v4 02/12] ACPICA: IORT: Update for revision E.f > [PATCH v4 03/12] ACPI/IORT: Support CANWBS memory access flag > [PATCH v4 04/12] iommu/arm-smmu-v3: Report IOMMU_CAP_ENFORCE_CACHE_COHERENCY for CANWBS > [PATCH v4 05/12] iommu/arm-smmu-v3: Support IOMMU_GET_HW_INFO via struct arm_smmu_hw_info > [PATCH v4 06/12] iommu/arm-smmu-v3: Implement IOMMU_HWPT_ALLOC_NEST_PARENT > [PATCH v4 07/12] iommu/arm-smmu-v3: Expose the arm_smmu_attach interface > > Onto a branch. I've pushed these onto a new branch in the IOMMU tree: iommufd/arm-smmuv3-nested However, please can you give it a day or two to get some exposure in -next before you merge that into iommufd? I'll ping back here later in the week. Cheers, Will
On Tue, Nov 05, 2024 at 04:48:29PM +0000, Will Deacon wrote: > Hi Jason, > > On Fri, Nov 01, 2024 at 10:25:03AM -0300, Jason Gunthorpe wrote: > > On Fri, Nov 01, 2024 at 12:18:50PM +0000, Will Deacon wrote: > > > On Wed, Oct 30, 2024 at 09:20:44PM -0300, Jason Gunthorpe wrote: > > > > [This is now based on Nicolin's iommufd patches for vIOMMU and will need > > > > to go through the iommufd tree, please ack] > > > > > > Can't we separate out the SMMUv3 driver changes? They shouldn't depend on > > > Nicolin's work afaict. > > > > We can put everything before "iommu/arm-smmu-v3: Support > > IOMMU_VIOMMU_ALLOC" directly on a rc and share a branch with your tree. > > > > That patch and following all depend on Nicolin's work, as they rely on > > the VIOMMU due to how different ARM is from Intel. > > > > How about you take these patches: > > > > [PATCH v4 01/12] vfio: Remove VFIO_TYPE1_NESTING_IOMMU > > [PATCH v4 02/12] ACPICA: IORT: Update for revision E.f > > [PATCH v4 03/12] ACPI/IORT: Support CANWBS memory access flag > > [PATCH v4 04/12] iommu/arm-smmu-v3: Report IOMMU_CAP_ENFORCE_CACHE_COHERENCY for CANWBS > > [PATCH v4 05/12] iommu/arm-smmu-v3: Support IOMMU_GET_HW_INFO via struct arm_smmu_hw_info > > [PATCH v4 06/12] iommu/arm-smmu-v3: Implement IOMMU_HWPT_ALLOC_NEST_PARENT > > [PATCH v4 07/12] iommu/arm-smmu-v3: Expose the arm_smmu_attach interface > > > > Onto a branch. > > I've pushed these onto a new branch in the IOMMU tree: > > iommufd/arm-smmuv3-nested > > However, please can you give it a day or two to get some exposure in > -next before you merge that into iommufd? I'll ping back here later in > the week. Thanks, I will hope to put the other parts together on Friday then so it can also get its time in -next, as we are running out of days before the merge window 0-day is still complaining about some kconfig in Nicolin's patches so we need to settle that anyhow. Jason
On Tue, Nov 05, 2024 at 04:48:29PM +0000, Will Deacon wrote: > On Fri, Nov 01, 2024 at 10:25:03AM -0300, Jason Gunthorpe wrote: > > On Fri, Nov 01, 2024 at 12:18:50PM +0000, Will Deacon wrote: > > > On Wed, Oct 30, 2024 at 09:20:44PM -0300, Jason Gunthorpe wrote: > > > > [This is now based on Nicolin's iommufd patches for vIOMMU and will need > > > > to go through the iommufd tree, please ack] > > > > > > Can't we separate out the SMMUv3 driver changes? They shouldn't depend on > > > Nicolin's work afaict. > > > > We can put everything before "iommu/arm-smmu-v3: Support > > IOMMU_VIOMMU_ALLOC" directly on a rc and share a branch with your tree. > > > > That patch and following all depend on Nicolin's work, as they rely on > > the VIOMMU due to how different ARM is from Intel. > > > > How about you take these patches: > > > > [PATCH v4 01/12] vfio: Remove VFIO_TYPE1_NESTING_IOMMU > > [PATCH v4 02/12] ACPICA: IORT: Update for revision E.f > > [PATCH v4 03/12] ACPI/IORT: Support CANWBS memory access flag > > [PATCH v4 04/12] iommu/arm-smmu-v3: Report IOMMU_CAP_ENFORCE_CACHE_COHERENCY for CANWBS > > [PATCH v4 05/12] iommu/arm-smmu-v3: Support IOMMU_GET_HW_INFO via struct arm_smmu_hw_info > > [PATCH v4 06/12] iommu/arm-smmu-v3: Implement IOMMU_HWPT_ALLOC_NEST_PARENT > > [PATCH v4 07/12] iommu/arm-smmu-v3: Expose the arm_smmu_attach interface > > > > Onto a branch. > > I've pushed these onto a new branch in the IOMMU tree: > > iommufd/arm-smmuv3-nested > > However, please can you give it a day or two to get some exposure in > -next before you merge that into iommufd? I'll ping back here later in > the week. Not seen anything go bang in -next, so I think we can consider this branch stable now. Will
On Wed, Oct 30, 2024 at 09:20:44PM -0300, Jason Gunthorpe wrote: > Jason Gunthorpe (7): > iommu/arm-smmu-v3: Support IOMMU_DOMAIN_NESTED > iommu/arm-smmu-v3: Use S2FWB for NESTED domains > iommu/arm-smmu-v3: Allow ATS for IOMMU_DOMAIN_NESTED > > Nicolin Chen (5): > iommu/arm-smmu-v3: Support IOMMU_VIOMMU_ALLOC > iommu/arm-smmu-v3: Support IOMMU_HWPT_INVALIDATE using a VIOMMU object Applied to iommufd for-next along with all the dependencies and the additional hunk Zhangfei pointed out. Thanks, Jason
On Wed, 13 Nov 2024 at 02:29, Jason Gunthorpe <jgg@nvidia.com> wrote: > > On Wed, Oct 30, 2024 at 09:20:44PM -0300, Jason Gunthorpe wrote: > > Jason Gunthorpe (7): > > iommu/arm-smmu-v3: Support IOMMU_DOMAIN_NESTED > > iommu/arm-smmu-v3: Use S2FWB for NESTED domains > > iommu/arm-smmu-v3: Allow ATS for IOMMU_DOMAIN_NESTED > > > > Nicolin Chen (5): > > iommu/arm-smmu-v3: Support IOMMU_VIOMMU_ALLOC > > iommu/arm-smmu-v3: Support IOMMU_HWPT_INVALIDATE using a VIOMMU object > > Applied to iommufd for-next along with all the dependencies and the > additional hunk Zhangfei pointed out. Thanks Jason, I have verified on aarch64 based on your jason/smmuv3_nesting branch https://github.com/Linaro/linux-kernel-uadk/tree/6.12-wip https://github.com/Linaro/qemu/tree/6.12-wip Still need this hack https://github.com/Linaro/linux-kernel-uadk/commit/eaa194d954112cad4da7852e29343e546baf8683 One is adding iommu_dev_enable/disable_feature IOMMU_DEV_FEAT_SVA, which you have patchset before. The other is to temporarily ignore S2FWB or CANWBS. Thanks
On Wed, Nov 13, 2024 at 09:01:41AM +0800, Zhangfei Gao wrote: > On Wed, 13 Nov 2024 at 02:29, Jason Gunthorpe <jgg@nvidia.com> wrote: > > > > On Wed, Oct 30, 2024 at 09:20:44PM -0300, Jason Gunthorpe wrote: > > > Jason Gunthorpe (7): > > > iommu/arm-smmu-v3: Support IOMMU_DOMAIN_NESTED > > > iommu/arm-smmu-v3: Use S2FWB for NESTED domains > > > iommu/arm-smmu-v3: Allow ATS for IOMMU_DOMAIN_NESTED > > > > > > Nicolin Chen (5): > > > iommu/arm-smmu-v3: Support IOMMU_VIOMMU_ALLOC > > > iommu/arm-smmu-v3: Support IOMMU_HWPT_INVALIDATE using a VIOMMU object > > > > Applied to iommufd for-next along with all the dependencies and the > > additional hunk Zhangfei pointed out. > > Thanks Jason, I have verified on aarch64 based on your > jason/smmuv3_nesting branch Great, thanks > https://github.com/Linaro/linux-kernel-uadk/tree/6.12-wip > https://github.com/Linaro/qemu/tree/6.12-wip > > Still need this hack > https://github.com/Linaro/linux-kernel-uadk/commit/eaa194d954112cad4da7852e29343e546baf8683 > > One is adding iommu_dev_enable/disable_feature IOMMU_DEV_FEAT_SVA, > which you have patchset before. Yes, I have a more complete version of that here someplace. Need some help on vt-d but hope to get that done next cycle. > The other is to temporarily ignore S2FWB or CANWBS. Yes, ideally you should do that in the device FW, or perhaps via the Linux ACPI patching? Jason
On 11/13/24 09:23, Jason Gunthorpe wrote: >> https://github.com/Linaro/linux-kernel-uadk/tree/6.12-wip >> https://github.com/Linaro/qemu/tree/6.12-wip >> >> Still need this hack >> https://github.com/Linaro/linux-kernel-uadk/commit/ >> eaa194d954112cad4da7852e29343e546baf8683 >> >> One is adding iommu_dev_enable/disable_feature IOMMU_DEV_FEAT_SVA, >> which you have patchset before. > Yes, I have a more complete version of that here someplace. Need some > help on vt-d but hope to get that done next cycle. Can you please elaborate this a bit more? Are you talking about below change + ret = iommu_dev_enable_feature(idev->dev, IOMMU_DEV_FEAT_SVA); + if (ret) + return ret; in iommufd_fault_iopf_enable()? I have no idea about why SVA is affected when enabling iopf. -- baolu
Hi, Baolu On Wed, 13 Nov 2024 at 10:56, Baolu Lu <baolu.lu@linux.intel.com> wrote: > > On 11/13/24 09:23, Jason Gunthorpe wrote: > >> https://github.com/Linaro/linux-kernel-uadk/tree/6.12-wip > >> https://github.com/Linaro/qemu/tree/6.12-wip > >> > >> Still need this hack > >> https://github.com/Linaro/linux-kernel-uadk/commit/ > >> eaa194d954112cad4da7852e29343e546baf8683 > >> > >> One is adding iommu_dev_enable/disable_feature IOMMU_DEV_FEAT_SVA, > >> which you have patchset before. > > Yes, I have a more complete version of that here someplace. Need some > > help on vt-d but hope to get that done next cycle. > > Can you please elaborate this a bit more? Are you talking about below > change > > + ret = iommu_dev_enable_feature(idev->dev, IOMMU_DEV_FEAT_SVA); > + if (ret) > + return ret; > > in iommufd_fault_iopf_enable()? > > I have no idea about why SVA is affected when enabling iopf. In drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c, iommu_dev_enable_feature(IOMMU_DEV_FEAT_SVA) will real call iopf_queue_add_device, while iommu_dev_enable_feature(IOPF) only set flag. arm_smmu_dev_enable_feature case IOMMU_DEV_FEAT_SVA: arm_smmu_master_enable_sva(master) iopf_queue_add_device(master->smmu->evtq.iopf, dev); Thanks
On Wed, Nov 13, 2024 at 10:55:41AM +0800, Baolu Lu wrote: > On 11/13/24 09:23, Jason Gunthorpe wrote: > > > https://github.com/Linaro/linux-kernel-uadk/tree/6.12-wip > > > https://github.com/Linaro/qemu/tree/6.12-wip > > > > > > Still need this hack > > > https://github.com/Linaro/linux-kernel-uadk/commit/ > > > eaa194d954112cad4da7852e29343e546baf8683 > > > > > > One is adding iommu_dev_enable/disable_feature IOMMU_DEV_FEAT_SVA, > > > which you have patchset before. > > Yes, I have a more complete version of that here someplace. Need some > > help on vt-d but hope to get that done next cycle. > > Can you please elaborate this a bit more? Are you talking about below > change I need your help to remove IOMMU_DEV_FEAT_IOPF from the intel driver. I have a patch series that eliminates it from all the other drivers, and I wrote a patch to remove FEAT_SVA from intel.. > + ret = iommu_dev_enable_feature(idev->dev, IOMMU_DEV_FEAT_SVA); > + if (ret) > + return ret; > > in iommufd_fault_iopf_enable()? > > I have no idea about why SVA is affected when enabling iopf. It is ARM not implementing the API correctly. Only SVA turns on the page fault reporting mechanism. In the new world the page fault reporting should be managed during domain attachment. If the domain is fault capable then faults should be delivered to that domain. That is the correct time to setup the iopf mechanism as well. So I fixed that and now ARM and AMD both have no-op implementations of IOMMU_DEV_FEAT_IOPF and IOMMU_DEV_FEAT_SVA. Thus I'd like to remove it entirely. Jason
On 11/14/24 00:43, Jason Gunthorpe wrote: > On Wed, Nov 13, 2024 at 10:55:41AM +0800, Baolu Lu wrote: >> On 11/13/24 09:23, Jason Gunthorpe wrote: >>>> https://github.com/Linaro/linux-kernel-uadk/tree/6.12-wip >>>> https://github.com/Linaro/qemu/tree/6.12-wip >>>> >>>> Still need this hack >>>> https://github.com/Linaro/linux-kernel-uadk/commit/ >>>> eaa194d954112cad4da7852e29343e546baf8683 >>>> >>>> One is adding iommu_dev_enable/disable_feature IOMMU_DEV_FEAT_SVA, >>>> which you have patchset before. >>> Yes, I have a more complete version of that here someplace. Need some >>> help on vt-d but hope to get that done next cycle. >> >> Can you please elaborate this a bit more? Are you talking about below >> change > > I need your help to remove IOMMU_DEV_FEAT_IOPF from the intel > driver. I have a patch series that eliminates it from all the other > drivers, and I wrote a patch to remove FEAT_SVA from intel.. Yes, sure. Let's make this happen in the next cycle. FEAT_IOPF could be removed. IOPF manipulation can be handled in the domain attachment path. A per-device refcount can be implemented. This count increments with each iopf-capable domain attachment and decrements with each detachment. PCI PRI is enabled for the first iopf-capable domain and disabled when the last one is removed. Probably we can also solve the PF/VF sharing PRI issue. With iopf moved to the domain attach path and hardware capability checks to the SVA domain allocation path, FEAT_SVA becomes essentially a no-op. > >> + ret = iommu_dev_enable_feature(idev->dev, IOMMU_DEV_FEAT_SVA); >> + if (ret) >> + return ret; >> >> in iommufd_fault_iopf_enable()? >> >> I have no idea about why SVA is affected when enabling iopf. > > It is ARM not implementing the API correctly. Only SVA turns on the > page fault reporting mechanism. > > In the new world the page fault reporting should be managed during > domain attachment. If the domain is fault capable then faults should > be delivered to that domain. That is the correct time to setup the > iopf mechanism as well. > > So I fixed that and now ARM and AMD both have no-op implementations of > IOMMU_DEV_FEAT_IOPF and IOMMU_DEV_FEAT_SVA. Thus I'd like to remove it > entirely. Thank you for the explanation. -- baolu