| Message ID | 20211027104428.1059740-1-eric.auger@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | SMMUv3 Nested Stage Setup (IOMMU part) | expand |
Hi, Eric On 2021/10/27 下午6:44, Eric Auger wrote: > This series brings the IOMMU part of HW nested paging support > in the SMMUv3. > > The SMMUv3 driver is adapted to support 2 nested stages. > > The IOMMU API is extended to convey the guest stage 1 > configuration and the hook is implemented in the SMMUv3 driver. > > This allows the guest to own the stage 1 tables and context > descriptors (so-called PASID table) while the host owns the > stage 2 tables and main configuration structures (STE). > > This work mainly is provided for test purpose as the upper > layer integration is under rework and bound to be based on > /dev/iommu instead of VFIO tunneling. In this version we also get > rid of the MSI BINDING ioctl, assuming the guest enforces > flat mapping of host IOVAs used to bind physical MSI doorbells. > In the current QEMU integration this is achieved by exposing > RMRs to the guest, using Shameer's series [1]. This approach > is RFC as the IORT spec is not really meant to do that > (single mapping flag limitation). > > Best Regards > > Eric > > This series (Host) can be found at: > https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16 > This includes a rebased VFIO integration (although not meant > to be upstreamed) > > Guest kernel branch can be found at: > https://github.com/eauger/linux/tree/shameer_rmrr_v7 > featuring [1] > > QEMU integration (still based on VFIO and exposing RMRs) > can be found at: > https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10 > (use iommu=nested-smmuv3 ARM virt option) > > Guest dependency: > [1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node Thanks a lot for upgrading these patches. I have basically verified these patches on HiSilicon Kunpeng920. And integrated them to these branches. https://github.com/Linaro/linux-kernel-uadk/tree/uacce-devel-5.16 https://github.com/Linaro/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10 Though they are provided for test purpose, Tested-by: Zhangfei Gao <zhangfei.gao@linaro.org> Thanks
Hi Eric, > This series brings the IOMMU part of HW nested paging support > in the SMMUv3. > > The SMMUv3 driver is adapted to support 2 nested stages. > > The IOMMU API is extended to convey the guest stage 1 > configuration and the hook is implemented in the SMMUv3 driver. > > This allows the guest to own the stage 1 tables and context > descriptors (so-called PASID table) while the host owns the > stage 2 tables and main configuration structures (STE). > > This work mainly is provided for test purpose as the upper > layer integration is under rework and bound to be based on > /dev/iommu instead of VFIO tunneling. In this version we also get > rid of the MSI BINDING ioctl, assuming the guest enforces > flat mapping of host IOVAs used to bind physical MSI doorbells. > In the current QEMU integration this is achieved by exposing > RMRs to the guest, using Shameer's series [1]. This approach > is RFC as the IORT spec is not really meant to do that > (single mapping flag limitation). > > Best Regards > > Eric > > This series (Host) can be found at: > https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16 > This includes a rebased VFIO integration (although not meant > to be upstreamed) > > Guest kernel branch can be found at: > https://github.com/eauger/linux/tree/shameer_rmrr_v7 > featuring [1] > > QEMU integration (still based on VFIO and exposing RMRs) > can be found at: > https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10 > (use iommu=nested-smmuv3 ARM virt option) > > Guest dependency: > [1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node > > History: > > v15 -> v16: > - guest RIL must support RIL > - additional checks in the cache invalidation hook > - removal of the MSI BINDING ioctl (tentative replacement > by RMRs) > > > Eric Auger (9): > iommu: Introduce attach/detach_pasid_table API > iommu: Introduce iommu_get_nesting > iommu/smmuv3: Allow s1 and s2 configs to coexist > iommu/smmuv3: Get prepared for nested stage support > iommu/smmuv3: Implement attach/detach_pasid_table > iommu/smmuv3: Allow stage 1 invalidation with unmanaged ASIDs > iommu/smmuv3: Implement cache_invalidate > iommu/smmuv3: report additional recoverable faults > iommu/smmuv3: Disallow nested mode in presence of HW MSI regions Hi Eric, I validated the reworked test patches in v16 from the given branches with Kernel v5.15 & Qemu v6.2. Verified them with NVMe PCI device assigned to Guest VM. Sorry, forgot to update earlier. Tested-by: Sumit Gupta <sumitg@nvidia.com> Thanks, Sumit Gupta
Hi Zhangfei, On 12/3/21 1:27 PM, Zhangfei Gao wrote: > > Hi, Eric > > On 2021/10/27 下午6:44, Eric Auger wrote: >> This series brings the IOMMU part of HW nested paging support >> in the SMMUv3. >> >> The SMMUv3 driver is adapted to support 2 nested stages. >> >> The IOMMU API is extended to convey the guest stage 1 >> configuration and the hook is implemented in the SMMUv3 driver. >> >> This allows the guest to own the stage 1 tables and context >> descriptors (so-called PASID table) while the host owns the >> stage 2 tables and main configuration structures (STE). >> >> This work mainly is provided for test purpose as the upper >> layer integration is under rework and bound to be based on >> /dev/iommu instead of VFIO tunneling. In this version we also get >> rid of the MSI BINDING ioctl, assuming the guest enforces >> flat mapping of host IOVAs used to bind physical MSI doorbells. >> In the current QEMU integration this is achieved by exposing >> RMRs to the guest, using Shameer's series [1]. This approach >> is RFC as the IORT spec is not really meant to do that >> (single mapping flag limitation). >> >> Best Regards >> >> Eric >> >> This series (Host) can be found at: >> https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16 >> This includes a rebased VFIO integration (although not meant >> to be upstreamed) >> >> Guest kernel branch can be found at: >> https://github.com/eauger/linux/tree/shameer_rmrr_v7 >> featuring [1] >> >> QEMU integration (still based on VFIO and exposing RMRs) >> can be found at: >> https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10 >> (use iommu=nested-smmuv3 ARM virt option) >> >> Guest dependency: >> [1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node > > Thanks a lot for upgrading these patches. > > I have basically verified these patches on HiSilicon Kunpeng920. > And integrated them to these branches. > https://github.com/Linaro/linux-kernel-uadk/tree/uacce-devel-5.16 > https://github.com/Linaro/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10 > > Though they are provided for test purpose, > > Tested-by: Zhangfei Gao <zhangfei.gao@linaro.org> Thank you very much. As you mentioned, until we do not have the /dev/iommu integration this is maintained for testing purpose. The SMMU changes shouldn't be much impacted though. The added value of this respin was to propose an MSI binding solution based on RMRRs which simplify things at kernel level. Thanks! Eric > > Thanks >
Hi Sumit, On 12/3/21 2:13 PM, Sumit Gupta wrote: > Hi Eric, > >> This series brings the IOMMU part of HW nested paging support >> in the SMMUv3. >> >> The SMMUv3 driver is adapted to support 2 nested stages. >> >> The IOMMU API is extended to convey the guest stage 1 >> configuration and the hook is implemented in the SMMUv3 driver. >> >> This allows the guest to own the stage 1 tables and context >> descriptors (so-called PASID table) while the host owns the >> stage 2 tables and main configuration structures (STE). >> >> This work mainly is provided for test purpose as the upper >> layer integration is under rework and bound to be based on >> /dev/iommu instead of VFIO tunneling. In this version we also get >> rid of the MSI BINDING ioctl, assuming the guest enforces >> flat mapping of host IOVAs used to bind physical MSI doorbells. >> In the current QEMU integration this is achieved by exposing >> RMRs to the guest, using Shameer's series [1]. This approach >> is RFC as the IORT spec is not really meant to do that >> (single mapping flag limitation). >> >> Best Regards >> >> Eric >> >> This series (Host) can be found at: >> https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16 >> This includes a rebased VFIO integration (although not meant >> to be upstreamed) >> >> Guest kernel branch can be found at: >> https://github.com/eauger/linux/tree/shameer_rmrr_v7 >> featuring [1] >> >> QEMU integration (still based on VFIO and exposing RMRs) >> can be found at: >> https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10 >> (use iommu=nested-smmuv3 ARM virt option) >> >> Guest dependency: >> [1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node >> >> History: >> >> v15 -> v16: >> - guest RIL must support RIL >> - additional checks in the cache invalidation hook >> - removal of the MSI BINDING ioctl (tentative replacement >> by RMRs) >> >> >> Eric Auger (9): >> iommu: Introduce attach/detach_pasid_table API >> iommu: Introduce iommu_get_nesting >> iommu/smmuv3: Allow s1 and s2 configs to coexist >> iommu/smmuv3: Get prepared for nested stage support >> iommu/smmuv3: Implement attach/detach_pasid_table >> iommu/smmuv3: Allow stage 1 invalidation with unmanaged ASIDs >> iommu/smmuv3: Implement cache_invalidate >> iommu/smmuv3: report additional recoverable faults >> iommu/smmuv3: Disallow nested mode in presence of HW MSI regions > Hi Eric, > > I validated the reworked test patches in v16 from the given > branches with Kernel v5.15 & Qemu v6.2. Verified them with > NVMe PCI device assigned to Guest VM. > Sorry, forgot to update earlier. > > Tested-by: Sumit Gupta <sumitg@nvidia.com> Thank you very much! Best Regards Eric > > Thanks, > Sumit Gupta >
On 2021/12/7 下午6:27, Eric Auger wrote: > Hi Zhangfei, > > On 12/3/21 1:27 PM, Zhangfei Gao wrote: >> Hi, Eric >> >> On 2021/10/27 下午6:44, Eric Auger wrote: >>> This series brings the IOMMU part of HW nested paging support >>> in the SMMUv3. >>> >>> The SMMUv3 driver is adapted to support 2 nested stages. >>> >>> The IOMMU API is extended to convey the guest stage 1 >>> configuration and the hook is implemented in the SMMUv3 driver. >>> >>> This allows the guest to own the stage 1 tables and context >>> descriptors (so-called PASID table) while the host owns the >>> stage 2 tables and main configuration structures (STE). >>> >>> This work mainly is provided for test purpose as the upper >>> layer integration is under rework and bound to be based on >>> /dev/iommu instead of VFIO tunneling. In this version we also get >>> rid of the MSI BINDING ioctl, assuming the guest enforces >>> flat mapping of host IOVAs used to bind physical MSI doorbells. >>> In the current QEMU integration this is achieved by exposing >>> RMRs to the guest, using Shameer's series [1]. This approach >>> is RFC as the IORT spec is not really meant to do that >>> (single mapping flag limitation). >>> >>> Best Regards >>> >>> Eric >>> >>> This series (Host) can be found at: >>> https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16 >>> This includes a rebased VFIO integration (although not meant >>> to be upstreamed) >>> >>> Guest kernel branch can be found at: >>> https://github.com/eauger/linux/tree/shameer_rmrr_v7 >>> featuring [1] >>> >>> QEMU integration (still based on VFIO and exposing RMRs) >>> can be found at: >>> https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10 >>> (use iommu=nested-smmuv3 ARM virt option) >>> >>> Guest dependency: >>> [1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node >> Thanks a lot for upgrading these patches. >> >> I have basically verified these patches on HiSilicon Kunpeng920. >> And integrated them to these branches. >> https://github.com/Linaro/linux-kernel-uadk/tree/uacce-devel-5.16 >> https://github.com/Linaro/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10 >> >> Though they are provided for test purpose, >> >> Tested-by: Zhangfei Gao <zhangfei.gao@linaro.org> > Thank you very much. As you mentioned, until we do not have the > /dev/iommu integration this is maintained for testing purpose. The SMMU > changes shouldn't be much impacted though. > The added value of this respin was to propose an MSI binding solution > based on RMRRs which simplify things at kernel level. Current RMRR solution requires uefi enabled, and QEMU_EFI.fd has to be provided to start qemu. Any plan to support dtb as well, which will be simpler since no need QEMU_EFI.fd anymore. Thanks
Hi Zhangfei, On 12/7/21 11:35 AM, Zhangfei Gao wrote: > > > On 2021/12/7 下午6:27, Eric Auger wrote: >> Hi Zhangfei, >> >> On 12/3/21 1:27 PM, Zhangfei Gao wrote: >>> Hi, Eric >>> >>> On 2021/10/27 下午6:44, Eric Auger wrote: >>>> This series brings the IOMMU part of HW nested paging support >>>> in the SMMUv3. >>>> >>>> The SMMUv3 driver is adapted to support 2 nested stages. >>>> >>>> The IOMMU API is extended to convey the guest stage 1 >>>> configuration and the hook is implemented in the SMMUv3 driver. >>>> >>>> This allows the guest to own the stage 1 tables and context >>>> descriptors (so-called PASID table) while the host owns the >>>> stage 2 tables and main configuration structures (STE). >>>> >>>> This work mainly is provided for test purpose as the upper >>>> layer integration is under rework and bound to be based on >>>> /dev/iommu instead of VFIO tunneling. In this version we also get >>>> rid of the MSI BINDING ioctl, assuming the guest enforces >>>> flat mapping of host IOVAs used to bind physical MSI doorbells. >>>> In the current QEMU integration this is achieved by exposing >>>> RMRs to the guest, using Shameer's series [1]. This approach >>>> is RFC as the IORT spec is not really meant to do that >>>> (single mapping flag limitation). >>>> >>>> Best Regards >>>> >>>> Eric >>>> >>>> This series (Host) can be found at: >>>> https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16 >>>> This includes a rebased VFIO integration (although not meant >>>> to be upstreamed) >>>> >>>> Guest kernel branch can be found at: >>>> https://github.com/eauger/linux/tree/shameer_rmrr_v7 >>>> featuring [1] >>>> >>>> QEMU integration (still based on VFIO and exposing RMRs) >>>> can be found at: >>>> https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10 >>>> (use iommu=nested-smmuv3 ARM virt option) >>>> >>>> Guest dependency: >>>> [1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node >>> Thanks a lot for upgrading these patches. >>> >>> I have basically verified these patches on HiSilicon Kunpeng920. >>> And integrated them to these branches. >>> https://github.com/Linaro/linux-kernel-uadk/tree/uacce-devel-5.16 >>> https://github.com/Linaro/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10 >>> >>> Though they are provided for test purpose, >>> >>> Tested-by: Zhangfei Gao <zhangfei.gao@linaro.org> >> Thank you very much. As you mentioned, until we do not have the >> /dev/iommu integration this is maintained for testing purpose. The SMMU >> changes shouldn't be much impacted though. >> The added value of this respin was to propose an MSI binding solution >> based on RMRRs which simplify things at kernel level. > > Current RMRR solution requires uefi enabled, > and QEMU_EFI.fd has to be provided to start qemu. > > Any plan to support dtb as well, which will be simpler since no need > QEMU_EFI.fd anymore. Yes the solution is based on ACPI IORT nodes. No clue if some DT integration is under work. Shameer? Thanks Eric > > Thanks > >
> -----Original Message----- > From: Eric Auger [mailto:eric.auger@redhat.com] > Sent: 07 December 2021 11:06 > To: Zhangfei Gao <zhangfei.gao@linaro.org>; eric.auger.pro@gmail.com; > iommu@lists.linux-foundation.org; linux-kernel@vger.kernel.org; > kvm@vger.kernel.org; kvmarm@lists.cs.columbia.edu; joro@8bytes.org; > will@kernel.org; robin.murphy@arm.com; jean-philippe@linaro.org; > zhukeqian <zhukeqian1@huawei.com> > Cc: alex.williamson@redhat.com; jacob.jun.pan@linux.intel.com; > yi.l.liu@intel.com; kevin.tian@intel.com; ashok.raj@intel.com; > maz@kernel.org; peter.maydell@linaro.org; vivek.gautam@arm.com; > Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>; > wangxingang <wangxingang5@huawei.com>; jiangkunkun > <jiangkunkun@huawei.com>; yuzenghui <yuzenghui@huawei.com>; > nicoleotsuka@gmail.com; chenxiang (M) <chenxiang66@hisilicon.com>; > sumitg@nvidia.com; nicolinc@nvidia.com; vdumpa@nvidia.com; > zhangfei.gao@gmail.com; lushenming@huawei.com; vsethi@nvidia.com > Subject: Re: [RFC v16 0/9] SMMUv3 Nested Stage Setup (IOMMU part) > > Hi Zhangfei, > > On 12/7/21 11:35 AM, Zhangfei Gao wrote: > > > > > > On 2021/12/7 下午6:27, Eric Auger wrote: > >> Hi Zhangfei, > >> > >> On 12/3/21 1:27 PM, Zhangfei Gao wrote: > >>> Hi, Eric > >>> > >>> On 2021/10/27 下午6:44, Eric Auger wrote: > >>>> This series brings the IOMMU part of HW nested paging support > >>>> in the SMMUv3. > >>>> > >>>> The SMMUv3 driver is adapted to support 2 nested stages. > >>>> > >>>> The IOMMU API is extended to convey the guest stage 1 > >>>> configuration and the hook is implemented in the SMMUv3 driver. > >>>> > >>>> This allows the guest to own the stage 1 tables and context > >>>> descriptors (so-called PASID table) while the host owns the > >>>> stage 2 tables and main configuration structures (STE). > >>>> > >>>> This work mainly is provided for test purpose as the upper > >>>> layer integration is under rework and bound to be based on > >>>> /dev/iommu instead of VFIO tunneling. In this version we also get > >>>> rid of the MSI BINDING ioctl, assuming the guest enforces > >>>> flat mapping of host IOVAs used to bind physical MSI doorbells. > >>>> In the current QEMU integration this is achieved by exposing > >>>> RMRs to the guest, using Shameer's series [1]. This approach > >>>> is RFC as the IORT spec is not really meant to do that > >>>> (single mapping flag limitation). > >>>> > >>>> Best Regards > >>>> > >>>> Eric > >>>> > >>>> This series (Host) can be found at: > >>>> https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16 > >>>> This includes a rebased VFIO integration (although not meant > >>>> to be upstreamed) > >>>> > >>>> Guest kernel branch can be found at: > >>>> https://github.com/eauger/linux/tree/shameer_rmrr_v7 > >>>> featuring [1] > >>>> > >>>> QEMU integration (still based on VFIO and exposing RMRs) > >>>> can be found at: > >>>> > https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10 > >>>> (use iommu=nested-smmuv3 ARM virt option) > >>>> > >>>> Guest dependency: > >>>> [1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node > >>> Thanks a lot for upgrading these patches. > >>> > >>> I have basically verified these patches on HiSilicon Kunpeng920. > >>> And integrated them to these branches. > >>> https://github.com/Linaro/linux-kernel-uadk/tree/uacce-devel-5.16 > >>> https://github.com/Linaro/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10 > >>> > >>> Though they are provided for test purpose, > >>> > >>> Tested-by: Zhangfei Gao <zhangfei.gao@linaro.org> > >> Thank you very much. As you mentioned, until we do not have the > >> /dev/iommu integration this is maintained for testing purpose. The SMMU > >> changes shouldn't be much impacted though. > >> The added value of this respin was to propose an MSI binding solution > >> based on RMRRs which simplify things at kernel level. > > > > Current RMRR solution requires uefi enabled, > > and QEMU_EFI.fd has to be provided to start qemu. > > > > Any plan to support dtb as well, which will be simpler since no need > > QEMU_EFI.fd anymore. > Yes the solution is based on ACPI IORT nodes. No clue if some DT > integration is under work. Shameer? There was some attempt in the past to create identity mappings using DT. This is the latest I can find on it, https://lore.kernel.org/linux-iommu/YTelDHx2REIIvV%2FN@orome.fritz.box/T/ Thanks, Shameer