| Message ID | 1401987808-23596-5-git-send-email-a.motakis@virtualopensystems.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
> -----Original Message----- > From: iommu-bounces@lists.linux-foundation.org [mailto:iommu- > bounces@lists.linux-foundation.org] On Behalf Of Antonios Motakis > Sent: Thursday, June 05, 2014 10:33 PM > To: alex.williamson@redhat.com; kvmarm@lists.cs.columbia.edu; > iommu@lists.linux-foundation.org > Cc: kvm@vger.kernel.org; eric.auger@linaro.org; open list; > will.deacon@arm.com; a.rigo@virtualopensystems.com; Yoder Stuart-B08248; > moderated list:ARM SMMU DRIVER; Antonios Motakis; > tech@virtualopensystems.com; christoffer.dall@linaro.org > Subject: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability > IOMMU_CAP_INTR_REMAP > > With an ARM SMMU, interrupt remapping should always be safe from the > SMMU's point of view, as it is properly handled by the GIC. > > Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com> > --- > drivers/iommu/arm-smmu.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c index > 15ab2af..ff29402 100644 > --- a/drivers/iommu/arm-smmu.c > +++ b/drivers/iommu/arm-smmu.c > @@ -1544,7 +1544,7 @@ static int arm_smmu_domain_has_cap(struct > iommu_domain *domain, > if (smmu_domain->root_cfg.smmu->features & > ARM_SMMU_FEAT_COHERENT_WALK) > caps |= IOMMU_CAP_CACHE_COHERENCY; > > - caps |= IOMMU_CAP_NOEXEC; > + caps |= IOMMU_CAP_NOEXEC | IOMMU_CAP_INTR_REMAP; > > return !!(cap & caps); > } Why mention this as an IOMMU capability when IOMMU doesn't support it? Also, wouldn't this depend on the GIC IP version? What are we gaining by adding this IOMMU capability list? -Varun
On Thu, Jun 05, 2014 at 07:03:12PM +0200, Antonios Motakis wrote: > With an ARM SMMU, interrupt remapping should always be safe from the > SMMU's point of view, as it is properly handled by the GIC. > > Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com> > --- > drivers/iommu/arm-smmu.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > index 15ab2af..ff29402 100644 > --- a/drivers/iommu/arm-smmu.c > +++ b/drivers/iommu/arm-smmu.c > @@ -1544,7 +1544,7 @@ static int arm_smmu_domain_has_cap(struct iommu_domain *domain, > if (smmu_domain->root_cfg.smmu->features & ARM_SMMU_FEAT_COHERENT_WALK) > caps |= IOMMU_CAP_CACHE_COHERENCY; > > - caps |= IOMMU_CAP_NOEXEC; > + caps |= IOMMU_CAP_NOEXEC | IOMMU_CAP_INTR_REMAP; > > return !!(cap & caps); > } > -- > 1.8.3.2 > What does IOMMU_CAP_INTR_REMAP signify exactly? Is there docs/examples somewhere I can look at? (A quick scan of the Linux souce code doesn't reveal much, and I'm not sure if this is purely MSI related or what...) Thanks, -Christoffer
On Sun, Jun 08, 2014 at 12:31:29PM +0200, Christoffer Dall wrote: > On Thu, Jun 05, 2014 at 07:03:12PM +0200, Antonios Motakis wrote: > > With an ARM SMMU, interrupt remapping should always be safe from the > > SMMU's point of view, as it is properly handled by the GIC. > > > > Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com> > > --- > > drivers/iommu/arm-smmu.c | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > > index 15ab2af..ff29402 100644 > > --- a/drivers/iommu/arm-smmu.c > > +++ b/drivers/iommu/arm-smmu.c > > @@ -1544,7 +1544,7 @@ static int arm_smmu_domain_has_cap(struct iommu_domain *domain, > > if (smmu_domain->root_cfg.smmu->features & ARM_SMMU_FEAT_COHERENT_WALK) > > caps |= IOMMU_CAP_CACHE_COHERENCY; > > > > - caps |= IOMMU_CAP_NOEXEC; > > + caps |= IOMMU_CAP_NOEXEC | IOMMU_CAP_INTR_REMAP; > > > > return !!(cap & caps); > > } > > -- > > 1.8.3.2 > > > What does IOMMU_CAP_INTR_REMAP signify exactly? Is there docs/examples > somewhere I can look at? (A quick scan of the Linux souce code doesn't > reveal much, and I'm not sure if this is purely MSI related or what...) The flag was introduced for x86 IOMMUs to detect whether an IOMMU in the system has and enables interrupt remapping to allow safe device assignment to KVM guests. Without interrupt remapping a malicious guest could attack the host with MSIs from the attached device. How are PCI MSIs implemented on ARM with SMMU and GIC enabled. MSIs are only memory dma transactions in the end, is it guaranteed on ARM that a device only sends MSI transactions it is allowed to? Joerg
On Mon, Jun 16, 2014 at 03:53:44PM +0100, Joerg Roedel wrote: > On Sun, Jun 08, 2014 at 12:31:29PM +0200, Christoffer Dall wrote: > > On Thu, Jun 05, 2014 at 07:03:12PM +0200, Antonios Motakis wrote: > > > With an ARM SMMU, interrupt remapping should always be safe from the > > > SMMU's point of view, as it is properly handled by the GIC. > > > > > > Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com> > > > --- > > > drivers/iommu/arm-smmu.c | 2 +- > > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > > > index 15ab2af..ff29402 100644 > > > --- a/drivers/iommu/arm-smmu.c > > > +++ b/drivers/iommu/arm-smmu.c > > > @@ -1544,7 +1544,7 @@ static int arm_smmu_domain_has_cap(struct iommu_domain *domain, > > > if (smmu_domain->root_cfg.smmu->features & ARM_SMMU_FEAT_COHERENT_WALK) > > > caps |= IOMMU_CAP_CACHE_COHERENCY; > > > > > > - caps |= IOMMU_CAP_NOEXEC; > > > + caps |= IOMMU_CAP_NOEXEC | IOMMU_CAP_INTR_REMAP; > > > > > > return !!(cap & caps); > > > } > > > -- > > > 1.8.3.2 > > > > > What does IOMMU_CAP_INTR_REMAP signify exactly? Is there docs/examples > > somewhere I can look at? (A quick scan of the Linux souce code doesn't > > reveal much, and I'm not sure if this is purely MSI related or what...) > > The flag was introduced for x86 IOMMUs to detect whether an IOMMU in the > system has and enables interrupt remapping to allow safe device > assignment to KVM guests. Without interrupt remapping a malicious guest > could attack the host with MSIs from the attached device. > > How are PCI MSIs implemented on ARM with SMMU and GIC enabled. MSIs are > only memory dma transactions in the end, is it guaranteed on ARM that a > device only sends MSI transactions it is allowed to? MSIs look just like memory accesses made by the device, so the SMMU will translate them to point at the GIC ITS (doorbell). The ITS then has tables to work out how to route the MSI. So, if IOMMU_CAP_INTR_REMAP is simply supposed to indicate that the SMMU can translate MSIs to point somewhere else, then the ARM SMMU can always do it. If it's supposed to indicate that the actual MSI payload can be filtered/routed, then that requires the GIC ITS. The part I'm unsure of is how VFIO knows where to map the MSIs to. That requires knowledge of the physical and virtual doorbell pages -- is that discoverable in the API? Will
On Mon, Jun 16, 2014 at 04:13:29PM +0100, Will Deacon wrote: > MSIs look just like memory accesses made by the device, so the SMMU > will translate them to point at the GIC ITS (doorbell). The ITS then > has tables to work out how to route the MSI. > > So, if IOMMU_CAP_INTR_REMAP is simply supposed to indicate that the > SMMU can translate MSIs to point somewhere else, then the ARM SMMU can > always do it. If it's supposed to indicate that the actual MSI > payload can be filtered/routed, then that requires the GIC ITS. > > The part I'm unsure of is how VFIO knows where to map the MSIs to. > That requires knowledge of the physical and virtual doorbell pages -- > is that discoverable in the API? VFIO does not care about the actual routing, it only cares that the device can not send interrupts it is not allowed to send (e.g. interrupts to vectors used by other devices or, on x86, exception vectors). If that is guaranteed by the SMMU or the GIC ITS hardware and driver then it is fine to set this flag. Joerg
On Mon, Jun 16, 2014 at 04:21:58PM +0100, Joerg Roedel wrote: > On Mon, Jun 16, 2014 at 04:13:29PM +0100, Will Deacon wrote: > > MSIs look just like memory accesses made by the device, so the SMMU > > will translate them to point at the GIC ITS (doorbell). The ITS then > > has tables to work out how to route the MSI. > > > > So, if IOMMU_CAP_INTR_REMAP is simply supposed to indicate that the > > SMMU can translate MSIs to point somewhere else, then the ARM SMMU can > > always do it. If it's supposed to indicate that the actual MSI > > payload can be filtered/routed, then that requires the GIC ITS. > > > > The part I'm unsure of is how VFIO knows where to map the MSIs to. > > That requires knowledge of the physical and virtual doorbell pages -- > > is that discoverable in the API? > > VFIO does not care about the actual routing, it only cares that the > device can not send interrupts it is not allowed to send (e.g. > interrupts to vectors used by other devices or, on x86, exception vectors). > If that is guaranteed by the SMMU or the GIC ITS hardware and driver > then it is fine to set this flag. Ok, thanks. In which case, I think this is really a combined property of the SMMU and the interrupt controller, so we might need some extra code so that the SMMU can check that the interrupt controller for the device is also capable of interrupt remapping. Will
On Mon, 2014-06-16 at 17:21 +0200, Joerg Roedel wrote: > On Mon, Jun 16, 2014 at 04:13:29PM +0100, Will Deacon wrote: > > MSIs look just like memory accesses made by the device, so the SMMU > > will translate them to point at the GIC ITS (doorbell). The ITS then > > has tables to work out how to route the MSI. > > > > So, if IOMMU_CAP_INTR_REMAP is simply supposed to indicate that the > > SMMU can translate MSIs to point somewhere else, then the ARM SMMU can > > always do it. If it's supposed to indicate that the actual MSI > > payload can be filtered/routed, then that requires the GIC ITS. > > > > The part I'm unsure of is how VFIO knows where to map the MSIs to. > > That requires knowledge of the physical and virtual doorbell pages -- > > is that discoverable in the API? > > VFIO does not care about the actual routing, it only cares that the > device can not send interrupts it is not allowed to send (e.g. > interrupts to vectors used by other devices or, on x86, exception vectors). > If that is guaranteed by the SMMU or the GIC ITS hardware and driver > then it is fine to set this flag. Yep, I agree. Thanks, Alex
On Mon, Jun 16, 2014 at 04:25:26PM +0100, Will Deacon wrote: > Ok, thanks. In which case, I think this is really a combined property of > the SMMU and the interrupt controller, so we might need some extra code > so that the SMMU can check that the interrupt controller for the device > is also capable of interrupt remapping. Right, that this is part of IOMMU code has more or less historic reasons on x86. Interrupt remapping is purely implemented in the IOMMU there, so on ARM some clue-code between interrupt controler and smmu is needed. Joerg
Forgive me if this discussion is not relative here, but I thought it is. How is VFIO restricting devices from writing to MSI/MSI-X, Is all the vector area is mapped by VFIO to trap the accesses. I am asking this because we might need to emulate ITS somewhere either in KVM or VFIO to provide direct access to devices. And I don't see any mentions on that. I think this flag needs to be set by ITS emulation. Regards, Tirumalesh. -----Original Message----- From: kvmarm-bounces@lists.cs.columbia.edu [mailto:kvmarm-bounces@lists.cs.columbia.edu] On Behalf Of Joerg Roedel Sent: Monday, June 16, 2014 8:39 AM To: Will Deacon Cc: stuart.yoder@freescale.com; kvm@vger.kernel.org; open list; iommu@lists.linux-foundation.org; alex.williamson@redhat.com; moderated list:ARM SMMU DRIVER; tech@virtualopensystems.com; kvmarm@lists.cs.columbia.edu; Christoffer Dall Subject: Re: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability IOMMU_CAP_INTR_REMAP On Mon, Jun 16, 2014 at 04:25:26PM +0100, Will Deacon wrote: > Ok, thanks. In which case, I think this is really a combined property > of the SMMU and the interrupt controller, so we might need some extra > code so that the SMMU can check that the interrupt controller for the > device is also capable of interrupt remapping. Right, that this is part of IOMMU code has more or less historic reasons on x86. Interrupt remapping is purely implemented in the IOMMU there, so on ARM some clue-code between interrupt controler and smmu is needed. Joerg
When I say emulating ITS, I mean translating guest ITS commands to physical ITS commands and placing them in physical queue. Regards, Tirumalesh. -----Original Message----- From: kvmarm-bounces@lists.cs.columbia.edu [mailto:kvmarm-bounces@lists.cs.columbia.edu] On Behalf Of Chalamarla, Tirumalesh Sent: Thursday, June 26, 2014 11:08 AM To: Joerg Roedel; Will Deacon Cc: kvm@vger.kernel.org; open list; alex.williamson@redhat.com; stuart.yoder@freescale.com; iommu@lists.linux-foundation.org; tech@virtualopensystems.com; kvmarm@lists.cs.columbia.edu; moderated list:ARM SMMU DRIVER Subject: RE: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability IOMMU_CAP_INTR_REMAP Forgive me if this discussion is not relative here, but I thought it is. How is VFIO restricting devices from writing to MSI/MSI-X, Is all the vector area is mapped by VFIO to trap the accesses. I am asking this because we might need to emulate ITS somewhere either in KVM or VFIO to provide direct access to devices. And I don't see any mentions on that. I think this flag needs to be set by ITS emulation. Regards, Tirumalesh. -----Original Message----- From: kvmarm-bounces@lists.cs.columbia.edu [mailto:kvmarm-bounces@lists.cs.columbia.edu] On Behalf Of Joerg Roedel Sent: Monday, June 16, 2014 8:39 AM To: Will Deacon Cc: stuart.yoder@freescale.com; kvm@vger.kernel.org; open list; iommu@lists.linux-foundation.org; alex.williamson@redhat.com; moderated list:ARM SMMU DRIVER; tech@virtualopensystems.com; kvmarm@lists.cs.columbia.edu; Christoffer Dall Subject: Re: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability IOMMU_CAP_INTR_REMAP On Mon, Jun 16, 2014 at 04:25:26PM +0100, Will Deacon wrote: > Ok, thanks. In which case, I think this is really a combined property > of the SMMU and the interrupt controller, so we might need some extra > code so that the SMMU can check that the interrupt controller for the > device is also capable of interrupt remapping. Right, that this is part of IOMMU code has more or less historic reasons on x86. Interrupt remapping is purely implemented in the IOMMU there, so on ARM some clue-code between interrupt controler and smmu is needed. Joerg
Sorry there was a type,
The question is:
How is VFIO restricting software from writing to MSI/MSI-X vectors of the device.
-----Original Message-----
From: Chalamarla, Tirumalesh
Sent: Thursday, June 26, 2014 11:16 AM
To: Chalamarla, Tirumalesh; Joerg Roedel; Will Deacon
Cc: kvm@vger.kernel.org; open list; alex.williamson@redhat.com; stuart.yoder@freescale.com; iommu@lists.linux-foundation.org; tech@virtualopensystems.com; kvmarm@lists.cs.columbia.edu; moderated list:ARM SMMU DRIVER
Subject: RE: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability IOMMU_CAP_INTR_REMAP
When I say emulating ITS, I mean translating guest ITS commands to physical ITS commands and placing them in physical queue.
Regards,
Tirumalesh.
-----Original Message-----
From: kvmarm-bounces@lists.cs.columbia.edu [mailto:kvmarm-bounces@lists.cs.columbia.edu] On Behalf Of Chalamarla, Tirumalesh
Sent: Thursday, June 26, 2014 11:08 AM
To: Joerg Roedel; Will Deacon
Cc: kvm@vger.kernel.org; open list; alex.williamson@redhat.com; stuart.yoder@freescale.com; iommu@lists.linux-foundation.org; tech@virtualopensystems.com; kvmarm@lists.cs.columbia.edu; moderated list:ARM SMMU DRIVER
Subject: RE: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability IOMMU_CAP_INTR_REMAP
Forgive me if this discussion is not relative here, but I thought it is.
How is VFIO restricting devices from writing to MSI/MSI-X, Is all the vector area is mapped by VFIO to trap the accesses. I am asking this because we might need to emulate ITS somewhere either in KVM or VFIO to provide direct access to devices.
And I don't see any mentions on that. I think this flag needs to be set by ITS emulation.
Regards,
Tirumalesh.
-----Original Message-----
From: kvmarm-bounces@lists.cs.columbia.edu [mailto:kvmarm-bounces@lists.cs.columbia.edu] On Behalf Of Joerg Roedel
Sent: Monday, June 16, 2014 8:39 AM
To: Will Deacon
Cc: stuart.yoder@freescale.com; kvm@vger.kernel.org; open list; iommu@lists.linux-foundation.org; alex.williamson@redhat.com; moderated list:ARM SMMU DRIVER; tech@virtualopensystems.com; kvmarm@lists.cs.columbia.edu; Christoffer Dall
Subject: Re: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability IOMMU_CAP_INTR_REMAP
On Mon, Jun 16, 2014 at 04:25:26PM +0100, Will Deacon wrote:
> Ok, thanks. In which case, I think this is really a combined property
> of the SMMU and the interrupt controller, so we might need some extra
> code so that the SMMU can check that the interrupt controller for the
> device is also capable of interrupt remapping.
Right, that this is part of IOMMU code has more or less historic reasons on x86. Interrupt remapping is purely implemented in the IOMMU there, so on ARM some clue-code between interrupt controler and smmu is needed.
Joerg
On Thu, 2014-06-26 at 18:41 +0000, Chalamarla, Tirumalesh wrote: > Sorry there was a type, > > The question is: > > How is VFIO restricting software from writing to MSI/MSI-X vectors of the device. All interrupts are configured via ioctl, not MSI config space or the MSI-X vector table in MMIO space. VFIO protects the MSI config area by virtualizing it (you can't actually write the physical enable bit or address/data through VFIO). The MSI-X vector table is protected by preventing read, write, or mmap access to it. QEMU provides further virtualization above the basics provided by VFIO. We really can't guarantee that devices don't have backdoors to configure these though. See the realtek quirk in QEMU for an example of a device that has such a backdoor. That's why we require interrupt remapping, so that a device that does this can only hurt the guest, and require the user to opt-out if they feel they have a sufficiently trusted guest. Thanks, Alex > > -----Original Message----- > From: Chalamarla, Tirumalesh > Sent: Thursday, June 26, 2014 11:16 AM > To: Chalamarla, Tirumalesh; Joerg Roedel; Will Deacon > Cc: kvm@vger.kernel.org; open list; alex.williamson@redhat.com; stuart.yoder@freescale.com; iommu@lists.linux-foundation.org; tech@virtualopensystems.com; kvmarm@lists.cs.columbia.edu; moderated list:ARM SMMU DRIVER > Subject: RE: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability IOMMU_CAP_INTR_REMAP > > When I say emulating ITS, I mean translating guest ITS commands to physical ITS commands and placing them in physical queue. > > Regards, > Tirumalesh. > > -----Original Message----- > From: kvmarm-bounces@lists.cs.columbia.edu [mailto:kvmarm-bounces@lists.cs.columbia.edu] On Behalf Of Chalamarla, Tirumalesh > Sent: Thursday, June 26, 2014 11:08 AM > To: Joerg Roedel; Will Deacon > Cc: kvm@vger.kernel.org; open list; alex.williamson@redhat.com; stuart.yoder@freescale.com; iommu@lists.linux-foundation.org; tech@virtualopensystems.com; kvmarm@lists.cs.columbia.edu; moderated list:ARM SMMU DRIVER > Subject: RE: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability IOMMU_CAP_INTR_REMAP > > Forgive me if this discussion is not relative here, but I thought it is. > > How is VFIO restricting devices from writing to MSI/MSI-X, Is all the vector area is mapped by VFIO to trap the accesses. I am asking this because we might need to emulate ITS somewhere either in KVM or VFIO to provide direct access to devices. > And I don't see any mentions on that. I think this flag needs to be set by ITS emulation. > > Regards, > Tirumalesh. > > -----Original Message----- > From: kvmarm-bounces@lists.cs.columbia.edu [mailto:kvmarm-bounces@lists.cs.columbia.edu] On Behalf Of Joerg Roedel > Sent: Monday, June 16, 2014 8:39 AM > To: Will Deacon > Cc: stuart.yoder@freescale.com; kvm@vger.kernel.org; open list; iommu@lists.linux-foundation.org; alex.williamson@redhat.com; moderated list:ARM SMMU DRIVER; tech@virtualopensystems.com; kvmarm@lists.cs.columbia.edu; Christoffer Dall > Subject: Re: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability IOMMU_CAP_INTR_REMAP > > On Mon, Jun 16, 2014 at 04:25:26PM +0100, Will Deacon wrote: > > Ok, thanks. In which case, I think this is really a combined property > > of the SMMU and the interrupt controller, so we might need some extra > > code so that the SMMU can check that the interrupt controller for the > > device is also capable of interrupt remapping. > > Right, that this is part of IOMMU code has more or less historic reasons on x86. Interrupt remapping is purely implemented in the IOMMU there, so on ARM some clue-code between interrupt controler and smmu is needed. > > > Joerg > > > _______________________________________________ > kvmarm mailing list > kvmarm@lists.cs.columbia.edu > https://lists.cs.columbia.edu/mailman/listinfo/kvmarm > _______________________________________________ > kvmarm mailing list > kvmarm@lists.cs.columbia.edu > https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Thanks for the clarification Alex, That’s exactly my point, why are we relying on QEMU or something else to emulate the MSI space when we can directly give access to devices using ITS (of course with a small emulation code).
This way we are also benefited from all ITS services like VCPU migration etc.
What about non QEMU VFIO users, for example, if I wanted to use VFIO to assign a device to a user process I don't need to depend on QEMU. I thought this is one of the main goals of vfio to make it independent of hypervisors.
Thanks,
Tirumalesh.
-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com]
Sent: Thursday, June 26, 2014 12:00 PM
To: Chalamarla, Tirumalesh
Cc: Joerg Roedel; Will Deacon; kvm@vger.kernel.org; open list; stuart.yoder@freescale.com; iommu@lists.linux-foundation.org; tech@virtualopensystems.com; kvmarm@lists.cs.columbia.edu; moderated list:ARM SMMU DRIVER
Subject: Re: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability IOMMU_CAP_INTR_REMAP
On Thu, 2014-06-26 at 18:41 +0000, Chalamarla, Tirumalesh wrote:
> Sorry there was a type,
>
> The question is:
>
> How is VFIO restricting software from writing to MSI/MSI-X vectors of the device.
All interrupts are configured via ioctl, not MSI config space or the MSI-X vector table in MMIO space. VFIO protects the MSI config area by virtualizing it (you can't actually write the physical enable bit or address/data through VFIO). The MSI-X vector table is protected by preventing read, write, or mmap access to it. QEMU provides further virtualization above the basics provided by VFIO. We really can't guarantee that devices don't have backdoors to configure these though.
See the realtek quirk in QEMU for an example of a device that has such a backdoor. That's why we require interrupt remapping, so that a device that does this can only hurt the guest, and require the user to opt-out if they feel they have a sufficiently trusted guest. Thanks,
Alex
>
> -----Original Message-----
> From: Chalamarla, Tirumalesh
> Sent: Thursday, June 26, 2014 11:16 AM
> To: Chalamarla, Tirumalesh; Joerg Roedel; Will Deacon
> Cc: kvm@vger.kernel.org; open list; alex.williamson@redhat.com;
> stuart.yoder@freescale.com; iommu@lists.linux-foundation.org;
> tech@virtualopensystems.com; kvmarm@lists.cs.columbia.edu; moderated
> list:ARM SMMU DRIVER
> Subject: RE: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability
> IOMMU_CAP_INTR_REMAP
>
> When I say emulating ITS, I mean translating guest ITS commands to physical ITS commands and placing them in physical queue.
>
> Regards,
> Tirumalesh.
>
> -----Original Message-----
> From: kvmarm-bounces@lists.cs.columbia.edu
> [mailto:kvmarm-bounces@lists.cs.columbia.edu] On Behalf Of Chalamarla,
> Tirumalesh
> Sent: Thursday, June 26, 2014 11:08 AM
> To: Joerg Roedel; Will Deacon
> Cc: kvm@vger.kernel.org; open list; alex.williamson@redhat.com;
> stuart.yoder@freescale.com; iommu@lists.linux-foundation.org;
> tech@virtualopensystems.com; kvmarm@lists.cs.columbia.edu; moderated
> list:ARM SMMU DRIVER
> Subject: RE: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability
> IOMMU_CAP_INTR_REMAP
>
> Forgive me if this discussion is not relative here, but I thought it is.
>
> How is VFIO restricting devices from writing to MSI/MSI-X, Is all the vector area is mapped by VFIO to trap the accesses. I am asking this because we might need to emulate ITS somewhere either in KVM or VFIO to provide direct access to devices.
> And I don't see any mentions on that. I think this flag needs to be set by ITS emulation.
>
> Regards,
> Tirumalesh.
>
> -----Original Message-----
> From: kvmarm-bounces@lists.cs.columbia.edu
> [mailto:kvmarm-bounces@lists.cs.columbia.edu] On Behalf Of Joerg
> Roedel
> Sent: Monday, June 16, 2014 8:39 AM
> To: Will Deacon
> Cc: stuart.yoder@freescale.com; kvm@vger.kernel.org; open list;
> iommu@lists.linux-foundation.org; alex.williamson@redhat.com;
> moderated list:ARM SMMU DRIVER; tech@virtualopensystems.com;
> kvmarm@lists.cs.columbia.edu; Christoffer Dall
> Subject: Re: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability
> IOMMU_CAP_INTR_REMAP
>
> On Mon, Jun 16, 2014 at 04:25:26PM +0100, Will Deacon wrote:
> > Ok, thanks. In which case, I think this is really a combined
> > property of the SMMU and the interrupt controller, so we might need
> > some extra code so that the SMMU can check that the interrupt
> > controller for the device is also capable of interrupt remapping.
>
> Right, that this is part of IOMMU code has more or less historic reasons on x86. Interrupt remapping is purely implemented in the IOMMU there, so on ARM some clue-code between interrupt controler and smmu is needed.
>
>
> Joerg
>
>
> _______________________________________________
> kvmarm mailing list
> kvmarm@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
> _______________________________________________
> kvmarm mailing list
> kvmarm@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
On Thu, 2014-06-26 at 19:10 +0000, Chalamarla, Tirumalesh wrote: > Thanks for the clarification Alex, That’s exactly my point, why are we relying on QEMU or something else to emulate the MSI space when we can directly give access to devices using ITS (of course with a small emulation code). > This way we are also benefited from all ITS services like VCPU migration etc. I have no idea what ITS is. > What about non QEMU VFIO users, for example, if I wanted to use VFIO to assign a device to a user process I don't need to depend on QEMU. I thought this is one of the main goals of vfio to make it independent of hypervisors. Where did QEMU become a requirement? Maybe I'm missing something in the ARM part of the conversation that got chopped off, but this is exactly why we have the VFIO/QEMU split that we do. VFIO provides basic virtualization for config space and restricts access to other areas that users shouldn't be allowed to change. QEMU is just one example of a userspace VFIO driver. QEMU takes the decomposed device exposed through the VFIO ABI and re-creates a PCI device out of it. VFIO itself has no dependency on QEMU. Thanks, Alex > -----Original Message----- > From: Alex Williamson [mailto:alex.williamson@redhat.com] > Sent: Thursday, June 26, 2014 12:00 PM > To: Chalamarla, Tirumalesh > Cc: Joerg Roedel; Will Deacon; kvm@vger.kernel.org; open list; stuart.yoder@freescale.com; iommu@lists.linux-foundation.org; tech@virtualopensystems.com; kvmarm@lists.cs.columbia.edu; moderated list:ARM SMMU DRIVER > Subject: Re: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability IOMMU_CAP_INTR_REMAP > > On Thu, 2014-06-26 at 18:41 +0000, Chalamarla, Tirumalesh wrote: > > Sorry there was a type, > > > > The question is: > > > > How is VFIO restricting software from writing to MSI/MSI-X vectors of the device. > > All interrupts are configured via ioctl, not MSI config space or the MSI-X vector table in MMIO space. VFIO protects the MSI config area by virtualizing it (you can't actually write the physical enable bit or address/data through VFIO). The MSI-X vector table is protected by preventing read, write, or mmap access to it. QEMU provides further virtualization above the basics provided by VFIO. We really can't guarantee that devices don't have backdoors to configure these though. > See the realtek quirk in QEMU for an example of a device that has such a backdoor. That's why we require interrupt remapping, so that a device that does this can only hurt the guest, and require the user to opt-out if they feel they have a sufficiently trusted guest. Thanks, > > Alex > > > > > -----Original Message----- > > From: Chalamarla, Tirumalesh > > Sent: Thursday, June 26, 2014 11:16 AM > > To: Chalamarla, Tirumalesh; Joerg Roedel; Will Deacon > > Cc: kvm@vger.kernel.org; open list; alex.williamson@redhat.com; > > stuart.yoder@freescale.com; iommu@lists.linux-foundation.org; > > tech@virtualopensystems.com; kvmarm@lists.cs.columbia.edu; moderated > > list:ARM SMMU DRIVER > > Subject: RE: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability > > IOMMU_CAP_INTR_REMAP > > > > When I say emulating ITS, I mean translating guest ITS commands to physical ITS commands and placing them in physical queue. > > > > Regards, > > Tirumalesh. > > > > -----Original Message----- > > From: kvmarm-bounces@lists.cs.columbia.edu > > [mailto:kvmarm-bounces@lists.cs.columbia.edu] On Behalf Of Chalamarla, > > Tirumalesh > > Sent: Thursday, June 26, 2014 11:08 AM > > To: Joerg Roedel; Will Deacon > > Cc: kvm@vger.kernel.org; open list; alex.williamson@redhat.com; > > stuart.yoder@freescale.com; iommu@lists.linux-foundation.org; > > tech@virtualopensystems.com; kvmarm@lists.cs.columbia.edu; moderated > > list:ARM SMMU DRIVER > > Subject: RE: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability > > IOMMU_CAP_INTR_REMAP > > > > Forgive me if this discussion is not relative here, but I thought it is. > > > > How is VFIO restricting devices from writing to MSI/MSI-X, Is all the vector area is mapped by VFIO to trap the accesses. I am asking this because we might need to emulate ITS somewhere either in KVM or VFIO to provide direct access to devices. > > And I don't see any mentions on that. I think this flag needs to be set by ITS emulation. > > > > Regards, > > Tirumalesh. > > > > -----Original Message----- > > From: kvmarm-bounces@lists.cs.columbia.edu > > [mailto:kvmarm-bounces@lists.cs.columbia.edu] On Behalf Of Joerg > > Roedel > > Sent: Monday, June 16, 2014 8:39 AM > > To: Will Deacon > > Cc: stuart.yoder@freescale.com; kvm@vger.kernel.org; open list; > > iommu@lists.linux-foundation.org; alex.williamson@redhat.com; > > moderated list:ARM SMMU DRIVER; tech@virtualopensystems.com; > > kvmarm@lists.cs.columbia.edu; Christoffer Dall > > Subject: Re: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability > > IOMMU_CAP_INTR_REMAP > > > > On Mon, Jun 16, 2014 at 04:25:26PM +0100, Will Deacon wrote: > > > Ok, thanks. In which case, I think this is really a combined > > > property of the SMMU and the interrupt controller, so we might need > > > some extra code so that the SMMU can check that the interrupt > > > controller for the device is also capable of interrupt remapping. > > > > Right, that this is part of IOMMU code has more or less historic reasons on x86. Interrupt remapping is purely implemented in the IOMMU there, so on ARM some clue-code between interrupt controler and smmu is needed. > > > > > > Joerg > > > > > > _______________________________________________ > > kvmarm mailing list > > kvmarm@lists.cs.columbia.edu > > https://lists.cs.columbia.edu/mailman/listinfo/kvmarm > > _______________________________________________ > > kvmarm mailing list > > kvmarm@lists.cs.columbia.edu > > https://lists.cs.columbia.edu/mailman/listinfo/kvmarm > > >
On Thu, Jun 26, 2014 at 08:36:24PM +0100, Alex Williamson wrote: > On Thu, 2014-06-26 at 19:10 +0000, Chalamarla, Tirumalesh wrote: > > Thanks for the clarification Alex, That’s exactly my point, why are we > > relying on QEMU or something else to emulate the MSI space when we can > > directly give access to devices using ITS (of course with a small > > emulation code). This way we are also benefited from all ITS services > > like VCPU migration etc. > > I have no idea what ITS is. ITS is the MSI doorbell for GICv3 (ARM's latest interrupt controller). I agree that we will need an ITS emulation if we want to use MSIs in the guest, and I believe that Marc (CC'd) had already started thinking about that. > > What about non QEMU VFIO users, for example, if I wanted to use VFIO to > > assign a device to a user process I don't need to depend on QEMU. I > > thought this is one of the main goals of vfio to make it independent of > > hypervisors. > > Where did QEMU become a requirement? Maybe I'm missing something in the > ARM part of the conversation that got chopped off, but this is exactly > why we have the VFIO/QEMU split that we do. VFIO provides basic > virtualization for config space and restricts access to other areas that > users shouldn't be allowed to change. QEMU is just one example of a > userspace VFIO driver. QEMU takes the decomposed device exposed through > the VFIO ABI and re-creates a PCI device out of it. VFIO itself has no > dependency on QEMU. Thanks, I also don't understand the QEMU part here. The MSI emulation would be in the kernel, just like the GICv2 emulation that we already have. For userspace drivers, wouldn't you just use eventfd rather than bother with emulating MSIs? Finally, the interrupt remapping part is about the SMMU preventing MSI writes to arbitrary portions of the host address space. The ITS is about routing interrupts to CPUs. Will
Marc,
What is your opinion on ITS emulation . is it should be part of KVM or VFIO.
Also this code needs to depend on ITS host driver a lot, Host ITS driver needs to have an interface for this code to use.
Thanks,
Tirumalesh
-----Original Message-----
From: Will Deacon [mailto:will.deacon@arm.com]
Sent: Friday, June 27, 2014 1:47 AM
To: Alex Williamson
Cc: Chalamarla, Tirumalesh; Joerg Roedel; kvm@vger.kernel.org; open list; stuart.yoder@freescale.com; iommu@lists.linux-foundation.org; tech@virtualopensystems.com; kvmarm@lists.cs.columbia.edu; moderated list:ARM SMMU DRIVER; marc.zyngier@arm.com
Subject: Re: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability IOMMU_CAP_INTR_REMAP
On Thu, Jun 26, 2014 at 08:36:24PM +0100, Alex Williamson wrote:
> On Thu, 2014-06-26 at 19:10 +0000, Chalamarla, Tirumalesh wrote:
> > Thanks for the clarification Alex, That’s exactly my point, why are
> > we relying on QEMU or something else to emulate the MSI space when
> > we can directly give access to devices using ITS (of course with a
> > small emulation code). This way we are also benefited from all ITS
> > services like VCPU migration etc.
>
> I have no idea what ITS is.
ITS is the MSI doorbell for GICv3 (ARM's latest interrupt controller).
I agree that we will need an ITS emulation if we want to use MSIs in the guest, and I believe that Marc (CC'd) had already started thinking about that.
> > What about non QEMU VFIO users, for example, if I wanted to use VFIO to
> > assign a device to a user process I don't need to depend on QEMU. I
> > thought this is one of the main goals of vfio to make it independent of
> > hypervisors.
>
> Where did QEMU become a requirement? Maybe I'm missing something in
> the ARM part of the conversation that got chopped off, but this is
> exactly why we have the VFIO/QEMU split that we do. VFIO provides
> basic virtualization for config space and restricts access to other
> areas that users shouldn't be allowed to change. QEMU is just one
> example of a userspace VFIO driver. QEMU takes the decomposed device
> exposed through the VFIO ABI and re-creates a PCI device out of it.
> VFIO itself has no dependency on QEMU. Thanks,
I also don't understand the QEMU part here. The MSI emulation would be in the kernel, just like the GICv2 emulation that we already have. For userspace drivers, wouldn't you just use eventfd rather than bother with emulating MSIs?
Finally, the interrupt remapping part is about the SMMU preventing MSI writes to arbitrary portions of the host address space. The ITS is about routing interrupts to CPUs.
Will
On Fri, Jun 27 2014 at 10:57:28 PM, "Chalamarla, Tirumalesh" <Tirumalesh.Chalamarla@caviumnetworks.com> wrote: > Marc, > > What is your opinion on ITS emulation . is it should be part > of KVM or VFIO. Making any sort of emulation part of VFIO sounds quite wrong. That's not what VFIO is about, at all. Emulation belongs to the hypervisor, and nowhere else. > Also this code needs to depend on ITS host driver a lot, Host > ITS driver needs to have an interface for this code to use. You can share the command interface as some form of library, but that's about it. There is no more relationship between the ITS driver and the ITS emulation as there is between the GIC driver and its emulation counterpart. M. > Thanks, > Tirumalesh > > -----Original Message----- > From: Will Deacon [mailto:will.deacon@arm.com] > Sent: Friday, June 27, 2014 1:47 AM > To: Alex Williamson > Cc: Chalamarla, Tirumalesh; Joerg Roedel; kvm@vger.kernel.org; open list; stuart.yoder@freescale.com; iommu@lists.linux-foundation.org; tech@virtualopensystems.com; kvmarm@lists.cs.columbia.edu; moderated list:ARM SMMU DRIVER; marc.zyngier@arm.com > Subject: Re: [RFC PATCH v6 04/20] iommu/arm-smmu: add capability IOMMU_CAP_INTR_REMAP > > On Thu, Jun 26, 2014 at 08:36:24PM +0100, Alex Williamson wrote: >> On Thu, 2014-06-26 at 19:10 +0000, Chalamarla, Tirumalesh wrote: >> > Thanks for the clarification Alex, That’s exactly my point, why are >> > we relying on QEMU or something else to emulate the MSI space when >> > we can directly give access to devices using ITS (of course with a >> > small emulation code). This way we are also benefited from all ITS >> > services like VCPU migration etc. >> >> I have no idea what ITS is. > > ITS is the MSI doorbell for GICv3 (ARM's latest interrupt controller). > > I agree that we will need an ITS emulation if we want to use MSIs in >> the guest, and I believe that Marc (CC'd) had already started >> thinking about that. > > >> > What about non QEMU VFIO users, for example, if I wanted to use VFIO to >> > assign a device to a user process I don't need to depend on QEMU. I >> > thought this is one of the main goals of vfio to make it independent of >> > hypervisors. >> >> Where did QEMU become a requirement? Maybe I'm missing something in >> the ARM part of the conversation that got chopped off, but this is >> exactly why we have the VFIO/QEMU split that we do. VFIO provides >> basic virtualization for config space and restricts access to other >> areas that users shouldn't be allowed to change. QEMU is just one >> example of a userspace VFIO driver. QEMU takes the decomposed device >> exposed through the VFIO ABI and re-creates a PCI device out of it. >> VFIO itself has no dependency on QEMU. Thanks, > > I also don't understand the QEMU part here. The MSI emulation would be >> in the kernel, just like the GICv2 emulation that we already >> have. For userspace drivers, wouldn't you just use eventfd rather >> than bother with emulating MSIs? > > Finally, the interrupt remapping part is about the SMMU preventing MSI >> writes to arbitrary portions of the host address space. The ITS is >> about routing interrupts to CPUs. > > Will
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c index 15ab2af..ff29402 100644 --- a/drivers/iommu/arm-smmu.c +++ b/drivers/iommu/arm-smmu.c @@ -1544,7 +1544,7 @@ static int arm_smmu_domain_has_cap(struct iommu_domain *domain, if (smmu_domain->root_cfg.smmu->features & ARM_SMMU_FEAT_COHERENT_WALK) caps |= IOMMU_CAP_CACHE_COHERENCY; - caps |= IOMMU_CAP_NOEXEC; + caps |= IOMMU_CAP_NOEXEC | IOMMU_CAP_INTR_REMAP; return !!(cap & caps); }
With an ARM SMMU, interrupt remapping should always be safe from the SMMU's point of view, as it is properly handled by the GIC. Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com> --- drivers/iommu/arm-smmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)