| Message ID | 20241031085713.6156-1-roger.pau@citrix.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | [v4] x86/io-apic: fix directed EOI when using AMD-Vi interrupt remapping | expand |
On Thu, Oct 31, 2024 at 09:57:13AM +0100, Roger Pau Monne wrote: > When using AMD-Vi interrupt remapping the vector field in the IO-APIC RTE is > repurposed to contain part of the offset into the remapping table. Previous to > 2ca9fbd739b8 Xen had logic so that the offset into the interrupt remapping > table would match the vector. Such logic was mandatory for end of interrupt to > work, since the vector field (even when not containing a vector) is used by the > IO-APIC to find for which pin the EOI must be performed. > > A simple solution wold be to read the IO-APIC RTE each time an EOI is to be > performed, so the raw value of the vector field can be obtained. However > that's likely to perform poorly. Instead introduce a cache to store the > EOI handles when using interrupt remapping, so that the IO-APIC driver can > translate pins into EOI handles without having to read the IO-APIC RTE entry. > Note that to simplify the logic such cache is used unconditionally when > interrupt remapping is enabled, even if strictly it would only be required > for AMD-Vi. > > Reported-by: Willi Junga <xenproject@ymy.be> > Suggested-by: David Woodhouse <dwmw@amazon.co.uk> > Fixes: 2ca9fbd739b8 ('AMD IOMMU: allocate IRTE entries instead of using a static mapping') > Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Very lightly, but Tested-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> with this patch, touchpad works on Framework AMD, _without_ ioapic_ack=new option. > --- > Changes since v3: > - Remove the usage of a sentinel value (again). > - Allocate an initialize the cache in enable_IO_APIC(). > - Fix MISRA compliance. > - Use xvmalloc. > > Changes since v2: > - Restore sentinel value. > > Changes since v1: > - s/apic_pin_eoi/io_apic_pin_eoi/. > - Expand comment about io_apic_pin_eoi usage and layout. > - Use uint8_t instead of unsigned int as array type. > - Do not use a sentinel value. > --- > xen/arch/x86/io_apic.c | 76 +++++++++++++++++++++++++++++++++++++++--- > 1 file changed, 71 insertions(+), 5 deletions(-) > > diff --git a/xen/arch/x86/io_apic.c b/xen/arch/x86/io_apic.c > index e40d2f7dbd75..4e5ee2b890a0 100644 > --- a/xen/arch/x86/io_apic.c > +++ b/xen/arch/x86/io_apic.c > @@ -29,6 +29,7 @@ > #include <xen/acpi.h> > #include <xen/keyhandler.h> > #include <xen/softirq.h> > +#include <xen/xvmalloc.h> > > #include <asm/hpet.h> > #include <asm/mc146818rtc.h> > @@ -71,6 +72,24 @@ static int apic_pin_2_gsi_irq(int apic, int pin); > > static vmask_t *__read_mostly vector_map[MAX_IO_APICS]; > > +/* > + * Store the EOI handle when using interrupt remapping. > + * > + * If using AMD-Vi interrupt remapping the IO-APIC redirection entry remapped > + * format repurposes the vector field to store the offset into the Interrupt > + * Remap table. This breaks directed EOI, as the CPU vector no longer matches > + * the contents of the RTE vector field. Add a translation cache so that > + * directed EOI uses the value in the RTE vector field when interrupt remapping > + * is enabled. > + * > + * Intel VT-d Xen code still stores the CPU vector in the RTE vector field when > + * using the remapped format, but use the translation cache uniformly in order > + * to avoid extra logic to differentiate between VT-d and AMD-Vi. > + * > + * The matrix is accessed as [#io-apic][#pin]. > + */ > +static uint8_t **io_apic_pin_eoi; > + > static void share_vector_maps(unsigned int src, unsigned int dst) > { > unsigned int pin; > @@ -273,6 +292,17 @@ void __ioapic_write_entry( > { > __io_apic_write(apic, 0x11 + 2 * pin, eu.w2); > __io_apic_write(apic, 0x10 + 2 * pin, eu.w1); > + /* > + * Might be called before io_apic_pin_eoi is allocated. Entry will be > + * initialized to the RTE value once the cache is allocated. > + * > + * The vector field is only cached for raw RTE writes when using IR. > + * In that case the vector field might have been repurposed to store > + * something different than the CPU vector, and hence need to be cached > + * for performing EOI. > + */ > + if ( io_apic_pin_eoi ) > + io_apic_pin_eoi[apic][pin] = e.vector; > } > else > iommu_update_ire_from_apic(apic, pin, e.raw); > @@ -288,18 +318,36 @@ static void ioapic_write_entry( > spin_unlock_irqrestore(&ioapic_lock, flags); > } > > -/* EOI an IO-APIC entry. Vector may be -1, indicating that it should be > +/* > + * EOI an IO-APIC entry. Vector may be -1, indicating that it should be > * worked out using the pin. This function expects that the ioapic_lock is > * being held, and interrupts are disabled (or there is a good reason not > * to), and that if both pin and vector are passed, that they refer to the > - * same redirection entry in the IO-APIC. */ > + * same redirection entry in the IO-APIC. > + * > + * If using Interrupt Remapping the vector is always ignored because the RTE > + * remapping format might have repurposed the vector field and a cached value > + * of the EOI handle to use is obtained based on the provided apic and pin > + * values. > + */ > static void __io_apic_eoi(unsigned int apic, unsigned int vector, unsigned int pin) > { > /* Prefer the use of the EOI register if available */ > if ( ioapic_has_eoi_reg(apic) ) > { > - /* If vector is unknown, read it from the IO-APIC */ > - if ( vector == IRQ_VECTOR_UNASSIGNED ) > + if ( io_apic_pin_eoi ) > + /* > + * If the EOI handle is cached use it. When using AMD-Vi IR the CPU > + * vector no longer matches the vector field in the RTE, because > + * the RTE remapping format repurposes the field. > + * > + * The value in the RTE vector field must always be used to signal > + * which RTE to EOI, hence use the cached value which always > + * mirrors the contents of the raw RTE vector field. > + */ > + vector = io_apic_pin_eoi[apic][pin]; > + else if ( vector == IRQ_VECTOR_UNASSIGNED ) > + /* If vector is unknown, read it from the IO-APIC */ > vector = __ioapic_read_entry(apic, pin, true).vector; > > *(IO_APIC_BASE(apic)+16) = vector; > @@ -1317,12 +1365,30 @@ void __init enable_IO_APIC(void) > vector_map[apic] = vector_map[0]; > } > > + if ( iommu_intremap != iommu_intremap_off ) > + { > + io_apic_pin_eoi = xvmalloc_array(typeof(*io_apic_pin_eoi), nr_ioapics); > + BUG_ON(!io_apic_pin_eoi); > + } > + > for(apic = 0; apic < nr_ioapics; apic++) { > int pin; > - /* See if any of the pins is in ExtINT mode */ > + > + if ( io_apic_pin_eoi ) > + { > + io_apic_pin_eoi[apic] = xvmalloc_array(typeof(**io_apic_pin_eoi), > + nr_ioapic_entries[apic]); > + BUG_ON(!io_apic_pin_eoi[apic]); > + } > + > + /* See if any of the pins is in ExtINT mode and cache EOI handle */ > for (pin = 0; pin < nr_ioapic_entries[apic]; pin++) { > struct IO_APIC_route_entry entry = ioapic_read_entry(apic, pin, false); > > + if ( io_apic_pin_eoi ) > + io_apic_pin_eoi[apic][pin] = > + ioapic_read_entry(apic, pin, true).vector; > + > /* If the interrupt line is enabled and in ExtInt mode > * I have found the pin where the i8259 is connected. > */ > -- > 2.46.0 > >
On 31.10.2024 09:57, Roger Pau Monne wrote: > @@ -71,6 +72,24 @@ static int apic_pin_2_gsi_irq(int apic, int pin); > > static vmask_t *__read_mostly vector_map[MAX_IO_APICS]; > > +/* > + * Store the EOI handle when using interrupt remapping. > + * > + * If using AMD-Vi interrupt remapping the IO-APIC redirection entry remapped > + * format repurposes the vector field to store the offset into the Interrupt > + * Remap table. This breaks directed EOI, as the CPU vector no longer matches > + * the contents of the RTE vector field. Add a translation cache so that > + * directed EOI uses the value in the RTE vector field when interrupt remapping > + * is enabled. > + * > + * Intel VT-d Xen code still stores the CPU vector in the RTE vector field when > + * using the remapped format, but use the translation cache uniformly in order > + * to avoid extra logic to differentiate between VT-d and AMD-Vi. > + * > + * The matrix is accessed as [#io-apic][#pin]. > + */ > +static uint8_t **io_apic_pin_eoi; __ro_after_init? > @@ -273,6 +292,17 @@ void __ioapic_write_entry( > { > __io_apic_write(apic, 0x11 + 2 * pin, eu.w2); > __io_apic_write(apic, 0x10 + 2 * pin, eu.w1); > + /* > + * Might be called before io_apic_pin_eoi is allocated. Entry will be > + * initialized to the RTE value once the cache is allocated. With the movement of the allocation to enable_IO_APIC(), isn't this part of the comment stale now? > + * The vector field is only cached for raw RTE writes when using IR. > + * In that case the vector field might have been repurposed to store > + * something different than the CPU vector, and hence need to be cached > + * for performing EOI. > + */ > + if ( io_apic_pin_eoi ) > + io_apic_pin_eoi[apic][pin] = e.vector; The conditional here is necessary anyway, isn't it (for the allocation being conditional itself)? With the adjustments (or clarification of why they cannot be made) Reviewed-by: Jan Beulich <jbeulich@suse.com> If the adjustments can be confirmed I'd also be happy to make them while committing, to save another round-trip. Jan
On Mon, Nov 04, 2024 at 10:56:36AM +0100, Jan Beulich wrote: > On 31.10.2024 09:57, Roger Pau Monne wrote: > > @@ -71,6 +72,24 @@ static int apic_pin_2_gsi_irq(int apic, int pin); > > > > static vmask_t *__read_mostly vector_map[MAX_IO_APICS]; > > > > +/* > > + * Store the EOI handle when using interrupt remapping. > > + * > > + * If using AMD-Vi interrupt remapping the IO-APIC redirection entry remapped > > + * format repurposes the vector field to store the offset into the Interrupt > > + * Remap table. This breaks directed EOI, as the CPU vector no longer matches > > + * the contents of the RTE vector field. Add a translation cache so that > > + * directed EOI uses the value in the RTE vector field when interrupt remapping > > + * is enabled. > > + * > > + * Intel VT-d Xen code still stores the CPU vector in the RTE vector field when > > + * using the remapped format, but use the translation cache uniformly in order > > + * to avoid extra logic to differentiate between VT-d and AMD-Vi. > > + * > > + * The matrix is accessed as [#io-apic][#pin]. > > + */ > > +static uint8_t **io_apic_pin_eoi; > > __ro_after_init? Oh, yes indeed, allocations are static after init. > > @@ -273,6 +292,17 @@ void __ioapic_write_entry( > > { > > __io_apic_write(apic, 0x11 + 2 * pin, eu.w2); > > __io_apic_write(apic, 0x10 + 2 * pin, eu.w1); > > + /* > > + * Might be called before io_apic_pin_eoi is allocated. Entry will be > > + * initialized to the RTE value once the cache is allocated. > > With the movement of the allocation to enable_IO_APIC(), isn't this part of > the comment stale now? There are still paths that call __ioapic_write_entry() before enable_IO_APIC(). See for example how x2apic_bsp_setup() makes use of save_IO_APIC_setup() ahead of enable_IO_APIC(). > > + * The vector field is only cached for raw RTE writes when using IR. > > + * In that case the vector field might have been repurposed to store > > + * something different than the CPU vector, and hence need to be cached > > + * for performing EOI. > > + */ > > + if ( io_apic_pin_eoi ) > > + io_apic_pin_eoi[apic][pin] = e.vector; > > The conditional here is necessary anyway, isn't it (for the allocation > being conditional itself)? Indeed, the matrix won't be allocated if interrupt remapping is not enabled. > With the adjustments (or clarification of why they cannot be made) > Reviewed-by: Jan Beulich <jbeulich@suse.com> Thanks. > If the adjustments can be confirmed I'd also be happy to make them while > committing, to save another round-trip. I agree with the __ro_after_init, see my reply to the code comment. Regards, Roger.
diff --git a/xen/arch/x86/io_apic.c b/xen/arch/x86/io_apic.c index e40d2f7dbd75..4e5ee2b890a0 100644 --- a/xen/arch/x86/io_apic.c +++ b/xen/arch/x86/io_apic.c @@ -29,6 +29,7 @@ #include <xen/acpi.h> #include <xen/keyhandler.h> #include <xen/softirq.h> +#include <xen/xvmalloc.h> #include <asm/hpet.h> #include <asm/mc146818rtc.h> @@ -71,6 +72,24 @@ static int apic_pin_2_gsi_irq(int apic, int pin); static vmask_t *__read_mostly vector_map[MAX_IO_APICS]; +/* + * Store the EOI handle when using interrupt remapping. + * + * If using AMD-Vi interrupt remapping the IO-APIC redirection entry remapped + * format repurposes the vector field to store the offset into the Interrupt + * Remap table. This breaks directed EOI, as the CPU vector no longer matches + * the contents of the RTE vector field. Add a translation cache so that + * directed EOI uses the value in the RTE vector field when interrupt remapping + * is enabled. + * + * Intel VT-d Xen code still stores the CPU vector in the RTE vector field when + * using the remapped format, but use the translation cache uniformly in order + * to avoid extra logic to differentiate between VT-d and AMD-Vi. + * + * The matrix is accessed as [#io-apic][#pin]. + */ +static uint8_t **io_apic_pin_eoi; + static void share_vector_maps(unsigned int src, unsigned int dst) { unsigned int pin; @@ -273,6 +292,17 @@ void __ioapic_write_entry( { __io_apic_write(apic, 0x11 + 2 * pin, eu.w2); __io_apic_write(apic, 0x10 + 2 * pin, eu.w1); + /* + * Might be called before io_apic_pin_eoi is allocated. Entry will be + * initialized to the RTE value once the cache is allocated. + * + * The vector field is only cached for raw RTE writes when using IR. + * In that case the vector field might have been repurposed to store + * something different than the CPU vector, and hence need to be cached + * for performing EOI. + */ + if ( io_apic_pin_eoi ) + io_apic_pin_eoi[apic][pin] = e.vector; } else iommu_update_ire_from_apic(apic, pin, e.raw); @@ -288,18 +318,36 @@ static void ioapic_write_entry( spin_unlock_irqrestore(&ioapic_lock, flags); } -/* EOI an IO-APIC entry. Vector may be -1, indicating that it should be +/* + * EOI an IO-APIC entry. Vector may be -1, indicating that it should be * worked out using the pin. This function expects that the ioapic_lock is * being held, and interrupts are disabled (or there is a good reason not * to), and that if both pin and vector are passed, that they refer to the - * same redirection entry in the IO-APIC. */ + * same redirection entry in the IO-APIC. + * + * If using Interrupt Remapping the vector is always ignored because the RTE + * remapping format might have repurposed the vector field and a cached value + * of the EOI handle to use is obtained based on the provided apic and pin + * values. + */ static void __io_apic_eoi(unsigned int apic, unsigned int vector, unsigned int pin) { /* Prefer the use of the EOI register if available */ if ( ioapic_has_eoi_reg(apic) ) { - /* If vector is unknown, read it from the IO-APIC */ - if ( vector == IRQ_VECTOR_UNASSIGNED ) + if ( io_apic_pin_eoi ) + /* + * If the EOI handle is cached use it. When using AMD-Vi IR the CPU + * vector no longer matches the vector field in the RTE, because + * the RTE remapping format repurposes the field. + * + * The value in the RTE vector field must always be used to signal + * which RTE to EOI, hence use the cached value which always + * mirrors the contents of the raw RTE vector field. + */ + vector = io_apic_pin_eoi[apic][pin]; + else if ( vector == IRQ_VECTOR_UNASSIGNED ) + /* If vector is unknown, read it from the IO-APIC */ vector = __ioapic_read_entry(apic, pin, true).vector; *(IO_APIC_BASE(apic)+16) = vector; @@ -1317,12 +1365,30 @@ void __init enable_IO_APIC(void) vector_map[apic] = vector_map[0]; } + if ( iommu_intremap != iommu_intremap_off ) + { + io_apic_pin_eoi = xvmalloc_array(typeof(*io_apic_pin_eoi), nr_ioapics); + BUG_ON(!io_apic_pin_eoi); + } + for(apic = 0; apic < nr_ioapics; apic++) { int pin; - /* See if any of the pins is in ExtINT mode */ + + if ( io_apic_pin_eoi ) + { + io_apic_pin_eoi[apic] = xvmalloc_array(typeof(**io_apic_pin_eoi), + nr_ioapic_entries[apic]); + BUG_ON(!io_apic_pin_eoi[apic]); + } + + /* See if any of the pins is in ExtINT mode and cache EOI handle */ for (pin = 0; pin < nr_ioapic_entries[apic]; pin++) { struct IO_APIC_route_entry entry = ioapic_read_entry(apic, pin, false); + if ( io_apic_pin_eoi ) + io_apic_pin_eoi[apic][pin] = + ioapic_read_entry(apic, pin, true).vector; + /* If the interrupt line is enabled and in ExtInt mode * I have found the pin where the i8259 is connected. */
When using AMD-Vi interrupt remapping the vector field in the IO-APIC RTE is repurposed to contain part of the offset into the remapping table. Previous to 2ca9fbd739b8 Xen had logic so that the offset into the interrupt remapping table would match the vector. Such logic was mandatory for end of interrupt to work, since the vector field (even when not containing a vector) is used by the IO-APIC to find for which pin the EOI must be performed. A simple solution wold be to read the IO-APIC RTE each time an EOI is to be performed, so the raw value of the vector field can be obtained. However that's likely to perform poorly. Instead introduce a cache to store the EOI handles when using interrupt remapping, so that the IO-APIC driver can translate pins into EOI handles without having to read the IO-APIC RTE entry. Note that to simplify the logic such cache is used unconditionally when interrupt remapping is enabled, even if strictly it would only be required for AMD-Vi. Reported-by: Willi Junga <xenproject@ymy.be> Suggested-by: David Woodhouse <dwmw@amazon.co.uk> Fixes: 2ca9fbd739b8 ('AMD IOMMU: allocate IRTE entries instead of using a static mapping') Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> --- Changes since v3: - Remove the usage of a sentinel value (again). - Allocate an initialize the cache in enable_IO_APIC(). - Fix MISRA compliance. - Use xvmalloc. Changes since v2: - Restore sentinel value. Changes since v1: - s/apic_pin_eoi/io_apic_pin_eoi/. - Expand comment about io_apic_pin_eoi usage and layout. - Use uint8_t instead of unsigned int as array type. - Do not use a sentinel value. --- xen/arch/x86/io_apic.c | 76 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 71 insertions(+), 5 deletions(-)