| Message ID | 20190905140018.5139-5-jonnyc@amazon.com (mailing list archive) |
|---|---|
| State | Superseded, archived |
| Headers | show |
| Series | Amazon's Annapurna Labs DT-based PCIe host controller driver | expand |
s/Add quirk to disable/Disable/ in subject On Thu, Sep 05, 2019 at 05:00:18PM +0300, Jonathan Chocron wrote: > The Root Port (identified by [1c36:0031]) doesn't support MSI-X. On some > platforms it is configured to not advertise the capability at all, while > on others it (mistakenly) does. This causes a panic during > initialization by the pcieport driver, since it tries to configure the > MSI-X capability. Specifically, when trying to access the MSI-X table > a "non-existing addr" exception occurs. MSI-X configuration is all in memory space (not config space), so I guess this device has a BAR (maybe more) and the MSI-X table access would be to something in that BAR? Or is there junk in the capability so we try to access something completely outside the BAR? [I wonder if it'd be worth adding some validation to make sure the Table and PBA areas are contained in the BAR?] The reason I'm curious about all this is because on the PCI side this is probably an Unsupported Request or similar error, and on *most* platforms, this does not cause a panic. If it was a read, the read usually gets ~0 data, and writes are dropped. So my concern is that we'll avoid this panic by disabling MSI, but we'll see other panics in other places. > Example stacktrace snippet: [Remove the timestamps since they're not really relevant and indent the quote a couple spaces] > [ 1.632363] SError Interrupt on CPU2, code 0xbf000000 -- SError > [ 1.632364] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-Jonny-14847-ge76f1d4a1828-dirty #33 > [ 1.632365] Hardware name: Annapurna Labs Alpine V3 EVP (DT) > [ 1.632365] pstate: 80000005 (Nzcv daif -PAN -UAO) > [ 1.632366] pc : __pci_enable_msix_range+0x4e4/0x608 > [ 1.632367] lr : __pci_enable_msix_range+0x498/0x608 > [ 1.632367] sp : ffffff80117db700 > [ 1.632368] x29: ffffff80117db700 x28: 0000000000000001 > [ 1.632370] x27: 0000000000000001 x26: 0000000000000000 > [ 1.632372] x25: ffffffd3e9d8c0b0 x24: 0000000000000000 > [ 1.632373] x23: 0000000000000000 x22: 0000000000000000 > [ 1.632375] x21: 0000000000000001 x20: 0000000000000000 > [ 1.632376] x19: ffffffd3e9d8c000 x18: ffffffffffffffff > [ 1.632378] x17: 0000000000000000 x16: 0000000000000000 > [ 1.632379] x15: ffffff80116496c8 x14: ffffffd3e9844503 > [ 1.632380] x13: ffffffd3e9844502 x12: 0000000000000038 > [ 1.632382] x11: ffffffffffffff00 x10: 0000000000000040 > [ 1.632384] x9 : ffffff801165e270 x8 : ffffff801165e268 > [ 1.632385] x7 : 0000000000000002 x6 : 00000000000000b2 > [ 1.632387] x5 : ffffffd3e9d8c2c0 x4 : 0000000000000000 > [ 1.632388] x3 : 0000000000000000 x2 : 0000000000000000 > [ 1.632390] x1 : 0000000000000000 x0 : ffffffd3e9844680 > [ 1.632392] Kernel panic - not syncing: Asynchronous SError Interrupt > [ 1.632393] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-Jonny-14847-ge76f1d4a1828-dirty #33 > [ 1.632394] Hardware name: Annapurna Labs Alpine V3 EVP (DT) > [ 1.632394] Call trace: > [ 1.632395] dump_backtrace+0x0/0x140 > [ 1.632395] show_stack+0x14/0x20 > [ 1.632396] dump_stack+0xa8/0xcc > [ 1.632396] panic+0x140/0x334 > [ 1.632397] nmi_panic+0x6c/0x70 > [ 1.632398] arm64_serror_panic+0x74/0x88 > [ 1.632398] __pte_error+0x0/0x28 > [ 1.632399] el1_error+0x84/0xf8 > [ 1.632400] __pci_enable_msix_range+0x4e4/0x608 > [ 1.632400] pci_alloc_irq_vectors_affinity+0xdc/0x150 > [ 1.632401] pcie_port_device_register+0x2b8/0x4e0 > [ 1.632402] pcie_portdrv_probe+0x34/0xf0 > > Notice that this quirk also disables MSI (which may work, but hasn't > been tested nor has a current use case), since currently there is no > standard way to disable only MSI-X. > > Signed-off-by: Jonathan Chocron <jonnyc@amazon.com> > Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com> > Reviewed-by: Andrew Murray <andrew.murray@arm.com> > --- > drivers/pci/quirks.c | 18 ++++++++++++++++++ > 1 file changed, 18 insertions(+) > > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c > index 8fe765592943..5a8ea5fdeae7 100644 > --- a/drivers/pci/quirks.c > +++ b/drivers/pci/quirks.c > @@ -2977,6 +2977,24 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0x10a1, > quirk_msi_intx_disable_qca_bug); > DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0xe091, > quirk_msi_intx_disable_qca_bug); > + > +/* > + * Amazon's Annapurna Labs 1c36:0031 Root Ports don't support MSI-X, so it > + * should be disabled on platforms where the device (mistakenly) advertises it. > + * > + * Notice that this quirk also disables MSI (which may work, but hasn't been > + * tested), since currently there is no standard way to disable only MSI-X. > + * > + * The 0031 device id is reused for other non Root Port device types, > + * therefore the quirk is registered for the PCI_CLASS_BRIDGE_PCI class. > + */ > +static void quirk_al_msi_disable(struct pci_dev *dev) > +{ > + dev->no_msi = 1; > + pci_warn(dev, "Disabling MSI/MSI-X\n"); > +} > +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031, > + PCI_CLASS_BRIDGE_PCI, 8, quirk_al_msi_disable); > #endif /* CONFIG_PCI_MSI */ > > /* > -- > 2.17.1 >
On Sat, Sep 07, 2019 at 11:55:42AM -0500, Bjorn Helgaas wrote: > s/Add quirk to disable/Disable/ in subject > > On Thu, Sep 05, 2019 at 05:00:18PM +0300, Jonathan Chocron wrote: > > The Root Port (identified by [1c36:0031]) doesn't support MSI-X. On some > > platforms it is configured to not advertise the capability at all, while > > on others it (mistakenly) does. This causes a panic during > > initialization by the pcieport driver, since it tries to configure the > > MSI-X capability. Specifically, when trying to access the MSI-X table > > a "non-existing addr" exception occurs. > > MSI-X configuration is all in memory space (not config space), so I > guess this device has a BAR (maybe more) and the MSI-X table access > would be to something in that BAR? Or is there junk in the capability > so we try to access something completely outside the BAR? [I wonder if > it'd be worth adding some validation to make sure the Table and PBA > areas are contained in the BAR?] > > The reason I'm curious about all this is because on the PCI side this > is probably an Unsupported Request or similar error, and on *most* > platforms, this does not cause a panic. If it was a read, the read > usually gets ~0 data, and writes are dropped. > > So my concern is that we'll avoid this panic by disabling MSI, but > we'll see other panics in other places. > > > Example stacktrace snippet: > > [Remove the timestamps since they're not really relevant and indent > the quote a couple spaces] Jonathan, can you please clarify and update the patch so that we can proceed with the series and try to get it into v5.4 ? Thanks, Lorenzo > > > [ 1.632363] SError Interrupt on CPU2, code 0xbf000000 -- SError > > [ 1.632364] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-Jonny-14847-ge76f1d4a1828-dirty #33 > > [ 1.632365] Hardware name: Annapurna Labs Alpine V3 EVP (DT) > > [ 1.632365] pstate: 80000005 (Nzcv daif -PAN -UAO) > > [ 1.632366] pc : __pci_enable_msix_range+0x4e4/0x608 > > [ 1.632367] lr : __pci_enable_msix_range+0x498/0x608 > > [ 1.632367] sp : ffffff80117db700 > > [ 1.632368] x29: ffffff80117db700 x28: 0000000000000001 > > [ 1.632370] x27: 0000000000000001 x26: 0000000000000000 > > [ 1.632372] x25: ffffffd3e9d8c0b0 x24: 0000000000000000 > > [ 1.632373] x23: 0000000000000000 x22: 0000000000000000 > > [ 1.632375] x21: 0000000000000001 x20: 0000000000000000 > > [ 1.632376] x19: ffffffd3e9d8c000 x18: ffffffffffffffff > > [ 1.632378] x17: 0000000000000000 x16: 0000000000000000 > > [ 1.632379] x15: ffffff80116496c8 x14: ffffffd3e9844503 > > [ 1.632380] x13: ffffffd3e9844502 x12: 0000000000000038 > > [ 1.632382] x11: ffffffffffffff00 x10: 0000000000000040 > > [ 1.632384] x9 : ffffff801165e270 x8 : ffffff801165e268 > > [ 1.632385] x7 : 0000000000000002 x6 : 00000000000000b2 > > [ 1.632387] x5 : ffffffd3e9d8c2c0 x4 : 0000000000000000 > > [ 1.632388] x3 : 0000000000000000 x2 : 0000000000000000 > > [ 1.632390] x1 : 0000000000000000 x0 : ffffffd3e9844680 > > [ 1.632392] Kernel panic - not syncing: Asynchronous SError Interrupt > > [ 1.632393] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-Jonny-14847-ge76f1d4a1828-dirty #33 > > [ 1.632394] Hardware name: Annapurna Labs Alpine V3 EVP (DT) > > [ 1.632394] Call trace: > > [ 1.632395] dump_backtrace+0x0/0x140 > > [ 1.632395] show_stack+0x14/0x20 > > [ 1.632396] dump_stack+0xa8/0xcc > > [ 1.632396] panic+0x140/0x334 > > [ 1.632397] nmi_panic+0x6c/0x70 > > [ 1.632398] arm64_serror_panic+0x74/0x88 > > [ 1.632398] __pte_error+0x0/0x28 > > [ 1.632399] el1_error+0x84/0xf8 > > [ 1.632400] __pci_enable_msix_range+0x4e4/0x608 > > [ 1.632400] pci_alloc_irq_vectors_affinity+0xdc/0x150 > > [ 1.632401] pcie_port_device_register+0x2b8/0x4e0 > > [ 1.632402] pcie_portdrv_probe+0x34/0xf0 > > > > Notice that this quirk also disables MSI (which may work, but hasn't > > been tested nor has a current use case), since currently there is no > > standard way to disable only MSI-X. > > > > Signed-off-by: Jonathan Chocron <jonnyc@amazon.com> > > Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com> > > Reviewed-by: Andrew Murray <andrew.murray@arm.com> > > --- > > drivers/pci/quirks.c | 18 ++++++++++++++++++ > > 1 file changed, 18 insertions(+) > > > > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c > > index 8fe765592943..5a8ea5fdeae7 100644 > > --- a/drivers/pci/quirks.c > > +++ b/drivers/pci/quirks.c > > @@ -2977,6 +2977,24 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0x10a1, > > quirk_msi_intx_disable_qca_bug); > > DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0xe091, > > quirk_msi_intx_disable_qca_bug); > > + > > +/* > > + * Amazon's Annapurna Labs 1c36:0031 Root Ports don't support MSI-X, so it > > + * should be disabled on platforms where the device (mistakenly) advertises it. > > + * > > + * Notice that this quirk also disables MSI (which may work, but hasn't been > > + * tested), since currently there is no standard way to disable only MSI-X. > > + * > > + * The 0031 device id is reused for other non Root Port device types, > > + * therefore the quirk is registered for the PCI_CLASS_BRIDGE_PCI class. > > + */ > > +static void quirk_al_msi_disable(struct pci_dev *dev) > > +{ > > + dev->no_msi = 1; > > + pci_warn(dev, "Disabling MSI/MSI-X\n"); > > +} > > +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031, > > + PCI_CLASS_BRIDGE_PCI, 8, quirk_al_msi_disable); > > #endif /* CONFIG_PCI_MSI */ > > > > /* > > -- > > 2.17.1 > >
On Sat, 2019-09-07 at 11:55 -0500, Bjorn Helgaas wrote: > s/Add quirk to disable/Disable/ in subject > > On Thu, Sep 05, 2019 at 05:00:18PM +0300, Jonathan Chocron wrote: > > The Root Port (identified by [1c36:0031]) doesn't support MSI-X. On > > some > > platforms it is configured to not advertise the capability at all, > > while > > on others it (mistakenly) does. This causes a panic during > > initialization by the pcieport driver, since it tries to configure > > the > > MSI-X capability. Specifically, when trying to access the MSI-X > > table > > a "non-existing addr" exception occurs. > > MSI-X configuration is all in memory space (not config space), so I > guess this device has a BAR (maybe more) and the MSI-X table access > would be to something in that BAR? Or is there junk in the > capability > so we try to access something completely outside the BAR? [I wonder > if > it'd be worth adding some validation to make sure the Table and PBA > areas are contained in the BAR?] > There is no actual memory allocated for the MSI-X table behind the BAR, so it attempts to access non existing regs, resulting in the SError. > The reason I'm curious about all this is because on the PCI side this > is probably an Unsupported Request or similar error, and on *most* > platforms, this does not cause a panic. If it was a read, the read > usually gets ~0 data, and writes are dropped. > I'm not entirely sure how this is implemented in the HW (i.e. accessing the Host controller's memory space), but I assume that an AXI error returns, which translates to the Serror. I can verify with our HW engineers if you think it is blocking. > So my concern is that we'll avoid this panic by disabling MSI, but > we'll see other panics in other places. > > > Example stacktrace snippet: > > [Remove the timestamps since they're not really relevant and indent > the quote a couple spaces] > Ack. > > [ 1.632363] SError Interrupt on CPU2, code 0xbf000000 -- SError > > [ 1.632364] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1- > > Jonny-14847-ge76f1d4a1828-dirty #33 > > [ 1.632365] Hardware name: Annapurna Labs Alpine V3 EVP (DT) > > [ 1.632365] pstate: 80000005 (Nzcv daif -PAN -UAO) > > [ 1.632366] pc : __pci_enable_msix_range+0x4e4/0x608 > > [ 1.632367] lr : __pci_enable_msix_range+0x498/0x608 > > [ 1.632367] sp : ffffff80117db700 > > [ 1.632368] x29: ffffff80117db700 x28: 0000000000000001 > > [ 1.632370] x27: 0000000000000001 x26: 0000000000000000 > > [ 1.632372] x25: ffffffd3e9d8c0b0 x24: 0000000000000000 > > [ 1.632373] x23: 0000000000000000 x22: 0000000000000000 > > [ 1.632375] x21: 0000000000000001 x20: 0000000000000000 > > [ 1.632376] x19: ffffffd3e9d8c000 x18: ffffffffffffffff > > [ 1.632378] x17: 0000000000000000 x16: 0000000000000000 > > [ 1.632379] x15: ffffff80116496c8 x14: ffffffd3e9844503 > > [ 1.632380] x13: ffffffd3e9844502 x12: 0000000000000038 > > [ 1.632382] x11: ffffffffffffff00 x10: 0000000000000040 > > [ 1.632384] x9 : ffffff801165e270 x8 : ffffff801165e268 > > [ 1.632385] x7 : 0000000000000002 x6 : 00000000000000b2 > > [ 1.632387] x5 : ffffffd3e9d8c2c0 x4 : 0000000000000000 > > [ 1.632388] x3 : 0000000000000000 x2 : 0000000000000000 > > [ 1.632390] x1 : 0000000000000000 x0 : ffffffd3e9844680 > > [ 1.632392] Kernel panic - not syncing: Asynchronous SError > > Interrupt > > [ 1.632393] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1- > > Jonny-14847-ge76f1d4a1828-dirty #33 > > [ 1.632394] Hardware name: Annapurna Labs Alpine V3 EVP (DT) > > [ 1.632394] Call trace: > > [ 1.632395] dump_backtrace+0x0/0x140 > > [ 1.632395] show_stack+0x14/0x20 > > [ 1.632396] dump_stack+0xa8/0xcc > > [ 1.632396] panic+0x140/0x334 > > [ 1.632397] nmi_panic+0x6c/0x70 > > [ 1.632398] arm64_serror_panic+0x74/0x88 > > [ 1.632398] __pte_error+0x0/0x28 > > [ 1.632399] el1_error+0x84/0xf8 > > [ 1.632400] __pci_enable_msix_range+0x4e4/0x608 > > [ 1.632400] pci_alloc_irq_vectors_affinity+0xdc/0x150 > > [ 1.632401] pcie_port_device_register+0x2b8/0x4e0 > > [ 1.632402] pcie_portdrv_probe+0x34/0xf0 > > > > Notice that this quirk also disables MSI (which may work, but > > hasn't > > been tested nor has a current use case), since currently there is > > no > > standard way to disable only MSI-X. > > > > Signed-off-by: Jonathan Chocron <jonnyc@amazon.com> > > Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com> > > Reviewed-by: Andrew Murray <andrew.murray@arm.com> > > --- > > drivers/pci/quirks.c | 18 ++++++++++++++++++ > > 1 file changed, 18 insertions(+) > > > > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c > > index 8fe765592943..5a8ea5fdeae7 100644 > > --- a/drivers/pci/quirks.c > > +++ b/drivers/pci/quirks.c > > @@ -2977,6 +2977,24 @@ > > DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0x10a1, > > quirk_msi_intx_disable_qca_bug); > > DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0xe091, > > quirk_msi_intx_disable_qca_bug); > > + > > +/* > > + * Amazon's Annapurna Labs 1c36:0031 Root Ports don't support MSI- > > X, so it > > + * should be disabled on platforms where the device (mistakenly) > > advertises it. > > + * > > + * Notice that this quirk also disables MSI (which may work, but > > hasn't been > > + * tested), since currently there is no standard way to disable > > only MSI-X. > > + * > > + * The 0031 device id is reused for other non Root Port device > > types, > > + * therefore the quirk is registered for the PCI_CLASS_BRIDGE_PCI > > class. > > + */ > > +static void quirk_al_msi_disable(struct pci_dev *dev) > > +{ > > + dev->no_msi = 1; > > + pci_warn(dev, "Disabling MSI/MSI-X\n"); > > +} > > +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, > > 0x0031, > > + PCI_CLASS_BRIDGE_PCI, 8, > > quirk_al_msi_disable); > > #endif /* CONFIG_PCI_MSI */ > > > > /* > > -- > > 2.17.1 > >
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 8fe765592943..5a8ea5fdeae7 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -2977,6 +2977,24 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0x10a1, quirk_msi_intx_disable_qca_bug); DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0xe091, quirk_msi_intx_disable_qca_bug); + +/* + * Amazon's Annapurna Labs 1c36:0031 Root Ports don't support MSI-X, so it + * should be disabled on platforms where the device (mistakenly) advertises it. + * + * Notice that this quirk also disables MSI (which may work, but hasn't been + * tested), since currently there is no standard way to disable only MSI-X. + * + * The 0031 device id is reused for other non Root Port device types, + * therefore the quirk is registered for the PCI_CLASS_BRIDGE_PCI class. + */ +static void quirk_al_msi_disable(struct pci_dev *dev) +{ + dev->no_msi = 1; + pci_warn(dev, "Disabling MSI/MSI-X\n"); +} +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031, + PCI_CLASS_BRIDGE_PCI, 8, quirk_al_msi_disable); #endif /* CONFIG_PCI_MSI */ /*