| Message ID | 20161117233031.2003.43054.stgit@bhelgaas-glaptop.roam.corp.google.com (mailing list archive) |
|---|---|
| State | New, archived |
| Delegated to: | Bjorn Helgaas |
| Headers | show |
On Thu, Nov 17, 2016 at 05:30:31PM -0600, Bjorn Helgaas wrote: >From: Noa Osherovich <noaos@mellanox.com> > >Mellanox devices were marked as having INTx masking ability broken. As a >result, the VFIO driver fails to start when more than one device function >is passed-through to a VM if both have the same INTx pin. > >Prior to Connect-IB, Mellanox devices exposed to the operating system one >PCI function per all ports. Starting from Connect-IB, the devices are >function-per-port. When passing the second function to a VM, VFIO will >fail to start. > >Exclude ConnectX-4, ConnectX4-Lx and Connect-IB from the list of Mellanox >devices marked as having broken INTx masking: > >- ConnectX-4 and ConnectX4-LX firmware version is checked. If INTx > masking is supported, we unmark the broken INTx masking. >- Connect-IB does not support INTx currently so will not cause any > problem. > >[bhelgaas: call pci_disable_device() always, after iounmap()] >Fixes: 11e42532ada3 ("PCI: Assume all Mellanox devices have broken INTx masking") >Signed-off-by: Noa Osherovich <noaos@mellanox.com> >Signed-off-by: Bjorn Helgaas <bhelgaas@google.com> >Reviewed-by: Or Gerlitz <ogerlitz@mellanox.com> >Reviewed-by: Gavin Shan <gwshan@linux.vnet.ibm.com> >--- > drivers/pci/quirks.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++--- > 1 file changed, 54 insertions(+), 3 deletions(-) > >diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c >index f0515b7..496ca13 100644 >--- a/drivers/pci/quirks.c >+++ b/drivers/pci/quirks.c >@@ -3209,13 +3209,25 @@ static u16 mellanox_broken_intx_devs[] = { > PCI_DEVICE_ID_MELLANOX_CONNECTX2, > PCI_DEVICE_ID_MELLANOX_CONNECTX3, > PCI_DEVICE_ID_MELLANOX_CONNECTX3_PRO, >- PCI_DEVICE_ID_MELLANOX_CONNECTIB, >- PCI_DEVICE_ID_MELLANOX_CONNECTX4, >- PCI_DEVICE_ID_MELLANOX_CONNECTX4_LX, > }; > >+#define CONNECTX_4_CURR_MAX_MINOR 99 >+#define CONNECTX_4_INTX_SUPPORT_MINOR 14 >+ >+/* >+ * Check ConnectX-4/LX FW version to see if it supports legacy interrupts. >+ * If so, don't mark it as broken. >+ * FW minor > 99 means older FW version format and no INTx masking support. >+ * FW minor < 14 means new FW version format and no INTx masking support. >+ */ > static void mellanox_check_broken_intx_masking(struct pci_dev *pdev) > { >+ __be32 __iomem *fw_ver; >+ u16 fw_major; >+ u16 fw_minor; >+ u16 fw_subminor; >+ u32 fw_maj_min; >+ u32 fw_sub_min; > int i; > > for (i = 0; i < ARRAY_SIZE(mellanox_broken_intx_devs); i++) { >@@ -3224,6 +3236,45 @@ static void mellanox_check_broken_intx_masking(struct pci_dev *pdev) > return; > } > } >+ >+ /* Getting here means Connect-IB cards and up. Connect-IB has no INTx >+ * support so shouldn't be checked further >+ */ >+ if (pdev->device == PCI_DEVICE_ID_MELLANOX_CONNECTIB) >+ return; >+ >+ if (pdev->device != PCI_DEVICE_ID_MELLANOX_CONNECTX4 && >+ pdev->device != PCI_DEVICE_ID_MELLANOX_CONNECTX4_LX) >+ return; >+ >+ /* For ConnectX-4 and ConnectX-4LX, need to check FW support */ >+ if (pci_enable_device_mem(pdev)) { >+ dev_warn(&pdev->dev, "Can't enable device memory\n"); >+ return; >+ } >+ >+ fw_ver = ioremap(pci_resource_start(pdev, 0), 4); >+ if (!fw_ver) { >+ dev_warn(&pdev->dev, "Can't map ConnectX-4 initialization segment\n"); Bjorn, pci_disable_device() is still missed here. Could you please amend it? >+ return; >+ } >+ >+ /* Reading from resource space should be 32b aligned */ >+ fw_maj_min = ioread32be(fw_ver); >+ fw_sub_min = ioread32be(fw_ver + 1); >+ fw_major = fw_maj_min & 0xffff; >+ fw_minor = fw_maj_min >> 16; >+ fw_subminor = fw_sub_min & 0xffff; >+ if (fw_minor > CONNECTX_4_CURR_MAX_MINOR || >+ fw_minor < CONNECTX_4_INTX_SUPPORT_MINOR) { >+ dev_warn(&pdev->dev, "ConnectX-4: FW %u.%u.%u doesn't support INTx masking, disabling. Please upgrade FW to %d.14.1100 and up for INTx support\n", >+ fw_major, fw_minor, fw_subminor, pdev->device == >+ PCI_DEVICE_ID_MELLANOX_CONNECTX4 ? 12 : 14); >+ pdev->broken_intx_masking = 1; >+ } >+ >+ iounmap(fw_ver); >+ pci_disable_device(pdev); > } > DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MELLANOX, PCI_ANY_ID, > mellanox_check_broken_intx_masking); > -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Nov 18, 2016 at 02:38:47PM +1100, Gavin Shan wrote: > On Thu, Nov 17, 2016 at 05:30:31PM -0600, Bjorn Helgaas wrote: > >From: Noa Osherovich <noaos@mellanox.com> > > > >Mellanox devices were marked as having INTx masking ability broken. As a > >result, the VFIO driver fails to start when more than one device function > >is passed-through to a VM if both have the same INTx pin. > > > >Prior to Connect-IB, Mellanox devices exposed to the operating system one > >PCI function per all ports. Starting from Connect-IB, the devices are > >function-per-port. When passing the second function to a VM, VFIO will > >fail to start. > > > >Exclude ConnectX-4, ConnectX4-Lx and Connect-IB from the list of Mellanox > >devices marked as having broken INTx masking: > > > >- ConnectX-4 and ConnectX4-LX firmware version is checked. If INTx > > masking is supported, we unmark the broken INTx masking. > >- Connect-IB does not support INTx currently so will not cause any > > problem. > > > >[bhelgaas: call pci_disable_device() always, after iounmap()] > >Fixes: 11e42532ada3 ("PCI: Assume all Mellanox devices have broken INTx masking") > >Signed-off-by: Noa Osherovich <noaos@mellanox.com> > >Signed-off-by: Bjorn Helgaas <bhelgaas@google.com> > >Reviewed-by: Or Gerlitz <ogerlitz@mellanox.com> > >Reviewed-by: Gavin Shan <gwshan@linux.vnet.ibm.com> > >--- > > drivers/pci/quirks.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++--- > > 1 file changed, 54 insertions(+), 3 deletions(-) > > > >diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c > >index f0515b7..496ca13 100644 > >--- a/drivers/pci/quirks.c > >+++ b/drivers/pci/quirks.c > >@@ -3209,13 +3209,25 @@ static u16 mellanox_broken_intx_devs[] = { > > PCI_DEVICE_ID_MELLANOX_CONNECTX2, > > PCI_DEVICE_ID_MELLANOX_CONNECTX3, > > PCI_DEVICE_ID_MELLANOX_CONNECTX3_PRO, > >- PCI_DEVICE_ID_MELLANOX_CONNECTIB, > >- PCI_DEVICE_ID_MELLANOX_CONNECTX4, > >- PCI_DEVICE_ID_MELLANOX_CONNECTX4_LX, > > }; > > > >+#define CONNECTX_4_CURR_MAX_MINOR 99 > >+#define CONNECTX_4_INTX_SUPPORT_MINOR 14 > >+ > >+/* > >+ * Check ConnectX-4/LX FW version to see if it supports legacy interrupts. > >+ * If so, don't mark it as broken. > >+ * FW minor > 99 means older FW version format and no INTx masking support. > >+ * FW minor < 14 means new FW version format and no INTx masking support. > >+ */ > > static void mellanox_check_broken_intx_masking(struct pci_dev *pdev) > > { > >+ __be32 __iomem *fw_ver; > >+ u16 fw_major; > >+ u16 fw_minor; > >+ u16 fw_subminor; > >+ u32 fw_maj_min; > >+ u32 fw_sub_min; > > int i; > > > > for (i = 0; i < ARRAY_SIZE(mellanox_broken_intx_devs); i++) { > >@@ -3224,6 +3236,45 @@ static void mellanox_check_broken_intx_masking(struct pci_dev *pdev) > > return; > > } > > } > >+ > >+ /* Getting here means Connect-IB cards and up. Connect-IB has no INTx > >+ * support so shouldn't be checked further > >+ */ > >+ if (pdev->device == PCI_DEVICE_ID_MELLANOX_CONNECTIB) > >+ return; > >+ > >+ if (pdev->device != PCI_DEVICE_ID_MELLANOX_CONNECTX4 && > >+ pdev->device != PCI_DEVICE_ID_MELLANOX_CONNECTX4_LX) > >+ return; > >+ > >+ /* For ConnectX-4 and ConnectX-4LX, need to check FW support */ > >+ if (pci_enable_device_mem(pdev)) { > >+ dev_warn(&pdev->dev, "Can't enable device memory\n"); > >+ return; > >+ } > >+ > >+ fw_ver = ioremap(pci_resource_start(pdev, 0), 4); > >+ if (!fw_ver) { > >+ dev_warn(&pdev->dev, "Can't map ConnectX-4 initialization segment\n"); > > Bjorn, pci_disable_device() is still missed here. Could you please > amend it? Fixed and re-pushed, thanks! > >+ return; > >+ } > >+ > >+ /* Reading from resource space should be 32b aligned */ > >+ fw_maj_min = ioread32be(fw_ver); > >+ fw_sub_min = ioread32be(fw_ver + 1); > >+ fw_major = fw_maj_min & 0xffff; > >+ fw_minor = fw_maj_min >> 16; > >+ fw_subminor = fw_sub_min & 0xffff; > >+ if (fw_minor > CONNECTX_4_CURR_MAX_MINOR || > >+ fw_minor < CONNECTX_4_INTX_SUPPORT_MINOR) { > >+ dev_warn(&pdev->dev, "ConnectX-4: FW %u.%u.%u doesn't support INTx masking, disabling. Please upgrade FW to %d.14.1100 and up for INTx support\n", > >+ fw_major, fw_minor, fw_subminor, pdev->device == > >+ PCI_DEVICE_ID_MELLANOX_CONNECTX4 ? 12 : 14); > >+ pdev->broken_intx_masking = 1; > >+ } > >+ > >+ iounmap(fw_ver); > >+ pci_disable_device(pdev); > > } > > DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MELLANOX, PCI_ANY_ID, > > mellanox_check_broken_intx_masking); > > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-pci" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index f0515b7..496ca13 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -3209,13 +3209,25 @@ static u16 mellanox_broken_intx_devs[] = { PCI_DEVICE_ID_MELLANOX_CONNECTX2, PCI_DEVICE_ID_MELLANOX_CONNECTX3, PCI_DEVICE_ID_MELLANOX_CONNECTX3_PRO, - PCI_DEVICE_ID_MELLANOX_CONNECTIB, - PCI_DEVICE_ID_MELLANOX_CONNECTX4, - PCI_DEVICE_ID_MELLANOX_CONNECTX4_LX, }; +#define CONNECTX_4_CURR_MAX_MINOR 99 +#define CONNECTX_4_INTX_SUPPORT_MINOR 14 + +/* + * Check ConnectX-4/LX FW version to see if it supports legacy interrupts. + * If so, don't mark it as broken. + * FW minor > 99 means older FW version format and no INTx masking support. + * FW minor < 14 means new FW version format and no INTx masking support. + */ static void mellanox_check_broken_intx_masking(struct pci_dev *pdev) { + __be32 __iomem *fw_ver; + u16 fw_major; + u16 fw_minor; + u16 fw_subminor; + u32 fw_maj_min; + u32 fw_sub_min; int i; for (i = 0; i < ARRAY_SIZE(mellanox_broken_intx_devs); i++) { @@ -3224,6 +3236,45 @@ static void mellanox_check_broken_intx_masking(struct pci_dev *pdev) return; } } + + /* Getting here means Connect-IB cards and up. Connect-IB has no INTx + * support so shouldn't be checked further + */ + if (pdev->device == PCI_DEVICE_ID_MELLANOX_CONNECTIB) + return; + + if (pdev->device != PCI_DEVICE_ID_MELLANOX_CONNECTX4 && + pdev->device != PCI_DEVICE_ID_MELLANOX_CONNECTX4_LX) + return; + + /* For ConnectX-4 and ConnectX-4LX, need to check FW support */ + if (pci_enable_device_mem(pdev)) { + dev_warn(&pdev->dev, "Can't enable device memory\n"); + return; + } + + fw_ver = ioremap(pci_resource_start(pdev, 0), 4); + if (!fw_ver) { + dev_warn(&pdev->dev, "Can't map ConnectX-4 initialization segment\n"); + return; + } + + /* Reading from resource space should be 32b aligned */ + fw_maj_min = ioread32be(fw_ver); + fw_sub_min = ioread32be(fw_ver + 1); + fw_major = fw_maj_min & 0xffff; + fw_minor = fw_maj_min >> 16; + fw_subminor = fw_sub_min & 0xffff; + if (fw_minor > CONNECTX_4_CURR_MAX_MINOR || + fw_minor < CONNECTX_4_INTX_SUPPORT_MINOR) { + dev_warn(&pdev->dev, "ConnectX-4: FW %u.%u.%u doesn't support INTx masking, disabling. Please upgrade FW to %d.14.1100 and up for INTx support\n", + fw_major, fw_minor, fw_subminor, pdev->device == + PCI_DEVICE_ID_MELLANOX_CONNECTX4 ? 12 : 14); + pdev->broken_intx_masking = 1; + } + + iounmap(fw_ver); + pci_disable_device(pdev); } DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MELLANOX, PCI_ANY_ID, mellanox_check_broken_intx_masking);