| Message ID | 20170818213210.15145.15340.stgit@bhelgaas-glaptop.roam.corp.google.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On 8/18/2017 5:32 PM, Bjorn Helgaas wrote: > + if ((*l & 0xffff) != 0x0001) > + return true; /* not a CRS completion */ > This version certainly looks cleaner. However, it breaks pci_flr_wait(). If some root port doesn't support CRS and returns 0xFFFFFFFF, pci_bus_wait_crs() function returns true. pci_flr_wait() prematurely bails out from here. pci_flr_wait() { + ret = pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000); + if (ret) + return; } We can change the return code to false above but then we break pci_bus_read_dev_vendor_id() function. That's why, I was interested in creating a pci_bus_crs_visibility_supported() helper function that would check for the magic 0x0001 value and return true. Otherwise, false. pci_bus_read_dev_vendor_id() would do this pci_bus_read_dev_vendor_id() { ... if (pci_bus_crs_visibility_supported()) return pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000); return true } Similar pattern for pci_flr_wait().
On Mon, Aug 21, 2017 at 09:53:56AM -0400, Sinan Kaya wrote: > On 8/18/2017 5:32 PM, Bjorn Helgaas wrote: > > + if ((*l & 0xffff) != 0x0001) > > + return true; /* not a CRS completion */ > > > > This version certainly looks cleaner. However, it breaks pci_flr_wait(). > > If some root port doesn't support CRS and returns 0xFFFFFFFF, pci_bus_wait_crs() > function returns true. pci_flr_wait() prematurely bails out from here. > > > pci_flr_wait() > { > > + ret = pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000); > + if (ret) > + return; > > } > > We can change the return code to false above but then we break pci_bus_read_dev_vendor_id() > function. > > That's why, I was interested in creating a pci_bus_crs_visibility_supported() helper > function that would check for the magic 0x0001 value and return true. Otherwise, false. > > pci_bus_read_dev_vendor_id() would do this > > pci_bus_read_dev_vendor_id() > { > ... > if (pci_bus_crs_visibility_supported()) > return pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000); > > return true > } > > Similar pattern for pci_flr_wait(). I think that makes sense. We'd want to check for CRS SV being enabled, e.g., maybe read PCI_EXP_RTCTL_CRSSVE back in pci_enable_crs() and cache it somewhere. Maybe a crs_sv_enabled bit in the root port's pci_dev, and check it with something like what pcie_root_rcb_set() does?
On 8/21/2017 3:18 PM, Bjorn Helgaas wrote: >> pci_bus_read_dev_vendor_id() >> { >> ... >> if (pci_bus_crs_visibility_supported()) >> return pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000); >> >> return true >> } >> >> Similar pattern for pci_flr_wait(). Sorry for the poor choice of function name. I was thinking of something like this. bool pci_bus_crs_pending(u32 l) { return (l & 0xFFFF) == 0x0001 } if (pci_bus_crs_pending(id)) return pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000); > I think that makes sense. We'd want to check for CRS SV being > enabled, e.g., maybe read PCI_EXP_RTCTL_CRSSVE back in > pci_enable_crs() and cache it somewhere. Maybe a crs_sv_enabled bit > in the root port's pci_dev, and check it with something like what > pcie_root_rcb_set() does? > You can observe CRS under the following conditions 1. root port <-> endpoint 2. bridge <-> endpoint 3. root port<->bridge I was relying on the fact that we are reading 0x001 as an indication that this device detected CRS. Maybe, this is too indirect. If we also want to capture the capability, I think the right thing is to check the parent capability. bool pci_bus_crs_vis_supported(struct pci_dev *bridge) { if (device type(bridge) == root port) return read(root_crs_register_reg); if (device type(bridge) == switch) return read(switch_crs_register); return false; } bool pci_bus_crs_pending(struct pci_dev *dev, u32 l) { if !pci_bus_crs_vis_supported(dev->parent) return false; return (l & 0xFFFF) == 0x0001; } I'll prototype this. Let me know if you have concerns.
On Mon, Aug 21, 2017 at 03:37:06PM -0400, Sinan Kaya wrote: > On 8/21/2017 3:18 PM, Bjorn Helgaas wrote: > ... > if (pci_bus_crs_pending(id)) > return pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000); > > > I think that makes sense. We'd want to check for CRS SV being > > enabled, e.g., maybe read PCI_EXP_RTCTL_CRSSVE back in > > pci_enable_crs() and cache it somewhere. Maybe a crs_sv_enabled bit > > in the root port's pci_dev, and check it with something like what > > pcie_root_rcb_set() does? > > > > You can observe CRS under the following conditions > > 1. root port <-> endpoint > 2. bridge <-> endpoint > 3. root port<->bridge > > I was relying on the fact that we are reading 0x001 as an indication that > this device detected CRS. Maybe, this is too indirect. > > If we also want to capture the capability, I think the right thing is to > check the parent capability. > > bool pci_bus_crs_vis_supported(struct pci_dev *bridge) > { > if (device type(bridge) == root port) > return read(root_crs_register_reg); > > if (device type(bridge) == switch) > return read(switch_crs_register); I don't understand this part. AFAIK, CRS SV is only a feature of root ports. The capability and enable bits are in the Root Capabilities and Root Control registers. It's certainly true that a device below a switch can respond with a CRS completion, but the switch is not the requester, and my understanding is that it would not take any action on the completion other than passing it upstream.
On 8/21/2017 4:23 PM, Bjorn Helgaas wrote: > On Mon, Aug 21, 2017 at 03:37:06PM -0400, Sinan Kaya wrote: >> On 8/21/2017 3:18 PM, Bjorn Helgaas wrote: >> ... >> if (pci_bus_crs_pending(id)) >> return pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000); >> >>> I think that makes sense. We'd want to check for CRS SV being >>> enabled, e.g., maybe read PCI_EXP_RTCTL_CRSSVE back in >>> pci_enable_crs() and cache it somewhere. Maybe a crs_sv_enabled bit >>> in the root port's pci_dev, and check it with something like what >>> pcie_root_rcb_set() does? >>> >> >> You can observe CRS under the following conditions >> >> 1. root port <-> endpoint >> 2. bridge <-> endpoint >> 3. root port<->bridge >> >> I was relying on the fact that we are reading 0x001 as an indication that >> this device detected CRS. Maybe, this is too indirect. >> >> If we also want to capture the capability, I think the right thing is to >> check the parent capability. >> >> bool pci_bus_crs_vis_supported(struct pci_dev *bridge) >> { >> if (device type(bridge) == root port) >> return read(root_crs_register_reg); >> >> if (device type(bridge) == switch) >> return read(switch_crs_register); > > I don't understand this part. AFAIK, CRS SV is only a feature of root > ports. The capability and enable bits are in the Root Capabilities > and Root Control registers. > No question about it. > It's certainly true that a device below a switch can respond with a > CRS completion, but the switch is not the requester, and my > understanding is that it would not take any action on the completion > other than passing it upstream. > I saw some bridge references in the spec for CRS. I was going to do some research for it. You answered my question. I was curious how this would impact the behavior. "Bridge Configuration Retry Enable – When Set, this bit enables PCI Express to PCI/PCI-X bridges to return Configuration Request Retry Status (CRS) in response to Configuration Requests that target devices below the bridge. Refer to the PCI Express to PCI/PCI-X Bridge Specification, Revision 1.0 for further details."
On Mon, Aug 21, 2017 at 04:32:26PM -0400, Sinan Kaya wrote: > On 8/21/2017 4:23 PM, Bjorn Helgaas wrote: > > On Mon, Aug 21, 2017 at 03:37:06PM -0400, Sinan Kaya wrote: > >> On 8/21/2017 3:18 PM, Bjorn Helgaas wrote: > >> ... > >> if (pci_bus_crs_pending(id)) > >> return pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000); > >> > >>> I think that makes sense. We'd want to check for CRS SV being > >>> enabled, e.g., maybe read PCI_EXP_RTCTL_CRSSVE back in > >>> pci_enable_crs() and cache it somewhere. Maybe a crs_sv_enabled bit > >>> in the root port's pci_dev, and check it with something like what > >>> pcie_root_rcb_set() does? > >>> > >> > >> You can observe CRS under the following conditions > >> > >> 1. root port <-> endpoint > >> 2. bridge <-> endpoint > >> 3. root port<->bridge > >> > >> I was relying on the fact that we are reading 0x001 as an indication that > >> this device detected CRS. Maybe, this is too indirect. > >> > >> If we also want to capture the capability, I think the right thing is to > >> check the parent capability. > >> > >> bool pci_bus_crs_vis_supported(struct pci_dev *bridge) > >> { > >> if (device type(bridge) == root port) > >> return read(root_crs_register_reg); > >> > >> if (device type(bridge) == switch) > >> return read(switch_crs_register); > > > > I don't understand this part. AFAIK, CRS SV is only a feature of root > > ports. The capability and enable bits are in the Root Capabilities > > and Root Control registers. > > > > No question about it. > > > It's certainly true that a device below a switch can respond with a > > CRS completion, but the switch is not the requester, and my > > understanding is that it would not take any action on the completion > > other than passing it upstream. > > > > I saw some bridge references in the spec for CRS. I was going to do > some research for it. You answered my question. I was curious how this > would impact the behavior. > > "Bridge Configuration Retry Enable – When Set, this bit enables PCI Express > to PCI/PCI-X bridges to return Configuration Request Retry Status (CRS) in > response to Configuration Requests that target devices below the bridge. > Refer to the PCI Express to PCI/PCI-X Bridge Specification, Revision 1.0 for > further details." (The above is from PCIe r3.1, sec 7.8.4, Device Control Register, and also discussed in sec 4.3 of the PCIe-to-PCI/PCI-X bridge spec.) In any event, the Bridge Configuration Retry Enable only determines whether the bridge ever returns a CRS completion. The bridge itself never converts a CRS completion into the 0x0001 vendor ID.
On 8/21/2017 3:37 PM, Sinan Kaya wrote: > bool pci_bus_crs_pending(struct pci_dev *dev, u32 l) > { > if !pci_bus_crs_vis_supported(dev->parent) > return false; Apparently, I can't do this. By the time, we come to here from vendor id read function, the topology has not been set up yet. I'm getting an exception: [ 6.067392] [<ffff00000846c878>] pci_bus_crs_visibility_pending+0x4/0x7c [ 6.074085] [<ffff00000846cccc>] pci_scan_single_device+0x40/0xb4 [ 6.080170] [<ffff00000846cd90>] pci_scan_slot+0x50/0xe8 [ 6.085474] [<ffff00000846dc14>] pci_scan_child_bus+0x30/0x108 [ 6.091300] [<ffff0000084bab94>] acpi_pci_root_create+0x184/0x1f0 [ 6.097388] [<ffff000008091dc8>] pci_acpi_scan_root+0x188/0x1d4 [ 6.103298] [<ffff0000084ba7a8>] acpi_pci_root_add+0x38c/0x44c [ 6.109125] [<ffff0000084b4d94>] acpi_bus_attach+0xe0/0x1ac [ 6.114689] [<ffff0000084b4e08>] acpi_bus_attach+0x154/0x1ac [ 6.120340] [<ffff0000084b4e08>] acpi_bus_attach+0x154/0x1ac [ 6.125991] [<ffff0000084b6608>] acpi_bus_scan+0x60/0x70 [ 6.131297] [<ffff0000091c87f8>] acpi_scan_init+0xd8/0x228 [ 6.136774] [<ffff0000091c84e0>] acpi_init+0x2d4/0x328 [ 6.141905] [<ffff0000091a0c88>] do_one_initcall+0x80/0x108 [ 6.147469] [<ffff0000091a0e98>] kernel_init_freeable+0x188/0x228 [ 6.153556] [<ffff000008c9bcbc>] kernel_init+0x10/0xfc [ 6.158687] [<ffff000008082ec0>] ret_from_fork+0x10/0x50 > > return (l & 0xFFFF) == 0x0001; > }
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 22e061738c6f..b0857052c04a 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -235,6 +235,7 @@ enum pci_bar_type { pci_bar_mem64, /* A 64-bit memory BAR */ }; +bool pci_bus_wait_crs(struct pci_bus *bus, int devfn, u32 *l, int timeout); bool pci_bus_read_dev_vendor_id(struct pci_bus *bus, int devfn, u32 *pl, int crs_timeout); int pci_setup_device(struct pci_dev *dev); diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 08ea844ac4ba..342a86640c6b 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -1824,33 +1824,26 @@ struct pci_dev *pci_alloc_dev(struct pci_bus *bus) } EXPORT_SYMBOL(pci_alloc_dev); -bool pci_bus_read_dev_vendor_id(struct pci_bus *bus, int devfn, u32 *l, - int crs_timeout) +bool pci_bus_wait_crs(struct pci_bus *bus, int devfn, u32 *l, int timeout) { int delay = 1; - if (pci_bus_read_config_dword(bus, devfn, PCI_VENDOR_ID, l)) - return false; + if ((*l & 0xffff) != 0x0001) + return true; /* not a CRS completion */ - /* some broken boards return 0 or ~0 if a slot is empty: */ - if (*l == 0xffffffff || *l == 0x00000000 || - *l == 0x0000ffff || *l == 0xffff0000) - return false; + if (!timeout) + return false; /* CRS, but caller doesn't want to wait */ /* - * Configuration Request Retry Status. Some root ports return the - * actual device ID instead of the synthetic ID (0xFFFF) required - * by the PCIe spec. Ignore the device ID and only check for - * (vendor id == 1). + * We got the reserved Vendor ID that indicates a completion with + * Configuration Request Retry Status (CRS). Retry until we get a + * valid Vendor ID or we time out. */ while ((*l & 0xffff) == 0x0001) { - if (!crs_timeout) - return false; - msleep(delay); delay *= 2; - if (delay > crs_timeout) { + if (delay > timeout) { printk(KERN_WARNING "pci %04x:%02x:%02x.%d: not responding\n", pci_domain_nr(bus), bus->number, PCI_SLOT(devfn), PCI_FUNC(devfn)); @@ -1863,6 +1856,20 @@ bool pci_bus_read_dev_vendor_id(struct pci_bus *bus, int devfn, u32 *l, return true; } + +bool pci_bus_read_dev_vendor_id(struct pci_bus *bus, int devfn, u32 *l, + int timeout) +{ + if (pci_bus_read_config_dword(bus, devfn, PCI_VENDOR_ID, l)) + return false; + + /* some broken boards return 0 or ~0 if a slot is empty: */ + if (*l == 0xffffffff || *l == 0x00000000 || + *l == 0x0000ffff || *l == 0xffff0000) + return false; + + return pci_bus_wait_crs(bus, devfn, l, timeout); +} EXPORT_SYMBOL(pci_bus_read_dev_vendor_id); /*