| Message ID | 20240223152124.20042-5-johan+linaro@kernel.org (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | Manivannan Sadhasivam |
| Headers | show |
| Series | arm64: dts: qcom: sc8280xp: PCIe fixes and GICv3 ITS enable | expand |
On Fri, Feb 23, 2024 at 04:21:16PM +0100, Johan Hovold wrote: > Commit 9f4f3dfad8cf ("PCI: qcom: Enable ASPM for platforms supporting > 1.9.0 ops") started enabling ASPM unconditionally when the hardware > claims to support it. This triggers Correctable Errors for some PCIe > devices on machines like the Lenovo ThinkPad X13s, which could indicate > an incomplete driver ASPM implementation or that the hardware does in > fact not support L0s. Are there any more details about this? Do the errors occur around suspend/resume, a power state transition, or some other event? Might other DWC-based devices be susceptible? Is there a specific driver you suspect might be incomplete? Do you want the DT approach because the problem is believed to be platform-specific? Otherwise, maybe we should consider reverting 9f4f3dfad8cf until the problem is understood? Could this be done via a quirk like quirk_disable_aspm_l0s()? That currently uses pci_disable_link_state(), which I don't think is completely safe because it leaves the possibility that drivers or users could re-enable L0s, e.g., via sysfs. This patch is nice because IIUC it directly changes PCI_EXP_LNKCAP, which avoids that issue, but quirk_disable_aspm_l0s() could conceivably be reimplemented to cache PCI_EXP_LNKCAP in struct pci_dev so quirks could override it, as we do with struct pci_dev.devcap. > Add support for disabling ASPM L0s in the devicetree when it is not > supported on a particular machine and controller. > > Note that only the 1.9.0 ops enable ASPM currently. > > Fixes: 9f4f3dfad8cf ("PCI: qcom: Enable ASPM for platforms supporting 1.9.0 ops") > Cc: stable@vger.kernel.org # 6.7 > Signed-off-by: Johan Hovold <johan+linaro@kernel.org> > --- > drivers/pci/controller/dwc/pcie-qcom.c | 20 ++++++++++++++++++++ > 1 file changed, 20 insertions(+) > > diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c > index 09d485df34b9..0fb5dc06d2ef 100644 > --- a/drivers/pci/controller/dwc/pcie-qcom.c > +++ b/drivers/pci/controller/dwc/pcie-qcom.c > @@ -273,6 +273,25 @@ static int qcom_pcie_start_link(struct dw_pcie *pci) > return 0; > } > > +static void qcom_pcie_clear_aspm_l0s(struct dw_pcie *pci) > +{ > + u16 offset; > + u32 val; > + > + if (!of_property_read_bool(pci->dev->of_node, "aspm-no-l0s")) > + return; > + > + offset = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP); > + > + dw_pcie_dbi_ro_wr_en(pci); > + > + val = readl(pci->dbi_base + offset + PCI_EXP_LNKCAP); > + val &= ~PCI_EXP_LNKCAP_ASPM_L0S; > + writel(val, pci->dbi_base + offset + PCI_EXP_LNKCAP); > + > + dw_pcie_dbi_ro_wr_dis(pci); > +} > + > static void qcom_pcie_clear_hpc(struct dw_pcie *pci) > { > u16 offset = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP); > @@ -962,6 +981,7 @@ static int qcom_pcie_init_2_7_0(struct qcom_pcie *pcie) > > static int qcom_pcie_post_init_2_7_0(struct qcom_pcie *pcie) > { > + qcom_pcie_clear_aspm_l0s(pcie->pci); > qcom_pcie_clear_hpc(pcie->pci); > > return 0; > -- > 2.43.0 >
On Fri, Feb 23, 2024 at 04:10:00PM -0600, Bjorn Helgaas wrote: > On Fri, Feb 23, 2024 at 04:21:16PM +0100, Johan Hovold wrote: > > Commit 9f4f3dfad8cf ("PCI: qcom: Enable ASPM for platforms supporting > > 1.9.0 ops") started enabling ASPM unconditionally when the hardware > > claims to support it. This triggers Correctable Errors for some PCIe > > devices on machines like the Lenovo ThinkPad X13s, which could indicate > > an incomplete driver ASPM implementation or that the hardware does in > > fact not support L0s. > > Are there any more details about this? Do the errors occur around > suspend/resume, a power state transition, or some other event? Might > other DWC-based devices be susceptible? Is there a specific driver > you suspect might be incomplete? I see these errors when the devices in question are active as well as idle (not during suspend/resume). For example, when running iperf3 or fio to test the wifi and nvme, but I also see this occasionally for a wifi device which is (supposedly) not active (e.g. a handful errors over night). I skimmed Qualcomm's driver and noted that there are some registers related to ASPM which that driver updates, while the mainline driver leaves them at their default settings, but I essentially only mentioned that the ASPM implementation may be incomplete as a theoretical possibility. The somewhat erratic ASPM behaviour for one of the modems also suggests that some further tweak/quirk may be needed, and I was hoping to catch Mani's interest by reporting it. But based on what I've since heard from Qualcomm, it seems like these correctable error may be a known issue with the hardware (e.g. seen also with Windows), which is also why we decided to disable it for all controllers on these two platforms where I've seen this in v2. > Do you want the DT approach because the problem is believed to be > platform-specific? Otherwise, maybe we should consider reverting > 9f4f3dfad8cf until the problem is understood? Enabling ASPM gave a very significant improvement in battery life on the Lenovo ThinkPad X13s, from 10.5 h to 15 h, so reverting is not really an option there. And with L0s disabled, the AER error reports about correctable errors (that prevent enabling the GIC ITS and possibly degrades performance somewhat) are gone. I don't know for sure if there are further Qualcomm platform that are affected by this so I also don't want to use a too big of a hammer. The devicetree property allows us to disable L0s only after confirming that it's needed, and we can always extend this to broader classes of device when/if we learn more. > Could this be done via a quirk like quirk_disable_aspm_l0s()? That > currently uses pci_disable_link_state(), which I don't think is > completely safe because it leaves the possibility that drivers or > users could re-enable L0s, e.g., via sysfs. That was my first approach, thinking that it was the endpoint devices which did not really support L0s. But initially it seemed like the wifi controller on the CRD was not affected by this, while the same controller on the X13s was. That made me conclude that this is not just a property of the device but (also) of the controller and/or machine. I then noticed that we already had some controller drivers implementing 'aspm-no-l0s' and decided to go with that. > This patch is nice because IIUC it directly changes PCI_EXP_LNKCAP, > which avoids that issue, but quirk_disable_aspm_l0s() could > conceivably be reimplemented to cache PCI_EXP_LNKCAP in struct pci_dev > so quirks could override it, as we do with struct pci_dev.devcap. Johan
On Tue, Feb 27, 2024 at 04:29:15PM +0100, Johan Hovold wrote: > On Fri, Feb 23, 2024 at 04:10:00PM -0600, Bjorn Helgaas wrote: > > On Fri, Feb 23, 2024 at 04:21:16PM +0100, Johan Hovold wrote: > > > Commit 9f4f3dfad8cf ("PCI: qcom: Enable ASPM for platforms supporting > > > 1.9.0 ops") started enabling ASPM unconditionally when the hardware > > > claims to support it. This triggers Correctable Errors for some PCIe > > > devices on machines like the Lenovo ThinkPad X13s, which could indicate > > > an incomplete driver ASPM implementation or that the hardware does in > > > fact not support L0s. > > > > Are there any more details about this? Do the errors occur around > > suspend/resume, a power state transition, or some other event? Might > > other DWC-based devices be susceptible? Is there a specific driver > > you suspect might be incomplete? > > I see these errors when the devices in question are active as well as > idle (not during suspend/resume). For example, when running iperf3 or > fio to test the wifi and nvme, but I also see this occasionally for a > wifi device which is (supposedly) not active (e.g. a handful errors over > night). > > I skimmed Qualcomm's driver and noted that there are some registers > related to ASPM which that driver updates, while the mainline driver > leaves them at their default settings, but I essentially only mentioned > that the ASPM implementation may be incomplete as a theoretical > possibility. The somewhat erratic ASPM behaviour for one of the modems > also suggests that some further tweak/quirk may be needed, and I was > hoping to catch Mani's interest by reporting it. > > But based on what I've since heard from Qualcomm, it seems like these > correctable error may be a known issue with the hardware (e.g. seen > also with Windows), which is also why we decided to disable it for all > controllers on these two platforms where I've seen this in v2. > > > Do you want the DT approach because the problem is believed to be > > platform-specific? Otherwise, maybe we should consider reverting > > 9f4f3dfad8cf until the problem is understood? > > Enabling ASPM gave a very significant improvement in battery life on the > Lenovo ThinkPad X13s, from 10.5 h to 15 h, so reverting is not really an > option there. Ah, I missed that you're only disabling L0s, but leaving L1 enabled, thanks! And given that the v1.9.0 ops that enable ASPM are used on a bunch of platforms, and L0s seems to work fine on most of them, we wouldn't want to disable L0s for everybody, so this seems like the right solution. Bjorn
diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c index 09d485df34b9..0fb5dc06d2ef 100644 --- a/drivers/pci/controller/dwc/pcie-qcom.c +++ b/drivers/pci/controller/dwc/pcie-qcom.c @@ -273,6 +273,25 @@ static int qcom_pcie_start_link(struct dw_pcie *pci) return 0; } +static void qcom_pcie_clear_aspm_l0s(struct dw_pcie *pci) +{ + u16 offset; + u32 val; + + if (!of_property_read_bool(pci->dev->of_node, "aspm-no-l0s")) + return; + + offset = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP); + + dw_pcie_dbi_ro_wr_en(pci); + + val = readl(pci->dbi_base + offset + PCI_EXP_LNKCAP); + val &= ~PCI_EXP_LNKCAP_ASPM_L0S; + writel(val, pci->dbi_base + offset + PCI_EXP_LNKCAP); + + dw_pcie_dbi_ro_wr_dis(pci); +} + static void qcom_pcie_clear_hpc(struct dw_pcie *pci) { u16 offset = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP); @@ -962,6 +981,7 @@ static int qcom_pcie_init_2_7_0(struct qcom_pcie *pcie) static int qcom_pcie_post_init_2_7_0(struct qcom_pcie *pcie) { + qcom_pcie_clear_aspm_l0s(pcie->pci); qcom_pcie_clear_hpc(pcie->pci); return 0;
Commit 9f4f3dfad8cf ("PCI: qcom: Enable ASPM for platforms supporting 1.9.0 ops") started enabling ASPM unconditionally when the hardware claims to support it. This triggers Correctable Errors for some PCIe devices on machines like the Lenovo ThinkPad X13s, which could indicate an incomplete driver ASPM implementation or that the hardware does in fact not support L0s. Add support for disabling ASPM L0s in the devicetree when it is not supported on a particular machine and controller. Note that only the 1.9.0 ops enable ASPM currently. Fixes: 9f4f3dfad8cf ("PCI: qcom: Enable ASPM for platforms supporting 1.9.0 ops") Cc: stable@vger.kernel.org # 6.7 Signed-off-by: Johan Hovold <johan+linaro@kernel.org> --- drivers/pci/controller/dwc/pcie-qcom.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)