From patchwork Tue Oct 1 08:34:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jian-Hong Pan X-Patchwork-Id: 13817641 X-Patchwork-Delegate: kw@linux.com Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B58519ABC4 for ; Tue, 1 Oct 2024 08:35:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727771740; cv=none; b=ctLkJs9Vh9x8btEn9MPc0/pqgdOSoRWCTZvbJZi2JiBfZ+6EB1aa8IcbbR0bb6OVe84crIqVQ4ZVR0kgQH7QA6+4uUIkZ1xIAmckApKTuRxHMPMGq/G5AdfTdrMM73hdbjxit4kWp78VLBtxUmrWa/Tw1e9IXLETQLg2PIe0vKs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727771740; c=relaxed/simple; bh=BWMkhzEh6lpnZXzCb4xgOx6TwYjnZdmi3ryFFcevWsg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NVPMtdHJDHvIHK7t8AT9VVZa0MsN8Rlhq3ohRDUHo0YackYpD9f8WPEHuIIRRL7/5lQgoAlj3bEqWeUEh9rofczeCJVuPwG3/bor6m/PkE7cRDnzCC9yT5a+jdiOC3KNFonhBi6a0Gak5tscEqmBq7HHURsDxrw2syB9th/4rIc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=endlessos.org; spf=pass smtp.mailfrom=endlessos.org; dkim=pass (2048-bit key) header.d=endlessos.org header.i=@endlessos.org header.b=MuCVceCD; arc=none smtp.client-ip=209.85.210.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=endlessos.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=endlessos.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=endlessos.org header.i=@endlessos.org header.b="MuCVceCD" Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-71db62281aeso839339b3a.0 for ; Tue, 01 Oct 2024 01:35:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=endlessos.org; s=google; t=1727771738; x=1728376538; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9Y8hsdN8SedF9rTxGC/1IRk82j7hLWsvR2mtT1iPaYo=; b=MuCVceCDzpcEmj7LW544foLLD3oOEBeceRIsUM4oSIWDbm+JPtdTLQN7F9KWs5RgEH aiGqFW7bjYZ96arCwjM7dNm9uO55x8h/fJRvFjTHhQ7rmw5tLrfJEjuUuEYMXftZJvpQ g2qQ4wFAS1LfAWAakHOvLQcX5t5BMwa3tUXYbHxw11pCNv+7s20UIOLR2IeoWwk9KX88 0GlyQTrQknkobcmvkXbHEKDMpsU9AURdnqcSylrt7H5hsjzRXXyzwRI+OoDk4ka0mpZT gQPRRWugj80+VeUL6J4sbLlDGTBBqnH4kOBOrb4rwt8ODD7nUWHiz7wvu224Y7lF442i JYcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727771738; x=1728376538; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9Y8hsdN8SedF9rTxGC/1IRk82j7hLWsvR2mtT1iPaYo=; b=RSlmhecB2TJhzY6KZaHJND7Q3Cf/Dow3Khc2J2rBCc/QZOxFAjeZWecfb4fH4Sntsq QXLl5vnWE48SRLwb1BetfsMUp+BCkZrexDj1A9asJagcvZ+0nn38ZgDZEqd6kmgzncdX UvBmYTNe8FE09o9MZ+ZTI9CtDavpaCjvBwbm6NDypXmSXmVhFQ6/Ofuv4IJU9+7JxK73 VxEw/BAcFQfWYDv5N2UPNJ76PbLws7ortx7VhtupIVdMC2sDCpSNhcS636qT3a96/dGM RgC9KrC7gEQp5P35wHZ/d3H0m2dTaHysiraNC5G/ifo51jOOZRtsGv0aPTdFqxo62fU4 nY3Q== X-Forwarded-Encrypted: i=1; AJvYcCXPtLHHtrydSJahQey0XX4XSmFqehg5gQJLc8EzgpHOv7pAmqrLs82aq4S8pEiFk3rKjFY7U5o6gR8=@vger.kernel.org X-Gm-Message-State: AOJu0Ywbvk7+lPjmrcpc+xpNAiUdeRx5ZcMiHbcnJ0XQJRiUGYDnyrMf oIrp6xDoH90IuvB66i9W6zfksPe+GV1fd2PO4aAUk1qP0cAYCpWG5t9ilMZnj68= X-Google-Smtp-Source: AGHT+IH9G+MfxrojB8phcCD1N5/Snag/pVEgMjttCtEBMh9QOUWC+VYONC08uOzAV+qBVihgGlHZzQ== X-Received: by 2002:a62:f241:0:b0:71d:c192:173a with SMTP id d2e1a72fcca58-71dc1921797mr281837b3a.10.1727771737616; Tue, 01 Oct 2024 01:35:37 -0700 (PDT) Received: from localhost.localdomain ([123.51.167.56]) by smtp.googlemail.com with ESMTPSA id d2e1a72fcca58-71b26516148sm7521573b3a.110.2024.10.01.01.35.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Oct 2024 01:35:37 -0700 (PDT) From: Jian-Hong Pan To: Bjorn Helgaas Cc: Johan Hovold , David Box , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Kuppuswamy Sathyanarayanan , Nirmal Patel , Jonathan Derrick , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux@endlessos.org, Jian-Hong Pan Subject: [PATCH v12 1/3] PCI: vmd: Set PCI devices to D0 before enable PCI PM's L1 substates Date: Tue, 1 Oct 2024 16:34:38 +0800 Message-ID: <20241001083438.10070-4-jhp@endlessos.org> X-Mailer: git-send-email 2.46.2 In-Reply-To: <20241001083438.10070-2-jhp@endlessos.org> References: <20241001083438.10070-2-jhp@endlessos.org> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The remapped PCIe Root Port and the child device have PCI PM L1 substates capability, but they are disabled originally. Here is a failed example on ASUS B1400CEAE: Capabilities: [900 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- L1_PM_Substates+ PortCommonModeRestoreTime=32us PortTPowerOnTime=10us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=101376ns L1SubCtl2: T_PwrOn=50us Power on all of the VMD remapped PCI devices to D0 before enable PCI-PM L1 PM Substates by following "PCIe r6.0, sec 5.5.4". Link: https://bugzilla.kernel.org/show_bug.cgi?id=218394 Signed-off-by: Jian-Hong Pan Reviewed-by: Kuppuswamy Sathyanarayanan --- v2: - Power on the VMD remapped devices with pci_set_power_state_locked() - Prepare the PCIe LTR parameters before enable L1 Substates - Add note into the comments of both pci_enable_link_state() and pci_enable_link_state_locked() for kernel-doc. - The original patch set can be split as individual patches. v3: - Re-send for the missed version information. - Split drivers/pci/pcie/aspm.c modification into following patches. - Fix the comment for enasuring the PCI devices in D0. v4: - The same v5: - Tweak the commit title and message - Change the goto label from out_enable_link_state to out_state_change v6~8: - The same v9: - Update L1 PM Substates information against kernel v6.11 in commit message v10~12: - The same drivers/pci/controller/vmd.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index 264a180403a0..11870d1fc818 100644 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -740,11 +740,9 @@ static int vmd_pm_enable_quirk(struct pci_dev *pdev, void *userdata) if (!(features & VMD_FEAT_BIOS_PM_QUIRK)) return 0; - pci_enable_link_state_locked(pdev, PCIE_LINK_STATE_ALL); - pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_LTR); if (!pos) - return 0; + goto out_state_change; /* * Skip if the max snoop LTR is non-zero, indicating BIOS has set it @@ -752,7 +750,7 @@ static int vmd_pm_enable_quirk(struct pci_dev *pdev, void *userdata) */ pci_read_config_dword(pdev, pos + PCI_LTR_MAX_SNOOP_LAT, <r_reg); if (!!(ltr_reg & (PCI_LTR_VALUE_MASK | PCI_LTR_SCALE_MASK))) - return 0; + goto out_state_change; /* * Set the default values to the maximum required by the platform to @@ -764,6 +762,13 @@ static int vmd_pm_enable_quirk(struct pci_dev *pdev, void *userdata) pci_write_config_dword(pdev, pos + PCI_LTR_MAX_SNOOP_LAT, ltr_reg); pci_info(pdev, "VMD: Default LTR value set by driver\n"); +out_state_change: + /* + * Ensure devices are in D0 before enabling PCI-PM L1 PM Substates, per + * PCIe r6.0, sec 5.5.4. + */ + pci_set_power_state_locked(pdev, PCI_D0); + pci_enable_link_state_locked(pdev, PCIE_LINK_STATE_ALL); return 0; } From patchwork Tue Oct 1 08:34:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jian-Hong Pan X-Patchwork-Id: 13817642 X-Patchwork-Delegate: kw@linux.com Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 950B319ABDE for ; Tue, 1 Oct 2024 08:36:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727771781; cv=none; b=fUYOkBgiK57REiNsXrxFLdXNrcK9tJhO/AKofDNFP8XjCHTeE3UnZbemf8lj2wOCvSXDor55w8CqaaU/6Q1k2ZiG7pisUOV/oPCEzacaw8fFmPcS4wLectYur/bkIgrgEzhlyqrv0b7EBaYZwqvLzwvY6209Mgv/mhcEc5JX21o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727771781; c=relaxed/simple; bh=X762HUvHe79Sfszvlpr6AaCEObfNJ8XwLibj0VikeaA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Of2+Sdkq6s5M0CQqeYxvkDiyZjuz4a8Hc+oTeG0PJRPf52spCROoU18/ID2faTJ8nDIqd1vJHPrIFyXKyJwu9FqFbxxPHPbSt6jhMUvmWLj3PfcvlaeV1RMvmhWFrhex1QCO5e9ZTbvb8/aYtuJjojhV5kJWDnkAV8mA7lh4orI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=endlessos.org; spf=pass smtp.mailfrom=endlessos.org; dkim=pass (2048-bit key) header.d=endlessos.org header.i=@endlessos.org header.b=V3N4fUM+; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=endlessos.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=endlessos.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=endlessos.org header.i=@endlessos.org header.b="V3N4fUM+" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-7178df70f28so4112958b3a.2 for ; Tue, 01 Oct 2024 01:36:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=endlessos.org; s=google; t=1727771779; x=1728376579; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9imT704LRAGL7Jvi8rBCp96QNuC359qUawNx+2cRI5g=; b=V3N4fUM+jzchbc3zlZM9EluBS4pOABr07F1u+GBSOZkqin3P8yVF1GDZMNusbKOdEi Hgvr7oSDYd1Ujw1c/kSin6tWYdTa8SJEZzlIhsmUT4NH8shCD8S1ndojm87oSuEmE5Bb TV3OKupMa6+XibwSOmJzqQYRRYkCP2fHFmMhNmElO1f9u4vuO8bXPfrybgytdlM1IES9 6N9yJqja1LCdmkbPv1UdPEtUTRx5llBaBz9hm4A7rE366FKRSvZxz/s/nCg1PCp7v5Wj gn9zhqpQQbyXbBUiWyTwbhTR0bc0d4cM9AJE3s5q9NZiEHk3xp3F0kfE0XMsOZEZjWQg ux6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727771779; x=1728376579; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9imT704LRAGL7Jvi8rBCp96QNuC359qUawNx+2cRI5g=; b=S+D7cpUZZRr7nFcuShLin3x603NyLXdCMNS9HKbAj52JGlLray5ePFa4fO5kGlY0zN C5mTS3v+nBZiiGlaNDMQAvZW9oIBwR2VC55ZahKTwWNQRBGhDtZiVFbX3Slb1eTsS3GO YQs3ulocPuxt+i2KA6e3BocKBZLC3H4a3+hFNKhHfvNr5nJLmVcUfe/y0LWhU9m/Egf6 VBUOfoPnyyi7BGM/Ha3d7hANYOEA8ywpbWvqi+giMqj1N5q5XF4rU9JghST9T0+kKdvk qfVC0ue+3M2y1b9UCwMi4Gev4rDnNNUEo2lS8RnPMAPQ5vz2ibhVNkvPyqswkAb7qwYy 4A0w== X-Forwarded-Encrypted: i=1; AJvYcCWHvTvFHKy4TSjpUwlDnEFqCgvY+ogNhw3+u0MI1VGHGTqENZsb9l47/TcjcoouhPgRzAlpexvK5wg=@vger.kernel.org X-Gm-Message-State: AOJu0YzI6fftRS37zih49jBdORbxpclXNgLQcuqMEcu1N+j+6n10bmPU XfXEKRRZNPpatBDKCQf4R9hQ2fBqsTdbGsYOAB1CJiedaESryG9CZMH4b5dzsDg= X-Google-Smtp-Source: AGHT+IGTYZspzAErDSrmscoQdRmq6XKHRbgb09e/7smZlcCyPbeS3xskua8DxYNdwUkvMDs21Dj2bw== X-Received: by 2002:a05:6a00:39a2:b0:717:8d5d:94d6 with SMTP id d2e1a72fcca58-71b25f4416cmr23558820b3a.12.1727771778836; Tue, 01 Oct 2024 01:36:18 -0700 (PDT) Received: from localhost.localdomain ([123.51.167.56]) by smtp.googlemail.com with ESMTPSA id d2e1a72fcca58-71b26516148sm7521573b3a.110.2024.10.01.01.36.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Oct 2024 01:36:18 -0700 (PDT) From: Jian-Hong Pan To: Bjorn Helgaas Cc: Johan Hovold , David Box , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Kuppuswamy Sathyanarayanan , Nirmal Patel , Jonathan Derrick , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux@endlessos.org, Jian-Hong Pan , Bjorn Helgaas Subject: [PATCH v12 2/3] PCI/ASPM: Add notes about enabling PCI-PM L1SS to pci_enable_link_state(_locked) Date: Tue, 1 Oct 2024 16:34:40 +0800 Message-ID: <20241001083438.10070-6-jhp@endlessos.org> X-Mailer: git-send-email 2.46.2 In-Reply-To: <20241001083438.10070-2-jhp@endlessos.org> References: <20241001083438.10070-2-jhp@endlessos.org> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 According to "PCIe r6.0, sec 5.5.4", add note about D0 requirement in pci_enable_link_state() kernel-doc. Signed-off-by: Jian-Hong Pan Acked-by: Bjorn Helgaas Reviewed-by: Kuppuswamy Sathyanarayanan --- v3: - Fix as readable comments v4: - The same v5: - Tweak and simplify the commit message v6~12: - The same drivers/pci/pcie/aspm.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index cee2365e54b8..bd0a8a05647e 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -1442,6 +1442,9 @@ static int __pci_enable_link_state(struct pci_dev *pdev, int state, bool locked) * touch the LNKCTL register. Also note that this does not enable states * disabled by pci_disable_link_state(). Return 0 or a negative errno. * + * Note: Ensure devices are in D0 before enabling PCI-PM L1 PM Substates, per + * PCIe r6.0, sec 5.5.4. + * * @pdev: PCI device * @state: Mask of ASPM link states to enable */ @@ -1458,6 +1461,9 @@ EXPORT_SYMBOL(pci_enable_link_state); * can't touch the LNKCTL register. Also note that this does not enable states * disabled by pci_disable_link_state(). Return 0 or a negative errno. * + * Note: Ensure devices are in D0 before enabling PCI-PM L1 PM Substates, per + * PCIe r6.0, sec 5.5.4. + * * @pdev: PCI device * @state: Mask of ASPM link states to enable * From patchwork Tue Oct 1 08:34:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jian-Hong Pan X-Patchwork-Id: 13817643 X-Patchwork-Delegate: kw@linux.com Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 84D1A199FD7 for ; Tue, 1 Oct 2024 08:36:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727771814; cv=none; b=mFAj9wext5bX8hkluZqFIPfx8+87O+fom4YZxREiIQRhowOzfrlaQnaiJGIay5iIPR1m6V4vLCJCjBrY6azXmN+HRxkt2CrnzvKJlbtLkBX7RZXZH9AvydLcfBaGvo/0Xd2slRzv6JvP+XFO/Whuh6Mti9YVvEkptaxYKS0eD9U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727771814; c=relaxed/simple; bh=YHdjDc5QvQTv8dnV4uC66vaWOywXjzF86Je+wGkstBA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=J6yECbGFw6tUerj1P5wFZ8Ch38UKEHyEzwOihZVm2+rbFdYq0fI1H1AHuPEhxzqQOtiiZPfQSFz6nTKpfVcz4nnNFa4RDBCxHrGBsjoVqS6N01raOTB/GIWz7xdFBc+GB55xMQgar2gHLwNx3iDjv1scMQYcdTHPYwb1+xtZX/c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=endlessos.org; spf=pass smtp.mailfrom=endlessos.org; dkim=pass (2048-bit key) header.d=endlessos.org header.i=@endlessos.org header.b=ARtI2gQc; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=endlessos.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=endlessos.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=endlessos.org header.i=@endlessos.org header.b="ARtI2gQc" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-71970655611so4861930b3a.0 for ; Tue, 01 Oct 2024 01:36:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=endlessos.org; s=google; t=1727771812; x=1728376612; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/Z8NJ9hl39zdRTB+NTLcPqploqSnWXD1+zopcYnbALw=; b=ARtI2gQcMgKPmZ9OGkN8dtW8Nal73fVZJFK6MsJZlCkDhftoLNtUfZWeJhrQ7P4U51 WkUu7HwYed0XUsPpSIWYBCN8GL5q6spwXaUdwrGxGaFwOqrabElD0U3DR4Hn1ksJD8B/ zwNbcqdx/e8Gq7ufHeWem4+/huBgFX9XXmi/blpzBcl6nJ5lUJOsMRaUwcGTayb92wEw 19CTkEaz79Pj1jCk6Uk/7WtR7jFtW7x6eT/PDBvPWOh+YVTCVKgbm2Uqa4vzYcOtMTf9 jsFAzNkT51aBJdXGxjjSbaIXNS7eGXtf6+I4VjSOvpMnpa+BA45Gk7UZUF+M+u053uE7 psJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727771812; x=1728376612; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/Z8NJ9hl39zdRTB+NTLcPqploqSnWXD1+zopcYnbALw=; b=rMYvTXohPCtbuT1OtUtIfDIwmIRfzKEYIRpRNhbmBMsV2eX0bppnJr1NGpE08ieKDY ie9THh08NrFBCD1iJmk5t4kbBFPZdxgijqpXia0neLEvG0VXDSmQ0GSeqHWcjVho8Xyi Qx0hK1Rac2s+HZ8PpWR492zzslRVX5WL7vlYytl/tduUMTMMC2a8Kg+H/7jYgoJ5dn/G jjDAqi4Yf+1UuWUiSi0rFpmzKLKKrHiwKpXxKUcSrsg54GoepyRKKBjMdfE7ZpsAfaLI 7Ykz+2gIZ/hTm6KWm+ImTUcZMqo30zXuuR6+U190wFVact4AVZvAHTyjjqu+1imJdUEo dSaQ== X-Forwarded-Encrypted: i=1; AJvYcCXH0yKwg0wD73cWgTJHa0ovHqsX9NxiehBj4rouxqXrfjAnNDz0S41OTaTb2kDQYvLNTce48seAlOY=@vger.kernel.org X-Gm-Message-State: AOJu0YxQ6Ay5muN/+PzEWC58O2xczGdoSp4viBvhI0zp42w7jxZ/te4J aaDexjazLvB1KRgN2+b16BuCVhXCDe4xPUYzq5Evx2wWsYfUZqZX+oWeWz7XD0o= X-Google-Smtp-Source: AGHT+IHicMH52afTHUITowduriNVU2SjBxq7UkdTpUK5KcNuYFt1T1q2ODACpF5e9RkCC942eEKIZw== X-Received: by 2002:a05:6a00:1ac6:b0:70d:2b95:d9c0 with SMTP id d2e1a72fcca58-71b25f6b4f2mr23654157b3a.14.1727771811736; Tue, 01 Oct 2024 01:36:51 -0700 (PDT) Received: from localhost.localdomain ([123.51.167.56]) by smtp.googlemail.com with ESMTPSA id d2e1a72fcca58-71b26516148sm7521573b3a.110.2024.10.01.01.36.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Oct 2024 01:36:51 -0700 (PDT) From: Jian-Hong Pan To: Bjorn Helgaas Cc: Johan Hovold , David Box , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Kuppuswamy Sathyanarayanan , Nirmal Patel , Jonathan Derrick , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux@endlessos.org, Jian-Hong Pan Subject: [PATCH v12 3/3] PCI/ASPM: Make pci_save_aspm_l1ss_state save both child and parent's L1SS configuration Date: Tue, 1 Oct 2024 16:34:42 +0800 Message-ID: <20241001083438.10070-8-jhp@endlessos.org> X-Mailer: git-send-email 2.46.2 In-Reply-To: <20241001083438.10070-2-jhp@endlessos.org> References: <20241001083438.10070-2-jhp@endlessos.org> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 PCI devices' parameters on the VMD bus have been programmed properly originally. But, cleared after pci_reset_bus() and have not been restored correctly. This leads the link's L1.2 between PCIe Root Port and child device gets wrong configs. Here is a failed example on ASUS B1400CEAE with enabled VMD. Both PCIe bridge and NVMe device should have the same LTR1.2_Threshold value. However, they are configured as different values in this case: 10000:e0:06.0 PCI bridge [0604]: Intel Corporation 11th Gen Core Processor PCIe Controller [8086:9a09] (rev 01) (prog-if 00 [Normal decode]) ... Capabilities: [200 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=45us PortTPowerOnTime=50us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=0ns L1SubCtl2: T_PwrOn=0us 10000:e1:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Blue SN550 NVMe SSD [15b7:5009] (rev 01) (prog-if 02 [NVM Express]) ... Capabilities: [900 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- L1_PM_Substates+ PortCommonModeRestoreTime=32us PortTPowerOnTime=10us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=101376ns L1SubCtl2: T_PwrOn=50us Here is VMD mapped PCI device tree: -+-[0000:00]-+-00.0 Intel Corporation Device 9a04 | ... \-[10000:e0]-+-06.0-[e1]----00.0 Sandisk Corp WD Blue SN550 NVMe SSD \-17.0 Intel Corporation Tiger Lake-LP SATA Controller When pci_reset_bus() resets the bus [e1] of the NVMe, it only saves and restores NVMe's state before and after reset. Then, when it restores the NVMe's state, ASPM code restores L1SS for both the parent bridge and the NVMe in pci_restore_aspm_l1ss_state(). The NVMe's L1SS is restored correctly. But, the parent bridge's L1SS is restored with a wrong value 0x0 because the parent bridge's L1SS wasn't saved by pci_save_aspm_l1ss_state() before reset. So, if the PCI device has a parent, make pci_save_aspm_l1ss_state() save the parent's L1SS configuration, too. This is symmetric on pci_restore_aspm_l1ss_state(). Link: https://lore.kernel.org/linux-pci/CAPpJ_eexU0gCHMbXw_z924WxXw0+B6SdS4eG9oGpEX1wmnMLkQ@mail.gmail.com/ Closes: https://bugzilla.kernel.org/show_bug.cgi?id=218394 Fixes: 17423360a27a ("PCI/ASPM: Save L1 PM Substates Capability for suspend/resume") Suggested-by: Ilpo Järvinen Signed-off-by: Jian-Hong Pan Reviewed-by: Ilpo Järvinen Reviewed-by: David E. Box --- v9: - Drop the v8 fix about drivers/pci/pcie/aspm.c. Use this in VMD instead. v10: - Drop the v9 fix about drivers/pci/controller/vmd.c - Fix in PCIe ASPM to make it symmetric between pci_save_aspm_l1ss_state() and pci_restore_aspm_l1ss_state() v11: - Introduce __pci_save_aspm_l1ss_state as a resusable helper function which is same as the original pci_configure_aspm_l1ss - Make pci_save_aspm_l1ss_state invoke __pci_save_aspm_l1ss_state for both child and parent devices - Smooth the commit message v12: - Update the commit message drivers/pci/pcie/aspm.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index bd0a8a05647e..17cdf372f7e0 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -79,7 +79,7 @@ void pci_configure_aspm_l1ss(struct pci_dev *pdev) ERR_PTR(rc)); } -void pci_save_aspm_l1ss_state(struct pci_dev *pdev) +static void __pci_save_aspm_l1ss_state(struct pci_dev *pdev) { struct pci_cap_saved_state *save_state; u16 l1ss = pdev->l1ss; @@ -101,6 +101,24 @@ void pci_save_aspm_l1ss_state(struct pci_dev *pdev) pci_read_config_dword(pdev, l1ss + PCI_L1SS_CTL1, cap++); } +void pci_save_aspm_l1ss_state(struct pci_dev *pdev) +{ + struct pci_dev *parent; + + __pci_save_aspm_l1ss_state(pdev); + + /* + * To be symmetric on pci_restore_aspm_l1ss_state(), save parent's L1 + * substate configuration, if the parent has not saved state. + */ + if (!pdev->bus || !pdev->bus->self) + return; + + parent = pdev->bus->self; + if (!parent->state_saved) + __pci_save_aspm_l1ss_state(parent); +} + void pci_restore_aspm_l1ss_state(struct pci_dev *pdev) { struct pci_cap_saved_state *pl_save_state, *cl_save_state;