From patchwork Tue Sep 13 13:18:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vidya Sagar X-Patchwork-Id: 12974798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A43FC54EE9 for ; Tue, 13 Sep 2022 13:18:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231922AbiIMNSr (ORCPT ); Tue, 13 Sep 2022 09:18:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49002 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230352AbiIMNSo (ORCPT ); Tue, 13 Sep 2022 09:18:44 -0400 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2087.outbound.protection.outlook.com [40.107.92.87]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 370A313DC3; Tue, 13 Sep 2022 06:18:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CjOEiaGcSjpYg6z1/FmAfk4COYY+qG61OKJD0g/sULz77ZzAZOsnng/HumDbYyp6AM11QDLZZgDht9kw6blC7HNA+TJSLrPXa8JiezbInqH7t2jSgW5Q3DX+ivYA39ZzPmcp5u9Ekgsjg3Q7GVpry45SMZf/uUn3Qzdus+eGFsaSgJ4fxDU8tgM42qnWYT4W3r48guk+Thwxwza8qD5cKtLe8TUi5GbikxS6d2Ip3QmVa9+k78KmFa70S52hmX74QtsfZoJr/cZ0gsUgO3lxtPGTNhKmd8QqaLMaVeZv7BcyRMj4KnoSCqqxwaSyPYRczaDUZourxLedWF/B5KozCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rjTTW9uCqu/TR/XcgNjJsUEmt4AGhGJtjdcDibrAMD8=; b=YcxTyHo1OGZQKU5QBhR9u9B5QRPIoEWBBqc6jheIyi4XU8//O4Bu7YzCsimUW0/Z2ueL7h1ezeNTUEIYai4b1PEcXVBhm0Y0dvg4TxRwf9KKDOKN5AajebPx4U9P/CDwAocSkZa2ewgADNtaaN2yAycJwA5Spj47lEYiqz0xaiQR6gAo1hHe1TSdQ6tzVHbdlslWq+Gmhuc39DuzUeoBMoRUPbLxCgcFIFzep164J4ta2ZhTSSj1FVFohu/I9T9WAlZAGNZlHCnG8qNEDPKbJwTP3mWWhKPcEkuIsvNfufUb197KC5X1WPTXG3pIX59ofQH89WigDP5xHgo5FqHGPg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.235) smtp.rcpttodomain=linux.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rjTTW9uCqu/TR/XcgNjJsUEmt4AGhGJtjdcDibrAMD8=; b=o+FQyXTQX1j5wklfhPNKUANY6cVnBAv8hMZm3omJdV8q2C00S43GQg7s8WEaZ6SlVye1oNhKJMD1/9hgnr4zo39jOZpS/EYHcdvPNrl8X1LZ+yDctBLGh1Xjdb+EdyqHkmdobO/5t2O9QLF8jcRJQThniqMYWL71ixcnfl+vVswAJf6sO9VKASkf+/mKbsOEwM8KXq1KtT0u9JM2iWItSNcPSHveXCNw31uEoZbVDfQhtHgQdTSHKKcHfsNrHuzWxuEaDEoeT+aWXsEkRS6V8Qedd1quRXOazLcSWoI2NN+d7SxIKYzyTdylhAs8uyeCUbABq99jfweky+mKcs02Jg== Received: from MW4PR04CA0048.namprd04.prod.outlook.com (2603:10b6:303:6a::23) by BL3PR12MB6473.namprd12.prod.outlook.com (2603:10b6:208:3b9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5612.22; Tue, 13 Sep 2022 13:18:38 +0000 Received: from CO1NAM11FT064.eop-nam11.prod.protection.outlook.com (2603:10b6:303:6a:cafe::5f) by MW4PR04CA0048.outlook.office365.com (2603:10b6:303:6a::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5612.19 via Frontend Transport; Tue, 13 Sep 2022 13:18:37 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.235) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.235 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.235; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.235) by CO1NAM11FT064.mail.protection.outlook.com (10.13.175.77) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5612.13 via Frontend Transport; Tue, 13 Sep 2022 13:18:37 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.38; Tue, 13 Sep 2022 13:18:36 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Tue, 13 Sep 2022 06:18:34 -0700 Received: from vidyas-desktop.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.986.29 via Frontend Transport; Tue, 13 Sep 2022 06:18:30 -0700 From: Vidya Sagar To: , , , , , , , , CC: , , , , , , , Subject: [PATCH V4 1/2] PCI/ASPM: Refactor ASPM L1SS control register programming Date: Tue, 13 Sep 2022 18:48:21 +0530 Message-ID: <20220913131822.16557-2-vidyas@nvidia.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220913131822.16557-1-vidyas@nvidia.com> References: <20220913131822.16557-1-vidyas@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1NAM11FT064:EE_|BL3PR12MB6473:EE_ X-MS-Office365-Filtering-Correlation-Id: 7b281b1a-7dc3-4ceb-8f5a-08da958a77bb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: IxB3frbLTHLE20xRFS0JMejxMrmjScuyCC0q3p4NUVZrj8nD3E1oFZoq9jWY8iKlSKAynaE+WVvEfJtYyNCHUi742WR9zGe8OIfhSYPrjdZAokLKBW6jl9RkMDlt9vM5RmguCQz+hvEbJjfdStve+aWl1rFykRT3VyaPlCt0nHQI15zVFcT5eXSbhzhyp70zaMvUEy+IjuuWJmn7fY42cJpejIIetrgpwFSvoilabrE2p1qeoJuSn5sj2zq4FgnnWKpcz2G4taL5Bbb3m5VmJnvp318kKhYPjHa/NujTKvPjjaiVlPNL+CmXjanafs6evXHyNcNmTB/B80ZpZe4U14ZJsqskv8vVWCK+iV0y5uGRFEFlnrCUiMA8bZWWzWxU8P67eGw/xPgXl/h3IZX2VHPOQZXxCILMfezndfDOwN/NWyj36WMaLMi0RFZvR2qbfPnuRSTwD6wb9yqam5/b9BH1TG743wfn1UpZcPZsBPJZUezpf6cbNoocyi68kqw6L8ZLpoSvjc9owBGmuurfURk0vEfPW0FaHx0Ja3bDt47M1wUW1tGZIdXeHcqt77fY6ugOm7K5rQEqa7X60jv9eusI5EUxkT6XWXKN9HczotnvHGflKPm3uV63F9SXAoErXVWRPb1saiiicQR7vR4N3sVG85HvsYvsBPwtsbPGzHxFhcp2MSWQgKMQO9qtHbaEXAVzD4mBLPmt17zGC0qfrfPZSRQR8Cf4nS77HFimpWUc8cygITvynU1GRaZ4T0t6TFhShjDs+MBghATTk57fd5aycRjVrc7UwhWkL79Tr+I5gXqoSN2mBeL1RcvcxJ6q1NIKPRLAh/NEvrcLoRTkBJYlm4OTU03WhHv7tEN7bTfvrc7C84ducnmIAW7EtpFk X-Forefront-Antispam-Report: CIP:12.22.5.235;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:InfoNoRecords;CAT:NONE;SFS:(13230022)(4636009)(346002)(39860400002)(396003)(136003)(376002)(451199015)(46966006)(40470700004)(36840700001)(7696005)(26005)(70206006)(186003)(4326008)(8676002)(83380400001)(316002)(7416002)(40460700003)(47076005)(110136005)(426003)(1076003)(356005)(70586007)(336012)(478600001)(5660300002)(41300700001)(2906002)(82310400005)(36756003)(81166007)(40480700001)(2616005)(6666004)(86362001)(6636002)(8936002)(82740400003)(54906003)(36860700001)(32563001)(36900700001)(2101003)(473944003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Sep 2022 13:18:37.1902 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7b281b1a-7dc3-4ceb-8f5a-08da958a77bb X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[12.22.5.235];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT064.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR12MB6473 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Refactor the code to extract the command code out to program Control Registers-1 & 2 of L1 Sub-States capability to a new function aspm_program_l1ss() and call it for both parent and child devices. Signed-off-by: Vidya Sagar --- V4: * New patch in this series drivers/pci/pcie/aspm.c | 63 +++++++++++++++++++---------------------- 1 file changed, 29 insertions(+), 34 deletions(-) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index a8aec190986c..ecbe3af4188d 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -455,6 +455,31 @@ static void pci_clear_and_set_dword(struct pci_dev *pdev, int pos, pci_write_config_dword(pdev, pos, val); } +static void aspm_program_l1ss(struct pci_dev *dev, u32 ctl1, u32 ctl2) +{ + u16 l1ss = dev->l1ss; + u32 l1_2_enable; + + /* + * Per PCIe r6.0, sec 5.5.4, T_POWER_ON in PCI_L1SS_CTL2 must be + * programmed prior to setting the L1.2 enable bits in PCI_L1SS_CTL1. + */ + pci_write_config_dword(dev, l1ss + PCI_L1SS_CTL2, ctl2); + + /* + * In addition, Common_Mode_Restore_Time and LTR_L1.2_THRESHOLD in + * PCI_L1SS_CTL1 must be programmed *before* setting the L1.2 + * enable bits, even though they're all in PCI_L1SS_CTL1. + */ + l1_2_enable = ctl1 & PCI_L1SS_CTL1_L1_2_MASK; + ctl1 &= ~PCI_L1SS_CTL1_L1_2_MASK; + + pci_write_config_dword(dev, l1ss + PCI_L1SS_CTL1, ctl1); + if (l1_2_enable) + pci_write_config_dword(dev, l1ss + PCI_L1SS_CTL1, + ctl1 | l1_2_enable); +} + /* Calculate L1.2 PM substate timing parameters */ static void aspm_calc_l1ss_info(struct pcie_link_state *link, u32 parent_l1ss_cap, u32 child_l1ss_cap) @@ -464,7 +489,6 @@ static void aspm_calc_l1ss_info(struct pcie_link_state *link, u32 t_common_mode, t_power_on, l1_2_threshold, scale, value; u32 ctl1 = 0, ctl2 = 0; u32 pctl1, pctl2, cctl1, cctl2; - u32 pl1_2_enables, cl1_2_enables; if (!(link->aspm_support & ASPM_STATE_L1_2_MASK)) return; @@ -513,39 +537,10 @@ static void aspm_calc_l1ss_info(struct pcie_link_state *link, ctl2 == pctl2 && ctl2 == cctl2) return; - /* Disable L1.2 while updating. See PCIe r5.0, sec 5.5.4, 7.8.3.3 */ - pl1_2_enables = pctl1 & PCI_L1SS_CTL1_L1_2_MASK; - cl1_2_enables = cctl1 & PCI_L1SS_CTL1_L1_2_MASK; - - if (pl1_2_enables || cl1_2_enables) { - pci_clear_and_set_dword(child, child->l1ss + PCI_L1SS_CTL1, - PCI_L1SS_CTL1_L1_2_MASK, 0); - pci_clear_and_set_dword(parent, parent->l1ss + PCI_L1SS_CTL1, - PCI_L1SS_CTL1_L1_2_MASK, 0); - } - - /* Program T_POWER_ON times in both ports */ - pci_write_config_dword(parent, parent->l1ss + PCI_L1SS_CTL2, ctl2); - pci_write_config_dword(child, child->l1ss + PCI_L1SS_CTL2, ctl2); - - /* Program Common_Mode_Restore_Time in upstream device */ - pci_clear_and_set_dword(parent, parent->l1ss + PCI_L1SS_CTL1, - PCI_L1SS_CTL1_CM_RESTORE_TIME, ctl1); - - /* Program LTR_L1.2_THRESHOLD time in both ports */ - pci_clear_and_set_dword(parent, parent->l1ss + PCI_L1SS_CTL1, - PCI_L1SS_CTL1_LTR_L12_TH_VALUE | - PCI_L1SS_CTL1_LTR_L12_TH_SCALE, ctl1); - pci_clear_and_set_dword(child, child->l1ss + PCI_L1SS_CTL1, - PCI_L1SS_CTL1_LTR_L12_TH_VALUE | - PCI_L1SS_CTL1_LTR_L12_TH_SCALE, ctl1); - - if (pl1_2_enables || cl1_2_enables) { - pci_clear_and_set_dword(parent, parent->l1ss + PCI_L1SS_CTL1, 0, - pl1_2_enables); - pci_clear_and_set_dword(child, child->l1ss + PCI_L1SS_CTL1, 0, - cl1_2_enables); - } + aspm_program_l1ss(parent, + ctl1 | (pctl1 & PCI_L1SS_CTL1_L1_2_MASK), ctl2); + aspm_program_l1ss(child, + ctl1 | (cctl1 & PCI_L1SS_CTL1_L1_2_MASK), ctl2); } static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist) From patchwork Tue Sep 13 13:18:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vidya Sagar X-Patchwork-Id: 12974799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 043D1C6FA89 for ; Tue, 13 Sep 2022 13:18:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232005AbiIMNSt (ORCPT ); Tue, 13 Sep 2022 09:18:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49062 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231935AbiIMNSr (ORCPT ); Tue, 13 Sep 2022 09:18:47 -0400 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2086.outbound.protection.outlook.com [40.107.94.86]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D452F1B794; Tue, 13 Sep 2022 06:18:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nIYikmm8rRxBsDQGmvKdBtjPpbMSNhe0hg9b0CJ45flCAywtMwI6OmwQirMDxt19mzfnBwr8ak38yqpAiOUwImg1yVasXNZJTSXC+2FxJS71Zh6kmr6PWSOw572Wv1f2ey+oBfBEpZPcGdyVthXOnnrzn76CAjivuc/bDspILZLEsfvpmrwsOBc95en+bF1GH53+t1/2AVWWRbjIImp05v9ZF1izr7rhwDn48UE6IUNRlcHyan2xD5z+xv4ykHe2RIdXY/buW2Rl0GcyYjJTiwKbwtywJbPktnqDIJjzb0IALSxv1C3iTD2+dnG9Enrz9qAYOuk802xxwS3ASJqDlQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=P9SjNsXe/a7WO6rUvFjyaMd6e8+9yOGsJhR12RYydNU=; b=AO0QgLkOrqJbsS1ivynzCFSAVeIySvLt1SjyWBwtn1pgW6T9+U0E8Aa2IYgFScTeaB+itqBGFsCfdjS/9ukTE8okZHtJbFgI5g1crKspXwBPMk69ReUMn5GBhuwlOHdaYOokq/D8f7frj/PcPrN9yvjiBsZbMZ21HUTVcQ6iq/8uFuMXvcIGxCDobCoQVj3xnJIuBk9hXeC5Ar62mmQySAh/6ShCnQ6u8C5evqRgWIOkbsLgpfyAvgafSr+SNQP1f5+SHAl7GIl9QPWOIb0y3J24h5Uel1aSDzQInUJ8hucTQFakkkY/Qno466tYVDcn69sG0V7hlN/bNcxfdUIWtg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.238) smtp.rcpttodomain=linux.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P9SjNsXe/a7WO6rUvFjyaMd6e8+9yOGsJhR12RYydNU=; b=XyzxofOe78ErfXZSyj19bgjmkb6Z4b6w4eoj3R+YEeRLRWq4ZX8MYkb6o2sRT65QqaKZG4YnQWd+Kkwgickv7m9kwzoT4x+rOF6k7EKKWzXsuCGAtCK8YViSQ1lmAj+uDoemTOysdy76dBU0yXuGuzkhpuQh2bB4biT+QIV/4ifBLxz7weXYCsuj3dlCSd51Nb077yOV8Byk8DYeWE9qZLgDhPcYGPvwVs3aATCD5g3k6ocC6E0s9A8B8GRqqWY52lR1f2der8KHonkIrLfF6IYHTpzoMaYzbXM3WdsLe06+1ho5zx2R49DSmvCWxCcJ2jrehpRH3rp8C4idjpogDA== Received: from BN9PR03CA0314.namprd03.prod.outlook.com (2603:10b6:408:112::19) by DM8PR12MB5462.namprd12.prod.outlook.com (2603:10b6:8:24::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5612.19; Tue, 13 Sep 2022 13:18:42 +0000 Received: from BL02EPF0000C405.namprd05.prod.outlook.com (2603:10b6:408:112:cafe::97) by BN9PR03CA0314.outlook.office365.com (2603:10b6:408:112::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5612.14 via Frontend Transport; Tue, 13 Sep 2022 13:18:42 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.238) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.238 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.238; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.238) by BL02EPF0000C405.mail.protection.outlook.com (10.167.241.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5612.12 via Frontend Transport; Tue, 13 Sep 2022 13:18:42 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by DRHQMAIL105.nvidia.com (10.27.9.14) with Microsoft SMTP Server (TLS) id 15.0.1497.38; Tue, 13 Sep 2022 13:18:41 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Tue, 13 Sep 2022 06:18:40 -0700 Received: from vidyas-desktop.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.986.29 via Frontend Transport; Tue, 13 Sep 2022 06:18:36 -0700 From: Vidya Sagar To: , , , , , , , , CC: , , , , , , , Subject: [PATCH V4 2/2] PCI/ASPM: Save/restore L1SS Capability for suspend/resume Date: Tue, 13 Sep 2022 18:48:22 +0530 Message-ID: <20220913131822.16557-3-vidyas@nvidia.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220913131822.16557-1-vidyas@nvidia.com> References: <20220913131822.16557-1-vidyas@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0000C405:EE_|DM8PR12MB5462:EE_ X-MS-Office365-Filtering-Correlation-Id: f972e8dd-8a04-4c6f-d6d4-08da958a7acc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: W33o6MojczfPAdqof8ts+OzCZxvjzG4ctCz1MXxbimayiYN9+gcdcZYEdAy5GSG9I1AMwyTzL141AEgZjn4Jne0q3EcE/TpseWwFpbSWlSINcckQ48fMFLBQjwDbORP6WN2MvCjq3Vezc3gNfns2pBPZrAcV7tnIaT0otu8EAjeVXkgXEDVZ72iLaO0N2iZHuCBcQoYdYWJYqen5M8doR564asO4mC2pCRk30dV/JMkwpU1rC5oryOQJaqTmXCS4X+Ne/sE9x32/xghPm836K9vvbIDErfFF72qliQlGlvfBMP2qy36peUCCRng/fzi1Gz89frxEBwB9zWRwLyt8BzN3zlFvWHGJXQp/fvVVdPDZ3k1wPb5E4STWARARuoyjeoIeRQWdoVT2S2U7EX81OXWmb9E+NW2MeMLIwRPHil44koSVjlvWlTQPTYOy2cafXQ+IqCwvwNjQM8ygVPlOBcrAnaZiN35aV0DMa8lM7tAzddxiVtE1lWt/dxtIOajXkyBdprpo29abCFuSM3dRWcm6QydosCHOP5RPbSuvP8S6a8qf1POkyOjgrGsuWUwNo36EChUWbd/ycnlUU/3D1MsEwDcohvAKJAh97iLGfvpV3Ns8sc9wiNivqHoChkPzxalPkgg95RcRKn3JgtLw8OxnvOvHwPCEmf1ekBte52vHIkcmmkrYZYI11blTOUSVmnZMyau3tKUhh2V+66x6ZJykh/UKQgdmYx7UEAQD7hnM9iLVGrt7Exa8EtdMHuQNQ/xKWIrRIg8NEh3OLMaP4QoUHgDgNyxZO2rMWrZPL1Jh7MgAOrNvHAgd04tcM4/WMd9ojqzA/NMYHFxR9LB4X36+A8esDehIVF5dE76Y9AzxBotxtqmOJVWqUnjUTgsq X-Forefront-Antispam-Report: CIP:12.22.5.238;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:InfoNoRecords;CAT:NONE;SFS:(13230022)(4636009)(346002)(136003)(39860400002)(376002)(396003)(451199015)(36840700001)(46966006)(40470700004)(70586007)(40460700003)(478600001)(6636002)(6666004)(26005)(36860700001)(8676002)(110136005)(316002)(2616005)(81166007)(54906003)(40480700001)(7696005)(186003)(8936002)(70206006)(426003)(83380400001)(356005)(36756003)(15650500001)(86362001)(1076003)(7416002)(41300700001)(4326008)(2906002)(5660300002)(336012)(82310400005)(82740400003)(47076005)(32563001)(2101003)(36900700001)(473944003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Sep 2022 13:18:42.2741 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f972e8dd-8a04-4c6f-d6d4-08da958a7acc X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[12.22.5.238];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0000C405.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM8PR12MB5462 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Previously ASPM L1 Substates control registers (CTL1 and CTL2) weren't saved and restored during suspend/resume leading to L1 Substates configuration being lost post-resume. Save the L1 Substates control registers so that the configuration is retained post-resume. Signed-off-by: Vidya Sagar --- V4: * Address review comments from Bjorn * Use the API aspm_program_l1ss() to restore L1SS registers V3: * Disabled L1.2 enable fields while restoring Control-1 register drivers/pci/pci.c | 7 +++++++ drivers/pci/pci.h | 4 ++++ drivers/pci/pcie/aspm.c | 43 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 54 insertions(+) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 95bc329e74c0..68a49fbaabde 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1663,6 +1663,7 @@ int pci_save_state(struct pci_dev *dev) return i; pci_save_ltr_state(dev); + pci_save_aspm_l1ss_state(dev); pci_save_dpc_state(dev); pci_save_aer_state(dev); pci_save_ptm_state(dev); @@ -1769,6 +1770,7 @@ void pci_restore_state(struct pci_dev *dev) * LTR itself (in the PCIe capability). */ pci_restore_ltr_state(dev); + pci_restore_aspm_l1ss_state(dev); pci_restore_pcie_state(dev); pci_restore_pasid_state(dev); @@ -3485,6 +3487,11 @@ void pci_allocate_cap_save_buffers(struct pci_dev *dev) if (error) pci_err(dev, "unable to allocate suspend buffer for LTR\n"); + error = pci_add_ext_cap_save_buffer(dev, PCI_EXT_CAP_ID_L1SS, + 2 * sizeof(u32)); + if (error) + pci_err(dev, "unable to allocate suspend buffer for ASPM-L1SS\n"); + pci_allocate_vc_save_buffers(dev); } diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 785f31086313..365a844ec430 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -561,10 +561,14 @@ bool pcie_wait_for_link(struct pci_dev *pdev, bool active); void pcie_aspm_init_link_state(struct pci_dev *pdev); void pcie_aspm_exit_link_state(struct pci_dev *pdev); void pcie_aspm_powersave_config_link(struct pci_dev *pdev); +void pci_save_aspm_l1ss_state(struct pci_dev *dev); +void pci_restore_aspm_l1ss_state(struct pci_dev *dev); #else static inline void pcie_aspm_init_link_state(struct pci_dev *pdev) { } static inline void pcie_aspm_exit_link_state(struct pci_dev *pdev) { } static inline void pcie_aspm_powersave_config_link(struct pci_dev *pdev) { } +static inline void pci_save_aspm_l1ss_state(struct pci_dev *dev) { } +static inline void pci_restore_aspm_l1ss_state(struct pci_dev *dev) { } #endif #ifdef CONFIG_PCIE_ECRC diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index ecbe3af4188d..dc2e21c7a9d4 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -721,6 +721,49 @@ static void pcie_config_aspm_l1ss(struct pcie_link_state *link, u32 state) PCI_L1SS_CTL1_L1SS_MASK, val); } +void pci_save_aspm_l1ss_state(struct pci_dev *dev) +{ + struct pci_cap_saved_state *save_state; + u16 l1ss = dev->l1ss; + u32 *cap; + + if (!pci_is_pcie(dev)) + return; + + if (!l1ss) + return; + + save_state = pci_find_saved_ext_cap(dev, PCI_EXT_CAP_ID_L1SS); + if (!save_state) + return; + + cap = (u32 *)&save_state->cap.data[0]; + pci_read_config_dword(dev, l1ss + PCI_L1SS_CTL2, cap++); + pci_read_config_dword(dev, l1ss + PCI_L1SS_CTL1, cap++); +} + +void pci_restore_aspm_l1ss_state(struct pci_dev *dev) +{ + struct pci_cap_saved_state *save_state; + u32 *cap, ctl1, ctl2; + u16 l1ss = dev->l1ss; + + if (!pci_is_pcie(dev)) + return; + + if (!l1ss) + return; + + save_state = pci_find_saved_ext_cap(dev, PCI_EXT_CAP_ID_L1SS); + if (!save_state) + return; + + cap = (u32 *)&save_state->cap.data[0]; + ctl2 = *cap++; + ctl1 = *cap; + aspm_program_l1ss(dev, ctl1, ctl2); +} + static void pcie_config_aspm_dev(struct pci_dev *pdev, u32 val) { pcie_capability_clear_and_set_word(pdev, PCI_EXP_LNKCTL,