From patchwork Sat Oct 24 20:55:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Ian Kumlien X-Patchwork-Id: 11854981 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 83B9614B4 for ; Sat, 24 Oct 2020 20:55:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 62DC621D41 for ; Sat, 24 Oct 2020 20:55:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mUSZaQyb" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1764537AbgJXUz4 (ORCPT ); Sat, 24 Oct 2020 16:55:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55964 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1764536AbgJXUz4 (ORCPT ); Sat, 24 Oct 2020 16:55:56 -0400 Received: from mail-lj1-x243.google.com (mail-lj1-x243.google.com [IPv6:2a00:1450:4864:20::243]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6BA7C0613CE for ; Sat, 24 Oct 2020 13:55:55 -0700 (PDT) Received: by mail-lj1-x243.google.com with SMTP id h20so5480046lji.9 for ; Sat, 24 Oct 2020 13:55:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8hyL9oNr8H/MDAXfwjeewU2xEG5E9TY5x+M8oCEvG88=; b=mUSZaQybR6UIq1VitNgzkASF++k0yN3aMCELYEDG+w45aggzWT+woJS3NjR+bk0+bl E9bXnbbUieWNBL7lVNNYxjmDpQ+JQUXcXOJYzbOh+O6GoYpx+VgDjv8eC/Jr2B2bdwTI vJt3XLS4Uw4XL3uxZsp6X/iDvoETYWEyK6uDfFaCWdujf4LD2Dwspmr1UXNH4jDyvETs ljI5OWerrGdlL1/8HAj7JZxESchMHafIrdBiVRnv5L15S5oJFOYxao09Orz3GmJ+OV1T wUqNuadCtjGRtun27AVDG9AAjJ88mhy/+cilpJUwOVf0QaaPZFjK1w5SBAqzyIWVXOfD e9jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8hyL9oNr8H/MDAXfwjeewU2xEG5E9TY5x+M8oCEvG88=; b=NfDYuQ46UWPpvfJ7h10C8I1hxDNbAxcz6a/L6NFiV/WqTFvQMfKz4tb07RbIAVSRPg 8TD0/p+JjiPeN4tXYXC6joP4aRNpSUlGV0pC5Z4EHJeUMtfWsL1ADPD/hkJbYKXgtSuH dbajQho6qVGnA6t7bd76LQOk2jzCYn4kbXKA4bHP6BNQcML6RqWAXf2wNDld6wAjMU0R N+PJkDtlHXLixkfvxROppsU5hSuqzITW1wbTkc6oSxByLZ3smHYRKncH0C0OblhEnawi 7U4lAlYdtntuBIsDx02DJj/nJcsXBr6YWvsHvvUQh8f/mAPShsl+dQuoFJOiHm0kzkbj Zecg== X-Gm-Message-State: AOAM530YFHwYIqdvYNhQIWW53fH/A8gJaa/ENaM+TOPPXwZ3YEgLiSiW F+Wu/h2Vwn0PGhYAzAJma3g= X-Google-Smtp-Source: ABdhPJzmtA7uXURgTxQlKNKVHborymXpt50XD4X7vRXsetRjIMPeiZcCZ+c+czDztfgPlirB4pgLeQ== X-Received: by 2002:a2e:8184:: with SMTP id e4mr3234602ljg.383.1603572954207; Sat, 24 Oct 2020 13:55:54 -0700 (PDT) Received: from octa.pomac.com (c-f9c8225c.013-195-6c756e10.bbcust.telenor.se. [92.34.200.249]) by smtp.gmail.com with ESMTPSA id f26sm246815lfc.302.2020.10.24.13.55.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 24 Oct 2020 13:55:53 -0700 (PDT) From: Ian Kumlien To: kai.heng.feng@canonical.com, linux-pci@vger.kernel.org, alexander.duyck@gmail.com, refactormyself@gmail.com, puranjay12@gmail.com Cc: Ian Kumlien Subject: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check Date: Sat, 24 Oct 2020 22:55:46 +0200 Message-Id: <20201024205548.1837770-1-ian.kumlien@gmail.com> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201022183030.GA513862@bjorn-Precision-5520> References: <20201022183030.GA513862@bjorn-Precision-5520> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Make pcie_aspm_check_latency comply with the PCIe spec, specifically: "5.4.1.2.2. Exit from the L1 State" Which makes it clear that each switch is required to initiate a transition within 1μs from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1. The current code doesn't take the maximum latency into account. From the example: +----------------+ | | | Root complex | | | | +-----+ | | |32 μs| | +----------------+ | | Link 1 | +----------------+ | |8 μs| | | +----+ | | Switch A | | +----+ | | |8 μs| | +----------------+ | | Link 2 | +----------------+ | |32 μs| | | +-----+ | | Switch B | | +-----+ | | |32 μs| | +----------------+ | | Link 3 | +----------------+ | |8μs| | | +---+ | | Endpoint C | | | | | +----------------+ Links 1, 2 and 3 are all in L1 state - endpoint C initiates the transition to L0 at time T. Since switch B takes 32 μs to exit L1 on it's ports, Link 3 will transition to L0 at T+32 (longest time considering T+8 for endpoint C and T+32 for switch B). Switch B is required to initiate a transition from the L1 state on it's upstream port after no more than 1 μs from the beginning of the transition from L1 state on the downstream port. Therefore, transition from L1 to L0 will begin on link 2 at T+1, this will cascade up the path. The path will exit L1 at T+34. On my specific system: 03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a) Exit latency Acceptable latency Tree: L1 L0s L1 L0s ---------- ------- ----- ------- ------ 00:01.2 <32 us - | 01:00.0 <32 us - |- 02:03.0 <32 us - | \03:00.0 <16 us <2us <64 us <512ns | \- 02:04.0 <32 us - \04:00.0 <64 us unlimited <64 us <512ns 04:00.0's latency is the same as the maximum it allows so as we walk the path the first switchs startup latency will pass the acceptable latency limit for the link, and as a side-effect it fixes my issues with 03:00.0. Without this patch, 03:00.0 misbehaves and only gives me ~40 mbit/s over links with 6 or more hops. With this patch I'm back to a maximum of ~933 mbit/s. The original code path did: 04:00:0-02:04.0 max latency 64 -> ok 02:04.0-01:00.0 max latency 32 +1 -> ok 01:00.0-00:01.2 max latency 32 +2 -> ok And thus didn't see any L1 ASPM latency issues. The new code does: 04:00:0-02:04.0 max latency 64 -> ok 02:04.0-01:00.0 max latency 64 +1 -> latency exceeded 01:00.0-00:01.2 max latency 64 +2 -> latency exceeded It correctly identifies the issue. For reference, pcie information: https://bugzilla.kernel.org/show_bug.cgi?id=209725 Kai-Heng Feng has a machine that will not boot with ASPM without this patch, information is documented here: https://bugzilla.kernel.org/show_bug.cgi?id=209671 Signed-off-by: Ian Kumlien Tested-by: Kai-Heng Feng --- drivers/pci/pcie/aspm.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index 253c30cc1967..c03ead0f1013 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev, static void pcie_aspm_check_latency(struct pci_dev *endpoint) { - u32 latency, l1_switch_latency = 0; + u32 latency, l1_max_latency = 0, l1_switch_latency = 0; struct aspm_latency *acceptable; struct pcie_link_state *link; @@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint) if ((link->aspm_capable & ASPM_STATE_L0S_DW) && (link->latency_dw.l0s > acceptable->l0s)) link->aspm_capable &= ~ASPM_STATE_L0S_DW; + /* * Check L1 latency. - * Every switch on the path to root complex need 1 - * more microsecond for L1. Spec doesn't mention L0s. + * + * PCIe r5.0, sec 5.4.1.2.2 states: + * A Switch is required to initiate an L1 exit transition on its + * Upstream Port Link after no more than 1 μs from the beginning of an + * L1 exit transition on any of its Downstream Port Links. * * The exit latencies for L1 substates are not advertised * by a device. Since the spec also doesn't mention a way @@ -469,11 +473,13 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint) * L1 exit latencies advertised by a device include L1 * substate latencies (and hence do not do any check). */ - latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1); - if ((link->aspm_capable & ASPM_STATE_L1) && - (latency + l1_switch_latency > acceptable->l1)) - link->aspm_capable &= ~ASPM_STATE_L1; - l1_switch_latency += 1000; + if (link->aspm_capable & ASPM_STATE_L1) { + latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1); + l1_max_latency = max_t(u32, latency, l1_max_latency); + if (l1_max_latency + l1_switch_latency > acceptable->l1) + link->aspm_capable &= ~ASPM_STATE_L1; + l1_switch_latency += 1000; + } link = link->parent; } From patchwork Sat Oct 24 20:55:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Kumlien X-Patchwork-Id: 11854983 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2816392C for ; Sat, 24 Oct 2020 20:56:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0AED722269 for ; Sat, 24 Oct 2020 20:56:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hp7r/Hx2" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1764538AbgJXU4A (ORCPT ); Sat, 24 Oct 2020 16:56:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55974 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1764536AbgJXU4A (ORCPT ); Sat, 24 Oct 2020 16:56:00 -0400 Received: from mail-lf1-x141.google.com (mail-lf1-x141.google.com [IPv6:2a00:1450:4864:20::141]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A9076C0613CE for ; Sat, 24 Oct 2020 13:55:59 -0700 (PDT) Received: by mail-lf1-x141.google.com with SMTP id 77so6709180lfl.2 for ; Sat, 24 Oct 2020 13:55:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kQSwDx/8ku9V+K5hldQkcp4a98iVQPx94jQsc+N/fFY=; b=hp7r/Hx2x7+Z9Y50FJ7M5wmV+nlF2f49Ma6L6SWmZY5/Dl5G9wR0HxsNc5X+1KEQSS PuENmVrJtxnlOjDs9xnEJl4OgjFnvKsi6ttjWrYTdhPAO7qJrO5NK9Upaee6ypP7r/2l YOU4r01avoU4AWuuLQlXvekHj/gXTCI00mCZ3xrd8QGcvjE+NQGT5s6mRf/VhRiFJUWJ NiDnj+4IHY4SQCsS9G3vpT+QFBCBx/OslFHBArd0BD4S51CyYhowwEcj83aUFoPEbeYx NoJByQ45L04BKehzOTawHBOKPrL7/4yF9Otw0TBkPwWop8ecTdCkh4/tcN3atzmOeuAu R9Mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kQSwDx/8ku9V+K5hldQkcp4a98iVQPx94jQsc+N/fFY=; b=r1NfBQnGsMW5x0ScQww2DM+UGkRl5d4Q0mbQ8CrZ5uBCpZNUSPAPwPAuQah9cUD5l+ /Vc4Li3YjUDTby/trgJeDxTHm5GOgy1Hd4Ue+nTyv8CQ+LMkAbxnvx/QqjNGab2n7Vup nhEw1rL0A+8/ZX43+1xVq3HOLVEX2Sl4lO3qYX6Jr6RLXyW8byCUo/cXONw6P7OwfXsu M85s6wKwEnqPeSeiC2VQqjU1B3lB3Dq+X12yJY3MnaWrb6fQ1IIJkJ7sCJ1zL5JFAvRk zh7jbqub6sIgkmL5fe5VJU4CTSggJxj14pbeH6skFlYE7eoimTZgEo5uauDjI6tVj656 cruw== X-Gm-Message-State: AOAM532gD4lRZRryWGgIs7IKJW3A3i9RmHhVEFKpYYCktkwkUMb7jjVT mlRhxNafQMVYo4Z0PCDaf6Co8Uul1NxtMA== X-Google-Smtp-Source: ABdhPJyosCRFQCL7wCIuidxQAwzVFjn6I2pbllpwD336aznIoG307ytSgQWf4jtUW41/pmn4huanhA== X-Received: by 2002:a05:6512:786:: with SMTP id x6mr2925906lfr.550.1603572958124; Sat, 24 Oct 2020 13:55:58 -0700 (PDT) Received: from octa.pomac.com (c-f9c8225c.013-195-6c756e10.bbcust.telenor.se. [92.34.200.249]) by smtp.gmail.com with ESMTPSA id f26sm246815lfc.302.2020.10.24.13.55.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 24 Oct 2020 13:55:57 -0700 (PDT) From: Ian Kumlien To: kai.heng.feng@canonical.com, linux-pci@vger.kernel.org, alexander.duyck@gmail.com, refactormyself@gmail.com, puranjay12@gmail.com Cc: Ian Kumlien Subject: [PATCH 2/3] PCI/ASPM: Fix L0s max latency check Date: Sat, 24 Oct 2020 22:55:47 +0200 Message-Id: <20201024205548.1837770-2-ian.kumlien@gmail.com> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201024205548.1837770-1-ian.kumlien@gmail.com> References: <20201022183030.GA513862@bjorn-Precision-5520> <20201024205548.1837770-1-ian.kumlien@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From what I have been able to figure out, it seems like LOs path latency is cumulative, so the max path latency should be the sum of all links maximum latency. Signed-off-by: Ian Kumlien --- drivers/pci/pcie/aspm.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index c03ead0f1013..dbe3ce60c1ff 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -434,7 +434,8 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev, static void pcie_aspm_check_latency(struct pci_dev *endpoint) { - u32 latency, l1_max_latency = 0, l1_switch_latency = 0; + u32 latency, l1_max_latency = 0, l1_switch_latency = 0, + l0s_latency_up = 0, l0s_latency_dw = 0; struct aspm_latency *acceptable; struct pcie_link_state *link; @@ -448,14 +449,18 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint) while (link) { /* Check upstream direction L0s latency */ - if ((link->aspm_capable & ASPM_STATE_L0S_UP) && - (link->latency_up.l0s > acceptable->l0s)) - link->aspm_capable &= ~ASPM_STATE_L0S_UP; + if (link->aspm_capable & ASPM_STATE_L0S_UP) { + l0s_latency_up += link->latency_up.l0s; + if (l0s_latency_up > acceptable->l0s) + link->aspm_capable &= ~ASPM_STATE_L0S_UP; + } /* Check downstream direction L0s latency */ - if ((link->aspm_capable & ASPM_STATE_L0S_DW) && - (link->latency_dw.l0s > acceptable->l0s)) - link->aspm_capable &= ~ASPM_STATE_L0S_DW; + if (link->aspm_capable & ASPM_STATE_L0S_DW) { + l0s_latency_dw += link->latency_dw.l0s; + if (l0s_latency_dw > acceptable->l0s) + link->aspm_capable &= ~ASPM_STATE_L0S_DW; + } /* * Check L1 latency. From patchwork Sat Oct 24 20:55:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Kumlien X-Patchwork-Id: 11854985 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3918D92C for ; Sat, 24 Oct 2020 20:56:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1CA2722276 for ; Sat, 24 Oct 2020 20:56:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kksQsO+Y" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1764539AbgJXU4T (ORCPT ); Sat, 24 Oct 2020 16:56:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1764536AbgJXU4T (ORCPT ); Sat, 24 Oct 2020 16:56:19 -0400 Received: from mail-lj1-x241.google.com (mail-lj1-x241.google.com [IPv6:2a00:1450:4864:20::241]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6391BC0613CE for ; Sat, 24 Oct 2020 13:56:03 -0700 (PDT) Received: by mail-lj1-x241.google.com with SMTP id i2so5472947ljg.4 for ; Sat, 24 Oct 2020 13:56:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xZT+5FZe1JJratYtVvddwvaIyTNrCamHP4CwNxkpWBg=; b=kksQsO+YLUDQhdI6/GTYGjAMlsX+XhJmP9nmjr55rRr8wFUGaT8J6R29s3rLdnoX0Y Leijw7ntpumAcSwf+O+jEqqIa9ESP9ZhUgEHUtMsJS1H58uuhhtvEyLULSeLAB9wCNCL zgsEwiqk+vzNcRtB5SRvpl4IXUbRFYTzXXoyQMBy6dj9qGsJjR6D4ngiuo77wWtFHZar O73LHWPOReO070ZqnEBjiiXt4e2rj8+u8T0pgK8fuN1iTx3E3VwoaPbzYPuonX8xGxtL pNVCfe4NRb0Y9JTHG144WaFdMHauoDVaKuT0+vvTk9U2DWKOhkZ6/qdGOztPJ0L/TOU1 6voQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xZT+5FZe1JJratYtVvddwvaIyTNrCamHP4CwNxkpWBg=; b=ebJvZqcnxnGr/B2WLY5ZDYoLJjS4pqa7gsCOjkLt+j3C4PoM03HsFJaVVEr1N7QEsY WRqWfAzB/z/AOouusMWiFd75uUCFrxe2zlcJMolhtNmUMPU6TY4W/8+2Tr8FjmicntPi sq/yV6vDkYX9J+fKl6MsDz8qNf643AFvWWXZkVTibsQNJtigPuR8M8QPbGaoo/Z1EDpX UZYkprN2thPlgCt0wSxChC1IWkuQ++yUkW8ZSNLSPdCRmbfhSC6tHtjW0yHN2s1vKrkl AcMScxnd4lxKwFHAVZsIqc2EfGLVyhZ7rZmj5DypXsJ3Zd/4UeKni2DT6mVDB7vdiq/y CtmQ== X-Gm-Message-State: AOAM531Vzj6+REpsmvkPpsPsjqTK1hzQg4pFHlXsLO6ns2HsN7hFAkRe bK0UsIp5sw2RIieWCWKjuZ4= X-Google-Smtp-Source: ABdhPJzViab6/TRRJFUoCbURv6LeDvTlYJVhQGMCpzlZ+mt7l/aOfLSN/JWNCSlg0iJ/CIWO7AzAcQ== X-Received: by 2002:a2e:9905:: with SMTP id v5mr2965374lji.222.1603572961852; Sat, 24 Oct 2020 13:56:01 -0700 (PDT) Received: from octa.pomac.com (c-f9c8225c.013-195-6c756e10.bbcust.telenor.se. [92.34.200.249]) by smtp.gmail.com with ESMTPSA id f26sm246815lfc.302.2020.10.24.13.56.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 24 Oct 2020 13:56:01 -0700 (PDT) From: Ian Kumlien To: kai.heng.feng@canonical.com, linux-pci@vger.kernel.org, alexander.duyck@gmail.com, refactormyself@gmail.com, puranjay12@gmail.com Cc: Ian Kumlien Subject: [PATCH 3/3] [RFC] PCI/ASPM: Print L1/L0s latency messages per endpoint Date: Sat, 24 Oct 2020 22:55:48 +0200 Message-Id: <20201024205548.1837770-3-ian.kumlien@gmail.com> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201024205548.1837770-1-ian.kumlien@gmail.com> References: <20201022183030.GA513862@bjorn-Precision-5520> <20201024205548.1837770-1-ian.kumlien@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Print information on where along the path we disable L1 or L0s per endpoint. I think the infromation could be useful in the normal dmesg. Signed-off-by: Ian Kumlien --- drivers/pci/pcie/aspm.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index dbe3ce60c1ff..8bec24119f3e 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -436,6 +436,7 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint) { u32 latency, l1_max_latency = 0, l1_switch_latency = 0, l0s_latency_up = 0, l0s_latency_dw = 0; + bool aspm_disable = 0; struct aspm_latency *acceptable; struct pcie_link_state *link; @@ -447,19 +448,35 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint) link = endpoint->bus->self->link_state; acceptable = &link->acceptable[PCI_FUNC(endpoint->devfn)]; +#define aspm_info(device, type, down, up) \ + if (!aspm_disable) \ + { \ + pr_cont("pci %s: ASPM latency exceeded, disabling: %s:%s-%s", \ + pci_name(device), type, pci_name(down), pci_name(up)); \ + aspm_disable = 1; \ + } \ + else \ + pr_cont(", %s:%s-%s", type, pci_name(down), pci_name(up)); + while (link) { /* Check upstream direction L0s latency */ if (link->aspm_capable & ASPM_STATE_L0S_UP) { l0s_latency_up += link->latency_up.l0s; if (l0s_latency_up > acceptable->l0s) + { link->aspm_capable &= ~ASPM_STATE_L0S_UP; + aspm_info(endpoint, "L0s-up", link->downstream, link->pdev); + } } /* Check downstream direction L0s latency */ if (link->aspm_capable & ASPM_STATE_L0S_DW) { l0s_latency_dw += link->latency_dw.l0s; if (l0s_latency_dw > acceptable->l0s) + { link->aspm_capable &= ~ASPM_STATE_L0S_DW; + aspm_info(endpoint, "L0s-dw", link->downstream, link->pdev); + } } /* @@ -482,12 +499,18 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint) latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1); l1_max_latency = max_t(u32, latency, l1_max_latency); if (l1_max_latency + l1_switch_latency > acceptable->l1) + { link->aspm_capable &= ~ASPM_STATE_L1; + aspm_info(endpoint, "L1", link->downstream, link->pdev); + } l1_switch_latency += 1000; } link = link->parent; } + if (aspm_disable) + pr_cont("\n"); +#undef aspm_info } /*