From patchwork Fri Jun 14 15:09:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ilpo_J=C3=A4rvinen?= X-Patchwork-Id: 13698805 X-Patchwork-Delegate: bhelgaas@google.com Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.14]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 643B516D30E; Fri, 14 Jun 2024 15:10:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.14 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718377811; cv=none; b=i3isJZUNzjWyQ7T99Lu17ty1g0gI34nN4N8rCBHkWdJrXRxrJjQqxLcyLojLL1UMgZFycV9W46KiNd9qAV9CEHdyRmP2EJ5bt22DYTS2ek6Qn44NNcZ0riOXEGIJckiD7c8LZc31Qu8VR/9LurWrVHdTXtnSMc4Ue82Im6OHsXY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718377811; c=relaxed/simple; bh=yet1N1drQ+gSy2qH8OrtJ7oc9RaHN37lFGg7skrGI6w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=dh4OUaN7+3JTy+9Ug+WRccT9tEAytYE3WPn0aEXGPbpSASwl6BWJ2JinasKgtfuiWEUX0sv3gf/uo4Zgz4pL0+OG8zkngWmFi2OHbCyVo6EQw1+59ndVRZK/UkLLij7qiPfISvAaM5YxqGAqoR11ZhR1j674Q9ouOS9LLoxYenw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=HEz8wesY; arc=none smtp.client-ip=198.175.65.14 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="HEz8wesY" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1718377809; x=1749913809; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=yet1N1drQ+gSy2qH8OrtJ7oc9RaHN37lFGg7skrGI6w=; b=HEz8wesYx88ZMzbTPqdrNLjnR2NlughNuCqqJItK8H81Q5Tn6wIxOHDA hgjnmW/JNcE0+9j+wCwjYkh4+8ZrZkC2crum+F6ThZcKYMRl4dL57/8A2 zv3oEIK+FXE/z5Z0cSMV7Bw4dQOe5jXmRk8mNC/4rruQU0HGBRhU3L/GN JF9WAVadHPoP0Rkd20asaQOyztmiX1Qjmzlek4WxUwHnIn5pWJ/shCX0C B0Hal3ci8mOy/q4psTwqZsED22bAr0QQZPQ3Y7Qh8x/J2ulBuKNuMmjV+ 3oC8+PN6gg7iae6/NgCTPnrVrTBs52YazDlGmB2m0w5lsB7y9WOcvqmgR A==; X-CSE-ConnectionGUID: Zve/ZjXtRv285OFFWlZVNA== X-CSE-MsgGUID: 4JC1z8GsRlmxuCyFLSZjxQ== X-IronPort-AV: E=McAfee;i="6700,10204,11103"; a="19089300" X-IronPort-AV: E=Sophos;i="6.08,238,1712646000"; d="scan'208";a="19089300" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2024 08:10:09 -0700 X-CSE-ConnectionGUID: YvITFwDYSqaSI+mpVpHiFg== X-CSE-MsgGUID: azTck6y0QB+2w961eiii3g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.08,238,1712646000"; d="scan'208";a="40629261" Received: from ijarvine-desk1.ger.corp.intel.com (HELO localhost) ([10.245.247.222]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2024 08:10:06 -0700 From: =?utf-8?q?Ilpo_J=C3=A4rvinen?= To: Bjorn Helgaas , linux-pci@vger.kernel.org, Lukas Wunner , linux-kernel@vger.kernel.org Cc: Borislav Petkov , linux-edac@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Mahesh J Salgaonkar , Oliver O'Halloran , Tony Luck , =?utf-8?q?Ilpo_J=C3=A4rvinen?= Subject: [PATCH 1/3] PCI: Refactor pcie_update_link_speed() Date: Fri, 14 Jun 2024 18:09:19 +0300 Message-Id: <20240614150921.29724-2-ilpo.jarvinen@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240614150921.29724-1-ilpo.jarvinen@linux.intel.com> References: <20240614150921.29724-1-ilpo.jarvinen@linux.intel.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 pcie_update_link_speed() is passed the Link Status register but not all callers have that value at hand nor need the value. Refactor pcie_update_link_speed() to include reading the Link Status register into pcie_update_link_speed() and create __pcie_update_link_speed() which can be used by the hotplug code that has the register value at hand beforehand (and needs the value for other purposes). Signed-off-by: Ilpo Järvinen --- drivers/pci/hotplug/pciehp_hpc.c | 2 +- drivers/pci/pci.h | 7 ++++++- drivers/pci/probe.c | 12 +++++++----- 3 files changed, 14 insertions(+), 7 deletions(-) diff --git a/drivers/pci/hotplug/pciehp_hpc.c b/drivers/pci/hotplug/pciehp_hpc.c index b1d0a1b3917d..0d818110af6d 100644 --- a/drivers/pci/hotplug/pciehp_hpc.c +++ b/drivers/pci/hotplug/pciehp_hpc.c @@ -319,7 +319,7 @@ int pciehp_check_link_status(struct controller *ctrl) return -1; } - pcie_update_link_speed(ctrl->pcie->port->subordinate, lnk_status); + __pcie_update_link_speed(ctrl->pcie->port->subordinate, lnk_status); if (!found) { ctrl_info(ctrl, "Slot(%s): No device found\n", diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 602b270c4408..47d8ccf6230c 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -297,7 +297,12 @@ enum pci_bus_speed pcie_get_speed_cap(struct pci_dev *dev); enum pcie_link_width pcie_get_width_cap(struct pci_dev *dev); void __pcie_print_link_status(struct pci_dev *dev, bool verbose); void pcie_report_downtraining(struct pci_dev *dev); -void pcie_update_link_speed(struct pci_bus *bus, u16 link_status); + +static inline void __pcie_update_link_speed(struct pci_bus *bus, u16 linksta) +{ + bus->cur_bus_speed = pcie_link_speed[linksta & PCI_EXP_LNKSTA_CLS]; +} +void pcie_update_link_speed(struct pci_bus *bus); /* Single Root I/O Virtualization */ struct pci_sriov { diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 6512b4fca229..42dbcc401443 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -742,9 +742,13 @@ const char *pci_speed_string(enum pci_bus_speed speed) } EXPORT_SYMBOL_GPL(pci_speed_string); -void pcie_update_link_speed(struct pci_bus *bus, u16 linksta) +void pcie_update_link_speed(struct pci_bus *bus) { - bus->cur_bus_speed = pcie_link_speed[linksta & PCI_EXP_LNKSTA_CLS]; + struct pci_dev *bridge = bus->self; + u16 linksta; + + pcie_capability_read_word(bridge, PCI_EXP_LNKSTA, &linksta); + __pcie_update_link_speed(bus, linksta); } EXPORT_SYMBOL_GPL(pcie_update_link_speed); @@ -827,13 +831,11 @@ static void pci_set_bus_speed(struct pci_bus *bus) if (pci_is_pcie(bridge)) { u32 linkcap; - u16 linksta; pcie_capability_read_dword(bridge, PCI_EXP_LNKCAP, &linkcap); bus->max_bus_speed = pcie_link_speed[linkcap & PCI_EXP_LNKCAP_SLS]; - pcie_capability_read_word(bridge, PCI_EXP_LNKSTA, &linksta); - pcie_update_link_speed(bus, linksta); + pcie_update_link_speed(bus); } } From patchwork Fri Jun 14 15:09:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ilpo_J=C3=A4rvinen?= X-Patchwork-Id: 13698806 X-Patchwork-Delegate: bhelgaas@google.com Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.14]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C72016D4D5; Fri, 14 Jun 2024 15:10:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.14 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718377821; cv=none; b=Sbo21LL194NUHTZ1ktjSvl0EqeCvxvOFe5W69vVRDu234RqfeS55maBWbIRpuFTIRQ+8JJKJIq9u61yLsP8K3ua7irCaVx4eRa5di9jy84I+4J20vlbiJ5dlVPzXAt4AohgK7PF/SdWdz1m5vQeZJJq8O/2jpEMNAOqD62fJg+Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718377821; c=relaxed/simple; bh=2BhGPo1WCFqZFCU81zY9ZpAs/37Sjw+rNqSOq5xzdv0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=A9TeRrcOyHGdWJjCpFK6NQYhOj76ieB1zfFenc5zY67PeXl4IXvmiXCUqrHNKOvSSywMaG0ie06/bYVSIVWJa4F3KTZ0ZkTZ6oIBjNjNp1+lTvVWT/s2P5zpZ9MfZxeSejKsna7RFBIrmctbZOhpkucKQjha1cCFAkjM6PZrzAg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=aI4LK+oU; arc=none smtp.client-ip=198.175.65.14 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="aI4LK+oU" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1718377819; x=1749913819; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2BhGPo1WCFqZFCU81zY9ZpAs/37Sjw+rNqSOq5xzdv0=; b=aI4LK+oU2r3joKwYNd9gvvIQ2AdjkjoydqnIDg5z4abVRULyz3MgpUG1 /umq7txUAh41f3tuzr8SoyMJqCinR7uoYpym36N6VAuRztF9j2LOg0T4H IEt/OFj+YyE207DSFhnxlUJFq5pWKXgKwAW/9DXl1i3IJaEdo3iw/QvD1 mYkiz5cUmpgCFuZK9R64Xirer8sbwTdLMCqMAtPlq2DhjlA6rXYHjnWuG oNUaoZo0z+ik0cXcJTja8ope0AJ3APs6MK3p69WVFcjOdUGeUPaOQAW2M cbRljdwnsutr83cQexdCBXj1WUEvHNFX/4cb8Z4n9V9wKC1S8qiVjUrZM A==; X-CSE-ConnectionGUID: GyQeAZEaS0WCD1CRoHHthQ== X-CSE-MsgGUID: uJd7fVHrSjuPBK7szJJLuQ== X-IronPort-AV: E=McAfee;i="6700,10204,11103"; a="19089331" X-IronPort-AV: E=Sophos;i="6.08,238,1712646000"; d="scan'208";a="19089331" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2024 08:10:19 -0700 X-CSE-ConnectionGUID: 8avQhssiREyKq7bvDUdCYg== X-CSE-MsgGUID: LeVgUj+IRxC25+Im9tu9xQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.08,238,1712646000"; d="scan'208";a="40629356" Received: from ijarvine-desk1.ger.corp.intel.com (HELO localhost) ([10.245.247.222]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2024 08:10:15 -0700 From: =?utf-8?q?Ilpo_J=C3=A4rvinen?= To: Bjorn Helgaas , linux-pci@vger.kernel.org, Lukas Wunner , linux-kernel@vger.kernel.org Cc: Borislav Petkov , linux-edac@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Mahesh J Salgaonkar , Oliver O'Halloran , Tony Luck , =?utf-8?q?Ilpo_J=C3=A4rvinen?= Subject: [PATCH 2/3] PCI: Track Flit Mode Status & print it with link status Date: Fri, 14 Jun 2024 18:09:20 +0300 Message-Id: <20240614150921.29724-3-ilpo.jarvinen@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240614150921.29724-1-ilpo.jarvinen@linux.intel.com> References: <20240614150921.29724-1-ilpo.jarvinen@linux.intel.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 PCIe r6.0 added Flit mode that mainly alters HW behavior but some OS visible changes are also because of it. The OS visible changes include differences in the layout of some capabilities and interpretation of the TLP headers (in diagnostics situations). To be able to determine which mode the PCIe link is using, store the Flit Mode Status (PCIe r6.1 sec 7.5.3.20) information in addition to the link speed into struct pci_bus in pcie_update_link_speed(). Signed-off-by: Ilpo Järvinen --- drivers/pci/hotplug/pciehp_hpc.c | 5 +++-- drivers/pci/pci.c | 12 ++++++++---- drivers/pci/pci.h | 3 ++- drivers/pci/probe.c | 5 +++-- include/linux/pci.h | 1 + 5 files changed, 17 insertions(+), 9 deletions(-) diff --git a/drivers/pci/hotplug/pciehp_hpc.c b/drivers/pci/hotplug/pciehp_hpc.c index 0d818110af6d..893c022a74cb 100644 --- a/drivers/pci/hotplug/pciehp_hpc.c +++ b/drivers/pci/hotplug/pciehp_hpc.c @@ -291,8 +291,8 @@ static void pcie_wait_for_presence(struct pci_dev *pdev) int pciehp_check_link_status(struct controller *ctrl) { struct pci_dev *pdev = ctrl_dev(ctrl); + u16 lnk_status, linksta2; bool found; - u16 lnk_status; if (!pcie_wait_for_link(pdev, true)) { ctrl_info(ctrl, "Slot(%s): No link\n", slot_name(ctrl)); @@ -319,7 +319,8 @@ int pciehp_check_link_status(struct controller *ctrl) return -1; } - __pcie_update_link_speed(ctrl->pcie->port->subordinate, lnk_status); + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA2, &linksta2); + __pcie_update_link_speed(ctrl->pcie->port->subordinate, lnk_status, linksta2); if (!found) { ctrl_info(ctrl, "Slot(%s): No device found\n", diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index db2e8f21f95c..ff47f6fdcb5a 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -6164,21 +6164,25 @@ void __pcie_print_link_status(struct pci_dev *dev, bool verbose) enum pci_bus_speed speed, speed_cap; struct pci_dev *limiting_dev = NULL; u32 bw_avail, bw_cap; + char *flit_mode = ""; bw_cap = pcie_bandwidth_capable(dev, &speed_cap, &width_cap); bw_avail = pcie_bandwidth_available(dev, &limiting_dev, &speed, &width); + if (dev->bus && dev->bus->flit_mode) + flit_mode = ", in Flit mode"; + if (bw_avail >= bw_cap && verbose) - pci_info(dev, "%u.%03u Gb/s available PCIe bandwidth (%s x%d link)\n", + pci_info(dev, "%u.%03u Gb/s available PCIe bandwidth (%s x%d link)%s\n", bw_cap / 1000, bw_cap % 1000, - pci_speed_string(speed_cap), width_cap); + pci_speed_string(speed_cap), width_cap, flit_mode); else if (bw_avail < bw_cap) - pci_info(dev, "%u.%03u Gb/s available PCIe bandwidth, limited by %s x%d link at %s (capable of %u.%03u Gb/s with %s x%d link)\n", + pci_info(dev, "%u.%03u Gb/s available PCIe bandwidth, limited by %s x%d link at %s (capable of %u.%03u Gb/s with %s x%d link)%s\n", bw_avail / 1000, bw_avail % 1000, pci_speed_string(speed), width, limiting_dev ? pci_name(limiting_dev) : "", bw_cap / 1000, bw_cap % 1000, - pci_speed_string(speed_cap), width_cap); + pci_speed_string(speed_cap), width_cap, flit_mode); } /** diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 47d8ccf6230c..54f5a1584ed8 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -298,9 +298,10 @@ enum pcie_link_width pcie_get_width_cap(struct pci_dev *dev); void __pcie_print_link_status(struct pci_dev *dev, bool verbose); void pcie_report_downtraining(struct pci_dev *dev); -static inline void __pcie_update_link_speed(struct pci_bus *bus, u16 linksta) +static inline void __pcie_update_link_speed(struct pci_bus *bus, u16 linksta, u16 linksta2) { bus->cur_bus_speed = pcie_link_speed[linksta & PCI_EXP_LNKSTA_CLS]; + bus->flit_mode = linksta2 & PCI_EXP_LNKSTA2_FLIT; } void pcie_update_link_speed(struct pci_bus *bus); diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 42dbcc401443..781b8ed892cf 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -745,10 +745,11 @@ EXPORT_SYMBOL_GPL(pci_speed_string); void pcie_update_link_speed(struct pci_bus *bus) { struct pci_dev *bridge = bus->self; - u16 linksta; + u16 linksta, linksta2; pcie_capability_read_word(bridge, PCI_EXP_LNKSTA, &linksta); - __pcie_update_link_speed(bus, linksta); + pcie_capability_read_word(bridge, PCI_EXP_LNKSTA2, &linksta2); + __pcie_update_link_speed(bus, linksta, linksta2); } EXPORT_SYMBOL_GPL(pcie_update_link_speed); diff --git a/include/linux/pci.h b/include/linux/pci.h index d21c704aaffa..a67294042bc5 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -680,6 +680,7 @@ struct pci_bus { struct bin_attribute *legacy_mem; /* Legacy mem */ unsigned int is_added:1; unsigned int unsafe_warn:1; /* warned about RW1C config write */ + bool flit_mode; /* Link in Flit mode */ }; #define to_pci_bus(n) container_of(n, struct pci_bus, dev) From patchwork Fri Jun 14 15:09:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ilpo_J=C3=A4rvinen?= X-Patchwork-Id: 13698807 X-Patchwork-Delegate: bhelgaas@google.com Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.14]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 90AC813E046; Fri, 14 Jun 2024 15:10:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.14 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718377831; cv=none; b=qqFqWfgYcWM+jQDq7/yavGWQbYktD6pv8jev0flD03pqSgcVUO+2g8QF55y/FH0HyrSi1vii8rEwTFevyFUUPOnGAONxuLhqKjC64kIinzVaWZDuxM06ciMGxtaZ+4112U+1kEYJ2oaQZyp9Ju8PHGJM1KxB7HmczwJoIXRwhLk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718377831; c=relaxed/simple; bh=o2NPSa264TsX89kQvYw6c2W52Obbl82MGm8ffWtcZv8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=bdnIN1OwxUFfNzx4eGE1eUP8xeHfLUCkFqfoKdx9bq0Eq7I9PSJfe+Hti/qNZ2d1mZu38gNybNivjRiuxIqkwP2rgEuZbh9UytwYZ3sk4s2gOuE2QNnbnJIcrAS2PmzHlGejj+t0s1rVqCYvFn+yrWxz+0o/+Ot5rnZ5AdgwlVY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=IfMjiQiY; arc=none smtp.client-ip=198.175.65.14 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="IfMjiQiY" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1718377829; x=1749913829; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=o2NPSa264TsX89kQvYw6c2W52Obbl82MGm8ffWtcZv8=; b=IfMjiQiY+zZY90LHPml5OV/GBJ16fLXub9YZsuqW1Vap+tS2gu58YP+Y Ie1CAABCF5E2kWyg0eyvKiS9dkWV7uuAfTj0MKFmNgp+NbEZCImtx8DWv 7x+6oIitHDepVt5Iwt0TsL7pbgMkZ7svLaHYCdGPNS5qHw0mbtnnkq/lp XHUmFA3ZmWSDW6gBjSrC9g0LYj8e6vYe90wPKm+kQ1BfgSRN+45Yz7Mq9 0lWHBDM14i1fR2A5Gs/6ycSVFZghLvxZ9Vzrcf+5cD0UmMWA1yDFHuXcx M+PkOH0mphyrE0Jyrl9Rfx7W+bFCeGPv1cGUg4cpfrh3bVLTNbw8h/Hcu A==; X-CSE-ConnectionGUID: sGOmc6Q2R3y54cpEPizb7g== X-CSE-MsgGUID: yHirxdPWQSuQX/2x8Zcviw== X-IronPort-AV: E=McAfee;i="6700,10204,11103"; a="19089356" X-IronPort-AV: E=Sophos;i="6.08,238,1712646000"; d="scan'208";a="19089356" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2024 08:10:29 -0700 X-CSE-ConnectionGUID: N1Dwe7OqQf+rOj+D1l+Iew== X-CSE-MsgGUID: 5c6fQ8PESlmv98LU/Q1+Dg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.08,238,1712646000"; d="scan'208";a="40629403" Received: from ijarvine-desk1.ger.corp.intel.com (HELO localhost) ([10.245.247.222]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2024 08:10:25 -0700 From: =?utf-8?q?Ilpo_J=C3=A4rvinen?= To: Bjorn Helgaas , linux-pci@vger.kernel.org, Lukas Wunner , Mahesh J Salgaonkar , "Oliver O'Halloran" , Tony Luck , Borislav Petkov , linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-edac@vger.kernel.org Cc: =?utf-8?q?Ilpo_J=C3=A4rvinen?= Subject: [PATCH 3/3] PCI: Handle TLP Log in Flit mode Date: Fri, 14 Jun 2024 18:09:21 +0300 Message-Id: <20240614150921.29724-4-ilpo.jarvinen@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240614150921.29724-1-ilpo.jarvinen@linux.intel.com> References: <20240614150921.29724-1-ilpo.jarvinen@linux.intel.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Flit mode introduced in PCIe r6.0 alters how the TLP Header Log is presented through AER and DPC Capability registers. The TLP Prefix Log Register is not present with Flit mode and the register becomes extension for TLP Header Log (PCIe r6.1 secs 7.8.4.12 & 7.9.14.13). Adapt pcie_read_tlp_log() and struct pcie_tlp_log to read and store also the extended TLP Header Log when the link is in Flit mode. As Prefix Log and Extended TLP Header are not present at the same time, C union can be used. Determining whether the error occurred while the Link was in Flit mode is bit complicated. In case of AER, Advanced Error Capabilities and Control Register directly tells whether the error was logged in Flit mode or not (PCIe r6.1 sec 7.8.4.7). DPC Capability (PCIe r6.1 sec 7.9.14), unfortunately, does not contain the same information. Unlike AER, the DPC capability does not provide way to discern whether the error was logged in Flit mode (this is confirmed by PCI WG to be an oversight in the spec). DPC will bring link down immediately following an error, which make it impossible to acquire the Flit mode status directly from the Link Status 2 register because Flit Mode Status is only set in certain Link states (PCIe r6.1 sec 7.5.3.20). As a workaround, use the flit_mode value stored into the struct pci_bus. Signed-off-by: Ilpo Järvinen --- drivers/pci/pci.h | 5 +-- drivers/pci/pcie/aer.c | 4 ++- drivers/pci/pcie/dpc.c | 23 +++++++++++--- drivers/pci/pcie/tlp.c | 57 ++++++++++++++++++++++++----------- include/linux/aer.h | 13 ++++++-- include/ras/ras_event.h | 12 ++++---- include/uapi/linux/pci_regs.h | 6 +++- 7 files changed, 85 insertions(+), 35 deletions(-) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 54f5a1584ed8..12e8cfd04ebe 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -425,8 +425,9 @@ int aer_get_device_error_info(struct pci_dev *dev, struct aer_err_info *info); void aer_print_error(struct pci_dev *dev, struct aer_err_info *info); int pcie_read_tlp_log(struct pci_dev *dev, int where, int where2, - unsigned int tlp_len, struct pcie_tlp_log *log); -unsigned int aer_tlp_log_len(struct pci_dev *dev); + unsigned int tlp_len, bool flit, + struct pcie_tlp_log *log); +unsigned int aer_tlp_log_len(struct pci_dev *dev, u32 aercc); void pcie_print_tlp_log(const struct pci_dev *dev, const struct pcie_tlp_log *log, const char *pfx); #endif /* CONFIG_PCIEAER */ diff --git a/drivers/pci/pcie/aer.c b/drivers/pci/pcie/aer.c index efb9e728fe94..6beb856964e1 100644 --- a/drivers/pci/pcie/aer.c +++ b/drivers/pci/pcie/aer.c @@ -1241,7 +1241,9 @@ int aer_get_device_error_info(struct pci_dev *dev, struct aer_err_info *info) info->tlp_header_valid = 1; pcie_read_tlp_log(dev, aer + PCI_ERR_HEADER_LOG, aer + PCI_ERR_PREFIX_LOG, - aer_tlp_log_len(dev), &info->tlp); + aer_tlp_log_len(dev, aercc), + aercc & PCI_ERR_CAP_TLP_LOG_FLIT, + &info->tlp); } } diff --git a/drivers/pci/pcie/dpc.c b/drivers/pci/pcie/dpc.c index 598f74384471..a374f3216ce2 100644 --- a/drivers/pci/pcie/dpc.c +++ b/drivers/pci/pcie/dpc.c @@ -219,7 +219,9 @@ static void dpc_process_rp_pio_error(struct pci_dev *pdev) goto clear_status; pcie_read_tlp_log(pdev, cap + PCI_EXP_DPC_RP_PIO_HEADER_LOG, cap + PCI_EXP_DPC_RP_PIO_TLPPREFIX_LOG, - dpc_tlp_log_len(pdev), &tlp_log); + dpc_tlp_log_len(pdev), + pdev->subordinate->flit_mode, + &tlp_log); pcie_print_tlp_log(pdev, &tlp_log, ""); if (pdev->dpc_rp_log_size < 5) @@ -398,12 +400,23 @@ void pci_dpc_init(struct pci_dev *pdev) /* Quirks may set dpc_rp_log_size if device or firmware is buggy */ if (!pdev->dpc_rp_log_size) { + u16 flags; + int ret; + + ret = pcie_capability_read_word(pdev, PCI_EXP_FLAGS, &flags); + if (ret) + return; + pdev->dpc_rp_log_size = FIELD_GET(PCI_EXP_DPC_RP_PIO_LOG_SIZE, cap); - if (pdev->dpc_rp_log_size < 4 || pdev->dpc_rp_log_size > 9) { - pci_err(pdev, "RP PIO log size %u is invalid\n", - pdev->dpc_rp_log_size); - pdev->dpc_rp_log_size = 0; + if (FIELD_GET(PCI_EXP_FLAGS_FLIT, flags)) { + pdev->dpc_rp_log_size += FIELD_GET(PCI_EXP_DPC_RP_PIO_LOG_SIZE4, cap) << 4; + } else { + if (pdev->dpc_rp_log_size < 4 || pdev->dpc_rp_log_size > 9) { + pci_err(pdev, "RP PIO log size %u is invalid\n", + pdev->dpc_rp_log_size); + pdev->dpc_rp_log_size = 0; + } } } } diff --git a/drivers/pci/pcie/tlp.c b/drivers/pci/pcie/tlp.c index 097ac8514e96..5e68b1516f68 100644 --- a/drivers/pci/pcie/tlp.c +++ b/drivers/pci/pcie/tlp.c @@ -7,6 +7,7 @@ #include #include +#include #include #include @@ -15,11 +16,15 @@ /** * aer_tlp_log_len - Calculates AER Capability TLP Header/Prefix Log length * @dev: PCIe device + * @aercc: AER Capabilities and Control register value * * Return: TLP Header/Prefix Log length */ -unsigned int aer_tlp_log_len(struct pci_dev *dev) +unsigned int aer_tlp_log_len(struct pci_dev *dev, u32 aercc) { + if (aercc & PCI_ERR_CAP_TLP_LOG_FLIT) + return FIELD_GET(PCI_ERR_CAP_TLP_LOG_SIZE, aercc); + return 4 + dev->eetlp_prefix_max; } @@ -46,6 +51,7 @@ unsigned int dpc_tlp_log_len(struct pci_dev *dev) * @where: PCI Config offset of TLP Header Log * @where2: PCI Config offset of TLP Prefix Log * @tlp_len: TLP Log length (Header Log + TLP Prefix Log in DWORDs) + * @flit: TLP Logged in Flit mode * @log: TLP Log structure to fill * * Fill @log from TLP Header Log registers, e.g., AER or DPC. @@ -53,28 +59,34 @@ unsigned int dpc_tlp_log_len(struct pci_dev *dev) * Return: 0 on success and filled TLP Log structure, <0 on error. */ int pcie_read_tlp_log(struct pci_dev *dev, int where, int where2, - unsigned int tlp_len, struct pcie_tlp_log *log) + unsigned int tlp_len, bool flit, struct pcie_tlp_log *log) { unsigned int i; int off, ret; - u32 *to; + + if (tlp_len > ARRAY_SIZE(log->dw)) + tlp_len = ARRAY_SIZE(log->dw); memset(log, 0, sizeof(*log)); for (i = 0; i < tlp_len; i++) { - if (i < 4) { + if (i < 4) off = where + i * 4; - to = &log->dw[i]; - } else { + else off = where2 + (i - 4) * 4; - to = &log->prefix[i - 4]; - } - ret = pci_read_config_dword(dev, off, to); + ret = pci_read_config_dword(dev, off, &log->dw[i]); if (ret) return pcibios_err_to_errno(ret); } + /* + * Hard-code non-Flit mode to 4 DWORDs, for now. The exact length + * can only be known if the TLP is parsed. + */ + log->header_len = flit ? tlp_len : 4; + log->flit = flit; + return 0; } @@ -89,21 +101,30 @@ int pcie_read_tlp_log(struct pci_dev *dev, int where, int where2, void pcie_print_tlp_log(const struct pci_dev *dev, const struct pcie_tlp_log *log, const char *pfx) { - char buf[(10 + 1) * (4 + ARRAY_SIZE(log->prefix)) + 14 + 1]; + char buf[(10 + 1) * PCIE_TLP_LOG_MAXLEN + 1]; unsigned int i; int len; len = scnprintf(buf, sizeof(buf), "%#010x %#010x %#010x %#010x", log->dw[0], log->dw[1], log->dw[2], log->dw[3]); - if (log->prefix[0]) - len += scnprintf(buf + len, sizeof(buf) - len, " E-E Prefixes:"); - for (i = 0; i < ARRAY_SIZE(log->prefix); i++) { - if (!log->prefix[i]) - break; - len += scnprintf(buf + len, sizeof(buf) - len, - " %#010x", log->prefix[i]); + if (log->flit) { + for (i = 4; i < log->header_len; i++) { + len += scnprintf(buf + len, sizeof(buf) - len, + " %#010x", log->dw[i]); + } + } else { + if (log->prefix[0]) + len += scnprintf(buf + len, sizeof(buf) - len, + " E-E Prefixes:"); + for (i = 0; i < ARRAY_SIZE(log->prefix); i++) { + if (!log->prefix[i]) + break; + len += scnprintf(buf + len, sizeof(buf) - len, + " %#010x", log->prefix[i]); + } } - pci_err(dev, "%sTLP Header: %s\n", pfx, buf); + pci_err(dev, "%sTLP Header%s: %s\n", pfx, + log->flit ? " (Flit)" : "", buf); } diff --git a/include/linux/aer.h b/include/linux/aer.h index dc498adaa1c8..96d1f83e4a65 100644 --- a/include/linux/aer.h +++ b/include/linux/aer.h @@ -18,9 +18,18 @@ struct pci_dev; +#define PCIE_TLP_LOG_MAXLEN 14 + struct pcie_tlp_log { - u32 dw[4]; - u32 prefix[4]; + union { + u32 dw[PCIE_TLP_LOG_MAXLEN]; + struct { + u32 _do_not_use[4]; + u32 prefix[4]; + }; + }; + u8 header_len; /* Length of the Logged TLP Header in DWORDs */ + bool flit; /* TLP was logged when in Flit mode */ }; struct aer_capability_regs { diff --git a/include/ras/ras_event.h b/include/ras/ras_event.h index 7c47151d5c72..316d216e0573 100644 --- a/include/ras/ras_event.h +++ b/include/ras/ras_event.h @@ -309,7 +309,7 @@ TRACE_EVENT(aer_event, __field( u32, status ) __field( u8, severity ) __field( u8, tlp_header_valid) - __array( u32, tlp_header, 4 ) + __array( u32, tlp_header, PCIE_TLP_LOG_MAXLEN ) ), TP_fast_assign( @@ -318,10 +318,10 @@ TRACE_EVENT(aer_event, __entry->severity = severity; __entry->tlp_header_valid = tlp_header_valid; if (tlp_header_valid) { - __entry->tlp_header[0] = tlp->dw[0]; - __entry->tlp_header[1] = tlp->dw[1]; - __entry->tlp_header[2] = tlp->dw[2]; - __entry->tlp_header[3] = tlp->dw[3]; + int i; + + for (i = 0; i < PCIE_TLP_LOG_MAXLEN; i++) + __entry->tlp_header[i] = tlp->dw[i]; } ), @@ -334,7 +334,7 @@ TRACE_EVENT(aer_event, __print_flags(__entry->status, "|", aer_correctable_errors) : __print_flags(__entry->status, "|", aer_uncorrectable_errors), __entry->tlp_header_valid ? - __print_array(__entry->tlp_header, 4, 4) : + __print_array(__entry->tlp_header, PCIE_TLP_LOG_MAXLEN, 4) : "Not available") ); diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h index b6f9012a3fc4..161b67525447 100644 --- a/include/uapi/linux/pci_regs.h +++ b/include/uapi/linux/pci_regs.h @@ -485,6 +485,7 @@ #define PCI_EXP_TYPE_RC_EC 0xa /* Root Complex Event Collector */ #define PCI_EXP_FLAGS_SLOT 0x0100 /* Slot implemented */ #define PCI_EXP_FLAGS_IRQ 0x3e00 /* Interrupt message number */ +#define PCI_EXP_FLAGS_FLIT 0x8000 /* Flit Mode Supported */ #define PCI_EXP_DEVCAP 0x04 /* Device capabilities */ #define PCI_EXP_DEVCAP_PAYLOAD 0x00000007 /* Max_Payload_Size */ #define PCI_EXP_DEVCAP_PHANTOM 0x00000018 /* Phantom functions */ @@ -788,6 +789,8 @@ #define PCI_ERR_CAP_ECRC_GENE 0x00000040 /* ECRC Generation Enable */ #define PCI_ERR_CAP_ECRC_CHKC 0x00000080 /* ECRC Check Capable */ #define PCI_ERR_CAP_ECRC_CHKE 0x00000100 /* ECRC Check Enable */ +#define PCI_ERR_CAP_TLP_LOG_FLIT 0x00040000 /* TLP was logged in Flit Mode */ +#define PCI_ERR_CAP_TLP_LOG_SIZE 0x00f80000 /* Logged TLP Size (only in Flit mode) */ #define PCI_ERR_HEADER_LOG 0x1c /* Header Log Register (16 bytes) */ #define PCI_ERR_ROOT_COMMAND 0x2c /* Root Error Command */ #define PCI_ERR_ROOT_CMD_COR_EN 0x00000001 /* Correctable Err Reporting Enable */ @@ -1038,8 +1041,9 @@ #define PCI_EXP_DPC_CAP_RP_EXT 0x0020 /* Root Port Extensions */ #define PCI_EXP_DPC_CAP_POISONED_TLP 0x0040 /* Poisoned TLP Egress Blocking Supported */ #define PCI_EXP_DPC_CAP_SW_TRIGGER 0x0080 /* Software Triggering Supported */ -#define PCI_EXP_DPC_RP_PIO_LOG_SIZE 0x0F00 /* RP PIO Log Size */ +#define PCI_EXP_DPC_RP_PIO_LOG_SIZE 0x0F00 /* RP PIO Log Size [3:0] */ #define PCI_EXP_DPC_CAP_DL_ACTIVE 0x1000 /* ERR_COR signal on DL_Active supported */ +#define PCI_EXP_DPC_RP_PIO_LOG_SIZE4 0x2000 /* RP PIO Log Size [4] */ #define PCI_EXP_DPC_CTL 0x06 /* DPC control */ #define PCI_EXP_DPC_CTL_EN_FATAL 0x0001 /* Enable trigger on ERR_FATAL message */