From patchwork Tue Nov 12 11:58:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shuai Xue X-Patchwork-Id: 13872154 Received: from out30-130.freemail.mail.aliyun.com (out30-130.freemail.mail.aliyun.com [115.124.30.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 634251F778B; Tue, 12 Nov 2024 11:59:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731412756; cv=none; b=kSXADqmRA5D9GglMFBz0AZVLvVQcHqGPEGHjwioibsxTy8Mjb0K6IMURhygSbvHfGLU9Hg7ZKqDWMGitBkRWEH/gVYPeSwti1roZhwpRU2wHtuN1fTEslfXC0oTnO18KyvAi2JbAaz3Tv1ZKrqo8VGeiH6OEEhbP3Jj8EUp0Pr0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731412756; c=relaxed/simple; bh=Ap/l3omHJrawAz/AJvQPhDYsp+GUXJzj9zpoEP54D5Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=vEwuU1FQs7UuqCA6+9okkJAmOQBUdv6kP7/S4oT41SoIc6UPA63xHqTFKsOHP8QJ+WTENdgS7LauxWKYJ3+CJs2E7Jes2ZRxdX0I+J6y1UdJMTSA3I39dDERIXQuWN7gJDyxZjIXxRif/ZRNG3gJWvr0IH7xwB5KwzDZ25cYc3g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=nMl2PzJO; arc=none smtp.client-ip=115.124.30.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="nMl2PzJO" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1731412745; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=235CE37JET+UgH7IPnuOtnr+0hOs1QWXhkBWfkJD5SA=; b=nMl2PzJOAvABO6xTPixIuXGLAV7BgUProlcz0rM+dwSQs5Scxk5U4TrzeRLk9Ype6bJXwcJmCzt5yU2QYFkFkcumsCZxVfNbTVvK+Jt+8FSP4n7r0NNNYFIQVMGNntraNvVzIGay8wRMxbesKz0SDJokBWIdiPx/JVpAlCkdvWo= Received: from localhost.localdomain(mailfrom:xueshuai@linux.alibaba.com fp:SMTPD_---0WJHYqws_1731412744 cluster:ay36) by smtp.aliyun-inc.com; Tue, 12 Nov 2024 19:59:04 +0800 From: Shuai Xue To: lukas@wunner.de, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux-edac@vger.kernel.org Cc: bhelgaas@google.com, tony.luck@intel.com, bp@alien8.de, xueshuai@linux.alibaba.com Subject: [PATCH v2 1/2] PCI: hotplug: Add a generic RAS tracepoint for hotplug event Date: Tue, 12 Nov 2024 19:58:51 +0800 Message-ID: <20241112115852.52980-2-xueshuai@linux.alibaba.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20241112115852.52980-1-xueshuai@linux.alibaba.com> References: <20241112115852.52980-1-xueshuai@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-edac@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Hotplug events are critical indicators for analyzing hardware health, particularly in AI supercomputers where surprise link downs can significantly impact system performance and reliability. The failure characterization analysis illustrates the significance of failures caused by the Infiniband link errors. Meta observes that 2% in a machine learning cluster and 6% in a vision application cluster of Infiniband failures co-occur with GPU failures, such as falling off the bus, which may indicate a correlation with PCIe.[1] Add a generic RAS tracepoint for hotplug event to help healthy check. [1]https://arxiv.org/abs/2410.21680 Signed-off-by: Shuai Xue --- drivers/pci/hotplug/trace.h | 68 +++++++++++++++++++++++++++++++++++++ include/uapi/linux/pci.h | 7 ++++ 2 files changed, 75 insertions(+) create mode 100644 drivers/pci/hotplug/trace.h diff --git a/drivers/pci/hotplug/trace.h b/drivers/pci/hotplug/trace.h new file mode 100644 index 000000000000..a56e22534bc0 --- /dev/null +++ b/drivers/pci/hotplug/trace.h @@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#if !defined(_TRACE_HW_EVENT_PCI_HP_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_HW_EVENT_PCI_HP_H + +#include + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM hotplug + +#define PCI_HP_TRANS_STATE \ + EM(PCI_HOTPLUG_LINK_UP, "Link Up") \ + EM(PCI_HOTPLUG_LINK_DOWN, "Link Down") \ + EM(PCI_HOTPLUG_CARD_PRESENT, "Card present") \ + EMe(PCI_HOTPLUG_CARD_NO_PRESENT, "Card not present") + +/* Enums require being exported to userspace, for user tool parsing */ +#undef EM +#undef EMe +#define EM(a, b) TRACE_DEFINE_ENUM(a); +#define EMe(a, b) TRACE_DEFINE_ENUM(a); + +PCI_HP_TRANS_STATE + +/* + * Now redefine the EM() and EMe() macros to map the enums to the strings + * that will be printed in the output. + */ +#undef EM +#undef EMe +#define EM(a, b) {a, b}, +#define EMe(a, b) {a, b} + +TRACE_EVENT(pci_hp_event, + + TP_PROTO(const char *port_name, + const char *slot, + const int trans_state), + + TP_ARGS(port_name, slot, trans_state), + + TP_STRUCT__entry( + __string( port_name, port_name ) + __string( slot, slot ) + __field( int, trans_state ) + ), + + TP_fast_assign( + __assign_str(port_name); + __assign_str(slot); + __entry->trans_state = trans_state; + ), + + TP_printk("%s slot:%s, trans_state:%s\n", + __get_str(port_name), + __get_str(slot), + __print_symbolic(__entry->trans_state, PCI_HP_TRANS_STATE) + ) +); + +#endif /* _TRACE_HW_EVENT_PCI_HP_H */ + +#undef TRACE_INCLUDE_PATH +#define TRACE_INCLUDE_PATH ../../drivers/pci/hotplug +#undef TRACE_INCLUDE_FILE +#define TRACE_INCLUDE_FILE trace + +/* This part must be outside protection */ +#include diff --git a/include/uapi/linux/pci.h b/include/uapi/linux/pci.h index a769eefc5139..1f751785a43d 100644 --- a/include/uapi/linux/pci.h +++ b/include/uapi/linux/pci.h @@ -39,4 +39,11 @@ #define PCIIOC_MMAP_IS_MEM (PCIIOC_BASE | 0x02) /* Set mmap state to MEM space. */ #define PCIIOC_WRITE_COMBINE (PCIIOC_BASE | 0x03) /* Enable/disable write-combining. */ +enum pci_hotplug_trans_type { + PCI_HOTPLUG_LINK_UP, + PCI_HOTPLUG_LINK_DOWN, + PCI_HOTPLUG_CARD_PRESENT, + PCI_HOTPLUG_CARD_NO_PRESENT, +}; + #endif /* _UAPILINUX_PCI_H */ From patchwork Tue Nov 12 11:58:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shuai Xue X-Patchwork-Id: 13872155 Received: from out30-100.freemail.mail.aliyun.com (out30-100.freemail.mail.aliyun.com [115.124.30.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C9B061F778C; Tue, 12 Nov 2024 11:59:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.100 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731412756; cv=none; b=QAj6vUzHo2/Kvq0QzHpk82K8SVV9tjRzWH3AayvMgQqLkmVF/ExktonBL0a9kL0QOD9DCUaWfFIKvTkJVz3X8GGB9rDaUtT7ITOyMSxKEJpzCs9EMHS2gcvcnMnVa3czoasiUP8anxWkTnxpnFvidSh7XtG3wxKBCDMxYWuw13U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731412756; c=relaxed/simple; bh=MZPW3KmuIhosXQiR1sUnukYbEwz7Ryg7lEeyBumQp40=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Iq1Ik83ncUoB5KpXZ4KwSF64IuwrQ+uJ5yD9que9bVxNVh10/LBssavEHkRHJ6d+mEAUyvUS2l+51G2Mqmz/8JQsPsHXe3uFf5Qxrcdd4f7Fqh/ybWNwBrqx0oBLVikpDyeatLtCYnQSLR2nAV+3FzUR5y/hbc2OvpHOa/jOl64= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=NxR/95k/; arc=none smtp.client-ip=115.124.30.100 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="NxR/95k/" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1731412745; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=7E4XIGVwFEQzfYl+p4x9wcXN7MWG+kSABSQ6uIWRSCk=; b=NxR/95k/U6jw94d2UPcWCY1K3lF9uGrIE7e183A1FdtzswL2uOyTKGBtk1ykqjMrlREU1W5FRlaLk0x6D6afQqrBK6bTer/ax/79XkplSxmjpYG1yTNOWvdFO1Wyt4HUDz49ziseVRoFjMhBZOei+zaYRaJ1G0+f/8YoNE4zsXs= Received: from localhost.localdomain(mailfrom:xueshuai@linux.alibaba.com fp:SMTPD_---0WJHYqx9_1731412744 cluster:ay36) by smtp.aliyun-inc.com; Tue, 12 Nov 2024 19:59:05 +0800 From: Shuai Xue To: lukas@wunner.de, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux-edac@vger.kernel.org Cc: bhelgaas@google.com, tony.luck@intel.com, bp@alien8.de, xueshuai@linux.alibaba.com Subject: [PATCH v2 2/2] pci: pciehp: Generate tracepoints for hotplug event Date: Tue, 12 Nov 2024 19:58:52 +0800 Message-ID: <20241112115852.52980-3-xueshuai@linux.alibaba.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20241112115852.52980-1-xueshuai@linux.alibaba.com> References: <20241112115852.52980-1-xueshuai@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-edac@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Generate tracepoints for hotplug event. The output like below: $ echo 1 > /sys/kernel/debug/tracing/events/hotplug/pci_hp_event/enable $ cat /sys/kernel/debug/tracing/trace_pipe <...>-206 [001] ..... 40.373870: pci_hp_event: 0000:00:02.0 slot:10, trans_state:Link Down <...>-206 [001] ..... 40.374871: pci_hp_event: 0000:00:02.0 slot:10, trans_state:Card not present Signed-off-by: Shuai Xue --- drivers/pci/hotplug/pciehp_ctrl.c | 33 +++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-) diff --git a/drivers/pci/hotplug/pciehp_ctrl.c b/drivers/pci/hotplug/pciehp_ctrl.c index dcdbfcf404dd..ba099cb19f5e 100644 --- a/drivers/pci/hotplug/pciehp_ctrl.c +++ b/drivers/pci/hotplug/pciehp_ctrl.c @@ -21,6 +21,9 @@ #include #include "pciehp.h" +#define CREATE_TRACE_POINTS +#include "trace.h" + /* The following routines constitute the bulk of the hotplug controller logic */ @@ -239,12 +242,20 @@ void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events) case ON_STATE: ctrl->state = POWEROFF_STATE; mutex_unlock(&ctrl->state_lock); - if (events & PCI_EXP_SLTSTA_DLLSC) + if (events & PCI_EXP_SLTSTA_DLLSC) { ctrl_info(ctrl, "Slot(%s): Link Down\n", slot_name(ctrl)); - if (events & PCI_EXP_SLTSTA_PDC) + trace_pci_hp_event(pci_name(ctrl->pcie->port), + slot_name(ctrl), + PCI_HOTPLUG_LINK_DOWN); + } + if (events & PCI_EXP_SLTSTA_PDC) { ctrl_info(ctrl, "Slot(%s): Card not present\n", slot_name(ctrl)); + trace_pci_hp_event(pci_name(ctrl->pcie->port), + slot_name(ctrl), + PCI_HOTPLUG_CARD_NO_PRESENT); + } pciehp_disable_slot(ctrl, SURPRISE_REMOVAL); break; default: @@ -264,6 +275,9 @@ void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events) INDICATOR_NOOP); ctrl_info(ctrl, "Slot(%s): Card not present\n", slot_name(ctrl)); + trace_pci_hp_event(pci_name(ctrl->pcie->port), + slot_name(ctrl), + PCI_HOTPLUG_CARD_NO_PRESENT); } mutex_unlock(&ctrl->state_lock); return; @@ -276,12 +290,19 @@ void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events) case OFF_STATE: ctrl->state = POWERON_STATE; mutex_unlock(&ctrl->state_lock); - if (present) + if (present) { ctrl_info(ctrl, "Slot(%s): Card present\n", slot_name(ctrl)); - if (link_active) - ctrl_info(ctrl, "Slot(%s): Link Up\n", - slot_name(ctrl)); + trace_pci_hp_event(pci_name(ctrl->pcie->port), + slot_name(ctrl), + PCI_HOTPLUG_CARD_PRESENT); + } + if (link_active) { + ctrl_info(ctrl, "Slot(%s): Link Up\n", slot_name(ctrl)); + trace_pci_hp_event(pci_name(ctrl->pcie->port), + slot_name(ctrl), + PCI_HOTPLUG_LINK_UP); + } ctrl->request_result = pciehp_enable_slot(ctrl); break; default: