From patchwork Fri Jan 20 03:46:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiaqi Yan X-Patchwork-Id: 13109101 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24ED5C46467 for ; Fri, 20 Jan 2023 03:46:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AE6DB6B0073; Thu, 19 Jan 2023 22:46:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A963D6B0078; Thu, 19 Jan 2023 22:46:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 986B66B007B; Thu, 19 Jan 2023 22:46:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7E0CE6B0073 for ; Thu, 19 Jan 2023 22:46:32 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4A9251605EF for ; Fri, 20 Jan 2023 03:46:32 +0000 (UTC) X-FDA: 80373790224.23.943E707 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf28.hostedemail.com (Postfix) with ESMTP id 880BAC0005 for ; Fri, 20 Jan 2023 03:46:29 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=n9vjHRZG; spf=pass (imf28.hostedemail.com: domain of 3lA7KYwgKCP0onfvn3fslttlqj.htrqnsz2-rrp0fhp.twl@flex--jiaqiyan.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3lA7KYwgKCP0onfvn3fslttlqj.htrqnsz2-rrp0fhp.twl@flex--jiaqiyan.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674186389; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+XoYfvivMm4eK9U0FZ9oxFt/e5wfF2TATZ3CeGQNbi4=; b=sLJFT9A9FuCHMkPW3TGmVpqjm4zfLmCd5vX7YXHdoPOsIJtRfJ/OYlkSO5I4oaZh1yVfbn Z4dineoKqx1mkrR02ToZBF94Q9T5/y90yfo4GMzS53ax3kCy+PwlScvnjjvKYU+0kAJsS5 HTOZuFBTygxlqNXpU7NQHR1pL8N3My4= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=n9vjHRZG; spf=pass (imf28.hostedemail.com: domain of 3lA7KYwgKCP0onfvn3fslttlqj.htrqnsz2-rrp0fhp.twl@flex--jiaqiyan.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3lA7KYwgKCP0onfvn3fslttlqj.htrqnsz2-rrp0fhp.twl@flex--jiaqiyan.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674186389; a=rsa-sha256; cv=none; b=snq2+otCt+ieVjmSdoWtr5j1JKXN9r0QDOSRfpIvaIbCTzOJgtvZzFBitWom3Kjhs9pfnH oh1OtEEdqUcncExcl5586SHRD+G006BidW7ATXyXaE3FUjogpS5dAiXwzMKTovCl1f2r8E V18L2oaTMfz7ouL+fpFNOmABONLZJ5M= Received: by mail-pj1-f74.google.com with SMTP id h6-20020a17090aa88600b00223fccff2efso4365662pjq.6 for ; Thu, 19 Jan 2023 19:46:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+XoYfvivMm4eK9U0FZ9oxFt/e5wfF2TATZ3CeGQNbi4=; b=n9vjHRZGnsz/IMMNj5Wx2W8jqZ6ZwkZ1+QsS4bYe6GGSs21nvGHNE437QEtVkHY5qY lapCC1WEk/RbNA62cnhio/RJWqRmDYlD6G2tkhg1Ondc61+wmfNXA6mF5MwKY22LuStG 9/I6+KX4Vu0e/mfstLju8YIEjkjub7UgWHqx18vhtD8HTU+N75EJ+aOJzGo4g8sfi9yT LwF1gNbpmB0CLcpHtCWz3VsAPn63/LxDRxpmJoCVR9LEg+AmZrOqX93bgsrl14p4tfNq vK9unWMDwvlQbrlifyJER7AtqOEhTDSs4vAK0nvtrB8Yhfp2YCNXKCJkW8ijyRZZQ+TF l+iA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+XoYfvivMm4eK9U0FZ9oxFt/e5wfF2TATZ3CeGQNbi4=; b=UWrSSH6gwh4jjQlzHt3P+wlh8cCs83Fvx0a4rWS0MYaohmvPiwvIfo5KNDolKIGb7G 7RVZJrXQymdKWpL1XlveecQPiQbUS81XDPwSmToJ0IIEK9rfbvgiBr6c2isN8EFhIkeX JKvaDtPGulQUudZMkCRhMKMymIMxFsfuWGFK66bmOXhNefz8HwA36TUcIXsdgGX3ojAa s3S4oZk7/Chyo/5rhgdlJrghePVmpB41cI77/JCYUhWB1GzIif59sVqDoV7lD/PCTrw4 uUcSuUi+YFmbwJytakvoCQusqWzkk9COuFB+pGAPt00F4vvuskHBJbQm5X/gjOJURDXs 9FmQ== X-Gm-Message-State: AFqh2koDiVAy0zkQoLe4Xo9t5BedLGseStkPk1+90ZfhhnSJTegAMbPQ SvN99/SOdvaEnAUK3r1VpKbs15j846qopg== X-Google-Smtp-Source: AMrXdXtTUrIqVjvBaNXV3f8Rx6f4oKdykpfGHDnW449gp+XvYvD73y8BnRbDQd746fWpJLK29zIc0xbBHtaNlg== X-Received: from yjq3.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:272f]) (user=jiaqiyan job=sendgmr) by 2002:a17:902:d894:b0:193:405:1704 with SMTP id b20-20020a170902d89400b0019304051704mr1324884plz.31.1674186388275; Thu, 19 Jan 2023 19:46:28 -0800 (PST) Date: Fri, 20 Jan 2023 03:46:20 +0000 In-Reply-To: <20230120034622.2698268-1-jiaqiyan@google.com> Mime-Version: 1.0 References: <20230120034622.2698268-1-jiaqiyan@google.com> X-Mailer: git-send-email 2.39.0.246.g2a6d74b583-goog Message-ID: <20230120034622.2698268-2-jiaqiyan@google.com> Subject: [PATCH v2 1/3] mm: memory-failure: Add memory failure stats to sysfs From: Jiaqi Yan To: tony.luck@intel.com, naoya.horiguchi@nec.com Cc: jiaqiyan@google.com, duenwen@google.com, rientjes@google.com, linux-mm@kvack.org, shy828301@gmail.com, akpm@linux-foundation.org, wangkefeng.wang@huawei.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 880BAC0005 X-Rspam-User: X-Stat-Signature: 3reohmrqpuzo33ogqew9hnfq8npef3n6 X-HE-Tag: 1674186389-599683 X-HE-Meta: U2FsdGVkX18FfhFIe3vJwaGJNGTkFqzCkShSmyu8Pi7jhKxURHczzxcm7V+9+sEKGMWOEOp/e4tyBB6PKD3gP34jnhloHDWbmxidniAWY6plAKDWWz3FTWUR1wmagDBiDwW029ga85/XUMPLa73FoUN+bzbqbEW90/DkQLYFrsaHgg/xxq95c/9rnBn1XiWv5yZRk1zw7Whi3Xgk/G+JKyO6VG5S2A1F5hvJ4lCP1CE9q+ZuO8I9tGi3lw3W5fsYEhCihN3jiRps7TlLarK0Cr0zvaRmJ7a34uzitbTMHtL/dWWFGr2JSVp3fRnQnY//TI/JW1XqUeBRk7tSK77gC5j3AAvqoPIVeHzRG08BTcUQhHhqAiz4dsAwX7YkSKF1ENxxJ3QzOe/CucKxtp1Fb3oQLSgji5j7XVsUdc6PjRLAZb5uSmocfxmIxUgImv5Bi7aT9U1yPyHoKPWHALnjnLTIxBRR6Zi7m+MIL5oN+Q0BNbSN9x4E+pGkkySCezWKt/pcFfB7sqxKGwQucbtM7rdLbpUqOm98/YXyf+1nzF9PRwCmJyPDa44yF5q2ach6OcRoHxK4zjKjZHQsIOEhE0zhBhpnVHpRQDOjZd+pOVX3rjnbkGS9Y+qO8eZRxh07gvMOHcDdKinW3JRSnwSUXWDnxX9CcYuCWqvsTw44dtqtzJDDsRGPlt2egGTXeQCuw3RcftoAaHTeqGjXtRtxJHhTdegmGcmSMJljRRultEjrerb6i7y8+IPtf5nz1Kw35au0Ep8FGdToKUSP7FYXr4KhoguK4TLWQj72/D9fy0OWDRxlhsc1ZLKJtNY6OoqjNvlCRjd8w2X3yguWJs4EtXul2DAKJxSXaBqgHvXgtD0/1AgSdb6aZn9JkA+RDVTmVfRtwxFBBVfngw06inuGGJjNuiG9MgEKrTYXtFXOWAqlKs7E8BHlw871lqUsjl2ptcMBXo2WkCUn81oHadM 7A5Kal9k 0XM/j1pBIccLR0T6MD4R23IRsbnqvQQ65R/JzrJLyOWBm2s6OHl3BBfpcCehhZOVBain/K1WAdTSUB1FQKZnrIVgLqP1qYRQaovhlVJssgTxzsDkMkl4w424O1el9eMsUJ9thVQRp3oN6mNveKillm8ckgF1sFtdNh8YC2rqp/fzxUcrKHiNivZ0g/26QjS8RHduff5rqLQmDsrmDudVDqOUfAnVZj8grhuVCgw0xbW3QozP5HWgDtPSGOo3Tm93K5T0i1X/AeHvkUMJ78m5sOCbo9YYXauXGpoSK2A7aoXlRxUOGXS4s4DHlnOg0II+12tWiX1efHlwZoiW5+Z0Ej7waY/JGOUSANiWmFVmliEQkSj0/KyLIGJRrC5vS4lO1M611mZfVLwwQ8aTeUAEDSJg8Rd8ZZiXkomh2ZkEl3I5t3ZvoCLv5aI1mItO2nTDwBZX4h4amXQaFdE8se56z0AcM/ARFGD1EQyLrd6oDpLdhwLStZ1+lh165ngBfPNd9HLB2JJLgCoBZjjcnCDTc/NsRjMmD2//c6g7Z6VstdVvjdSwClpW8qYSy4ucZEg7LgWWwzHoUJ/aXAtKANwPp1J0bC5Ati+RO6T/HY2tXV7JnplLo40RAPtb42GGPDZw2yqlHG9cSyLukPdU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Today kernel provides following memory error info to userspace, but each has its own disadvantage * HardwareCorrupted in /proc/meminfo: number of bytes poisoned in total, not per NUMA node stats though * ras:memory_failure_event: only available after explicitly enabled * /dev/mcelog provides many useful info about the MCEs, but doesn't capture how memory_failure recovered memory MCEs * kernel logs: userspace needs to process log text Exposes per NUMA node memory error stats as sysfs entries: /sys/devices/system/node/node${X}/memory_failure/total /sys/devices/system/node/node${X}/memory_failure/recovered /sys/devices/system/node/node${X}/memory_failure/ignored /sys/devices/system/node/node${X}/memory_failure/failed /sys/devices/system/node/node${X}/memory_failure/delayed These counters describe how many raw pages are poisoned and after the attempted recoveries by the kernel, their resolutions: how many are recovered, ignored, failed, or delayed respectively. The following math holds for the statistics: * total = recovered + ignored + failed + delayed Acked-by: David Rientjes Signed-off-by: Jiaqi Yan Acked-by: Naoya Horiguchi --- drivers/base/node.c | 3 +++ include/linux/mm.h | 5 +++++ include/linux/mmzone.h | 28 ++++++++++++++++++++++++++++ mm/memory-failure.c | 35 +++++++++++++++++++++++++++++++++++ 4 files changed, 71 insertions(+) diff --git a/drivers/base/node.c b/drivers/base/node.c index faf3597a96da..b46db17124f3 100644 --- a/drivers/base/node.c +++ b/drivers/base/node.c @@ -586,6 +586,9 @@ static const struct attribute_group *node_dev_groups[] = { &node_dev_group, #ifdef CONFIG_HAVE_ARCH_NODE_DEV_GROUP &arch_node_dev_group, +#endif +#ifdef CONFIG_MEMORY_FAILURE + &memory_failure_attr_group, #endif NULL }; diff --git a/include/linux/mm.h b/include/linux/mm.h index f3f196e4d66d..888576884eb9 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3521,6 +3521,11 @@ enum mf_action_page_type { MF_MSG_UNKNOWN, }; +/* + * Sysfs entries for memory failure handling statistics. + */ +extern const struct attribute_group memory_failure_attr_group; + #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) extern void clear_huge_page(struct page *page, unsigned long addr_hint, diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index cd28a100d9e4..2c537b31fa7b 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1110,6 +1110,31 @@ struct deferred_split { }; #endif +#ifdef CONFIG_MEMORY_FAILURE +/* + * Per NUMA node memory failure handling statistics. + */ +struct memory_failure_stats { + /* + * Number of raw pages poisoned. + * Cases not accounted: memory outside kernel control, offline page, + * arch-specific memory_failure (SGX), hwpoison_filter() filtered + * error events, and unpoison actions from hwpoison_unpoison. + */ + unsigned long total; + /* + * Recovery results of poisoned raw pages handled by memory_failure, + * in sync with mf_result. + * total = ignored + failed + delayed + recovered. + * total * PAGE_SIZE * #nodes = /proc/meminfo/HardwareCorrupted. + */ + unsigned long ignored; + unsigned long failed; + unsigned long delayed; + unsigned long recovered; +}; +#endif + /* * On NUMA machines, each NUMA node would have a pg_data_t to describe * it's memory layout. On UMA machines there is a single pglist_data which @@ -1253,6 +1278,9 @@ typedef struct pglist_data { #ifdef CONFIG_NUMA struct memory_tier __rcu *memtier; #endif +#ifdef CONFIG_MEMORY_FAILURE + struct memory_failure_stats mf_stats; +#endif } pg_data_t; #define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages) diff --git a/mm/memory-failure.c b/mm/memory-failure.c index c77a9e37e27e..c628f1db3a4d 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -87,6 +87,41 @@ inline void num_poisoned_pages_sub(unsigned long pfn, long i) memblk_nr_poison_sub(pfn, i); } +/** + * MF_ATTR_RO - Create sysfs entry for each memory failure statistics. + * @_name: name of the file in the per NUMA sysfs directory. + */ +#define MF_ATTR_RO(_name) \ +static ssize_t _name##_show(struct device *dev, \ + struct device_attribute *attr, \ + char *buf) \ +{ \ + struct memory_failure_stats *mf_stats = \ + &NODE_DATA(dev->id)->mf_stats; \ + return sprintf(buf, "%lu\n", mf_stats->_name); \ +} \ +static DEVICE_ATTR_RO(_name) + +MF_ATTR_RO(total); +MF_ATTR_RO(ignored); +MF_ATTR_RO(failed); +MF_ATTR_RO(delayed); +MF_ATTR_RO(recovered); + +static struct attribute *memory_failure_attr[] = { + &dev_attr_total.attr, + &dev_attr_ignored.attr, + &dev_attr_failed.attr, + &dev_attr_delayed.attr, + &dev_attr_recovered.attr, + NULL, +}; + +const struct attribute_group memory_failure_attr_group = { + .name = "memory_failure", + .attrs = memory_failure_attr, +}; + /* * Return values: * 1: the page is dissolved (if needed) and taken off from buddy, From patchwork Fri Jan 20 03:46:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiaqi Yan X-Patchwork-Id: 13109102 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCFE4C38159 for ; Fri, 20 Jan 2023 03:46:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 591E76B0078; Thu, 19 Jan 2023 22:46:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5407F6B007B; Thu, 19 Jan 2023 22:46:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3DFE46B007D; Thu, 19 Jan 2023 22:46:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 30B806B0078 for ; Thu, 19 Jan 2023 22:46:34 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 08EFA1A0D8E for ; Fri, 20 Jan 2023 03:46:34 +0000 (UTC) X-FDA: 80373790308.15.C1B08B0 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf29.hostedemail.com (Postfix) with ESMTP id 49EB0120009 for ; Fri, 20 Jan 2023 03:46:32 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=aSRPxEVI; spf=pass (imf29.hostedemail.com: domain of 3lw7KYwgKCAIlkcsk0cpiqqing.eqonkpwz-oomxcem.qti@flex--jiaqiyan.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3lw7KYwgKCAIlkcsk0cpiqqing.eqonkpwz-oomxcem.qti@flex--jiaqiyan.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674186392; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Vh3ILj9DKG5irjvgNrvZrFCXWlHJpNIyi1XxviyUiJM=; b=Wj8yOt2887JFYlZXidEDhvuyfGUfnz4K5Aqa2joNRnjyS07LszgORjQ6P2DjGHdthlxY/v flBHnFA+wevzi6yMz56P6A3hEuePJrlE5PA8Tuka0K0+8+Wtwso+Xrb0XB0GZxkJbD/FP4 hH51H4rBWajLcHKI6zk6uylcBWiVMV0= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=aSRPxEVI; spf=pass (imf29.hostedemail.com: domain of 3lw7KYwgKCAIlkcsk0cpiqqing.eqonkpwz-oomxcem.qti@flex--jiaqiyan.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3lw7KYwgKCAIlkcsk0cpiqqing.eqonkpwz-oomxcem.qti@flex--jiaqiyan.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674186392; a=rsa-sha256; cv=none; b=J/z2vYFzRyvXdIIjhZDcJEEfroh7ugohG3XHBQFkrNhlVZUEcMCQev/YyqHqaGQ0eEyeMp 411RtexUnGnNsToV7lHw6Oksra0IzYsFxHvJk7bZyUNIvEYBZ5d6HadY4ndQNlFC0Rk0ER BYdtJDzY0PwWF78CssTh0FUMaoFHqUs= Received: by mail-pg1-f202.google.com with SMTP id s76-20020a632c4f000000b0049ceb0f185eso1963588pgs.7 for ; Thu, 19 Jan 2023 19:46:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Vh3ILj9DKG5irjvgNrvZrFCXWlHJpNIyi1XxviyUiJM=; b=aSRPxEVI/CVfxAGsMfP16EgxYh+QuktFjPe2t5kv3/qEgUxjHLp4r6OtjgHRV8Kbrd jvnYw7fObLCpkDzA+e+MgM36k26uqwJvxgHtKwbA01sSZmMlfF0l2FOCbwlToWLLW5zu 03tYuST+/8k57H/QGaeKnV+ahL4l6moZiWp4acOJILenZCd0xs7W5aOkfvhBtAb0aobC 0TkLXTyjuVQLkoKItW17lq3kAMM7PAvu11NycnYWXdwP/j0FVNjwgWO8wjjlPBJpYsvO VAU+31DeCLAF0ipyJE/7CmzaUXLqNKnRzw5EZjMm8xU1drRBbw2bweyV+LMWTj+NTCdt TPJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Vh3ILj9DKG5irjvgNrvZrFCXWlHJpNIyi1XxviyUiJM=; b=oOCv7NXjs2heeFwuabp5GVXSHk9pu9rt2D83X0xQZ9eF5IHsDPJHcC4xQvk4Svv8bc WySFtVe1nfZ9L0B7iC4n9PptOBMKHWeZDkbEgd5KuDHgjFYmqtK+1qQlrFkwenaz+O0n Q+UVfOW47olc/J7YWqcZR+W+0B2byTHXi0BGxTlU6XoFb8nw3oworWU2p+vOYbr3XKNy juQhOm6sap0F80hjGf2lhNrjsrMzqtTs0RDX+aKXbKEq6w4BIzxsUc893x0q5RU8reKS MjyL0OPL2lh82ofJQ+f12FmLHe9Oh9E5CNa5VN2XwKhDZUyZrkcdUcVZB9IX+Pa2Uvc4 mDTw== X-Gm-Message-State: AFqh2kqWDGknEPiZ67KbFn9JBVsKsrjXrObxOwz6aSkYkqk0pAXPfAtD RYWrjGmiuBbMScvKIvJv3Zf4XjNYIgcXWQ== X-Google-Smtp-Source: AMrXdXtsoYxNlsEAqRJH/XdnlwWvTvE7TChG6zzsq7buK/fDm6KdEHEH4do4apgl0fyFmz8UQSzifEEqW7fGnA== X-Received: from yjq3.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:272f]) (user=jiaqiyan job=sendgmr) by 2002:a17:90a:4945:b0:22b:ae7e:9698 with SMTP id c63-20020a17090a494500b0022bae7e9698mr53693pjh.170.1674186391025; Thu, 19 Jan 2023 19:46:31 -0800 (PST) Date: Fri, 20 Jan 2023 03:46:21 +0000 In-Reply-To: <20230120034622.2698268-1-jiaqiyan@google.com> Mime-Version: 1.0 References: <20230120034622.2698268-1-jiaqiyan@google.com> X-Mailer: git-send-email 2.39.0.246.g2a6d74b583-goog Message-ID: <20230120034622.2698268-3-jiaqiyan@google.com> Subject: [PATCH v2 2/3] mm: memory-failure: Bump memory failure stats to pglist_data From: Jiaqi Yan To: tony.luck@intel.com, naoya.horiguchi@nec.com Cc: jiaqiyan@google.com, duenwen@google.com, rientjes@google.com, linux-mm@kvack.org, shy828301@gmail.com, akpm@linux-foundation.org, wangkefeng.wang@huawei.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 49EB0120009 X-Stat-Signature: rowihiagzii9ep1axso8pj56qcaaiozb X-Rspam-User: X-HE-Tag: 1674186392-510706 X-HE-Meta: U2FsdGVkX184LGwP10ErsUwqawqvyCOUdbTHnluypX0f4u+3476x9vLeQOogxNxiPQmWVZoNxhyrGXBDPXPsJ+WxKcP6DbWewt65gzwjWsBAA2Cn0Pc8vckcyx0KcXicdmK6TZBebv8/IbQx6fV2iE4PkFo3WoE1W4c/+QmP0x4Sti2RLlLTKun/Zz51i2fXMJGEebWfgNDrP8ckqMIiQUGELZdmNh/2CM5R0BM1lI7ABOwpuuzEeig0bWC1nYggmxGYPzOcOAsMQPkcmIuPo3N4lD9ZufzT8dsu+PpoKJgsSRsJoc6AS05prL4zfTA8gAw546TUNv37VdjfY+8gG+qNfVBZ7XL1NR8HP5vzUWiCTHevX9USYP9VJwxQSHmY4nV2FrtQhcZDTJ/afWaoQ82GMRIrr3davCukuPghvhQx9MJOeh2lumEKreywmbCjzK9rAmCV7z/2VDA6+6hDO/0iDRa+AfII2egMsLCdo1EGs3s5oMNZFMgQKGKdKth+etX454IcmT787lwMm0Xn1bpVAV4E2gh6EbfFdpnchNbObtMT+OF24USHXewGIumCoyNG//U/X/sGxfbc2NyaT+LoHQLMsDkW9M6YknafCg5EICsv0OFT4xcbb+mnegbVqNxYI3kSnB8jAa41bXrJcnsxZt7tzIYofnAmfj8oCGuSoE+SYPHL5eUXKazwi1JktDxUL5nCLGv6E2r9Z3jj7vwHgNrwgQgOZP+QelM19Q9yZeeFPVxWxWodiIlUmFk4IAmu2J3R2Kk/dbGVkjO2eIJAEEr/fXyuG0ZP4E4KsVBOGCYKbe2xu4hYSj4bpAAzJpjmRKJI6Dj4U7E9Rjl9s6IeSeKYDsP412Yj+dqTVFr3F0ZCNjlnJv1gKpWbKpHx2ik4zyNnyzFdUythbZkBJaduoyfusKFNIJ987JuvJiO+st6GaG5G6sZJ1zokWDGg3p26bhS9sChpohXiAiN vCcCR6fU p/PreSBgb3EBp9FZ9fWJ0cfOpPOhXKQI1DtZkR1IYNIGc0FnWAooXvvmj3nZjuTkRrfWfJjXJ72UY4oiwb08QqHvhnhDblR2sOv3d/n5Gm/MeLLcogksnLSekAfTIBld+sVq+t9up6AocmsgYu4T7FH9/MvV5mrlnS+VdfEQLzpzw9ViWMG/00ZOAEjjw5s5BXJXS55zUZQD6Q0C1DHqUaubn2FVR+gF/Hu0AOLTnDH9xvdY/GzKQf5xit4oA2pa3KLvvGqjMELKU7vqGt+KygOlRrTAEobKcplwWpuUkO0K1nNA42iUed3umG8sETg+KvpbX42+dUNs9ozr5EG14EMcaNTDS0GcqHMoyK1fJxopTGQchah2Av+Iuu2Fp93+4vtJf8Y7NxC/PsrsW6XH58zKfB0zbmWAxYeoGCLP16Fheer/8PYB5jk8myEwStwAx/aMHG5KWOnE3qZAeWXZtZnZxGhFfBtHonEd3VXno+zeO8QMxeu5vPHNhBAxp9eKyW6ING10Krfs/H+p1JjOqkscjsCOAn6qgcooWxBYD0WV/V71AbXUvAp+npIYha/ZPn9YDKyOL6xQBt6SJKgrXAlH7D93/qBHrQ09n+c+1fW/o42gq0ytBnCmySFySk1fTYyAtF92PIuiV+M2CdanXL4HllR5NLyF+Y2y29czyyrbcD5F5gxevWU0ShQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Right before memory_failure finishes its handling, accumulate poisoned page's resolution counters to pglist_data's memory_failure_stats, so as to update the corresponding sysfs entries. Tested: 1) Start an application to allocate memory buffer chunks 2) Convert random memory buffer addresses to physical addresses 3) Inject memory errors using EINJ at chosen physical addresses 4) Access poisoned memory buffer and recover from SIGBUS 5) Check counter values under /sys/devices/system/node/node*/memory_failure/* Acked-by: David Rientjes Signed-off-by: Jiaqi Yan Acked-by: Naoya Horiguchi --- mm/memory-failure.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/mm/memory-failure.c b/mm/memory-failure.c index c628f1db3a4d..f4990839ea66 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -1227,6 +1227,39 @@ static struct page_state error_states[] = { #undef slab #undef reserved +static void update_per_node_mf_stats(unsigned long pfn, + enum mf_result result) +{ + int nid = MAX_NUMNODES; + struct memory_failure_stats *mf_stats = NULL; + + nid = pfn_to_nid(pfn); + if (unlikely(nid < 0 || nid >= MAX_NUMNODES)) { + WARN_ONCE(1, "Memory failure: pfn=%#lx, invalid nid=%d", pfn, nid); + return; + } + + mf_stats = &NODE_DATA(nid)->mf_stats; + switch (result) { + case MF_IGNORED: + ++mf_stats->ignored; + break; + case MF_FAILED: + ++mf_stats->failed; + break; + case MF_DELAYED: + ++mf_stats->delayed; + break; + case MF_RECOVERED: + ++mf_stats->recovered; + break; + default: + WARN_ONCE(1, "Memory failure: mf_result=%d is not properly handled", result); + break; + } + ++mf_stats->total; +} + /* * "Dirty/Clean" indication is not 100% accurate due to the possibility of * setting PG_dirty outside page lock. See also comment above set_page_dirty(). @@ -1237,6 +1270,9 @@ static int action_result(unsigned long pfn, enum mf_action_page_type type, trace_memory_failure_event(pfn, type, result); num_poisoned_pages_inc(pfn); + + update_per_node_mf_stats(pfn, result); + pr_err("%#lx: recovery action for %s: %s\n", pfn, action_page_types[type], action_name[result]); From patchwork Fri Jan 20 03:46:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiaqi Yan X-Patchwork-Id: 13109103 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFD29C38159 for ; Fri, 20 Jan 2023 03:46:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 67A506B007B; Thu, 19 Jan 2023 22:46:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 629436B007D; Thu, 19 Jan 2023 22:46:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F1626B007E; Thu, 19 Jan 2023 22:46:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3F5EB6B007B for ; Thu, 19 Jan 2023 22:46:37 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2093AA046E for ; Fri, 20 Jan 2023 03:46:37 +0000 (UTC) X-FDA: 80373790434.10.4AC65E6 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf24.hostedemail.com (Postfix) with ESMTP id 52DA9180002 for ; Fri, 20 Jan 2023 03:46:35 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=e71kIHCk; spf=pass (imf24.hostedemail.com: domain of 3mg7KYwgKCAUonfvn3fslttlqj.htrqnsz2-rrp0fhp.twl@flex--jiaqiyan.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3mg7KYwgKCAUonfvn3fslttlqj.htrqnsz2-rrp0fhp.twl@flex--jiaqiyan.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674186395; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=06MvJe4l/KqF1Ll/YZLPcYuuXzR4uJPBN/DkB4GWgdE=; b=dtDItK3flecqg8l9+PUHc+ai6GigS/WGus2VdDX980MoPugtu+B7skAHQi3zjgdlIQ8xb2 XBgRVrxHfkdrZSQjcQjNpShwm9hzppa2O4ikViAN2qSi4u6Bq9dPbXBo/1SghpD8H9dmBt ZGhqvkwONCb7YQRd/sGJgP5wrZZ19Ns= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=e71kIHCk; spf=pass (imf24.hostedemail.com: domain of 3mg7KYwgKCAUonfvn3fslttlqj.htrqnsz2-rrp0fhp.twl@flex--jiaqiyan.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3mg7KYwgKCAUonfvn3fslttlqj.htrqnsz2-rrp0fhp.twl@flex--jiaqiyan.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674186395; a=rsa-sha256; cv=none; b=dJ7/QjKjAR1FiumfSaolALFaIf6blAz+RmNaevZ2Ih/UcwqDIubRpisSJkKinf9ILOAo+g H8F+VG9LIlhCy8cWK5lk6zT4gGqVu/KiwuelGnCVZ4gQfXFj/3JtsPDOc5Y6MD7qH9fDrB 53bD5oOgMNlqGPybFXNf1Ul3al5eWHg= Received: by mail-pg1-f202.google.com with SMTP id q130-20020a632a88000000b004a03cfb3ac6so1966642pgq.6 for ; Thu, 19 Jan 2023 19:46:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=06MvJe4l/KqF1Ll/YZLPcYuuXzR4uJPBN/DkB4GWgdE=; b=e71kIHCkW/LvlHDupWAsK5NpmznpAvZSG7iLgy4DFNYX0JZGbVNA9yI/YanRAh5jHO c3tDrn45VhXxH4sAsfJmK0PYMgLjAoWKqCdlyOWeikVzLHwaMGMhbyH/btApiclGeamz Uw4xD+dE6b4Zd2LLyWYC6mW7+A8MJxY/v+u9JbOL0v1qYZES2i7HGl+4r+icoZZxUejz vywc8BSZWBx0F7sDeXhCmk5TTRpXyqWxB9YbIBRqdrTPP7QRT8r1jqxEj++wGFTeifX7 sd5AqZcOg2fYyFenatNOwmmZdUwyYlc7gO2y0Nx8W3Hkixzjr/0OOpFuTD+KfYzfCpCq XpvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=06MvJe4l/KqF1Ll/YZLPcYuuXzR4uJPBN/DkB4GWgdE=; b=GjznX39Xfd4aZn6qUO5cpA4a+3S3Re6AXLLjZziPbsSrnMU0QKwH+l1vJkKDk/5sLC xTGQam/hRcSGgeRUSyMj7UZk3nIsm4BbsEbZS2c2dMaXq79x5K9eYG/8p0GPCOFNdkx8 VXpYq9KskS57AGclpM7mJfLXkmQalfuKoSF2LXTrBm7lw/vKoT6aOLz251Tim6N0I4K7 F7nTIi1Y0ZD6DhQqtjsjlwpGOOWYG49erCBttmG9v+PD8Fc4BEtT26J/j3icQuPVDNXp hW6GrAnGb8yM0ObG6RhBHuJ78lNpAwmuqUwJaSjdlcDaW7mvo114UQGRjbHHjkQJ5yj0 /GQw== X-Gm-Message-State: AFqh2kp77FpTBcQ9xesMsIbbnuRVqE1PwjzUzlFrbOWu0sWZnnpnhQck /hI+4UMRJ1bQPpVpJyPvErc57eQ10wImpw== X-Google-Smtp-Source: AMrXdXvT78Hwi6DJimmbWWCwy16zUFM/vAmoYSNClNFZ23V0Xgag7aeeBSTlEHML8jQB0j9/dO7Gxr3xBJty+g== X-Received: from yjq3.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:272f]) (user=jiaqiyan job=sendgmr) by 2002:a17:902:d1d2:b0:189:8ea3:7455 with SMTP id g18-20020a170902d1d200b001898ea37455mr1232271plb.19.1674186394289; Thu, 19 Jan 2023 19:46:34 -0800 (PST) Date: Fri, 20 Jan 2023 03:46:22 +0000 In-Reply-To: <20230120034622.2698268-1-jiaqiyan@google.com> Mime-Version: 1.0 References: <20230120034622.2698268-1-jiaqiyan@google.com> X-Mailer: git-send-email 2.39.0.246.g2a6d74b583-goog Message-ID: <20230120034622.2698268-4-jiaqiyan@google.com> Subject: [PATCH v2 3/3] mm: memory-failure: Document memory failure stats From: Jiaqi Yan To: tony.luck@intel.com, naoya.horiguchi@nec.com Cc: jiaqiyan@google.com, duenwen@google.com, rientjes@google.com, linux-mm@kvack.org, shy828301@gmail.com, akpm@linux-foundation.org, wangkefeng.wang@huawei.com X-Rspamd-Queue-Id: 52DA9180002 X-Stat-Signature: dra4jrgfwoene937mg8hs3bpghjdpbpy X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1674186395-578276 X-HE-Meta: U2FsdGVkX18TYxbxKEfjI4rwLAuL7rv1g1IvDfnTR4bXQ2Mxc7YSDO0ygwkATmlAKG53Iz0olDGffRyI0CTfCdzOYrwWhOueU3bAtXushMIq58SPMKYYWIo4G6aYxlrD1YRv93DR1qsfsFG1DHqvy8BFWKFj9w8aXSeNLpMGHJx6/2fJJcPWRyPiTYpPYfzReWs55LjUscWV/ZUHyLCAnkFnf4G8yjjFzSDCmT+tKVFpRQz8Y12c3t+SOBlEhGE1VvPb5NqZNMV40OI4gCds94XurEr7sjSoV1bUvtSdbQefXYxCcysYN4cOf5vyIqvAQdSd7yBFmYekTlNCOZ5Dew2mosZCR65zXCYiT+i+WviFvKh/Pmua0Xmom5Eq1m/eTXy2obs3fQd9SOpSJwB6DYfv0gboO+2MPtQmFyS3PQWajT5q4W9UXiAEdpon21VaCGBmKTuEMZD2ISaxZyhJEoXqROdq8YqYIsE3q+FIYA1tT5JyUJRn54G5AKy+nK6lX8elFoCPkYWfRe2ddcHamHPFBGsB6G51UZJRhFSeEc2gnPVaABIVPFD+x4Zjc3QemnHk0zvlzCy9eQ+4k/+3HjMNW24NQpoZtCTrQvTnYUNrd3Cw0M7VVA5xGAaQiClUiCbW5rh3WOx3ax5sDfSdPw2q5c4XdrSTAz20wYVNAk/zSnhXsA4kLLTIoZp57SdRNsGl9IshOYNxfgybfbgJpLZpcCcg5IGc8BiKI3HRIHD+G1/zrTEp3qZdLcipmHxOQPzLnKr5VrMi7tPM5JO+8TMFhN97Nx584Rxms+9YHk24bqZYs939f3gCl5jxNLWww0TbgyFThSpB+WCG4PfpJIVO6wuAJ6GBRcGofd1OKxYiWhfzbr5tRxHzAD6Bobdg3KbzGYSxTv1jFj9gHWfz01Jq6LKkjSwEFRd22NXTBSHGkUe8sTXaRdy9KSHEMIy/qeW1hjQmAgQ2FGELQ72 /pdR59fU 1RHsMAfsEQWjp6t2ZIvt+lV+qEkrzkdxO70CEpWqEiL+9jBvIiP3TtjRR+zdeUuSXO1M2y+EjBfDEezIHO1VVJzxSR8BrOxkm3ygoURC5k2mOs0054I29TOkCpProZtKOhKhEMI/8OOJTgOyMnJdEdA0g5eu/y0baGevo95ScaPOclhB3JbkghmWaXEbMSKmZ46sCV8j+ElxSRQNWJ00LpVg8mY6mGPhI2KOeQpwz/deo4fE8+Z2MCuFCQAMXLbhcEm0XdH08rjvpeA0yS2kxfjolsPor5BaTRbgzgx1AYILGhTILS5cEGI6dbaoscz2NJEjM0SNH8VV078VBpia81btX70dsofbXEwyI2+LZaWM+V+RrBvG93Cy4TqoL9PSrKSiebJOf8Yu61zzGXh9dwKLSvixlaqGOKodQ74eNd7nyD1BW21SK/uTwfuoEjbFWFzaBjbuA3JY9PItWJ7wNxgmM5OZuOfGq68vgnCHikWMGyKIMP3cXRt7FkmExp/AKSOEWGagAmxOsFRNwFy7RS5WBs21TnoLokuyzgaQQwD1UMVcW94Pvqv48UO2ZEIp81WuGGeyV07pdISFkgx/c9k42OGHbX8u8sgD/EPptYI6CKqlbY+KPwn8wtv2Pc4gTsIA1VRVdqAr0IEuLYDB74EvKIUy6RWX4/d6jASpvZvKkD1tdTb5M9HLNj+oY4473WSs8f/bHMe+8up/OVxf4C/VafeiDH4UGFaBlz+QCRjMoOsK2oDjmv6vVkA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add documentation for memory_failure's per NUMA node sysfs entries Signed-off-by: Jiaqi Yan Acked-by: Naoya Horiguchi --- Documentation/ABI/stable/sysfs-devices-node | 39 +++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/Documentation/ABI/stable/sysfs-devices-node b/Documentation/ABI/stable/sysfs-devices-node index 8db67aa472f1..402af4b2b905 100644 --- a/Documentation/ABI/stable/sysfs-devices-node +++ b/Documentation/ABI/stable/sysfs-devices-node @@ -182,3 +182,42 @@ Date: November 2021 Contact: Jarkko Sakkinen Description: The total amount of SGX physical memory in bytes. + +What: /sys/devices/system/node/nodeX/memory_failure/total +Date: January 2023 +Contact: Jiaqi Yan +Description: + The total number of raw poisoned pages (pages containing + corrupted data due to memory errors) on a NUMA node. + +What: /sys/devices/system/node/nodeX/memory_failure/ignored +Date: January 2023 +Contact: Jiaqi Yan +Description: + Of the raw poisoned pages on a NUMA node, how many pages are + ignored by memory error recovery attempt, usually because + support for this type of pages is unavailable, and kernel + gives up the recovery. + +What: /sys/devices/system/node/nodeX/memory_failure/failed +Date: January 2023 +Contact: Jiaqi Yan +Description: + Of the raw poisoned pages on a NUMA node, how many pages are + failed by memory error recovery attempt. This usually means + a key recovery operation failed. + +What: /sys/devices/system/node/nodeX/memory_failure/delayed +Date: January 2023 +Contact: Jiaqi Yan +Description: + Of the raw poisoned pages on a NUMA node, how many pages are + delayed by memory error recovery attempt. Delayed poisoned + pages usually will be retried by kernel. + +What: /sys/devices/system/node/nodeX/memory_failure/recovered +Date: January 2023 +Contact: Jiaqi Yan +Description: + Of the raw poisoned pages on a NUMA node, how many pages are + recovered by memory error recovery attempt.