From patchwork Mon Jan 16 19:38:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiaqi Yan X-Patchwork-Id: 13103609 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3125C54EBE for ; Mon, 16 Jan 2023 19:39:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 639DD6B0072; Mon, 16 Jan 2023 14:39:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5E9836B0073; Mon, 16 Jan 2023 14:39:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B0E86B0075; Mon, 16 Jan 2023 14:39:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3A31C6B0072 for ; Mon, 16 Jan 2023 14:39:10 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id EF92CC09E5 for ; Mon, 16 Jan 2023 19:39:09 +0000 (UTC) X-FDA: 80361675618.24.7093DFB Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 6064D180018 for ; Mon, 16 Jan 2023 19:39:08 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=aqqslGEg; spf=pass (imf24.hostedemail.com: domain of 326fFYwgKCGYNMEUMcERKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jiaqiyan.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=326fFYwgKCGYNMEUMcERKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jiaqiyan.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673897948; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=y/OSdyEbYRZocK4yFWVn0AhgkvdoyazW3O79hcPe4kU=; b=K3wj4DGGavbpVm8l/CLph26tsn5cRs6YrXVoUitdTQj7OuAAiq8JG352dbCy/xJhhZ0X18 jox37RlWJVt/4K6aG32Ybe1LePTIFkm5+0ezwY/MDavJ+Vtto8GhfgicHFYaltJEOs226e kJwC0zzL6twab40lduQMTA+qdEwkI6s= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=aqqslGEg; spf=pass (imf24.hostedemail.com: domain of 326fFYwgKCGYNMEUMcERKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jiaqiyan.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=326fFYwgKCGYNMEUMcERKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jiaqiyan.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673897948; a=rsa-sha256; cv=none; b=PL6k9OlV6sJojbtosOGvc8nPrE2d6ArlnZwcBez5FfoNiu9/zP4UidWxUrSiLQ6XXfcV0X CFGSvRDBrcaKPeX768HBh3tChzrBDvEsxy5ioVkYzHnW121blUzT/OJ116rriOezNBLamz B+qE41HJCJUENUBjQKDsfnwAp3IyRg0= Received: by mail-pl1-f201.google.com with SMTP id l15-20020a170902f68f00b001948ddc7cddso3073842plg.2 for ; Mon, 16 Jan 2023 11:39:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=y/OSdyEbYRZocK4yFWVn0AhgkvdoyazW3O79hcPe4kU=; b=aqqslGEg5xmkrkGKm96Dt1GwoU6sEleLu4JcnefYoIIyBZnVS/X769cm97CZAoYHRC YzvffI1FlxEAW0S3lYI428bgd2PUVlSEjdQlDvol9F6YnHMPFHZf7ytQmcSaOnRUe1B0 796g0nOdiPUq1F3esT96PSvYbXfNVTCo6iTSiG38juJkwFvxnIK1f4ZAXy+CODh3Ddxo 9dfQgzeZCQ4jjzYKcQjTMYFaDJsJ6BolpL8sn0bexPIKu7yyfWA+5zeeZozV6VcDl6jz 2OxyZtnTc8/wEgkafl/cvxVuK7zKQGd0bHfqHfyVQIPYKUkDDRTdyvB/scnQnmBrEB0D q+hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=y/OSdyEbYRZocK4yFWVn0AhgkvdoyazW3O79hcPe4kU=; b=r1+DKpGog7WocE64QJBt5f4qUF7dmb8cNksdqO2re6yItH+UzFpbS9CaI0X5U+WR7F viVBBzA7dl3ig1qzobhUYyTdBCcIurvomNWQkD9WUD7/ZmuS94ZeecK1tujzwF0Y/Z/1 /LFyg8hmnJJc0ZXG9l5GkapoPEBk7yZs2A5EGjYPxuoBY19vO+/aed3KYC+yOp1Ujg/W vQnVonR19bCdkGhN9lZ85rE4+0npK2aR7ULlPU+T3jtgSwRZU+IQGCPXG8k0KybPI5zl fA8NWUlZQX9U3M34LkuCrIhUGR4zsLmoFHPS4VJtOENskehFcPBXsIB9pw4c/WPbpnh+ FXDw== X-Gm-Message-State: AFqh2koQR/Ar2EkjYJiHDMxST/TD4tRfB4EtjeeQ23KCrcGBefh3TWP4 RS05fmFZ6d0uAcFzS5O1Qb7vRofS/F+zJw== X-Google-Smtp-Source: AMrXdXtuWxkNEogqlgaqC0XFYapYxhKgSVuZDdoBsrO632fp4bBIsOdVKhMWCvjLs891THOt58WjAZrOr99xCA== X-Received: from yjq3.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:272f]) (user=jiaqiyan job=sendgmr) by 2002:a62:2f03:0:b0:58d:b5d2:fce2 with SMTP id v3-20020a622f03000000b0058db5d2fce2mr65423pfv.52.1673897947079; Mon, 16 Jan 2023 11:39:07 -0800 (PST) Date: Mon, 16 Jan 2023 19:38:59 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230116193902.1315236-1-jiaqiyan@google.com> Subject: [PATCH v1 0/3] Introduce per NUMA node memory error statistics From: Jiaqi Yan To: tony.luck@intel.com, naoya.horiguchi@nec.com Cc: jiaqiyan@google.com, duenwen@google.com, rientjes@google.com, linux-mm@kvack.org, shy828301@gmail.com, akpm@linux-foundation.org, wangkefeng.wang@huawei.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 6064D180018 X-Stat-Signature: g3azszegaaijuqkndstnexrmtk9bb68d X-HE-Tag: 1673897948-223924 X-HE-Meta: U2FsdGVkX18AgZyjIFuam2Ks4ZTreXpokOwtLpZb6Hl8OpNlnW6JlDnhCx/iCKpnXnHv9XnzUgvmv/z9UaHQZ77Qx8I2XOpXgtOOI+1s/D4qSu/Ivj/r0AtiwdZM9QYfVhsJrV6XEd2dx3jaip9lEUB0DSXP1eAbqDG517khFgdr05sBiNAFkC7kQvEUO7xumi/N/yn9myX62TLuH1YglmoBmiKKAZa+uBYEBKdF5/mshLKDl21VnQtpVI7RwS3SCjxTpYNdOTVIXjLpq03W9sda0EI2tLJAyizOeqnoXFdp5OI5sW8otuffB392lEjDWOEvwJXXRP5RaqrqsLEEGFXmwYVGqqX99fiNI2jX26nZZqhB7cPWm3BoNvkttwhCTY5iG1NHRBbGzjdmWzAGfMrhcMublqXgjsvkx4YTHBF+ymeoFBuJsaxxmUQ59LpPVIt+zdVlRpzb/rUrCskeqDZnzerw0YL6Oa8vNDzW7U1JpbhzBl4lKeR79Iz6Wn+47vvGAVSi3OkyahBNAvd6mNLOo33cj8MsYbHvsSGi34Fv3tACDbK6IqhJqaa0JCTGFAPhTdzKIUyUGe0kkeHhWRdHRllv3mRUx4l7F5Y07PShcKVdq99wy9iw7rjOARiKML8XSVN9LQUgMtg3GOqg1qgxs3aOcwdwDUlIWp558OE7darepfmijmAJKfKbzki71RvuYCF+TPLtKMK+of82o7qOTklRqX3/cJPOtY//eTF9RUXT11go76UzKXUSsgz/BDeZquEbb2x//VXLXMBO9W0vr21I878jW7kLLEik6RbIn+ji+qA4v4yMs7wR6a3thy8LUkVOo5bYg+O5XBLxZeoHzyUwA5UpFqDHVaj7TWZZn7W31pmW/6qrIn8PbW7gJKG0i2scKyHDg0YEqq0B8YCAQjlGmeUeWKBb33Uf6Yymy5C+5PBkAEB8R0DxdVkVrCLdNcbXXeORoqvsK0o tx6dhqpB hrP8fxn1uBu7pKKx/s8TOu4BvN9fuoz1MAJIkUIvdhWnvp8RrHPQvdbIDULjms2hR8qxUQwYzO2q6S6GoZki4ZwfzEf4xdkDLD8vKutnw3BxhobaF027RfwVCzsSaQSKsLJ9uptPdA71edKenf90iyn72/u5nE06G7sOp1FUUgBqgw/YvDo7Mp79LFj6MsUO1Th/36O2CXzKVwNHON5+wu27MYcXAmSwIRkwR/A66oMGiO9WNPAXxja/5M4Dyp8NgwKp3HrMo70PFjpELraTx2cnsEBpRHNmhhzrMAsQW1YFM8iEmxFTkqhgW+j9F7qGfwj5Dzoqlf4UOxnrpXWT6RZzcMO2Lp4jSpmVTMBOxMJtg9WkLY9cCIrVPvmIjGp+vF9A3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.005438, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Background ========== In the RFC for Kernel Support of Memory Error Detection [1], one advantage of software-based scanning over hardware patrol scrubber is the ability to make statistics visible to system administrators. The statistics include 2 categories: * Memory error statistics, for example, how many memory error are encountered, how many of them are recovered by the kernel. Note these memory errors are non-fatal to kernel: during the machine check exception (MCE) handling kernel already classified MCE's severity to be unnecessary to panic (but either action required or optional). * Scanner statistics, for example how many times the scanner have fully scanned a NUMA node, how many errors are first detected by the scanner. The memory error statistics are useful to userspace and actually not specific to scanner detected memory errors, and are the focus of this RFC. Motivation ========== Memory error stats are important to userspace but insufficient in kernel today. Datacenter administrators can better monitor a machine's memory health with the visible stats. For example, while memory errors are inevitable on servers with 10+ TB memory, starting server maintenance when there are only 1~2 recovered memory errors could be overreacting; in cloud production environment maintenance usually means live migrate all the workload running on the server and this usually causes nontrivial disruption to the customer. Providing insight into the scope of memory errors on a system helps to determine the appropriate follow-up action. In addition, the kernel's existing memory error stats need to be standardized so that userspace can reliably count on their usefulness. Today kernel provides following memory error info to userspace, but they are not sufficient or have disadvantages: * HardwareCorrupted in /proc/meminfo: number of bytes poisoned in total, not per NUMA node stats though * ras:memory_failure_event: only available after explicitly enabled * /dev/mcelog provides many useful info about the MCEs, but doesn't capture how memory_failure recovered memory MCEs * kernel logs: userspace needs to process log text Exposing memory error stats is also a good start for the in-kernel memory error detector. Today the data source of memory error stats are either direct memory error consumption, or hardware patrol scrubber detection (when signaled as UCNA; these signaled as SRAO are not handled by memory_failure). Once in-kernel memory scanner is implemented, it will be the main source as it is usually configured to scan memory DIMMs constantly and faster than hardware patrol scrubber. How Implemented =============== As Naoya pointed out [2], exposing memory error statistics to userspace is useful independent of software or hardware scanner. Therefore we implement the memory error statistics independent of the in-kernel memory error detector. It exposes the following per NUMA node memory error counters: /sys/devices/system/node/node${X}/memory_failure/pages_poisoned /sys/devices/system/node/node${X}/memory_failure/pages_recovered /sys/devices/system/node/node${X}/memory_failure/pages_ignored /sys/devices/system/node/node${X}/memory_failure/pages_failed /sys/devices/system/node/node${X}/memory_failure/pages_delayed These counters describe how many raw pages are poisoned and after the attempted recoveries by the kernel, their resolutions: how many are recovered, ignored, failed, or delayed respectively. This approach can be easier to extend for future use cases than /proc/meminfo, trace event, and log. The following math holds for the statistics: * pages_poisoned = pages_recovered + pages_ignored + pages_failed + pages_delayed * pages_poisoned * page_size = /proc/meminfo/HardwareCorrupted These memory error stats are reset during machine boot. The 1st commit introduces these sysfs entries. The 2nd commit populates memory error stats every time memory_failure finishes memory error recovery. The 3rd commit adds documentations for introduced stats. [1] https://lore.kernel.org/linux-mm/7E670362-C29E-4626-B546-26530D54F937@gmail.com/T/#mc22959244f5388891c523882e61163c6e4d703af [2] https://lore.kernel.org/linux-mm/7E670362-C29E-4626-B546-26530D54F937@gmail.com/T/#m52d8d7a333d8536bd7ce74253298858b1c0c0ac6 Jiaqi Yan (3): mm: memory-failure: Add memory failure stats to sysfs mm: memory-failure: Bump memory failure stats to pglist_data mm: memory-failure: Document memory failure stats Documentation/ABI/stable/sysfs-devices-node | 39 +++++++++++ drivers/base/node.c | 3 + include/linux/mm.h | 5 ++ include/linux/mmzone.h | 28 ++++++++ mm/memory-failure.c | 71 +++++++++++++++++++++ 5 files changed, 146 insertions(+)