From patchwork Fri Jul 7 20:19:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiaqi Yan X-Patchwork-Id: 13305252 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4B3DEB64DA for ; Fri, 7 Jul 2023 20:19:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B4486B007B; Fri, 7 Jul 2023 16:19:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 464408D0001; Fri, 7 Jul 2023 16:19:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2427E6B007E; Fri, 7 Jul 2023 16:19:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 155F06B007B for ; Fri, 7 Jul 2023 16:19:21 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id DBFCCAEBD8 for ; Fri, 7 Jul 2023 20:19:20 +0000 (UTC) X-FDA: 80985930480.18.2E057BE Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf09.hostedemail.com (Postfix) with ESMTP id 02D7B140012 for ; Fri, 7 Jul 2023 20:19:18 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Lm821Q9A; spf=pass (imf09.hostedemail.com: domain of 3RXOoZAgKCPgonfvn3fslttlqj.htrqnsz2-rrp0fhp.twl@flex--jiaqiyan.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3RXOoZAgKCPgonfvn3fslttlqj.htrqnsz2-rrp0fhp.twl@flex--jiaqiyan.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688761159; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lhHFjAXtuMYeiDPTb9enUa1sagPmUht2lF56qHA5nAY=; b=OeEfjD9di87XrkLbIIVCh2gm5t0mDsSgXeEC8KsmJFaIjLYvYmY2Hdrj7sYglmdSKrtMFJ pDUgBrlX6g6gtyAcU8sJ8MDbT8bwIbaG4NQI41jbsm3+wpC4xHPXEOv2FOm7/ePbhRz1tU HMoYGdw4ND/4OMPDW0yq80b7TAFbkdY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688761159; a=rsa-sha256; cv=none; b=Bv4USIkwt/eNAn6ny18I8qDseyJ3OTv15U9xr04gRKVr5jkFcrmdDsFh69dqM4T+Mikxi4 PSzNBDT6wwna7TxaQ3x07NhlT8erl2Uw8lKI6HP1bjxmZOMItwbzVKHAURQYp6+B04knNH 16aF9f+aUVlr+/D6G5iMwoG+3wy6A78= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Lm821Q9A; spf=pass (imf09.hostedemail.com: domain of 3RXOoZAgKCPgonfvn3fslttlqj.htrqnsz2-rrp0fhp.twl@flex--jiaqiyan.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3RXOoZAgKCPgonfvn3fslttlqj.htrqnsz2-rrp0fhp.twl@flex--jiaqiyan.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-55bf2bf1cdeso3172003a12.3 for ; Fri, 07 Jul 2023 13:19:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1688761158; x=1691353158; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lhHFjAXtuMYeiDPTb9enUa1sagPmUht2lF56qHA5nAY=; b=Lm821Q9A8z9/jwUhSfYSBmBg3a1CJgB8uC9S29RR8iamr7rpAYSfO0Eeq787JeWICj aSBaqXte1TU+6+8Sh0EwU0DomQ2dskj2yaXJhdXbMBNjJ8QUuqx8PFueYsGG0LTCGMQM AG5oIBFVs6KD00JEaeesIKk4hiB7jSUeRJ5U9ZVeA4Xg20DZQrowBV2mwu3oDdpyaXg2 VSPJ+oq7NOFgZNlDG3dULstPjbk7FgX8TBv4efKwHSj4dG2SHkzR24FKkM89NmGdxa+V ORZx2FMfNdL9uzR1qedaRjJLe7tuSJorO9cCwuwQmen2ZaI5XXih1iy8y9odfFxhoErc 4nvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688761158; x=1691353158; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lhHFjAXtuMYeiDPTb9enUa1sagPmUht2lF56qHA5nAY=; b=g8IiFpL/cQ/SdAKbTjF0ilhM8WRHT7g3KNR9oUgttBz14Wu+rRQCy+6u+OXsV/3V2v b2nIVLHbMKEJt3ogbYnfJ/tMfhOT2li6Z71+DZ8taxQgji6uSSb4pQf5yjkHAS3O/Igl xPru/+QGAeRqYz8yF58UKqsHO6X+F9rE2vmleiG6QH/am3bukCTn6RasUTPCIMvd6Ol7 6Gs2BhZSRFp8KwsUexIKe1T39WU7+fEYJ3/MSF/K6CG6nswtXvSLFuWUoBQNxBQSDG7/ zF6CYXvVNpn1cRPTq16VbZt2jjy0cnCgoF719lgsEvU+EKLbzsOeSg2symZkUIBk2j+T 2cuQ== X-Gm-Message-State: ABy/qLbrRfqLmvtGdo7hr7aCIV5t6blT9Oq6Y88PTV9rUI+YOwUOTRQz cbuJtSlUbjMU3mUr1SasPqAijxPidH2sKw== X-Google-Smtp-Source: APBJJlFoY61HjNHb0rFEYpqM/xY1hMlpJ1lQz10syaUR+Qp6bGufuof79IX3dfbrgyCAU0fML5WpoZTLRKI8TQ== X-Received: from yjq3.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:272f]) (user=jiaqiyan job=sendgmr) by 2002:a63:7f51:0:b0:557:5649:381 with SMTP id p17-20020a637f51000000b0055756490381mr4012081pgn.3.1688761157717; Fri, 07 Jul 2023 13:19:17 -0700 (PDT) Date: Fri, 7 Jul 2023 20:19:02 +0000 In-Reply-To: <20230707201904.953262-1-jiaqiyan@google.com> Mime-Version: 1.0 References: <20230707201904.953262-1-jiaqiyan@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230707201904.953262-3-jiaqiyan@google.com> Subject: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON From: Jiaqi Yan To: akpm@linux-foundation.org, mike.kravetz@oracle.com, naoya.horiguchi@nec.com Cc: songmuchun@bytedance.com, shy828301@gmail.com, linmiaohe@huawei.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, duenwen@google.com, axelrasmussen@google.com, jthoughton@google.com, Jiaqi Yan X-Rspamd-Queue-Id: 02D7B140012 X-Rspam-User: X-Stat-Signature: qrsybqkfqmmkn7joemcy4my1n7toff9x X-Rspamd-Server: rspam03 X-HE-Tag: 1688761158-276183 X-HE-Meta: U2FsdGVkX18fAtlur9ul9QZA/GigGDEkqaF8aMddJhbAk1GtnREfEIVxagfRexOBY0IC6UDHURZud4Dj4QU2U7W97aB8npgyZr3b/OpVEtNSIMITLoik18Ct/OpXbldnyxY47yw/XSJYoL+iemfgYESW8YdObVn3pqCcasNRsZ4HcV6hL8mWuIYtcTFZjlNbJXATHJc1UbGD67oHqUMfUyD/Wv3h/CfM0czOBBO/m2S8uGq1JqhJZHVmF4mnSZQxZ1MaFxYidOOrCb4eWjIu47PVw4XADiMiaDd5TutwoaoOkQ/RevLJRpglLTmPfLi12P6UX3jgYxEYeU8lC2oprhPbs0zKslLTHh48uVPCeuuELApPWZb9HXH1NvBPfXEabj4rVZMtQ9yk7gOpBSHgGpgk7i0mxjMZmSemHsTOsZHuRjCh8lKIdVcDsrFNAfKlOHnaJTl+kfbtmkfrb0TAGlrBqu0H20TWpSk4IKdvU7XjCDcvXooW55XZAPsPnM8dLWpky+2F4QDcnQAjJZODVInMZE8m4eA9JjWhVx4bsKgAL56ry6Mbj0cVKY16SLvW2AaYRQNw8I1UJ8JKnMzEI6p8ExzGufAdM3yLATMM3w8LknfnQTAYtWujMhxvO3Pbfp3nqi41kBirPpX8mncRbNUPjJMnQBZ77nIqm0vo06qojvO4KNFaqaGE6Fp0IfXlFelH+B+u3fEnFtoe7HI6p7ugFkgilOtxULh1Tp3wdebLXvdQi3+XgobuaJMAXmGzREDnsSQam6Yv64ZyPGA1buAS08JZJP+eyM9Gs7rnTUb/V9vOurqPI47JjPe/Z8Q+dqoqTv3svf/lIdhP5+eRe49KCRP/bLgHdmp0TenLIHJGwSNVbEc9ege9xP1p86sjP7anNgnAhwCklQWZ2v2UBtTTkGE85UfdIqcdhNqBBYefWOLOP08TfUR1py9Ae59vFnOvn7RvXETkZhavAMQ jw+yzjdS wdoHlVoo2pWM6312AgrnmRiuOIdkuLvy5sx7VcSQNXwQkZ31a9XMxL9EosPFoxOTTKG7SrONHmUe/PQkMT1WJvHeZh3d3W+FCG5oD07STTs3wWm3a0YG48MKHo1w97mZlhhdjOf4EDesEYTjrET2uZysuZ2RipevoG2K8hEr4qVeOqyZ3tjir5d6Vx0anIzbR5q5s5FwR5mwWsaSZem8CG3qblSXWYAKUkTYoghmGv2cikAFRiI725fZzfQWS/mfz0XmV606qyNajeCkR+03k6S/i876MxdD/Fm0DuC5vgatk/eHELE4NN66Eb/68SUNWwwC2o9O5usIPKwH2EekLBFhgwfM2IdQCuKkAicZWPI6XzqiA2BBWvuBohyAhblnC/6Ht03VaPZXnuFOv/Ju/8mwsPftpYERS0rxyMXFq6HMCL3X8vRQRw0UFs6WaK/aq6no/bV6NGe4QQIpGkIsocSm5A8smZ26yBQn/V514uRp3UGNLIuUrlv3HKrAo74R6LhybbxVsQUbnSpP7+aDPAD1H2kA/a915IWq3Pr1jpcTNuZoRAAxn+azhTrSmfnHI9mWPJxVRaGbRUwa6JZWhhjt4co2x/NiDd2MZWvf36rOC0pLKzFXcAq/KkS7QgpAwEclaI/gR0WVELnqv5AuMZEPiCfNasyo4mOrhcrWZ1Amen+o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add the functionality, is_raw_hwp_subpage, to tell if a subpage of a hugetlb folio is a raw HWPOISON page. This functionality relies on RawHwpUnreliable to be not set; otherwise hugepage's raw HWPOISON list becomes meaningless. is_raw_hwp_subpage needs to hold hugetlb_lock in order to synchronize with __get_huge_page_for_hwpoison, who iterates and inserts an entry to raw_hwp_list. llist itself doesn't ensure insertion is synchornized with the iterating used by __is_raw_hwp_list. Caller can minimize the overhead of lock cycles by first checking if folio / head page's HWPOISON flag is set. Exports this functionality to be immediately used in the read operation for hugetlbfs. Reviewed-by: Mike Kravetz Reviewed-by: Naoya Horiguchi Signed-off-by: Jiaqi Yan Reviewed-by: Miaohe Lin --- include/linux/hugetlb.h | 19 +++++++++++++++++++ include/linux/mm.h | 7 +++++++ mm/hugetlb.c | 10 ++++++++++ mm/memory-failure.c | 34 ++++++++++++++++++++++++---------- 4 files changed, 60 insertions(+), 10 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ca3c8e10f24a..4a745af98525 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1007,6 +1007,25 @@ void hugetlb_register_node(struct node *node); void hugetlb_unregister_node(struct node *node); #endif +/* + * Struct raw_hwp_page represents information about "raw error page", + * constructing singly linked list from ->_hugetlb_hwpoison field of folio. + */ +struct raw_hwp_page { + struct llist_node node; + struct page *page; +}; + +static inline struct llist_head *raw_hwp_list_head(struct folio *folio) +{ + return (struct llist_head *)&folio->_hugetlb_hwpoison; +} + +/* + * Check if a given raw @subpage in a hugepage @folio is HWPOISON. + */ +bool is_raw_hwp_subpage(struct folio *folio, struct page *subpage); + #else /* CONFIG_HUGETLB_PAGE */ struct hstate {}; diff --git a/include/linux/mm.h b/include/linux/mm.h index 74f1be743ba2..edaa18b6f731 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3680,6 +3680,7 @@ extern const struct attribute_group memory_failure_attr_group; extern void memory_failure_queue(unsigned long pfn, int flags); extern int __get_huge_page_for_hwpoison(unsigned long pfn, int flags, bool *migratable_cleared); +extern bool __is_raw_hwp_subpage(struct folio *folio, struct page *subpage); void num_poisoned_pages_inc(unsigned long pfn); void num_poisoned_pages_sub(unsigned long pfn, long i); struct task_struct *task_early_kill(struct task_struct *tsk, int force_early); @@ -3694,6 +3695,12 @@ static inline int __get_huge_page_for_hwpoison(unsigned long pfn, int flags, return 0; } +static inline bool __is_raw_hwp_subpage(struct folio *folio, + struct page *subpage) +{ + return false; +} + static inline void num_poisoned_pages_inc(unsigned long pfn) { } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bce28cca73a1..9c608d2f6630 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -7373,6 +7373,16 @@ int get_huge_page_for_hwpoison(unsigned long pfn, int flags, return ret; } +bool is_raw_hwp_subpage(struct folio *folio, struct page *subpage) +{ + bool ret; + + spin_lock_irq(&hugetlb_lock); + ret = __is_raw_hwp_subpage(folio, subpage); + spin_unlock_irq(&hugetlb_lock); + return ret; +} + void folio_putback_active_hugetlb(struct folio *folio) { spin_lock_irq(&hugetlb_lock); diff --git a/mm/memory-failure.c b/mm/memory-failure.c index a08677dcf953..5b6c8ceb13c0 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -1813,18 +1813,32 @@ EXPORT_SYMBOL_GPL(mf_dax_kill_procs); #endif /* CONFIG_FS_DAX */ #ifdef CONFIG_HUGETLB_PAGE -/* - * Struct raw_hwp_page represents information about "raw error page", - * constructing singly linked list from ->_hugetlb_hwpoison field of folio. - */ -struct raw_hwp_page { - struct llist_node node; - struct page *page; -}; -static inline struct llist_head *raw_hwp_list_head(struct folio *folio) +bool __is_raw_hwp_subpage(struct folio *folio, struct page *subpage) { - return (struct llist_head *)&folio->_hugetlb_hwpoison; + struct llist_head *raw_hwp_head; + struct raw_hwp_page *p, *tmp; + bool ret = false; + + if (!folio_test_hwpoison(folio)) + return false; + + /* + * When RawHwpUnreliable is set, kernel lost track of which subpages + * are HWPOISON. So return as if ALL subpages are HWPOISONed. + */ + if (folio_test_hugetlb_raw_hwp_unreliable(folio)) + return true; + + raw_hwp_head = raw_hwp_list_head(folio); + llist_for_each_entry_safe(p, tmp, raw_hwp_head->first, node) { + if (subpage == p->page) { + ret = true; + break; + } + } + + return ret; } static unsigned long __folio_free_raw_hwp(struct folio *folio, bool move_flag)