From patchwork Thu Apr 13 13:13:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Liu Shixin X-Patchwork-Id: 13210254 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BFC0C77B61 for ; Thu, 13 Apr 2023 12:24:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 763CD6B0072; Thu, 13 Apr 2023 08:24:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 713216B0074; Thu, 13 Apr 2023 08:24:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 60260900002; Thu, 13 Apr 2023 08:24:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 518CA6B0072 for ; Thu, 13 Apr 2023 08:24:02 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1ABA41C5CFA for ; Thu, 13 Apr 2023 12:24:02 +0000 (UTC) X-FDA: 80676284724.04.021ED8C Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf13.hostedemail.com (Postfix) with ESMTP id CB2562000F for ; Thu, 13 Apr 2023 12:23:58 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf13.hostedemail.com: domain of liushixin2@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=liushixin2@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681388640; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references; bh=K1kY2pbb1GUWbzmwx2Nf2SeaxRWsMV5fefl5xcvFB54=; b=vqjTCye6JtkxqGODr2xzZ9TH3gkJS3jB8GlcnFhxvj6xRIrBKoucYVm9FmzReilKCyO1ue eGqGoOlbyhX/kSCtwN44pdcw/CvtGF+vh9KikuPueOg/xzfs0toyuXQ4d8tQaliETFWk4D uyz9sFgq3R5i1Yu5R8DVhxuVTHfFP+c= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf13.hostedemail.com: domain of liushixin2@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=liushixin2@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681388640; a=rsa-sha256; cv=none; b=UQCR+8YjDyzX+YEEmR7ML1oVDXgkBsD1aKbrLeAkgIkIjmvbr+EnkRRJhm4M8glqkf3Stq XskjMdf5wLJBH49cEZb5fsdRwtBZ8kCDSqU6A0/bYk/aOko6JDxlszEDM/oxddCGwwB0Jp wAUb6F5D6mMaiSYi7NEPAJ76RIApBDg= Received: from dggpemm100009.china.huawei.com (unknown [172.30.72.56]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4PxzDZ4VB2znbw1; Thu, 13 Apr 2023 20:20:14 +0800 (CST) Received: from huawei.com (10.175.113.32) by dggpemm100009.china.huawei.com (7.185.36.113) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Thu, 13 Apr 2023 20:23:50 +0800 From: Liu Shixin To: Andrew Morton , Mike Kravetz , Naoya Horiguchi , Tony Luck CC: Miaohe Lin , Muchun Song , , , Liu Shixin Subject: [PATCH -next v2] mm: hwpoison: support recovery from HugePage copy-on-write faults Date: Thu, 13 Apr 2023 21:13:49 +0800 Message-ID: <20230413131349.2524210-1-liushixin2@huawei.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Originating-IP: [10.175.113.32] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To dggpemm100009.china.huawei.com (7.185.36.113) X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: CB2562000F X-Stat-Signature: uee5qfqkrywcu3pusfbqc5pooxondhni X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1681388638-532583 X-HE-Meta: U2FsdGVkX18wVfO4lZUlBMgDuv7buRYB4zEiDsOnzzeLgKUJmv10hMRrgZyuA2WDTl/k+64JvSYcJJzwQrLF29fv/dzzPpuvjGvFFW2RAtzOYVVti2Q07JwN/Al2sCt3HLuR+CKT4rPhYzTnFtGGGChCJewBEpLeDhotFT/Gzwn5Tzx2OBmizxi2cXnPdno776yVz6g0bB1qswFgDVvM8AqyNTs75puAElzt9HewBbR7DxYtPCpF99GBlbe2a9COTqBTjYGiplCAWLpDVoNu7aeVbv/jmY35q299+ut+yqQPE2CaQ2RSkiIa+uoWMuA5YTNff+9HdHZKSUHHPFpGP7p7HToFX8gMnm83ZvHEuUbo/v/VsngSJqgyTDl/8agb4Xj3xCFBwtsBd3P/PIqsUYgUdYeUL0mhtvR+g1yZijg1pAP2ZMR1F+cTTbVVOfhfCAjPS7xu/6p6yhBtx6CUPIMey8Capvq99IrcivJjAdbRR164d7jNMM6JK/RS7BqXtZaiMLL70rsIm1EVrqfAnUkV6HPvaxgRXdOMZP41D66jt3KUw8anKLI/vCUQzMBrEhDiuYHjiCe9Ds1BZYyFBPG2reLldc9UYQvOiVwR72YbJl97halUUo0CGMzTtasuAtD9/GH+Rvf/y920LYNHbFWQnbkTusA91tUk8pclz8snM1mbrRuG//soYeNV4O8PdBVWQ90KyCY0LUnNMhu5od91hpM29kgrrf9RRdMAmFwdAtuZQK0rStz0UjaGIdBjB4ocDscjwNYKB/BLvAZgXfcg3jufmiF2Z71QdfE/njHIav7GZ2qhpUiqtvsFA2p9PVZQ6j6f9BcVxmZkPiRhXeu50n4QX2Q6FndytVHRzS+jJ465S36y4aNbE/8VOZ0K5miqQ1hP+e5lmoJsAczbIF53ggEAJjSmAKg08h6thNiQDQCLqJueTIEmId+vw0shVQ8Fd8Qn/fPrsiPAsys NktD+0Qr aoJVEdOLUAZzadFwzokaUghpX4FuEZg//zA0WVouCFNXDNf5OFCBMOMRgDB1YkZLwVk8dmgcLdNSl1dMPOY9giwLMwpSRQsnSMyoEQeGY6SNthWQz+I8MLjKFVrLPJDVFP6qPy8ZMn1J9g1NmhYl8VN8yUg04XtCCq8dDSMO/4bwGumPTBtIBC1srOg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: copy-on-write of hugetlb user pages with uncorrectable errors will result in a kernel crash. This is because the copy is performed in kernel mode and in general we can not handle accessing memory with such errors while in kernel mode. Commit a873dfe1032a ("mm, hwpoison: try to recover from copy-on write faults") introduced the routine copy_user_highpage_mc() to gracefully handle copying of user pages with uncorrectable errors. However, the separate hugetlb copy-on-write code paths were not modified as part of commit a873dfe1032a. Modify hugetlb copy-on-write code paths to use copy_mc_user_highpage() so that they can also gracefully handle uncorrectable errors in user pages. This involves changing the hugetlb specific routine copy_user_large_folio() from type void to int so that it can return an error. Modify the hugetlb userfaultfd code in the same way so that it can return -EHWPOISON if it encounters an uncorrectable error. Signed-off-by: Liu Shixin Acked-by: Mike Kravetz Reviewed-by: Naoya Horiguchi --- v2: - Update changelog and code based on newest branch, thanks for Mike's changelog. include/linux/mm.h | 6 ++--- mm/hugetlb.c | 17 ++++++++++--- mm/memory.c | 59 +++++++++++++++++++++++++++++----------------- 3 files changed, 55 insertions(+), 27 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 88df1f146121..46c9ae9a7f9b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3730,9 +3730,9 @@ extern const struct attribute_group memory_failure_attr_group; extern void clear_huge_page(struct page *page, unsigned long addr_hint, unsigned int pages_per_huge_page); -void copy_user_large_folio(struct folio *dst, struct folio *src, - unsigned long addr_hint, - struct vm_area_struct *vma); +int copy_user_large_folio(struct folio *dst, struct folio *src, + unsigned long addr_hint, + struct vm_area_struct *vma); long copy_folio_from_user(struct folio *dst_folio, const void __user *usr_src, bool allow_pagefault); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 80c0bf2a4f64..155bb6712be1 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5142,10 +5142,14 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, ret = PTR_ERR(new_folio); break; } - copy_user_large_folio(new_folio, + ret = copy_user_large_folio(new_folio, page_folio(ptepage), addr, dst_vma); put_page(ptepage); + if (ret) { + folio_put(new_folio); + break; + } /* Install the new hugetlb folio if src pte stable */ dst_ptl = huge_pte_lock(h, dst, dst_pte); @@ -5662,7 +5666,10 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, goto out_release_all; } - copy_user_large_folio(new_folio, page_folio(old_page), address, vma); + if (copy_user_large_folio(new_folio, page_folio(old_page), address, vma)) { + ret = VM_FAULT_HWPOISON_LARGE; + goto out_release_all; + } __folio_mark_uptodate(new_folio); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, haddr, @@ -6305,9 +6312,13 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, *foliop = NULL; goto out; } - copy_user_large_folio(folio, *foliop, dst_addr, dst_vma); + ret = copy_user_large_folio(folio, *foliop, dst_addr, dst_vma); folio_put(*foliop); *foliop = NULL; + if (ret) { + folio_put(folio); + goto out; + } } /* diff --git a/mm/memory.c b/mm/memory.c index 1607f0862f65..f2a33337631c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5734,12 +5734,12 @@ EXPORT_SYMBOL(__might_fault); * operation. The target subpage will be processed last to keep its * cache lines hot. */ -static inline void process_huge_page( +static inline int process_huge_page( unsigned long addr_hint, unsigned int pages_per_huge_page, - void (*process_subpage)(unsigned long addr, int idx, void *arg), + int (*process_subpage)(unsigned long addr, int idx, void *arg), void *arg) { - int i, n, base, l; + int i, n, base, l, ret; unsigned long addr = addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); @@ -5753,7 +5753,9 @@ static inline void process_huge_page( /* Process subpages at the end of huge page */ for (i = pages_per_huge_page - 1; i >= 2 * n; i--) { cond_resched(); - process_subpage(addr + i * PAGE_SIZE, i, arg); + ret = process_subpage(addr + i * PAGE_SIZE, i, arg); + if (ret) + return ret; } } else { /* If target subpage in second half of huge page */ @@ -5762,7 +5764,9 @@ static inline void process_huge_page( /* Process subpages at the begin of huge page */ for (i = 0; i < base; i++) { cond_resched(); - process_subpage(addr + i * PAGE_SIZE, i, arg); + ret = process_subpage(addr + i * PAGE_SIZE, i, arg); + if (ret) + return ret; } } /* @@ -5774,10 +5778,15 @@ static inline void process_huge_page( int right_idx = base + 2 * l - 1 - i; cond_resched(); - process_subpage(addr + left_idx * PAGE_SIZE, left_idx, arg); + ret = process_subpage(addr + left_idx * PAGE_SIZE, left_idx, arg); + if (ret) + return ret; cond_resched(); - process_subpage(addr + right_idx * PAGE_SIZE, right_idx, arg); + ret = process_subpage(addr + right_idx * PAGE_SIZE, right_idx, arg); + if (ret) + return ret; } + return 0; } static void clear_gigantic_page(struct page *page, @@ -5795,11 +5804,12 @@ static void clear_gigantic_page(struct page *page, } } -static void clear_subpage(unsigned long addr, int idx, void *arg) +static int clear_subpage(unsigned long addr, int idx, void *arg) { struct page *page = arg; clear_user_highpage(page + idx, addr); + return 0; } void clear_huge_page(struct page *page, @@ -5816,7 +5826,7 @@ void clear_huge_page(struct page *page, process_huge_page(addr_hint, pages_per_huge_page, clear_subpage, page); } -static void copy_user_gigantic_page(struct folio *dst, struct folio *src, +static int copy_user_gigantic_page(struct folio *dst, struct folio *src, unsigned long addr, struct vm_area_struct *vma, unsigned int pages_per_huge_page) @@ -5830,8 +5840,13 @@ static void copy_user_gigantic_page(struct folio *dst, struct folio *src, src_page = folio_page(src, i); cond_resched(); - copy_user_highpage(dst_page, src_page, addr + i*PAGE_SIZE, vma); + if (copy_mc_user_highpage(dst_page, src_page, + addr + i*PAGE_SIZE, vma)) { + memory_failure_queue(page_to_pfn(src_page), 0); + return -EHWPOISON; + } } + return 0; } struct copy_subpage_arg { @@ -5840,16 +5855,20 @@ struct copy_subpage_arg { struct vm_area_struct *vma; }; -static void copy_subpage(unsigned long addr, int idx, void *arg) +static int copy_subpage(unsigned long addr, int idx, void *arg) { struct copy_subpage_arg *copy_arg = arg; - copy_user_highpage(copy_arg->dst + idx, copy_arg->src + idx, - addr, copy_arg->vma); + if (copy_mc_user_highpage(copy_arg->dst + idx, copy_arg->src + idx, + addr, copy_arg->vma)) { + memory_failure_queue(page_to_pfn(copy_arg->src + idx), 0); + return -EHWPOISON; + } + return 0; } -void copy_user_large_folio(struct folio *dst, struct folio *src, - unsigned long addr_hint, struct vm_area_struct *vma) +int copy_user_large_folio(struct folio *dst, struct folio *src, + unsigned long addr_hint, struct vm_area_struct *vma) { unsigned int pages_per_huge_page = folio_nr_pages(dst); unsigned long addr = addr_hint & @@ -5860,13 +5879,11 @@ void copy_user_large_folio(struct folio *dst, struct folio *src, .vma = vma, }; - if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { - copy_user_gigantic_page(dst, src, addr, vma, - pages_per_huge_page); - return; - } + if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) + return copy_user_gigantic_page(dst, src, addr, vma, + pages_per_huge_page); - process_huge_page(addr_hint, pages_per_huge_page, copy_subpage, &arg); + return process_huge_page(addr_hint, pages_per_huge_page, copy_subpage, &arg); } long copy_folio_from_user(struct folio *dst_folio,