From patchwork Thu Mar 30 13:40:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peng Zhang X-Patchwork-Id: 13194191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C26AEC6FD1D for ; Thu, 30 Mar 2023 13:41:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 47B256B007E; Thu, 30 Mar 2023 09:41:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 42D856B0080; Thu, 30 Mar 2023 09:41:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 27E33900002; Thu, 30 Mar 2023 09:41:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 131F46B007E for ; Thu, 30 Mar 2023 09:41:40 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B73D8160F97 for ; Thu, 30 Mar 2023 13:41:39 +0000 (UTC) X-FDA: 80625677118.24.E3CD9F0 Received: from szxga08-in.huawei.com (szxga08-in.huawei.com [45.249.212.255]) by imf21.hostedemail.com (Postfix) with ESMTP id 46AC11C001F for ; Thu, 30 Mar 2023 13:41:35 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf21.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.255 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680183696; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jEnN6sQShKpyDIfHL89cAUPTseGoSUSDQgcWk1SCPzc=; b=MBpPWlsO3KP8iDu08MFrB+B+P1KBl5xslm7Ph2jTfKEGSGZjRG2fZZ6oHmPqJNqKZyFkvG /sjPreVPSkTzo+mDOxJ2UO6XJuS26Bp1lToFEQ3JA2MF2ZkQIbW+gmWItnU7xY8O9EKmG5 OzAG8L+PdvlRvGXzO+yutQxzbrZ+/nY= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf21.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.255 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680183696; a=rsa-sha256; cv=none; b=gkA67E8aXLE0Rg6xfMAC71nW2QgTNt2UW42QxlQhenXpEwK4a9BUgZNInjotW8YJuB+lqr vNtrsnR0iWDO9RFMi6UaUx7CaCRVx5DlHvDUO5L6dlP5orDlAgXCHoufbflqK25+CjA22N 7EOwP86YXqbq2WviqCJpEPysYiIsjQA= Received: from kwepemm600020.china.huawei.com (unknown [172.30.72.56]) by szxga08-in.huawei.com (SkyGuard) with ESMTP id 4PnPd04wmDz17QYv; Thu, 30 Mar 2023 21:38:12 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by kwepemm600020.china.huawei.com (7.193.23.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Thu, 30 Mar 2023 21:41:28 +0800 From: Peng Zhang To: , , , , CC: , , , , , ZhangPeng Subject: [PATCH v4 1/6] userfaultfd: convert mfill_atomic_pte_copy() to use a folio Date: Thu, 30 Mar 2023 21:40:40 +0800 Message-ID: <20230330134045.375163-2-zhangpeng362@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230330134045.375163-1-zhangpeng362@huawei.com> References: <20230330134045.375163-1-zhangpeng362@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To kwepemm600020.china.huawei.com (7.193.23.147) X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 46AC11C001F X-Stat-Signature: kippi41bzzhbagoccb9u7y5godxmek8b X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680183695-902959 X-HE-Meta: U2FsdGVkX1/W2NNrLTdIJF/ekTJLQ46p1iqEmqTdC9+IydqbM4QT8g6man1efaPb81IydiEvYe9/nOS1kxlUyJAC7teL9ZxhS/z1IiSwcPWtPhhIAYAWwf52jd1f5nuzDu3EZB11KWjDgVYPX/gSTbxoKkWaeSjDBrIh0OeP3pzVJ+55h1DwStZKcHZJDsJG3a/inhYmdeStvB03radCe1JuNDF384g88AXHeKFmh3LnAuf8rvJjuibN9xaiUkVfWfL7vRShxjVjH6NPus9MpoJfnlAgIfTFaqv53M06oIna3iIcfJwStk4//3DHdsDNGlvGNfxz99p3GfTOCNuinj4RZad7tTcHS30bpYXhYOSFXhA5FDPc9TMx+jdvY+181VPv9+ZI0j05Jj/CZkIY5jDI1mMRlPf40cbtQwI2UYSK8PpEfHIWzOQUXPAXv+PPfIEqa5dumEsOEbP1k0zmSRPH+5L01opr2f8vN6HG4las0JJvCeaoPNkEPXI+suoakNuDbHJKOFBTZRB0IDc2N2t9l/4Wz3mVHs5h3lc2EM1/y5EvPUDT2o4t/OjWGPqE7Wa1hRefFWYhPKJ7QKwlzFAFgedaH48JCw/yY1axSh2N8TskpXm3ytn7uKKT6p/eCgWwMS2ouyyxbb3eTqtxX/yQbgkb3hwnfvuh+J/Col6lfjg1z1ZOvi56t6azp2qHrlwPr7UyUXwBpDgJZHD1ewhVsxUEjx06yHqrMSWi4cO/YwtDgdwlkbvLDqkXOSwDoaBRFqE67R2l66ZmvOS1DdOh8ENl1YDw4FajHAfKXMyQbykxOqB5FOQcg2kI+p2BRMuQn364sHO4gAwPFPjV/jEd8nKiZTSEBPKB/spFWyJMcRI08dEipQFNJZycfEeQbjxJCtSZiHypIeWnz/7vjq6znBreE3jkPiBmEpBbgt4tYG2L2HRTiuGuud+NUQzdFwptAb85tdg2fMsIiio CdWB0KGG zRwoRiZgwov1HnJZUCY5vWWRfLmpHNN4UqncQqT1qDw2H/Mn865FDUV8ArWUvPtegPrPlnzgmsaInb6PK8akds7Y65S8Vqivrd4wldutx9yf6bc55l09h3nUGYnpaVRDvg2twZSyyC86Y6H3tna39pakGHFZfhWrGiCmpH+nkj3R/Lztqtz6I98Xzy5dNhC0e2NoYa+97Ay9YhMt0aaMw8duiI86Ogavo38D/QGK6IF5nkuNpiz3wnVP0+cbNYLVcsviyQzEQ6RntPQK8D7DiN+Fnkg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: ZhangPeng Call vma_alloc_folio() directly instead of alloc_page_vma(). Add an assertion that this is a single-page folio and removes several calls to compound_head(). Signed-off-by: ZhangPeng Reviewed-by: Sidhartha Kumar --- mm/userfaultfd.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 7f1b5f8b712c..efa9e1d681ee 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -137,15 +137,15 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd, { void *page_kaddr; int ret; - struct page *page; + struct folio *folio; if (!*pagep) { ret = -ENOMEM; - page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, dst_vma, dst_addr); - if (!page) + folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, dst_vma, dst_addr, false); + if (!folio) goto out; - page_kaddr = kmap_local_page(page); + page_kaddr = kmap_local_folio(folio, 0); /* * The read mmap_lock is held here. Despite the * mmap_lock being read recursive a deadlock is still @@ -171,36 +171,37 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd, /* fallback to copy_from_user outside mmap_lock */ if (unlikely(ret)) { ret = -ENOENT; - *pagep = page; + *pagep = &folio->page; /* don't free the page */ goto out; } - flush_dcache_page(page); + flush_dcache_folio(folio); } else { - page = *pagep; + folio = page_folio(*pagep); + VM_BUG_ON_FOLIO(folio_test_large(folio), folio); *pagep = NULL; } /* - * The memory barrier inside __SetPageUptodate makes sure that + * The memory barrier inside __folio_mark_uptodate makes sure that * preceding stores to the page contents become visible before * the set_pte_at() write. */ - __SetPageUptodate(page); + __folio_mark_uptodate(folio); ret = -ENOMEM; - if (mem_cgroup_charge(page_folio(page), dst_vma->vm_mm, GFP_KERNEL)) + if (mem_cgroup_charge(folio, dst_vma->vm_mm, GFP_KERNEL)) goto out_release; ret = mfill_atomic_install_pte(dst_pmd, dst_vma, dst_addr, - page, true, flags); + &folio->page, true, flags); if (ret) goto out_release; out: return ret; out_release: - put_page(page); + folio_put(folio); goto out; } From patchwork Thu Mar 30 13:40:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peng Zhang X-Patchwork-Id: 13194189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72940C761A6 for ; Thu, 30 Mar 2023 13:41:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EDC366B0074; Thu, 30 Mar 2023 09:41:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E8B396B007D; Thu, 30 Mar 2023 09:41:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D52F96B0080; Thu, 30 Mar 2023 09:41:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BD3D96B0074 for ; Thu, 30 Mar 2023 09:41:37 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8D2BD1209C4 for ; Thu, 30 Mar 2023 13:41:37 +0000 (UTC) X-FDA: 80625677034.18.C9B0007 Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [45.249.212.189]) by imf07.hostedemail.com (Postfix) with ESMTP id 18C8640006 for ; Thu, 30 Mar 2023 13:41:34 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf07.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.189 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680183695; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sqFhQUDXgIlzpli4zj+/EhLZe7JDjqB2+tFA0MAoZA8=; b=2Jq4yAPIFd+Xq9mh2fJp0L1MlACAE++VAGJNvEoFyudzHzxwpHO4xlVg9HgIYb0YXioyWq Z32o0zOgl+uc+SZICODnzeM6hgficgdvyhFBb9Du0LzKG4QtwRN8DsnmEQhu7lE7EmAPvH L2PtOJqfwMoOMs61nkdhT6zKEHmEv1o= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf07.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.189 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680183695; a=rsa-sha256; cv=none; b=Z6JRDfFzR8L7bAUoFCU/WP/eHcMO0DEYqHV1kLmv+Dyzz6xG7t1XN8nqyVExb9yjoyXNVr I7zNylRk2/3EuechcVv9F7wvZTsOSwZsQreumGcBc0FYSGc4SwTy0SjYhyhq8SKYxERtAj Ctx6Xbk2t/G0zJ/CHMMNfisfjzbjbKI= Received: from kwepemm600020.china.huawei.com (unknown [172.30.72.53]) by szxga03-in.huawei.com (SkyGuard) with ESMTP id 4PnPhD0nzCzKqDF; Thu, 30 Mar 2023 21:41:00 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by kwepemm600020.china.huawei.com (7.193.23.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Thu, 30 Mar 2023 21:41:29 +0800 From: Peng Zhang To: , , , , CC: , , , , , ZhangPeng Subject: [PATCH v4 2/6] userfaultfd: use kmap_local_page() in copy_huge_page_from_user() Date: Thu, 30 Mar 2023 21:40:41 +0800 Message-ID: <20230330134045.375163-3-zhangpeng362@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230330134045.375163-1-zhangpeng362@huawei.com> References: <20230330134045.375163-1-zhangpeng362@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To kwepemm600020.china.huawei.com (7.193.23.147) X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 18C8640006 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: gkh699q4qr11zrzbozfira73g1ntgi5x X-HE-Tag: 1680183694-925252 X-HE-Meta: U2FsdGVkX1+rJSwS0JiHK1hzIbdN1pgl6/2xjoOgzrarlu8sLZYD45CqD7a/qiZK2hHvdC0V8oS3IumUtxLPp4THZHVsw1nga4d1UnLgkM8ioJCTJRy321SfRm0UA6Ur35wNyEVBTJ64NqVP2K5MPRqoTLNXwxdbTyfsQ5VMhG8zovbfPGgGlMyvDN9N8/DWN5pODIiK1klBu2C0oLvFuaZyhoGW2MR7da2LBuFN6pQsMR6usEtaPYtT7b8xvUhCDJkaELPVG/i02bSQ8A52vy7szkiHRuQTWrFSBY+5rcVEtn+B1OHwEamhCUibWlGZkX6NaUpcOyuRBKSIlTSSZrCNl6+78z3jHg0SW/iNAgdexFcn1YzaxPwMVAKLj7sNVRW5Th6AeqbBYxJa6Mk+CiKwg3JP1ulS3bFo2GPgC4FYWEPCPTzfRhbHJaBuxMvyUnJW8fuoD3lkP5YMCh/Hh0IBeEkstwk5pZ/MktaBwO+ROUhV62/AcJH7EKc5JLmymODbhTPPLlSEm3RVOk7J6EPAFJTZTJG5cCeXF90nAr7hsqHyVbAncEded6EmYrgOab3Piyto3N/PGd4N2QA7kiuTqwGEkD+w4fcASFMhZUG930hE36RfhYwGQcipi4kw9uG5OWyoTGV97rO9ky5sH9Ix8IkBfj8PEYmoGt1e+fzodxw1mrpv/e26eq/DtwceCOfgmIa7kywa5+JxgbYB2yZ/1i1pnwTwl6JCbwz0D97CLvMCGCr50rKPziOZbA1LVfdj9WpmyB7sqnA2FMBBKhNIrM/B6n6NhhAq6TATZViUinF8xEiNy5AOnR1bWiNoJlZf2pta3s97wRVAW0p+ScZq4SJJQ8OukqVV0t27h2wryRRCU3/aFIM1VeavREiEsXwEFtchLNUtll4U3btneYqlMTwbIGWdUyztAksOXRV32xB4CAdod5q3Mfc2D8PPL6Sb3CdxdrGouWRfOC4 SVRNfBMD 8AoxfOeLfeI4Aj5H40bs+ch/sEjBA3bzQnIqHyQ4p9WmkV2nhxgsDTL3d7Z9eWQGNcYEO2hqEncbaKTZqcvqeDsIwS0Z1bJOCgA6v/XtglS2LWZsAyjsm3DWRCqAfq6RsIyDkAGE7od3Kt1q/GxFfjZYvIOL6FAYxkaUpalivuaQeKkc21/CMzUiejgfaM9vl6VcB2RRZmki+2n5kdHdngDkYtq4J2gA4OOK760HorQMq7Bf2P21yEljekgt0yrmSGgKu4xa3w46xuEQlB7CfNP2unq91gAxjyFFYK6oWiQlbnfDwnnt31wHbwXRttGgZz9suYO6rmyveUp7ORjk+7wrQ1qPurqTtgv48Gmwcz3mk9xY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: ZhangPeng kmap() and kmap_atomic() are being deprecated in favor of kmap_local_page() which is appropriate for any thread local context.[1] Let's replace the kmap() and kmap_atomic() with kmap_local_page() in copy_huge_page_from_user(). When allow_pagefault is false, disable page faults to prevent potential deadlock.[2] [1] https://lore.kernel.org/all/20220813220034.806698-1-ira.weiny@intel.com/ [2] https://lkml.kernel.org/r/20221025220136.2366143-1-ira.weiny@intel.com Signed-off-by: ZhangPeng --- mm/memory.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index f77fccb5310c..c47b8991410a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5866,16 +5866,14 @@ long copy_huge_page_from_user(struct page *dst_page, for (i = 0; i < pages_per_huge_page; i++) { subpage = nth_page(dst_page, i); - if (allow_pagefault) - page_kaddr = kmap(subpage); - else - page_kaddr = kmap_atomic(subpage); + page_kaddr = kmap_local_page(subpage); + if (!allow_pagefault) + pagefault_disable(); rc = copy_from_user(page_kaddr, usr_src + i * PAGE_SIZE, PAGE_SIZE); - if (allow_pagefault) - kunmap(subpage); - else - kunmap_atomic(page_kaddr); + if (!allow_pagefault) + pagefault_enable(); + kunmap_local(page_kaddr); ret_val -= (PAGE_SIZE - rc); if (rc) From patchwork Thu Mar 30 13:40:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peng Zhang X-Patchwork-Id: 13194194 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5EC24C761AF for ; Thu, 30 Mar 2023 13:41:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F06AA900006; Thu, 30 Mar 2023 09:41:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E8C24900005; Thu, 30 Mar 2023 09:41:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C927F900006; Thu, 30 Mar 2023 09:41:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AD7C2900002 for ; Thu, 30 Mar 2023 09:41:43 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E879F1C6629 for ; Thu, 30 Mar 2023 13:41:40 +0000 (UTC) X-FDA: 80625677160.08.F17904A Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf14.hostedemail.com (Postfix) with ESMTP id B42CE100003 for ; Thu, 30 Mar 2023 13:41:37 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf14.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680183699; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ECjVASrWJxczVOli1Z/HrBt3MPaQ/5CFM40MrwtxHk4=; b=skqneaL6BCsXMZFX77FzLIV7+TY1Q3PiUKL0Juob26WoLqwKhimCeCtnXa95Nb7kxT11eC VFoCxYMQV7p4S1Bk+Y3aGmXeKQHKefJln9kGqgxBV9dkJ5sUidXp6eMSgtkNdM1bCPtEQu XNM55b3EovH6lCtAs6fYbWxRqwtPmFk= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf14.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680183699; a=rsa-sha256; cv=none; b=IPwQCux9hr5Ci6dch/wS8qJCcHGxd3+6vWTjxwy+/0kQhsW1TdVmv6dLB53wtLbC/s3U5z rwLj17NzUCgvoXEzvU+MvmHK0JHS9FV45jfYVrrD0W1QY+nSRj70Vv4sEzIckPq+jfo0tu j+ZFq3QAsA+AG8Uv3OttksDhdgC8CO0= Received: from kwepemm600020.china.huawei.com (unknown [172.30.72.55]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4PnPgT5JJhzrYdQ; Thu, 30 Mar 2023 21:40:21 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by kwepemm600020.china.huawei.com (7.193.23.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Thu, 30 Mar 2023 21:41:30 +0800 From: Peng Zhang To: , , , , CC: , , , , , ZhangPeng Subject: [PATCH v4 3/6] userfaultfd: convert copy_huge_page_from_user() to copy_folio_from_user() Date: Thu, 30 Mar 2023 21:40:42 +0800 Message-ID: <20230330134045.375163-4-zhangpeng362@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230330134045.375163-1-zhangpeng362@huawei.com> References: <20230330134045.375163-1-zhangpeng362@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To kwepemm600020.china.huawei.com (7.193.23.147) X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: B42CE100003 X-Stat-Signature: jhaeoqsaaeryforbykqkethe4ehqz7nm X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680183697-948648 X-HE-Meta: U2FsdGVkX1/FOGZvG3NyUQb5/WZfXjFesTa3rjx6V0j6f1ZV502bOZ9kssfexlvXaoPFi1JJS8rhZv+G4ruVobx1OjTKp24ivzINnhB8zbY1wOTR/ELoVrO3feIIo3qWQh8shkSOJb2smhshdIGUGY97YCxZG20tsNfZ5SYR1cd0tPlnrErod/hllVfBINs6kblxg5j5N6gtNoC+l9pXm49ZXpatTSZPFSFFSYdEfYo/rheQHelDsj42vHsxbruJATYPs+CsDGUoxrb/aaY29BZOamCYczYDV+hi8x4mTM9J7KSNY1Cu3DOU74+XegjGAYmTt6K/UNj3RhAfoDH4s4kZOnnBsdu1JEcEfIly6vuUm+pzioor56rcs/9X6IEhIqVpxqPltcxTSZelW/a5PAWqHcWvP5zLRg+gz1eForF+lADlc1Tg79+KIA17E+ymQdsh1O6EcaIEdCWaa4qigHUUcFJN+b+rJN2BDvDJu5qW+E4Ius/Xfvyf2HAf4cPeyjUQ7BQlj7SktvfVZYLZUri9tn2yLFMmgi4dtPW/Oreo+QrQ3pCCgQ/nsYpLRQYI2Klu3uz2xdA2BHNWm56XnPXkT/MVInegcfueU9Bh1h4gSE25qk2pNldfSViOOPyKal8JodII+4MjhKEoWa59Hk/F4qnY8YNsOjW9rbbdskeSJSEyjiToZw3924geO1TL2vCCCbu74oDcHWFPrsz2fRhuQnyfo02i7TrJhItEtGg24Ml09IezvbpS8RcsL88bLyjhqBfjYM4a6wPCreIx9wi9ElsU8z15H57CBp416HcWKU3K6tJBbpNgDoCGH1cQlHCp0L9YKbemNpW6QL5RS5t+06QeZPncVmhhmUnZtRiGXHhzaaS55XQy9ya6AvOL8oGoKfaOG2nJObHI2mPu5Tz26RwzgMyfdNQclY5QPUica4XqJU3H8kZa66E8/0u6r+n0HRhgXtQIj30dZjX 4w5ScY2u uZ8pFVIMci9DkzdGiOEEReohs+jLjtPIe1C/6sl0tjgDz2i/qAQctNO8zUf450CirUxQ8pvR7eVmO6PN5f/8JBu7CnGt6R6bHBir7XERXeD3DvtZHRdvkMlz3ef3WjSL3jAE5bnPuzMhdKhJc+6rVS6j+GvI+MfZRvYvEtI37XnFrdRx0tkmlTCSJLgIMn+YflIq2js8OaYgY3HPhPkTWF+iKQDAlwX419hXe3wQG0oBGcF7+/zRD+yKkJzDkYKibzzZLl9y4sITkZgJ12Bccw7OC6yKC5EMVsvhZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: ZhangPeng Replace copy_huge_page_from_user() with copy_folio_from_user(). copy_folio_from_user() does the same as copy_huge_page_from_user(), but takes in a folio instead of a page. Convert page_kaddr to kaddr in copy_folio_from_user() to do indenting cleanup. Signed-off-by: ZhangPeng Reviewed-by: Sidhartha Kumar --- include/linux/mm.h | 7 +++---- mm/hugetlb.c | 5 ++--- mm/memory.c | 26 ++++++++++++-------------- mm/userfaultfd.c | 6 ++---- 4 files changed, 19 insertions(+), 25 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index e249208f8fbe..cf4d773ca506 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3682,10 +3682,9 @@ extern void copy_user_huge_page(struct page *dst, struct page *src, unsigned long addr_hint, struct vm_area_struct *vma, unsigned int pages_per_huge_page); -extern long copy_huge_page_from_user(struct page *dst_page, - const void __user *usr_src, - unsigned int pages_per_huge_page, - bool allow_pagefault); +long copy_folio_from_user(struct folio *dst_folio, + const void __user *usr_src, + bool allow_pagefault); /** * vma_is_special_huge - Are transhuge page-table entries considered special? diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 7e4a80769c9e..aade1b513474 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6217,9 +6217,8 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, goto out; } - ret = copy_huge_page_from_user(&folio->page, - (const void __user *) src_addr, - pages_per_huge_page(h), false); + ret = copy_folio_from_user(folio, (const void __user *) src_addr, + false); /* fallback to copy_from_user outside mmap_lock */ if (unlikely(ret)) { diff --git a/mm/memory.c b/mm/memory.c index c47b8991410a..9d59dad319b3 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5854,35 +5854,33 @@ void copy_user_huge_page(struct page *dst, struct page *src, process_huge_page(addr_hint, pages_per_huge_page, copy_subpage, &arg); } -long copy_huge_page_from_user(struct page *dst_page, - const void __user *usr_src, - unsigned int pages_per_huge_page, - bool allow_pagefault) +long copy_folio_from_user(struct folio *dst_folio, + const void __user *usr_src, + bool allow_pagefault) { - void *page_kaddr; + void *kaddr; unsigned long i, rc = 0; - unsigned long ret_val = pages_per_huge_page * PAGE_SIZE; + unsigned int nr_pages = folio_nr_pages(dst_folio); + unsigned long ret_val = nr_pages * PAGE_SIZE; struct page *subpage; - for (i = 0; i < pages_per_huge_page; i++) { - subpage = nth_page(dst_page, i); - page_kaddr = kmap_local_page(subpage); + for (i = 0; i < nr_pages; i++) { + subpage = folio_page(dst_folio, i); + kaddr = kmap_local_page(subpage); if (!allow_pagefault) pagefault_disable(); - rc = copy_from_user(page_kaddr, - usr_src + i * PAGE_SIZE, PAGE_SIZE); + rc = copy_from_user(kaddr, usr_src + i * PAGE_SIZE, PAGE_SIZE); if (!allow_pagefault) pagefault_enable(); - kunmap_local(page_kaddr); + kunmap_local(kaddr); ret_val -= (PAGE_SIZE - rc); if (rc) break; - flush_dcache_page(subpage); - cond_resched(); } + flush_dcache_folio(dst_folio); return ret_val; } #endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HUGETLBFS */ diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index efa9e1d681ee..b453a4d2a0d3 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -422,10 +422,8 @@ static __always_inline ssize_t mfill_atomic_hugetlb( mmap_read_unlock(dst_mm); BUG_ON(!page); - err = copy_huge_page_from_user(page, - (const void __user *)src_addr, - vma_hpagesize / PAGE_SIZE, - true); + err = copy_folio_from_user(page_folio(page), + (const void __user *)src_addr, true); if (unlikely(err)) { err = -EFAULT; goto out; From patchwork Thu Mar 30 13:40:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peng Zhang X-Patchwork-Id: 13194193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7769C6FD1D for ; Thu, 30 Mar 2023 13:41:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 25893900004; Thu, 30 Mar 2023 09:41:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 209B7900002; Thu, 30 Mar 2023 09:41:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05BF7900004; Thu, 30 Mar 2023 09:41:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D8B8F900002 for ; Thu, 30 Mar 2023 09:41:41 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id AE2D740F3F for ; Thu, 30 Mar 2023 13:41:41 +0000 (UTC) X-FDA: 80625677202.02.5859ECA Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by imf20.hostedemail.com (Postfix) with ESMTP id C8A051C001A for ; Thu, 30 Mar 2023 13:41:38 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680183699; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CZNHzSt0tHTjqMCHSS9dyn9pURJtyKXkj/QjHG9AFDE=; b=jALVhDYCEcnUqdEEDbQqAlXInsFBicKBNsWQ2cMKnlkwCA9g5Uz4rmTBt0szQm5HU5M0W/ HWJzNPWoer6qW6stiwIafY9eOikHJIqUIy+qSU3a96aHE/hYf8jX2Xb533uVjVWhwL9tcM TMKIBji0O0B4BYbULDvcUcyTPwTFLTo= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680183699; a=rsa-sha256; cv=none; b=BTGR6BApSIOpuESJJ3ogWmPRLeYsuvK1ZGbepwjECtaYoBluA4pV2hsT2IbzNe3/Q6aCSS WOupA4QqlfDBYjr5fjykvoeZJCZ1fpXC9Ut27LbbuyyBd3k/Asv60zvi3undateKuhbCH+ ukgSLVMEwJVIPO+XiN9j+F99JWRARl8= Received: from kwepemm600020.china.huawei.com (unknown [172.30.72.56]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4PnPf41JkyzKwBL; Thu, 30 Mar 2023 21:39:08 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by kwepemm600020.china.huawei.com (7.193.23.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Thu, 30 Mar 2023 21:41:31 +0800 From: Peng Zhang To: , , , , CC: , , , , , ZhangPeng Subject: [PATCH v4 4/6] userfaultfd: convert mfill_atomic_hugetlb() to use a folio Date: Thu, 30 Mar 2023 21:40:43 +0800 Message-ID: <20230330134045.375163-5-zhangpeng362@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230330134045.375163-1-zhangpeng362@huawei.com> References: <20230330134045.375163-1-zhangpeng362@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To kwepemm600020.china.huawei.com (7.193.23.147) X-CFilter-Loop: Reflected X-Stat-Signature: fu8b4uyh88j9zs3tqaiqkh4oeoti4uam X-Rspam-User: X-Rspamd-Queue-Id: C8A051C001A X-Rspamd-Server: rspam06 X-HE-Tag: 1680183698-593148 X-HE-Meta: U2FsdGVkX1+GIVGep+JJE9O67cz5krNI14gMo+MFBZF9V5s2H89dR/F9ekII2jDYBU0vKTRVLqaDvs5O2wyQKSTxkgGLfiiKwoqv1+NTDvK2F+G3UW7RQt3PBY82bmq3KAbfmYJ2jXrgx0KR9lNMPprQpuMyMqb5RQOo1oZ7IgkzaN6LRIf3Yp+JjrqGvcBSWzXub7h/u9JUQ5S2rDrH9TvOjWpiPfFgjR3HtI940dI21+AG20u1cHF9fUxPpuUoObC4+hNb4KoHgCmuncXOS0bs4O/BmZciQ8fAq0HwgkaO20MWdnT2v3Ptkky6jIpm9l7iX5W5/62nfC2OwAcy2V74VfVLR81qiIleW2jMeYhor/Os3i6HTdKJdUnMz88l27Cao4/m+YK2992OXtcgxZEg08FNDoAe9EJ1h/KcfdPPaBrNtOjMdJ4PwblYVJDnx8BQc3wu/JnWPPDNd+ENMRHl/9Zer7glEgYGBonhO9g9s3AJUKXL9HoHjP6IaSqFwJkRDLghtY0A/IhZlAXaNznEjYTr//020jRIsSwjG75spuGgYMM8wXUVk4sXSv7ZLrQbp4ySJACLOQw0lm5SO3VvRf5wD/PiF4Qan6o5qwnc9P2eqTt9KFtGrt7ZsCBxLZbOqFRrApC9W2q3V4PeTTrrxLZN4erLywHdg5cyTu9UhWHXrlcE4u9gkiRDq1VsCByr1TnudF2qXio1kK1hRoGxMOXdohQPZkePnKUboNdbJPy4wEt4ooGy9WaR5uSB/QXaqknCcm+mLEBkZ3FxOPIZGWo6G5Zm3FKFxLaCabDmpd3kP35mkRlRkeNYvFJtVht1JDBH5e1BGRYhRX6G/+EDhViz81HgaDzdvJg64o5HNwnjbgEya7I8A+PvymhC3TD9yRJvXDIPaNPZOERhGxz3xJB0TYPVjVVeDnqFQNkT1cLR85joSRxs/i6pgdwMCEmwu5WM60Dhf0UwQQh 0PEnfrMI THCis3G2mXIDHcHu6d2U6Li0eSN7NtwAmTfKcCp/sEUPNHIVugh/tYEpdDBNlpeBrC+rf3R58YcgXdVhfTzmczDbT1Zy+LD36TGFf227O8SLpdhHyxn522Wva2+G7Zws8VFa1r47xG9E0El3gmevAJ2b8RJ+MSHYj+7cFdnDRE6gqYvdDHE8BZqkct50pIBluiU0fwINLjFUNlexKk/ogBqEQDFNAzebC/WfEN0d6H/LbDbPOT4N46uwqiuk+IoV+nRw4rCoBCiGNLPeBebeAJweqQiEOQdnixwtm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: ZhangPeng Convert hugetlb_mfill_atomic_pte() to take in a folio pointer instead of a page pointer. Convert mfill_atomic_hugetlb() to use a folio. Signed-off-by: ZhangPeng Reviewed-by: Sidhartha Kumar --- include/linux/hugetlb.h | 4 ++-- mm/hugetlb.c | 26 +++++++++++++------------- mm/userfaultfd.c | 16 ++++++++-------- 3 files changed, 23 insertions(+), 23 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 2a758bcd6719..28703fe22386 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -163,7 +163,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, unsigned long dst_addr, unsigned long src_addr, uffd_flags_t flags, - struct page **pagep); + struct folio **foliop); #endif /* CONFIG_USERFAULTFD */ bool hugetlb_reserve_pages(struct inode *inode, long from, long to, struct vm_area_struct *vma, @@ -397,7 +397,7 @@ static inline int hugetlb_mfill_atomic_pte(pte_t *dst_pte, unsigned long dst_addr, unsigned long src_addr, uffd_flags_t flags, - struct page **pagep) + struct folio **foliop) { BUG(); return 0; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index aade1b513474..c88f856ec2e2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6178,7 +6178,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, unsigned long dst_addr, unsigned long src_addr, uffd_flags_t flags, - struct page **pagep) + struct folio **foliop) { struct mm_struct *dst_mm = dst_vma->vm_mm; bool is_continue = uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE); @@ -6201,8 +6201,8 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, if (IS_ERR(folio)) goto out; folio_in_pagecache = true; - } else if (!*pagep) { - /* If a page already exists, then it's UFFDIO_COPY for + } else if (!*foliop) { + /* If a folio already exists, then it's UFFDIO_COPY for * a non-missing case. Return -EEXIST. */ if (vm_shared && @@ -6237,33 +6237,33 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, ret = -ENOMEM; goto out; } - *pagep = &folio->page; - /* Set the outparam pagep and return to the caller to + *foliop = folio; + /* Set the outparam foliop and return to the caller to * copy the contents outside the lock. Don't free the - * page. + * folio. */ goto out; } } else { if (vm_shared && hugetlbfs_pagecache_present(h, dst_vma, dst_addr)) { - put_page(*pagep); + folio_put(*foliop); ret = -EEXIST; - *pagep = NULL; + *foliop = NULL; goto out; } folio = alloc_hugetlb_folio(dst_vma, dst_addr, 0); if (IS_ERR(folio)) { - put_page(*pagep); + folio_put(*foliop); ret = -ENOMEM; - *pagep = NULL; + *foliop = NULL; goto out; } - copy_user_huge_page(&folio->page, *pagep, dst_addr, dst_vma, + copy_user_huge_page(&folio->page, &(*foliop)->page, dst_addr, dst_vma, pages_per_huge_page(h)); - put_page(*pagep); - *pagep = NULL; + folio_put(*foliop); + *foliop = NULL; } /* diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index b453a4d2a0d3..9e95af5ce2dd 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -322,7 +322,7 @@ static __always_inline ssize_t mfill_atomic_hugetlb( pte_t *dst_pte; unsigned long src_addr, dst_addr; long copied; - struct page *page; + struct folio *folio; unsigned long vma_hpagesize; pgoff_t idx; u32 hash; @@ -342,7 +342,7 @@ static __always_inline ssize_t mfill_atomic_hugetlb( src_addr = src_start; dst_addr = dst_start; copied = 0; - page = NULL; + folio = NULL; vma_hpagesize = vma_kernel_pagesize(dst_vma); /* @@ -411,7 +411,7 @@ static __always_inline ssize_t mfill_atomic_hugetlb( } err = hugetlb_mfill_atomic_pte(dst_pte, dst_vma, dst_addr, - src_addr, flags, &page); + src_addr, flags, &folio); hugetlb_vma_unlock_read(dst_vma); mutex_unlock(&hugetlb_fault_mutex_table[hash]); @@ -420,9 +420,9 @@ static __always_inline ssize_t mfill_atomic_hugetlb( if (unlikely(err == -ENOENT)) { mmap_read_unlock(dst_mm); - BUG_ON(!page); + BUG_ON(!folio); - err = copy_folio_from_user(page_folio(page), + err = copy_folio_from_user(folio, (const void __user *)src_addr, true); if (unlikely(err)) { err = -EFAULT; @@ -433,7 +433,7 @@ static __always_inline ssize_t mfill_atomic_hugetlb( dst_vma = NULL; goto retry; } else - BUG_ON(page); + BUG_ON(folio); if (!err) { dst_addr += vma_hpagesize; @@ -450,8 +450,8 @@ static __always_inline ssize_t mfill_atomic_hugetlb( out_unlock: mmap_read_unlock(dst_mm); out: - if (page) - put_page(page); + if (folio) + folio_put(folio); BUG_ON(copied < 0); BUG_ON(err > 0); BUG_ON(!copied && !err); From patchwork Thu Mar 30 13:40:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peng Zhang X-Patchwork-Id: 13194192 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 345C7C761A6 for ; Thu, 30 Mar 2023 13:41:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B5A26900003; Thu, 30 Mar 2023 09:41:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B0AC2900002; Thu, 30 Mar 2023 09:41:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A99A900003; Thu, 30 Mar 2023 09:41:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8B063900002 for ; Thu, 30 Mar 2023 09:41:40 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4A0D8C0F36 for ; Thu, 30 Mar 2023 13:41:40 +0000 (UTC) X-FDA: 80625677160.16.412DD21 Received: from szxga08-in.huawei.com (szxga08-in.huawei.com [45.249.212.255]) by imf14.hostedemail.com (Postfix) with ESMTP id BF70A10000C for ; Thu, 30 Mar 2023 13:41:36 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf14.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.255 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680183698; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=E35cqwRK5yY+181xYgcO5dDDmWMRjom4aSv6GskoIZE=; b=SNE+aOWL56az6KP4oFZHKSBwXCSf4VGy8NIU8UYl1BD6Vi9kKuk7Ob71cDs31gqkf6DYO0 0izX2e3tEUtgfKFTsxFP+Yf3d9IvFlc2qI02/nlf3uuenj9G6EChgGCadLUjtwr+dXrGaL cxUdYhZVmhOk8/bxqcsMXNZE+yITgsM= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf14.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.255 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680183698; a=rsa-sha256; cv=none; b=QaXe7+8+GuL7PvsCFWsqtVmHMHn4sw3Li6rRx+JZ4YfXXAGcw7sIsSQiUeoicwZ7DGVDza JagLMP76AU51J8SnK1Oh1XM2HTnwYiaOFNm67+ihuk7SiDZqj0EVajcnzR8Di307cJAloe ilOBfIwYjXRCDYk9ewYqHI5a5aj3j5I= Received: from kwepemm600020.china.huawei.com (unknown [172.30.72.54]) by szxga08-in.huawei.com (SkyGuard) with ESMTP id 4PnPd40Cp3z17QWN; Thu, 30 Mar 2023 21:38:16 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by kwepemm600020.china.huawei.com (7.193.23.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Thu, 30 Mar 2023 21:41:32 +0800 From: Peng Zhang To: , , , , CC: , , , , , ZhangPeng Subject: [PATCH v4 5/6] mm: convert copy_user_huge_page() to copy_user_folio() Date: Thu, 30 Mar 2023 21:40:44 +0800 Message-ID: <20230330134045.375163-6-zhangpeng362@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230330134045.375163-1-zhangpeng362@huawei.com> References: <20230330134045.375163-1-zhangpeng362@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To kwepemm600020.china.huawei.com (7.193.23.147) X-CFilter-Loop: Reflected X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: BF70A10000C X-Rspam-User: X-Stat-Signature: h7i5w49pnmqow75kuwdy6hdo1i7q338i X-HE-Tag: 1680183696-983831 X-HE-Meta: U2FsdGVkX18rRts88X4IiG2Onfj1WFG67lllvBPaf+y4/KjNyw5btC1IM+tCpRqsyKU0JF5Fs5JqL53gHU3xeuiTwN6f+4Un7eMTOIuydhrqGD4rpbEzfWms7ZuW2Sqj4uaIwL1+Qbz9RO1GlUoOatU8Ui7pEMYJQHK2URk/WqXV22dDTHTLytmcF2WNBWCnzJrJhqfoBqxUs6DteuiL0BSbHRsokp7WHsjCwGwlxN4p0vLYa6eY8Os2mApNNFLwHrYunk3H1wsDxQHT47NbN0XcaMPTkuRRgPlWSpOtCkI8DxPmzSjzdwmP6GabTjmin9h8e2WffAMITLt3OO6WXvG8oVcgPpUOPQQPO1xLJMMuLPQvrdffj8h/iu9MKIbpF8lfwsmmugms/h9PVEFk8CdQNXPhVrNnSwetWkqUu2VZxK11DJl0ShCgox9hIEAE8GfbnKHnE0GiPoPWYcLN+vBOfqxHbM0Ov6J42B8kwOdxlWogew8MPKyjJf137Q8B3mVfviM/PVihMB33EFh06hDiafa5WioPd//vYeJ3yWQdsL7Xg8qeAeZSV4kXLyLDN5m7N5c/umes6gOFFyPJj9ShtYDDKQEsaDk0TNwODGwB48URgBb8NPrm6ebL+Zxlb0WFr1YfxhjwoNfd3/lWhOkeZmHMlWaxBQmhLM/JhjUwGgJSh/jKa4hAN6O1pUbuAtgaO6RyBzwhglz+9BS7fZIHWX7AahXclR80xSQCmKg67HDRmh0WHlKsila6jnug7/ej/MtT8wwSBsoiaQjAVv99Wr14dxWyPBfL/UglAkWj2cacv4XPCA4cHJpqpzjmhtS2QbjHZUIM86PXyhLVj2pBQOhZepob8Rw8tnt+/e+VfDPOROmBLxvGgUyJ0dPhv5MiPglRYPhzPVGUcThvHubiU5gaagctwo4g+8gB3DzHNZsQAHfN7R5eC4hz3nfZFJ5FNdmE9oP+FGxsNUK 4JeJwcay oo9bD/XzAyq97DlSedqKqIwVDe3UJGb+N3VuSXWzvktsgXDz2Ym/r7GDYWtWKH0cfA+wOQft2617quRPXrLJ34N2f305z9JRi8U094UezOjGMyLfNME7PtcKEPM80ag/f2Yf1ifwCzI4nGDXlFd7Kk3ZG8Sfzy8g8WAxC1y9V0eQtz1wcqaip+HzlYzIX54rvVUokJcfKqtAb1RdARaXGhd06vWYHSjH4CluP3yTPLEAeLJ1+34czBpWjAr4eEVG7tGAfK6cyD2TRRVF+PbFjZHR96A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: ZhangPeng Replace copy_user_huge_page() with copy_user_folio(). copy_user_folio() does the same as copy_user_huge_page(), but takes in folios instead of pages. Convert copy_user_gigantic_page() to take in folios. Signed-off-by: ZhangPeng --- include/linux/mm.h | 8 ++++---- mm/hugetlb.c | 12 ++++++------ mm/memory.c | 28 ++++++++++++++-------------- 3 files changed, 24 insertions(+), 24 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index cf4d773ca506..7b8fb88fb14d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3678,10 +3678,10 @@ extern const struct attribute_group memory_failure_attr_group; extern void clear_huge_page(struct page *page, unsigned long addr_hint, unsigned int pages_per_huge_page); -extern void copy_user_huge_page(struct page *dst, struct page *src, - unsigned long addr_hint, - struct vm_area_struct *vma, - unsigned int pages_per_huge_page); +void copy_user_folio(struct folio *dst, struct folio *src, + unsigned long addr_hint, + struct vm_area_struct *vma, + unsigned int pages_per_huge_page); long copy_folio_from_user(struct folio *dst_folio, const void __user *usr_src, bool allow_pagefault); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c88f856ec2e2..fc3017a6a818 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5097,8 +5097,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, ret = PTR_ERR(new_folio); break; } - copy_user_huge_page(&new_folio->page, ptepage, addr, dst_vma, - npages); + copy_user_folio(new_folio, page_folio(ptepage), addr, dst_vma, + npages); put_page(ptepage); /* Install the new hugetlb folio if src pte stable */ @@ -5616,8 +5616,8 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, goto out_release_all; } - copy_user_huge_page(&new_folio->page, old_page, address, vma, - pages_per_huge_page(h)); + copy_user_folio(new_folio, page_folio(old_page), address, vma, + pages_per_huge_page(h)); __folio_mark_uptodate(new_folio); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, haddr, @@ -6260,8 +6260,8 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, *foliop = NULL; goto out; } - copy_user_huge_page(&folio->page, &(*foliop)->page, dst_addr, dst_vma, - pages_per_huge_page(h)); + copy_user_folio(folio, *foliop, dst_addr, dst_vma, + pages_per_huge_page(h)); folio_put(*foliop); *foliop = NULL; } diff --git a/mm/memory.c b/mm/memory.c index 9d59dad319b3..8bddbb927806 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5801,21 +5801,21 @@ void clear_huge_page(struct page *page, process_huge_page(addr_hint, pages_per_huge_page, clear_subpage, page); } -static void copy_user_gigantic_page(struct page *dst, struct page *src, - unsigned long addr, - struct vm_area_struct *vma, - unsigned int pages_per_huge_page) +static void copy_user_gigantic_page(struct folio *dst, struct folio *src, + unsigned long addr, + struct vm_area_struct *vma, + unsigned int pages_per_huge_page) { int i; - struct page *dst_base = dst; - struct page *src_base = src; + struct page *dst_page; + struct page *src_page; for (i = 0; i < pages_per_huge_page; i++) { - dst = nth_page(dst_base, i); - src = nth_page(src_base, i); + dst_page = folio_page(dst, i); + src_page = folio_page(src, i); cond_resched(); - copy_user_highpage(dst, src, addr + i*PAGE_SIZE, vma); + copy_user_highpage(dst_page, src_page, addr + i*PAGE_SIZE, vma); } } @@ -5833,15 +5833,15 @@ static void copy_subpage(unsigned long addr, int idx, void *arg) addr, copy_arg->vma); } -void copy_user_huge_page(struct page *dst, struct page *src, - unsigned long addr_hint, struct vm_area_struct *vma, - unsigned int pages_per_huge_page) +void copy_user_folio(struct folio *dst, struct folio *src, + unsigned long addr_hint, struct vm_area_struct *vma, + unsigned int pages_per_huge_page) { unsigned long addr = addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); struct copy_subpage_arg arg = { - .dst = dst, - .src = src, + .dst = &dst->page, + .src = &src->page, .vma = vma, }; From patchwork Thu Mar 30 13:40:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peng Zhang X-Patchwork-Id: 13194195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA491C761A6 for ; Thu, 30 Mar 2023 13:41:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 28237900002; Thu, 30 Mar 2023 09:41:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1B913280001; Thu, 30 Mar 2023 09:41:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D7B17900002; Thu, 30 Mar 2023 09:41:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BF481900005 for ; Thu, 30 Mar 2023 09:41:43 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9434A1209C8 for ; Thu, 30 Mar 2023 13:41:43 +0000 (UTC) X-FDA: 80625677286.12.02F01F3 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf22.hostedemail.com (Postfix) with ESMTP id 0016AC001B for ; Thu, 30 Mar 2023 13:41:40 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf22.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680183701; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ahDIrwpjV7IDBnWOTGVf77qmCasBuiHazajE/lFOPL0=; b=7APXX9A6cZsYlO+Tqnk6Iazdo2IwP0qm2ng7jSEbjwzv3SDq2AzevB+E4eAA3LDYqV2A3F yKj5ESlyoZsLsLdkO076Z4S0e7MU7Tr23HsghqXJYCR0TDtSo4+NgMOqSWjwC61QJBKKtz TWWXX/tMF9Mc/T1l1jpCicaenJDdQ8I= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf22.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680183701; a=rsa-sha256; cv=none; b=SVK/ggtawWj19KKRTrCMGMVoGA1ez3TKMf3zMSo668D/HKV6q5lHE6CUm9/ImTA+0mGlyM sKL2sffRI8bHjSV6yXyWTAW88tq+WnaozyKDXw00bKjHVCvTto+PWrOmNww0BpRGcUgH30 BfuD0fDuIS/zPFnoPS4q3AlJHbR5iVM= Received: from kwepemm600020.china.huawei.com (unknown [172.30.72.54]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4PnPd413YbzgZhs; Thu, 30 Mar 2023 21:38:16 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by kwepemm600020.china.huawei.com (7.193.23.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Thu, 30 Mar 2023 21:41:33 +0800 From: Peng Zhang To: , , , , CC: , , , , , ZhangPeng Subject: [PATCH v4 6/6] userfaultfd: convert mfill_atomic() to use a folio Date: Thu, 30 Mar 2023 21:40:45 +0800 Message-ID: <20230330134045.375163-7-zhangpeng362@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230330134045.375163-1-zhangpeng362@huawei.com> References: <20230330134045.375163-1-zhangpeng362@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To kwepemm600020.china.huawei.com (7.193.23.147) X-CFilter-Loop: Reflected X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 0016AC001B X-Rspam-User: X-Stat-Signature: m8a1yss6tm5a5cbbsc43cor3j8qekeiu X-HE-Tag: 1680183700-871311 X-HE-Meta: U2FsdGVkX1+qGIyJmVg9b+ijYMsAUSnyT8sOAkBaOj8K7P4lXoiiI4OcSz9cv5V5C8Itsyhwj4badXHtgaTVp9occkFqcUUSSebSxvNU9CB9CPc5HTXSTe7U5UphmnYR8sQHmBLg80zLA8UFWM82uuZhvaVsjvE6zUX8X+CGHGXEGrevlaeYH4E3aMmQ6S+F5UHz8f1pBZyQcXx6qasSFdfenugefZzCu+KmzxSG9EuCLHnne78+fMdSFAUsY6WTNWzAXja1gdea6/VhdpsBcQLbFtbn+G2FaakWgTk7HelqydbaPZJyGzfeTgxo9iPNAAD555tAFTe+mLKudjXHf/fkoDAhRD+072ZXUYeXUQ1MBBmFvH+sCtS4gL9EsmgSUoKwblRshSjctRyZEkQv5vIvDE2CZFjtuSLbqULbTOHX++tuNQNBjl6isxRGaY8RBQfNJQfdfrS457FIlkPjv+dJA4bYd2U+bsvcOJyfayUpLfAMAPGu9ldxNK+Ug5hb6wxZPk/lkO8PjxM80Ko0z1rYgVNSD4cap3EdeegA8xVdY6XQwvkbKpTvUoazrPE0dptacsc3L80K4blp3cumMvwZw3ijvaC/Lwl1B0bQRMi5TtVnTKFUbbN6PerAeb4L6YsR0LDR/Kww1MN1sAcigjBt/W110SwoGfQsfupxPfmkd8ki/M/TaxyLX8NMKJr0C0kXKHGMApUlMC/iX670kQBwC+rt3XgDrSuiwusfUygukrX591rvXbvBN8i1CAFdya4jHwobTKCGotUNthhLy8PbP1V9670fzkudAXSwcKwvr1cLLn6s+tm2AKKgNBhgtxTdTpbx8qRnLMROvSGW1s9pqSJ3NVEPpoxGG8oJ6RDznls+sK2dHcpedZr8+kYezvC1b7wDa6LRv2gllmUntnhmDuUZgG/8ssnwN2WwMOYtqdRv3E18Hxs01vmGfXRVkF3PAZ6kcGAxy/7qaCS dt1VjmPR asgzhj6qNdCUqjB7Ow/3VVgiwFXiDzaWIeqDoadT5E+EIxnTTEqg8S4kJkC66p+sz1Ye8L0z639eNPK4le0hKCU2pspbnxfdNrQhkvhGoJErMoDZh6ldjgfi4yYg8HlqyOGZ6ItSioF2gFIAtzSChJ8GZr4TWmTaByXo+v14ogVV78A/VhWKuKE2eA/PbjNU9p37PlgxzzOyJ2UOvKludry1AVfd1pvDM7H2SZX8fSmMPno3GszYvBCUA545G+wkvgzSExvEMHYNDqSgUox+jVObX6Iedy4v0nrRq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: ZhangPeng Convert mfill_atomic_pte_copy(), shmem_mfill_atomic_pte() and mfill_atomic_pte() to take in a folio pointer. Convert mfill_atomic() to use a folio. Signed-off-by: ZhangPeng --- include/linux/shmem_fs.h | 4 ++-- mm/shmem.c | 16 ++++++++-------- mm/userfaultfd.c | 34 +++++++++++++++++----------------- 3 files changed, 27 insertions(+), 27 deletions(-) diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h index 3bb8d21edbb3..9e151ba45068 100644 --- a/include/linux/shmem_fs.h +++ b/include/linux/shmem_fs.h @@ -158,10 +158,10 @@ extern int shmem_mfill_atomic_pte(pmd_t *dst_pmd, unsigned long dst_addr, unsigned long src_addr, uffd_flags_t flags, - struct page **pagep); + struct folio **foliop); #else /* !CONFIG_SHMEM */ #define shmem_mfill_atomic_pte(dst_pmd, dst_vma, dst_addr, \ - src_addr, flags, pagep) ({ BUG(); 0; }) + src_addr, flags, foliop) ({ BUG(); 0; }) #endif /* CONFIG_SHMEM */ #endif /* CONFIG_USERFAULTFD */ diff --git a/mm/shmem.c b/mm/shmem.c index 6c08f5a75d3a..9218c955f482 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2548,7 +2548,7 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd, unsigned long dst_addr, unsigned long src_addr, uffd_flags_t flags, - struct page **pagep) + struct folio **foliop) { struct inode *inode = file_inode(dst_vma->vm_file); struct shmem_inode_info *info = SHMEM_I(inode); @@ -2566,14 +2566,14 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd, * and now we find ourselves with -ENOMEM. Release the page, to * avoid a BUG_ON in our caller. */ - if (unlikely(*pagep)) { - put_page(*pagep); - *pagep = NULL; + if (unlikely(*foliop)) { + folio_put(*foliop); + *foliop = NULL; } return -ENOMEM; } - if (!*pagep) { + if (!*foliop) { ret = -ENOMEM; folio = shmem_alloc_folio(gfp, info, pgoff); if (!folio) @@ -2605,7 +2605,7 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd, /* fallback to copy_from_user outside mmap_lock */ if (unlikely(ret)) { - *pagep = &folio->page; + *foliop = folio; ret = -ENOENT; /* don't free the page */ goto out_unacct_blocks; @@ -2616,9 +2616,9 @@ int shmem_mfill_atomic_pte(pmd_t *dst_pmd, clear_user_highpage(&folio->page, dst_addr); } } else { - folio = page_folio(*pagep); + folio = *foliop; VM_BUG_ON_FOLIO(folio_test_large(folio), folio); - *pagep = NULL; + *foliop = NULL; } VM_BUG_ON(folio_test_locked(folio)); diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 9e95af5ce2dd..74b122887901 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -133,13 +133,13 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd, unsigned long dst_addr, unsigned long src_addr, uffd_flags_t flags, - struct page **pagep) + struct folio **foliop) { void *page_kaddr; int ret; struct folio *folio; - if (!*pagep) { + if (!*foliop) { ret = -ENOMEM; folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, dst_vma, dst_addr, false); if (!folio) @@ -171,16 +171,16 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd, /* fallback to copy_from_user outside mmap_lock */ if (unlikely(ret)) { ret = -ENOENT; - *pagep = &folio->page; + *foliop = folio; /* don't free the page */ goto out; } flush_dcache_folio(folio); } else { - folio = page_folio(*pagep); + folio = *foliop; VM_BUG_ON_FOLIO(folio_test_large(folio), folio); - *pagep = NULL; + *foliop = NULL; } /* @@ -471,7 +471,7 @@ static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd, unsigned long dst_addr, unsigned long src_addr, uffd_flags_t flags, - struct page **pagep) + struct folio **foliop) { ssize_t err; @@ -494,14 +494,14 @@ static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd, if (uffd_flags_mode_is(flags, MFILL_ATOMIC_COPY)) err = mfill_atomic_pte_copy(dst_pmd, dst_vma, dst_addr, src_addr, - flags, pagep); + flags, foliop); else err = mfill_atomic_pte_zeropage(dst_pmd, dst_vma, dst_addr); } else { err = shmem_mfill_atomic_pte(dst_pmd, dst_vma, dst_addr, src_addr, - flags, pagep); + flags, foliop); } return err; @@ -519,7 +519,7 @@ static __always_inline ssize_t mfill_atomic(struct mm_struct *dst_mm, pmd_t *dst_pmd; unsigned long src_addr, dst_addr; long copied; - struct page *page; + struct folio *folio; /* * Sanitize the command parameters: @@ -534,7 +534,7 @@ static __always_inline ssize_t mfill_atomic(struct mm_struct *dst_mm, src_addr = src_start; dst_addr = dst_start; copied = 0; - page = NULL; + folio = NULL; retry: mmap_read_lock(dst_mm); @@ -630,16 +630,16 @@ static __always_inline ssize_t mfill_atomic(struct mm_struct *dst_mm, BUG_ON(pmd_trans_huge(*dst_pmd)); err = mfill_atomic_pte(dst_pmd, dst_vma, dst_addr, - src_addr, flags, &page); + src_addr, flags, &folio); cond_resched(); if (unlikely(err == -ENOENT)) { void *page_kaddr; mmap_read_unlock(dst_mm); - BUG_ON(!page); + BUG_ON(!folio); - page_kaddr = kmap_local_page(page); + page_kaddr = kmap_local_folio(folio, 0); err = copy_from_user(page_kaddr, (const void __user *) src_addr, PAGE_SIZE); @@ -648,10 +648,10 @@ static __always_inline ssize_t mfill_atomic(struct mm_struct *dst_mm, err = -EFAULT; goto out; } - flush_dcache_page(page); + flush_dcache_folio(folio); goto retry; } else - BUG_ON(page); + BUG_ON(folio); if (!err) { dst_addr += PAGE_SIZE; @@ -668,8 +668,8 @@ static __always_inline ssize_t mfill_atomic(struct mm_struct *dst_mm, out_unlock: mmap_read_unlock(dst_mm); out: - if (page) - put_page(page); + if (folio) + folio_put(folio); BUG_ON(copied < 0); BUG_ON(err > 0); BUG_ON(!copied && !err);