From patchwork Mon Jun 26 17:14:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13293230 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21F61EB64DA for ; Mon, 26 Jun 2023 17:14:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 941E88D0006; Mon, 26 Jun 2023 13:14:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8CBDF8D0001; Mon, 26 Jun 2023 13:14:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E1068D0006; Mon, 26 Jun 2023 13:14:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 701EF8D0001 for ; Mon, 26 Jun 2023 13:14:47 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id F17A0140216 for ; Mon, 26 Jun 2023 17:14:46 +0000 (UTC) X-FDA: 80945548572.16.F0BB77D Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf17.hostedemail.com (Postfix) with ESMTP id 13E9540033 for ; Mon, 26 Jun 2023 17:14:44 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=none; spf=pass (imf17.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687799685; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zeQagWoBaXA3rXEOQANZ4Snh6DOQHHPxdX/QvLLvgp4=; b=bOVeSl3zRdRz7bRwdK4fhIkc2O3ImpAUG9iiomPXJqLk29PKASBRaPi441Jsjsn8/DE6IF AaHP+aOqILrakEfXOx1LXil2HuqvkcAG/KkqErbqRtmKeNPMafh9y34D4+14I7MfmpEqp+ UjylVEWQZpPOkTkGbj9U5evncyAS3Uw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687799685; a=rsa-sha256; cv=none; b=O3ES0rUk9/n2MXFoyVg0SUkrwUMHAU0gc4N/ACbD/r5BCVUoavXNauHCZxw7EFoq8BCs2r byECT5jlVgXOYxeysbb5Y6ySao23Wom2g6EV94vE432D/ZMMwrTLklYU8lzZQ823dHZ0F9 /+0rPc9QNPBy6SOGHT4l4/soJm4PE4c= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=none; spf=pass (imf17.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 31D9613D5; Mon, 26 Jun 2023 10:15:28 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 484F93F663; Mon, 26 Jun 2023 10:14:41 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Geert Uytterhoeven , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-alpha@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-s390@vger.kernel.org Subject: [PATCH v1 01/10] mm: Expose clear_huge_page() unconditionally Date: Mon, 26 Jun 2023 18:14:21 +0100 Message-Id: <20230626171430.3167004-2-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230626171430.3167004-1-ryan.roberts@arm.com> References: <20230626171430.3167004-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Stat-Signature: 34b57wjndd6zr4o9grxi14fss1oaeeoi X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 13E9540033 X-Rspam-User: X-HE-Tag: 1687799684-29020 X-HE-Meta: U2FsdGVkX18PNKv7V+75DVg3FwY9CvxnDzhmy6GbsBNRfafUZATgwKnoM+fVsQ6umeevj/FZ+qrPw02tOIfRDdrbUTaMdaSWNBDlgQVZSalBOKYwjW77RVkuoK2r9ipSqdA3UKUJVaYoxxWmeCKpkO9I1XVyB0iaPikxRHb2w5C2evEYnL/z5zJebR6XZWSedZ6aOUrvYhGZUr/XyVvthvZLCPj9X469g8LoP7TrTFQfcsfMObql6FJ3A7gng+xtaIm1Ql9CYFy1z9HEZrNIImtaPeMoJaJzB8vG07LcUqI9iSJ5g/cKMbqL69eO4/Ee06+zwIngfhLrrWKGHMhgn6EiOmSqaqPUopNPmkn7hLzmC4x+cedkRCTLROl2m7lRknPNAYhwf9Iq2n8xxLul3qK07syjHEZbhwV3tl1t8Xw+n86zkHOSCX6gQ9FDKc7FmHnQ+mtrbb/pU3A49CwTZ856ezKiEge5to9G6gkjvZ2fiEUCm5wlDxRKFKfCiRXGE2kC/a5Q92hSDojQR0DjptrSwrnpwFmovylZsjqpJYuf8UAjfltUIGdcMnrJWllFmaS6bwZA6eSLDtdiDN0a3Z6SWccqEFRJ3YD07DyA5yKTU9SvYgNg/9qxM3gPBE+uBqRg+CoT3F63cAJtSvZ4HTLJXcGRxzJFrFsm6hHq0yk7poxeFpSYmW1RhYlrmFuqYVDzF3/a1OkY3tqkhbREGrAZ4j/yT46w+cVMcWnq1Q7Tmz0/h15B8as/rzMZnSEoDCrWZjaWCSUlC+o/1uXaxt1cj7hu+WwMC8xiOo/WnkAof798kgqnNSZ4aMC75vD12QcqVOGoqTMltUhfwawvIGFR8hqr9poJ5fNhV3bgkgfrF49xX4mkMqBu3iURibBkK6sfnb7e7tPQM2ZOF1Gj25H2ulA8fmiXdgHUBVgleDQBoj5mwqKYshCi/9yQtnm01gloOYjr59XedXjK/9T KGxcDUla PnAmQgfb6pAQPQnwuNcnLPLMOGiERaBud3juNjbE+rN1MFl6mBine414rI5eERv5a8GCQ0swUebcmb+IrQAlA9dnfrBGkkycy+FtsadU65DmRHRoZ3hxgYiZDsFRisIz+0w7UXVXsJSW9u0YomIDvsRpfcEZYXyCA6lMccm6VjOocjNKiSJRSKcAbwOTngfnMBiMB29KKDMjTtPI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for extending vma_alloc_zeroed_movable_folio() to allocate a arbitrary order folio, expose clear_huge_page() unconditionally, so that it can be used to zero the allocated folio in the generic implementation of vma_alloc_zeroed_movable_folio(). Signed-off-by: Ryan Roberts --- include/linux/mm.h | 3 ++- mm/memory.c | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7f1741bd870a..7e3bf45e6491 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3684,10 +3684,11 @@ enum mf_action_page_type { */ extern const struct attribute_group memory_failure_attr_group; -#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) extern void clear_huge_page(struct page *page, unsigned long addr_hint, unsigned int pages_per_huge_page); + +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) int copy_user_large_folio(struct folio *dst, struct folio *src, unsigned long addr_hint, struct vm_area_struct *vma); diff --git a/mm/memory.c b/mm/memory.c index fb30f7523550..3d4ea668c4d1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5741,7 +5741,6 @@ void __might_fault(const char *file, int line) EXPORT_SYMBOL(__might_fault); #endif -#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) /* * Process all subpages of the specified huge page with the specified * operation. The target subpage will be processed last to keep its @@ -5839,6 +5838,7 @@ void clear_huge_page(struct page *page, process_huge_page(addr_hint, pages_per_huge_page, clear_subpage, page); } +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) static int copy_user_gigantic_page(struct folio *dst, struct folio *src, unsigned long addr, struct vm_area_struct *vma, From patchwork Mon Jun 26 17:14:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13293231 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4143EEB64DA for ; Mon, 26 Jun 2023 17:14:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D39818D0007; Mon, 26 Jun 2023 13:14:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CE90E8D0001; Mon, 26 Jun 2023 13:14:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD8558D0007; Mon, 26 Jun 2023 13:14:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B03168D0001 for ; Mon, 26 Jun 2023 13:14:50 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 75B7F1604D7 for ; Mon, 26 Jun 2023 17:14:50 +0000 (UTC) X-FDA: 80945548740.04.4C00C0B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf23.hostedemail.com (Postfix) with ESMTP id BC88814001B for ; Mon, 26 Jun 2023 17:14:48 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf23.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687799688; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YKOYDnZuFPt4HShEut0g3IABiQ9Dibq44QzpCDeDSkU=; b=bdYLkc24TOzXCWvHmqNY5N/h6UFFQZ3Pr7HPyKpebkCgd7qaIbmlFJVWKmFABhIpsLq9Yv 4HS0/JhUZ0ojoFDREgivKyi0ijluLxAFjWIePE3bBe4IEuKzgd8N/aNHD5weHROUUZpQ5d bZFJ+B6MijCwQGJZoPoXw4PydcrIiBA= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf23.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687799688; a=rsa-sha256; cv=none; b=msm91tFNkVG62f1aUrnOzNEy7KbXrwpPXxlsDFDbcbxRSd2YinSZxQziYn/LwVIybUAuTa DBoUJydjlPyU3L9hljBvwlbRb1CTuUiKjbWUcU4ixS6tGwK9NIULEJIyCM2aSsJFcBWXp6 NcdU+6N/YV2jZOWUhFTZzAKlb0dWyJA= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5BD8D143D; Mon, 26 Jun 2023 10:15:31 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 577523F663; Mon, 26 Jun 2023 10:14:44 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Geert Uytterhoeven , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-alpha@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-s390@vger.kernel.org Subject: [PATCH v1 02/10] mm: pass gfp flags and order to vma_alloc_zeroed_movable_folio() Date: Mon, 26 Jun 2023 18:14:22 +0100 Message-Id: <20230626171430.3167004-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230626171430.3167004-1-ryan.roberts@arm.com> References: <20230626171430.3167004-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: BC88814001B X-Stat-Signature: eh7h8o41bidt49zisbirtstk9jrsprdq X-Rspam-User: X-HE-Tag: 1687799688-797535 X-HE-Meta: U2FsdGVkX18eUbjlBfrSWtUIVyvvalFSFP76ToD3/2q6duChMvVaWTtAf8zSRwaZKV1T2ziftzhY4lk9JX100UU2jc9BulxpUSLL/lWBTsBHIxWt2h8x2Wt6BpK/F6YAOzijs04qo1jz9TNrBFWRyFgV3XLeeyuMDNfFVADbByZcgPdQuHeHesSlkjvCPb4jVHwLCtlwb+m9JdoozeDP2MYE5NhjWHVlhEEN+NrcT0kUBt8SkxF1QBEua18LN2BdEuViS8pM8H7W3EzaX2KiitP93DaIIVgsnPfNo78FPkkIh8xebW4PhaNHuoLOZqUADSC3QY6O/SlZEsSefKHFGNvauU9gqzN9FA4b8WvsnMfTDK/0OWXV/17kouVulvpx9TJ3gHtkxVbSPxB7LCqd0qU0oF2jKOFOlpxvJMx+Z8XZn4BCvik1KUifwkPq9GluW08L9cpnNTIAH6PajCoqt+qYs9m0jLLBDifVtIWSzcEn9JHIFBb++GH1WpFfEOyhoF+RrDawz0igWL8105SSQIhE+fTOwGkR9FlHUbRJNMu2B5QG8PuAPOu0IaHqK0Ts7mrAz6PCWEE8JuOE1PMpIRBAO9ZiZ2Q7Aep0luKUP4IwcfE0FLunuMNLZb3g7UW8WZ5ZNbY4lyDIoFQG051MygOgkXPAnNP/WL84NdZaLba89g5kKZGtWByI2qYJQdYBAJ3bDmZXWjnqpt3CnOCuWsv9uyaf0R2FYUeS1m4WBghaSjuUa5J/up1i1WXHTBFw+z6+zd578wmkudcqoBcj2yF3M4OEw+7HatXP/x/4UZ04bNdlkhP6KqieGcr/XtOH/5dNSPF1kzSrwFvdAZtjSFHmS6jul70nHvNy4kMEZB11Bi8Z+N92l1vjJDpjy4PfZtTK7ks+8ElmTF1W3jHQswsn4+HwRsks+wYCIQyH2ji9A2VVHS8E5dKjiFFD/TkHdhj1lNX4eTs34m6TOSo YvdccmbS CMxgPzUMvKECIfTkMnjiryHq7n9U170eegzHxGpezg0e6e3/sGoAkm3BNiJH+Uau5H//ZKH92tbhbO29lpZAZnQ3ts2S0ZoBM5DGJOA9a+AZcLLiNqoaNOMQzSWPJUjepTGMcRzjk18FZNF6odRre+klHsTBOlMtbs4Zdc/srIpgjoCNPBZ2WjV6TMv/exJiuUV+0iHczKbfK+Sg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allow allocation of large folios with vma_alloc_zeroed_movable_folio(). This prepares the ground for large anonymous folios. The generic implementation of vma_alloc_zeroed_movable_folio() now uses clear_huge_page() to zero the allocated folio since it may now be a non-0 order. Currently the function is always called with order 0 and no extra gfp flags, so no functional change intended. But a subsequent commit will take advantage of the new parameters to allocate large folios. The extra gfp flags will be used to control the reclaim policy. Signed-off-by: Ryan Roberts --- arch/alpha/include/asm/page.h | 5 +++-- arch/arm64/include/asm/page.h | 3 ++- arch/arm64/mm/fault.c | 7 ++++--- arch/ia64/include/asm/page.h | 5 +++-- arch/m68k/include/asm/page_no.h | 7 ++++--- arch/s390/include/asm/page.h | 5 +++-- arch/x86/include/asm/page.h | 5 +++-- include/linux/highmem.h | 23 +++++++++++++---------- mm/memory.c | 5 +++-- 9 files changed, 38 insertions(+), 27 deletions(-) diff --git a/arch/alpha/include/asm/page.h b/arch/alpha/include/asm/page.h index 4db1ebc0ed99..6fc7fe91b6cb 100644 --- a/arch/alpha/include/asm/page.h +++ b/arch/alpha/include/asm/page.h @@ -17,8 +17,9 @@ extern void clear_page(void *page); #define clear_user_page(page, vaddr, pg) clear_page(page) -#define vma_alloc_zeroed_movable_folio(vma, vaddr) \ - vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false) +#define vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order) \ + vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO | (gfp), \ + order, vma, vaddr, false) extern void copy_page(void * _to, void * _from); #define copy_user_page(to, from, vaddr, pg) copy_page(to, from) diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h index 2312e6ee595f..47710852f872 100644 --- a/arch/arm64/include/asm/page.h +++ b/arch/arm64/include/asm/page.h @@ -30,7 +30,8 @@ void copy_highpage(struct page *to, struct page *from); #define __HAVE_ARCH_COPY_HIGHPAGE struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, - unsigned long vaddr); + unsigned long vaddr, + gfp_t gfp, int order); #define vma_alloc_zeroed_movable_folio vma_alloc_zeroed_movable_folio void tag_clear_highpage(struct page *to); diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 6045a5117ac1..0a43c3b3f190 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -961,9 +961,10 @@ NOKPROBE_SYMBOL(do_debug_exception); * Used during anonymous page fault handling. */ struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, - unsigned long vaddr) + unsigned long vaddr, + gfp_t gfp, int order) { - gfp_t flags = GFP_HIGHUSER_MOVABLE | __GFP_ZERO; + gfp_t flags = GFP_HIGHUSER_MOVABLE | __GFP_ZERO | gfp; /* * If the page is mapped with PROT_MTE, initialise the tags at the @@ -973,7 +974,7 @@ struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, if (vma->vm_flags & VM_MTE) flags |= __GFP_ZEROTAGS; - return vma_alloc_folio(flags, 0, vma, vaddr, false); + return vma_alloc_folio(flags, order, vma, vaddr, false); } void tag_clear_highpage(struct page *page) diff --git a/arch/ia64/include/asm/page.h b/arch/ia64/include/asm/page.h index 310b09c3342d..ebdf04274023 100644 --- a/arch/ia64/include/asm/page.h +++ b/arch/ia64/include/asm/page.h @@ -82,10 +82,11 @@ do { \ } while (0) -#define vma_alloc_zeroed_movable_folio(vma, vaddr) \ +#define vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order) \ ({ \ struct folio *folio = vma_alloc_folio( \ - GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false); \ + GFP_HIGHUSER_MOVABLE | __GFP_ZERO | (gfp), \ + order, vma, vaddr, false); \ if (folio) \ flush_dcache_folio(folio); \ folio; \ diff --git a/arch/m68k/include/asm/page_no.h b/arch/m68k/include/asm/page_no.h index 060e4c0e7605..4a2fe57fef5e 100644 --- a/arch/m68k/include/asm/page_no.h +++ b/arch/m68k/include/asm/page_no.h @@ -3,7 +3,7 @@ #define _M68K_PAGE_NO_H #ifndef __ASSEMBLY__ - + extern unsigned long memory_start; extern unsigned long memory_end; @@ -13,8 +13,9 @@ extern unsigned long memory_end; #define clear_user_page(page, vaddr, pg) clear_page(page) #define copy_user_page(to, from, vaddr, pg) copy_page(to, from) -#define vma_alloc_zeroed_movable_folio(vma, vaddr) \ - vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false) +#define vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order) \ + vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO | (gfp), \ + order, vma, vaddr, false) #define __pa(vaddr) ((unsigned long)(vaddr)) #define __va(paddr) ((void *)((unsigned long)(paddr))) diff --git a/arch/s390/include/asm/page.h b/arch/s390/include/asm/page.h index 8a2a3b5d1e29..b749564140f1 100644 --- a/arch/s390/include/asm/page.h +++ b/arch/s390/include/asm/page.h @@ -73,8 +73,9 @@ static inline void copy_page(void *to, void *from) #define clear_user_page(page, vaddr, pg) clear_page(page) #define copy_user_page(to, from, vaddr, pg) copy_page(to, from) -#define vma_alloc_zeroed_movable_folio(vma, vaddr) \ - vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false) +#define vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order) \ + vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO | (gfp), \ + order, vma, vaddr, false) /* * These are used to make use of C type-checking.. diff --git a/arch/x86/include/asm/page.h b/arch/x86/include/asm/page.h index d18e5c332cb9..34deab1a8dae 100644 --- a/arch/x86/include/asm/page.h +++ b/arch/x86/include/asm/page.h @@ -34,8 +34,9 @@ static inline void copy_user_page(void *to, void *from, unsigned long vaddr, copy_page(to, from); } -#define vma_alloc_zeroed_movable_folio(vma, vaddr) \ - vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false) +#define vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order) \ + vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO | (gfp), \ + order, vma, vaddr, false) #ifndef __pa #define __pa(x) __phys_addr((unsigned long)(x)) diff --git a/include/linux/highmem.h b/include/linux/highmem.h index 4de1dbcd3ef6..b9a9b0340557 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -209,26 +209,29 @@ static inline void clear_user_highpage(struct page *page, unsigned long vaddr) #ifndef vma_alloc_zeroed_movable_folio /** - * vma_alloc_zeroed_movable_folio - Allocate a zeroed page for a VMA. - * @vma: The VMA the page is to be allocated for. - * @vaddr: The virtual address the page will be inserted into. - * - * This function will allocate a page suitable for inserting into this - * VMA at this virtual address. It may be allocated from highmem or + * vma_alloc_zeroed_movable_folio - Allocate a zeroed folio for a VMA. + * @vma: The start VMA the folio is to be allocated for. + * @vaddr: The virtual address the folio will be inserted into. + * @gfp: Additional gfp falgs to mix in or 0. + * @order: The order of the folio (2^order pages). + * + * This function will allocate a folio suitable for inserting into this + * VMA starting at this virtual address. It may be allocated from highmem or * the movable zone. An architecture may provide its own implementation. * - * Return: A folio containing one allocated and zeroed page or NULL if + * Return: A folio containing 2^order allocated and zeroed pages or NULL if * we are out of memory. */ static inline struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, - unsigned long vaddr) + unsigned long vaddr, gfp_t gfp, int order) { struct folio *folio; - folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vaddr, false); + folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE | gfp, + order, vma, vaddr, false); if (folio) - clear_user_highpage(&folio->page, vaddr); + clear_huge_page(&folio->page, vaddr, 1U << order); return folio; } diff --git a/mm/memory.c b/mm/memory.c index 3d4ea668c4d1..367bbbb29d91 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3073,7 +3073,8 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) goto oom; if (is_zero_pfn(pte_pfn(vmf->orig_pte))) { - new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); + new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address, + 0, 0); if (!new_folio) goto oom; } else { @@ -4087,7 +4088,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) /* Allocate our own private page. */ if (unlikely(anon_vma_prepare(vma))) goto oom; - folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); + folio = vma_alloc_zeroed_movable_folio(vma, vmf->address, 0, 0); if (!folio) goto oom; From patchwork Mon Jun 26 17:14:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13293232 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 104EBEB64D7 for ; Mon, 26 Jun 2023 17:14:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9DE028D0003; Mon, 26 Jun 2023 13:14:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 968498D0001; Mon, 26 Jun 2023 13:14:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 793918D0003; Mon, 26 Jun 2023 13:14:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 69E1F8D0001 for ; Mon, 26 Jun 2023 13:14:53 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D116B1407EE for ; Mon, 26 Jun 2023 17:14:52 +0000 (UTC) X-FDA: 80945548824.05.E0DB974 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf23.hostedemail.com (Postfix) with ESMTP id 252E914001F for ; Mon, 26 Jun 2023 17:14:50 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf23.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687799691; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lSO1RBWRS/esyQ/yVe9ST1uKh8G+pWrwI0yEsZeuvk8=; b=hodJ+4H7fTQJ1bglzy3tMK9b7IVxTRAaAZ6qjP9pN0FUK40wCWU8QJFzlgN2qq9xS6Gsx9 bY9o4o9zFTdr6/KnItaXX1BbT3iqOBIJ9umYUuztZSlyT3cj9HIu+Gm0X3+FT5jaAU6uFE 4id12+UmIj1TBOim9DM+3Ofivhj2pXo= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf23.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687799691; a=rsa-sha256; cv=none; b=jroANzCzQwQi8bPe6FEgZa792M34RTyZ9/JjXCUN6Ue1DJDqEYbHWSc7Y8J2Pf7nsR1cYq ja9Ps5tBMpO2le9AZ+SM8eW2wiIw01Cv9MXx+rlHiucxdZwFCl4eSQEQmKakEvM+OEMHfL 055G1jQBTrSwMPPAEIIKO/GMaROBlZ4= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6A1EA1474; Mon, 26 Jun 2023 10:15:34 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 80DFE3F663; Mon, 26 Jun 2023 10:14:47 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Geert Uytterhoeven , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-alpha@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-s390@vger.kernel.org Subject: [PATCH v1 03/10] mm: Introduce try_vma_alloc_movable_folio() Date: Mon, 26 Jun 2023 18:14:23 +0100 Message-Id: <20230626171430.3167004-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230626171430.3167004-1-ryan.roberts@arm.com> References: <20230626171430.3167004-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: wjqi6g9x5aykbmotkuz4wt1g4cotcbzp X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 252E914001F X-HE-Tag: 1687799690-140058 X-HE-Meta: U2FsdGVkX18dQN0F4qTmpaZeuRTWMyR09sZ/5cUGen6rzky2O/ZLpNTF1LBjjLUQ/MrufG+6ZrSd1SAm18isKWF4/ztLhQDHpDSQuaMQvaDNplrKSD/eHRgjfe8GwxGZj8wDXmOTFxQB/3dUx3x0DfXjJMSJXzQGlnSlWDx2fJcGxIBODuJwVU2TQEOymACv9Arx7ezJefhB8FSt9IRV6CKz+OS11Mr7X/Xyh4D1Lw2vRmaLYla0K6wVlq/ZckWnbVySfwfa+YjCvaVJ+tsejM7EqxGFkeIM1ltaXx9ajjh4oDMXxCbkStovcScbqHmMdRlDjLye0IXD5PjGECY2LupqcV7XvkH7VMqUdw1SUiTAHzkBCU4Vq7LNSzDdRrGcbrx5v/Ca32+17/zFvciEiUKDEHmvqY9/yQUoiAFZh3OZ+VN9YBHEcIdqaAYb7ER/V9rglbMGJh++4x2HR4zfL5+ug+K40Eq1nvJn+lHqqb4D/5YWr2Nwvv0S/nzkSUer/vW3HXVeVyEepOrdNjalxMESdNVFf0sjxrvyRp5BPVbYEJCFijOZMEO0M/Xvboa4jjQY7aGHu2eM4hLRvaw/0pzCaVAaBKXhQ8F8oYImX9ff8iQPVg8ioWu9QRpKFAgrWnxxQk0MHH2OGx2/9XpNDzHb48iUb5v0Z3Mhg9dKV3q5DGU1dxWRrQOzpDeFKa0vPRVJI8dKQ/I+A5hmjjT7sn10EQEXxcZDFDgd5rsXnWBmnzRsvCkm61PfQdMBhRM/007kuQbB6HlFn09K/jTIhtgfxUQSVoKoQZZvx0yI+HK3IzlhlEYgxUdnVatCpkapy0x4kyI0LqoV4hTk+s8DetaQ+m0cPGZvEtKGGF0rRu3j9mOMKfF5d7sEyx7wpCn4N3rJpUotda2GZRa5l53RfhNqXBpI0UGJUreIrwo8d0TOTJPNpJcWdmEPBRXdPrUHFSA7lxQTwE0MbpO046s NGsh+R7W iINdX1fJ1hbkzCXWt26auIAswc2byjGCX54hI6k1Pz/m1/DZJcdGdj4FgE6P0+CT8rTihJB8e3RpPJ2W/FXXntvunbBBML7PjzLJHxSjvWPIJolp6Bwjtxw3O+aZs8v83byjEjnZhQv8e6dL6p7gIBPVhfF8d4f36zjwgUTQFhTSgg2osSNktSbTA+VDXkBeJOCcJBlC3/UEhgVw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Opportunistically attempt to allocate high-order folios in highmem, optionally zeroed. Retry with lower orders all the way to order-0, until success. Although, of note, order-1 allocations are skipped since a large folio must be at least order-2 to work with the THP machinery. The user must check what they got with folio_order(). This will be used to oportunistically allocate large folios for anonymous memory with a sensible fallback under memory pressure. For attempts to allocate non-0 orders, we set __GFP_NORETRY to prevent high latency due to reclaim, instead preferring to just try for a lower order. The same approach is used by the readahead code when allocating large folios. Signed-off-by: Ryan Roberts --- mm/memory.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 367bbbb29d91..53896d46e686 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3001,6 +3001,39 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) return 0; } +static inline struct folio *vma_alloc_movable_folio(struct vm_area_struct *vma, + unsigned long vaddr, int order, bool zeroed) +{ + gfp_t gfp = order > 0 ? __GFP_NORETRY | __GFP_NOWARN : 0; + + if (zeroed) + return vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order); + else + return vma_alloc_folio(GFP_HIGHUSER_MOVABLE | gfp, order, vma, + vaddr, false); +} + +/* + * Opportunistically attempt to allocate high-order folios, retrying with lower + * orders all the way to order-0, until success. order-1 allocations are skipped + * since a folio must be at least order-2 to work with the THP machinery. The + * user must check what they got with folio_order(). vaddr can be any virtual + * address that will be mapped by the allocated folio. + */ +static struct folio *try_vma_alloc_movable_folio(struct vm_area_struct *vma, + unsigned long vaddr, int order, bool zeroed) +{ + struct folio *folio; + + for (; order > 1; order--) { + folio = vma_alloc_movable_folio(vma, vaddr, order, zeroed); + if (folio) + return folio; + } + + return vma_alloc_movable_folio(vma, vaddr, 0, zeroed); +} + /* * Handle write page faults for pages that can be reused in the current vma * From patchwork Mon Jun 26 17:14:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13293233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B2F2EB64D7 for ; Mon, 26 Jun 2023 17:14:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DFE668D0008; Mon, 26 Jun 2023 13:14:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DAC8C8D0001; Mon, 26 Jun 2023 13:14:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C74E38D0008; Mon, 26 Jun 2023 13:14:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B4A708D0001 for ; Mon, 26 Jun 2023 13:14:56 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 3C6781C7ECB for ; Mon, 26 Jun 2023 17:14:56 +0000 (UTC) X-FDA: 80945548992.24.D287448 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf13.hostedemail.com (Postfix) with ESMTP id 59D652002F for ; Mon, 26 Jun 2023 17:14:54 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf13.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687799694; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JK++ZwPDFjFzL1t38tJLvARD4YCY0yS3OUbZtrWhuTw=; b=3wriQVhHOkrpGHULt9bSGmUKEj9BoHgZW5OPd9OrXoS5suxFFXzujwtJ8iJjkZOK0kllTB DZTBOGyBS+VZ3J4PWsCvF1s+eyYBmFmDng1p8ie7VF0zwDrruAyo3lvlLS/GzxL7ni91Dc Xh8PFY5cmvEYCUKPQjMsOuRgrsfzHRc= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf13.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687799694; a=rsa-sha256; cv=none; b=Y/X7CaS3gRoTW4XYFXg+RnyiVmtdvTWrnfp3Q9wggYZO8XEa2ek/7Awb1LV3CNEpJ7ErFk bZOGdHHzs8UK3TCdKVK5H5RVkv5GTBtx8eRq1CFABPS+/nlrwa8hnKwY4o27BWpr9qIJ06 XDkKrZ8PxceLL5dHkjYDw70VVhlaFJM= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7835F1480; Mon, 26 Jun 2023 10:15:37 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8F06B3F663; Mon, 26 Jun 2023 10:14:50 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Geert Uytterhoeven , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-alpha@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-s390@vger.kernel.org Subject: [PATCH v1 04/10] mm: Implement folio_add_new_anon_rmap_range() Date: Mon, 26 Jun 2023 18:14:24 +0100 Message-Id: <20230626171430.3167004-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230626171430.3167004-1-ryan.roberts@arm.com> References: <20230626171430.3167004-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 59w1w7uxdh1c1e4mtpuuahzwkni6z7fw X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 59D652002F X-HE-Tag: 1687799694-300580 X-HE-Meta: U2FsdGVkX1/AwofiCKFS3/vVWtYGvbYfrorEzums0K+tiH4BJecLl61k7u8hfkXPSd6SB3xFRox9tBIJ/8wsYi+PhV52Y2LCzLVs0g8hZvfpuyrJeLRv5fo9riJlYIXpLYvts8N2STHhGvu1YOybYEDb8Oz5sb07Xt54302wAyvIxbNgc+I2jVAhajmJO4DroEgowueoXqCVWw9+LJdEuii0Y35RGX+KAEn78Rmi7qEwEIJs8Ys1J7+vKv3jagqBXq7U172hlUwgRsarlU3D0G/AkfTy5rzQzRHb+5EWSZv/ARmzped7PE65157jvt90zKhR52I9kDtcXHgNij4U5k53nv8gtO5nqjPZbqA3q4+grbEivHFrpMsToieL6qVf5XW+SG+buy5oKD7EO9r18v1VqYN2z8EVonWl7UM7bsWWdI+dwv6HVk7juTNsnrVT69pWGUFyWVJddA72Pn9k+ZYJyOhZU8ck2wk/lIz1VQRWZi34NJEbLIg95yAR5wycSX+aLCsKDdYZXQ5cj8sGPufmAkazZlok2pUKZRbXiJTmW6ke1lXXgUPvSW98o84pLqqy+XdnPOyb8pTHRTQbTlBUmwegP8QhDjba992NQdYlf293Or6ih1Go8kxlat54+H9v8Rwmlky6olSd8X7ojODaTTxB2J+c5+ZMXp0gQaGhEN+9bHId1bCTknew1AVqHPiDwCSv5j6YEB6HXgaweSuD+rYvUkizjn5sDBV4W8BRd5oXIbnevGCjAgTq7WqeoDcV0aDz0w+AjFG+7sEoFfUdpVOnNSn4vVp++9GvPJ3p2KPMwANA7ZtOF91DKFxD7brlNQLhM3mWdXkEh5mSgwO82LeM7CyEWQRaaEMd462d7kEBtA9WAygXtF5x5urosrLIEacYdXq1xh9H3AHn5Y71Hkx7O6a+mzc5LMhXgwXfHJddK+aoNTsfffgEizxVLNPDfiAARz8QrQySBjw WD42h/8+ rWRBDzZdkeS4RMpYq9fE2153/FLTgwOfObAzQvmARj0epErgNRRVAHwxv51eo3bi2DRJO4KPaYk+50QvjU/Ynz32vNrBV7FV52wg4eUBcmElPSXhpcolZ1Ck2XE0ElvPkU0RPNhK4mttpVVYPDz3keidF3oJMiwmNuRFKQ39jfakU0kIbHKzA8t9I8eBmO42sRpS2qdFnBboN2iw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like folio_add_new_anon_rmap() but batch-rmaps a range of pages belonging to a folio, for effciency savings. All pages are accounted as small pages. Signed-off-by: Ryan Roberts --- include/linux/rmap.h | 2 ++ mm/rmap.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index a3825ce81102..15433a3d0cbf 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -196,6 +196,8 @@ void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address); void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +void folio_add_new_anon_rmap_range(struct folio *folio, struct page *page, + int nr, struct vm_area_struct *vma, unsigned long address); void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr, diff --git a/mm/rmap.c b/mm/rmap.c index 1d8369549424..4050bcea7ae7 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1305,6 +1305,49 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, __page_set_anon_rmap(folio, &folio->page, vma, address, 1); } +/** + * folio_add_new_anon_rmap_range - Add mapping to a set of pages within a new + * anonymous potentially large folio. + * @folio: The folio containing the pages to be mapped + * @page: First page in the folio to be mapped + * @nr: Number of pages to be mapped + * @vma: the vm area in which the mapping is added + * @address: the user virtual address of the first page to be mapped + * + * Like folio_add_new_anon_rmap() but batch-maps a range of pages within a folio + * using non-THP accounting. Like folio_add_new_anon_rmap(), the inc-and-test is + * bypassed and the folio does not have to be locked. All pages in the folio are + * individually accounted. + * + * As the folio is new, it's assumed to be mapped exclusively by a single + * process. + */ +void folio_add_new_anon_rmap_range(struct folio *folio, struct page *page, + int nr, struct vm_area_struct *vma, unsigned long address) +{ + int i; + + VM_BUG_ON_VMA(address < vma->vm_start || + address + (nr << PAGE_SHIFT) > vma->vm_end, vma); + __folio_set_swapbacked(folio); + + if (folio_test_large(folio)) { + /* increment count (starts at 0) */ + atomic_set(&folio->_nr_pages_mapped, nr); + } + + for (i = 0; i < nr; i++) { + /* increment count (starts at -1) */ + atomic_set(&page->_mapcount, 0); + __page_set_anon_rmap(folio, page, vma, address, 1); + page++; + address += PAGE_SIZE; + } + + __lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr); + +} + /** * folio_add_file_rmap_range - add pte mapping to page range of a folio * @folio: The folio to add the mapping to From patchwork Mon Jun 26 17:14:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13293234 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFAD3EB64D7 for ; Mon, 26 Jun 2023 17:15:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 60D508E0001; Mon, 26 Jun 2023 13:15:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5BD558D0001; Mon, 26 Jun 2023 13:15:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4856D8E0001; Mon, 26 Jun 2023 13:15:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 39DE98D0001 for ; Mon, 26 Jun 2023 13:15:00 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9CB428080E for ; Mon, 26 Jun 2023 17:14:59 +0000 (UTC) X-FDA: 80945549118.09.C2B3625 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf21.hostedemail.com (Postfix) with ESMTP id 9765C1C001F for ; Mon, 26 Jun 2023 17:14:57 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687799697; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MCPPpPFh5Z8cSR0+xeGHeJRTVorBAm/+P3hCtLuZNVU=; b=Od8zKHRS3aBjjZM4v0tCmy27LsCYmuzG5iVoeWpkTubVZ/JGMBOa1/O5OsXKQdIqT1N29i 5kuTCnHFBQa5yZ8Su5ymENS3SVS4InLfBEhDcghTDAmlcAgm1D+5M6lPyMbwhFfIge84JQ MQHmsKzyoIN79ZtUGKM8eaFdB4Brj7w= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687799697; a=rsa-sha256; cv=none; b=RsmWKCId6ylGvPSxrRodOJUIbZrj1WgwuWLAqVmY4tNetZY5JMetJLxL7hKu4IY4CSGk7r hfXdNNdTTFQIOBSGZc9sn68X757aeHRtWE/le2ZyzwFiT9jmRXR16JGKHFf/xp/yWFMX/+ bekZCril0PNGh8GD1fMnG60hrbMdFjE= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 864C1150C; Mon, 26 Jun 2023 10:15:40 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9D3913F663; Mon, 26 Jun 2023 10:14:53 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Geert Uytterhoeven , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-alpha@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-s390@vger.kernel.org Subject: [PATCH v1 05/10] mm: Implement folio_remove_rmap_range() Date: Mon, 26 Jun 2023 18:14:25 +0100 Message-Id: <20230626171430.3167004-6-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230626171430.3167004-1-ryan.roberts@arm.com> References: <20230626171430.3167004-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Stat-Signature: 878hbi6whwjkw3b7bcygp5azous5tqkt X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 9765C1C001F X-Rspam-User: X-HE-Tag: 1687799697-656476 X-HE-Meta: U2FsdGVkX1/P1684RUXwacHcP2J81U/uZJdoDnZd6FEFq2GdKiiE2cL7dMSRRLISH7QLjimxZBFDJAFriutrmG7DefX83ifNW4Vp0W0nsOaBtJ7qo559+f33h765fNSddAyU1ZJfMMV9vOPUfh+W+Dre6jNG15yderqd7Rl65fHaf+u0Y2Gm0qzwu/y4xG63JZcS3DR73RncRxv2nCPNww5wfhdDWOzK8RgbfqSe4s2zBLeB4s5/qOmMsOmI6m5m/bVIoMjsqzTn5cgQRbXZLpEB2cqtnpw/5gLVvfMb0mCu0DBUJGMvnYKn+Rxe7skIl6bm5axOMK47gXByk97c8L6D9LfpdPWvxqQlZhYfnVG/TVLteOkvJQ2nyyOhfiZB6q24fJnNAS3+Eho4t8ufoM6zINTfzX3Gyze8owg7U25iueAxzuPxxXPm2hapqmMYMeII5AI1xulOnv02Q0E3vQU/VjMk2dZGzWEdTJ6xCCGJNlWZuY2g+w4ATeKeNTZdW82uF896UXAVZf/cpzWWLt8H4SCFwtaAs/nBhx1mhCh86MUKnoZvCTSgBAUg8YqdUh8WvpupATNqyKJuqKlWuwzTSVkOB6/EJ/rG3ukbAvxRzeDF4JAdiRq6T/G7WMi4L0+5a7pbtLKOBqgk25ynQfiFI9jXlHnwePHiFg0XJF4RCDzfsicpnADEKs2NdgRSYH6jE8bKWG7yIYgpMQ+SLWuAjN29k+acs/WPlKqqChMuZtfNxG7xeJJfPLECYf9Sh8R8n+PUDNxgOD3Gha9OXtXq2+/aR7s0fA+WZy20japNBWqxkVw9+cUjChypt4+ADXkNTHkbgKU7PcT1Mqm5KnPvzAbxnGKDV4YTrAnfUmHH2k0EOKZsNrvtY/pUOk+4q/TIjZP5HOSrrZD1kqNnEkoXaiA6eI6Te6VlF1ZL7XH07nCb5VuatXg2SQSPrACE9ZWS1AhgVP522IbQJ++ YqI965iy 5uxeTCT5XjEKG/l50e5mTKn93BCJhNjMhYXH4w20Cku0rgeXGHlt3++CG4uu5mvCqSHamPatJmVHksMgG0oo14CVMXFl+GKLq44HcmZ4ZzoHPOXSwzzYmr0RiUyICyuildiQSLq620K91I5va1vji6cWrEX37NKYMfRCyOUdPQPM9DkOwY2TscQaNcReVCCn7N0K4q1rRvJAfXDE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like page_remove_rmap() but batch-removes the rmap for a range of pages belonging to a folio, for effciency savings. All pages are accounted as small pages. Signed-off-by: Ryan Roberts --- include/linux/rmap.h | 2 ++ mm/rmap.c | 62 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 15433a3d0cbf..50f50e4cb0f8 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -204,6 +204,8 @@ void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr, struct vm_area_struct *, bool compound); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); +void folio_remove_rmap_range(struct folio *folio, struct page *page, + int nr, struct vm_area_struct *vma); void hugepage_add_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); diff --git a/mm/rmap.c b/mm/rmap.c index 4050bcea7ae7..ac1d93d43f2b 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1434,6 +1434,68 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, folio_add_file_rmap_range(folio, page, nr_pages, vma, compound); } +/* + * folio_remove_rmap_range - take down pte mappings from a range of pages + * belonging to a folio. All pages are accounted as small pages. + * @folio: folio that all pages belong to + * @page: first page in range to remove mapping from + * @nr: number of pages in range to remove mapping from + * @vma: the vm area from which the mapping is removed + * + * The caller needs to hold the pte lock. + */ +void folio_remove_rmap_range(struct folio *folio, struct page *page, + int nr, struct vm_area_struct *vma) +{ + atomic_t *mapped = &folio->_nr_pages_mapped; + int nr_unmapped = 0; + int nr_mapped; + bool last; + enum node_stat_item idx; + + VM_BUG_ON_FOLIO(folio_test_hugetlb(folio), folio); + + if (!folio_test_large(folio)) { + /* Is this the page's last map to be removed? */ + last = atomic_add_negative(-1, &page->_mapcount); + nr_unmapped = last; + } else { + for (; nr != 0; nr--, page++) { + /* Is this the page's last map to be removed? */ + last = atomic_add_negative(-1, &page->_mapcount); + if (last) { + /* Page still mapped if folio mapped entirely */ + nr_mapped = atomic_dec_return_relaxed(mapped); + if (nr_mapped < COMPOUND_MAPPED) + nr_unmapped++; + } + } + } + + if (nr_unmapped) { + idx = folio_test_anon(folio) ? NR_ANON_MAPPED : NR_FILE_MAPPED; + __lruvec_stat_mod_folio(folio, idx, -nr_unmapped); + + /* + * Queue anon THP for deferred split if we have just unmapped at + * least 1 page, while at least 1 page remains mapped. + */ + if (folio_test_large(folio) && folio_test_anon(folio)) + if (nr_mapped) + deferred_split_folio(folio); + } + + /* + * It would be tidy to reset folio_test_anon mapping when fully + * unmapped, but that might overwrite a racing page_add_anon_rmap + * which increments mapcount after us but sets mapping before us: + * so leave the reset to free_pages_prepare, and remember that + * it's only reliable while mapped. + */ + + munlock_vma_folio(folio, vma, false); +} + /** * page_remove_rmap - take down pte mapping from a page * @page: page to remove mapping from From patchwork Mon Jun 26 17:14:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13293235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29EBDEB64DA for ; Mon, 26 Jun 2023 17:15:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B64AC8D0009; Mon, 26 Jun 2023 13:15:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AEE698D0001; Mon, 26 Jun 2023 13:15:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9DC838D0009; Mon, 26 Jun 2023 13:15:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 921468D0001 for ; Mon, 26 Jun 2023 13:15:02 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 532A01407F7 for ; Mon, 26 Jun 2023 17:15:02 +0000 (UTC) X-FDA: 80945549244.11.8D0B734 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf11.hostedemail.com (Postfix) with ESMTP id 8E32C4000A for ; Mon, 26 Jun 2023 17:15:00 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf11.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687799700; a=rsa-sha256; cv=none; b=35hSiAVoD9YfH5OhFyaKG1eFZnRGYU5Di6lyiCMpaMxYhNZc/ScVHHPoWZt1adpTI+LPnG FFaFNh2l6qmV2n34vwm2v78uTzGYAIQCSnAW4nmwHrgVFM2dWAExZU09RAed1Hk59tPvTE /ALKX8kh5T2DnFX1rTdv9AlXS8y/vV0= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf11.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687799700; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9LDnxtcMGKzMwMeO2DGDBjGbhx12txEm8328niFWh5M=; b=aN7LuhG5A3Gh4QFrX/Bvjbd74hhqTqShLDBeDrguVzwAZtGXo6X2zEurqbesego9JOWJvg HfMGuImjJyKpfM9lZQxIkkOVQVKrmrz/3aXwkXpMQUKq4hKPP4QaUqClWZp6dpZ1Xfl8IE 6P8Zps2XWTKnhwp0wekKSijNe3amQWU= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 94AD9152B; Mon, 26 Jun 2023 10:15:43 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id AB7DB3F663; Mon, 26 Jun 2023 10:14:56 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Geert Uytterhoeven , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-alpha@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-s390@vger.kernel.org Subject: [PATCH v1 06/10] mm: Allow deferred splitting of arbitrary large anon folios Date: Mon, 26 Jun 2023 18:14:26 +0100 Message-Id: <20230626171430.3167004-7-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230626171430.3167004-1-ryan.roberts@arm.com> References: <20230626171430.3167004-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 8E32C4000A X-Stat-Signature: 6sfpnth7dzgnjmixfot589ajgtiuercn X-Rspam-User: X-HE-Tag: 1687799700-537119 X-HE-Meta: U2FsdGVkX1/UCvXlJnFWQza3NP+WmTWce/2deM3afEgU3XoP0mmvonzjnJeNFXQgKjp43q29JMBtiFR+3bjTsUYLHhWsrKkgV1rpDYPprpDJkSd8J/ZbQE2nlr3McXvzATTbxxiDO+ErWnNJfe7fXrK4RhpCRME2Zi57a2PRU6qxlcIEJhDtwDYg5Cc3OpOKW3Hl7+jVQVUbE3jBOXk8x7GhTt9RY2ZFeJ1Rjim0ZLPB5FGlZl2ZU3NNIU71UJ0wh0YdbNQHTxlMaAwoahMZEt7eVGAif66rbIUkgtcAhcHcN4mOa30Aj/X2HSFf0+77+KJGXlkeTL2Zi5pC4Z6vV9e3FBAbQFd6I67yb1as/jRS4L0bz3qtxrpI8XYaQ6dUTAQgsmzhGv5FmW1M2tP4L5iVzDe8ggABzlFi1J/9ZAPynFfVZqeej9ftGxEWKJyfBuRi6MlXSoNPNjAr/p1+xPOS2Ka9u8cGCkBJ63PBLSTCHBA4O5VI/0bkt47UEoqU7OMOOGy3oF1JgSyDifriiPXBfq45gjZHq1QGSSeRxCf9/+ZnzcqTUSR3DNiJrhr+R2bdbyWHp6B9Q9aN5vHnmQfKqFSPfNCzpIk0Ca7Rtjfzqd70F7XpVh1fAb/6ihdziM4pwcozPRB7PVT4DAvyJDeoLzjKMI7+G22DmrVWLCwuWalUOijeIyqQFX/Mue/oLKBhzv437UPiZ3ugV3MXOXL5xOXSHRPf8SZnOlkgQVyNxrJPlZPO2fqzoujMf4YjD7kLAthqlZVGF5sYqVIeqB+4UCUumMQh13UA7ASOdzYjpMc2/NpDmH4zJ5HmV2mNB7creVTaGwesR/8RRFAmcJwj/uUT7qjVzZWC8f92LAJsIHvJjRFjqCJGDySwf/fc99XlYEwsQQProqA5yoLf42AYLL2qceaPjxdU+5WKmZIq5fBzo9X5nyqhGJiR7fwjCvXSiQojnHnt/3cmK+t z+16zNYo vtA2M5fxkOBpchzRbfXYaPH+xK3ekxghDo634c2T8xNaG1LQRiL7eABtDcFyvoDB0rTZpE3vQScz0DQlvokuUrWkrXIoZ/ZDvhWjpM1su8zfRCPd/H6JfsZ0t3vDtN7Z/jS7ghr/RBnIqGITXW08YoCCNQXbkhVe77Cra X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With the introduction of large folios for anonymous memory, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized. Signed-off-by: Ryan Roberts Reviewed-by: Yu Zhao Reviewed-by: Yin Fengwei --- mm/rmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/rmap.c b/mm/rmap.c index ac1d93d43f2b..3d11c5fb6090 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1567,7 +1567,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, * page of the folio is unmapped and at least one page * is still mapped. */ - if (folio_test_pmd_mappable(folio) && folio_test_anon(folio)) + if (folio_test_large(folio) && folio_test_anon(folio)) if (!compound || nr < nr_pmdmapped) deferred_split_folio(folio); } From patchwork Mon Jun 26 17:14:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13293236 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50F77EB64DC for ; Mon, 26 Jun 2023 17:15:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DFCB98D0001; Mon, 26 Jun 2023 13:15:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D5E3C8E0002; Mon, 26 Jun 2023 13:15:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C27A58D000A; Mon, 26 Jun 2023 13:15:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B5DB58D0001 for ; Mon, 26 Jun 2023 13:15:05 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6E0D416068B for ; Mon, 26 Jun 2023 17:15:05 +0000 (UTC) X-FDA: 80945549370.10.8626114 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf15.hostedemail.com (Postfix) with ESMTP id 7B340A000C for ; Mon, 26 Jun 2023 17:15:03 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; spf=pass (imf15.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687799703; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=igZkz1a6pk30y7Cb39JwkU8z+NvNmcA3pvWODxUTXD0=; b=rdAl/Ju7RMQRCRmW9xnjusYVgkm5WNruEseGkUi2xKDFi0dlM54b3kKpNBdX32OSesdUmH Igaa8KN5q6Td0rms/qpcsjcuYB7PEfVQxka9oxh5N9NzyprD1EtroJlk+Db2ZUwh+yFUje eZJXg9c5aRrgyTFhsLf9wu03T2qwg8M= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687799703; a=rsa-sha256; cv=none; b=pofwEk0lmpMzdvX46qJYcZdLvB+qd1rbf2FHtwTYP0hWG8VVNgRi4FOubVEQ51fmKBrKgB uhEoJivQ9TWp0T0NicXpQXITDTA6UT6Tt6dLGf73BoxKHV6aAf0xL6rxI2lp7c4k2rm3ZA Wn20xTtUsmMRC9IulBXZrRRNYSdcKhA= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; spf=pass (imf15.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A3C721595; Mon, 26 Jun 2023 10:15:46 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id BA1553F663; Mon, 26 Jun 2023 10:14:59 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Geert Uytterhoeven , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-alpha@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-s390@vger.kernel.org Subject: [PATCH v1 07/10] mm: Batch-zap large anonymous folio PTE mappings Date: Mon, 26 Jun 2023 18:14:27 +0100 Message-Id: <20230626171430.3167004-8-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230626171430.3167004-1-ryan.roberts@arm.com> References: <20230626171430.3167004-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Stat-Signature: kpftabh1jdk85hqom4kx1u5kph7xjbow X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 7B340A000C X-Rspam-User: X-HE-Tag: 1687799703-27496 X-HE-Meta: U2FsdGVkX1/m7pippmDam8Ex7b7VfWzgJ4f7v9A2nXwspfCE/q7hCQNaaziXs3VKri1pDlfo8y4rilcMObpwtLqiHMSZC2RhIV35/Qvy58gnkeXWdqQcqcPrmHkCoLUxDC7wYtsPgNdCcoVuvCEKpKRzSEBVr4OqZzbUqCQSvHoxXukEvL38Qb+t/ZYJavhwyRKtcWQLdhZpL/9C3GUPov5PXcyPcLqY6yHH9zgux9LnDE93G4w7IJZqtkWa2zsuGL/i/tOYW0ntxpdcwt+wJLBeqEd/vc3LltdNbZklC4BR1wvn+DBjOHpDmIiMYc9Ors/p2e1KVUIr5LuluSD+/uvX4O5VJWB+NNY6IZDphVygmqlSXO+WL1rJIrUI/USGZa4rVWCm8FAsDJQk1Itug8PCVekfdhb3HT8eE6NxgegpMPXjBrdioRc9TSDCNoHiEymI7JqUttUV29OatUJ3/TOWrFVAAEB39qY1S+UfJDWo65B/c8fMc6O56ysAYeqYj4YC0dM0ORLYCpvMEwR3CE3fohu+yE7SmIgY2+eeqeSRN3361pLdoH/3NLCaUWHkbNAjTEhzX8r7kLxcy4MoujXTXGuCQMMj/Ui+rjO/H7s1ywKNsGNipMUhADZaBMoFkjpLfuJQPtyYRr7nXPklWRP6miYJXHeO8cQToxoFhIcmi0sAOjiJOuIijqOPO7AFn54fv9yXLVvvrueFQbx6naP7iusGfcDRgL4zJpJ0DCqUrF08L2n1xCxLfBTl3p/zNu2sY4JyfZJ/2pBAqkwRnKRi3pMuLUXciApkLl/r79qbTTRCGAJPdrjdZTKfNJ79wTFfaysjC3A1aqCIM+yZRwlkGRZT/jnPbFqgdEcMssuXciVyfeGjIRioQ48O0yMgXBaBjXAIccq5abMcc5PYfsDaFfbatWX23eZeOCHGS1Jh6DPGTHIiWTETbEEDl4HGrBjKudcJ1d0L/vEFaoJ kRYxBgeM yaFZTrb9+qfnS1CQA/hmqEpRIvrYhu9cPlFwRiaaClV1er7KTusTb0aSYB+I4qK+eCiYdxwN5gX0EhDMlu4AgKy2LCZRgQ9vQPRNZsE3cjCPMxGOfSO24oyyV+zqupjzJ1N9rcpP3A2R64HKmpEduA6OjVA4qX3irLMUG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows batching the rmap removal with folio_remove_rmap_range(), which means we avoid spuriously adding a partially unmapped folio to the deferrred split queue in the common case, which reduces split queue lock contention. Previously each page was removed from the rmap individually with page_remove_rmap(). If the first page belonged to a large folio, this would cause page_remove_rmap() to conclude that the folio was now partially mapped and add the folio to the deferred split queue. But subsequent calls would cause the folio to become fully unmapped, meaning there is no value to adding it to the split queue. Signed-off-by: Ryan Roberts --- mm/memory.c | 119 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 53896d46e686..9165ed1b9fc2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -914,6 +914,57 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma return 0; } +static inline unsigned long page_addr(struct page *page, + struct page *anchor, unsigned long anchor_addr) +{ + unsigned long offset; + unsigned long addr; + + offset = (page_to_pfn(page) - page_to_pfn(anchor)) << PAGE_SHIFT; + addr = anchor_addr + offset; + + if (anchor > page) { + if (addr > anchor_addr) + return 0; + } else { + if (addr < anchor_addr) + return ULONG_MAX; + } + + return addr; +} + +static int calc_anon_folio_map_pgcount(struct folio *folio, + struct page *page, pte_t *pte, + unsigned long addr, unsigned long end) +{ + pte_t ptent; + int floops; + int i; + unsigned long pfn; + + end = min(page_addr(&folio->page + folio_nr_pages(folio), page, addr), + end); + floops = (end - addr) >> PAGE_SHIFT; + pfn = page_to_pfn(page); + pfn++; + pte++; + + for (i = 1; i < floops; i++) { + ptent = ptep_get(pte); + + if (!pte_present(ptent) || + pte_pfn(ptent) != pfn) { + return i; + } + + pfn++; + pte++; + } + + return floops; +} + /* * Copy one pte. Returns 0 if succeeded, or -EAGAIN if one preallocated page * is required to copy this pte. @@ -1379,6 +1430,44 @@ zap_install_uffd_wp_if_needed(struct vm_area_struct *vma, pte_install_uffd_wp_if_needed(vma, addr, pte, pteval); } +static unsigned long zap_anon_pte_range(struct mmu_gather *tlb, + struct vm_area_struct *vma, + struct page *page, pte_t *pte, + unsigned long addr, unsigned long end, + bool *full_out) +{ + struct folio *folio = page_folio(page); + struct mm_struct *mm = tlb->mm; + pte_t ptent; + int pgcount; + int i; + bool full; + + pgcount = calc_anon_folio_map_pgcount(folio, page, pte, addr, end); + + for (i = 0; i < pgcount;) { + ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + tlb_remove_tlb_entry(tlb, pte, addr); + full = __tlb_remove_page(tlb, page, 0); + + if (unlikely(page_mapcount(page) < 1)) + print_bad_pte(vma, addr, ptent, page); + + i++; + page++; + pte++; + addr += PAGE_SIZE; + + if (unlikely(full)) + break; + } + + folio_remove_rmap_range(folio, page - i, i, vma); + + *full_out = full; + return i; +} + static unsigned long zap_pte_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, @@ -1415,6 +1504,36 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, page = vm_normal_page(vma, addr, ptent); if (unlikely(!should_zap_page(details, page))) continue; + + /* + * Batch zap large anonymous folio mappings. This allows + * batching the rmap removal, which means we avoid + * spuriously adding a partially unmapped folio to the + * deferrred split queue in the common case, which + * reduces split queue lock contention. Require the VMA + * to be anonymous to ensure that none of the PTEs in + * the range require zap_install_uffd_wp_if_needed(). + */ + if (page && PageAnon(page) && vma_is_anonymous(vma)) { + bool full; + int pgcount; + + pgcount = zap_anon_pte_range(tlb, vma, + page, pte, addr, end, &full); + + rss[mm_counter(page)] -= pgcount; + pgcount--; + pte += pgcount; + addr += pgcount << PAGE_SHIFT; + + if (unlikely(full)) { + force_flush = 1; + addr += PAGE_SIZE; + break; + } + continue; + } + ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); tlb_remove_tlb_entry(tlb, pte, addr); From patchwork Mon Jun 26 17:14:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13293237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D79CEB64D9 for ; Mon, 26 Jun 2023 17:15:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A7B568E0003; Mon, 26 Jun 2023 13:15:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A031D8E0002; Mon, 26 Jun 2023 13:15:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8CAF18E0003; Mon, 26 Jun 2023 13:15:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7AEE88E0002 for ; Mon, 26 Jun 2023 13:15:08 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4F8191404C1 for ; Mon, 26 Jun 2023 17:15:08 +0000 (UTC) X-FDA: 80945549496.07.7054426 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf21.hostedemail.com (Postfix) with ESMTP id 79FF71C0011 for ; Mon, 26 Jun 2023 17:15:06 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687799706; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AKvNVAfIA7FhYjb4NI628m6rn2oSwSbs9tNtgVulh4k=; b=I+qrHDmmKbvvAaB35Dcxg8pN7N/KTyhSCca/PNE/ezhRMICZuXx6LVdY+NU2Xcn9FSTt2Y VFHx30oB2Bk/2xwO3HkNkDRN/+OyOUO/Un8aMZeSouwKTH7KvAYPqDMFI/BuuNl5ALbGI5 yZmhUG1K6kzto/y13b3GEXHnSnx5Y3g= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687799706; a=rsa-sha256; cv=none; b=6mCulrwVTsDSOfLwAGn855i7nXKuyGe1thyrXmlgIlvRs+6BwVShaVbRzEyZ0q9nw7HROy NSoXwL6jWCNou8viKH97Omr/RD8M1ISFXsidv2ySrUFDrr2+/P9j1zgqeOaOb5sM1P976L H0Gza1GHTnX8OSkt0YPX3/2btNFfA+s= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B2AB21596; Mon, 26 Jun 2023 10:15:49 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C8DE63F663; Mon, 26 Jun 2023 10:15:02 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Geert Uytterhoeven , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-alpha@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-s390@vger.kernel.org Subject: [PATCH v1 08/10] mm: Kconfig hooks to determine max anon folio allocation order Date: Mon, 26 Jun 2023 18:14:28 +0100 Message-Id: <20230626171430.3167004-9-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230626171430.3167004-1-ryan.roberts@arm.com> References: <20230626171430.3167004-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 79FF71C0011 X-Rspam-User: X-Stat-Signature: i3qut5i8xems3e3f8xqquju8een5c3rb X-Rspamd-Server: rspam01 X-HE-Tag: 1687799706-524166 X-HE-Meta: U2FsdGVkX1/CNna9oM5bMYgPf2x/dKeW8+hYUrdYLZ6zHK2ftsMicwMfKwD2KJLnZKebTL74LVOUAGxPbkvEosUoMihPKxgr8LHWajeHuvytbhIYuMbXK+DH1XhSRUhhBu0YomGtVlLkHL7pnqPAqrsa1prZcb9DKPvPzRD7FwUE2FhJ0zUiYeTaoq+ZFRDMh55lR/WxkDdso2Zi1lwN1PyY3pwXNq3i5F/laP2ikxfDGOxQIEzQixF7xstLtVTZi7cAxR86A0jbOm3lgsXNe59vCcHBe2Na/CoMbh2N7PPLPty8oTrwMxjLPoAmUgO/HXnEBjgQ3xBwyb3ka+IeSYieIaR0ltw2V/N5HKnvmmtJDkSNov2cWsZ7te4+cEfd0shMjXYU+tk8TO2LJYjx/5pink+4zqyL1pnFaOa3/1E7Pfkt01wDDG+SAxLlvR1zgOix01s1jc6gXSITUbEvjoi/d0ajRd0FUMBYJtfzNwtpC6VHSZAcV79U1lMfdU1UDI/bE3LKDCcD2Q4oaXv9QTCI4O/rs90V7T+jZxDb0gTXjYQaRI1x+8Olvz7qDLnzxjmKQzpfn8J5lPJWW4My+S8rn3KJRPgnj0JRvXOAx73bhDtbWWWfzaYEuID9cZQGiBsnksqGvRoTuZqpYdqKWNfDDXKUWF9gY1GuDPGr5tucxcd8Gp71nlziQCjW1MPdFkAg9DpOrH48STDZtCaXbGQjONyh8AP3y90OEU6kSDV7wHkSt6o7TvthhX31ZUpKxhJFBQFFnBZIEOsKxXFC0DiPutQYtT56L2S5E+ahFFDVI/5N8V+BdqgIV7ccVWyQ6/dOzTE+U9ZrRpecPLGihetK8OGfj8/WOdxcQYvlF+Hq0HW3yNPge7nH5TUQQQF9zUC+2B5Ue+Zxu4YE0arUmpMks8+oSGgM7cC9ViOUvIwL1xCIjfpm5+hLt2C/usbMfmo3ug2zjopIlZcXJvh pASjcOJh vmpGMgok5Ff6g+ZZzSpvYcR7lW/pqYZBeeTgm8/VetHsKpOhYgODXftTDuQJWBH5lMmMoGUYrQadH7WP7Rjq1vJt72yWU6P09beHWbwINCuyM3LNvCMQp982YJUY8Km4BZTAry7GxbzMLoqU+0nYqo7h4JQn3dq9Lw5+M X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For variable-order anonymous folios, we need to determine the order that we will allocate. From a SW perspective, the higher the order we allocate, the less overhead we will have; fewer faults, fewer folios in lists, etc. But of course there will also be more memory wastage as the order increases. From a HW perspective, there are memory block sizes that can be beneficial to reducing TLB pressure. arm64, for example, has the ability to map "contpte" sized chunks (64K for a 4K base page, 2M for 16K and 64K base pages) such that one of these chunks only uses a single TLB entry. So we let the architecture specify the order of the maximally beneficial mapping unit when PTE-mapped. Furthermore, because in some cases, this order may be quite big (and therefore potentially wasteful of memory), allow the arch to specify 2 values; One is the max order for a mapping that _would not_ use THP if all size and alignment constraints were met, and the other is the max order for a mapping that _would_ use THP if all those constraints were met. Implement this with Kconfig by introducing some new options to allow the architecture to declare that it supports large anonymous folios along with these 2 preferred max order values. Then introduce a user-facing option, LARGE_ANON_FOLIO, which defaults to disabled and can only be enabled if the architecture has declared its support. When disabled, it forces the max order values, LARGE_ANON_FOLIO_NOTHP_ORDER_MAX and LARGE_ANON_FOLIO_THP_ORDER_MAX to 0, meaning only a single page is ever allocated. Signed-off-by: Ryan Roberts --- mm/Kconfig | 39 +++++++++++++++++++++++++++++++++++++++ mm/memory.c | 8 ++++++++ 2 files changed, 47 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 7672a22647b4..f4ba48c37b75 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1208,4 +1208,43 @@ config PER_VMA_LOCK source "mm/damon/Kconfig" +config ARCH_SUPPORTS_LARGE_ANON_FOLIO + def_bool n + help + An arch should select this symbol if wants to allow LARGE_ANON_FOLIO + to be enabled. It must also set the following integer values: + - ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX + - ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX + +config ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX + int + help + The maximum size of folio to allocate for an anonymous VMA PTE-mapping + that does not have the MADV_HUGEPAGE hint set. + +config ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX + int + help + The maximum size of folio to allocate for an anonymous VMA PTE-mapping + that has the MADV_HUGEPAGE hint set. + +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on ARCH_SUPPORTS_LARGE_ANON_FOLIO + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible. This reduces the number of page faults, as well as other + per-page overheads to improve performance for many workloads. + +config LARGE_ANON_FOLIO_NOTHP_ORDER_MAX + int + default 0 if !LARGE_ANON_FOLIO + default ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX + +config LARGE_ANON_FOLIO_THP_ORDER_MAX + int + default 0 if !LARGE_ANON_FOLIO + default ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX + endmenu diff --git a/mm/memory.c b/mm/memory.c index 9165ed1b9fc2..a8f7e2b28d7a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3153,6 +3153,14 @@ static struct folio *try_vma_alloc_movable_folio(struct vm_area_struct *vma, return vma_alloc_movable_folio(vma, vaddr, 0, zeroed); } +static inline int max_anon_folio_order(struct vm_area_struct *vma) +{ + if (hugepage_vma_check(vma, vma->vm_flags, false, true, true)) + return CONFIG_LARGE_ANON_FOLIO_THP_ORDER_MAX; + else + return CONFIG_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX; +} + /* * Handle write page faults for pages that can be reused in the current vma * From patchwork Mon Jun 26 17:14:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13293238 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38086EB64D7 for ; Mon, 26 Jun 2023 17:15:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C4F048E0005; Mon, 26 Jun 2023 13:15:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BFF468E0002; Mon, 26 Jun 2023 13:15:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD1388E0005; Mon, 26 Jun 2023 13:15:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9D8B78E0002 for ; Mon, 26 Jun 2023 13:15:11 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4949714079A for ; Mon, 26 Jun 2023 17:15:11 +0000 (UTC) X-FDA: 80945549622.02.2E01796 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf12.hostedemail.com (Postfix) with ESMTP id 6694740026 for ; Mon, 26 Jun 2023 17:15:09 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf12.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687799709; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6MmZkyfi0JXPji2JdUjitVD1cfMLJ46y/AmVM9gAT7Q=; b=tTGs2qSFFdkMNsDP23azLWRXZOY0WwH5wNkrGHHEQvCv0NWJ7QI7YxqOpAqRFGRCujlePc L8dKYUbYE2SpQ9QEZw88C0Pk9Vimnp5QhHrdhjciIEhxeM+Ub9YLxjgbBex39byg3TOf1A JPQ7W1DBECWIMcbCH0yKY8p+ZvDf9Mk= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf12.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687799709; a=rsa-sha256; cv=none; b=42nsDS8gHXNHS3oyukwO8QsxLQcB+psLG3LmeI18fzAT+8G1SwiDR1+Zar5kPA5L3Xppch BtUc2/vJ3JcE0qu2Xwdpzkw6TSjpiJF82u3nHiQa6KeQyNnFDdSlts4hbPtm4hkKDFgURd 7TOPWrPznQZVwqu4MLMHFrZYAyE/4p0= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C111215A1; Mon, 26 Jun 2023 10:15:52 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D7EBC3F663; Mon, 26 Jun 2023 10:15:05 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Geert Uytterhoeven , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-alpha@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-s390@vger.kernel.org Subject: [PATCH v1 09/10] arm64: mm: Declare support for large anonymous folios Date: Mon, 26 Jun 2023 18:14:29 +0100 Message-Id: <20230626171430.3167004-10-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230626171430.3167004-1-ryan.roberts@arm.com> References: <20230626171430.3167004-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 6694740026 X-Stat-Signature: d6i9hb9w4iba3b89ikxr6ikrcw7dygwq X-Rspam-User: X-HE-Tag: 1687799709-304830 X-HE-Meta: U2FsdGVkX1/4lXiPvUvbX8zpVMRHp22R4QNuMdQGzsDm5eziBZdoR91KRLwZZlSGi+oM1ESmScTXcgxA9qI3fePDVcHzAj8ZeoO1TXOz630LNP/tRY7JvhDcwABYpWVECAR35+ZZB9UbFy0gVsJ3NiuCNllzipmjs1ChAhQPEH+9kH9tSzcrjyqRXP4Rr93KTU00fOPcyJYdlzpmdgJwkHW+EovywdB2yDNsX9O98jw6CLOu1RBgNQvuuNY6cp1sXd/rmRMz+Ssj81SvQgxp6pyxK2UvffoxBF3x5RRV4sQG/dKhyC1DWhL7Dim43ogilpNLlKF7Gb+i43Gobh30+Bc9nK4KcdwY95l5SXVy68pyA0wlSwwN7ndqQDeCTwhY2b07bGhgNJsc7OSo/J4NJcOZqtkQfRo/nTzZxe4+2Jqa8LDvpluyVv/RiSym2YKoSVcQZFFTGFtwoqqVGyBoq+2eP4UPHlEWvHh+jaHpUeh2qjaerVGCzHvqSpUhxJWNiIfkv96Ob6vNCQFu9t1rkhXVvBXJkvtSTMaMVH54d63tASgVjP5rzIrIqIg/YZcFG7PDG++QkD2/0RVBl/iVv6Wkvcd6JxZQPxMD07p5t3Nw2B/2aJjjs8WbicjkYuPxpslvWPKDuXs4k2xH2kgpgkRVdwsiDoBF6CHyVDXEQpdeOo3AebRZYHKO+lgu4gUFkc/8HNWvmD1598xdSGQ1hVB8b5a2hphusJcxsS1cJOaImw+mHh802kDukoEyvRnnCe2hZtdo8pbgrnK2KtL0QqPB+HJsnlhPXKzSZ4pE1TfIXcCqa9xWGjzil2nvjP7Zi4j5PS3ChhG9yN+3fo5q0JfZe7tfEqoe4noEI7kmrlfTVDZtc4oFB2C7V5LHyoFjJPaxqvFtBst1VNF4f5x9lKtua0pswzi3+UXOEXJJPd0XA6FyktzBajTjLAIbTy8wukbfNphnhOZP7OF6uSW 2Hhh6OKF Oa3z72gcSTyn6ygGqajIF9mPC9DYUWRaVqEuaHCzBuB97Dh/o1c5wM/gkJmkxe8xEAwsf6GGN+xNS48+l/dpv2f37st0/szHP+kWN+l0K7p9ScJHW/U62TF4VHkuT69ixPRyvZSoFgTqhBN2VNnvtGItzXbLlKvnWlvK6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For the unhinted case, when THP is not permitted for the vma, don't allow anything bigger than 64K. This means we don't waste too much memory. Additionally, for 4K pages this is the contpte size, and for 16K, this is (usually) the HPA size when the uarch feature is implemented. For the hinted case, when THP is permitted for the vma, allow the contpte size for all page size configurations; 64K for 4K, 2M for 16K and 2M for 64K. Signed-off-by: Ryan Roberts --- arch/arm64/Kconfig | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 343e1e1cae10..0e91b5bc8cd9 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -243,6 +243,7 @@ config ARM64 select TRACE_IRQFLAGS_SUPPORT select TRACE_IRQFLAGS_NMI_SUPPORT select HAVE_SOFTIRQ_ON_OWN_STACK + select ARCH_SUPPORTS_LARGE_ANON_FOLIO help ARM 64-bit (AArch64) Linux support. @@ -281,6 +282,18 @@ config ARM64_CONT_PMD_SHIFT default 5 if ARM64_16K_PAGES default 4 +config ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX + int + default 0 if ARM64_64K_PAGES # 64K (1 page) + default 2 if ARM64_16K_PAGES # 64K (4 pages; benefits from HPA where HW supports it) + default 4 if ARM64_4K_PAGES # 64K (16 pages; eligible for contpte-mapping) + +config ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX + int + default 5 if ARM64_64K_PAGES # 2M (32 page; eligible for contpte-mapping) + default 7 if ARM64_16K_PAGES # 2M (128 pages; eligible for contpte-mapping) + default 4 if ARM64_4K_PAGES # 64K (16 pages; eligible for contpte-mapping) + config ARCH_MMAP_RND_BITS_MIN default 14 if ARM64_64K_PAGES default 16 if ARM64_16K_PAGES From patchwork Mon Jun 26 17:14:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13293239 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40B13EB64D7 for ; Mon, 26 Jun 2023 17:15:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DAE168E0006; Mon, 26 Jun 2023 13:15:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D5E2D8E0002; Mon, 26 Jun 2023 13:15:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BFF2D8E0006; Mon, 26 Jun 2023 13:15:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B28798E0002 for ; Mon, 26 Jun 2023 13:15:15 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5ECDD1207D3 for ; Mon, 26 Jun 2023 17:15:15 +0000 (UTC) X-FDA: 80945549790.15.816A66B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf16.hostedemail.com (Postfix) with ESMTP id 7202418000A for ; Mon, 26 Jun 2023 17:15:12 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf16.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687799712; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ayx1XUOP8Jqvje8vyoH06qGKySRPMl/T4IoLYSq146A=; b=3WIU7PPKklb5TVgT2HneXczEmYmQfrSWWJui62mXlatFdsKeSUrGO87DSn2BGUAyl4015M U0bls+DMuqIZGO1xMfEM+txG2CU7whuZlU636/g142IOMXAUALt2WOuqr48vRJpewf8WCk qQN3kApIXsYkabP/eyxPoMxgXhmJNmg= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf16.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687799712; a=rsa-sha256; cv=none; b=clucTa/HsRemtjqPfB7uqB43XoDhqR74t//xmfYLe8Kt1pCp5L9PkSjr6FXq7KCjn1J9zi RDyElnhul7eTnrUdSZ/eFMgFdXkldydRHa5DtEIThnJ9Wc0azwo3zsm/RFzALOHAWlgC05 tWjM0qGCNDbcGY57fvCVwfRo1/oqTyI= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D0EE615DB; Mon, 26 Jun 2023 10:15:55 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E66B63F663; Mon, 26 Jun 2023 10:15:08 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Geert Uytterhoeven , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-alpha@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-s390@vger.kernel.org Subject: [PATCH v1 10/10] mm: Allocate large folios for anonymous memory Date: Mon, 26 Jun 2023 18:14:30 +0100 Message-Id: <20230626171430.3167004-11-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230626171430.3167004-1-ryan.roberts@arm.com> References: <20230626171430.3167004-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 7202418000A X-Stat-Signature: seuqhu5mq11k977mu6p3qt7je1hkngy3 X-HE-Tag: 1687799712-799889 X-HE-Meta: U2FsdGVkX19lAyrMgDLOHeyFmSR6jsE+DaVv2Hcg11t+Bil3rbP+LUDamVBBSKO4T0SzbTdAc+lMJw0aBccsKPy2I+4IBb7FlqqqRexcvA460JFt7RxKIIZsK7ndnAzUAjwGaZapOvPhI/yf4TuJiedVRG8ld63XkDetpyNNzMlUXk487cdhxbs8lB26bJ2I8RRup7HZWlhUqtIV1Rc6AIzWWtikv1mNVaZCHpy7L/xf+rtkZecBdT2Nn6MAecfawNvT/lVQtxeaq44iSIzYmx/y5mPGPCFfE2MpL0lykOEmgtJVA6ZvNnSepKNwYZYDPV1w8O+PszfVY5vGVp4o8QEtyBqqqs/TpWIKqta+U4Psj5/a/fgOzalE0obLrveS6vW7QwwS/QnJgEnuE/16r9vKueWC5olQV7IvF79XZvpENt3ihSB8wG69YM+IXWokaCYmzg7v90x2mL0M5oUmPW1XH8RYiQl/HouhC8mkKom4RGTbVCncFVLQlSNp1rCJEmNwpKJCzCu8Klm4gufGyyls6al4D3zEQP7GXK4GSE0cjHtURCyQj7mz4MeiGv3G6FxjJeVGrFE+7GuqR1KHR8js9liM2h5ByOUYEEIZa54AaUR0jeCkQZWRzgfJzq+I7CBKSaMrwnlyrBBbG9PEx8DoSFpNI21sta4CQ1rXLatvAUwN5N6p0oVPBMahB+hSgZep32MtYKCVbBnH9U0lbeL6tZDEaCQoGSONUyvhUkOJ55qJ/NKozNEmciAwQz8mKp28jj8YKQC1Tpk0Dg/B5OnVSKLIKiJJyO6LpmwiN04hB/GPtERpOespBrdblhGh1H28SKlavUAhMotHN3llIdAbo8qWBeTt0T7aTGwfWrksTNPRhkSfWunkAvj00qr7ueXfQO4Gh1EEPDm74DksR1+FmIPPPjqQn4v9TjxclL5zDiQ/0UhMSoGiH++zCI56T6w79vtT0mpTqkNY0F9 xg4qtquF BZh9EjBJDvOi2QecuhCqjMvSeqn/g/wGRDfjGSXDurMRCn+EV1QiREUxWJkpsA2no5ewmnXqxOLv7V293QLYAYsOmS58BvYppXlqwM41oMOfVBHf8jtmCdrI+MKh4EJuyH9WK3T3luaCdV5NBepLk/WfznyoEjnlTK2MX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With all of the enabler patches in place, modify the anonymous memory write allocation path so that it opportunistically attempts to allocate a large folio up to `max_anon_folio_order()` size (This value is ultimately configured by the architecture). This reduces the number of page faults, reduces the size of (e.g. LRU) lists, and generally improves performance by batching what were per-page operations into per-(large)-folio operations. If CONFIG_LARGE_ANON_FOLIO is not enabled (the default) then `max_anon_folio_order()` always returns 0, meaning we get the existing allocation behaviour. Signed-off-by: Ryan Roberts --- mm/memory.c | 159 +++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 144 insertions(+), 15 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index a8f7e2b28d7a..d23c44cc5092 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3161,6 +3161,90 @@ static inline int max_anon_folio_order(struct vm_area_struct *vma) return CONFIG_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX; } +/* + * Returns index of first pte that is not none, or nr if all are none. + */ +static inline int check_ptes_none(pte_t *pte, int nr) +{ + int i; + + for (i = 0; i < nr; i++) { + if (!pte_none(ptep_get(pte++))) + return i; + } + + return nr; +} + +static int calc_anon_folio_order_alloc(struct vm_fault *vmf, int order) +{ + /* + * The aim here is to determine what size of folio we should allocate + * for this fault. Factors include: + * - Order must not be higher than `order` upon entry + * - Folio must be naturally aligned within VA space + * - Folio must not breach boundaries of vma + * - Folio must be fully contained inside one pmd entry + * - Folio must not overlap any non-none ptes + * + * Additionally, we do not allow order-1 since this breaks assumptions + * elsewhere in the mm; THP pages must be at least order-2 (since they + * store state up to the 3rd struct page subpage), and these pages must + * be THP in order to correctly use pre-existing THP infrastructure such + * as folio_split(). + * + * As a consequence of relying on the THP infrastructure, if the system + * does not support THP, we always fallback to order-0. + * + * Note that the caller may or may not choose to lock the pte. If + * unlocked, the calculation should be considered an estimate that will + * need to be validated under the lock. + */ + + struct vm_area_struct *vma = vmf->vma; + int nr; + unsigned long addr; + pte_t *pte; + pte_t *first_set = NULL; + int ret; + + if (has_transparent_hugepage()) { + order = min(order, PMD_SHIFT - PAGE_SHIFT); + + for (; order > 1; order--) { + nr = 1 << order; + addr = ALIGN_DOWN(vmf->address, nr << PAGE_SHIFT); + pte = vmf->pte - ((vmf->address - addr) >> PAGE_SHIFT); + + /* Check vma bounds. */ + if (addr < vma->vm_start || + addr + (nr << PAGE_SHIFT) > vma->vm_end) + continue; + + /* Ptes covered by order already known to be none. */ + if (pte + nr <= first_set) + break; + + /* Already found set pte in range covered by order. */ + if (pte <= first_set) + continue; + + /* Need to check if all the ptes are none. */ + ret = check_ptes_none(pte, nr); + if (ret == nr) + break; + + first_set = pte + ret; + } + + if (order == 1) + order = 0; + } else + order = 0; + + return order; +} + /* * Handle write page faults for pages that can be reused in the current vma * @@ -4201,6 +4285,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) struct folio *folio; vm_fault_t ret = 0; pte_t entry; + unsigned long addr; + int order = uffd_wp ? 0 : max_anon_folio_order(vma); + int pgcount = BIT(order); /* File mapping without ->vm_ops ? */ if (vma->vm_flags & VM_SHARED) @@ -4242,24 +4329,44 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); return handle_userfault(vmf, VM_UFFD_MISSING); } - goto setpte; + if (uffd_wp) + entry = pte_mkuffd_wp(entry); + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); + + /* No need to invalidate - it was non-present before */ + update_mmu_cache(vma, vmf->address, vmf->pte); + goto unlock; } - /* Allocate our own private page. */ +retry: + /* + * Estimate the folio order to allocate. We are not under the ptl here + * so this estiamte needs to be re-checked later once we have the lock. + */ + vmf->pte = pte_offset_map(vmf->pmd, vmf->address); + order = calc_anon_folio_order_alloc(vmf, order); + pte_unmap(vmf->pte); + + /* Allocate our own private folio. */ if (unlikely(anon_vma_prepare(vma))) goto oom; - folio = vma_alloc_zeroed_movable_folio(vma, vmf->address, 0, 0); + folio = try_vma_alloc_movable_folio(vma, vmf->address, order, true); if (!folio) goto oom; + /* We may have been granted less than we asked for. */ + order = folio_order(folio); + pgcount = BIT(order); + addr = ALIGN_DOWN(vmf->address, pgcount << PAGE_SHIFT); + if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL)) goto oom_free_page; folio_throttle_swaprate(folio, GFP_KERNEL); /* * The memory barrier inside __folio_mark_uptodate makes sure that - * preceding stores to the page contents become visible before - * the set_pte_at() write. + * preceding stores to the folio contents become visible before + * the set_ptes() write. */ __folio_mark_uptodate(folio); @@ -4268,11 +4375,31 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); - if (vmf_pte_changed(vmf)) { - update_mmu_tlb(vma, vmf->address, vmf->pte); - goto release; + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl); + + /* + * Ensure our estimate above is still correct; we could have raced with + * another thread to service a fault in the region. + */ + if (order == 0) { + if (vmf_pte_changed(vmf)) { + update_mmu_tlb(vma, vmf->address, vmf->pte); + goto release; + } + } else if (check_ptes_none(vmf->pte, pgcount) != pgcount) { + pte_t *pte = vmf->pte + ((vmf->address - addr) >> PAGE_SHIFT); + + /* If faulting pte was allocated by another, exit early. */ + if (!pte_none(ptep_get(pte))) { + update_mmu_tlb(vma, vmf->address, pte); + goto release; + } + + /* Else try again, with a lower order. */ + pte_unmap_unlock(vmf->pte, vmf->ptl); + folio_put(folio); + order--; + goto retry; } ret = check_stable_address_space(vma->vm_mm); @@ -4286,16 +4413,18 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) return handle_userfault(vmf, VM_UFFD_MISSING); } - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_ref_add(folio, pgcount - 1); + + add_mm_counter(vma->vm_mm, MM_ANONPAGES, pgcount); + folio_add_new_anon_rmap_range(folio, &folio->page, pgcount, vma, addr); folio_add_lru_vma(folio, vma); -setpte: + if (uffd_wp) entry = pte_mkuffd_wp(entry); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); + set_ptes(vma->vm_mm, addr, vmf->pte, entry, pgcount); /* No need to invalidate - it was non-present before */ - update_mmu_cache(vma, vmf->address, vmf->pte); + update_mmu_cache_range(vma, addr, vmf->pte, pgcount); unlock: pte_unmap_unlock(vmf->pte, vmf->ptl); return ret;