From patchwork Fri Jul 14 16:17:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13313836 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0ECEC001B0 for ; Fri, 14 Jul 2023 16:17:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 786DA900005; Fri, 14 Jul 2023 12:17:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 739028E000A; Fri, 14 Jul 2023 12:17:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5FFA9900005; Fri, 14 Jul 2023 12:17:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4EB2C8E000A for ; Fri, 14 Jul 2023 12:17:47 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 1CEFCA02AD for ; Fri, 14 Jul 2023 16:17:47 +0000 (UTC) X-FDA: 81010723374.16.CDB6557 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf20.hostedemail.com (Postfix) with ESMTP id 580DB1C0003 for ; Fri, 14 Jul 2023 16:17:45 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf20.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689351465; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MtFe+SBpb58GfXPkNa/WZIUSJbi6BRi5IT0DzQBQLj0=; b=VIAjOpzshnGnsqAtrFweu8j/6ZvEq0aRlK5NDNxqIJ+Ux4WyIwRyAmIQ1+s0MY+m5yJOaP 5q8j1IJQ7cyFG3olf7tvuaCGJAB340KhbBuUElNj6ugC4EYm3b6Xy4AFqAUADXgtfXNwKI JnWf5IpIq6IdOd+ZFzFO5CdkFT7EgF8= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf20.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689351465; a=rsa-sha256; cv=none; b=GoscbgQ1vx/Bl64VN/rJKyZwiPE5qzlaVaGPnaSAWlHppgsvolpoWFZ/p8MMlw+E4fLrfr wpUraNJRbbcEXW9O5VPDFhAOVjh0J01/Nku6Neit903NzYhKDHfQZ6hvEyRsSNteisaG/B vyTal5Wxy/SpG0TigtTOD7fjGQr9uBA= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 56B861570; Fri, 14 Jul 2023 09:18:26 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 089E33F740; Fri, 14 Jul 2023 09:17:41 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 1/4] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap() Date: Fri, 14 Jul 2023 17:17:30 +0100 Message-Id: <20230714161733.4144503-1-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230714160407.4142030-1-ryan.roberts@arm.com> References: <20230714160407.4142030-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 580DB1C0003 X-Stat-Signature: 95pynny8c97qrjjeoxg39psakezdmkhd X-HE-Tag: 1689351465-412295 X-HE-Meta: U2FsdGVkX1+4s1iqRDX03VUB1YwzWgLhLHveUAjWjIzdHWnpOpfC0sar9/WP1BkpdKDfRv+Im7fVivv6lanaAxGqTrgyK2GtMB2hghDtb+g3Acmk9V0e3CTplqaJw5QFQVanjphi3dEjfseCjnqpFth83l1PiqAjhl06+04KD8SoXMlIAPSbLUr4Lq+Bk5tI8EY3FUsvN/xnXseqPAYWmMIAsvUzEmFAIGAhkpLNHbvAalaXULDHkTolrgno5othRCsMdBgFw03wU0Ro2Vqi1sMgjKEgibMABWyQmK6jcc8wbqlWM9SKwKsv/vfQg0WZ0kRr94srSoSw5e4JEMbhzU2LATTZ3fti8FNaMl7nXCtXG1aVB07w27QlquzVEA1Y5Qp2GJ1jzznf0PXXUKjbQDbTSgI3NdYWijIg5g45J4KzJUcPWhR/xV7ikCS6wDiVp7JJJPY28N/ERGepWV8Gbjcl/Hg/N5macO4RtGRyUc1yShp725IjgupLyRLRn7zKrzqkk0W9FIZdaTKjvO9KYPwdPrdmWDOyTHsWTRxQmXGyAu5nEgyKXvyErjrzK5j2PF/tH8b5kiyM6vSlWUsGgwufxAkBGlwY7rOavzwfapNJM+kxyPwOW/324RG7m/ZfN36K+qmqQmBJDFeR90xCDUFZp6pEMHest7xtD1Scb0TWXkYU7gCYM5GBEecDi+oUve6ImtsmBPWtpkVXISt0JU/2MCZjaNXRdgI3EBEtXlS0zNhxzwuiELiSVB7ezl9Owl1lbdHn/bGU96/6lyvWtPIhVDV5s7r1OKdR9jxiGRrCU3Ny4FxCiqgwSs/W8z8Nu9sUQIVcSfDiOopjcIhPTPFbewT3SB9hKdc+KuRhwSKBYiHGOz3bpz0wl2yNlkLJ0FT4BdrmIOyZyTO/NsH4RNWbpJ+6pvHWhHWFQfVqROa2EVZGJ/+0n4Fi6U2sHn4NYJMntFOm82Os8iq6PhX THBIIQIp dJ3a/0mYRHe+jRhxogUyu+AMUSErnJTP56DzMOMnVvfvbwHqVOksJFGY5mvWSSEyuzWku0F/aEPvMdHX5h+YipzHl1k94yiQ+i37uT4wjxdO65NV4yejlG28JLxSKNUqDY5yRJXB3M80p9bbhxcsAbJm3eBaJCArTIY3p6Skd/vBRpN1mQDZ8JeeLx9ajMiA83W1TpO4hAK53e8D+Fpb9l2CtduNQNCvhXN3Blryg6OotQyxuVKF/kbYRGQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for FLEXIBLE_THP support, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the order-0 folio (or base page) scheme. Signed-off-by: Ryan Roberts Reviewed-by: Yu Zhao Reviewed-by: Yin Fengwei --- mm/rmap.c | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) -- 2.25.1 diff --git a/mm/rmap.c b/mm/rmap.c index 0c0d8857dfce..f293d072368a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1278,31 +1278,45 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, * This means the inc-and-test can be bypassed. * The folio does not have to be locked. * - * If the folio is large, it is accounted as a THP. As the folio + * If the folio is pmd-mappable, it is accounted as a THP. As the folio * is new, it's assumed to be mapped exclusively by a single process. */ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address) { - int nr; + int nr = folio_nr_pages(folio); - VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma); + VM_BUG_ON_VMA(address < vma->vm_start || + address + (nr << PAGE_SHIFT) > vma->vm_end, vma); __folio_set_swapbacked(folio); - if (likely(!folio_test_pmd_mappable(folio))) { + if (!folio_test_large(folio)) { /* increment count (starts at -1) */ atomic_set(&folio->_mapcount, 0); - nr = 1; + __page_set_anon_rmap(folio, &folio->page, vma, address, 1); + } else if (!folio_test_pmd_mappable(folio)) { + int i; + + for (i = 0; i < nr; i++) { + struct page *page = folio_page(folio, i); + + /* increment count (starts at -1) */ + atomic_set(&page->_mapcount, 0); + __page_set_anon_rmap(folio, page, vma, + address + (i << PAGE_SHIFT), 1); + } + + /* increment count (starts at 0) */ + atomic_set(&folio->_nr_pages_mapped, nr); } else { /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); atomic_set(&folio->_nr_pages_mapped, COMPOUND_MAPPED); - nr = folio_nr_pages(folio); + __page_set_anon_rmap(folio, &folio->page, vma, address, 1); __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr); } __lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr); - __page_set_anon_rmap(folio, &folio->page, vma, address, 1); } /** From patchwork Fri Jul 14 16:17:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13313837 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A843EB64DC for ; Fri, 14 Jul 2023 16:17:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 90EE28E000A; Fri, 14 Jul 2023 12:17:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8C078900006; Fri, 14 Jul 2023 12:17:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 70FE68E000F; Fri, 14 Jul 2023 12:17:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5B7708E000A for ; Fri, 14 Jul 2023 12:17:49 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 332CE160301 for ; Fri, 14 Jul 2023 16:17:49 +0000 (UTC) X-FDA: 81010723458.21.B7AC580 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf06.hostedemail.com (Postfix) with ESMTP id 791EF18002E for ; Fri, 14 Jul 2023 16:17:47 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf06.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689351467; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RwsmSgL54KvIcN9UUxItjXrDFbdgmMPuqsBrxLvJGfQ=; b=vf2T3Q6Z2O9UGRT3ZXGwnzIRatSMSX5oG1iK16SBb6exwhgKo6lGnoJyVR+0xjuGzRYhfo GcU2f+KF+AnDjhq/WQeIT4ehVxrtAn3h2Np2vepznuC+j9MPsaUXffbTBUfdUxsNRoQWtl V/E3AnfMTc1nSWJh+3GIaG5S33b2sn8= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf06.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689351467; a=rsa-sha256; cv=none; b=nr9lvwP8/gDnlgFlvl39SE844kvkgx2qdoxt3UWgL9E7Vv9rC9HuU0orRvrl1fzGC7gle2 9hwiTS0zfse6lahPY7bFBklF3ReU0iJIuJ0hnNIt6NrtF2Q2ia3mQpMLhd1MtOdVVLjvg8 z6ORveN85IREO/6jVxfZgOrO6BakVGs= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9EB8E1576; Fri, 14 Jul 2023 09:18:28 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 527963F740; Fri, 14 Jul 2023 09:17:44 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 2/4] mm: Default implementation of arch_wants_pte_order() Date: Fri, 14 Jul 2023 17:17:31 +0100 Message-Id: <20230714161733.4144503-2-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230714160407.4142030-1-ryan.roberts@arm.com> References: <20230714160407.4142030-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 791EF18002E X-Stat-Signature: jc9jx4enr799x5bjqgwyacyohog9yk1y X-HE-Tag: 1689351467-710830 X-HE-Meta: U2FsdGVkX18Ho40TwBz2kUL9CfHN9NkCZeb+rdQq2CLpkCgfdWlla3inqkWoy16O/kDV8Lgl65f3a+UenRhKdNBMtVE5You2Zi4tiAAKC036Xh6YTD905nVvrpFYakESNC5vyct5eF1C/7HjEUguqsgsYPqoZFTH5Fi60Dhze3t+PD87RiFRGHpsMMxv9w8NsnNLwLkiOlBAzEsj9Ketubi3+muTB56G2g7BIqIIPgVOsZOF5M77j6PCZYiQaxseW6HKPANd+RWVffEBCSzXz0NS5LCpVwoI2LOQ2RpB8x2tPVK+5ynCFSeWgE+JxHX1qfaqJCd4xLOaGsdTjW7uieMR+uyhqYWNA+P1NYiUDrYaFeJC0lzNXq5FpbnvGRWk5bPVrhF8nNxl/3k33plkoDmegs6eY7qAkO+bgV4kpnGz/2ZTguKTdqtbPmBEiof8RaLMcgNUA7w90KVpe5kraTtKziT5qbQkK44r/KorajeIZCfVgWoHxX7bbCUjlTEFNtn2sjwuW8nlvM2wSfwGDVKnGgPPAVMr+QbSwYgKUjFrdBTVvVqpZvq64S5IfyeLUg4w6SdxQnG1eZXTj2yd3t266iELkgQfnrtYuty12WPDmkbQJM/tKfxoTtZHwJVuyQttD5UpNDupxMsQcAnhh65GuepWDv6O7ZFc9DbJVF2lt28su7Nc2ZGSEWnYnXNui6XuN3lniNAeosFEIvdw8lhjfswW76JXlxJk/5sT15y9RxtPzogExpxkjuhdylNB83DzhTeWWdKqUSHkBqi81MNZkWoNNz7gcs4MG3S+GMA6NaTqmHMObcaeeAn2DnC5hFMtA3Dv4bIOvz3UgvjUw1kfSGFvnsOSWql5u69HpBwNVYvHpyL8UUNFhO78JPyREboqaQce8NrdWOsUsKTlQ7bkg2cwLkxgVCC9WP7KQ8ncvAGnFQ2wDxgWzsKoBBM+IogjdL1YWRWlQomwcLe UIiO9wgI 8W9z874cvIbrGs5m3ou71bgckqR0wfRM36VJqAjlD2idbN6pS/EVXXCnCw2+2ITzMcLpsYnMylUc8BmO5IZQ7A21z0oBSB/HEsm9zU2q5nF8ldwhAstWl6lvTqre6K/wQgGrTvYNu/csespZcaBJ6iii3KFiW9b3BI7C6ba5rJBLgXfGT05LGbgnNCoOfQfn7Y9CpHD3Yigg+SRw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: arch_wants_pte_order() can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous. The first user for this hint will be FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs. Here we add the default implementation of the function, used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order. Signed-off-by: Ryan Roberts Reviewed-by: Yu Zhao Reviewed-by: Yin Fengwei --- include/linux/pgtable.h | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 5063b482e34f..2a1d83775837 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) } #endif +#ifndef arch_wants_pte_order +/* + * Returns preferred folio order for pte-mapped memory. Must be in range [0, + * PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios + * to be at least order-2. Negative value implies that the HW has no preference + * and mm will choose it's own default order. + */ +static inline int arch_wants_pte_order(void) +{ + return -1; +} +#endif + #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address, From patchwork Fri Jul 14 16:17:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13313838 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 860E9C001B0 for ; Fri, 14 Jul 2023 16:17:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1BC3C900007; Fri, 14 Jul 2023 12:17:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 16B41900006; Fri, 14 Jul 2023 12:17:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 00B75900007; Fri, 14 Jul 2023 12:17:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E4D43900006 for ; Fri, 14 Jul 2023 12:17:51 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id AF56316021A for ; Fri, 14 Jul 2023 16:17:51 +0000 (UTC) X-FDA: 81010723542.26.84034FB Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf29.hostedemail.com (Postfix) with ESMTP id CB00712002F for ; Fri, 14 Jul 2023 16:17:49 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; spf=pass (imf29.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689351469; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tX8967EIh0HI9YMP7qJcM0Jgu1MCSasQRcYW7+VsDaw=; b=tapaKJuvEXIQlqmiVAEWWB0/5+7g1xlD9xnZqYQCGi7aiRWEclydbn80RZZIx1nHcyv+xH 4tJxNm/zLRkEkzLNMhuYie/wLqrYHgzYpCB1FKMar4xjIS2jxwPbO/CXStxeAaIs/P89R9 n7elYeHRK9vfyMipr11jDn1s5A2InIk= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; spf=pass (imf29.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689351469; a=rsa-sha256; cv=none; b=Swu7+b5HBK4tKqpVHbyTmBiA8ZErOwVz+UyuvHILmCmHEGqYgnK31U8bnXgoz72YAuUMC0 NssnWZTaz0tVq/JxzQxavznp3ZBE03FfLPW1Y/2CheFVO7VFgz/SswFSoOlGFIUEdquCkv 3qsnGiqb3qcttcjmX8KUyxdFt9moFt0= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0E83215A1; Fri, 14 Jul 2023 09:18:31 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9C00F3F740; Fri, 14 Jul 2023 09:17:46 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 3/4] mm: FLEXIBLE_THP for improved performance Date: Fri, 14 Jul 2023 17:17:32 +0100 Message-Id: <20230714161733.4144503-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230714160407.4142030-1-ryan.roberts@arm.com> References: <20230714160407.4142030-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: CB00712002F X-Rspam-User: X-Stat-Signature: 6g3upsk5sydjnmui4if68u43yena6u4h X-Rspamd-Server: rspam01 X-HE-Tag: 1689351469-328171 X-HE-Meta: U2FsdGVkX18THKcJS4BtlC1X/okOFKz/JFD2DAdNyBWe7jIIoOZDloU7Bf8gatASe5TETi0djbmcMphHAGgQs/jHWkErNFG2I3Uf0ZbV6bwd5p+lASMbFZkoc7onZnjgLFCclvYp21YmRKkA4iuSml7UBBSBl/HKzK3PjYXUCkPkcLwhKDzesAfXyzULhWT9dZrY580G3KeCJgGN+snz9Gl6s2Nty4mPqKOZNtJlGMN6QdezNNLruQUkpDUlq9h5tVq9U9Xylxryo1ULjM6mDMQ/wUf1FZzeqYOm2UL6ELEH7ABb6IcgllqZapXtFDt8WkS4KKzNLerGM3+GW4wIh8r6kBhb5DZKU0oeFQUARcO0akZKBToXyhI7zuHN1xHLfFL1HomLvgYuB4yNWEdQtG4uQynl4xFSW/MDK6tqoW85qGlTjaZIBgZ+TPDrJJOUD9TNlqrz0IT2TQgOm6vGA0nYA6tUjpu+JNmWrr7qEZ2+yyBLyA9iybJAe+ra7CZN4q/tHM+0rCHh1kDIhKyBMR/gbu+CGtsq0tJj463BnF5gHg2rdeZKcNPgdT5EZpocY7o7xRyJYOgyEHusgcg9cdzq6fbu/r6ZUfvx1TIupT9nLIIpvnc5SAKMz/8t+t8ou/nT1Eb3sgjypKgWghz03KWud/n+XGeuu3dX4qIBO7RxUmLvslb1/gZHJE65Z5tEVuUbiPC61dsbxzo1hj3JOSskrQ8Aj29wJ3Q7AIg/v4g21nN04765TMBNkf3ZsJDOVvYd3IZjMsGyfVX9o5/CsXchgponlNXP0k2hY8iXRERblWWw04e9Y9KTVfhsdBAhvXPpnoqGpUwmZqtGI3JPSQp7/u4y8lU5LABcXKyfQG/5wc/LKZSs6A2sisZVdXFJ8lvHEuTMLu+GKGI1DjiSRvEnQUpNd2Qlekw5mifWg5GKaK2rwJsc0PrjkJws6g6EOKnqEDFbY277nTVeAcu wvu6l/pU T7qFKijs3QRuIoIF2Fuz9gSMlN4401BGLCHPzvl7HArTYi7iDVvx4fOqNmxWWBdmtM968mVWkCsgSeCZkqUEk9Kkko0Wcy9UWfJSVBoPTjlBojzQfeoZhWepNvhjxSZ3M17rZZU/XAsT7iE5VTizXoblQzOff0V+dTTP4bVopDBshPriyUbhGwL+Mp6HcS0jRYB8FdP0wVr7VP+o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce FLEXIBLE_THP feature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio. The new behaviour is hidden behind the new FLEXIBLE_THP Kconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first. When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request. Otherwise, the return value of arch_wants_pte_order() is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not have MADV_HUGEPAGE), then arch_wants_pte_order() is limited by the new cmdline parameter, `flexthp_unhinted_max`. This allows for a performance boost without requiring any explicit opt-in from the workload while allowing the sysadmin to tune between performance and internal fragmentation. arch_wants_pte_order() can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required. If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first PAGE_ALLOC_COSTLY_ORDER, then order-0. Signed-off-by: Ryan Roberts --- .../admin-guide/kernel-parameters.txt | 10 + mm/Kconfig | 10 + mm/memory.c | 187 ++++++++++++++++-- 3 files changed, 190 insertions(+), 17 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index a1457995fd41..405d624e2191 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1497,6 +1497,16 @@ See Documentation/admin-guide/sysctl/net.rst for fb_tunnels_only_for_init_ns + flexthp_unhinted_max= + [KNL] Requires CONFIG_FLEXIBLE_THP enabled. The maximum + folio size that will be allocated for an anonymous vma + that has neither explicitly opted in nor out of using + transparent hugepages. The size must be a power-of-2 in + the range [PAGE_SIZE, PMD_SIZE). A larger size improves + performance by reducing page faults, while a smaller + size reduces internal fragmentation. Default: max(64K, + PAGE_SIZE). Format: size[KMG]. + floppy= [HW] See Documentation/admin-guide/blockdev/floppy.rst. diff --git a/mm/Kconfig b/mm/Kconfig index 09130434e30d..26c5e51ef11d 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -848,6 +848,16 @@ config READ_ONLY_THP_FOR_FS support of file THPs will be developed in the next few release cycles. +config FLEXIBLE_THP + bool "Flexible order THP" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endif # TRANSPARENT_HUGEPAGE # diff --git a/mm/memory.c b/mm/memory.c index 01f39e8144ef..e8bc729efb9d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4050,6 +4050,148 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) return ret; } +static bool vmf_pte_range_changed(struct vm_fault *vmf, int nr_pages) +{ + int i; + + if (nr_pages == 1) + return vmf_pte_changed(vmf); + + for (i = 0; i < nr_pages; i++) { + if (!pte_none(ptep_get_lockless(vmf->pte + i))) + return true; + } + + return false; +} + +#ifdef CONFIG_FLEXIBLE_THP +static int flexthp_unhinted_max_order = + ilog2(SZ_64K > PAGE_SIZE ? SZ_64K : PAGE_SIZE) - PAGE_SHIFT; + +static int __init parse_flexthp_unhinted_max(char *s) +{ + unsigned long long size = memparse(s, NULL); + + if (!is_power_of_2(size) || size < PAGE_SIZE || size > PMD_SIZE) { + pr_warn("flexthp: flexthp_unhinted_max=%s must be power-of-2 between PAGE_SIZE (%lu) and PMD_SIZE (%lu), ignoring\n", + s, PAGE_SIZE, PMD_SIZE); + return 1; + } + + flexthp_unhinted_max_order = ilog2(size) - PAGE_SHIFT; + + /* THP machinery requires at least 3 struct pages for meta data. */ + if (flexthp_unhinted_max_order == 1) + flexthp_unhinted_max_order--; + + return 1; +} + +__setup("flexthp_unhinted_max=", parse_flexthp_unhinted_max); + +static int anon_folio_order(struct vm_area_struct *vma) +{ + int order; + + /* + * If THP is explicitly disabled for either the vma, the process or the + * system, then this is very likely intended to limit internal + * fragmentation; in this case, don't attempt to allocate a large + * anonymous folio. + * + * Else, if the vma is eligible for thp, allocate a large folio of the + * size preferred by the arch. Or if the arch requested a very small + * size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, + * which still meets the arch's requirements but means we still take + * advantage of SW optimizations (e.g. fewer page faults). + * + * Finally if thp is enabled but the vma isn't eligible, take the + * arch-preferred size and limit it to the flexthp_unhinted_max cmdline + * parameter. This allows a sysadmin to tune performance vs internal + * fragmentation. + */ + + if ((vma->vm_flags & VM_NOHUGEPAGE) || + test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags) || + !hugepage_flags_enabled()) + order = 0; + else { + order = max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_ORDER); + + if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true)) + order = min(order, flexthp_unhinted_max_order); + } + + return order; +} + +static int alloc_anon_folio(struct vm_fault *vmf, struct folio **folio) +{ + int i; + gfp_t gfp; + pte_t *pte; + unsigned long addr; + struct vm_area_struct *vma = vmf->vma; + int prefer = anon_folio_order(vma); + int orders[] = { + prefer, + prefer > PAGE_ALLOC_COSTLY_ORDER ? PAGE_ALLOC_COSTLY_ORDER : 0, + 0, + }; + + *folio = NULL; + + if (vmf_orig_pte_uffd_wp(vmf)) + goto fallback; + + for (i = 0; orders[i]; i++) { + addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << orders[i]); + if (addr >= vma->vm_start && + addr + (PAGE_SIZE << orders[i]) <= vma->vm_end) + break; + } + + if (!orders[i]) + goto fallback; + + pte = pte_offset_map(vmf->pmd, vmf->address & PMD_MASK); + if (!pte) + return -EAGAIN; + + for (; orders[i]; i++) { + addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << orders[i]); + vmf->pte = pte + pte_index(addr); + if (!vmf_pte_range_changed(vmf, 1 << orders[i])) + break; + } + + vmf->pte = NULL; + pte_unmap(pte); + + gfp = vma_thp_gfp_mask(vma); + + for (; orders[i]; i++) { + addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << orders[i]); + *folio = vma_alloc_folio(gfp, orders[i], vma, addr, true); + if (*folio) { + clear_huge_page(&(*folio)->page, addr, 1 << orders[i]); + return 0; + } + } + +fallback: + *folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); + return *folio ? 0 : -ENOMEM; +} +#else +static inline int alloc_anon_folio(struct vm_fault *vmf, struct folio **folio) +{ + *folio = vma_alloc_zeroed_movable_folio(vmf->vma, vmf->address); + return *folio ? 0 : -ENOMEM; +} +#endif + /* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked. @@ -4057,11 +4199,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) { + int i = 0; + int nr_pages = 1; bool uffd_wp = vmf_orig_pte_uffd_wp(vmf); struct vm_area_struct *vma = vmf->vma; struct folio *folio; vm_fault_t ret = 0; pte_t entry; + unsigned long addr; /* File mapping without ->vm_ops ? */ if (vma->vm_flags & VM_SHARED) @@ -4101,10 +4246,15 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) /* Allocate our own private page. */ if (unlikely(anon_vma_prepare(vma))) goto oom; - folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); + ret = alloc_anon_folio(vmf, &folio); + if (unlikely(ret == -EAGAIN)) + return 0; if (!folio) goto oom; + nr_pages = folio_nr_pages(folio); + addr = ALIGN_DOWN(vmf->address, nr_pages * PAGE_SIZE); + if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL)) goto oom_free_page; folio_throttle_swaprate(folio, GFP_KERNEL); @@ -4116,17 +4266,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) */ __folio_mark_uptodate(folio); - entry = mk_pte(&folio->page, vma->vm_page_prot); - entry = pte_sw_mkyoung(entry); - if (vma->vm_flags & VM_WRITE) - entry = pte_mkwrite(pte_mkdirty(entry)); - - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl); if (!vmf->pte) goto release; - if (vmf_pte_changed(vmf)) { - update_mmu_tlb(vma, vmf->address, vmf->pte); + if (vmf_pte_range_changed(vmf, nr_pages)) { + for (i = 0; i < nr_pages; i++) + update_mmu_tlb(vma, addr + PAGE_SIZE * i, vmf->pte + i); goto release; } @@ -4141,16 +4286,24 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) return handle_userfault(vmf, VM_UFFD_MISSING); } - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_ref_add(folio, nr_pages - 1); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + folio_add_new_anon_rmap(folio, vma, addr); folio_add_lru_vma(folio, vma); + + for (i = 0; i < nr_pages; i++) { + entry = mk_pte(folio_page(folio, i), vma->vm_page_prot); + entry = pte_sw_mkyoung(entry); + if (vma->vm_flags & VM_WRITE) + entry = pte_mkwrite(pte_mkdirty(entry)); setpte: - if (uffd_wp) - entry = pte_mkuffd_wp(entry); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); + if (uffd_wp) + entry = pte_mkuffd_wp(entry); + set_pte_at(vma->vm_mm, addr + PAGE_SIZE * i, vmf->pte + i, entry); - /* No need to invalidate - it was non-present before */ - update_mmu_cache(vma, vmf->address, vmf->pte); + /* No need to invalidate - it was non-present before */ + update_mmu_cache(vma, addr + PAGE_SIZE * i, vmf->pte + i); + } unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); From patchwork Fri Jul 14 16:17:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13313839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87AC8C001DC for ; Fri, 14 Jul 2023 16:17:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 16D3F900008; Fri, 14 Jul 2023 12:17:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 11DBC900006; Fri, 14 Jul 2023 12:17:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0A0B900008; Fri, 14 Jul 2023 12:17:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DF124900006 for ; Fri, 14 Jul 2023 12:17:53 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B4073A0322 for ; Fri, 14 Jul 2023 16:17:53 +0000 (UTC) X-FDA: 81010723626.01.4A0000F Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf28.hostedemail.com (Postfix) with ESMTP id CB157C0023 for ; Fri, 14 Jul 2023 16:17:51 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689351471; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=71smc1TZNhMZlMSAnG9fgcjZlzgTrk3pagdzxHJYIq4=; b=1jGUbkGw90sUzbkhRBv0f4uEq+D8Rqsu4Acz+ZpnwAXgMl33wrHqjOY6GwIGZrMuJfVBSy VLi4f5OuXNb6Lh8nQQ2JmmpBfPhoT2kqpefhkB4ti2PXZVNqK0hcE8Wn7Chaov91sE0cGI 0NN/7lTSGaYNgj8i5hVmlYtWBqBTSfw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689351471; a=rsa-sha256; cv=none; b=lt5QyYXj6yBz7/xFeou1iakJP+cXBAF3g7feHVrJWEst/+PeB9yqL8DfxZtJH2DtHp5/Tr gfV0/G4FlBogoJDAifIwleUWl19n7gBT0jgJw9FDO/qzP/hOGiLTbnDGgbXN2CyjRdy1rx jnKpOm7JV7LLf/lKhHhuwSOLj6t4eHU= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5789215BF; Fri, 14 Jul 2023 09:18:33 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0B8643F740; Fri, 14 Jul 2023 09:17:48 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 4/4] arm64: mm: Override arch_wants_pte_order() Date: Fri, 14 Jul 2023 17:17:33 +0100 Message-Id: <20230714161733.4144503-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230714160407.4142030-1-ryan.roberts@arm.com> References: <20230714160407.4142030-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: CB157C0023 X-Rspam-User: X-Stat-Signature: nt4x3nudto4nd165f6s99iacwg7eo7ai X-Rspamd-Server: rspam03 X-HE-Tag: 1689351471-552725 X-HE-Meta: U2FsdGVkX1/kuQ0OTm5xJfO6MguAY1Zfp0Rsy4VOGV8cQirjfDNBmgo2M7ilvoqiowWdnR7jev3dzoKMXCkcZtI1K3NIzj74eTPZIuFV+dZxHx73VJkbdgdYk+xe5NXR74h0AWEFO6Ye9ivkC3CitKfOhYQBokCXZeaptENk7SLCzLArlgF0WSRG7uEqdQcpGKU7krW+4jK24n95s82IlAaO2ptP0VNpd4hhIEJw0WViIbPYGBkPgAK7wpgLFJMqg9DO7b/9fUdn5XvAdQbAwPreZ99BNJXGPsoaaUG7My3A0i//yEqeJVSW/6ik57CE5SzTRfd2mrkcyDS+g4xbs7JrVf9BgRv2DGSk5dmSoaHhcarRiFxW26vPYOLsvEDi6hUqsV9aCAQJHOcn3hxFpmCLQBmzChOz/6RRnlwrqD8+mUtSircunWQfn4qsHty3moINa3X3jfdToMPOIxMUybtCEBvXdaCraLaDH2uiafx4zrbPWuMexeJZaX645zaoJEEDnbOn0z1yFivCpcJEFtpILFT2uQAWEDVO6f/xD1jNB2tuH5t2PdPXvzULOILGrE5bCdG09ZM7ewc9wxs6DjxHxf9hHD5FX3lzeq++1sQPWmiKTjg2wzEbEKTqHLU2HxllXpCikqrzr5NZkVqdfQYD63c4lBBDWdzuOnphQnYkk+rV1pHF75my3g9cy5AzuHk9jpd9wOyrDByIV8ZN2OzzVbQ6pkWd/AX9Qs9nAtmvm1ckc39h2SUScasLlG+HKgHke1ANAjZM0n4D+0pycmltgX4ObSnfZ7590wdDcHY4WVXv7A+4icSaetOtbxSXbifMDlCPexKmW50BvAMkAFAjmHBkdK9N1E6uo8L8QpOYJqsL+ppbZKfPc6fTbk1LE3l9tWtsBL98unY2GB2TF0lmLH+muytvigr5Wgl5Hrx8lizaH1j+KV/q1E40ujM5aIbCcpt86Cps8uXrN9C Bt/QKRak 9mdgWbkMI5tWKo6hpZh1mzEm0gENUF6ieXMMwSEri1iONO63fOTlrbELNYhzxopMokvwIB4LC/1vte2H4zsIdy2mprsn0JlFtcOyYj22YC6b/JeCSkXQM/nBKqH0rbGoLqOCnQfo9c9KFrxsYD2OI2iYn55dvbfpeniEYjz031JUdEulA5TzlVO7fBNsVGpNDrc+BzU7fnM9zLUg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Define an arch-specific override of arch_wants_pte_order() so that when FLEXIBLE_THP is enabled, large folios will be allocated for anonymous memory with an order that is compatible with arm64's contpte mappings. Signed-off-by: Ryan Roberts Reviewed-by: Yu Zhao --- arch/arm64/include/asm/pgtable.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 0bd18de9fd97..d00bb26fe28f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1106,6 +1106,12 @@ extern pte_t ptep_modify_prot_start(struct vm_area_struct *vma, extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); + +#define arch_wants_pte_order arch_wants_pte_order +static inline int arch_wants_pte_order(void) +{ + return CONT_PTE_SHIFT - PAGE_SHIFT; +} #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */