From patchwork Mon Jul 3 13:53:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13300152 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D33D1C0015E for ; Mon, 3 Jul 2023 13:53:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5EC858E00BA; Mon, 3 Jul 2023 09:53:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 576CC90000B; Mon, 3 Jul 2023 09:53:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C7A38E00C2; Mon, 3 Jul 2023 09:53:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 12BF58E00BA for ; Mon, 3 Jul 2023 09:53:48 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C7BA8408BB for ; Mon, 3 Jul 2023 13:53:47 +0000 (UTC) X-FDA: 80970443694.20.FFEE7CB Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf13.hostedemail.com (Postfix) with ESMTP id 0D0D820028 for ; Mon, 3 Jul 2023 13:53:45 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf13.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688392426; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oKmEDpPRks/ilB7GL5ZE6RM4bSXQ4CYbCYmtoGtRwVc=; b=d6oeWahn2D1DmT1B8Dg4hCcbkN81yzzBV8yemHt8zY8J+FT4ynVl/PkjNgZYfKKWlXa2tQ ysAYwy4Pk0ICm6oY1gIbB7uCPnybXk02TTS6pMPTWm2W2GMXWs1C78D2xKZOx9R3O1FB1E wCbde6Vi6JVvjZuhe/YhXI05wGl/aaw= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf13.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688392426; a=rsa-sha256; cv=none; b=ozjZZa7/fH6A8CeOyaS/VFYk9WOwnvzqnNpU5kGGLMwXspU5CRTQtNszT3FMIQGm6WYmuX XAeGKlX2WQN+2ZIEskF1fhWzlcm7NAGWqjy3C2k0OM9gZYYGg+eyCrqgPwrLaZL7DWuIC5 1/Ku0YXxBxYcXy4eAOtAJEE0axlBgBk= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 71661C14; Mon, 3 Jul 2023 06:54:27 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EDAF83F73F; Mon, 3 Jul 2023 06:53:42 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 1/5] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap() Date: Mon, 3 Jul 2023 14:53:26 +0100 Message-Id: <20230703135330.1865927-2-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230703135330.1865927-1-ryan.roberts@arm.com> References: <20230703135330.1865927-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0D0D820028 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 7txs33h4nhi6pifed1z8p5dxasxx7uw4 X-HE-Tag: 1688392425-698102 X-HE-Meta: U2FsdGVkX1+ehkYH3XE48vRW1xr08nJhRT9fw8GTmjrz8qazoW8Ui85BPhEIFp/mEjXHpBib3Cp+gqFPQHAUsTClZTf50X+9XGLInHb2Ya3sdz6qNjKc1TQJY8/uxuLdg6QnmvimLfk1FAx6+cgX62ScUgdYsouAwBGM+4CHvOMKdzSJ2G6LusB2LC0aZbM9sJzXaOTNzxPu76YT5jKizoMqp2Lg0oGGtBjV9gSs7g3Yag5E2vOtKpGWzFaAAujphGb89iY6mXhvilAy/KgnDucQpx8dIXLqEDE2CUzTXlGUBodUph6LnAxeVMJXcxraPTerL9zZ763PMROYYR22uScHRuHoNTvQJ+p+iPGHhlBXtmi279avckMfsPhN5GJTLf8rpJ9k+enMFhSRKptVCoxzgfTu61A+RZ1R5CNwim226gpGtud8I5Ewh/H25PZzu9oCCiopb+z2oYPUI5SX87652cvQTwW1jt6kGzJkMHe4+OxTf/KYmqkXJOsvdQi9MAB8IC5tT7B3j5xuuHFwlkzf5/X9PCT1wVsllDgFEOI70bT4DggWkjmxj/1MfIyYLgnVj43PEKlSCvkJsVNXA14lQn2Gj5OvmCyhVev+SFm+jb1ZYH245XdNBxkJcvh4g3VdJerdK81XMx1BhNyvzDa/a9iXOfAEGvRn07/luh+0ECF/pC96ZMKOOBs5JOGrZRqc3myVfCdpE5nKw02nxsBKJgRV0ZziR1Vr3TTbeWWrsE6+M25yEn0y5KPekjE1MU6xtfirubmxRM0DOJidWcestZS1yAUYf8Ro3qw07e+fRQZdMOVwhJ1Ur10QYXWMeInOWOCacCaFK26LzGGMx8YyrCTi5OazVVKHVHhjcpXKajvWc+Xl3ib/FouUfC617jU5ldBK5/dQEXX2tJkxcc8vkLI+grlLVwDBcFJWSblhAKKLzR4QxJAB7RJmgqrjnBSPlj/aAFMyhyjoIny nux3z7C4 CTRaJsP/d4q8ASmFvMy+c1eFQ8zkzVSs1vTl51pCEZu58y24YNskmq4R+A3PkxskASKc8ofh3jtwcDv4s2DM+iXPIox9Uf3sfr3x7wTXtE4g6VXmWB81COE0GWuBljSq5JW4/GbeWxsybcf+tKk03QVmx4w+QeCJE1gEZjw+WCSPJFXvyCHsuX3iOmeIry69ICmNjWqGVp+lvJSs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for FLEXIBLE_THP support, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the "small" pages scheme. Signed-off-by: Ryan Roberts Reviewed-by: Yu Zhao Reviewed-by: Yin, Fengwei --- mm/rmap.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 1d8369549424..82ef5ba363d1 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1278,31 +1278,43 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, * This means the inc-and-test can be bypassed. * The folio does not have to be locked. * - * If the folio is large, it is accounted as a THP. As the folio + * If the folio is pmd-mappable, it is accounted as a THP. As the folio * is new, it's assumed to be mapped exclusively by a single process. */ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address) { - int nr; + int nr = folio_nr_pages(folio); + int i; + struct page *page; - VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma); + VM_BUG_ON_VMA(address < vma->vm_start || + address + (nr << PAGE_SHIFT) > vma->vm_end, vma); __folio_set_swapbacked(folio); - if (likely(!folio_test_pmd_mappable(folio))) { + if (!folio_test_large(folio)) { /* increment count (starts at -1) */ atomic_set(&folio->_mapcount, 0); - nr = 1; + __page_set_anon_rmap(folio, &folio->page, vma, address, 1); + } else if (!folio_test_pmd_mappable(folio)) { + /* increment count (starts at 0) */ + atomic_set(&folio->_nr_pages_mapped, nr); + + page = &folio->page; + for (i = 0; i < nr; i++, page++, address += PAGE_SIZE) { + /* increment count (starts at -1) */ + atomic_set(&page->_mapcount, 0); + __page_set_anon_rmap(folio, page, vma, address, 1); + } } else { /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); atomic_set(&folio->_nr_pages_mapped, COMPOUND_MAPPED); - nr = folio_nr_pages(folio); __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr); + __page_set_anon_rmap(folio, &folio->page, vma, address, 1); } __lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr); - __page_set_anon_rmap(folio, &folio->page, vma, address, 1); } /** From patchwork Mon Jul 3 13:53:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13300153 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 811D8EB64DC for ; Mon, 3 Jul 2023 13:53:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 06E1490000C; Mon, 3 Jul 2023 09:53:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F12B690000B; Mon, 3 Jul 2023 09:53:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D64AE90000C; Mon, 3 Jul 2023 09:53:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BEF3990000B for ; Mon, 3 Jul 2023 09:53:49 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7A82D1C8991 for ; Mon, 3 Jul 2023 13:53:49 +0000 (UTC) X-FDA: 80970443778.07.0F63DEA Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf25.hostedemail.com (Postfix) with ESMTP id A3988A0004 for ; Mon, 3 Jul 2023 13:53:47 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf25.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688392427; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qDlU2rGsKBPfwZSw0bLPzWmjupT8xRy+8rDXmR7/VPM=; b=KER8PTTZT0is1E8H+eWkGO0q4ER8rVajpQIOjDN0z8Yl6c3DW0agX99hgYejVnY1Aafvv8 bdvdZVYqjfWfjR59PhYnH1PrR/L5lrR6xQwqAJPJxhJvRXqyARD+zJ668CQD1BxMg7FhNf vuzuH43IwzcZqFsnWyQ1dIehgQxyHCY= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf25.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688392427; a=rsa-sha256; cv=none; b=0KgSvbZDrxmFao7D1eTrH/ZpgLqlIhyxgUVAiB7CeWIDvLqZLJeFa4KPIhYFpPKrC8mJYp MqdlWsUwUV8OslwGgWgwbd6SQXNeBCJ5mbgULorh7sMAlSORia1nJi8dqbA9rJrInquX+H lf6n5QBeEOoOi9/KqrtEdujlKYGhgjc= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6A42A1424; Mon, 3 Jul 2023 06:54:29 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E6A393F73F; Mon, 3 Jul 2023 06:53:44 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 2/5] mm: Allow deferred splitting of arbitrary large anon folios Date: Mon, 3 Jul 2023 14:53:27 +0100 Message-Id: <20230703135330.1865927-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230703135330.1865927-1-ryan.roberts@arm.com> References: <20230703135330.1865927-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: A3988A0004 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: sm966ht6xraru6fhm94rim5489ugtfjn X-HE-Tag: 1688392427-113007 X-HE-Meta: U2FsdGVkX1+FmGH1wXy5DXOQBvZ3PWqc+q+78NH5rW/dZdOWw/W/0ZKxDDZMbUu3Eu0xddQTiXYS2zmuQTvemHABwzqR1kRWND6HgCRL5IGUbtoHamBLFnT/sqYJw2GX3EasDW8c3hDgwj0vQt+bCw7RgfKef03O8kbwxnKLBn2tNr7QFmkavmWyxOBlzuEKIfOc/dcxVHyxYqQn2zmyyUZh3fozTLehEAl2UQt/gQFPas8XeJ7AwdrAFGjxCtkLcHeWDD20u1nSy9F7dh28M2+4dTntIiomPg6kUXhM+J426RBh3opmfQMlnlsplOdHBcH6Y1uY5KYsi3mlJWaQ5zFREnXrbyF5QVzWafmxuNsXX65P1S7+bUYW96zLiFgrF/4dPJBtowosT2HjpOd+4rZ4SKFVkQvNKtFBUNS3Ys/q0pBDWhHp1T+wKVfPXpGcolm6+7UyAFeZceOpA/IQGfl3ZMspLjWcAvG/qVK7a8bVrAFb4579kGWV+ReZ552DZnvuJZNst3xZhwwbXc+3XzCC8fNcOucoR0qpeUB6kxszlOiDaMqIRqcUxqvk8yMGzSsJEyFoWlIDiZYK9gHjbEQlvgJ1kH4BmhXycxZluyd8HGVhNCmOEh/PUbft9TrzdnhS6dTzOA+GWprhQOyFqgIFf/OYhgnlrceBeI/jWlm6gWnOoIt6EhdNXixUa4lsKXYNRM2w3VwLnk6Qk3rix7L0ryHEZKDfUbAMuH8W8xd4zdH5YuUfVrphycjTP8cvdpAn+i9OzvXfEPm03YgxsK8Azsr5lsg0MusR7yOuLZCphyNV2enJgyHEaRRryaMzDjyIHEiDsTEfWtS8HgBeIvwRygWupHm4yWmuLjoghLP9BrTDDixHB1DRJDsil/6EdZOWwoCq8i2oFraswibYntBdxEheZYL+YVFvJUMYGs8kvZ7088xMbnUfwEqkLLvVuqOSITL9m5KX0dmETri Syl602B2 BtcQruhNmdjG9UCbpMFNBBmeotiwPrTv04qi/7F0HOKTlWArM2V5Y401yyl1Yhw1WKB/i9XvWSC43cONiRpnOupME9ZWSEa835guhOu12Ew1AYMPgjVP8QjkOdY3k1KmTca1re8cGQpD1ySSiTCcjYtOt0AQoKwIZASfVyc6XbB4NCpeFvAbx8CMov2lWvf/AtvfLQgPPUy3mzfI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With the introduction of large folios for anonymous memory, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized. Signed-off-by: Ryan Roberts Reviewed-by: Yu Zhao Reviewed-by: Yin Fengwei --- mm/rmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/rmap.c b/mm/rmap.c index 82ef5ba363d1..bbcb2308a1c5 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1474,7 +1474,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, * page of the folio is unmapped and at least one page * is still mapped. */ - if (folio_test_pmd_mappable(folio) && folio_test_anon(folio)) + if (folio_test_large(folio) && folio_test_anon(folio)) if (!compound || nr < nr_pmdmapped) deferred_split_folio(folio); } From patchwork Mon Jul 3 13:53:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13300154 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6021BEB64DD for ; Mon, 3 Jul 2023 13:53:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DDC0A90000B; Mon, 3 Jul 2023 09:53:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D659F280001; Mon, 3 Jul 2023 09:53:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C064E90000D; Mon, 3 Jul 2023 09:53:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 995A990000B for ; Mon, 3 Jul 2023 09:53:51 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 68D07120910 for ; Mon, 3 Jul 2023 13:53:51 +0000 (UTC) X-FDA: 80970443862.14.69ADBFA Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf02.hostedemail.com (Postfix) with ESMTP id AC80780013 for ; Mon, 3 Jul 2023 13:53:49 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688392429; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=USdbSWQAhhNgz/kbRbFMcj5UNwofuLjMnykBwRvmpy4=; b=CkvvpLWGDwzy3V0C3D4ngVmbdglGrsooTRlb5Mtbold1591QWRaxbGIGGhOrJDzWy49mhq 4C11JqbRF10QGPnk7c8Wxj4v0/hASzChsbIqhMpCKsvt+e4rbZmjCCqyqDhoIHr1glYbf6 u5mZDTG8X5vEDw0RqNnpBGn8ap06Yjo= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688392429; a=rsa-sha256; cv=none; b=TWZ7PajOETh5dkBd0UnskFkBoYx/UK30nzpNN9xyybroxx9r6yxEGbS7K07pSa0B8JH9Cu 5J8eRPOijEiFkQxyjTIqFKUeYGnUiQGo6DJAzXdHBP2wTQtlE56Ys5UBBGy3iGCg99MIlS J57tAB3/gLmlwyo+Ow70K0UyfnhpR4A= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6399A11FB; Mon, 3 Jul 2023 06:54:31 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DF28C3F73F; Mon, 3 Jul 2023 06:53:46 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order() Date: Mon, 3 Jul 2023 14:53:28 +0100 Message-Id: <20230703135330.1865927-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230703135330.1865927-1-ryan.roberts@arm.com> References: <20230703135330.1865927-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: AC80780013 X-Rspam-User: X-Stat-Signature: zuwhdwyjd9fumrsnkn14epak7zphfauf X-Rspamd-Server: rspam01 X-HE-Tag: 1688392429-296835 X-HE-Meta: U2FsdGVkX18YrJdgAgAPtLVNHiJ8wjMCuAwLDdjQFj2t+1ERdlqDZ+6m7FSdORN82WlCPgKYry0TMjurKurvfUhUgX9lx7kLjS+xFi8JaWqlPVhkQy+hAUBX84RLM9Vs3KmtBfCJFEwY0eH9qCE9PFR/UW8RinW1TuTdoVZBlHmCLbehJ+KZFDJXHx4pwbF+arE1Fid4n+L8ULvQ4aB3jrmAK5/GbCAvqLnW5KBacpbWMJcB87ATlXquznWjKsS3u0iCyY/blrIH8HyzAeb1i0Yj1wJBxNqNYwpyg8eiWmgo9y6a96VW/4IVrsQiW8JJ3w55vepH/maR6P/RuYa3p8NHw1PCIt9s0nbbM6t0DLZXxIH3nnBeOqFL5mzRKcCK23hg3RsDKVVZCM83pFarLqUOzGuSP7+j9Uwum34mlcF9GDmI8AA+xIojXV3XQVT6Zg/edIRVeXiYhzX1PmGkuHCJPF2BEJWEi1gwxBFKpXqnGFeELlDV3VTTVKMCRh4QIRLP34+FijPIxAtcEJNeOC3TzL2NcJ7G7b10+qmyOY+wCbWsmAtujfaj0eUJ79EPtR/vvpsSReuxZEA/SWK16gf+ijwokk6EyNwRPQozmTvKGWAfCf4XL00+lDXZPAV6EGJ15ep6qO1zghWFB8T2GUKk1r953gIidy0z9TKe23t9ELqVtsc5/YYoG8j7EsraADpoXZVeevNPWfqwpT1Rui9XpgueOpGSpuTGVRXn5QtTBIkZMoeb34t50lO+5Av8Zl9C/EJEnDSBurQEsEOWYzDjYrd72xgz++U9fo/3aB0q9igAocxVKhKOLMzo0b6MoPOuoHeizFWQZCq8HC5lH1DoEKXE2RB4Y7/MFXYdDYkXTAzZZO0avraXoSnPDc/kBKV3v3TryHQBiu1MYz3/CyZTQHzi8JJbE3MWh5HfUFfIb9v0MRoM2AVHg8tZSbrcBQ4DwUs+Yvb93EKuGFs KSRFWucJ RBmK4Yf+d5G4vA55lac7PugRrvDjRh79rXCcIbOEcdagT5kWv8jBal8zR7JYs5PCOIpE0fPQ9daKU9z5Z4ji1AJYSL5dn470/s+PwK6/s3eEv5ApuFaSZTs6MqHimL7mS9IiSAGnXTUVqm6xhVWgtoWgy4PV4Q+IqnG1eIPGaNGPkPowrXgxxqfSFCUUg89ae2ToxRXoxVWYfcveGZIqQ00X6cT4Smnrl7jy23zNwEsn0/v0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: arch_wants_pte_order() can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous. The first user for this hint will be FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs. Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K. Signed-off-by: Ryan Roberts Reviewed-by: Yu Zhao --- include/linux/pgtable.h | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index a661a17173fa..f7e38598f20b 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -13,6 +13,7 @@ #include #include #include +#include #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) - \ defined(__PAGETABLE_PMD_FOLDED) != CONFIG_PGTABLE_LEVELS @@ -336,6 +337,18 @@ static inline bool arch_has_hw_pte_young(void) } #endif +#ifndef arch_wants_pte_order +/* + * Returns preferred folio order for pte-mapped memory. Must be in range [0, + * PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios + * to be at least order-2. + */ +static inline int arch_wants_pte_order(struct vm_area_struct *vma) +{ + return ilog2(SZ_64K >> PAGE_SHIFT); +} +#endif + #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address, From patchwork Mon Jul 3 13:53:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13300155 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65610EB64DC for ; Mon, 3 Jul 2023 13:53:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EEE26280002; Mon, 3 Jul 2023 09:53:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E7721280001; Mon, 3 Jul 2023 09:53:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D1705280002; Mon, 3 Jul 2023 09:53:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BB316280001 for ; Mon, 3 Jul 2023 09:53:53 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8EE7180426 for ; Mon, 3 Jul 2023 13:53:53 +0000 (UTC) X-FDA: 80970443946.24.62F4863 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf22.hostedemail.com (Postfix) with ESMTP id A5005C0022 for ; Mon, 3 Jul 2023 13:53:51 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf22.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688392431; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=M42aKf0Dh3v26rYqL7fJB4T37PH00qdaAFo246Anczo=; b=AOL/EjvlY+iNkBiOLmmd8tXrDz0YxinwwjHyajo0S9nAUCrN9WWP6Zkcca+3uzeLhFmUaK 2QvSM7UsZYuhK2npvKGQMmfUL1gZKRCekeAIJ0SaHAKrWAkXU04y+1KJxGRL/Q4CxECbat 2UpCWIeDnEKsZvgmj69u7BLa7QeP3WI= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf22.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688392431; a=rsa-sha256; cv=none; b=Oo8ITAXib8QJVUzR60g1skKd4CiZc9P2GDvoyBtcfl+KwpbfHNOfQv9R7vs0R0XaL1YklS IULof2BdcJQeRtUVkOfhO50crQ1M+vpRb8cPdo2uSNc3RaokW7/YVUnByXwgrtqqx6Ol9z IsMYeK5+y6exku1OyoXAlMlZsFo5j1g= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 76F33143D; Mon, 3 Jul 2023 06:54:33 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D88B03F73F; Mon, 3 Jul 2023 06:53:48 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance Date: Mon, 3 Jul 2023 14:53:29 +0100 Message-Id: <20230703135330.1865927-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230703135330.1865927-1-ryan.roberts@arm.com> References: <20230703135330.1865927-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: A5005C0022 X-Stat-Signature: rp3bjioykout9tgtgnk9qj8zr85u4914 X-HE-Tag: 1688392431-616853 X-HE-Meta: U2FsdGVkX1/2Qor9mGAftAmq/CLkREmLSOUEfHgnXxCY0reweXT/s41IVtOEaZafq9Wra5L0dPdGFzEhOCptmIHTPwUSw4GP7fxAvnbidHmnMgHyHnRuON49cdmSq6tigOmelxMoODutDlZJ/9CWFJ0g9OxfC/9NfjCbOIKo6tDP6NcspKgyIuDS2oWcQy8P2QpMW0WGN64JLUSH1SgGUMkcItz95Fg0D8EkkG1TvxF6IFNCEXRGpMceUhNIeD7cWLHu60hMCPFW+GNsA+r8vH/UK6QPlhvYIJ8+ewtCUE1VbCMGt4O6m0wGbU8JX3vMIxqePnlhd9WAWwJboZTF0zKPMkhc+Y2/iUZvuMIfnQR/IxhGTZqW+/KEg9zKDBgPyN3iGe0PMxWP3CroGxx0cHi1g96G7XY4WOTSmdee2SxWoex+COw1RatPq+NaruKsoT7pAjM/w4HqTvxqD5h2XcV0LeVshe6ejWQJ6ysV2KaYVQ2Myqv6H/4jgC2E+SrXQKrtZX3lTD49mOB68qQQVqb3Mqfo6HS05NkBAEZ/OT7FXmjMqgF+EPCTE/BLIDtiyqaiGrq8NG74FrknWZzienFZmyzrG/0g7dDDU1Hso3Yi97Bf4PspeileGGdX5Yjs/e1ZyFDlHyG7rpVo/Oif2d0u/KrodAGLuEJp/u6ri79UHIwvzrmZIpLbxT5bsd8HyfTRhfydLsZyCrc0WjTbB2QiUBuYac2+NINRNZhE8oaUt7WLy6xWVrKmojAm8I96hlObLmVTCTLlQCoW/p0yNerW7izUPUsXjn9Cx46Ino6BPiLiLscgtbc96M4KmcDC1/j/7QXwI8gLP8p6VMycf9gsynRD7VXZ+JkauRioQSZSiKsEUOumNjd9Z0Htg/rAWP5r6xhpzkEoJEZmZ+uuZTemIFMSz3yWJWmNOD8+H4LE2k17vYigGQjbBwrGHIKnuNHuXu2J48CrJ6/EcUC RKTIRx2u Ft5hwTXFOt03pfmu1TB07eBdlgLPKJJdqVIWcpMNY6GBhqLhG46tvEd8gkErURW9lfZN0iZxQn8j4505uFq5isIbKF6w7GHYN0T0rEd0csXoilbXMeom40Wg27RHHI/WEFCCXtQ1BBhj5lyMOD5dmMC99Y0yGD9uLt09e6TdS2ckUycnvDCKPrxIOtXGsABnqdl5wINaaeGOSE4E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce FLEXIBLE_THP feature, which allows anonymous memory to be allocated in large folios of a specified order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio. The new behaviour is hidden behind the new FLEXIBLE_THP Kconfig, which defaults to disabled for now; there is a long list of todos to make FLEXIBLE_THP robust with existing features (e.g. compaction, mlock, some madvise ops, etc). These items will be tackled in subsequent patches. When enabled, the preferred folio order is as returned by arch_wants_pte_order(), which may be overridden by the arch as it sees fit. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required. If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order. Signed-off-by: Ryan Roberts --- mm/Kconfig | 10 ++++ mm/memory.c | 168 ++++++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 165 insertions(+), 13 deletions(-) diff --git a/mm/Kconfig b/mm/Kconfig index 7672a22647b4..1c06b2c0a24e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -822,6 +822,16 @@ config READ_ONLY_THP_FOR_FS support of file THPs will be developed in the next few release cycles. +config FLEXIBLE_THP + bool "Flexible order THP" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even if the order of the folio is smaller than the PMD + order. This reduces the number of page faults, as well as other + per-page overheads to improve performance for many workloads. + endif # TRANSPARENT_HUGEPAGE # diff --git a/mm/memory.c b/mm/memory.c index fb30f7523550..abe2ea94f3f5 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3001,6 +3001,116 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) return 0; } +#ifdef CONFIG_FLEXIBLE_THP +/* + * Allocates, zeros and returns a folio of the requested order for use as + * anonymous memory. + */ +static struct folio *alloc_anon_folio(struct vm_area_struct *vma, + unsigned long addr, int order) +{ + gfp_t gfp; + struct folio *folio; + + if (order == 0) + return vma_alloc_zeroed_movable_folio(vma, addr); + + gfp = vma_thp_gfp_mask(vma); + folio = vma_alloc_folio(gfp, order, vma, addr, true); + if (folio) + clear_huge_page(&folio->page, addr, folio_nr_pages(folio)); + + return folio; +} + +/* + * Preferred folio order to allocate for anonymous memory. + */ +#define max_anon_folio_order(vma) arch_wants_pte_order(vma) +#else +#define alloc_anon_folio(vma, addr, order) \ + vma_alloc_zeroed_movable_folio(vma, addr) +#define max_anon_folio_order(vma) 0 +#endif + +/* + * Returns index of first pte that is not none, or nr if all are none. + */ +static inline int check_ptes_none(pte_t *pte, int nr) +{ + int i; + + for (i = 0; i < nr; i++) { + if (!pte_none(ptep_get(pte++))) + return i; + } + + return nr; +} + +static int calc_anon_folio_order_alloc(struct vm_fault *vmf, int order) +{ + /* + * The aim here is to determine what size of folio we should allocate + * for this fault. Factors include: + * - Order must not be higher than `order` upon entry + * - Folio must be naturally aligned within VA space + * - Folio must be fully contained inside one pmd entry + * - Folio must not breach boundaries of vma + * - Folio must not overlap any non-none ptes + * + * Additionally, we do not allow order-1 since this breaks assumptions + * elsewhere in the mm; THP pages must be at least order-2 (since they + * store state up to the 3rd struct page subpage), and these pages must + * be THP in order to correctly use pre-existing THP infrastructure such + * as folio_split(). + * + * Note that the caller may or may not choose to lock the pte. If + * unlocked, the result is racy and the user must re-check any overlap + * with non-none ptes under the lock. + */ + + struct vm_area_struct *vma = vmf->vma; + int nr; + unsigned long addr; + pte_t *pte; + pte_t *first_set = NULL; + int ret; + + order = min(order, PMD_SHIFT - PAGE_SHIFT); + + for (; order > 1; order--) { + nr = 1 << order; + addr = ALIGN_DOWN(vmf->address, nr << PAGE_SHIFT); + pte = vmf->pte - ((vmf->address - addr) >> PAGE_SHIFT); + + /* Check vma bounds. */ + if (addr < vma->vm_start || + addr + (nr << PAGE_SHIFT) > vma->vm_end) + continue; + + /* Ptes covered by order already known to be none. */ + if (pte + nr <= first_set) + break; + + /* Already found set pte in range covered by order. */ + if (pte <= first_set) + continue; + + /* Need to check if all the ptes are none. */ + ret = check_ptes_none(pte, nr); + if (ret == nr) + break; + + first_set = pte + ret; + } + + if (order == 1) + order = 0; + + return order; +} + /* * Handle write page faults for pages that can be reused in the current vma * @@ -3073,7 +3183,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) goto oom; if (is_zero_pfn(pte_pfn(vmf->orig_pte))) { - new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); + new_folio = alloc_anon_folio(vma, vmf->address, 0); if (!new_folio) goto oom; } else { @@ -4040,6 +4150,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) struct folio *folio; vm_fault_t ret = 0; pte_t entry; + int order; + int pgcount; + unsigned long addr; /* File mapping without ->vm_ops ? */ if (vma->vm_flags & VM_SHARED) @@ -4081,24 +4194,51 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); return handle_userfault(vmf, VM_UFFD_MISSING); } - goto setpte; + if (uffd_wp) + entry = pte_mkuffd_wp(entry); + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); + + /* No need to invalidate - it was non-present before */ + update_mmu_cache(vma, vmf->address, vmf->pte); + goto unlock; + } + + /* + * If allocating a large folio, determine the biggest suitable order for + * the VMA (e.g. it must not exceed the VMA's bounds, it must not + * overlap with any populated PTEs, etc). We are not under the ptl here + * so we will need to re-check that we are not overlapping any populated + * PTEs once we have the lock. + */ + order = uffd_wp ? 0 : max_anon_folio_order(vma); + if (order > 0) { + vmf->pte = pte_offset_map(vmf->pmd, vmf->address); + order = calc_anon_folio_order_alloc(vmf, order); + pte_unmap(vmf->pte); } - /* Allocate our own private page. */ + /* Allocate our own private folio. */ if (unlikely(anon_vma_prepare(vma))) goto oom; - folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); + folio = alloc_anon_folio(vma, vmf->address, order); + if (!folio && order > 0) { + order = 0; + folio = alloc_anon_folio(vma, vmf->address, order); + } if (!folio) goto oom; + pgcount = 1 << order; + addr = ALIGN_DOWN(vmf->address, pgcount << PAGE_SHIFT); + if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL)) goto oom_free_page; folio_throttle_swaprate(folio, GFP_KERNEL); /* * The memory barrier inside __folio_mark_uptodate makes sure that - * preceding stores to the page contents become visible before - * the set_pte_at() write. + * preceding stores to the folio contents become visible before + * the set_ptes() write. */ __folio_mark_uptodate(folio); @@ -4107,11 +4247,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl); if (vmf_pte_changed(vmf)) { update_mmu_tlb(vma, vmf->address, vmf->pte); goto release; + } else if (order > 0 && check_ptes_none(vmf->pte, pgcount) != pgcount) { + goto release; } ret = check_stable_address_space(vma->vm_mm); @@ -4125,16 +4266,17 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) return handle_userfault(vmf, VM_UFFD_MISSING); } - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_ref_add(folio, pgcount - 1); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, pgcount); + folio_add_new_anon_rmap(folio, vma, addr); folio_add_lru_vma(folio, vma); -setpte: + if (uffd_wp) entry = pte_mkuffd_wp(entry); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); + set_ptes(vma->vm_mm, addr, vmf->pte, entry, pgcount); /* No need to invalidate - it was non-present before */ - update_mmu_cache(vma, vmf->address, vmf->pte); + update_mmu_cache_range(vma, addr, vmf->pte, pgcount); unlock: pte_unmap_unlock(vmf->pte, vmf->ptl); return ret; From patchwork Mon Jul 3 13:53:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13300156 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2B8BC001B1 for ; Mon, 3 Jul 2023 13:53:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F155A280003; Mon, 3 Jul 2023 09:53:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E9E10280001; Mon, 3 Jul 2023 09:53:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF028280003; Mon, 3 Jul 2023 09:53:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B13CF280001 for ; Mon, 3 Jul 2023 09:53:55 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 882311C8987 for ; Mon, 3 Jul 2023 13:53:55 +0000 (UTC) X-FDA: 80970444030.05.C99B865 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf12.hostedemail.com (Postfix) with ESMTP id BA3B64000B for ; Mon, 3 Jul 2023 13:53:53 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688392433; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0Xs0/r8JxC0IODzuzWxhN6nl3kLwEjLKvoCF58989ko=; b=Y+qtV42ke3CNIvOUjQkyKSkjCl80PvuAXL0DqSTd8ySX8glq3y1sgKDSi3gICCQzaFjJ+5 XaWUZMquNLJZRR1ud63x9AbwrXICwhQ6CHw3Ga/y9hKvbqO7Qe48Gi9k2RmkCIlSNiLDmK PFsm1CGx7FplKJz2GEzt/JI0N444s34= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688392433; a=rsa-sha256; cv=none; b=0J37pGjpEpD3c78dJfrqu0OvLSixRF811V+dJ/6x9fUgLbtkS1MvopKZ1VvkThJ4SoB6oT hjpQ3AYg8I+3ZU9HhBflXvAI6JYqj8QPtu3X24xQ59kDufcJt3clYfCZ+IJExeHFYKUSGR an/xe3uUhmSj4q+obyfd6MPACQnLF18= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6FD552F4; Mon, 3 Jul 2023 06:54:35 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EBD643F73F; Mon, 3 Jul 2023 06:53:50 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 5/5] arm64: mm: Override arch_wants_pte_order() Date: Mon, 3 Jul 2023 14:53:30 +0100 Message-Id: <20230703135330.1865927-6-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230703135330.1865927-1-ryan.roberts@arm.com> References: <20230703135330.1865927-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Stat-Signature: dfnuxq4yr9u5tx4dk34woca9sjsrxz4j X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: BA3B64000B X-Rspam-User: X-HE-Tag: 1688392433-997186 X-HE-Meta: U2FsdGVkX19IMcged2qWwfax5vaTnZ/lBcWGqvhJmjJQ4y8G+D5oMM59wfGhrGf1TEjZfThvqzfPeglrrLZJw5wcy3tpqQeuEaqo4Spmy/LkbVTKQlKQWp7HewJRsSfSxVqdaR3I0k5N3y3ilabr157OmOBUXKn+bela+Vt0E78H97RQDZmWJNuj/hHsc7Y34HyBizMGpnGT30XnC9mn+c+md3bNZjDGlrtamRBaqXEoOclF4J8YZ22YoR5taYGjcg/CwEjhCQUp62aewyFntN1HZg4qW1uGN8V8LpkjgPogO6nA3F5bWqskKK+EwcU8FCwJVdoLUQhy61fkJBYo1VP2ztvoHqJKtJB8IhQjjjIXOCu0i6Qz+ABhBXNhdtPbpMdClLtERV3E/EXwz8VbHYkl2lYhCshOjV9vAKyyBrpBmUpB12tCapIEwXrFAPgWD1HmM+UXbf+L8GTI2aMVlb1C9OO/h0koWv3FJ/E9W5IYhVPma2fSifJjkA1YL9U9EDXrO0oPGSlwZ2Pz2yMc6xAHOcM0Pv0zdihywdvkWaUB09ejobr6+D0Tdohmspvutbs21FkV+GdtrkA9GNQOjCPPj1faDaG3/mro4hl9fBo1htAQ1Cp9iwswDc6W0ORQRgBWntRpbOXtzPPNGKnEVJro02yA34+xBz//mipbyUf3k/fQP9+kVelDIpodXM9hVmhu9G7OmPgPrJ6njQU2ba0eqqWM9HdZn4e5XVrq0HD9puNqesFgzpFU2bvza348manEJoVFQrmZURlge0Bt7hXQ48EVhxWDmnYQve5PioZWcDKE7QHE94clphwSgJ0mo9DKqasGOmCP8gKIIAmq9DmPTjxrnsUJRqhEiQ14xlzMSXTfosbeQxzrNOzUU7ENjQVqKpaDOiFEzMWO0Po8q2jnovkD84obB36XEOfzg9O+YRqCHvsPJSDsfyh/a3KMyeOJWIC1TH9+xggRjN1 1X8yAaY7 tpOg8WrJmTcPKlfDLDSM77+O7N1w5g7k/5IesFgstK2JY0n7DjFupPsNloihP3G3iUArG68qFYYzyYATEfGuRarijJIZFCRoEn5geQy28VUCu0uBj8l8soXhKM6ApyYZ648R3YwqMpIngTWGte5gTjKLD/ljleoHR6iQPMouoOOzw7sXzoPkG3QgNqIPYtC5xK12sayZEZT3NUck= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Define an arch-specific override of arch_wants_pte_order() so that when FLEXIBLE_THP is enabled, large folios will be allocated for anonymous memory with an order that is compatible with arm64's contpte mappings. arch_wants_pte_order() returns an order according to the following policy: For the unhinted case, when THP is not requested for the vma, don't allow anything bigger than 64K. This means we don't waste too much memory. Additionally, for 4K pages this is the contpte size, and for 16K, this is (usually) the HPA size when the uarch feature is implemented. For the hinted case, when THP is requested for the vma, allow the contpte size for all page size configurations; 64K for 4K, 2M for 16K and 2M for 64K. Additionally, the THP and NOTHP order constants are defined using Kconfig so it is possible to override them at build time. Signed-off-by: Ryan Roberts --- arch/arm64/Kconfig | 12 ++++++++++++ arch/arm64/include/asm/pgtable.h | 4 ++++ arch/arm64/mm/mmu.c | 8 ++++++++ 3 files changed, 24 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 343e1e1cae10..689c5bf13dc1 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -281,6 +281,18 @@ config ARM64_CONT_PMD_SHIFT default 5 if ARM64_16K_PAGES default 4 +config ARM64_PTE_ORDER_NOTHP + int + default 0 if ARM64_64K_PAGES # 64K (1 page) + default 2 if ARM64_16K_PAGES # 64K (4 pages; benefits from HPA where HW supports it) + default 4 if ARM64_4K_PAGES # 64K (16 pages; eligible for contpte-mapping) + +config ARM64_PTE_ORDER_THP + int + default 5 if ARM64_64K_PAGES # 2M (32 pages; eligible for contpte-mapping) + default 7 if ARM64_16K_PAGES # 2M (128 pages; eligible for contpte-mapping) + default 4 if ARM64_4K_PAGES # 64K (16 pages; eligible for contpte-mapping) + config ARCH_MMAP_RND_BITS_MIN default 14 if ARM64_64K_PAGES default 16 if ARM64_16K_PAGES diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 6fd012663a01..8463d5f9f307 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1117,6 +1117,10 @@ extern pte_t ptep_modify_prot_start(struct vm_area_struct *vma, extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); + +#define arch_wants_pte_order arch_wants_pte_order +extern int arch_wants_pte_order(struct vm_area_struct *vma); + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index af6bc8403ee4..8556c4a9b507 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1481,3 +1481,11 @@ void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte { set_pte_at(vma->vm_mm, addr, ptep, pte); } + +int arch_wants_pte_order(struct vm_area_struct *vma) +{ + if (hugepage_vma_check(vma, vma->vm_flags, false, true, true)) + return CONFIG_ARM64_PTE_ORDER_THP; + else + return CONFIG_ARM64_PTE_ORDER_NOTHP; +}