From patchwork Thu Jun 22 14:41:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13289249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 799E9EB64D8 for ; Thu, 22 Jun 2023 14:42:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EB55D8D0005; Thu, 22 Jun 2023 10:42:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E3FA48D0001; Thu, 22 Jun 2023 10:42:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CB8288D0005; Thu, 22 Jun 2023 10:42:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BA00E8D0001 for ; Thu, 22 Jun 2023 10:42:34 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7F5481601B1 for ; Thu, 22 Jun 2023 14:42:34 +0000 (UTC) X-FDA: 80930649828.06.A4E57CB Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf01.hostedemail.com (Postfix) with ESMTP id CEF8D40002 for ; Thu, 22 Jun 2023 14:42:31 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687444952; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0wmgZ1gAShkaExAhWff4a7KZ8yvQfuwLze+OBa4plIM=; b=lVozCjkJJVr7+xT21Ox5T7hnnM0AkhNZpj484B20nXqHuCUt/nDhIlYFeDekAKNGCwuA0y Oq0YIAPCSMXIDzsZIKfYDJHuiq/8Dc8y5rjCP/tN5LqOlhfTE6AKx+pekJMBN3nuy660L5 TZDHToWCKELlFP10AFeReIFEUVV6yCU= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687444952; a=rsa-sha256; cv=none; b=gOMeBL4IKlWmZcRHkO9+Thog1Yjr6YG4zIhjDdYRlOb9DE7kszu7QOixS91W6Lq8aWB950 7qAvAzNU5g/j4RX9CLH7tul+tQbc5pCA3QBXlAYIOtI+p9D+LzY5OHmjjTdjvS0029GGJs mic06sBHao6GKjP4nWH5f5am2fjvDMI= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 927601042; Thu, 22 Jun 2023 07:43:14 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0E8A43F663; Thu, 22 Jun 2023 07:42:27 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 01/14] arm64/mm: set_pte(): New layer to manage contig bit Date: Thu, 22 Jun 2023 15:41:56 +0100 Message-Id: <20230622144210.2623299-2-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230622144210.2623299-1-ryan.roberts@arm.com> References: <20230622144210.2623299-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: CEF8D40002 X-Rspam-User: X-Stat-Signature: 8xefqu83qqi1mek53aqqf65iz5nra3to X-Rspamd-Server: rspam01 X-HE-Tag: 1687444951-175093 X-HE-Meta: U2FsdGVkX1/a5SF/hQhgaDgRYy9N9FngOLpCAIDglvaiRm6fJgw0myvYoK6/kB26/w6ECQOCpJp8Fv1w48JPPvCCL/MVO5gDRLttpZ5Z1tLo4ZeAUJkcBjHrBZVd6RVHOdyPHgBBoL5L9u0m6GXRs7UY1RiIbuBP+v/ufcZExd0Yq2JBvqi94paGvYih2imJYOAcLglhPSIltON7/5NC6gI9oGExuYlj2x1mF+LuY1l5Nr2yWqlTLLvu1Dd2wC48lbj5D7FaFFiLYYEawmHXdd00ap56kxxQMT05YiZwdfIPYK8Z86iSNaGh4wA2ytImDwaBg2yNKh/CfAK5G/rFNxZCGuGRL9ITee9304yztVgHLwB6B/Q3mm/1lIcLZu0YijZVYrhz4WJbx4ANfBa8sqN3VOZNuE02PkkHjxfN18h+sQJVPyp9O345jgnDU+Z9oTfYvPopBOfDSb0SGCbYJtISPc28ctBse6JfDnvAFo3IBdAetLKvRDEB5+AAgcyiNoRHh9wuRp4HOfw4uDrgkv+YgKusMmuF0o4LQiNMJ4PVt7slf3uqYCyWP/kRdgqOFxaQba2tmhLZx8080BHxEWaVj2IXRgui0fLDuUMZCf7GZGQXMlGR+XTCxYByN1DLkgWUOyjsQBULvthUlkMCXn2bIJxt6Wy7P1hAiPvVF4ABesoHD9UIJyZf8mWez4V8IaPXQUJ/vxR6JYgVxAOtLdmC+AgppYgkiG+K/fwRyVRlkIVktZsOjkKo5f7znVtwZZ8onK7f3w82zuVXzvswSC5UEcnpNImXOY1fbXegymH2IC2MGzK/34nbP7pQ6yAEHjGAUrOqPkG9FKm+FtddMI7QZfePChntpGd/gn7niZL9zsxL20gigrpElbK6hvijr/YIweshO3xzPhQqpwiiQljuVKC0uyputwESukc7aQR8XAPfMJJ6zYYiyD7+gfkPX24yepjLFubwztwaBh1 qJ/OB0Sb Ms+5EWAXYyrtQo/gIaJg3sWZvLEl7OLSsLKyu5W6xxEmZ8WIvZhUGM5cKDV2l7gow+yNq/GLoo4gq/2WuE2M38sWMzenKqT4H+7MFi+puPP0dfIyUF9Br/nvX9N1v/W0lmdKsu/n1KGGl04Doj6FNnqcLRwROIV7zd6aNaOz1TbHtfWsPP80tOQDIZitjrOx3ZHIV9bJgq8YtK65nFMpLxdpyFhfq9W+cysdW0upXDxISWIzeYTzLqQbuHitm4ly5mYVVDtqv7+StST3H+Hkm36ibZw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. No behavioural changes intended. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 23 ++++++++++++++++++++--- arch/arm64/kernel/efi.c | 2 +- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/kasan_init.c | 4 ++-- arch/arm64/mm/mmu.c | 2 +- arch/arm64/mm/pageattr.c | 2 +- arch/arm64/mm/trans_pgd.c | 4 ++-- 7 files changed, 28 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 6fd012663a01..7f5ce5687466 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -93,7 +93,8 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) #define pte_none(pte) (!pte_val(pte)) -#define pte_clear(mm,addr,ptep) set_pte(ptep, __pte(0)) +#define pte_clear(mm, addr, ptep) \ + __set_pte(ptep, __pte(0)) #define pte_page(pte) (pfn_to_page(pte_pfn(pte))) /* @@ -260,7 +261,7 @@ static inline pte_t pte_mkdevmap(pte_t pte) return set_pte_bit(pte, __pgprot(PTE_DEVMAP | PTE_SPECIAL)); } -static inline void set_pte(pte_t *ptep, pte_t pte) +static inline void __set_pte(pte_t *ptep, pte_t pte) { WRITE_ONCE(*ptep, pte); @@ -352,7 +353,7 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr, __check_safe_pte_update(mm, ptep, pte); - set_pte(ptep, pte); + __set_pte(ptep, pte); } static inline void set_ptes(struct mm_struct *mm, unsigned long addr, @@ -1117,6 +1118,22 @@ extern pte_t ptep_modify_prot_start(struct vm_area_struct *vma, extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); + +/* + * The below functions constitute the public API that arm64 presents to the + * core-mm to manipulate PTE entries within the their page tables (or at least + * this is the subset of the API that arm64 needs to implement). These public + * versions will automatically and transparently apply the contiguous bit where + * it makes sense to do so. Therefore any users that are contig-aware (e.g. + * hugetlb, kernel mapper) should NOT use these APIs, but instead use the + * private versions, which are prefixed with double underscore. + */ + +static inline void set_pte(pte_t *ptep, pte_t pte) +{ + __set_pte(ptep, pte); +} + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index baab8dd3ead3..7a28b6a08a82 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -115,7 +115,7 @@ static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data) else if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL) && system_supports_bti() && spd->has_bti) pte = set_pte_bit(pte, __pgprot(PTE_GP)); - set_pte(ptep, pte); + __set_pte(ptep, pte); return 0; } diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index c0a3301203bd..51cd4501816d 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -121,7 +121,7 @@ void __set_fixmap(enum fixed_addresses idx, ptep = fixmap_pte(addr); if (pgprot_val(flags)) { - set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, flags)); + __set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, flags)); } else { pte_clear(&init_mm, addr, ptep); flush_tlb_kernel_range(addr, addr+PAGE_SIZE); diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index e969e68de005..40125b217195 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -112,7 +112,7 @@ static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, if (!early) memset(__va(page_phys), KASAN_SHADOW_INIT, PAGE_SIZE); next = addr + PAGE_SIZE; - set_pte(ptep, pfn_pte(__phys_to_pfn(page_phys), PAGE_KERNEL)); + __set_pte(ptep, pfn_pte(__phys_to_pfn(page_phys), PAGE_KERNEL)); } while (ptep++, addr = next, addr != end && pte_none(READ_ONCE(*ptep))); } @@ -275,7 +275,7 @@ static void __init kasan_init_shadow(void) * so we should make sure that it maps the zero page read-only. */ for (i = 0; i < PTRS_PER_PTE; i++) - set_pte(&kasan_early_shadow_pte[i], + __set_pte(&kasan_early_shadow_pte[i], pfn_pte(sym_to_pfn(kasan_early_shadow_page), PAGE_KERNEL_RO)); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index af6bc8403ee4..c84dc87d08b9 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -178,7 +178,7 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, do { pte_t old_pte = READ_ONCE(*ptep); - set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); + __set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); /* * After the PTE entry has been populated once, we diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 8e2017ba5f1b..057097acf9e0 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -41,7 +41,7 @@ static int change_page_range(pte_t *ptep, unsigned long addr, void *data) pte = clear_pte_bit(pte, cdata->clear_mask); pte = set_pte_bit(pte, cdata->set_mask); - set_pte(ptep, pte); + __set_pte(ptep, pte); return 0; } diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index 4ea2eefbc053..f9997b226614 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -40,7 +40,7 @@ static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) * read only (code, rodata). Clear the RDONLY bit from * the temporary mappings we use during restore. */ - set_pte(dst_ptep, pte_mkwrite(pte)); + __set_pte(dst_ptep, pte_mkwrite(pte)); } else if (debug_pagealloc_enabled() && !pte_none(pte)) { /* * debug_pagealloc will removed the PTE_VALID bit if @@ -53,7 +53,7 @@ static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) */ BUG_ON(!pfn_valid(pte_pfn(pte))); - set_pte(dst_ptep, pte_mkpresent(pte_mkwrite(pte))); + __set_pte(dst_ptep, pte_mkpresent(pte_mkwrite(pte))); } } From patchwork Thu Jun 22 14:41:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13289250 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2994BEB64DB for ; Thu, 22 Jun 2023 14:42:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B83408D0006; Thu, 22 Jun 2023 10:42:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B31C18D0001; Thu, 22 Jun 2023 10:42:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 89A3D8D0006; Thu, 22 Jun 2023 10:42:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 771A28D0001 for ; Thu, 22 Jun 2023 10:42:36 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 519C01C763E for ; Thu, 22 Jun 2023 14:42:36 +0000 (UTC) X-FDA: 80930649912.17.B59FBAB Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf15.hostedemail.com (Postfix) with ESMTP id 87A78A0004 for ; Thu, 22 Jun 2023 14:42:34 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; spf=pass (imf15.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687444954; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fYvhdkX/ZWLQAoUTwjzI4mzSzxcxmlI7JPHyvYv9B00=; b=wth4ZqebjR1E3NKSGQSRDBpN1vPzDvF/5zFeg1T71igFqrqxwQb5vU1SgqiBKMayGAs+3e 1fQofLX3optTI7oprdItYGsYsKXCnDJ21TVfMigI5cZbBVqMsLUTRbvgN0wWWOQcSRcPEo UVYbpMzEMEZRDXrQiSPA/LzG7LVPL60= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687444954; a=rsa-sha256; cv=none; b=0RrNPntf6U//9Fs5ZCCg2SLt5iMArKH4U72uuVunTEJs9T6Agq12PjjksJeClS1bXe3vuX LmwxXNBUAfXF/kE41+ZMsXXFgJ21/SDaxGR7yjnIoqFKWpCYsOWCUgFMLkwzRo5o/kEQbe l0qg3PDexoergdpkRGs89TO64jjr6r8= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; spf=pass (imf15.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6CB6C113E; Thu, 22 Jun 2023 07:43:17 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DC8543F663; Thu, 22 Jun 2023 07:42:30 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 02/14] arm64/mm: set_ptes()/set_pte_at(): New layer to manage contig bit Date: Thu, 22 Jun 2023 15:41:57 +0100 Message-Id: <20230622144210.2623299-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230622144210.2623299-1-ryan.roberts@arm.com> References: <20230622144210.2623299-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Stat-Signature: 7ne97e3p6qr65u88of1b358mzfdamjaw X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 87A78A0004 X-Rspam-User: X-HE-Tag: 1687444954-455576 X-HE-Meta: U2FsdGVkX19TiFOsNGZfasnE64xip4Mq7DFkTY/rL2zqavclOscPSljCnRKi0CvldqPfUhrdWCBthZz/us6mVAHC3G6iikwcoYaWphVYeNhNt/M+dqlc73C2D0FqTOE5M4TKpDWomf8czDCmHxdu150Z250aeK4rlbvSLuYuF+408oC6YXXEb5cju3Rqhw3MIyo8GvfosASJ1SodkVU98GtypEXvtwg7CekieWA7KdI503Vmhn7QV93eSX1dFVgRJ80zyWSnPotzs6MjSvZpJacteVC6X3T755EpfD7LClY8f3bAKS4PSMCbfBNwVvP4fkMo0Zr+P4AraDbmaYGZ6g9gKRvTz/yoxiwdJuZXnfd4alxn38ahrQ0v2HzLRJaMRil1yL5T6/Jm98KQ+8Fx3pFNbGVNZsWoUWqnqDrPM0gMqLyjt1up2KVBnsA18YZs5LllNyJ4sCnjlki//4MJ9TadF8OK9K9Uv8is1cVZY0qTDjmdbX6D92nFoJ4x1mncYD3ldOchfoaqmedMw5ZxOBd+/NZVxLGbfzIj0+pjKf66HUczJvzdMnOoxVcR0mjKjWCplxgyHhFb2cOW3vAL7r5edQOT7NHcD9acKxuVvJt2tA1B+pOF9dd3KSCb4tyy9oJLqv35e1AqxPUuNYo+EHl+v5ANYBExJzWNq+E+fwpNHQD6RbAHo3kaF2DLzTKwlFeKNuP4mrjDW7mIOIYbhRql89ihW15nobgvWOM9ehKyFvXS9DoVn1T6OBS6YyWTZ9FgXnUj7Q1kNyh8QXRVCfONFqY7r5bm3CWC0JM+/7IWWKKBLkVpua+LoPxdh6qae2ea9pEi4Uyb1llSB63TSgTOP8KNfGGGwR6+OLeL+32uDPsM9GwV0+CkW9dg+h3ECg3q7lZW9+FDlAGjqnfEjSu+vTX7qgkUxX+tLURygsiQTWRGZ7aE8FmgKgFokXZ/E6Gkkmj9VahkEZWuNNu MRv6s4Px oknsPlrW2Ay2in9hVCUHgIWE4/pKxJ4RGc4wQSfEawQIe+ddw1g6XI8mlSJ3O4GxLgPwMpBlOamjAx0tpZB6oKS2yRa2RWC4B9UwcUK43B1cIDoW1Epc9wd9mlJbdekfH9wjz4avtxpT1/y1I2t8qc1r1F9T3OlrANVd+p6DkQSj8D9sARtwgWuOyXx4GH54zVfAO00Fw42PP9ZvPZfzJqP9lK3NH5qLUw+pWvZ2tDsmqL7Bv5Yxw780cIULseMC5F1OvTeW92FXXZ/mUKrw/bT9i1A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. set_pte_at() is a core macro that forwards to set_ptes() (with nr=1). Instead of creating a __set_pte_at() internal macro, convert all arch users to use set_ptes()/__set_ptes() directly, as appropriate. Callers in hugetlb may benefit from calling __set_ptes() once for their whole range rather than managing their own loop. This is left for future improvement. No behavioural changes intended. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 12 +++++++++--- arch/arm64/kernel/mte.c | 2 +- arch/arm64/kvm/guest.c | 2 +- arch/arm64/mm/fault.c | 2 +- arch/arm64/mm/hugetlbpage.c | 10 +++++----- 5 files changed, 17 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7f5ce5687466..84919a3c558e 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -356,7 +356,7 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr, __set_pte(ptep, pte); } -static inline void set_ptes(struct mm_struct *mm, unsigned long addr, +static inline void __set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr) { page_table_check_ptes_set(mm, addr, ptep, pte, nr); @@ -370,7 +370,6 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr, pte_val(pte) += PAGE_SIZE; } } -#define set_ptes set_ptes /* * Huge pte definitions. @@ -1067,7 +1066,7 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) #endif /* CONFIG_ARM64_MTE */ /* - * On AArch64, the cache coherency is handled via the set_pte_at() function. + * On AArch64, the cache coherency is handled via the __set_ptes() function. */ static inline void update_mmu_cache_range(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, unsigned int nr) @@ -1134,6 +1133,13 @@ static inline void set_pte(pte_t *ptep, pte_t pte) __set_pte(ptep, pte); } +#define set_ptes set_ptes +static inline void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + __set_ptes(mm, addr, ptep, pte, nr); +} + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 7e89968bd282..9b248549a020 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -90,7 +90,7 @@ int memcmp_pages(struct page *page1, struct page *page2) /* * If the page content is identical but at least one of the pages is * tagged, return non-zero to avoid KSM merging. If only one of the - * pages is tagged, set_pte_at() may zero or change the tags of the + * pages is tagged, __set_ptes() may zero or change the tags of the * other page via mte_sync_tags(). */ if (page_mte_tagged(page1) || page_mte_tagged(page2)) diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index 20280a5233f6..478df2edcf99 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -1087,7 +1087,7 @@ int kvm_vm_ioctl_mte_copy_tags(struct kvm *kvm, } else { /* * Only locking to serialise with a concurrent - * set_pte_at() in the VMM but still overriding the + * __set_ptes() in the VMM but still overriding the * tags, hence ignoring the return value. */ try_page_mte_tagging(page); diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 6045a5117ac1..d3a64624ed88 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -191,7 +191,7 @@ static void show_pte(unsigned long addr) * * It needs to cope with hardware update of the accessed/dirty state by other * agents in the system and can safely skip the __sync_icache_dcache() call as, - * like set_pte_at(), the PTE is never changed from no-exec to exec here. + * like __set_ptes(), the PTE is never changed from no-exec to exec here. * * Returns whether or not the PTE actually changed. */ diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 95364e8bdc19..31a1da655bf1 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -264,12 +264,12 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, ncontig = num_contig_ptes(folio_size(folio), &pgsize); for (i = 0; i < ncontig; i++, ptep++) - set_pte_at(mm, addr, ptep, pte); + __set_ptes(mm, addr, ptep, pte, 1); return; } if (!pte_cont(pte)) { - set_pte_at(mm, addr, ptep, pte); + __set_ptes(mm, addr, ptep, pte, 1); return; } @@ -281,7 +281,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, clear_flush(mm, addr, ptep, pgsize, ncontig); for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); + __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); } pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, @@ -496,7 +496,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, hugeprot = pte_pgprot(pte); for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); + __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); return 1; } @@ -525,7 +525,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pfn = pte_pfn(pte); for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); + __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); } pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, From patchwork Thu Jun 22 14:41:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13289251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0968FEB64DA for ; Thu, 22 Jun 2023 14:42:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9AD308D0007; Thu, 22 Jun 2023 10:42:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 935B28D0001; Thu, 22 Jun 2023 10:42:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 788018D0007; Thu, 22 Jun 2023 10:42:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 63B9A8D0001 for ; Thu, 22 Jun 2023 10:42:39 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 44D711C8421 for ; Thu, 22 Jun 2023 14:42:39 +0000 (UTC) X-FDA: 80930650038.18.E021E44 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf13.hostedemail.com (Postfix) with ESMTP id 73EED2001E for ; Thu, 22 Jun 2023 14:42:37 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf13.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687444957; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0oPSCCAWlfmzkzgt7vGIxZAnqWTVpAld++EiGzSncrg=; b=x/DFB4S7FL3f4VN3JoKqa3ZVTouUoc2H3/oDcitAMS/LG/Yp+eWonlkNq1lgN2rLqyhi1q BZCy0/gQM4hiecu+k90AXiOB8ekR882I9eCq/FtWZs3jQU3rSOWLLjzSP105BVZcLTnnfr AdayLR0ezTQ7gFCs4/6ELtvYN+ZU2CE= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf13.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687444957; a=rsa-sha256; cv=none; b=OAKotWw9GxTXpyDOy4uChlVRYP4eJ28jMfi1MM2BBpVaYwDH7Ivj0UnvdZrR9gFoOwrOfi dQ3m6KdYimz3rCe1ogiasJi0OhzUUElTu+XGWWgSf92lwTg1P5lb5qcK+WJ7mySJ/kDKVv W259+SHLnIfFTRSy6gW5O+qiX85q0mE= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4662B14BF; Thu, 22 Jun 2023 07:43:20 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B63163F663; Thu, 22 Jun 2023 07:42:33 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 03/14] arm64/mm: pte_clear(): New layer to manage contig bit Date: Thu, 22 Jun 2023 15:41:58 +0100 Message-Id: <20230622144210.2623299-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230622144210.2623299-1-ryan.roberts@arm.com> References: <20230622144210.2623299-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 73EED2001E X-Stat-Signature: zym34t8kzncuwofym6oye14x69zk45t3 X-HE-Tag: 1687444957-219652 X-HE-Meta: U2FsdGVkX187ZHz9JiIOGCBcLw779OOgMSQTMnW4pKXufbQ5zXG0rUxMNVlmeAmbKPAkz4jJUVRNVnkanUOtqwJyu5EOSQFfH8YQGJd8MTqK12VtCqilbN/roXFjMZvtdZjVQtjgxwpI3YtN0XuyAE7AOJ83Thy9IfmbG77c5o6YvsfnDbdAxIH/Vqm4LQt+pgLAtUekyjHUIfKUK4w94ICFiTWEoo1VUlPP+YGR/iJErDvHhtM2oNrA5U8CtnIx4+vbQw63g6IN4ukr2PCTRZhaDtCToqi+hUQABRhRXFhUU18AX9I2lDZG+Z3IPGAy3xkX/ASB9fUXaHieByN85DI+NmBnfDejx7S7sS0Lv/dEu4PCnoJ8PUJrjwEXr58M45xQtY0OmnoE02g+IxHxnJ5+thoA2K7Zok1sZgxYrzaMjMzmNSpyReh+B9CbzJmUyQFnNADkXtn8w2rA1rHDUM5AiuLWo1X61xwI/a6qkbgpEXWZkNi2L/znmR+qvEIj2IV1QrsbRrZ7unFo3Fnrn99S4/ovhGvXQ6bJI8ctTprpwBTWEnpxKkAqznvaX3Kbc75bB9+8zUfRWtSI6p9COndyjrsIv20xWRMs9JYpu+Ahz5KchHD6cPzPL6HNNq5EiYmrre7hZ6zPhJ+aPSO74wZKXPYg2ynfOr0oW6rAydOgK927iwWIupAguNR6AuMYi9ErV3g1hlkAJGjYJgabT12I94IWSUyFS5Yw7GulVCbx0BtxI63jQ+XSLp56xh9im+1SUHQvC8Ntbk5miHrY5/yVUlW6Ocg54YqmelXkI+xA8twnPZppnGotgtS9PwBB2FTdWLh6CC8MyHvr7SSNFofRPFhZ5v6Ny4uDXumaZHldQuo1ahARSedHETehE5Jx0Dlwd31m+nQ7Ty5EAJyf+Cd/0nIfx5JTJSsiCKKtw/kZk3WCK2iXHmLQqYJFsNQ4MvL/9LbWSxUboRTsVzz fFni8S96 /csGZ0b1gaJPbEyumFcaPkXum8DdUkKqHssrHEHt5K5imMJzNPPilRSRQb2jAPTs/g3W7xs5nP/kRo2HT6trqbCvjmH0hHBli7TW76oEl7fiJiDa+7x8PdPd1ZgtqnzhstiJxS9znNLIPgZ/Pp6e/glhaHQ8O8WdOu6/qM76tGB8ffn6yJfp5vKS/uvXzIe+MqFCgOuFXgkFbsmlhggBKzp5rThhmmf9hpk3Fo4+G/oixtHyXR2kHx37tFCH+o2lWszaaO6M9G0esatJJlBpICjZrxg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. No behavioural changes intended. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 8 +++++++- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/hugetlbpage.c | 4 ++-- arch/arm64/mm/mmu.c | 2 +- 4 files changed, 11 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 84919a3c558e..06b5dca469f5 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -93,7 +93,7 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) #define pte_none(pte) (!pte_val(pte)) -#define pte_clear(mm, addr, ptep) \ +#define __pte_clear(mm, addr, ptep) \ __set_pte(ptep, __pte(0)) #define pte_page(pte) (pfn_to_page(pte_pfn(pte))) @@ -1140,6 +1140,12 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr, __set_ptes(mm, addr, ptep, pte, nr); } +static inline void pte_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + __pte_clear(mm, addr, ptep); +} + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index 51cd4501816d..bfc02568805a 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -123,7 +123,7 @@ void __set_fixmap(enum fixed_addresses idx, if (pgprot_val(flags)) { __set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, flags)); } else { - pte_clear(&init_mm, addr, ptep); + __pte_clear(&init_mm, addr, ptep); flush_tlb_kernel_range(addr, addr+PAGE_SIZE); } } diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 31a1da655bf1..eebd3107c7d2 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -236,7 +236,7 @@ static void clear_flush(struct mm_struct *mm, unsigned long i, saddr = addr; for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - pte_clear(mm, addr, ptep); + __pte_clear(mm, addr, ptep); flush_tlb_range(&vma, saddr, addr); } @@ -418,7 +418,7 @@ void huge_pte_clear(struct mm_struct *mm, unsigned long addr, ncontig = num_contig_ptes(sz, &pgsize); for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - pte_clear(mm, addr, ptep); + __pte_clear(mm, addr, ptep); } pte_t huge_ptep_get_and_clear(struct mm_struct *mm, diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index c84dc87d08b9..085a7e3eec98 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -853,7 +853,7 @@ static void unmap_hotplug_pte_range(pmd_t *pmdp, unsigned long addr, continue; WARN_ON(!pte_present(pte)); - pte_clear(&init_mm, addr, ptep); + __pte_clear(&init_mm, addr, ptep); flush_tlb_kernel_range(addr, addr + PAGE_SIZE); if (free_mapped) free_hotplug_page_range(pte_page(pte), From patchwork Thu Jun 22 14:41:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13289252 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BBB67EB64D8 for ; Thu, 22 Jun 2023 14:42:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5671C8D0008; Thu, 22 Jun 2023 10:42:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F16D8D0001; Thu, 22 Jun 2023 10:42:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3902F8D0008; Thu, 22 Jun 2023 10:42:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 28A318D0001 for ; Thu, 22 Jun 2023 10:42:42 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 05D281C785A for ; Thu, 22 Jun 2023 14:42:41 +0000 (UTC) X-FDA: 80930650164.23.713D757 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf14.hostedemail.com (Postfix) with ESMTP id 22437100017 for ; Thu, 22 Jun 2023 14:42:39 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687444960; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AS1XtnxRscbzxA0kaRz9FvWsJTVqgcu/WVonKiamoVo=; b=PjpItEAcPAMEgRhJqDN79tYlRmf0T3d3gGa4gzsAq4KwHSVFoN2A44ZyfZDodQNb3ILsiu W7TdrOWcfIIB7trQRDJAnqowEcZJyTLBJ7l1xQhk9qv69pz08T1cCMH0OFUzb/dMJ9WGAd tkWLN4IKu3rV14R8FgIeJuHjI/BzHVs= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687444960; a=rsa-sha256; cv=none; b=dyILnZ/oUXMRfrVPU+IqInCCDMDkRy3vede2x5Exing9fZofF/gh8q4gHq8HDNoypZfkrW uC49+8hKa4L9pC9SwuHiCH2P2N+YgUg+xwGN3F37QXNwyKSp2LhAkBYIyz9sA35fnzRHkU cYoUwB5H3FbbyBxdrtwPIkm0udiRURs= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 200F6C14; Thu, 22 Jun 2023 07:43:23 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8FFBD3F663; Thu, 22 Jun 2023 07:42:36 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 04/14] arm64/mm: ptep_get_and_clear(): New layer to manage contig bit Date: Thu, 22 Jun 2023 15:41:59 +0100 Message-Id: <20230622144210.2623299-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230622144210.2623299-1-ryan.roberts@arm.com> References: <20230622144210.2623299-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 22437100017 X-Rspam-User: X-Stat-Signature: hjhpqymhzoycipgkd3ijkkkumd5fcnt9 X-Rspamd-Server: rspam01 X-HE-Tag: 1687444959-957302 X-HE-Meta: U2FsdGVkX1+Zhr75MKx+baeBOdOywKXK4oJpE+FUujAbPEDLPYiqvr3N4lLruQ0M0GT3WMC/I0ppXA3BdEv/2/dsbDNNl5QWnQ7/+VF4HKBDrIageuUCLQB0mK3cTvt+GRNcSPhVULx/m2DBvAJu5wl0SwNZvxbmPKVPI5XKi6DF7njEcjJLW2BU56XtKHlfziGqLMZg01L+kZ0qJf24dAbkBvbU1MLzbzObWT4dSFkXAoUhcrDUKQ4CwH4QcbtOUBhVkrUQ7Juv9au/f6MYIpZkjrksSuk6PGzAmgoCjfE2p/AQR+8pdPvhej4UJvuFNl2RkL9MD4R3t6t28OoFTx7WX62taXx5bdJgeYPu5VhVclV/UW/U+8D+2qsX5bgSUeJdn8pIaUUk+n5zjCFLKbtDwf/y9/oEivXxxERrNgD25BJ2JeYw0PVcKmVKC3qtfCF2yM8nQQWL5HQygjSwZ7/aHNGL88QE04TIAKGLxTD2dxUQ3WZ6I0YYMTi1bYr7fBZ0RHx1LOdfB0bzhjvX1b6helHxzdxkeR92cDBvTAwKdRWwT/wUB+AuSlluLYtBHL9g4lQAW3GNXx9fWz+kR6TinaM6x0qfaAlC0cqSyGer0KXUvfRezFP6rVhlv0PZp0fgCCPIpqc/7bFVPoNO8LgnIvhr0xMvQFL/suBD9V+WZRoNmEl+tTv9xBTnUqPBhs4mALnGorEl/pZEmuz6KTYlmTmUS30a/4QHUkOFp9uG2WiY9nAlCDEOZRYP2EXNLebQbGfehMbibdMIBWOKjvHpZbCZkf8lkg7LADXpBpYuDXw02z3SAGGBrKEme9SwdJEd0WwsdyuAqJigqVj/84MRpviX1odrSifX56HWADw2LaO6XYaeusbWcT86iYxgG81HWN/wLDyMv4ALVbIlFxZ/FwEse7qbZdbBGUOEVbjONZUoI0MiZwAGVaZQvgwwfrnYwh57q7NrNP4OI8M YprG6WFA /IWAozMeD04Itm2eCv/mJWKDLfSvyWW37x6UJbSQfnjr55ehmD/Gxf1eXUhxwhcroxOzRtEf+NP7T84Jlw3iy1EX31JMfv5CUx1KjllY6/gRF5bmFT/UIOONYx6eAEmQnmeOHWEefkFCPjOSpU3tdlafKTIra7KrXl77J5O0Fc0XLMvgt9Do3WG70JWuBpcJ4z7ZXJO9P8FjyCBSNsiza4p1Sc6J4Xxtfw720affbIAFXqOwMK60HJE4s1N/OJFNWYZgHAr+LqNhxalCaq9P+XaOyiw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. No behavioural changes intended. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 10 ++++++++-- arch/arm64/mm/hugetlbpage.c | 4 ++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 06b5dca469f5..2a525e72537d 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -941,8 +941,7 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -#define __HAVE_ARCH_PTEP_GET_AND_CLEAR -static inline pte_t ptep_get_and_clear(struct mm_struct *mm, +static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, unsigned long address, pte_t *ptep) { pte_t pte = __pte(xchg_relaxed(&pte_val(*ptep), 0)); @@ -1146,6 +1145,13 @@ static inline void pte_clear(struct mm_struct *mm, __pte_clear(mm, addr, ptep); } +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR +static inline pte_t ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + return __ptep_get_and_clear(mm, addr, ptep); +} + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index eebd3107c7d2..931a17f3c3fb 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -188,7 +188,7 @@ static pte_t get_clear_contig(struct mm_struct *mm, unsigned long i; for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) { - pte_t pte = ptep_get_and_clear(mm, addr, ptep); + pte_t pte = __ptep_get_and_clear(mm, addr, ptep); /* * If HW_AFDBM is enabled, then the HW could turn on @@ -429,7 +429,7 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, pte_t orig_pte = ptep_get(ptep); if (!pte_cont(orig_pte)) - return ptep_get_and_clear(mm, addr, ptep); + return __ptep_get_and_clear(mm, addr, ptep); ncontig = find_num_contig(mm, addr, ptep, &pgsize); From patchwork Thu Jun 22 14:42:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13289253 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DF57EB64D8 for ; Thu, 22 Jun 2023 14:42:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A22D28D0009; Thu, 22 Jun 2023 10:42:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9AAF68D0001; Thu, 22 Jun 2023 10:42:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 84D1A8D0009; Thu, 22 Jun 2023 10:42:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 759F18D0001 for ; Thu, 22 Jun 2023 10:42:45 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2E9CB1C8421 for ; Thu, 22 Jun 2023 14:42:45 +0000 (UTC) X-FDA: 80930650290.20.BC8E5D1 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf28.hostedemail.com (Postfix) with ESMTP id 0E52DC0017 for ; Thu, 22 Jun 2023 14:42:42 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687444963; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IEufNZScMSs5MVwG9Ps+h77UK8rEp6c++7KnxZ9ZyzQ=; b=yxcQAteu0gHZ3///SjtAQ+slE9F7Qik0Y8kNITO4NTozQ8lisVRahJ8/hyvCpJLerk/Jzu /vsDdctavuMRo/iHl8LeofAHBe93C+ddioUGIFpmrqu16EKL1Dp6h18xuGpwDslOqnpLwF d0fEjanE90bKS4ldHhKrD89Tizd5/yo= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687444963; a=rsa-sha256; cv=none; b=neWx8HBeG5u01/OVEOUiD1QYDE6sAz7S66RC2IpNfFvm8aN5iMNTpUYR5lb9NBqZ3eUPK9 GrGNOszITK8Lv6RxeWaogtm/tuhch6iSchuyv9/UPneF2WtalUW5ivocimSmB3CVZF9rsH mGJN76lkC7yn4M2LFPzyeEuB0clANh8= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EDA091042; Thu, 22 Jun 2023 07:43:25 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6998E3F663; Thu, 22 Jun 2023 07:42:39 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 05/14] arm64/mm: ptep_test_and_clear_young(): New layer to manage contig bit Date: Thu, 22 Jun 2023 15:42:00 +0100 Message-Id: <20230622144210.2623299-6-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230622144210.2623299-1-ryan.roberts@arm.com> References: <20230622144210.2623299-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0E52DC0017 X-Rspam-User: X-Stat-Signature: f14pa46r6ziq6ipkb8m49i4nkkp5bowm X-Rspamd-Server: rspam01 X-HE-Tag: 1687444962-387283 X-HE-Meta: U2FsdGVkX18UY5rG3iEPjb7V5th3DkewUSkZqQVF+yaeRvnQ7BGu2iCb6+SzZOiExc+Lw6YmCw/HeZkiZfTwrDkhAapjt9qo2EEEo1wlLI0xY/lMN8CwbBwWmQo3B+XNS8tkqAIO3494VTsdMDeolMinxCXETDoQ998juVsJnTXcayMEjgJH678FHs7gbjNCT7yBQN3H3O8Xg7TCmSIs/89/7G+8ljPlB+jc+EGDYNMi/RF4ptpw5Np7PghoyHV/uKB8Z+Bh0ceTe7YRSNsHCqHdIZcRDVFuRHPavMINRPaPI7M/1UyvwpWZTxgdep4GRtAJIQ8fcDuQomXlKm0kS8E1e9ChPqVYPRVnQqLnD31QwRos2blGY/y17aXSLs6XdHJ18WWK5TblabCbCya4v+59HdSjUpq1wjH6hirz1V1RfACHIXPGWQ4SKBMLk/ALBiNPX85jqkyknqS3PztNdPqzDmK0Dqsixm16OwL6hmBI+7io8F8JjdOkEGPQJ5k9UtHINW7yTXreLSkFgSWB497gz+snoIPujm6jPF1Bg/ZZdwQgHmMZJ4/5fdpw+08mktpUubu//ZgyV5n6ztOq2wjUsCbORycxLFMxMKABRmSR7xJNQIdlj7Isz8btdTbMeUeagKiVs8GRtlzWCfuIro830Y1nZ/WuUN2DqwZKBIIWCrjEeAcBOL05+qGzg2mVr3kqQg1gKpVOE25IFR+YggzW84pD9IRoygGDCmpjGaaOAdQtJJrLaX445OSmjCI5mq/PGOw/YlFtSAnIb/dnQu17wRiZEVUyi5RfKi1VNbQI3OX+b7bwUxJ3MjoGqFaDgoUDbnFMdXmN2KsA5ErBGk65XNgB12kHvvdhpCVLePS9aZpxohPctXIeNaw59T5xE+mzbDCwRU8m3MAjbH+KaiwPUJwsRoSNzFKdyzM0Kr2+Pm2RkY3YqtvtxHwOUSlwUTtgFKmIkHEtlwlJ4Sf bwS3fktd glt3kcYW+anbZq/hmbD9EBdYY/Q6lgVw5Qay3jE4x4dxv1VFiPN174KWeVHfRlS/IzX9iQQR/gMcvlSs0MNVDz+uklQTvKPX11Bk2y3BqKhg7mv32HM5QZ+P9FS0srSf+uQErq2oEzLExgCYRspWRiTKUvwBH/1QdHSmVb8z7sWKcDDSuAbuloaA3qVZl7dUCPxMMcqD4a6HWgj2fHsILB/AvVpWTrpWj2bcfNpR4YQWWUN4DA6WxFKmAaKwKhznvVbE5lkT4kBUFVpapPotFcW9uDA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. No behavioural changes intended. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 2a525e72537d..1f4efa17cc39 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -887,8 +887,9 @@ static inline bool pud_user_accessible_page(pud_t pud) /* * Atomic pte/pmd modifications. */ -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG -static inline int __ptep_test_and_clear_young(pte_t *ptep) +static inline int __ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, + pte_t *ptep) { pte_t old_pte, pte; @@ -903,18 +904,11 @@ static inline int __ptep_test_and_clear_young(pte_t *ptep) return pte_young(pte); } -static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, - unsigned long address, - pte_t *ptep) -{ - return __ptep_test_and_clear_young(ptep); -} - #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH static inline int ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { - int young = ptep_test_and_clear_young(vma, address, ptep); + int young = __ptep_test_and_clear_young(vma, address, ptep); if (young) { /* @@ -937,7 +931,7 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { - return ptep_test_and_clear_young(vma, address, (pte_t *)pmdp); + return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp); } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ @@ -1152,6 +1146,13 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, return __ptep_get_and_clear(mm, addr, ptep); } +#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG +static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + return __ptep_test_and_clear_young(vma, addr, ptep); +} + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ From patchwork Thu Jun 22 14:42:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13289257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7B44EB64DB for ; Thu, 22 Jun 2023 14:43:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F2D298D000D; Thu, 22 Jun 2023 10:42:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DCAB18D000C; Thu, 22 Jun 2023 10:42:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF4AD8D000D; Thu, 22 Jun 2023 10:42:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A9E328D000C for ; Thu, 22 Jun 2023 10:42:57 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C7DEFA0BC7 for ; Thu, 22 Jun 2023 14:42:47 +0000 (UTC) X-FDA: 80930650374.01.C731AE4 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf10.hostedemail.com (Postfix) with ESMTP id BDBC6C0009 for ; Thu, 22 Jun 2023 14:42:45 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687444965; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RSoTM1LsxTuhuluqju/tIeLVsAKwQnd+CnuMIDbAx3Q=; b=b6j/3tivsoYsuFkpUDn1kRuCuY2LvJ3vzlvKKA+e/k9135zg+sNDxGvM1hSFuf9REPGiYt B63SLd9hnFKHmgj9CDtdXdnzDliLOa5DKnDHTS8aZEIgxHXL3Qb2dqGhzDmH6GhxuppYBm lcHwSB7JYM1APzkWXhzykk2GXv+e8a0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687444965; a=rsa-sha256; cv=none; b=tIJEehYxtHIbQsLUsnYG++uuBmVG46T6gdvoOch5JqgeNJ4QAXimRJreyCsgMktza3H0bo 3AgMOiaCVbGK9k4AaGad0gi5KkPAq0wbfEOI+DRGeteiE7QNTkwFcAVw9wYA6Clbd9stXw h1Tmu0I8NhgRWKM9DgGvIqSHUbKZWjA= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C707A113E; Thu, 22 Jun 2023 07:43:28 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 431FC3F663; Thu, 22 Jun 2023 07:42:42 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 06/14] arm64/mm: ptep_clear_flush_young(): New layer to manage contig bit Date: Thu, 22 Jun 2023 15:42:01 +0100 Message-Id: <20230622144210.2623299-7-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230622144210.2623299-1-ryan.roberts@arm.com> References: <20230622144210.2623299-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BDBC6C0009 X-Rspam-User: X-Stat-Signature: r5qnmydx8apofgindt9w3cgumsgo4q7r X-Rspamd-Server: rspam03 X-HE-Tag: 1687444965-697583 X-HE-Meta: U2FsdGVkX1+OkSsuD1c2ussIjJii7D0qs3sDbjUsmHeiJ13cCSlVNXDZ0+QQN2IIC4NFo795nyKZ3qM8nlweHZ/fir9b6lhs71wGzWhaTgNwzpmrYJPzsRCcLwGB8TVflgDC+DXVZ7UKTOESx5bIqTwoUxGMbuXOYCgmPiH8mMYNtEACJy9KWh6jGJNf6cLsRlij0vNVNVvyVIcbPOhoXqvCDXjiVATWcsj2qOW7DpJCnbemTodpHyljGnfIBYGp0gWdh76rNCc5zbsldoaGm1rzlvwpd7Qbs7qfC/leVEEW5AcPuCchu7Wr1NNvbL8D7n3+MmiUxKcxm+KIO1/vM7jE5bKrcqJO8R3w/6SpaMEtFnUfZE32p3wngDeaNBuqPOc1VcD+4n5Jr7vuBiqNXzPujkoAKyTpgYybeI0F/AUEtrvdNtHWN6ZFa4TKPI1vWz83x/+Bw4K41ia79Saj3R15FTLYGFY1Vx3af/7EIw9sbZanPNLagBYMJfcLZY1sD3/t9sPEqN2t0EwGt4U5D3GaZ8tKVBhdSyWAiDimxfpppVorO5AdQccUafBbxoGBwL4iSCRhtxr6E9edTsPqlf30jv02GxCqg7Kx87tzvV6qyHh/fs57kBeyU9lmoXoZKtYmagyXK6wW3Wy4ET1nWzKbpQYfcvviKSHFtq6MS6JEqK0+2nWZt6aWFtnMRhj6DoVrHaYDvHAarnWHLos/sePkifRUmgbLfxkYwNoHsKdaxJPYRvcOgHbcHLAy796gM5Xq0SdsrUtBIBhyBjN0OAERRN+Q8HNw/NjzbNOD6Pe8yjfFBgggxJsCCtGzCi4DVj8PvwwC05bvD8MCMbPzGHxX9gzVqY2BwHQskBRuzAGgiPY+uDMbdtBJfyXb+RSFIBFghJfHLxROD9JSqBlKYkAQF0fTdVIDziuk1V8LLyJhF1YHWEk6ORzerU5zHiD80D0HUjONwjlPYlK04yN s3d/KxTy aihde49ubXowjC6az23IgA/dIqKvirnWrVfBqxjGDyXL7AgTJvKV6YxT1lwQVweOqBIBXSeztjmZMdluxKN0Aduooadgrty/oFU4ZQWUYipjPWnN+oDQL44SzhfVBp0eT/ZTNkUV9usYs7lPSjzpHkjWMTsAg3JdBTeZxH5HF/ZmzWRPbnIIkXTWTmH5cHOPJl46yIyZSgObBp8XD2k4mlSee3rmuDXZhmak5oLQge7znsh4Yupw2CbNCXk4r1yKvIXc615ikt8G1GdS5W8U9jYlOSQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. No behavioural changes intended. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 1f4efa17cc39..450428b11c49 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -137,7 +137,7 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) * so that we don't erroneously return false for pages that have been * remapped as PROT_NONE but are yet to be flushed from the TLB. * Note that we can't make any assumptions based on the state of the access - * flag, since ptep_clear_flush_young() elides a DSB when invalidating the + * flag, since __ptep_clear_flush_young() elides a DSB when invalidating the * TLB. */ #define pte_accessible(mm, pte) \ @@ -904,8 +904,7 @@ static inline int __ptep_test_and_clear_young(struct vm_area_struct *vma, return pte_young(pte); } -#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH -static inline int ptep_clear_flush_young(struct vm_area_struct *vma, +static inline int __ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { int young = __ptep_test_and_clear_young(vma, address, ptep); @@ -1153,6 +1152,13 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, return __ptep_test_and_clear_young(vma, addr, ptep); } +#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH +static inline int ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + return __ptep_clear_flush_young(vma, addr, ptep); +} + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ From patchwork Thu Jun 22 14:42:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13289256 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2640EB64DA for ; Thu, 22 Jun 2023 14:42:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 740288D0001; Thu, 22 Jun 2023 10:42:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C57E8D000C; Thu, 22 Jun 2023 10:42:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 564A58D0001; Thu, 22 Jun 2023 10:42:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 338268D000C for ; Thu, 22 Jun 2023 10:42:57 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 9BAEFA0BD6 for ; Thu, 22 Jun 2023 14:42:50 +0000 (UTC) X-FDA: 80930650500.24.427EBF8 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf30.hostedemail.com (Postfix) with ESMTP id 9FB4780009 for ; Thu, 22 Jun 2023 14:42:48 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687444968; a=rsa-sha256; cv=none; b=Uiwz1SYXYwBhVKe1vMxAsN9EYA9Go/YH013r5aZDiKK8b0iC1cVlgtxlthf3LsFmSBRSis Rjp61ojGNxDbRm64rPxyAkX5OmmuGVjrOiciHNwjZoG+Yb4O1Ea7MOrFK0ANobWp3vK9sK aPHmE6PJR7+cTUGU8lXcCtVSrKK+Zuk= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687444968; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LGmsfh5ujWnepK3ZoOUTFN5hSel/MimSSYZZvfFi/dY=; b=jRw2yVRzyw77xzpcxGjLyjCXuI2XeK6cDoFbm0HDLnopO4Jb/94E9Sce3r/QDq7KgxLFWK 2mz0ZY12QxvpuW4TVh8/TY86Fs4h2bZn7hcCez3sLfQ5uSw2GNfCf6E1thbDVyAe7dON/r Zz4kmaDmaYBCZZd6i7+Vsfb6/UNGnnc= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A0A93C14; Thu, 22 Jun 2023 07:43:31 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 1CB8C3F663; Thu, 22 Jun 2023 07:42:45 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 07/14] arm64/mm: ptep_set_wrprotect(): New layer to manage contig bit Date: Thu, 22 Jun 2023 15:42:02 +0100 Message-Id: <20230622144210.2623299-8-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230622144210.2623299-1-ryan.roberts@arm.com> References: <20230622144210.2623299-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 9FB4780009 X-Stat-Signature: 3dn4tost8ezp5fafeedtocxoh364r4nz X-HE-Tag: 1687444968-736522 X-HE-Meta: U2FsdGVkX1+9Ell8q/xkNquMIXSOyt3E9I3fCevRZpvK9wGi2oWwi3xmgel6w6tR3db2wIhHvMgR8w63AAIBV9F1vwuczYgCoHArUkJmZwCe5IYQvJ/01+L0bdT+9Z2b0Q1xZjnvLHBD5cKPl3lafwL9fP9/N3IvC1nrFcn7NrUSIUicgjsMIK/WAKlVTyoT8UhE3JJ5mpiwtThl2MJp36PuYz0bpEdCgISGyN1hQhQ7T0OjUptYVkTHYB52IP+/9q2/hUUJy6xOcMWUBa17u643ftI9FRlNdqltVZuTU7EjqWcf2YD01N9k3krL7BU6hDk0lc2oVf6QHF9A6VXSEHMMBP8gHr5O9VDUT6Vs2D9DYpqsXOFMzcs9HmbSlrG+lWinlie3TFImE3vkDQXmPnP/6/q8T7UMROECSBMQkZwCTVk0oJLs+UUdK05H0iG4xNzuku6ysFCHFFIJnGTcRijpGXOVXWpIutwPrBSCrH8xre1cp+TTnbgB3xelY7020Yy/Uj1MEXaXbWv+VKm76BRJb7OlSVmH+WvPZxAJBhyWKBT+nFsdrkjHfH0rap6RL0nGcuMmGFKrFi3kuMjLzYSIh7ZPpBZF3SCNEtFZwm4ZV8p2QcUrSpUQzQAHcioMita1eBKCgVPBlyc6GzQUevXG+JfSZePnVP9xS4NE7uTsn1tCdPtloYJN2Q/MsSgSb0XJITsU4NEEbKOxxae4ikhHnTBbuxBCv6Y989o5nTAGYiy/G+M6mxdPjNOn+aiJX4IJKoE9aJVLWDoSeqoP6jfdWuOM1zWkfvp90Tfo9agMVw9SqEFnaP01SzxN/Yks9EKn1JXzq5E/W2tWQqQ3rcRe10vvd/5y1rHaB4XjT8jbcUukcdTLW6etWVbXGXQUs0CqxOOkqkfxx+Csd00NU1tD9PMJMIVJ9IteHzzX22BRsXIdhuuQ0RuZe2FBeXUny0+CIAo2k6buEvKmzRF MopKlJae 5PE9HlmIdlCzkutRDI558vsSuXaSiWboLSqH/HKew5rzc/47qXIZpR8l+P3vcI+Jn70PIhRBvZ14AJMdn+Mnf4tVXoOFHKnUM2fiohepbm66b+jktg+wvbGrKthoD6wej1kilY0xU+xPOP6qWbHtQ6T3T6tE0AfLgtM5lXhCsWu3+GPBaCQnYnSEOpwkp9UAUjB9lwxSKO0fWP+REQDJZCZJkgyUMjXYbBpx7ZA/P3paJlAhncH+x4Lj08PEbeSxg5nz1rmGe7DRdxNhH7I/nCsWb6w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. No behavioural changes intended. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 15 +++++++++++---- arch/arm64/mm/hugetlbpage.c | 2 +- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 450428b11c49..2fcc3b19c873 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -958,11 +958,11 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ /* - * ptep_set_wrprotect - mark read-only while trasferring potential hardware + * __ptep_set_wrprotect - mark read-only while trasferring potential hardware * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. */ -#define __HAVE_ARCH_PTEP_SET_WRPROTECT -static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep) +static inline void __ptep_set_wrprotect(struct mm_struct *mm, + unsigned long address, pte_t *ptep) { pte_t old_pte, pte; @@ -980,7 +980,7 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long address, pmd_t *pmdp) { - ptep_set_wrprotect(mm, address, (pte_t *)pmdp); + __ptep_set_wrprotect(mm, address, (pte_t *)pmdp); } #define pmdp_establish pmdp_establish @@ -1159,6 +1159,13 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, return __ptep_clear_flush_young(vma, addr, ptep); } +#define __HAVE_ARCH_PTEP_SET_WRPROTECT +static inline void ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + __ptep_set_wrprotect(mm, addr, ptep); +} + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 931a17f3c3fb..7d5eb71db396 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -511,7 +511,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pte_t pte; if (!pte_cont(READ_ONCE(*ptep))) { - ptep_set_wrprotect(mm, addr, ptep); + __ptep_set_wrprotect(mm, addr, ptep); return; } From patchwork Thu Jun 22 14:42:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13289254 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02626EB64D8 for ; Thu, 22 Jun 2023 14:42:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9AD338D000A; Thu, 22 Jun 2023 10:42:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9351A8D0001; Thu, 22 Jun 2023 10:42:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D78D8D000A; Thu, 22 Jun 2023 10:42:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6F2C98D0001 for ; Thu, 22 Jun 2023 10:42:53 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 3B7E11C888D for ; Thu, 22 Jun 2023 14:42:53 +0000 (UTC) X-FDA: 80930650626.05.E333F88 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf03.hostedemail.com (Postfix) with ESMTP id 403A72001E for ; Thu, 22 Jun 2023 14:42:51 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687444971; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8+4aKe5InG/sT9ahgDIcgoSk7tRyyrQ1fi5c8NU1sPo=; b=CxiOKhq2SM4PuP9N1lXiB3kISKf9RmdaZnHs8LZIu6f10Z/1foT4FI+pid2Oh1F+t5z9cx c3j3DeuU0qO1uoUx7UtNZxgJFKoqBrQlanaGwH1sz0sJukquQ66f8NBtew7Jficp4ZMiA1 b9mF2tUFb/XlGZlGFPV/+cRFOaaeX4g= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687444971; a=rsa-sha256; cv=none; b=tEZpgk/5wv86RYAx9TrrMn8CO0twEHrab0f4BbNydyMaPn/6GgW2dmRoNHUAlEbSfuvhEe +wS7uhNACh5Ux35HIR7J/rWzZYh/kh5m9eC5CHbz7Gcd3i/gr5cekf4/TEFXcZXhDuFA9y G+7/hE5MJSG5jnS+UoPRAZxOEKx1kUA= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7A4C61042; Thu, 22 Jun 2023 07:43:34 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EAAD23F663; Thu, 22 Jun 2023 07:42:47 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 08/14] arm64/mm: ptep_set_access_flags(): New layer to manage contig bit Date: Thu, 22 Jun 2023 15:42:03 +0100 Message-Id: <20230622144210.2623299-9-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230622144210.2623299-1-ryan.roberts@arm.com> References: <20230622144210.2623299-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 403A72001E X-Rspam-User: X-Stat-Signature: rqb4rzth8zt8sndyzztp8cwzj6puc69r X-Rspamd-Server: rspam03 X-HE-Tag: 1687444971-797809 X-HE-Meta: U2FsdGVkX198C3mqLG9qCBOK4+7yPLzc7seQGlU5TlxCJBlV07AJtR+dLSXrd//YvfDGXVc2r/kYkK0Els2yRszFIG9NN0g0uV9Lc3HA5F1zrKp+UsHt1E3hO4NQ/36SnA97fo9fXrB8Y+wYjov17pBppGMF+Kz9E9mXYHLtuLyZY2om0Y213WyoL3nSf0v3a+sG3YR6C3g1dyeISTR49lZv/aL7h3aR1+TfDc/IfHKgyCENvXb5i8MnZ5SUTuty+nnwedLw5KaJNmpB3VHdTTRXs+SI7sYM5R+GoQRogayxfASycC+mcZziIwOkdD5ZNdNRIyqmoKqZj+EG/a52nG1kz+Ps301oOECqsofvlWXD1qAAgMimd9874SA8sjcL81K9wzOR1NMnddrvXSDXrMb8r2EsVepZFNNCZXPg2wW6vcfL2/naAy0DE9CHEMuQusgF1BHBQbg2Uo2CaEc7OhMQ0J7pVqJ1PJ07jpSooG2lLJTvqNh1i/W7XFYaiAXMoc9Cl2zJJmQGf2/vCof01HWaGJkReSqrYjG6EDbFeKfHnLbg1pFfqTrs9Ehpbn71mw9Ai73/pOS25oDhnqpqz2rBajOf+j1KNUNPbmvKiiv4wcFr6d00CT+wV274FmnW0Rdak+vhemZFmUHmVuNHpZfjSDdMFxrkiAdxGwJT4O7AO1kUbVtkSl8wfwS8dNWm+HIg+57nL0qY/5ccKBQur8O/ZNCTG2owibqz0II3Xb/4BJNsF7F9vrtit8p63A9F/B8LtaDsLNQupp/nScOUEgqKGuVQvzs8VByx9xvetTLbxZIN9pVpKzvow1sr2vEKCgoFotnkq+yiwiXmNDph+isT8MO516pdhEUY+AH/0398TtP4rK6Vr4+5ceUt6VzpWQbsrARoH4fMNotvmSBjuk9J4wg7x3aZbwejYSMibfpOJwkZihzKJVxCvbYj6tkLAmcLD0ONa3oO7diQje1 beJcuk0K kuFU/0TE6x51enDBZ2GlFsMVJX9nk7Bd10IxsLITfLla7EPZzic2YFfA5Re40NlOLIa6lCBleFxNn17jppFvDkAJcVGTZNDdrf4WaFwz+lgKIPYsHxHbe3VUIqVKCkmVAViYMQ3DDtJvoy7BGzRyP1HUQat/yjhlERlSIVfuauffJMSLP9YgB7DN+RmpIE80cPsVMKMP3Jho/+zxuAxYX+a+a3Y9iKsYis2gzDEsexmbcHOr7IgV1wByfDZRfPDK1aaVprckFXYqnyMOvh5LnQh8huw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. No behavioural changes intended. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 16 ++++++++++++---- arch/arm64/mm/fault.c | 6 +++--- arch/arm64/mm/hugetlbpage.c | 2 +- 3 files changed, 16 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 2fcc3b19c873..ff79578fd806 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -311,7 +311,7 @@ static inline void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, /* * Check for potential race with hardware updates of the pte - * (ptep_set_access_flags safely changes valid ptes without going + * (__ptep_set_access_flags safely changes valid ptes without going * through an invalid entry). */ VM_WARN_ONCE(!pte_young(pte), @@ -842,8 +842,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) return pte_pmd(pte_modify(pmd_pte(pmd), newprot)); } -#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS -extern int ptep_set_access_flags(struct vm_area_struct *vma, +extern int __ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty); @@ -853,7 +852,8 @@ static inline int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t entry, int dirty) { - return ptep_set_access_flags(vma, address, (pte_t *)pmdp, pmd_pte(entry), dirty); + return __ptep_set_access_flags(vma, address, (pte_t *)pmdp, + pmd_pte(entry), dirty); } static inline int pud_devmap(pud_t pud) @@ -1166,6 +1166,14 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, __ptep_set_wrprotect(mm, addr, ptep); } +#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS +static inline int ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty) +{ + return __ptep_set_access_flags(vma, addr, ptep, entry, dirty); +} + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index d3a64624ed88..f5a7a5ff6814 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -195,9 +195,9 @@ static void show_pte(unsigned long addr) * * Returns whether or not the PTE actually changed. */ -int ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep, - pte_t entry, int dirty) +int __ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep, + pte_t entry, int dirty) { pteval_t old_pteval, pteval; pte_t pte = READ_ONCE(*ptep); diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 7d5eb71db396..9a87b1c5661a 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -477,7 +477,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, pte_t orig_pte; if (!pte_cont(pte)) - return ptep_set_access_flags(vma, addr, ptep, pte, dirty); + return __ptep_set_access_flags(vma, addr, ptep, pte, dirty); ncontig = find_num_contig(mm, addr, ptep, &pgsize); dpfn = pgsize >> PAGE_SHIFT; From patchwork Thu Jun 22 14:42:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13289255 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BCE8EB64D8 for ; Thu, 22 Jun 2023 14:42:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1BE998D000B; Thu, 22 Jun 2023 10:42:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0F8F48D0001; Thu, 22 Jun 2023 10:42:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EDBF68D000B; Thu, 22 Jun 2023 10:42:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id DB42C8D0001 for ; Thu, 22 Jun 2023 10:42:56 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 5C09B140195 for ; Thu, 22 Jun 2023 14:42:56 +0000 (UTC) X-FDA: 80930650752.07.7672EF0 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf28.hostedemail.com (Postfix) with ESMTP id 5077FC0022 for ; Thu, 22 Jun 2023 14:42:53 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687444974; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DGRckod/otio/3IodOPAW6O8qQnfn5Pej2aY1zORQlE=; b=mDs8DNKiJQ3hITDiJbCqAOhsqr+PB0GBsysMPQfoNhdjD/tHqSaKc/+WHGRxLyoj/zrJJh peeVczg6MR0nAfEPbfjGTThtdXacqIFszxobCK8QsjXkR31hLlJYW6b6MuQGYpEP69vRdW FcQpl376Rwt7vrqrBh2K5A79oQArBV0= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687444974; a=rsa-sha256; cv=none; b=1X07SK+DmSZMdtOlS4eb0t6dtIEMordIAzqA1dmMqJ/HzPkVqKDhgVeYPIziFQx6Z0dxAM MaIIZUguMwLPMtBKLsEufZ0b/W4Ne5nccEbjoEAXUsOwR1pTXtFsp6IQs5SKBtfzT7JWpD PBmJjcLDGqhKjzt01/yJwYKPJ7MHQLY= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6E9EB14BF; Thu, 22 Jun 2023 07:43:37 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C4A133F663; Thu, 22 Jun 2023 07:42:50 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 09/14] arm64/mm: ptep_get(): New layer to manage contig bit Date: Thu, 22 Jun 2023 15:42:04 +0100 Message-Id: <20230622144210.2623299-10-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230622144210.2623299-1-ryan.roberts@arm.com> References: <20230622144210.2623299-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5077FC0022 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: qsk7zbrg4sx1g7gku4q7ao7nexdgq7tg X-HE-Tag: 1687444973-305684 X-HE-Meta: U2FsdGVkX19PMrlrsdJDfGuNbOLkyyacOsNugpizMmnDmQ3rzq8yb9MTeq88TBDNKMuSz1rvBpWfWQxb7/orJh6fVhvc7eixj0b384EUw35ELTYTBAVOPnQdKLMHYTyWvEzakDMQVsq1yZUifBStij0OmF+aL5DvC4xfI7g7Vl1WXAJVZPMnoFeUCpz7u+DrsA1AKfi8ofp1XpqjIJOOGVvrVQFWX+MFMebMT8YydEet3T8FgsGpWfrc3d4C69T6Cp63eSXFKvDfw685fMKWZuDYdlNHrAwwu0IZ+JnniRdMK8NjsLG6bb2jh2KUjS+WRqpLeropwPsJCX1cat13m1AfUIt/tFmjcBiUlx8tXfEe+2cfOW9KQO121uYLmzgLZ5Gw8e8tU7JCgcmwoXvfz4JcE5oOpD/p3NTsbWciXulzrY/DE6f4vPazNyvUOlikcA+1H3WUkLRNqKtuJaG7q3iSavU4sBK0qMcn3lCUqv1GJI+LVqxjEG5Rab/KAoRL0H5LohFBUgsAVJZXJv/lSryFU+xNb+A2egGXtPZNfplDvEGEMALd2nFmDhBW4zYS5A/EeJjOCtHG6oPA5wI+tYEPzVPRUi8buOlHkOLT19BDjTsAnYVR6T1uKxvFV2czpqYf62mgTHQSOy6sseh4supMyPN1FczQzgVr7+DT6zqvfTshRZP2KJIWLP414dB5pbGYUJKxIAsVyGAS0MpjAD5++ZJA42oNmHytJqtk20MdykHuJMzzKHn/BSAhdc6RhgKg9BOjUNtmA93FwtBQ4QnKvgw+4Um7fC/Sn9QlpMXXQ91d9EtvPAp9fditNPFGkOtWTBDIiVdFXMZRn0b5aXtYln1HZfGs8qcxEtUNGVdFS5Cyvltca96pybWLOPyztVmG+tHsu/CkRPJBQ6Tdz2j390UjXuxtxYcHx5erRY0spTvD5NAu2hkrT41WmBbXV4tw/SFllAGSVnD0yCE 06/4bY9p z7EftyndbPuM8hkvCMJKX/WeCgtGOVlp53RdVcpbSg3LTNnylF6McILlBPQCunSXAdadxBITZIkTTfSP77yhtYB8vBkmeLtE9BC+QvM4rVoQikj5Afwo2EfDXGQiOHoJ7bf37DuBhzMKHbl7rjdS87qqHpqgtEKSlME+vrdz7ODJqVtZMISj18M8MdFBY1E4Ia+McCAm8FTi2nrjf98Q5lxL5RoA8Bz+ewkfebtZxEozEwk+lFVC+SW83ajMHMPdCjLIoxVY1nHPws5kvhZRu2buIAg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. arm64 did not previously define an arch-specific ptep_get(), so override the default version in the arch code, and also define the private __ptep_get() version. Currently they both do the same thing that the default version does (READ_ONCE()). Some arch users (hugetlb) were already using ptep_get() so convert those to the private API. While other callsites were doing direct READ_ONCE(), so convert those to use the appropriate (public/private) API too. There are some core kernel locations that directly dereference the ptep, so these will need to be updated separately. No behavioural changes intended. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 19 +++++++++++++++---- arch/arm64/kernel/efi.c | 2 +- arch/arm64/mm/fault.c | 4 ++-- arch/arm64/mm/hugetlbpage.c | 18 +++++++++--------- arch/arm64/mm/kasan_init.c | 2 +- arch/arm64/mm/mmu.c | 12 ++++++------ arch/arm64/mm/pageattr.c | 4 ++-- arch/arm64/mm/trans_pgd.c | 2 +- 8 files changed, 37 insertions(+), 26 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index ff79578fd806..31df4d73f9ac 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -275,6 +275,11 @@ static inline void __set_pte(pte_t *ptep, pte_t pte) } } +static inline pte_t __ptep_get(pte_t *ptep) +{ + return READ_ONCE(*ptep); +} + extern void __sync_icache_dcache(pte_t pteval); bool pgattr_change_is_safe(u64 old, u64 new); @@ -302,7 +307,7 @@ static inline void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, if (!IS_ENABLED(CONFIG_DEBUG_VM)) return; - old_pte = READ_ONCE(*ptep); + old_pte = __ptep_get(ptep); if (!pte_valid(old_pte) || !pte_valid(pte)) return; @@ -339,7 +344,7 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr, */ if (system_supports_mte() && pte_access_permitted(pte, false) && !pte_special(pte)) { - pte_t old_pte = READ_ONCE(*ptep); + pte_t old_pte = __ptep_get(ptep); /* * We only need to synchronise if the new PTE has tags enabled * or if swapping in (in which case another mapping may have @@ -893,7 +898,7 @@ static inline int __ptep_test_and_clear_young(struct vm_area_struct *vma, { pte_t old_pte, pte; - pte = READ_ONCE(*ptep); + pte = __ptep_get(ptep); do { old_pte = pte; pte = pte_mkold(pte); @@ -966,7 +971,7 @@ static inline void __ptep_set_wrprotect(struct mm_struct *mm, { pte_t old_pte, pte; - pte = READ_ONCE(*ptep); + pte = __ptep_get(ptep); do { old_pte = pte; pte = pte_wrprotect(pte); @@ -1120,6 +1125,12 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, * private versions, which are prefixed with double underscore. */ +#define ptep_get ptep_get +static inline pte_t ptep_get(pte_t *ptep) +{ + return __ptep_get(ptep); +} + static inline void set_pte(pte_t *ptep, pte_t pte) { __set_pte(ptep, pte); diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index 7a28b6a08a82..9536fbce77a2 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -106,7 +106,7 @@ static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data) { struct set_perm_data *spd = data; const efi_memory_desc_t *md = spd->md; - pte_t pte = READ_ONCE(*ptep); + pte_t pte = __ptep_get(ptep); if (md->attribute & EFI_MEMORY_RO) pte = set_pte_bit(pte, __pgprot(PTE_RDONLY)); diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index f5a7a5ff6814..3193526b226d 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -177,7 +177,7 @@ static void show_pte(unsigned long addr) break; ptep = pte_offset_map(pmdp, addr); - pte = READ_ONCE(*ptep); + pte = __ptep_get(ptep); pr_cont(", pte=%016llx", pte_val(pte)); pte_unmap(ptep); } while(0); @@ -200,7 +200,7 @@ int __ptep_set_access_flags(struct vm_area_struct *vma, pte_t entry, int dirty) { pteval_t old_pteval, pteval; - pte_t pte = READ_ONCE(*ptep); + pte_t pte = __ptep_get(ptep); if (pte_same(pte, entry)) return 0; diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 9a87b1c5661a..82b2036dbe2f 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -152,14 +152,14 @@ pte_t huge_ptep_get(pte_t *ptep) { int ncontig, i; size_t pgsize; - pte_t orig_pte = ptep_get(ptep); + pte_t orig_pte = __ptep_get(ptep); if (!pte_present(orig_pte) || !pte_cont(orig_pte)) return orig_pte; ncontig = num_contig_ptes(page_size(pte_page(orig_pte)), &pgsize); for (i = 0; i < ncontig; i++, ptep++) { - pte_t pte = ptep_get(ptep); + pte_t pte = __ptep_get(ptep); if (pte_dirty(pte)) orig_pte = pte_mkdirty(orig_pte); @@ -184,7 +184,7 @@ static pte_t get_clear_contig(struct mm_struct *mm, unsigned long pgsize, unsigned long ncontig) { - pte_t orig_pte = ptep_get(ptep); + pte_t orig_pte = __ptep_get(ptep); unsigned long i; for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) { @@ -426,7 +426,7 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, { int ncontig; size_t pgsize; - pte_t orig_pte = ptep_get(ptep); + pte_t orig_pte = __ptep_get(ptep); if (!pte_cont(orig_pte)) return __ptep_get_and_clear(mm, addr, ptep); @@ -449,11 +449,11 @@ static int __cont_access_flags_changed(pte_t *ptep, pte_t pte, int ncontig) { int i; - if (pte_write(pte) != pte_write(ptep_get(ptep))) + if (pte_write(pte) != pte_write(__ptep_get(ptep))) return 1; for (i = 0; i < ncontig; i++) { - pte_t orig_pte = ptep_get(ptep + i); + pte_t orig_pte = __ptep_get(ptep + i); if (pte_dirty(pte) != pte_dirty(orig_pte)) return 1; @@ -510,7 +510,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, size_t pgsize; pte_t pte; - if (!pte_cont(READ_ONCE(*ptep))) { + if (!pte_cont(__ptep_get(ptep))) { __ptep_set_wrprotect(mm, addr, ptep); return; } @@ -535,7 +535,7 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, size_t pgsize; int ncontig; - if (!pte_cont(READ_ONCE(*ptep))) + if (!pte_cont(__ptep_get(ptep))) return ptep_clear_flush(vma, addr, ptep); ncontig = find_num_contig(mm, addr, ptep, &pgsize); @@ -569,7 +569,7 @@ pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr * when the permission changes from executable to non-executable * in cases where cpu is affected with errata #2645198. */ - if (pte_user_exec(READ_ONCE(*ptep))) + if (pte_user_exec(__ptep_get(ptep))) return huge_ptep_clear_flush(vma, addr, ptep); } return huge_ptep_get_and_clear(vma->vm_mm, addr, ptep); diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 40125b217195..65074cf7f3a3 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -113,7 +113,7 @@ static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, memset(__va(page_phys), KASAN_SHADOW_INIT, PAGE_SIZE); next = addr + PAGE_SIZE; __set_pte(ptep, pfn_pte(__phys_to_pfn(page_phys), PAGE_KERNEL)); - } while (ptep++, addr = next, addr != end && pte_none(READ_ONCE(*ptep))); + } while (ptep++, addr = next, addr != end && pte_none(__ptep_get(ptep))); } static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr, diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 085a7e3eec98..d5dc36ff3827 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -176,7 +176,7 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, ptep = pte_set_fixmap_offset(pmdp, addr); do { - pte_t old_pte = READ_ONCE(*ptep); + pte_t old_pte = __ptep_get(ptep); __set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); @@ -185,7 +185,7 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, * only allow updates to the permission attributes. */ BUG_ON(!pgattr_change_is_safe(pte_val(old_pte), - READ_ONCE(pte_val(*ptep)))); + pte_val(__ptep_get(ptep)))); phys += PAGE_SIZE; } while (ptep++, addr += PAGE_SIZE, addr != end); @@ -848,7 +848,7 @@ static void unmap_hotplug_pte_range(pmd_t *pmdp, unsigned long addr, do { ptep = pte_offset_kernel(pmdp, addr); - pte = READ_ONCE(*ptep); + pte = __ptep_get(ptep); if (pte_none(pte)) continue; @@ -981,7 +981,7 @@ static void free_empty_pte_table(pmd_t *pmdp, unsigned long addr, do { ptep = pte_offset_kernel(pmdp, addr); - pte = READ_ONCE(*ptep); + pte = __ptep_get(ptep); /* * This is just a sanity check here which verifies that @@ -1000,7 +1000,7 @@ static void free_empty_pte_table(pmd_t *pmdp, unsigned long addr, */ ptep = pte_offset_kernel(pmdp, 0UL); for (i = 0; i < PTRS_PER_PTE; i++) { - if (!pte_none(READ_ONCE(ptep[i]))) + if (!pte_none(__ptep_get(ptep++))) return; } @@ -1470,7 +1470,7 @@ pte_t ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr, pte * when the permission changes from executable to non-executable * in cases where cpu is affected with errata #2645198. */ - if (pte_user_exec(READ_ONCE(*ptep))) + if (pte_user_exec(ptep_get(ptep))) return ptep_clear_flush(vma, addr, ptep); } return ptep_get_and_clear(vma->vm_mm, addr, ptep); diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 057097acf9e0..624b0b0982e3 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -36,7 +36,7 @@ bool can_set_direct_map(void) static int change_page_range(pte_t *ptep, unsigned long addr, void *data) { struct page_change_data *cdata = data; - pte_t pte = READ_ONCE(*ptep); + pte_t pte = __ptep_get(ptep); pte = clear_pte_bit(pte, cdata->clear_mask); pte = set_pte_bit(pte, cdata->set_mask); @@ -246,5 +246,5 @@ bool kernel_page_present(struct page *page) return true; ptep = pte_offset_kernel(pmdp, addr); - return pte_valid(READ_ONCE(*ptep)); + return pte_valid(__ptep_get(ptep)); } diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index f9997b226614..b130a65092c1 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -32,7 +32,7 @@ static void *trans_alloc(struct trans_pgd_info *info) static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) { - pte_t pte = READ_ONCE(*src_ptep); + pte_t pte = __ptep_get(src_ptep); if (pte_valid(pte)) { /* From patchwork Thu Jun 22 14:42:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13289258 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4F39EB64D8 for ; Thu, 22 Jun 2023 14:43:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 36D628D000E; Thu, 22 Jun 2023 10:43:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F4438D000C; Thu, 22 Jun 2023 10:43:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16E6B8D000E; Thu, 22 Jun 2023 10:43:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 019108D000C for ; Thu, 22 Jun 2023 10:42:59 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D48F1140B97 for ; Thu, 22 Jun 2023 14:42:58 +0000 (UTC) X-FDA: 80930650836.14.E7DD88D Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf29.hostedemail.com (Postfix) with ESMTP id 12EA112001F for ; Thu, 22 Jun 2023 14:42:56 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf29.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687444977; a=rsa-sha256; cv=none; b=A1nGlQYW4t5Nf55h26PCwQZgHrxFFy+4dHKwGRN10JghndHiZ/3Ise1oE+LcETLeZvJccB 71k4Lh8t5UkhzFdcKdBIqkayJvTvfQB0ZEISOcd2/OA182xS3LhLHTS766vacwc2vBIiK5 H8NQ5wWQUYX5BDAH0ppbtI12dWh7g20= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf29.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687444977; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WAztuwwCVy8qP9L2dunxZQePeRG4wFUsOFlwxBwE1/E=; b=ZKTbkr8QUdnFnNNk6OUqCknu8IAX0Cs+GZeqm3Or6EY/iad8rimSLmcI9Zztlt0qDt2dHS GR4A67yEYL769Y5GhgXJWQjxQt+hT1fQNs1j8TcAXyGlnflJVibT3BrsPYn85lJ+Uskd7S sZikZ6E8lEuezUGSaPcB6lc/Y9aEkno= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4843F113E; Thu, 22 Jun 2023 07:43:40 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B85DE3F663; Thu, 22 Jun 2023 07:42:53 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 10/14] arm64/mm: Split __flush_tlb_range() to elide trailing DSB Date: Thu, 22 Jun 2023 15:42:05 +0100 Message-Id: <20230622144210.2623299-11-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230622144210.2623299-1-ryan.roberts@arm.com> References: <20230622144210.2623299-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 12EA112001F X-Stat-Signature: j4fw4egcy7qqjbhykxzw8gip6ycsc54k X-Rspam-User: X-HE-Tag: 1687444976-408846 X-HE-Meta: U2FsdGVkX18I9TgbQZ2VezJZhMylEDpkLpWDmR6VqBhkZisgIJ+DYGx9HC5pGwnfQ9EPMSoo0kXM6v2r0b0NX3j+UHUpq2AxOxXaRtrd11eWV6kZaQhUraQvWp+BzBdHMQmh/YIjimWC4Flbp54+87eb8sTiDerxA6qr8Li9nsKw0T5Am6MS6yv2B86hECIhgGWkrd61qNUpc2mchH9/HJM0XCjXCGiWHIGpCTrOU3sPBpY+nZO9BH4QXI78GyDvZL3F5xyDza/gtp1apOmRb55cVX443BMvOguFddIfZSVnp486MBJEvT4zAGGvSbqcMK4N4spgW/NKMvqyypgKHXQRXLtT7DniXWk5EYalmiZ9CwQi7E8Hpu0MFCtR1jFsddHEaGmHcDKJeYFh6kRxNmRHPfhwyrg1Q5y2/JIaflq6eToN2NSZhtylIXwEEdE+NiYvXj3iL42T86FJGnmQP/7JiyS/ORc0WW2g68Prg3Y7cyIPOlAE0kzvEL845qv2pRaGxT7a4UuqgfLXyqL1O81vCTzD5xCWhk6yJlWA9s7Eh4vcmxM9dxammWajXw2Hn2jaztkiCPtZ2druXucSqeLlujGAHyqLf8kkhiIHRvjhGDDX0yttyCc/7WRw0PpnUxJIODx90ARp83IrkGnKGY2QKcONqEMYIdV2rWeeVLzH+egha9K7oA8bseCZlsjj2CXndkxLPDj/uDbSskVG0v5fya1d/H8qA8nOpKj2uWe875cRL1YRFNcg/wZaM3qg+GJtsRJz7VcaW+juGEznrYt0WnqPWzX26Zb3H9nLGoJTuG9MXrrjMAXdY35c3vmmgmu6JHJ3bmSxuD7VB9Dt4tixJ2wOZqG0biwMTPrc2ydfb98j7sKLgkrRueMLPPdOBl7Hql83zcGeWaLWslYhmT/8qnKmY/DprSNbXhhuKpwBdDuZUbDvlXrY6W//oLI94wdvE7KTOIPNllVfK++ nc+7zmPi nIo+U6L7KtxD5cQWdfNPVIPyWeF9BwyfeYib8en+YMEv+6Bc6cyLzrdprkBnYmDHUo1HLuYH1+HpR4AvHyFYHE6Hw7JB8Vuzn4hxzfzZssFA7u7Sp5Do+8PMzQlP3DLs9wuYqJqqYULVswQ/NG7OL/oEgjVQ8dnR+nn40lBGFYsEONGuRNJ7O/qM42FMXvQAZiZrtrSRTUxiyFpkWr/wYVMny9khevcM7gcktldrZD1cmVbW/bW/+qclWqkVqdnx9wmH5j/wicSWh/bOV6Ab9BypYyA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Split __flush_tlb_range() into __flush_tlb_range_nosync() + __flush_tlb_range(), in the same way as the existing flush_tlb_page() arrangement. This allows calling __flush_tlb_range_nosync() to elide the trailing DSB. Forthcoming "contpte" code will take advantage of this when clearing the young bit from a contiguous range of ptes. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/tlbflush.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 412a3b9a3c25..de1f5d9a546e 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -278,7 +278,7 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, */ #define MAX_TLBI_OPS PTRS_PER_PTE -static inline void __flush_tlb_range(struct vm_area_struct *vma, +static inline void __flush_tlb_range_nosync(struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long stride, bool last_level, int tlb_level) @@ -357,6 +357,15 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, } scale++; } +} + +static inline void __flush_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end, + unsigned long stride, bool last_level, + int tlb_level) +{ + __flush_tlb_range_nosync(vma, start, end, stride, + last_level, tlb_level); dsb(ish); } From patchwork Thu Jun 22 14:42:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13289259 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF94AEB64DB for ; Thu, 22 Jun 2023 14:43:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2812B8D000F; Thu, 22 Jun 2023 10:43:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 20F858D000C; Thu, 22 Jun 2023 10:43:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 00F3D8D000F; Thu, 22 Jun 2023 10:43:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D21118D000C for ; Thu, 22 Jun 2023 10:43:02 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id EA8BF140B23 for ; Thu, 22 Jun 2023 14:43:01 +0000 (UTC) X-FDA: 80930650962.10.B5B94B8 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf09.hostedemail.com (Postfix) with ESMTP id 1A4F4140023 for ; Thu, 22 Jun 2023 14:42:59 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf09.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687444980; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=upLk2HLYgqwAW4MQvQEZ09lc7CQdtN33REcZ0u51mAs=; b=UHUdZrg/Nzav96wtKLy20GDa5lBtxO+jGz0x5scqbfd/umTqkP6x9qFMX3fg75p2bOxb14 54nwS9G45nomN7OBC4naru+jqxNDj220TB+i5t0OneHO5U2Jf5cEItW+El6bEbLuNQ/PGr rzcPTAyyCeBKuigXkuEgDv2s829PWNU= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf09.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687444980; a=rsa-sha256; cv=none; b=Z9bw6VwTsMYkioYGE9heTBn41ZAM2fLE/b79sea79aA4VxW0crwTntOaPVUVIPG3zOzVfY ErFR89nRgSzhbtpwnfuOsr5eFbh2eZMwB+CuioDwwz5UimUP7dNk9N/D0LhIdtK24lq5wA 1qdfomicGw3S6lUa6/guTlg/zlactP8= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 22AEF1515; Thu, 22 Jun 2023 07:43:43 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 925DC3F663; Thu, 22 Jun 2023 07:42:56 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 11/14] arm64/mm: Wire up PTE_CONT for user mappings Date: Thu, 22 Jun 2023 15:42:06 +0100 Message-Id: <20230622144210.2623299-12-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230622144210.2623299-1-ryan.roberts@arm.com> References: <20230622144210.2623299-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1A4F4140023 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: jkiem3aszq6sxqy9phmeoi4j6hxy58bi X-HE-Tag: 1687444979-774749 X-HE-Meta: U2FsdGVkX19d6WYND4z92VyXdsBc+xZ3u7RImIvJBlX7pLV2YAKjDgsbTbvfH7d7VWWgIdXDhTbCUNaeCZeMjaJ6B4QFLR+p67gt7jX1TEhunJLammtwLLiX5yCcEvHnMZMPOTNkzyaMISZ62wEVej3/HjuUK0F19OfbCcSahNCETkX5OcQ05WBK3tS4/ivYBQBbJAztneFXEW7nbOw2CpGiSQGlSr9xumQlKALFEXcl1hDuqYz9WRlNVqcvgkHzDM1n23zPtC3QFMp9TP3ZJPca0peaKwJMVkjHzQsvD9fmyZr4TTpcmj7IC/ZB6b1mdTJDyGqyWrpMm5uOepsm7ZIf+SG4LknA/nXxgHHA5ijWRiPnPKkF8rycon9zH/l2xJx8DkG6IPn1WLjjB5tYv/0x0I99vgYyGc4aHXhcQMB2K7G2rYOf4GHN4jwuztG7ViX/+XXpczpucEBSarKfZOmUT8uW1Zi2G/caz2V4lUtv3NVUvmYDB0V1nBh2ASmwwUANGfwkhzhfFqhcEbtU6QzAy0LCKh3TwT5TrO3ZJnPE/7+FaY/eEu62psQmS/4z3hWbvXh/Jae8dXPLDgcnWMEcaEOFsPhj7fbLiVEETekbEJ8PMttExS68+T9MFbuCavxc48mI+pAqtXaO9/sFbcR5SyDOGRCgvX8P/MUmsYKYbJ12rHtYwsMSLtytX0FeAii8SEkLlVg1wj9hHBFkMUfLa3UTvkL32rF4T4o3sVQKoV+UoeYEFtgpifbg4aXJ502QNyJa8QSM0IMrIpHxwBtR7XEPAY/nACFfQXHDcYQRS7PWWmBJFrS81p93DpNSbFkiiwA03bGHVmg5J5uy93oNIWfeQJI68gI9YOU7OjDSURK41hEltQPIsGYb/yhoRmMKkFZIdppk8tuXi4xWowxIN/EtkRaRlTfsaboyDsqnQETmMN69kGp9cxIPyZlnVU8xs5BiUHzy1Ax2g4X a5c8qBQD 3jA6rpjq3lmCD/cHDCnv1lTm4RkCffw81qT4ptMKgkpw3jmy4VMTX6X+yCxs78bFfnCwpBP61SrWlAydlU6qMQKV5Dk1PBySjrTaP4A0pxaxyc+cs9q83SQPvN17jphBa6dGI9AO/kGoFP142mUIfjnu15ZgOfc73GaXEpCykwrndoQSEKdw6wYpya5cuOdNHFKTQK/2XUsfMbk/7cqiZ2msUGE4esMQ6CVzwwVsBFmTBAVu7yGiWUKCboZ3Z/Sx50ALPsEBNoqPhCnK/M9Z+wToQlw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With the ptep API sufficiently refactored, we can now introduce a new "contpte" API layer, which transparently manages the PTE_CONT bit for user mappings. Whenever it detects a set of PTEs that meet the requirements for a contiguous range, the PTEs are re-painted with the PTE_CONT bit. This initial change provides a baseline that can be optimized in future commits. That said, fold/unfold operations (which imply tlb invalidation) are avoided where possible with a few tricks for access/dirty bit management. Write-enable and write-protect modifications are likely non-optimal and likely incure a regression in fork() performance. This will be addressed separately. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 137 ++++++++++++- arch/arm64/mm/Makefile | 3 +- arch/arm64/mm/contpte.c | 334 +++++++++++++++++++++++++++++++ 3 files changed, 466 insertions(+), 8 deletions(-) create mode 100644 arch/arm64/mm/contpte.c diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 31df4d73f9ac..17ea534bc5b0 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1115,6 +1115,71 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); +/* + * The contpte APIs are used to transparently manage the contiguous bit in ptes + * where it is possible and makes sense to do so. The PTE_CONT bit is considered + * a private implementation detail of the public ptep API (see below). + */ +extern void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); +extern void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); +extern pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); +extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); +extern void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr); +extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); +extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); +extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty); + +static inline pte_t *contpte_align_down(pte_t *ptep) +{ + return (pte_t *)(ALIGN_DOWN((unsigned long)ptep >> 3, CONT_PTES) << 3); +} + +static inline bool contpte_is_enabled(struct mm_struct *mm) +{ + /* + * Don't attempt to apply the contig bit to kernel mappings, because + * dynamically adding/removing the contig bit can cause page faults. + * These racing faults are ok for user space, since they get serialized + * on the PTL. But kernel mappings can't tolerate faults. + */ + + return mm != &init_mm; +} + +static inline void contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + /* + * Only bother trying if both the virtual and physical addresses are + * aligned and correspond to the last entry in a contig range. The core + * code mostly modifies ranges from low to high, so this is the likely + * the last modification in the contig range, so a good time to fold. + */ + + bool valign = ((unsigned long)ptep >> 3) % CONT_PTES == CONT_PTES - 1; + bool palign = pte_pfn(pte) % CONT_PTES == CONT_PTES - 1; + + if (contpte_is_enabled(mm) && + pte_present(pte) && !pte_cont(pte) && + valign && palign) + __contpte_try_fold(mm, addr, ptep, pte); +} + +static inline void contpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + if (contpte_is_enabled(mm) && + pte_present(pte) && pte_cont(pte)) + __contpte_try_unfold(mm, addr, ptep, pte); +} + /* * The below functions constitute the public API that arm64 presents to the * core-mm to manipulate PTE entries within the their page tables (or at least @@ -1122,30 +1187,68 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, * versions will automatically and transparently apply the contiguous bit where * it makes sense to do so. Therefore any users that are contig-aware (e.g. * hugetlb, kernel mapper) should NOT use these APIs, but instead use the - * private versions, which are prefixed with double underscore. + * private versions, which are prefixed with double underscore. All of these + * APIs except for ptep_get_lockless() are expected to be called with the PTL + * held. */ #define ptep_get ptep_get static inline pte_t ptep_get(pte_t *ptep) { - return __ptep_get(ptep); + pte_t pte = __ptep_get(ptep); + + if (!pte_present(pte) || !pte_cont(pte)) + return pte; + + return contpte_ptep_get(ptep, pte); +} + +#define ptep_get_lockless ptep_get_lockless +static inline pte_t ptep_get_lockless(pte_t *ptep) +{ + pte_t pte = __ptep_get(ptep); + + if (!pte_present(pte) || !pte_cont(pte)) + return pte; + + return contpte_ptep_get_lockless(ptep); } static inline void set_pte(pte_t *ptep, pte_t pte) { - __set_pte(ptep, pte); + /* + * We don't have the mm or vaddr so cannot unfold or fold contig entries + * (since it requires tlb maintenance). set_pte() is not used in core + * code, so this should never even be called. Regardless do our best to + * service any call and emit a warning if there is any attempt to set a + * pte on top of an existing contig range. + */ + pte_t orig_pte = __ptep_get(ptep); + + WARN_ON_ONCE(pte_present(orig_pte) && pte_cont(orig_pte)); + __set_pte(ptep, pte_mknoncont(pte)); } #define set_ptes set_ptes static inline void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr) { - __set_ptes(mm, addr, ptep, pte, nr); + pte = pte_mknoncont(pte); + + if (!contpte_is_enabled(mm)) + __set_ptes(mm, addr, ptep, pte, nr); + else if (nr == 1) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __set_ptes(mm, addr, ptep, pte, nr); + contpte_try_fold(mm, addr, ptep, pte); + } else + contpte_set_ptes(mm, addr, ptep, pte, nr); } static inline void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); __pte_clear(mm, addr, ptep); } @@ -1153,6 +1256,7 @@ static inline void pte_clear(struct mm_struct *mm, static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); return __ptep_get_and_clear(mm, addr, ptep); } @@ -1160,21 +1264,33 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) { - return __ptep_test_and_clear_young(vma, addr, ptep); + pte_t orig_pte = __ptep_get(ptep); + + if (!pte_present(orig_pte) || !pte_cont(orig_pte)) + return __ptep_test_and_clear_young(vma, addr, ptep); + + return contpte_ptep_test_and_clear_young(vma, addr, ptep); } #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH static inline int ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) { - return __ptep_clear_flush_young(vma, addr, ptep); + pte_t orig_pte = __ptep_get(ptep); + + if (!pte_present(orig_pte) || !pte_cont(orig_pte)) + return __ptep_clear_flush_young(vma, addr, ptep); + + return contpte_ptep_clear_flush_young(vma, addr, ptep); } #define __HAVE_ARCH_PTEP_SET_WRPROTECT static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); __ptep_set_wrprotect(mm, addr, ptep); + contpte_try_fold(mm, addr, ptep, __ptep_get(ptep)); } #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS @@ -1182,7 +1298,14 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty) { - return __ptep_set_access_flags(vma, addr, ptep, entry, dirty); + pte_t orig_pte = __ptep_get(ptep); + + entry = pte_mknoncont(entry); + + if (!pte_present(orig_pte) || !pte_cont(orig_pte)) + return __ptep_set_access_flags(vma, addr, ptep, entry, dirty); + + return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); } #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index dbd1bc95967d..70b6aba09b5d 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -2,7 +2,8 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ cache.o copypage.o flush.o \ ioremap.o mmap.o pgd.o mmu.o \ - context.o proc.o pageattr.o fixmap.o + context.o proc.o pageattr.o fixmap.o \ + contpte.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c new file mode 100644 index 000000000000..e8e4a298fd53 --- /dev/null +++ b/arch/arm64/mm/contpte.c @@ -0,0 +1,334 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2023 ARM Ltd. + */ + +#include +#include + +static void ptep_clear_flush_range(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, int nr) +{ + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + unsigned long start_addr = addr; + int i; + + for (i = 0; i < nr; i++, ptep++, addr += PAGE_SIZE) + __pte_clear(mm, addr, ptep); + + __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); +} + +static bool ptep_any_present(pte_t *ptep, int nr) +{ + int i; + + for (i = 0; i < nr; i++, ptep++) { + if (pte_present(__ptep_get(ptep))) + return true; + } + + return false; +} + +static void contpte_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, bool fold) +{ + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + unsigned long start_addr; + pte_t *start_ptep; + int i; + + start_ptep = ptep = contpte_align_down(ptep); + start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); + pte = pfn_pte(ALIGN_DOWN(pte_pfn(pte), CONT_PTES), pte_pgprot(pte)); + pte = fold ? pte_mkcont(pte) : pte_mknoncont(pte); + + for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) { + pte_t ptent = __ptep_get_and_clear(mm, addr, ptep); + + if (pte_dirty(ptent)) + pte = pte_mkdirty(pte); + + if (pte_young(ptent)) + pte = pte_mkyoung(pte); + } + + __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); + + __set_ptes(mm, start_addr, start_ptep, pte, CONT_PTES); +} + +void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + /* + * We have already checked that the virtual and pysical addresses are + * correctly aligned for a contig mapping in contpte_try_fold() so the + * remaining checks are to ensure that the contig range is fully covered + * by a single folio, and ensure that all the ptes are present with + * contiguous PFNs and matching prots. + */ + + struct page *page = pte_page(pte); + struct folio *folio = page_folio(page); + unsigned long folio_saddr = addr - (page - &folio->page) * PAGE_SIZE; + unsigned long folio_eaddr = folio_saddr + folio_nr_pages(folio) * PAGE_SIZE; + unsigned long cont_saddr = ALIGN_DOWN(addr, CONT_PTE_SIZE); + unsigned long cont_eaddr = cont_saddr + CONT_PTE_SIZE; + unsigned long pfn; + pgprot_t prot; + pte_t subpte; + pte_t *orig_ptep; + int i; + + if (folio_saddr > cont_saddr || folio_eaddr < cont_eaddr) + return; + + pfn = pte_pfn(pte) - ((addr - cont_saddr) >> PAGE_SHIFT); + prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); + orig_ptep = ptep; + ptep = contpte_align_down(ptep); + + for (i = 0; i < CONT_PTES; i++, ptep++, pfn++) { + subpte = __ptep_get(ptep); + subpte = pte_mkold(pte_mkclean(subpte)); + + if (!pte_present(subpte) || + pte_pfn(subpte) != pfn || + pgprot_val(pte_pgprot(subpte)) != pgprot_val(prot)) + return; + } + + contpte_fold(mm, addr, orig_ptep, pte, true); +} + +void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + /* + * We have already checked that the ptes are contiguous in + * contpte_try_unfold(), so we can unfold unconditionally here. + */ + + contpte_fold(mm, addr, ptep, pte, false); +} + +pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte) +{ + /* + * Gather access/dirty bits, which may be populated in any of the ptes + * of the contig range. We are guarranteed to be holding the PTL, so any + * contiguous range cannot be unfolded or otherwise modified under our + * feet. + */ + + pte_t pte; + int i; + + ptep = contpte_align_down(ptep); + + for (i = 0; i < CONT_PTES; i++, ptep++) { + pte = __ptep_get(ptep); + + /* + * Deal with the partial contpte_ptep_get_and_clear_full() case, + * where some of the ptes in the range may be cleared but others + * are still to do. See contpte_ptep_get_and_clear_full(). + */ + if (pte_val(pte) == 0) + continue; + + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + + return orig_pte; +} + +pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) +{ + /* + * Gather access/dirty bits, which may be populated in any of the ptes + * of the contig range. We may not be holding the PTL, so any contiguous + * range may be unfolded/modified/refolded under our feet. + */ + + pte_t orig_pte; + pgprot_t orig_prot; + pte_t *ptep; + unsigned long pfn; + pte_t pte; + pgprot_t prot; + int i; + +retry: + orig_pte = __ptep_get(orig_ptep); + + if (!pte_present(orig_pte) || !pte_cont(orig_pte)) + return orig_pte; + + orig_prot = pte_pgprot(pte_mkold(pte_mkclean(orig_pte))); + ptep = contpte_align_down(orig_ptep); + pfn = pte_pfn(orig_pte) - (orig_ptep - ptep); + + for (i = 0; i < CONT_PTES; i++, ptep++, pfn++) { + pte = __ptep_get(ptep); + prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); + + if (!pte_present(pte) || !pte_cont(pte) || + pte_pfn(pte) != pfn || + pgprot_val(prot) != pgprot_val(orig_prot)) + goto retry; + + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + + return orig_pte; +} + +void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + unsigned long next; + unsigned long end = addr + (nr << PAGE_SHIFT); + unsigned long pfn = pte_pfn(pte); + pgprot_t prot = pte_pgprot(pte); + pte_t orig_pte; + + do { + next = pte_cont_addr_end(addr, end); + nr = (next - addr) >> PAGE_SHIFT; + pte = pfn_pte(pfn, prot); + + if (((addr | next | (pfn << PAGE_SHIFT)) & ~CONT_PTE_MASK) == 0) + pte = pte_mkcont(pte); + else + pte = pte_mknoncont(pte); + + /* + * If operating on a partial contiguous range then we must first + * unfold the contiguous range if it was previously folded. + * Otherwise we could end up with overlapping tlb entries. + */ + if (nr != CONT_PTES) + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + + /* + * If we are replacing ptes that were contiguous or if the new + * ptes are contiguous and any of the ptes being replaced are + * present, we need to clear and flush the range to prevent + * overlapping tlb entries. + */ + orig_pte = __ptep_get(ptep); + if ((pte_present(orig_pte) && pte_cont(orig_pte)) || + (pte_cont(pte) && ptep_any_present(ptep, nr))) + ptep_clear_flush_range(mm, addr, ptep, nr); + + __set_ptes(mm, addr, ptep, pte, nr); + + addr = next; + ptep += nr; + pfn += nr; + + } while (addr != end); +} + +int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + /* + * ptep_clear_flush_young() technically requires us to clear the access + * flag for a _single_ pte. However, the core-mm code actually tracks + * access/dirty per folio, not per page. And since we only create a + * contig range when the range is covered by a single folio, we can get + * away with clearing young for the whole contig range here, so we avoid + * having to unfold. + */ + + int i; + int young = 0; + + ptep = contpte_align_down(ptep); + addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); + + for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) + young |= __ptep_test_and_clear_young(vma, addr, ptep); + + return young; +} + +int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + int young; + + young = contpte_ptep_test_and_clear_young(vma, addr, ptep); + addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); + + if (young) { + /* + * See comment in __ptep_clear_flush_young(); same rationale for + * eliding the trailing DSB applies here. + */ + __flush_tlb_range_nosync(vma, addr, addr + CONT_PTE_SIZE, + PAGE_SIZE, true, 3); + } + + return young; +} + +int contpte_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty) +{ + pte_t orig_pte; + int i; + + /* + * Gather the access/dirty bits for the contiguous range. If nothing has + * changed, its a noop. + */ + orig_pte = ptep_get(ptep); + if (pte_val(orig_pte) == pte_val(entry)) + return 0; + + /* + * We can fix up access/dirty bits without having to unfold/fold the + * contig range. But if the write bit is changing, we need to go through + * the full unfold/fold cycle. + */ + if (pte_write(orig_pte) == pte_write(entry)) { + /* + * No need to flush here; This is always "more permissive" so we + * can only be _adding_ the access or dirty bit. And since the + * tlb can't cache an entry without the AF set and the dirty bit + * is a SW bit, there can be no confusion. For HW access + * management, we technically only need to update the flag on a + * single pte in the range. But for SW access management, we + * need to update all the ptes to prevent extra faults. + */ + ptep = contpte_align_down(ptep); + addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); + + for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) + __ptep_set_access_flags(vma, addr, ptep, entry, 0); + } else { + /* + * No need to flush in __ptep_set_access_flags() because we just + * flushed the whole range in __contpte_try_unfold(). + */ + __contpte_try_unfold(vma->vm_mm, addr, ptep, orig_pte); + __ptep_set_access_flags(vma, addr, ptep, entry, 0); + contpte_try_fold(vma->vm_mm, addr, ptep, entry); + } + + return 1; +} From patchwork Thu Jun 22 14:42:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13289260 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A54B3EB64D8 for ; Thu, 22 Jun 2023 14:43:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9880A8D0010; Thu, 22 Jun 2023 10:43:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 872518D000C; Thu, 22 Jun 2023 10:43:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A0088D0010; Thu, 22 Jun 2023 10:43:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4E36D8D000C for ; Thu, 22 Jun 2023 10:43:05 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C52EC120B88 for ; Thu, 22 Jun 2023 14:43:04 +0000 (UTC) X-FDA: 80930651088.17.FB6AD7C Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf06.hostedemail.com (Postfix) with ESMTP id 0262218000D for ; Thu, 22 Jun 2023 14:43:02 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf06.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687444983; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dsj19i4WO7Xm5RCy72VgkaRoah4zje0Yv8SkvWhAYkk=; b=VsNNGeZlC1MNiTrqGFQlydoImI5PPPsAn4zRx5GZEEq965y52VZ20dMT7oQaWgt6VmmN4E ftRApaGA72SvzItV/8i96IqX7aMq0B57+Ajajyt9PT8WcrD5dNADNtcTT/OJ3+Kih1n/4C TqM8KRTqZg+9rQvpRYz/DR9Pi/O/RcM= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf06.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687444983; a=rsa-sha256; cv=none; b=XTxldbtdVorvV/W4PY5yf0kI4pPMYsl6e2VbNPR8Il3bK2OprMlnOfHrncWeWg0d8ZdIw3 2HfSSA2OzW4bQFu85LOuvdEbQcVHt0kdDZ1Hid9ixu9XDpGLtw/DtR35KrPhvITnPMXHxr Ew1na1hoUTcBAjZLUK6tOHusNkPicPQ= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id F0FCAC14; Thu, 22 Jun 2023 07:43:45 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6C4793F663; Thu, 22 Jun 2023 07:42:59 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 12/14] arm64/mm: Add ptep_get_and_clear_full() to optimize process teardown Date: Thu, 22 Jun 2023 15:42:07 +0100 Message-Id: <20230622144210.2623299-13-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230622144210.2623299-1-ryan.roberts@arm.com> References: <20230622144210.2623299-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0262218000D X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: m749xbdy88waju8mcsr9rjxd4tf5d8e8 X-HE-Tag: 1687444982-501742 X-HE-Meta: U2FsdGVkX19j9FBaIhV4XfOrWKdLzdimEM6FYC7xGaZT8SbkMM5zEWPA7yL5E8Zp1uAShEi93Z9YcaQA9rPdAm4qSfVJUDJXqmCyPLRNaq25UMUOH3Gs7JB79pR3HGgLkgkPEpEEXqLvKu6EsHqhwMUcL5jiI214cvFTz1/xpwmHOYB5O02STgg8A5tkmCoKsk39p/jy3KgKGeX8npyWhK09WukWsAhuMLjOeA0GSI7XU0Bwrslr8TaT1pkO5UaMBskx+A9SfQrhH5hkeGtdWvO2g0KPWwE1ExRhdY6a3CJvc/3xy/phvhVWEUT3V/H7Sn0FFB4yJ+ibrn3OUJUZdol1bjC3nWmgqmk4/TGgo1K8kKDytR1QUtcszcYss40MTmwx67eEB5EMY1DLMcACVs7WbvrPvZZeARcdpyYLaOt46R/PhQ6+hTYUE3nFqqOYsmAPHfwLzBW8dUNMM4btCwnD+gYhzpqlMrhvXSI0Folj1VMUvfu4l5XfSAinm6KfsF0ClnqpvXiaQc5zx0aOINvy9bU3fgcsvBaHQGwVP4k3Kky+i72v9zhwieqetxVIyR8HF0UO9m8QhzJRlGN7s1MbzQW16+B8P9aJ4FEus+PXZIL2AJEfVzJQb+YjQZlekhbu9S4Rq+PutuszCYH19+8BJ4X8ZjF1gWgCQklTXBflfcEGds/1elGLaAYsLKal1sC7TemjOgXtkur77TvWlOEDkFKKAjnHNTw+CYBF2oxCu+RG1d/4otm293X/sEChofzPNPNEqDq+pIJfUgbe8ASeuUnI0zzPmjjBn0HHJgLyo6nYV0GqrEQSBrq1Po/SqyMe0ulgEfvqun6EVqsCYGgHdLtTtKerOjWOv+oee2lFZUMKhWCYjdHg+5NgUQrj7jMlT5/btLOB56Ux5ad35yupyxAF1L4pXU33PprBdu9fHOSr2AAj5z7DZPHkpbsIFJQqA9DC3weojzabk4Q 3lF/bFrT lwq8b3L1fLI4JBYWkfQ2Jbt99S/wpmBtCbHPqHP5VOEAk3x2HmpcGUD9WTav6tgoLlbN3RFKaxdSU059slGcu7zIc1eHMi6kOn1a5g8tDorcM0yopFOdlLeTolxGIfB9v/npdUsjCuBZJSf2OVvNQKzpJTRf7T9IZDTVhcH6/TbcplFaWfScSwsYhbvEoefZWbeB87Yj6L97zJ6mGK2Vn3V7MRW7y8Sy/qZ5q1Aqs1jO3mkgiqzCBCcmHfwRHpIs2r2ikeV1ccA/Ez2kevuhvA+z0ZQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: ptep_get_and_clear_full() adds a 'full' parameter which is not present for the fallback ptep_get_and_clear() function. 'full' is set to 1 when a full address space teardown is in progress. We use this information to optimize arm64_sys_exit_group() by avoiding unfolding (and therefore tlbi) contiguous ranges. Instead we just clear the PTE but allow all the contiguous neighbours to keep their contig bit set, because we know we are about to clear the rest too. Before this optimization, the cost of arm64_sys_exit_group() exploded to 32x what it was before PTE_CONT support was wired up, when compiling the kernel. With this optimization in place, we are back down to the original cost. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 18 ++++++++- arch/arm64/mm/contpte.c | 68 ++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 17ea534bc5b0..5963da651da7 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1128,6 +1128,8 @@ extern pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); extern void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); +extern pte_t contpte_ptep_get_and_clear_full(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, @@ -1252,12 +1254,24 @@ static inline void pte_clear(struct mm_struct *mm, __pte_clear(mm, addr, ptep); } +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR_FULL +static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, int full) +{ + pte_t orig_pte = __ptep_get(ptep); + + if (!pte_present(orig_pte) || !pte_cont(orig_pte) || !full) { + contpte_try_unfold(mm, addr, ptep, orig_pte); + return __ptep_get_and_clear(mm, addr, ptep); + } else + return contpte_ptep_get_and_clear_full(mm, addr, ptep); +} + #define __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - return __ptep_get_and_clear(mm, addr, ptep); + return ptep_get_and_clear_full(mm, addr, ptep, 0); } #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index e8e4a298fd53..0b585d1c4c94 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -241,6 +241,74 @@ void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, } while (addr != end); } +pte_t contpte_ptep_get_and_clear_full(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + /* + * When doing a full address space teardown, we can avoid unfolding the + * contiguous range, and therefore avoid the associated tlbi. Instead, + * just clear the pte. The caller is promising to call us for every pte, + * so every pte in the range will be cleared by the time the tlbi is + * issued. + * + * However, this approach will leave the ptes in an inconsistent state + * until ptep_get_and_clear_full() has been called for every pte in the + * range. This could cause ptep_get() to fail to return the correct + * access/dirty bits, if ptep_get() calls are interleved with + * ptep_get_and_clear_full() (which they are). Solve this by copying the + * access/dirty bits to every pte in the range so that ptep_get() still + * sees them if we have already cleared pte that the hw chose to update. + * Note that a full teardown will only happen when the process is + * exiting, so we do not expect anymore accesses and therefore no more + * access/dirty bit updates, so there is no race here. + */ + + pte_t *orig_ptep = ptep; + pte_t pte; + bool flags_propagated = false; + bool dirty = false; + bool young = false; + int i; + + /* First, gather access and dirty bits. */ + ptep = contpte_align_down(orig_ptep); + for (i = 0; i < CONT_PTES; i++, ptep++) { + pte = __ptep_get(ptep); + + /* + * If we find a zeroed PTE, contpte_ptep_get_and_clear_full() + * must have already been called for it, so we have already + * propagated the flags to the other ptes. + */ + if (pte_val(pte) == 0) { + flags_propagated = true; + break; + } + + if (pte_dirty(pte)) + dirty = true; + + if (pte_young(pte)) + young = true; + } + + /* Now copy the access and dirty bits into each pte in the range. */ + if (!flags_propagated) { + ptep = contpte_align_down(orig_ptep); + for (i = 0; i < CONT_PTES; i++, ptep++) { + pte = __ptep_get(ptep); + + if (dirty) + pte = pte_mkdirty(pte); + + if (young) + pte = pte_mkyoung(pte); + } + } + + return __ptep_get_and_clear(mm, addr, orig_ptep); +} + int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) { From patchwork Thu Jun 22 14:42:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13289261 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DBA42EB64DA for ; Thu, 22 Jun 2023 14:43:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F6F28D0011; Thu, 22 Jun 2023 10:43:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 27EBA8D000C; Thu, 22 Jun 2023 10:43:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F8E48D0011; Thu, 22 Jun 2023 10:43:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id EEB388D000C for ; Thu, 22 Jun 2023 10:43:07 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B5CF040BE9 for ; Thu, 22 Jun 2023 14:43:07 +0000 (UTC) X-FDA: 80930651214.15.8335378 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf16.hostedemail.com (Postfix) with ESMTP id C0493180014 for ; Thu, 22 Jun 2023 14:43:05 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687444985; a=rsa-sha256; cv=none; b=YmRRW20O/RhwOxQqhTp9GHF9o74ZpIp+gKALB9CFOECZRDCWc8yV9TDlGBosgcDQHI7ZxN 5ib2K6Q3NKwgeq8KLVwqDfZ7QQzHvV23REaS4quZdpOuxAvx0KrnLTLLJ6fXS7peRL9ANH 7H1hx1ldN5x0uaK6tw8gB2RwfSzknNU= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687444985; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1U98Qo4K46blocWc+Tr5yK1y0iUP+/AjAftybug9D74=; b=eoSS0WojXfTiz0Id0kBCn6G26Uoj7XzdL3cXtIHVbd5zVIPfqZiu7UwbUEi2bYX8vA1pzY Xahc14NVJ5Kz84f+fCwa2g7OLx8ZM4vfmW3uvc68Vy4Ot9k1Q0S6kdgwibkqjFUEd6ngO1 pf2oTUsZPKydUgqMPpNVTgPnnOuS6Xo= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CB0721516; Thu, 22 Jun 2023 07:43:48 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 46D553F663; Thu, 22 Jun 2023 07:43:02 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 13/14] mm: Batch-copy PTE ranges during fork() Date: Thu, 22 Jun 2023 15:42:08 +0100 Message-Id: <20230622144210.2623299-14-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230622144210.2623299-1-ryan.roberts@arm.com> References: <20230622144210.2623299-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: C0493180014 X-Stat-Signature: peaok9amf6es56axtx1mdnc8tjzupycm X-HE-Tag: 1687444985-488209 X-HE-Meta: U2FsdGVkX19F+c+6jkyOlQwGO5ddcopPkKZy8c7T6fEwbpUbmvPi5TIrODgSlFcnWyTpX+EgUIO6+ukdu1HLPWK1SN9byotqOI+FdMPEzbXJ8ssnqV2FSe83/EdMoQZ8jqx1+NplfHBNKdCeRuLx0eK2wKWiKGwho5bu/Amlz0zAGVshrwCT8YAA11mhCuMD428pRh9no6XHTlaI69w0pA9lTsVHKsLEhlCYLJkm9CjN4Wi+UBHjVFHiLAwd7sIbM+01igeUE+DvxuONErPXklx3lwkgz9SbyKpGfJ8UsIAuWPRsjRtjJnkWqMFsL3VmPgparzeLR6lc6r/CSzTI7mbmy45rCjNyo2X6JKxdBH2hiNenThbfnEMMpdi+uSQTz+ujFquxIx+DLnuJogm+nhApOKZ8nX3oFUfJ0V3RzK+fvXBWjYId3USEnHm2subNAeAVdE+85jXvI0fdp5tDVjM9mAIGszVQfjexaJreCIDQgchXGfiUsyg76lus5PCXni2HQ46+p51HSp2cRzueqAVASkLeT3NcKyrihWQTADphABQPNFVaY8bsafFsnZarPp3m+HXygr4uF1hSoB0Hyl68a8C0zpFUfPw7dPpoPaqFDO9FAbsMlum8PTezPJrJx9hDCfor2Wzmpf1grGbaBv1oqDELrWbojQX8otz61yNYuT1OR6EOB8FkyVZC3tidDb7MLsXSFD1YUPvdJmJ+d7IsQOEFHCt8xNU4Vah3FnYJ/RaajMceBZnOv+JR3GaFcYA3xbYH+3KqtgOlmGmC8LwLtBjs8DArk+LzVW8/LjGC9AdvjyjKbKDR7W0+9vnt1WzfuetrgBpWS3RZTy59zQaI6U4wWtZ5mOzPIFZkJ8G/zw6akkISf6h3liJ5vPPTSASw0YK/CsK4J3kzm6prwY4jjPNeUTDV2NjXrKNRMBAKdPdEq0YrqThIjxCbFUf1gVyssDNreS+cWJNZPNJ ylv8SzzF ISHnXQNgtGmmBqOEc4pswu2+9gOjagOHfU3QXU+5plKSGWfNp61BtkepMIahKiE3ahPUzRuBM7uS6yTLHBxbDP6UJrL1wPXRYNhMJk2/MKDmFj7SKOTim9ck8pRkvqsHHWWTtn2zzInCptlJfnF2YUoN+pE4TBrBxi615e6rwSF3vsJ54vkst9aPhfoLyWZhOdsIkptrC5WOnUnNPrAa1XYI+SVJlDV1/xej2eHBJADwPAoE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert copy_pte_range() to copy a set of ptes that map a physically contiguous block of memory in a batch. This will likely improve performance by a tiny amount due to batching the folio reference count management and calling set_ptes() rather than making individual calls to set_pte_at(). However, the primary motivation for this change is to reduce the number of tlb maintenance operations that the arm64 backend has to perform during fork, now that it transparently supports the "contiguous bit" in its ptes. By write-protecting the parent using the new ptep_set_wrprotects() (note the 's' at the end) function, the backend can avoid having to unfold contig ranges of PTEs, which is expensive, when all ptes in the range are being write-protected. Similarly, by using set_ptes() rather than set_pte_at() to set up ptes in the child, the backend does not need to fold a contiguous range once they are all populated - they can be initially populated as a contiguous range in the first place. This change addresses the core-mm refactoring only, and introduces ptep_set_wrprotects() with a default implementation that calls ptep_set_wrprotect() for each pte in the range. A separate change will implement ptep_set_wrprotects() in the arm64 backend to realize the performance improvement. Signed-off-by: Ryan Roberts --- include/linux/pgtable.h | 13 ++++ mm/memory.c | 149 +++++++++++++++++++++++++++++++--------- 2 files changed, 128 insertions(+), 34 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index a661a17173fa..6a7b28d520de 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -547,6 +547,19 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres } #endif +#ifndef ptep_set_wrprotects +struct mm_struct; +static inline void ptep_set_wrprotects(struct mm_struct *mm, + unsigned long address, pte_t *ptep, + unsigned int nr) +{ + unsigned int i; + + for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) + ptep_set_wrprotect(mm, address, ptep); +} +#endif + /* * On some architectures hardware does not set page access bit when accessing * memory page, it is responsibility of software setting this bit. It brings diff --git a/mm/memory.c b/mm/memory.c index fb30f7523550..9a041cc31c74 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -911,57 +911,126 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma /* Uffd-wp needs to be delivered to dest pte as well */ pte = pte_mkuffd_wp(pte); set_pte_at(dst_vma->vm_mm, addr, dst_pte, pte); - return 0; + return 1; +} + +static inline unsigned long page_addr(struct page *page, + struct page *anchor, unsigned long anchor_addr) +{ + unsigned long offset; + unsigned long addr; + + offset = (page_to_pfn(page) - page_to_pfn(anchor)) << PAGE_SHIFT; + addr = anchor_addr + offset; + + if (anchor > page) { + if (addr > anchor_addr) + return 0; + } else { + if (addr < anchor_addr) + return ULONG_MAX; + } + + return addr; +} + +static int calc_anon_folio_map_pgcount(struct folio *folio, + struct page *page, pte_t *pte, + unsigned long addr, unsigned long end) +{ + pte_t ptent; + int floops; + int i; + unsigned long pfn; + + end = min(page_addr(&folio->page + folio_nr_pages(folio), page, addr), + end); + floops = (end - addr) >> PAGE_SHIFT; + pfn = page_to_pfn(page); + pfn++; + pte++; + + for (i = 1; i < floops; i++) { + ptent = ptep_get(pte); + + if (!pte_present(ptent) || + pte_pfn(ptent) != pfn) { + return i; + } + + pfn++; + pte++; + } + + return floops; } /* - * Copy one pte. Returns 0 if succeeded, or -EAGAIN if one preallocated page - * is required to copy this pte. + * Copy set of contiguous ptes. Returns number of ptes copied if succeeded + * (always gte 1), or -EAGAIN if one preallocated page is required to copy the + * first pte. */ static inline int -copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, - pte_t *dst_pte, pte_t *src_pte, unsigned long addr, int *rss, - struct folio **prealloc) +copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, + pte_t *dst_pte, pte_t *src_pte, + unsigned long addr, unsigned long end, + int *rss, struct folio **prealloc) { struct mm_struct *src_mm = src_vma->vm_mm; unsigned long vm_flags = src_vma->vm_flags; pte_t pte = ptep_get(src_pte); struct page *page; struct folio *folio; + bool anon; + int nr; + int i; page = vm_normal_page(src_vma, addr, pte); - if (page) + if (page) { folio = page_folio(page); - if (page && folio_test_anon(folio)) { - /* - * If this page may have been pinned by the parent process, - * copy the page immediately for the child so that we'll always - * guarantee the pinned page won't be randomly replaced in the - * future. - */ - folio_get(folio); - if (unlikely(page_try_dup_anon_rmap(page, false, src_vma))) { - /* Page may be pinned, we have to copy. */ - folio_put(folio); - return copy_present_page(dst_vma, src_vma, dst_pte, src_pte, - addr, rss, prealloc, page); + anon = folio_test_anon(folio); + nr = calc_anon_folio_map_pgcount(folio, page, src_pte, addr, end); + + for (i = 0; i < nr; i++, page++) { + if (anon) { + /* + * If this page may have been pinned by the + * parent process, copy the page immediately for + * the child so that we'll always guarantee the + * pinned page won't be randomly replaced in the + * future. + */ + if (unlikely(page_try_dup_anon_rmap( + page, false, src_vma))) { + if (i != 0) + break; + /* Page may be pinned, we have to copy. */ + return copy_present_page(dst_vma, src_vma, + dst_pte, src_pte, + addr, rss, + prealloc, page); + } + rss[MM_ANONPAGES]++; + VM_BUG_ON(PageAnonExclusive(page)); + } else { + page_dup_file_rmap(page, false); + rss[mm_counter_file(page)]++; + } } - rss[MM_ANONPAGES]++; - } else if (page) { - folio_get(folio); - page_dup_file_rmap(page, false); - rss[mm_counter_file(page)]++; - } + + nr = i; + folio_ref_add(folio, nr); + } else + nr = 1; /* * If it's a COW mapping, write protect it both * in the parent and the child */ if (is_cow_mapping(vm_flags) && pte_write(pte)) { - ptep_set_wrprotect(src_mm, addr, src_pte); + ptep_set_wrprotects(src_mm, addr, src_pte, nr); pte = pte_wrprotect(pte); } - VM_BUG_ON(page && folio_test_anon(folio) && PageAnonExclusive(page)); /* * If it's a shared mapping, mark it clean in @@ -974,8 +1043,8 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, if (!userfaultfd_wp(dst_vma)) pte = pte_clear_uffd_wp(pte); - set_pte_at(dst_vma->vm_mm, addr, dst_pte, pte); - return 0; + set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr); + return nr; } static inline struct folio *page_copy_prealloc(struct mm_struct *src_mm, @@ -1065,15 +1134,28 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, */ WARN_ON_ONCE(ret != -ENOENT); } - /* copy_present_pte() will clear `*prealloc' if consumed */ - ret = copy_present_pte(dst_vma, src_vma, dst_pte, src_pte, - addr, rss, &prealloc); + /* copy_present_ptes() will clear `*prealloc' if consumed */ + ret = copy_present_ptes(dst_vma, src_vma, dst_pte, src_pte, + addr, end, rss, &prealloc); + /* * If we need a pre-allocated page for this pte, drop the * locks, allocate, and try again. */ if (unlikely(ret == -EAGAIN)) break; + + /* + * Positive return value is the number of ptes copied. + */ + VM_WARN_ON_ONCE(ret < 1); + progress += 8 * ret; + ret--; + dst_pte += ret; + src_pte += ret; + addr += ret << PAGE_SHIFT; + ret = 0; + if (unlikely(prealloc)) { /* * pre-alloc page cannot be reused by next time so as @@ -1084,7 +1166,6 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, folio_put(prealloc); prealloc = NULL; } - progress += 8; } while (dst_pte++, src_pte++, addr += PAGE_SIZE, addr != end); arch_leave_lazy_mmu_mode(); From patchwork Thu Jun 22 14:42:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13289262 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E1A6C0015E for ; Thu, 22 Jun 2023 14:43:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B6098D0012; Thu, 22 Jun 2023 10:43:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 53FAF8D000C; Thu, 22 Jun 2023 10:43:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2CD048D0012; Thu, 22 Jun 2023 10:43:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 138D78D000C for ; Thu, 22 Jun 2023 10:43:11 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id DAC51AFD7D for ; Thu, 22 Jun 2023 14:43:10 +0000 (UTC) X-FDA: 80930651340.16.1C1CB5C Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf07.hostedemail.com (Postfix) with ESMTP id D7F6940025 for ; Thu, 22 Jun 2023 14:43:08 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf07.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687444989; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3VXUsl66xsyGZxd0YSBpF82rlW1iasetGIFfl5S2d8w=; b=Z4tKw1g82LUB2WSDQYkDLqFzVsPdwJ0fezsNsXKPyTwAdvO/bnJ7P8BRgPf3XBBarMjSaA Engk3AsQpkaF/mCuKuc82pvFg3COtl5HD79rTOqAZ+Unodev70OkJikzJMU7jdkrX/ADvK 8Rp0PRPtU5tof8iQTP7zvjpV2GIWPvA= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf07.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687444989; a=rsa-sha256; cv=none; b=Kr+d+zNgRbTR8eJcAYVOo2/EKn5dthvLdc/jnhmEvCpA1UMVwogze1JRZBTsBwLftpQSj3 deie2xxOAkVuX99uz72UxABuryKOmxCL2wA/8KpBrpryUURWQur+crgGVLBgB4w1LxKsqM 3bR2YzOgPe3t+DNL5JE6krzdU9E5888= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A45B71042; Thu, 22 Jun 2023 07:43:51 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 207E63F663; Thu, 22 Jun 2023 07:43:05 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 14/14] arm64/mm: Implement ptep_set_wrprotects() to optimize fork() Date: Thu, 22 Jun 2023 15:42:09 +0100 Message-Id: <20230622144210.2623299-15-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230622144210.2623299-1-ryan.roberts@arm.com> References: <20230622144210.2623299-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 1h7fygr8qbsf9qsywzafk5ymedd7fzb7 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D7F6940025 X-HE-Tag: 1687444988-105489 X-HE-Meta: U2FsdGVkX18nK/4anJd9DfoV8UrSMtHMrG2bs5Xc12HwaIBXifVi7RB4eJSjxdciA20cBOKPjJqLAZESthr1CR7FAxCUM6crFpdAUxjDXPI2v7hnlSZ33ip3/lXKdO40bxaSi7dZh+RRznMbnTOdvt8uvKdcqaokz37B5tfeVPE6BmNbr2+RS7rpj2jPk+PBeGHL+zTm9mrZ8Aq6u6jeZQEWM5t9g4Bn0RVWGkCKHmTaEKHVxryPncebzEn1QrGslRGybo6MgJA9N5+Ws98+6PqCVdc5mNT5H8pWoLPpCUJc1DE7lFHrqpCVEoscyCvXaqApCnLveUJUTswTD34oMyMZUtvXx5dloBhLIKqVHADKWWL5PqmsLM547DIr3qoOG6ntVsItm5Vkjm2OKXQ8EDaPNAEmwYOCEa//iCINeD3X8nIVBipYjQl5A13IdPFZlwuNks1x+tW2GDdWoM0+uVBcIEYX9bhtJ6YmH/8zM5RiwkmxFgTpd7HMCg5MJT1tRTEW5ExJRCE9tx+xUhv5p5zuxuxpiD0AX0dNiRxcD6Ay98Vw8c+pM4tIo6Edn9E3vxB9bnYBfUlXIFEVM5CtIkPmOz/nK+9lPeCayxCzoazoFYCnakivfGA+JKRA/0U1G7Ul1tqbpye6rw3zt4wvkZScDz0TtDRO6I7jRIjf57g6WfeBI8F4ydPRcd40uP26e+KHSnZuQCcuD0Szyr0mAPPH9ebDYosX7jQrqDxnVHCd1ZJsK8gr3nDF+fj9Xg4tSVRXejRIKKqNhJrMI3/EnZKy+mZ624OKGU+IJsnA+wP7ln2UPfMbatqf5S5gffyPPvo/XH0F5IDIjl8xgLmaqqdKWVE5/1XzvdKPZCIDPX/diGT5kvIqyP+ez6vP7j6vRIbG2DoyLBF5cp/GBLC987bbI+RYQHDDzptWF8qG+ZU16+j+fe7xz/RT6q+7Xdaxgtq+LeULykXBE8RHK51 GckIqT9v 87bOpVWX028AnBzJjwvdXUrLvmKR0lwbk4bntb8RuJCPAAdPUqCI709i09J/pwfSbl08R7FMBSNA1kv6muqYMaoh8QM69H+ae5O6zrJL5VJVn49kwtkc6ebtXCQg+tLFfIJMTZNp9XrdmGc2Jzd/7w+cAeGIdbW6SPXVYLZPGc68NO9kTYybLQT7RFUo10YrjmWkOmZWYF2euE59ZOX9lm/emFWeYyagpK3JbRZh0sD+21WflE4L46jFDs8635u6pT0XKpLvWjTd5K5v8ZGq0kS/+sg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With the core-mm changes in place to batch-copy ptes during fork, we can take advantage of this in arm64 to greatly reduce the number of tlbis we have to issue, and recover the lost fork performance incured when adding support for transparent contiguous ptes. If we are write-protecting a whole contig range, we can apply the write-protection to the whole range and know that it won't change whether the range should have the contiguous bit set or not. For ranges smaller than the contig range, we will still have to unfold, apply the write-protection, then fold if the change now means the range is foldable. Performance tested with the following test written for the will-it-scale framework: ------- char *testcase_description = "fork and exit"; void testcase(unsigned long long *iterations, unsigned long nr) { int pid; char *mem; mem = malloc(SZ_128M); assert(mem); memset(mem, 1, SZ_128M); while (1) { pid = fork(); assert(pid >= 0); if (!pid) exit(0); waitpid(pid, NULL, 0); (*iterations)++; } } ------- I see huge performance regression when PTE_CONT support was added, then the regression is mostly fixed with the addition of this change. The following shows regression relative to before PTE_CONT was enabled (bigger negative value is bigger regression): | cpus | before opt | after opt | |-------:|-------------:|------------:| | 1 | -10.4% | -5.2% | | 8 | -15.4% | -3.5% | | 16 | -38.7% | -3.7% | | 24 | -57.0% | -4.4% | | 32 | -65.8% | -5.4% | Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 48 ++++++++++++++++++++++---------- arch/arm64/mm/contpte.c | 41 +++++++++++++++++++++++++++ arch/arm64/mm/hugetlbpage.c | 2 +- 3 files changed, 75 insertions(+), 16 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 5963da651da7..479a9e5ab756 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -963,21 +963,25 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ /* - * __ptep_set_wrprotect - mark read-only while trasferring potential hardware + * __ptep_set_wrprotects - mark read-only while trasferring potential hardware * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. */ -static inline void __ptep_set_wrprotect(struct mm_struct *mm, - unsigned long address, pte_t *ptep) +static inline void __ptep_set_wrprotects(struct mm_struct *mm, + unsigned long address, pte_t *ptep, + unsigned int nr) { pte_t old_pte, pte; - - pte = __ptep_get(ptep); - do { - old_pte = pte; - pte = pte_wrprotect(pte); - pte_val(pte) = cmpxchg_relaxed(&pte_val(*ptep), - pte_val(old_pte), pte_val(pte)); - } while (pte_val(pte) != pte_val(old_pte)); + unsigned int i; + + for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) { + pte = __ptep_get(ptep); + do { + old_pte = pte; + pte = pte_wrprotect(pte); + pte_val(pte) = cmpxchg_relaxed(&pte_val(*ptep), + pte_val(old_pte), pte_val(pte)); + } while (pte_val(pte) != pte_val(old_pte)); + } } #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -985,7 +989,7 @@ static inline void __ptep_set_wrprotect(struct mm_struct *mm, static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long address, pmd_t *pmdp) { - __ptep_set_wrprotect(mm, address, (pte_t *)pmdp); + __ptep_set_wrprotects(mm, address, (pte_t *)pmdp, 1); } #define pmdp_establish pmdp_establish @@ -1134,6 +1138,8 @@ extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +extern void contpte_set_wrprotects(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr); extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); @@ -1298,13 +1304,25 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, return contpte_ptep_clear_flush_young(vma, addr, ptep); } +#define ptep_set_wrprotects ptep_set_wrprotects +static inline void ptep_set_wrprotects(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + if (!contpte_is_enabled(mm)) + __ptep_set_wrprotects(mm, addr, ptep, nr); + else if (nr == 1) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __ptep_set_wrprotects(mm, addr, ptep, 1); + contpte_try_fold(mm, addr, ptep, __ptep_get(ptep)); + } else + contpte_set_wrprotects(mm, addr, ptep, nr); +} + #define __HAVE_ARCH_PTEP_SET_WRPROTECT static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - __ptep_set_wrprotect(mm, addr, ptep); - contpte_try_fold(mm, addr, ptep, __ptep_get(ptep)); + ptep_set_wrprotects(mm, addr, ptep, 1); } #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 0b585d1c4c94..4f666697547d 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -353,6 +353,47 @@ int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, return young; } +void contpte_set_wrprotects(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + unsigned long next; + unsigned long end = addr + (nr << PAGE_SHIFT); + + do { + next = pte_cont_addr_end(addr, end); + nr = (next - addr) >> PAGE_SHIFT; + + /* + * If wrprotecting an entire contig range, we can avoid + * unfolding. Just set wrprotect and wait for the later + * mmu_gather flush to invalidate the tlb. Until the flush, the + * page may or may not be wrprotected. After the flush, it is + * guarranteed wrprotected. If its a partial range though, we + * must unfold, because we can't have a case where CONT_PTE is + * set but wrprotect applies to a subset of the PTEs; this would + * cause it to continue to be unpredictable after the flush. + */ + if (nr != CONT_PTES) + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + + __ptep_set_wrprotects(mm, addr, ptep, nr); + + addr = next; + ptep += nr; + + /* + * If applying to a partial contig range, the change could have + * made the range foldable. Use the last pte in the range we + * just set for comparison, since contpte_try_fold() only + * triggers when acting on the last pte in the contig range. + */ + if (nr != CONT_PTES) + contpte_try_fold(mm, addr - PAGE_SIZE, ptep - 1, + __ptep_get(ptep - 1)); + + } while (addr != end); +} + int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 82b2036dbe2f..ecf7bfa761c3 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -511,7 +511,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pte_t pte; if (!pte_cont(__ptep_get(ptep))) { - __ptep_set_wrprotect(mm, addr, ptep); + __ptep_set_wrprotects(mm, addr, ptep, 1); return; }