From patchwork Fri Apr 11 09:16:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047916 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46350C36010 for ; Fri, 11 Apr 2025 09:18:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DAE5B2801A7; Fri, 11 Apr 2025 05:18:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D5C6428019B; Fri, 11 Apr 2025 05:18:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD4262801A7; Fri, 11 Apr 2025 05:18:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9A82928019B for ; Fri, 11 Apr 2025 05:18:12 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 43F0F82281 for ; Fri, 11 Apr 2025 09:18:13 +0000 (UTC) X-FDA: 83321211666.25.052D4D2 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf21.hostedemail.com (Postfix) with ESMTP id A5E511C0007 for ; Fri, 11 Apr 2025 09:18:11 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744363091; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LvMNmUJw5YHbXOAt8Uq/tq9yqI5eyvYGL9Ei7Fvb7sQ=; b=7T1KCVE9Id6oOE3xzxEEADOHpAekEfeoeGGEGuTR4MbLwkDrCW1r8chLL2AV0zrMdeY6vh fz62dvaemFpeBOX2El3K1R18dEdsoIm72YaAuaZl6u8SfO9oALV73zMAMCZ3xeaSZ+SLR8 QL64VZFPF1sHxWvSumgu2x9xp2ilgA0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744363091; a=rsa-sha256; cv=none; b=WkbgaaSx4QJNtSqDm3RJfki6274PvQ0387ETIbYymOFOP+MIyCGF6nbh9Z82XNbJT4cVZA a7w7qluw4JRPOUao0eMih7YSdm79x0bEzWLzzIRgdTqvMQPM9eZw0IB0RM8wyGiUdCF8qM qJbUZrFnpVJSjWt2I3Uxo6X9aIUWZeY= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A04EC1596; Fri, 11 Apr 2025 02:18:10 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 1A69D3F6A8; Fri, 11 Apr 2025 02:18:06 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 18/18] arm64: mm: Batch kpkeys level switches Date: Fri, 11 Apr 2025 10:16:31 +0100 Message-ID: <20250411091631.954228-19-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-Stat-Signature: 5bdbre3nwjgdspxi3qiausdra3q6cfuw X-Rspam-User: X-Rspamd-Queue-Id: A5E511C0007 X-Rspamd-Server: rspam08 X-HE-Tag: 1744363091-79388 X-HE-Meta: U2FsdGVkX1+HtKM16tMBrtnK9CJKpiP1anX3DPJpycdj1x4FC96CMd28DAjhF5DhEiAgU8Rctoqc3pBeYfRUQHHfw3wVpq916fjGLr8Hx4Z0LQTPDRGPrZ3KxbnOcAbMlry+A0t2VQ8V/zGXMeGfE3PJxL/6aSqA54B69S732HuayBy4iJxSWx9aQ8e5e7+ZVAGr4FtzB9P8Y40eB8yQP0boVCmMLRyzxz5s5e5j8ncC/cJQUMqg0GFCoArzlA8AL22GDQ77HZ+hItDJ+yvayXFuTd+Of5RJ6iEzB7Rz8RldrLGnA+UOPy9z685NNWnbsu/BimLq27304KehR9Q2JtHB4wjgvSzOkCYR9AtQRostus85B5QyKph9KTjnVjKp768v6yrNje6lAtE5U1i8MuqEYfXDZuivq7iPkhctfkD7YqjjMNqW9WPBH8oCpJDWI5p3MZqWII+evoTtMkBQd558yP/dmOaR9SXYUjIek1vEtgMAencbVeSalwIJ7HDCe38TjYHkOYKEOgVz2zm4EPuj+b9qV9QhGbAlxUZoRQxJuedN380gc5KJOh8FtZ66RZNK98PrrteZh3aMv5grRIRtekOE615P2V/OwfVwWd2j5rgTdZ+GmphLpCy4FSjvLHpsnWZYu6yuXUme+nXo87ptFWlwLuYyT9uyUN194Ho7iYWUm51ozRbHZAxAMKOWfcNqo530CUs0AC8hPb9Bf9bYip4wF4a3d4jBpJjHl02ef+a1q1h16Nlo2wg/7vcKIB+xvRYImkJejYdOsRnuG+r420hOUDfQW9oja7NNgYRaVm3S+phGjZJRUdsi7NzFj0V1H4mYLxIW6fA5I048UxkqRV+OIy1PE4kPqF5QgoS62nnDI7bZsLxKuh4g+UAlEyO2a0mUIhDnHJ8VZOkavCbmKPypXADyEHI+j2S4MA3TThvvopFIVmPIIN5+Fd7bXjNbcd3DSYF6SznvRkj D806cpyr 8JZhRfK/mbRVRZ7LRUxRnrrzwtzVTXa03ToQtv1DqW/eJAOW88+xREFGC/aFneZQFv9rDAnkldSv/9GsErZ3dOV68WaBrROtupqfRTcYvvUkAYq3XhnG/bIo1MT7FIBbQX8DHBWKg8YFSWnwp8djqEU7jLE/9PMrDByIiSCaN40QEpZ4KWm+miQpe0g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The kpkeys_hardened_pgtables feature currently switches kpkeys level in every helper that writes to page tables, such as set_pte(). With kpkeys implemented using POE, this entails a pair of ISBs whenever such helper is called. A simple way to reduce this overhead is to make use of the lazy_mmu mode, which has recently been adopted on arm64 to batch barriers (DSB/ISB) when updating kernel pgtables [1]. Reusing the TIF_LAZY_MMU flag introduced by this series, we amend the kpkeys_hardened_pgtables guard so that no level switch (i.e. POR_EL1 update) is issued while that flag is set. Instead, we switch to KPKEYS_LVL_PGTABLES when entering lazy_mmu mode, and restore the previous level when exiting it. Restoring the previous kpkeys level requires storing the original value of POR_EL1 somewhere. This is a full 64-bit value so we cannot simply use a TIF flag, but since lazy_mmu sections cannot nest, some sort of thread-local variable would do the trick. There is no straightforward way to reuse current->thread.por_el1 for that purpose - this is where the current value of POR_EL1 is stored on a context switch, i.e. the value corresponding to KPKEYS_LVL_PGTABLES inside a lazy_mmu section. Instead, we add a new member to thread_struct to hold that value temporarily. This isn't optimal as that member is unused outside of lazy_mmu sections, but it is the simplest option. A further optimisation this patch makes is to merge the ISBs when exiting lazy_mmu mode. That is, if an ISB is going to be issued by emit_pte_barriers() because kernel pgtables were modified in the lazy_mmu section, we skip the ISB after restoring POR_EL1. This is done by checking TIF_LAZY_MMU_PENDING and ensuring that POR_EL1 is restored before emit_pte_barriers() is called. Checking TIF_LAZY_MMU flag in all pgtable writers is currently overkill, as lazy_mmu sections are only used at the lowest level of page tables. In other words, set_pgd() (for instance) will never be called with TIF_LAZY_MMU set. However, such higher-level helpers are called relatively infrequently and the overhead of checking a TIF flag is low. The flag is therefore checked in all cases for simplicity's sake, just like in [1]. [1] https://lore.kernel.org/linux-mm/20250304150444.3788920-1-ryan.roberts@arm.com/ Signed-off-by: Kevin Brodsky --- arch/arm64/include/asm/pgtable.h | 37 +++++++++++++++++++++++++++++- arch/arm64/include/asm/processor.h | 1 + 2 files changed, 37 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7929b79cd6b1..61dee76be515 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -43,11 +43,40 @@ #ifdef CONFIG_KPKEYS_HARDENED_PGTABLES KPKEYS_GUARD_COND(kpkeys_hardened_pgtables, KPKEYS_LVL_PGTABLES, - kpkeys_hardened_pgtables_enabled()) + kpkeys_hardened_pgtables_enabled() && + !test_thread_flag(TIF_LAZY_MMU)) #else KPKEYS_GUARD_NOOP(kpkeys_hardened_pgtables) #endif +static void kpkeys_lazy_mmu_enter(void) +{ + if (!kpkeys_hardened_pgtables_enabled()) + return; + + current->thread.por_el1_lazy_mmu = kpkeys_set_level(KPKEYS_LVL_PGTABLES); +} + +static void kpkeys_lazy_mmu_exit(void) +{ + u64 saved_por_el1; + + if (!kpkeys_hardened_pgtables_enabled()) + return; + + saved_por_el1 = current->thread.por_el1_lazy_mmu; + + /* + * We skip any barrier if TIF_LAZY_MMU_PENDING is set: + * emit_pte_barriers() will issue an ISB just after this function + * returns. + */ + if (test_thread_flag(TIF_LAZY_MMU_PENDING)) + __kpkeys_set_pkey_reg_nosync(saved_por_el1); + else + arch_kpkeys_restore_pkey_reg(saved_por_el1); +} + static inline void emit_pte_barriers(void) { /* @@ -83,6 +112,7 @@ static inline void arch_enter_lazy_mmu_mode(void) VM_WARN_ON(test_thread_flag(TIF_LAZY_MMU)); set_thread_flag(TIF_LAZY_MMU); + kpkeys_lazy_mmu_enter(); } static inline void arch_flush_lazy_mmu_mode(void) @@ -93,6 +123,11 @@ static inline void arch_flush_lazy_mmu_mode(void) static inline void arch_leave_lazy_mmu_mode(void) { + /* + * The ordering should be preserved to allow kpkeys_lazy_mmu_exit() + * to skip any barrier when TIF_LAZY_MMU_PENDING is set. + */ + kpkeys_lazy_mmu_exit(); arch_flush_lazy_mmu_mode(); clear_thread_flag(TIF_LAZY_MMU); } diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 0afaf96ca699..14a4b483098d 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -186,6 +186,7 @@ struct thread_struct { u64 tpidr2_el0; u64 por_el0; u64 por_el1; + u64 por_el1_lazy_mmu; #ifdef CONFIG_ARM64_GCS unsigned int gcs_el0_mode; unsigned int gcs_el0_locked;