From patchwork Fri Apr 11 09:16:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047925 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3831CC369A2 for ; Fri, 11 Apr 2025 09:23:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=0TInbeeHsJ1w6rsCRtJk/Jeb1cP06TeabMr1jT8HfEs=; b=J2oMe4svO8Xq00nSoO5h8Hha/3 L6GsN8JX0DG+1NEoRwsq63FjKf6i0dfikHGWn+xSw7WpxDAec07mF4bjYA6kb9GHlanhw4SJlXoMD uFcjHqEEDmLCbS3I4vK1Vm4l5zxtg9lhLTi/9tX49HxMMYzj+exQXO6N8rPqRQ8li8et6npXzsOZY osO3m8gfFUHEb1x0X7Zkd59/PJ4QlJoqgRiGZ0m48Q3CYHruSrzE8HKB37xt8dhA1aak9kIXHV8cG xSsRUuWE4bfUoBdam9p5k7Yj5EK6ym/t2FlEFmnAqXCF9IgGiFvTDuI69RIrGv18qBloOcwreikIS Km1c00gg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3Abk-0000000DC7b-159O; Fri, 11 Apr 2025 09:23:32 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AVR-0000000DALL-34xg for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:17:02 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 081411596; Fri, 11 Apr 2025 02:17:01 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 763A83F6A8; Fri, 11 Apr 2025 02:16:57 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 01/18] mm: Introduce kpkeys Date: Fri, 11 Apr 2025 10:16:14 +0100 Message-ID: <20250411091631.954228-2-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_021701_858464_61C2CBF6 X-CRM114-Status: GOOD ( 25.21 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org kpkeys is a simple framework to enable the use of protection keys (pkeys) to harden the kernel itself. This patch introduces the basic API in : a couple of functions to set and restore the pkey register and macros to define guard objects. kpkeys introduces a new concept on top of pkeys: the kpkeys level. Each level is associated to a set of permissions for the pkeys managed by the kpkeys framework. kpkeys_set_level(lvl) sets those permissions according to lvl, and returns the original pkey register, to be later restored by kpkeys_restore_pkey_reg(). To start with, only KPKEYS_LVL_DEFAULT is available, which is meant to grant RW access to KPKEYS_PKEY_DEFAULT (i.e. all memory since this is the only available pkey for now). Because each architecture implementing pkeys uses a different representation for the pkey register, and may reserve certain pkeys for specific uses, support for kpkeys must be explicitly indicated by selecting ARCH_HAS_KPKEYS and defining the following functions in , in addition to the macros provided in : - arch_kpkeys_set_level() - arch_kpkeys_restore_pkey_reg() - arch_kpkeys_enabled() Signed-off-by: Kevin Brodsky --- include/asm-generic/kpkeys.h | 17 ++++++ include/linux/kpkeys.h | 113 +++++++++++++++++++++++++++++++++++ mm/Kconfig | 2 + 3 files changed, 132 insertions(+) create mode 100644 include/asm-generic/kpkeys.h create mode 100644 include/linux/kpkeys.h diff --git a/include/asm-generic/kpkeys.h b/include/asm-generic/kpkeys.h new file mode 100644 index 000000000000..ab819f157d6a --- /dev/null +++ b/include/asm-generic/kpkeys.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __ASM_GENERIC_KPKEYS_H +#define __ASM_GENERIC_KPKEYS_H + +#ifndef KPKEYS_PKEY_DEFAULT +#define KPKEYS_PKEY_DEFAULT 0 +#endif + +/* + * Represents a pkey register value that cannot be used, typically disabling + * access to all keys. + */ +#ifndef KPKEYS_PKEY_REG_INVAL +#define KPKEYS_PKEY_REG_INVAL 0 +#endif + +#endif /* __ASM_GENERIC_KPKEYS_H */ diff --git a/include/linux/kpkeys.h b/include/linux/kpkeys.h new file mode 100644 index 000000000000..faa6e2615798 --- /dev/null +++ b/include/linux/kpkeys.h @@ -0,0 +1,113 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _LINUX_KPKEYS_H +#define _LINUX_KPKEYS_H + +#include +#include + +#define KPKEYS_LVL_DEFAULT 0 + +#define KPKEYS_LVL_MIN KPKEYS_LVL_DEFAULT +#define KPKEYS_LVL_MAX KPKEYS_LVL_DEFAULT + +#define __KPKEYS_GUARD(name, set_level, restore_pkey_reg, set_arg, ...) \ + __DEFINE_CLASS_IS_CONDITIONAL(name, false); \ + DEFINE_CLASS(name, u64, \ + restore_pkey_reg, set_level, set_arg); \ + static inline void *class_##name##_lock_ptr(u64 *_T) \ + { return _T; } + +/** + * KPKEYS_GUARD_NOOP() - define a guard type that does nothing + * @name: the name of the guard type + * @cond_arg: an argument specification (optional) + * + * Define a guard type that does nothing, useful to match a real guard type + * that is defined under an #ifdef. @cond_arg may optionally be passed to match + * a guard defined using KPKEYS_GUARD_COND(). + */ +#define KPKEYS_GUARD_NOOP(name, ...) \ + __KPKEYS_GUARD(name, 0, (void)_T, ##__VA_ARGS__, void) + +#ifdef CONFIG_ARCH_HAS_KPKEYS + +#include + +/** + * KPKEYS_GUARD_COND() - define a guard type that conditionally switches to + * a given kpkeys level + * @name: the name of the guard type + * @level: the kpkeys level to switch to + * @cond: an expression that is evaluated as condition + * @cond_arg: an argument specification for the condition (optional) + * + * Define a guard type that switches to @level if @cond evaluates to true, and + * does nothing otherwise. @cond_arg may be specified to give access to a + * caller-defined argument to @cond. + */ +#define KPKEYS_GUARD_COND(name, level, cond, ...) \ + __KPKEYS_GUARD(name, \ + cond ? kpkeys_set_level(level) \ + : KPKEYS_PKEY_REG_INVAL, \ + kpkeys_restore_pkey_reg(_T), \ + ##__VA_ARGS__, void) + +/** + * KPKEYS_GUARD() - define a guard type that switches to a given kpkeys level + * if kpkeys are enabled + * @name: the name of the guard type + * @level: the kpkeys level to switch to + * + * Define a guard type that switches to @level if the system supports kpkeys. + */ +#define KPKEYS_GUARD(name, level) \ + KPKEYS_GUARD_COND(name, level, arch_kpkeys_enabled()) + +/** + * kpkeys_set_level() - switch kpkeys level + * @level: the level to switch to + * + * Switches the kpkeys level to the specified value. @level must be a + * compile-time constant. The arch-specific pkey register will be updated + * accordingly, and the original value returned. + * + * Return: the original pkey register value if the register was written to, or + * KPKEYS_PKEY_REG_INVAL otherwise (no write to the register was + * required). + */ +static __always_inline u64 kpkeys_set_level(int level) +{ + BUILD_BUG_ON_MSG(!__builtin_constant_p(level), + "kpkeys_set_level() only takes constant levels"); + BUILD_BUG_ON_MSG(level < KPKEYS_LVL_MIN || level > KPKEYS_LVL_MAX, + "Invalid level passed to kpkeys_set_level()"); + + return arch_kpkeys_set_level(level); +} + +/** + * kpkeys_restore_pkey_reg() - restores a pkey register value + * @pkey_reg: the pkey register value to restore + * + * This function is meant to be passed the value returned by kpkeys_set_level(), + * in order to restore the pkey register to its original value (thus restoring + * the original kpkeys level). + */ +static __always_inline void kpkeys_restore_pkey_reg(u64 pkey_reg) +{ + if (pkey_reg != KPKEYS_PKEY_REG_INVAL) + arch_kpkeys_restore_pkey_reg(pkey_reg); +} + +#else /* CONFIG_ARCH_HAS_KPKEYS */ + +#include + +static inline bool arch_kpkeys_enabled(void) +{ + return false; +} + +#endif /* CONFIG_ARCH_HAS_KPKEYS */ + +#endif /* _LINUX_KPKEYS_H */ diff --git a/mm/Kconfig b/mm/Kconfig index e113f713b493..819ef5b70695 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1131,6 +1131,8 @@ config ARCH_USES_HIGH_VMA_FLAGS bool config ARCH_HAS_PKEYS bool +config ARCH_HAS_KPKEYS + bool config ARCH_USES_PG_ARCH_2 bool From patchwork Fri Apr 11 09:16:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 018C3C36010 for ; Fri, 11 Apr 2025 09:27:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=DAvN/ymhOC9RH1764Pqdo1P8xtkdW4aJF6g43VRuQIg=; b=OL4X9jJveMJ5p2Faya7/1pOMhj y/JyCpq77x6tGLzj7DXlFjpFcyT3K4DIlLnoonhh9FKi3JWCdS2d0MveVQEL/WKcwA5vOxnIbAyKs uakCLMOHUFI+k94l+JAP7coEvaUfn61rMW6vUiCYMk2e9EroIowycWNLhDqebHzWTaOgaE4dorTu+ wosREdFdU9lWLgMtnPxLIfT9kktMWan7WUQuYZM9Ae6aOSkcBcCsTRtVLgCDGItO2OW4kl1os8cz/ sR6Wf4p73t4fQHM496hypWXX8dxkYyRJd3UzDtXJO9tQK2n4EMUPACJJaAt6C/DKLVsB6KJOdh9Qx SWfUkJLw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AfJ-0000000DCmx-2f9G; Fri, 11 Apr 2025 09:27:13 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AVZ-0000000DANU-07tg for linux-arm-kernel@bombadil.infradead.org; Fri, 11 Apr 2025 09:17:09 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=DAvN/ymhOC9RH1764Pqdo1P8xtkdW4aJF6g43VRuQIg=; b=fRYvgOC4NwSVIQJLGd+IRh0BH8 fDvl2p1DFY0d3u2CP7mAGFlXa9DuKuRpSqXZZN9Ya3wlhoWofNPLEocZHvniYSyx58S+Ib7Z859Sa crYvkzL/0VuWHaF6gNEtJ6jUZDv7sJclOF+8wTfPliWn1Im+HIR70fFFLmFW2GAa+FkFxnbs6NDfr cKvcH4sOLuVRm7rVTB1/TMbP9fSg+lFjhoXYpNueeWaEsV/x6AvZ29fI4EJkI+KrkonAsqpEvP55q AZNKtGgAbKvEkOYhXbmSBI+WZ3y5HS37nAOPYVfL7eFs2kZhEX4zszxDc9Ci8SL+lTmThODccidh8 yICw2UKw==; Received: from foss.arm.com ([217.140.110.172]) by desiato.infradead.org with esmtp (Exim 4.98.1 #2 (Red Hat Linux)) id 1u3AVW-00000008zi1-0ZR7 for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:17:07 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 19C221692; Fri, 11 Apr 2025 02:17:05 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 87FBC3F6A8; Fri, 11 Apr 2025 02:17:01 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 02/18] set_memory: Introduce set_memory_pkey() stub Date: Fri, 11 Apr 2025 10:16:15 +0100 Message-ID: <20250411091631.954228-3-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_101706_398427_C3E46F70 X-CRM114-Status: GOOD ( 10.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce a new function, set_memory_pkey(), which sets the protection key (pkey) of pages in the specified linear mapping range. Architectures implementing kernel pkeys (kpkeys) must provide a suitable implementation; an empty stub is added as fallback. Signed-off-by: Kevin Brodsky --- include/linux/set_memory.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/include/linux/set_memory.h b/include/linux/set_memory.h index 3030d9245f5a..7b3a8bfde3c6 100644 --- a/include/linux/set_memory.h +++ b/include/linux/set_memory.h @@ -84,4 +84,11 @@ static inline int set_memory_decrypted(unsigned long addr, int numpages) } #endif /* CONFIG_ARCH_HAS_MEM_ENCRYPT */ +#ifndef CONFIG_ARCH_HAS_KPKEYS +static inline int set_memory_pkey(unsigned long addr, int numpages, int pkey) +{ + return 0; +} +#endif + #endif /* _LINUX_SET_MEMORY_H_ */ From patchwork Fri Apr 11 09:16:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047938 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 66112C36010 for ; Fri, 11 Apr 2025 09:29:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=4bn2iTHRxV4Y+JziL3tQkHFaUTnfOTGLKiy637DrfqM=; b=YvzSkhMW3XAfVc29qlO+ZkxxE3 l5IHWSeJqjHi4PzfdviL1K4T0RmbwAF3Dju9Zd6kNBJ2640REMSFORiXy4tcnRGLnRBKtY+yMC0jK yy3CNuuXxfy4/eSYXhAPp261ia50Imj5+wWfegfzC8Bec9jG6iHnUUkiR6K15wFBbdpt+tawFIFza XXw4OZ0P8qzV1ZhJkYNzbCGZ7b3XG900JQ1Sq7JJGhsWe4kD+zGFjRCz9rU6NxnV8SJmyDJiWpHqz dSzsFC9Y42tZJ9zGVg7l44bvcaWnymz4aEz0azMoTMk9L/yi+54RPd6sYha1j4+JgSxRok/psrL79 cVLK3+Lw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3Ah7-0000000DDAy-1Hz1; Fri, 11 Apr 2025 09:29:05 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AVZ-0000000DAO9-39Za for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:17:11 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2C2C7106F; Fri, 11 Apr 2025 02:17:09 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9A1553F6A8; Fri, 11 Apr 2025 02:17:05 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 03/18] arm64: mm: Enable overlays for all EL1 indirect permissions Date: Fri, 11 Apr 2025 10:16:16 +0100 Message-ID: <20250411091631.954228-4-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_021709_831908_25C72FA8 X-CRM114-Status: GOOD ( 11.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In preparation of using POE inside the kernel, enable "Overlay applied" for all stage 1 base permissions in PIR_EL1. This ensures that the permissions set in POR_EL1 affect all kernel mappings. Signed-off-by: Kevin Brodsky --- arch/arm64/include/asm/pgtable-prot.h | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h index 7830d031742e..c863d8e51281 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -181,13 +181,13 @@ static inline bool __pure lpa2_is_enabled(void) PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_GCS), PIE_NONE_O) | \ PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_GCS_RO), PIE_NONE_O) | \ PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_EXECONLY), PIE_NONE_O) | \ - PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_READONLY_EXEC), PIE_R) | \ - PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_SHARED_EXEC), PIE_RW) | \ - PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_READONLY), PIE_R) | \ - PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_SHARED), PIE_RW) | \ - PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_ROX), PIE_RX) | \ - PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_EXEC), PIE_RWX) | \ - PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_RO), PIE_R) | \ - PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL), PIE_RW)) + PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_READONLY_EXEC), PIE_R_O) | \ + PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_SHARED_EXEC), PIE_RW_O) | \ + PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_READONLY), PIE_R_O) | \ + PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_SHARED), PIE_RW_O) | \ + PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_ROX), PIE_RX_O) | \ + PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_EXEC), PIE_RWX_O) | \ + PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_RO), PIE_R_O) | \ + PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL), PIE_RW_O)) #endif /* __ASM_PGTABLE_PROT_H */ From patchwork Fri Apr 11 09:16:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B43B1C36010 for ; Fri, 11 Apr 2025 09:31:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=AUz8ih9TJh8rve7uDxN9yr0bb8Udhpj9bSc6wxAckKg=; b=X5jf7C5A4h71HGaYqCtQl6Qnjz UKHm9TxqCfIjjfWWe/mVT7Prb0sECWWPj8ZHqUrjWzU3CcXry983soHoW78KMV/AoU2IFs6nWRStI qIgeIr40F2wTfekl7U7rEHK16rlLrvxVIExHX8bntIGWlAF+S+rOWNbcmRAnefbhkVu2TCcI5qzRN P8uHqwLG4Bsxj8pVo6f0cBvol8FBCdf9KK7Xmipsy9Z0ATLTfBlovv+hW4lU9VdhVEqvn6ofSnaV0 ivTt1RGuKCMny/ZyuaN0RqGMYVmHMnmDcEhWxBVd/67HHNDPUXwZymcWF41pRiA2Pm9ZS7sXw3k4v DTTh3k2g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3Aiu-0000000DDRT-003a; Fri, 11 Apr 2025 09:30:56 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AVe-0000000DAQ1-0Hh5 for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:17:15 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3F8CD1596; Fri, 11 Apr 2025 02:17:13 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id AC79A3F6A8; Fri, 11 Apr 2025 02:17:09 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 04/18] arm64: Introduce por_elx_set_pkey_perms() helper Date: Fri, 11 Apr 2025 10:16:17 +0100 Message-ID: <20250411091631.954228-5-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_021714_187433_DDEE73F7 X-CRM114-Status: GOOD ( 14.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce a helper that sets the permissions of a given pkey (POIndex) in the POR_ELx format, and make use of it in arch_set_user_pkey_access(). Signed-off-by: Kevin Brodsky --- arch/arm64/include/asm/por.h | 7 +++++++ arch/arm64/mm/mmu.c | 26 ++++++++++---------------- 2 files changed, 17 insertions(+), 16 deletions(-) diff --git a/arch/arm64/include/asm/por.h b/arch/arm64/include/asm/por.h index d913d5b529e4..bffb4d2b1246 100644 --- a/arch/arm64/include/asm/por.h +++ b/arch/arm64/include/asm/por.h @@ -31,4 +31,11 @@ static inline bool por_elx_allows_exec(u64 por, u8 pkey) return perm & POE_X; } +static inline u64 por_elx_set_pkey_perms(u64 por, u8 pkey, u64 perms) +{ + u64 shift = POR_ELx_PERM_SHIFT(pkey); + + return (por & ~(POE_MASK << shift)) | (perms << shift); +} + #endif /* _ASM_ARM64_POR_H */ diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 8fcf59ba39db..89a331e699d2 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1578,8 +1578,8 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp) #ifdef CONFIG_ARCH_HAS_PKEYS int arch_set_user_pkey_access(struct task_struct *tsk, int pkey, unsigned long init_val) { - u64 new_por; - u64 old_por; + u64 new_perms; + u64 por; if (!system_supports_poe()) return -ENOSPC; @@ -1593,25 +1593,19 @@ int arch_set_user_pkey_access(struct task_struct *tsk, int pkey, unsigned long i return -EINVAL; /* Set the bits we need in POR: */ - new_por = POE_RWX; + new_perms = POE_RWX; if (init_val & PKEY_DISABLE_WRITE) - new_por &= ~POE_W; + new_perms &= ~POE_W; if (init_val & PKEY_DISABLE_ACCESS) - new_por &= ~POE_RW; + new_perms &= ~POE_RW; if (init_val & PKEY_DISABLE_READ) - new_por &= ~POE_R; + new_perms &= ~POE_R; if (init_val & PKEY_DISABLE_EXECUTE) - new_por &= ~POE_X; + new_perms &= ~POE_X; - /* Shift the bits in to the correct place in POR for pkey: */ - new_por = POR_ELx_PERM_PREP(pkey, new_por); - - /* Get old POR and mask off any old bits in place: */ - old_por = read_sysreg_s(SYS_POR_EL0); - old_por &= ~(POE_MASK << POR_ELx_PERM_SHIFT(pkey)); - - /* Write old part along with new part: */ - write_sysreg_s(old_por | new_por, SYS_POR_EL0); + por = read_sysreg_s(SYS_POR_EL0); + por = por_elx_set_pkey_perms(por, pkey, new_perms); + write_sysreg_s(por, SYS_POR_EL0); return 0; } From patchwork Fri Apr 11 09:16:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D6EF8C36010 for ; Fri, 11 Apr 2025 09:32:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=aJoDwb/u3Nzlv4jn8+XZaODNbWbOa/Nrb+B9eqkeeBc=; b=n/8S9JtzMMLHP/kYpFWbuygtEW Z19i5ziCANOybUCSNc3WaGGSUo56rx7jiYZpMeIa5irKwspcgGqK6gDg2gBlV87J0HdSsAFlIw+6V djqgdnJyF/CtHhi+LZbEyBhBBpnq3rXXM1NUg56aC2vkl0kuburoxPo14xY24IJH2osCz/A6pWtGf 69GML20XE8us+HaXtY6Qyp66hvNLFVBzssy9t8cob9Sw1Mf318jZY2350Zh3Gqd64MBjBd8giXNuF wIA+kIdHP77ZI2ZLaKgUbhJ9gW/qVEOmOBWeRqyxU0hIHz4ncq1seQOiJkfmRxdZ/ag59Cyi+yo1w kztSNDcw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3Akh-0000000DDbW-2nQ6; Fri, 11 Apr 2025 09:32:47 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AVh-0000000DARr-3fpF for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:17:19 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5339C106F; Fri, 11 Apr 2025 02:17:17 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id BFD4B3F6A8; Fri, 11 Apr 2025 02:17:13 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 05/18] arm64: Implement asm/kpkeys.h using POE Date: Fri, 11 Apr 2025 10:16:18 +0100 Message-ID: <20250411091631.954228-6-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_021717_954241_869CE5C4 X-CRM114-Status: GOOD ( 13.86 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Implement the kpkeys interface if CONFIG_ARM64_POE is enabled. The permissions for KPKEYS_PKEY_DEFAULT (pkey 0) are set to RWX as this pkey is also used for code mappings. Signed-off-by: Kevin Brodsky --- arch/arm64/include/asm/kpkeys.h | 49 +++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 arch/arm64/include/asm/kpkeys.h diff --git a/arch/arm64/include/asm/kpkeys.h b/arch/arm64/include/asm/kpkeys.h new file mode 100644 index 000000000000..3b0ab5e7dd22 --- /dev/null +++ b/arch/arm64/include/asm/kpkeys.h @@ -0,0 +1,49 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __ASM_KPKEYS_H +#define __ASM_KPKEYS_H + +#include +#include +#include + +#include + +static inline bool arch_kpkeys_enabled(void) +{ + return system_supports_poe(); +} + +#ifdef CONFIG_ARM64_POE + +static inline u64 por_set_kpkeys_level(u64 por, int level) +{ + por = por_elx_set_pkey_perms(por, KPKEYS_PKEY_DEFAULT, POE_RWX); + + return por; +} + +static __always_inline void __kpkeys_set_pkey_reg_nosync(u64 pkey_reg) +{ + write_sysreg_s(pkey_reg, SYS_POR_EL1); +} + +static __always_inline int arch_kpkeys_set_level(int level) +{ + u64 prev_por = read_sysreg_s(SYS_POR_EL1); + u64 new_por = por_set_kpkeys_level(prev_por, level); + + __kpkeys_set_pkey_reg_nosync(new_por); + isb(); + + return prev_por; +} + +static __always_inline void arch_kpkeys_restore_pkey_reg(u64 pkey_reg) +{ + __kpkeys_set_pkey_reg_nosync(pkey_reg); + isb(); +} + +#endif /* CONFIG_ARM64_POE */ + +#endif /* __ASM_KPKEYS_H */ From patchwork Fri Apr 11 09:16:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047942 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 40A21C36010 for ; Fri, 11 Apr 2025 09:34:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=oHokXrrIxgFozOdPdKGv7pm7boYyoVv2BPENAFW6e/0=; b=0zrjhwoB+7xOLlvumuvH19qu3J wsXfkzr1T8hBYHaGUxhU999Q7FxB/d4GnwP9kcmQv/4gvGTh6Fej+L7Gg7y1LkZbcVbHUf7faI690 +grjcbiyjHGqOjSoyVAvMt77IRcoMe3FnHoiNZBcC8YxWCAqLWyEW1M7pAdryuu3ULZ5nWhXMqnP1 5eP5IKh3f7S6KyJqDL+KIy9GaGF4PSR6Uy/rXzsTc8oHk6B1PAy8c3hTB0ZFgvebe6Tx+ZBEfwwDJ RJyA7+9M+LLrMpAc71dq4XbhdmTm1xdOEbtxEAAOnFHlqTbM4Bfh6brF7tdA/+4imL8L5jeahoNHc ObIB4RiA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AmU-0000000DDlF-1yB8; Fri, 11 Apr 2025 09:34:38 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AVm-0000000DATf-0K6S for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:17:23 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 65F051692; Fri, 11 Apr 2025 02:17:21 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D39293F6A8; Fri, 11 Apr 2025 02:17:17 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 06/18] arm64: set_memory: Implement set_memory_pkey() Date: Fri, 11 Apr 2025 10:16:19 +0100 Message-ID: <20250411091631.954228-7-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_021722_150853_4FD8504E X-CRM114-Status: GOOD ( 13.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Implement set_memory_pkey() using POE if supported. Signed-off-by: Kevin Brodsky --- arch/arm64/include/asm/set_memory.h | 4 ++++ arch/arm64/mm/pageattr.c | 25 +++++++++++++++++++++++++ 2 files changed, 29 insertions(+) diff --git a/arch/arm64/include/asm/set_memory.h b/arch/arm64/include/asm/set_memory.h index 90f61b17275e..b6cd6de34abf 100644 --- a/arch/arm64/include/asm/set_memory.h +++ b/arch/arm64/include/asm/set_memory.h @@ -19,4 +19,8 @@ bool kernel_page_present(struct page *page); int set_memory_encrypted(unsigned long addr, int numpages); int set_memory_decrypted(unsigned long addr, int numpages); +#ifdef CONFIG_ARCH_HAS_KPKEYS +int set_memory_pkey(unsigned long addr, int numpages, int pkey); +#endif + #endif /* _ASM_ARM64_SET_MEMORY_H */ diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 39fd1f7ff02a..9721a74adbe2 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include @@ -292,6 +293,30 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid) return set_memory_valid(addr, nr, valid); } +#ifdef CONFIG_ARCH_HAS_KPKEYS +int set_memory_pkey(unsigned long addr, int numpages, int pkey) +{ + unsigned long set_prot = 0; + + if (!system_supports_poe()) + return 0; + + if (!__is_lm_address(addr)) + return -EINVAL; + + if (pkey >= arch_max_pkey()) + return -EINVAL; + + set_prot |= pkey & BIT(0) ? PTE_PO_IDX_0 : 0; + set_prot |= pkey & BIT(1) ? PTE_PO_IDX_1 : 0; + set_prot |= pkey & BIT(2) ? PTE_PO_IDX_2 : 0; + + return __change_memory_common(addr, PAGE_SIZE * numpages, + __pgprot(set_prot), + __pgprot(PTE_PO_IDX_MASK)); +} +#endif + #ifdef CONFIG_DEBUG_PAGEALLOC /* * This is - apart from the return value - doing the same From patchwork Fri Apr 11 09:16:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047955 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EED47C36010 for ; Fri, 11 Apr 2025 09:36:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=WSd0yigUChVpRcAhfJwe0J3l1gFAvojA/sDA8zvgUxg=; b=GrzlSP3DTlwl7brcTZvFfEVzia t28E3HvDS2J5YUKNZ9reVR0+RV2Bgar68AE638zjDKdc4EDom0m51WRRCMLKviZyzi2ojIQyuPguu P4autXqGA6dgxqW4X1qisbgX+28JVq02P0+JD7SFZzGZUxQ4+/RPz1/8H3BD6rhnpln01xudRE8zS LoAm30rOnmcGOagLEtSPvy8xFWz8XzZNiRqbJr6Rr1Hih7E9OFB013xWKvKOetJHtUn/LBSkc0VNK DgmAyk6n6Ciuuw+d4Qx3zaXVvsWu13t7X/jwzp5+zIbWnyqSgMlXBGcyy1h0Q+roUn3YD2F1mDwqV V/uNkQXA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AoG-0000000DDsN-0WcQ; Fri, 11 Apr 2025 09:36:28 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AVq-0000000DAVF-07Vr for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:17:27 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 783311596; Fri, 11 Apr 2025 02:17:25 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E5EB13F6A8; Fri, 11 Apr 2025 02:17:21 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 07/18] arm64: Reset POR_EL1 on exception entry Date: Fri, 11 Apr 2025 10:16:20 +0100 Message-ID: <20250411091631.954228-8-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_021726_160686_45B9F7C7 X-CRM114-Status: GOOD ( 19.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org POR_EL1 will be modified, through the kpkeys framework, in order to grant temporary RW access to certain keys. If an exception occurs in the middle of a "critical section" where POR_EL1 is set to a privileged value, it is preferable to reset it to its default value upon taking the exception to minimise the amount of code running at higher kpkeys level. This patch implements the reset of POR_EL1 on exception entry, storing the original value in a new pt_regs field and restoring on exception return. To avoid an expensive ISB, the register is only reset if the interrupted value isn't the default. No check is made on the return path as an ISB occurs anyway as part of ERET. Signed-off-by: Kevin Brodsky --- arch/arm64/include/asm/kpkeys.h | 10 ++++++++++ arch/arm64/include/asm/por.h | 4 ++++ arch/arm64/include/asm/ptrace.h | 4 ++++ arch/arm64/kernel/asm-offsets.c | 3 +++ arch/arm64/kernel/entry.S | 24 +++++++++++++++++++++++- 5 files changed, 44 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kpkeys.h b/arch/arm64/include/asm/kpkeys.h index 3b0ab5e7dd22..79ae33388088 100644 --- a/arch/arm64/include/asm/kpkeys.h +++ b/arch/arm64/include/asm/kpkeys.h @@ -8,6 +8,14 @@ #include +/* + * Equivalent to por_set_kpkeys_level(0, KPKEYS_LVL_DEFAULT), but can also be + * used in assembly. + */ +#define POR_EL1_INIT POR_ELx_PERM_PREP(KPKEYS_PKEY_DEFAULT, POE_RWX) + +#ifndef __ASSEMBLY__ + static inline bool arch_kpkeys_enabled(void) { return system_supports_poe(); @@ -46,4 +54,6 @@ static __always_inline void arch_kpkeys_restore_pkey_reg(u64 pkey_reg) #endif /* CONFIG_ARM64_POE */ +#endif /* __ASSEMBLY__ */ + #endif /* __ASM_KPKEYS_H */ diff --git a/arch/arm64/include/asm/por.h b/arch/arm64/include/asm/por.h index bffb4d2b1246..58dce4b8021b 100644 --- a/arch/arm64/include/asm/por.h +++ b/arch/arm64/include/asm/por.h @@ -10,6 +10,8 @@ #define POR_EL0_INIT POR_ELx_PERM_PREP(0, POE_RWX) +#ifndef __ASSEMBLY__ + static inline bool por_elx_allows_read(u64 por, u8 pkey) { u8 perm = POR_ELx_PERM_GET(pkey, por); @@ -38,4 +40,6 @@ static inline u64 por_elx_set_pkey_perms(u64 por, u8 pkey, u64 perms) return (por & ~(POE_MASK << shift)) | (perms << shift); } +#endif /* __ASSEMBLY__ */ + #endif /* _ASM_ARM64_POR_H */ diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h index 47ff8654c5ec..e907df4225d4 100644 --- a/arch/arm64/include/asm/ptrace.h +++ b/arch/arm64/include/asm/ptrace.h @@ -166,6 +166,10 @@ struct pt_regs { u64 orig_x0; s32 syscallno; u32 pmr; +#ifdef CONFIG_ARM64_POE + u64 por_el1; + u64 __unused; +#endif u64 sdei_ttbr1; struct frame_record_meta stackframe; diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index eb1a840e4110..a69c8ba0e085 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -75,6 +75,9 @@ int main(void) DEFINE(S_SYSCALLNO, offsetof(struct pt_regs, syscallno)); DEFINE(S_SDEI_TTBR1, offsetof(struct pt_regs, sdei_ttbr1)); DEFINE(S_PMR, offsetof(struct pt_regs, pmr)); +#ifdef CONFIG_ARM64_POE + DEFINE(S_POR_EL1, offsetof(struct pt_regs, por_el1)); +#endif DEFINE(S_STACKFRAME, offsetof(struct pt_regs, stackframe)); DEFINE(S_STACKFRAME_TYPE, offsetof(struct pt_regs, stackframe.type)); DEFINE(PT_REGS_SIZE, sizeof(struct pt_regs)); diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 5ae2a34b50bd..bc042eabd730 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -279,6 +280,19 @@ alternative_else_nop_endif .else add x21, sp, #PT_REGS_SIZE get_current_task tsk +#ifdef CONFIG_ARM64_POE +alternative_if_not ARM64_HAS_S1POE + b 1f +alternative_else_nop_endif + mrs_s x0, SYS_POR_EL1 + str x0, [sp, #S_POR_EL1] + mov x1, #POR_EL1_INIT + cmp x0, x1 + b.eq 1f + msr_s SYS_POR_EL1, x1 + isb +1: +#endif /* CONFIG_ARM64_POE */ .endif /* \el == 0 */ mrs x22, elr_el1 mrs x23, spsr_el1 @@ -409,7 +423,15 @@ alternative_else_nop_endif mte_set_user_gcr tsk, x0, x1 apply_ssbd 0, x0, x1 - .endif + .else +#ifdef CONFIG_ARM64_POE +alternative_if ARM64_HAS_S1POE + ldr x0, [sp, #S_POR_EL1] + msr_s SYS_POR_EL1, x0 + /* No explicit ISB; we rely on ERET */ +alternative_else_nop_endif +#endif /* CONFIG_ARM64_POE */ + .endif /* \el == 0 */ msr elr_el1, x21 // set up the return data msr spsr_el1, x22 From patchwork Fri Apr 11 09:16:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047956 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BF36DC369A8 for ; Fri, 11 Apr 2025 09:38:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=kfkzdaySghlKYvqTb+hEmk4OPnsFL6Vuh0dxNa6E2UQ=; b=EqA0Vy6HMjpf80dEK0crlmmNEx 4ZgipCWfeQqj2vV0/OQ3pgBKEveqQRGVqAW76A5WLBgvSkcDRc1wMuzXcaQxOAU+/z6RCTf/eF/Om dHVv+Ts+ZEG7USmSEvgpNCIL9uOyK50GNu0J+WMC4dEPI2I1Tn94aHOLE2c3INJ/9TPYJSk22m0PC ChbKwg+cwMVTGZUVIKOhc8D65QHaSJhouTFDBCz1pUblxMabXeGg9UMVw9FPdFa2d8NaOGXIOp5vr k8cS1EFk09jwt8PmBlIsm4dZq7hHKWqG0yRWdO1plbNpupF6UduQ8pV73+dMZB8HR2maisDVrsvNC +gDR3qsA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3Aq1-0000000DE2o-3TXi; Fri, 11 Apr 2025 09:38:17 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AVu-0000000DAWz-0T17 for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:17:31 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8B014106F; Fri, 11 Apr 2025 02:17:29 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 042073F6A8; Fri, 11 Apr 2025 02:17:25 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 08/18] arm64: Context-switch POR_EL1 Date: Fri, 11 Apr 2025 10:16:21 +0100 Message-ID: <20250411091631.954228-9-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_021730_184861_5408D114 X-CRM114-Status: GOOD ( 15.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org POR_EL1 is about to be used by the kpkeys framework, modifying it for (typically small) sections of code. If an exception occurs during that window and scheduling occurs, we must ensure that POR_EL1 is context-switched as needed (saving the old value and restoring the new one). An ISB is needed to ensure the write takes effect, so we skip it if the new value is the same as the old, like for POR_EL0. Signed-off-by: Kevin Brodsky --- arch/arm64/include/asm/processor.h | 1 + arch/arm64/kernel/process.c | 9 +++++++++ 2 files changed, 10 insertions(+) diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 1bf1a3b16e88..0afaf96ca699 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -185,6 +185,7 @@ struct thread_struct { u64 svcr; u64 tpidr2_el0; u64 por_el0; + u64 por_el1; #ifdef CONFIG_ARM64_GCS unsigned int gcs_el0_mode; unsigned int gcs_el0_locked; diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 45a55fe81788..1f38c29b0c95 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -418,6 +418,9 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) ptrauth_thread_init_kernel(p); + if (system_supports_poe()) + p->thread.por_el1 = read_sysreg_s(SYS_POR_EL1); + if (likely(!args->fn)) { *childregs = *current_pt_regs(); childregs->regs[0] = 0; @@ -639,6 +642,12 @@ static void permission_overlay_switch(struct task_struct *next) if (current->thread.por_el0 != next->thread.por_el0) { write_sysreg_s(next->thread.por_el0, SYS_POR_EL0); } + + current->thread.por_el1 = read_sysreg_s(SYS_POR_EL1); + if (current->thread.por_el1 != next->thread.por_el1) { + write_sysreg_s(next->thread.por_el1, SYS_POR_EL1); + isb(); + } } /* From patchwork Fri Apr 11 09:16:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047957 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EB1CDC369A8 for ; Fri, 11 Apr 2025 09:40:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=D6OJ6eTJLaGqk3wgOaTb1V4BCC/cGWqmQ6ZcX7KPIXg=; b=dSZ98FUYVgGwJpAp0OMVdR7hcj S32pKmHLEftkeB+3AgXP7NgzEi0gFi98YeEc8IXiz1KJp9WXLyW/Nyf0gtISR2yAMyNQeJDeXE9r0 BR6n4Pciob4FpH9O0H9a5heJNoNEAiMijHN6QtMuQATPMV93vu0U2PAk1kP/K/UN42e1XXg1siQcN gv0ufxrRscbSrr5rOLYNCcUw0mqXcPBuv1H2tG0+I4xhBJapfnvb6/MG02YruDQ3MHT8GI4dQ+Kqp VPemLoW3yzPK1B0v8MhQvNaQIrcZ0nlGrBNAJSVvw7qmNSgSvdJbnyZTRd6qxVLSXsit4PHV2tc+l Yq6O+JDA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3Arp-0000000DEEX-2SCs; Fri, 11 Apr 2025 09:40:09 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AVy-0000000DAYZ-1Act for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:17:35 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A4C361713; Fri, 11 Apr 2025 02:17:33 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 1681D3F6A8; Fri, 11 Apr 2025 02:17:29 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 09/18] arm64: Enable kpkeys Date: Fri, 11 Apr 2025 10:16:22 +0100 Message-ID: <20250411091631.954228-10-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_021734_361436_B733498F X-CRM114-Status: GOOD ( 13.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This is the final step to enable kpkeys on arm64. We enable POE at EL1 by setting TCR2_EL1.POE, and initialise POR_EL1 to the default value, enabling access to the default pkey/POIndex (0). An ISB is added so that POE restrictions are enforced immediately. Having done this, we can now select ARCH_HAS_KPKEYS if ARM64_POE is enabled. Signed-off-by: Kevin Brodsky --- arch/arm64/Kconfig | 1 + arch/arm64/kernel/cpufeature.c | 5 ++++- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a182295e6f08..5f81db8134da 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2198,6 +2198,7 @@ config ARM64_POE def_bool y select ARCH_USES_HIGH_VMA_FLAGS select ARCH_HAS_PKEYS + select ARCH_HAS_KPKEYS help The Permission Overlay Extension is used to implement Memory Protection Keys. Memory Protection Keys provides a mechanism for diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 9c4d6d552b25..2043f04c26ef 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -76,6 +76,7 @@ #include #include #include +#include #include #include @@ -2409,8 +2410,10 @@ static void cpu_enable_mops(const struct arm64_cpu_capabilities *__unused) #ifdef CONFIG_ARM64_POE static void cpu_enable_poe(const struct arm64_cpu_capabilities *__unused) { - sysreg_clear_set(REG_TCR2_EL1, 0, TCR2_EL1_E0POE); + write_sysreg_s(POR_EL1_INIT, SYS_POR_EL1); + sysreg_clear_set(REG_TCR2_EL1, 0, TCR2_EL1_E0POE | TCR2_EL1_POE); sysreg_clear_set(CPACR_EL1, 0, CPACR_EL1_E0POE); + isb(); } #endif From patchwork Fri Apr 11 09:16:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047968 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2679AC36010 for ; Fri, 11 Apr 2025 09:42:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=HDIDxyDyfX0gdKCGHJVPMPRL70ks6utR2npUV1XvMpc=; b=s924OqUKc5clMawFlhra7tLiTK DswCXub5KdgdTCWquvxbjVcPkcYwNKmWBdCO1+65zd+2IjKYX361onubAUlRTZNMYfImgFaeb3p/K 6yPJxTQeKvNiGppLtsDliuZuBNpp7S6CuJakyls43evmuMfpOncSB5Zzb4nbNTrvi6D6Ipz35XoKA tEyQIfiMu3dZh438tZd8fPfDIl4x+bSyGaLhdt48+sx0Ild5lR4UzmXnqodQEREoFZZPXOYwbWEWk ZTz/P4XzohEmrVA9UT0TfpYHH5YytxnlGvfOkwLqLw33k6hnyLmFPMOQ7So5CzgxBm8aQV+pTGLzE 7NLBbqbg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3Atc-0000000DEaq-1CtT; Fri, 11 Apr 2025 09:42:00 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AW2-0000000DAap-191h for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:17:39 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AD61A1596; Fri, 11 Apr 2025 02:17:37 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 286563F6A8; Fri, 11 Apr 2025 02:17:34 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 10/18] mm: Introduce kernel_pgtables_set_pkey() Date: Fri, 11 Apr 2025 10:16:23 +0100 Message-ID: <20250411091631.954228-11-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_021738_398759_820F4D52 X-CRM114-Status: GOOD ( 18.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org kernel_pgtables_set_pkey() allows setting the pkey of all page table pages in swapper_pg_dir, recursively. This will be needed by kpkeys_hardened_pgtables, as it relies on all PTPs being mapped with a non-default pkey. Those initial kernel page tables cannot practically be assigned a non-default pkey right when they are allocated, so mutating them during (early) boot is required. Signed-off-by: Kevin Brodsky --- include/linux/mm.h | 2 + mm/memory.c | 137 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 139 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index ef420f4dc72c..dd1b918dc294 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4240,6 +4240,8 @@ int arch_get_shadow_stack_status(struct task_struct *t, unsigned long __user *st int arch_set_shadow_stack_status(struct task_struct *t, unsigned long status); int arch_lock_shadow_stack_status(struct task_struct *t, unsigned long status); +int kernel_pgtables_set_pkey(int pkey); + /* * mseal of userspace process's system mappings. diff --git a/mm/memory.c b/mm/memory.c index 2d8c265fc7d6..37c2bb35faea 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -76,6 +76,8 @@ #include #include #include +#include +#include #include @@ -7376,3 +7378,138 @@ void vma_pgtable_walk_end(struct vm_area_struct *vma) if (is_vm_hugetlb_page(vma)) hugetlb_vma_unlock_read(vma); } + +static int __init set_page_pkey(void *p, int pkey) +{ + unsigned long addr = (unsigned long)p; + + /* + * swapper_pg_dir itself will be made read-only by mark_rodata_ro() + * so there is no point in changing its pkey. + */ + if (p == swapper_pg_dir) + return 0; + + return set_memory_pkey(addr, 1, pkey); +} + +static int __init set_pkey_pte(pmd_t *pmd, int pkey) +{ + pte_t *pte; + int err; + + pte = pte_offset_kernel(pmd, 0); + err = set_page_pkey(pte, pkey); + + return err; +} + +static int __init set_pkey_pmd(pud_t *pud, int pkey) +{ + pmd_t *pmd; + int i, err = 0; + + pmd = pmd_offset(pud, 0); + + err = set_page_pkey(pmd, pkey); + if (err) + return err; + + for (i = 0; i < PTRS_PER_PMD; i++) { + if (pmd_none(pmd[i]) || pmd_bad(pmd[i]) || pmd_leaf(pmd[i])) + continue; + err = set_pkey_pte(&pmd[i], pkey); + if (err) + break; + } + + return err; +} + +static int __init set_pkey_pud(p4d_t *p4d, int pkey) +{ + pud_t *pud; + int i, err = 0; + + if (mm_pmd_folded(&init_mm)) + return set_pkey_pmd((pud_t *)p4d, pkey); + + pud = pud_offset(p4d, 0); + + err = set_page_pkey(pud, pkey); + if (err) + return err; + + for (i = 0; i < PTRS_PER_PUD; i++) { + if (pud_none(pud[i]) || pud_bad(pud[i]) || pud_leaf(pud[i])) + continue; + err = set_pkey_pmd(&pud[i], pkey); + if (err) + break; + } + + return err; +} + +static int __init set_pkey_p4d(pgd_t *pgd, int pkey) +{ + p4d_t *p4d; + int i, err = 0; + + if (mm_pud_folded(&init_mm)) + return set_pkey_pud((p4d_t *)pgd, pkey); + + p4d = p4d_offset(pgd, 0); + + err = set_page_pkey(p4d, pkey); + if (err) + return err; + + for (i = 0; i < PTRS_PER_P4D; i++) { + if (p4d_none(p4d[i]) || p4d_bad(p4d[i]) || p4d_leaf(p4d[i])) + continue; + err = set_pkey_pud(&p4d[i], pkey); + if (err) + break; + } + + return err; +} + +/** + * kernel_pgtables_set_pkey - set pkey for all kernel page table pages + * @pkey: pkey to set the page table pages to + * + * Walks swapper_pg_dir setting the protection key of every page table page (at + * all levels) to @pkey. swapper_pg_dir itself is left untouched as it is + * expected to be mapped read-only by mark_rodata_ro(). + * + * No-op if the architecture does not support kpkeys. + */ +int __init kernel_pgtables_set_pkey(int pkey) +{ + pgd_t *pgd = swapper_pg_dir; + int i, err = 0; + + if (!arch_kpkeys_enabled()) + return 0; + + spin_lock(&init_mm.page_table_lock); + + if (mm_p4d_folded(&init_mm)) { + err = set_pkey_p4d(pgd, pkey); + goto out; + } + + for (i = 0; i < PTRS_PER_PGD; i++) { + if (pgd_none(pgd[i]) || pgd_bad(pgd[i]) || pgd_leaf(pgd[i])) + continue; + err = set_pkey_p4d(&pgd[i], pkey); + if (err) + break; + } + +out: + spin_unlock(&init_mm.page_table_lock); + return err; +} From patchwork Fri Apr 11 09:16:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047969 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6A049C36010 for ; Fri, 11 Apr 2025 09:44:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Vu1673XrmeD4DkkrL7shuinMre40KndBvHpM+lv2ifE=; b=gcsmAlxhF3s4WC8mH/vM3jFTzJ hk/tCb0fSgxp8PHA6etnkNyxBBpDkRmzIK7gO8xjxKIvNy2EVS33FJIBA+c4CmoERc/zCW+Msx9cb cCA6RQ+xlvoADbKvYQEUYyceVuNvaJx1SA2WZjkDLgms1GLHZ1tfB3HmfcEQJ7GLIiOQkGFMDvdF7 miByIAKll2iL8EflnbYzIws3+0+ar0UtjKLoLCeG1RGGqaX1+OH1r5T2wRoD1XqaS9uH/QonQP6ir dZBlV7BMa9dEFz3l9/EG2pzg556NJhpr8Puot7KUKd1UR1G6LTy/8Th8dXg9e6gWTYw9lLiPBvg35 Y85zod1g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AvO-0000000DEsi-47t1; Fri, 11 Apr 2025 09:43:50 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AWA-0000000DAe5-1mNM for linux-arm-kernel@bombadil.infradead.org; Fri, 11 Apr 2025 09:17:46 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Vu1673XrmeD4DkkrL7shuinMre40KndBvHpM+lv2ifE=; b=cEcgYpeB5UpuyyXTih+XOxxDbv 7L9uO/lc9/G63wXuB5cTl0/OG1UBOgHxs7bNknVhQn9PXt/n2cJUoUGEzX577n/EF5rv3bXDGZMRl wfiS2XIp+pwoOQlH5VrE4jvQMnJwU6Ruj6BNh61xp5eYkzjkznito8Aw7pbFk3XeQsaZsDKbwUBub VTf8DRJpDaEiQOc4Gvb6euoEd/aUAHmqlrOBVGESMRju4yT5ZwQEmYIENcf+kU5nuJZQqzVak4HXa GRVQzgIHlgAAZLpQehDoU5C2/ClK72KQ9nM9A0LQICuZTAyIyLfLB6fUhV94zgFLfg1DoAI6/Pl43 U32Lc4Jw==; Received: from foss.arm.com ([217.140.110.172]) by desiato.infradead.org with esmtp (Exim 4.98.1 #2 (Red Hat Linux)) id 1u3AW7-00000008zjJ-0itX for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:17:45 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1DC4D106F; Fri, 11 Apr 2025 02:17:42 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8C2003F6A8; Fri, 11 Apr 2025 02:17:38 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 11/18] mm: Introduce kpkeys_hardened_pgtables Date: Fri, 11 Apr 2025 10:16:24 +0100 Message-ID: <20250411091631.954228-12-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_101743_769846_2BFCEFB6 X-CRM114-Status: GOOD ( 25.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org kpkeys_hardened_pgtables is a hardening feature based on kpkeys. It aims to prevent the corruption of page tables by: 1. mapping all page table pages, both kernel and user, with a privileged pkey (KPKEYS_PKEY_PGTABLES), and 2. granting write access to that pkey only when running at a higher kpkeys level (KPKEYS_LVL_PGTABLES). The feature is exposed as CONFIG_KPKEYS_HARDENED_PGTABLES; it requires explicit architecture opt-in by selecting ARCH_HAS_KPKEYS_HARDENED_PGTABLES, since much of the page table handling is arch-specific. This patch introduces an API to modify the PTPs' pkey. Because this API is going to be called from low-level pgtable helpers, it must be inactive on boot and explicitly switched on if and when kpkeys become available. A static key is used for that purpose; it is the responsibility of each architecture supporting kpkeys_hardened_pgtables to call kpkeys_hardened_pgtables_enable() as early as possible to switch on that static key. The initial kernel page tables are also walked to set their pkey, since they have already been allocated at that point. Signed-off-by: Kevin Brodsky --- include/asm-generic/kpkeys.h | 4 +++ include/linux/kpkeys.h | 46 ++++++++++++++++++++++++++++++++++- mm/Kconfig | 3 +++ mm/Makefile | 1 + mm/kpkeys_hardened_pgtables.c | 44 +++++++++++++++++++++++++++++++++ security/Kconfig.hardening | 12 +++++++++ 6 files changed, 109 insertions(+), 1 deletion(-) create mode 100644 mm/kpkeys_hardened_pgtables.c diff --git a/include/asm-generic/kpkeys.h b/include/asm-generic/kpkeys.h index ab819f157d6a..cec92334a9f3 100644 --- a/include/asm-generic/kpkeys.h +++ b/include/asm-generic/kpkeys.h @@ -2,6 +2,10 @@ #ifndef __ASM_GENERIC_KPKEYS_H #define __ASM_GENERIC_KPKEYS_H +#ifndef KPKEYS_PKEY_PGTABLES +#define KPKEYS_PKEY_PGTABLES 1 +#endif + #ifndef KPKEYS_PKEY_DEFAULT #define KPKEYS_PKEY_DEFAULT 0 #endif diff --git a/include/linux/kpkeys.h b/include/linux/kpkeys.h index faa6e2615798..5f4b096374ba 100644 --- a/include/linux/kpkeys.h +++ b/include/linux/kpkeys.h @@ -4,11 +4,15 @@ #include #include +#include + +struct folio; #define KPKEYS_LVL_DEFAULT 0 +#define KPKEYS_LVL_PGTABLES 1 #define KPKEYS_LVL_MIN KPKEYS_LVL_DEFAULT -#define KPKEYS_LVL_MAX KPKEYS_LVL_DEFAULT +#define KPKEYS_LVL_MAX KPKEYS_LVL_PGTABLES #define __KPKEYS_GUARD(name, set_level, restore_pkey_reg, set_arg, ...) \ __DEFINE_CLASS_IS_CONDITIONAL(name, false); \ @@ -110,4 +114,44 @@ static inline bool arch_kpkeys_enabled(void) #endif /* CONFIG_ARCH_HAS_KPKEYS */ +#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES + +DECLARE_STATIC_KEY_FALSE(kpkeys_hardened_pgtables_key); + +static inline bool kpkeys_hardened_pgtables_enabled(void) +{ + return static_branch_unlikely(&kpkeys_hardened_pgtables_key); +} + +int kpkeys_protect_pgtable_memory(struct folio *folio); +int kpkeys_unprotect_pgtable_memory(struct folio *folio); + +/* + * Enables kpkeys_hardened_pgtables and switches existing kernel page tables to + * a privileged pkey (KPKEYS_PKEY_PGTABLES). + * + * Should be called as early as possible by architecture code, after (k)pkeys + * are initialised and before any user task is spawned. + */ +void kpkeys_hardened_pgtables_enable(void); + +#else /* CONFIG_KPKEYS_HARDENED_PGTABLES */ + +static inline bool kpkeys_hardened_pgtables_enabled(void) +{ + return false; +} + +static inline int kpkeys_protect_pgtable_memory(struct folio *folio) +{ + return 0; +} +static inline int kpkeys_unprotect_pgtable_memory(struct folio *folio) +{ + return 0; +} +static inline void kpkeys_hardened_pgtables_enable(void) {} + +#endif /* CONFIG_KPKEYS_HARDENED_PGTABLES */ + #endif /* _LINUX_KPKEYS_H */ diff --git a/mm/Kconfig b/mm/Kconfig index 819ef5b70695..ae1cb209a1ac 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1133,6 +1133,9 @@ config ARCH_HAS_PKEYS bool config ARCH_HAS_KPKEYS bool +# ARCH_HAS_KPKEYS must be selected when selecting this option +config ARCH_HAS_KPKEYS_HARDENED_PGTABLES + bool config ARCH_USES_PG_ARCH_2 bool diff --git a/mm/Makefile b/mm/Makefile index e7f6bbf8ae5f..0f30cc85c6f1 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -148,3 +148,4 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o obj-$(CONFIG_EXECMEM) += execmem.o obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_PT_RECLAIM) += pt_reclaim.o +obj-$(CONFIG_KPKEYS_HARDENED_PGTABLES) += kpkeys_hardened_pgtables.o diff --git a/mm/kpkeys_hardened_pgtables.c b/mm/kpkeys_hardened_pgtables.c new file mode 100644 index 000000000000..931fa97bc8a7 --- /dev/null +++ b/mm/kpkeys_hardened_pgtables.c @@ -0,0 +1,44 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include + +DEFINE_STATIC_KEY_FALSE(kpkeys_hardened_pgtables_key); + +int kpkeys_protect_pgtable_memory(struct folio *folio) +{ + unsigned long addr = (unsigned long)folio_address(folio); + unsigned int order = folio_order(folio); + int ret = 0; + + if (kpkeys_hardened_pgtables_enabled()) + ret = set_memory_pkey(addr, 1 << order, KPKEYS_PKEY_PGTABLES); + + WARN_ON(ret); + return ret; +} + +int kpkeys_unprotect_pgtable_memory(struct folio *folio) +{ + unsigned long addr = (unsigned long)folio_address(folio); + unsigned int order = folio_order(folio); + int ret = 0; + + if (kpkeys_hardened_pgtables_enabled()) + ret = set_memory_pkey(addr, 1 << order, KPKEYS_PKEY_DEFAULT); + + WARN_ON(ret); + return ret; +} + +void __init kpkeys_hardened_pgtables_enable(void) +{ + int ret; + + if (!arch_kpkeys_enabled()) + return; + + static_branch_enable(&kpkeys_hardened_pgtables_key); + ret = kernel_pgtables_set_pkey(KPKEYS_PKEY_PGTABLES); + WARN_ON(ret); +} diff --git a/security/Kconfig.hardening b/security/Kconfig.hardening index c17366ce8224..c2b0987768ca 100644 --- a/security/Kconfig.hardening +++ b/security/Kconfig.hardening @@ -334,6 +334,18 @@ config BUG_ON_DATA_CORRUPTION If unsure, say N. +config KPKEYS_HARDENED_PGTABLES + bool "Harden page tables using kernel pkeys" + depends on ARCH_HAS_KPKEYS_HARDENED_PGTABLES + help + This option makes all page tables mostly read-only by + allocating them with a non-default protection key (pkey) and + only enabling write access to that pkey in routines that are + expected to write to page table entries. + + This option has no effect if the system does not support + kernel pkeys. + endmenu config CC_HAS_RANDSTRUCT From patchwork Fri Apr 11 09:16:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 22296C36010 for ; Fri, 11 Apr 2025 09:45:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=nv2oqsrFF3oYfOvHVkuk73o0k7HCEGakiogCU9B9JNg=; b=UxGJdPKjFLUmNHDOkI31h5vDhN G03InOHwbm/JCHIf9cCRnqRCGAvpoWbnfAK1i5FA3SgbcWUBtj4zUUdfZR6BOzXTiZRfgDT6kQhBp PM/1dSrZ4liVBDM9cmvpZ2Ruf+1jQ7hoqOnRUVkghUoxKYS/njjbBAJtUrl3o4a3YeAo3gTdlv5nv DP+oKIyQH1D62mK9OiK2EurPFPnwDeyNCtzdWFlu6P1thLNEJEmmaAubaq1ktBf05abDAdaSlZUb5 92cY9JXghIGn76OKxQsGNYF45513PmhgZ7bBvqQJV8xpyI+aw3boCGmOkPX1Zc8vtjWUX1kC8YN50 SPuRgg4g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AxC-0000000DF8b-33is; Fri, 11 Apr 2025 09:45:42 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AWA-0000000DAeG-3IhV for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:17:48 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 30AA31596; Fri, 11 Apr 2025 02:17:46 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9E54D3F6A8; Fri, 11 Apr 2025 02:17:42 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 12/18] mm: Allow __pagetable_ctor() to fail Date: Fri, 11 Apr 2025 10:16:25 +0100 Message-ID: <20250411091631.954228-13-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_021746_914626_FBFB14A7 X-CRM114-Status: GOOD ( 12.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In preparation for adding construction hooks (that may fail) to __pagetable_ctor(), make __pagetable_ctor() return a bool, propagate it to pagetable_*_ctor() and handle failure in the generic {pud,p4d,pgd}_alloc. Signed-off-by: Kevin Brodsky --- include/asm-generic/pgalloc.h | 15 ++++++++++++--- include/linux/mm.h | 21 ++++++++++----------- 2 files changed, 22 insertions(+), 14 deletions(-) diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 3c8ec3bfea44..3e184f3ca37a 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -178,7 +178,10 @@ static inline pud_t *__pud_alloc_one_noprof(struct mm_struct *mm, unsigned long if (!ptdesc) return NULL; - pagetable_pud_ctor(ptdesc); + if (!pagetable_pud_ctor(ptdesc)) { + pagetable_free(ptdesc); + return NULL; + } return ptdesc_address(ptdesc); } #define __pud_alloc_one(...) alloc_hooks(__pud_alloc_one_noprof(__VA_ARGS__)) @@ -232,7 +235,10 @@ static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long if (!ptdesc) return NULL; - pagetable_p4d_ctor(ptdesc); + if (!pagetable_p4d_ctor(ptdesc)) { + pagetable_free(ptdesc); + return NULL; + } return ptdesc_address(ptdesc); } #define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__)) @@ -276,7 +282,10 @@ static inline pgd_t *__pgd_alloc_noprof(struct mm_struct *mm, unsigned int order if (!ptdesc) return NULL; - pagetable_pgd_ctor(ptdesc); + if (!pagetable_pgd_ctor(ptdesc)) { + pagetable_free(ptdesc); + return NULL; + } return ptdesc_address(ptdesc); } #define __pgd_alloc(...) alloc_hooks(__pgd_alloc_noprof(__VA_ARGS__)) diff --git a/include/linux/mm.h b/include/linux/mm.h index dd1b918dc294..43e35d41b850 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3077,12 +3077,13 @@ static inline bool ptlock_init(struct ptdesc *ptdesc) { return true; } static inline void ptlock_free(struct ptdesc *ptdesc) {} #endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ -static inline void __pagetable_ctor(struct ptdesc *ptdesc) +static inline bool __pagetable_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc); __folio_set_pgtable(folio); lruvec_stat_add_folio(folio, NR_PAGETABLE); + return true; } static inline void pagetable_dtor(struct ptdesc *ptdesc) @@ -3105,8 +3106,7 @@ static inline bool pagetable_pte_ctor(struct mm_struct *mm, { if (mm != &init_mm && !ptlock_init(ptdesc)) return false; - __pagetable_ctor(ptdesc); - return true; + return __pagetable_ctor(ptdesc); } pte_t *___pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp); @@ -3213,8 +3213,7 @@ static inline bool pagetable_pmd_ctor(struct mm_struct *mm, if (mm != &init_mm && !pmd_ptlock_init(ptdesc)) return false; ptdesc_pmd_pts_init(ptdesc); - __pagetable_ctor(ptdesc); - return true; + return __pagetable_ctor(ptdesc); } /* @@ -3236,19 +3235,19 @@ static inline spinlock_t *pud_lock(struct mm_struct *mm, pud_t *pud) return ptl; } -static inline void pagetable_pud_ctor(struct ptdesc *ptdesc) +static inline bool pagetable_pud_ctor(struct ptdesc *ptdesc) { - __pagetable_ctor(ptdesc); + return __pagetable_ctor(ptdesc); } -static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) +static inline bool pagetable_p4d_ctor(struct ptdesc *ptdesc) { - __pagetable_ctor(ptdesc); + return __pagetable_ctor(ptdesc); } -static inline void pagetable_pgd_ctor(struct ptdesc *ptdesc) +static inline bool pagetable_pgd_ctor(struct ptdesc *ptdesc) { - __pagetable_ctor(ptdesc); + return __pagetable_ctor(ptdesc); } extern void __init pagecache_init(void); From patchwork Fri Apr 11 09:16:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047971 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 53FD0C369A9 for ; Fri, 11 Apr 2025 09:47:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Dt/XoONL9y6fQB1g9UJTjYyCBLEHEeGxmo0qFMBni9E=; b=WBdlVze8q9H+rBu3DS4DqtqFZf wJUuFMjYfJGqH06E8S4SXW/7Pd0B4k+zb7+gjeipzDN3wfLzG+hedMw50A6owcNYCyE7XeW9m65GS 3kD7BM8ngMUXWRZRHmZrLtj3ojoQrlQnKepv+pwosDTjOE/ta+qrvr1dsDNu4zc3v216bhM8YoGgI /f4sKofFhAp4iZyfgPEhT9zIi1sginwyXgX25XwIi4DXZSNRHka/9+Y2S/u65PIYfU5A7YRpCcapL MEU0m9d7A0B+pkzFUUkpEaK0CfKAaSFzJ//B/rus5IOfCmL6zV3PrM6VW5ar6c4Uz7oqNSMagD+iI kcdKSRWw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3Ayy-0000000DFKJ-1aa2; Fri, 11 Apr 2025 09:47:32 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AWF-0000000DAg8-0diD for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:17:52 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 44125106F; Fri, 11 Apr 2025 02:17:50 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B0DF23F6A8; Fri, 11 Apr 2025 02:17:46 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 13/18] mm: Map page tables with privileged pkey Date: Fri, 11 Apr 2025 10:16:26 +0100 Message-ID: <20250411091631.954228-14-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_021751_230375_3129A47B X-CRM114-Status: GOOD ( 11.02 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org If CONFIG_KPKEYS_HARDENED_PGTABLES is enabled, map allocated page table pages using a privileged pkey (KPKEYS_PKEY_PGTABLES), so that page tables can only be written under guard(kpkeys_hardened_pgtables). This patch is a no-op if CONFIG_KPKEYS_HARDENED_PGTABLES is disabled (default). Signed-off-by: Kevin Brodsky --- include/linux/mm.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 43e35d41b850..411707dabe65 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -33,6 +33,7 @@ #include #include #include +#include struct mempolicy; struct anon_vma; @@ -3083,6 +3084,8 @@ static inline bool __pagetable_ctor(struct ptdesc *ptdesc) __folio_set_pgtable(folio); lruvec_stat_add_folio(folio, NR_PAGETABLE); + if (kpkeys_protect_pgtable_memory(folio)) + return false; return true; } @@ -3093,6 +3096,7 @@ static inline void pagetable_dtor(struct ptdesc *ptdesc) ptlock_free(ptdesc); __folio_clear_pgtable(folio); lruvec_stat_sub_folio(folio, NR_PAGETABLE); + kpkeys_unprotect_pgtable_memory(folio); } static inline void pagetable_dtor_free(struct ptdesc *ptdesc) From patchwork Fri Apr 11 09:16:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047972 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 25C9BC36010 for ; Fri, 11 Apr 2025 09:49:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Cd8SVz56UFDTvkQjpDWLbjdDftI6n7YlnUbxT3vQe+A=; b=Q1XklHJoOkp8rhluktUfgy8Qtx Re0EMrS0hj1046HKvOguUH1fkO5zNK1dDPIIAVokIX/zbgOgmI60QMn6xaAAdu21t3U5cvKSxQZhs gEG8KUTSLrqxSrFWIgYRzsKHl+dI76BW1dutmvX/XUl01j4XiOTNjrfTJsg+u4UW6X5zsD7KVCgTa M9ZL0YaKsoHBv0EdpeH/aaBdruLzzgmEJd6Ha74rXyz5S4StD8vkrhA8SotZ0l8AKTj3E1I0lL7y4 apxzxjDvLeaeqUefFk8wctWkF9hHxjONa+s1CFb8XxNpGZ1383yqd8GlsRCiOcrxnL522QTziJHyo 2RcTFJzA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3B0l-0000000DFVr-0MVD; Fri, 11 Apr 2025 09:49:23 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AWI-0000000DAhO-3hnz for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:17:56 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5604C1692; Fri, 11 Apr 2025 02:17:54 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C41DE3F6A8; Fri, 11 Apr 2025 02:17:50 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 14/18] arm64: kpkeys: Support KPKEYS_LVL_PGTABLES Date: Fri, 11 Apr 2025 10:16:27 +0100 Message-ID: <20250411091631.954228-15-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_021754_962985_A40D3558 X-CRM114-Status: GOOD ( 12.68 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Enable RW access to KPKEYS_PKEY_PGTABLES (used to map page table pages) if switching to KPKEYS_LVL_PGTABLES, otherwise only grant RO access. Signed-off-by: Kevin Brodsky --- arch/arm64/include/asm/kpkeys.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kpkeys.h b/arch/arm64/include/asm/kpkeys.h index 79ae33388088..64d6e22740ec 100644 --- a/arch/arm64/include/asm/kpkeys.h +++ b/arch/arm64/include/asm/kpkeys.h @@ -12,7 +12,8 @@ * Equivalent to por_set_kpkeys_level(0, KPKEYS_LVL_DEFAULT), but can also be * used in assembly. */ -#define POR_EL1_INIT POR_ELx_PERM_PREP(KPKEYS_PKEY_DEFAULT, POE_RWX) +#define POR_EL1_INIT (POR_ELx_PERM_PREP(KPKEYS_PKEY_DEFAULT, POE_RWX) | \ + POR_ELx_PERM_PREP(KPKEYS_PKEY_PGTABLES, POE_R)) #ifndef __ASSEMBLY__ @@ -26,6 +27,8 @@ static inline bool arch_kpkeys_enabled(void) static inline u64 por_set_kpkeys_level(u64 por, int level) { por = por_elx_set_pkey_perms(por, KPKEYS_PKEY_DEFAULT, POE_RWX); + por = por_elx_set_pkey_perms(por, KPKEYS_PKEY_PGTABLES, + level == KPKEYS_LVL_PGTABLES ? POE_RW : POE_R); return por; } From patchwork Fri Apr 11 09:16:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047996 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6309BC36010 for ; Fri, 11 Apr 2025 09:51:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=kfWEv7Ptu52vQV052WCDfhDg7Nqo7t04Slp2nei36Js=; b=p3rCOy749wZD+qwhZqSM3mmYWK rNi0S54nGYUlpzBHzzH/FriP2zL57WEhto1bgDeqTcBnKUzxooooCY5zNAxvujkEiQf3iZDaFwRKl TvoFU+RPviXr1CAFxkjMIa/R84r+8ncOjORZU+Jm62QxWUYgpBi9a89Phmr4PSW1FceaECyjTPKmI h8LolUajDwBSjXBfqlP/FIe9zfLvIpbTB3Ulv+aqrw7SvRGuRNNiewpJVT/eXDj0xtScTY/wfX5Cn oZygwSvfGFFOaFf/oxeVOk7wMKadP46Jfb6FRkx1iHQbE2hybQtj4ZWqkLE0dQlzBGxIkK+A3lTgs 5F7+jeEA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3B2W-0000000DFfU-3fkj; Fri, 11 Apr 2025 09:51:12 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AWN-0000000DAj4-1rvZ for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:18:00 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9EFFC106F; Fri, 11 Apr 2025 02:17:58 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D6AEE3F6A8; Fri, 11 Apr 2025 02:17:54 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 15/18] arm64: mm: Guard page table writes with kpkeys Date: Fri, 11 Apr 2025 10:16:28 +0100 Message-ID: <20250411091631.954228-16-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_021759_569769_675B612C X-CRM114-Status: GOOD ( 16.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When CONFIG_KPKEYS_HARDENED_PGTABLES is enabled, page tables (both user and kernel) are mapped with a privileged pkey in the linear mapping. As a result, they can only be written at an elevated kpkeys level. Introduce a kpkeys guard that sets POR_EL1 appropriately to allow writing to page tables, and use this guard wherever necessary. The scope is kept as small as possible, so that POR_EL1 is quickly reset to its default value. Where atomics are involved, the guard's scope encompasses the whole loop to avoid switching POR_EL1 unnecessarily. This patch is a no-op if CONFIG_KPKEYS_HARDENED_PGTABLES is disabled (default). Signed-off-by: Kevin Brodsky --- arch/arm64/include/asm/pgtable.h | 22 +++++++++++++++++++++- arch/arm64/mm/fault.c | 2 ++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 6c780c428a6d..7929b79cd6b1 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -39,6 +39,14 @@ #include #include #include +#include + +#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES +KPKEYS_GUARD_COND(kpkeys_hardened_pgtables, KPKEYS_LVL_PGTABLES, + kpkeys_hardened_pgtables_enabled()) +#else +KPKEYS_GUARD_NOOP(kpkeys_hardened_pgtables) +#endif static inline void emit_pte_barriers(void) { @@ -366,6 +374,7 @@ static inline pte_t pte_clear_uffd_wp(pte_t pte) static inline void __set_pte_nosync(pte_t *ptep, pte_t pte) { + guard(kpkeys_hardened_pgtables)(); WRITE_ONCE(*ptep, pte); } @@ -843,6 +852,7 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) } #endif /* __PAGETABLE_PMD_FOLDED */ + guard(kpkeys_hardened_pgtables)(); WRITE_ONCE(*pmdp, pmd); if (pmd_valid(pmd)) @@ -909,6 +919,7 @@ static inline void set_pud(pud_t *pudp, pud_t pud) return; } + guard(kpkeys_hardened_pgtables)(); WRITE_ONCE(*pudp, pud); if (pud_valid(pud)) @@ -990,6 +1001,7 @@ static inline void set_p4d(p4d_t *p4dp, p4d_t p4d) return; } + guard(kpkeys_hardened_pgtables)(); WRITE_ONCE(*p4dp, p4d); queue_pte_barriers(); } @@ -1118,6 +1130,7 @@ static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) return; } + guard(kpkeys_hardened_pgtables)(); WRITE_ONCE(*pgdp, pgd); queue_pte_barriers(); } @@ -1317,6 +1330,7 @@ static inline int __ptep_test_and_clear_young(struct vm_area_struct *vma, { pte_t old_pte, pte; + guard(kpkeys_hardened_pgtables)(); pte = __ptep_get(ptep); do { old_pte = pte; @@ -1363,7 +1377,10 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, static inline pte_t ptep_get_and_clear_anysz(struct mm_struct *mm, pte_t *ptep, unsigned long pgsize) { - pte_t pte = __pte(xchg_relaxed(&pte_val(*ptep), 0)); + pte_t pte; + + scoped_guard(kpkeys_hardened_pgtables) + pte = __pte(xchg_relaxed(&pte_val(*ptep), 0)); switch (pgsize) { case PAGE_SIZE: @@ -1434,6 +1451,7 @@ static inline void ___ptep_set_wrprotect(struct mm_struct *mm, { pte_t old_pte; + guard(kpkeys_hardened_pgtables)(); do { old_pte = pte; pte = pte_wrprotect(pte); @@ -1467,6 +1485,7 @@ static inline void __clear_young_dirty_pte(struct vm_area_struct *vma, { pte_t old_pte; + guard(kpkeys_hardened_pgtables)(); do { old_pte = pte; @@ -1514,6 +1533,7 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t pmd) { page_table_check_pmd_set(vma->vm_mm, pmdp, pmd); + guard(kpkeys_hardened_pgtables)(); return __pmd(xchg_relaxed(&pmd_val(*pmdp), pmd_val(pmd))); } #endif diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index ec0a337891dd..5681553e2db9 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -220,6 +220,8 @@ int __ptep_set_access_flags(struct vm_area_struct *vma, if (pte_same(pte, entry)) return 0; + guard(kpkeys_hardened_pgtables)(); + /* only preserve the access flags and write permission */ pte_val(entry) &= PTE_RDONLY | PTE_AF | PTE_WRITE | PTE_DIRTY; From patchwork Fri Apr 11 09:16:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047997 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CA23AC36010 for ; Fri, 11 Apr 2025 09:53:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=OWDvh2bFfbHkJemGgaNc6HxyvRtDNFqCF7qScL2ZdLs=; b=NlHSEDaftJpGjyDrsJvGqZKybM EMzvo88BQAIADKqG7FPc7NUCQiayhmcHYywTdx+KCQ3v/XBn/ErBATbh7wwT5ULXYwF+o9SpLIubS Bl4e6ZT55seNf/6hqj32o1cfFlV/utartEL/SR2qE4JGLnIO7vw/t4kUmKh9s4BZAohDDcFBYuOq8 GLFS8SO71tbPvqaj1hvyeTPObPm5exSaIPaxkTl+wzvgu+pZYoU28dHjSJEWhf18GbfrDOTOhp5Fk SfRW/HhsnD2dZRNbhrnYHQdUF4CXjhm1RsmWKSEnOz+3F0tEzuQm6DG9ONAnGQeHqToPprTDEzUnC OzQD8WNg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3B4J-0000000DFu8-2PWT; Fri, 11 Apr 2025 09:53:03 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AWR-0000000DAm1-0ByG for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:18:04 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7BCCF1596; Fri, 11 Apr 2025 02:18:02 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E90163F792; Fri, 11 Apr 2025 02:17:58 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 16/18] arm64: Enable kpkeys_hardened_pgtables support Date: Fri, 11 Apr 2025 10:16:29 +0100 Message-ID: <20250411091631.954228-17-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_021803_131516_A4E0D0EC X-CRM114-Status: GOOD ( 12.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org kpkeys_hardened_pgtables should be enabled as early as possible (if selected). It does however require kpkeys being available, which means on arm64 POE being detected and enabled. POE is a boot feature, so calling kpkeys_hardened_pgtables_enable() just after setup_boot_cpu_features() in smp_prepare_boot_cpu() is the best we can do. With that done, all the bits are in place and we can advertise support for kpkeys_hardened_pgtables by selecting ARCH_HAS_KPKEYS_HARDENED_PGTABLES if ARM64_POE is enabled. Signed-off-by: Kevin Brodsky --- arch/arm64/Kconfig | 1 + arch/arm64/kernel/smp.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 5f81db8134da..3d90b895e2ef 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2199,6 +2199,7 @@ config ARM64_POE select ARCH_USES_HIGH_VMA_FLAGS select ARCH_HAS_PKEYS select ARCH_HAS_KPKEYS + select ARCH_HAS_KPKEYS_HARDENED_PGTABLES help The Permission Overlay Extension is used to implement Memory Protection Keys. Memory Protection Keys provides a mechanism for diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 3b3f6b56e733..074cab55f9db 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -35,6 +35,7 @@ #include #include #include +#include #include #include @@ -468,6 +469,7 @@ void __init smp_prepare_boot_cpu(void) if (system_uses_irq_prio_masking()) init_gic_priority_masking(); + kpkeys_hardened_pgtables_enable(); kasan_init_hw_tags(); /* Init percpu seeds for random tags after cpus are set up. */ kasan_init_sw_tags(); From patchwork Fri Apr 11 09:16:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14047998 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 15832C36010 for ; Fri, 11 Apr 2025 09:55:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=bdMqiFDP4vUIg95dX44Zi6HrmWZHm2nf87Jx6pv15M8=; b=xV7nEQ8BAij7j7yYI9kvw9qGCq XRdu2lCt4HIg8hiVNjrzdhQIPswxTRbTDjkO6qFgck1BYQdbQnxZg8lVzDhX+EV6uwXF2/kWa/EQN M9MlCX0gpxVvW1y18U9DmH7wlZKylMpCIRf8qn24PdyRb6+yVBM3TKPUvMiQ9lkONAmCjE3tQS48R kHUsuhjrm8t8Xx0wg8Fv0vLgJMgOFkakf4JG+yl8yrgdBg9aItvnw3uidxRHXtMRUY8bICqgPbGTL H0fXaWvVYidIq5qbnUM42AR5EJ0tM4wvtaLny/dWLGjCch9MsAz4udMAAK8FPTO0JMY2afAPsFGxK QD0Aj5xQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3B65-0000000DGGW-10C9; Fri, 11 Apr 2025 09:54:53 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AWV-0000000DApE-2Dbo for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:18:08 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8E785106F; Fri, 11 Apr 2025 02:18:06 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 07A993F6A8; Fri, 11 Apr 2025 02:18:02 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 17/18] mm: Add basic tests for kpkeys_hardened_pgtables Date: Fri, 11 Apr 2025 10:16:30 +0100 Message-ID: <20250411091631.954228-18-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_021807_659304_EB7893E1 X-CRM114-Status: GOOD ( 21.39 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add basic tests for the kpkeys_hardened_pgtables feature: try to perform a direct write to kernel and user page table entries and ensure it fails. Signed-off-by: Kevin Brodsky --- mm/Makefile | 1 + mm/tests/kpkeys_hardened_pgtables_kunit.c | 97 +++++++++++++++++++++++ security/Kconfig.hardening | 12 +++ 3 files changed, 110 insertions(+) create mode 100644 mm/tests/kpkeys_hardened_pgtables_kunit.c diff --git a/mm/Makefile b/mm/Makefile index 0f30cc85c6f1..778df78c65d5 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -149,3 +149,4 @@ obj-$(CONFIG_EXECMEM) += execmem.o obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_PT_RECLAIM) += pt_reclaim.o obj-$(CONFIG_KPKEYS_HARDENED_PGTABLES) += kpkeys_hardened_pgtables.o +obj-$(CONFIG_KPKEYS_HARDENED_PGTABLES_KUNIT_TEST) += tests/kpkeys_hardened_pgtables_kunit.o diff --git a/mm/tests/kpkeys_hardened_pgtables_kunit.c b/mm/tests/kpkeys_hardened_pgtables_kunit.c new file mode 100644 index 000000000000..6e29721262b2 --- /dev/null +++ b/mm/tests/kpkeys_hardened_pgtables_kunit.c @@ -0,0 +1,97 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include + +KUNIT_DEFINE_ACTION_WRAPPER(vfree_wrapper, vfree, const void *); + +static void write_linear_map_pte(struct kunit *test) +{ + pte_t *ptep; + pte_t pte; + int ret; + + if (!arch_kpkeys_enabled()) + kunit_skip(test, "kpkeys are not supported"); + + /* + * The choice of address is mostly arbitrary - we just need something + * that falls in the linear mapping, such as the address of a global + * variable. + */ + ptep = virt_to_kpte((unsigned long)&init_mm); + KUNIT_ASSERT_NOT_NULL_MSG(test, ptep, "Failed to get PTE"); + + pte = ptep_get(ptep); + pte = set_pte_bit(pte, __pgprot(PTE_WRITE)); + ret = copy_to_kernel_nofault(ptep, &pte, sizeof(pte)); + KUNIT_EXPECT_EQ_MSG(test, ret, -EFAULT, + "Direct PTE write wasn't prevented"); +} + +static void write_kernel_vmalloc_pte(struct kunit *test) +{ + void *mem; + pte_t *ptep; + pte_t pte; + int ret; + + if (!arch_kpkeys_enabled()) + kunit_skip(test, "kpkeys are not supported"); + + mem = vmalloc(PAGE_SIZE); + KUNIT_ASSERT_NOT_NULL(test, mem); + ret = kunit_add_action_or_reset(test, vfree_wrapper, mem); + KUNIT_ASSERT_EQ(test, ret, 0); + + ptep = virt_to_kpte((unsigned long)mem); + KUNIT_ASSERT_NOT_NULL_MSG(test, ptep, "Failed to get PTE"); + + pte = ptep_get(ptep); + pte = set_pte_bit(pte, __pgprot(PTE_WRITE)); + ret = copy_to_kernel_nofault(ptep, &pte, sizeof(pte)); + KUNIT_EXPECT_EQ_MSG(test, ret, -EFAULT, + "Direct PTE write wasn't prevented"); +} + +static void write_user_pmd(struct kunit *test) +{ + pmd_t *pmdp; + pmd_t pmd; + unsigned long uaddr; + int ret; + + if (!arch_kpkeys_enabled()) + kunit_skip(test, "kpkeys are not supported"); + + uaddr = kunit_vm_mmap(test, NULL, 0, PAGE_SIZE, PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE | MAP_POPULATE, 0); + KUNIT_ASSERT_NE_MSG(test, uaddr, 0, "Could not create userspace mm"); + + /* We passed MAP_POPULATE so a PMD should already be allocated */ + pmdp = pmd_off(current->mm, uaddr); + KUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, "Failed to get PMD"); + + pmd = pmdp_get(pmdp); + pmd = set_pmd_bit(pmd, __pgprot(PROT_SECT_NORMAL)); + ret = copy_to_kernel_nofault(pmdp, &pmd, sizeof(pmd)); + KUNIT_EXPECT_EQ_MSG(test, ret, -EFAULT, + "Direct PMD write wasn't prevented"); +} + +static struct kunit_case kpkeys_hardened_pgtables_test_cases[] = { + KUNIT_CASE(write_linear_map_pte), + KUNIT_CASE(write_kernel_vmalloc_pte), + KUNIT_CASE(write_user_pmd), + {} +}; + +static struct kunit_suite kpkeys_hardened_pgtables_test_suite = { + .name = "Hardened pgtables using kpkeys", + .test_cases = kpkeys_hardened_pgtables_test_cases, +}; +kunit_test_suite(kpkeys_hardened_pgtables_test_suite); + +MODULE_DESCRIPTION("Tests for the kpkeys_hardened_pgtables feature"); +MODULE_LICENSE("GPL"); diff --git a/security/Kconfig.hardening b/security/Kconfig.hardening index c2b0987768ca..cc3458510e97 100644 --- a/security/Kconfig.hardening +++ b/security/Kconfig.hardening @@ -346,6 +346,18 @@ config KPKEYS_HARDENED_PGTABLES This option has no effect if the system does not support kernel pkeys. +config KPKEYS_HARDENED_PGTABLES_KUNIT_TEST + tristate "KUnit tests for kpkeys_hardened_pgtables" if !KUNIT_ALL_TESTS + depends on KPKEYS_HARDENED_PGTABLES + depends on KUNIT + default KUNIT_ALL_TESTS + help + Enable this option to check that the kpkeys_hardened_pgtables feature + functions as intended, i.e. prevents arbitrary writes to user and + kernel page tables. + + If unsure, say N. + endmenu config CC_HAS_RANDSTRUCT From patchwork Fri Apr 11 09:16:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14048006 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 23750C36010 for ; Fri, 11 Apr 2025 09:56:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=LvMNmUJw5YHbXOAt8Uq/tq9yqI5eyvYGL9Ei7Fvb7sQ=; b=jtLAGiU7p+HcI9eKa37lflbmfq lg/j65A+xLrhCQuFfXxdQ8isO/ZnR7Q3mB+wFAdOUE7JgCs22S5d7DIBaLV5A+gp9ENMXPvw/FX8+ 9AyotuqXw93ihTNhCd8KdGr4WT1bcqDQ1CWtP7pnBby5WHsf1EH0oFO7ylVe4RO35+/L4zWxypI0J p8B6I8yO7t7ctwqXFYeEW8jvqcbtr4q9PMTfxFlHrBETMvRx6T/ho+3iS2OTyeGPAHIQVK72hiO/B j4IICida776PRbBzK9q38HO+R7oGY3QCsWJUIyjcMkF7Edz2G2AJwLesBACFgVXW5TqlYP/zMlavf 06fbbO6A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3B7t-0000000DGj1-03RZ; Fri, 11 Apr 2025 09:56:45 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u3AWZ-0000000DAr9-119F for linux-arm-kernel@lists.infradead.org; Fri, 11 Apr 2025 09:18:13 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A04EC1596; Fri, 11 Apr 2025 02:18:10 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 1A69D3F6A8; Fri, 11 Apr 2025 02:18:06 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Andrew Morton , Mark Brown , Catalin Marinas , Dave Hansen , David Hildenbrand , Ira Weiny , Jann Horn , Jeff Xu , Joey Gouly , Kees Cook , Linus Walleij , Andy Lutomirski , Marc Zyngier , Peter Zijlstra , Pierre Langlois , Quentin Perret , Rick Edgecombe , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-arm-kernel@lists.infradead.org, x86@kernel.org Subject: [RFC PATCH v4 18/18] arm64: mm: Batch kpkeys level switches Date: Fri, 11 Apr 2025 10:16:31 +0100 Message-ID: <20250411091631.954228-19-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250411091631.954228-1-kevin.brodsky@arm.com> References: <20250411091631.954228-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250411_021811_372264_D5A3B7FA X-CRM114-Status: GOOD ( 21.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The kpkeys_hardened_pgtables feature currently switches kpkeys level in every helper that writes to page tables, such as set_pte(). With kpkeys implemented using POE, this entails a pair of ISBs whenever such helper is called. A simple way to reduce this overhead is to make use of the lazy_mmu mode, which has recently been adopted on arm64 to batch barriers (DSB/ISB) when updating kernel pgtables [1]. Reusing the TIF_LAZY_MMU flag introduced by this series, we amend the kpkeys_hardened_pgtables guard so that no level switch (i.e. POR_EL1 update) is issued while that flag is set. Instead, we switch to KPKEYS_LVL_PGTABLES when entering lazy_mmu mode, and restore the previous level when exiting it. Restoring the previous kpkeys level requires storing the original value of POR_EL1 somewhere. This is a full 64-bit value so we cannot simply use a TIF flag, but since lazy_mmu sections cannot nest, some sort of thread-local variable would do the trick. There is no straightforward way to reuse current->thread.por_el1 for that purpose - this is where the current value of POR_EL1 is stored on a context switch, i.e. the value corresponding to KPKEYS_LVL_PGTABLES inside a lazy_mmu section. Instead, we add a new member to thread_struct to hold that value temporarily. This isn't optimal as that member is unused outside of lazy_mmu sections, but it is the simplest option. A further optimisation this patch makes is to merge the ISBs when exiting lazy_mmu mode. That is, if an ISB is going to be issued by emit_pte_barriers() because kernel pgtables were modified in the lazy_mmu section, we skip the ISB after restoring POR_EL1. This is done by checking TIF_LAZY_MMU_PENDING and ensuring that POR_EL1 is restored before emit_pte_barriers() is called. Checking TIF_LAZY_MMU flag in all pgtable writers is currently overkill, as lazy_mmu sections are only used at the lowest level of page tables. In other words, set_pgd() (for instance) will never be called with TIF_LAZY_MMU set. However, such higher-level helpers are called relatively infrequently and the overhead of checking a TIF flag is low. The flag is therefore checked in all cases for simplicity's sake, just like in [1]. [1] https://lore.kernel.org/linux-mm/20250304150444.3788920-1-ryan.roberts@arm.com/ Signed-off-by: Kevin Brodsky --- arch/arm64/include/asm/pgtable.h | 37 +++++++++++++++++++++++++++++- arch/arm64/include/asm/processor.h | 1 + 2 files changed, 37 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7929b79cd6b1..61dee76be515 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -43,11 +43,40 @@ #ifdef CONFIG_KPKEYS_HARDENED_PGTABLES KPKEYS_GUARD_COND(kpkeys_hardened_pgtables, KPKEYS_LVL_PGTABLES, - kpkeys_hardened_pgtables_enabled()) + kpkeys_hardened_pgtables_enabled() && + !test_thread_flag(TIF_LAZY_MMU)) #else KPKEYS_GUARD_NOOP(kpkeys_hardened_pgtables) #endif +static void kpkeys_lazy_mmu_enter(void) +{ + if (!kpkeys_hardened_pgtables_enabled()) + return; + + current->thread.por_el1_lazy_mmu = kpkeys_set_level(KPKEYS_LVL_PGTABLES); +} + +static void kpkeys_lazy_mmu_exit(void) +{ + u64 saved_por_el1; + + if (!kpkeys_hardened_pgtables_enabled()) + return; + + saved_por_el1 = current->thread.por_el1_lazy_mmu; + + /* + * We skip any barrier if TIF_LAZY_MMU_PENDING is set: + * emit_pte_barriers() will issue an ISB just after this function + * returns. + */ + if (test_thread_flag(TIF_LAZY_MMU_PENDING)) + __kpkeys_set_pkey_reg_nosync(saved_por_el1); + else + arch_kpkeys_restore_pkey_reg(saved_por_el1); +} + static inline void emit_pte_barriers(void) { /* @@ -83,6 +112,7 @@ static inline void arch_enter_lazy_mmu_mode(void) VM_WARN_ON(test_thread_flag(TIF_LAZY_MMU)); set_thread_flag(TIF_LAZY_MMU); + kpkeys_lazy_mmu_enter(); } static inline void arch_flush_lazy_mmu_mode(void) @@ -93,6 +123,11 @@ static inline void arch_flush_lazy_mmu_mode(void) static inline void arch_leave_lazy_mmu_mode(void) { + /* + * The ordering should be preserved to allow kpkeys_lazy_mmu_exit() + * to skip any barrier when TIF_LAZY_MMU_PENDING is set. + */ + kpkeys_lazy_mmu_exit(); arch_flush_lazy_mmu_mode(); clear_thread_flag(TIF_LAZY_MMU); } diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 0afaf96ca699..14a4b483098d 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -186,6 +186,7 @@ struct thread_struct { u64 tpidr2_el0; u64 por_el0; u64 por_el1; + u64 por_el1_lazy_mmu; #ifdef CONFIG_ARM64_GCS unsigned int gcs_el0_mode; unsigned int gcs_el0_locked;