From patchwork Tue Oct 1 22:59:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 13818921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E0BDCF3189 for ; Tue, 1 Oct 2024 23:03:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0BC61440160; Tue, 1 Oct 2024 19:03:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 06C0868002B; Tue, 1 Oct 2024 19:03:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E27D2440160; Tue, 1 Oct 2024 19:03:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C290D68002B for ; Tue, 1 Oct 2024 19:03:54 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 864B78153F for ; Tue, 1 Oct 2024 23:03:54 +0000 (UTC) X-FDA: 82626562788.15.FAC3439 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf15.hostedemail.com (Postfix) with ESMTP id A2C5BA0006 for ; Tue, 1 Oct 2024 23:03:52 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=HUx8wIEC; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf15.hostedemail.com: domain of broonie@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=broonie@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727823738; a=rsa-sha256; cv=none; b=H5ZmV+uJ0CY18X4aiDsPI5I7mIsV5CJuth7h8eNpyAXxDckvQeBJfHITv8EF4AILGyjvUp WdsksKtUPuJsL4xYrSO+U4vb5EUwYYDQ/6PzDVjaL/TFnMlrty1g8eQ2a8CMvFf3PqYRHo YE215uecf3vODYKvRE7AOuSb0AJPfBY= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=HUx8wIEC; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf15.hostedemail.com: domain of broonie@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=broonie@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727823738; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rkhNPrB550gX7mvmN+oI1OSILfZOjyJO3ye3nAYBDYg=; b=D603GfZjt0S89RhSXuFfb9/At76iBRtYeuBSJeJCmFgbIP488Xjj7co2pcN47YS2Wo5I4+ MrmyNiM75F7idui77l1yR3LRcDJk0byfyARkqumyPjIjQzMMfiOdVa9ftRmQbHL6G/AJkI OH3LPt+DQGU9t8GZ/uocqEA0628vfjs= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id BBA375C05A0; Tue, 1 Oct 2024 23:03:47 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9525BC4CEC6; Tue, 1 Oct 2024 23:03:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1727823831; bh=B+jxXTUkTSef0cvYwALMxetqKDkj+z2yhbykonxuIvw=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=HUx8wIECR9lToVgEuuRmL1LTcof2SadUqhIeeq87HK6Vh5V0HYCpD8XsImAgHM0+i t9MgwEkfFPeiUZrwIsl1rbxozChdiha+NTSyAa1S7yVv7u6cfe+F5QUsqLXI7DHTOH px22G4UkMeFRmJidB5PidQ7keLIJUIo/fue980rAKoStkksx0adimrxQVGaZpjzIOk qm0jRfMf3bI1fSa9ha3mn9r6pDP84uJzOaF2uEuMJ0I72tNBqhhgMzrm6m/EqHGi+B kUtOj7EaSX2hkutQLSZ3Ozf5B0Y83qSLL/ev1DtwcbqyiH4Dfsjk+WQQXjCKMF1iPO k2Z0V5P25+TqA== From: Mark Brown Date: Tue, 01 Oct 2024 23:59:00 +0100 Subject: [PATCH v13 21/40] arm64/gcs: Context switch GCS state for EL0 MIME-Version: 1.0 Message-Id: <20241001-arm64-gcs-v13-21-222b78d87eee@kernel.org> References: <20241001-arm64-gcs-v13-0-222b78d87eee@kernel.org> In-Reply-To: <20241001-arm64-gcs-v13-0-222b78d87eee@kernel.org> To: Catalin Marinas , Will Deacon , Jonathan Corbet , Andrew Morton , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Arnd Bergmann , Oleg Nesterov , Eric Biederman , Shuah Khan , "Rick P. Edgecombe" , Deepak Gupta , Ard Biesheuvel , Szabolcs Nagy , Kees Cook Cc: "H.J. Lu" , Paul Walmsley , Palmer Dabbelt , Albert Ou , Florian Weimer , Christian Brauner , Thiago Jung Bauermann , Ross Burton , David Spickett , Yury Khrustalev , Wilco Dijkstra , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Mark Brown X-Mailer: b4 0.15-dev-99b12 X-Developer-Signature: v=1; a=openpgp-sha256; l=7335; i=broonie@kernel.org; h=from:subject:message-id; bh=B+jxXTUkTSef0cvYwALMxetqKDkj+z2yhbykonxuIvw=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBm/H7UABcgRepLC8/BSr/31M9cejBZhSyBGfKixQ4S CY8NnOqJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZvx+1AAKCRAk1otyXVSH0C3MB/ 9LkMlFMmSzQNqWzPn4u2mp4miFf2NFWEYmSg15LjV8XbEQl6b8hH0BEHWBfNCxBQOtDj/yW/9fkL4/ 1GS6g9GYcoXPS8j4UBZVxHhFOS5d1nATMSVL4vmF2wVD7HeHxcLbLnU+jnfoyM58HBHLRhiwcyyXrO CKZTtB1PWAIwtA8YMiTSPURfF/u+ngYFysvvoizBewFLIIg08trjdUoSjHZ2yefmXcif6Z4UMao5fh Jt+OQEWq6p/cgvHM4d/H2L95mmxDgM/w61k+s/R/bP91pdTY2mFnsYL/21j/lZPLw7LykOnCK5mHn6 numrEQSMpJt8eRfpLOQwyzWjsIyudp X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-Rspamd-Queue-Id: A2C5BA0006 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: f46qc1bbknyhytbdgi1jnwpftfdrir5q X-HE-Tag: 1727823832-417885 X-HE-Meta: U2FsdGVkX1/5bYsAddvDGJ6mEsjSKk/8R7LbC5DYpynJ1b5aqfMkOgA5HU7dzmZ0dtLAP8p5Yylm/JZbx4Q+n2z7bRJZ7Q0CqYeDGjOvjoPJrE5UEHMsWcz3UWJUS5SCNmeZJW31w5p8W7s/nfYDiBfuYg1Eu7szBFZL3+NPHwNJk+nNXpOS0vyVFkncTFZzYMBYiIOKUZhOL3lVRGrpjngBC1fnMlYEEoBpnIS+Nn2nFF3aLI9pX6f8CuATb2fB+d0QtQcEze6YcPLXu9zFQBk8/46GOalliGva2O5mQSP73AZT7Ww3KIksVm2KGrhpEFWtXubozD50qug/pOCriZDSHVXzUZUxqmTpdZXL/l64yiEHlxQ25BpMRXXEisN2vlJ9p+wbP7jUX46p8g518jrXyRnd2+R7EXUbCEcNCM9GtiCcMSmB4YG5dYHy+lCs9s1CiQovH3RlYsIZ521gOp7VpdUs1aVstezDaAwF+1fLFuWhHjRNvGNL784tJx5lvaf6X3ZCYmHBASN5QrC20lVC0QHeWacvPRrg7A00ytoRuAFf4wix0uGoTyysmsFB+qzkqaHTMDjERksM0bSMECYs83Pq8avKy8UpPBqm/E+j0RLsDjawnHQ8FUN2zbjWmlMxKVDSIanWv2tHGDkUadDBi8Vqx3HviVqo6xYpRYiSX7i4BLKBMmxmkBoNcVMSW0wLZ1G+bdgmieTrwBBAo+dqonQ8M5rXEdM8aCnFVj3LaXM7qGoC85E42jHzsNnPs6w4v6n4ui9/wRQNxE1IxUMSgrlMIP8UeGhxTAZppOwTN7zUZJyivfCsqSd2mJZCjVzqOIyABfDk42X7/2gz7uS1ynHSd0Ah4wzAl1rwxsJBwc8NuU09iINimHhlD5S+qNFbE/YNRX5tLOvXt1idSCzmV/WBmv7LEyW5FZAiZSRoVUfuGEyyQcBtd0kcwgjPSAnBpfZ7hhLtRk0Ynbr oo0C3gPL 65MTkxVXOThlelYSSlhsL++wmkJOUXYz110knmWy9PnQFkNQtGAI+Je2D84mI+/kuXq6HSTwnNAqJSOZe04fHdAy66QLQhJsOVpFxuSpi1Svfr81JYSMGt0Ji5oDPiTVfrDsvHDU5WwNMOyFMcyzk+wRrPsqrt4nGpTRUgi+Hpb8s2gIUlGz8IqnyFb1a0NgGlXG3pOzqnxaJ/oIsryfxFvlr9g4KK3nI1+m+oPvJGiXwQOHtW6QycJvHHJMNLqsq3xBcqPd+YjUNNPiPdL/pXli2GuwuVU7iXF2sFSRq4MJGCBZYIzzQGjK3c+uI2CYwSTKOEnkwtnQItMnwnz+DGmxg4/icboLdQ/b/jKx0EUBslZp4Fk8J+TEE06DMGVQhoNR+xizRtZZvEQTvGZzX51sIocbUspnOiR8hlM3QR1aXhMTXHc3JsYFU+A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are two registers controlling the GCS state of EL0, GCSPR_EL0 which is the current GCS pointer and GCSCRE0_EL1 which has enable bits for the specific GCS functionality enabled for EL0. Manage these on context switch and process lifetime events, GCS is reset on exec(). Also ensure that any changes to the GCS memory are visible to other PEs and that changes from other PEs are visible on this one by issuing a GCSB DSYNC when moving to or from a thread with GCS. Since the current GCS configuration of a thread will be visible to userspace we store the configuration in the format used with userspace and provide a helper which configures the system register as needed. On systems that support GCS we always allow access to GCSPR_EL0, this facilitates reporting of GCS faults if userspace implements disabling of GCS on error - the GCS can still be discovered and examined even if GCS has been disabled. Reviewed-by: Catalin Marinas Reviewed-by: Thiago Jung Bauermann Signed-off-by: Mark Brown --- arch/arm64/include/asm/gcs.h | 24 +++++++++++++++ arch/arm64/include/asm/processor.h | 6 ++++ arch/arm64/kernel/process.c | 62 ++++++++++++++++++++++++++++++++++++++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/gcs.c | 42 ++++++++++++++++++++++++++ 5 files changed, 135 insertions(+) diff --git a/arch/arm64/include/asm/gcs.h b/arch/arm64/include/asm/gcs.h index 7c5e95218db6..04594ef59dad 100644 --- a/arch/arm64/include/asm/gcs.h +++ b/arch/arm64/include/asm/gcs.h @@ -48,4 +48,28 @@ static inline u64 gcsss2(void) return Xt; } +#ifdef CONFIG_ARM64_GCS + +static inline bool task_gcs_el0_enabled(struct task_struct *task) +{ + return current->thread.gcs_el0_mode & PR_SHADOW_STACK_ENABLE; +} + +void gcs_set_el0_mode(struct task_struct *task); +void gcs_free(struct task_struct *task); +void gcs_preserve_current_state(void); + +#else + +static inline bool task_gcs_el0_enabled(struct task_struct *task) +{ + return false; +} + +static inline void gcs_set_el0_mode(struct task_struct *task) { } +static inline void gcs_free(struct task_struct *task) { } +static inline void gcs_preserve_current_state(void) { } + +#endif + #endif diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 1438424f0064..5260788247d8 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -185,6 +185,12 @@ struct thread_struct { u64 svcr; u64 tpidr2_el0; u64 por_el0; +#ifdef CONFIG_ARM64_GCS + unsigned int gcs_el0_mode; + u64 gcspr_el0; + u64 gcs_base; + u64 gcs_size; +#endif }; static inline unsigned int thread_get_vl(struct thread_struct *thread, diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 0540653fbf38..aedcf332f422 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -49,6 +49,7 @@ #include #include #include +#include #include #include #include @@ -280,6 +281,25 @@ static void flush_poe(void) write_sysreg_s(POR_EL0_INIT, SYS_POR_EL0); } +#ifdef CONFIG_ARM64_GCS + +static void flush_gcs(void) +{ + if (!system_supports_gcs()) + return; + + gcs_free(current); + current->thread.gcs_el0_mode = 0; + write_sysreg_s(GCSCRE0_EL1_nTR, SYS_GCSCRE0_EL1); + write_sysreg_s(0, SYS_GCSPR_EL0); +} + +#else + +static void flush_gcs(void) { } + +#endif + void flush_thread(void) { fpsimd_flush_thread(); @@ -287,6 +307,7 @@ void flush_thread(void) flush_ptrace_hw_breakpoint(current); flush_tagged_addr_state(); flush_poe(); + flush_gcs(); } void arch_release_task_struct(struct task_struct *tsk) @@ -484,6 +505,46 @@ static void entry_task_switch(struct task_struct *next) __this_cpu_write(__entry_task, next); } +#ifdef CONFIG_ARM64_GCS + +void gcs_preserve_current_state(void) +{ + current->thread.gcspr_el0 = read_sysreg_s(SYS_GCSPR_EL0); +} + +static void gcs_thread_switch(struct task_struct *next) +{ + if (!system_supports_gcs()) + return; + + /* GCSPR_EL0 is always readable */ + gcs_preserve_current_state(); + write_sysreg_s(next->thread.gcspr_el0, SYS_GCSPR_EL0); + + if (current->thread.gcs_el0_mode != next->thread.gcs_el0_mode) + gcs_set_el0_mode(next); + + /* + * Ensure that GCS memory effects of the 'prev' thread are + * ordered before other memory accesses with release semantics + * (or preceded by a DMB) on the current PE. In addition, any + * memory accesses with acquire semantics (or succeeded by a + * DMB) are ordered before GCS memory effects of the 'next' + * thread. This will ensure that the GCS memory effects are + * visible to other PEs in case of migration. + */ + if (task_gcs_el0_enabled(current) || task_gcs_el0_enabled(next)) + gcsb_dsync(); +} + +#else + +static void gcs_thread_switch(struct task_struct *next) +{ +} + +#endif + /* * Handle sysreg updates for ARM erratum 1418040 which affects the 32bit view of * CNTVCT, various other errata which require trapping all CNTVCT{,_EL0} @@ -580,6 +641,7 @@ struct task_struct *__switch_to(struct task_struct *prev, cntkctl_thread_switch(prev, next); ptrauth_thread_switch_user(next); permission_overlay_switch(next); + gcs_thread_switch(next); /* * Complete any pending TLB or cache maintenance on this CPU in case diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 2fc8c6dd0407..fc92170a8f37 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -11,6 +11,7 @@ obj-$(CONFIG_TRANS_TABLE) += trans_pgd.o obj-$(CONFIG_TRANS_TABLE) += trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o obj-$(CONFIG_ARM64_MTE) += mteswap.o +obj-$(CONFIG_ARM64_GCS) += gcs.o KASAN_SANITIZE_physaddr.o += n obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/arm64/mm/gcs.c b/arch/arm64/mm/gcs.c new file mode 100644 index 000000000000..f8f4f984a247 --- /dev/null +++ b/arch/arm64/mm/gcs.c @@ -0,0 +1,42 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include +#include +#include +#include + +#include +#include + +/* + * Apply the GCS mode configured for the specified task to the + * hardware. + */ +void gcs_set_el0_mode(struct task_struct *task) +{ + u64 gcscre0_el1 = GCSCRE0_EL1_nTR; + + if (task->thread.gcs_el0_mode & PR_SHADOW_STACK_ENABLE) + gcscre0_el1 |= GCSCRE0_EL1_RVCHKEN | GCSCRE0_EL1_PCRSEL; + + if (task->thread.gcs_el0_mode & PR_SHADOW_STACK_WRITE) + gcscre0_el1 |= GCSCRE0_EL1_STREn; + + if (task->thread.gcs_el0_mode & PR_SHADOW_STACK_PUSH) + gcscre0_el1 |= GCSCRE0_EL1_PUSHMEn; + + write_sysreg_s(gcscre0_el1, SYS_GCSCRE0_EL1); +} + +void gcs_free(struct task_struct *task) +{ + if (!system_supports_gcs()) + return; + + if (task->thread.gcs_base) + vm_munmap(task->thread.gcs_base, task->thread.gcs_size); + + task->thread.gcspr_el0 = 0; + task->thread.gcs_base = 0; + task->thread.gcs_size = 0; +}