From patchwork Mon Aug 7 22:00:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 13345140 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A57AFC04FE0 for ; Mon, 7 Aug 2023 22:03:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 302606B0088; Mon, 7 Aug 2023 18:03:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 28B2C900002; Mon, 7 Aug 2023 18:03:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 104EB8D0001; Mon, 7 Aug 2023 18:03:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id EFD956B0088 for ; Mon, 7 Aug 2023 18:03:28 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id B6BEA140B43 for ; Mon, 7 Aug 2023 22:03:28 +0000 (UTC) X-FDA: 81098685696.19.57092C4 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf01.hostedemail.com (Postfix) with ESMTP id DC2634000D for ; Mon, 7 Aug 2023 22:03:26 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=VuLILJWP; spf=pass (imf01.hostedemail.com: domain of broonie@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=broonie@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691445807; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sAhQux4ErUwiAv3XkaoNHj4MussaVgrlpmB3FlGsirg=; b=eAaw+bhypFx0J0HdfCMDfTh55KAS8TTcVK50zBZ6w3hrdi5xIIWvDsr1VYVbFEDtRohtIT M3P4IFTHTHVFNZUWqmLX7xp47CZgB56ioMofD9rg3u9bGDGC7hveAqKJQ4+NBqN2f2rQm9 f787kUh8+IHTjzBktkXgC6lx6IjYEDk= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=VuLILJWP; spf=pass (imf01.hostedemail.com: domain of broonie@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=broonie@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691445807; a=rsa-sha256; cv=none; b=gJ0yrxKRpNTLZrr3D/1KelGaoS2ES04KGyIUy0A2YoLMDt6NOEXgoAl1dYhDfgx3QYCkqV rEwJiWtV3z2FiCzdqCQkyqPN49w9Xzei7xBz8OnEbQkgALvldiGxMKFbq0XIBnAIS2xCAL sEUDgbSJvewEb1UKCEWAoJyIkaifrd4= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id DE3C36223A; Mon, 7 Aug 2023 22:03:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8567DC433CB; Mon, 7 Aug 2023 22:03:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1691445805; bh=QhfMLGzeDJZ/KjsMbq99IZf6UTeyPwLI/WOXtKz/xL4=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=VuLILJWPIJadXtEHv2oIy8BJNKgnIXVAS/dgClg0pz+Ewm8YFgPvsIJnNPATVpyqz eS5nfXd2vmWyaFX4k72Cg58qfEBF9RRkZwtohs9dUl3qclPPgdvPFtE20j5XPXhEve 0nlIsPHmgi7pCqqWMBTaAgG9BIn2c4ex3KpT8ZHpOXgQbDTBpF7UJFAlfcC+a05RRl JUXkKzKTF068aPIK7b7YwdzMcx2BZuTDGlP4xsMOuM7CQp3K+225Lscfy8k75GBC9u zPOY91pFev4+rPZaDOyXK2z9GqCZBUkx/iP06LkOGCadnqfYu1Ohf49jXFofmXy+U0 l/jKRuVEvJbcQ== From: Mark Brown Date: Mon, 07 Aug 2023 23:00:23 +0100 Subject: [PATCH v4 18/36] arm64/gcs: Context switch GCS state for EL0 MIME-Version: 1.0 Message-Id: <20230807-arm64-gcs-v4-18-68cfa37f9069@kernel.org> References: <20230807-arm64-gcs-v4-0-68cfa37f9069@kernel.org> In-Reply-To: <20230807-arm64-gcs-v4-0-68cfa37f9069@kernel.org> To: Catalin Marinas , Will Deacon , Jonathan Corbet , Andrew Morton , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Arnd Bergmann , Oleg Nesterov , Eric Biederman , Kees Cook , Shuah Khan , "Rick P. Edgecombe" , Deepak Gupta , Ard Biesheuvel , Szabolcs Nagy Cc: "H.J. Lu" , Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Mark Brown X-Mailer: b4 0.13-dev-034f2 X-Developer-Signature: v=1; a=openpgp-sha256; l=6697; i=broonie@kernel.org; h=from:subject:message-id; bh=QhfMLGzeDJZ/KjsMbq99IZf6UTeyPwLI/WOXtKz/xL4=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBk0WmmusCyKUu51cqJMqf5Bw/uaxABoBgQjuoGzyAe MmQYzfWJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZNFppgAKCRAk1otyXVSH0EXAB/ 4nnEGl8Q3w16Xpg88arEB4lmAcTGyxORLpfPHeac6iue8O/TLWKOCXdjayTpOii/GtRvS1jb/SyuCS sDNWfJYNfm4QggTAN7jRNXCyvXEepPkGv78svX5LulPJmL9W1P3Xe0VpUQIN5IWLYSvg7h9QW0mlFF Gdks1AfSyf7O9l8ENq+5B3j4/0g2H8KcbfuTRT6sUpapldGN14IuqdDTmvLkdmt1R7QIZY5S94CqWY SZ7XaiBujnlDo6pYpwUSbKPaVye026n7WEJbzUCBX3uL8KFUxptZlrgW2rqDd5WYALPk87/Ml6WtvX QOGj/3WXXTje6Z5Wo56bBFis1XkIzz X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-Rspamd-Queue-Id: DC2634000D X-Rspam-User: X-Stat-Signature: kd7phyj9jaqq5496za3judhgztgi317s X-Rspamd-Server: rspam01 X-HE-Tag: 1691445806-129785 X-HE-Meta: U2FsdGVkX1/5WC2YNzwv9MXNEKeqZhYhclmgCKDL58xG04Saj697YbSYH/N2flH7LSDVKJx4NdUB9T//Bq/DtDlxwDIyYhwI/EfO3dQCtCGrbor0TyjSF49LgmHinkNBcxMfhvDLvo0y2QpZbB8brOeDEF7VzIz59FQAeyVC93udEFvGyX53xWDTY9eUyzilyzx6xYHhpjfsUAOB6m+gHmSwArXLrSfZN1XKdwM6dbVyJ21AaXOCML6sEwcN1H7KlPExsRo8ffWwYdlu0BsxzLmB+nZ4es1JUIiu6C5hxandFam48z2N23TjtH5l2V1PV7z4CLv++GM3A+Nge+3ZGn0szCn+ab2n9ebCOCMy6hKdGH4IgzpeBd6vwEs5LKdXgguqnIJM7OFdRXHbivgrMHY6idS9donZkYogNtnV4q5F1cu2Zd6voqokeZLtxjpGJuB/0dV3VOUKN/Hw+dsEIU5/9CmTcfuCVQwc9RmYg5VKXkgiZuQb/J5irX1bMU+C1MI6vg9XTzm6qpFs2WDDHuzfBOHNj+yjImUQAw2nT/cE5TU5qWCbrnOY6bJnY4rx3dcHFXb/FNCFMBdl4DHoQeMjjyXiAV3c8iGDlYFFMdSyD1vsOcz4timlu4T5jVFRWpBvnJwROYJuZx/pUO3d+RiXV6Vd/c8VXFLD/J1BF0BgDjrNgls6R01e1f+DgejY0U6jMHlIRmEgNMES28410w3ZnweFgFjQSTITJxDUhbPEa8znkldeWFet5mjoOYZzID5aJDX0Rrp7vzvGUMLzwXQ40bwo2wo2l37J9bIl1bH/gzQz/y8u7z61H5tkTTyMNbruoNUaRiYTVKMc9dnjQs36nRr9mgIdylaYwaXvNPziH7qeEEN2+k0/NTW6Fxc6TsQPD6O/LyUL3mdoeuGrhL59El9Eku8VWGu8GuBRaRzbinWxT6jjVwj+8X597fk2HcQBMo1fYMW1TNugj3P djlxkBZ8 g8w9dCr+mCBuxGPCpYOnH30Xr7R05BtvDkSsJHhCfOgSGf3fhOi+rGf7vhv2rSpdGTUsel0tvMclUa46RBZ2GQqWQNdV1bFTYR1eHYtd5BYfL3edIWkXHrbzAvSv4LzKzJPrRma8gnwlqwvy0RiAnZfqmp3LYTRq5IoFJz9qkYZqUfJF049nb9Eh3+hPvgJWomd6rQgKeNu6cAS92lGyTMnuIg0xd4DdjFVURIlgAKfeGlxowAuS9mSLftE/AXuaP6QQ0f+YNYPuS5k1jBxSxoxj6rQFzJQCZFcZLtg9uNcRsJtTljfnkK19lx4/rwn1w2+/5w2W9R4VOFJVFbkVOHc9WUHK+kCFLJwpkdNp9GcUodtb9OdHPf4kCIjIxKYj2InR+qP5rkO145miEjZYf6nsWvUBDBrTcrO+/sU60yKNciX63MmSuiryBHrd9w/xai/RJPjDa5qCEOXjc8+IU9DXRQQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There are two registers controlling the GCS state of EL0, GCSPR_EL0 which is the current GCS pointer and GCSCRE0_EL1 which has enable bits for the specific GCS functionality enabled for EL0. Manage these on context switch and process lifetime events, GCS is reset on exec(). Also ensure that any changes to the GCS memory are visible to other PEs and that changes from other PEs are visible on this one by issuing a GCSB DSYNC when moving to or from a thread with GCS. Since the current GCS configuration of a thread will be visible to userspace we store the configuration in the format used with userspace and provide a helper which configures the system register as needed. On systems that support GCS we always allow access to GCSPR_EL0, this facilitates reporting of GCS faults if userspace implements disabling of GCS on error - the GCS can still be discovered and examined even if GCS has been disabled. Signed-off-by: Mark Brown --- arch/arm64/include/asm/gcs.h | 24 +++++++++++++++++ arch/arm64/include/asm/processor.h | 6 +++++ arch/arm64/kernel/process.c | 55 ++++++++++++++++++++++++++++++++++++++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/gcs.c | 39 +++++++++++++++++++++++++++ 5 files changed, 125 insertions(+) diff --git a/arch/arm64/include/asm/gcs.h b/arch/arm64/include/asm/gcs.h index 7c5e95218db6..04594ef59dad 100644 --- a/arch/arm64/include/asm/gcs.h +++ b/arch/arm64/include/asm/gcs.h @@ -48,4 +48,28 @@ static inline u64 gcsss2(void) return Xt; } +#ifdef CONFIG_ARM64_GCS + +static inline bool task_gcs_el0_enabled(struct task_struct *task) +{ + return current->thread.gcs_el0_mode & PR_SHADOW_STACK_ENABLE; +} + +void gcs_set_el0_mode(struct task_struct *task); +void gcs_free(struct task_struct *task); +void gcs_preserve_current_state(void); + +#else + +static inline bool task_gcs_el0_enabled(struct task_struct *task) +{ + return false; +} + +static inline void gcs_set_el0_mode(struct task_struct *task) { } +static inline void gcs_free(struct task_struct *task) { } +static inline void gcs_preserve_current_state(void) { } + +#endif + #endif diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 3918f2a67970..f1551228a143 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -179,6 +179,12 @@ struct thread_struct { u64 sctlr_user; u64 svcr; u64 tpidr2_el0; +#ifdef CONFIG_ARM64_GCS + unsigned int gcs_el0_mode; + u64 gcspr_el0; + u64 gcs_base; + u64 gcs_size; +#endif }; static inline unsigned int thread_get_vl(struct thread_struct *thread, diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 0fcc4eb1a7ab..b8a42471aea3 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -48,6 +48,7 @@ #include #include #include +#include #include #include #include @@ -271,12 +272,31 @@ static void flush_tagged_addr_state(void) clear_thread_flag(TIF_TAGGED_ADDR); } +#ifdef CONFIG_ARM64_GCS + +static void flush_gcs(void) +{ + if (system_supports_gcs()) { + gcs_free(current); + current->thread.gcs_el0_mode = 0; + write_sysreg_s(0, SYS_GCSCRE0_EL1); + write_sysreg_s(0, SYS_GCSPR_EL0); + } +} + +#else + +static void flush_gcs(void) { } + +#endif + void flush_thread(void) { fpsimd_flush_thread(); tls_thread_flush(); flush_ptrace_hw_breakpoint(current); flush_tagged_addr_state(); + flush_gcs(); } void arch_release_task_struct(struct task_struct *tsk) @@ -474,6 +494,40 @@ static void entry_task_switch(struct task_struct *next) __this_cpu_write(__entry_task, next); } +#ifdef CONFIG_ARM64_GCS + +void gcs_preserve_current_state(void) +{ + if (task_gcs_el0_enabled(current)) + current->thread.gcspr_el0 = read_sysreg_s(SYS_GCSPR_EL0); +} + +static void gcs_thread_switch(struct task_struct *next) +{ + if (!system_supports_gcs()) + return; + + gcs_preserve_current_state(); + + /* + * Ensure that GCS changes are observable by/from other PEs in + * case of migration. + */ + if (task_gcs_el0_enabled(current) || task_gcs_el0_enabled(next)) + gcsb_dsync(); + + gcs_set_el0_mode(next); + write_sysreg_s(next->thread.gcspr_el0, SYS_GCSPR_EL0); +} + +#else + +static void gcs_thread_switch(struct task_struct *next) +{ +} + +#endif + /* * ARM erratum 1418040 handling, affecting the 32bit view of CNTVCT. * Ensure access is disabled when switching to a 32bit task, ensure @@ -533,6 +587,7 @@ struct task_struct *__switch_to(struct task_struct *prev, ssbs_thread_switch(next); erratum_1418040_thread_switch(next); ptrauth_thread_switch_user(next); + gcs_thread_switch(next); /* * Complete any pending TLB or cache maintenance on this CPU in case diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index dbd1bc95967d..4e7cb2f02999 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -10,6 +10,7 @@ obj-$(CONFIG_TRANS_TABLE) += trans_pgd.o obj-$(CONFIG_TRANS_TABLE) += trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o obj-$(CONFIG_ARM64_MTE) += mteswap.o +obj-$(CONFIG_ARM64_GCS) += gcs.o KASAN_SANITIZE_physaddr.o += n obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/arm64/mm/gcs.c b/arch/arm64/mm/gcs.c new file mode 100644 index 000000000000..b0a67efc522b --- /dev/null +++ b/arch/arm64/mm/gcs.c @@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include +#include +#include +#include + +#include +#include + +/* + * Apply the GCS mode configured for the specified task to the + * hardware. + */ +void gcs_set_el0_mode(struct task_struct *task) +{ + u64 gcscre0_el1 = GCSCRE0_EL1_nTR; + + if (task->thread.gcs_el0_mode & PR_SHADOW_STACK_ENABLE) + gcscre0_el1 |= GCSCRE0_EL1_RVCHKEN | GCSCRE0_EL1_PCRSEL; + + if (task->thread.gcs_el0_mode & PR_SHADOW_STACK_WRITE) + gcscre0_el1 |= GCSCRE0_EL1_STREn; + + if (task->thread.gcs_el0_mode & PR_SHADOW_STACK_PUSH) + gcscre0_el1 |= GCSCRE0_EL1_PUSHMEn; + + write_sysreg_s(gcscre0_el1, SYS_GCSCRE0_EL1); +} + +void gcs_free(struct task_struct *task) +{ + if (task->thread.gcs_base) + vm_munmap(task->thread.gcs_base, task->thread.gcs_size); + + task->thread.gcspr_el0 = 0; + task->thread.gcs_base = 0; + task->thread.gcs_size = 0; +}