From patchwork Sun Mar 19 00:15:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13180159 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68F66C7618A for ; Sun, 19 Mar 2023 00:16:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C0F6728001D; Sat, 18 Mar 2023 20:16:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B20CC280001; Sat, 18 Mar 2023 20:16:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8FC2728001D; Sat, 18 Mar 2023 20:16:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7381C280001 for ; Sat, 18 Mar 2023 20:16:49 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4CE871A0C22 for ; Sun, 19 Mar 2023 00:16:49 +0000 (UTC) X-FDA: 80583732138.06.C3085DB Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf21.hostedemail.com (Postfix) with ESMTP id 49F0E1C0012 for ; Sun, 19 Mar 2023 00:16:47 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=QagjewQz; spf=pass (imf21.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679185007; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=UTD47oSwK8HDyM7Tg5IVAtcUOtNetxtvbzv1n8npU2M=; b=avbWd/m19VZ5Ub3WswNkTy5C3fL+YKZhcNH/Tn7KX/dYAJU9W34TRlAcGuFf66yRbiKqKx wNRXMfLsqE5698SDe5Sv6inLY3iQ1blTBVF94Cs06B1dr9tf/WHybvQG3Na+4fz1oFfPkL V87YERjB5l0JIAWosX9r+wPKalDoFgw= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=QagjewQz; spf=pass (imf21.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679185007; a=rsa-sha256; cv=none; b=SqMGgB/IIkHrK8345qzr9nTuAFvX7m+TFfzOk9XPdjcssm0QVW+CeWoCTf2f2pKJ8Wyj5y YbWByO/xiyI5rK/FEvv03U7iF9dLVfoi66HVVo0stIyEJueEXiODCQq6TKglxh3Cucj7iC eznDEnTLfrsbLtcl94dcRMTfpeQyNcg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679185007; x=1710721007; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=dBIkMgtPKqQv5kVHDuakiGU5eUx6LQnshjLSaxY9/+4=; b=QagjewQzoIHjvWKRGwgunWgwL48K5j1MHTVlpXTpxInLK8UkefvrE8tw cTfqTMG3LuEUzsiXcp83FfokKDORF1Qm1HiEQtnIM4XmCae5OB+rOj6hj hl7If89wc2FyiVulViNhqM0J2BRAerm61dFSO56Q1+8hmlH3A2uFPZVHj gwRvbIwmrN1Gm0eeWDmM2te0xE6GuM6otq+hQ2zptXF9zy5vdp/pko4dW QQ527XkKr1JpnNweaMindW+hlCvm+wkgUpYyZtqSJhtU8bIgxSzVbvL5u PCZ1Jb/F++8eqBe5YbWPr7VhmS2eOj6UjuEWwOqErNCm9uSCGZJj5WBtn Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10653"; a="338491432" X-IronPort-AV: E=Sophos;i="5.98,272,1673942400"; d="scan'208";a="338491432" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2023 17:16:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10653"; a="749672945" X-IronPort-AV: E=Sophos;i="5.98,272,1673942400"; d="scan'208";a="749672945" Received: from bmahatwo-mobl1.gar.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.135.34.5]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2023 17:16:44 -0700 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com, szabolcs.nagy@arm.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v8 31/40] x86/shstk: Introduce routines modifying shstk Date: Sat, 18 Mar 2023 17:15:26 -0700 Message-Id: <20230319001535.23210-32-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230319001535.23210-1-rick.p.edgecombe@intel.com> References: <20230319001535.23210-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 49F0E1C0012 X-Rspam-User: X-Stat-Signature: rry1347rte3t39m7ujcjh796q5dptsyn X-HE-Tag: 1679185007-210644 X-HE-Meta: U2FsdGVkX18Yu2HL2JzVkYIxoU6TOdxcnJzVp6HgvGqVLUZrhyBIXb/stOJb582Ka5GDkyoc2JfTL0iIGLBTp3lZY3nsQwDcfRfjl4dIlBDppmKiwJtfuXGaGEl8kI+MEI4VmIWMR83ol76qu4wpm0Q3aoRaF56qvB8JX8AUFUrtbXHNfcIE1gvQXfcQ/4wiqNZlhXpbNu+YaOy0UFJ9A52Xs5Aq5tGMwFt3afpZKrIvhbmj3fndJp1C3JF43srqoMFdkxNE5c3T8n9S70PnZDWNhBtV5jYDqKVBZ/YsnBubgR00MgihHINO4dVYXCEw7ctt7WkINdN/jD9w0Um9zHlDUYfAeBFdzftWzxxrE7A5I58rZxXxjxoUU16w5imIRKDzSXpU21sQfLD5egiopI82Y8ilKo26OAFOVBXqjjEwst7yzz6yJLbBN9vYqrhcLPeCKyuDBBj6qN2TO7PG2qtRhVQoANQHr7ZRAndHAjLKLL4MVd/omLOrtv1uawdFwwx8ynHvetrNgCyqFC66g7lGeh2xUr9HI142BBxjrckuHVYHj0rjXTHKikF6fhx0EEzY9zORH8lOAZ5r9G0IAZ/0qmtlkF8oyJ5xFa1l2d8TXkIvaZzSHW1pp7Gs8pgjnZP2L5qeHcBv6kTo91/Ohl1OGymBpx1F6zC9zs8Q/ksB6RKjIkUPmx3MN+NnKxx3D/KDHYezlaKDCpCs0ItQc6slSdpIDLSNePYGg6rKjhadZT4Q5zDbA24hEVbVSV/GF8HwOGvpUATY864/uH7t59jP2nX2NfKj3CVPt4PRoBqXmmOHHPTl8HKOE14RMfq3WKUa+PfisFJBlOffx0hPxU0blo97b3lOx7qyiNWmfC2D8D89E9ZrhYI4WgsE3CK8a6gdVq9XKgQmUa9oPRznS4NEnrYJRtGi645C4fKFmmrl0hGOZY2nyIslCavCuXw9pAqLzHHLEBRfOK1w6h/ uaQfLzrX 6pk/qxE5vNCTS34aJy3ECazpzEF1xhAD9gdEkj2NuKPQmLPHJPwYzbnIhbh9oZNKr6kyVnsxrYK+kT3rqqTM8nSXwulJdZ2mDmiOsHeXe/zvsq77JNVef4KICzmVmfCxfYG85NmfnwTNmePxGuZrEo+J3JEbNoTNZmY7Cnpv+eZRDWoC64KVgqNzm8cYln/jA29IO5mkufI57ZpcEy4BgPlVpX8jZcTq8GSELmM64sBzDWmYULrrddM3ImrVMRNJ28UnMj6SieYDyKB8vcjTQuGfuy5IAZP7XD39fe++q3mZtw1Z92kyXbhGoLoh+NNnXOH6rV3+Mn0o8Xs5a8pfjdoDdV95m4DUU/Ejh X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Shadow stacks are normally written to via CALL/RET or specific CET instructions like RSTORSSP/SAVEPREVSSP. However, sometimes the kernel will need to write to the shadow stack directly using the ring-0 only WRUSS instruction. A shadow stack restore token marks a restore point of the shadow stack, and the address in a token must point directly above the token, which is within the same shadow stack. This is distinctively different from other pointers on the shadow stack, since those pointers point to executable code area. Introduce token setup and verify routines. Also introduce WRUSS, which is a kernel-mode instruction but writes directly to user shadow stack. In future patches that enable shadow stack to work with signals, the kernel will need something to denote the point in the stack where sigreturn may be called. This will prevent attackers calling sigreturn at arbitrary places in the stack, in order to help prevent SROP attacks. To do this, something that can only be written by the kernel needs to be placed on the shadow stack. This can be accomplished by setting bit 63 in the frame written to the shadow stack. Userspace return addresses can't have this bit set as it is in the kernel range. It also can't be a valid restore token. Co-developed-by: Yu-cheng Yu Signed-off-by: Yu-cheng Yu Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook Acked-by: Mike Rapoport (IBM) Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook --- v8: - Update commit log verbiage (Boris) - Use define instead of magic BIT(63) (Boris) v5: - Fix typo in commit log v3: - Drop shstk_check_rstor_token() - Fail put_shstk_data() if bit 63 is set in the data (Kees) - Add comment in create_rstor_token() (Kees) - Pull in create_rstor_token() changes from future patch (Kees) v2: - Add data helpers for writing to shadow stack. --- arch/x86/include/asm/special_insns.h | 13 +++++ arch/x86/kernel/shstk.c | 75 ++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+) diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h index de48d1389936..d6cd9344f6c7 100644 --- a/arch/x86/include/asm/special_insns.h +++ b/arch/x86/include/asm/special_insns.h @@ -202,6 +202,19 @@ static inline void clwb(volatile void *__p) : [pax] "a" (p)); } +#ifdef CONFIG_X86_USER_SHADOW_STACK +static inline int write_user_shstk_64(u64 __user *addr, u64 val) +{ + asm_volatile_goto("1: wrussq %[val], (%[addr])\n" + _ASM_EXTABLE(1b, %l[fail]) + :: [addr] "r" (addr), [val] "r" (val) + :: fail); + return 0; +fail: + return -EFAULT; +} +#endif /* CONFIG_X86_USER_SHADOW_STACK */ + #define nop() asm volatile ("nop") static inline void serialize(void) diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index bd9cdc3a7338..e22928c63ffc 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -25,6 +25,8 @@ #include #include +#define SS_FRAME_SIZE 8 + static bool features_enabled(unsigned long features) { return current->thread.features & features; @@ -40,6 +42,35 @@ static void features_clr(unsigned long features) current->thread.features &= ~features; } +/* + * Create a restore token on the shadow stack. A token is always 8-byte + * and aligned to 8. + */ +static int create_rstor_token(unsigned long ssp, unsigned long *token_addr) +{ + unsigned long addr; + + /* Token must be aligned */ + if (!IS_ALIGNED(ssp, 8)) + return -EINVAL; + + addr = ssp - SS_FRAME_SIZE; + + /* + * SSP is aligned, so reserved bits and mode bit are a zero, just mark + * the token 64-bit. + */ + ssp |= BIT(0); + + if (write_user_shstk_64((u64 __user *)addr, (u64)ssp)) + return -EFAULT; + + if (token_addr) + *token_addr = addr; + + return 0; +} + static unsigned long alloc_shstk(unsigned long size) { int flags = MAP_ANONYMOUS | MAP_PRIVATE | MAP_ABOVE4G; @@ -157,6 +188,50 @@ unsigned long shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long cl return addr + size; } +static unsigned long get_user_shstk_addr(void) +{ + unsigned long long ssp; + + fpregs_lock_and_load(); + + rdmsrl(MSR_IA32_PL3_SSP, ssp); + + fpregs_unlock(); + + return ssp; +} + +#define SHSTK_DATA_BIT BIT(63) + +static int put_shstk_data(u64 __user *addr, u64 data) +{ + if (WARN_ON_ONCE(data & SHSTK_DATA_BIT)) + return -EINVAL; + + /* + * Mark the high bit so that the sigframe can't be processed as a + * return address. + */ + if (write_user_shstk_64(addr, data | SHSTK_DATA_BIT)) + return -EFAULT; + return 0; +} + +static int get_shstk_data(unsigned long *data, unsigned long __user *addr) +{ + unsigned long ldata; + + if (unlikely(get_user(ldata, addr))) + return -EFAULT; + + if (!(ldata & SHSTK_DATA_BIT)) + return -EINVAL; + + *data = ldata & ~SHSTK_DATA_BIT; + + return 0; +} + void shstk_free(struct task_struct *tsk) { struct thread_shstk *shstk = &tsk->thread.shstk;