From patchwork Thu Jan 19 21:23:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13108805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D551C004D4 for ; Thu, 19 Jan 2023 21:24:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6617B280004; Thu, 19 Jan 2023 16:24:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C326280001; Thu, 19 Jan 2023 16:24:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3CC35280004; Thu, 19 Jan 2023 16:24:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 24705280001 for ; Thu, 19 Jan 2023 16:24:11 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E9C90A0275 for ; Thu, 19 Jan 2023 21:24:10 +0000 (UTC) X-FDA: 80372826660.17.7C11D82 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by imf27.hostedemail.com (Postfix) with ESMTP id DB19140014 for ; Thu, 19 Jan 2023 21:24:08 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=TZuy+3QB; spf=pass (imf27.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674163449; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=bs+9z9aIIF14yXerZa6rGEtRQZc7+0mOjtqR3MupuLg=; b=RDmeCse5P5BxAPu08PpDb3c2pdnOkoG8CMnPRGfhLTZVMNgjqlF3vu3VhlozGtC9rUXmtL 5b5MvXSz9MOlUw/vdxuOi7NeSeQGBYrpj73qQGvigSwQcVKXm++lsSRMi4R548Hq3bj2KC Ejpd9FNqpLTR4cHHiMDhj30h1WDEp6k= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=TZuy+3QB; spf=pass (imf27.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674163449; a=rsa-sha256; cv=none; b=hNpzFfAoU2CNUt0qCtjGLzGKc36qXgZ+mFA3tismSTcsEYYS+yQwvHcxKjxO3fQ2fVY8iW 4QROMrliWKVNQX68NqrI4D6jAIngfVo3nTGtNEIziESUAwwE3QxL4jPCaQPqBVJSv731u9 V7VkW62C+xwW7JnpmgWrwyayzTUkVuw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163449; x=1705699449; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Ifxi0e32c84RQATdXOs3rUNHxe57SwrjYewWnLIO6BE=; b=TZuy+3QBd59FA8ZPS8UpStBkuM46jx68KuWSDcL+4PDkWKCg4Vumo7aB 3t1JxBu9rQm+xOauKJVRNW3cEs220/LA647pcvFAmKJCZMKk3zoetWLqM OiXe/Q0J6taMc+ZlOQUdFCibWqrPrIm3bSrZ9Q4JPoiTU5lDjA5/05XYQ 0pgx+5Cs6RNQ3QHnwBbxzOcZvfXZd2n2B/ZKOoEyCahs+xExGCUghfMcG UHLcHNLfi+K1ll+OmxG4ndHZsht9oF8FuzjjgKcoBgBdKOKzDpSxNeo3K 0Bv980bgjHEWr27EgsPQOo2KAO+KZRiLF7OOSXiRC05eXThBmDhmLSYOX g==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119839" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119839" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:08 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139135" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139135" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:06 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 27/39] x86/shstk: Add user-mode shadow stack support Date: Thu, 19 Jan 2023 13:23:05 -0800 Message-Id: <20230119212317.8324-28-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: DB19140014 X-Stat-Signature: h1ytforowmbf7mfs37j53rb1c68jusz7 X-Rspam-User: X-HE-Tag: 1674163448-461478 X-HE-Meta: U2FsdGVkX189KgycEYk3sZ0KomY3PUSvAibsgykC5n/geZr/inQYwy6jxIL8aXDsUDvjRcx6FAQ61R+A/aTPYbCFlB+66Gn6L9Gha/9P57X3B254DHwFZYdGwLgWioz0WrkUdSVxoFPxfNBCxnbubAyT+gLF9lJiyqNrSgk+MxNnioKF9E91CvfuOLLRkr1YSn4QPsX6D08N8XcCZ0PBJVhov4T7tHZJPC+VvLvIyMvYvqJJFzYeoYOKj/4VouJHOqX1E9wvnoD9yy7xk1ttypBjy+/GVjRWUw3AISorYlf4lAgXUaGO2HMbed+GLNXttWWAFJZSVajLIAEsVfbu+fzTA78Cqudlzk+mWRABgTqSWbpXb2N1gjMlqxo0FvPJcK58BiqteTMnaC+IzjrnMr87kqi1FGSC7zIs+2H+LYZlN6cvijX4iUfC0/IhA0NbvEOJmVbEWC04Hr5aWOTQDdbdAWxLzlM2AgKFpae10tZKoGzGZK+2ap+7GiebmpEj5jHWFWhHZEdjWXAPn931T0mnrt5d8Qng9ZxuBzQhxhuN7hEGzuQa6vyaGFje2C6gS/hGtja9ZqAPs5FqCOblTcED5HLiHxdjA7aB+a6PnUKSgFFTGY14i1njc2S44PBHZ1aYLGPB8Cp0uLVr/wtN1bHd5dvo3xNTV2dhIjkD37FMy5pfM7Wmcf4cqFX0aG42RFl43qefNdHTSO1jg6G8/QU4cV85pEaeQ1w+A2FRzB/EwVtnwLRrhS96QJehwxeUeL1ZbghOSQjI1oH3i549c59Vv2nwGJ4e71SNvEH+q64Ehx8JN1fObUyNOqm9CTV8pizhrZ7aW6im0vln3Uoqvu0ONqsHEo443TgtOQh6M1yYMiHz5zz0xj3VfT2KZ1JFtYnUvNok65FgDZXiDioeoj9dP+fEjs2A3zdZ/9FTDT2ZJyYOqIcjb0T0HE1rNAijMlFBqp+p45nyf0deQ3q xF7TBhXN 0fFr1tMNUx8M31JntP1wD0UAg8sdIEQ9tIeW0T1NZpf+KBI9FC7vcavsyu4ZEBP+90N1YBDCX4tZ2BYst0uKNkhx5a5/DaoOSw1vThn60hTp/lOlJYWPslTH3SAkFTHKI5HUq8xG2Dka5QCo+uIWXoiT1w4cX9SV6RGJvghZ2Zavs52/pfi4IdDHzzjG/90b0JzgsqJ0fS8CVZgnnDP1aMK8BcR6LZOmJ2/2CUpXiA054Lup2a5BYWrVtUJO3QmyQEkvUxALR4TjwTGTxNUXuAe9/syAinVVIbxIR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu Introduce basic shadow stack enabling/disabling/allocation routines. A task's shadow stack is allocated from memory with VM_SHADOW_STACK flag and has a fixed size of min(RLIMIT_STACK, 4GB). Keep the task's shadow stack address and size in thread_struct. This will be copied when cloning new threads, but needs to be cleared during exec, so add a function to do this. Do not support IA32 emulation or x32. Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook Reviewed-by: Kees Cook --- v5: - Switch to EOPNOTSUPP - Use MAP_ABOVE4G - Move set_clr_bits_msrl() to patch where it is first used v4: - Just set MSR_IA32_U_CET when disabling shadow stack, since we don't have IBT yet. (Peterz) v3: - Use define for set_clr_bits_msrl() (Kees) - Make some functions static (Kees) - Change feature_foo() to features_foo() (Kees) - Centralize shadow stack size rlimit checks (Kees) - Disable x32 support v2: - Get rid of unnessary shstk->base checks - Don't support IA32 emulation arch/x86/include/asm/processor.h | 2 + arch/x86/include/asm/shstk.h | 7 ++ arch/x86/include/uapi/asm/prctl.h | 3 + arch/x86/kernel/shstk.c | 146 ++++++++++++++++++++++++++++++ 4 files changed, 158 insertions(+) diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index e0734f417273..3c257a1a0757 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -479,6 +479,8 @@ struct thread_struct { #ifdef CONFIG_X86_USER_SHADOW_STACK unsigned long features; unsigned long features_locked; + + struct thread_shstk shstk; #endif /* Floating point and extended processor state */ diff --git a/arch/x86/include/asm/shstk.h b/arch/x86/include/asm/shstk.h index 58f9ee675be0..f40414a982e8 100644 --- a/arch/x86/include/asm/shstk.h +++ b/arch/x86/include/asm/shstk.h @@ -8,12 +8,19 @@ struct task_struct; #ifdef CONFIG_X86_USER_SHADOW_STACK +struct thread_shstk { + u64 base; + u64 size; +}; + long shstk_prctl(struct task_struct *task, int option, unsigned long features); void reset_thread_features(void); +void shstk_free(struct task_struct *p); #else static inline long shstk_prctl(struct task_struct *task, int option, unsigned long features) { return -EINVAL; } static inline void reset_thread_features(void) {} +static inline void shstk_free(struct task_struct *p) {} #endif /* CONFIG_X86_USER_SHADOW_STACK */ #endif /* __ASSEMBLY__ */ diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index b2b3b7200b2d..7dfd9dc00509 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -26,4 +26,7 @@ #define ARCH_SHSTK_DISABLE 0x5002 #define ARCH_SHSTK_LOCK 0x5003 +/* ARCH_SHSTK_ features bits */ +#define ARCH_SHSTK_SHSTK (1ULL << 0) + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 41ed6552e0a5..f39e5d3b9303 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -8,14 +8,160 @@ #include #include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include #include +static bool features_enabled(unsigned long features) +{ + return current->thread.features & features; +} + +static void features_set(unsigned long features) +{ + current->thread.features |= features; +} + +static void features_clr(unsigned long features) +{ + current->thread.features &= ~features; +} + +static unsigned long alloc_shstk(unsigned long size) +{ + int flags = MAP_ANONYMOUS | MAP_PRIVATE | MAP_ABOVE4G; + struct mm_struct *mm = current->mm; + unsigned long addr, unused; + + mmap_write_lock(mm); + addr = do_mmap(NULL, addr, size, PROT_READ, flags, + VM_SHADOW_STACK | VM_WRITE, 0, &unused, NULL); + + mmap_write_unlock(mm); + + return addr; +} + +static unsigned long adjust_shstk_size(unsigned long size) +{ + if (size) + return PAGE_ALIGN(size); + + return PAGE_ALIGN(min_t(unsigned long long, rlimit(RLIMIT_STACK), SZ_4G)); +} + +static void unmap_shadow_stack(u64 base, u64 size) +{ + while (1) { + int r; + + r = vm_munmap(base, size); + + /* + * vm_munmap() returns -EINTR when mmap_lock is held by + * something else, and that lock should not be held for a + * long time. Retry it for the case. + */ + if (r == -EINTR) { + cond_resched(); + continue; + } + + /* + * For all other types of vm_munmap() failure, either the + * system is out of memory or there is bug. + */ + WARN_ON_ONCE(r); + break; + } +} + +static int shstk_setup(void) +{ + struct thread_shstk *shstk = ¤t->thread.shstk; + unsigned long addr, size; + + /* Already enabled */ + if (features_enabled(ARCH_SHSTK_SHSTK)) + return 0; + + /* Also not supported for 32 bit and x32 */ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) || in_32bit_syscall()) + return -EOPNOTSUPP; + + size = adjust_shstk_size(0); + addr = alloc_shstk(size); + if (IS_ERR_VALUE(addr)) + return PTR_ERR((void *)addr); + + fpregs_lock_and_load(); + wrmsrl(MSR_IA32_PL3_SSP, addr + size); + wrmsrl(MSR_IA32_U_CET, CET_SHSTK_EN); + fpregs_unlock(); + + shstk->base = addr; + shstk->size = size; + features_set(ARCH_SHSTK_SHSTK); + + return 0; +} + void reset_thread_features(void) { + memset(¤t->thread.shstk, 0, sizeof(struct thread_shstk)); current->thread.features = 0; current->thread.features_locked = 0; } +void shstk_free(struct task_struct *tsk) +{ + struct thread_shstk *shstk = &tsk->thread.shstk; + + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) || + !features_enabled(ARCH_SHSTK_SHSTK)) + return; + + if (!tsk->mm) + return; + + unmap_shadow_stack(shstk->base, shstk->size); +} + + +static int shstk_disable(void) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return -EOPNOTSUPP; + + /* Already disabled? */ + if (!features_enabled(ARCH_SHSTK_SHSTK)) + return 0; + + fpregs_lock_and_load(); + /* Disable WRSS too when disabling shadow stack */ + wrmsrl(MSR_IA32_U_CET, 0); + wrmsrl(MSR_IA32_PL3_SSP, 0); + fpregs_unlock(); + + shstk_free(current); + features_clr(ARCH_SHSTK_SHSTK); + + return 0; +} + long shstk_prctl(struct task_struct *task, int option, unsigned long features) { if (option == ARCH_SHSTK_LOCK) {