From patchwork Mon Feb 27 22:29:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154232 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C62E9C7EE31 for ; Mon, 27 Feb 2023 22:31:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 539276B0072; Mon, 27 Feb 2023 17:31:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E7516B0074; Mon, 27 Feb 2023 17:31:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3887D6B0075; Mon, 27 Feb 2023 17:31:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 28E136B0072 for ; Mon, 27 Feb 2023 17:31:32 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id F39AC1204AA for ; Mon, 27 Feb 2023 22:31:31 +0000 (UTC) X-FDA: 80514519582.15.3AD3B2A Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf13.hostedemail.com (Postfix) with ESMTP id C823220009 for ; Mon, 27 Feb 2023 22:31:29 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=GnUGuh71; spf=pass (imf13.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537090; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=jlTKmZYifm3wG4j38mda0fuVjIqxC/45mxPbP7FCqjE=; b=wR3BEQFNSLcuiOFhMSEh0C8zJJSTSq7+oPb6vRAe+9Z/TbDjGPi7dTMI4ddA7uehLN0z9b Wuw3H9vaBdUWHrpfpFa5nbObYWCLNeblSkV7pmQq5UbYh88zs+R3lLUX6DP6U/blZBrqp4 fkJr9YrYJATuFdeLM+ERMkMgIoaCVrY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=GnUGuh71; spf=pass (imf13.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537090; a=rsa-sha256; cv=none; b=Rq+xiKO5bXnbIVOd/xN23yyF2onGeutGCkTC52BG2QItN/DaP/gJYcVr+m2Y87lW6wSI4J gAb8Em1e/cGase+eLAZmdqxjp9Iwg4lmSDX/E+aNPzl5LlBSUQjffUDiygk075KtIP6PMg 1qnK7RMCQpkL5mko9eJJZaTWcCE+9KU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537089; x=1709073089; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=J0932fBh2XRteOmToVVnjH63Fz97/Mj9Z7Xua8i0NMA=; b=GnUGuh71wruHatVKB2sEk4RS2aJsMMzQ/N7+ICqZFVfIUeK4TIhYYGYu mQZ/KLpfNq63ly+WRLVrmnbHSOWqUzAcEgdvS6SjuU9wuI2ngFonGk5E6 LgWdefboLesB0UCe6C3b3/EAuZ65nobiDWfbFpvKQT9TdToM0KEF/BNsB qP4Gy88EVbe1n5r+Eoya/Z4wQ1yL3CG3vZkKXMaN20qSw949xkObehicT 9EdUt5gq/8b06Mu92hGCMXLg0Wv91CWhpVybhndbVIyepnIHHj2PXXxCS fiNvCDP+mgh2Zb1XN4v1WpwAcWrOBReI98ps+kGV+hchhVjwjxu/jBP2a A==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657010" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657010" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:08 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024335" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024335" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:07 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description Date: Mon, 27 Feb 2023 14:29:17 -0800 Message-Id: <20230227222957.24501-2-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: C823220009 X-Stat-Signature: o4agoqu8xos3fznsujddt4kay9d6dabk X-Rspam-User: X-HE-Tag: 1677537089-49088 X-HE-Meta: U2FsdGVkX180a3/gfYnhvkGm5X35EFJSvEWd38NP4bBhLwFJ/DBSyqGolkQ8bHUvgIpzdnjRp48rxdY36I29fY1WBocXq1FGrUFQ1jw5Qakh92brfGuSZOBFUpYq9NMShvw3fBUUlvs1xJr72qkFgDf4n+61ajmABXqdubgbEMkz1sy+3J3H0cWkd+Snp89ItP02atwiBUDyvxWyviJeJ9G9HJjnh3yB3yhublU5G+soI65Pmrirb6xN3I+YLms/8SKVpg9fNgvz2SyKsfa1M+eM9tmXuVU6cLBe1aE8escjB+HgFmVqFG01+8j11v5QVXh8ARFjMUyalVtTfk3UZMBsozbSt/pz8ZvJNyiDj3i/In12Eyo0Ef9zQ2fRtsjzrZFk2FaIaswhPzGqC6xgBgsF8sPNkRE0C+0a+LpR5MU6QC29o7a6E8vuu7FOD/r2mwU3qib/9b8kOkX4vXtE3zNsroXf+acYXx3BQA9KvlzLkFsxsbSGWtWVeTi7RIQHfGwv3eS1mbvvZG/EH/V6ZvfkDJqcAu6pLHllPl5WNWzwa4eSdaI5/A9LK0J8FClmW8QBis+VhNGflDhX+pyCPG87XenY7hu0ICzFl437EGKejm/nB5+evDJeddcWkNJE12hOtQbsqR7DF/LqxHQuRjHoayvb5lxdvZ3W2ajKYCNWu8CSxmLMLwSOB1tmoiQhOU2JzM7+7QxGs6I21XGOODtahUDKH39H0EomdjWZAo3+gZ58oknQqQNaRtXXJeBStyvbOjQT0YP1DH1boDNoopwiEwsLp7NkOYv5G+cKYQ0bjd/kjv/z/zdG5ylsbHxWdJDYfpw/7vuTiRWXvBwG4fxX93yPBarcdfzwakPYSMnmPXsqsUKzapq+AWdHD/uSJZgS3hwFuzN/eM3X77ZMrDnUzFoM9e+6TG2z2BdiAbXUr34z7wzjGPZ3CCNl8DFN+CrlO4jAiwPDl6/DInE rJv1gT+P H5FmAegyeAgkbyMiOkSAn2mUzxnimKA2WNm0fDcNE6fhkAN2unSlrVqUCCuMfow6i0m+JD+xcZ+tneH3u7LnkxjJpZA8G5HLlih2KYZEWh0KquQqRRfv8FlBsxG4CAqjg2GaXmfM/um0qkrpXNhCnUREll1aEu1QP5q4pP7+ELjjSxtdi9joqoSt1bV8lpjUc6IiVtqDs2XO+X4huB+BiAqyq5dMMYzdboU8lO7GCs8GcaBh73otGlox1L/2RuwklHpQphdVtdbeBFsfQSruKKeP567TA8Wng/V0qkVOsomCsL5mJfCXyqpouGG3Lbt6Dop2eZzEOb0UQ74VmL4kGhqSPjOPcU9FINRH7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu Introduce a new document on Control-flow Enforcement Technology (CET). Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook --- v5: - Literal format tweaks (Bagas Sanjaya) - Update EOPNOTSUPP text due to unification after comment from (Kees) - Update 32 bit signal support with new behavior - Remove capitalization on shadow stack (Boris) - Fix typo v4: - Drop clearcpuid piece (Boris) - Add some info about 32 bit v3: - Clarify kernel IBT is supported by the kernel. (Kees, Andrew Cooper) - Clarify which arch_prctl's can take multiple bits. (Kees) - Describe ASLR characteristics of thread shadow stacks. (Kees) - Add exec section. (Andrew Cooper) - Fix some capitalization (Bagas Sanjaya) - Update new location of enablement status proc. - Add info about new user_shstk software capability. - Add more info about what the kernel pushes to the shadow stack on signal. v2: - Updated to new arch_prctl() API - Add bit about new proc status --- Documentation/x86/index.rst | 1 + Documentation/x86/shstk.rst | 166 ++++++++++++++++++++++++++++++++++++ 2 files changed, 167 insertions(+) create mode 100644 Documentation/x86/shstk.rst diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst index c73d133fd37c..8ac64d7de4dc 100644 --- a/Documentation/x86/index.rst +++ b/Documentation/x86/index.rst @@ -22,6 +22,7 @@ x86-specific Documentation mtrr pat intel-hfi + shstk iommu intel_txt amd-memory-encryption diff --git a/Documentation/x86/shstk.rst b/Documentation/x86/shstk.rst new file mode 100644 index 000000000000..f2e6f323cf68 --- /dev/null +++ b/Documentation/x86/shstk.rst @@ -0,0 +1,166 @@ +.. SPDX-License-Identifier: GPL-2.0 + +====================================================== +Control-flow Enforcement Technology (CET) Shadow Stack +====================================================== + +CET Background +============== + +Control-flow Enforcement Technology (CET) is term referring to several +related x86 processor features that provides protection against control +flow hijacking attacks. The HW feature itself can be set up to protect +both applications and the kernel. + +CET introduces shadow stack and indirect branch tracking (IBT). Shadow stack +is a secondary stack allocated from memory and cannot be directly modified by +applications. When executing a CALL instruction, the processor pushes the +return address to both the normal stack and the shadow stack. Upon +function return, the processor pops the shadow stack copy and compares it +to the normal stack copy. If the two differ, the processor raises a +control-protection fault. IBT verifies indirect CALL/JMP targets are intended +as marked by the compiler with 'ENDBR' opcodes. Not all CPU's have both Shadow +Stack and Indirect Branch Tracking. Today in the 64-bit kernel, only userspace +shadow stack and kernel IBT are supported. + +Requirements to use Shadow Stack +================================ + +To use userspace shadow stack you need HW that supports it, a kernel +configured with it and userspace libraries compiled with it. + +The kernel Kconfig option is X86_USER_SHADOW_STACK, and it can be disabled +with the kernel parameter: nousershstk. + +To build a user shadow stack enabled kernel, Binutils v2.29 or LLVM v6 or later +are required. + +At run time, /proc/cpuinfo shows CET features if the processor supports +CET. "user_shstk" means that userspace shadow stack is supported on the current +kernel and HW. + +Application Enabling +==================== + +An application's CET capability is marked in its ELF note and can be verified +from readelf/llvm-readelf output:: + + readelf -n | grep -a SHSTK + properties: x86 feature: SHSTK + +The kernel does not process these applications markers directly. Applications +or loaders must enable CET features using the interface described in section 4. +Typically this would be done in dynamic loader or static runtime objects, as is +the case in GLIBC. + +Enabling arch_prctl()'s +======================= + +Elf features should be enabled by the loader using the below arch_prctl's. They +are only supported in 64 bit user applications. + +arch_prctl(ARCH_SHSTK_ENABLE, unsigned long feature) + Enable a single feature specified in 'feature'. Can only operate on + one feature at a time. + +arch_prctl(ARCH_SHSTK_DISABLE, unsigned long feature) + Disable a single feature specified in 'feature'. Can only operate on + one feature at a time. + +arch_prctl(ARCH_SHSTK_LOCK, unsigned long features) + Lock in features at their current enabled or disabled status. 'features' + is a mask of all features to lock. All bits set are processed, unset bits + are ignored. The mask is ORed with the existing value. So any feature bits + set here cannot be enabled or disabled afterwards. + +The return values are as follows. On success, return 0. On error, errno can +be:: + + -EPERM if any of the passed feature are locked. + -ENOTSUPP if the feature is not supported by the hardware or + kernel. + -EINVAL arguments (non existing feature, etc) + +The feature's bits supported are:: + + ARCH_SHSTK_SHSTK - Shadow stack + ARCH_SHSTK_WRSS - WRSS + +Currently shadow stack and WRSS are supported via this interface. WRSS +can only be enabled with shadow stack, and is automatically disabled +if shadow stack is disabled. + +Proc Status +=========== +To check if an application is actually running with shadow stack, the +user can read the /proc/$PID/status. It will report "wrss" or "shstk" +depending on what is enabled. The lines look like this:: + + x86_Thread_features: shstk wrss + x86_Thread_features_locked: shstk wrss + +Implementation of the Shadow Stack +================================== + +Shadow Stack Size +----------------- + +A task's shadow stack is allocated from memory to a fixed size of +MIN(RLIMIT_STACK, 4 GB). In other words, the shadow stack is allocated to +the maximum size of the normal stack, but capped to 4 GB. However, +a compat-mode application's address space is smaller, each of its thread's +shadow stack size is MIN(1/4 RLIMIT_STACK, 4 GB). + +Signal +------ + +By default, the main program and its signal handlers use the same shadow +stack. Because the shadow stack stores only return addresses, a large +shadow stack covers the condition that both the program stack and the +signal alternate stack run out. + +When a signal happens, the old pre-signal state is pushed on the stack. When +shadow stack is enabled, the shadow stack specific state is pushed onto the +shadow stack. Today this is only the old SSP (shadow stack pointer), pushed +in a special format with bit 63 set. On sigreturn this old SSP token is +verified and restored by the kernel. The kernel will also push the normal +restorer address to the shadow stack to help userspace avoid a shadow stack +violation on the sigreturn path that goes through the restorer. + +So the shadow stack signal frame format is as follows:: + + |1...old SSP| - Pointer to old pre-signal ssp in sigframe token format + (bit 63 set to 1) + | ...| - Other state may be added in the future + + +32 bit ABI signals are not supported in shadow stack processes. Linux prevents +32 bit execution while shadow stack is enabled by the allocating shadow stack's +outside of the 32 bit address space. When execution enters 32 bit mode, either +via far call or returning to userspace, a #GP is generated by the hardware +which, will be delivered to the process as a segfault. When transitioning to +userspace the register's state will be as if the userspace ip being returned to +caused the segfault. + +Fork +---- + +The shadow stack's vma has VM_SHADOW_STACK flag set; its PTEs are required +to be read-only and dirty. When a shadow stack PTE is not RO and dirty, a +shadow access triggers a page fault with the shadow stack access bit set +in the page fault error code. + +When a task forks a child, its shadow stack PTEs are copied and both the +parent's and the child's shadow stack PTEs are cleared of the dirty bit. +Upon the next shadow stack access, the resulting shadow stack page fault +is handled by page copy/re-use. + +When a pthread child is created, the kernel allocates a new shadow stack +for the new thread. New shadow stack's behave like mmap() with respect to +ASLR behavior. + +Exec +---- + +On exec, shadow stack features are disabled by the kernel. At which point, +userspace can choose to re-enable, or lock them. From patchwork Mon Feb 27 22:29:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FA93C7EE33 for ; Mon, 27 Feb 2023 22:31:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 894C06B0075; Mon, 27 Feb 2023 17:31:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7F5296B0078; Mon, 27 Feb 2023 17:31:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6476D6B007B; Mon, 27 Feb 2023 17:31:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4AA0E6B0075 for ; Mon, 27 Feb 2023 17:31:33 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 06A9D80B19 for ; Mon, 27 Feb 2023 22:31:33 +0000 (UTC) X-FDA: 80514519666.06.E1BA504 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf07.hostedemail.com (Postfix) with ESMTP id 13BEF40011 for ; Mon, 27 Feb 2023 22:31:30 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=UYYDe0XA; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537091; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=YBXDrbHF64FKzHpmMbwe8SmzWZN2R6oOuvtFVnYBnws=; b=xMNrWkEYBkaDK/nxMNUJlSJcUCrai9/97Q3IAwnG8v38QTMrEWaIBQkp2cCurPneSFZeTU NG6U8LnXHD6cg+2sHIEuBlZ1fbr+Y6AKpvMCvqq/HUz7lTompvbIbtCtM5GmegihAekqXL JzZJEWPxG4IpcZ7c7QdWtoJut5TeyDc= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=UYYDe0XA; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537091; a=rsa-sha256; cv=none; b=8gmeDeU31SwRkAt8LZvxv7ocjy0ora5Er1bo66jr30VntS9UxZYluR/sKPFYmzgcLTySPM U25aaKMwc5RZacuQuv5QSlNrSwDKpWJlmuAzXiQiAzO3p75mn8/30oIXA+U42Cfy3FdFo2 5TZQ0GHxSB0mvPcZ7kPq0Lzj+NsNWZw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537091; x=1709073091; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=lXB/Va8eFkKMhXnCWr4ZLsBS0D4WgDtbHK72wob+dBQ=; b=UYYDe0XAxW3rjPka/O3Ow3eSZzNSMpiajqranujPLO5fXJR436mX5kPT X5QaO1z4YiMfgRVmw8zNFFt7zErKjI2rSu60uIEw1/c5md5hJ+N3PM+Z4 8fKnRViioC9fSWATheil+hYDdM2VZKkcC1uySgu0L3dinhChMBrKrzJr5 vqXb4h+sF5senFiHBWe/bTqtbZ7KxV91RD7gTucXaUEabSD4bjXkaLik8 lC4kQzDkI5SI1xjxIcAq/m2K1fTTsR6YVcWHRP9Rz9XMH8pRoYZmlqSOa dBfkPLZt7SPd6w51ZCqkvaQYuSIs99XVm5kjZ4LJogQyQxm+oI8Sdy9Pp Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657029" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657029" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:08 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024341" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024341" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:07 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 02/41] x86/shstk: Add Kconfig option for shadow stack Date: Mon, 27 Feb 2023 14:29:18 -0800 Message-Id: <20230227222957.24501-3-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 13BEF40011 X-Stat-Signature: qci4aniewxzzdnngptqa86dq1wqu4b7d X-Rspam-User: X-HE-Tag: 1677537090-928050 X-HE-Meta: U2FsdGVkX1+9UzGbOadv/ebxKR5i149OXg/qz1zt7PgGVeOtXLsw7NUdDRB1Ywsn6oNeOi67SKA8TJRANGYCNoiIkVsplu7E34r8rSEtccESrix5tuzVNPKSA/fwHJfGPey0EElyi9oMoLPoNe+3m4mtiBt5SVZ0upWncDtqctE7DcgKHGrnybf3JaSL7hlBoLvtoXD1KJWdIy5Z2DvFwkNFhZrPAbhJceB4IFYNxrPTy0C7pNe3PHoofcz8eSuYB+x9I9+tU2dkcDlXX5FlUbw+7fZh1aPICirLvQO8FlJbxYzMMQwT2O4ph5GptgmhdvRENhgITbB34nGoZe//kYOxZ/6mF9KPIsUvf9sBNvi8l+AMmiLL4+x0EUSWBqDdu4lWUkbLXm6xi7rv09VPvnx9aZTa1N3mwizT0HJJQg7zJNy7tjYYV2b88BurYNDjc8NpLqlX3ZHpvlMTPyGXN6xRlFWXFGeL1rMEiIR0kZIAZoyxt3LBAz6j1GFulpvUEyqPeXV12rZzu5fsdQWSDy9hkeYl8LhocD72hxGBOhJ64qGr8YPKmBmln6Cy5a7w0sLAmXzZndV78F8pOE38TwvPeiAK8ovjFpQQxJO3lxMHhF+JlXNfsogGk0Q1l/MZeeAlTb7jwyjO1+0Xh43Ty3QAr1OUrVRH3T0HZh4a6xsp8oP6c2mm/ix78XpkzHjOovcN5hd5dRn7EsnykqC8JS0hTpYn2msbdZtYzTltIBmCKO3zoqPtC5LFx/xkyFLBquhpvEzCct4iMT1fBgfWKVGxNxaJSXh/lObiqNImq8RqpGEOxIGFmZnO/6Vv0gKt58jA/cRqJsUz33ZRJLR+9pRTEjHEgel9xtRF9vYQNyux+qhmvBKWJXOzA+F19EXPcmSOuqFVf5A4mq3GxF3PcVLHJqUlrSgvU5YLV5YUlmCrOsZQeJgFAiJ9Ia1p73yCZ7P7+V6C/o3xuPevXFk mqeyXQvq oyXJolIci8Tt/UyL+aSsZQ2YARHUmZCMpoe2SlFcyeTJoEeTCnE8Lnrezy0yQqMZGJ2Ov0Io/mk+PMDdytfhdYn5tldGbRTue1ahkA38ufXTDaRD5hYc0xMI6H3BsjdWyKVSDwTOVk4g7gHZDjbiBFqyLOBiO2UIo1FDcdYM8nS+P9PF4C8ocktmESspzZ7QzMTbEsJFp4UWVDjKBtgkU+dCVZTe/fQyjuMGY5kSOGJbYZRtGbm6Ihvc1+uaKFyVwfXrMJZdQ19LmuN+RopBtekwoXMAiprlB2rrxUdj1hYApp+Fwp01brHAa88HZjk+qqG1m3JB8ug2gVQjFib6QcesBJLedDcj511ElYso3bqhqG1qMZDdB6KQa+Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu Shadow stack provides protection for applications against function return address corruption. It is active when the processor supports it, the kernel has CONFIG_X86_SHADOW_STACK enabled, and the application is built for the feature. This is only implemented for the 64-bit kernel. When it is enabled, legacy non-shadow stack applications continue to work, but without protection. Since there is another feature that utilizes CET (Kernel IBT) that will share implementation with shadow stacks, create CONFIG_CET to signify that at least one CET feature is configured. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook --- v5: - Remove capitalization of shadow stack (Boris) v3: - Add X86_CET (Kees) - Add back WRUSS dependency (Kees) - Fix verbiage (Dave) - Change from promt to bool (Kirill) - Add more to commit log v2: - Remove already wrong kernel size increase info (tlgx) - Change prompt to remove "Intel" (tglx) - Update line about what CPUs are supported (Dave) Yu-cheng v25: - Remove X86_CET and use X86_SHADOW_STACK directly. --- arch/x86/Kconfig | 24 ++++++++++++++++++++++++ arch/x86/Kconfig.assembler | 5 +++++ 2 files changed, 29 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index a825bf031f49..f03791b73f9f 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1851,6 +1851,11 @@ config CC_HAS_IBT (CC_IS_CLANG && CLANG_VERSION >= 140000)) && \ $(as-instr,endbr64) +config X86_CET + def_bool n + help + CET features configured (Shadow stack or IBT) + config X86_KERNEL_IBT prompt "Indirect Branch Tracking" def_bool y @@ -1858,6 +1863,7 @@ config X86_KERNEL_IBT # https://github.com/llvm/llvm-project/commit/9d7001eba9c4cb311e03cd8cdc231f9e579f2d0f depends on !LD_IS_LLD || LLD_VERSION >= 140000 select OBJTOOL + select X86_CET help Build the kernel with support for Indirect Branch Tracking, a hardware support course-grain forward-edge Control Flow Integrity @@ -1952,6 +1958,24 @@ config X86_SGX If unsure, say N. +config X86_USER_SHADOW_STACK + bool "X86 userspace shadow stack" + depends on AS_WRUSS + depends on X86_64 + select ARCH_USES_HIGH_VMA_FLAGS + select X86_CET + help + Shadow stack protection is a hardware feature that detects function + return address corruption. This helps mitigate ROP attacks. + Applications must be enabled to use it, and old userspace does not + get protection "for free". + + CPUs supporting shadow stacks were first released in 2020. + + See Documentation/x86/shstk.rst for more information. + + If unsure, say N. + config EFI bool "EFI runtime service support" depends on ACPI diff --git a/arch/x86/Kconfig.assembler b/arch/x86/Kconfig.assembler index b88f784cb02e..8ad41da301e5 100644 --- a/arch/x86/Kconfig.assembler +++ b/arch/x86/Kconfig.assembler @@ -24,3 +24,8 @@ config AS_GFNI def_bool $(as-instr,vgf2p8mulb %xmm0$(comma)%xmm1$(comma)%xmm2) help Supported by binutils >= 2.30 and LLVM integrated assembler + +config AS_WRUSS + def_bool $(as-instr,wrussq %rax$(comma)(%rbx)) + help + Supported by binutils >= 2.31 and LLVM integrated assembler From patchwork Mon Feb 27 22:29:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154234 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B934C64ED6 for ; Mon, 27 Feb 2023 22:31:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0F2346B0078; Mon, 27 Feb 2023 17:31:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0A4FD6B007B; Mon, 27 Feb 2023 17:31:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E371A6B007D; Mon, 27 Feb 2023 17:31:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D50B56B0078 for ; Mon, 27 Feb 2023 17:31:33 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9969E40A27 for ; Mon, 27 Feb 2023 22:31:33 +0000 (UTC) X-FDA: 80514519666.03.FFA8BCF Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf28.hostedemail.com (Postfix) with ESMTP id 9D005C000F for ; Mon, 27 Feb 2023 22:31:31 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=LV4qOqFY; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537091; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=m82UYX5aQssRgl8psmBP9zJk+xjZFbSau758VDt+A0c=; b=OhmSwWbdMUoN4FtyEIt0IwUy/CdfnvuC3aVjWoP4ylUdCIM7deaFNc4gE901q04En8vMiI ec8Y7kGX/fI3H1FknbM6hGpcJVE/ZQhafZridzX1ey/r+76/sTCJ4AaUnM9qwVZe0X4Ds8 gv/7fJOJpKFlxJqkT8lFn2w7gKApgow= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=LV4qOqFY; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537091; a=rsa-sha256; cv=none; b=8oAAlECNNEZ57UO7YJH4reeDojuN9qvhtsgpDCRFVldbMIh+lPtNpCAmaBLPuSxEzRKJEA CXzIIfCK2HxllKTpQeROlDtz+JZtqoUqwJX9LVqq5jHpj7CuNMR5KOcQrkGVg9BSRlcED/ xC7Qp9Cv7C14O7iN9YWXJbWjeRnBBTQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537091; x=1709073091; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=9fd4C/mVhbbyV9RlyDxCvsqphPPcMSnaWW7TQtu1kzU=; b=LV4qOqFY1LOQS6MRtaXtSF2F8GOfsuwIa93bqgUdxrqb0ONKhlXuVXe7 2Dn3SU4h34ECi4DtHHorWS150pT8yg4lc6Ou6so61luRnptR+YRsZeldo +gEOK30b7+lWV/5AiHsK6pAMhKV9Xs0ga1QkgOuO42nNBXU+HHgdTZmYF aybdl2mt75spd7uOs2+WFHA14mzC+/dTwi1IQmMA3ZmPLydIwKE5gu2Tq t1MSytdf44fv8NR5Pg5DE52uqVVWVENAPoH4rC3JKyv97Qbo7zEDMhC/h MePYbHQY1cFtl9/Q1R7Kxge3Id1+uxmqgwLJM5+WyclqDhjr3Pehwl9wB w==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657051" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657051" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:09 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024349" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024349" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:08 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 03/41] x86/cpufeatures: Add CPU feature flags for shadow stacks Date: Mon, 27 Feb 2023 14:29:19 -0800 Message-Id: <20230227222957.24501-4-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 9D005C000F X-Stat-Signature: xwzwb71e7rjw5oytbit7mf1zu9qy31kb X-Rspam-User: X-HE-Tag: 1677537091-668506 X-HE-Meta: U2FsdGVkX1/sNtNsrfOLOoiUBoatk0ftzzyHn4L/MXp/VE+/P73KMVIdFVxrzr4kWw95IDVdxoRqQMUvoe+c4RrFy7n4PpFAaohQDSX6Vf4bL+G1gWs8vzxhGBqk6fcHo6itgvneHGxWsWr+d9uq28t0ASSW/mjDGcAHsFFcwqpv70vijVdlAjstd87ckB0/175jUUGRk4wsIFc259keIvZv1umm2Pt2k1Vs4g76w6TC5jiThzYyZTmZ20rHX6JvV+fUyGVXk09FU6e/Rmv1dCIQE9KMsrYHc/JAqbG2x799R9kMtxO/sxmerrsuLAyz1PfrH5IUnyooPXVvKrjKMn7cqvhMKAPOngZ01DdRHGDTSWvIswgogMiEYGS52bkSshgmv+2gxDYUO7MkN/icr0Nv3u3f/SmdImUYHear4MU5nDI6vOTyFg9UWPDh9d3rZKPpap6fv1PX+UJN9rnGzGmG+y3pmUdWkyQs+gwqH5OllqlEPzuELz3CydB4i0DaqfTkqInI4+oRJRK19qBtmPcIK+KuoSkKNW5IEOCJSnxFwGO/20buw0cWUi8NX/kS4jSTDhk9LpECygL1pEUNZLaJnoIK2tfHLs4hGh1PlYsZrYUa6BbPEmYyjw+SgEjgjrdfbkbfDM1xOK9OSxGih3uXvE0xZ1++pOUhvyWXucEGDXW8OMgjXdybA4k34qp7gqb3Rgwvd5KjhtdImcONJpLBOwZ8mtiItMO8g0P3GLM/pIKSkFOPZosRp74DqJqrFLXj9qGDPN2/Ms+sPxDPFu9FIsM47IG0b2IIq/DYsmHY2DllPaLJQbGoLlg2U9g+NdzOWB0IIHzCVQs6TUW+aRrqi2tBHkqK1gxNz75wLNHRxJvBtV0LDkXVtSVeIkoMHKljNGM2TopsZXFkVk9UKc5B6oC+l0ExJTdyIqkg06gaNgswHDcE0cgOe6M/LLIrgkQVeVc4G8WiVfOMz/f ID24nsn6 yX729RctneAGXR7u6qga9jYpLjSqxeXlU6So4wK5Ns7pfxBKHl4+zizAfPxvTsqizlgIL6xNKP8NAGZuECuegCe0vtwlP9QqRsu39hKzIKiHLpNiCr7FfbHzK3J3LOYCfdRlf+RJzyx0memGAPliieBWl9O4YxM9sjklh1Z5OffW3nlVd4/AXwj5BIM9jf2umf7ckn67EQKuiX9E0HkDu44Xwhru1g4XzRuQgVDn7DfVjMRI2pxfJ/o93QRJEho07LJSm6dCMQy0I0hxslKATjCyO85BuwWGBIz5IE6yXUXDBcb4N5/Sq/e/KQUY2OYtoXJQD3weqEu595DbYZV9GxPAT0FMPE+JbQCMh X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu The Control-Flow Enforcement Technology contains two related features, one of which is Shadow Stacks. Future patches will utilize this feature for shadow stack support in KVM, so add a CPU feature flags for Shadow Stacks (CPUID.(EAX=7,ECX=0):ECX[bit 7]). To protect shadow stack state from malicious modification, the registers are only accessible in supervisor mode. This implementation context-switches the registers with XSAVES. Make X86_FEATURE_SHSTK depend on XSAVES. The shadow stack feature, enumerated by the CPUID bit described above, encompasses both supervisor and userspace support for shadow stack. In near future patches, only userspace shadow stack will be enabled. In expectation of future supervisor shadow stack support, create a software CPU capability to enumerate kernel utilization of userspace shadow stack support. This user shadow stack bit should depend on the HW "shstk" capability and that logic will be implemented in future patches. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook --- v5: - Drop "shstk" from cpuinfo (Boris) - Remove capitalization on shadow stack (Boris) v3: - Add user specific shadow stack cpu cap (Andrew Cooper) - Drop reviewed-bys from Boris and Kees due to the above change. v2: - Remove IBT reference in commit log (Kees) - Describe xsaves dependency using text from (Dave) v1: - Remove IBT, can be added in a follow on IBT series. --- arch/x86/include/asm/cpufeatures.h | 2 ++ arch/x86/include/asm/disabled-features.h | 8 +++++++- arch/x86/kernel/cpu/cpuid-deps.c | 1 + 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 389ea336258f..d01afabcf03e 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -309,6 +309,7 @@ #define X86_FEATURE_MSR_TSX_CTRL (11*32+20) /* "" MSR IA32_TSX_CTRL (Intel) implemented */ #define X86_FEATURE_SMBA (11*32+21) /* "" Slow Memory Bandwidth Allocation */ #define X86_FEATURE_BMEC (11*32+22) /* "" Bandwidth Monitoring Event Configuration */ +#define X86_FEATURE_USER_SHSTK (11*32+23) /* Shadow stack support for user mode applications */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */ @@ -375,6 +376,7 @@ #define X86_FEATURE_OSPKE (16*32+ 4) /* OS Protection Keys Enable */ #define X86_FEATURE_WAITPKG (16*32+ 5) /* UMONITOR/UMWAIT/TPAUSE Instructions */ #define X86_FEATURE_AVX512_VBMI2 (16*32+ 6) /* Additional AVX512 Vector Bit Manipulation Instructions */ +#define X86_FEATURE_SHSTK (16*32+ 7) /* "" Shadow stack */ #define X86_FEATURE_GFNI (16*32+ 8) /* Galois Field New Instructions */ #define X86_FEATURE_VAES (16*32+ 9) /* Vector AES */ #define X86_FEATURE_VPCLMULQDQ (16*32+10) /* Carry-Less Multiplication Double Quadword */ diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index 5dfa4fb76f4b..505f78ddca82 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -99,6 +99,12 @@ # define DISABLE_TDX_GUEST (1 << (X86_FEATURE_TDX_GUEST & 31)) #endif +#ifdef CONFIG_X86_USER_SHADOW_STACK +#define DISABLE_USER_SHSTK 0 +#else +#define DISABLE_USER_SHSTK (1 << (X86_FEATURE_USER_SHSTK & 31)) +#endif + /* * Make sure to add features to the correct mask */ @@ -114,7 +120,7 @@ #define DISABLED_MASK9 (DISABLE_SGX) #define DISABLED_MASK10 0 #define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ - DISABLE_CALL_DEPTH_TRACKING) + DISABLE_CALL_DEPTH_TRACKING|DISABLE_USER_SHSTK) #define DISABLED_MASK12 0 #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 diff --git a/arch/x86/kernel/cpu/cpuid-deps.c b/arch/x86/kernel/cpu/cpuid-deps.c index f6748c8bd647..e462c1d3800a 100644 --- a/arch/x86/kernel/cpu/cpuid-deps.c +++ b/arch/x86/kernel/cpu/cpuid-deps.c @@ -81,6 +81,7 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_XFD, X86_FEATURE_XSAVES }, { X86_FEATURE_XFD, X86_FEATURE_XGETBV1 }, { X86_FEATURE_AMX_TILE, X86_FEATURE_XFD }, + { X86_FEATURE_SHSTK, X86_FEATURE_XSAVES }, {} }; From patchwork Mon Feb 27 22:29:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7D0EC64ED8 for ; Mon, 27 Feb 2023 22:31:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 70E896B007B; Mon, 27 Feb 2023 17:31:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6E5816B007D; Mon, 27 Feb 2023 17:31:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 55CDD6B007E; Mon, 27 Feb 2023 17:31:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 460B46B007B for ; Mon, 27 Feb 2023 17:31:34 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 2239A140928 for ; Mon, 27 Feb 2023 22:31:34 +0000 (UTC) X-FDA: 80514519708.29.3314B35 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf13.hostedemail.com (Postfix) with ESMTP id 1BDAE20009 for ; Mon, 27 Feb 2023 22:31:31 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="bp4/wOhm"; spf=pass (imf13.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537092; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=IwpMesFlHIDR54EkEjKtDtUDsVUQ1W/vfrUDOH5zIbg=; b=SPJUr4AsHt/YhNxMXweJd32aFkqQH7BSf9Y9OqoU66seO3BtzY14xrFwaRD9hi/liTVZg0 VhPokIsJrYWGDmc4R8p5UWaOAaGC+DBsAGkYu6GH/WxHDDYkkH4Z2SeknEKnmmxTbs2kCg Zlh7b+zY502C/oOuKCLjAU7tX8k8UEo= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="bp4/wOhm"; spf=pass (imf13.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537092; a=rsa-sha256; cv=none; b=JgL1Py+3vpuNCWvH8zVrVZmVj7Azy73ByBNPdyPawh5AfI12VbKsmH76jZix/l8+NLiXSZ pFDBRwVBS0zfzWLqanxa+pABVDMhN6f0xMO5LuK57DO6BJFKM0AO4rvxIpzlktMpfSwwlr IDcyxc0JvcnwzK4kr6fRJWRpDCDB56w= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537092; x=1709073092; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Bg+qgKp1F47Y8UIietH1jC4hvVo4rClMm1kv9uYsMSQ=; b=bp4/wOhm2V6fL0qXlaX+94pWqzoaB5zUU7e3HTpJP1zmE1I7qvNDUmaP p+E9tzoCO+0GecbbhJgjJ7pYUz6YIYTeP97u4OJiaONSVfreVxUpvoht4 bWWsm8n7xApU9c7e5hVa0Alcf0m+xZx/4qVyuEDAVUlA8zyMV+STb+WpF nBkYWJiRplNh4o4jvV1u+L27ngb2w2quICp93TyB6eFuS/wpQktTkH+y+ qF85U61PvzoiwEtRrGBKeR7PTag1Cqn4TYUDyt/ZLm9wxvR4dutfkQAqJ 1PClZ63pDzJ5VWje4opXTBwpP6Ns7pViRxuutKLwnLPi7UXXeqpk8/WFn w==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657074" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657074" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:10 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024368" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024368" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:09 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 04/41] x86/cpufeatures: Enable CET CR4 bit for shadow stack Date: Mon, 27 Feb 2023 14:29:20 -0800 Message-Id: <20230227222957.24501-5-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 1BDAE20009 X-Stat-Signature: ndub5akk855gieskiqrtd8ip8pztoknd X-Rspam-User: X-HE-Tag: 1677537091-715931 X-HE-Meta: U2FsdGVkX19j10Pw3ZU23jVyqGlxe1qRuVNZKfCJj9NZoQcvkcMXH3bQoTFxO1Cduo2/a25RIjdBK+xB6OZWBRX18fBetybG8VdO9XxAgD/okOzZU/lEAyQ6yel7X9PFhSU+Wf/Nn9DRYznCfHpaoiBbXanc6FK96pxE8I1gSufDB+FkOJkFQ/OrGB2nMjaG7tkl7hfvgspdlnXr2xcmPGOLQ93/GOQRZ2sVrVMWRbY+r3TCWzWiTQsPc2rPHdFhV2xjhlHS2McL1fGH3CJCiB5fpMcfMOjfHchmuRzDF7mUksMzZNazN8TPNJA72yTiZ4z3HbRjATnMg4amfoinQRFkgsQY2aFEha28pZxKQmcr2S3+K/+VGdzmWgjEI3xlwY+wbhnZaSI/zVCoCzeJhvsEg9MlNI/oiCWoFwTt2huulV9oIin26141GFrqB/ioJYZiP0UdxoYXzYaJvEIfGr7W3q3uFuc9MbZVftNnJT/I6x559Pcvd6LktNoQkpCrABzwhLaPhuvt3erBXHQXZZ8Z8fk7BOUeHhmzTkQ+LGWEy80IMq8byAdJentjLW3eoLu+BmGwwfP5nzwUZHB7Gvs1F9b5a0l6cq8lJGnrAmH5M8i58mtthgI/UBHE8jmVegXR82AurWEjkLV7D2PJsZVYidkzuB/GumslKuvkX2lAB7YWRcIIrnRwYnsYWg0qJRp7oCW8H8aN5xkLXf2u4+uq4t4AHcjhng0rGpBsUpM8PmigJDwy+/8YD+SSmX3xVKFCuKRLHn32ssgLUwJ2fOBYVm2o7/2HqZzf3A1gz3k4m3xmeXRrYFMwyFTRCMCCD7Mi0SfhowImDyPtxVwAQayrqC7BF3lyN91zVF3UGeqUEOhln94v2bNzpNXYfCqSlX4aU1SpoNqLAvMfBI/kbYGWJNIXgL4e0NNzRwHGplshUDKO2nKespjQqvSNR+Uu6NN5tsWZdxXqd+ux34U MJ+Qn+yO EJE1qUsvbgix6O6C3Vmsoq5ZslqwOXEb5WFW3zlDa2JRBUg2uVI7ZD4EErS/xg7YslpSf2LtLirISzhnymr2U+V8nyQMZgB4kJ8aglpIeA6YaecVGDzxClHOjE2AuDPvhtDQ4n30mqq6zbVwgaioHESMZ5MP71U0moyTsYo8NoZUK0DihTsysV0ftaeeV1+5WvBdt/+P6NLlbBr3xQma3U1fn3Ea3sh/If193qD4Tg2M5sX+40aaOJCQlhHzyufp/Jg4IDmeQXC8MV/D/0GEZsLx+PC5AQwbHQZ+NSp7VsSWn/pHMVubDpNnD+AhhcmH00X9GhQl5QuqlfAQASg1w+KQKyyoM0lkk9urD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu Setting CR4.CET is a prerequisite for utilizing any CET features, most of which also require setting MSRs. Kernel IBT already enables the CET CR4 bit when it detects IBT HW support and is configured with kernel IBT. However, future patches that enable userspace shadow stack support will need the bit set as well. So change the logic to enable it in either case. Clear MSR_IA32_U_CET in cet_disable() so that it can't live to see userspace in a new kexec-ed kernel that has CR4.CET set from kernel IBT. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook --- v5: - Remove #ifdeffery (Boris) v4: - Add back dedicated command line disable: "nousershstk" (Boris) v3: - Remove stay new line (Boris) - Simplify commit log (Andrew Cooper) v2: - In the shadow stack case, go back to only setting CR4.CET if the kernel is compiled with user shadow stack support. - Clear MSR_IA32_U_CET as well. (PeterZ) --- arch/x86/kernel/cpu/common.c | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 8cd4126d8253..cc686e5039be 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -600,27 +600,43 @@ __noendbr void ibt_restore(u64 save) static __always_inline void setup_cet(struct cpuinfo_x86 *c) { - u64 msr = CET_ENDBR_EN; + bool user_shstk, kernel_ibt; - if (!HAS_KERNEL_IBT || - !cpu_feature_enabled(X86_FEATURE_IBT)) + if (!IS_ENABLED(CONFIG_X86_CET)) return; - wrmsrl(MSR_IA32_S_CET, msr); + kernel_ibt = HAS_KERNEL_IBT && cpu_feature_enabled(X86_FEATURE_IBT); + user_shstk = cpu_feature_enabled(X86_FEATURE_SHSTK) && + IS_ENABLED(CONFIG_X86_USER_SHADOW_STACK); + + if (!kernel_ibt && !user_shstk) + return; + + if (user_shstk) + set_cpu_cap(c, X86_FEATURE_USER_SHSTK); + + if (kernel_ibt) + wrmsrl(MSR_IA32_S_CET, CET_ENDBR_EN); + else + wrmsrl(MSR_IA32_S_CET, 0); + cr4_set_bits(X86_CR4_CET); - if (!ibt_selftest()) { + if (kernel_ibt && !ibt_selftest()) { pr_err("IBT selftest: Failed!\n"); wrmsrl(MSR_IA32_S_CET, 0); setup_clear_cpu_cap(X86_FEATURE_IBT); - return; } } __noendbr void cet_disable(void) { - if (cpu_feature_enabled(X86_FEATURE_IBT)) - wrmsrl(MSR_IA32_S_CET, 0); + if (!(cpu_feature_enabled(X86_FEATURE_IBT) || + cpu_feature_enabled(X86_FEATURE_SHSTK))) + return; + + wrmsrl(MSR_IA32_S_CET, 0); + wrmsrl(MSR_IA32_U_CET, 0); } /* @@ -1482,6 +1498,9 @@ static void __init cpu_parse_early_param(void) if (cmdline_find_option_bool(boot_command_line, "noxsaves")) setup_clear_cpu_cap(X86_FEATURE_XSAVES); + if (cmdline_find_option_bool(boot_command_line, "nousershstk")) + setup_clear_cpu_cap(X86_FEATURE_USER_SHSTK); + arglen = cmdline_find_option(boot_command_line, "clearcpuid", arg, sizeof(arg)); if (arglen <= 0) return; From patchwork Mon Feb 27 22:29:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154236 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 534F7C64ED9 for ; Mon, 27 Feb 2023 22:31:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AB91B6B007D; Mon, 27 Feb 2023 17:31:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A18536B007E; Mon, 27 Feb 2023 17:31:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 86A796B0080; Mon, 27 Feb 2023 17:31:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 735796B007D for ; Mon, 27 Feb 2023 17:31:35 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 57B1A1C5DEA for ; Mon, 27 Feb 2023 22:31:35 +0000 (UTC) X-FDA: 80514519750.16.F8BDE9C Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf07.hostedemail.com (Postfix) with ESMTP id 47F8040011 for ; Mon, 27 Feb 2023 22:31:33 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=NSIF8lDA; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537093; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=AYA2D21wCSZCsg77UCdaKj768b2rQFr+02Vl43G5Tcc=; b=ieKBmrCsj/flr7oP2/LueiNfU3n7rD48PDIbncsGYmGpppzzlRcGpdZQqXNnggPKlYrXHe POpEyOJA/Xgx3ZMswrdCLrxwN+NO6Q3uKYho6S0SuAyRWp8H3Lk0qSJxNZO5Vlh6bZqWQS 2sUNICX5qIOk6UtdqKaT03RWftMG+rs= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=NSIF8lDA; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537093; a=rsa-sha256; cv=none; b=j79qyTyWofsBJkE3SHVBytWKAk8Lmse4P1M5YfTbCKhR/Oh1ZqPnmy4JCFVJVo5scr/kLF zScffmWEWjABShjQo7EEtZFc+3dVXs2o/AhnkN8m/K1qwP7MfxSjy7x5L8ZNy5vwzadLss D8gTbnkCPpgVrvdpM39RkGFqsj7Eqf8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537093; x=1709073093; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=3cQeDPuVgUbdnvCzorxup8sw+mhzntiHilKqyamxhz4=; b=NSIF8lDAtCLaPdgUdLczrpAHYzubrv6RHCxVEqZ+FcHHPZ3PdnYNE6V+ ZK6NvSEhNG9rrzJ0BbjwBO5ShAn93cK/xSgpqLHo8CtLTUdZGQG1fbQXH oOnDkk46EJs/ug3+b/xmuZSimFnavmhjidyV20aPne25MfjfdrYkvd2XH EWtkHt95hF1XwtaqxUvA1XImq/4Pn74Au7wly3uOU/GROjf4NshLkLkk8 rr44i+9A7y5HpbQTBDEBZkSc3VS9uEZYiZ9x1LrehIgLZ7wQoRsQOf3QV xH98w6khxXkZkbKxXAxztv6IA6Q/3n1UE0UBWz5S3lcNazJxwvVVJrOwD A==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657106" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657106" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:11 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024382" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024382" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:10 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 05/41] x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states Date: Mon, 27 Feb 2023 14:29:21 -0800 Message-Id: <20230227222957.24501-6-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 47F8040011 X-Stat-Signature: uaewuf64sbd6ewxwkpt184b5dju75b7d X-Rspam-User: X-HE-Tag: 1677537093-330837 X-HE-Meta: U2FsdGVkX18Cyt8gheBGhNbK0KvW8l0jPlrlMzElqX+uWQwdHS0IGjvMmTkwy6ZuEyO7mUc9Qojk5be1NuCP/Jovy0d7Umv6bcePPSdgsQ0oYQDZVDQRGfk4th3Fy6k2S/+4sz5FlgKjbN5l6isofxe3qf20jNgUa5amZyPFPPd0SRiAI0yckqCOsKvplEpGm6k/uCrRUIWs66d9jWkjvSdUNhjpymW1h4Z3De2GsmNk7315MAn5DcUPq916gI7pbWJ1vBUKNS83ogFm35KsACMFLFpi0/9UcQaf9YIhb5iPAcEWEE+C3Pfn0h/WmMwlZ3ffMU/gfDEH1S7Jzdfqp35oy+lFSQRhIbws0W6T9e5CGJ8eFdqArG1e+u1BXLPTSY8vvZ2KPPODsBDDmRTMP8UPWvLITPJA7nZT9zci71EA20z62jHzecTcYYcEH1M9b4W7Gvl3qlf39g0ZJYhc/YUiz91qmEcqV4dA44hGu2Fm7khFKdiZLYFZ9LC9abjU32y1u3yvb4k8qojvKhVVLnf6IaJas2H6mAeqT+4FQsAsoePoTk+2oluPxs7PvzV6PVPNEf+X7/lEK9KEQ+o55Ec/w2dJcnbzl6FLOIWBSosYEj/07j8KV2TtlyLN0vC4bJqR3+RNxUgqVtA+jlzEMMe6ktRDvpk581wJGLys9LXnb4x9D+7lpGBVlQ1qHVKE8oCyQe8lvT9BteTWpTR1IJipENt5tGMdKNzLjl5neJwqTH0T4ckZtF8F4zPYM79XskLM31h9t6+RMgW3x+lUy3/71cxa3dP+jGv6rWH42qUoBknnWyz+hKk+tmXPH+IvEtBSYCJ5TIoMb5irUOGMLMNpq1qbuNcfgj/lG34hVZy/Uy2ePL6pTe15QWsuWPfUo/BCLzNplUaS2XaWfh9sAVqpuHy2cOdZN007H0Qf0iEEUWPu2dL41WCwlSx2tXISdJqdAGMKWm2zJCIrn5J s5jU5xxm qK0p7k1YGdFm4t9ZwmakeqgJcRmyJUrwzW3hG8UprIveZSbbkBDeklnjkr+T4uJ8ADrv2qvpMov5rrwZCf5IatPCYLHnSYgS07j+ml4Fgs6e57r2jeSNauxydhh2ytlpD+r6lSQlAwweTsR+Oh+dsTdtYEiAb76pTSzxkuMB6LLKS/Nbu4GU8Z1lmgeuxQLRaa/WtfW29p5zBi8VH1/EL+33VXzhTtcnVc8zwkKO6QOEXkcbaTHqYljJbcQJE9neuYmQEHXqyNUgIMOk2A51L2ulWUXLlU/531UiU501MnZbbrdypAQkj4pLIaR0N/Jau+dcfWtRUqvGkk/cq2GWWimpORtLVq/ILm8/E X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu Shadow stack register state can be managed with XSAVE. The registers can logically be separated into two groups: * Registers controlling user-mode operation * Registers controlling kernel-mode operation The architecture has two new XSAVE state components: one for each group of those groups of registers. This lets an OS manage them separately if it chooses. Future patches for host userspace and KVM guests will only utilize the user-mode registers, so only configure XSAVE to save user-mode registers. This state will add 16 bytes to the xsave buffer size. Future patches will use the user-mode XSAVE area to save guest user-mode CET state. However, VMCS includes new fields for guest CET supervisor states. KVM can use these to save and restore guest supervisor state, so host supervisor XSAVE support is not required. Adding this exacerbates the already unwieldy if statement in check_xstate_against_struct() that handles warning about un-implemented xfeatures. So refactor these check's by having XCHECK_SZ() set a bool when it actually check's the xfeature. This ends up exceeding 80 chars, but was better on balance than other options explored. Pass the bool as pointer to make it clear that XCHECK_SZ() can change the variable. While configuring user-mode XSAVE, clarify kernel-mode registers are not managed by XSAVE by defining the xfeature in XFEATURE_MASK_SUPERVISOR_UNSUPPORTED, like is done for XFEATURE_MASK_PT. This serves more of a documentation as code purpose, and functionally, only enables a few safety checks. Both XSAVE state components are supervisor states, even the state controlling user-mode operation. This is a departure from earlier features like protection keys where the PKRU state is a normal user (non-supervisor) state. Having the user state be supervisor-managed ensures there is no direct, unprivileged access to it, making it harder for an attacker to subvert CET. To facilitate this privileged access, define the two user-mode CET MSRs, and the bits defined in those MSRs relevant to future shadow stack enablement patches. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook --- v5: - Move comments from end of lines in cet_user_state struct (Boris) v3: - Add missing "is" in commit log (Boris) - Change to case statement for struct size checking (Boris) - Adjust commas on xfeature_names (Kees, Boris) v2: - Change name to XFEATURE_CET_KERNEL_UNUSED (peterz) KVM refresh: - Reword commit log using some verbiage posted by Dave Hansen - Remove unlikely to be used supervisor cet xsave struct - Clarify that supervisor cet state is not saved by xsave - Remove unused supervisor MSRs --- arch/x86/include/asm/fpu/types.h | 16 +++++- arch/x86/include/asm/fpu/xstate.h | 6 ++- arch/x86/kernel/fpu/xstate.c | 90 +++++++++++++++---------------- 3 files changed, 61 insertions(+), 51 deletions(-) diff --git a/arch/x86/include/asm/fpu/types.h b/arch/x86/include/asm/fpu/types.h index 7f6d858ff47a..eb810074f1e7 100644 --- a/arch/x86/include/asm/fpu/types.h +++ b/arch/x86/include/asm/fpu/types.h @@ -115,8 +115,8 @@ enum xfeature { XFEATURE_PT_UNIMPLEMENTED_SO_FAR, XFEATURE_PKRU, XFEATURE_PASID, - XFEATURE_RSRVD_COMP_11, - XFEATURE_RSRVD_COMP_12, + XFEATURE_CET_USER, + XFEATURE_CET_KERNEL_UNUSED, XFEATURE_RSRVD_COMP_13, XFEATURE_RSRVD_COMP_14, XFEATURE_LBR, @@ -138,6 +138,8 @@ enum xfeature { #define XFEATURE_MASK_PT (1 << XFEATURE_PT_UNIMPLEMENTED_SO_FAR) #define XFEATURE_MASK_PKRU (1 << XFEATURE_PKRU) #define XFEATURE_MASK_PASID (1 << XFEATURE_PASID) +#define XFEATURE_MASK_CET_USER (1 << XFEATURE_CET_USER) +#define XFEATURE_MASK_CET_KERNEL (1 << XFEATURE_CET_KERNEL_UNUSED) #define XFEATURE_MASK_LBR (1 << XFEATURE_LBR) #define XFEATURE_MASK_XTILE_CFG (1 << XFEATURE_XTILE_CFG) #define XFEATURE_MASK_XTILE_DATA (1 << XFEATURE_XTILE_DATA) @@ -252,6 +254,16 @@ struct pkru_state { u32 pad; } __packed; +/* + * State component 11 is Control-flow Enforcement user states + */ +struct cet_user_state { + /* user control-flow settings */ + u64 user_cet; + /* user shadow stack pointer */ + u64 user_ssp; +}; + /* * State component 15: Architectural LBR configuration state. * The size of Arch LBR state depends on the number of LBRs (lbr_depth). diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index cd3dd170e23a..d4427b88ee12 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -50,7 +50,8 @@ #define XFEATURE_MASK_USER_DYNAMIC XFEATURE_MASK_XTILE_DATA /* All currently supported supervisor features */ -#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID) +#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID | \ + XFEATURE_MASK_CET_USER) /* * A supervisor state component may not always contain valuable information, @@ -77,7 +78,8 @@ * Unsupported supervisor features. When a supervisor feature in this mask is * supported in the future, move it to the supported supervisor feature mask. */ -#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT) +#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT | \ + XFEATURE_MASK_CET_KERNEL) /* All supervisor states including supported and unsupported states. */ #define XFEATURE_MASK_SUPERVISOR_ALL (XFEATURE_MASK_SUPERVISOR_SUPPORTED | \ diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 714166cc25f2..13a80521dd51 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -39,26 +39,26 @@ */ static const char *xfeature_names[] = { - "x87 floating point registers" , - "SSE registers" , - "AVX registers" , - "MPX bounds registers" , - "MPX CSR" , - "AVX-512 opmask" , - "AVX-512 Hi256" , - "AVX-512 ZMM_Hi256" , - "Processor Trace (unused)" , + "x87 floating point registers", + "SSE registers", + "AVX registers", + "MPX bounds registers", + "MPX CSR", + "AVX-512 opmask", + "AVX-512 Hi256", + "AVX-512 ZMM_Hi256", + "Processor Trace (unused)", "Protection Keys User registers", "PASID state", - "unknown xstate feature" , - "unknown xstate feature" , - "unknown xstate feature" , - "unknown xstate feature" , - "unknown xstate feature" , - "unknown xstate feature" , - "AMX Tile config" , - "AMX Tile data" , - "unknown xstate feature" , + "Control-flow User registers", + "Control-flow Kernel registers (unused)", + "unknown xstate feature", + "unknown xstate feature", + "unknown xstate feature", + "unknown xstate feature", + "AMX Tile config", + "AMX Tile data", + "unknown xstate feature", }; static unsigned short xsave_cpuid_features[] __initdata = { @@ -73,6 +73,7 @@ static unsigned short xsave_cpuid_features[] __initdata = { [XFEATURE_PT_UNIMPLEMENTED_SO_FAR] = X86_FEATURE_INTEL_PT, [XFEATURE_PKRU] = X86_FEATURE_PKU, [XFEATURE_PASID] = X86_FEATURE_ENQCMD, + [XFEATURE_CET_USER] = X86_FEATURE_SHSTK, [XFEATURE_XTILE_CFG] = X86_FEATURE_AMX_TILE, [XFEATURE_XTILE_DATA] = X86_FEATURE_AMX_TILE, }; @@ -276,6 +277,7 @@ static void __init print_xstate_features(void) print_xstate_feature(XFEATURE_MASK_Hi16_ZMM); print_xstate_feature(XFEATURE_MASK_PKRU); print_xstate_feature(XFEATURE_MASK_PASID); + print_xstate_feature(XFEATURE_MASK_CET_USER); print_xstate_feature(XFEATURE_MASK_XTILE_CFG); print_xstate_feature(XFEATURE_MASK_XTILE_DATA); } @@ -344,6 +346,7 @@ static __init void os_xrstor_booting(struct xregs_state *xstate) XFEATURE_MASK_BNDREGS | \ XFEATURE_MASK_BNDCSR | \ XFEATURE_MASK_PASID | \ + XFEATURE_MASK_CET_USER | \ XFEATURE_MASK_XTILE) /* @@ -446,14 +449,15 @@ static void __init __xstate_dump_leaves(void) } \ } while (0) -#define XCHECK_SZ(sz, nr, nr_macro, __struct) do { \ - if ((nr == nr_macro) && \ - WARN_ONCE(sz != sizeof(__struct), \ - "%s: struct is %zu bytes, cpu state %d bytes\n", \ - __stringify(nr_macro), sizeof(__struct), sz)) { \ +#define XCHECK_SZ(sz, nr, __struct) ({ \ + if (WARN_ONCE(sz != sizeof(__struct), \ + "[%s]: struct is %zu bytes, cpu state %d bytes\n", \ + xfeature_names[nr], sizeof(__struct), sz)) { \ __xstate_dump_leaves(); \ } \ -} while (0) + true; \ +}) + /** * check_xtile_data_against_struct - Check tile data state size. @@ -527,36 +531,28 @@ static bool __init check_xstate_against_struct(int nr) * Ask the CPU for the size of the state. */ int sz = xfeature_size(nr); + /* * Match each CPU state with the corresponding software * structure. */ - XCHECK_SZ(sz, nr, XFEATURE_YMM, struct ymmh_struct); - XCHECK_SZ(sz, nr, XFEATURE_BNDREGS, struct mpx_bndreg_state); - XCHECK_SZ(sz, nr, XFEATURE_BNDCSR, struct mpx_bndcsr_state); - XCHECK_SZ(sz, nr, XFEATURE_OPMASK, struct avx_512_opmask_state); - XCHECK_SZ(sz, nr, XFEATURE_ZMM_Hi256, struct avx_512_zmm_uppers_state); - XCHECK_SZ(sz, nr, XFEATURE_Hi16_ZMM, struct avx_512_hi16_state); - XCHECK_SZ(sz, nr, XFEATURE_PKRU, struct pkru_state); - XCHECK_SZ(sz, nr, XFEATURE_PASID, struct ia32_pasid_state); - XCHECK_SZ(sz, nr, XFEATURE_XTILE_CFG, struct xtile_cfg); - - /* The tile data size varies between implementations. */ - if (nr == XFEATURE_XTILE_DATA) - check_xtile_data_against_struct(sz); - - /* - * Make *SURE* to add any feature numbers in below if - * there are "holes" in the xsave state component - * numbers. - */ - if ((nr < XFEATURE_YMM) || - (nr >= XFEATURE_MAX) || - (nr == XFEATURE_PT_UNIMPLEMENTED_SO_FAR) || - ((nr >= XFEATURE_RSRVD_COMP_11) && (nr <= XFEATURE_RSRVD_COMP_16))) { + switch (nr) { + case XFEATURE_YMM: return XCHECK_SZ(sz, nr, struct ymmh_struct); + case XFEATURE_BNDREGS: return XCHECK_SZ(sz, nr, struct mpx_bndreg_state); + case XFEATURE_BNDCSR: return XCHECK_SZ(sz, nr, struct mpx_bndcsr_state); + case XFEATURE_OPMASK: return XCHECK_SZ(sz, nr, struct avx_512_opmask_state); + case XFEATURE_ZMM_Hi256: return XCHECK_SZ(sz, nr, struct avx_512_zmm_uppers_state); + case XFEATURE_Hi16_ZMM: return XCHECK_SZ(sz, nr, struct avx_512_hi16_state); + case XFEATURE_PKRU: return XCHECK_SZ(sz, nr, struct pkru_state); + case XFEATURE_PASID: return XCHECK_SZ(sz, nr, struct ia32_pasid_state); + case XFEATURE_XTILE_CFG: return XCHECK_SZ(sz, nr, struct xtile_cfg); + case XFEATURE_CET_USER: return XCHECK_SZ(sz, nr, struct cet_user_state); + case XFEATURE_XTILE_DATA: check_xtile_data_against_struct(sz); return true; + default: XSTATE_WARN_ON(1, "No structure for xstate: %d\n", nr); return false; } + return true; } From patchwork Mon Feb 27 22:29:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E76EAC64ED6 for ; Mon, 27 Feb 2023 22:31:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1687D6B007E; Mon, 27 Feb 2023 17:31:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0FCBC6B0080; Mon, 27 Feb 2023 17:31:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E35F16B0081; Mon, 27 Feb 2023 17:31:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D6A5A6B007E for ; Mon, 27 Feb 2023 17:31:35 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A6237160977 for ; Mon, 27 Feb 2023 22:31:35 +0000 (UTC) X-FDA: 80514519750.10.FEDF25F Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf28.hostedemail.com (Postfix) with ESMTP id C2B9CC000E for ; Mon, 27 Feb 2023 22:31:33 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=drISy8Ph; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537094; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=MvWt7ReyhF/wn7NboBw5a1sJLt/UxAkay4sKp6Ow5WM=; b=IOte67wezZdvn79i0jgALiJeepv/VCzXu4EBqfkddqdwiIAnoRLZ4YBgO2iTwRH56JCpQu LCgZAFlyt+Nk/bGDhkC4EVx13K/9zZHWlz6JGzAv9moCWYjlC2qoQT2LUoazFwsgydWbM1 myISWS9AB/Z6Dn5g4Fj7hk/nMI2GUzI= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=drISy8Ph; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537094; a=rsa-sha256; cv=none; b=4NsoeMMqQ4htDet3T31rzsBVwxr7JrulBHPaBPZBchbxDBSNJBLrVhyRZbUKJ/dmRA7/Yt 5GDrr6o+JTONsF4SB2QKzaGPTeFcaEbSw5VCyHGAo5Tses3c+bcgX3fjQquUnsFwpd9Gaq anUAELLYy6UbbjIIplvkHvg209lBCt4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537093; x=1709073093; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=D7r3sPYzuOicvyDyAiCcjJlcjrFZlOzlMPKscQsbvdA=; b=drISy8Phx9ojyjpgcEFsd/xyYBD6/+Thpa3Wfy2w258MtzF0/ZPPyBQw 7j7oswAS1oVGSDzDs0Nv1p56+vSga1alNvgB37n7JYUKnm6/6rasgCuW+ Kylhf01j8LJkfjM70F7s+ZKdp1KxYVQT4xwu3UxROC3wNHffmlQGAgSlW c8An4A79SpmvfzEatSIAfIauZDAvsX41svFP+jK6Q+Q+sTxSA/42sKrey qmWStOkIcNtJRm6aXK12b//cV7oCNPoSO/308bA7DST5AlO9f1tgHtGQq GpcqmzyA2gA1yWdzMOCKeKL3Ko5f85SJg4TLA5/mEMSbDYMqO279e4136 g==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657134" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657134" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:12 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024393" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024393" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:11 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v7 06/41] x86/fpu: Add helper for modifying xstate Date: Mon, 27 Feb 2023 14:29:22 -0800 Message-Id: <20230227222957.24501-7-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: C2B9CC000E X-Stat-Signature: 8wtuo3tdikjia3748i5w5garehty64ry X-Rspam-User: X-HE-Tag: 1677537093-810099 X-HE-Meta: U2FsdGVkX1+XOJYtrSIfDQf8mVX1FjhvydMkQC0RrrMSZExcyxWccO6FJYxep2fG8t6HYaaT7d+zzgei8OE5xzO32KkEsLjXnkGyg1no7vaX7d7zimb9Q4VJA7WfdSfG7F0NEyfvwnD27ap4HViSSA0w2bdOahNrRM7ZtCaqUj6hPK/TvO5RpSPkUynPfxPTo3lYscOZkL3Gs74Ww2Jg2ZEhQQXMz1WFPCOUDvC3+R1XwBkb6o38V4QLlDWZdtsUVS2RPuVwGUQjwL48SXSoxIJNuNkndtA3gQQtuMo5k4mxRA+wYp6H9IrDVhjcorBBvYj1V66oLfIkjtATvX5O6hqPqj8CyxMZNHrpuklsIMKegGyzREksjnvtcVNlXIkLBqPWBR2/WrbJv263zuTDpZfaHNBjc48XsGsSiLOUf9fVRFkx6doWkeRkDqaPw5O58w8TTvcFwlV+Wmozp+Qfsn4CdFYfv/6HqN1xL7WyEqILymQbVLvqMaAL01Gk+sbzH/GFxUhofeKb+q+8jA854hCUETYr3lo9fb2sPgOkrdiDljhvf9HFS4s9vFRXqy1yFZcM7sff3a4ljA9bOKsc3EB6JV7GbIwLZq5L3hGG9XfyxISCmZV7Bc1hmVtHcs/Yrh7DzHExjjirpgls3xiRzk4oQ6vDXUyct9T6LFVPM7s9IrISqILw05jukp9VhdN203nFRODwmWWrvA1QB+ysaUJO8xgSIVcr3ctJi8qS6sMvGG1Ui5flz3/NlSDmRu7ab1q28b2um2UhB2QBIp2AAzxTXZJk19ABvuNYv0IT6EK4nc2QbGCgBySc/4PBdEbD2wX5OrUIEAxV1oKoo/iBXZSNKyFvjzKXLaJo1kgRsnimTFZzz3DLKGGtt5m++C/rxrWyN95+qGmYPrIdBts7F4CFL+gERQ9ZJmdu8nJUzH3TAUsVzXdMhTp6SQdcntKLeogNyiwYboButfFlXde lwrBgyWA Eq8dbYktoovEOwIbYmiKp4qsDItbFSumkAiAe7IR4x9aSe8T4PG6h9WHICLEgS20hZnE1H/5TCR3RvLOhlqp9ymiYU/74WKHU63j/DaUIDy+iCNW5t+uXmfFMrFnOnPxpQVAfF4qqykzDVSz+y1XekSw9NY28PZBdL9bWDUCMol8x0nH3gnJWFtH43EVrjtP31+KKSmBHCEKQcAirBKfK6sGeFk6MxScc6vNHpClxd3iznSRv8Mb8dYe7b97tDevJ+dWDUBZuTS9IJGdb/URfgg1P0cTlyw91yWN3VPM2nzlILC+8+wR4f1Os34VTJD/tZeNSrHbtsvs4p2ITTyOXp9DBtTkzabRS2lDv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Just like user xfeatures, supervisor xfeatures can be active in the registers or present in the task FPU buffer. If the registers are active, the registers can be modified directly. If the registers are not active, the modification must be performed on the task FPU buffer. When the state is not active, the kernel could perform modifications directly to the buffer. But in order for it to do that, it needs to know where in the buffer the specific state it wants to modify is located. Doing this is not robust against optimizations that compact the FPU buffer, as each access would require computing where in the buffer it is. The easiest way to modify supervisor xfeature data is to force restore the registers and write directly to the MSRs. Often times this is just fine anyway as the registers need to be restored before returning to userspace. Do this for now, leaving buffer writing optimizations for the future. Add a new function fpregs_lock_and_load() that can simultaneously call fpregs_lock() and do this restore. Also perform some extra sanity checks in this function since this will be used in non-fpu focused code. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Suggested-by: Thomas Gleixner Signed-off-by: Rick Edgecombe --- v6: - Drop "but appear to work" (Boris) v5: - Fix spelling error (Boris) - Don't export fpregs_lock_and_load() (Boris) v3: - Rename to fpregs_lock_and_load() to match the unlocking fpregs_unlock(). (Kees) - Elaborate in comment about helper. (Dave) v2: - Drop optimization of writing directly the buffer, and change API accordingly. - fpregs_lock_and_load() suggested by tglx - Some commit log verbiage from dhansen --- arch/x86/include/asm/fpu/api.h | 9 +++++++++ arch/x86/kernel/fpu/core.c | 18 ++++++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/arch/x86/include/asm/fpu/api.h b/arch/x86/include/asm/fpu/api.h index 503a577814b2..aadc6893dcaa 100644 --- a/arch/x86/include/asm/fpu/api.h +++ b/arch/x86/include/asm/fpu/api.h @@ -82,6 +82,15 @@ static inline void fpregs_unlock(void) preempt_enable(); } +/* + * FPU state gets lazily restored before returning to userspace. So when in the + * kernel, the valid FPU state may be kept in the buffer. This function will force + * restore all the fpu state to the registers early if needed, and lock them from + * being automatically saved/restored. Then FPU state can be modified safely in the + * registers, before unlocking with fpregs_unlock(). + */ +void fpregs_lock_and_load(void); + #ifdef CONFIG_X86_DEBUG_FPU extern void fpregs_assert_state_consistent(void); #else diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index caf33486dc5e..f851558b673f 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -753,6 +753,24 @@ void switch_fpu_return(void) } EXPORT_SYMBOL_GPL(switch_fpu_return); +void fpregs_lock_and_load(void) +{ + /* + * fpregs_lock() only disables preemption (mostly). So modifying state + * in an interrupt could screw up some in progress fpregs operation. + * Warn about it. + */ + WARN_ON_ONCE(!irq_fpu_usable()); + WARN_ON_ONCE(current->flags & PF_KTHREAD); + + fpregs_lock(); + + fpregs_assert_state_consistent(); + + if (test_thread_flag(TIF_NEED_FPU_LOAD)) + fpregs_restore_userregs(); +} + #ifdef CONFIG_X86_DEBUG_FPU /* * If current FPU state according to its tracking (loaded FPU context on this From patchwork Mon Feb 27 22:29:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154238 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D0BCC64ED8 for ; Mon, 27 Feb 2023 22:31:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B05706B0080; Mon, 27 Feb 2023 17:31:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AB6686B0081; Mon, 27 Feb 2023 17:31:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 86A8A6B0082; Mon, 27 Feb 2023 17:31:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7918D6B0080 for ; Mon, 27 Feb 2023 17:31:36 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3D1C4AAEAF for ; Mon, 27 Feb 2023 22:31:36 +0000 (UTC) X-FDA: 80514519792.02.9D289A0 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf13.hostedemail.com (Postfix) with ESMTP id 478982000A for ; Mon, 27 Feb 2023 22:31:34 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Xs7f2G5A; spf=pass (imf13.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537094; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=tsJuTqRpMXZtpcwHwPCHx0KxUqD4hID8NTn5X+g1d6E=; b=piFQBjV8vvk4pE64dTbMZnOJ8em3W8fXEOiDlDdz7opHijQXunmMiqPyNJUdUau9rCtE62 wpxxX62838CSHjeJNpIwDtNw9Ha0/79DH3Hz0UxXg+CVOIVh5IIz1mB+J8aPgCWEyo0Oh7 qLFpANqDdopRTVrZE4aGloTBVaAGMhE= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Xs7f2G5A; spf=pass (imf13.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537094; a=rsa-sha256; cv=none; b=OXMkgMNOdIJC5NoFPaifebsxdhjZOmYES50uAg23c7j6sMhwoFC63pZuqlJlSL4zBdBJZL /15fNNYRBnsV15/VJAKpLeH/EewqqFIfFvbY97btSAOUEjFDiAiSPI5tMD+D4D/qZu4mMX ukmC2L/oudAECExSqUa2qJCE8GRZu2I= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537094; x=1709073094; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=2jjnDaU5BZRKVaf5H9IX0n9VX/dF6zxdHp6xtU3HToE=; b=Xs7f2G5AkvO/Vv4tQ2/XhYtS7IjVNxEhADiaDTBVSzlo8LKjsOfNYQgP a4nx0xpFrz9FCl5qSufcjE5zR8R2+/kjP9LNPo98KaSBEF4nY2HcHhdgh NljEXR/aqc090QEem4vIAiA1ZLcSizyZL3oEc8g3Ff7NPtniJsi//1q9j NoGrqAbzjv/vgXWghdi2I8jrDNXNwi+qj0qV+/yM06JiWrch+BHe0Qhaj minqhjn39W9ZwqZxckv1pIoQhdUDV31u+BjcZWED4KCEetuX8+RljGKeA u0fzRK3jobg01iKTQq1FUokzxfAP5DwK1I1AI/Mlpwx46ykQaaMEbSqne w==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657157" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657157" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:12 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024403" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024403" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:12 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v7 07/41] x86: Move control protection handler to separate file Date: Mon, 27 Feb 2023 14:29:23 -0800 Message-Id: <20230227222957.24501-8-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 478982000A X-Stat-Signature: 48dyd8jy89e8wq7wphon93h7x1mxjxgo X-Rspam-User: X-HE-Tag: 1677537094-732176 X-HE-Meta: U2FsdGVkX1+6g1YiStY4NNKiAZdNqcBQIGvW/NX7pL+w2alOJY0ftGArsvujhaPYCLS/xVuNtS40j741BD7hd7U/6VhsdLd0bqZ6qVkSmazzpaE/Dn29dzP6nGKdYzCzeD3sCog/euWlewp26C11DXFJ1Y9zZjKjEgyxqGo5givIMWisa5yisEt7z+cDJuRhw2IdFpzINV8kRhRL/uqNC2mtCDU5WQy2aBqtV2BZpOLC6GSCL9mNkPm85Qz3DNyb1aFoEyuDei0rCkZXKOvrJw/4srt/TsD3/orQC1O9FFRY3A+rxyobx3xjo98xNim0m3r9F3i0knP7IVFx3aj3CZthslCouJSsUXzVA1JVY9TUElHQ9q5QlJlU1aJzblreMHaEuXRIYgPv1IXQRdND1BmbgvXa5cebWuO6ConjArnl7SnhIjSbMMX3+INewAkPbYq5REZgE9VcBBhM3oSaAI7Ao1NdSlMAAqeM3F4kktcBbwMpQBObnyrux7NaKzqEIPgDz48nUG1ozxTczX09A2m0fAL2XdXc7KOKR7/ah+ZBS+dhRNl3nEoPXqyGF+5jSOZV293ocYAfH4HOxA6jTFUlJBpSXKC9CDCvka0teYajnFgfjczzbxEXQKKMmHohrFOyc71+rsejfB5lSyGunrRwqN4RBm2DIO2olAbNUn9MaOEXxIDXUNANtS15Ad/ixFg7/0PjZjzJejtaw36QXZ4x9jDF7ycFMjWQafpR0QiBruiN1VgItt+gUL+UlGWDxeOOcpTw07XXvs89AiUPhhy95kfMG1pNrE2jbzT/Dk906XHlKK3ttgkPjiXrTp4FwkFAHL2TkZPMJw4OUcdbnN+gusXM+u8tId749yn8V+fcSDnjJat1xhJSYjycQ7ZDyI7FqCShGo7IcCwa2LTzw80yZZHZMCa7hDGFkp+Oon4HLEhjzRygAiFvamGzxc/e5UMb6c7nQVGPuEPTnCV oxbWSa0E 11NIoq57Q/qrMxRcqYD1jTZqRQPy7JG0Zr6hdQC/cshPdpnK0dgv6rbZ54ZIxOyVfgFdYzExwY1MFauvwcPW/NcEmwR8SYrrcH+R3QshX1y0FORq397cMryQE65fD4EddvDbUbl+9ApUAUuM8i5fgbrEWk7Rx45vNcFjHz4xLNME7hVLjx6nZ3KwY+rYUiHj2Nk0wFAr0onPMxx1XEkc0PD5C6ubI+HZHCOCvvYIo8g/DJzCB98kQPchf3i0uwnH+6ziFfNofzO5Gfh9WV0RBO2avv/fAy6R7BfmgI6GORKBYRoq0k0b6rY/4/LfLWHufsA3h3ciE232X8YLSUCZfQ53n5L2Gzyco9d+A X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Today the control protection handler is defined in traps.c and used only for the kernel IBT feature. To reduce ifdeffery, move it to it's own file. In future patches, functionality will be added to make this handler also handle user shadow stack faults. So name the file cet.c. No functional change. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Rick Edgecombe --- v6: - Split move to cet.c and shadow stack enhancements to fault handler to separate files. (Kees) --- arch/x86/kernel/Makefile | 2 ++ arch/x86/kernel/cet.c | 76 ++++++++++++++++++++++++++++++++++++++++ arch/x86/kernel/traps.c | 75 --------------------------------------- 3 files changed, 78 insertions(+), 75 deletions(-) create mode 100644 arch/x86/kernel/cet.c diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index dd61752f4c96..92446f1dedd7 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -144,6 +144,8 @@ obj-$(CONFIG_CFI_CLANG) += cfi.o obj-$(CONFIG_CALL_THUNKS) += callthunks.o +obj-$(CONFIG_X86_CET) += cet.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y) diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c new file mode 100644 index 000000000000..7ad22b705b64 --- /dev/null +++ b/arch/x86/kernel/cet.c @@ -0,0 +1,76 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include + +static __ro_after_init bool ibt_fatal = true; + +extern void ibt_selftest_ip(void); /* code label defined in asm below */ + +enum cp_error_code { + CP_EC = (1 << 15) - 1, + + CP_RET = 1, + CP_IRET = 2, + CP_ENDBR = 3, + CP_RSTRORSSP = 4, + CP_SETSSBSY = 5, + + CP_ENCL = 1 << 15, +}; + +DEFINE_IDTENTRY_ERRORCODE(exc_control_protection) +{ + if (!cpu_feature_enabled(X86_FEATURE_IBT)) { + pr_err("Unexpected #CP\n"); + BUG(); + } + + if (WARN_ON_ONCE(user_mode(regs) || (error_code & CP_EC) != CP_ENDBR)) + return; + + if (unlikely(regs->ip == (unsigned long)&ibt_selftest_ip)) { + regs->ax = 0; + return; + } + + pr_err("Missing ENDBR: %pS\n", (void *)instruction_pointer(regs)); + if (!ibt_fatal) { + printk(KERN_DEFAULT CUT_HERE); + __warn(__FILE__, __LINE__, (void *)regs->ip, TAINT_WARN, regs, NULL); + return; + } + BUG(); +} + +/* Must be noinline to ensure uniqueness of ibt_selftest_ip. */ +noinline bool ibt_selftest(void) +{ + unsigned long ret; + + asm (" lea ibt_selftest_ip(%%rip), %%rax\n\t" + ANNOTATE_RETPOLINE_SAFE + " jmp *%%rax\n\t" + "ibt_selftest_ip:\n\t" + UNWIND_HINT_FUNC + ANNOTATE_NOENDBR + " nop\n\t" + + : "=a" (ret) : : "memory"); + + return !ret; +} + +static int __init ibt_setup(char *str) +{ + if (!strcmp(str, "off")) + setup_clear_cpu_cap(X86_FEATURE_IBT); + + if (!strcmp(str, "warn")) + ibt_fatal = false; + + return 1; +} + +__setup("ibt=", ibt_setup); diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index d317dc3d06a3..cc223e60aba2 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -213,81 +213,6 @@ DEFINE_IDTENTRY(exc_overflow) do_error_trap(regs, 0, "overflow", X86_TRAP_OF, SIGSEGV, 0, NULL); } -#ifdef CONFIG_X86_KERNEL_IBT - -static __ro_after_init bool ibt_fatal = true; - -extern void ibt_selftest_ip(void); /* code label defined in asm below */ - -enum cp_error_code { - CP_EC = (1 << 15) - 1, - - CP_RET = 1, - CP_IRET = 2, - CP_ENDBR = 3, - CP_RSTRORSSP = 4, - CP_SETSSBSY = 5, - - CP_ENCL = 1 << 15, -}; - -DEFINE_IDTENTRY_ERRORCODE(exc_control_protection) -{ - if (!cpu_feature_enabled(X86_FEATURE_IBT)) { - pr_err("Unexpected #CP\n"); - BUG(); - } - - if (WARN_ON_ONCE(user_mode(regs) || (error_code & CP_EC) != CP_ENDBR)) - return; - - if (unlikely(regs->ip == (unsigned long)&ibt_selftest_ip)) { - regs->ax = 0; - return; - } - - pr_err("Missing ENDBR: %pS\n", (void *)instruction_pointer(regs)); - if (!ibt_fatal) { - printk(KERN_DEFAULT CUT_HERE); - __warn(__FILE__, __LINE__, (void *)regs->ip, TAINT_WARN, regs, NULL); - return; - } - BUG(); -} - -/* Must be noinline to ensure uniqueness of ibt_selftest_ip. */ -noinline bool ibt_selftest(void) -{ - unsigned long ret; - - asm (" lea ibt_selftest_ip(%%rip), %%rax\n\t" - ANNOTATE_RETPOLINE_SAFE - " jmp *%%rax\n\t" - "ibt_selftest_ip:\n\t" - UNWIND_HINT_FUNC - ANNOTATE_NOENDBR - " nop\n\t" - - : "=a" (ret) : : "memory"); - - return !ret; -} - -static int __init ibt_setup(char *str) -{ - if (!strcmp(str, "off")) - setup_clear_cpu_cap(X86_FEATURE_IBT); - - if (!strcmp(str, "warn")) - ibt_fatal = false; - - return 1; -} - -__setup("ibt=", ibt_setup); - -#endif /* CONFIG_X86_KERNEL_IBT */ - #ifdef CONFIG_X86_F00F_BUG void handle_invalid_op(struct pt_regs *regs) #else From patchwork Mon Feb 27 22:29:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154239 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B04F8C64ED9 for ; Mon, 27 Feb 2023 22:31:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 47A2B6B0082; Mon, 27 Feb 2023 17:31:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3B7A96B0087; Mon, 27 Feb 2023 17:31:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E884D6B0083; Mon, 27 Feb 2023 17:31:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id CEDAD6B0081 for ; Mon, 27 Feb 2023 17:31:37 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A05CD1C6282 for ; Mon, 27 Feb 2023 22:31:37 +0000 (UTC) X-FDA: 80514519834.19.1CAA7D7 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf07.hostedemail.com (Postfix) with ESMTP id 9F71D40015 for ; Mon, 27 Feb 2023 22:31:35 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=XByo1FDF; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537095; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=yakbtHkioUXAlihHk1t0g8Fq+fju/GbWCuUw9K+uZPI=; b=NS6BKhFDS6JRUPl+IbX6ztH7d5mwNZp4bS1MCGFDq8Bv/5+93/ezsoKyfHNEQ2DYxMo+TF MCO8HnWl1NqhCx3UCK2fzsNik+CfU8KWDItIUDiE4KWe0V4uu3+A8KXSXZWs5WrSzuzj/U ne44FsKIpWHBAcKyS5/sdr7GBoCudzI= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=XByo1FDF; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537095; a=rsa-sha256; cv=none; b=M9vqknTcjnE8ZXzVi52uSaYN9vS7TxyQgDw1FUdKvG8fIJShmLWiRdmLt5G3U6qJla2abJ ZVri++2Smm3ffFZJt7XVC1ladj06lIGuzUwBe84qcxgHFSlo58eyVGuhmEV8526QZiBZGL E8pJP8YrubxT25T9CQO3qHF90vgXQBM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537095; x=1709073095; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=T4rGtw5DEsV2/UtZGHsMITMIzMqQGlL724CmcN7T2S8=; b=XByo1FDFTcM8f6DLxRnOOeK7y2wRrO7vn0LIhmS3njrjS4jG1nunqYFp 09VgB9MhrjeS5kXvxSMvy4HlmfBOOWWQjf6jzaSfW1nfYYI/022axAM16 M6nf6WvP8poiI7UFr+7sboI8nZqa08xFmWTMez8XYR1ns0ihgm6A4i+sB SkVjT1JqbBf/C44bVyI5tukYStoBgIOddTn6icMQd0MgVGavow94YKyzz zXZHaxZdfdq+zszFm73AVpzoSTr6pQyOo7PufOzooOVUdPpYpsjaabant 4HkwaN7oC1KCe19sCNasnhHEGSUzjbv8LganNfiA4zu894QjGAHuqrTwV g==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657180" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657180" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:13 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024412" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024412" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:12 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu , Michael Kerrisk Subject: [PATCH v7 08/41] x86/shstk: Add user control-protection fault handler Date: Mon, 27 Feb 2023 14:29:24 -0800 Message-Id: <20230227222957.24501-9-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 9F71D40015 X-Stat-Signature: zp35pjbkq8oxw6do6nwqfwpfrh7stgk3 X-Rspam-User: X-HE-Tag: 1677537095-918548 X-HE-Meta: U2FsdGVkX1/Eb5Z2nbqJtq0kGgNdvCH/Ve5kYJkpxoNM6KUpFf99hTzqTlfR0tMNr3JFyXZ5OHcVbZSAxqyaqjIDlrkgPNsmH0de86L4MAPqegA+DDIsRsD08WBDhk2txc0asoyDfuIYY60V6R/Z1QT9ERS4irnutSsgo46ECFMC//d4fQhVXBiiPHCSvJ+2nC/DXUppcgzwfr58CL2VA4vcC31TnKK/4liRtEmoWF+WfyQkqMXT+tbQCpj2gMI23yOL+rSQxYVEK6zdKypOkzUk1ej+2KUPhPcJCgPTk7pdKANBsfSWXglEaxYkKgoWTFwVxhJPOHz1Q+KOaL/0E/v7CiU3BxvpeyEtbdvxQZFM0LqQAloly3dy9eAuY/Os3T9W1nGae3MBO8iSkiPAdZ7+3u286UWnP2aHbyt0ud5dF2CRHDysfy/SxPTobgkvsN5gF1Obp4rMcUVqzYZsE9bjPHY4KLIAEUcBY87reedtL4PbuFvkwIcJCwyoctE8hqrOYS5PCalNwoe9LGhYzfPuev6Xw0t5t96jxsSp3f2CrGvmoVjju9SxAUouQiRVcOWKtcnqEa3ZV/Rz84phvhww+ru6sVAfBFPWSeB5YZQJbNLkMZMW7ryalu7inYdkJUS1vHRYu+1y5yTS+u/f7gvNL02xo/YwUbfaWUuGvViTekksjLPH+7pFQGpbTX8uDhjWTeBFjMeJn4r3aJ2ZlYftGxYE1XRekrEhUbWgwzGI2LDNCW2j1a8dpcU45SKAtWpSVjn3Vi4FOvXQSwTYQmlyRjcQUZ+EAvMOqk4lAFr4ymUG8OtL/Qe2b+3f4fMvYtddFTvFkTNNvvKgmZcdzFcmFHC6JhJs4Ebw0jkWRgCIv95AJUlIDEZtd04h5jK04VTX011KAKTNhi7vGjGEhNLX9K+yzhRpvor2nkzU1Fy+cnerZ7lTOmeX10TBL+GsNL+g9+ebFkfwMSvuW89 4fCTBZfi AwVq76nbMutQJmadwTyPjSC94FQaE9KZSWMiTytpy0WddE5XR6cZbdCM2VoHfXbJVQPPjtfyLnLS7dgLAiqmqZdlwhEkxsGC3WjpAdYgQmQaIe40vo9pxfpTjTnmEvt4erm7An5/sY1/F6e4VOYyS+8Bu4woDmeCtBOINauwhF3Rafso6uxYgcZtNZG3TnQsjxb4VTF/VH0INAC1Ka//A/Cd6jiDWpGkqBieHpwg0r8hv45Y4aEWw8ua+34rfp8HsQUZ3g8jIX43SN7FDpQODcOmz/KCDIe+czVt0q8ynAzKJFXjvJ/5hCbeeD1d5NvNhMsqHzhrgwVz201k0aoOjGe2ZgkWxLioTGDttaeIKhGtWi5j9LzQlA5XW2A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu A control-protection fault is triggered when a control-flow transfer attempt violates Shadow Stack or Indirect Branch Tracking constraints. For example, the return address for a RET instruction differs from the copy on the shadow stack. There already exists a control-protection fault handler for handling kernel IBT faults. Refactor this fault handler into separate user and kernel handlers, like the page fault handler. Add a control-protection handler for usermode. To avoid ifdeffery, put them both in a new file cet.c, which is compiled in the case of either of the two CET features supported in the kernel: kernel IBT or user mode shadow stack. Move some static inline functions from traps.c into a header so they can be used in cet.c. Opportunistically fix a comment in the kernel IBT part of the fault handler that is on the end of the line instead of preceding it. Keep the same behavior for the kernel side of the fault handler, except for converting a BUG to a WARN in the case of a #CP happening when the feature is missing. This unifies the behavior with the new shadow stack code, and also prevents the kernel from crashing under this situation which is potentially recoverable. The control-protection fault handler works in a similar way as the general protection fault handler. It provides the si_code SEGV_CPERR to the signal handler. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook Cc: Michael Kerrisk --- v7: - Adjust alignment of WARN statement v6: - Split into separate patches (Kees) - Change to "x86/shstk" in commit log (Boris) v5: - Move to separate file to avoid ifdeffery (Boris) - Improvements to commit log (Boris) - Rename control_protection_err (Boris) - Move comment from end of line in IBT fault handler (Boris) v3: - Shorten user/kernel #CP handler function names (peterz) - Restore CP_ENDBR check to kernel handler (peterz) - Utilize CONFIG_X86_CET (Kees) - Unify "unexpected" warnings (Andrew Cooper) - Use 2d array for error code chars (Andrew Cooper) - Add comment about why to read SSP MSR before enabling interrupts v2: - Integrate with kernel IBT fault handler - Update printed messages. (Dave) - Remove array_index_nospec() usage. (Dave) - Remove IBT messages. (Dave) - Add enclave error code bit processing it case it can get triggered somehow. - Add extra "unknown" in control_protection_err. --- arch/arm/kernel/signal.c | 2 +- arch/arm64/kernel/signal.c | 2 +- arch/arm64/kernel/signal32.c | 2 +- arch/sparc/kernel/signal32.c | 2 +- arch/sparc/kernel/signal_64.c | 2 +- arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/idtentry.h | 2 +- arch/x86/include/asm/traps.h | 12 +++ arch/x86/kernel/cet.c | 94 +++++++++++++++++++++--- arch/x86/kernel/idt.c | 2 +- arch/x86/kernel/signal_32.c | 2 +- arch/x86/kernel/signal_64.c | 2 +- arch/x86/kernel/traps.c | 12 --- arch/x86/xen/enlighten_pv.c | 2 +- arch/x86/xen/xen-asm.S | 2 +- include/uapi/asm-generic/siginfo.h | 3 +- 16 files changed, 117 insertions(+), 34 deletions(-) diff --git a/arch/arm/kernel/signal.c b/arch/arm/kernel/signal.c index e07f359254c3..9a3c9de5ac5e 100644 --- a/arch/arm/kernel/signal.c +++ b/arch/arm/kernel/signal.c @@ -681,7 +681,7 @@ asmlinkage void do_rseq_syscall(struct pt_regs *regs) */ static_assert(NSIGILL == 11); static_assert(NSIGFPE == 15); -static_assert(NSIGSEGV == 9); +static_assert(NSIGSEGV == 10); static_assert(NSIGBUS == 5); static_assert(NSIGTRAP == 6); static_assert(NSIGCHLD == 6); diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c index 06a02707f488..19b6b292892c 100644 --- a/arch/arm64/kernel/signal.c +++ b/arch/arm64/kernel/signal.c @@ -1341,7 +1341,7 @@ void __init minsigstksz_setup(void) */ static_assert(NSIGILL == 11); static_assert(NSIGFPE == 15); -static_assert(NSIGSEGV == 9); +static_assert(NSIGSEGV == 10); static_assert(NSIGBUS == 5); static_assert(NSIGTRAP == 6); static_assert(NSIGCHLD == 6); diff --git a/arch/arm64/kernel/signal32.c b/arch/arm64/kernel/signal32.c index 4700f8522d27..bbd542704730 100644 --- a/arch/arm64/kernel/signal32.c +++ b/arch/arm64/kernel/signal32.c @@ -460,7 +460,7 @@ void compat_setup_restart_syscall(struct pt_regs *regs) */ static_assert(NSIGILL == 11); static_assert(NSIGFPE == 15); -static_assert(NSIGSEGV == 9); +static_assert(NSIGSEGV == 10); static_assert(NSIGBUS == 5); static_assert(NSIGTRAP == 6); static_assert(NSIGCHLD == 6); diff --git a/arch/sparc/kernel/signal32.c b/arch/sparc/kernel/signal32.c index dad38960d1a8..82da8a2d769d 100644 --- a/arch/sparc/kernel/signal32.c +++ b/arch/sparc/kernel/signal32.c @@ -751,7 +751,7 @@ asmlinkage int do_sys32_sigstack(u32 u_ssptr, u32 u_ossptr, unsigned long sp) */ static_assert(NSIGILL == 11); static_assert(NSIGFPE == 15); -static_assert(NSIGSEGV == 9); +static_assert(NSIGSEGV == 10); static_assert(NSIGBUS == 5); static_assert(NSIGTRAP == 6); static_assert(NSIGCHLD == 6); diff --git a/arch/sparc/kernel/signal_64.c b/arch/sparc/kernel/signal_64.c index 570e43e6fda5..b4e410976e0d 100644 --- a/arch/sparc/kernel/signal_64.c +++ b/arch/sparc/kernel/signal_64.c @@ -562,7 +562,7 @@ void do_notify_resume(struct pt_regs *regs, unsigned long orig_i0, unsigned long */ static_assert(NSIGILL == 11); static_assert(NSIGFPE == 15); -static_assert(NSIGSEGV == 9); +static_assert(NSIGSEGV == 10); static_assert(NSIGBUS == 5); static_assert(NSIGTRAP == 6); static_assert(NSIGCHLD == 6); diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index 505f78ddca82..652e366b68a0 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -105,6 +105,12 @@ #define DISABLE_USER_SHSTK (1 << (X86_FEATURE_USER_SHSTK & 31)) #endif +#ifdef CONFIG_X86_KERNEL_IBT +#define DISABLE_IBT 0 +#else +#define DISABLE_IBT (1 << (X86_FEATURE_IBT & 31)) +#endif + /* * Make sure to add features to the correct mask */ @@ -128,7 +134,7 @@ #define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ DISABLE_ENQCMD) #define DISABLED_MASK17 0 -#define DISABLED_MASK18 0 +#define DISABLED_MASK18 (DISABLE_IBT) #define DISABLED_MASK19 0 #define DISABLED_MASK20 0 #define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 21) diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h index 72184b0b2219..69e26f48d027 100644 --- a/arch/x86/include/asm/idtentry.h +++ b/arch/x86/include/asm/idtentry.h @@ -618,7 +618,7 @@ DECLARE_IDTENTRY_RAW_ERRORCODE(X86_TRAP_DF, xenpv_exc_double_fault); #endif /* #CP */ -#ifdef CONFIG_X86_KERNEL_IBT +#ifdef CONFIG_X86_CET DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_CP, exc_control_protection); #endif diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h index 47ecfff2c83d..75e0dabf0c45 100644 --- a/arch/x86/include/asm/traps.h +++ b/arch/x86/include/asm/traps.h @@ -47,4 +47,16 @@ void __noreturn handle_stack_overflow(struct pt_regs *regs, struct stack_info *info); #endif +static inline void cond_local_irq_enable(struct pt_regs *regs) +{ + if (regs->flags & X86_EFLAGS_IF) + local_irq_enable(); +} + +static inline void cond_local_irq_disable(struct pt_regs *regs) +{ + if (regs->flags & X86_EFLAGS_IF) + local_irq_disable(); +} + #endif /* _ASM_X86_TRAPS_H */ diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index 7ad22b705b64..cc10d8be9d74 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -4,10 +4,6 @@ #include #include -static __ro_after_init bool ibt_fatal = true; - -extern void ibt_selftest_ip(void); /* code label defined in asm below */ - enum cp_error_code { CP_EC = (1 << 15) - 1, @@ -20,15 +16,80 @@ enum cp_error_code { CP_ENCL = 1 << 15, }; -DEFINE_IDTENTRY_ERRORCODE(exc_control_protection) +static const char cp_err[][10] = { + [0] = "unknown", + [1] = "near ret", + [2] = "far/iret", + [3] = "endbranch", + [4] = "rstorssp", + [5] = "setssbsy", +}; + +static const char *cp_err_string(unsigned long error_code) +{ + unsigned int cpec = error_code & CP_EC; + + if (cpec >= ARRAY_SIZE(cp_err)) + cpec = 0; + return cp_err[cpec]; +} + +static void do_unexpected_cp(struct pt_regs *regs, unsigned long error_code) +{ + WARN_ONCE(1, "Unexpected %s #CP, error_code: %s\n", + user_mode(regs) ? "user mode" : "kernel mode", + cp_err_string(error_code)); +} + +static DEFINE_RATELIMIT_STATE(cpf_rate, DEFAULT_RATELIMIT_INTERVAL, + DEFAULT_RATELIMIT_BURST); + +static void do_user_cp_fault(struct pt_regs *regs, unsigned long error_code) { - if (!cpu_feature_enabled(X86_FEATURE_IBT)) { - pr_err("Unexpected #CP\n"); - BUG(); + struct task_struct *tsk; + unsigned long ssp; + + /* + * An exception was just taken from userspace. Since interrupts are disabled + * here, no scheduling should have messed with the registers yet and they + * will be whatever is live in userspace. So read the SSP before enabling + * interrupts so locking the fpregs to do it later is not required. + */ + rdmsrl(MSR_IA32_PL3_SSP, ssp); + + cond_local_irq_enable(regs); + + tsk = current; + tsk->thread.error_code = error_code; + tsk->thread.trap_nr = X86_TRAP_CP; + + /* Ratelimit to prevent log spamming. */ + if (show_unhandled_signals && unhandled_signal(tsk, SIGSEGV) && + __ratelimit(&cpf_rate)) { + pr_emerg("%s[%d] control protection ip:%lx sp:%lx ssp:%lx error:%lx(%s)%s", + tsk->comm, task_pid_nr(tsk), + regs->ip, regs->sp, ssp, error_code, + cp_err_string(error_code), + error_code & CP_ENCL ? " in enclave" : ""); + print_vma_addr(KERN_CONT " in ", regs->ip); + pr_cont("\n"); } - if (WARN_ON_ONCE(user_mode(regs) || (error_code & CP_EC) != CP_ENDBR)) + force_sig_fault(SIGSEGV, SEGV_CPERR, (void __user *)0); + cond_local_irq_disable(regs); +} + +static __ro_after_init bool ibt_fatal = true; + +/* code label defined in asm below */ +extern void ibt_selftest_ip(void); + +static void do_kernel_cp_fault(struct pt_regs *regs, unsigned long error_code) +{ + if ((error_code & CP_EC) != CP_ENDBR) { + do_unexpected_cp(regs, error_code); return; + } if (unlikely(regs->ip == (unsigned long)&ibt_selftest_ip)) { regs->ax = 0; @@ -74,3 +135,18 @@ static int __init ibt_setup(char *str) } __setup("ibt=", ibt_setup); + +DEFINE_IDTENTRY_ERRORCODE(exc_control_protection) +{ + if (user_mode(regs)) { + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + do_user_cp_fault(regs, error_code); + else + do_unexpected_cp(regs, error_code); + } else { + if (cpu_feature_enabled(X86_FEATURE_IBT)) + do_kernel_cp_fault(regs, error_code); + else + do_unexpected_cp(regs, error_code); + } +} diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c index a58c6bc1cd68..5074b8420359 100644 --- a/arch/x86/kernel/idt.c +++ b/arch/x86/kernel/idt.c @@ -107,7 +107,7 @@ static const __initconst struct idt_data def_idts[] = { ISTG(X86_TRAP_MC, asm_exc_machine_check, IST_INDEX_MCE), #endif -#ifdef CONFIG_X86_KERNEL_IBT +#ifdef CONFIG_X86_CET INTG(X86_TRAP_CP, asm_exc_control_protection), #endif diff --git a/arch/x86/kernel/signal_32.c b/arch/x86/kernel/signal_32.c index 9027fc088f97..c12624bc82a3 100644 --- a/arch/x86/kernel/signal_32.c +++ b/arch/x86/kernel/signal_32.c @@ -402,7 +402,7 @@ int ia32_setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs) */ static_assert(NSIGILL == 11); static_assert(NSIGFPE == 15); -static_assert(NSIGSEGV == 9); +static_assert(NSIGSEGV == 10); static_assert(NSIGBUS == 5); static_assert(NSIGTRAP == 6); static_assert(NSIGCHLD == 6); diff --git a/arch/x86/kernel/signal_64.c b/arch/x86/kernel/signal_64.c index 13a1e6083837..0e808c72bf7e 100644 --- a/arch/x86/kernel/signal_64.c +++ b/arch/x86/kernel/signal_64.c @@ -403,7 +403,7 @@ void sigaction_compat_abi(struct k_sigaction *act, struct k_sigaction *oact) */ static_assert(NSIGILL == 11); static_assert(NSIGFPE == 15); -static_assert(NSIGSEGV == 9); +static_assert(NSIGSEGV == 10); static_assert(NSIGBUS == 5); static_assert(NSIGTRAP == 6); static_assert(NSIGCHLD == 6); diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index cc223e60aba2..18fb9d620824 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -77,18 +77,6 @@ DECLARE_BITMAP(system_vectors, NR_VECTORS); -static inline void cond_local_irq_enable(struct pt_regs *regs) -{ - if (regs->flags & X86_EFLAGS_IF) - local_irq_enable(); -} - -static inline void cond_local_irq_disable(struct pt_regs *regs) -{ - if (regs->flags & X86_EFLAGS_IF) - local_irq_disable(); -} - __always_inline int is_valid_bugaddr(unsigned long addr) { if (addr < TASK_SIZE_MAX) diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c index bb59cc6ddb2d..9c29cd5393cc 100644 --- a/arch/x86/xen/enlighten_pv.c +++ b/arch/x86/xen/enlighten_pv.c @@ -640,7 +640,7 @@ static struct trap_array_entry trap_array[] = { TRAP_ENTRY(exc_coprocessor_error, false ), TRAP_ENTRY(exc_alignment_check, false ), TRAP_ENTRY(exc_simd_coprocessor_error, false ), -#ifdef CONFIG_X86_KERNEL_IBT +#ifdef CONFIG_X86_CET TRAP_ENTRY(exc_control_protection, false ), #endif }; diff --git a/arch/x86/xen/xen-asm.S b/arch/x86/xen/xen-asm.S index 4a184f6e4e4d..7cdcb4ce6976 100644 --- a/arch/x86/xen/xen-asm.S +++ b/arch/x86/xen/xen-asm.S @@ -148,7 +148,7 @@ xen_pv_trap asm_exc_page_fault xen_pv_trap asm_exc_spurious_interrupt_bug xen_pv_trap asm_exc_coprocessor_error xen_pv_trap asm_exc_alignment_check -#ifdef CONFIG_X86_KERNEL_IBT +#ifdef CONFIG_X86_CET xen_pv_trap asm_exc_control_protection #endif #ifdef CONFIG_X86_MCE diff --git a/include/uapi/asm-generic/siginfo.h b/include/uapi/asm-generic/siginfo.h index ffbe4cec9f32..0f52d0ac47c5 100644 --- a/include/uapi/asm-generic/siginfo.h +++ b/include/uapi/asm-generic/siginfo.h @@ -242,7 +242,8 @@ typedef struct siginfo { #define SEGV_ADIPERR 7 /* Precise MCD exception */ #define SEGV_MTEAERR 8 /* Asynchronous ARM MTE error */ #define SEGV_MTESERR 9 /* Synchronous ARM MTE exception */ -#define NSIGSEGV 9 +#define SEGV_CPERR 10 /* Control protection fault */ +#define NSIGSEGV 10 /* * SIGBUS si_codes From patchwork Mon Feb 27 22:29:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154240 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71AB1C7EE2E for ; Mon, 27 Feb 2023 22:31:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E20B6B0081; Mon, 27 Feb 2023 17:31:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6E64C6B0085; Mon, 27 Feb 2023 17:31:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 474C26B0081; Mon, 27 Feb 2023 17:31:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0337F6B0081 for ; Mon, 27 Feb 2023 17:31:38 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D407340A27 for ; Mon, 27 Feb 2023 22:31:37 +0000 (UTC) X-FDA: 80514519834.20.E44B7BB Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf28.hostedemail.com (Postfix) with ESMTP id EDA75C000E for ; Mon, 27 Feb 2023 22:31:35 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=XUlKpAPD; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537096; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=BsGO9+iSszn3nu39GKw6/CO3iDfwWfjJ8YMO59zeTxU=; b=PJ49Ix/LnsAgRdI8PWXpLzWXS3PXaSEnLEv1Hs3JKlIzMwi02S7OoK1e7E+Eu1Sl1RZj+i tGawtd/eZjBrh4cNtL2NuUPMyUTDB62sHwZjfC6LGq3iOVcr0eBYxhNErZ+iqvJTgxDwC0 4i6V5OW1cI5SCQOKuRBsfbL6zFB9/0Y= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=XUlKpAPD; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537096; a=rsa-sha256; cv=none; b=6QNiS3wS6NIxaV9aWIv2DRqOCRY7/oWecOFVZGSnu/AumvHYz9K55QY8wcwNqr0SboKB0U lqtV/prZ4S0DhEYBKkwFrW0dDj5IjufSJR5YaTk3+qlbXzT/8km+QiWf3I/ZG5ekOaLu1c oAQBUcCx1UyBfQNHESQjW6NP0Kt4LgM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537096; x=1709073096; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=A4W74hb/mIAsetG4DQoyU98QDtxo6WTGpJJK3DMv6sU=; b=XUlKpAPDQSojiQPNyJIkqX8EbAm1FeGK5C+y/GGDh9HSCVcRRxCS350W rmNJN11SeSCoL0oPuTiucSyZd6rjz5quJMXZUJwjFVXfiHxJfjJdH7sti /Mn5KkSzF41EQNt3dFeQRoFpIwwMHrAXLKVJjiQSD4a8ZjzuQGh2azd9W 9sVkh0sghnfaB7+jLPLOEURg/FIqbEEXnFcp+OssGZ4Gi8QqbbKhBZdZV LF/Fa4mV/YazUALeDFNQ2NbquEjDq8dvdVQVa+XzDhZ6Tjsq/h18w256L j8pnvYEnvUobaJ/0cI9ksXz7PiOQVnmxKd8peRVhjXgPuKH0JXjVezuHf Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657206" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657206" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:14 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024424" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024424" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:13 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu , Christoph Hellwig Subject: [PATCH v7 09/41] x86/mm: Remove _PAGE_DIRTY from kernel RO pages Date: Mon, 27 Feb 2023 14:29:25 -0800 Message-Id: <20230227222957.24501-10-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: EDA75C000E X-Stat-Signature: ojp8kyaamd3mkd148zdxxmqnsczstazj X-Rspam-User: X-HE-Tag: 1677537095-617122 X-HE-Meta: U2FsdGVkX19T6pM8a40c/Y+2sgyMYqLt+XzMh/iK5gKtMlO2RZtSS2TNpqEpPBWYOx55Jq04As/+Tly3MgdxvrjP8U2Z06aWfPLI36PWRtQSZ/6VO2+seBTKvYst4Px9y2ta17gvG9CB55BQBwR6uObYA4e4jtRRYJop4gkb/CgF6QZiBQKCRjV3LRpEqc0JnHVngdgyC3zqM9qNUI2WdoYWjhl7W/gbesVzmGtKo8++/ocVCyTjrJ/ugi9xr6PJhEJYLEnY/60Dov1S7i3qeeF2h4dKAW2S27up32aDyQUQuPTwPaHnHUPp6Sfia/sMkkJaEdjif/JzCU0k/QdvZD1wZD94CiIem2aWaAk1Qi0jRgar7caVn6m7rUc7yzOzaWfouH7J++5hB4BIDfDK7ODV6Bnkws7dXRRQuAlwhLqcPBdJHsvzayOBon/cn3P5ARycrMpGp/muV+WuVCCwEwZzcHtt1R5Gqca87w9kCZoCuA5Npt2tqVVDZAhOw9SODGfq1mBw4Z2Mv/B+/fdd5CAd1O15xPPj3YdT0NjpaN0yGhwXk0WIQ3W0/6qHEXLAm3p8zws0aW4sKHX0tbCqbtLJEDo6gO9Iv6LhQxltsvaQR5MPrkpFwrkwV5IDrOfiX1DUhrJbd20R4TCW8k+ctcyvOj8LSKOr5CYSGvOH+LG7iBQyZQ/0w4/LY3Vn1Gm/MS1az8xC2FSG891K/Z5yZIBZ84PFsGuHF+ICAIqekbEN/0tJCOWpwFcDLMbTOgGxdQrB+oBQp3CQjArMbe0IBzo+yeSasm3asNzngJ+JGi1Iu7IFcyes4a5r7siqXAQgXATK/k6HNA594LuJxMIEVWXBOtLXraJHin3q5f4icYjJL6ekOS1fmsZBPz2Yv9+rh7HPq4ieiJPpu9lXJ3kRyG3nQed/QdEpamOxQCweKOUyz7nt4t9QuxytmwPRbosxE6h0idE84a4nJfqrfDv HI8/GTyD tQAiXJ0iAZuGdWufNSeyYaBJhMU//FMghmjb/8IpQEjB1CM9HMlrXIC+w0385jMtQ44S9WCD7cDSbNzGTOQqP51q4jUbgJjADTGl/4JZDvfpcW1d2GqC0q99HOC9jgKDcwv6yJxUBO5o38EYpalXPH8AdP0oYhaPi2HCCaWyL4lTgAnsuQvpx88t9m7A2Xvq3wHUf5nMaWdKG8GA25DYSX07xKQW63lYvrx92eN93x90e8nvdgHQxzTopQaD5BeaZ01zAxoIs/QyrLOilV6C3hyCfqE/zAnOetMppm/t7PuNzOrboK4EY97CAH5PzjM2xeFh+I/WX6yEJZABmKrSLakgvswAPaQ/txutMH4ANikhr+h6XIUYxqW6eMLsv4h0ksJgL0Y7tLG6RtfnwUpDTBs5zmJuFtgH+RNCes+LswDYTnsTk9POuKmuZ2iHNCKrzkJZY43cAx1HlIVo/vKgXNRL/OACfWPPXCShPm7Z+zOCE+et/If6KKUzUpsOTr/xUitoh7wGf62lrmKXZjZetxVUNYyNtSdJpuAPUkeIUtK2NXdiyyc2KcGsvvw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu New processors that support Shadow Stack regard Write=0,Dirty=1 PTEs as shadow stack pages. In normal cases, it can be helpful to create Write=1 PTEs as also Dirty=1 if HW dirty tracking is not needed, because if the Dirty bit is not already set the CPU has to set Dirty=1 when the memory gets written to. This creates additional work for the CPU. So traditional wisdom was to simply set the Dirty bit whenever you didn't care about it. However, it was never really very helpful for read-only kernel memory. When CR4.CET=1 and IA32_S_CET.SH_STK_EN=1, some instructions can write to such supervisor memory. The kernel does not set IA32_S_CET.SH_STK_EN, so avoiding kernel Write=0,Dirty=1 memory is not strictly needed for any functional reason. But having Write=0,Dirty=1 kernel memory doesn't have any functional benefit either, so to reduce ambiguity between shadow stack and regular Write=0 pages, remove Dirty=1 from any kernel Write=0 PTEs. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: "H. Peter Anvin" Cc: Kees Cook Cc: Thomas Gleixner Cc: Dave Hansen Cc: Christoph Hellwig Cc: Andy Lutomirski Cc: Ingo Molnar Cc: Borislav Petkov Cc: Peter Zijlstra --- v6: - Also remove dirty from newly added set_memory_rox() v5: - Spelling and grammar in commit log (Boris) v3: - Update commit log (Andrew Cooper, Peterz) v2: - Normalize PTE bit descriptions between patches --- arch/x86/include/asm/pgtable_types.h | 6 +++--- arch/x86/mm/pat/set_memory.c | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index 447d4bee25c4..0646ad00178b 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -192,10 +192,10 @@ enum page_cache_mode { #define _KERNPG_TABLE (__PP|__RW| 0|___A| 0|___D| 0| 0| _ENC) #define _PAGE_TABLE_NOENC (__PP|__RW|_USR|___A| 0|___D| 0| 0) #define _PAGE_TABLE (__PP|__RW|_USR|___A| 0|___D| 0| 0| _ENC) -#define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX|___D| 0|___G) -#define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0|___D| 0|___G) +#define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX| 0| 0|___G) +#define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0| 0| 0|___G) #define __PAGE_KERNEL_NOCACHE (__PP|__RW| 0|___A|__NX|___D| 0|___G| __NC) -#define __PAGE_KERNEL_VVAR (__PP| 0|_USR|___A|__NX|___D| 0|___G) +#define __PAGE_KERNEL_VVAR (__PP| 0|_USR|___A|__NX| 0| 0|___G) #define __PAGE_KERNEL_LARGE (__PP|__RW| 0|___A|__NX|___D|_PSE|___G) #define __PAGE_KERNEL_LARGE_EXEC (__PP|__RW| 0|___A| 0|___D|_PSE|___G) #define __PAGE_KERNEL_WP (__PP|__RW| 0|___A|__NX|___D| 0|___G| __WP) diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index 356758b7d4b4..1b5c0dc9f32b 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -2073,12 +2073,12 @@ int set_memory_nx(unsigned long addr, int numpages) int set_memory_ro(unsigned long addr, int numpages) { - return change_page_attr_clear(&addr, numpages, __pgprot(_PAGE_RW), 0); + return change_page_attr_clear(&addr, numpages, __pgprot(_PAGE_RW | _PAGE_DIRTY), 0); } int set_memory_rox(unsigned long addr, int numpages) { - pgprot_t clr = __pgprot(_PAGE_RW); + pgprot_t clr = __pgprot(_PAGE_RW | _PAGE_DIRTY); if (__supported_pte_mask & _PAGE_NX) clr.pgprot |= _PAGE_NX; From patchwork Mon Feb 27 22:29:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F559C64ED6 for ; Mon, 27 Feb 2023 22:31:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BC81F6B0083; Mon, 27 Feb 2023 17:31:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B9E5B6B0087; Mon, 27 Feb 2023 17:31:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A3F4A6B0085; Mon, 27 Feb 2023 17:31:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 703436B0088 for ; Mon, 27 Feb 2023 17:31:38 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 538BC80A5D for ; Mon, 27 Feb 2023 22:31:38 +0000 (UTC) X-FDA: 80514519876.16.7461DE9 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf13.hostedemail.com (Postfix) with ESMTP id 6F90E2000A for ; Mon, 27 Feb 2023 22:31:36 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=d34L9Opn; spf=pass (imf13.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537096; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=Ic4aJhbI+buGyHxHvTWd0ForjxPqTZwiqz8ffS9E0kM=; b=3kaYwXuRtL60IhEVz65uKBvwqp6rQRXTBuuuSdthfaqq/783X506Y0e1NE3PLoR6hPLwx4 NqCRpy+tYzbpeuqjAMYchGKre2V0jAH6aDzx9LS9XCtPpwoZFCvr5EVFKIUQ2krr1UwYp0 ySyeJOD6YItSJVpeH1iVc/+60uhti48= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=d34L9Opn; spf=pass (imf13.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537096; a=rsa-sha256; cv=none; b=tbi5zDkHuGqodaKv+M1QAVd/NlEzK+RPcs5oFbl97ZOVv4ns3tJ/uCGTKVIMJcxTkTdVt+ JBORwNW/GgYTa3xibAHTzf9JU45xoj+VJUCD/JRZP6BjD5ZaEM7RlZuaBmCSl5tp66e0Y+ UEIjgNOE1UyxJHAh5/xtBF5i+AL6ysk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537096; x=1709073096; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=PJnxzPZePTV1h98WOAU1laXgjeyErhdpAXi/giQ17HE=; b=d34L9OpntE2O1+Q44yEnhTSH4NXzOGdimGmUXHH7fYkloXAjpRcy0zhT fGHRfiCuzXe5wMeNbDCJpZze3JBUgRqCjwuaHfJkRRpHiG3aXxGvNRHXg W0orhz2mPzuJrn/IHqYup7rxKF/739d6J0b7fLHAv7HXmm6xhFm2XkMC9 EI0xljwvoylLmpuSKmhR4wY0zt6X96LjpvyJ4/Q4b6McZMrrO4rQ78bDG FInK3Z7AMoz29EC4DAH0D61RousS1+/Nit1obeO0RaeViGqtDNP8W509C Wlz/hTYrfeabXgk0LqgMMYxpn5n4p3bf1otedxR6kdSnxIrPGhILCcyyU g==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657230" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657230" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:15 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024440" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024440" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:14 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 10/41] x86/mm: Move pmd_write(), pud_write() up in the file Date: Mon, 27 Feb 2023 14:29:26 -0800 Message-Id: <20230227222957.24501-11-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 6F90E2000A X-Stat-Signature: 5ias751wqqep9nk4453bjgtesw1p3fy3 X-Rspam-User: X-HE-Tag: 1677537096-637179 X-HE-Meta: U2FsdGVkX1+iCcMgrvO1VF6MPlp5wzkjKScz1Lf73JTyT55t/AGcREUHU0HqJfdb7TTPNcIU8TIg/btYi/vI6wylnGd65/ZemPIq8RPuzuwKVhOg1CN+jy2Dg8NQiS4R75OjIpsv0wEP6PWOKQxTc9xawaD/11guGQ5oAeulBjJE5mNeqALcEdvpP/L5J4nmY/XSLzWyvM102eU/W3j8bOuPoeBDMASxa53BGSgMdQ6UaGl6L6oe2jg788IONq6Vyj8MPyX4maoXtpZvG9Ep05agO0BlNoybWRrtdtLb4mDp8dJI0ocCQT6tGGR1oeV0e+HHEQYYk0ovc66Rtt/xc30h80+OuoYx7Tud0obpkV4KGq2vioH3IDDV6pVCmAQnCiMGPULwPrxMXH81mGNdGEFaGaMZsr9Ft4O+bBXrcDM2BpRH9n0RJPc3hTBygpCyd127wHFDgKp9Hg79llajIS9elOZ2q04wZ+THRSZVJxyN2PHx18XjTAs7HFKma2rZDMhYBOW2NACDGUOlOFZUelB2J0etCAeOkii7Ca9K08QAGmYS56dzRkWmgw2NI0HGo9etdJFXwASU5gWfBt0YuvFLW6mQJKTl+8bcgxWOdJgWGO6e+Dnf3L/z7qIdwo0T5b1hFgHr54pu7uiQ7fRFKWWSG36LZYa/E01wkLLkm273E0NVSNSniw9lKfnZXTfZFo0eiv2VztNIxCVSBfJqmVOYI/YlHU+r1ofFXxMarwnZez/R77YRf/EzOFKVicS9vR3mqIp7oyqSehFtI7QniAiFUNNffBnQ1r2d4xuMPeRehNmgvIaD1MP3eKS8vnG8icE9VGQuqCwS1DX8AokQkAHlPNIn/hjjktv3hTP1w0+iVH7SQOZDjfPY9Bh6IgVAkQ2sT5ICuk/DQcFD+5/w88TIxEa00VJt7OlFwD4SRMq6h4SM9OrWrpKt+CtOgyuY/jXF4g+EGAAcGfnOss4 G484DiUf DeFxkoxBkD9i6qvqXGz9JAKZTiJ7ahy3ID8Zesi416yhos2nUhEO7u6HvBJN0zikcT4yqhwTabicHX9va9VyE+dobaIdb4vuulBrRaB3/vXC3r0IHhkXe1ZmD0mR70OC2wvFFcBac1tgTGbMdR9WPSkBV3JCdKgZgA/qiZvADV3BkANhqfppi47gxTYuCO6p5MIk+al8fCYWArvGOfRz9L8AYfiBh2ZS197/ThNiwJZVzp3N6a4gBylLWNfhw5OkL3ipWx0mW73AOBbdvtfdMkW7rqFCElaQrmf+PW/2CSCTuTWCE3ME/Hy95yHGkIai7sMrzR4w65Q1Ook0kq96GzdKlJaC042BMxw9CzjjHrtTybbBRX69XN0McsvaMxww+eEpx64pN05kO4vo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu To prepare the introduction of _PAGE_SAVED_DIRTY, move pmd_write() and pud_write() up in the file, so that they can be used by other helpers below. No functional changes. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Yu-cheng Yu Reviewed-by: Kirill A. Shutemov Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/pgtable.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 7425f32e5293..56eea96502c6 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -160,6 +160,18 @@ static inline int pte_write(pte_t pte) return pte_flags(pte) & _PAGE_RW; } +#define pmd_write pmd_write +static inline int pmd_write(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_RW; +} + +#define pud_write pud_write +static inline int pud_write(pud_t pud) +{ + return pud_flags(pud) & _PAGE_RW; +} + static inline int pte_huge(pte_t pte) { return pte_flags(pte) & _PAGE_PSE; @@ -1120,12 +1132,6 @@ extern int pmdp_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); -#define pmd_write pmd_write -static inline int pmd_write(pmd_t pmd) -{ - return pmd_flags(pmd) & _PAGE_RW; -} - #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) @@ -1155,12 +1161,6 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, clear_bit(_PAGE_BIT_RW, (unsigned long *)pmdp); } -#define pud_write pud_write -static inline int pud_write(pud_t pud) -{ - return pud_flags(pud) & _PAGE_RW; -} - #ifndef pmdp_establish #define pmdp_establish pmdp_establish static inline pmd_t pmdp_establish(struct vm_area_struct *vma, From patchwork Mon Feb 27 22:29:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154242 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACF81C64ED9 for ; Mon, 27 Feb 2023 22:31:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 514B16B0085; Mon, 27 Feb 2023 17:31:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4C0706B0089; Mon, 27 Feb 2023 17:31:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C54F6B0088; Mon, 27 Feb 2023 17:31:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 16C346B0085 for ; Mon, 27 Feb 2023 17:31:40 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id EE82BC065E for ; Mon, 27 Feb 2023 22:31:39 +0000 (UTC) X-FDA: 80514519918.27.13EE1C2 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf07.hostedemail.com (Postfix) with ESMTP id D0FB340011 for ; Mon, 27 Feb 2023 22:31:37 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=S1CYLG4J; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537098; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kf7MHkInNN4jdhKGfQzSmJle/kstjdoC6HAG2DBSO1E=; b=hVhbOLfVayl5HS8u2tbOg91zCmwo8HFd6kAl6HwBe2zqTuvWRzwMRPtJRNljuqFE9Q7oJI EaMsKiCGw1oBfKFbSU4bj/Qs7L1gPM69ySEyLi756R7jyZNVCUHJykVlrPoRUFDiqPlqaD mpGiyFwYcEcwUw2KJnzGIkiFCsvOGPc= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=S1CYLG4J; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537098; a=rsa-sha256; cv=none; b=FrYmLl8oYRIgo6IlttmZvT5P26eMscF5+e8Pn7Hzt1HpqBlB0KzRB5aIMR/rgAFZK+4Tjt SwhWy5JOJ88Xw9plJG78ZOsxYpE2lizDDJs1vSH5bSyVls0pkwGybZyLmIWZ011yj0HjyR KPnxo8HIyYVoqG9k1jA3/xxNgc2emPk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537097; x=1709073097; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=h3g9dfsaSYOQw9Djcf7aO6npKOb7bN/WODuQBbF8SE0=; b=S1CYLG4Jo/xtZfBlB5jMSYoqTEWFoLAjjmm+xUVkZfSESKeJm1kgfiem 6HuGCKCGJjb+SPcAQGeiTmVasCB+LMK3Cq4zRQepy+h8uHeR1ewuxJG0E IHp1QwkbmdkUesKjK5VT6MtDu6VmnEYtnP2LOGTOLhm/jxFFO5pYQ7f+W J6N8wy5Nzog7GGg+8PssIMkSMmVQDfv4KK1nyUmM3URYvBEv+HuxofcWp 1+IsHvNRbcWetf8EfQTcAGAaZsc3QDIGYXxIEOner8ehUzs55gMg86Jbi aT/ilsO50uEPsZgEgUkx6lZYz8F/NJWszsY307BmWdraA7XV00pMGHCsJ Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657256" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657256" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:16 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024459" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024459" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:15 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, linux-arm-kernel@lists.infradead.org, linux-s390@vger.kernel.org, xen-devel@lists.xenproject.org Subject: [PATCH v7 11/41] mm: Introduce pte_mkwrite_kernel() Date: Mon, 27 Feb 2023 14:29:27 -0800 Message-Id: <20230227222957.24501-12-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: D0FB340011 X-Stat-Signature: coaode6geyu4xk7hpzt67cooqu85ka8c X-Rspam-User: X-HE-Tag: 1677537097-62634 X-HE-Meta: U2FsdGVkX1+Km4RJEfOtviuXtdrKwcorKHt8yPyClwfyuvEu4b7gp0cFnlIQiINylyy/dzVq2f3lNHIbRR/6I5FzV/5sWT5dBzL7kTjjyDd4+FgaOamaByckE8rHAPJZQ0OvO4nNI/t/cEHbxMjzWuoNL3og8V8/FGpTaZPhtgQukkQJRML7cgmQfMEeIiJrupgqz7iwRA8bhk/xr1e1p1+nTX+/hVGoyZYkd05Z5H2lWtJsyQ7LDJUXizEagfE2DjpVp2a6XTunvKfskXdqyWqp8O8TrhON/uztxBcuDjAH1+amF6pMSnV+9m7E7SL14+go+y8zuc+28Y8MzVA5EKmD6pAZysSW8hF/3R0kw+GTjXVa7ShE0G7HZBcwKUtpgeAGFYJ99YDOw5hf8mmNzGMpg7EgvmiUT6r6bCNveOczV442YeI9kbnL4sJHsCusEtjfqGKIxK4820VqFWhrxojw5xaTOEOYAvFLQ5BhzRRi34y0PItIqDM0Af0veqIRAPLfo5s/S2CXdVHAvQi5uLUnX/AgNfJlSQakQ+3/7R+wVYt9NKveaZvNhOlhJCZoKoHEv3Fap98ibpLC602P9mggxBIaFhqNJq4LK71SfiOvtZICsY1tfXaLF0aEfgxcvMcscqobxVyeiMMePkWRuKt3i3XnV11BsaA34KGV3FRDoSxzKfTZy0U9D5fNcOkbP8Xu8ZNDDYGoIWlaAWkWkF09hlBFIg1GVT8GlZv9uZKWuzHDM8ijYMNBpt/NrKPU80kwkgEoi6ViHtlUJLo6YoZDtdgcRVUtKcQZ5GgQj3PLQHRxCbKWxpdtNnvIzKs6dSCNRwXBj12JaTaU/cKDSYKLGEa/KtFtl/DRQzZkds94kCOIB9akk1mi1tzg+ZockEM/LHvMQ4rhf39XDOgal9NQXBDfphGH89z+hm/tEHlHK948bZTBtLT8UyApfaIk0TmhwgLbKUvi5WXQCHN GgV0iZLY 0eOt7hAlfPubtFUqhBffFIctdL58x0DIYNoAN0JWj4caG/xja6vhYjcGrBBR0kAyPJThjvVPEmTa7igZM8YbXoS/MI59s05mIDqIoGOWidB2B1tPo5yKIlFAeIBRanA/RN2bVk+ibOb+ZNSqQuES4V3xwozvBsNCG1rUgWfTLA9e21Pl4F8B+HDGpd8zWFkpOOKO9LXa81k76NvK21/br01Un8IJ34Yz3e/91ltNhz1Oe77dpwXkfZJngNBsqlRWGfNzf7csyeohnhI7rWv6pOaD3fzR+8MGu665GULfLZBuA2yyUza7h18lPoJxgNKHY4MwXGih4eMegQLitYQtvxCI0LAzGp/67iI6o9EIVuigpD5gzkbcKQkmrQPDFnXPaBR8nolkgAW9X2iXg5J1wavp51oNwPFmnAQzUzZL+NqqWQhJeosfekiIDidEOU8fgvazz19v6Ytjwa/LHpSKAJ9zlrhRsi9xWwIdpEoit3J6bwX9+Y/6DmuULjXaXpErmYZFvEPFw75GSB2ynY7b3oqYY8F7IyhRwzD36U1irAw6rKv/i2U+txKDgezMmmS1UYM7ChM4PmMFDAifhpd4nfQufdSgaK3TrXQDwOsFrHpHDVZ7aSab8sZtJSK6d/kmTaBN2tRkEuhuBKSk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. One of these changes is to allow for pte_mkwrite() to create different types of writable memory (the existing conventionally writable type and also the new shadow stack type). Future patches will convert pte_mkwrite() to take a VMA in order to facilitate this, however there are places in the kernel where pte_mkwrite() is called outside of the context of a VMA. These are for kernel memory. So create a new variant called pte_mkwrite_kernel() and switch the kernel users over to it. Have pte_mkwrite() and pte_mkwrite_kernel() be the same for now. Future patches will introduce changes to make pte_mkwrite() take a VMA. Only do this for architectures that need it because they call pte_mkwrite() in arch code without an associated VMA. Since it will only currently be used in arch code, so do not include it in arch_pgtable_helpers.rst. Cc: linux-doc@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org Cc: linux-s390@vger.kernel.org Cc: xen-devel@lists.xenproject.org Cc: linux-arch@vger.kernel.org Cc: linux-mm@kvack.org Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Acked-by: David Hildenbrand Reviewed-by: Kees Cook Suggested-by: David Hildenbrand Signed-off-by: Rick Edgecombe --- Hi Non-x86 Arch’s, x86 has a feature that allows for the creation of a special type of writable memory (shadow stack) that is only writable in limited specific ways. Previously, changes were proposed to core MM code to teach it to decide when to create normally writable memory or the special shadow stack writable memory, but David Hildenbrand suggested[0] to change pXX_mkwrite() to take a VMA, so awareness of shadow stack memory can be moved into x86 code. Since pXX_mkwrite() is defined in every arch, it requires some tree-wide changes. So that is why you are seeing some patches out of a big x86 series pop up in your arch mailing list. There is no functional change. After this refactor, the shadow stack series goes on to use the arch helpers to push shadow stack memory details inside arch/x86. Testing was just 0-day build testing. Hopefully that is enough context. Thanks! [0] https://lore.kernel.org/lkml/0e29a2d0-08d8-bcd6-ff26-4bea0e4037b0@redhat.com/#t v6: - New patch --- arch/arm64/include/asm/pgtable.h | 7 ++++++- arch/arm64/mm/trans_pgd.c | 4 ++-- arch/s390/include/asm/pgtable.h | 7 ++++++- arch/s390/mm/pageattr.c | 2 +- arch/x86/include/asm/pgtable.h | 7 ++++++- arch/x86/xen/mmu_pv.c | 2 +- 6 files changed, 22 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index b6ba466e2e8a..cccf8885792e 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -180,13 +180,18 @@ static inline pmd_t set_pmd_bit(pmd_t pmd, pgprot_t prot) return pmd; } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite_kernel(pte_t pte) { pte = set_pte_bit(pte, __pgprot(PTE_WRITE)); pte = clear_pte_bit(pte, __pgprot(PTE_RDONLY)); return pte; } +static inline pte_t pte_mkwrite(pte_t pte) +{ + return pte_mkwrite_kernel(pte); +} + static inline pte_t pte_mkclean(pte_t pte) { pte = clear_pte_bit(pte, __pgprot(PTE_DIRTY)); diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index 4ea2eefbc053..5c07e68d80ea 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -40,7 +40,7 @@ static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) * read only (code, rodata). Clear the RDONLY bit from * the temporary mappings we use during restore. */ - set_pte(dst_ptep, pte_mkwrite(pte)); + set_pte(dst_ptep, pte_mkwrite_kernel(pte)); } else if (debug_pagealloc_enabled() && !pte_none(pte)) { /* * debug_pagealloc will removed the PTE_VALID bit if @@ -53,7 +53,7 @@ static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) */ BUG_ON(!pfn_valid(pte_pfn(pte))); - set_pte(dst_ptep, pte_mkpresent(pte_mkwrite(pte))); + set_pte(dst_ptep, pte_mkpresent(pte_mkwrite_kernel(pte))); } } diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 2c70b4d1263d..d4943f2d3f00 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -1005,7 +1005,7 @@ static inline pte_t pte_wrprotect(pte_t pte) return set_pte_bit(pte, __pgprot(_PAGE_PROTECT)); } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite_kernel(pte_t pte) { pte = set_pte_bit(pte, __pgprot(_PAGE_WRITE)); if (pte_val(pte) & _PAGE_DIRTY) @@ -1013,6 +1013,11 @@ static inline pte_t pte_mkwrite(pte_t pte) return pte; } +static inline pte_t pte_mkwrite(pte_t pte) +{ + return pte_mkwrite_kernel(pte); +} + static inline pte_t pte_mkclean(pte_t pte) { pte = clear_pte_bit(pte, __pgprot(_PAGE_DIRTY)); diff --git a/arch/s390/mm/pageattr.c b/arch/s390/mm/pageattr.c index 85195c18b2e8..4ee5fe5caa23 100644 --- a/arch/s390/mm/pageattr.c +++ b/arch/s390/mm/pageattr.c @@ -96,7 +96,7 @@ static int walk_pte_level(pmd_t *pmdp, unsigned long addr, unsigned long end, if (flags & SET_MEMORY_RO) new = pte_wrprotect(new); else if (flags & SET_MEMORY_RW) - new = pte_mkwrite(pte_mkdirty(new)); + new = pte_mkwrite_kernel(pte_mkdirty(new)); if (flags & SET_MEMORY_NX) new = set_pte_bit(new, __pgprot(_PAGE_NOEXEC)); else if (flags & SET_MEMORY_X) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 56eea96502c6..3607f2572f9e 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -364,11 +364,16 @@ static inline pte_t pte_mkyoung(pte_t pte) return pte_set_flags(pte, _PAGE_ACCESSED); } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite_kernel(pte_t pte) { return pte_set_flags(pte, _PAGE_RW); } +static inline pte_t pte_mkwrite(pte_t pte) +{ + return pte_mkwrite_kernel(pte); +} + static inline pte_t pte_mkhuge(pte_t pte) { return pte_set_flags(pte, _PAGE_PSE); diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c index ee29fb558f2e..a23f04243c19 100644 --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -150,7 +150,7 @@ void make_lowmem_page_readwrite(void *vaddr) if (pte == NULL) return; /* vaddr missing */ - ptev = pte_mkwrite(*pte); + ptev = pte_mkwrite_kernel(*pte); if (HYPERVISOR_update_va_mapping(address, ptev, 0)) BUG(); From patchwork Mon Feb 27 22:29:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 673BDC64ED8 for ; Mon, 27 Feb 2023 22:31:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E6636B0087; Mon, 27 Feb 2023 17:31:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 786F16B008A; Mon, 27 Feb 2023 17:31:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50F696B0087; Mon, 27 Feb 2023 17:31:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 391C36B0085 for ; Mon, 27 Feb 2023 17:31:40 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1C4581A07C0 for ; Mon, 27 Feb 2023 22:31:40 +0000 (UTC) X-FDA: 80514519960.25.59338A0 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf28.hostedemail.com (Postfix) with ESMTP id 17C97C000E for ; Mon, 27 Feb 2023 22:31:37 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Nqhmx5ZK; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537098; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TG4W8GNWHkgVckZvhzg7bpbibSBgnztcU0bXpGrX6WY=; b=g0dLpxl6b/yWp5TY/S6trqUmgrZAgTnNAjZncJNVQsIENvuVyVExiK77EtK/kyx/mRXUi7 3aKdmKLgbdnwwvd/LZIeV6SLWMheGxqf+GgM4Amzbvw8dk55Uy9R3uKIZk180yACvYeOCw GWixTTYi+THJOz9/XwkhUnyJWXCb+pE= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Nqhmx5ZK; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537098; a=rsa-sha256; cv=none; b=RFepmKNfMh7abmL9xOz6+149y1v+a019eJE+zebrLEvxdRKoqoZRk1h0afYAPNzSSMS3fV 4vwHnOvJCU6kL+MeGUfHxRVswrPxvFrmj9EA5+v2+w7qHNiv+KcceIdiXvAJt4y3TzK/U2 SLvwxHPOQxwrj+k43gDXMm/90zvHUy0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537098; x=1709073098; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ErfroshrbQualKPLAsK/DpeAwNdzrJEBOvME1abf8WI=; b=Nqhmx5ZKzHljEeH00gT5i44iLXYa7NpAkPb8hFL7qXz4pEgX7LagsqpX d0c8XY5RBp1JnzzBtX0U5bKEKyjhyhs90zFrYUSMFLWqENL3FelydyXg4 I20zi2Fl8EXTussAjQaGcguC6aMhSSSIP/vonsYP8Sj4sk/cbWQwiTPty qNreCsEY8626jLsQeU1rE9vzzVeBrW5Hzm4hNdpsIIM6jE4e5njrQIToP UjohPNSwnnWHk+nfJzNuAkYgx9JOUd352WkHjzS8vmXwZkZSJRQ6lfGPu zvX4e79/1fIguxJBGbbVhCV05FR9aZMhXvWjUX7APRNNKMv3KgvZ5pZMT w==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657286" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657286" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:16 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024471" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024471" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:15 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, linux-s390@vger.kernel.org Subject: [PATCH v7 12/41] s390/mm: Introduce pmd_mkwrite_kernel() Date: Mon, 27 Feb 2023 14:29:28 -0800 Message-Id: <20230227222957.24501-13-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 17C97C000E X-Stat-Signature: g8u46pmn7hyeqfbny83n58wtjeyde5tn X-Rspam-User: X-HE-Tag: 1677537097-83426 X-HE-Meta: U2FsdGVkX1+RuZA7+Yq6IijgWoL1z+Dk7tNavQQOJkYb9/dhxuZonphelRFIQ99z0LhU8UhOI2kNUF6O+GK3nCLERYCZW/GkJFyBDXd9M/dV43z2vvjYuoQSAWetkxUOAZTPxlzLav4RKi9EPRn6JrmhuEB8rQPuIWtHbO79Gk4ABQtcCSK7W9OsB6J+NKGKKYet+NOUL+Ve5fWgRsdUZVB1mUPCrd3wmR7OznLGkOBy0TGf1QpAWtffZbco8PtNYXnoCzr6zJ1c5HXJabK2Qs+00bN9hbeor9Nkux8+Nr07KWEb0LYz3nDoY2d/67JZHQohSoADZEFw6qDJCH90quL686u9so+IPoSVAEa7V1aIlp4l7wIpCXA76ovq2JMLHdpHkBrZKZkvAFFTWoGoxHIuLQlY2qbmoi7IcURlqSbXW/x4OkQslNfoGA2W7vmGuw6zt5SQKMcnsnvFD+RufORd97pH7YrKBjIf311Qs/HbIqVOB1QkdWOXERuowC86vLMsN4ItP8j/fBBJkQS7mCbHhhS12hhgExnAkzBaEFesmiy3DoerCHgQCAX24jvZZsWLyfHSkaPN0wbtEV4XKhBwh3ysfhnjZYav7HP0+TWLYFX3DpY2I2AZOgZJRl85VjpEBLi1+3O9y2CEkzR2Hr5hQ9nJfLQei9jKRdmLteYXzbEIMJyOh3ZigXtjNwJSe7yq/pIkNtjPAEtS0/tpja4rMRTy/WeKJFtIKP0qD+zmK86l+qd+r+2AHrilUTvw/Up2mA5dkav9qXywc4btQhHZ2thJ2EGwTiezYp1PtWUHgh2Qmyeq1BTS7BS+xLCHnh5krUzneo3n/ArNf5NaJhyLabAKzqs+AydZEq2vVDiExnAG0NPWWWvkRQrs/uKV+KuMCWOwOshEj5812tu1XB03auxGr+ELEEZI+uYLSdTrhytRlMw4DHeFxwjfnKwWTKYVQZpFQ3PS7TPG0ij ru2kt9Tr SBe8yLhnyt54QG/cqEMEgDfuuoNTt5GHQegkytSndoEs5Cdi60rT1U/zVa3M4aJxfWa9t67QWLXxReYUFIers2itI+iuKXlGN9kIJqo5iRbXTbtT2LUyvrinZ3crh+HEpbUYJ2PqFCYCU2Ne8YICteub/I4D49YCwvEzWCuy4NUe4cE7mWtwiumSUk+i+5OR8sntq/lTLdUidQg9zjiuFzWPcwUWQhwfKV+B65Cx1ODpr9FPReyCGi9GWWfyoF2myDFfxLCwD2OmE0f9ofyu054xIxB8t5bEeT+ZrpUcsWzER8zmWT2+dZXJ1TYfjTRHm3cJzDC2n0MyVqbefnAwwbHQaVlLX3Z13b3FQVikNnyWgWAChkCIp7BI2tTWLAKGVGO7uw6/Wk78SKK5m4JBHC7U/4FXVni5FcoPbFj29WfF2wuD6btZ/GDP+gTG07oDlzvQ1QdJCc2WpAyHlO5PBWwFKxAe99b0/UG92dznEorYSSmTjXfj45G3VeKusf+nARh3vD3fCm29G93S/khWmo7ZG0tUx3nA1O+7rZeqbEQeg+ME= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. One of these changes is to allow for pmd_mkwrite() to create different types of writable memory (the existing conventionally writable type and also the new shadow stack type). Future patches will convert pmd_mkwrite() to take a VMA in order to facilitate this, however there are places in the kernel where pmd_mkwrite() is called outside of the context of a VMA. These are for kernel memory. So create a new variant called pmd_mkwrite_kernel() and switch the kernel users over to it. Have pmd_mkwrite() and pmd_mkwrite_kernel() be the same for now. Future patches will introduce changes to make pmd_mkwrite() take a VMA. Only do this for architectures that need it because they call pmd_mkwrite() in arch code without an associated VMA. Since it will only currently be used in arch code, so do not include it in arch_pgtable_helpers.rst. Cc: linux-kernel@vger.kernel.org Cc: linux-s390@vger.kernel.org Cc: linux-arch@vger.kernel.org Cc: linux-mm@kvack.org Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Acked-by: Heiko Carstens Acked-by: David Hildenbrand Reviewed-by: Kees Cook Suggested-by: David Hildenbrand Signed-off-by: Rick Edgecombe --- Hi Non-x86 Arch’s, x86 has a feature that allows for the creation of a special type of writable memory (shadow stack) that is only writable in limited specific ways. Previously, changes were proposed to core MM code to teach it to decide when to create normally writable memory or the special shadow stack writable memory, but David Hildenbrand suggested[0] to change pXX_mkwrite() to take a VMA, so awareness of shadow stack memory can be moved into x86 code. Since pXX_mkwrite() is defined in every arch, it requires some tree-wide changes. So that is why you are seeing some patches out of a big x86 series pop up in your arch mailing list. There is no functional change. After this refactor, the shadow stack series goes on to use the arch helpers to push shadow stack memory details inside arch/x86. Testing was just 0-day build testing. Hopefully that is enough context. Thanks! [0] https://lore.kernel.org/lkml/0e29a2d0-08d8-bcd6-ff26-4bea0e4037b0@redhat.com/#t v6: - New patch --- arch/s390/include/asm/pgtable.h | 7 ++++++- arch/s390/mm/pageattr.c | 2 +- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index d4943f2d3f00..deeb918cae1d 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -1491,7 +1491,7 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd) return set_pmd_bit(pmd, __pgprot(_SEGMENT_ENTRY_PROTECT)); } -static inline pmd_t pmd_mkwrite(pmd_t pmd) +static inline pmd_t pmd_mkwrite_kernel(pmd_t pmd) { pmd = set_pmd_bit(pmd, __pgprot(_SEGMENT_ENTRY_WRITE)); if (pmd_val(pmd) & _SEGMENT_ENTRY_DIRTY) @@ -1499,6 +1499,11 @@ static inline pmd_t pmd_mkwrite(pmd_t pmd) return pmd; } +static inline pmd_t pmd_mkwrite(pmd_t pmd) +{ + return pmd_mkwrite_kernel(pmd); +} + static inline pmd_t pmd_mkclean(pmd_t pmd) { pmd = clear_pmd_bit(pmd, __pgprot(_SEGMENT_ENTRY_DIRTY)); diff --git a/arch/s390/mm/pageattr.c b/arch/s390/mm/pageattr.c index 4ee5fe5caa23..7b6967dfacd0 100644 --- a/arch/s390/mm/pageattr.c +++ b/arch/s390/mm/pageattr.c @@ -146,7 +146,7 @@ static void modify_pmd_page(pmd_t *pmdp, unsigned long addr, if (flags & SET_MEMORY_RO) new = pmd_wrprotect(new); else if (flags & SET_MEMORY_RW) - new = pmd_mkwrite(pmd_mkdirty(new)); + new = pmd_mkwrite_kernel(pmd_mkdirty(new)); if (flags & SET_MEMORY_NX) new = set_pmd_bit(new, __pgprot(_SEGMENT_ENTRY_NOEXEC)); else if (flags & SET_MEMORY_X) From patchwork Mon Feb 27 22:29:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154245 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBE70C64ED6 for ; Mon, 27 Feb 2023 22:31:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4C2C76B0088; Mon, 27 Feb 2023 17:31:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 473DD6B0089; Mon, 27 Feb 2023 17:31:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 253BD6B008A; Mon, 27 Feb 2023 17:31:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id EC9D96B0088 for ; Mon, 27 Feb 2023 17:31:42 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id B906A1A0920 for ; Mon, 27 Feb 2023 22:31:42 +0000 (UTC) X-FDA: 80514520044.02.F64ADA9 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf28.hostedemail.com (Postfix) with ESMTP id 845C5C0017 for ; Mon, 27 Feb 2023 22:31:40 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="W/r7AGup"; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537100; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=24H1afPok8BTe5Dv6zBGDosuJmpobZUXSlWAjRK+yOM=; b=nHLuYsVLgRtrwuPETcoJiTA+BnEVbEQrn30sxitbsyeazZzmlquTjLxYOhycxSvH/jLE1T v3SDkIMBaCtGVKLUCahNbxbn+hWavpsT3rAy3XqAQespof07FbEbFzzKaK9JrVv5ujPF3K gc6F/ZfzpWq6DAy/4cvqSxSfG3oYHe0= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="W/r7AGup"; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537100; a=rsa-sha256; cv=none; b=zUtEZmKCH9e7hxcsoDHMQLNNGiXCqUnDxgmmO/indJsAz3TatR+INMVvOWEiW6j8Sm94gr kbc3D8QovPi6ih2gtnnHljHSJ0/rUJgtOGmoNEyKhMfbzmGurVJX8QmApStZ0ovsB7wkhk EXbrC98anvCrIASLCZAiGrpg3cBaUhg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537100; x=1709073100; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jtLiGLS5ozXsIx63+p41sv5RK32EOGBRgxF7nrBF+NQ=; b=W/r7AGupzMp5I7dacRJWLK3uqFEgb3cZFV+jc1vGngrRPFOqv//LC3l8 hCQOHjI4QrDsopO9+E23c3Gp7KRNoT14hRSkvn+EFmY1as1c0bqEzbMSO KG7xGQLVfyPWpbS0UsYdslYCDW0RVJLpFhnmcAvF3wGbtCwc9KMtS389A Tn0KfTASCsdN/0tgD2XLXVUwK8L3nZQZ8xoauEHzTQcvazpken+FsaNJD oLNY5AGPHphNeG4uh1QGGyk9hkTmwOGtNbIWv7fSGr3PepGLs6El1/mWR 6gUKNPmWyO83JJSqd9rpAyHlWMRgfkNUQPQ/n++baSFTliPGMnohsQBRh Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657329" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657329" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:21 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024498" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024498" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:16 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, linux-alpha@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-ia64@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, Michal Simek , Dinh Nguyen , linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-um@lists.infradead.org, xen-devel@lists.xenproject.org Subject: [PATCH v7 13/41] mm: Make pte_mkwrite() take a VMA Date: Mon, 27 Feb 2023 14:29:29 -0800 Message-Id: <20230227222957.24501-14-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 845C5C0017 X-Stat-Signature: ddnanu58d638usk85n3fjthwhtzamjkh X-Rspam-User: X-HE-Tag: 1677537100-416828 X-HE-Meta: U2FsdGVkX1+3r18KCZuNGVQdguwIwM/bkakZyfbD3PASK9cKsNA048vIydLHALP6XqSyQm/7HsadhEn1JSmAT2F48IVoRAyGo3HNnvtz9WuPoJyz9pbvvggVy4I7jF0UdcCA9MWvLi+gIdOdwPR9cjU4aHXPda1KcX2M9Uy2YpsYJEnej4vIkTBAmn/3JwnU7X0n/Yj9vlwOVb82bSWhij839DT4QAbdd03GHHqdNAU5WiH0Qe+Z/UZgjdPt4R7U9BOI9IN5+9F8JrVFDz2ZFZAlaM1odJpsI4kIC3bswHmDfL7YHejxxhzDzD5VUKQJyyB6FFq/XhGw4CH0aWLAVjJQvrCPtoIh1XnTzWh/jCf0FhzzIMXQJexQQ91nR8SuTuJ3lXwt/2BXNDqbMTInGUWkaM6l33afD/3cQk/CflsKCjOZvPt1/g5w5r1YjoVHIWXWdZ2pC9cgMzJPUK7OHGRhEsaHal0Q3Otz1hNMONQ47Rf75oy22VBsFe51wBeCHG3cCDaguLlL7IwjbVpTpPwyU4gzXpFTTNCrCWy7a8K4O56eqINPdSdmg/66dqMOIrjvKc+AKLmGpUhNlF6Yj6GgrIWnn+COCV4FgIunQcoIliVcV3mtP0LBRx6kOsw5uaSTUx6AhILFpy97lrZAPZrDlRa677xh6M8jaXd2g6OSOXpP9RgVX9wuafgpA0OhITkprwjrb5zKdNBjXGW1qXMP/Hz4LXzMP4iqWtxyeTR42r3r8+qJH2BC/ciOIvZlHrS/+y7QyuLjGE+Od41c5nfvgfpfEBoHJ9qJTRiY9t54ok0N1xD3RpQQtvWxsxMVcbrS0muQ/vLxUK9DwmF1KTY+oH1iFv1FhytR46Dv/N88tsBpsBHnN0hA96llHdlAscgzxKzICwlZNkzfyX5xw5bk1GmxEnfE4MXZEGkADHwGzG+fG01dgXimidlXJCL4puHffLOCZmh0WnGSpeN IgEYv2oX YKHh2qhNlHKkXx2Ks3uA3VCMDeGfBgQ+IuzV2cSOiW2zkcbVqqV/NIQZhoS0wb4D1sTXX5kYTvEDaeFcNtH2UbKe+2BA4p7ad3wgWkT8sHEH3cezJcC38msaLy9Znimf7prDstnkoHoAUIfuxXeyCQonyR6Nvlr3bKnTA6kKq8KKKIdlLhG8xzV9p1fSLGs9ZLv4UD7qRyN/pkztZFSZ3VyLqsDy//8Okw7TzpyOlYtk6YdYMaZ1CIQYTO2Zd+GHpmr9SBiV+3OnjHYmnPIVu3WQQMYnmV2SWQmomw25qSsR78BJw5KtiaGoXnQTW4EulLU9MoAdDe2p6h3/wrgmUAhIjcAlX2YQGFmLtHnSIg43ccYFGZHz6YJA8D9EXJRZvNe6vYEjEE0UE5T8nhnTzK8lF4mI0Quz93xmvXroS4ryT+iPlgAl3j/EDNUemNk9jkVYKpwX89XQPP+Gw+oh4xbyWAIoCq3ylF8r6ot44uDc6fhnphdx8M0n4QcSukEXHURxfATGsixPHkjJmbIspfW5YTQqoZzRcYYgbGJ5MxAcDWJnAmRA0Lc/8RkWKUhBN0jGXpiEzvuF2JdWmxf1v+aK8fxVHdaMsUYn55K79y9FWLc6g4/XCWoWNeGvvQ3GqGD2U4PSzDKzDDOkKXbQtc6WPjhc4NGYjTioEocVafDTP6dO2j1FZAZZ1qiCAm6pqylMTK6ESi9DhCAVjgYEtfDjOG0yEsACpD98eAsqcxrTGxNwYL+dH++M8spqE2MxavbKzAdlAd9fvUSYKh+Ni4d9IZrZvQOpuaV+MgB1ypFkr6diqKdw2J3fsBWYelpksAiNk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. One of these unusual properties is that shadow stack memory is writable, but only in limited ways. These limits are applied via a specific PTE bit combination. Nevertheless, the memory is writable, and core mm code will need to apply the writable permissions in the typical paths that call pte_mkwrite(). In addition to VM_WRITE, the shadow stack VMA's will have a flag denoting that they are special shadow stack flavor of writable memory. So make pte_mkwrite() take a VMA, so that the x86 implementation of it can know to create regular writable memory or shadow stack memory. Apply the same changes for pmd_mkwrite() and huge_pte_mkwrite(). No functional change. Cc: linux-doc@vger.kernel.org Cc: linux-kernel@vger.kernel.org Cc: linux-alpha@vger.kernel.org Cc: linux-snps-arc@lists.infradead.org Cc: linux-arm-kernel@lists.infradead.org Cc: linux-csky@vger.kernel.org Cc: linux-hexagon@vger.kernel.org Cc: linux-ia64@vger.kernel.org Cc: loongarch@lists.linux.dev Cc: linux-m68k@lists.linux-m68k.org Cc: Michal Simek Cc: Dinh Nguyen Cc: linux-mips@vger.kernel.org Cc: linux-openrisc@vger.kernel.org Cc: linux-parisc@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org Cc: linux-riscv@lists.infradead.org Cc: linux-s390@vger.kernel.org Cc: linux-sh@vger.kernel.org Cc: sparclinux@vger.kernel.org Cc: linux-um@lists.infradead.org Cc: xen-devel@lists.xenproject.org Cc: linux-arch@vger.kernel.org Cc: linux-mm@kvack.org Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Acked-by: Michael Ellerman Acked-by: David Hildenbrand Reviewed-by: Kees Cook Suggested-by: David Hildenbrand Signed-off-by: Rick Edgecombe --- Hi Non-x86 Arch’s, x86 has a feature that allows for the creation of a special type of writable memory (shadow stack) that is only writable in limited specific ways. Previously, changes were proposed to core MM code to teach it to decide when to create normally writable memory or the special shadow stack writable memory, but David Hildenbrand suggested[0] to change pXX_mkwrite() to take a VMA, so awareness of shadow stack memory can be moved into x86 code. Since pXX_mkwrite() is defined in every arch, it requires some tree-wide changes. So that is why you are seeing some patches out of a big x86 series pop up in your arch mailing list. There is no functional change. After this refactor, the shadow stack series goes on to use the arch helpers to push shadow stack memory details inside arch/x86. Testing was just 0-day build testing. Hopefully that is enough context. Thanks! [0] https://lore.kernel.org/lkml/0e29a2d0-08d8-bcd6-ff26-4bea0e4037b0@redhat.com/#t v6: - New patch --- Documentation/mm/arch_pgtable_helpers.rst | 9 ++++++--- arch/alpha/include/asm/pgtable.h | 6 +++++- arch/arc/include/asm/hugepage.h | 2 +- arch/arc/include/asm/pgtable-bits-arcv2.h | 7 ++++++- arch/arm/include/asm/pgtable-3level.h | 7 ++++++- arch/arm/include/asm/pgtable.h | 2 +- arch/arm64/include/asm/pgtable.h | 4 ++-- arch/csky/include/asm/pgtable.h | 2 +- arch/hexagon/include/asm/pgtable.h | 2 +- arch/ia64/include/asm/pgtable.h | 2 +- arch/loongarch/include/asm/pgtable.h | 4 ++-- arch/m68k/include/asm/mcf_pgtable.h | 2 +- arch/m68k/include/asm/motorola_pgtable.h | 6 +++++- arch/m68k/include/asm/sun3_pgtable.h | 6 +++++- arch/microblaze/include/asm/pgtable.h | 2 +- arch/mips/include/asm/pgtable.h | 6 +++--- arch/nios2/include/asm/pgtable.h | 2 +- arch/openrisc/include/asm/pgtable.h | 2 +- arch/parisc/include/asm/pgtable.h | 6 +++++- arch/powerpc/include/asm/book3s/32/pgtable.h | 2 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 4 ++-- arch/powerpc/include/asm/nohash/32/pgtable.h | 2 +- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 2 +- arch/powerpc/include/asm/nohash/64/pgtable.h | 2 +- arch/riscv/include/asm/pgtable.h | 6 +++--- arch/s390/include/asm/hugetlb.h | 4 ++-- arch/s390/include/asm/pgtable.h | 4 ++-- arch/sh/include/asm/pgtable_32.h | 10 ++++++++-- arch/sparc/include/asm/pgtable_32.h | 2 +- arch/sparc/include/asm/pgtable_64.h | 6 +++--- arch/um/include/asm/pgtable.h | 2 +- arch/x86/include/asm/pgtable.h | 6 ++++-- arch/xtensa/include/asm/pgtable.h | 2 +- include/asm-generic/hugetlb.h | 4 ++-- include/linux/mm.h | 2 +- mm/debug_vm_pgtable.c | 16 ++++++++-------- mm/huge_memory.c | 6 +++--- mm/hugetlb.c | 4 ++-- mm/memory.c | 4 ++-- mm/migrate_device.c | 2 +- mm/mprotect.c | 2 +- mm/userfaultfd.c | 2 +- 42 files changed, 106 insertions(+), 69 deletions(-) diff --git a/Documentation/mm/arch_pgtable_helpers.rst b/Documentation/mm/arch_pgtable_helpers.rst index 30d9a09f01f4..78ac3ff2fe1d 100644 --- a/Documentation/mm/arch_pgtable_helpers.rst +++ b/Documentation/mm/arch_pgtable_helpers.rst @@ -46,7 +46,8 @@ PTE Page Table Helpers +---------------------------+--------------------------------------------------+ | pte_mkclean | Creates a clean PTE | +---------------------------+--------------------------------------------------+ -| pte_mkwrite | Creates a writable PTE | +| pte_mkwrite | Creates a writable PTE of the type specified by | +| | the VMA. | +---------------------------+--------------------------------------------------+ | pte_wrprotect | Creates a write protected PTE | +---------------------------+--------------------------------------------------+ @@ -118,7 +119,8 @@ PMD Page Table Helpers +---------------------------+--------------------------------------------------+ | pmd_mkclean | Creates a clean PMD | +---------------------------+--------------------------------------------------+ -| pmd_mkwrite | Creates a writable PMD | +| pmd_mkwrite | Creates a writable PMD of the type specified by | +| | the VMA. | +---------------------------+--------------------------------------------------+ | pmd_wrprotect | Creates a write protected PMD | +---------------------------+--------------------------------------------------+ @@ -222,7 +224,8 @@ HugeTLB Page Table Helpers +---------------------------+--------------------------------------------------+ | huge_pte_mkdirty | Creates a dirty HugeTLB | +---------------------------+--------------------------------------------------+ -| huge_pte_mkwrite | Creates a writable HugeTLB | +| huge_pte_mkwrite | Creates a writable HugeTLB of the type specified | +| | by the VMA. | +---------------------------+--------------------------------------------------+ | huge_pte_wrprotect | Creates a write protected HugeTLB | +---------------------------+--------------------------------------------------+ diff --git a/arch/alpha/include/asm/pgtable.h b/arch/alpha/include/asm/pgtable.h index ba43cb841d19..fb5d207c2a89 100644 --- a/arch/alpha/include/asm/pgtable.h +++ b/arch/alpha/include/asm/pgtable.h @@ -256,9 +256,13 @@ extern inline int pte_young(pte_t pte) { return pte_val(pte) & _PAGE_ACCESSED; extern inline pte_t pte_wrprotect(pte_t pte) { pte_val(pte) |= _PAGE_FOW; return pte; } extern inline pte_t pte_mkclean(pte_t pte) { pte_val(pte) &= ~(__DIRTY_BITS); return pte; } extern inline pte_t pte_mkold(pte_t pte) { pte_val(pte) &= ~(__ACCESS_BITS); return pte; } -extern inline pte_t pte_mkwrite(pte_t pte) { pte_val(pte) &= ~_PAGE_FOW; return pte; } extern inline pte_t pte_mkdirty(pte_t pte) { pte_val(pte) |= __DIRTY_BITS; return pte; } extern inline pte_t pte_mkyoung(pte_t pte) { pte_val(pte) |= __ACCESS_BITS; return pte; } +extern inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) +{ + pte_val(pte) &= ~_PAGE_FOW; + return pte; +} /* * The smp_rmb() in the following functions are required to order the load of diff --git a/arch/arc/include/asm/hugepage.h b/arch/arc/include/asm/hugepage.h index 5001b796fb8d..223a96967188 100644 --- a/arch/arc/include/asm/hugepage.h +++ b/arch/arc/include/asm/hugepage.h @@ -21,7 +21,7 @@ static inline pmd_t pte_pmd(pte_t pte) } #define pmd_wrprotect(pmd) pte_pmd(pte_wrprotect(pmd_pte(pmd))) -#define pmd_mkwrite(pmd) pte_pmd(pte_mkwrite(pmd_pte(pmd))) +#define pmd_mkwrite(pmd, vma) pte_pmd(pte_mkwrite(pmd_pte(pmd), (vma))) #define pmd_mkdirty(pmd) pte_pmd(pte_mkdirty(pmd_pte(pmd))) #define pmd_mkold(pmd) pte_pmd(pte_mkold(pmd_pte(pmd))) #define pmd_mkyoung(pmd) pte_pmd(pte_mkyoung(pmd_pte(pmd))) diff --git a/arch/arc/include/asm/pgtable-bits-arcv2.h b/arch/arc/include/asm/pgtable-bits-arcv2.h index 6e9f8ca6d6a1..a5b8bc955015 100644 --- a/arch/arc/include/asm/pgtable-bits-arcv2.h +++ b/arch/arc/include/asm/pgtable-bits-arcv2.h @@ -87,7 +87,6 @@ PTE_BIT_FUNC(mknotpresent, &= ~(_PAGE_PRESENT)); PTE_BIT_FUNC(wrprotect, &= ~(_PAGE_WRITE)); -PTE_BIT_FUNC(mkwrite, |= (_PAGE_WRITE)); PTE_BIT_FUNC(mkclean, &= ~(_PAGE_DIRTY)); PTE_BIT_FUNC(mkdirty, |= (_PAGE_DIRTY)); PTE_BIT_FUNC(mkold, &= ~(_PAGE_ACCESSED)); @@ -95,6 +94,12 @@ PTE_BIT_FUNC(mkyoung, |= (_PAGE_ACCESSED)); PTE_BIT_FUNC(mkspecial, |= (_PAGE_SPECIAL)); PTE_BIT_FUNC(mkhuge, |= (_PAGE_HW_SZ)); +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) +{ + pte_val(pte) |= (_PAGE_WRITE); + return pte; +} + static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) { return __pte((pte_val(pte) & _PAGE_CHG_MASK) | pgprot_val(newprot)); diff --git a/arch/arm/include/asm/pgtable-3level.h b/arch/arm/include/asm/pgtable-3level.h index 106049791500..df071a807610 100644 --- a/arch/arm/include/asm/pgtable-3level.h +++ b/arch/arm/include/asm/pgtable-3level.h @@ -202,11 +202,16 @@ static inline pmd_t pmd_##fn(pmd_t pmd) { pmd_val(pmd) op; return pmd; } PMD_BIT_FUNC(wrprotect, |= L_PMD_SECT_RDONLY); PMD_BIT_FUNC(mkold, &= ~PMD_SECT_AF); -PMD_BIT_FUNC(mkwrite, &= ~L_PMD_SECT_RDONLY); PMD_BIT_FUNC(mkdirty, |= L_PMD_SECT_DIRTY); PMD_BIT_FUNC(mkclean, &= ~L_PMD_SECT_DIRTY); PMD_BIT_FUNC(mkyoung, |= PMD_SECT_AF); +static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) +{ + pmd_val(pmd) |= L_PMD_SECT_RDONLY; + return pmd; +} + #define pmd_mkhuge(pmd) (__pmd(pmd_val(pmd) & ~PMD_TABLE_BIT)) #define pmd_pfn(pmd) (((pmd_val(pmd) & PMD_MASK) & PHYS_MASK) >> PAGE_SHIFT) diff --git a/arch/arm/include/asm/pgtable.h b/arch/arm/include/asm/pgtable.h index a58ccbb406ad..39ad1ae1308d 100644 --- a/arch/arm/include/asm/pgtable.h +++ b/arch/arm/include/asm/pgtable.h @@ -227,7 +227,7 @@ static inline pte_t pte_wrprotect(pte_t pte) return set_pte_bit(pte, __pgprot(L_PTE_RDONLY)); } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { return clear_pte_bit(pte, __pgprot(L_PTE_RDONLY)); } diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index cccf8885792e..913bf370f74a 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -187,7 +187,7 @@ static inline pte_t pte_mkwrite_kernel(pte_t pte) return pte; } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { return pte_mkwrite_kernel(pte); } @@ -492,7 +492,7 @@ static inline int pmd_trans_huge(pmd_t pmd) #define pmd_cont(pmd) pte_cont(pmd_pte(pmd)) #define pmd_wrprotect(pmd) pte_pmd(pte_wrprotect(pmd_pte(pmd))) #define pmd_mkold(pmd) pte_pmd(pte_mkold(pmd_pte(pmd))) -#define pmd_mkwrite(pmd) pte_pmd(pte_mkwrite(pmd_pte(pmd))) +#define pmd_mkwrite(pmd, vma) pte_pmd(pte_mkwrite(pmd_pte(pmd), (vma))) #define pmd_mkclean(pmd) pte_pmd(pte_mkclean(pmd_pte(pmd))) #define pmd_mkdirty(pmd) pte_pmd(pte_mkdirty(pmd_pte(pmd))) #define pmd_mkyoung(pmd) pte_pmd(pte_mkyoung(pmd_pte(pmd))) diff --git a/arch/csky/include/asm/pgtable.h b/arch/csky/include/asm/pgtable.h index d4042495febc..c2f92c991e37 100644 --- a/arch/csky/include/asm/pgtable.h +++ b/arch/csky/include/asm/pgtable.h @@ -176,7 +176,7 @@ static inline pte_t pte_mkold(pte_t pte) return pte; } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { pte_val(pte) |= _PAGE_WRITE; if (pte_val(pte) & _PAGE_MODIFIED) diff --git a/arch/hexagon/include/asm/pgtable.h b/arch/hexagon/include/asm/pgtable.h index 59393613d086..14ab9c789c0e 100644 --- a/arch/hexagon/include/asm/pgtable.h +++ b/arch/hexagon/include/asm/pgtable.h @@ -300,7 +300,7 @@ static inline pte_t pte_wrprotect(pte_t pte) } /* pte_mkwrite - mark page as writable */ -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { pte_val(pte) |= _PAGE_WRITE; return pte; diff --git a/arch/ia64/include/asm/pgtable.h b/arch/ia64/include/asm/pgtable.h index 21c97e31a28a..f879dd626da6 100644 --- a/arch/ia64/include/asm/pgtable.h +++ b/arch/ia64/include/asm/pgtable.h @@ -268,7 +268,7 @@ ia64_phys_addr_valid (unsigned long addr) * access rights: */ #define pte_wrprotect(pte) (__pte(pte_val(pte) & ~_PAGE_AR_RW)) -#define pte_mkwrite(pte) (__pte(pte_val(pte) | _PAGE_AR_RW)) +#define pte_mkwrite(pte, vma) (__pte(pte_val(pte) | _PAGE_AR_RW)) #define pte_mkold(pte) (__pte(pte_val(pte) & ~_PAGE_A)) #define pte_mkyoung(pte) (__pte(pte_val(pte) | _PAGE_A)) #define pte_mkclean(pte) (__pte(pte_val(pte) & ~_PAGE_D)) diff --git a/arch/loongarch/include/asm/pgtable.h b/arch/loongarch/include/asm/pgtable.h index d28fb9dbec59..ebf645f40298 100644 --- a/arch/loongarch/include/asm/pgtable.h +++ b/arch/loongarch/include/asm/pgtable.h @@ -390,7 +390,7 @@ static inline pte_t pte_mkdirty(pte_t pte) return pte; } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { pte_val(pte) |= _PAGE_WRITE; if (pte_val(pte) & _PAGE_MODIFIED) @@ -490,7 +490,7 @@ static inline int pmd_write(pmd_t pmd) return !!(pmd_val(pmd) & _PAGE_WRITE); } -static inline pmd_t pmd_mkwrite(pmd_t pmd) +static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) { pmd_val(pmd) |= _PAGE_WRITE; if (pmd_val(pmd) & _PAGE_MODIFIED) diff --git a/arch/m68k/include/asm/mcf_pgtable.h b/arch/m68k/include/asm/mcf_pgtable.h index 13741c1245e1..37d77e055016 100644 --- a/arch/m68k/include/asm/mcf_pgtable.h +++ b/arch/m68k/include/asm/mcf_pgtable.h @@ -211,7 +211,7 @@ static inline pte_t pte_mkold(pte_t pte) return pte; } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { pte_val(pte) |= CF_PAGE_WRITABLE; return pte; diff --git a/arch/m68k/include/asm/motorola_pgtable.h b/arch/m68k/include/asm/motorola_pgtable.h index ec0dc19ab834..c4e8eb76286d 100644 --- a/arch/m68k/include/asm/motorola_pgtable.h +++ b/arch/m68k/include/asm/motorola_pgtable.h @@ -155,7 +155,6 @@ static inline int pte_young(pte_t pte) { return pte_val(pte) & _PAGE_ACCESSED; static inline pte_t pte_wrprotect(pte_t pte) { pte_val(pte) |= _PAGE_RONLY; return pte; } static inline pte_t pte_mkclean(pte_t pte) { pte_val(pte) &= ~_PAGE_DIRTY; return pte; } static inline pte_t pte_mkold(pte_t pte) { pte_val(pte) &= ~_PAGE_ACCESSED; return pte; } -static inline pte_t pte_mkwrite(pte_t pte) { pte_val(pte) &= ~_PAGE_RONLY; return pte; } static inline pte_t pte_mkdirty(pte_t pte) { pte_val(pte) |= _PAGE_DIRTY; return pte; } static inline pte_t pte_mkyoung(pte_t pte) { pte_val(pte) |= _PAGE_ACCESSED; return pte; } static inline pte_t pte_mknocache(pte_t pte) @@ -168,6 +167,11 @@ static inline pte_t pte_mkcache(pte_t pte) pte_val(pte) = (pte_val(pte) & _CACHEMASK040) | m68k_supervisor_cachemode; return pte; } +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) +{ + pte_val(pte) &= ~_PAGE_RONLY; + return pte; +} #define swapper_pg_dir kernel_pg_dir extern pgd_t kernel_pg_dir[128]; diff --git a/arch/m68k/include/asm/sun3_pgtable.h b/arch/m68k/include/asm/sun3_pgtable.h index e582b0484a55..2a06bea51a1e 100644 --- a/arch/m68k/include/asm/sun3_pgtable.h +++ b/arch/m68k/include/asm/sun3_pgtable.h @@ -143,10 +143,14 @@ static inline int pte_young(pte_t pte) { return pte_val(pte) & SUN3_PAGE_ACCESS static inline pte_t pte_wrprotect(pte_t pte) { pte_val(pte) &= ~SUN3_PAGE_WRITEABLE; return pte; } static inline pte_t pte_mkclean(pte_t pte) { pte_val(pte) &= ~SUN3_PAGE_MODIFIED; return pte; } static inline pte_t pte_mkold(pte_t pte) { pte_val(pte) &= ~SUN3_PAGE_ACCESSED; return pte; } -static inline pte_t pte_mkwrite(pte_t pte) { pte_val(pte) |= SUN3_PAGE_WRITEABLE; return pte; } static inline pte_t pte_mkdirty(pte_t pte) { pte_val(pte) |= SUN3_PAGE_MODIFIED; return pte; } static inline pte_t pte_mkyoung(pte_t pte) { pte_val(pte) |= SUN3_PAGE_ACCESSED; return pte; } static inline pte_t pte_mknocache(pte_t pte) { pte_val(pte) |= SUN3_PAGE_NOCACHE; return pte; } +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) +{ + pte_val(pte) |= SUN3_PAGE_WRITEABLE; + return pte; +} // use this version when caches work... //static inline pte_t pte_mkcache(pte_t pte) { pte_val(pte) &= SUN3_PAGE_NOCACHE; return pte; } // until then, use: diff --git a/arch/microblaze/include/asm/pgtable.h b/arch/microblaze/include/asm/pgtable.h index d1b8272abcd9..5b83e82f8d7e 100644 --- a/arch/microblaze/include/asm/pgtable.h +++ b/arch/microblaze/include/asm/pgtable.h @@ -266,7 +266,7 @@ static inline pte_t pte_mkread(pte_t pte) \ { pte_val(pte) |= _PAGE_USER; return pte; } static inline pte_t pte_mkexec(pte_t pte) \ { pte_val(pte) |= _PAGE_USER | _PAGE_EXEC; return pte; } -static inline pte_t pte_mkwrite(pte_t pte) \ +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) \ { pte_val(pte) |= _PAGE_RW; return pte; } static inline pte_t pte_mkdirty(pte_t pte) \ { pte_val(pte) |= _PAGE_DIRTY; return pte; } diff --git a/arch/mips/include/asm/pgtable.h b/arch/mips/include/asm/pgtable.h index 791389bf3c12..06efd567144a 100644 --- a/arch/mips/include/asm/pgtable.h +++ b/arch/mips/include/asm/pgtable.h @@ -309,7 +309,7 @@ static inline pte_t pte_mkold(pte_t pte) return pte; } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { pte.pte_low |= _PAGE_WRITE; if (pte.pte_low & _PAGE_MODIFIED) { @@ -364,7 +364,7 @@ static inline pte_t pte_mkold(pte_t pte) return pte; } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { pte_val(pte) |= _PAGE_WRITE; if (pte_val(pte) & _PAGE_MODIFIED) @@ -626,7 +626,7 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd) return pmd; } -static inline pmd_t pmd_mkwrite(pmd_t pmd) +static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) { pmd_val(pmd) |= _PAGE_WRITE; if (pmd_val(pmd) & _PAGE_MODIFIED) diff --git a/arch/nios2/include/asm/pgtable.h b/arch/nios2/include/asm/pgtable.h index 0f5c2564e9f5..edd458518e0e 100644 --- a/arch/nios2/include/asm/pgtable.h +++ b/arch/nios2/include/asm/pgtable.h @@ -129,7 +129,7 @@ static inline pte_t pte_mkold(pte_t pte) return pte; } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { pte_val(pte) |= _PAGE_WRITE; return pte; diff --git a/arch/openrisc/include/asm/pgtable.h b/arch/openrisc/include/asm/pgtable.h index 3eb9b9555d0d..fd40aec189d1 100644 --- a/arch/openrisc/include/asm/pgtable.h +++ b/arch/openrisc/include/asm/pgtable.h @@ -250,7 +250,7 @@ static inline pte_t pte_mkold(pte_t pte) return pte; } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { pte_val(pte) |= _PAGE_WRITE; return pte; diff --git a/arch/parisc/include/asm/pgtable.h b/arch/parisc/include/asm/pgtable.h index e2950f5db7c9..89f62137e67f 100644 --- a/arch/parisc/include/asm/pgtable.h +++ b/arch/parisc/include/asm/pgtable.h @@ -331,8 +331,12 @@ static inline pte_t pte_mkold(pte_t pte) { pte_val(pte) &= ~_PAGE_ACCESSED; retu static inline pte_t pte_wrprotect(pte_t pte) { pte_val(pte) &= ~_PAGE_WRITE; return pte; } static inline pte_t pte_mkdirty(pte_t pte) { pte_val(pte) |= _PAGE_DIRTY; return pte; } static inline pte_t pte_mkyoung(pte_t pte) { pte_val(pte) |= _PAGE_ACCESSED; return pte; } -static inline pte_t pte_mkwrite(pte_t pte) { pte_val(pte) |= _PAGE_WRITE; return pte; } static inline pte_t pte_mkspecial(pte_t pte) { pte_val(pte) |= _PAGE_SPECIAL; return pte; } +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) +{ + pte_val(pte) |= _PAGE_WRITE; + return pte; +} /* * Huge pte definitions. diff --git a/arch/powerpc/include/asm/book3s/32/pgtable.h b/arch/powerpc/include/asm/book3s/32/pgtable.h index 7bf1fe7297c6..10d9a1d2aca9 100644 --- a/arch/powerpc/include/asm/book3s/32/pgtable.h +++ b/arch/powerpc/include/asm/book3s/32/pgtable.h @@ -498,7 +498,7 @@ static inline pte_t pte_mkpte(pte_t pte) return pte; } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { return __pte(pte_val(pte) | _PAGE_RW); } diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index 4acc9690f599..be0636522d36 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -600,7 +600,7 @@ static inline pte_t pte_mkexec(pte_t pte) return __pte_raw(pte_raw(pte) | cpu_to_be64(_PAGE_EXEC)); } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { /* * write implies read, hence set both @@ -1071,7 +1071,7 @@ static inline pte_t *pmdp_ptep(pmd_t *pmd) #define pmd_mkdirty(pmd) pte_pmd(pte_mkdirty(pmd_pte(pmd))) #define pmd_mkclean(pmd) pte_pmd(pte_mkclean(pmd_pte(pmd))) #define pmd_mkyoung(pmd) pte_pmd(pte_mkyoung(pmd_pte(pmd))) -#define pmd_mkwrite(pmd) pte_pmd(pte_mkwrite(pmd_pte(pmd))) +#define pmd_mkwrite(pmd, vma) pte_pmd(pte_mkwrite(pmd_pte(pmd), (vma))) #ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY #define pmd_soft_dirty(pmd) pte_soft_dirty(pmd_pte(pmd)) diff --git a/arch/powerpc/include/asm/nohash/32/pgtable.h b/arch/powerpc/include/asm/nohash/32/pgtable.h index fec56d965f00..7bfbcb9ba55b 100644 --- a/arch/powerpc/include/asm/nohash/32/pgtable.h +++ b/arch/powerpc/include/asm/nohash/32/pgtable.h @@ -171,7 +171,7 @@ void unmap_kernel_page(unsigned long va); do { pte_update(mm, addr, ptep, ~0, 0, 0); } while (0) #ifndef pte_mkwrite -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { return __pte(pte_val(pte) | _PAGE_RW); } diff --git a/arch/powerpc/include/asm/nohash/32/pte-8xx.h b/arch/powerpc/include/asm/nohash/32/pte-8xx.h index 1a89ebdc3acc..f32450eb270a 100644 --- a/arch/powerpc/include/asm/nohash/32/pte-8xx.h +++ b/arch/powerpc/include/asm/nohash/32/pte-8xx.h @@ -101,7 +101,7 @@ static inline int pte_write(pte_t pte) #define pte_write pte_write -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { return __pte(pte_val(pte) & ~_PAGE_RO); } diff --git a/arch/powerpc/include/asm/nohash/64/pgtable.h b/arch/powerpc/include/asm/nohash/64/pgtable.h index 287e25864ffa..589009555877 100644 --- a/arch/powerpc/include/asm/nohash/64/pgtable.h +++ b/arch/powerpc/include/asm/nohash/64/pgtable.h @@ -85,7 +85,7 @@ #ifndef __ASSEMBLY__ /* pte_clear moved to later in this file */ -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { return __pte(pte_val(pte) | _PAGE_RW); } diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index d8d8de0ded99..fed1b81fbe07 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -338,7 +338,7 @@ static inline pte_t pte_wrprotect(pte_t pte) /* static inline pte_t pte_mkread(pte_t pte) */ -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { return __pte(pte_val(pte) | _PAGE_WRITE); } @@ -624,9 +624,9 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd) return pte_pmd(pte_mkyoung(pmd_pte(pmd))); } -static inline pmd_t pmd_mkwrite(pmd_t pmd) +static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) { - return pte_pmd(pte_mkwrite(pmd_pte(pmd))); + return pte_pmd(pte_mkwrite(pmd_pte(pmd), vma)); } static inline pmd_t pmd_wrprotect(pmd_t pmd) diff --git a/arch/s390/include/asm/hugetlb.h b/arch/s390/include/asm/hugetlb.h index ccdbccfde148..558f7eef9c4d 100644 --- a/arch/s390/include/asm/hugetlb.h +++ b/arch/s390/include/asm/hugetlb.h @@ -102,9 +102,9 @@ static inline int huge_pte_dirty(pte_t pte) return pte_dirty(pte); } -static inline pte_t huge_pte_mkwrite(pte_t pte) +static inline pte_t huge_pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { - return pte_mkwrite(pte); + return pte_mkwrite(pte, vma); } static inline pte_t huge_pte_mkdirty(pte_t pte) diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index deeb918cae1d..8f2c743da0eb 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -1013,7 +1013,7 @@ static inline pte_t pte_mkwrite_kernel(pte_t pte) return pte; } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { return pte_mkwrite_kernel(pte); } @@ -1499,7 +1499,7 @@ static inline pmd_t pmd_mkwrite_kernel(pmd_t pmd) return pmd; } -static inline pmd_t pmd_mkwrite(pmd_t pmd) +static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) { return pmd_mkwrite_kernel(pmd); } diff --git a/arch/sh/include/asm/pgtable_32.h b/arch/sh/include/asm/pgtable_32.h index 21952b094650..9f2dcb9eafc8 100644 --- a/arch/sh/include/asm/pgtable_32.h +++ b/arch/sh/include/asm/pgtable_32.h @@ -351,6 +351,12 @@ static inline void set_pte(pte_t *ptep, pte_t pte) #define PTE_BIT_FUNC(h,fn,op) \ static inline pte_t pte_##fn(pte_t pte) { pte.pte_##h op; return pte; } +#define PTE_BIT_FUNC_VMA(h,fn,op) \ +static inline pte_t pte_##fn(pte_t pte, struct vm_area_struct *vma) \ +{ \ + pte.pte_##h op; \ + return pte; \ +} #ifdef CONFIG_X2TLB /* @@ -359,11 +365,11 @@ static inline pte_t pte_##fn(pte_t pte) { pte.pte_##h op; return pte; } * kernel permissions), we attempt to couple them a bit more sanely here. */ PTE_BIT_FUNC(high, wrprotect, &= ~(_PAGE_EXT_USER_WRITE | _PAGE_EXT_KERN_WRITE)); -PTE_BIT_FUNC(high, mkwrite, |= _PAGE_EXT_USER_WRITE | _PAGE_EXT_KERN_WRITE); +PTE_BIT_FUNC_VMA(high, mkwrite, |= _PAGE_EXT_USER_WRITE | _PAGE_EXT_KERN_WRITE); PTE_BIT_FUNC(high, mkhuge, |= _PAGE_SZHUGE); #else PTE_BIT_FUNC(low, wrprotect, &= ~_PAGE_RW); -PTE_BIT_FUNC(low, mkwrite, |= _PAGE_RW); +PTE_BIT_FUNC_VMA(low, mkwrite, |= _PAGE_RW); PTE_BIT_FUNC(low, mkhuge, |= _PAGE_SZHUGE); #endif diff --git a/arch/sparc/include/asm/pgtable_32.h b/arch/sparc/include/asm/pgtable_32.h index d4330e3c57a6..3e8836179456 100644 --- a/arch/sparc/include/asm/pgtable_32.h +++ b/arch/sparc/include/asm/pgtable_32.h @@ -241,7 +241,7 @@ static inline pte_t pte_mkold(pte_t pte) return __pte(pte_val(pte) & ~SRMMU_REF); } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { return __pte(pte_val(pte) | SRMMU_WRITE); } diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h index 2dc8d4641734..c5cd5c03f557 100644 --- a/arch/sparc/include/asm/pgtable_64.h +++ b/arch/sparc/include/asm/pgtable_64.h @@ -466,7 +466,7 @@ static inline pte_t pte_mkclean(pte_t pte) return __pte(val); } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { unsigned long val = pte_val(pte), mask; @@ -756,11 +756,11 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd) return __pmd(pte_val(pte)); } -static inline pmd_t pmd_mkwrite(pmd_t pmd) +static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) { pte_t pte = __pte(pmd_val(pmd)); - pte = pte_mkwrite(pte); + pte = pte_mkwrite(pte, vma); return __pmd(pte_val(pte)); } diff --git a/arch/um/include/asm/pgtable.h b/arch/um/include/asm/pgtable.h index a70d1618eb35..963479c133b7 100644 --- a/arch/um/include/asm/pgtable.h +++ b/arch/um/include/asm/pgtable.h @@ -207,7 +207,7 @@ static inline pte_t pte_mkyoung(pte_t pte) return(pte); } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { if (unlikely(pte_get_bits(pte, _PAGE_RW))) return pte; diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 3607f2572f9e..66c514808276 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -369,7 +369,9 @@ static inline pte_t pte_mkwrite_kernel(pte_t pte) return pte_set_flags(pte, _PAGE_RW); } -static inline pte_t pte_mkwrite(pte_t pte) +struct vm_area_struct; + +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { return pte_mkwrite_kernel(pte); } @@ -470,7 +472,7 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd) return pmd_set_flags(pmd, _PAGE_ACCESSED); } -static inline pmd_t pmd_mkwrite(pmd_t pmd) +static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) { return pmd_set_flags(pmd, _PAGE_RW); } diff --git a/arch/xtensa/include/asm/pgtable.h b/arch/xtensa/include/asm/pgtable.h index fc7a14884c6c..d72632d9c53c 100644 --- a/arch/xtensa/include/asm/pgtable.h +++ b/arch/xtensa/include/asm/pgtable.h @@ -262,7 +262,7 @@ static inline pte_t pte_mkdirty(pte_t pte) { pte_val(pte) |= _PAGE_DIRTY; return pte; } static inline pte_t pte_mkyoung(pte_t pte) { pte_val(pte) |= _PAGE_ACCESSED; return pte; } -static inline pte_t pte_mkwrite(pte_t pte) +static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { pte_val(pte) |= _PAGE_WRITABLE; return pte; } #define pgprot_noncached(prot) \ diff --git a/include/asm-generic/hugetlb.h b/include/asm-generic/hugetlb.h index d7f6335d3999..e86c830728de 100644 --- a/include/asm-generic/hugetlb.h +++ b/include/asm-generic/hugetlb.h @@ -20,9 +20,9 @@ static inline unsigned long huge_pte_dirty(pte_t pte) return pte_dirty(pte); } -static inline pte_t huge_pte_mkwrite(pte_t pte) +static inline pte_t huge_pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { - return pte_mkwrite(pte); + return pte_mkwrite(pte, vma); } #ifndef __HAVE_ARCH_HUGE_PTE_WRPROTECT diff --git a/include/linux/mm.h b/include/linux/mm.h index 1f79667824eb..af652444fbba 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1163,7 +1163,7 @@ void free_compound_page(struct page *page); static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma) { if (likely(vma->vm_flags & VM_WRITE)) - pte = pte_mkwrite(pte); + pte = pte_mkwrite(pte, vma); return pte; } diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c index af59cc7bd307..7bc5592900bc 100644 --- a/mm/debug_vm_pgtable.c +++ b/mm/debug_vm_pgtable.c @@ -109,10 +109,10 @@ static void __init pte_basic_tests(struct pgtable_debug_args *args, int idx) WARN_ON(!pte_same(pte, pte)); WARN_ON(!pte_young(pte_mkyoung(pte_mkold(pte)))); WARN_ON(!pte_dirty(pte_mkdirty(pte_mkclean(pte)))); - WARN_ON(!pte_write(pte_mkwrite(pte_wrprotect(pte)))); + WARN_ON(!pte_write(pte_mkwrite(pte_wrprotect(pte), args->vma))); WARN_ON(pte_young(pte_mkold(pte_mkyoung(pte)))); WARN_ON(pte_dirty(pte_mkclean(pte_mkdirty(pte)))); - WARN_ON(pte_write(pte_wrprotect(pte_mkwrite(pte)))); + WARN_ON(pte_write(pte_wrprotect(pte_mkwrite(pte, args->vma)))); WARN_ON(pte_dirty(pte_wrprotect(pte_mkclean(pte)))); WARN_ON(!pte_dirty(pte_wrprotect(pte_mkdirty(pte)))); } @@ -153,7 +153,7 @@ static void __init pte_advanced_tests(struct pgtable_debug_args *args) pte = pte_mkclean(pte); set_pte_at(args->mm, args->vaddr, args->ptep, pte); flush_dcache_page(page); - pte = pte_mkwrite(pte); + pte = pte_mkwrite(pte, args->vma); pte = pte_mkdirty(pte); ptep_set_access_flags(args->vma, args->vaddr, args->ptep, pte, 1); pte = ptep_get(args->ptep); @@ -199,10 +199,10 @@ static void __init pmd_basic_tests(struct pgtable_debug_args *args, int idx) WARN_ON(!pmd_same(pmd, pmd)); WARN_ON(!pmd_young(pmd_mkyoung(pmd_mkold(pmd)))); WARN_ON(!pmd_dirty(pmd_mkdirty(pmd_mkclean(pmd)))); - WARN_ON(!pmd_write(pmd_mkwrite(pmd_wrprotect(pmd)))); + WARN_ON(!pmd_write(pmd_mkwrite(pmd_wrprotect(pmd), args->vma))); WARN_ON(pmd_young(pmd_mkold(pmd_mkyoung(pmd)))); WARN_ON(pmd_dirty(pmd_mkclean(pmd_mkdirty(pmd)))); - WARN_ON(pmd_write(pmd_wrprotect(pmd_mkwrite(pmd)))); + WARN_ON(pmd_write(pmd_wrprotect(pmd_mkwrite(pmd, args->vma)))); WARN_ON(pmd_dirty(pmd_wrprotect(pmd_mkclean(pmd)))); WARN_ON(!pmd_dirty(pmd_wrprotect(pmd_mkdirty(pmd)))); /* @@ -253,7 +253,7 @@ static void __init pmd_advanced_tests(struct pgtable_debug_args *args) pmd = pmd_mkclean(pmd); set_pmd_at(args->mm, vaddr, args->pmdp, pmd); flush_dcache_page(page); - pmd = pmd_mkwrite(pmd); + pmd = pmd_mkwrite(pmd, args->vma); pmd = pmd_mkdirty(pmd); pmdp_set_access_flags(args->vma, vaddr, args->pmdp, pmd, 1); pmd = READ_ONCE(*args->pmdp); @@ -928,8 +928,8 @@ static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) pte = mk_huge_pte(page, args->page_prot); WARN_ON(!huge_pte_dirty(huge_pte_mkdirty(pte))); - WARN_ON(!huge_pte_write(huge_pte_mkwrite(huge_pte_wrprotect(pte)))); - WARN_ON(huge_pte_write(huge_pte_wrprotect(huge_pte_mkwrite(pte)))); + WARN_ON(!huge_pte_write(huge_pte_mkwrite(huge_pte_wrprotect(pte), args->vma))); + WARN_ON(huge_pte_write(huge_pte_wrprotect(huge_pte_mkwrite(pte, args->vma)))); #ifdef CONFIG_ARCH_WANT_GENERAL_HUGETLB pte = pfn_pte(args->fixed_pmd_pfn, args->page_prot); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 4fc43859e59a..aaf815838144 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -555,7 +555,7 @@ __setup("transparent_hugepage=", setup_transparent_hugepage); pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) { if (likely(vma->vm_flags & VM_WRITE)) - pmd = pmd_mkwrite(pmd); + pmd = pmd_mkwrite(pmd, vma); return pmd; } @@ -1580,7 +1580,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) pmd = pmd_modify(oldpmd, vma->vm_page_prot); pmd = pmd_mkyoung(pmd); if (writable) - pmd = pmd_mkwrite(pmd); + pmd = pmd_mkwrite(pmd, vma); set_pmd_at(vma->vm_mm, haddr, vmf->pmd, pmd); update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); spin_unlock(vmf->ptl); @@ -1926,7 +1926,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, /* See change_pte_range(). */ if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && !pmd_write(entry) && can_change_pmd_writable(vma, addr, entry)) - entry = pmd_mkwrite(entry); + entry = pmd_mkwrite(entry, vma); ret = HPAGE_PMD_NR; set_pmd_at(mm, addr, pmd, entry); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 07abcb6eb203..6af471bdcff8 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4900,7 +4900,7 @@ static pte_t make_huge_pte(struct vm_area_struct *vma, struct page *page, if (writable) { entry = huge_pte_mkwrite(huge_pte_mkdirty(mk_huge_pte(page, - vma->vm_page_prot))); + vma->vm_page_prot)), vma); } else { entry = huge_pte_wrprotect(mk_huge_pte(page, vma->vm_page_prot)); @@ -4916,7 +4916,7 @@ static void set_huge_ptep_writable(struct vm_area_struct *vma, { pte_t entry; - entry = huge_pte_mkwrite(huge_pte_mkdirty(huge_ptep_get(ptep))); + entry = huge_pte_mkwrite(huge_pte_mkdirty(huge_ptep_get(ptep)), vma); if (huge_ptep_set_access_flags(vma, address, ptep, entry, 1)) update_mmu_cache(vma, address, ptep); } diff --git a/mm/memory.c b/mm/memory.c index f456f3b5049c..d0972d2d6f36 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4067,7 +4067,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) entry = mk_pte(&folio->page, vma->vm_page_prot); entry = pte_sw_mkyoung(entry); if (vma->vm_flags & VM_WRITE) - entry = pte_mkwrite(pte_mkdirty(entry)); + entry = pte_mkwrite(pte_mkdirty(entry), vma); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); @@ -4755,7 +4755,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) pte = pte_modify(old_pte, vma->vm_page_prot); pte = pte_mkyoung(pte); if (writable) - pte = pte_mkwrite(pte); + pte = pte_mkwrite(pte, vma); ptep_modify_prot_commit(vma, vmf->address, vmf->pte, old_pte, pte); update_mmu_cache(vma, vmf->address, vmf->pte); pte_unmap_unlock(vmf->pte, vmf->ptl); diff --git a/mm/migrate_device.c b/mm/migrate_device.c index d30c9de60b0d..df3f5e9d5f76 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -646,7 +646,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, } entry = mk_pte(page, vma->vm_page_prot); if (vma->vm_flags & VM_WRITE) - entry = pte_mkwrite(pte_mkdirty(entry)); + entry = pte_mkwrite(pte_mkdirty(entry), vma); } ptep = pte_offset_map_lock(mm, pmdp, addr, &ptl); diff --git a/mm/mprotect.c b/mm/mprotect.c index 1d4843c97c2a..381163a41e88 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -198,7 +198,7 @@ static long change_pte_range(struct mmu_gather *tlb, if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && !pte_write(ptent) && can_change_pte_writable(vma, addr, ptent)) - ptent = pte_mkwrite(ptent); + ptent = pte_mkwrite(ptent, vma); ptep_modify_prot_commit(vma, addr, pte, oldpte, ptent); if (pte_needs_flush(oldpte, ptent)) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 53c3d916ff66..3db6f87c0aca 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -75,7 +75,7 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, if (page_in_cache && !vm_shared) writable = false; if (writable) - _dst_pte = pte_mkwrite(_dst_pte); + _dst_pte = pte_mkwrite(_dst_pte, dst_vma); if (wp_copy) _dst_pte = pte_mkuffd_wp(_dst_pte); From patchwork Mon Feb 27 22:29:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154246 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 837A4C64ED9 for ; Mon, 27 Feb 2023 22:31:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9BCBC6B0089; Mon, 27 Feb 2023 17:31:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 905EC6B0093; Mon, 27 Feb 2023 17:31:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D3B26B008A; Mon, 27 Feb 2023 17:31:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 3DB0D6B008C for ; Mon, 27 Feb 2023 17:31:45 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E91411A07C0 for ; Mon, 27 Feb 2023 22:31:44 +0000 (UTC) X-FDA: 80514520128.08.EB76C0F Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf07.hostedemail.com (Postfix) with ESMTP id D081440005 for ; Mon, 27 Feb 2023 22:31:42 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=hawtsNaS; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537103; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=ORlNTzzAb+0Pu9bloKiS26koMHV7fWpkcUCLy6J+J0E=; b=gEht1Tjk0oPh8Z/unA/3MLk+Lhm+8WzYy6xl/L4y/qvFurxSYOkmlom1aUpAJUoMyLWaeG 1B0tb16c3tl2TFv0GK7QBiksz/1BbhQwciX1ejGF+qAdG6vBMd5SRAXclczoMBB5cDzGiA /NrCA/qCxXbK0w5WAKVhSRYuCdOMTqo= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=hawtsNaS; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537103; a=rsa-sha256; cv=none; b=2TeFlXFyzupd9/MqU4wvPSibRkvX/6hlVahDNA0JGrYXIXi12GWZzFy1PgtyOAu6Fn5+B6 FNvL0Sf5x0oeIqodlkGBZ+EPNAvIE+xgKOMLcnitZIks7Uuoqc35tOo1mQs60pVBkICIpD wkcOpE7bDWLxNKbhkZQZFodQT1xSsKw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537102; x=1709073102; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=GSuTSaHBIXzx6uwemEhog5YXgFQyGTsYLIh2QPWPoMs=; b=hawtsNaSACylLBfHxzgKe5pKmDXVxliCGJ9b10UyUlrw0oMu6fLcDuas Y/XLAun28bbIn9qpbMp6Y4rcnU714xCUjVOa3hoUg+Ni+5xFHJ1c2gy8F GK7cVsqVxUeOwPKrSxMbDN9AQBMECf85bOw+pDr7MdZgHumSSCkXfpk3K /X/9ZcDP43za3sq3J/wfBINFGPfTkz44PtvKmi6CFK43tbnzbfLR9vi9P F116GP5+JwwtW6p1dD4/Ua76Hw4exIbcgMHvpChCjzfzCMgUX03kc1HLi fWkAEcdsqc4QDzBzTTdt7PxKKOx9tqHtLYllR8wit3i25dQTCa8d9rpbR Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657372" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657372" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:23 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024565" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024565" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:17 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 14/41] x86/mm: Introduce _PAGE_SAVED_DIRTY Date: Mon, 27 Feb 2023 14:29:30 -0800 Message-Id: <20230227222957.24501-15-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: D081440005 X-Stat-Signature: ef3f6ppohgw3s1ci48chojaqha3fikek X-HE-Tag: 1677537102-862031 X-HE-Meta: U2FsdGVkX1+koUz1FOziM4dClKjOVR8l6uVT9SeTr3IxanfFm2aTHagmK/qylFp8ix6oZulYrhbHbwbacts8VTJWkLILgCiny2ihVmvfW1o3p87uGiYXjanitz1UNrLwOCYJw0kaG6daFyLMrAwIAY25j26C2A12IQhMRedXdGB4U5z6aYl801ZmlTkBiIoQoLkrmDtaR2yRr9vIlZ+dUBqcaHEYUhI0sVstkKCsW0nZNYSTFL4uvk9ZOVcspORwotlChRS7YFvrArEflcvk2A1lwvoK3T8u9Q4LLZPmQtSb1DzBqrQ/MLHdDCKkYBQbtNcCsPC2smbrzYRzZ9r3inTJgWZImViWAEi1sGCknuUD32/crNJ7R81eJvWIJc54codkuVZCbYFPp32Z3ijU0CXq3VKoRpLWcsWjYI0bfhQDs5D5W7Tp2Ngvj40SW5sJP3/eNr83gBXiCa7eN1ZFTXLJq3oQPMhFn5yx4ze/qnPvD4NdM3PvdCpG1bmEOl505zFDx3kC4QnxzEg8abvzEhkos9jebEcG7o+uk8fTAzRpmrwhZdvon1IfpUFASkI4dhCoaxNgtMpZhynuHLa3G34aZEXxXf0PmHAYnXCjyQim7DJDMOEUjmS/fSE/xM8+MjA2KgM5tAXrZ3PfcaBm1XrNqX0bNzlnirrTbR3ZEEo0GI/+fDCranQYVViRL+hnlmx5vimA+MuFykxh10npe9dwJG4wpMBspPYxHHxyiZNDcy2ooDnXvU9YneifVFjaWdQiFlSpxSTlmn53XRVZDGcmjooX83HLi2PJ5sIWNrrsJY0vjK4qC5V57XUdZwUKFEPOR5n6IHLaEvcTdujgdWV92xvSYE7SzPAcMdK43GFZiI/U1vj2Ufekon3Ee36Q1GvFRuXWgJptbe21wc9ls6/cvX4z7KvOrSy6nabNz08wgQiT6VM64m7vbfe1B4oy6D5XkW7aOj2Ui3nGPuu Pb6SItXf V/QwMIbNO2V8LsaqQIrQoDV64fwtd7LqqrLZjoLs21YUq0GnnosYJWF8FHGz86jAHzRf6Gdz4KEnID0piqS/MgKIWpPWUcfoImiHSab5QZWkzYeEsCGYpfJ+HhUQcfyw5nGQnEZcylvCHfVsm73AnpcvA/fQ2K7lCCDky1VqaTP+xMc/mn3bbrEWfW0P7AXzS6ahgd3R85N1C/ozS1Q3EPgIyIFS4BG7zLCSQ2EY/r40FIcz/wE+QIke2Id21AFVfykuXs+L6us7tolZJaEniXuKbHvd9hf72BgJIVZ8cLNTu2Sz8/Sxrg0X89nMFZklIeU0q+a73zvjSyaSZa5nkf7bk47/8x2el99PfFlWFNVW+zKLcwcjKXcR45srqd+9uRVKfb8S5iaxIc34= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some OSes have a greater dependence on software available bits in PTEs than Linux. That left the hardware architects looking for a way to represent a new memory type (shadow stack) within the existing bits. They chose to repurpose a lightly-used state: Write=0,Dirty=1. So in order to support shadow stack memory, Linux should avoid creating memory with this PTE bit combination unless it intends for it to be shadow stack. The reason it's lightly used is that Dirty=1 is normally set by HW _before_ a write. A write with a Write=0 PTE would typically only generate a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1 *and* generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware which supports shadow stacks will no longer exhibit this oddity. So that leaves Write=0,Dirty=1 PTEs created in software. To avoid inadvertently created shadow stack memory, in places where Linux normally creates Write=0,Dirty=1, it can use the software-defined _PAGE_SAVED_DIRTY in place of the hardware _PAGE_DIRTY. In other words, whenever Linux needs to create Write=0,Dirty=1, it instead creates Write=0,SavedDirty=1 except for shadow stack, which is Write=0,Dirty=1. There are six bits left available to software in the 64-bit PTE after consuming a bit for _PAGE_SAVED_DIRTY. No space is consumed in 32-bit kernels because shadow stacks are not enabled there. Implement only the infrastructure for _PAGE_SAVED_DIRTY. Changes to actually begin creating _PAGE_SAVED_DIRTY PTEs will follow once other pieces are in place. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Co-developed-by: Yu-cheng Yu Signed-off-by: Yu-cheng Yu Signed-off-by: Rick Edgecombe --- v7: - Use lightly edited comment verbiage from (David Hildenbrand) - Update commit log to reduce verbosity (David Hildenbrand) v6: - Rename _PAGE_COW to _PAGE_SAVED_DIRTY (David Hildenbrand) - Add _PAGE_SAVED_DIRTY to _PAGE_CHG_MASK v5: - Fix log, comments and whitespace (Boris) - Remove capitalization on shadow stack (Boris) v4: - Teach pte_flags_need_flush() about _PAGE_COW bit - Break apart patch for better bisectability v3: - Add comment around _PAGE_TABLE in response to comment from (Andrew Cooper) - Check for PSE in pmd_shstk (Andrew Cooper) - Get to the point quicker in commit log (Andrew Cooper) - Clarify and reorder commit log for why the PTE bit examples have multiple entries. Apply same changes for comment. (peterz) - Fix comment that implied dirty bit for COW was a specific x86 thing (peterz) - Fix swapping of Write/Dirty (PeterZ) --- arch/x86/include/asm/pgtable.h | 79 ++++++++++++++++++++++++++++ arch/x86/include/asm/pgtable_types.h | 50 +++++++++++++++--- arch/x86/include/asm/tlbflush.h | 3 +- 3 files changed, 123 insertions(+), 9 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 66c514808276..7360783f2140 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -301,6 +301,45 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear) return native_make_pte(v & ~clear); } +/* + * Write protection operations can result in Dirty=1,Write=0 PTEs. But in the + * case of X86_FEATURE_USER_SHSTK, the software SavedDirty bit is used, since + * the Dirty=1,Write=0 will result in the memory being treated as shadow stack + * by the HW. So when creating dirty, write-protected memory, a software bit is + * used _PAGE_BIT_SAVED_DIRTY. The following functions pte_mksaveddirty() and + * pte_clear_saveddirty() take a conventional dirty, write-protected PTE + * (Write=0,Dirty=1) and transition it to the shadow stack compatible + * version. (Write=0,SavedDirty=1). + */ +static inline pte_t pte_mksaveddirty(pte_t pte) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return pte; + + pte = pte_clear_flags(pte, _PAGE_DIRTY); + return pte_set_flags(pte, _PAGE_SAVED_DIRTY); +} + +static inline pte_t pte_clear_saveddirty(pte_t pte) +{ + /* + * _PAGE_SAVED_DIRTY is unnecessary on !X86_FEATURE_USER_SHSTK kernels, + * since the HW dirty bit can be used without creating shadow stack + * memory. See the _PAGE_SAVED_DIRTY definition for more details. + */ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return pte; + + /* + * PTE is getting copied-on-write, so it will be dirtied + * if writable, or made shadow stack if shadow stack and + * being copied on access. Set the dirty bit for both + * cases. + */ + pte = pte_set_flags(pte, _PAGE_DIRTY); + return pte_clear_flags(pte, _PAGE_SAVED_DIRTY); +} + static inline pte_t pte_wrprotect(pte_t pte) { return pte_clear_flags(pte, _PAGE_RW); @@ -420,6 +459,26 @@ static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) return native_make_pmd(v & ~clear); } +/* See comments above pte_mksaveddirty() */ +static inline pmd_t pmd_mksaveddirty(pmd_t pmd) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return pmd; + + pmd = pmd_clear_flags(pmd, _PAGE_DIRTY); + return pmd_set_flags(pmd, _PAGE_SAVED_DIRTY); +} + +/* See comments above pte_mksaveddirty() */ +static inline pmd_t pmd_clear_saveddirty(pmd_t pmd) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return pmd; + + pmd = pmd_set_flags(pmd, _PAGE_DIRTY); + return pmd_clear_flags(pmd, _PAGE_SAVED_DIRTY); +} + static inline pmd_t pmd_wrprotect(pmd_t pmd) { return pmd_clear_flags(pmd, _PAGE_RW); @@ -491,6 +550,26 @@ static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) return native_make_pud(v & ~clear); } +/* See comments above pte_mksaveddirty() */ +static inline pud_t pud_mksaveddirty(pud_t pud) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return pud; + + pud = pud_clear_flags(pud, _PAGE_DIRTY); + return pud_set_flags(pud, _PAGE_SAVED_DIRTY); +} + +/* See comments above pte_mksaveddirty() */ +static inline pud_t pud_clear_saveddirty(pud_t pud) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return pud; + + pud = pud_set_flags(pud, _PAGE_DIRTY); + return pud_clear_flags(pud, _PAGE_SAVED_DIRTY); +} + static inline pud_t pud_mkold(pud_t pud) { return pud_clear_flags(pud, _PAGE_ACCESSED); diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index 0646ad00178b..56b374d1bffb 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -21,7 +21,8 @@ #define _PAGE_BIT_SOFTW2 10 /* " */ #define _PAGE_BIT_SOFTW3 11 /* " */ #define _PAGE_BIT_PAT_LARGE 12 /* On 2MB or 1GB pages */ -#define _PAGE_BIT_SOFTW4 58 /* available for programmer */ +#define _PAGE_BIT_SOFTW4 57 /* available for programmer */ +#define _PAGE_BIT_SOFTW5 58 /* available for programmer */ #define _PAGE_BIT_PKEY_BIT0 59 /* Protection Keys, bit 1/4 */ #define _PAGE_BIT_PKEY_BIT1 60 /* Protection Keys, bit 2/4 */ #define _PAGE_BIT_PKEY_BIT2 61 /* Protection Keys, bit 3/4 */ @@ -34,6 +35,15 @@ #define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3 /* software dirty tracking */ #define _PAGE_BIT_DEVMAP _PAGE_BIT_SOFTW4 +/* + * Indicates a Saved Dirty bit page. + */ +#ifdef CONFIG_X86_USER_SHADOW_STACK +#define _PAGE_BIT_SAVED_DIRTY _PAGE_BIT_SOFTW5 /* Saved Dirty bit */ +#else +#define _PAGE_BIT_SAVED_DIRTY 0 +#endif + /* If _PAGE_BIT_PRESENT is clear, we use these: */ /* - if the user mapped it with PROT_NONE; pte_present gives true */ #define _PAGE_BIT_PROTNONE _PAGE_BIT_GLOBAL @@ -117,6 +127,25 @@ #define _PAGE_SOFTW4 (_AT(pteval_t, 0)) #endif +/* + * The hardware requires shadow stack to be Write=0,Dirty=1. However, + * there are valid cases where the kernel might create read-only PTEs that + * are dirty (e.g., fork(), mprotect(), uffd-wp(), soft-dirty tracking). In + * this case, the _PAGE_SAVED_DIRTY bit is used instead of the HW-dirty bit, + * to avoid creating a wrong "shadow stack" PTEs. Such PTEs have + * (Write=0,SavedDirty=1,Dirty=0) set. + * + * Note that on processors without shadow stack support, the + * _PAGE_SAVED_DIRTY remains unused. + */ +#ifdef CONFIG_X86_USER_SHADOW_STACK +#define _PAGE_SAVED_DIRTY (_AT(pteval_t, 1) << _PAGE_BIT_SAVED_DIRTY) +#else +#define _PAGE_SAVED_DIRTY (_AT(pteval_t, 0)) +#endif + +#define _PAGE_DIRTY_BITS (_PAGE_DIRTY | _PAGE_SAVED_DIRTY) + #define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE) /* @@ -125,9 +154,9 @@ * instance, and is *not* included in this mask since * pte_modify() does modify it. */ -#define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ - _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY | \ - _PAGE_SOFT_DIRTY | _PAGE_DEVMAP | _PAGE_ENC | \ +#define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ + _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY_BITS | \ + _PAGE_SOFT_DIRTY | _PAGE_DEVMAP | _PAGE_ENC | \ _PAGE_UFFD_WP) #define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE) @@ -186,12 +215,17 @@ enum page_cache_mode { #define PAGE_READONLY __pg(__PP| 0|_USR|___A|__NX| 0| 0| 0) #define PAGE_READONLY_EXEC __pg(__PP| 0|_USR|___A| 0| 0| 0| 0) -#define __PAGE_KERNEL (__PP|__RW| 0|___A|__NX|___D| 0|___G) -#define __PAGE_KERNEL_EXEC (__PP|__RW| 0|___A| 0|___D| 0|___G) -#define _KERNPG_TABLE_NOENC (__PP|__RW| 0|___A| 0|___D| 0| 0) -#define _KERNPG_TABLE (__PP|__RW| 0|___A| 0|___D| 0| 0| _ENC) +/* + * Page tables needs to have Write=1 in order for any lower PTEs to be + * writable. This includes shadow stack memory (Write=0, Dirty=1) + */ #define _PAGE_TABLE_NOENC (__PP|__RW|_USR|___A| 0|___D| 0| 0) #define _PAGE_TABLE (__PP|__RW|_USR|___A| 0|___D| 0| 0| _ENC) +#define _KERNPG_TABLE_NOENC (__PP|__RW| 0|___A| 0|___D| 0| 0) +#define _KERNPG_TABLE (__PP|__RW| 0|___A| 0|___D| 0| 0| _ENC) + +#define __PAGE_KERNEL (__PP|__RW| 0|___A|__NX|___D| 0|___G) +#define __PAGE_KERNEL_EXEC (__PP|__RW| 0|___A| 0|___D| 0|___G) #define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX| 0| 0|___G) #define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0| 0| 0|___G) #define __PAGE_KERNEL_NOCACHE (__PP|__RW| 0|___A|__NX|___D| 0|___G| __NC) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index cda3118f3b27..6c5ef14060a8 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -273,7 +273,8 @@ static inline bool pte_flags_need_flush(unsigned long oldflags, const pteval_t flush_on_clear = _PAGE_DIRTY | _PAGE_PRESENT | _PAGE_ACCESSED; const pteval_t software_flags = _PAGE_SOFTW1 | _PAGE_SOFTW2 | - _PAGE_SOFTW3 | _PAGE_SOFTW4; + _PAGE_SOFTW3 | _PAGE_SOFTW4 | + _PAGE_SAVED_DIRTY; const pteval_t flush_on_change = _PAGE_RW | _PAGE_USER | _PAGE_PWT | _PAGE_PCD | _PAGE_PSE | _PAGE_GLOBAL | _PAGE_PAT | _PAGE_PAT_LARGE | _PAGE_PKEY_BIT0 | _PAGE_PKEY_BIT1 | From patchwork Mon Feb 27 22:29:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1D08C83005 for ; Mon, 27 Feb 2023 22:32:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D3CFE6B008A; Mon, 27 Feb 2023 17:31:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A41A06B0096; Mon, 27 Feb 2023 17:31:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 69A3B6B0093; Mon, 27 Feb 2023 17:31:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 275F56B008A for ; Mon, 27 Feb 2023 17:31:47 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 09FD51A05A5 for ; Mon, 27 Feb 2023 22:31:47 +0000 (UTC) X-FDA: 80514520254.15.D691236 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf24.hostedemail.com (Postfix) with ESMTP id 08E2A180009 for ; Mon, 27 Feb 2023 22:31:44 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=RhOrvd+M; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537105; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=5N0lFCbFu8gwpBsmCT1OeqTBC2+wwCTV8SigqvCE8fg=; b=0qKZWl/i0Gd48ANjZDXhFskK+dSaN9BWVXjJWc+Q9h4NR3mcI02LOHyKQvTA94ycgFGOjd 81H/iAppktyOm73AN9thMg7RTcAYs+UefFxVg0y94JIxal/ubG2c93PZyY7Pt5PjsoVVs7 XNKUUiNdiG2ABeYCxlEAuSjQDUwCHFg= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=RhOrvd+M; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537105; a=rsa-sha256; cv=none; b=TQI6bQH9HSMtJlpu/QB7YWve2F0xkTTIW0TsEIqZkGWU9ajc3g9bdRgR5BOfhfwjHwZKLm saGOgXOSTL8OBqik4UvfD6QAy6G0KtOtKPJNvhH11yiT4YP8WCvopzbglAlWtuH/py0o2f LQxndwLafqPcFJels7tm5n22uW/xyYw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537105; x=1709073105; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Pl+T1+xmSBSUgFqJaHPWJP+BmegywDM6GYKzVKSMXoQ=; b=RhOrvd+MjCj46UIbdgPZrlU22DVQXSsU7b2FM6A7cxk1nLipVM56ZbdM AqO+nUCr0cgh/9q0V4vNNnHMJXRfydVZc0pV9t6u5mVUzcpIAzRjQMwdG UesWZs6T8XK/4W/Fxt0C8kd9eVYqX/CEde4a5qk+kjBwMn7pbOzdy4RO9 mlD8SwXe0k/GNUz52u3G2+cK3dEs1sNNT3X0XQrsX68cqIoVDNnQRR4/i OfX80XlVfYcxyksZPqDjbBpJzSMdccDe6EZBBdAya3IRDNE4FQm//I/N3 Wr6cOBN7HV/27TSBc/UIETMkJznvNhFgNCwPzpQ0CEohpLWgG6mc/f0t1 g==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657391" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657391" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:23 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024576" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024576" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:18 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 15/41] x86/mm: Update ptep/pmdp_set_wrprotect() for _PAGE_SAVED_DIRTY Date: Mon, 27 Feb 2023 14:29:31 -0800 Message-Id: <20230227222957.24501-16-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Queue-Id: 08E2A180009 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: qjq8bzwap5z6m3pdkgi6or9au1qufrq9 X-HE-Tag: 1677537104-818712 X-HE-Meta: U2FsdGVkX1/pnCdx1qcUQcLLFV+Nch0rLI2/0/iyy1MRJwEn4MU5lr1nJ0clGngnwIbNTEt8FZpiPHsKluJM1D2WZv0ftXKFYaLenbxzp4bJ+GzNRHGq/xD2gGtYfs2Ys43bpCmU4gjhpxaNim+VA/s6uuMKRcGaNuPK7Akf6AcdSg5BfSytxVUQvbIs4UdjGj4fggeBodGc7egCm5xYpV9bR+N8IQiw7pAScFfBqzGx30JEbe5L1YXaf9sES68FxJceN/FoRcWvY54mEkuVd3EX8MKykipOE0JDML7Kn1eNkqocUS/WvPWfxuh2wO/IQVPzRFHISVzayc8kJY1lCo0NihA9inlIOMxBQVdIO2nW5LZW7mevLHyx773lTME+1tzOe46ROGNHRdqRJmjaarx9oasXc6wSq1yqMNlT0ZI7GzggvRTqMQnL0rPvVUb4WrlNN2mCbzyqmzB3Mnudh0gUrdGpcT533CjdmRkf+ibzP6haY6YW1ywS7jJKHA/7K7CWe8qQXJZ6an8PkJ9hHeSzAO8IAnaRqGRz9mxcCQWes3HeieDUhjVtZCkmjlisOg041iYprHTZbOUtbuhl0fGdisrROJZ+L8P1WKGZjvH5UrLqSxXhJUAxlS9zblBa6ennFtYhWPfdH9UXpewokKOWVJRg1yyZLnWbX4SXbCR1as7OuJW1ELQ/bTmpaAh9F/IfJZ8Z0zcjFXghXMEfFf1yV8gBN9jRof87CuU6QkOpcQV3ghQruiMxwe0aBykAamOV4GvINFPb4QWGStBB7WD++yemtB4Utjuh6zZKi1qN11VlU41y2bdXKoPwdNmA2FebZd0kpL2PdDVemncfFQ8KrI9vGFbO4N2ZJHKK8ncC7YrV3nS1cOdPFsskKOWQUXOAJY3WnD2LrzNH2RTcXrAvGd+BnTDgCIg20O8CWaZ9BY4p5EIVOWUEzHT8Zq3Qc8Z3uBQrXbnRjrsWScR Y5ZYi8ZR 6k5eFH6UJiXEbC4+NQmfgv/PTw07Dmj7ItEXNxIo6+N+XB1ekCyZoSBmDXqMHPS+GYaAhRq5T+oT4BSDn4SrOC+DJ7up9QfsRnq6B3ydMToTfsVGboKVIZhru3eKGOsJ+n4HRZzxVbuSfCF5G1H52iHhMd4rjYG4XfefSuY585yQfuUcrRpwZP4Zacf8JGdjscubo5I7hQ2p6GrOzH6VTul77O9ZFGau4L5AjK/XDpMHKPAEGxk8NbtU08HUAVRmt0UGyl4raCcmqoaT9Ma3cD7S6qFUAKn4v7R7JEOiKuUsvFVKfRT5xub+mCUYMzsmVpZ8m8ve3D0bRd/zDKcaK58wVFFqc7Iv/3r+nxWxjOEsttMDpIbWL7l3h4G2wLCCFpXS5bXzyLMjj8rs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu When shadow stack is in use, Write=0,Dirty=1 PTE are preserved for shadow stack. Copy-on-write PTEs then have Write=0,SavedDirty=1. When a PTE goes from Write=1,Dirty=1 to Write=0,SavedDirty=1, it could become a transient shadow stack PTE in two cases: 1. Some processors can start a write but end up seeing a Write=0 PTE by the time they get to the Dirty bit, creating a transient shadow stack PTE. However, this will not occur on processors supporting shadow stack, and a TLB flush is not necessary. 2. When _PAGE_DIRTY is replaced with _PAGE_SAVED_DIRTY non-atomically, a transient shadow stack PTE can be created as a result. Thus, prevent that with cmpxchg. In the case of pmdp_set_wrprotect(), for nopmd configs the ->pmd operated on does not exist and the logic would need to be different. Although the extra functionality will normally be optimized out when user shadow stacks are not configured, also exclude it in the preprocessor stage so that it will still compile. User shadow stack is not supported there by Linux anyway. Leave the cpu_feature_enabled() check so that the functionality also gets disabled based on runtime detection of the feature. Similarly, compile it out in ptep_set_wrprotect() due to a clang warning on i386. Like above, the code path should get optimized out on i386 since shadow stack is not supported on 32 bit kernels, but this makes the compiler happy. Dave Hansen, Jann Horn, Andy Lutomirski, and Peter Zijlstra provided many insights to the issue. Jann Horn provided the cmpxchg solution. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe --- v6: - Fix comment and log to update for _PAGE_COW being replaced with _PAGE_SAVED_DIRTY. v5: - Commit log verbiage and formatting (Boris) - Remove capitalization on shadow stack (Boris) - Fix i386 warning on recent clang v3: - Remove unnecessary #ifdef (Dave Hansen) v2: - Compile out some code due to clang build error - Clarify commit log (dhansen) - Normalize PTE bit descriptions between patches (dhansen) - Update comment with text from (dhansen) --- arch/x86/include/asm/pgtable.h | 35 ++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 7360783f2140..349fcab0405a 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1192,6 +1192,23 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { +#ifdef CONFIG_X86_USER_SHADOW_STACK + /* + * Avoid accidentally creating shadow stack PTEs + * (Write=0,Dirty=1). Use cmpxchg() to prevent races with + * the hardware setting Dirty=1. + */ + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) { + pte_t old_pte, new_pte; + + old_pte = READ_ONCE(*ptep); + do { + new_pte = pte_wrprotect(old_pte); + } while (!try_cmpxchg(&ptep->pte, &old_pte.pte, new_pte.pte)); + + return; + } +#endif clear_bit(_PAGE_BIT_RW, (unsigned long *)&ptep->pte); } @@ -1244,6 +1261,24 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) { +#ifdef CONFIG_X86_USER_SHADOW_STACK + /* + * Avoid accidentally creating shadow stack PTEs + * (Write=0,Dirty=1). Use cmpxchg() to prevent races with + * the hardware setting Dirty=1. + */ + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) { + pmd_t old_pmd, new_pmd; + + old_pmd = READ_ONCE(*pmdp); + do { + new_pmd = pmd_wrprotect(old_pmd); + } while (!try_cmpxchg(&pmdp->pmd, &old_pmd.pmd, new_pmd.pmd)); + + return; + } +#endif + clear_bit(_PAGE_BIT_RW, (unsigned long *)pmdp); } From patchwork Mon Feb 27 22:29:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154244 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B830C64ED8 for ; Mon, 27 Feb 2023 22:31:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6E6666B0092; Mon, 27 Feb 2023 17:31:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 671856B0089; Mon, 27 Feb 2023 17:31:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4753C6B0092; Mon, 27 Feb 2023 17:31:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 23BED6B0089 for ; Mon, 27 Feb 2023 17:31:45 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E81E5A015E for ; Mon, 27 Feb 2023 22:31:44 +0000 (UTC) X-FDA: 80514520128.05.8335055 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf28.hostedemail.com (Postfix) with ESMTP id DCBCDC0010 for ; Mon, 27 Feb 2023 22:31:42 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=oGm8Uyvi; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537103; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=JM2zGYV7PoLVXGJvwopM6rwJjQzMW8sxNmugfzubGvI=; b=xkWNBzWNr7f8PgIrJqU0uWB1J3EeCekDd7c0RPcqOzjeEDtvXYVHqJ094e/WfY177GuQj2 J6ta07JRN63XtBpVsDI9BSe3TbDegKRd7zm3FYVzM9tz7HdptTRTLNStFtyYDHMalz8a89 cNJzAgiZcY6a5KI8LTiFRs5zeoIvNsQ= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=oGm8Uyvi; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537103; a=rsa-sha256; cv=none; b=l672QDwG3KvVaHi9zYdJ/KT+WDxmnLKjN4oz3m4/UoO4iVQrHv4aaLOSaur7dRR2JPohw+ 8ys3E3NJuaMbPRLPW742VSHktf6gXrbfolvgPo7poH8GUzpsfYewB/9FpUGDOniEzS9R7e Jz3gqdr1h1pybUStU8RJbPERpXV4dTM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537102; x=1709073102; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=7g52pRKve1b7v6zvBkrhRB55FfgM/NC0gEJKTUhkXLA=; b=oGm8Uyvi3FiXZpV6Unknsr9dBFleajYx3u8yPeyDaBKWESdk4qXwnKYS lABzaVB3j09hgZfeOzK7RrfxNPeqht1zl6DFH09RREC4JFGx8HN9s6Jm5 emC9jZnOxXegoRMTB1/0M8YlP1Sy9kDWSoj1zZEtx0jJmDZDoHLozCl5R aOpheCToUOXwDkeIXKcoa0YlxgvIvuBBgmi4Tt/0253JHZ7YGc3/IvG1F DxZ9iZDWcRjo4kZ07g7sxfrk8c/stfe1+BYAMhfuzzmetEjxmQYwCORgK 9aq36Tp3dKR2iEtCj+CGs6cJK44cC/uvRPOkTJkHJ9/9snbM+RHsUAt84 g==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657359" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657359" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:22 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024583" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024583" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:19 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 16/41] x86/mm: Start actually marking _PAGE_SAVED_DIRTY Date: Mon, 27 Feb 2023 14:29:32 -0800 Message-Id: <20230227222957.24501-17-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: DCBCDC0010 X-Stat-Signature: ow48uwbh8edact96qip8hyx8uk8xphnr X-Rspam-User: X-HE-Tag: 1677537102-524469 X-HE-Meta: U2FsdGVkX18lbxOurdyqijROPgJQMP1YvVr8o7TGgY0gf49Xuk36PThMPoazGX9lC3GxNI2PTIZSISJDTMvVPsNZLBIGqNxL32bO9706Mdiw7eGck8aQi5Iecia8dKS+qqLlBzEHoa2SzKEGRsITodiSl3aqBZEdNhODaQE7+NHOTTH01y64jdIFa6FMhJ8PQBtt4jsSfxQzfG4XdAPl1s6wLMEQH1M7bI1YYMRJxFWGirEgPuvTiK7Cdebihw6I3A9zT4M0/WHhAB7gVh2tUGKg8y4nFoQ1/5d6VO8Ym1QHgjn27+KgGd2vSn10eef8XBW9XDDvay8IDv+xCYEzzHpo/VD2wljt4vuzADLbI8iV/7tOJuyuAHLKQ9AXWgWjxJ6cXNC35FnjAaqFCvz3uujEddt13GRG3t17QG7EJ+K6UbB3Qu9ilLvsV19VXKSdbUznvpQkW4iPbkz4SPrFqItrJ1HX++dP4+jR0L4zxK1XEDBlSw/NXwzAEGF21SyidbPpbWwgaYwaFp9yOPhGsWfwr9XrkfZV6HBmmtFL3J4UxnlWrRBKaV40jeGm78LBZ32TZOoC8lKyyp3Qmc7fH04eGPe9g4uhlBY9lPYjbe/wOYookeTDHY9Gu2Ry1s1JT8DokyqzQ6PXPAEh5bsBUxrrUqwBDj767FOL8zhrJOOu3xH3wgCbDgXiutsWObSYN4yUPob6QTgq796f970awNn9LdNezl7pKni91cPk4GyJ6eYWY1R5iqnoNf5VvK9H8+TjJc1njt6U7EI1n7eY2mZdUxLeubp96IOpaB6ksQk0uJGnMDIxda2VncKnNnTyJe8xS4MS+OKqKjweZ9GXEizBO2gVawYub/fMMiHDFnIlwZedFWzrCgVFcHwZgxCsH1ushAnTy4ng61jZgkCSy+DPgN1SqYMEfNjdp4IeluuOD134+AcfFmbn1TtmPgF7xrQidAKjSjXfbf19cb3 PKaW8Hmr gYyacNnD7SO/OMGwIwaf5jWhBoGzZHRb1zcozs1rmZBKK1TQr4m9G+MYuYcyprpi5tj62EMlWPt6kC9LCL+RyKRIWsRbvhxXga2DXohs3n6EaBhZKcpdJQ6hElBEiMEGQjKIWbQw7gK2Go3/vMttvtFII6cYIXELFFELpggJVf4ZUQYRutw1+0EUlKWD5UC+T7kMvct70UXdyZn7q5SvVnhgXASVPPzl/1FJYbrdenNUH38g9sn40mw/oZZoJ0Woye2nhUcVezJAzB6N07qPlGa3Hc3CEuG9kb4eT1jeri5oQOxHr6E7pc9oyYhKvnmrNYRD/fT4J6nGLmhXcx1F/aD6iioJ27HA9BFE9l4A/bUb11p6KYYdSrr9WRsGxnfRLTp1LrK+n003E3Cl8Inks4suCGg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The recently introduced _PAGE_SAVED_DIRTY should be used instead of the HW Dirty bit whenever a PTE is Write=0, in order to not inadvertently create shadow stack PTEs. Update pte_mk*() helpers to do this, and apply the same changes to pmd and pud. For pte_modify() this is a bit trickier. It takes a "raw" pgprot_t which was not necessarily created with any of the existing PTE bit helpers. That means that it can return a pte_t with Write=0,Dirty=1, a shadow stack PTE, when it did not intend to create one. Modify it to also move _PAGE_DIRTY to _PAGE_SAVED_DIRTY. To avoid creating Write=0,Dirty=1 PTEs, pte_modify() needs to avoid: 1. Marking Write=0 PTEs Dirty=1 2. Marking Dirty=1 PTEs Write=0 The first case cannot happen as the existing behavior of pte_modify() is to filter out any Dirty bit passed in newprot. Handle the second case by shifting _PAGE_DIRTY=1 to _PAGE_SAVED_DIRTY=1 if the PTE was write protected by the pte_modify() call. Apply the same changes to pmd_modify(). Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Co-developed-by: Yu-cheng Yu Signed-off-by: Yu-cheng Yu Signed-off-by: Rick Edgecombe --- v6: - Rename _PAGE_COW to _PAGE_SAVED_DIRTY (David Hildenbrand) - Open code _PAGE_SAVED_DIRTY part in pte_modify() (Boris) - Change the logic so the open coded part is not too ugly - Merge pte_modify() patch with this one because of the above v4: - Break part patch for better bisectability --- arch/x86/include/asm/pgtable.h | 168 ++++++++++++++++++++++++++++----- 1 file changed, 145 insertions(+), 23 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 349fcab0405a..05dfdbdf96b4 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -124,9 +124,17 @@ extern pmdval_t early_pmd_flags; * The following only work if pte_present() is true. * Undefined behaviour if not.. */ -static inline int pte_dirty(pte_t pte) +static inline bool pte_dirty(pte_t pte) { - return pte_flags(pte) & _PAGE_DIRTY; + return pte_flags(pte) & _PAGE_DIRTY_BITS; +} + +static inline bool pte_shstk(pte_t pte) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return false; + + return (pte_flags(pte) & (_PAGE_RW | _PAGE_DIRTY)) == _PAGE_DIRTY; } static inline int pte_young(pte_t pte) @@ -134,9 +142,18 @@ static inline int pte_young(pte_t pte) return pte_flags(pte) & _PAGE_ACCESSED; } -static inline int pmd_dirty(pmd_t pmd) +static inline bool pmd_dirty(pmd_t pmd) { - return pmd_flags(pmd) & _PAGE_DIRTY; + return pmd_flags(pmd) & _PAGE_DIRTY_BITS; +} + +static inline bool pmd_shstk(pmd_t pmd) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return false; + + return (pmd_flags(pmd) & (_PAGE_RW | _PAGE_DIRTY | _PAGE_PSE)) == + (_PAGE_DIRTY | _PAGE_PSE); } #define pmd_young pmd_young @@ -145,9 +162,9 @@ static inline int pmd_young(pmd_t pmd) return pmd_flags(pmd) & _PAGE_ACCESSED; } -static inline int pud_dirty(pud_t pud) +static inline bool pud_dirty(pud_t pud) { - return pud_flags(pud) & _PAGE_DIRTY; + return pud_flags(pud) & _PAGE_DIRTY_BITS; } static inline int pud_young(pud_t pud) @@ -157,13 +174,21 @@ static inline int pud_young(pud_t pud) static inline int pte_write(pte_t pte) { - return pte_flags(pte) & _PAGE_RW; + /* + * Shadow stack pages are logically writable, but do not have + * _PAGE_RW. Check for them separately from _PAGE_RW itself. + */ + return (pte_flags(pte) & _PAGE_RW) || pte_shstk(pte); } #define pmd_write pmd_write static inline int pmd_write(pmd_t pmd) { - return pmd_flags(pmd) & _PAGE_RW; + /* + * Shadow stack pages are logically writable, but do not have + * _PAGE_RW. Check for them separately from _PAGE_RW itself. + */ + return (pmd_flags(pmd) & _PAGE_RW) || pmd_shstk(pmd); } #define pud_write pud_write @@ -342,7 +367,16 @@ static inline pte_t pte_clear_saveddirty(pte_t pte) static inline pte_t pte_wrprotect(pte_t pte) { - return pte_clear_flags(pte, _PAGE_RW); + pte = pte_clear_flags(pte, _PAGE_RW); + + /* + * Blindly clearing _PAGE_RW might accidentally create + * a shadow stack PTE (Write=0,Dirty=1). Move the hardware + * dirty value to the software bit. + */ + if (pte_dirty(pte)) + pte = pte_mksaveddirty(pte); + return pte; } #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP @@ -380,7 +414,7 @@ static inline pte_t pte_clear_uffd_wp(pte_t pte) static inline pte_t pte_mkclean(pte_t pte) { - return pte_clear_flags(pte, _PAGE_DIRTY); + return pte_clear_flags(pte, _PAGE_DIRTY_BITS); } static inline pte_t pte_mkold(pte_t pte) @@ -395,7 +429,19 @@ static inline pte_t pte_mkexec(pte_t pte) static inline pte_t pte_mkdirty(pte_t pte) { - return pte_set_flags(pte, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + pteval_t dirty = _PAGE_DIRTY; + + /* Avoid creating Dirty=1,Write=0 PTEs */ + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && !pte_write(pte)) + dirty = _PAGE_SAVED_DIRTY; + + return pte_set_flags(pte, dirty | _PAGE_SOFT_DIRTY); +} + +static inline pte_t pte_mkwrite_shstk(pte_t pte) +{ + /* pte_clear_saveddirty() also sets Dirty=1 */ + return pte_clear_saveddirty(pte); } static inline pte_t pte_mkyoung(pte_t pte) @@ -412,7 +458,12 @@ struct vm_area_struct; static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) { - return pte_mkwrite_kernel(pte); + pte = pte_mkwrite_kernel(pte); + + if (pte_dirty(pte)) + pte = pte_clear_saveddirty(pte); + + return pte; } static inline pte_t pte_mkhuge(pte_t pte) @@ -481,7 +532,15 @@ static inline pmd_t pmd_clear_saveddirty(pmd_t pmd) static inline pmd_t pmd_wrprotect(pmd_t pmd) { - return pmd_clear_flags(pmd, _PAGE_RW); + pmd = pmd_clear_flags(pmd, _PAGE_RW); + /* + * Blindly clearing _PAGE_RW might accidentally create + * a shadow stack PMD (RW=0, Dirty=1). Move the hardware + * dirty value to the software bit. + */ + if (pmd_dirty(pmd)) + pmd = pmd_mksaveddirty(pmd); + return pmd; } #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP @@ -508,12 +567,23 @@ static inline pmd_t pmd_mkold(pmd_t pmd) static inline pmd_t pmd_mkclean(pmd_t pmd) { - return pmd_clear_flags(pmd, _PAGE_DIRTY); + return pmd_clear_flags(pmd, _PAGE_DIRTY_BITS); } static inline pmd_t pmd_mkdirty(pmd_t pmd) { - return pmd_set_flags(pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + pmdval_t dirty = _PAGE_DIRTY; + + /* Avoid creating (HW)Dirty=1, Write=0 PMDs */ + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && !pmd_write(pmd)) + dirty = _PAGE_SAVED_DIRTY; + + return pmd_set_flags(pmd, dirty | _PAGE_SOFT_DIRTY); +} + +static inline pmd_t pmd_mkwrite_shstk(pmd_t pmd) +{ + return pmd_clear_saveddirty(pmd); } static inline pmd_t pmd_mkdevmap(pmd_t pmd) @@ -533,7 +603,12 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd) static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) { - return pmd_set_flags(pmd, _PAGE_RW); + pmd = pmd_set_flags(pmd, _PAGE_RW); + + if (pmd_dirty(pmd)) + pmd = pmd_clear_saveddirty(pmd); + + return pmd; } static inline pud_t pud_set_flags(pud_t pud, pudval_t set) @@ -577,17 +652,32 @@ static inline pud_t pud_mkold(pud_t pud) static inline pud_t pud_mkclean(pud_t pud) { - return pud_clear_flags(pud, _PAGE_DIRTY); + return pud_clear_flags(pud, _PAGE_DIRTY_BITS); } static inline pud_t pud_wrprotect(pud_t pud) { - return pud_clear_flags(pud, _PAGE_RW); + pud = pud_clear_flags(pud, _PAGE_RW); + + /* + * Blindly clearing _PAGE_RW might accidentally create + * a shadow stack PUD (RW=0, Dirty=1). Move the hardware + * dirty value to the software bit. + */ + if (pud_dirty(pud)) + pud = pud_mksaveddirty(pud); + return pud; } static inline pud_t pud_mkdirty(pud_t pud) { - return pud_set_flags(pud, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + pudval_t dirty = _PAGE_DIRTY; + + /* Avoid creating (HW)Dirty=1, Write=0 PUDs */ + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && !pud_write(pud)) + dirty = _PAGE_SAVED_DIRTY; + + return pud_set_flags(pud, dirty | _PAGE_SOFT_DIRTY); } static inline pud_t pud_mkdevmap(pud_t pud) @@ -607,7 +697,11 @@ static inline pud_t pud_mkyoung(pud_t pud) static inline pud_t pud_mkwrite(pud_t pud) { - return pud_set_flags(pud, _PAGE_RW); + pud = pud_set_flags(pud, _PAGE_RW); + + if (pud_dirty(pud)) + pud = pud_clear_saveddirty(pud); + return pud; } #ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY @@ -724,6 +818,8 @@ static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask); static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) { pteval_t val = pte_val(pte), oldval = val; + bool wr_protected; + pte_t pte_result; /* * Chop off the NX bit (if present), and add the NX portion of @@ -732,17 +828,43 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) val &= _PAGE_CHG_MASK; val |= check_pgprot(newprot) & ~_PAGE_CHG_MASK; val = flip_protnone_guard(oldval, val, PTE_PFN_MASK); - return __pte(val); + + pte_result = __pte(val); + + /* + * Do the saveddirty fixup if the PTE was just write protected and + * it's dirty. + */ + wr_protected = (oldval & _PAGE_RW) && !(val & _PAGE_RW); + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && wr_protected && + (val & _PAGE_DIRTY)) + pte_result = pte_mksaveddirty(pte_result); + + return pte_result; } static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) { pmdval_t val = pmd_val(pmd), oldval = val; + bool wr_protected; + pmd_t pmd_result; - val &= _HPAGE_CHG_MASK; + val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY); val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK); - return __pmd(val); + + pmd_result = __pmd(val); + + /* + * Do the saveddirty fixup if the PMD was just write protected and + * it's dirty. + */ + wr_protected = (oldval & _PAGE_RW) && !(val & _PAGE_RW); + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && wr_protected && + (val & _PAGE_DIRTY)) + pmd_result = pmd_mksaveddirty(pmd_result); + + return pmd_result; } /* From patchwork Mon Feb 27 22:29:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154248 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16B92C7EE3A for ; Mon, 27 Feb 2023 22:31:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B059E6B008C; Mon, 27 Feb 2023 17:31:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 86B176B008A; Mon, 27 Feb 2023 17:31:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5AA966B0095; Mon, 27 Feb 2023 17:31:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 340266B008C for ; Mon, 27 Feb 2023 17:31:47 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 01B4BAB051 for ; Mon, 27 Feb 2023 22:31:46 +0000 (UTC) X-FDA: 80514520254.15.29CE3F6 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf28.hostedemail.com (Postfix) with ESMTP id 14789C0010 for ; Mon, 27 Feb 2023 22:31:44 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="Yc1L1//M"; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537105; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=/86SfoVQQ8nfu81V3XQo7ipVqL9sa+8xwOyr++XyGK4=; b=2qteqjz7+58J5iAo8ioeOJ8A2Lwc1ztAPSyJVyh2Sb9PSHpc7Ob0SUBj5jIvR9u3Yi64gd REQB5cnyVV6Fk//oxof8wNpxK+9LRxliAbPKxkgUzFnO8oXhCnQYt+/p11Cxx1X8q4N85u bxvAKpCdzEupJF5w7EuJ1/cOMNx8a7A= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="Yc1L1//M"; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537105; a=rsa-sha256; cv=none; b=JF3QkmczJZL6PoX4c9ZJVJDrNX6T2+KjFXxqNaK10LJZ9T7d3C1G8lh7EqUtloPSfJX++k x6gWlZjmB9kN3omkF7yt1u+cLuBXN3epTyb2tQ8pdNGTxZUOBfWzF13JB9bsRPeet7y7W9 NrLYaDpQ5flp3I1DUpbMHmku6JFq0pk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537105; x=1709073105; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=CMdfKd48MBBKKroi4bf9lEd66dWh8s6uGF9cifN2Nfs=; b=Yc1L1//M3ArliK0JF8e+hJp7gs7lduea3rhSrRBrCNd0zKb28U4QhD7h bMQjQAEROVUT3KmxeaPEw6tKgTMryOEcMvbU8FQdWZanwUIXZlKbVdbJv uAFq811qUXSyhiXimcVbPSSPLq7Hi/8WOfcBGpyNgojBtGYAl6lFgB11E yrS39P6pIITwDR8L53HDtwcx6p3cWTUf63rOTnwJuzzLdciCSn7G8bWzS WAPJm3OXSwJzwv3OWNpmvXc3kp9VfbUfjUDd9vt0gf7LWwUiLI1xcOm3O PZUnzI9f/LoPH+1KhI9GIgc8jFHDX4HfW/dhqQNwMSRB8FglGkXKXa5TN g==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657413" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657413" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:23 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024589" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024589" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:19 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu , Peter Xu Subject: [PATCH v7 17/41] mm: Move VM_UFFD_MINOR_BIT from 37 to 38 Date: Mon, 27 Feb 2023 14:29:33 -0800 Message-Id: <20230227222957.24501-18-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 14789C0010 X-Stat-Signature: i7omett9ayu1txuje6z3jmp31j947pdu X-Rspam-User: X-HE-Tag: 1677537104-258285 X-HE-Meta: U2FsdGVkX1+yeh+Ak1Qy3LE0KzM/PcOVMfHF5sJO7dQYFfVjR7wSFSi5iDshohQUB5PGiGMgRQj6nGK2BP6SOlgwDA7Z/S2Gb1ljzFxh9kw7z7lPJDJsyPUZTnxZ2IsN3a9HoBq3fsaFfkyNItQxQpDgFH4Hdjprx4QB64KOzbfyuh8AKRDFOChm3gPUpGOQomr45SSWtaq8oQrpeKiBYZSxjQ9B2qtHW3Pia0B2ar+wnm9h3ntr9KA6SWXsiD8uRJ/mBz0oFiLK7tH0a4aonmxmumWES//VHYbiXotGjekldv6Ue5CBmTvyWqmhJPBORdJOPruVbXQa64BvFVXkp4GzzFxVCgYBrMIiajiOlNgxWIg2MFiyJGMWOfQXounZAo7GQ6WPGzviTh3WFxlEuXVmx0L4hcjhwyawtK9hXSLX0h08rHbgZqW3jHwT5DUgS8uCANr1dH2mvypfwkWvwgJWdOgA2VKL3es9adFbirKmPu9dfkGgG4sVsBe1kbTQKQeyv+dlo58W/2gYpFDq57wQbhUWvgK870Ad3cykRGnI0l7dhDPrOqMEY5g+teN2elABhEp+JRoOqq/JfOIbovael6rGqD7JYyCVyE6TZUN5UXIBh8Utp9an0sZnfPgqNbKBxZvgBrLetKVNPMy6COPKgSyRdgtWxo373erlr1TuOOXjD3IPufFnTHehz5zquFRTCjyM45jt6C4+bq8CV4z/rcRV2dnu+THjWPp63roaWBGrfCePb5N2MmTsl5MJtZb6yKCcmSD/33jcBWDQVUwCpkdVCvPw90Jb2WWCwxBNJQFP1VgQCKD4N+TNgz0MPV9ZzCdn4cBnyNqHPHajfQnKSDzuBSXgMAoLyy5tzhBPijeyuPSRprnemY19mdXSDINOnQ6so6svtpLLUUYOeOduIAW+oENAm/JNvhevV/GpXmAfOoPzPM49wZuL9gLU2vPiQo0Ek2s03Awo7kg CJ2xuHLT CDWqWIZAU64GYi9R8oVQ8t5kLsKUKOcdrJQnXCCxqvlDlNlBOMrkao7TA9Z+WopNGzv6gU0I8z6AHy9iPHgukSF9xQjxHCTbBixXxfk2XZQjudvMIADh6b7GlE1iHQmsgUZJDplrY8Ews/5wggdNv818zTXQ0+8b8XTfV9GgP8TufYAR7qna3pNZVbg8KyKElxfDeaa+ePLPR5b1H2LkMdtAIWFQT7hMdhJBor3wAHYZLmNiJr2JHrNpweG+ryUQbk8TKqrmMYA/7frNud/1vWV8PhIIeqAY9vZLxxRdTlPIdJKz7C1zWENG2S58f3oSQGVqEFaKhNwoys3GIpP/a4NmtvU9iRbPkzN8huASt3VJnhQuUuSVsHl9rDtQ4pxZ6t45+3L8yaNkNBup8J86DeH6OnRqSIhFRsQba5U3eZcgJ+X6irzrdh/114YsJVDPEzta+fC/Ly1QOy+q4K4z01KoXuA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. Future patches will introduce a new VM flag VM_SHADOW_STACK that will be VM_HIGH_ARCH_BIT_5. VM_HIGH_ARCH_BIT_1 through VM_HIGH_ARCH_BIT_4 are bits 32-36, and bit 37 is the unrelated VM_UFFD_MINOR_BIT. For the sake of order, make all VM_HIGH_ARCH_BITs stay together by moving VM_UFFD_MINOR_BIT from 37 to 38. This will allow VM_SHADOW_STACK to be introduced as 37. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Acked-by: Peter Xu Signed-off-by: Yu-cheng Yu Reviewed-by: Axel Rasmussen Signed-off-by: Rick Edgecombe Cc: Peter Xu Cc: Mike Kravetz --- include/linux/mm.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index af652444fbba..a1b31caae013 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -377,7 +377,7 @@ extern unsigned int kobjsize(const void *objp); #endif #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR -# define VM_UFFD_MINOR_BIT 37 +# define VM_UFFD_MINOR_BIT 38 # define VM_UFFD_MINOR BIT(VM_UFFD_MINOR_BIT) /* UFFD minor faults */ #else /* !CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */ # define VM_UFFD_MINOR VM_NONE From patchwork Mon Feb 27 22:29:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154247 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6170FC64ED8 for ; Mon, 27 Feb 2023 22:31:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 86E3F6B0098; Mon, 27 Feb 2023 17:31:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 70A506B008C; Mon, 27 Feb 2023 17:31:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4EAC86B0096; Mon, 27 Feb 2023 17:31:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3D72A6B0093 for ; Mon, 27 Feb 2023 17:31:47 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 14AC540A3E for ; Mon, 27 Feb 2023 22:31:47 +0000 (UTC) X-FDA: 80514520254.15.8196269 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf07.hostedemail.com (Postfix) with ESMTP id 14AB240012 for ; Mon, 27 Feb 2023 22:31:44 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ftZK3VjD; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537105; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=wceL11ZZ0s1oZMAkJv9xADmkIBB3swH4yWQ/WScjmXo=; b=6LcRb1zafXLbIKdWglSqWCRUONK2PKx83iWVMM18TErQ316nN+3Cl2MR7ndABtefuR66JC FpNuGYRxxFriX+NR0Hvwt/3dCifZURmP6Uov+NnsnUVEqsllcQJs7m2wzn39+YYcy04XPb VtkgEFlbAz+2Ad7jf2LxrmZMCUtsOVI= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ftZK3VjD; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537105; a=rsa-sha256; cv=none; b=DvEla1ZWntNFOw1uOInq3aHqQ3NYIJDiMXr+LWDu9jmSQrSB+9SgzogJ0s1hyXZ89OcMLU WuYgBGl1sqVxAUNb6EcT1sHF0mD5WPIIV9quNQPxEgFvGuEDpxC9Vsvpq6UUldUEZVsLkw ENZU7jmKdfbDHdCdSFtrM20KARbqYeE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537105; x=1709073105; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=9ua4eoVpPkNNqQnm/7DlucqvtoyDtbjdtgZW9gHkRAo=; b=ftZK3VjDUU/ccOTrMcU5/bedxgGIQDCx9nQNey6lztLzyz1D4jDL+dSW T/1MyPJlpApUjsenOSEZR6iAaTh9Q0G4TXVz2b1CN3WdRzb6lfDaXjNps UOh8NTyxJcjYDpL2FNFqvZP5DzbHsz4CpWXV0OYzmcQCejCTIZB8qL2cG NlV/0lXQu1D7kPk0Gdb+xObD3HVJCjvOkM6DCxKLAInlEtbGCdR7hP+s0 RqocO8VszTvB/gbHg0v/Dcpm7hCSaVmgFSTJ7tiQcrglm4fqFBhMCqP7x 75CJs1hip1LGP1C6dct4mJtKySbUflhqWcjjDxv4LogRmf+EFyTvPtq7i g==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657454" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657454" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:24 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024595" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024595" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:20 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 18/41] mm: Introduce VM_SHADOW_STACK for shadow stack memory Date: Mon, 27 Feb 2023 14:29:34 -0800 Message-Id: <20230227222957.24501-19-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 14AB240012 X-Stat-Signature: dygjpzmbifmugn5fhxthp3xnfyqjs6p8 X-HE-Tag: 1677537104-433197 X-HE-Meta: U2FsdGVkX1/CneG0Rl2zbyTGlv7R10p0pzdlaeZiZ8VThbBnf4020kZ7uWb75yJvErFbIYxnTw1rLiqdAuoyBxPXpkKgevvtu0GR6FTA06k/Vdbp/G7b+ovkVCMdXB8vMvcW3h1o3kNQErhax8M/G+uwUsRwlqyPdppJMBveWzDrbSBT/H1ecxdKCgzHg6snUK9xL71zD3IB3V0x5GxhcisFQaVz2aKxIcrVJu86BsBxBgFg8+wWVnsE95dlhc19PVk5CM3qJDlkSQlMxGqZtMX26ZBCs6ZmPxtCKzRd/Lc7yW8zGGMgXufw2vB8lZN+zICKKJUnBd2dqxFYyYvpxQeD7l6yS2KeFPSL6q0UsNEL+hfieeuNTU+Zs1kC0vMGmZ45qPC+a9av2/+mDpHZyjcEVZgfbQ83EeC4jFP56aHgxV3b8cApO8FuD3c26zB0n2mOiG9/vN9119wlPkmf7mpXsHhmooI/1zD6iiuQ9F9qrzrm6YGG8Wmw9NW46uNtR8GLMeciclxNSxUrL7YGfkBDMX1WH5AldlH1kEz6gh/S7wun8gksZbi5q+27zItE+maycAaC92ATKqKsmnd67etMHp3WK0Mb+lJCi/904e95frVkBMrcauDM2ARYgQtV+QWsoq/F63WJetiZUCXt7oUt9MVA7xqMSQHzCWBUsUT1Jmq6LHvF5sPQapyK/gW6Sk86u5i5WQf/jGmzwhdn/69k310XssaYX2xlAjeybyKTkfgkfhAeWXWItqhtSHYeWnAkPiKdbnANT+07lrjHo22bwh4YZ95diRLdw7ZgofLmovcxZ3NSFqMKl8vi/+OHOEfPzXdkujqOb8JJHfKE9Kj7/ZCXagunadG0lT99925R9YWSlvLvEgZWV8iEu42Ebblw+2WxQUrxR8fqDfrnRQ3S+qfZHnLAF8cmHtHajfgz7j8gIlon0guw+iAl46XcKmMhCA2YqFLZ10KTgUa GzfWfvUV yia1nH/GJ/mKLXAhC6Ik+qvP2vYgxRogbRzl88BEjXh4dk+dx2wuQNyZwxZ0b/GC+nXev7eni8wfFI13DPgOG8mKomyhDQ9AuWVmGsRGBl7c3y5HI8aktaSnPWIL9R0+XomZpDbQ0dU1g+uADz6egUeqq7wSiMGYPn+lGYtUJMY6UliSmahkFZAw0d/P0PaLcpDNolW7wNAKOarGmDUXdAWfqfiMMXlosBgq1W5G6mwoDkYxfn403XfcL1ejtuP2Dwwt+6pqu2GJmHxq0Ui0QpEr6PDuRFReBR8z/g3eT0Kvnll1T0WAdAthrTgqxbGj6JbsipmYagdxxC9HdO/TxcEQ+sRlILm4/rjTNJ/2r1vNBEBZ9wSovV5LgYHz+Fjr5003PLX7GcKtvl6U= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu New hardware extensions implement support for shadow stack memory, such as x86 Control-flow Enforcement Technology (CET). Add a new VM flag to identify these areas, for example, to be used to properly indicate shadow stack PTEs to the hardware. Shadow stack VMA creation will be tightly controlled and limited to anonymous memory to make the implementation simpler and since that is all that is required. The solution will rely on pte_mkwrite() to create the shadow stack PTEs, so it will not be required for vm_get_page_prot() to learn how to create shadow stack memory. For this reason document that VM_SHADOW_STACK should not be mixed with VM_SHARED. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Reviewed-by: Kirill A. Shutemov Signed-off-by: Yu-cheng Yu Signed-off-by: Rick Edgecombe Cc: Kees Cook --- v7: - Use lightly edited commit log verbiage from (David Hildenbrand) - Add explanation for VM_SHARED limitation (David Hildenbrand) v6: - Add comment about VM_SHADOW_STACK not being allowed with VM_SHARED (David Hildenbrand) v3: - Drop arch specific change in arch_vma_name(). The memory can show as anonymous (Kirill) - Change CONFIG_ARCH_HAS_SHADOW_STACK to CONFIG_X86_USER_SHADOW_STACK in show_smap_vma_flags() (Boris) --- Documentation/filesystems/proc.rst | 1 + fs/proc/task_mmu.c | 3 +++ include/linux/mm.h | 8 ++++++++ 3 files changed, 12 insertions(+) diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index 9d5fd9424e8b..8b314df7ccdf 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -564,6 +564,7 @@ encoded manner. The codes are the following: mt arm64 MTE allocation tags are enabled um userfaultfd missing tracking uw userfaultfd wr-protect tracking + ss shadow stack page == ======================================= Note that there is no guarantee that every flag and associated mnemonic will diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 6a96e1713fd5..324b092c2ac9 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -711,6 +711,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR [ilog2(VM_UFFD_MINOR)] = "ui", #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */ +#ifdef CONFIG_X86_USER_SHADOW_STACK + [ilog2(VM_SHADOW_STACK)] = "ss", +#endif }; size_t i; diff --git a/include/linux/mm.h b/include/linux/mm.h index a1b31caae013..097544afb1aa 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -326,11 +326,13 @@ extern unsigned int kobjsize(const void *objp); #define VM_HIGH_ARCH_BIT_2 34 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_3 35 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_4 36 /* bit only usable on 64-bit architectures */ +#define VM_HIGH_ARCH_BIT_5 37 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_0 BIT(VM_HIGH_ARCH_BIT_0) #define VM_HIGH_ARCH_1 BIT(VM_HIGH_ARCH_BIT_1) #define VM_HIGH_ARCH_2 BIT(VM_HIGH_ARCH_BIT_2) #define VM_HIGH_ARCH_3 BIT(VM_HIGH_ARCH_BIT_3) #define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4) +#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5) #endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */ #ifdef CONFIG_ARCH_HAS_PKEYS @@ -346,6 +348,12 @@ extern unsigned int kobjsize(const void *objp); #endif #endif /* CONFIG_ARCH_HAS_PKEYS */ +#ifdef CONFIG_X86_USER_SHADOW_STACK +# define VM_SHADOW_STACK VM_HIGH_ARCH_5 /* Should not be set with VM_SHARED */ +#else +# define VM_SHADOW_STACK VM_NONE +#endif + #if defined(CONFIG_X86) # define VM_PAT VM_ARCH_1 /* PAT reserves whole VMA at once (x86) */ #elif defined(CONFIG_PPC) From patchwork Mon Feb 27 22:29:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154250 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75BB0C87FDE for ; Mon, 27 Feb 2023 22:32:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9D14B6B009B; Mon, 27 Feb 2023 17:31:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 891FB6B0093; Mon, 27 Feb 2023 17:31:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 620DF6B0096; Mon, 27 Feb 2023 17:31:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3CF836B0093 for ; Mon, 27 Feb 2023 17:31:49 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1A5CB1C605F for ; Mon, 27 Feb 2023 22:31:49 +0000 (UTC) X-FDA: 80514520338.27.2FBE7E0 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf24.hostedemail.com (Postfix) with ESMTP id 32491180003 for ; Mon, 27 Feb 2023 22:31:47 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=oDgDMJzf; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537107; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=sRvzlorvbRuacARz1OmsuoI0l9txi5DKbUTizxVaiSw=; b=LrsmPiB3UpY7GQpRZP391wIyjP+MymRWVBxDsd2HyHrdorPX6lmGOyuLBGC+6XBaybPe/F /glSUNnz3ZEdXqForkQ/Y6npuG9uhiCN4yXXdpXKeiDy1aLhOqEnV2gOy6NjglDwnnGsb+ 0XCIEXTWbU/aBaosHY+Pr7MgYlGWnZc= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=oDgDMJzf; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537107; a=rsa-sha256; cv=none; b=Ck9cGpzKFwvk2kO0SQCNqZGejqnaNRGEqkGodADDIEjQCoWN0hANRZj/2dPh7rwrU0Cb0f lVJdsIFf+ixDMYZcSOEspp1MPNATmdYLnnGNOsza9HOfJ8BLV3ZaFIu/3qXa609Eye9R91 TKajSz2GayFVnFh6g0Lx3n3zmX4Que4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537107; x=1709073107; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=s8OgmYq9WCVDjcUq/7zu6NLjqIcWEXmfAG1/rmhO7do=; b=oDgDMJzfK5KbN9aAdNNSRWeoyFGOavRY87AdMALT/4guYx7PEXpcjqbH 0tDZy9kT3o14ebjwGAwAB4sN86a7TxRsyZoV2TcJ7ycbus3iINgQPVWQj 5eDDXtIraxDAc0L8b6NjvEbc+nQYcEzP6O0KXu/jJEZkoP3sKr/mfrqVI 5AIN+g4HbcSmILqeAtOWYn+Nz1ydwxEb/NnALCXYP6mzs6MVEX4XLONKv 35EVak536vFdwBhpqXJpOWNe4//dNSl0MypjIyE13d2qcAqnsDxOSCz15 UiPrYx3siINZWLp5gUOyJG9Gqhr44nYAq85nxTr0ZJ+2ZkNnMcaiBBXbG A==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657498" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657498" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:24 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024608" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024608" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:20 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 19/41] x86/mm: Check shadow stack page fault errors Date: Mon, 27 Feb 2023 14:29:35 -0800 Message-Id: <20230227222957.24501-20-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Queue-Id: 32491180003 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: orypk9hdsd88je71apy7pe7mkn3axq5y X-HE-Tag: 1677537107-332969 X-HE-Meta: U2FsdGVkX19ZahD+biDeuXVYXoP6ZXlQuv+SRL1GEXXI8aSSvpIuOpQZfljkKt1GGvKcqF5YPSQopMVxQGq2AmwD2C8RmgD6lSBtyQMJIB0HuSmA8PWunh6R+XAf5ynwphgcfJGH046otma9PTtQc7Af1K4br4H1HW/uLxjgYIEIIr6+d9jaDEN3GcisIZYJA9uDmYcyztUVdXq+z2bDz+NnPXN3PlyrHjsc2lIZ6p1ehx+KgabwBnpd7KI9O6cm1emLm+6d4rOxGRuJLTmEY2Dmvigo+vgANw04Vhd6xdMpSbzGglBslITdypQfuDM78VGslzKD5UI20OPHO2M2nQVJoH9h/zzpYpoSxCMOdL+wEiVKPUbSMJz5r46jVDdkK8g9yVMu8uCFrmx/oCj7Ryc9NBI1shlzIBcXtEkqaTgWD3I9Gn4i6k511CKqKI80r5lKI2mBVOxR1B0RGngGGrdULwGfKRo8lL7zFmqMajjEqBXEBhrVFEsuwsoAauHwPmgzneT+RnalAftFiOB0hW1Zk58a2IvlL+ah6MYWbSxpqE1LSgAVNHH3CY8wGig7aOXLxCuOgGBcXQlBWMsoVqE+YG9Ureg9NfxKg6fdKfoluGZrC8jqs3c/FUCVI2zT/fVuA+uKJvxd14lvkLTLbf0s4DjgTgnYDOAi5wyznRuOsjaFyFHjHsUDCxmJQAoC/al7R5yqnMq8oqNGurWXgRYHJRkutyokEhTHsh05TOajObh+orKIHeij6/fOkmNPIdxbNyTu4+NG+UPZG1qIjauzLpCqMOK3Ud2n58Zpg9w/Ri6LG8kS9dtNVD21ShDkWccBbCxUBKbCxP2ytJdMSlGHiXqluzBjVfX8hk0drpaGWN3ONqt+nHyHOYWkZODUlMBN/2mkib+h8Je7/wqbv3ZLLJPCY+uaOOPQu4OoH3BA5qzDDTshOw1qMVFOC/iDBowk+QImI2AIKp6+by7 ls/V4kIK rVao3OsZ4LuswT73ggwvc8I7qx0UYMlrqWzltNqtbreijJLlLcabSx03YWuIKSjzmEyIxAOvVIqsiy96Jm0qUtinQ9uR7UbCP1FR+/KVwhKh/vGwCpwibC9xbSnRrljjU1cXMSw50LfAuhbr4iyZHASiQIxYuD9IjDlMOiYolBz5zQII6fiESC6YtShyqVGV/0eEkWWwQkowKlLx2oDYtzWZdid7qD7tjrLOx+v6G1DNBYiu7g9BFaoPRtYZB4mJiYEgGpEeaUcT+5ST2NhscPlF8RNirvLIya/7w1iGitKUjmIoV0iZlM5LDmKysHunPQ/yChef9CUVP62H24Hx0mFeG4l5IhvB0qCJU X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu The CPU performs "shadow stack accesses" when it expects to encounter shadow stack mappings. These accesses can be implicit (via CALL/RET instructions) or explicit (instructions like WRSS). Shadow stack accesses to shadow-stack mappings can result in faults in normal, valid operation just like regular accesses to regular mappings. Shadow stacks need some of the same features like delayed allocation, swap and copy-on-write. The kernel needs to use faults to implement those features. The architecture has concepts of both shadow stack reads and shadow stack writes. Any shadow stack access to non-shadow stack memory will generate a fault with the shadow stack error code bit set. This means that, unlike normal write protection, the fault handler needs to create a type of memory that can be written to (with instructions that generate shadow stack writes), even to fulfill a read access. So in the case of COW memory, the COW needs to take place even with a shadow stack read. Otherwise the page will be left (shadow stack) writable in userspace. So to trigger the appropriate behavior, set FAULT_FLAG_WRITE for shadow stack accesses, even if the access was a shadow stack read. For the purpose of making this clearer, consider the following example. If a process has a shadow stack, and forks, the shadow stack PTEs will become read-only due to COW. If the CPU in one process performs a shadow stack read access to the shadow stack, for example executing a RET and causing the CPU to read the shadow stack copy of the return address, then in order for the fault to be resolved the PTE will need to be set with shadow stack permissions. But then the memory would be changeable from userspace (from CALL, RET, WRSS, etc). So this scenario needs to trigger COW, otherwise the shared page would be changeable from both processes. Shadow stack accesses can also result in errors, such as when a shadow stack overflows, or if a shadow stack access occurs to a non-shadow-stack mapping. Also, generate the errors for invalid shadow stack accesses. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe --- v7: - Update comment in fault handler (David Hildenbrand) v6: - Update comment due to rename of Cow bit to SavedDirty v5: - Add description of COW example (Boris) - Replace "permissioned" (Boris) - Remove capitalization of shadow stack (Boris) v4: - Further improve comment talking about FAULT_FLAG_WRITE (Peterz) v3: - Improve comment talking about using FAULT_FLAG_WRITE (Peterz) --- arch/x86/include/asm/trap_pf.h | 2 ++ arch/x86/mm/fault.c | 31 +++++++++++++++++++++++++++++++ 2 files changed, 33 insertions(+) diff --git a/arch/x86/include/asm/trap_pf.h b/arch/x86/include/asm/trap_pf.h index 10b1de500ab1..afa524325e55 100644 --- a/arch/x86/include/asm/trap_pf.h +++ b/arch/x86/include/asm/trap_pf.h @@ -11,6 +11,7 @@ * bit 3 == 1: use of reserved bit detected * bit 4 == 1: fault was an instruction fetch * bit 5 == 1: protection keys block access + * bit 6 == 1: shadow stack access fault * bit 15 == 1: SGX MMU page-fault */ enum x86_pf_error_code { @@ -20,6 +21,7 @@ enum x86_pf_error_code { X86_PF_RSVD = 1 << 3, X86_PF_INSTR = 1 << 4, X86_PF_PK = 1 << 5, + X86_PF_SHSTK = 1 << 6, X86_PF_SGX = 1 << 15, }; diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index a498ae1fbe66..776b92339cfe 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1117,8 +1117,22 @@ access_error(unsigned long error_code, struct vm_area_struct *vma) (error_code & X86_PF_INSTR), foreign)) return 1; + /* + * Shadow stack accesses (PF_SHSTK=1) are only permitted to + * shadow stack VMAs. All other accesses result in an error. + */ + if (error_code & X86_PF_SHSTK) { + if (unlikely(!(vma->vm_flags & VM_SHADOW_STACK))) + return 1; + if (unlikely(!(vma->vm_flags & VM_WRITE))) + return 1; + return 0; + } + if (error_code & X86_PF_WRITE) { /* write, present and write, not present: */ + if (unlikely(vma->vm_flags & VM_SHADOW_STACK)) + return 1; if (unlikely(!(vma->vm_flags & VM_WRITE))) return 1; return 0; @@ -1310,6 +1324,23 @@ void do_user_addr_fault(struct pt_regs *regs, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address); + /* + * For conventionally writable pages, a read can be serviced with a + * read only PTE. But for shadow stack, there isn't a concept of + * read-only shadow stack memory. If it a PTE has the shadow stack + * permission, it can be modified via CALL and RET instructions. So + * core MM needs to fault in a writable PTE and do things it already + * does for write faults. + * + * Shadow stack accesses (read or write) need to be serviced with + * shadow stack permission memory, which always include write + * permissions. So in the case of a shadow stack read access, treat it + * as a WRITE fault. This will make sure that MM will prepare + * everything (e.g., break COW) such that maybe_mkwrite() can create a + * proper shadow stack PTE. + */ + if (error_code & X86_PF_SHSTK) + flags |= FAULT_FLAG_WRITE; if (error_code & X86_PF_WRITE) flags |= FAULT_FLAG_WRITE; if (error_code & X86_PF_INSTR) From patchwork Mon Feb 27 22:29:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B957C7EE37 for ; Mon, 27 Feb 2023 22:32:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C33F76B0093; Mon, 27 Feb 2023 17:31:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9CEDC6B0099; Mon, 27 Feb 2023 17:31:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7CE2A6B0095; Mon, 27 Feb 2023 17:31:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6278D6B0099 for ; Mon, 27 Feb 2023 17:31:49 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3B62AAACE5 for ; Mon, 27 Feb 2023 22:31:49 +0000 (UTC) X-FDA: 80514520338.27.8E4F269 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf28.hostedemail.com (Postfix) with ESMTP id 2BF81C000F for ; Mon, 27 Feb 2023 22:31:46 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=KLbbJMxZ; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537107; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=8ZDtthj99O2BCHZCVaXrdETfLIFDAgx1jaaLams/1qY=; b=bM7of9JHohiRHEf3gjpqiy5ZbE+OgHrWBwMcxXqiaM4bsLHhgyGgHSs329Ib5cPyQvHVBc KGUYAXlWc1Kpyc+50AL1Go4pXiusGKsAe0Y8c9+/zwJQuSCNn1TiCuojb6ErCymp1XoDHV QF0hUSmdP9WrGMIXaHPdb9mkXSOZXtc= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=KLbbJMxZ; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537107; a=rsa-sha256; cv=none; b=1zQiZ+79bDxiad2h9K0Go1ohEwBFb1kTmbD1Ij4PzCYA4MV/uevMP1kFAS8kmWhde5c7H6 b5d8Xx2RI8n+71js9X2c0wqdKNorvUMRWqSusbb9TscoczNsuAZGhW6aP8waBkybvAH6oc d3sUvpcOJynBirjO6xU9R+rw19KACAM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537107; x=1709073107; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=st63KrmVo0FweiQrqPB8uPrGYWL/hy9Xgx1d9xVLyzs=; b=KLbbJMxZAAxCVkebNTflhR8XxxSQWa+52suNxqT6r0CkUlOOAiRh03rR BjdEtMqcQwKFCW5yRHl1Rk0vmhmfd3SG7oiuEvUA+0mv8kkNlcfoDuidk UfUMpce/mhzPA8YVB+CyMLiRGDEZIsFMlGU94FTt3CTk7mujU19rX/9Ia kcaWfUgAJzqOLRCJnEYuvHMSo1VEiW2mJyCeOHTRfpyy335ucP+kpl4WJ ntPIwiiLuznGS1xbzJ5V8IpOQbGV//Ul/gp1PAfSUObSZjzGy7IwkQhh6 oO1OqdNOziE2QoRWV2x4SVuvVN+CK1QA6gP1XcTtbZKln8pDwAcNcFpMq Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657491" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657491" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:24 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024618" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024618" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:21 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v7 20/41] x86/mm: Teach pte_mkwrite() about stack memory Date: Mon, 27 Feb 2023 14:29:36 -0800 Message-Id: <20230227222957.24501-21-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 2BF81C000F X-Stat-Signature: 9udtwijnh3ma1yao5d34eh7chxgptb68 X-Rspam-User: X-HE-Tag: 1677537106-792140 X-HE-Meta: U2FsdGVkX1+LMxHHE9TctaJZx7NeHQjs+mkBywYx5330U/WIL32SkVYOyTM5ulaD7eMmHLIZcwFhKI5gPZ4vFZPeJVIFaHahdceg46zZr0iDHwGCQ8vfDj0/iuWm3yWJw3oVAG4lDECufAWwz7bg6rZoG7OGA+em16BsTP26PK02KmzdE0kfHMGY9qUsDYLsWvn6u0dPPXeRRzqMiXsyIuKCKpCUkrNDQgvhi1wa+s0dsBlVCU597IO9gqXsXkbrK/ExQtod05ozJ66XXPKqBZHIXbAjFmZ+TIGruzsdcoeyWo8g/WlC+gkcuZgVG5BdPjk3rgKaAfCcZo31A62q/7ZhZNOYJfv+xyCPjR9+N56hy98Zdgu/yYIUgxlIGOBf4Kj1XZrcbp3kaklc4wRTqf+6BucFWifXw86XAZrp0tj3rhWP3/AUuG9fXdlHpE3QpEbtlUiWEme8hMOk0IpO9LXAFxHBqm3RseuNkot08cS9bCC8oKKXk8l+KYOGT2yK8eJCY4r5Oju/5rOxf0I1hxExx88VdjpW9fAvJM5GmqC22OuSLm6ci9ZfECX8zB9R0jP8QjnergC3AZpdI/LIcmzWrah2JtCBUbml+49UpiOfYDon9BbnOImZGctV9lPeL8fd1PDEuXH8vcuPXsVH8ETV3xqexpglf2caAd0SFCNaFN+hXGOp+j6bqyQmdSe7fjC3WDUnNS9+u6AQ1y0eyuTHzlRwPAwABu+/Xu27k3JGuBMUnV6l6G7Q6mbquzQyroxgRd/s8SQewxAgkUN50hsCnguMRr6miy2TRgsSJdxCbP4pLXoiHcXR0wul4je2QmRnzxbV72LNtObFjApd7QnH72aAPDi6FFLr63pfq5SAENhjWneAVWGbn33YzOS6i1smrqEfJn55XMCM/NB2PKyY1INqrmWSEf9D495t6kilgeZSeKbQsvbI5OGFajX3jF+KEcb7WXD8G7t9F// YLuVv34e Q0cEdorQHWpm/xhw71GdTDyQGjzAlZ5IwwZbB039RaYbMTL7J7ymaWh9MsX1IZ5jXYZduRIq3NqPqwxKbG2jbZFjAT7Z3rNKx66CObWFVut7OEAE7hb2ubX5p+x6DSJ59kd5vpda6GrcvTRT+ntbBXwZq1eiwgzSsny8IBlHw4GGo2HV7s+aO0DHGr7ZCKKmAL8iwUNOFyVBvvrOCLSu6idpCje4RK/wzvZjQHJ0nEhTWLIGWbJBAjjDnXL/vPuNmJovDdMUtUFJSg1jnX7JA8WI16Qoh1ntRjU84Toa/Bw2IeFnh8DGZ8Z6usAnpKt9Ohpr3qkf1do2EwKggM8hIgnBqRHkXgYa/kbj1c9R44us1qFXfZcNpb1eKSKbGPCFiomhwynzMU/ojxSw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If a VMA has the VM_SHADOW_STACK flag, it is shadow stack memory. So when it is made writable with pte_mkwrite(), it should create shadow stack memory, not conventionally writable memory. Now that pte_mkwrite() takes a VMA, and places where shadow stack memory might be created pass one, pte_mkwrite() can know when it should do this. So make pte_mkwrite() create shadow stack memory when the VMA has the VM_SHADOW_STACK flag. Do the same thing for pmd_mkwrite(). This requires referencing VM_SHADOW_STACK in these functions, which are currently defined in pgtable.h, however mm.h (where VM_SHADOW_STACK is located) can't be pulled in without causing problems for files that reference pgtable.h. So also move pte/pmd_mkwrite() into pgtable.c, where they can safely reference VM_SHADOW_STACK. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Rick Edgecombe --- v6: - New patch --- arch/x86/include/asm/pgtable.h | 20 ++------------------ arch/x86/mm/pgtable.c | 26 ++++++++++++++++++++++++++ 2 files changed, 28 insertions(+), 18 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 05dfdbdf96b4..d81e7ec27507 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -456,15 +456,7 @@ static inline pte_t pte_mkwrite_kernel(pte_t pte) struct vm_area_struct; -static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) -{ - pte = pte_mkwrite_kernel(pte); - - if (pte_dirty(pte)) - pte = pte_clear_saveddirty(pte); - - return pte; -} +pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma); static inline pte_t pte_mkhuge(pte_t pte) { @@ -601,15 +593,7 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd) return pmd_set_flags(pmd, _PAGE_ACCESSED); } -static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) -{ - pmd = pmd_set_flags(pmd, _PAGE_RW); - - if (pmd_dirty(pmd)) - pmd = pmd_clear_saveddirty(pmd); - - return pmd; -} +pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma); static inline pud_t pud_set_flags(pud_t pud, pudval_t set) { diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index e4f499eb0f29..98856bcc8102 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -880,3 +880,29 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) #endif /* CONFIG_X86_64 */ #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ + +pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHADOW_STACK) + return pte_mkwrite_shstk(pte); + + pte = pte_mkwrite_kernel(pte); + + if (pte_dirty(pte)) + pte = pte_clear_saveddirty(pte); + + return pte; +} + +pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHADOW_STACK) + return pmd_mkwrite_shstk(pmd); + + pmd = pmd_set_flags(pmd, _PAGE_RW); + + if (pmd_dirty(pmd)) + pmd = pmd_clear_saveddirty(pmd); + + return pmd; +} From patchwork Mon Feb 27 22:29:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154254 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F2A5C64ED9 for ; Mon, 27 Feb 2023 22:32:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9F5E36B0096; Mon, 27 Feb 2023 17:31:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9CA9A6B0099; Mon, 27 Feb 2023 17:31:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8458C6B009A; Mon, 27 Feb 2023 17:31:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7335E6B0096 for ; Mon, 27 Feb 2023 17:31:51 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3FE79C03CF for ; Mon, 27 Feb 2023 22:31:51 +0000 (UTC) X-FDA: 80514520422.07.7C6CA57 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf24.hostedemail.com (Postfix) with ESMTP id 4FF01180017 for ; Mon, 27 Feb 2023 22:31:49 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=g+DJINxj; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537109; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=YgqLQ+1vy4zj1LyeIgLZtOZ931vA8iNoFckEDb4u1rM=; b=iOWmyNFWY0FQNfnykULgUt2IqUXI6ycz60RQ9ICjBaPCoxR3QAfq3+Ol0UiUV4LJ/9rue3 UY8PJiba5w4nFBXRt/hjNWQ2NSNvyLYB/7xI9OwybhWkh8BdlveJlXNRxupNqnZev4En2p engEriEKbm+VEpGAlE6omslucAuZjLk= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=g+DJINxj; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537109; a=rsa-sha256; cv=none; b=MI58fw4ASbPkocnv/w1vz+5HXBdiCgWmLxZhUTjBGCKwy2QP5iZkySpfAkI+OpSV4tRVAR htoYVsiSzgYJs6U7AR8/edIXY0CXzKY8MVfpqcWGZ+VIVjQZlqN98AOJm3R1ewtfy4YJMA kIA3IAsHyzB+ouHZinSCvMUKBunjhxI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537109; x=1709073109; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=iMZ7pHmGnRJHR2PLdV4imtJgm0cCw3zHhRVdTgGS/n0=; b=g+DJINxjOjxEF6A4JPjXV91iTyxXuYevzdcPfyyX0LorssUd4FWTT2R/ 6OJdRrd+pq58OC6JcTqFT9r2w27Sia81ZQKQ8vlRazFPrGabziKciVtkH JoKl1Dn9+KZi769RR5FBWSgjbuAQ9mv2I1rHsURVtmCoZLpWVLo2j7Rdg 4NXckOWbEiHOpJMgWvMJo3QTJ/n9+g/yvliOlFvl0fhgfs/hAvyrLYj+G UyQfthQBIg9GvdJ5ZRBBA2W9PfHuwh1UElwFnxmZHlEgYhRh8hwuPLcUx fow28QnZDw5a1tq8dTmBIAI6l3zSg9IzB+VsspzdXObrKznimqKIf3UI+ w==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657514" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657514" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:25 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024623" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024623" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:22 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 21/41] mm: Add guard pages around a shadow stack. Date: Mon, 27 Feb 2023 14:29:37 -0800 Message-Id: <20230227222957.24501-22-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Queue-Id: 4FF01180017 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 8odb98hqqsqzk9rwryhda57bxek7rew7 X-HE-Tag: 1677537109-261903 X-HE-Meta: U2FsdGVkX19A1SIhePhcrOuMGihZM+2OfXMCXcY4ZPSVwsq/vdYbn+0gulJ8/djp8T+oiWCCOr/PYxmXdo8/myjWGkjd4Jm8yMmMtOs7nwtzVU55rTITAk3sA260UO+rUE8g1Taw2NYhFFiGb1zl2Ww2dyUTiZvqGrLDrPEJ8rGJCZ7wyZgP6pNAbFI4nKSuQfx9GWQZ81b85dU2jUzwQmXXoIcZo+1dYEtrrH1DLyyPJBkc7eEaSrrpvJw5ADgXcFIGW7BuxlyGoz7ejcSMdPIbejfenPOhvPYsg3QTwCfXnFwkzPG45YjA/yIcZTQleykh3hJp1gs+BuLty0adEInf6OCswdj2Zic+BVkzkVtHD2nOFEyA+6mRyfcgyX2K6e4WPSO54CeK+W+kzA0quT3UVLWOIOPcdanS5qXirg0w0eQ+9ybPPLsVCMP30VJNZO2WGKhmtQTF5A4Y+5JhCV2LADrk7US/rJpCmeHQCzxmnchMMkgs2YseocHvEkMO/pWq+jnwoW4ZAZlHfizHi/3yafffRvXg9Pykvwdp/dFoaSua1al3HO/QFB8P0DN/rrsxMCO1CxNk85e7E0WLPbrbTNan3adWIduHTbRwtUAJMy0fx7BPuuqX4oQsgfap4pggTciM1Cnr+iaNfW5xQuZZschf29gs0PQdOGjmIHoi6s8HFwWDB95K1mnCnP5pyRvlsi0uOAT6M/u9SHR9iTU4VjSkInVQDLUgRDRYcjyXaiNFFf5HPMvxE+Uw5/lDuTPjKBRxiDOGyZroE5/RtVWm6orDTkXup0kOvOYNLhb+a9pa3yDW/AHzyiq6GRLxJs9jBBzgu/PKevp9+GfLJ2gaNUtB2oEdAvTefuGtFCiC9XPkvmld9dtOX4GX3UK3QzwZ6Y5V5GJPmHF7Y2pOxM/bZlcZ+7nJTkgpFtVWyrO5kzRAKjGi9sOmdrtcl4vDyFPGPgWXW02djvf5jSd Tm296xYX DmxPbqng4FTzs40SqX8RrPeiKGc0qlVPgADO8EVHAUKnxgbtroMmaFOaQoWigCp31Lc5OoicyY+Md+MdjXWcI1SpNpgVTDlYdQpd+AoE3e3E7JujK2xI82xd/fiL3+hjewTu5BonjrIUWd/25EmEOozUqup2TkojroQ0C1NtEFbbr72o4QiNhIs6HJh4R0+34jdx7+vfmtN13qp3ZBGYKwuF898RxPVNvX274xmJ8vj3ecZ7Yr+An36Ev38PHbe5l3u1wzbT9kVD8sRD/BxKR6hZ3VbaahGNOolHZzqE1r7iQzkqQZZpv8XtY1EgQYErL70bZiXumL0yLN0XVS+9pUk35cN65FyG7eEcV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. The architecture of shadow stack constrains the ability of userspace to move the shadow stack pointer (SSP) in order to prevent corrupting or switching to other shadow stacks. The RSTORSSP can move the ssp to different shadow stacks, but it requires a specially placed token in order to do this. However, the architecture does not prevent incrementing the stack pointer to wander onto an adjacent shadow stack. To prevent this in software, enforce guard pages at the beginning of shadow stack vmas, such that there will always be a gap between adjacent shadow stacks. Make the gap big enough so that no userspace SSP changing operations (besides RSTORSSP), can move the SSP from one stack to the next. The SSP can increment or decrement by CALL, RET and INCSSP. CALL and RET can move the SSP by a maximum of 8 bytes, at which point the shadow stack would be accessed. The INCSSP instruction can also increment the shadow stack pointer. It is the shadow stack analog of an instruction like: addq $0x80, %rsp However, there is one important difference between an ADD on %rsp and INCSSP. In addition to modifying SSP, INCSSP also reads from the memory of the first and last elements that were "popped". It can be thought of as acting like this: READ_ONCE(ssp); // read+discard top element on stack ssp += nr_to_pop * 8; // move the shadow stack READ_ONCE(ssp-8); // read+discard last popped stack element The maximum distance INCSSP can move the SSP is 2040 bytes, before it would read the memory. Therefore a single page gap will be enough to prevent any operation from shifting the SSP to an adjacent stack, since it would have to land in the gap at least once, causing a fault. This could be accomplished by using VM_GROWSDOWN, but this has a downside. The behavior would allow shadow stack's to grow, which is unneeded and adds a strange difference to how most regular stacks work. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook --- v5: - Fix typo in commit log v4: - Drop references to 32 bit instructions - Switch to generic code to drop __weak (Peterz) v2: - Use __weak instead of #ifdef (Dave Hansen) - Only have start gap on shadow stack (Andy Luto) - Create stack_guard_start_gap() to not duplicate code in an arch version of vm_start_gap() (Dave Hansen) - Improve commit log partly with verbiage from (Dave Hansen) Yu-cheng v25: - Move SHADOW_STACK_GUARD_GAP to arch/x86/mm/mmap.c. --- include/linux/mm.h | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 097544afb1aa..6a093daced88 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3107,15 +3107,36 @@ struct vm_area_struct *vma_lookup(struct mm_struct *mm, unsigned long addr) return mtree_load(&mm->mm_mt, addr); } +static inline unsigned long stack_guard_start_gap(struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_GROWSDOWN) + return stack_guard_gap; + + /* + * Shadow stack pointer is moved by CALL, RET, and INCSSPQ. + * INCSSPQ moves shadow stack pointer up to 255 * 8 = ~2 KB + * and touches the first and the last element in the range, which + * triggers a page fault if the range is not in a shadow stack. + * Because of this, creating 4-KB guard pages around a shadow + * stack prevents these instructions from going beyond. + * + * Creation of VM_SHADOW_STACK is tightly controlled, so a vma + * can't be both VM_GROWSDOWN and VM_SHADOW_STACK + */ + if (vma->vm_flags & VM_SHADOW_STACK) + return PAGE_SIZE; + + return 0; +} + static inline unsigned long vm_start_gap(struct vm_area_struct *vma) { + unsigned long gap = stack_guard_start_gap(vma); unsigned long vm_start = vma->vm_start; - if (vma->vm_flags & VM_GROWSDOWN) { - vm_start -= stack_guard_gap; - if (vm_start > vma->vm_start) - vm_start = 0; - } + vm_start -= gap; + if (vm_start > vma->vm_start) + vm_start = 0; return vm_start; } From patchwork Mon Feb 27 22:29:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154253 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F0A6C7EE31 for ; Mon, 27 Feb 2023 22:32:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1DC066B0095; Mon, 27 Feb 2023 17:31:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 16A846B0099; Mon, 27 Feb 2023 17:31:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DEF176B0096; Mon, 27 Feb 2023 17:31:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9920A6B009A for ; Mon, 27 Feb 2023 17:31:49 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 74672AACE5 for ; Mon, 27 Feb 2023 22:31:49 +0000 (UTC) X-FDA: 80514520338.22.5A864A6 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf07.hostedemail.com (Postfix) with ESMTP id 4C9EB40003 for ; Mon, 27 Feb 2023 22:31:47 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=emI5I9Wf; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537107; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=+kqsDAD87DHH8at7rwF6//m6V9jqq8z7byrF3dP8xHg=; b=kTh3yEoM4gWJcrPKkjljoEBVs21jwNAS+eltVgKuJ95tOdrBKAzL7YPke6OrDBJxEIW2Mo OdkMF1tT7lzillu6LJQwn+1ew5TBRt7bAa8Tez3g7DTEIo3z+ebKUNDO9piu5M355HRpnj sAbNJk/QgQSm8F2Vbi+HAucHVK32tOo= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=emI5I9Wf; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537107; a=rsa-sha256; cv=none; b=fErVvVe04aqlbIv0Eip4j7osm8budYuqiQ4ghhTRSErFznBeO41HDCdEFVNC+lsS8zKzKh MjsBblyKW5jYWOMTnkNz5xuSsQ7HOx22yfVjGdwmz2UHPDqhTXMVo4mvv5BYECUfF2BOiH yivOd3g3ONc6Dpgj7b7t/r35+gCEEMM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537107; x=1709073107; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=ZIfeT04mKir/YNhHsgsFjFvjvr/0UP7Swg3kZIhAJQw=; b=emI5I9Wf0/c85qRhbZNcoqugc3ao3FY6fP+QezXEE9FqJ+sFt35EpV2R 9g3FjuUd/Za0RnjRzL+RpFh61pDajbPoaN7mVvhFcQUoRZA19GsE2d57s Tn454XQ/+UaK6OxuoQap3bspazxr0Q9+8naffNRntLoqm767YpCs1Mc6Q joIu0nDPz4UcFbZpbrSenriREVPCxizwPsbLtKBBKkbWgRf0kUR24Rj8M 2qgMpktC5F5/K+WYbYJTh+O1LxoPFeSqhqzNvIR1K5Rzy4/Jq4tKjt719 UEVTkHcL46vly5g0czthGM/ilfIXL765NdJ4njMqwXVCmAmwQJI7FHY2d A==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657508" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657508" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:24 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024631" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024631" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:23 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 22/41] mm/mmap: Add shadow stack pages to memory accounting Date: Mon, 27 Feb 2023 14:29:38 -0800 Message-Id: <20230227222957.24501-23-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 4C9EB40003 X-Stat-Signature: sd6yhq6upgbhjej69wppg7bfbc5zqtak X-HE-Tag: 1677537107-635563 X-HE-Meta: U2FsdGVkX19hdnm5iuyKLHL7V6LDWTmFOpkoUPoFx07SVCk77y0bps10AGv9tI8XlRof5DFroAvyazdvSNlPk9Jb4gl499upi4pg5qKO12fqkrov2GGWQPv3ltLsD2eHe5WScpFkgjxeFsTl8pUTO5AbKdv2F9PMEnx0D/7E8twgM0wSFsfTzSTxbRy/Mz6zKKmKB3t9F0XyIK83r4zMgdXw/jpjKiiOcFbcbX+FjGAYqLLdMN12yYVXC4gnZ00LqjMc8A02GRe7cnf6BYoR0pTPqn8fUiiqjvlQr5CDOuB+GpRU3P7JHpqvTSmo5O2D1pUgGoMafPi4F1A8TbnMDELNJP8Yp8IT0KGVgNEi/X/ufibeBg3SsGrtxnwocdtfJvvo9uYP6PX+zj2j9DXscmYprS+LDm4wDwPlUlss24GMvQ4eQajPsPzoO8BnaWPsPu8yUI2yjgh/acr84pU8PuESxJU0MFcV7rGXPcDi2VTGSyGbuzqR/jWJgVuzjhV7lCOcMZYKFMJFERQ7Wbmk4mH6omYROiVTOfL9/wiI9wm1Yssaijv1EtpA1vjAsHGx+L7hJL2dI+qynedykCkBs0bas9O5sh7rEKyQJs03MCXVLmJurV5HtIX2NLGoQy4VVeBoY4gllkC5hHL0JQuT+6vWJYrH4wQ9pZhOEc0Bay93c7/C7AT0TDljYaTQwytXauw9E3KhM7LP/iiACEpFabp6Z/q6qvvvEb6VIurBNHjvFV7VdhHy/d3K+yfbc0Odq3QdbY2p+P0KFmGvw7mQMGrWdseSzvJaH+nU/Gd2yOWwIIZgidxiij6YGmoLqMBqYC9mif2qL7oCSbFNjFW2OKkptABEc913H00AXSzI8O6pAd24cEsUY2PFczJdkx1HS5gyEwRr0cK9vPq2eH4Q391aKzh1OFEhjnY3/Vp9IZ3Ab9LGf51eorcC+jPk9rOeUmK4nJvZutgqse42sFj n64IBXgX eeX9sRZj77iY67COnkzmc6MYvbn1+u3IpwYXulCmWeaZ8be5V/MLBBvV7zHGnzkcCNPztAKI7k2mdIcCLsugY1Be+PMUKMLRoxX2ihG8ktU8gHvDxdt2eFSP42j3d9gAmwr75kf12b7I+t34oHOTE6DGf4HQZgnd3DO6Y8JCoCeul575xJth/qf262arSvjrtRt5djxMvO08eqFdBWqoOJqZUJGigviqGxZpWQzoQ4hVKd9oPnmNhAOtwQfM6yBEaq2erPnC3vQwoifeUFj/FAzdFCGmlNGsRYlfLVBUY6tvLnCLCihA3B738uc+HhVNST5aQ/5e2JRT7cQITmCHJ62IUp+HmSl/cJc4M X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. Account shadow stack pages to stack memory. Do this by adding a VM_SHADOW_STACK check in is_stack_mapping(). Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook Acked-by: David Hildenbrand --- v7: - Change is_stack_mapping() to know about VM_SHADOW_STACK so the additions in vm_stat_account() can be dropped. (David Hildenbrand) v3: - Remove unneeded VM_SHADOW_STACK check in accountable_mapping() (Kirill) v2: - Remove is_shadow_stack_mapping() and just change it to directly bitwise and VM_SHADOW_STACK. Yu-cheng v26: - Remove redundant #ifdef CONFIG_MMU. Yu-cheng v25: - Remove #ifdef CONFIG_ARCH_HAS_SHADOW_STACK for is_shadow_stack_mapping(). --- mm/internal.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 7920a8b7982e..1d13d5580f64 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -491,14 +491,14 @@ static inline bool is_exec_mapping(vm_flags_t flags) } /* - * Stack area - automatically grows in one direction + * Stack area * - * VM_GROWSUP / VM_GROWSDOWN VMAs are always private anonymous: - * do_mmap() forbids all other combinations. + * VM_GROWSUP, VM_GROWSDOWN VMAs are always private + * anonymous. do_mmap() forbids all other combinations. */ static inline bool is_stack_mapping(vm_flags_t flags) { - return (flags & VM_STACK) == VM_STACK; + return ((flags & VM_STACK) == VM_STACK) || (flags & VM_SHADOW_STACK); } /* From patchwork Mon Feb 27 22:29:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154255 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F36E5C7EE3A for ; Mon, 27 Feb 2023 22:32:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C41E16B009A; Mon, 27 Feb 2023 17:31:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B2A616B009C; Mon, 27 Feb 2023 17:31:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A3A06B009D; Mon, 27 Feb 2023 17:31:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 87B3F6B009A for ; Mon, 27 Feb 2023 17:31:52 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5DA0D40A1C for ; Mon, 27 Feb 2023 22:31:52 +0000 (UTC) X-FDA: 80514520464.25.441FF0C Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf28.hostedemail.com (Postfix) with ESMTP id 5E60DC000F for ; Mon, 27 Feb 2023 22:31:49 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=lheS49sH; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537109; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=qAyyzHuhpB1+Uby4j0qT9fRmrng7fTbHGa4BAOyou28=; b=Tvpya5zHVN4FDSs4ew+zIYJC6eEjz8ufT33O0sMrHI/476prI1/SFllbZ0nnGCCk62usD9 92leXaLPi20QkmAF9U4Ge4g5pertzG6zb12uxXTnV77lHF2J8Nyktp7XFjsM/gVwpH3otR xJdLdv9K1H+pC5iDRz/yQji8bjeNHKs= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=lheS49sH; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537109; a=rsa-sha256; cv=none; b=uf000lPmDjjhX1hdrilfP21Q52XHSOgEtK3yQzQQFMBuOCUAW1gB2BA1cLIAGGJWW0xdr6 cyHsA6VLA/Mhaf7f+qX0Z672JF8z+Liie3yCQYkSU7MAFcNE6dSQlBpNYQ1sjtfF7ecTQG a4wiyZ5JW6O75jzIBnCz44oqwpTPcHA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537109; x=1709073109; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Wm5/KWrYYFgliIdaotUWNuGKUFxakUNYXMA7lYNP4nA=; b=lheS49sH3AyRf9AuP3pfgUdYLqTbaz4sXIROrYKAae282h3/AeXEEkry 7gz3FI/9ovSQQ3+aVKLDPs3y9UtEb1GsDU2zBMtUe1sqRo9tHBskzRb2F xW5C0to/dWAmaB3eAH+84L9Z5xYxkYtvIP6V8IYoP4/ZeLGFvMOMBmJBC WO4kSekasHl8IpRLkylNH8t3CmqpFBLWXKRXA/sJFtm9mTI1PXiIns871 H4QE1NpsgYDWYry637M7sCXI+UzxOXt9JMhyCI2Y0YP6LM1O0bIEDZYR2 SlnIuHen0Su1ELI9tdpvQ38RPz4wFzJHO0FYZBX2VVdaa5B1QjcZl7WD2 w==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657531" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657531" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:25 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024644" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024644" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:24 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 23/41] mm: Re-introduce vm_flags to do_mmap() Date: Mon, 27 Feb 2023 14:29:39 -0800 Message-Id: <20230227222957.24501-24-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 5E60DC000F X-Stat-Signature: 9ibs59eydxajmmk57qpxqpcanbkep1tw X-Rspam-User: X-HE-Tag: 1677537109-820704 X-HE-Meta: U2FsdGVkX1+Aovv8Hf6tu6iVBJUFSlSYWxWDes4wZJrqdkGkqtIo9dNntfQGpONEIyE5TOCqVblaeknmD5cUDTelyHrTk58l4HN0Nt3JO0LFIsE7XLTFicPyA5w4yXTmD9uV7bn5pn0xFrE47OspHa4xHWijvwJSX/MxE/C8BmUK0oh4S/BsdLTSmq7j4CnE3nl5yRbNv2tNFCqGnFGma4E9G7xyJZ1fflRzohcWwB10e17gdnD2WyGL1RNuTgLdiUChLfSUkHtXRWFMxM0RN4CP37z30/hrghzmF09djKAojrfcjYSkDO3meWcJWDzPztjyOqBV01SxuOgeFmJgxkfpSNxQenDDDUJfzcdHEUXYQpWew6WvGcMdnKczaogJlOfVEPyuOc2QhWjO0VhrQhDhWhwZbd0piH0nA6KA0RmBvsgvRNUixOoxMYFEgyAoUY/IM5a3YM7yps5Y0pMSDU2iZGbT0pJY49y4sZS4JW6YGlXX1D8WR+My+hEVUpYgmKKgkcmSUoMes453BbxuvfwRM/IV8QbqJMICexGqbN8atiNy4699IuB4t+GkYoR9aDvVLU+5sS07sL1FuSekNuTCcpbpYvMAXOHzFwwRI127TWy7W2qbsLIN0woccrRsWPKIK21TAlXJjIlWNUMCAijGSbB1jbU7qAuHgZXJcW6ED/zIs9ghvjno+VOiNhKM7tgQbay7qH27o4LAlDFoKlUDpD442TyKPSfxygyEYNyMI6ARG3oFO8y0pnGlaCq57N71H5icu14qJnStizI+4VR36n5F7Jcyr/oQEIPKI0GFLru1Ncc0+w1EDZPYRXm24g5452SytLeYgfosuFRxgUDUG4D9i5on5644qgWC8GjmiZcE1Kw0E30HwmkzVqGHiCsJ71OKalPQvcnxCZX7kTX8EtmmRUPUNM6C6VN9pqGEVHZz+SiRrwETLtckaHO2Lzsqc9vizIL6+IBecRY 3M1CnxyK TZ1oUqdI42c2GxG1OW2sHFZ7n16z22Evw7je1YnDfWWJVyAAlIFrmxqOCO2JDtbxDr+qHXsuSLu7irkPVnDtLvKmZAco1e+RsUDWYPXQd5ZF1RzyrzV2rinVPOSpMZt15/bWomOFVDtRIrBBd1e+kMtaraCOrmq2MwN/De1PktNOWg0Q9oP006hY4d4pF3XS7cuXR8JG1+u2YdmSnEWElR3hl8vS1XZ3D9/GqRXiDrzhs5l3s56jgEHyHdrcao/utkMpi0zD4ADfywsEpSOFIjOUd7cbfe4fXuqwo9ebhPBsvXrv472LP+rJ6uMuDOgCB4WmWsGxEHs2EeqJGGMfzIdw/ld1VguOrOr7h5u9tjnfFb+2harESXT9x3w6qh3Dk9Hgrgqbap8EukKCaMEXzyUta8AID4rYD2ecp9vQQPKgTJElKDjOCArxbp9xqDmDWm2bKOPgMpnwJSHopvRQtZLy5lMoQi6Dtm7LgkIPxa4p16BReP9hQTgMjytmz1iClu8DldS7AbDT+XTnoy3mRUwmH+sOXfUF/ERg50JkSc7Sjlqx06GN2LXJ9cw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu There was no more caller passing vm_flags to do_mmap(), and vm_flags was removed from the function's input by: commit 45e55300f114 ("mm: remove unnecessary wrapper function do_mmap_pgoff()"). There is a new user now. Shadow stack allocation passes VM_SHADOW_STACK to do_mmap(). Thus, re-introduce vm_flags to do_mmap(). Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Peter Collingbourne Reviewed-by: Kees Cook Reviewed-by: Kirill A. Shutemov Signed-off-by: Yu-cheng Yu Signed-off-by: Rick Edgecombe Cc: Andrew Morton Cc: Oleg Nesterov Cc: linux-mm@kvack.org --- fs/aio.c | 2 +- include/linux/mm.h | 3 ++- ipc/shm.c | 2 +- mm/mmap.c | 10 +++++----- mm/nommu.c | 4 ++-- mm/util.c | 2 +- 6 files changed, 12 insertions(+), 11 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index b0b17bd098bb..4a7576989719 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -558,7 +558,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned int nr_events) ctx->mmap_base = do_mmap(ctx->aio_ring_file, 0, ctx->mmap_size, PROT_READ | PROT_WRITE, - MAP_SHARED, 0, &unused, NULL); + MAP_SHARED, 0, 0, &unused, NULL); mmap_write_unlock(mm); if (IS_ERR((void *)ctx->mmap_base)) { ctx->mmap_size = 0; diff --git a/include/linux/mm.h b/include/linux/mm.h index 6a093daced88..87e46a9e0e93 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3014,7 +3014,8 @@ extern unsigned long mmap_region(struct file *file, unsigned long addr, struct list_head *uf); extern unsigned long do_mmap(struct file *file, unsigned long addr, unsigned long len, unsigned long prot, unsigned long flags, - unsigned long pgoff, unsigned long *populate, struct list_head *uf); + vm_flags_t vm_flags, unsigned long pgoff, unsigned long *populate, + struct list_head *uf); extern int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm, unsigned long start, size_t len, struct list_head *uf, bool downgrade); diff --git a/ipc/shm.c b/ipc/shm.c index 60e45e7045d4..576a543b7cff 100644 --- a/ipc/shm.c +++ b/ipc/shm.c @@ -1662,7 +1662,7 @@ long do_shmat(int shmid, char __user *shmaddr, int shmflg, goto invalid; } - addr = do_mmap(file, addr, size, prot, flags, 0, &populate, NULL); + addr = do_mmap(file, addr, size, prot, flags, 0, 0, &populate, NULL); *raddr = addr; err = 0; if (IS_ERR_VALUE(addr)) diff --git a/mm/mmap.c b/mm/mmap.c index 20f21f0949dd..eedae44dfc78 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1191,11 +1191,11 @@ static inline bool file_mmap_ok(struct file *file, struct inode *inode, */ unsigned long do_mmap(struct file *file, unsigned long addr, unsigned long len, unsigned long prot, - unsigned long flags, unsigned long pgoff, - unsigned long *populate, struct list_head *uf) + unsigned long flags, vm_flags_t vm_flags, + unsigned long pgoff, unsigned long *populate, + struct list_head *uf) { struct mm_struct *mm = current->mm; - vm_flags_t vm_flags; int pkey = 0; validate_mm(mm); @@ -1256,7 +1256,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr, * to. we assume access permissions have been handled by the open * of the memory object, so we don't do any here. */ - vm_flags = calc_vm_prot_bits(prot, pkey) | calc_vm_flag_bits(flags) | + vm_flags |= calc_vm_prot_bits(prot, pkey) | calc_vm_flag_bits(flags) | mm->def_flags | VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC; if (flags & MAP_LOCKED) @@ -2829,7 +2829,7 @@ SYSCALL_DEFINE5(remap_file_pages, unsigned long, start, unsigned long, size, file = get_file(vma->vm_file); ret = do_mmap(vma->vm_file, start, size, - prot, flags, pgoff, &populate, NULL); + prot, flags, 0, pgoff, &populate, NULL); fput(file); out: mmap_write_unlock(mm); diff --git a/mm/nommu.c b/mm/nommu.c index 57ba243c6a37..f6ddd084671f 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -1002,6 +1002,7 @@ unsigned long do_mmap(struct file *file, unsigned long len, unsigned long prot, unsigned long flags, + vm_flags_t vm_flags, unsigned long pgoff, unsigned long *populate, struct list_head *uf) @@ -1009,7 +1010,6 @@ unsigned long do_mmap(struct file *file, struct vm_area_struct *vma; struct vm_region *region; struct rb_node *rb; - vm_flags_t vm_flags; unsigned long capabilities, result; int ret; VMA_ITERATOR(vmi, current->mm, 0); @@ -1029,7 +1029,7 @@ unsigned long do_mmap(struct file *file, /* we've determined that we can make the mapping, now translate what we * now know into VMA flags */ - vm_flags = determine_vm_flags(file, prot, flags, capabilities); + vm_flags |= determine_vm_flags(file, prot, flags, capabilities); /* we're going to need to record the mapping */ diff --git a/mm/util.c b/mm/util.c index b8ed9dbc7fd5..a93e832f4065 100644 --- a/mm/util.c +++ b/mm/util.c @@ -539,7 +539,7 @@ unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr, if (!ret) { if (mmap_write_lock_killable(mm)) return -EINTR; - ret = do_mmap(file, addr, len, prot, flag, pgoff, &populate, + ret = do_mmap(file, addr, len, prot, flag, 0, pgoff, &populate, &uf); mmap_write_unlock(mm); userfaultfd_unmap_complete(mm, &uf); From patchwork Mon Feb 27 22:29:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154252 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F03DC7EE43 for ; Mon, 27 Feb 2023 22:32:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A3906B0099; Mon, 27 Feb 2023 17:31:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 07C7C6B009A; Mon, 27 Feb 2023 17:31:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE8B66B009C; Mon, 27 Feb 2023 17:31:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id CE0FD6B0099 for ; Mon, 27 Feb 2023 17:31:51 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A66B7806D6 for ; Mon, 27 Feb 2023 22:31:51 +0000 (UTC) X-FDA: 80514520422.04.E0959BB Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf07.hostedemail.com (Postfix) with ESMTP id B537540003 for ; Mon, 27 Feb 2023 22:31:49 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=TGrLU8H8; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537110; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=se2s0CGLP/xSzdXeus2cLfjx/jzzmrnWH6Cj7Cnle5Y=; b=Ic65Fn28CvT4Sbdy+h6s3RGa/iAQxveelBb52WcKlmDwbDYjNML1IJonfM8fDH7hFfNPYK LHWK7WaETWobbnacuLv1CuJWNf/bjs+jZURPwyFbR6+dLBtLhncF5kssZG9FEHE1VSB8So VDuYf9PNBPPGREggN10p2eMHNfB/YL4= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=TGrLU8H8; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537110; a=rsa-sha256; cv=none; b=G5gGzul5iHosOQ59+GFyh9GMD7kj6p60MOsVhfuijU2BAAUT8rdtGkIte3+w9L2jjIE3KM sB7efr4af1/W4PPsJWBeVVp1d8shdJgoQkbQVasBF2GMDVtqV2nu8Vy2GTQkFzh6At9Wll apKQiBWCzkOeuFoRfCV4qY4t1W6PFEU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537109; x=1709073109; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=g/mLbA9/W/OJJZ4lgb8cNHR8TFa9W9m/xYBMcnZGnDk=; b=TGrLU8H84lo76+Il3q/fc4WV+Tbxm5kfvpX7U+3nZ30qC7R8MCxXfy5M 5phzYuV5SYP51U8gll2iIjeoex0OotmucCnqy6vq5260p3zKutKUJjowc XsveOzGZXqfztocDDGytAT/nrTIHvmmt0faaVWNq7ZaPOFdNtFiLMzEaL fXGCvKe8YTaQfJmWAFPmiR+n6Nj5fMgdt8uOA/cbomguCnLppPRpbp1t+ 8lnQVL+BYEkBLE2igxWe0YUyapEIGvmQDE1UW963bvpfXBwo5ZjBVJ1p8 HybRnJhQIMKXmO1DABvRRsnZ8NsbgEQNQLaX6G8IkG3HJ/V+0HA9ldKOb g==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657566" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657566" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:25 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024659" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024659" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:25 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v7 24/41] mm: Don't allow write GUPs to shadow stack memory Date: Mon, 27 Feb 2023 14:29:40 -0800 Message-Id: <20230227222957.24501-25-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B537540003 X-Stat-Signature: t4snqdmaxeddo6mc9rn8r6amewhoq95n X-HE-Tag: 1677537109-171814 X-HE-Meta: U2FsdGVkX18mzwbKch672E2Bd0pNznZiCIAYCtT4jTukBax91OPYwthL8PMVabPijBYqouxJBudqcwMrhXedV1kAkxJRZJK+72tNTvJjtNONXTDLMjumpck3Rt+jrecUEejJ+pF5vvoWQUjWAknWCF+jNEvyR9TF2G9ukbBB/wgnTEPlHahkGkq851u7LW2RvYh7ZynBXiqzu6gzsEcFnA5jmqt1qqRWZXiIVNQBBOTrwa9dbWr53prgKTM8yGBPmJ9GW4GhlMvjrSezM2Qjq+DhwDC7AwIxdyi1jS0Q/Xt9yb43VrY4hpmo3Z1ZksfCG/zqjD2b/fdUYtt2ID7AiFgxgc9s1vjyeTYg/9MkcR+/s5WRQYlQucZtktkGw02WX17tLfH3krY7XuXdsDCdg5epAh0B0d4nk/87wez6z91nOO3Dd0dZHo2vIFhzSthKbSYDn5lYe1R+wdKESoYydYxPbVx9GuhHaWl5hgizWAGwToMfQ0iY4A2XtX8UudCswRb0GvczlfCMxRRzBH7zwkVcI1KL48DaSE6vYyWlRieZzERIjyAcaeQcklp89w5yZfVU75zbfPV8JeUDbxoL+JrabAAvnXn2gnja//5iaViHBGWEpIUU/OLjxi9IFFsU7A0BTmFAof7/9BlyttDHv9PuBXL9bNAtd5GNiyt9OiK6p2RaQUb3Gjd1PLT92zzpHhXJcxhsmy1gIZqk6Y0u+cBGkicSx5eJFs+d0ZW/3PWyxRndVydjaEl8R+Etsg0d6JGCYzypClRXymmpN52F6ZwyTKyCREJ6CrwCpQOC02M2aHTFeh7ZMVAGFCOd6wxN+0906DroP8PWDN09xhnlGkjEPE/8uBHZSNOpe/GPZT+xBY5EkIwvS5YwwNFXhs69sLS7E4LOmyYYxoNtqt/C7kDSSj7JAE2wai00MzKTazSkjXGcKioc4AdZ6jduE3XTOYnWL+nDmqRT3Exfjci Ln2HCgKV fiLui3CM8zGqppf7lTHYB0WifoHxk2Sm+d+6hP8N7NbnPljlGJDa2pYy0fl3g6ShYx0nTrForoB4X0blfudGfYQUkUujoouHLfQhGptWQteMZ+05PcsU7OI8lpIBLjB/kC6AwqjN3w2rsyUPue/D0cmQtjEuHRwhbtiBYbW+cvTti3fqI27pinfHCwWGTK0Kf/bcd1K1IcSiUBFIwmKiOMXNRjo4irw9UJ/RVxv6F4qd1NtgCXIyAG1pAEzhccrTy7tNueNiN/8L9Wg6aoHPXU/yqpFBW6GkTBZLMI75/WWHfJRU0gqYyr6Yr5gGl8g7QxHAUFu7SZg1Oglj62ByG7u8j0wiIa2y3DaQKXxugShZjTJyehoQLh+fRUg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. Shadow stack memory is writable only in very specific, controlled ways. However, since it is writable, the kernel treats it as such. As a result there remain many ways for userspace to trigger the kernel to write to shadow stack's via get_user_pages(, FOLL_WRITE) operations. To make this a little less exposed, block writable GUPs for shadow stack VMAs. Still allow FOLL_FORCE to write through shadow stack protections, as it does for read-only protections. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Acked-by: David Hildenbrand Reviewed-by: Kees Cook Signed-off-by: Rick Edgecombe --- v3: - Add comment in __pte_access_permitted() (Dave) - Remove unneeded shadow stack specific check in __pte_access_permitted() (Jann) --- arch/x86/include/asm/pgtable.h | 5 +++++ mm/gup.c | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index d81e7ec27507..2e3d8cca1195 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1638,6 +1638,11 @@ static inline bool __pte_access_permitted(unsigned long pteval, bool write) { unsigned long need_pte_bits = _PAGE_PRESENT|_PAGE_USER; + /* + * Write=0,Dirty=1 PTEs are shadow stack, which the kernel + * shouldn't generally allow access to, but since they + * are already Write=0, the below logic covers both cases. + */ if (write) need_pte_bits |= _PAGE_RW; diff --git a/mm/gup.c b/mm/gup.c index eab18ba045db..e7c7bcc0e268 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -978,7 +978,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) return -EFAULT; if (write) { - if (!(vm_flags & VM_WRITE)) { + if (!(vm_flags & VM_WRITE) || (vm_flags & VM_SHADOW_STACK)) { if (!(gup_flags & FOLL_FORCE)) return -EFAULT; /* hugetlb does not support FOLL_FORCE|FOLL_WRITE. */ From patchwork Mon Feb 27 22:29:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154256 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8AAABC7EE2E for ; Mon, 27 Feb 2023 22:32:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 256BA6B009C; Mon, 27 Feb 2023 17:31:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0CB9E6B009D; Mon, 27 Feb 2023 17:31:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DEA106B009E; Mon, 27 Feb 2023 17:31:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CA8446B009C for ; Mon, 27 Feb 2023 17:31:53 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9091AAACE2 for ; Mon, 27 Feb 2023 22:31:53 +0000 (UTC) X-FDA: 80514520506.15.2026C81 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf24.hostedemail.com (Postfix) with ESMTP id 6BF27180017 for ; Mon, 27 Feb 2023 22:31:51 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Iq0+aTfB; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537111; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=veDLluZZGIZBNK/wTadMCXK8+klKEMevtWOXQT8RUT0=; b=fyiSQi8CZrw1OhuwunLwzH0zDH5uBEAsAeaQ9Oo+tvp8ffn0uSQxlBTL+AqxXS73UarIzv Pm6gEgTi5kwrX5P08qqxBQfhT/C34R6irIV3mAls5qqsdNMQaqpJPjEX9bA0ymIR5lPprQ E75eCVyO/ft5pBsLeK92a8Ewg5KNy1U= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Iq0+aTfB; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537111; a=rsa-sha256; cv=none; b=VplEpVnEfqt/fRWknqsrzLpXBSNAGrLkhrnaa/tQUWrdEFQ37Pfp7himq9qrNWrXAQBcb0 TUcyl70Do58/T+pGQQRm1y4w+Mx+puoTGgIVF/8tfTzJRWxwBI+vq5lSw7sEhh45q5raTx sl1M/P359WWaSf/CKAtkRjdgXrz7dNM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537111; x=1709073111; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=FzRSuSkDyk6//OI0YkurvPKbmt+RZMfvZ31TH5a40rI=; b=Iq0+aTfBZY8f0yN+5xgGCYd2pMoXHdCHWiMvuWGo7gmHC188Uh6vkxd2 80JLk1woMT8f4NfUwnKvWluWSdBvdR1im4MUpACEadIsV67mr33zjeXQb WnW3zOregDU9+p+R7CuZqmD3e5KWILQFO1vk+1a6MLS/zplqRijSYROdf Qr72B6m2rwKc0ZjP4kbmV0o+BYCaJrP01pJrhu+OdAWhQb/LBv2VvOJnK 41ZfhFS90aVLTaCoVOuFGP3zXgbT2sCgoQRowKgrB/aAcbqI+u+LeQPOn NfZbW3t6DlL2Sv+9P1h7uiABDmEv0L3rk2hWlrlQyNXJfpmtrGmvSWMSW A==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657599" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657599" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:26 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024675" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024675" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:25 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v7 25/41] x86/mm: Introduce MAP_ABOVE4G Date: Mon, 27 Feb 2023 14:29:41 -0800 Message-Id: <20230227222957.24501-26-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Queue-Id: 6BF27180017 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: nmqg4b64oenxqk81e5nsdpdzn34gbws3 X-HE-Tag: 1677537111-358105 X-HE-Meta: U2FsdGVkX1/Xax84XrmqTjOY+RHEXae6oK35BEsZz+aM6iX4+14MyYABzBPCaAdtzIZ0GnNRS0gZKRqpQLDDJ9IOT9Rn7uyVZmWrsBwmW/WOolW0ApGDAdu7xqqh1O9CwZcg/E8IM2zvb4cz+MTaJPcWknH48Jez8BTps5nTamRfM15DP7yw1KnN+gZXWLFyaoepn0K860kaM8f5v9ZrVtQbNPCEtUBtJ7ZU9k4xsZJEj65uT6ceQeIWfs69TCZ54wQMDq4bZtou+C7LvV3K561tTTLKtDjqKI2EOH71QZ5iiVqS0NrwFHOzUlctmWp65vxy76gCaPY66IM1rVNTpuXeCQmOSYpoUMRuT0nxivLGjEiOIzPC1vUYHmeLBT8fSzOnH0HLumoklWcuWBikUjPhPhK6ydVsSzPOyQsEPKOissUuMKuzEMQxtkD4SljDbPdaligjDsLxHaR1OXhXqdt7rq4go+lahNKAqeRp5FDhiM6ekWMAkm67ZxcFQ5ceqI6zrrxIqu7BChUtEVKwBDdNsJICyUtgG2gtgpMmyz/PGaXUZ8aEqSNpSdJT7pv621F4dr8RcaEJFFwKDke48rv1eWfU57Hnp7S35e3MucNUWf17L3vZ5cAfMDC21/hsR34Fro5GCS6A3QemNqvFIykMIbiNQqPdBkPGOItmwlO7NaWY/8aoFXSZMIvgq4u48nWOZeoNok/eGRUcBOS7njhu0+7uP3zt6naAhjOs+wyCTi15Rl52xN3XuC7t/UiHOQqzl2dLErpoLT38iT01ZchlZKiyuSLF1IIZ9xaGNYVzqNuYb9AJwZX4uGHpB9nHZBwxzl9CWPwNARGbXbAz1PmNqobIooTRUIN9bahcDyY6/Y2h8fxdRrca+NEQeZUgQZYtRK8fejhA71x5mwexKc039gDsovbtFzXIo4CmzLw11h+HiRNebIvKFghePLInNsAPHlo+wOcJJ3xvh0M m3qB+UBr xJIBEGLjT9hL/LniZuiPA9rBURSkJhug6uCSRLFxUrVlxPRJ8bt6OkCQ1z7y8wSTaZ7Us4YBwzEYz8ox2V8FTGOhSvfD+zpMLqNjWzYkFAWMK0ngYloIsHYjXAayStmBW9jSTcP8YdHp4viwSLfhX2RkbeJDM2ugSvcYn2wIvW9Ei0LQ7dB9d+C9YdKXLx2au84lPYDYsLHM5KrEdF0K2XW+3uo6P/ScqNN7Xz8KLYphfuDZH1fVKMi/aZGs/THi/gcmGLjX3oPFZYw7mTptzMYSpvDY6q8GZXRR7H6e2AFeH1pizW/9JYt2IwpFbM84vsiEusR94hy1SQlsYW07rKB5SEef08gCSZvtk5gMwBR1ZjzD8NvDdT7UvAfdZWO1T2stmif/IJs8h304= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which require some core mm changes to function properly. One of the properties is that the shadow stack pointer (SSP), which is a CPU register that points to the shadow stack like the stack pointer points to the stack, can't be pointing outside of the 32 bit address space when the CPU is executing in 32 bit mode. It is desirable to prevent executing in 32 bit mode when shadow stack is enabled because the kernel can't easily support 32 bit signals. On x86 it is possible to transition to 32 bit mode without any special interaction with the kernel, by doing a "far call" to a 32 bit segment. So the shadow stack implementation can use this address space behavior as a feature, by enforcing that shadow stack memory is always crated outside of the 32 bit address space. This way userspace will trigger a general protection fault which will in turn trigger a segfault if it tries to transition to 32 bit mode with shadow stack enabled. This provides a clean error generating border for the user if they try attempt to do 32 bit mode shadow stack, rather than leave the kernel in a half working state for userspace to be surprised by. So to allow future shadow stack enabling patches to map shadow stacks out of the 32 bit address space, introduce MAP_ABOVE4G. The behavior is pretty much like MAP_32BIT, except that it has the opposite address range. The are a few differences though. If both MAP_32BIT and MAP_ABOVE4G are provided, the kernel will use the MAP_ABOVE4G behavior. Like MAP_32BIT, MAP_ABOVE4G is ignored in a 32 bit syscall. Since the default search behavior is top down, the normal kaslr base can be used for MAP_ABOVE4G. This is unlike MAP_32BIT which has to add it's own randomization in the bottom up case. For MAP_32BIT, only the bottom up search path is used. For MAP_ABOVE4G both are potentially valid, so both are used. In the bottomup search path, the default behavior is already consistent with MAP_ABOVE4G since mmap base should be above 4GB. Without MAP_ABOVE4G, the shadow stack will already normally be above 4GB. So without introducing MAP_ABOVE4G, trying to transition to 32 bit mode with shadow stack enabled would usually segfault anyway. This is already pretty decent guard rails. But the addition of MAP_ABOVE4G is some small complexity spent to make it make it more complete. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Rick Edgecombe --- v5: - New patch --- arch/x86/include/uapi/asm/mman.h | 1 + arch/x86/kernel/sys_x86_64.c | 6 +++++- include/linux/mman.h | 4 ++++ 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/uapi/asm/mman.h b/arch/x86/include/uapi/asm/mman.h index 775dbd3aff73..5a0256e73f1e 100644 --- a/arch/x86/include/uapi/asm/mman.h +++ b/arch/x86/include/uapi/asm/mman.h @@ -3,6 +3,7 @@ #define _ASM_X86_MMAN_H #define MAP_32BIT 0x40 /* only give out 32bit addresses */ +#define MAP_ABOVE4G 0x80 /* only map above 4GB */ #ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS #define arch_calc_vm_prot_bits(prot, key) ( \ diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c index 8cc653ffdccd..06378b5682c1 100644 --- a/arch/x86/kernel/sys_x86_64.c +++ b/arch/x86/kernel/sys_x86_64.c @@ -193,7 +193,11 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, info.flags = VM_UNMAPPED_AREA_TOPDOWN; info.length = len; - info.low_limit = PAGE_SIZE; + if (!in_32bit_syscall() && (flags & MAP_ABOVE4G)) + info.low_limit = 0x100000000; + else + info.low_limit = PAGE_SIZE; + info.high_limit = get_mmap_base(0); /* diff --git a/include/linux/mman.h b/include/linux/mman.h index cee1e4b566d8..40d94411d492 100644 --- a/include/linux/mman.h +++ b/include/linux/mman.h @@ -15,6 +15,9 @@ #ifndef MAP_32BIT #define MAP_32BIT 0 #endif +#ifndef MAP_ABOVE4G +#define MAP_ABOVE4G 0 +#endif #ifndef MAP_HUGE_2MB #define MAP_HUGE_2MB 0 #endif @@ -50,6 +53,7 @@ | MAP_STACK \ | MAP_HUGETLB \ | MAP_32BIT \ + | MAP_ABOVE4G \ | MAP_HUGE_2MB \ | MAP_HUGE_1GB) From patchwork Mon Feb 27 22:29:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 567D6C64ED8 for ; Mon, 27 Feb 2023 22:32:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 81FBC6B009D; Mon, 27 Feb 2023 17:31:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A9CF6B009E; Mon, 27 Feb 2023 17:31:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5AB636B009F; Mon, 27 Feb 2023 17:31:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 32E356B009E for ; Mon, 27 Feb 2023 17:31:54 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 10493120B0A for ; Mon, 27 Feb 2023 22:31:54 +0000 (UTC) X-FDA: 80514520548.10.8FBB805 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf07.hostedemail.com (Postfix) with ESMTP id 1E85C40015 for ; Mon, 27 Feb 2023 22:31:51 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=P7piIXvV; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537112; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=KYhEIBMklX69eNehPMqR0LfIIfvJ1fTdVv+Bbf8rMO4=; b=p4sI6/7f9UhPyynIwOZ2xZMtzyFtIhNAve5d0dcNLT4notnk6RsT0vxEylTAw/SpN+qljU rZX4qOHsY4kNwnktS/HxAJ00xTh+FtXQtA5mK3HIMEPVMnFA8olvUBf+KtZ1421r+aVz8z lu/5pX0h3eAQlQ7ra//ktB9FQvXcd0M= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=P7piIXvV; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537112; a=rsa-sha256; cv=none; b=g3gJ/9u5Emz2Q7QMpmdijSjSYkIVH5r2jyDvFhQPeNPF+m5c4S5/8oHFDgC5s93GQub4rh LZw32f1o3xH8PKvGyuAV2KSJ/olcpmvD/ngZUhM3DrsKytlSlUv/6dtuRY3SPCiEbIieN9 xueh7BQla1LaqxbFGQALEdgJW/y5IfA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537112; x=1709073112; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=nWbdJuzpIGeSMrabGm/MRjJVIT+DDdvBFWJsiZ1uvU4=; b=P7piIXvVfFCCQ4xfo0nDjM56PaaACtkeHXUGecnI1xFEGLEfZe6jZRY8 ODLxOrGHtarddDMsvPNW1AiqqCeU0AtZs2cK+t/8oepGBKiLN65uhNl+w stVFB4rQi5I3SsCks413DeX/WWCNSNteW3nTxH/qfs5yo46IZtNPRYS1I DeRWWRnDxTZbXwS+Eq+rv3G7+g8g+AULFqryvYMdxcwR4oIkSWtCWVynK DfMoOqLJACTESA3M7EY4T3spRYa+ZlJUdrCDtREWrPeYpy6+AdZj8VgFw ipVSl7jo8WDYJMlkXt9Kv7lCJ31mtpxJK+PgO9lAtBfbtUr8V7MA1leas A==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657628" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657628" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:27 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024686" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024686" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:26 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v7 26/41] mm: Warn on shadow stack memory in wrong vma Date: Mon, 27 Feb 2023 14:29:42 -0800 Message-Id: <20230227222957.24501-27-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 1E85C40015 X-Stat-Signature: rk9dsbzunpagmp45k1niy1opemiyf7r5 X-HE-Tag: 1677537111-7084 X-HE-Meta: U2FsdGVkX19dYaaaAAhhUClFumRdKhV9zoPUL81kCMC0s5rTvSDQcqb9FzRosJXMna0oPxOuRIk6lieD5MtM5y18vBx0dpq06TAo2l6Lv9FxvC3BffntNWenHU9kifXUafNCHURxfjaemcrHpmWJj729qF6ZQgZvxqcJypteGIxFRWY/h9LOCgRmz4QdVydiA1cHp2XERAEUdlO76I6e6Kbt7fkm2pGlK4JIInl/VOuhhgFbuAbDb5f52bK7oE6R5EKaQVTfLMsmObUEjVNfnoqQOpasTaMJYnOCT0Yyj+2CTj3Zh/dA81B40Fgqts+VRV+aj1kjVm4TkyvZh7+kEygmJJvtXL3FwWbXOrJ/JBqmnKu4/HsaUojZOGA6MwKuUssC18DdK1WrFBDw0zJAJMWopNiQ5QPEkQmKBhzQmbzd3XHhufB4phX8grH+0rTk2HOXiceVWStD6MbSEJ37+ZS6+hImlNh/gOaFR3bEzufT9zCY3rQ7SkpxXwm34iZrhJPYtMgshaumpUvw7A+Og/bWs5yUGHvpqXj66x1z2E0LAjAyFXVp49VWx01emWAB0bgUukbUhzY/GIHF8EM5fOai8Vk9a8dmufc51nfhxvr/YsuUaD/g8IQ/6FtE9aZOCGkfvzEbx2zkxyqAKheqEbYApGpbU798qmK6qiGitzv9jFoxymVqeIEEq6H/VpG1EtwLMRIGHnl64dME6jRk9QeVxRWUGqb0gDwI5eucyWFPkZyUqnxNoJgLPfcHlfmhNEm0p2GfL/ojryDrDrAqX3TpYr/BozYQYRwIukufe7tC8deu4B/eEDy4SrU/D7szlW06J6433dIuPNXRXkrLD2uBSe+6+g6ayOhIWUjDs8ynrcClJwBSXTdkQqVqMUBuK3Ru8S+3J2igCdigN/wpUCwX/RHjclfK/jPHow33nNN2GEtmvCL6fjkDHlE4B6JPBxyc9UXVQ7Cxq+cy8rc xT9XvsRP ZdQO2h5D3UhnrcdEeWZLhlVbp5Q25i3TWkdIUh0nwBn0Id6MQP5QF/mJtQe5gbCeRI6R8ZOO4WARrVBiTIFOklNlobyeLYgFK4QldpBr9tn+4c+PWM5isfdTbetM3QxZEuhe7BLsNCh6F0S0ISjrSwGwkVRLsfCZidJpQsMCGP6emnvvhhVQSQDE7/llMoHF9B7K6zcV+wEvEco7ezdyr9s0m7ZlhSnDs80a7en6/nfSfaBFK2qyzTb5Nk37Qtbh2KLVUvO5XszGggobtD2pgELBGhmC9TVlxstZ47RmEi6pINYeqEh9yAoL2XPBkWx7KkPTwmM2+rfg8b56vtb8jOi0i8X2Ck3vW7coj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. One sharp edge is that PTEs that are both Write=0 and Dirty=1 are treated as shadow by the CPU, but this combination used to be created by the kernel on x86. Previous patches have changed the kernel to now avoid creating these PTEs unless they are for shadow stack memory. In case any missed corners of the kernel are still creating PTEs like this for non-shadow stack memory, and to catch any re-introductions of the logic, warn if any shadow stack PTEs (Write=0, Dirty=1) are found in non-shadow stack VMAs when they are being zapped. This won't catch transient cases but should have decent coverage. It will be compiled out when shadow stack is not configured. In order to check if a pte is shadow stack in core mm code, add two arch breakouts arch_check_zapped_pte/pmd(). This will allow shadow stack specific code to be kept in arch/x86. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Rick Edgecombe --- v6: - Add arch breakout to remove shstk from core MM code. v5: - Fix typo in commit log v3: - New patch --- arch/x86/include/asm/pgtable.h | 6 ++++++ arch/x86/mm/pgtable.c | 12 ++++++++++++ include/linux/pgtable.h | 14 ++++++++++++++ mm/huge_memory.c | 1 + mm/memory.c | 1 + 5 files changed, 34 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 2e3d8cca1195..e5b3dce0d9fe 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1684,6 +1684,12 @@ static inline bool arch_has_hw_pte_young(void) return true; } +#define arch_check_zapped_pte arch_check_zapped_pte +void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte); + +#define arch_check_zapped_pmd arch_check_zapped_pmd +void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd); + #ifdef CONFIG_XEN_PV #define arch_has_hw_nonleaf_pmd_young arch_has_hw_nonleaf_pmd_young static inline bool arch_has_hw_nonleaf_pmd_young(void) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 98856bcc8102..afab0bc7862b 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -906,3 +906,15 @@ pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) return pmd; } + +void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte) +{ + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && + pte_shstk(pte)); +} + +void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd) +{ + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && + pmd_shstk(pmd)); +} diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index c63cd44777ec..4a8970b9fb11 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -291,6 +291,20 @@ static inline bool arch_has_hw_pte_young(void) } #endif +#ifndef arch_check_zapped_pte +static inline void arch_check_zapped_pte(struct vm_area_struct *vma, + pte_t pte) +{ +} +#endif + +#ifndef arch_check_zapped_pmd +static inline void arch_check_zapped_pmd(struct vm_area_struct *vma, + pmd_t pmd) +{ +} +#endif + #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index aaf815838144..24797be05fcb 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1689,6 +1689,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, */ orig_pmd = pmdp_huge_get_and_clear_full(vma, addr, pmd, tlb->fullmm); + arch_check_zapped_pmd(vma, orig_pmd); tlb_remove_pmd_tlb_entry(tlb, pmd, addr); if (vma_is_special_huge(vma)) { if (arch_needs_pgtable_deposit()) diff --git a/mm/memory.c b/mm/memory.c index d0972d2d6f36..c953c2c4588c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1389,6 +1389,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + arch_check_zapped_pte(vma, ptent); tlb_remove_tlb_entry(tlb, pte, addr); zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); From patchwork Mon Feb 27 22:29:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154258 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14FD1C64ED6 for ; Mon, 27 Feb 2023 22:32:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D014F6B009E; Mon, 27 Feb 2023 17:31:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CB12B6B00A0; Mon, 27 Feb 2023 17:31:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A74636B009E; Mon, 27 Feb 2023 17:31:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8B3096B009F for ; Mon, 27 Feb 2023 17:31:54 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5F722806D6 for ; Mon, 27 Feb 2023 22:31:54 +0000 (UTC) X-FDA: 80514520548.04.3B81FF5 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf28.hostedemail.com (Postfix) with ESMTP id 862FFC0011 for ; Mon, 27 Feb 2023 22:31:52 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=kwiVjhyG; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537112; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=JtebQR+v79yhlycXcnfuE5jL9g9etdSAEsiP4bdc7gU=; b=sduuxaCbgGWKS5Hm24tkvgCXLmESxVpSQ8POzm16IQqBvNpxxppi4j5bbRXlgO4W9rruuU zgwqNzqlrK7Ba58EwLbiKB5kBqnUQ3ru+Xma8JxWtLqUPGvDHFm6VmyXZsHpv9lCMcbYIc +7avm83l8J7yQZiGk7BBUSqAqDQD4Is= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=kwiVjhyG; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537112; a=rsa-sha256; cv=none; b=P/jXEyWDD1wJyn+Meo8aDAOgt+epNloPkw8OwDNUm8fQ/U1hDx6Oa7lCr60eYSiP0FXNEk k+XEJy63qAGzhEZcUtpPKTDMY2y2m9OVN0hbjCg2hxaHJmhZLXSwbJB47C3NHqSLXrcTs2 4svWg9/jfzhJx+4quxbMFCGQa1+OZfA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537112; x=1709073112; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=06DYB189OojzsE4lYBG6K2Lo6kRqspO9hy4UYyXmLjM=; b=kwiVjhyG0Xyg10PRyDMJj5yB6m4zR2UgIUh+cVNEHSr8aPENe96l/seP A3LfmKtYCvgGcZY2WsuATNeToJYjwaR4UYk+KjvwgmD6PtI2mg/7v5SyM k3lkPia1aA6WaaSce7GnoUs9pLml1iuO5m8ngJmkv/SXNb4j0OIPO6d7H AxjYaGifPXa+zc5jfp7tDP8mNLoK+AAvE95eAmTwzJgb/JRjtebaVA49u pCzFhvN5YQZ4JQ9faApK+1U1/5L+MSHejRqf+8GqUdBnUtHQrS28BQyOn 7sRRYW9OwxIB4N1KgRIQX6lNazgStOWpTDtJvMr6WlAJeL+LG9uxmXA/f A==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657650" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657650" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:27 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024693" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024693" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:27 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v7 27/41] x86/mm: Warn if create Write=0,Dirty=1 with raw prot Date: Mon, 27 Feb 2023 14:29:43 -0800 Message-Id: <20230227222957.24501-28-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 862FFC0011 X-Stat-Signature: s3cgbxmfmcku9m3s7xkyednb6ohf1k3w X-Rspam-User: X-HE-Tag: 1677537112-514311 X-HE-Meta: U2FsdGVkX1+/KBMRaiBha/R5az5SBdJlSEEzcPA87W6OmKQ4STfJr46QFWrBEPqkDK9o+tAbPw1uhfEuMmADAtzOzIbmiZvNxHGMr142btINLgPDKKxenwInq50QkmInN6Jxe9wtaPgY9oTsKnq/DifGdZPf7Y2HK73keqXhmtsAvM4gki2QsOSXpmzJ2u4JdnhrSKwBtQmgbWkXgInOS1/l5FF6calxmh3swRH4MdQFa9NnMBvnSSqVRPF7Mzzq32EWlxMSsqz5/bhoStF/aMbjvtTJiGm8VvkzPqjNn98mHIUrGgk9DXz0cXjTSGSQIrXkr4GRsCOTz1C4XF7pUx3bh4N917cI0HjT5B/bUsFzIY9dZmPyhgCej3CukKw3jdgUhB8LVwGuQndyHwKP10kVfyyArrofBia6v5udxiX/0WBnwDFE99es1+01Pfbe3N85A0Lf+1J277IjvIg62Eu00VDz1xQWvVynVYydXK3WqJkaXErpr2KOCZYwyVm43lLy1nMEIMS6uRF2Qb7nffBVWzmJLPvEhcqv3pItyVYvl6jqfLs8Vat/4PyfJiEnhEKFwHVfI+2/qU/OximEuLt4ed6YmuZxMmXRvhf2QJkWYAJpo3sLwYqsO2KQTYd3A57LPmLiFlYWSLuIqjEFrP6eBSpVxozuJRqpD4IS1ONoq90limSXVeDPz2AYZxgSCh8v/8t9Sfsk3MnshmsGW6rL4x1i58ZbBitY0r4iYqlzIsJemArXcB9vV/snuIqc82YxY/NfTUEMahJnTw0kacu4ioiH+alxjh30pS+b12TrTsjsJfhQAFwFuwdnKc0EfRTa4O1gsUzV/uzd5z9tRwZHuq0TUoc+riEfkn+U9a3G+WArYgLdRotHx9lfwuM/R8P8mJg7t8Sc52hsUAwxQPHGc7g5GUmGp0OVQ7ZiPWdyNDLlatqybmr+6FpuqqWpoSDhuVKKX6nynliXX0a R9y5ZcF3 Q0YQYmszAD4yP9EmKEGuWOa+i2Ggk99tP8khcaeo5T7NOAKZv753VdmYspQoBo2skXlInyZxEf9vC+yakEiN9iA9cCp7FhZowWIWZJKjXKZdVZgJGW8BztPoqHNGJPLn00pCQde+DfmYJhcBCoVCbRur9nAg1yUlbjgpxq1gjPmKkK5TyYfxsCZzpXhow20UYl52FKKozM7/CEW6ZhSbobxky2YURcKm4zLfZZ+qTF9j+UxhEvMcYApBksB6hF3ucuJlYVXOXsiDvoZsoduz0T63J1zeQQ8skiX6a7W2Q+rOSoz14ZNK/Lq4FV8b1Cw623j7NlQbYIJCW4L6CFNSIMCtZBAqUvHLNyFtL X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When user shadow stack is use, Write=0,Dirty=1 is treated by the CPU as shadow stack memory. So for shadow stack memory this bit combination is valid, but when Dirty=1,Write=1 (conventionally writable) memory is being write protected, the kernel has been taught to transition the Dirty=1 bit to SavedDirty=1, to avoid inadvertently creating shadow stack memory. It does this inside pte_wrprotect() because it knows the PTE is not intended to be a writable shadow stack entry, it is supposed to be write protected. However, when a PTE is created by a raw prot using mk_pte(), mk_pte() can't know whether to adjust Dirty=1 to SavedDirty=1. It can't distinguish between the caller intending to create a shadow stack PTE or needing the SavedDirty shift. The kernel has been updated to not do this, and so Write=0,Dirty=1 memory should only be created by the pte_mkfoo() helpers. Add a warning to make sure no new mk_pte() start doing this. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook --- v6: - New patch (Note, this has already been a useful warning, it caught the newly added set_memory_rox() doing this) --- arch/x86/include/asm/pgtable.h | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index e5b3dce0d9fe..7142f99d3fbb 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1032,7 +1032,15 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd) * (Currently stuck as a macro because of indirect forward reference * to linux/mm.h:page_to_nid()) */ -#define mk_pte(page, pgprot) pfn_pte(page_to_pfn(page), (pgprot)) +#define mk_pte(page, pgprot) \ +({ \ + pgprot_t __pgprot = pgprot; \ + \ + WARN_ON_ONCE(cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && \ + (pgprot_val(__pgprot) & (_PAGE_DIRTY | _PAGE_RW)) == \ + _PAGE_DIRTY); \ + pfn_pte(page_to_pfn(page), __pgprot); \ +}) static inline int pmd_bad(pmd_t pmd) { From patchwork Mon Feb 27 22:29:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154259 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB1E9C64ED9 for ; Mon, 27 Feb 2023 22:32:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D0306B009F; Mon, 27 Feb 2023 17:31:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 75A5F6B00A0; Mon, 27 Feb 2023 17:31:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 55F206B00A1; Mon, 27 Feb 2023 17:31:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 432FD6B009F for ; Mon, 27 Feb 2023 17:31:56 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 1BC5C16057A for ; Mon, 27 Feb 2023 22:31:56 +0000 (UTC) X-FDA: 80514520632.21.110A048 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf24.hostedemail.com (Postfix) with ESMTP id BD01C180017 for ; Mon, 27 Feb 2023 22:31:53 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=eYbM6e4P; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537114; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=1i/FzPbqzxTXG8dJ1u/9sE/uM7j0WUeqnnBeybnxIC8=; b=dz2yorgqmMdNKALXRjCoyqHDQfK5sykmJGailR2kuJ7p6DHFClbcrd3ScfgGPbPE04I5MM JlC54L3kWCQSPR9EvF2K+2KilvPIJ7FwncZiXmZRwqE2ape+AFYf6CM98yFi86L0uj9O+R eYMeHYNn9xuVAgt/bnM3EQDNS1zlZ4I= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=eYbM6e4P; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537114; a=rsa-sha256; cv=none; b=2Ln6Ge9zY9NwJtgD0z+7yT9IYUMgTM2NPZcz3dppSn8sesztjiV22qLEaWqjli8QO8zxJw 48iipXCQuMgShYK0q/dH/4L2wKgmdGGPraolWE5tNeJVuj9C5A3kQs9qGoyCDVQ4ws2xVD w4klK8iO66rrdAV6ia9SuprazfYgSE0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537114; x=1709073114; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=u1FQlCSURy2KLxFeOQDLZBSKCZJ7Cv/holb6byWhLUc=; b=eYbM6e4PdRl+e+8bTbp0YbDs4mVVKSD/jUfcgXI5uah3KJkopdTA/MyV 2tfSwupBJukxe7Nh5ZQWIuWcYtXS6e7wZv0Q7W02XqeOUQb2/Urgt4/Ej k24pS92lGDjimL/9r/XlLOEy5qt14prmcKoBv1TFlVRf1JT5wKl0l0z0k MpOYRV/XFzF0Zl1KWff2Fl2efnVnNZXdPSnGtyE8hSpgtHJdXLevP3kLm 0FvoIdwzGw6RJCtb7OjT+4VAV1ppRhrHKpceatWClQjWMp18Qr90W5naS bYDOiUUX1HFXoXj0l8+JQ/q+jTpbh/UoOp74dXUyL1OPSF1qhTMkgzDh+ w==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657674" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657674" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:28 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024703" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024703" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:27 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v7 28/41] x86: Introduce userspace API for shadow stack Date: Mon, 27 Feb 2023 14:29:44 -0800 Message-Id: <20230227222957.24501-29-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Queue-Id: BD01C180017 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: y79kux4yd7hhqdijmtmdns64g83k34k8 X-HE-Tag: 1677537113-215512 X-HE-Meta: U2FsdGVkX192bv2G0LxdCspJsYAlZV7UOVH1EDWmAn2WuzNx4OJnXO5t5+nfV/lhnApXRUT/zAasdoPKt1kpCdX7VRQoHch7zKJ9K5tMXbANorAQjA4TqxCUM4dOBK6aB++44XAjz6vguG4qaaTt2EuTQeYhSYwHAFVauOeYbbA2RyfJlzyMbCz8TGvuTUo00HMcgDN5XRSk7/f6fBSVcPSQsf1fi4J88hsRs1TS5TyFuPZPh5P9RsWDYRcaKdB2dPWg064Pua0DCsFQISzrBgYOFqiD+DQsOkvfoyQaTZpYugGEvtz9BvX4TRo52U3u0kRZzik6ciCS3KnpO5LR27AFER9jZCcLPI4uAufko6L/gnQ2Gb5ed0mDQIPNF2tGC4iTKreZD3vvul7ByLouSGeA+2IAEkzau/9G6fNY6O+++5ciJPdnJb2Ira0x4FB6zWnjumLj9+QwNeo6bSaZCTp82WHSUoAwdqMALVt3DtHNGc7QheDUwMbRyiqYYNQHel57cUy0b9J5OvDVQT8wM9h8fJ1U6pQrfh3dFy9Xf1jvcfb/eAQDiTQBN1vFwVBMSV/luYkZDZhm9/AYUs5o7GTcpdLLoRi0JJKeToyfSEpDpxeB/36NXkM3zFSB4tHpaMzkx+wGSfsX0Kd87w6tAgAsILkzvnmaCqCjUdib1Zc3YKbVP58DFP1k7jibBfhC2WrcbVQKSNzer9/fp7GmwGabBKwZam/u6GpGDsXdfTPhnih7Z8CF582GV3tXy9efc1sMtKYY5Ugol9798h59RHjhN4Uz+e9A8HPVVUtmDJx9rLG7SacaUx/9GLhzPM9WVtgVzxWd357OnMBeC++EZYwnAOlRH9EcAZdGLU9ON9pTgqDTOSH/xoNw4l6LM5zO4jwkmYn7MdVa8dyZjcIHShZQQ60dGPeDdDHsUWtjwbrxHr9cGl5AJ306s7NHkSLG3wnreFwfjO5Va6c+Q8c hZSz0IJK hKdcHhJXAyPZRpDOWfCGtt3cL6GA2e/GT1cos1ytvX6/BRCniB6kwh6W6fjApVRzynfUD13Gf+cnrgkCzq/0TXQFJoFjklAP/5pzi71dp2Rs/PvcXojon7h8iRvdgBDqR62gHOY8OSF+N1j7wHzkJvFjfKaJYVEnnJerFTcWshArXfrzCNwUTFmFQX6iaUh3/DI7AqXAQFt7A67uioRd9fioCoaYbxkoBmKVzqEaDb2AM7g0gVoi4r8+gtNvdAFTdkpFD9Ns0RYwT7TfrlUHhHdW0BeywCTW3YkcRq6QI4E0EZYc2bhnFLsYTvBvAXCM09d3HisEE1RJ2fG/AxJiC2mmEk2eVHBzeH6g1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Kirill A. Shutemov" Add three new arch_prctl() handles: - ARCH_SHSTK_ENABLE/DISABLE enables or disables the specified feature. Returns 0 on success or an error. - ARCH_SHSTK_LOCK prevents future disabling or enabling of the specified feature. Returns 0 on success or an error The features are handled per-thread and inherited over fork(2)/clone(2), but reset on exec(). This is preparation patch. It does not implement any features. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Kirill A. Shutemov [tweaked with feedback from tglx] Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe --- v4: - Remove references to CET and replace with shadow stack (Peterz) v3: - Move shstk.c Makefile changes earlier (Kees) - Add #ifdef around features_locked and features (Kees) - Encapsulate features reset earlier in reset_thread_features() so features and features_locked are not referenced in code that would be compiled !CONFIG_X86_USER_SHADOW_STACK. (Kees) - Fix typo in commit log (Kees) - Switch arch_prctl() numbers to avoid conflict with LAM v2: - Only allow one enable/disable per call (tglx) - Return error code like a normal arch_prctl() (Alexander Potapenko) - Make CET only (tglx) --- arch/x86/include/asm/processor.h | 6 +++++ arch/x86/include/asm/shstk.h | 21 +++++++++++++++ arch/x86/include/uapi/asm/prctl.h | 6 +++++ arch/x86/kernel/Makefile | 2 ++ arch/x86/kernel/process_64.c | 7 ++++- arch/x86/kernel/shstk.c | 44 +++++++++++++++++++++++++++++++ 6 files changed, 85 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/shstk.h create mode 100644 arch/x86/kernel/shstk.c diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index 8d73004e4cac..bd16e012b3e9 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -28,6 +28,7 @@ struct vm86; #include #include #include +#include #include #include @@ -475,6 +476,11 @@ struct thread_struct { */ u32 pkru; +#ifdef CONFIG_X86_USER_SHADOW_STACK + unsigned long features; + unsigned long features_locked; +#endif + /* Floating point and extended processor state */ struct fpu fpu; /* diff --git a/arch/x86/include/asm/shstk.h b/arch/x86/include/asm/shstk.h new file mode 100644 index 000000000000..ec753809f074 --- /dev/null +++ b/arch/x86/include/asm/shstk.h @@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_SHSTK_H +#define _ASM_X86_SHSTK_H + +#ifndef __ASSEMBLY__ +#include + +struct task_struct; + +#ifdef CONFIG_X86_USER_SHADOW_STACK +long shstk_prctl(struct task_struct *task, int option, unsigned long features); +void reset_thread_features(void); +#else +static inline long shstk_prctl(struct task_struct *task, int option, + unsigned long arg2) { return -EINVAL; } +static inline void reset_thread_features(void) {} +#endif /* CONFIG_X86_USER_SHADOW_STACK */ + +#endif /* __ASSEMBLY__ */ + +#endif /* _ASM_X86_SHSTK_H */ diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 500b96e71f18..b2b3b7200b2d 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -20,4 +20,10 @@ #define ARCH_MAP_VDSO_32 0x2002 #define ARCH_MAP_VDSO_64 0x2003 +/* Don't use 0x3001-0x3004 because of old glibcs */ + +#define ARCH_SHSTK_ENABLE 0x5001 +#define ARCH_SHSTK_DISABLE 0x5002 +#define ARCH_SHSTK_LOCK 0x5003 + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 92446f1dedd7..b366641703e3 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -146,6 +146,8 @@ obj-$(CONFIG_CALL_THUNKS) += callthunks.o obj-$(CONFIG_X86_CET) += cet.o +obj-$(CONFIG_X86_USER_SHADOW_STACK) += shstk.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 4e34b3b68ebd..71094c8a305f 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -514,6 +514,8 @@ start_thread_common(struct pt_regs *regs, unsigned long new_ip, load_gs_index(__USER_DS); } + reset_thread_features(); + loadsegment(fs, 0); loadsegment(es, _ds); loadsegment(ds, _ds); @@ -830,7 +832,10 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) case ARCH_MAP_VDSO_64: return prctl_map_vdso(&vdso_image_64, arg2); #endif - + case ARCH_SHSTK_ENABLE: + case ARCH_SHSTK_DISABLE: + case ARCH_SHSTK_LOCK: + return shstk_prctl(task, option, arg2); default: ret = -EINVAL; break; diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c new file mode 100644 index 000000000000..41ed6552e0a5 --- /dev/null +++ b/arch/x86/kernel/shstk.c @@ -0,0 +1,44 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * shstk.c - Intel shadow stack support + * + * Copyright (c) 2021, Intel Corporation. + * Yu-cheng Yu + */ + +#include +#include +#include + +void reset_thread_features(void) +{ + current->thread.features = 0; + current->thread.features_locked = 0; +} + +long shstk_prctl(struct task_struct *task, int option, unsigned long features) +{ + if (option == ARCH_SHSTK_LOCK) { + task->thread.features_locked |= features; + return 0; + } + + /* Don't allow via ptrace */ + if (task != current) + return -EINVAL; + + /* Do not allow to change locked features */ + if (features & task->thread.features_locked) + return -EPERM; + + /* Only support enabling/disabling one feature at a time. */ + if (hweight_long(features) > 1) + return -EINVAL; + + if (option == ARCH_SHSTK_DISABLE) { + return -EINVAL; + } + + /* Handle ARCH_SHSTK_ENABLE */ + return -EINVAL; +} From patchwork Mon Feb 27 22:29:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154260 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97DA1C64ED6 for ; Mon, 27 Feb 2023 22:32:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D53036B00A0; Mon, 27 Feb 2023 17:31:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C3B826B00A1; Mon, 27 Feb 2023 17:31:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F68F6B00A3; Mon, 27 Feb 2023 17:31:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8FC776B00A1 for ; Mon, 27 Feb 2023 17:31:56 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 66C69A012F for ; Mon, 27 Feb 2023 22:31:56 +0000 (UTC) X-FDA: 80514520632.16.D9BB519 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf07.hostedemail.com (Postfix) with ESMTP id 62ADF40015 for ; Mon, 27 Feb 2023 22:31:54 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="UbpjA/et"; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537114; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=Fd4CLx3Wb5bNuvpnqgjIwEqsxwps1hj9mXZuxyqW6+Y=; b=oBvuqKNLvKqaLZS5mmS8Ae40lfySPrtCWFOUHcJI3zEhes422T6NCG0sBs0rhyFgB0dqJL C1radJMDSR8uT3awDwMUZNcrva5VTXaJuo/ifdTVDsfMTOFnCAzs3670xNrfVwCFiEsDV8 sGrL/ReiFeKuGIR9Aq1+BUy4vxH3nk0= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="UbpjA/et"; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537114; a=rsa-sha256; cv=none; b=YZjdKkp8zH4zeTq5Z/FIkIQvUtMlrHDpe+s9RhMqcV6ESQP2WXJiDGts4oGYx1YhCIbUGS QFUaaT3hekLE6tR5L2yShpv7QbPvtiJYVZkOyMeq/vX+LdOi2G4vZaBCTUYS9y62IQVb/5 LsW587FImCFyJ5jDTXRYW4eXiCoMScU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537114; x=1709073114; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=iMlXlUo1ewZh0k0SEbwJZr7uR8+MhW0vqvkBqCbDW78=; b=UbpjA/etXuHTsMhSPwuYbSTX1t/CjQIS0II0kQyhDkOxb1nRewE1uARq Gs3eFoxJZZqnFKnPYhFtiWEfxLIThhPtii/gaw5PcQSzr+EuXO0q16aw9 r3F9JLx2nbzeHOmkYmxAhtrgoCkVHHAyWvr0fK9epg0Sz+JvXAatVailU 40p9nQ0RfLcsr+Ijwfi7Diiyw5D4x9Hm4ApgDWKVd4njxKiE4opNQKwQO MuLvaBkjNAE3KIUZ4BHA9c0EzaHcd4QUvpPH13oSuKydX0pemFp8BqelW S4JensxM8goHx9r2yZnW3VjLOOHyOTtbACVetS06COI6gdomsxylkSWqO g==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657703" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657703" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:29 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024714" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024714" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:28 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 29/41] x86/shstk: Add user-mode shadow stack support Date: Mon, 27 Feb 2023 14:29:45 -0800 Message-Id: <20230227222957.24501-30-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 62ADF40015 X-Stat-Signature: 1okgr4x66jtkjtooy7ntde1nqx6bjdxg X-HE-Tag: 1677537114-842864 X-HE-Meta: U2FsdGVkX185kytlsneS95ugxhUmdMyR57FLPJqYAXPlqbUTzVRn9l0ViSVvbSkbjeydBFzR7QQ3w+eimcIrEwdpYbidhE8I5lXUp5wx7TT+Fy9ZTHPRlkznLHaZGk2lhEnyutNKYwhCj6oL9mCSEdLQQb+1aYEOIDc0JecY+5ceBCzl7RcMQcJ6bD1Kj547SUyhv5MJpaMoTEGO/S1Zt8Bm4Wha4DtLrI6EMlRME4WBVl950GSCkTB4WDqIoGvMFp6mEtKNF8vmtAgOczIrp7NOcRBPWQP/U37tuYbVdXPrw7CiFpSvZ8sXc/X2ZXlbMtFgbO9i9WHcev6Exnh7AavLUxVyq84coX/yPdTSQriKHuJYuI9mSpaAzdLRBsS4yAe7Wv6joURhqv14+PzEPXJmj/ZPbaWZ/aRm8XSh+x8CiWakK+trjdZ8tUB94d6Ef9q/WZNz5+sopITTgUE3/6XnlrtEzeXeosFpa0i0UP1pLNrTskhKXn67F4dRiXj3EI5YUa/+p54rSjSq56i6YMDfvPnMymfiBZWHjJZLrDUb+tLL9tDJEH6GCPGdv/XPppxEG1OiJ8XBUQEDUeZ5S6PRAsuHcjJRIbgJ0G/j+65psdSPU+EAccaanB905eFNKmc2cPkof8XXKdG8xYLVIc9oGb071JIAHct09/UMuDLQKgoF5XgbGGYpIOXaptGO6+W4qTJxwkb2eFJrFeVh02ykxln8BxNx13AcmOIBNvyGev6Mdu7wB/OgVdqiFLnqSqoDvhWNfxwsiVTGCeJFGj9fRiny9pwPn3iCleMYFc5jIWJZMWhiZu4sKhOqPn53kcGH+wC9dr2b7BCZV7Ge5MjwtzdtBrAuCQGsM7gSoKyZUHxkAyzpGk0D+oOYlIntmeSlr0mzlqIb3I2r4YojJJpPDovKPhEQgv74j5ivO45temY3/1yVGd4B6S4vtUexkxFXNwPa1mDz7BLQVAc XNg8H27p C10sLKkSjZXYZJ2llCpRFU7DJA1qmzAvbQKHw3oiGXHpCPfpSLnON/YJnQ77bBYoVKrfJJWSKzr5JoSjB4LO8FQcZIFOFwB0bul913y99sd2ydlAsDP+1/AcennQJuueoDMd4FIN6t4bdiVKVan1bSeIxpGlf+0knMn/E/8gW5SSUHDwHbG1Xo766QXF3ApQI5eWy3hqKEsTZj5MmsVZB/fzT+U2ae8o/u9x6jGk4g5w6XqELzvb4K29uH72xEWR/3PmwujZScg9zXDPKW6HjD3WjawmWXAIMyRMYM+4q/in14daYQKH9loN8e+c8y+4lnUDEWhRAof/GBqfXt/dngUqw2pzpcF1zJOaT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu Introduce basic shadow stack enabling/disabling/allocation routines. A task's shadow stack is allocated from memory with VM_SHADOW_STACK flag and has a fixed size of min(RLIMIT_STACK, 4GB). Keep the task's shadow stack address and size in thread_struct. This will be copied when cloning new threads, but needs to be cleared during exec, so add a function to do this. 32 bit shadow stack is not expected to have many users and it will complicate the signal implementation. So do not support IA32 emulation or x32. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook --- v7: - Add explanation for not supporting 32 bit in commit log (Boris) v5: - Switch to EOPNOTSUPP - Use MAP_ABOVE4G - Move set_clr_bits_msrl() to patch where it is first used v4: - Just set MSR_IA32_U_CET when disabling shadow stack, since we don't have IBT yet. (Peterz) v3: - Use define for set_clr_bits_msrl() (Kees) - Make some functions static (Kees) - Change feature_foo() to features_foo() (Kees) - Centralize shadow stack size rlimit checks (Kees) - Disable x32 support v2: - Get rid of unnecessary shstk->base checks - Don't support IA32 emulation --- arch/x86/include/asm/processor.h | 2 + arch/x86/include/asm/shstk.h | 7 ++ arch/x86/include/uapi/asm/prctl.h | 3 + arch/x86/kernel/shstk.c | 145 ++++++++++++++++++++++++++++++ 4 files changed, 157 insertions(+) diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index bd16e012b3e9..ff98cd6d5af2 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -479,6 +479,8 @@ struct thread_struct { #ifdef CONFIG_X86_USER_SHADOW_STACK unsigned long features; unsigned long features_locked; + + struct thread_shstk shstk; #endif /* Floating point and extended processor state */ diff --git a/arch/x86/include/asm/shstk.h b/arch/x86/include/asm/shstk.h index ec753809f074..2b1f7c9b9995 100644 --- a/arch/x86/include/asm/shstk.h +++ b/arch/x86/include/asm/shstk.h @@ -8,12 +8,19 @@ struct task_struct; #ifdef CONFIG_X86_USER_SHADOW_STACK +struct thread_shstk { + u64 base; + u64 size; +}; + long shstk_prctl(struct task_struct *task, int option, unsigned long features); void reset_thread_features(void); +void shstk_free(struct task_struct *p); #else static inline long shstk_prctl(struct task_struct *task, int option, unsigned long arg2) { return -EINVAL; } static inline void reset_thread_features(void) {} +static inline void shstk_free(struct task_struct *p) {} #endif /* CONFIG_X86_USER_SHADOW_STACK */ #endif /* __ASSEMBLY__ */ diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index b2b3b7200b2d..7dfd9dc00509 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -26,4 +26,7 @@ #define ARCH_SHSTK_DISABLE 0x5002 #define ARCH_SHSTK_LOCK 0x5003 +/* ARCH_SHSTK_ features bits */ +#define ARCH_SHSTK_SHSTK (1ULL << 0) + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 41ed6552e0a5..3cb85224d856 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -8,14 +8,159 @@ #include #include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include #include +static bool features_enabled(unsigned long features) +{ + return current->thread.features & features; +} + +static void features_set(unsigned long features) +{ + current->thread.features |= features; +} + +static void features_clr(unsigned long features) +{ + current->thread.features &= ~features; +} + +static unsigned long alloc_shstk(unsigned long size) +{ + int flags = MAP_ANONYMOUS | MAP_PRIVATE | MAP_ABOVE4G; + struct mm_struct *mm = current->mm; + unsigned long addr, unused; + + mmap_write_lock(mm); + addr = do_mmap(NULL, addr, size, PROT_READ, flags, + VM_SHADOW_STACK | VM_WRITE, 0, &unused, NULL); + + mmap_write_unlock(mm); + + return addr; +} + +static unsigned long adjust_shstk_size(unsigned long size) +{ + if (size) + return PAGE_ALIGN(size); + + return PAGE_ALIGN(min_t(unsigned long long, rlimit(RLIMIT_STACK), SZ_4G)); +} + +static void unmap_shadow_stack(u64 base, u64 size) +{ + while (1) { + int r; + + r = vm_munmap(base, size); + + /* + * vm_munmap() returns -EINTR when mmap_lock is held by + * something else, and that lock should not be held for a + * long time. Retry it for the case. + */ + if (r == -EINTR) { + cond_resched(); + continue; + } + + /* + * For all other types of vm_munmap() failure, either the + * system is out of memory or there is bug. + */ + WARN_ON_ONCE(r); + break; + } +} + +static int shstk_setup(void) +{ + struct thread_shstk *shstk = ¤t->thread.shstk; + unsigned long addr, size; + + /* Already enabled */ + if (features_enabled(ARCH_SHSTK_SHSTK)) + return 0; + + /* Also not supported for 32 bit and x32 */ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) || in_32bit_syscall()) + return -EOPNOTSUPP; + + size = adjust_shstk_size(0); + addr = alloc_shstk(size); + if (IS_ERR_VALUE(addr)) + return PTR_ERR((void *)addr); + + fpregs_lock_and_load(); + wrmsrl(MSR_IA32_PL3_SSP, addr + size); + wrmsrl(MSR_IA32_U_CET, CET_SHSTK_EN); + fpregs_unlock(); + + shstk->base = addr; + shstk->size = size; + features_set(ARCH_SHSTK_SHSTK); + + return 0; +} + void reset_thread_features(void) { + memset(¤t->thread.shstk, 0, sizeof(struct thread_shstk)); current->thread.features = 0; current->thread.features_locked = 0; } +void shstk_free(struct task_struct *tsk) +{ + struct thread_shstk *shstk = &tsk->thread.shstk; + + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) || + !features_enabled(ARCH_SHSTK_SHSTK)) + return; + + if (!tsk->mm) + return; + + unmap_shadow_stack(shstk->base, shstk->size); +} + +static int shstk_disable(void) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return -EOPNOTSUPP; + + /* Already disabled? */ + if (!features_enabled(ARCH_SHSTK_SHSTK)) + return 0; + + fpregs_lock_and_load(); + /* Disable WRSS too when disabling shadow stack */ + wrmsrl(MSR_IA32_U_CET, 0); + wrmsrl(MSR_IA32_PL3_SSP, 0); + fpregs_unlock(); + + shstk_free(current); + features_clr(ARCH_SHSTK_SHSTK); + + return 0; +} + long shstk_prctl(struct task_struct *task, int option, unsigned long features) { if (option == ARCH_SHSTK_LOCK) { From patchwork Mon Feb 27 22:29:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154261 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F8AEC64ED8 for ; Mon, 27 Feb 2023 22:32:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4C5526B00A1; Mon, 27 Feb 2023 17:31:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4734B6B00A2; Mon, 27 Feb 2023 17:31:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 251116B00A3; Mon, 27 Feb 2023 17:31:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id EBBAC6B00A2 for ; Mon, 27 Feb 2023 17:31:56 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C3DF81A0905 for ; Mon, 27 Feb 2023 22:31:56 +0000 (UTC) X-FDA: 80514520632.15.34E70FE Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf28.hostedemail.com (Postfix) with ESMTP id B2A86C0011 for ; Mon, 27 Feb 2023 22:31:54 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=KPnTnQzZ; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537115; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=m52ALMa5MoA5LZgVka16gxe6V6oS249JLsTs5r6G+dI=; b=e2aTtWwRxOitEb0ECvh5GPLGnjYFN7jN6J8+dFwkYTiwvf8jZ+cgeDill7E9Y3rOzPXaiS Dr1DVr8JxDTa4qTXurYfs1paptRt6f13vvJMNeuKgz+lWRxYnz9BQ/chrq1g/zlZEkkhK1 pdGNrBJi5aeUrrAFT3HqEZuS65kqW+c= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=KPnTnQzZ; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537115; a=rsa-sha256; cv=none; b=ydWhjrF601/b0DckPFjIav8+cfcBkYcDiVK3EEeJSkLUZi9MakhEduETlt2ln2gVQx9UEJ ce1U3jtI0yZ+y2jxtP46e0IBMCn6s+7M/NTZhIzEl6b6rSwOJifijPCzhSZ/DV0D5+P/hj dKMzb89BCtHiTsciz2CscBjHmFooAxQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537114; x=1709073114; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=rnMkyBiyqCskyK36o+V760Wpmhf/ifdA1Q7ZDNUKK58=; b=KPnTnQzZPK8+Rkdr9TpXFxHan+z5ge/r1xG+x8zRy1X/Fs678l/oRNoT ZOEy2LcvRNXF+euoQX668IOvNvcabot7VgaIsTlvhs4t/5sZCTriE6xWO HsIrrFtVd0AEiKB3fkZUX3PdqVC/n1H+QMk0lXe2J5gRhJCToc2Pu8f7e IFpC1qHHj5BpdGSx3x2r4KMz12ekymFmqX5slTQ49eiA1BhwlQJGUf1fX 7QPwbIlAvNYH/eBkcAo4LIn1fB9lGMvy3OHGk4vIJJg/rJm+b24xDUJJd JGMgJYjF+zB8WIBJR3iM70oRG6l35t3k+01zAS+ZyVaFg7tjKyTmZVf4b w==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657730" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657730" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:29 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024728" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024728" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:29 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 30/41] x86/shstk: Handle thread shadow stack Date: Mon, 27 Feb 2023 14:29:46 -0800 Message-Id: <20230227222957.24501-31-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: B2A86C0011 X-Stat-Signature: sjqspq45kppn68rrx35jizrp1towxdiz X-Rspam-User: X-HE-Tag: 1677537114-243066 X-HE-Meta: U2FsdGVkX1/02rzGjdVgeic4wR4u2kAyCfYJCY1sHI5HPQWazeKJ3GhnNFhTt0GtQk2AaIUnH+uA//7VWjIIVtbDtD9WAnzmyM/4EUB/siie0K33L+lpNPlF05nXHUj4/gIqX9AZQt8187gIWLBsmqWWIac5A6a9JoR2WL5pW6jL2ZbXLvLtabwFFg50krG/ySkON2TFYdpIhPRwwCQB2RlUiIgBTxSRbJ4auyrNGVnYwOuBX6lmRI7fS2YWQdHVSbqcs/fUKzOpL0X+VYH3tCleNbLZI9jqmdShNK+JEqNvEnC3s2heJ1qZXkyCzGRV8W50QYXmb3lfSpPCN1CFgupNLdrnr1Pp66wbW+ZoEzTQk9MzlICxb/uAThD+KeVdM8B2p7nrFGdviXRM0SLySGDlNAK7PN8NjTccnXRXNFiS9oe6yqAx3xTzsAzNv9ScYzQ9+K/EHgIeIVKn1l1aDLHrysmxXmkuH5l58+rET9W4E7x4HgVs7UW4pK9V8+GO5C4ud9LPzrph5RmWMV7ttpWDYQn45PaHY2nqpVsHftxBsFD0gx9ZJofnnJeWbIsMICshCX/Rz2au2eKzBEiZyzxxVbDZOHbnbn7UZ5W/EtmHrdQZEtQlm7bSEmS18Fg9BH/pr8xWfobpoP2YI0bAv1D2s55ZgFYTGka8E747eb0sDMj178V3HXi8rnivzqfru4v/QU3D16VSd7mxuWlKFnSMH99vX4XVQBOZcFWL0wu7AZWp9IZ0b8cR7oU4L1EovXIEtIiJwXEyWppwbwcMw6knmLzRlAzEq4Kt/RPPWj1SRzIqJqb2U4EFM/+Mr8rA3wPGZXgBMucj1RlrCd0KYN/tLqyAI6rQid12Au/OhwgwI3tgwlK87vIHb2bBr3/WQAUQE1gOsNxMmwRExadHf9N+PfmMztX68N/OlIX+fvpRSvFkVJN7UOn01/0z5q6Z+ZMWtBigArByTgigUc1 u6APyxgr GPSJI7mm1zRRyx9JePJQrpBCLf0Qm5hB2FWtwmnjzd1lnTDZrqkfT3XzexbTSLkBkH9WghCJj/+gRK9uKnJcQCmqwj+m8ZrhyDq2/m2BpMKzJptrOCHTWK4fcU6SpLaGWKwl7icUhQ9eI1y13goRZX4skFAmCOoBjTCVo4Xf4LOCNqKpVJXW4dWmS71YuU7+D4O/EncjeLox2rELappsbugYvsN8cyUM/p9fbJnJc/z/oJXfQsQh8IAzf2Af9kReZaDDTiR5Twp+9ukBg+N8lHAHLFEx0vQdAwiOkm7ipnL/Ziv9Ih67mxeHt4w+lt5+LdUiRurrolcAN4HL7RRCBJIOJT5uTbrQQMhxJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu When a process is duplicated, but the child shares the address space with the parent, there is potential for the threads sharing a single stack to cause conflicts for each other. In the normal non-cet case this is handled in two ways. With regular CLONE_VM a new stack is provided by userspace such that the parent and child have different stacks. For vfork, the parent is suspended until the child exits. So as long as the child doesn't return from the vfork()/CLONE_VFORK calling function and sticks to a limited set of operations, the parent and child can share the same stack. For shadow stack, these scenarios present similar sharing problems. For the CLONE_VM case, the child and the parent must have separate shadow stacks. Instead of changing clone to take a shadow stack, have the kernel just allocate one and switch to it. Use stack_size passed from clone3() syscall for thread shadow stack size. A compat-mode thread shadow stack size is further reduced to 1/4. This allows more threads to run in a 32-bit address space. The clone() does not pass stack_size, which was added to clone3(). In that case, use RLIMIT_STACK size and cap to 4 GB. For shadow stack enabled vfork(), the parent and child can share the same shadow stack, like they can share a normal stack. Since the parent is suspended until the child terminates, the child will not interfere with the parent while executing as long as it doesn't return from the vfork() and overwrite up the shadow stack. The child can safely overwrite down the shadow stack, as the parent can just overwrite this later. So CET does not add any additional limitations for vfork(). Userspace implementing posix vfork() can actually prevent the child from returning from the vfork() calling function, using CET. Glibc does this by adjusting the shadow stack pointer in the child, so that the child receives a #CP if it tries to return from vfork() calling function. Free the shadow stack on thread exit by doing it in mm_release(). Skip this when exiting a vfork() child since the stack is shared in the parent. During this operation, the shadow stack pointer of the new thread needs to be updated to point to the newly allocated shadow stack. Since the ability to do this is confined to the FPU subsystem, change fpu_clone() to take the new shadow stack pointer, and update it internally inside the FPU subsystem. This part was suggested by Thomas Gleixner. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Suggested-by: Thomas Gleixner Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe --- v3: - Fix update_fpu_shstk() stub (Mike Rapoport) - Fix chunks around alloc_shstk() in wrong patch (Kees) - Fix stack_size/flags swap (Kees) - Use centralized stack size logic (Kees) v2: - Have fpu_clone() take new shadow stack pointer and update SSP in xsave buffer for new task. (tglx) v1: - Expand commit log. - Add more comments. - Switch to xsave helpers. Yu-cheng v30: - Update comments about clone()/clone3(). (Borislav Petkov) --- arch/x86/include/asm/fpu/sched.h | 3 ++- arch/x86/include/asm/mmu_context.h | 2 ++ arch/x86/include/asm/shstk.h | 7 +++++ arch/x86/kernel/fpu/core.c | 41 +++++++++++++++++++++++++++- arch/x86/kernel/process.c | 18 ++++++++++++- arch/x86/kernel/shstk.c | 43 ++++++++++++++++++++++++++++-- 6 files changed, 109 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/fpu/sched.h b/arch/x86/include/asm/fpu/sched.h index c2d6cd78ed0c..3c2903bbb456 100644 --- a/arch/x86/include/asm/fpu/sched.h +++ b/arch/x86/include/asm/fpu/sched.h @@ -11,7 +11,8 @@ extern void save_fpregs_to_fpstate(struct fpu *fpu); extern void fpu__drop(struct fpu *fpu); -extern int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal); +extern int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal, + unsigned long shstk_addr); extern void fpu_flush_thread(void); /* diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index e01aa74a6de7..9714f08d941b 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -147,6 +147,8 @@ do { \ #else #define deactivate_mm(tsk, mm) \ do { \ + if (!tsk->vfork_done) \ + shstk_free(tsk); \ load_gs_index(0); \ loadsegment(fs, 0); \ } while (0) diff --git a/arch/x86/include/asm/shstk.h b/arch/x86/include/asm/shstk.h index 2b1f7c9b9995..1399f4df098b 100644 --- a/arch/x86/include/asm/shstk.h +++ b/arch/x86/include/asm/shstk.h @@ -15,11 +15,18 @@ struct thread_shstk { long shstk_prctl(struct task_struct *task, int option, unsigned long features); void reset_thread_features(void); +int shstk_alloc_thread_stack(struct task_struct *p, unsigned long clone_flags, + unsigned long stack_size, + unsigned long *shstk_addr); void shstk_free(struct task_struct *p); #else static inline long shstk_prctl(struct task_struct *task, int option, unsigned long arg2) { return -EINVAL; } static inline void reset_thread_features(void) {} +static inline int shstk_alloc_thread_stack(struct task_struct *p, + unsigned long clone_flags, + unsigned long stack_size, + unsigned long *shstk_addr) { return 0; } static inline void shstk_free(struct task_struct *p) {} #endif /* CONFIG_X86_USER_SHADOW_STACK */ diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index f851558b673f..bc3de4aeb661 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -552,8 +552,41 @@ static inline void fpu_inherit_perms(struct fpu *dst_fpu) } } +#ifdef CONFIG_X86_USER_SHADOW_STACK +static int update_fpu_shstk(struct task_struct *dst, unsigned long ssp) +{ + struct cet_user_state *xstate; + + /* If ssp update is not needed. */ + if (!ssp) + return 0; + + xstate = get_xsave_addr(&dst->thread.fpu.fpstate->regs.xsave, + XFEATURE_CET_USER); + + /* + * If there is a non-zero ssp, then 'dst' must be configured with a shadow + * stack and the fpu state should be up to date since it was just copied + * from the parent in fpu_clone(). So there must be a valid non-init CET + * state location in the buffer. + */ + if (WARN_ON_ONCE(!xstate)) + return 1; + + xstate->user_ssp = (u64)ssp; + + return 0; +} +#else +static int update_fpu_shstk(struct task_struct *dst, unsigned long shstk_addr) +{ + return 0; +} +#endif + /* Clone current's FPU state on fork */ -int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal) +int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal, + unsigned long ssp) { struct fpu *src_fpu = ¤t->thread.fpu; struct fpu *dst_fpu = &dst->thread.fpu; @@ -613,6 +646,12 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal) if (use_xsave()) dst_fpu->fpstate->regs.xsave.header.xfeatures &= ~XFEATURE_MASK_PASID; + /* + * Update shadow stack pointer, in case it changed during clone. + */ + if (update_fpu_shstk(dst, ssp)) + return 1; + trace_x86_fpu_copy_src(src_fpu); trace_x86_fpu_copy_dst(dst_fpu); diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index b650cde3f64d..bf703f53fa49 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -48,6 +48,7 @@ #include #include #include +#include #include "process.h" @@ -119,6 +120,7 @@ void exit_thread(struct task_struct *tsk) free_vm86(t); + shstk_free(tsk); fpu__drop(fpu); } @@ -140,6 +142,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) struct inactive_task_frame *frame; struct fork_frame *fork_frame; struct pt_regs *childregs; + unsigned long shstk_addr = 0; int ret = 0; childregs = task_pt_regs(p); @@ -174,7 +177,13 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) frame->flags = X86_EFLAGS_FIXED; #endif - fpu_clone(p, clone_flags, args->fn); + /* Allocate a new shadow stack for pthread if needed */ + ret = shstk_alloc_thread_stack(p, clone_flags, args->stack_size, + &shstk_addr); + if (ret) + return ret; + + fpu_clone(p, clone_flags, args->fn, shstk_addr); /* Kernel thread ? */ if (unlikely(p->flags & PF_KTHREAD)) { @@ -220,6 +229,13 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) if (!ret && unlikely(test_tsk_thread_flag(current, TIF_IO_BITMAP))) io_bitmap_share(p); + /* + * If copy_thread() if failing, don't leak the shadow stack possibly + * allocated in shstk_alloc_thread_stack() above. + */ + if (ret) + shstk_free(p); + return ret; } diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 3cb85224d856..1d30295e0066 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -47,7 +47,7 @@ static unsigned long alloc_shstk(unsigned long size) unsigned long addr, unused; mmap_write_lock(mm); - addr = do_mmap(NULL, addr, size, PROT_READ, flags, + addr = do_mmap(NULL, 0, size, PROT_READ, flags, VM_SHADOW_STACK | VM_WRITE, 0, &unused, NULL); mmap_write_unlock(mm); @@ -126,6 +126,39 @@ void reset_thread_features(void) current->thread.features_locked = 0; } +int shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags, + unsigned long stack_size, unsigned long *shstk_addr) +{ + struct thread_shstk *shstk = &tsk->thread.shstk; + unsigned long addr, size; + + /* + * If shadow stack is not enabled on the new thread, skip any + * switch to a new shadow stack. + */ + if (!features_enabled(ARCH_SHSTK_SHSTK)) + return 0; + + /* + * For CLONE_VM, except vfork, the child needs a separate shadow + * stack. + */ + if ((clone_flags & (CLONE_VFORK | CLONE_VM)) != CLONE_VM) + return 0; + + size = adjust_shstk_size(stack_size); + addr = alloc_shstk(size); + if (IS_ERR_VALUE(addr)) + return PTR_ERR((void *)addr); + + shstk->base = addr; + shstk->size = size; + + *shstk_addr = addr + size; + + return 0; +} + void shstk_free(struct task_struct *tsk) { struct thread_shstk *shstk = &tsk->thread.shstk; @@ -134,7 +167,13 @@ void shstk_free(struct task_struct *tsk) !features_enabled(ARCH_SHSTK_SHSTK)) return; - if (!tsk->mm) + /* + * When fork() with CLONE_VM fails, the child (tsk) already has a + * shadow stack allocated, and exit_thread() calls this function to + * free it. In this case the parent (current) and the child share + * the same mm struct. + */ + if (!tsk->mm || tsk->mm != current->mm) return; unmap_shadow_stack(shstk->base, shstk->size); From patchwork Mon Feb 27 22:29:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154262 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32548C64ED6 for ; Mon, 27 Feb 2023 22:32:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7AB256B00A2; Mon, 27 Feb 2023 17:31:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 759236B00A3; Mon, 27 Feb 2023 17:31:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 585676B00A4; Mon, 27 Feb 2023 17:31:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4638F6B00A2 for ; Mon, 27 Feb 2023 17:31:58 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 284B780B6A for ; Mon, 27 Feb 2023 22:31:58 +0000 (UTC) X-FDA: 80514520716.04.99C99B3 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf24.hostedemail.com (Postfix) with ESMTP id 3CF0A180009 for ; Mon, 27 Feb 2023 22:31:56 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=TxBni539; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537116; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=lzo3Du4DYoE+b17QWEoI7riVHVFfp/EGQhzvABVyQ5w=; b=XPSUU0GTITuuPv/rhx9htBeWObhkptBdKFKCUKKU8/GTZ1hoccsub3ZaX5TvvQrMBX2Jbu 6AjSCMvo6VY/Z2x8/naQM5IvqZQGd5hGoPGn8nL+VP60Jyvl1ylmWJJR806mOYxAVaaRUr tz6s/bjgUDNdyPkC72O8nehMSDisuy8= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=TxBni539; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537116; a=rsa-sha256; cv=none; b=vOppUHgj0+Daah10gmwxYd64uIWh1dwmEayizinywExAUbJWxD2UWezbiHtoBHG8fjqQzr NzCShcYoiAP6tSXwtM1DAadAQC3rX+9iR6bgLD8yHEqQnI6DvaofO3TBzBI/i4U9mVN7bT TTkCv/YPr6BG57t4egsZ4VX+FO83H2g= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537116; x=1709073116; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=EFxkXhIou5r9xim+54dTmsDWNwff9pMdXLmKz8K9Fko=; b=TxBni539xjkuXvnQl43kXXKaR1fqXmmmUhUtYEbOTqigSkrLGIfetRR2 9hL1Vbc7K9Q2LEWNgdFa53wjQ2/IuzP5wCi8j28FXSxIeduRKUXhf2kW/ dQMqtNTpi5dUHmC7TEOtXl1XrkNvDKUkyAEJGL/EcuVpi9AKt4LBZxNNX zDQLua/2PivJVTmjU26eOQc2RDzVtttDzuRqcEzqDt97ZzNzuYx8SUmlv EZZ0c7hgB/IV8CXuOa9uDP3tk/lj8enRGPw0ON4rLrBJrOJTNgf/abNXp e6MEmfXtfnNDxieHAYAkhaJsunGiwJvBtQP+f44/n5zh4DPDGJCopTfnY Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657766" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657766" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:32 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024746" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024746" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:29 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 31/41] x86/shstk: Introduce routines modifying shstk Date: Mon, 27 Feb 2023 14:29:47 -0800 Message-Id: <20230227222957.24501-32-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Queue-Id: 3CF0A180009 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: xapoajsc8ruodfddbu93ya64xyfdrfqn X-HE-Tag: 1677537116-940168 X-HE-Meta: U2FsdGVkX18s+YqWUCPOElINU+aac81XbVaTuSdTc8NsN3TNuZc/G6n7FbSlssO5HOHLSD7UynMqUXXWsnQxOdRtP3D+1kSNvrM4O1T5pk5X4oT8bVNad2gDTTHir4Z1s7xneWovfQ6Tc1JlBul43XKx4XlpAdOTeMIsG3BPgZHo+r3PL3XPy8PM+TGZcJ++Gu0DJodpgUs1VAxEhC8UaPBm5WhMQUNGO/ktTpk+oPSD6rN1sqDzMaFfG8lC0dBZ+jQSyCXMxZ2asHDe+65mWN4tLW077Dc/cx4cXEZOfdcruO0RIh8q24n0JPPTCMb944I9fuY1rLo+fyBz1mPH1W6bK5SNotRPsrKcoztEirrwlTLZMIwd1MX/E6he+5mLC2jifAcBtQ/qy2TE8CHALt9baH68v3nKsYOXnH2X4BiqaQ5pnLXlVkuPbEICNKCf9rxZxwfSqNQE5ZJXNcBB0YZ/KIol6HDm6ZOn7wBNW9fYCMbubqB5WzD4SETk+3iQGV9kOgbHBQUKl2mcMRf2qmQ6us6CmfjNntVaZlne38R2wdHNR7SVJ/rBmk7u6TK3SKffmaJz2kWSognB4yAiWmVjK7KA+fkKCbien1QsF6TzpA6pAnQAhKDsB2Es4TEeUnKPIQQ0SRLW8Kb2B2XYhQJhuix2G42OoNJuMPvDkx1pM/fyTE9xrd3TsL16xQ9l4UQUxbzM+S/DSth5q/YlDw6WLbF4GTLMezdUToJe5BHELpJy5KEswgutdQNN+b7dEELv1xer6582RECvAAI/ArQg3mUk2Yf+mY/49XD8s4BZRPh9MkWMGPd0MchzYkg+q3jm1Nt1VtO6fUIlW+F/1ss8e+lt6wayNnAIw7nn77kIIwGa+3z/1ervbeZSy8ftp7AYLxs3ROGg/M7j9uV94yVJaC7pZrYhoy/gQQnd1TrQblWfV5Mwc9D4gyq14Qp88Gm73Kfj/spMd80+ZkR YeZIRnbL SoO3xN4rN8nQnc9sSLoisatIdS2ZV9XoATekCWXO8goEvw9VL63haJEo58cf8nNBgnEc6VLrWMre2jmPOWyX26Mhns9+MQORHYH0o7xV5wO0dZyVym+/s8eRvyEZXBJ5838jtZxB32c9gk2hmDoEsee+/mCQX+w7xMc41DS5/tUQO0FOqhPDZ3E8nn9R1yOTbALtqNBnqqPQ6JBVzfHZlTEfgd0mCTpMgMHj6zCIr+jmHQF/bzNk2MNQOOoRrnYe+80AJFgEy1K2q8PVJCGER53+4LbK8KI2UNC32vnY8/8wPfA5iVTDqwsXO4BhQt17vwuzJokY3RNyKeGCWOtumP4fnf1s0ACfwH+96 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu Shadow stacks are normally written to via CALL/RET or specific CET instructions like RSTORSSP/SAVEPREVSSP. However during some Linux operations the kernel will need to write to directly using the ring-0 only WRUSS instruction. A shadow stack restore token marks a restore point of the shadow stack, and the address in a token must point directly above the token, which is within the same shadow stack. This is distinctively different from other pointers on the shadow stack, since those pointers point to executable code area. Introduce token setup and verify routines. Also introduce WRUSS, which is a kernel-mode instruction but writes directly to user shadow stack. In future patches that enable shadow stack to work with signals, the kernel will need something to denote the point in the stack where sigreturn may be called. This will prevent attackers calling sigreturn at arbitrary places in the stack, in order to help prevent SROP attacks. To do this, something that can only be written by the kernel needs to be placed on the shadow stack. This can be accomplished by setting bit 63 in the frame written to the shadow stack. Userspace return addresses can't have this bit set as it is in the kernel range. It is also can't be a valid restore token. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook --- v5: - Fix typo in commit log v3: - Drop shstk_check_rstor_token() - Fail put_shstk_data() if bit 63 is set in the data (Kees) - Add comment in create_rstor_token() (Kees) - Pull in create_rstor_token() changes from future patch (Kees) v2: - Add data helpers for writing to shadow stack. v1: - Use xsave helpers. --- arch/x86/include/asm/special_insns.h | 13 +++++ arch/x86/kernel/shstk.c | 73 ++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+) diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h index de48d1389936..d6cd9344f6c7 100644 --- a/arch/x86/include/asm/special_insns.h +++ b/arch/x86/include/asm/special_insns.h @@ -202,6 +202,19 @@ static inline void clwb(volatile void *__p) : [pax] "a" (p)); } +#ifdef CONFIG_X86_USER_SHADOW_STACK +static inline int write_user_shstk_64(u64 __user *addr, u64 val) +{ + asm_volatile_goto("1: wrussq %[val], (%[addr])\n" + _ASM_EXTABLE(1b, %l[fail]) + :: [addr] "r" (addr), [val] "r" (val) + :: fail); + return 0; +fail: + return -EFAULT; +} +#endif /* CONFIG_X86_USER_SHADOW_STACK */ + #define nop() asm volatile ("nop") static inline void serialize(void) diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 1d30295e0066..13c02747386f 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -25,6 +25,8 @@ #include #include +#define SS_FRAME_SIZE 8 + static bool features_enabled(unsigned long features) { return current->thread.features & features; @@ -40,6 +42,35 @@ static void features_clr(unsigned long features) current->thread.features &= ~features; } +/* + * Create a restore token on the shadow stack. A token is always 8-byte + * and aligned to 8. + */ +static int create_rstor_token(unsigned long ssp, unsigned long *token_addr) +{ + unsigned long addr; + + /* Token must be aligned */ + if (!IS_ALIGNED(ssp, 8)) + return -EINVAL; + + addr = ssp - SS_FRAME_SIZE; + + /* + * SSP is aligned, so reserved bits and mode bit are a zero, just mark + * the token 64-bit. + */ + ssp |= BIT(0); + + if (write_user_shstk_64((u64 __user *)addr, (u64)ssp)) + return -EFAULT; + + if (token_addr) + *token_addr = addr; + + return 0; +} + static unsigned long alloc_shstk(unsigned long size) { int flags = MAP_ANONYMOUS | MAP_PRIVATE | MAP_ABOVE4G; @@ -159,6 +190,48 @@ int shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags, return 0; } +static unsigned long get_user_shstk_addr(void) +{ + unsigned long long ssp; + + fpregs_lock_and_load(); + + rdmsrl(MSR_IA32_PL3_SSP, ssp); + + fpregs_unlock(); + + return ssp; +} + +static int put_shstk_data(u64 __user *addr, u64 data) +{ + if (WARN_ON_ONCE(data & BIT(63))) + return -EINVAL; + + /* + * Mark the high bit so that the sigframe can't be processed as a + * return address. + */ + if (write_user_shstk_64(addr, data | BIT(63))) + return -EFAULT; + return 0; +} + +static int get_shstk_data(unsigned long *data, unsigned long __user *addr) +{ + unsigned long ldata; + + if (unlikely(get_user(ldata, addr))) + return -EFAULT; + + if (!(ldata & BIT(63))) + return -EINVAL; + + *data = ldata & ~BIT(63); + + return 0; +} + void shstk_free(struct task_struct *tsk) { struct thread_shstk *shstk = &tsk->thread.shstk; From patchwork Mon Feb 27 22:29:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154263 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA16DC64ED9 for ; Mon, 27 Feb 2023 22:32:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0F1D36B00A4; Mon, 27 Feb 2023 17:31:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 07C136B00A5; Mon, 27 Feb 2023 17:31:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE8856B00A6; Mon, 27 Feb 2023 17:31:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C85626B00A4 for ; Mon, 27 Feb 2023 17:31:58 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6B07216029E for ; Mon, 27 Feb 2023 22:31:58 +0000 (UTC) X-FDA: 80514520716.29.8724D0E Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf07.hostedemail.com (Postfix) with ESMTP id 79D6540011 for ; Mon, 27 Feb 2023 22:31:56 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=FogXzCaT; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537116; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=TY2JtR8ifKge5ml0A1+y7lAR6dgyU2LGxzFL7xG1GFc=; b=bQtujuQ4sUK76VXF3fnqN449f/nuMUSxGzMWtQZOjZXV5D0ialrYDwEZdK3IYYU6+ao8hN KIYN0yUUJKLgP/7RLYTfvf9mGHVNQ/oH+XhDRSOpyC2kr9U/KJqylD8RFJ++D47JMlIt98 B7wc0Ojipf8MbNCsGfk1t1GAKZC1JFY= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=FogXzCaT; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537116; a=rsa-sha256; cv=none; b=hpwNFbXCQJ7r6d7lkrg99fhmtWQmUOk1NLx39zGvILqfp7/B5cf6SBnBoPrFkyrUIwRd57 86G8r8WQeRKHPBiEzjqlfou8kpcguK2nP3BRm8XDoMu8BLuEMniJhBQ9HEgzeJNrFcxi/c AbnBu9LQ1Gzndcs7TOD8w6n6tiTBbgw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537116; x=1709073116; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=lP5MPQ2jM0Cm31tDJBvrQ8+bAxBLar4e9NDXZhzO2V4=; b=FogXzCaTjX0THiLfHwnE9Th1K1DmMo0jYEM9JHrEoS3blvZ7BdhOFsLf Pf0xN0atPx5y1uHgKdny8/vmuxhD2ANf3BXB2vcEe4GUE1aeunNE8n28E Yfo8OC4UoptUIDP/SIjkcelFUhuKpQynYPesOXSn2aVBJV67vXSJmHUIG gu1bxRiaPsNDaKeY0qiWi3x5lSayvI+nE405q+BoLc7hqgaM4dljHCk3/ hhcQnbWBDSwifJwLaTUeXV+rHR8fBp7vZem+w6jFmx3pOwfpHuwx6yvEU HrAsf2UOES48dhX7dp1LIJShmgUM4mPZCKXDFbsHhQ7WBpayWiLU18hi1 A==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657774" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657774" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:32 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024758" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024758" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:31 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 32/41] x86/shstk: Handle signals for shadow stack Date: Mon, 27 Feb 2023 14:29:48 -0800 Message-Id: <20230227222957.24501-33-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 79D6540011 X-Stat-Signature: gt1hhxu6bngzu8w94jpptayne86co1z4 X-HE-Tag: 1677537116-263128 X-HE-Meta: U2FsdGVkX1/+M3VCUQTYohjsA3DttMfyF/5lDaO55nwi4F8dBcQnGmYjp//0f6pewjjujRrGgpZe4siH8Ghb1h6rKqT9NBEbmmNQLS1k28ad6jfAxzawyXlQw755WzgTvVOpgsvHTck+XJ9cQbs/4/YOS1tJ2Na5M8E/o9LZGcql2YlWbPMXsakBH2OzQHRRkh74Dsnq71XZD4HQs6PGOc7794bAoENKTPghU4XrzLUe8WkudAqYdh/kWbDxan2vxddCy+JQCFXDQiNgVLzUVXqlbsMMeWlk+5QlOZluohRfz+kMDBLCgKU8Sb+dM+oAG0hf6M1Jb1WESA/EOxYVGYpHpxAYR9tQZhMR4sdyYYpgTew7g1mDpEhDzmm6JpvyZ2xFSlHaHZXY3JBN7b2A+SA7gclq4CHzzrP4rFO1OchVuyvP/+Ib8PqmVeJr/U4E/qh+q5tIvqVMHmsNehzVb7OQTld3pkhjUEf4jr9qyyIO3Fb9ntFO+xc1L4vNLIU3q7Iu0coZJEXgFN0/YRcwChK4+mvxsdRTYtGLfNTj3j1OXvPg1g4tAOiA6pVFNq/9PpIOc0uz8ur4Q4JwHy+XhRMM327xpBVCiNZcLUGwfnrQjAqeLLiHaU4lk9Qdwtt3/F1ySH3dsluR4FFw9vL67Pd4861mX5vW2yNE/7/W/HjUpyCH7fEcSzSj8pnuo9U6384gAal2IYexpjEH2LX27jvQQdHdaURhUNoxEW4GVhRi/SzxrbSI2zOQGqNjwvVSCe0GcaeaaNBoarsC86V/XeVi7HoySs2dQ22LQQEkY2HCeYHuGxxg8sADqXuT6wHLqo7xKJ5XlkBkwD4Y9bteGqFXH2HnTqUVsD8zjgVYjdUfyrJ1sw5kBnMrP+RzKVtwEnLl+ornhKKlcj9fbsZ+zrYdfir35dTx72Cw79CATHDhElLf73HrXANrij6BNktLtFfRkBY9pWDwhVSYrGp +JObXw04 JZc5iZbxzQY2BNa9bq6MkuKAyvcrzMJty4XR25YCx80B0iaIdEPr7nUAITcwxC0cuLdsp8AYyeb+0sPTZvwnsE07sM15t2/4myfII+yej77gwrrIb9DCq7OZI+JUir5CYDBWhYNCBd3XdnsvOqJ+owwgKWiks1ZP9Hj1Fiw2sS3vC1O5bgz7ussF9yVt4PCbI+E6k7DnxTb9W213syZY8wBtAikAnQm16/a99M7zOWWKg/kgKeQfzb37BlNcLzyajZ24p0dX7sfm5B5CxIAIkpSBcJs5N7fjbVNB2x/QgOrF9L+mtQtTQN2WqLns2S2exrJuBu+qRR4GKjKoOGVXK1e+8vlYJ23Vyrz+JqeVNov1iiyQmnV1gOfUDViUTZdymG6Qom5LLAYnXSaXewrAwq3IeAfW5FfW+uqqTgbq9ba8d/KygcQPmHX8+p1EOiHRSvQrQ8/ZU5M/imIg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu When a signal is handled normally the context is pushed to the stack before handling it. For shadow stacks, since the shadow stack only track's return addresses, there isn't any state that needs to be pushed. However, there are still a few things that need to be done. These things are userspace visible and which will be kernel ABI for shadow stacks. One is to make sure the restorer address is written to shadow stack, since the signal handler (if not changing ucontext) returns to the restorer, and the restorer calls sigreturn. So add the restorer on the shadow stack before handling the signal, so there is not a conflict when the signal handler returns to the restorer. The other thing to do is to place some type of checkable token on the thread's shadow stack before handling the signal and check it during sigreturn. This is an extra layer of protection to hamper attackers calling sigreturn manually as in SROP-like attacks. For this token we can use the shadow stack data format defined earlier. Have the data pushed be the previous SSP. In the future the sigreturn might want to return back to a different stack. Storing the SSP (instead of a restore offset or something) allows for future functionality that may want to restore to a different stack. So, when handling a signal push - the SSP pointing in the shadow stack data format - the restorer address below the restore token. In sigreturn, verify SSP is stored in the data format and pop the shadow stack. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Andy Lutomirski Cc: Cyrill Gorcunov Cc: Florian Weimer Cc: H. Peter Anvin Cc: Kees Cook --- v3: - Drop shstk_setup_rstor_token() (Kees) - Drop x32 signal support, since x32 support is dropped v2: - Switch to new shstk signal format v1: - Use xsave helpers. - Expand commit log. Yu-cheng v27: - Eliminate saving shadow stack pointer to signal context. --- arch/x86/include/asm/shstk.h | 5 ++ arch/x86/kernel/shstk.c | 98 ++++++++++++++++++++++++++++++++++++ arch/x86/kernel/signal.c | 1 + arch/x86/kernel/signal_64.c | 6 +++ 4 files changed, 110 insertions(+) diff --git a/arch/x86/include/asm/shstk.h b/arch/x86/include/asm/shstk.h index 1399f4df098b..acee68d30a07 100644 --- a/arch/x86/include/asm/shstk.h +++ b/arch/x86/include/asm/shstk.h @@ -6,6 +6,7 @@ #include struct task_struct; +struct ksignal; #ifdef CONFIG_X86_USER_SHADOW_STACK struct thread_shstk { @@ -19,6 +20,8 @@ int shstk_alloc_thread_stack(struct task_struct *p, unsigned long clone_flags, unsigned long stack_size, unsigned long *shstk_addr); void shstk_free(struct task_struct *p); +int setup_signal_shadow_stack(struct ksignal *ksig); +int restore_signal_shadow_stack(void); #else static inline long shstk_prctl(struct task_struct *task, int option, unsigned long arg2) { return -EINVAL; } @@ -28,6 +31,8 @@ static inline int shstk_alloc_thread_stack(struct task_struct *p, unsigned long stack_size, unsigned long *shstk_addr) { return 0; } static inline void shstk_free(struct task_struct *p) {} +static inline int setup_signal_shadow_stack(struct ksignal *ksig) { return 0; } +static inline int restore_signal_shadow_stack(void) { return 0; } #endif /* CONFIG_X86_USER_SHADOW_STACK */ #endif /* __ASSEMBLY__ */ diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 13c02747386f..40f0a55762a9 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -232,6 +232,104 @@ static int get_shstk_data(unsigned long *data, unsigned long __user *addr) return 0; } +static int shstk_push_sigframe(unsigned long *ssp) +{ + unsigned long target_ssp = *ssp; + + /* Token must be aligned */ + if (!IS_ALIGNED(*ssp, 8)) + return -EINVAL; + + if (!IS_ALIGNED(target_ssp, 8)) + return -EINVAL; + + *ssp -= SS_FRAME_SIZE; + if (put_shstk_data((void *__user)*ssp, target_ssp)) + return -EFAULT; + + return 0; +} + +static int shstk_pop_sigframe(unsigned long *ssp) +{ + unsigned long token_addr; + int err; + + err = get_shstk_data(&token_addr, (unsigned long __user *)*ssp); + if (unlikely(err)) + return err; + + /* Restore SSP aligned? */ + if (unlikely(!IS_ALIGNED(token_addr, 8))) + return -EINVAL; + + /* SSP in userspace? */ + if (unlikely(token_addr >= TASK_SIZE_MAX)) + return -EINVAL; + + *ssp = token_addr; + + return 0; +} + +int setup_signal_shadow_stack(struct ksignal *ksig) +{ + void __user *restorer = ksig->ka.sa.sa_restorer; + unsigned long ssp; + int err; + + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) || + !features_enabled(ARCH_SHSTK_SHSTK)) + return 0; + + if (!restorer) + return -EINVAL; + + ssp = get_user_shstk_addr(); + if (unlikely(!ssp)) + return -EINVAL; + + err = shstk_push_sigframe(&ssp); + if (unlikely(err)) + return err; + + /* Push restorer address */ + ssp -= SS_FRAME_SIZE; + err = write_user_shstk_64((u64 __user *)ssp, (u64)restorer); + if (unlikely(err)) + return -EFAULT; + + fpregs_lock_and_load(); + wrmsrl(MSR_IA32_PL3_SSP, ssp); + fpregs_unlock(); + + return 0; +} + +int restore_signal_shadow_stack(void) +{ + unsigned long ssp; + int err; + + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) || + !features_enabled(ARCH_SHSTK_SHSTK)) + return 0; + + ssp = get_user_shstk_addr(); + if (unlikely(!ssp)) + return -EINVAL; + + err = shstk_pop_sigframe(&ssp); + if (unlikely(err)) + return err; + + fpregs_lock_and_load(); + wrmsrl(MSR_IA32_PL3_SSP, ssp); + fpregs_unlock(); + + return 0; +} + void shstk_free(struct task_struct *tsk) { struct thread_shstk *shstk = &tsk->thread.shstk; diff --git a/arch/x86/kernel/signal.c b/arch/x86/kernel/signal.c index 004cb30b7419..356253e85ce9 100644 --- a/arch/x86/kernel/signal.c +++ b/arch/x86/kernel/signal.c @@ -40,6 +40,7 @@ #include #include #include +#include static inline int is_ia32_compat_frame(struct ksignal *ksig) { diff --git a/arch/x86/kernel/signal_64.c b/arch/x86/kernel/signal_64.c index 0e808c72bf7e..cacf2ede6217 100644 --- a/arch/x86/kernel/signal_64.c +++ b/arch/x86/kernel/signal_64.c @@ -175,6 +175,9 @@ int x64_setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs) frame = get_sigframe(ksig, regs, sizeof(struct rt_sigframe), &fp); uc_flags = frame_uc_flags(regs); + if (setup_signal_shadow_stack(ksig)) + return -EFAULT; + if (!user_access_begin(frame, sizeof(*frame))) return -EFAULT; @@ -260,6 +263,9 @@ SYSCALL_DEFINE0(rt_sigreturn) if (!restore_sigcontext(regs, &frame->uc.uc_mcontext, uc_flags)) goto badframe; + if (restore_signal_shadow_stack()) + goto badframe; + if (restore_altstack(&frame->uc.uc_stack)) goto badframe; From patchwork Mon Feb 27 22:29:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154264 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41E2CC7EE2E for ; Mon, 27 Feb 2023 22:32:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 930806B00A5; Mon, 27 Feb 2023 17:31:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8E00A6B00A6; Mon, 27 Feb 2023 17:31:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 732086B00A7; Mon, 27 Feb 2023 17:31:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 64E186B00A5 for ; Mon, 27 Feb 2023 17:31:59 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 291E3AAF96 for ; Mon, 27 Feb 2023 22:31:59 +0000 (UTC) X-FDA: 80514520758.26.0B141E4 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf28.hostedemail.com (Postfix) with ESMTP id 0715CC0010 for ; Mon, 27 Feb 2023 22:31:56 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=htfEXMEO; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537117; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=2FtOU/X+ROob91Drvw/u5zl2nlpfMJbMpmIA9iRxoYU=; b=NRgZ3k2x5yFskWHh3BBbYGsOIgO0+/4J9x5O4/1WIP40yqK4gkx7A9o7II3AX0YSVAMxHH SeN1sKBAVOryGF3GcHWEaCaHym1YMjPJBBza8k16FDLnj7LSHWR4WRmyWtWFSfJykthr/d tasBtT+bkIJ36wfXF9dSLdR1eg365kI= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=htfEXMEO; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537117; a=rsa-sha256; cv=none; b=MswoNePfg6RJJPxcTNKLi8vmS6BpsNcvySv3YDreJIfqe4dLZilUv5JkoJtozy2lzcPPDL gKMeESYlBDh9zStkxeRC2ZTnh/t64oDBTcZUqHjQyZ6+f2zrmyTFXNZ2Mwwb3Kb4jzInAd SUUk4P4S56/K5sKjfeBHjzoa/IcLSu4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537117; x=1709073117; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=0sN0qiJhpy5r3MZ28FxXbF8Pdb6RhgABRtgYTnCuxVM=; b=htfEXMEO+vgr93l0tjiPi9NCz8x5udGoDxEBnJvwstA8IpJQyRN9JA7s p0V1ODC4nztfKci4d/S9fI/9hXOWAuiS7KrIhE2iby1vFhnuK2zPLKkL4 izuyTD02fg7bSK8H2FJLjwrhwWd5M9V+DqK9l+ex3jzPhIVOPcoDbSayu /Vvx1+e0+fMguXDfZ/OlnMN9ijb9PCaU4u3sAoErqhL31U1ljr0NnA/ay uRwWCZxu/GpYoZ8zpTQz7Kv6WVsB/rfvysGtjJdxLMvSuKqFbXXW6N5Zy G0f0BmYUuNXW3V2zWeYbAlvj4XOExTBSvjaCoeyrQrHWoC6uHVs6f1BZo A==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657806" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657806" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:32 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024769" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024769" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:32 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v7 33/41] x86/shstk: Introduce map_shadow_stack syscall Date: Mon, 27 Feb 2023 14:29:49 -0800 Message-Id: <20230227222957.24501-34-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 0715CC0010 X-Stat-Signature: shs6qwhcyibqtew3un1jzwb7qb1jkpk4 X-Rspam-User: X-HE-Tag: 1677537116-255029 X-HE-Meta: U2FsdGVkX183rKdEFHVJq1et7kVivvuBZ0X2p9ZA4XyG8UdPdQ8l1uvaTwWhHLBkpUp1bkGzFI5s1Bl+rQoawM651vCJgSYfUZiODaO1qtBUcY3pFS9C9DVRu5N8QHLc06zWOKFGmqc4JAH0wZKE+CXJmQB54vBad4XivGC+8Re+ivtQ3/s4yU3fWZv8TdMZCH8NWwiLoBQK5auXY9iTOCW9X7O8pvpxdf9koeIPRjxVeE0brRv9sP+DsoMhXW+oah9gdIIGNdGspAqALm8kAVQ5rXVtJ4W5yMENnp6LxJ8tY/NYm99pRQs6pmSgLg4WUtdPyAinqCNHdw6vNcwsncdyzDo7GLnxN/gAEWH53tzMX1TRtFHmBy4o4gEWeyImRxEbuJ5A8kLsn2pc/dYluv4AuJhac9lHDxnDDLYtWZAlfTaTO09ZK6mSoptv5UBuUHiqCpQnwU0xKXfja63fBbvbOG7SjiAvouWS72nnLa+AvAfbTbaDb0BrhW2s8I01DJ8ch1ZOybxIPiUbRSKopwx1KsRzDQSSZVNQB4PbKptMZQIBDbb0WvtlzfX+HawvY1oQ/49YUGk0fM/otx8o3N5LqTg3mGoIan2lQSuNiOnphPyGnC9/g1lkL24LPBTW5yp5x/G+GlzywupgUST4RQNaQuN5bvs5pLVaScXY+d8VnJyEf5ZxLCNdI3PFiRLEubZr29SpItVGqM9ApXcMEv66tADzuzbNJ7IKh++n9KocJLST3EurpJ0td7Ueo386VawJ0K99N0HIZMuHHs8Oirc2r0LtsxAL/sAl8Ty573CqP1g4EwYCmEk6o0SsJHvpOmEPrzcbJaBjGVARnDiYy8wux37ZM5ipjQ/XkUWqoLwe09iC3wwbnzfvN+Dj2bDvl2FUBimjzgHWo1ZS72UpQ8GjiIi37nUu5WWsmjeZgQBL8FmN0t/Y4VUasd2157+GDUECOdaK+scLjNJmMOE bn8Ss1vv mQrwBltYnkXOQDeqDLmKP2aY8AqVkpO94weWHpSFqZV50LMUZQEtEGPgO22aeHVxRsXsZce5HEjLBtQYZLbtJ0n7D5aawiG8endxnasRKhx7ilFtdlpWXTrjT0463bgLjjKobxsyyvcEJXBjjWwvQ6juyu7GjMGGf0BVYGgn26DsrNdK0bLLm9WdlPf4jWZd+w2BDMetV6cljONha1U8s236FXWnmJnKOXUjWCsv354leS3Cdq3b6VztqNQyb48zEskEafqZ3mVlCwFWla6f4f2NxU0/uvBXVVM8FiIc712U2lEo3wUOFH3GI2wZU+/PLrXSmvNksKQp9utl+8TKg9zMaGw7eH0cir15BLS1JtpXYiKWNTO8Ncte4fC1toWa80I3ihoKs8w4wz4wUxrBDASoKNQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When operating with shadow stacks enabled, the kernel will automatically allocate shadow stacks for new threads, however in some cases userspace will need additional shadow stacks. The main example of this is the ucontext family of functions, which require userspace allocating and pivoting to userspace managed stacks. Unlike most other user memory permissions, shadow stacks need to be provisioned with special data in order to be useful. They need to be setup with a restore token so that userspace can pivot to them via the RSTORSSP instruction. But, the security design of shadow stack's is that they should not be written to except in limited circumstances. This presents a problem for userspace, as to how userspace can provision this special data, without allowing for the shadow stack to be generally writable. Previously, a new PROT_SHADOW_STACK was attempted, which could be mprotect()ed from RW permissions after the data was provisioned. This was found to not be secure enough, as other thread's could write to the shadow stack during the writable window. The kernel can use a special instruction, WRUSS, to write directly to userspace shadow stacks. So the solution can be that memory can be mapped as shadow stack permissions from the beginning (never generally writable in userspace), and the kernel itself can write the restore token. First, a new madvise() flag was explored, which could operate on the PROT_SHADOW_STACK memory. This had a couple downsides: 1. Extra checks were needed in mprotect() to prevent writable memory from ever becoming PROT_SHADOW_STACK. 2. Extra checks/vma state were needed in the new madvise() to prevent restore tokens being written into the middle of pre-used shadow stacks. It is ideal to prevent restore tokens being added at arbitrary locations, so the check was to make sure the shadow stack had never been written to. 3. It stood out from the rest of the madvise flags, as more of direct action than a hint at future desired behavior. So rather than repurpose two existing syscalls (mmap, madvise) that don't quite fit, just implement a new map_shadow_stack syscall to allow userspace to map and setup new shadow stacks in one step. While ucontext is the primary motivator, userspace may have other unforeseen reasons to setup it's own shadow stacks using the WRSS instruction. Towards this provide a flag so that stacks can be optionally setup securely for the common case of ucontext without enabling WRSS. Or potentially have the kernel set up the shadow stack in some new way. The following example demonstrates how to create a new shadow stack with map_shadow_stack: void *shstk = map_shadow_stack(addr, stack_size, SHADOW_STACK_SET_TOKEN); Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Rick Edgecombe --- v5: - Fix addr/mapped_addr (Kees) - Switch to EOPNOTSUPP (Kees suggested ENOTSUPP, but checkpatch suggests this) - Return error for addresses below 4G v3: - Change syscall common -> 64 (Kees) - Use bit shift notation instead of 0x1 for uapi header (Kees) - Call do_mmap() with MAP_FIXED_NOREPLACE (Kees) - Block unsupported flags (Kees) - Require size >= 8 to set token (Kees) v2: - Change syscall to take address like mmap() for CRIU's usage v1: - New patch (replaces PROT_SHADOW_STACK). --- arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/include/uapi/asm/mman.h | 3 ++ arch/x86/kernel/shstk.c | 59 ++++++++++++++++++++++---- include/linux/syscalls.h | 1 + include/uapi/asm-generic/unistd.h | 2 +- kernel/sys_ni.c | 1 + 6 files changed, 58 insertions(+), 9 deletions(-) diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl index c84d12608cd2..f65c671ce3b1 100644 --- a/arch/x86/entry/syscalls/syscall_64.tbl +++ b/arch/x86/entry/syscalls/syscall_64.tbl @@ -372,6 +372,7 @@ 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv 450 common set_mempolicy_home_node sys_set_mempolicy_home_node +451 64 map_shadow_stack sys_map_shadow_stack # # Due to a historical design error, certain syscalls are numbered differently diff --git a/arch/x86/include/uapi/asm/mman.h b/arch/x86/include/uapi/asm/mman.h index 5a0256e73f1e..8148bdddbd2c 100644 --- a/arch/x86/include/uapi/asm/mman.h +++ b/arch/x86/include/uapi/asm/mman.h @@ -13,6 +13,9 @@ ((key) & 0x8 ? VM_PKEY_BIT3 : 0)) #endif +/* Flags for map_shadow_stack(2) */ +#define SHADOW_STACK_SET_TOKEN (1ULL << 0) /* Set up a restore token in the shadow stack */ + #include #endif /* _ASM_X86_MMAN_H */ diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 40f0a55762a9..0a3decab70ee 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include @@ -71,19 +72,31 @@ static int create_rstor_token(unsigned long ssp, unsigned long *token_addr) return 0; } -static unsigned long alloc_shstk(unsigned long size) +static unsigned long alloc_shstk(unsigned long addr, unsigned long size, + unsigned long token_offset, bool set_res_tok) { int flags = MAP_ANONYMOUS | MAP_PRIVATE | MAP_ABOVE4G; struct mm_struct *mm = current->mm; - unsigned long addr, unused; + unsigned long mapped_addr, unused; - mmap_write_lock(mm); - addr = do_mmap(NULL, 0, size, PROT_READ, flags, - VM_SHADOW_STACK | VM_WRITE, 0, &unused, NULL); + if (addr) + flags |= MAP_FIXED_NOREPLACE; + mmap_write_lock(mm); + mapped_addr = do_mmap(NULL, addr, size, PROT_READ, flags, + VM_SHADOW_STACK | VM_WRITE, 0, &unused, NULL); mmap_write_unlock(mm); - return addr; + if (!set_res_tok || IS_ERR_VALUE(mapped_addr)) + goto out; + + if (create_rstor_token(mapped_addr + token_offset, NULL)) { + vm_munmap(mapped_addr, size); + return -EINVAL; + } + +out: + return mapped_addr; } static unsigned long adjust_shstk_size(unsigned long size) @@ -134,7 +147,7 @@ static int shstk_setup(void) return -EOPNOTSUPP; size = adjust_shstk_size(0); - addr = alloc_shstk(size); + addr = alloc_shstk(0, size, 0, false); if (IS_ERR_VALUE(addr)) return PTR_ERR((void *)addr); @@ -178,7 +191,7 @@ int shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags, return 0; size = adjust_shstk_size(stack_size); - addr = alloc_shstk(size); + addr = alloc_shstk(0, size, 0, false); if (IS_ERR_VALUE(addr)) return PTR_ERR((void *)addr); @@ -371,6 +384,36 @@ static int shstk_disable(void) return 0; } +SYSCALL_DEFINE3(map_shadow_stack, unsigned long, addr, unsigned long, size, unsigned int, flags) +{ + bool set_tok = flags & SHADOW_STACK_SET_TOKEN; + unsigned long aligned_size; + + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return -EOPNOTSUPP; + + if (flags & ~SHADOW_STACK_SET_TOKEN) + return -EINVAL; + + /* If there isn't space for a token */ + if (set_tok && size < 8) + return -EINVAL; + + if (addr && addr <= 0xFFFFFFFF) + return -EINVAL; + + /* + * An overflow would result in attempting to write the restore token + * to the wrong location. Not catastrophic, but just return the right + * error code and block it. + */ + aligned_size = PAGE_ALIGN(size); + if (aligned_size < size) + return -EOVERFLOW; + + return alloc_shstk(addr, aligned_size, size, set_tok); +} + long shstk_prctl(struct task_struct *task, int option, unsigned long features) { if (option == ARCH_SHSTK_LOCK) { diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h index 33a0ee3bcb2e..392dc11e3556 100644 --- a/include/linux/syscalls.h +++ b/include/linux/syscalls.h @@ -1058,6 +1058,7 @@ asmlinkage long sys_memfd_secret(unsigned int flags); asmlinkage long sys_set_mempolicy_home_node(unsigned long start, unsigned long len, unsigned long home_node, unsigned long flags); +asmlinkage long sys_map_shadow_stack(unsigned long addr, unsigned long size, unsigned int flags); /* * Architecture-specific system calls diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h index 45fa180cc56a..b12940ec5926 100644 --- a/include/uapi/asm-generic/unistd.h +++ b/include/uapi/asm-generic/unistd.h @@ -887,7 +887,7 @@ __SYSCALL(__NR_futex_waitv, sys_futex_waitv) __SYSCALL(__NR_set_mempolicy_home_node, sys_set_mempolicy_home_node) #undef __NR_syscalls -#define __NR_syscalls 451 +#define __NR_syscalls 452 /* * 32 bit systems traditionally used different diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c index 860b2dcf3ac4..cb9aebd34646 100644 --- a/kernel/sys_ni.c +++ b/kernel/sys_ni.c @@ -381,6 +381,7 @@ COND_SYSCALL(vm86old); COND_SYSCALL(modify_ldt); COND_SYSCALL(vm86); COND_SYSCALL(kexec_file_load); +COND_SYSCALL(map_shadow_stack); /* s390 */ COND_SYSCALL(s390_pci_mmio_read); From patchwork Mon Feb 27 22:29:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154265 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D22EEC64ED8 for ; Mon, 27 Feb 2023 22:32:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B53116B00A6; Mon, 27 Feb 2023 17:32:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ADB026B00A7; Mon, 27 Feb 2023 17:32:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 907546B00A8; Mon, 27 Feb 2023 17:32:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 81F866B00A6 for ; Mon, 27 Feb 2023 17:32:00 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5026E120306 for ; Mon, 27 Feb 2023 22:32:00 +0000 (UTC) X-FDA: 80514520800.09.6AE41A9 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf24.hostedemail.com (Postfix) with ESMTP id 70BD118000C for ; Mon, 27 Feb 2023 22:31:58 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=PX4+WgEM; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537118; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=PfVyx2ZUIDwJeog9TcVEcCcnDkbyw8PVB33z8FVE0wQ=; b=CuNjtpSKH/fm25zP6xgAG/OC4tQ8hjzQfGpKYLSW5b0zFxLTWaLJF8oQIiW5/eNl691+wA cOVVq20qPYtJlFlu/13nR/dGjGZMDt+nzu4NxfaFbDawgrbxTuUitDuBm8URof/momHxbY GT2Uy7OAh7uIIxgn3Hly+qLv6ezVjDY= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=PX4+WgEM; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537118; a=rsa-sha256; cv=none; b=AwsRsO2GtF93Vma7V6JcDmvMD35flZkzeunxaOP3iynOh/tivXKdlZ3Y9gArxCdWCjuLXl 65rwL4EhJZPx6iS2aWOwtJ2v7NnzuWRuuBJC/O52N/Wfl+IvU8slEq+sqksNt0yjUDVWzR LGzx6H6e7JU8YKhk4kXiwFJnvqOZMUM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537118; x=1709073118; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=HyAWoaPlsuor4NZWrHVRIBqQGXmP1Us+CQiWWbdiIWE=; b=PX4+WgEMCEtElX3vaZvxDIv4NCwVDsIUoWpnxXVvNmkkr/58yzrWxHpk rjlEGpPWpd+9V/5iB5FdZalRYJuIDOdibJNjp+wucTfA5NgLvWNziepIx J9UZMcJI12nQHp2urRIwOeuT53fTHTne3LereO/MkwczXt8JO5oHwU/jr vc2U2c0IIkWDOrB3C+A+3S6zAbllBk2xvbAVeFTxP5atrKRHzLmjBjzxD are2kNQudzMZ1t6OsmvCoqwcrnalOiQO7O/JignGqW5IdIlQTRSwo1P6i gxOOvEca1u49Z0NazR9ccaKk/U7Jxp4H8E2F322jtWSmMmVSzNspRPOT9 Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657824" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657824" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:33 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024775" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024775" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:32 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v7 34/41] x86/shstk: Support WRSS for userspace Date: Mon, 27 Feb 2023 14:29:50 -0800 Message-Id: <20230227222957.24501-35-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Queue-Id: 70BD118000C X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: ynx8i1br3c9o53dxnjpuft89akbc5ci7 X-HE-Tag: 1677537118-679228 X-HE-Meta: U2FsdGVkX1+HNF3yjJKx/IQtYLYnnMypd2GNngVmoXUeLnB7RVGLtUcFhJlVSPpAYNZNKEOFgBZ0b/HrNdMrBXWRgSUreH166zzVg5rKVK00Ii7velusEfLsxYjOGu3iBJPKbPD7rPfbgTqyPGMc0dbPH8EWYrTchmGpQGR5bYmRjC+wAGB/gUAaQVmqixP6rbvEeEeBDukP8ILCeGaOArGVn4IziPH//hmHUzi7E7bIPRIi0GQ8bxLQ8zvfzGccEWvaUPuqqn39ObVKQ8agUa782KeokxGMSC15pd96y8RGSXC127yI9M7IibigEy8Xtc7unPEqxLMbk01geoQrY5CWAYbcEOdUiqcxYhyWAVhTivo7h7KCrMD6vWzEVGBrtD4QD2OliRRrvZQZdaxgCDTSf2jPdJViQxxBFzf0T4UtXLzu0UQkWD039DrGJFKbVul3pWH1WT7ZbH82P9V08PeDynTIGND+pbNRaBePffrMKBvEFWQ5sMEBEyG0lI3xQAmqQYgFj0YEZQRlbsMANH/tQjOIY2zuiv3x29LjO99Kx5ea3X56zEq/SOt86fuBEm/4KF/Q6vx2INKA65sWKzDx/1+EnR/LUm/FpJhQpLCSA5fWVRU0lDL3AqkHf+duItdiymRMfDV5vMo5kor/fITWC0yoN7l2pfgEaFREcBURY5uaqAG4DDjHkT7YHFHkHP0o7qd266cQHCxjIvtiaWh3uKiKx8OshV1PU3+LLzmzFKq2LYucntOkr5xIkhY8E2gCCGOT9rFAijyXMJwclke/EQQcT7e/01Yzm4wzX/J+sv8/8CX8spohPf9W0/3rzxd4+AuRAnTlOLQHHB3O1HkPqld6hW2XUsW/lf7sEIfrja8JwBQMjoJiqvaSfhBVipjBjk1JsfI5R/nxySAzNKRNLXvSwowQaTrnxUCxsoEgvaHAue8bPz1umfSn6iToETJTYyqedO/RHsZSu0g smHtcnz5 fzgHNoI+maWP3f72gHXPkgmrZXGVD1ADeDa1RGcvk3tlKD0qsrptL6ASH/6LdGHt1oPkNLzJ5Fkvv+qERP3IvZWPctOfJJ/SzJIXoUVwB3gfblbV5iaGJ9o4sQJPV8+lMTAMdz8Nkm+jI89Ce40ZOmXglJGIZoBdp8qLlKbrgNGGrLWAlNbotkwClFPk0gmUpN/WB06gY8Ja+Z1WA33xs4TJ85X0gjAT4NPCVYSu65zT1SmUHdSK5gut/z+0QVTWwAzJeGgFHBwUSAEVEBoJZsJ3dHfCutXpUW9m+uZaT/KmyAfQJ4p4tVSNUzSVUTSfUppcxWoFyAgo7H1jakYI3Q6i17M0aePYl+zKr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For the current shadow stack implementation, shadow stacks contents can't easily be provisioned with arbitrary data. This property helps apps protect themselves better, but also restricts any potential apps that may want to do exotic things at the expense of a little security. The x86 shadow stack feature introduces a new instruction, WRSS, which can be enabled to write directly to shadow stack permissioned memory from userspace. Allow it to get enabled via the prctl interface. Only enable the userspace WRSS instruction, which allows writes to userspace shadow stacks from userspace. Do not allow it to be enabled independently of shadow stack, as HW does not support using WRSS when shadow stack is disabled. From a fault handler perspective, WRSS will behave very similar to WRUSS, which is treated like a user access from a #PF err code perspective. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Rick Edgecombe --- v6: - Make set_clr_bits_msrl() avoid side affects in 'msr' v5: - Switch to EOPNOTSUPP - Move set_clr_bits_msrl() to patch where it is first used - Commit log formatting v3: - Make wrss_control() static - Fix verbiage in commit log (Kees) v2: - Add some commit log verbiage from (Dave Hansen) v1: - New patch. --- arch/x86/include/asm/msr.h | 11 +++++++++++ arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/shstk.c | 32 ++++++++++++++++++++++++++++++- 3 files changed, 43 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/msr.h b/arch/x86/include/asm/msr.h index 65ec1965cd28..2d3b35c957ad 100644 --- a/arch/x86/include/asm/msr.h +++ b/arch/x86/include/asm/msr.h @@ -310,6 +310,17 @@ void msrs_free(struct msr *msrs); int msr_set_bit(u32 msr, u8 bit); int msr_clear_bit(u32 msr, u8 bit); +/* Helper that can never get accidentally un-inlined. */ +#define set_clr_bits_msrl(msr, set, clear) do { \ + u64 __val, __new_val, __msr = msr; \ + \ + rdmsrl(__msr, __val); \ + __new_val = (__val & ~(clear)) | (set); \ + \ + if (__new_val != __val) \ + wrmsrl(__msr, __new_val); \ +} while (0) + #ifdef CONFIG_SMP int rdmsr_on_cpu(unsigned int cpu, u32 msr_no, u32 *l, u32 *h); int wrmsr_on_cpu(unsigned int cpu, u32 msr_no, u32 l, u32 h); diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 7dfd9dc00509..e31495668056 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -28,5 +28,6 @@ /* ARCH_SHSTK_ features bits */ #define ARCH_SHSTK_SHSTK (1ULL << 0) +#define ARCH_SHSTK_WRSS (1ULL << 1) #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 0a3decab70ee..009cb3fa0ae5 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -363,6 +363,36 @@ void shstk_free(struct task_struct *tsk) unmap_shadow_stack(shstk->base, shstk->size); } +static int wrss_control(bool enable) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return -EOPNOTSUPP; + + /* + * Only enable wrss if shadow stack is enabled. If shadow stack is not + * enabled, wrss will already be disabled, so don't bother clearing it + * when disabling. + */ + if (!features_enabled(ARCH_SHSTK_SHSTK)) + return -EPERM; + + /* Already enabled/disabled? */ + if (features_enabled(ARCH_SHSTK_WRSS) == enable) + return 0; + + fpregs_lock_and_load(); + if (enable) { + set_clr_bits_msrl(MSR_IA32_U_CET, CET_WRSS_EN, 0); + features_set(ARCH_SHSTK_WRSS); + } else { + set_clr_bits_msrl(MSR_IA32_U_CET, 0, CET_WRSS_EN); + features_clr(ARCH_SHSTK_WRSS); + } + fpregs_unlock(); + + return 0; +} + static int shstk_disable(void) { if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) @@ -379,7 +409,7 @@ static int shstk_disable(void) fpregs_unlock(); shstk_free(current); - features_clr(ARCH_SHSTK_SHSTK); + features_clr(ARCH_SHSTK_SHSTK | ARCH_SHSTK_WRSS); return 0; } From patchwork Mon Feb 27 22:29:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154266 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6555FC64ED6 for ; Mon, 27 Feb 2023 22:32:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 365406B00A7; Mon, 27 Feb 2023 17:32:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F1056B00A8; Mon, 27 Feb 2023 17:32:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F2DD6B00A9; Mon, 27 Feb 2023 17:32:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DA2826B00A7 for ; Mon, 27 Feb 2023 17:32:00 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 95479C0A36 for ; Mon, 27 Feb 2023 22:32:00 +0000 (UTC) X-FDA: 80514520800.07.F97B69A Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf07.hostedemail.com (Postfix) with ESMTP id 9EA5B40003 for ; Mon, 27 Feb 2023 22:31:58 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=mab6eRah; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537118; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=v91OPFL+NXA/Nu8SB0fkXd3GzEnDmi+70tOEkSeZJRU=; b=1u2q/lsfFyN4AndAlONbULa91+W7ThwXRnhgdRM91Kmu7ZzNVvsOs2Jma1GaM4YmouZ0Tu jF1LLjiH5BeoQd1amMGZ3FvMoCICkmnHmMxmCMZhZu1j3rPDrXAFxHK74rHXK2gVbCDb3Z aDjW+GC70xQEMlsRvYpxrE4OqKqIO0E= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=mab6eRah; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537118; a=rsa-sha256; cv=none; b=vdzkqQ5C/y0tOp/VHk5j1IClZKa533nIJN9W6/8eKeH5tbSGXM8r0i74XwG3zv23m/BVk1 K7/cA+JDAIsRHC9MDZ2jJFmGtEeYnUQQJ/Hq7L6oU6JE+C6hfSylrU9teyqCJih2f5NTLm R+ytUH8N/cJ28rqSIIxJm3IpDXLETJg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537118; x=1709073118; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=IJ4r2WgIxo+o2NsnoMfw1hw6rWv+qYgJeXRvN9Qbwh8=; b=mab6eRahoTy3ACGnaJJdRQRtxXWiuuR9Fo7Z3OMo1eESaIoYkQogf8tP 883O8Yv2HIcBeTNWeMKR7m9fbwEsDqsLg8LoSTKzf3Akm5zxRG6SkCH7d y73CNVxVW2SGQDoqUTm5L/JIJjI37g5rJGGtzIVKh8oaKTZB9B600olbQ +ojl6q5Gst0dynHZlz7kMb7UCsWsatpTo+49/Th6Gjvw/N8TQMS4QRkWO pugLLeaYsJSE41U1LspBoolJJeFtJalLDhJ4lXtG3JMuXw2moq6SIFgXf WIiuCgQi5xogK/tAtPS2VOByJF3PbyiYJggwgjWy4YFcFZovtn3o4k2P8 Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657863" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657863" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:34 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024780" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024780" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:33 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v7 35/41] x86: Expose thread features in /proc/$PID/status Date: Mon, 27 Feb 2023 14:29:51 -0800 Message-Id: <20230227222957.24501-36-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 9EA5B40003 X-Stat-Signature: ww13k5xinutg6c7qzygunarkshcizp1r X-HE-Tag: 1677537118-720243 X-HE-Meta: U2FsdGVkX1/h7yCfTI2CcuwfaUn08VvA9Lyu5bXF6C+zERDYoeULcdibCxCVbeiKchl+fLVJ3wT2Xp6bMw9YwiBzwsZGetfZPlrZjFmHMG2m5i7+lbAD4F25Pivbze3U71xYe5uLqBsn/e+UCKXivtJP7p9TrMKxg6QRqy5YhpRJCkSkvr0cIDMbuP0PwcZSEHdD2dCp9EQVCMg6Q4UFixBHkCqDugEh6tvpp5QNni4zA678o4i7e2ZBd/iytI7cbRRGC3SAe44iEAVJoDPspEEhpMnU8b9WSJxQgRmyMmPFcOOQTE4u04tl2ywQ0lg9a6MzfPQyHmffaQy7Qwn2JCPhBLB79OWwdasVtbHGy+G6HB/AIbFdaKpfhgFjEAawGkCXtLWkDuR1U+LpZMXuGy1VL2N5+D4K+mz/Mh5ywkrZYOcJzFIl8J66yHA3K6lL2lcbSw2KCk8udP8jWgYffyP1KPsP55vgXFIn+EdkoZAJby6AxGdci92dLNqdEEePz97KHVLpC54J1ko/cJx+3J1aZpktqB04bcg3In2sJ3jpThYN2v/jVmD77ccT4Mcy4CP+2iHO6naNrGIgej3piVA3B0zUgWIIalIBgARIwTv+/+SpQfi43YGwIh3I1iDXC+h1oyaVu63Gdiese8Vg/OJ7DqeiOI5R/6ZWoeZtpHoA4LpXrJi1aZvAnUg+klJ+Z66goKOR9HstnJ2k7LMymHcW/EXs1jUgdH4a4f2OoUFGYaQlYQaN56l6p41wQeGXedfxR1RoFJEfu5cPyQ9dQQH2KUEx3jX5zpkAklfjsD05PcEfPszKmYkn9ZB7xQxpOp/Texswp0tNlcFhCMVaJMbCkkkW/GA7AM/9q/92t3cRjP1cdVYvVR2l9RVqWNx0EasVb51VrKw2uJtpKuh4SYTmP47WzVuxLZ5x5uxq7r/J2I8TihA8SI15fTjQ1qUCf1Mof/D2G0ULZGBB9Xg z+9BaUH4 YZlu6BL9vj3h+lz3ReUraFZzjnZ5Sl+kPHYHcT3/5Tw6adS8uj4blb+wZQpIrwNbTm85Myk/SU7apabZUdR0bSEaFpJIMfN2ii5oZRNr7nlVduKUwi3RgT1EkbGKUce6gADJTQAiIxehQMJdKZVk8tnjvtnO/2yvKaTzMYU7AizBhVYRVLwwjl0QWHeZkgkAu57L4W0T2kI/dWGxH4uiyjizuRPVHl3VBA1vd+7LEjdGJuPjcC7wIw68FkHvREKefGBMFhGjM87tR9NDISXPvg/smmXCt7CGqUYrA3CaX29+SoTUlFjiVU5IyJz/vkW+XvCwyDc5o/SFlpWS16iF5xFG3DjYEj7lZDWeoKyGSVCZ8XJkENah9ZT0z4srCk9KLkpQ99Dp8WNCU7ks= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Applications and loaders can have logic to decide whether to enable shadow stack. They usually don't report whether shadow stack has been enabled or not, so there is no way to verify whether an application actually is protected by shadow stack. Add two lines in /proc/$PID/status to report enabled and locked features. Since, this involves referring to arch specific defines in asm/prctl.h, implement an arch breakout to emit the feature lines. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Kirill A. Shutemov [Switched to CET, added to commit log] Signed-off-by: Rick Edgecombe --- v4: - Remove "CET" references v3: - Move to /proc/pid/status (Kees) v2: - New patch --- arch/x86/kernel/cpu/proc.c | 23 +++++++++++++++++++++++ fs/proc/array.c | 6 ++++++ include/linux/proc_fs.h | 2 ++ 3 files changed, 31 insertions(+) diff --git a/arch/x86/kernel/cpu/proc.c b/arch/x86/kernel/cpu/proc.c index 099b6f0d96bd..31c0e68f6227 100644 --- a/arch/x86/kernel/cpu/proc.c +++ b/arch/x86/kernel/cpu/proc.c @@ -4,6 +4,8 @@ #include #include #include +#include +#include #include "cpu.h" @@ -175,3 +177,24 @@ const struct seq_operations cpuinfo_op = { .stop = c_stop, .show = show_cpuinfo, }; + +#ifdef CONFIG_X86_USER_SHADOW_STACK +static void dump_x86_features(struct seq_file *m, unsigned long features) +{ + if (features & ARCH_SHSTK_SHSTK) + seq_puts(m, "shstk "); + if (features & ARCH_SHSTK_WRSS) + seq_puts(m, "wrss "); +} + +void arch_proc_pid_thread_features(struct seq_file *m, struct task_struct *task) +{ + seq_puts(m, "x86_Thread_features:\t"); + dump_x86_features(m, task->thread.features); + seq_putc(m, '\n'); + + seq_puts(m, "x86_Thread_features_locked:\t"); + dump_x86_features(m, task->thread.features_locked); + seq_putc(m, '\n'); +} +#endif /* CONFIG_X86_USER_SHADOW_STACK */ diff --git a/fs/proc/array.c b/fs/proc/array.c index 49283b8103c7..7ac43ecda1c2 100644 --- a/fs/proc/array.c +++ b/fs/proc/array.c @@ -428,6 +428,11 @@ static inline void task_thp_status(struct seq_file *m, struct mm_struct *mm) seq_printf(m, "THP_enabled:\t%d\n", thp_enabled); } +__weak void arch_proc_pid_thread_features(struct seq_file *m, + struct task_struct *task) +{ +} + int proc_pid_status(struct seq_file *m, struct pid_namespace *ns, struct pid *pid, struct task_struct *task) { @@ -451,6 +456,7 @@ int proc_pid_status(struct seq_file *m, struct pid_namespace *ns, task_cpus_allowed(m, task); cpuset_task_status_allowed(m, task); task_context_switch_counts(m, task); + arch_proc_pid_thread_features(m, task); return 0; } diff --git a/include/linux/proc_fs.h b/include/linux/proc_fs.h index 0260f5ea98fe..80ff8e533cbd 100644 --- a/include/linux/proc_fs.h +++ b/include/linux/proc_fs.h @@ -158,6 +158,8 @@ int proc_pid_arch_status(struct seq_file *m, struct pid_namespace *ns, struct pid *pid, struct task_struct *task); #endif /* CONFIG_PROC_PID_ARCH_STATUS */ +void arch_proc_pid_thread_features(struct seq_file *m, struct task_struct *task); + #else /* CONFIG_PROC_FS */ static inline void proc_root_init(void) From patchwork Mon Feb 27 22:29:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154267 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4431C7EE31 for ; Mon, 27 Feb 2023 22:32:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 81FA86B00A8; Mon, 27 Feb 2023 17:32:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7F6B36B00A9; Mon, 27 Feb 2023 17:32:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 648CE6B00AA; Mon, 27 Feb 2023 17:32:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 43FA16B00A9 for ; Mon, 27 Feb 2023 17:32:01 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 1FC0BC0A5F for ; Mon, 27 Feb 2023 22:32:01 +0000 (UTC) X-FDA: 80514520842.03.342E7D0 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf28.hostedemail.com (Postfix) with ESMTP id 337E4C0010 for ; Mon, 27 Feb 2023 22:31:58 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=QJ9mpQfh; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537119; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=LY7sRJ+HhyH/fRQUu6yhwRH2pCUOOjl5GAOCVlgILp0=; b=p/cJ/nQrxMI9h3kE7ktmohA7+7ytQ3Ch5tIfwzfQF8NdytKdekTu/1vwk7ZHaTPIeqkBfV zIXl5hCEi5tHJ4KbfEKVTeSkfl9sUQhKKrK2f2tGE8GzafDiOM1wjhUgRElu19HKpzH/tk wSQXDfvrN4EgZU5bBUuCGhSpmL1Xfbw= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=QJ9mpQfh; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537119; a=rsa-sha256; cv=none; b=qbMAel2XTtR4EaJO59zhvxQzXFldBcErm89iJyMWPYtU4n08eo7PAKjIgkl+9sWXEa5Qlz hMdsvcCiPXg94YpeRiMeOtQETQi8cnTWJiIjklzwW+59IzdTaT5UgggsKJAkisPoPrF2Pw okf712YZo/txgw9+dP5RAqEJBx79Lbs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537119; x=1709073119; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=u0fpzllgXo4dayZIZ+QVsU1muHhaSbmjAoWT0A5Xxuo=; b=QJ9mpQfheFI0ujQgeItINSK9vwxvSBQzUT5EaRhZy24DQtG+7PbqwPpd fISUdOcxUvovqryKxXKn1zXIDdt4CY33Yn9ZkR/1Szj+FUaxP6zzUiPdo SYNpD4t1XJhRi0cqRxFy7EXtMO648dN07tL6v9TlY1X0oAXF4UVZOMmWe t8RAGgbsUj218k+WAuyBNtEDe+/XiQFcHwgNzc9B8txi2fA/EwEGKHhnr 4LhaMe14ilCf9S5ZgoojyKJ57CeWzjmAjux/KrhxXnxGSwzNldNMRRT12 JpxZIXG+ahGzMtwpbruUNmLYlL9c5yeaIx7of5d18CM6EveABFZc/4zlC Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657885" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657885" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:34 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024790" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024790" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:34 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v7 36/41] x86/shstk: Wire in shadow stack interface Date: Mon, 27 Feb 2023 14:29:52 -0800 Message-Id: <20230227222957.24501-37-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 337E4C0010 X-Stat-Signature: 74khzximu5fg9ta8b7nah5m4wjk3tmhp X-Rspam-User: X-HE-Tag: 1677537118-26353 X-HE-Meta: U2FsdGVkX1/mOx2MPPALcBEPfO3VIZir3XcNPaLVxGpZC9gNA+k0AYbUJTNRWocQU3a6AoSmVA3YPALwXIYqXUOeKV0gLXmcwGDDmmJ9Nb8hdpnPv+NE1w4OK6y1mIQb9JK+juN/RCeizW84ElThU9Qsi+DxZfgUiMPR0Za6bqdHzMcMQfIvqYt4j4zffecr202Pr/sv/DbGCA1f+dQzMx1OAJ6ZowN+m6HvE8a5byO29gnKrq4s3eKLnsM6bJlcCqXgbEAJgntVv6DIGHxGp9doiBKBaBHFXNUsjQgbvAQ/nCmY4Pb7NB5jyv8Wois+QnQTU+Ie4ZRUqHKFgDNdllOrCzRMlisRXrAVTPztu3f0bf03QCTGGy8RZra1cpkzHWxb0mUENFAZOAvNFHQXNiihdrYi83s0QnHR+t96G7aB1o7cwMZZlHdhqaPICdzehof7JA5oMuJwpz4vDLe7ozAwWjPCCSAFsUXYuiHIsby7dynz3NmB25CbMuiI2UgrL74L6rCkYTD652e951tCBB5ChUj3Iz/vlhHglje6n6mUhlS2Vzk4IddDyxkWg3VJafDd6Io0oWw6QTh4+M6ZZrgiv/H1KHKbvykJXD3W9tI7dikUaAr19hRef/mQ0IVgSyAdS+06Z34r9eXyDTJCItJjpbt2LdVNWY4mrvArbnVYDL2hE0pY8wpc2K+VNLfLBk0CIMYmivXHlEjqqG4IfXP0n9xETRhjIQ8epdin7OogzSzpJF2npGrAn02jVR6kPFQNzSIIkRDx8mQaqscKsvIF61faYYcQjtELFQqsbQ8y27DlRgarQCR3J/U9kN/Ui6C2K1PPOZQBPW9qKWCDppOnwvpU7QNRO0xtN//GwTnDqIMuQ+UcianyTqq/y9DO6U2gwrQzN0oQAhguxAnaFAQDULJFcyW6knNQj8VWNAQjZ6F6EIg/m3wgycf5/lOzMZJ84RUv6buQfrtADkX 9oxRgByj 9SaHEOglP3I/K4Jm7Z5X8C7Ujd8qpkm7OsJ9DH+0jJrXVMUONB3HtJlncZqTSXNdjw+drlOv1miY/zsdkn3Rhkr0RHKIvQztJ0KdL6Mf3oadL8iBCC9gUQE1OThqnafjAx4QHocvjcMKv9PiDFPIBvCqwAj7/9q32KLRQUySyKnIA6XMDS/MxPEa3kqFQQQ9E/enOHykyoUDRkDL0OK6rYddMupZEQMPFpZZxA7DdvQ8ZE8yIE67K7P4oq+RHxt8GwTADeA343w6FSKhgFcnNzKB/FNvcopeSTIySSIha15sG3UVTnH8iSb1uiKkdRkMd4bLMpEP+ftMzSrTWz7qTBe4FsVkS/Ci+SZ8v X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The kernel now has the main shadow stack functionality to support applications. Wire in the WRSS and shadow stack enable/disable functions into the existing shadow stack API skeleton. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Rick Edgecombe --- v4: - Remove "CET" references v2: - Split from other patches --- arch/x86/kernel/shstk.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 009cb3fa0ae5..2faf9b45ac72 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -464,9 +464,17 @@ long shstk_prctl(struct task_struct *task, int option, unsigned long features) return -EINVAL; if (option == ARCH_SHSTK_DISABLE) { + if (features & ARCH_SHSTK_WRSS) + return wrss_control(false); + if (features & ARCH_SHSTK_SHSTK) + return shstk_disable(); return -EINVAL; } /* Handle ARCH_SHSTK_ENABLE */ + if (features & ARCH_SHSTK_SHSTK) + return shstk_setup(); + if (features & ARCH_SHSTK_WRSS) + return wrss_control(true); return -EINVAL; } From patchwork Mon Feb 27 22:29:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154269 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0ED5BC64ED6 for ; Mon, 27 Feb 2023 22:32:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 56F746B00AA; Mon, 27 Feb 2023 17:32:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4EB476B00AE; Mon, 27 Feb 2023 17:32:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 11A8E6B00AB; Mon, 27 Feb 2023 17:32:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E1F3C6B00A9 for ; Mon, 27 Feb 2023 17:32:02 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 96FA61408F8 for ; Mon, 27 Feb 2023 22:32:02 +0000 (UTC) X-FDA: 80514520884.22.CDB3B46 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf24.hostedemail.com (Postfix) with ESMTP id 9DABF180009 for ; Mon, 27 Feb 2023 22:32:00 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=CWfzECdc; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537120; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=ZybWw0rrCDhQS/b+Dz2vVvE91ri/kPPghseJnKjxSes=; b=Aje/h+JvbBCGhf7ti/ymn+Tz28vO7p2QNqUdPFOeyavc9+GX0e84P70EECMx9nJZDhcIiB UWFSuOCGYDOki356z1dPruUvw8I/GS19fyFMy6gbJepCn1e2hzqBNPExRw0xVspEQ5NOQg ZQZJxOpfjEudMMfMSYsjSRAFhe1H61U= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=CWfzECdc; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537121; a=rsa-sha256; cv=none; b=XpMV/pjNPFo91Ls09R3dE00jZHW667GJecP/hu4tvh3AwjtyjUFtHcmXaifIV1M4IygML0 t6AoR/k2EDN/wDS1rhC7vk2npjhLs4fXlxqflGAsCEz0mwjZDFnPzcq45DyFfV3kp9cmQV EITUeaYpUodldSvwpiUTwkDrtEylt9M= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537120; x=1709073120; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=cee+EewTjCtifSB1MytX2d+dveLwM02bV9ByAwGf9nE=; b=CWfzECdcNXyFIrDHDxXTqoBDYubgV3trKP83JhVAc0nPYO5JpJJV/TNA BBg/nPlWrqzGQw5+u0426doo9IrvcjOKl9hozRhg59HBgfpdy1tmdqu+P 2XkN2e7ogfCzJDf1Mxt+vf3V3liiYC5OOS8UeX3ncCl+SWYEoxl5L0ray rC+VW9WF/OcaP+X9aA4SJE3vKT7aoVWksT2nrt0MYGkcAljInTT3PX3qH DsUSUGgODf+9GCRGuZ9VexdTZf99jsupaqs+olVtB7o2PgartWhjbRGBS glEELBsv7ze4Pt9wIKi/oedz2jlVxJ2f5resIdhffdQj8LnyUjHcLxwNF A==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657914" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657914" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:35 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024799" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024799" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:34 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 37/41] selftests/x86: Add shadow stack test Date: Mon, 27 Feb 2023 14:29:53 -0800 Message-Id: <20230227222957.24501-38-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Queue-Id: 9DABF180009 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: shg8wimry3o6dubk3tsj6b5p1634ofsf X-HE-Tag: 1677537120-946675 X-HE-Meta: U2FsdGVkX1+y2SPpVqdlA6N76hayF688NRXzH1/n7tw7fuZwTKjrZcz1W5zht2PMi9YuU2RWImo1tAAn2Y8QJ9RNAQHw3vXkZeoGtA8RQ+0MvUrpNrtluOkfAIkN4sR/JaQ9k5QWkBgG6j5aZhXk2CRj9AsCGYEDKYK1jOVDfyZi140QjInEZgIb4NTMNfLIthIvDU5HQn7IV4FF6WuqMsR5t+fziOQnFlRoizvu3D15reVEuWbN84owR1sfAyRfiG73i57+rN4FQYXieoNv16+QbK3oAU/x8gqjX7sdyK5Xc4ei7DclCBaU+N2cNxHnMWuIrql/VO7IwH47RRrOd/vBM1O4IFfOWe8MRpcaqXm3QRRA2JxqGaUOJUD2QvUZcddX0dRhQqkfz/z9Ed3yM6Eoj03WzdhVRxnWfP+nu9Gt9GC0h5NNXIbs7WTaiFKpy4SuBsvX73zC5UXhZ460UPC4xsgKO0KqSGM16W9k+Y+C+i1llRx6H0D4spxl/ukkVstXEZamwkSyfzROZIlAegnhcNe9PZKMguuP77KRX1XE/oa22/ea7A6iY2DbNPIF8kXmTSJaYoxmwuZHwBvWpVsmRw017ndmVr2nmUGdLsmn55ySHqvh6WyykXiuv+YbopeXxtYYzfJ0R+a+PQ3+Du99YL5FfmY+xUMO0fDqveKacXGRZDIOQGgwgfnr+BrX80C5agV+vGkWDtRUkLa/AbNqIcfEHPwcBS9qxQkP8hg2qs9lHzAbiw226REmie+yW1zgET4GyycB00woHsKnBtSHt+nuAYlLeDPs1Z1dM6t1JsHHEJq3JTaIYV0r4rk4zjrCMr17Etn4WuNlvwiBA7x0mDG/W37Ok7avI49VFTYEyHaN3RVTKD/t0IeXI6wiI1KqqMVhtz7kSS0ClpOUugi+9jiJqHgbg6uLrTbE1QvgU/nwurFqK2Q1YKQj7GcXaaEvl76I9UaPWRztxNs kGofb7Ko BGZg9m7G0lrd4ReOtmujtVDANgesHUW0/3vL7PbnVxjTGo47WdZcq0+e2fxQvrLSe6/QRPhWCOAkNq5Zf3f8V1hOBkF8LHY7cP51Ls4ygaoSV1W3LYcU3ZQzz81LbFE86JAHREsOns6cMdcG8qWDwBymIB5Vfee5OcejwmOAsMuWuXvxZAzw7kw/3ebakq8OLgYuTojqt5dGTuVARht/WAxnR+eP5aoMrQ1sgQHLJ56Vv/zLINWPJ1WTnuL6h1VjifU617xzYlaHpDwEeVKiggT7/qMQD0GTM5lZ19feqfadhqeIrSpaahxWxtrzjvVI82khzBPwEsc/uLa6pY2bSz1b9MTpc9v/bXyPp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a simple selftest for exercising some shadow stack behavior: - map_shadow_stack syscall and pivot - Faulting in shadow stack memory - Handling shadow stack violations - GUP of shadow stack memory - mprotect() of shadow stack memory - Userfaultfd on shadow stack memory - 32 bit segmentation Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Co-developed-by: Yu-cheng Yu Signed-off-by: Yu-cheng Yu Signed-off-by: Rick Edgecombe --- v7: - Remove KHDR_INCLUDES and just add a copy of the defines (Boris) v6: - Tweak mprotect test - Code style tweaks v5: - Update 32 bit signal test with new ABI and better asm v4: - Add test for 32 bit signal ABI blocking v3: - Change "+m" to "=m" in write_shstk() (Andrew Cooper) - Fix userfaultfd test with transparent huge pages by doing a MADV_DONTNEED, since the token write faults in the while stack with huge pages. --- tools/testing/selftests/x86/Makefile | 2 +- .../testing/selftests/x86/test_shadow_stack.c | 695 ++++++++++++++++++ 2 files changed, 696 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/x86/test_shadow_stack.c diff --git a/tools/testing/selftests/x86/Makefile b/tools/testing/selftests/x86/Makefile index ca9374b56ead..cfc8a26ad151 100644 --- a/tools/testing/selftests/x86/Makefile +++ b/tools/testing/selftests/x86/Makefile @@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \ - corrupt_xstate_header amx + corrupt_xstate_header amx test_shadow_stack # Some selftests require 32bit support enabled also on 64bit systems TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall diff --git a/tools/testing/selftests/x86/test_shadow_stack.c b/tools/testing/selftests/x86/test_shadow_stack.c new file mode 100644 index 000000000000..94eb223456f6 --- /dev/null +++ b/tools/testing/selftests/x86/test_shadow_stack.c @@ -0,0 +1,695 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * This program test's basic kernel shadow stack support. It enables shadow + * stack manual via the arch_prctl(), instead of relying on glibc. It's + * Makefile doesn't compile with shadow stack support, so it doesn't rely on + * any particular glibc. As a result it can't do any operations that require + * special glibc shadow stack support (longjmp(), swapcontext(), etc). Just + * stick to the basics and hope the compiler doesn't do anything strange. + */ + +#define _GNU_SOURCE + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* + * Define the ABI defines if needed, so people can run the tests + * without building the headers. + */ +#ifndef __NR_map_shadow_stack +#define __NR_map_shadow_stack 451 + +#define SHADOW_STACK_SET_TOKEN (1ULL << 0) + +#define ARCH_SHSTK_ENABLE 0x5001 +#define ARCH_SHSTK_DISABLE 0x5002 +#define ARCH_SHSTK_LOCK 0x5003 +#define ARCH_SHSTK_UNLOCK 0x5004 +#define ARCH_SHSTK_STATUS 0x5005 + +#define ARCH_SHSTK_SHSTK (1ULL << 0) +#define ARCH_SHSTK_WRSS (1ULL << 1) +#endif + +#define SS_SIZE 0x200000 + +#if (__GNUC__ < 8) || (__GNUC__ == 8 && __GNUC_MINOR__ < 5) +int main(int argc, char *argv[]) +{ + printf("[SKIP]\tCompiler does not support CET.\n"); + return 0; +} +#else +void write_shstk(unsigned long *addr, unsigned long val) +{ + asm volatile("wrssq %[val], (%[addr])\n" + : "=m" (addr) + : [addr] "r" (addr), [val] "r" (val)); +} + +static inline unsigned long __attribute__((always_inline)) get_ssp(void) +{ + unsigned long ret = 0; + + asm volatile("xor %0, %0; rdsspq %0" : "=r" (ret)); + return ret; +} + +/* + * For use in inline enablement of shadow stack. + * + * The program can't return from the point where shadow stack gets enabled + * because there will be no address on the shadow stack. So it can't use + * syscall() for enablement, since it is a function. + * + * Based on code from nolibc.h. Keep a copy here because this can't pull in all + * of nolibc.h. + */ +#define ARCH_PRCTL(arg1, arg2) \ +({ \ + long _ret; \ + register long _num asm("eax") = __NR_arch_prctl; \ + register long _arg1 asm("rdi") = (long)(arg1); \ + register long _arg2 asm("rsi") = (long)(arg2); \ + \ + asm volatile ( \ + "syscall\n" \ + : "=a"(_ret) \ + : "r"(_arg1), "r"(_arg2), \ + "0"(_num) \ + : "rcx", "r11", "memory", "cc" \ + ); \ + _ret; \ +}) + +void *create_shstk(void *addr) +{ + return (void *)syscall(__NR_map_shadow_stack, addr, SS_SIZE, SHADOW_STACK_SET_TOKEN); +} + +void *create_normal_mem(void *addr) +{ + return mmap(addr, SS_SIZE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, 0, 0); +} + +void free_shstk(void *shstk) +{ + munmap(shstk, SS_SIZE); +} + +int reset_shstk(void *shstk) +{ + return madvise(shstk, SS_SIZE, MADV_DONTNEED); +} + +void try_shstk(unsigned long new_ssp) +{ + unsigned long ssp; + + printf("[INFO]\tnew_ssp = %lx, *new_ssp = %lx\n", + new_ssp, *((unsigned long *)new_ssp)); + + ssp = get_ssp(); + printf("[INFO]\tchanging ssp from %lx to %lx\n", ssp, new_ssp); + + asm volatile("rstorssp (%0)\n":: "r" (new_ssp)); + asm volatile("saveprevssp"); + printf("[INFO]\tssp is now %lx\n", get_ssp()); + + /* Switch back to original shadow stack */ + ssp -= 8; + asm volatile("rstorssp (%0)\n":: "r" (ssp)); + asm volatile("saveprevssp"); +} + +int test_shstk_pivot(void) +{ + void *shstk = create_shstk(0); + + if (shstk == MAP_FAILED) { + printf("[FAIL]\tError creating shadow stack: %d\n", errno); + return 1; + } + try_shstk((unsigned long)shstk + SS_SIZE - 8); + free_shstk(shstk); + + printf("[OK]\tShadow stack pivot\n"); + return 0; +} + +int test_shstk_faults(void) +{ + unsigned long *shstk = create_shstk(0); + + /* Read shadow stack, test if it's zero to not get read optimized out */ + if (*shstk != 0) + goto err; + + /* Wrss memory that was already read. */ + write_shstk(shstk, 1); + if (*shstk != 1) + goto err; + + /* Page out memory, so we can wrss it again. */ + if (reset_shstk((void *)shstk)) + goto err; + + write_shstk(shstk, 1); + if (*shstk != 1) + goto err; + + printf("[OK]\tShadow stack faults\n"); + return 0; + +err: + return 1; +} + +unsigned long saved_ssp; +unsigned long saved_ssp_val; +volatile bool segv_triggered; + +void __attribute__((noinline)) violate_ss(void) +{ + saved_ssp = get_ssp(); + saved_ssp_val = *(unsigned long *)saved_ssp; + + /* Corrupt shadow stack */ + printf("[INFO]\tCorrupting shadow stack\n"); + write_shstk((void *)saved_ssp, 0); +} + +void segv_handler(int signum, siginfo_t *si, void *uc) +{ + printf("[INFO]\tGenerated shadow stack violation successfully\n"); + + segv_triggered = true; + + /* Fix shadow stack */ + write_shstk((void *)saved_ssp, saved_ssp_val); +} + +int test_shstk_violation(void) +{ + struct sigaction sa; + + sa.sa_sigaction = segv_handler; + if (sigaction(SIGSEGV, &sa, NULL)) + return 1; + sa.sa_flags = SA_SIGINFO; + + segv_triggered = false; + + /* Make sure segv_triggered is set before violate_ss() */ + asm volatile("" : : : "memory"); + + violate_ss(); + + signal(SIGSEGV, SIG_DFL); + + printf("[OK]\tShadow stack violation test\n"); + + return !segv_triggered; +} + +/* Gup test state */ +#define MAGIC_VAL 0x12345678 +bool is_shstk_access; +void *shstk_ptr; +int fd; + +void reset_test_shstk(void *addr) +{ + if (shstk_ptr) + free_shstk(shstk_ptr); + shstk_ptr = create_shstk(addr); +} + +void test_access_fix_handler(int signum, siginfo_t *si, void *uc) +{ + printf("[INFO]\tViolation from %s\n", is_shstk_access ? "shstk access" : "normal write"); + + segv_triggered = true; + + /* Fix shadow stack */ + if (is_shstk_access) { + reset_test_shstk(shstk_ptr); + return; + } + + free_shstk(shstk_ptr); + create_normal_mem(shstk_ptr); +} + +bool test_shstk_access(void *ptr) +{ + is_shstk_access = true; + segv_triggered = false; + write_shstk(ptr, MAGIC_VAL); + + asm volatile("" : : : "memory"); + + return segv_triggered; +} + +bool test_write_access(void *ptr) +{ + is_shstk_access = false; + segv_triggered = false; + *(unsigned long *)ptr = MAGIC_VAL; + + asm volatile("" : : : "memory"); + + return segv_triggered; +} + +bool gup_write(void *ptr) +{ + unsigned long val; + + lseek(fd, (unsigned long)ptr, SEEK_SET); + if (write(fd, &val, sizeof(val)) < 0) + return 1; + + return 0; +} + +bool gup_read(void *ptr) +{ + unsigned long val; + + lseek(fd, (unsigned long)ptr, SEEK_SET); + if (read(fd, &val, sizeof(val)) < 0) + return 1; + + return 0; +} + +int test_gup(void) +{ + struct sigaction sa; + int status; + pid_t pid; + + sa.sa_sigaction = test_access_fix_handler; + if (sigaction(SIGSEGV, &sa, NULL)) + return 1; + sa.sa_flags = SA_SIGINFO; + + segv_triggered = false; + + fd = open("/proc/self/mem", O_RDWR); + if (fd == -1) + return 1; + + reset_test_shstk(0); + if (gup_read(shstk_ptr)) + return 1; + if (test_shstk_access(shstk_ptr)) + return 1; + printf("[INFO]\tGup read -> shstk access success\n"); + + reset_test_shstk(0); + if (gup_write(shstk_ptr)) + return 1; + if (test_shstk_access(shstk_ptr)) + return 1; + printf("[INFO]\tGup write -> shstk access success\n"); + + reset_test_shstk(0); + if (gup_read(shstk_ptr)) + return 1; + if (!test_write_access(shstk_ptr)) + return 1; + printf("[INFO]\tGup read -> write access success\n"); + + reset_test_shstk(0); + if (gup_write(shstk_ptr)) + return 1; + if (!test_write_access(shstk_ptr)) + return 1; + printf("[INFO]\tGup write -> write access success\n"); + + close(fd); + + /* COW/gup test */ + reset_test_shstk(0); + pid = fork(); + if (!pid) { + fd = open("/proc/self/mem", O_RDWR); + if (fd == -1) + exit(1); + + if (gup_write(shstk_ptr)) { + close(fd); + exit(1); + } + close(fd); + exit(0); + } + waitpid(pid, &status, 0); + if (WEXITSTATUS(status)) { + printf("[FAIL]\tWrite in child failed\n"); + return 1; + } + if (*(unsigned long *)shstk_ptr == MAGIC_VAL) { + printf("[FAIL]\tWrite in child wrote through to shared memory\n"); + return 1; + } + + printf("[INFO]\tCow gup write -> write access success\n"); + + free_shstk(shstk_ptr); + + signal(SIGSEGV, SIG_DFL); + + printf("[OK]\tShadow gup test\n"); + + return 0; +} + +int test_mprotect(void) +{ + struct sigaction sa; + + sa.sa_sigaction = test_access_fix_handler; + if (sigaction(SIGSEGV, &sa, NULL)) + return 1; + sa.sa_flags = SA_SIGINFO; + + segv_triggered = false; + + /* mprotect a shadow stack as read only */ + reset_test_shstk(0); + if (mprotect(shstk_ptr, SS_SIZE, PROT_READ) < 0) { + printf("[FAIL]\tmprotect(PROT_READ) failed\n"); + return 1; + } + + /* try to wrss it and fail */ + if (!test_shstk_access(shstk_ptr)) { + printf("[FAIL]\tShadow stack access to read-only memory succeeded\n"); + return 1; + } + + /* + * The shadow stack was reset above to resolve the fault, make the new one + * read-only. + */ + if (mprotect(shstk_ptr, SS_SIZE, PROT_READ) < 0) { + printf("[FAIL]\tmprotect(PROT_READ) failed\n"); + return 1; + } + + /* then back to writable */ + if (mprotect(shstk_ptr, SS_SIZE, PROT_WRITE | PROT_READ) < 0) { + printf("[FAIL]\tmprotect(PROT_WRITE) failed\n"); + return 1; + } + + /* then wrss to it and succeed */ + if (test_shstk_access(shstk_ptr)) { + printf("[FAIL]\tShadow stack access to mprotect() writable memory failed\n"); + return 1; + } + + free_shstk(shstk_ptr); + + signal(SIGSEGV, SIG_DFL); + + printf("[OK]\tmprotect() test\n"); + + return 0; +} + +char zero[4096]; + +static void *uffd_thread(void *arg) +{ + struct uffdio_copy req; + int uffd = *(int *)arg; + struct uffd_msg msg; + + if (read(uffd, &msg, sizeof(msg)) <= 0) + return (void *)1; + + req.dst = msg.arg.pagefault.address; + req.src = (__u64)zero; + req.len = 4096; + req.mode = 0; + + if (ioctl(uffd, UFFDIO_COPY, &req)) + return (void *)1; + + return (void *)0; +} + +int test_userfaultfd(void) +{ + struct uffdio_register uffdio_register; + struct uffdio_api uffdio_api; + struct sigaction sa; + pthread_t thread; + void *res; + int uffd; + + sa.sa_sigaction = test_access_fix_handler; + if (sigaction(SIGSEGV, &sa, NULL)) + return 1; + sa.sa_flags = SA_SIGINFO; + + uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK); + if (uffd < 0) { + printf("[SKIP]\tUserfaultfd unavailable.\n"); + return 0; + } + + reset_test_shstk(0); + + uffdio_api.api = UFFD_API; + uffdio_api.features = 0; + if (ioctl(uffd, UFFDIO_API, &uffdio_api)) + goto err; + + uffdio_register.range.start = (__u64)shstk_ptr; + uffdio_register.range.len = 4096; + uffdio_register.mode = UFFDIO_REGISTER_MODE_MISSING; + if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register)) + goto err; + + if (pthread_create(&thread, NULL, &uffd_thread, &uffd)) + goto err; + + reset_shstk(shstk_ptr); + test_shstk_access(shstk_ptr); + + if (pthread_join(thread, &res)) + goto err; + + if (test_shstk_access(shstk_ptr)) + goto err; + + free_shstk(shstk_ptr); + + signal(SIGSEGV, SIG_DFL); + + if (!res) + printf("[OK]\tUserfaultfd test\n"); + return !!res; +err: + free_shstk(shstk_ptr); + close(uffd); + signal(SIGSEGV, SIG_DFL); + return 1; +} + +/* + * Too complicated to pull it out of the 32 bit header, but also get the + * 64 bit one needed above. Just define a copy here. + */ +#define __NR_compat_sigaction 67 + +/* + * Call 32 bit signal handler to get 32 bit signals ABI. Make sure + * to push the registers that will get clobbered. + */ +int sigaction32(int signum, const struct sigaction *restrict act, + struct sigaction *restrict oldact) +{ + register long syscall_reg asm("eax") = __NR_compat_sigaction; + register long signum_reg asm("ebx") = signum; + register long act_reg asm("ecx") = (long)act; + register long oldact_reg asm("edx") = (long)oldact; + int ret = 0; + + asm volatile ("int $0x80;" + : "=a"(ret), "=m"(oldact) + : "r"(syscall_reg), "r"(signum_reg), "r"(act_reg), + "r"(oldact_reg) + : "r8", "r9", "r10", "r11" + ); + + return ret; +} + +sigjmp_buf jmp_buffer; + +void segv_gp_handler(int signum, siginfo_t *si, void *uc) +{ + segv_triggered = true; + + /* + * To work with old glibc, this can't rely on siglongjmp working with + * shadow stack enabled, so disable shadow stack before siglongjmp(). + */ + ARCH_PRCTL(ARCH_SHSTK_DISABLE, ARCH_SHSTK_SHSTK); + siglongjmp(jmp_buffer, -1); +} + +/* + * Transition to 32 bit mode and check that a #GP triggers a segfault. + */ +int test_32bit(void) +{ + struct sigaction sa; + struct sigaction *sa32; + + /* Create sigaction in 32 bit address range */ + sa32 = mmap(0, 4096, PROT_READ | PROT_WRITE, + MAP_32BIT | MAP_PRIVATE | MAP_ANONYMOUS, 0, 0); + sa32->sa_flags = SA_SIGINFO; + + sa.sa_sigaction = segv_gp_handler; + if (sigaction(SIGSEGV, &sa, NULL)) + return 1; + sa.sa_flags = SA_SIGINFO; + + segv_triggered = false; + + /* Make sure segv_triggered is set before triggering the #GP */ + asm volatile("" : : : "memory"); + + /* + * Set handler to somewhere in 32 bit address space + */ + sa32->sa_handler = (void *)sa32; + if (sigaction32(SIGUSR1, sa32, NULL)) + return 1; + + if (!sigsetjmp(jmp_buffer, 1)) + raise(SIGUSR1); + + if (segv_triggered) + printf("[OK]\t32 bit test\n"); + + return !segv_triggered; +} + +int main(int argc, char *argv[]) +{ + int ret = 0; + + if (ARCH_PRCTL(ARCH_SHSTK_ENABLE, ARCH_SHSTK_SHSTK)) { + printf("[SKIP]\tCould not enable Shadow stack\n"); + return 1; + } + + if (ARCH_PRCTL(ARCH_SHSTK_DISABLE, ARCH_SHSTK_SHSTK)) { + ret = 1; + printf("[FAIL]\tDisabling shadow stack failed\n"); + } + + if (ARCH_PRCTL(ARCH_SHSTK_ENABLE, ARCH_SHSTK_SHSTK)) { + printf("[SKIP]\tCould not re-enable Shadow stack\n"); + return 1; + } + + if (ARCH_PRCTL(ARCH_SHSTK_ENABLE, ARCH_SHSTK_WRSS)) { + printf("[SKIP]\tCould not enable WRSS\n"); + ret = 1; + goto out; + } + + /* Should have succeeded if here, but this is a test, so double check. */ + if (!get_ssp()) { + printf("[FAIL]\tShadow stack disabled\n"); + return 1; + } + + if (test_shstk_pivot()) { + ret = 1; + printf("[FAIL]\tShadow stack pivot\n"); + goto out; + } + + if (test_shstk_faults()) { + ret = 1; + printf("[FAIL]\tShadow stack fault test\n"); + goto out; + } + + if (test_shstk_violation()) { + ret = 1; + printf("[FAIL]\tShadow stack violation test\n"); + goto out; + } + + if (test_gup()) { + ret = 1; + printf("[FAIL]\tShadow shadow stack gup\n"); + goto out; + } + + if (test_mprotect()) { + ret = 1; + printf("[FAIL]\tShadow shadow mprotect test\n"); + goto out; + } + + if (test_userfaultfd()) { + ret = 1; + printf("[FAIL]\tUserfaultfd test\n"); + goto out; + } + + if (test_32bit()) { + ret = 1; + printf("[FAIL]\t32 bit test\n"); + } + + return ret; + +out: + /* + * Disable shadow stack before the function returns, or there will be a + * shadow stack violation. + */ + if (ARCH_PRCTL(ARCH_SHSTK_DISABLE, ARCH_SHSTK_SHSTK)) { + ret = 1; + printf("[FAIL]\tDisabling shadow stack failed\n"); + } + + return ret; +} +#endif From patchwork Mon Feb 27 22:29:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154268 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C456C87FDE for ; Mon, 27 Feb 2023 22:32:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2CB936B00A9; Mon, 27 Feb 2023 17:32:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 278666B00AA; Mon, 27 Feb 2023 17:32:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 055286B00AC; Mon, 27 Feb 2023 17:32:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E33596B00AA for ; Mon, 27 Feb 2023 17:32:02 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C4D6F1A08EE for ; Mon, 27 Feb 2023 22:32:02 +0000 (UTC) X-FDA: 80514520884.21.9F04559 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf07.hostedemail.com (Postfix) with ESMTP id CE6F540013 for ; Mon, 27 Feb 2023 22:32:00 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="Iw/fzHL4"; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537121; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=U7hvfVf5QRzvGljNI8gpRe6+fyv2kVLnLmpYI7wD5/Q=; b=nAOViyi9tICF4ew+s1Nhc/FIbrGD3eJZfagyaLllEaGnploG1uxIlJ74jwSum8mRQwHR+K DifJ4WrJ512+SGiIrlhmNQECaBBnm+U9p0Qh7o+ISWNNcam3O/9PwuxzvCKnAWCxJEfNbq DBKTYr1U/W/s9mYDCtQqatQjcH7iFB4= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="Iw/fzHL4"; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537121; a=rsa-sha256; cv=none; b=jo21rhveRTv8dPPFo1bOptEOOkLre6yxVK9mnjAk5egNnR17F+tkeHb9PhowH4Q3BCExeS qRpWMcHsktOp9Qj4rixva3ohS34hMqZ84tpKEV0PB4Gu0d2bIqHUgR/hpj/p51MH8rTdS9 3q4pvuAFr0tnvUtxMg7uK+owL7O3Mv4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537120; x=1709073120; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=AW2ARGp5U2MHiLvxwUeTShQ2Nof5LT8uoV2neNfX8mg=; b=Iw/fzHL4HzJeUj0+tjDFN+f2yxbzsXg/j2Wr0U+2++sdi7WV5emKEfML 0THQXbdAbkp0qmlWkHzw4O1LsGuSXhhNPI0BlubhceFsj5LJbTyUzuunK ZfDiPdNSgAACATATgn15CdosfK1TaZSkwlhCVkOY2s1YMXo8w6mM4mQLt oAryYDgCMum8eiH4RQcD3wv1CMzMZIX6vmo0NhHfQ0QZLM6jGin2KAY78 rNNdtKKSYFjK01TejGRoZcvBkdjwW6eMP9Qi5jZa2jLSwYB1XIzAF6F1B Eq8PyMcA6FHrNB7uHmMOQHKkPAKLtDs0ke42sYJhUqXjacBh8OD9vUpws w==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657939" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657939" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:36 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024808" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024808" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:35 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v7 38/41] x86/fpu: Add helper for initing features Date: Mon, 27 Feb 2023 14:29:54 -0800 Message-Id: <20230227222957.24501-39-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: CE6F540013 X-Stat-Signature: xqhhijscpkkijqrnckomqwzfbuiy4a7j X-HE-Tag: 1677537120-853737 X-HE-Meta: U2FsdGVkX1/HVeMHmfj6I91MuY3WHa7wPhJ4Jz6KaPGSPJ1sxde2c5W466Pz/oJ4zyuEPADJPu4L6f4O6rJoKZeRCHmgKm0eOdBTcwNddp6d2mHiGKvi8ujN8wFcE48EKTa35XbnWwqMD6XyjrcIj/aq9LcVfwkGC2Lcy2O8sFR5OpZFdcttA7Tyw/+xHjbTdems9c8U5oFkbFXtAgI7R74e8coNm4uM807VhckT7QMLfzahD9XIj2hq8friFXqmzsszhU4veJdixbCjHdvBtKEA0Kp4/XDKG+TzXIDqUuk3KIga0QU4WIiFtMfWM5o5GRI1AkTWDOuTfM8rAUeA29aaJx/GGuhOmnz8nfwjf4oIltgkGngE3G4z5WPFMDlUgvG9qPkMGWflZXP8HHG7EIDIrzibVmGmFWfeP+1JxwZ4u1FDQiY5Rzmsl8lAvY6jeahvyzAdEG/62bmJr4m7KVVU6efcatr7+s1tCdb6tRFQcB9qg+Zvid6k3mBGo4evjYSn9KZQsIs30h71fsYwK+d/QNvgp3PJyD+16buzXRnHhGDIFZMF4pX38/MlrpiGjU1cb8YER7JOtZSfFfN/rlBzUttpuoQXqlh0mklpymGa17xq1dc5O+e93dffuabHHjYIL2/f2pY9pc0eGDWQNGW2u8aHIY7G9+/4vO9icV//p9bViumtMSXPPJjMVXgDxDrPprDmm7Wy/cGiNqSJxecrt5fLk8fD9jOgOn8fGww0CjwrD/BfUm4RJG79z1zN7yehDraKF3iimJ6s7u2IHUUGe9s679iqBwKXV4wM3zVdpS+VJtFY1xEkrSjnD9OZ6ESemtI+/4Z8Q3xAkCxHO/roBdy1ZI9makbjITgHvAHERXn04cASflpG7Sn4Py/v5VcLe5ssFWgPXoz7aPCEUzcSHVK3Mz1f1RDoncLNJBgkTUn3y1ioryzNaA4Mc/EVd7wdDgoTeg/m4NIViXA Px6X+dgB mfKNp954uw33wqsAn/PGolUh12nDFqqVLpuJwc0bgLKDzF19cKhJhjMmAO2lSAu0t1ah0yu75cHJzNQmXs5tiyqu6c8yYVtUaQ+j335EOQeLSMuO4fvmhKYjP51L+Ijq7ykona3eU6GT3k7bWvQeWNvJGvM9mslP4y9w7iNeQb0GlAhbqNPLQ36Wx9LbYuYwLOZTgHrxWtF8PEEEpy7gK7/43o8dW5+fhM+N4PKoIn09FrFfToL/7eBwRqhwhiTmjg0qnsiSyp93c+jru/BH5DXKVOQSW4wAscFufskIaV7N1oRtMV9p9d0vzHgMFIIat7AgT838YqX8mUjZSZ+x7ikuYOnF6Cv2t5S3/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If an xfeature is saved in a buffer, the xfeature's bit will be set in xsave->header.xfeatures. The CPU may opt to not save the xfeature if it is in it's init state. In this case the xfeature buffer address cannot be retrieved with get_xsave_addr(). Future patches will need to handle the case of writing to an xfeature that may not be saved. So provide helpers to init an xfeature in an xsave buffer. This could of course be done directly by reaching into the xsave buffer, however this would not be robust against future changes to optimize the xsave buffer by compacting it. In that case the xsave buffer would need to be re-arranged as well. So the logic properly belongs encapsulated in a helper where the logic can be unified. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Rick Edgecombe --- v2: - New patch --- arch/x86/kernel/fpu/xstate.c | 58 +++++++++++++++++++++++++++++------- arch/x86/kernel/fpu/xstate.h | 6 ++++ 2 files changed, 53 insertions(+), 11 deletions(-) diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 13a80521dd51..3ff80be0a441 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -934,6 +934,24 @@ static void *__raw_xsave_addr(struct xregs_state *xsave, int xfeature_nr) return (void *)xsave + xfeature_get_offset(xcomp_bv, xfeature_nr); } +static int xsave_buffer_access_checks(int xfeature_nr) +{ + /* + * Do we even *have* xsave state? + */ + if (!boot_cpu_has(X86_FEATURE_XSAVE)) + return 1; + + /* + * We should not ever be requesting features that we + * have not enabled. + */ + if (WARN_ON_ONCE(!xfeature_enabled(xfeature_nr))) + return 1; + + return 0; +} + /* * Given the xsave area and a state inside, this function returns the * address of the state. @@ -954,17 +972,7 @@ static void *__raw_xsave_addr(struct xregs_state *xsave, int xfeature_nr) */ void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr) { - /* - * Do we even *have* xsave state? - */ - if (!boot_cpu_has(X86_FEATURE_XSAVE)) - return NULL; - - /* - * We should not ever be requesting features that we - * have not enabled. - */ - if (WARN_ON_ONCE(!xfeature_enabled(xfeature_nr))) + if (xsave_buffer_access_checks(xfeature_nr)) return NULL; /* @@ -984,6 +992,34 @@ void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr) return __raw_xsave_addr(xsave, xfeature_nr); } +/* + * Given the xsave area and a state inside, this function + * initializes an xfeature in the buffer. + * + * get_xsave_addr() will return NULL if the feature bit is + * not present in the header. This function will make it so + * the xfeature buffer address is ready to be retrieved by + * get_xsave_addr(). + * + * Inputs: + * xstate: the thread's storage area for all FPU data + * xfeature_nr: state which is defined in xsave.h (e.g. XFEATURE_FP, + * XFEATURE_SSE, etc...) + * Output: + * 1 if the feature cannot be inited, 0 on success + */ +int init_xfeature(struct xregs_state *xsave, int xfeature_nr) +{ + if (xsave_buffer_access_checks(xfeature_nr)) + return 1; + + /* + * Mark the feature inited. + */ + xsave->header.xfeatures |= BIT_ULL(xfeature_nr); + return 0; +} + #ifdef CONFIG_ARCH_HAS_PKEYS /* diff --git a/arch/x86/kernel/fpu/xstate.h b/arch/x86/kernel/fpu/xstate.h index a4ecb04d8d64..dc06f63063ee 100644 --- a/arch/x86/kernel/fpu/xstate.h +++ b/arch/x86/kernel/fpu/xstate.h @@ -54,6 +54,12 @@ extern void fpu__init_cpu_xstate(void); extern void fpu__init_system_xstate(unsigned int legacy_size); extern void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr); +extern int init_xfeature(struct xregs_state *xsave, int xfeature_nr); + +static inline int xfeature_saved(struct xregs_state *xsave, int xfeature_nr) +{ + return xsave->header.xfeatures & BIT_ULL(xfeature_nr); +} static inline u64 xfeatures_mask_supervisor(void) { From patchwork Mon Feb 27 22:29:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154270 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0288C88CB2 for ; Mon, 27 Feb 2023 22:32:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CDAD46B00AB; Mon, 27 Feb 2023 17:32:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B78366B00AD; Mon, 27 Feb 2023 17:32:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A2B76B00AE; Mon, 27 Feb 2023 17:32:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 884086B00AB for ; Mon, 27 Feb 2023 17:32:03 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4D8DB40A7F for ; Mon, 27 Feb 2023 22:32:03 +0000 (UTC) X-FDA: 80514520926.14.47801E0 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf28.hostedemail.com (Postfix) with ESMTP id 54851C0010 for ; Mon, 27 Feb 2023 22:32:01 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ImIeiC4n; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537121; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=bud3WTrvZ04cWHpJ8882/Hx0ishqCggJIVkuvRwOAFs=; b=RXZPqfOguIX9aTnV/UNHcT/QJ6IOo1ntSM5q/V/gZT+KznAIGsIVvp4objpwEnsKSHjSd/ mGu8roUBWt0nTS1AhnnZ/mNTwO/hxQptmfS/SCGmibjss8QSAS89MKgirVZPxgMzv3yE+K lidjqTN0C34/BgCM6RPgeqS0UacLda4= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ImIeiC4n; spf=pass (imf28.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537121; a=rsa-sha256; cv=none; b=vLXt9zfYRCK0R5VRyMrZBgT4rZ4+vHuLdmUBKSOSMRrAkzetoEEF9YZsYhO2A4GfI6uTnd iEMVos7EK2RKMZT//1dLjw4DflrKEkMGRz1TRPRVutXnmigJ/jnsPdZQ30O2irZGqxpwwV bzvIcUwu7uJU7tWBzRT7+v3inXaJCCk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537121; x=1709073121; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=UFVg3sIbuffnHjOtlNoaz9ZH+gYHjqQgnqMZvI5Wi8Y=; b=ImIeiC4nygVpzWj3ygGYlyJV6wivDkgHbD3B4p2RPHCRMRqeSZ3KHa9f pCKLckoQ8BA96Y2Pxh3At/7VyptjoTa2tC741Xd1lvni94c4Yh0+7OmRS ZSYwun1kJMhD4neR8p28iZvK0mhSXPwpx4aB03hLuhuaE94w7AynM5IWp zxy1l7s7qzmi8M3aImIfny5DRc96HwyNpCBpLyXAxZviVJaDXlr1TRF6T okv1Q7pemLmnyFO2/wnZQztrw+VmMENp+9WuT6VNTMLvGMcd0AfB3aigO Yv42XuVyRiUYvk+AhAzTvJJOdym6vgKtivd2JAKcW5SlijBaoAZGUccSr A==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657958" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657958" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:36 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024813" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024813" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:36 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 39/41] x86: Add PTRACE interface for shadow stack Date: Mon, 27 Feb 2023 14:29:55 -0800 Message-Id: <20230227222957.24501-40-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 54851C0010 X-Stat-Signature: f3bi1rcdyezgwfhkzfpw57yuwqjdm8ts X-Rspam-User: X-HE-Tag: 1677537121-148421 X-HE-Meta: U2FsdGVkX18xGDmOkS4nfkxOcRZHsupfnkL+XdWj6TyjiaaroUbcjZPJbqAOFliABnQZi3xYtPk/UuWEM4riM4Sr24WatwyvZWUMcVHnvMg/NINOwoppdjEKeY27EnAt6Mj53Mo65tql7rXtQX+uBEo76pquW0MyDvf3U39Z1JxwiU7TUkKHbb1IlPQRsWhU7mkEuFz4C0bZa2x92+HTvjl9iTjeOIFbO6h9XEV30w0hgl/Ys6CX9H47e2J3jhtAl8CGYnjSXRY4931viAqhR5nUY9qQG85HorDZrS1gxY2VtgS1Gcy8sJ2KLKTqmJ7zRQgPUMa1A/M5oFhKWrmwMDo2UL54v/Dh166v3tI9duZr03CnXvxVYohF/Wvca8OVUwJorSw7Af1gNzRIWFLCez5pB1vNj8rKN6mPQJNeD0ScFS3//i6LKUNLdrlklwnciHFL7YxTkcux8p7u4Bi/zSvkJk4Hc4MSduT2tvKU4AfIr1jaT36dXoa+x1MI47+BZVgzBKW21ELzsDrqRQFxq/PpYaHpJ3La5hiYvG/on1KE99MehvwErthGqiIIeVWhrCZsRz8tdUsabE1FuHrDPyBf5aAPJXNGRqVljgmDwtLP88e3ewZnzPQks562M/TGmuXAVzXGIXtnYsQOMfxdoZOSfDPe+d6mTSvnoWyEJmQqFozySxtZWgdWtYATaNjjEOZ2tZTXgfLpEMHvglXvUeRtfAX5UA+P7n8kg+ZUY/agoTqVCjLnZ7bdxUdzcjGQ+rYU5kNouN8FJlQitiR8oWNpHwLwYQN9dd/KwA654Nhiq5Fj2UQjbndCEC1xRTVHboibeNSIY48mYctlOOSTl9ik2JVS/qTGBilRmG9GsLD1diu4qJKJt0j5Waz7oUDaaBhL5/s8J+m35rDVnUFl6SmuHl1o2r1Dz4935bSDUxMxJmKgynHob0ZwwKbzKYPyYlj/unaXyvVE0xO99ks n49tcd/0 uyrkdc4oS52e01wsyYu6cH0zMssYkxsJUOLdaA7BbwMoHXi8OTKE8Nqv/Hum15y1qduS9KIw2mDysNNi9BoLymKs54Cd8gLG5/FcMFaVD9IEXqKrW7h5driSYPI9BbckDNtYdvk5656VrATbYoqpygWZS3H88Uq6OnNs1QA5nnW6yKE/E69I9ymzi1JXkcJ66/l/VC6ES9ywqBdN/miczI2n7efolv/C1Ho/8OcyTyAutt2nhRZyExCHHChL/tZH2Ez4fosf6H2gokqHFPtth9fTpaEO0ef5yZSwYMLoAQDX1jaIkD0FDzlivO64zpLNAkw4bYqRWf7wksZrFlyCKVexgVLVnZ6ioL4n+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu Some applications (like GDB) would like to tweak shadow stack state via ptrace. This allows for existing functionality to continue to work for seized shadow stack applications. Provide an regset interface for manipulating the shadow stack pointer (SSP). There is already ptrace functionality for accessing xstate, but this does not include supervisor xfeatures. So there is not a completely clear place for where to put the shadow stack state. Adding it to the user xfeatures regset would complicate that code, as it currently shares logic with signals which should not have supervisor features. Don't add a general supervisor xfeature regset like the user one, because it is better to maintain flexibility for other supervisor xfeatures to define their own interface. For example, an xfeature may decide not to expose all of it's state to userspace, as is actually the case for shadow stack ptrace functionality. A lot of enum values remain to be used, so just put it in dedicated shadow stack regset. The only downside to not having a generic supervisor xfeature regset, is that apps need to be enlightened of any new supervisor xfeature exposed this way (i.e. they can't try to have generic save/restore logic). But maybe that is a good thing, because they have to think through each new xfeature instead of encountering issues when new a new supervisor xfeature was added. By adding a shadow stack regset, it also has the effect of including the shadow stack state in a core dump, which could be useful for debugging. The shadow stack specific xstate includes the SSP, and the shadow stack and WRSS enablement status. Enabling shadow stack or wrss in the kernel involves more than just flipping the bit. The kernel is made aware that it has to do extra things when cloning or handling signals. That logic is triggered off of separate feature enablement state kept in the task struct. So the flipping on HW shadow stack enforcement without notifying the kernel to change its behavior would severely limit what an application could do without crashing, and the results would depend on kernel internal implementation details. There is also no known use for controlling this state via prtace today. So only expose the SSP, which is something that userspace already has indirect control over. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Signed-off-by: Yu-cheng Yu Signed-off-by: Yu-cheng Yu Signed-off-by: Rick Edgecombe Tested-by: Pengfei Xu --- v5: - Check shadow stack enablement status for tracee (rppt) - Fix typo in comment v4: - Make shadow stack only. Reduce to only supporting SSP register, and remove CET references (peterz) - Add comment to not use 0x203, because binutils already looks for it in coredumps. (Christina Schimpe) v3: - Drop dependence on thread.shstk.size, and use thread.features bits - Drop 32 bit support v2: - Check alignment on ssp. - Block IBT bits. - Handle init states instead of returning error. - Add verbose commit log justifying the design. --- arch/x86/include/asm/fpu/regset.h | 7 +-- arch/x86/kernel/fpu/regset.c | 86 +++++++++++++++++++++++++++++++ arch/x86/kernel/ptrace.c | 12 +++++ include/uapi/linux/elf.h | 2 + 4 files changed, 104 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/fpu/regset.h b/arch/x86/include/asm/fpu/regset.h index 4f928d6a367b..697b77e96025 100644 --- a/arch/x86/include/asm/fpu/regset.h +++ b/arch/x86/include/asm/fpu/regset.h @@ -7,11 +7,12 @@ #include -extern user_regset_active_fn regset_fpregs_active, regset_xregset_fpregs_active; +extern user_regset_active_fn regset_fpregs_active, regset_xregset_fpregs_active, + ssp_active; extern user_regset_get2_fn fpregs_get, xfpregs_get, fpregs_soft_get, - xstateregs_get; + xstateregs_get, ssp_get; extern user_regset_set_fn fpregs_set, xfpregs_set, fpregs_soft_set, - xstateregs_set; + xstateregs_set, ssp_set; /* * xstateregs_active == regset_fpregs_active. Please refer to the comment diff --git a/arch/x86/kernel/fpu/regset.c b/arch/x86/kernel/fpu/regset.c index 6d056b68f4ed..c806952d9496 100644 --- a/arch/x86/kernel/fpu/regset.c +++ b/arch/x86/kernel/fpu/regset.c @@ -8,6 +8,7 @@ #include #include #include +#include #include "context.h" #include "internal.h" @@ -174,6 +175,91 @@ int xstateregs_set(struct task_struct *target, const struct user_regset *regset, return ret; } +#ifdef CONFIG_X86_USER_SHADOW_STACK +int ssp_active(struct task_struct *target, const struct user_regset *regset) +{ + if (target->thread.features & ARCH_SHSTK_SHSTK) + return regset->n; + + return 0; +} + +int ssp_get(struct task_struct *target, const struct user_regset *regset, + struct membuf to) +{ + struct fpu *fpu = &target->thread.fpu; + struct cet_user_state *cetregs; + + if (!boot_cpu_has(X86_FEATURE_USER_SHSTK)) + return -ENODEV; + + sync_fpstate(fpu); + cetregs = get_xsave_addr(&fpu->fpstate->regs.xsave, XFEATURE_CET_USER); + if (!cetregs) { + /* + * The registers are the in the init state. The init values for + * these regs are zero, so just zero the output buffer. + */ + membuf_zero(&to, sizeof(cetregs->user_ssp)); + return 0; + } + + return membuf_write(&to, (unsigned long *)&cetregs->user_ssp, + sizeof(cetregs->user_ssp)); +} + +int ssp_set(struct task_struct *target, const struct user_regset *regset, + unsigned int pos, unsigned int count, + const void *kbuf, const void __user *ubuf) +{ + struct fpu *fpu = &target->thread.fpu; + struct xregs_state *xsave = &fpu->fpstate->regs.xsave; + struct cet_user_state *cetregs; + unsigned long user_ssp; + int r; + + if (!boot_cpu_has(X86_FEATURE_USER_SHSTK) || + !ssp_active(target, regset)) + return -ENODEV; + + r = user_regset_copyin(&pos, &count, &kbuf, &ubuf, &user_ssp, 0, -1); + if (r) + return r; + + /* + * Some kernel instructions (IRET, etc) can cause exceptions in the case + * of disallowed CET register values. Just prevent invalid values. + */ + if ((user_ssp >= TASK_SIZE_MAX) || !IS_ALIGNED(user_ssp, 8)) + return -EINVAL; + + fpu_force_restore(fpu); + + /* + * Don't want to init the xfeature until the kernel will definitely + * overwrite it, otherwise if it inits and then fails out, it would + * end up initing it to random data. + */ + if (!xfeature_saved(xsave, XFEATURE_CET_USER) && + WARN_ON(init_xfeature(xsave, XFEATURE_CET_USER))) + return -ENODEV; + + cetregs = get_xsave_addr(xsave, XFEATURE_CET_USER); + if (WARN_ON(!cetregs)) { + /* + * This shouldn't ever be NULL because it was successfully + * inited above if needed. The only scenario would be if an + * xfeature was somehow saved in a buffer, but not enabled in + * xsave. + */ + return -ENODEV; + } + + cetregs->user_ssp = user_ssp; + return 0; +} +#endif /* CONFIG_X86_USER_SHADOW_STACK */ + #if defined CONFIG_X86_32 || defined CONFIG_IA32_EMULATION /* diff --git a/arch/x86/kernel/ptrace.c b/arch/x86/kernel/ptrace.c index dfaa270a7cc9..095f04bdabdc 100644 --- a/arch/x86/kernel/ptrace.c +++ b/arch/x86/kernel/ptrace.c @@ -58,6 +58,7 @@ enum x86_regset_64 { REGSET64_FP, REGSET64_IOPERM, REGSET64_XSTATE, + REGSET64_SSP, }; #define REGSET_GENERAL \ @@ -1267,6 +1268,17 @@ static struct user_regset x86_64_regsets[] __ro_after_init = { .active = ioperm_active, .regset_get = ioperm_get }, +#ifdef CONFIG_X86_USER_SHADOW_STACK + [REGSET64_SSP] = { + .core_note_type = NT_X86_SHSTK, + .n = 1, + .size = sizeof(u64), + .align = sizeof(u64), + .active = ssp_active, + .regset_get = ssp_get, + .set = ssp_set + }, +#endif }; static const struct user_regset_view user_x86_64_view = { diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h index 68de6f4c4eee..103a1f2da86e 100644 --- a/include/uapi/linux/elf.h +++ b/include/uapi/linux/elf.h @@ -406,6 +406,8 @@ typedef struct elf64_shdr { #define NT_386_TLS 0x200 /* i386 TLS slots (struct user_desc) */ #define NT_386_IOPERM 0x201 /* x86 io permission bitmap (1=deny) */ #define NT_X86_XSTATE 0x202 /* x86 extended state using xsave */ +/* Old binutils treats 0x203 as a CET state */ +#define NT_X86_SHSTK 0x204 /* x86 SHSTK state */ #define NT_S390_HIGH_GPRS 0x300 /* s390 upper register halves */ #define NT_S390_TIMER 0x301 /* s390 timer register */ #define NT_S390_TODCMP 0x302 /* s390 TOD clock comparator register */ From patchwork Mon Feb 27 22:29:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154271 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4E26CA9EA0 for ; Mon, 27 Feb 2023 22:32:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C6D36B00AE; Mon, 27 Feb 2023 17:32:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 229DB6B00AF; Mon, 27 Feb 2023 17:32:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C9066B00B0; Mon, 27 Feb 2023 17:32:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E6DFC6B00AE for ; Mon, 27 Feb 2023 17:32:04 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C5AEC1C5AD0 for ; Mon, 27 Feb 2023 22:32:04 +0000 (UTC) X-FDA: 80514520968.01.250CD6F Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf24.hostedemail.com (Postfix) with ESMTP id BCC9B180009 for ; Mon, 27 Feb 2023 22:32:02 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=eyOA9yPc; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537123; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=njwT8/nseUUwtQUAH3WeQoFGEFftjwWBmHlQjdHCBPs=; b=qZTg7n1DWczbnn611gO7ghDP694czpBf+YeBGvVZsAqYhkuw+yiYi84a0SLzGjgjUQXvYB lJzUUoddSz/bKH12e2IZ2N+q3sBk+FTOIuWQuSuNrUci/Vt+WHshrvEJ34k8ErwqkhaI8E TuChGJrCvKGueGSvIQg5v8JELzfvd2o= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=eyOA9yPc; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537123; a=rsa-sha256; cv=none; b=amp4hdtb2wSPTyqph14IAxFSiM9fHZYdtSdex00/zFUl5njWUR4z/59HSgoYHM2fRLHehf uteLHxcEC1FynGNxcYs6t4pDCP62Gx1eQiJSPbvk8Ojmo0V7W4HLkYOUckE63S2fExAKwl 7ATWOfCrNzbr0nqPFXxVsztBTuoYI3I= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537122; x=1709073122; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=ND8Cme6dUKUNgrJvq4rzFWKNWp2gSll9XobN53PYr6I=; b=eyOA9yPcLqW8nbUF4vQARdTUev+INKPKdzAidptpl04iC4YXbfDIfvld vCcCz4FAssyIbzUSQtcfROezZn2h9FC98Luj5EN+Uz2E6s5B7oRZEIIV2 xRhKUPVu+9UCl+RF4cV06+7a04WBDKRo1cRzMWwB8qWenMps940kuhnIl iW9mvZUEeQWM1VaWIvptFghs2k39YEGQV5sfT/Osoz1q4zrRHdcyGMS+L bsSHE87B9b+Sa0UB5psiqVvRpUoTKwsHP255R8p2xCRHe8tCFBezNR3QA xYPrxVjcTzj2xOCbyUSAysgAhUINDSDXv7ZgONwDw0qpJgaPfOKxc+7nY A==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657983" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657983" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:37 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024819" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024819" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:36 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Mike Rapoport Subject: [PATCH v7 40/41] x86/shstk: Add ARCH_SHSTK_UNLOCK Date: Mon, 27 Feb 2023 14:29:56 -0800 Message-Id: <20230227222957.24501-41-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Queue-Id: BCC9B180009 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: fpnu1sir7hn193k76d5acjkegpkzgqxw X-HE-Tag: 1677537122-608467 X-HE-Meta: U2FsdGVkX18NodDSIlpZ8jk1ByaZ1vNwxzWzAZx0OsdRWFKrG6khNrXnbsQefV/cHy9tmCsKoO5T7SnPwAM7bqTMG7eU8Ym8+s5uqf+n1itrSHfEjvH1xQKTfSiSWsXH0tO8UD727EJr2mQQ4Lj6tsUn/FOD9JWtIlOs+ZEA19eLVLS1ZIJcFltRdEsz5j9yuiSC+astwqKOH0hTnEDOaVF2muDyjqg57eENz8SOGzwZLyPFDJrC9oQk0ufgKbSPI4VbXjjJJ7OxjiS+wBzV4SgWEhyCCXOWqI8V0CthLx7DS1Uvv0I3vGkMxYXBwiXj2fZX2jcxLJ6l+xEnc6MXwDPeAvUtIMNh2kraww2uN9jRBytCTUzMQZ5+tSpywWU4w/upEgmhBL0xFtGjAzfsaBIR6boCO8ErzLRK5yWLpvUngB1TIYTaMaLLrr1wtnenA4z94tiu0Ji+SMAFBrd/wS5H1PUPNQ9VfePieGSSGL/eKlungf4mH/YjBnJIoqG6Ns1CxKXX0sgJzOnjdpPtXbzQlcZwqaW85rJLWxZ03Nrpw4kBnljE0d/EWsmd8XRuNhYRsbIbIcrySCPJwIaVKLPzGhV4pU9sFppjHygECWO8tUrTBD1tg24B7YWoRlSmwdMu45Ai0HqqLCIXfrAtITGcGW+AD7vRzlS9XPeQvLbv+K/PRkfzI8/PTEDZ0b5p5L2sKsl3EZ1eCJgw5pbkJdz45TER2gRmjjlN8w0OK0MGU+y51S0UvDfxyDwTtJ3Lj1kREy/YW/mcOS7a9uWBhuNxOQxF+Fhn0XUyb6EL/m6qWhZkBqFzTCA7J2I+PpIiPkZDVE73+hEy2fXzdrjc944wVINc1iWRGOCwUwchgspvRxDrS/UhqkeGhV4HTuusH/TPmxuJPc1Xypp2zoehJJnoGp5Dzd6ReISNDkTQXH9u45WhSJc/z77Te6amCmWlYyQBoGecIQVHS7trpXt TmAKlFJJ Pv/nzQ3B5aKmRPQK/de2kqs6pbsxLydEmWr0DFiR2aOusgRS8lMIVYrZl1yPESzRR4duefIv/P3fbszdDVY/kNiJhPW/gS00uDwT7sHjeBIC3hq1RjALAs5m5+YYq+enkFtb12y2MUAPVZTqa3/jyDE/XZ9u3bxYh5uNTFyqsaNd4ImvetfwfnbKKh45jqkNiExlijpPW9jH8yph0sSaB9NqApTZXfR4FIPMmybk0voFFRzEfNnIzFSBeFOrZGxsmbK+CxuEhfcYq7JuXAGLMBeS/XGs7kUxONRGvj9sE/C3tZ4p2f0D/1+KzZFtGIt7iGa0wESYGeUg3HG9nZ9GlGvpEPa48P0kudIB9SpEoVEsEisq3cqi/E1UBKuojrURigVcg24jgj6uF+GjjDZzMDtvt1APejEXKItuy X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mike Rapoport Userspace loaders may lock features before a CRIU restore operation has the chance to set them to whatever state is required by the process being restored. Allow a way for CRIU to unlock features. Add it as an arch_prctl() like the other shadow stack operations, but restrict it being called by the ptrace arch_pctl() interface. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Mike Rapoport [Merged into recent API changes, added commit log and docs] Signed-off-by: Rick Edgecombe --- v4: - Add to docs that it is ptrace only. - Remove "CET" references v3: - Depend on CONFIG_CHECKPOINT_RESTORE (Kees) --- Documentation/x86/shstk.rst | 4 ++++ arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/process_64.c | 1 + arch/x86/kernel/shstk.c | 9 +++++++-- 4 files changed, 13 insertions(+), 2 deletions(-) diff --git a/Documentation/x86/shstk.rst b/Documentation/x86/shstk.rst index f2e6f323cf68..e8ed5fc0f7ae 100644 --- a/Documentation/x86/shstk.rst +++ b/Documentation/x86/shstk.rst @@ -73,6 +73,10 @@ arch_prctl(ARCH_SHSTK_LOCK, unsigned long features) are ignored. The mask is ORed with the existing value. So any feature bits set here cannot be enabled or disabled afterwards. +arch_prctl(ARCH_SHSTK_UNLOCK, unsigned long features) + Unlock features. 'features' is a mask of all features to unlock. All + bits set are processed, unset bits are ignored. Only works via ptrace. + The return values are as follows. On success, return 0. On error, errno can be:: diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index e31495668056..200efbbe5809 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -25,6 +25,7 @@ #define ARCH_SHSTK_ENABLE 0x5001 #define ARCH_SHSTK_DISABLE 0x5002 #define ARCH_SHSTK_LOCK 0x5003 +#define ARCH_SHSTK_UNLOCK 0x5004 /* ARCH_SHSTK_ features bits */ #define ARCH_SHSTK_SHSTK (1ULL << 0) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 71094c8a305f..d368854fa9c4 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -835,6 +835,7 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) case ARCH_SHSTK_ENABLE: case ARCH_SHSTK_DISABLE: case ARCH_SHSTK_LOCK: + case ARCH_SHSTK_UNLOCK: return shstk_prctl(task, option, arg2); default: ret = -EINVAL; diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 2faf9b45ac72..3197ff824809 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -451,9 +451,14 @@ long shstk_prctl(struct task_struct *task, int option, unsigned long features) return 0; } - /* Don't allow via ptrace */ - if (task != current) + /* Only allow via ptrace */ + if (task != current) { + if (option == ARCH_SHSTK_UNLOCK && IS_ENABLED(CONFIG_CHECKPOINT_RESTORE)) { + task->thread.features_locked &= ~features; + return 0; + } return -EINVAL; + } /* Do not allow to change locked features */ if (features & task->thread.features_locked) From patchwork Mon Feb 27 22:29:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13154272 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1427C64ED8 for ; Mon, 27 Feb 2023 22:32:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9CC906B00AF; Mon, 27 Feb 2023 17:32:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 97CE06B00B0; Mon, 27 Feb 2023 17:32:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F5D06B00B1; Mon, 27 Feb 2023 17:32:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 528AF6B00B0 for ; Mon, 27 Feb 2023 17:32:05 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 24493140948 for ; Mon, 27 Feb 2023 22:32:05 +0000 (UTC) X-FDA: 80514521010.01.53A89B1 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf07.hostedemail.com (Postfix) with ESMTP id 269894000C for ; Mon, 27 Feb 2023 22:32:02 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=VJfwSzu0; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537123; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=s3SKx8uA87FhB8rouLjfRqrxDp5yHfGp17WLKTIWOmg=; b=oxcuNAI3VmO+BVvjSKw80VeX3IjBqFXF1mNY7pjZJz/IYnFhzFzxTenNUVabd8vZkQ/YCl B+CxbgGRS4uUPr6x6N0nEFOAvP/fN/0DxQ0dhLPlZ6uj9jbl7F9k+QllR+u/yOV8TVP35x VQI8k4SCwxDJHjFU21MnwWRGuEn99zU= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=VJfwSzu0; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537123; a=rsa-sha256; cv=none; b=mHC4ucTyjqL1HtWtX9dq8x2btZVnXZ+nE5uTbngRDzIg58Vtohj9/Q0i+NgfOb3NL0moF2 xXNWOk9izK5ZB3LjK0vjfUeXk3e6IyoYjWLDt1GWNxMTPRkyUxVuQvRKbbv2p8bJGZzT+E xPp/llE5laFYWlgHVXzQpT+d5lEaOuY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537123; x=1709073123; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=dUBKpkq7gq2yOqJjADRSuNiVlevkRQtTmCsetlkcYh0=; b=VJfwSzu0Zj+YbUyeWPqv/F/UsZsFJ2JoPo2f1bso8OLj0/iObvkmuzXr 1KtVLXb3+AntPhYvCrHg9d7FhibyXIeK4OC7YzrQQhotFjoBBTJkUWFDE sNvfeooDFLtAta5Zz9afFcthpRdg67XSGyJjrPTY2EcuMhWDkQaKVQYvt IbKAHHw93HM5JMjof0OZy69+6QYPeexP+wzUbYpLGUjO0L6mH+yc5KSSB d2JkozeUKIh49OO6eI9fa0fUh/TetzztK9ozkNaZwFk5HhLzzEJUDQNtq kH3Bv7coN1e33xshqR1V+Yn26of3TGVif+2v8kAcQi8QT5A+623PAs3uc Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313658007" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313658007" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:38 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024827" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024827" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:37 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v7 41/41] x86/shstk: Add ARCH_SHSTK_STATUS Date: Mon, 27 Feb 2023 14:29:57 -0800 Message-Id: <20230227222957.24501-42-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 269894000C X-Stat-Signature: bt9uft31rhxi7kmzo95k1cfpyokumjwj X-HE-Tag: 1677537122-899982 X-HE-Meta: U2FsdGVkX19MspJwkd+q7i2dUYOC17WqxV60bB3dcB20sbxZEMkGTPOg72MMYLNbk1S6ff4HQlgQLEQlf6l9GCCAxcO5tYeD4Uwd8Q8UpoPvYwz7JS3LlxLnN/2h21JTGGs5i/5s2cYxXKrmEa7F8zu826bRF1R6FEv3taiDoYNP3B76tIpFzYTOxlcWZuvyPqsziM0pSDFJB1cykE8AyK6uLgh1xSuGGdvDDBV96JPIIjbb86Vzfsz8jU8eCUy84ZUZ2N5heP9WhFjqvDzSdcVJvs9BWeXUpxv3RkjjGq8Oo+IlySKi718IimO3oEGwtJy7wbTgTZWP+z/Nvjmzhl+Y2XIAlay7GZdiE7kyMQpIx1lY5UVnpCGVQCPfnla/ohWB/wSwMC8wbAd3GOF54weNJ1mBR88c4jkv+x8m25YwQ8H4A7RyYOnxq1JJekVDwpIKRYzLXSVn9Ymz3I9Ut3h7YvvKkcBz0k6ADJIkGqv1tB9FvqThHOOW58Ymf/13mPkKEY98tXWxUWc9D6x6uwjo3OHwtxUJoWuU4CK/t3IhHhCB9nkxr1ZsKWlzj4V2Fb6CSrQIgv0c4w/W8X9sQTRakYD7DjbR2YcS0FFkE5DPxcdxdlqySgqoEvwsiPmr8STDlVcFDwySRTIzhy3tjG/HMKTcJZQ58BtnOanw0XxZo9Le8Jc3zLDuTjdUFgYYBWLYeqFLd8dk1bYq5otc51qxSRPtopiUV6W44UhO7PR5ORea2Sfv90n4K8SKmKwNOBYEr/fXTAkXt/eH3hWQpQ1UlJyFtJSBFWd0uYcXlx3L+JsUuqurTTZUWdJInm0iwolxsixI5R8ZPvUyCzUj6fY2GaBdwGw8lgvBwX7ntffstfipBXshxsJqWFjWpQGxlevdJld+aBsSxG1KDQX6Jsl40E8xCFuwdMJX2ZJ9nEZCSkaYS129Nim/jeIVCVTsGKldb1+4LS9qNQMNcMj HBB+JkQO t7v7YTKj03La85g+EqI6tWw7wRep8vUd3nZYB5b3fhK3tXawdqmqJFqaKISj5eCsVGBjhb8ULVAWD217teBjr7hlf0j+wDcBk+j6T9mBYqV8vLilL8cxLRX+zxgDjORU5L1AX5DDy3dq7Ahb3dudgsKCB9qq7jDjatcCUtxgTz4ctfzW9EQHGi1uuJGLLQ+X/ybQHzqAe30/RTLKNTiX6colB9Q49UuTVDPCEM61VJ/Kv2s4DoUp13Jxa1xZm7153s0BaQHeenzDHvB2HyXoFhl19YGPthHWUxLsCzqmJdkryp4vUxNA+SJO2CE3OnWZnEvwgWKrsk89auD7Xm560rudZKRHs7c3hrqQzODXPmOgZPfDve+0IBDaEdgAYpuNXtJTW8C6hMiQwi/x37xQ6sN+deg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: CRIU and GDB need to get the current shadow stack and WRSS enablement status. This information is already available via /proc/pid/status, but this is inconvenient for CRIU because it involves parsing the text output in an area of the code where this is difficult. Provide a status arch_prctl(), ARCH_SHSTK_STATUS for retrieving the status. Have arg2 be a userspace address, and make the new arch_prctl simply copy the features out to userspace. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Suggested-by: Mike Rapoport Signed-off-by: Rick Edgecombe --- v5: - Fix typo in commit log v4: - New patch --- Documentation/x86/shstk.rst | 6 ++++++ arch/x86/include/asm/shstk.h | 2 +- arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/process_64.c | 1 + arch/x86/kernel/shstk.c | 8 +++++++- 5 files changed, 16 insertions(+), 2 deletions(-) diff --git a/Documentation/x86/shstk.rst b/Documentation/x86/shstk.rst index e8ed5fc0f7ae..7f4af798794e 100644 --- a/Documentation/x86/shstk.rst +++ b/Documentation/x86/shstk.rst @@ -77,6 +77,11 @@ arch_prctl(ARCH_SHSTK_UNLOCK, unsigned long features) Unlock features. 'features' is a mask of all features to unlock. All bits set are processed, unset bits are ignored. Only works via ptrace. +arch_prctl(ARCH_SHSTK_STATUS, unsigned long addr) + Copy the currently enabled features to the address passed in addr. The + features are described using the bits passed into the others in + 'features'. + The return values are as follows. On success, return 0. On error, errno can be:: @@ -84,6 +89,7 @@ be:: -ENOTSUPP if the feature is not supported by the hardware or kernel. -EINVAL arguments (non existing feature, etc) + -EFAULT if could not copy information back to userspace The feature's bits supported are:: diff --git a/arch/x86/include/asm/shstk.h b/arch/x86/include/asm/shstk.h index acee68d30a07..be9267897211 100644 --- a/arch/x86/include/asm/shstk.h +++ b/arch/x86/include/asm/shstk.h @@ -14,7 +14,7 @@ struct thread_shstk { u64 size; }; -long shstk_prctl(struct task_struct *task, int option, unsigned long features); +long shstk_prctl(struct task_struct *task, int option, unsigned long arg2); void reset_thread_features(void); int shstk_alloc_thread_stack(struct task_struct *p, unsigned long clone_flags, unsigned long stack_size, diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 200efbbe5809..1b85bc876c2d 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -26,6 +26,7 @@ #define ARCH_SHSTK_DISABLE 0x5002 #define ARCH_SHSTK_LOCK 0x5003 #define ARCH_SHSTK_UNLOCK 0x5004 +#define ARCH_SHSTK_STATUS 0x5005 /* ARCH_SHSTK_ features bits */ #define ARCH_SHSTK_SHSTK (1ULL << 0) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index d368854fa9c4..dde43caf196e 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -836,6 +836,7 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) case ARCH_SHSTK_DISABLE: case ARCH_SHSTK_LOCK: case ARCH_SHSTK_UNLOCK: + case ARCH_SHSTK_STATUS: return shstk_prctl(task, option, arg2); default: ret = -EINVAL; diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 3197ff824809..4069d5bbbe8c 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -444,8 +444,14 @@ SYSCALL_DEFINE3(map_shadow_stack, unsigned long, addr, unsigned long, size, unsi return alloc_shstk(addr, aligned_size, size, set_tok); } -long shstk_prctl(struct task_struct *task, int option, unsigned long features) +long shstk_prctl(struct task_struct *task, int option, unsigned long arg2) { + unsigned long features = arg2; + + if (option == ARCH_SHSTK_STATUS) { + return put_user(task->thread.features, (unsigned long __user *)arg2); + } + if (option == ARCH_SHSTK_LOCK) { task->thread.features_locked |= features; return 0;