From patchwork Mon Feb 27 22:29:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 13154254 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F2A5C64ED9 for ; Mon, 27 Feb 2023 22:32:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9F5E36B0096; Mon, 27 Feb 2023 17:31:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9CA9A6B0099; Mon, 27 Feb 2023 17:31:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8458C6B009A; Mon, 27 Feb 2023 17:31:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7335E6B0096 for ; Mon, 27 Feb 2023 17:31:51 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3FE79C03CF for ; Mon, 27 Feb 2023 22:31:51 +0000 (UTC) X-FDA: 80514520422.07.7C6CA57 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf24.hostedemail.com (Postfix) with ESMTP id 4FF01180017 for ; Mon, 27 Feb 2023 22:31:49 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=g+DJINxj; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677537109; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=YgqLQ+1vy4zj1LyeIgLZtOZ931vA8iNoFckEDb4u1rM=; b=iOWmyNFWY0FQNfnykULgUt2IqUXI6ycz60RQ9ICjBaPCoxR3QAfq3+Ol0UiUV4LJ/9rue3 UY8PJiba5w4nFBXRt/hjNWQ2NSNvyLYB/7xI9OwybhWkh8BdlveJlXNRxupNqnZev4En2p engEriEKbm+VEpGAlE6omslucAuZjLk= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=g+DJINxj; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677537109; a=rsa-sha256; cv=none; b=MI58fw4ASbPkocnv/w1vz+5HXBdiCgWmLxZhUTjBGCKwy2QP5iZkySpfAkI+OpSV4tRVAR htoYVsiSzgYJs6U7AR8/edIXY0CXzKY8MVfpqcWGZ+VIVjQZlqN98AOJm3R1ewtfy4YJMA kIA3IAsHyzB+ouHZinSCvMUKBunjhxI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677537109; x=1709073109; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=iMZ7pHmGnRJHR2PLdV4imtJgm0cCw3zHhRVdTgGS/n0=; b=g+DJINxjOjxEF6A4JPjXV91iTyxXuYevzdcPfyyX0LorssUd4FWTT2R/ 6OJdRrd+pq58OC6JcTqFT9r2w27Sia81ZQKQ8vlRazFPrGabziKciVtkH JoKl1Dn9+KZi769RR5FBWSgjbuAQ9mv2I1rHsURVtmCoZLpWVLo2j7Rdg 4NXckOWbEiHOpJMgWvMJo3QTJ/n9+g/yvliOlFvl0fhgfs/hAvyrLYj+G UyQfthQBIg9GvdJ5ZRBBA2W9PfHuwh1UElwFnxmZHlEgYhRh8hwuPLcUx fow28QnZDw5a1tq8dTmBIAI6l3zSg9IzB+VsspzdXObrKznimqKIf3UI+ w==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="313657514" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="313657514" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:25 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="848024623" X-IronPort-AV: E=Sophos;i="5.98,220,1673942400"; d="scan'208";a="848024623" Received: from leonqu-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.72.19]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 14:31:22 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v7 21/41] mm: Add guard pages around a shadow stack. Date: Mon, 27 Feb 2023 14:29:37 -0800 Message-Id: <20230227222957.24501-22-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230227222957.24501-1-rick.p.edgecombe@intel.com> References: <20230227222957.24501-1-rick.p.edgecombe@intel.com> X-Rspamd-Queue-Id: 4FF01180017 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 8odb98hqqsqzk9rwryhda57bxek7rew7 X-HE-Tag: 1677537109-261903 X-HE-Meta: U2FsdGVkX19A1SIhePhcrOuMGihZM+2OfXMCXcY4ZPSVwsq/vdYbn+0gulJ8/djp8T+oiWCCOr/PYxmXdo8/myjWGkjd4Jm8yMmMtOs7nwtzVU55rTITAk3sA260UO+rUE8g1Taw2NYhFFiGb1zl2Ww2dyUTiZvqGrLDrPEJ8rGJCZ7wyZgP6pNAbFI4nKSuQfx9GWQZ81b85dU2jUzwQmXXoIcZo+1dYEtrrH1DLyyPJBkc7eEaSrrpvJw5ADgXcFIGW7BuxlyGoz7ejcSMdPIbejfenPOhvPYsg3QTwCfXnFwkzPG45YjA/yIcZTQleykh3hJp1gs+BuLty0adEInf6OCswdj2Zic+BVkzkVtHD2nOFEyA+6mRyfcgyX2K6e4WPSO54CeK+W+kzA0quT3UVLWOIOPcdanS5qXirg0w0eQ+9ybPPLsVCMP30VJNZO2WGKhmtQTF5A4Y+5JhCV2LADrk7US/rJpCmeHQCzxmnchMMkgs2YseocHvEkMO/pWq+jnwoW4ZAZlHfizHi/3yafffRvXg9Pykvwdp/dFoaSua1al3HO/QFB8P0DN/rrsxMCO1CxNk85e7E0WLPbrbTNan3adWIduHTbRwtUAJMy0fx7BPuuqX4oQsgfap4pggTciM1Cnr+iaNfW5xQuZZschf29gs0PQdOGjmIHoi6s8HFwWDB95K1mnCnP5pyRvlsi0uOAT6M/u9SHR9iTU4VjSkInVQDLUgRDRYcjyXaiNFFf5HPMvxE+Uw5/lDuTPjKBRxiDOGyZroE5/RtVWm6orDTkXup0kOvOYNLhb+a9pa3yDW/AHzyiq6GRLxJs9jBBzgu/PKevp9+GfLJ2gaNUtB2oEdAvTefuGtFCiC9XPkvmld9dtOX4GX3UK3QzwZ6Y5V5GJPmHF7Y2pOxM/bZlcZ+7nJTkgpFtVWyrO5kzRAKjGi9sOmdrtcl4vDyFPGPgWXW02djvf5jSd Tm296xYX DmxPbqng4FTzs40SqX8RrPeiKGc0qlVPgADO8EVHAUKnxgbtroMmaFOaQoWigCp31Lc5OoicyY+Md+MdjXWcI1SpNpgVTDlYdQpd+AoE3e3E7JujK2xI82xd/fiL3+hjewTu5BonjrIUWd/25EmEOozUqup2TkojroQ0C1NtEFbbr72o4QiNhIs6HJh4R0+34jdx7+vfmtN13qp3ZBGYKwuF898RxPVNvX274xmJ8vj3ecZ7Yr+An36Ev38PHbe5l3u1wzbT9kVD8sRD/BxKR6hZ3VbaahGNOolHZzqE1r7iQzkqQZZpv8XtY1EgQYErL70bZiXumL0yLN0XVS+9pUk35cN65FyG7eEcV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. The architecture of shadow stack constrains the ability of userspace to move the shadow stack pointer (SSP) in order to prevent corrupting or switching to other shadow stacks. The RSTORSSP can move the ssp to different shadow stacks, but it requires a specially placed token in order to do this. However, the architecture does not prevent incrementing the stack pointer to wander onto an adjacent shadow stack. To prevent this in software, enforce guard pages at the beginning of shadow stack vmas, such that there will always be a gap between adjacent shadow stacks. Make the gap big enough so that no userspace SSP changing operations (besides RSTORSSP), can move the SSP from one stack to the next. The SSP can increment or decrement by CALL, RET and INCSSP. CALL and RET can move the SSP by a maximum of 8 bytes, at which point the shadow stack would be accessed. The INCSSP instruction can also increment the shadow stack pointer. It is the shadow stack analog of an instruction like: addq $0x80, %rsp However, there is one important difference between an ADD on %rsp and INCSSP. In addition to modifying SSP, INCSSP also reads from the memory of the first and last elements that were "popped". It can be thought of as acting like this: READ_ONCE(ssp); // read+discard top element on stack ssp += nr_to_pop * 8; // move the shadow stack READ_ONCE(ssp-8); // read+discard last popped stack element The maximum distance INCSSP can move the SSP is 2040 bytes, before it would read the memory. Therefore a single page gap will be enough to prevent any operation from shifting the SSP to an adjacent stack, since it would have to land in the gap at least once, causing a fault. This could be accomplished by using VM_GROWSDOWN, but this has a downside. The behavior would allow shadow stack's to grow, which is unneeded and adds a strange difference to how most regular stacks work. Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: Mike Rapoport (IBM) Reviewed-by: Kees Cook Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook --- v5: - Fix typo in commit log v4: - Drop references to 32 bit instructions - Switch to generic code to drop __weak (Peterz) v2: - Use __weak instead of #ifdef (Dave Hansen) - Only have start gap on shadow stack (Andy Luto) - Create stack_guard_start_gap() to not duplicate code in an arch version of vm_start_gap() (Dave Hansen) - Improve commit log partly with verbiage from (Dave Hansen) Yu-cheng v25: - Move SHADOW_STACK_GUARD_GAP to arch/x86/mm/mmap.c. --- include/linux/mm.h | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 097544afb1aa..6a093daced88 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3107,15 +3107,36 @@ struct vm_area_struct *vma_lookup(struct mm_struct *mm, unsigned long addr) return mtree_load(&mm->mm_mt, addr); } +static inline unsigned long stack_guard_start_gap(struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_GROWSDOWN) + return stack_guard_gap; + + /* + * Shadow stack pointer is moved by CALL, RET, and INCSSPQ. + * INCSSPQ moves shadow stack pointer up to 255 * 8 = ~2 KB + * and touches the first and the last element in the range, which + * triggers a page fault if the range is not in a shadow stack. + * Because of this, creating 4-KB guard pages around a shadow + * stack prevents these instructions from going beyond. + * + * Creation of VM_SHADOW_STACK is tightly controlled, so a vma + * can't be both VM_GROWSDOWN and VM_SHADOW_STACK + */ + if (vma->vm_flags & VM_SHADOW_STACK) + return PAGE_SIZE; + + return 0; +} + static inline unsigned long vm_start_gap(struct vm_area_struct *vma) { + unsigned long gap = stack_guard_start_gap(vma); unsigned long vm_start = vma->vm_start; - if (vma->vm_flags & VM_GROWSDOWN) { - vm_start -= stack_guard_gap; - if (vm_start > vma->vm_start) - vm_start = 0; - } + vm_start -= gap; + if (vm_start > vma->vm_start) + vm_start = 0; return vm_start; }