From patchwork Sun Mar 19 00:15:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 13180147 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FB03C76196 for ; Sun, 19 Mar 2023 00:16:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 11004280011; Sat, 18 Mar 2023 20:16:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 070D4280001; Sat, 18 Mar 2023 20:16:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D921A280011; Sat, 18 Mar 2023 20:16:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C1F61280001 for ; Sat, 18 Mar 2023 20:16:29 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8D9A4160C75 for ; Sun, 19 Mar 2023 00:16:29 +0000 (UTC) X-FDA: 80583731298.23.2C75200 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf21.hostedemail.com (Postfix) with ESMTP id 827611C000F for ; Sun, 19 Mar 2023 00:16:27 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=f+VSTnDm; spf=pass (imf21.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679184987; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=LspeC6VcDZY1g8y8TPAj26UpA/NokIsNWt2kToN9yyA=; b=c7rat+uX1ud7pUampGPTBvVNKE61SIokqOo241KwYicdUvo63mL9JxG08WAWzHEIr7hXyT icXVkBhQWSkYtIx0k/s16ghY4KCMGPfrwIYNh5l8Zc+vXckO4I0PhL5eEmYSjsNr8w3r0S RL3mtRLz4bwnNT9X6QLzwMp01hp1K90= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=f+VSTnDm; spf=pass (imf21.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679184987; a=rsa-sha256; cv=none; b=iQ021Rgco5u5ZESA4N0f2/Ij4vwcRpelhXwHfbFLNtCYHzeJyHQkSbASOk2APYAeaKgVWS rLKSuMQleT8dgw34chBW5I9GE+uF54MdGh4+xOxDpNDG19gRA3RpAJ18gRWRbOtHt+4OKn GrMBz72wgEffwkC/WsAqkryS65L/RJo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679184987; x=1710720987; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=dronpxQi47Q7lStj0Ifv5In98YSzpCoT6jYTioIlyo0=; b=f+VSTnDmBhLo7PpyRoGk3WAb2bn+vHQw8ktfzPXOYCRyPJsVU3ygiUYl GM5mZXIBx70++VOo8Y+YpZPllQVWtUYYz32HW3oYBMT3zV+eKglfLqQ1m tT6PEHMMdB2vmZzXP+vfyPsijzgxXg+kDPvQUiTeu4ih1iUnblItTkJgY uUMfknWk6XXK2v+DvrKtJ4V8574sjycQCbpG2RWgIfIUs5Fw1wmc4XhvP BMaX1DI/val7yFNwCNrcg7+8oPCNyTJDCmsgJOTvvlp194Wlv76XWWdDY 3f/T0zYm/VATByeGa1Hkg9Al5y7zwd7CN140QJeWv3QLvzC0boMPMVpRp Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10653"; a="338491151" X-IronPort-AV: E=Sophos;i="5.98,272,1673942400"; d="scan'208";a="338491151" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2023 17:16:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10653"; a="749672858" X-IronPort-AV: E=Sophos;i="5.98,272,1673942400"; d="scan'208";a="749672858" Received: from bmahatwo-mobl1.gar.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.135.34.5]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2023 17:16:25 -0700 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com, szabolcs.nagy@arm.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v8 19/40] x86/mm: Check shadow stack page fault errors Date: Sat, 18 Mar 2023 17:15:14 -0700 Message-Id: <20230319001535.23210-20-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230319001535.23210-1-rick.p.edgecombe@intel.com> References: <20230319001535.23210-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 827611C000F X-Rspam-User: X-Stat-Signature: bi1w1engend3qkgntq3cibaut7br4zjb X-HE-Tag: 1679184987-207557 X-HE-Meta: U2FsdGVkX1/0WAbe0wchSNe7V+ed3kcCPJkJUeF7yOfiwoEpuaao9RBwS00Qb0u+9p32PBQD4hoJ4BQpeLmpt7KRakbrd86uw6yuF6E8cw8h8OUFlHTH7s6kK+zSoU/WN1GBXH97NScEj6LWtoyJWrMk09j1/IMBKu7d/Ahq9u08K194CyYnnM6zTH7/Cmq/4SeS5SCr9o87k7HczvM2MFS3Ab+p2VZPcivvZOhixWe/Isg086yKwVKOluCtwppo/v4AciRDz6KMViBfi5fIi0uIDcM1jgaeTHIA0NEriQtO8E9qFcjrRD61aitiff7bL1RTY5EH9SEwDDvMzSPQWAqxn4k5leN0OqFXe9GAlV2n8Z0nJksev5EHMr3rO3elv4CRSll+y4NjGHhTulnTWMpCATvbT/ZUZLwRa4k+ZAO1BbAXTGtPnvBaOIl/beYBbh60nj7LxDErwQIMYc6t/ir7VPYiibaYqcjdI5EeNfI2G2iqD3qg4dJl7kr/MEKBA7ji0HJr1KWFuXrY3zqnwbPaZc2hHoay0vOeiOlzo+8JjgG1oEZuTSSPcUE9ruvatshoY3Lczy8eomc8A4BfEhDMY8ARUsZVG48UTTyES60MzhGe1gCAwtjf2cuIwGoIr3K+B+MomCohZ8UoIUFn+ubZw1p5w2JsUjDLPPlDEvQBPz+eU8nyz4XA7FHwyUR2fgggxBt2KURoMgC7PNos+Imlth5dhG9MuTX7xDcQW/IT+m6dlyoVY6vh+LSPxx+wCQNpHkthFC/p17jxBnh02ONd8uI1Ir3YltGFtoBtiZwX1aaQTJstvp/++gF8sTUFGeZiXaXfdlUbgIelTl7E/yz3xDDFQDvqErvvThtX73jTEmQ9Dl4bBZtSgE3OFKgO/PSd+tzWGTGyy9Ci9HwP73Oq6I76qCHtd9OZoof5Wdj57MlaRnmz/7tZwqkjztr69M0ipoRlpwcQnq9dk0+ GUq9VfKA dZTJOJQi/16kpwp8gxnExNb9yb1PaTAMu8eEGYx7XHhpmTX4b2AKEq4I9ECQgLhxQ/sZ1dTu21ETosLAC1JonLQNkmJpx2E1vosur7r9oFfwOFMKv81z3edyQ7H94sbYpgKMv02+TW0thW0D1AoYvVFfwXqmei0x9fQcd2yNSjXMFb2xhh77KfLVRIzWxPFiS3D1+eZsUJFbDLq0ZW6gyLV4XXVL++9dEFU1bpFvMK/7j9cmEfR/xgQeATQ8Rv9RuFjSFjrhhEK0VZbJyNZT1dKuCa353BBMjQX3S64sDdOd539YkbFzDe+1rqfEGe+FrYZgdSQkHs9AZdFOdif8r3MHt/UsWb+sCgL8N X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The CPU performs "shadow stack accesses" when it expects to encounter shadow stack mappings. These accesses can be implicit (via CALL/RET instructions) or explicit (instructions like WRSS). Shadow stack accesses to shadow-stack mappings can result in faults in normal, valid operation just like regular accesses to regular mappings. Shadow stacks need some of the same features like delayed allocation, swap and copy-on-write. The kernel needs to use faults to implement those features. The architecture has concepts of both shadow stack reads and shadow stack writes. Any shadow stack access to non-shadow stack memory will generate a fault with the shadow stack error code bit set. This means that, unlike normal write protection, the fault handler needs to create a type of memory that can be written to (with instructions that generate shadow stack writes), even to fulfill a read access. So in the case of COW memory, the COW needs to take place even with a shadow stack read. Otherwise the page will be left (shadow stack) writable in userspace. So to trigger the appropriate behavior, set FAULT_FLAG_WRITE for shadow stack accesses, even if the access was a shadow stack read. For the purpose of making this clearer, consider the following example. If a process has a shadow stack, and forks, the shadow stack PTEs will become read-only due to COW. If the CPU in one process performs a shadow stack read access to the shadow stack, for example executing a RET and causing the CPU to read the shadow stack copy of the return address, then in order for the fault to be resolved the PTE will need to be set with shadow stack permissions. But then the memory would be changeable from userspace (from CALL, RET, WRSS, etc). So this scenario needs to trigger COW, otherwise the shared page would be changeable from both processes. Shadow stack accesses can also result in errors, such as when a shadow stack overflows, or if a shadow stack access occurs to a non-shadow-stack mapping. Also, generate the errors for invalid shadow stack accesses. Co-developed-by: Yu-cheng Yu Signed-off-by: Yu-cheng Yu Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook Acked-by: Mike Rapoport (IBM) Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook --- v8: - Further tweak commit log (dhansen, Boris) v7: - Update comment in fault handler (David Hildenbrand) v6: - Update comment due to rename of Cow bit to SavedDirty v5: - Add description of COW example (Boris) - Replace "permissioned" (Boris) - Remove capitalization of shadow stack (Boris) --- arch/x86/include/asm/trap_pf.h | 2 ++ arch/x86/mm/fault.c | 22 ++++++++++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/arch/x86/include/asm/trap_pf.h b/arch/x86/include/asm/trap_pf.h index 10b1de500ab1..afa524325e55 100644 --- a/arch/x86/include/asm/trap_pf.h +++ b/arch/x86/include/asm/trap_pf.h @@ -11,6 +11,7 @@ * bit 3 == 1: use of reserved bit detected * bit 4 == 1: fault was an instruction fetch * bit 5 == 1: protection keys block access + * bit 6 == 1: shadow stack access fault * bit 15 == 1: SGX MMU page-fault */ enum x86_pf_error_code { @@ -20,6 +21,7 @@ enum x86_pf_error_code { X86_PF_RSVD = 1 << 3, X86_PF_INSTR = 1 << 4, X86_PF_PK = 1 << 5, + X86_PF_SHSTK = 1 << 6, X86_PF_SGX = 1 << 15, }; diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index a498ae1fbe66..7beb0ba6b2ec 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1117,8 +1117,22 @@ access_error(unsigned long error_code, struct vm_area_struct *vma) (error_code & X86_PF_INSTR), foreign)) return 1; + /* + * Shadow stack accesses (PF_SHSTK=1) are only permitted to + * shadow stack VMAs. All other accesses result in an error. + */ + if (error_code & X86_PF_SHSTK) { + if (unlikely(!(vma->vm_flags & VM_SHADOW_STACK))) + return 1; + if (unlikely(!(vma->vm_flags & VM_WRITE))) + return 1; + return 0; + } + if (error_code & X86_PF_WRITE) { /* write, present and write, not present: */ + if (unlikely(vma->vm_flags & VM_SHADOW_STACK)) + return 1; if (unlikely(!(vma->vm_flags & VM_WRITE))) return 1; return 0; @@ -1310,6 +1324,14 @@ void do_user_addr_fault(struct pt_regs *regs, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address); + /* + * Read-only permissions can not be expressed in shadow stack PTEs. + * Treat all shadow stack accesses as WRITE faults. This ensures + * that the MM will prepare everything (e.g., break COW) such that + * maybe_mkwrite() can create a proper shadow stack PTE. + */ + if (error_code & X86_PF_SHSTK) + flags |= FAULT_FLAG_WRITE; if (error_code & X86_PF_WRITE) flags |= FAULT_FLAG_WRITE; if (error_code & X86_PF_INSTR)