From patchwork Thu Jan 19 21:22:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 13108794 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4211C6379F for ; Thu, 19 Jan 2023 21:23:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 26A238E0002; Thu, 19 Jan 2023 16:23:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1CF938E0001; Thu, 19 Jan 2023 16:23:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DDAB88E0002; Thu, 19 Jan 2023 16:23:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C57768E0001 for ; Thu, 19 Jan 2023 16:23:53 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8E6C9120DFE for ; Thu, 19 Jan 2023 21:23:53 +0000 (UTC) X-FDA: 80372825946.08.6D36503 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by imf16.hostedemail.com (Postfix) with ESMTP id 67388180019 for ; Thu, 19 Jan 2023 21:23:51 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="SlJB/YHi"; spf=pass (imf16.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674163431; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=LjZx3/OxatauvlHEC9jJBaxJxgdku9VSDjOhpZOLExo=; b=rxoZCBB9xJlYn0EQKZ+BSwQ1L4s/cM9kYtIdxiVnroSu/2t3NVWG4kXs3yBnqZQJunxaeT kJplU0zJWxI8oW5IZADOaqL5I7eraGysdtvnpzpM0ecs+GBRdeZpa7Y35xo63kRzKba12i rhQzP1uNGxNzNWx/XFvIoW3wNlL60So= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="SlJB/YHi"; spf=pass (imf16.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674163431; a=rsa-sha256; cv=none; b=Czk0PcpqlrRN86+71mrsMf7UE/Ma58AjvcnN3bPx6AY9mJhWrh0UC3hOzVn0xpXe5cei1G y95cKN8eVQcdWYaM3zQQVspAJW186d3r14VDzJBSvrXoQNa04UPUVi8yuaApdBWc0Fz3Vj Ao4C64An72AoUOiw2rbjS0OMiyAPPC0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163431; x=1705699431; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=W0uo/pMYztKK7cUO1RHzIUdsd0CMMqFeZAfln1MTMss=; b=SlJB/YHitVhbEDbmCuYk4n/fO5q6q2iv0Mtbdzy6rD7PmZf+aPxNwCNO Iaqx6gwj1Waa/dJYPG1AC5Y3XhN6vl93g9WTWi2IXt8dSVxklddIn/di9 1SGMv+1oMtite9nk5TLUTPavNkkubCn6DLwOQP3LXr8VKWpNFSqN+TdPp mIYS7PCxjA8/ZXVrZJ5PnywbGHg7AxAG7i1KaXRR9j/KDGpDV2lu4y+H9 +TbQlACtV0d12OH2r8h9sM73dKVEHSF64Rl3QljNw2g9VMqx+UwkVv9ml wFdqILAoFBfKILa0/gBfmtZ6cBMMBEeuIvItcdFW39woQygtLVuapjRnd Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119528" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119528" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:50 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139070" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139070" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:49 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 16/39] x86/mm: Check shadow stack page fault errors Date: Thu, 19 Jan 2023 13:22:54 -0800 Message-Id: <20230119212317.8324-17-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 67388180019 X-Stat-Signature: tyhu7zo5ond7gy6yf7699k44de38pcgr X-Rspam-User: X-HE-Tag: 1674163431-936513 X-HE-Meta: U2FsdGVkX1+2ucrQqCBpgHp01+c21Vh9KU0N302t0t+df1ADKY4Lz0Y2cIlfhksC8iy/q9TbGVdZNacwO7N7SRXfSewut73/os4XJyO4AW0ugIngCsaj0gEcpYqqEBnriuAqHB53iX/owEvBi2KHcHgdCrbRfQRrtiHrcfdGTuEppenc1T1C8m36ION9G5dZAVV+v3JqUSCTcGEK8jwe5br0aoxo0G/G3id+OHRPuXhIrXHC3CMVeGc1feJT1HfX81dkSYonBkwuImOsrmJNUrfnOnMOeTT2wMsJLom0ZtxfFLaV2XkmDf5lqAcst5NS7uNbmmPO78owZ17YJEPvkiAksfrtmcXgiaWkJvy6H6RUddjEa7HppG52jUgMVb2UTQzj+eoGDOrNdYaxcMdCHOxSOC5g7PFqmbuG01tdovgBNpf+NrH8Sm8lpuSk7x7KApu6SdYBPAyEcGKsfnf9CMGZiERqKeWanSy4TnGjsXZrhYjAmdXZPkvyBMRRWVrw0Xn9CzpiCxr9gxrfurpmJq9XMWHCgWNCL0Ozpc2jdpcrBMVv3tlu7dubjaN7/C0G5HR2rB/TsEd4L9h0wKr9x63BTDYvkFBWYZTtrlQXJxpyDFZ4KQRaljsWwFftLLvLU5N1T/ZVQ7ecNRLLZihhoJWrXqpNg77ezglP09WEEy3v0UWOZhMwjHE22jJWp93YdIeSD2Jhdgx/EKzAgNb1jHvtWj6IhvXmReVy0yBXJlW9Zw7G2VYidWZ1/RAJVd8Qwpx7GrkAv85hW/GW8U+wTVkXKTVMW3a1jcpIovwt4iNThzjszs/JSRMoJv1NABZv8LCvKTdEGjQoZG0gIQfY2sK2tv+hfUTdUJUl6g5Vbtv9H0jeJnFWFpQlQDHdbRplYhVnHG9HQFYtG5J7P8SfPMonb5rWuxt6Cf8LTry5GT6pbRJuTRRzp0OG4bkP3pMelViA3468YovV33lyZ+w J2vM6cMu tsmP0lPPszuR6r1xkrkOMJtudEe1tzRZvQIPTAyIFG+2b7dmbh2KepgayAQsNDvE4s7W/EpvxNbP1+mezS53ZW6xfARH8qPMDhgVarA94ah1CoQiB4rkuwWFjamQELowxGQbTBIn3WT37YKWbBeOkMwAEiHCseR5A46ZtyvDPCCU4/olY9iZqXDr8e7OC0W64t0dJZNl9rqhXLpUWvLLGnzVcZkJ0W+Ydb+1yR1bnkN5ztdo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu The CPU performs "shadow stack accesses" when it expects to encounter shadow stack mappings. These accesses can be implicit (via CALL/RET instructions) or explicit (instructions like WRSS). Shadow stack accesses to shadow-stack mappings can result in faults in normal, valid operation just like regular accesses to regular mappings. Shadow stacks need some of the same features like delayed allocation, swap and copy-on-write. The kernel needs to use faults to implement those features. The architecture has concepts of both shadow stack reads and shadow stack writes. Any shadow stack access to non-shadow stack memory will generate a fault with the shadow stack error code bit set. This means that, unlike normal write protection, the fault handler needs to create a type of memory that can be written to (with instructions that generate shadow stack writes), even to fulfill a read access. So in the case of COW memory, the COW needs to take place even with a shadow stack read. Otherwise the page will be left (shadow stack) writable in userspace. So to trigger the appropriate behavior, set FAULT_FLAG_WRITE for shadow stack accesses, even if the access was a shadow stack read. For the purpose of making this clearer, consider the following example. If a process has a shadow stack, and forks, the shadow stack PTEs will become read-only due to COW. If the CPU in one process performs a shadow stack read access to the shadow stack, for example executing a RET and causing the CPU to read the shadow stack copy of the return address, then in order for the fault to be resolved the PTE will need to be set with shadow stack permissions. But then the memory would be changeable from userspace (from CALL, RET, WRSS, etc). So this scenario needs to trigger COW, otherwise the shared page would be changeable from both processes. Shadow stack accesses can also result in errors, such as when a shadow stack overflows, or if a shadow stack access occurs to a non-shadow-stack mapping. Also, generate the errors for invalid shadow stack accesses. Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook --- v5: - Add description of COW example (Boris) - Replace "permissioned" (Boris) - Remove capitalization of shadow stack (Boris) v4: - Further improve comment talking about FAULT_FLAG_WRITE (Peterz) v3: - Improve comment talking about using FAULT_FLAG_WRITE (Peterz) v2: - Update commit log with verbiage/feedback from Dave Hansen - Clarify reasoning for FAULT_FLAG_WRITE for all shadow stack accesses - Update comments with some verbiage from Dave Hansen arch/x86/include/asm/trap_pf.h | 2 ++ arch/x86/mm/fault.c | 38 ++++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/arch/x86/include/asm/trap_pf.h b/arch/x86/include/asm/trap_pf.h index 10b1de500ab1..afa524325e55 100644 --- a/arch/x86/include/asm/trap_pf.h +++ b/arch/x86/include/asm/trap_pf.h @@ -11,6 +11,7 @@ * bit 3 == 1: use of reserved bit detected * bit 4 == 1: fault was an instruction fetch * bit 5 == 1: protection keys block access + * bit 6 == 1: shadow stack access fault * bit 15 == 1: SGX MMU page-fault */ enum x86_pf_error_code { @@ -20,6 +21,7 @@ enum x86_pf_error_code { X86_PF_RSVD = 1 << 3, X86_PF_INSTR = 1 << 4, X86_PF_PK = 1 << 5, + X86_PF_SHSTK = 1 << 6, X86_PF_SGX = 1 << 15, }; diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 7b0d4ab894c8..070b50c87415 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1138,8 +1138,22 @@ access_error(unsigned long error_code, struct vm_area_struct *vma) (error_code & X86_PF_INSTR), foreign)) return 1; + /* + * Shadow stack accesses (PF_SHSTK=1) are only permitted to + * shadow stack VMAs. All other accesses result in an error. + */ + if (error_code & X86_PF_SHSTK) { + if (unlikely(!(vma->vm_flags & VM_SHADOW_STACK))) + return 1; + if (unlikely(!(vma->vm_flags & VM_WRITE))) + return 1; + return 0; + } + if (error_code & X86_PF_WRITE) { /* write, present and write, not present: */ + if (unlikely(vma->vm_flags & VM_SHADOW_STACK)) + return 1; if (unlikely(!(vma->vm_flags & VM_WRITE))) return 1; return 0; @@ -1331,6 +1345,30 @@ void do_user_addr_fault(struct pt_regs *regs, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address); + /* + * When a page becomes COW it changes from a shadow stack permission + * page (Write=0,Dirty=1) to (Write=0,Dirty=0,CoW=1), which is simply + * read-only to the CPU. When shadow stack is enabled, a RET would + * normally pop the shadow stack by reading it with a "shadow stack + * read" access. However, in the COW case the shadow stack memory does + * not have shadow stack permissions, it is read-only. So it will + * generate a fault. + * + * For conventionally writable pages, a read can be serviced with a + * read only PTE, and COW would not have to happen. But for shadow + * stack, there isn't the concept of read-only shadow stack memory. + * If it is shadow stack permission, it can be modified via CALL and + * RET instructions. So COW needs to happen before any memory can be + * mapped with shadow stack permissions. + * + * Shadow stack accesses (read or write) need to be serviced with + * shadow stack permission memory, so in the case of a shadow stack + * read access, treat it as a WRITE fault so both COW will happen and + * the write fault path will tickle maybe_mkwrite() and map the memory + * shadow stack. + */ + if (error_code & X86_PF_SHSTK) + flags |= FAULT_FLAG_WRITE; if (error_code & X86_PF_WRITE) flags |= FAULT_FLAG_WRITE; if (error_code & X86_PF_INSTR)