From patchwork Sun Mar 19 00:15:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 13180154 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 661FCC76196 for ; Sun, 19 Mar 2023 00:16:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C864280018; Sat, 18 Mar 2023 20:16:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 74FC0280001; Sat, 18 Mar 2023 20:16:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3586A280018; Sat, 18 Mar 2023 20:16:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 12A82280001 for ; Sat, 18 Mar 2023 20:16:41 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E7C5040C8A for ; Sun, 19 Mar 2023 00:16:40 +0000 (UTC) X-FDA: 80583731760.13.B61E715 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf11.hostedemail.com (Postfix) with ESMTP id E262F4000C for ; Sun, 19 Mar 2023 00:16:38 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=F7lDWYYa; spf=pass (imf11.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679184999; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=QkKdIBSM6+NW96X7S3iu9foVuWMjrTX9oD9LGR3hCIY=; b=NZKkPhCwh4IGLDKJVS+5+zYfpR7PTOfvore/W4jzeSOce4IcQ4alwUIqIFDoz+sgMZtdac 4hixMNSZ+1DSJCfjNEsrNwUFuYq7JZDDTXARKm0v/ERsb15Tk+9jw4hPqWV4XjzDPejB+Y AW+mLwW+ksA575ZgcUDCpf1z5J3IsYs= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=F7lDWYYa; spf=pass (imf11.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679184999; a=rsa-sha256; cv=none; b=5akc2nj2XjBDUcsSty6uK71IUiwnSsTX/xP6d+lCp9/bOBM5oIoXBU8DhQ70nRdjABdR6j aVBQFUWIQVaJ3S4upKzCemYouKsqicOI0sBPYTU5iqa9TIHjjBbSZhga7ApTBo+61TLVDf ff/tEJkc9KQHbJbWnUG/8kBFHNwGiKE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679184998; x=1710720998; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=hsI3rH3pI5zk+4z/rTXM9tpk9smi32a1UvWDoOYm7ZQ=; b=F7lDWYYaUSBbdg6LeJCmjYqebyoNrKsMJLNb7CG4PVJsAymM3BmkrMzO T1lcv2Nv2vCJ62RP2eFB0D3X+oyb84DdwFC8P2D9Qg2B6LOrlbLnm2rvZ M2gVUNEMXzoJK/DYTxyvtM3Fl+n/alyzfVTQ+fQGidB0V5BGhGjBSmPTY znf6xp7EdeOrxPr/HhqSfcbudif8V/BLUUl12lMRtTLo6xZjx0RXzQWw+ C+o5Li7BHyV0dN/Hyu9LBQuUgfA/107JyH9JpFB2mmXQahcREMnaM7IJs vtiiI8DYfsOYECG8+es6TBfCxnc4efqomSYelpb5p8eiieprn7pn15ypp g==; X-IronPort-AV: E=McAfee;i="6600,9927,10653"; a="338491318" X-IronPort-AV: E=Sophos;i="5.98,272,1673942400"; d="scan'208";a="338491318" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2023 17:16:38 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10653"; a="749672895" X-IronPort-AV: E=Sophos;i="5.98,272,1673942400"; d="scan'208";a="749672895" Received: from bmahatwo-mobl1.gar.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.135.34.5]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2023 17:16:36 -0700 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com, szabolcs.nagy@arm.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v8 26/40] mm: Warn on shadow stack memory in wrong vma Date: Sat, 18 Mar 2023 17:15:21 -0700 Message-Id: <20230319001535.23210-27-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230319001535.23210-1-rick.p.edgecombe@intel.com> References: <20230319001535.23210-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E262F4000C X-Rspam-User: X-Stat-Signature: wftnxgymp53xkhh83s8bw74re6ii14fb X-HE-Tag: 1679184998-564178 X-HE-Meta: U2FsdGVkX19WlYxsAiifssZf8y1d3EL3uMeIrteI606lNeUtGjV+qToPQNJloWmtcl/oD/Sro681OeOG5jspzBC8kXQXuUDL2ZuBLvMFLxZkESyMK+aMG1Czxk3u/QO0h93q+61wVYQY/ViXZoSKPeWmFHsCMeKyuRXfYcsv4+dMpC3Nt34VBtVSuF65A+rIv+Cg8aLU0WmEz28hgCpEXFkWEhDWZ9AmwZS3NJrXYIjbkLjfQ/wR/ORpmK9M15N2shgi+pw7M5G7sRoAQ8UfmTcmvxgRLbKVU+/AX8wg9XO9AgpNZ94XbrRc58W94684u2K5+s5oi8s/yzMKG4i/mRMJNYvBCCEptbsAcZXVYnFSOLU4Az/fQfl9CCN4n1F8keuo7L1/1x+4SiHxeRS54r8j3rpw3PnpVAic6Gjctt0Y8wZCoa95Pel3BmCIH8xZNowO9ZFu9tlwTNXHljZ5iZKVS8wBplTD+iGVTUsiNJKeO1l6iPtTYAXVwVX9nO/q6L2Li79xsPUSmX4/8LEO/ydHQP1quoiT/cmU6Byl3FXp/I4vM2hct7nGaC3kcmqnij3Em7A5eLsnHHWQHkUPHynZI1FrsrXgNYKZjwFnaig6eA7KWVZkCURdOj+wVB6wxlzuyVnUS1DTi5HxuHHzwuNbhbZVjShwE70gvBRzRzV3VY0bHYE1G4mfk0R3GTzQ4UWBNs23W8yNOa7UiGvmh3pRmXFZFWEiWTn3czogAXPuXDfznpieA2Ah1FU1bHVdU+20ntsFWZv0FP8alkBuqziAmHBlROaTLLiivxBloKa2+0FdpbgACkMooSc9KZdAiet6q9jLO7Si00O2kebkCBaxtNOEMkWPTxq20NNux3um9AncYAbCt4bAPbmrkE+XVjEML+wlAkgnhT0rbpGI1VQDnrmbmjEgiOHgxkBNR078khWTXmkCWEvDd9d4WJol/RKVNd2gRxcLi3ijGfB AomZ8eJI nNleQ2ZHQlzmrIAi+SgBvIyF5gMELTSfEkcZ5Ac3I7v7pBEqNmmUunW7lVUx0/8smNlIV9JpuLkbB82nzEwxCPgf5zVWmUEf96gpzpvRUM2/lGl2+wpLiw7TFk5DblmxQrbiJdy6b0SPMZP8b+v/C1VCzaHRkeoU+D93pESgyWCMjQucwJNB5LsXGioKfbIE29Ex4vTso2/0MsU2C2wod4t9+qixRAD8KC7VaVib6akxVOVOjVgdPwpDHqxrOT0XgnpzdkT4qnbXnuH5V9qh/uZLKMVMmxOxIp4aLcAapzHtiN5mJ6LXWBZbmmjGX3puQa6oTtseY5sfruNINDDbSpy3ebi1W/g+wjHQA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. One sharp edge is that PTEs that are both Write=0 and Dirty=1 are treated as shadow by the CPU, but this combination used to be created by the kernel on x86. Previous patches have changed the kernel to now avoid creating these PTEs unless they are for shadow stack memory. In case any missed corners of the kernel are still creating PTEs like this for non-shadow stack memory, and to catch any re-introductions of the logic, warn if any shadow stack PTEs (Write=0, Dirty=1) are found in non-shadow stack VMAs when they are being zapped. This won't catch transient cases but should have decent coverage. It will be compiled out when shadow stack is not configured. In order to check if a PTE is shadow stack in core mm code, add two arch breakouts arch_check_zapped_pte/pmd(). This will allow shadow stack specific code to be kept in arch/x86. Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook Acked-by: Mike Rapoport (IBM) Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Acked-by: David Hildenbrand --- v8: - Update commit log verbaige (Boris) v6: - Add arch breakout to remove shstk from core MM code. v5: - Fix typo in commit log v3: - New patch --- arch/x86/include/asm/pgtable.h | 6 ++++++ arch/x86/mm/pgtable.c | 12 ++++++++++++ include/linux/pgtable.h | 14 ++++++++++++++ mm/huge_memory.c | 1 + mm/memory.c | 1 + 5 files changed, 34 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 2e3d8cca1195..e5b3dce0d9fe 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1684,6 +1684,12 @@ static inline bool arch_has_hw_pte_young(void) return true; } +#define arch_check_zapped_pte arch_check_zapped_pte +void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte); + +#define arch_check_zapped_pmd arch_check_zapped_pmd +void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd); + #ifdef CONFIG_XEN_PV #define arch_has_hw_nonleaf_pmd_young arch_has_hw_nonleaf_pmd_young static inline bool arch_has_hw_nonleaf_pmd_young(void) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 98856bcc8102..afab0bc7862b 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -906,3 +906,15 @@ pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) return pmd; } + +void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte) +{ + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && + pte_shstk(pte)); +} + +void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd) +{ + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && + pmd_shstk(pmd)); +} diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index c63cd44777ec..4a8970b9fb11 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -291,6 +291,20 @@ static inline bool arch_has_hw_pte_young(void) } #endif +#ifndef arch_check_zapped_pte +static inline void arch_check_zapped_pte(struct vm_area_struct *vma, + pte_t pte) +{ +} +#endif + +#ifndef arch_check_zapped_pmd +static inline void arch_check_zapped_pmd(struct vm_area_struct *vma, + pmd_t pmd) +{ +} +#endif + #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index aaf815838144..24797be05fcb 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1689,6 +1689,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, */ orig_pmd = pmdp_huge_get_and_clear_full(vma, addr, pmd, tlb->fullmm); + arch_check_zapped_pmd(vma, orig_pmd); tlb_remove_pmd_tlb_entry(tlb, pmd, addr); if (vma_is_special_huge(vma)) { if (arch_needs_pgtable_deposit()) diff --git a/mm/memory.c b/mm/memory.c index d0972d2d6f36..c953c2c4588c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1389,6 +1389,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + arch_check_zapped_pte(vma, ptent); tlb_remove_tlb_entry(tlb, pte, addr); zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent);