From patchwork Fri Nov 4 22:35:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 13032661 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D9B8C433FE for ; Fri, 4 Nov 2022 22:39:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1CBED8E0011; Fri, 4 Nov 2022 18:39:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 156498E0007; Fri, 4 Nov 2022 18:39:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC3A68E0011; Fri, 4 Nov 2022 18:39:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C200F8E0007 for ; Fri, 4 Nov 2022 18:39:44 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A9815C0AC6 for ; Fri, 4 Nov 2022 22:39:44 +0000 (UTC) X-FDA: 80097228288.25.45B547B Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by imf17.hostedemail.com (Postfix) with ESMTP id 443A540005 for ; Fri, 4 Nov 2022 22:39:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1667601584; x=1699137584; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=xaioYoGXdFCH6JyKO39LB3gea0qfxKRbZWJ+Tm5uBgg=; b=TUxc7Exz3kxhqWXPE5186CDZ6ym1oJ+fCn2uMfUbZdU12Od6iCYV4uoF kFbOIHCVVjtz/aK4qqPDSZ2NksDCWw5cj/NnTpbnJ0WlEHdODyr0mLn8+ szyeN2pC7o1Lz5OVeU7uVqkJ3RPj+uuaUq6wCebvWMoQS1LuCAh71FE2K O1sxdBtb5OgIInSelO58nKDTR2X97vU7xVo2qfknU5eZfSw8MCir+VSAc zg0wYz6VnonVgDkFyvzYk2Mtte84QhsGxHO3Q9DNuWHYLLwS+6RuAPrVB qG0WL+asgU/IVgb17wxFrsCNCZ0Ep36aKZmvHqr97ERWq07rKRkOSB7tV A==; X-IronPort-AV: E=McAfee;i="6500,9779,10521"; a="297559680" X-IronPort-AV: E=Sophos;i="5.96,138,1665471600"; d="scan'208";a="297559680" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Nov 2022 15:39:43 -0700 X-IronPort-AV: E=McAfee;i="6500,9779,10521"; a="668514099" X-IronPort-AV: E=Sophos;i="5.96,138,1665471600"; d="scan'208";a="668514099" Received: from adhjerms-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.212.227.68]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Nov 2022 15:39:42 -0700 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V . Shankar" , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org Cc: rick.p.edgecombe@intel.com Subject: [PATCH v3 23/37] mm: Warn on shadow stack memory in wrong vma Date: Fri, 4 Nov 2022 15:35:50 -0700 Message-Id: <20221104223604.29615-24-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221104223604.29615-1-rick.p.edgecombe@intel.com> References: <20221104223604.29615-1-rick.p.edgecombe@intel.com> ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667601584; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=4C0U4C/Am5K4X0WaSC2NtdD6FJYGChD7H8vpOqQxCM8=; b=4HChdEBzwdFFXZo1Q74VH+UlgRXm3qBFKAjMuoCtO1VzSt96Jjjkl7K86VueoyT9WqqCtD UUncLEAZcl6zhoCPW1YczV5cryg/fZN3Ec1JEvVungFBBeA4PlaNkmTMEwSXDrG+vxE4GV l4OwNuKz+R0ixEg22GohiJlPo73SV/o= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=TUxc7Exz; spf=pass (imf17.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667601584; a=rsa-sha256; cv=none; b=zzQaDskadQplujqHLiBGR+/Qywv9R/bpQFYXNTjDLHRFLJc4h+1kIduLWXAz5LUYAtypiG IgnjjiW9+wH3Yfo/xjdijtSkt6TkGmssOnINQ2vTPK7WzyL+tn/Rxn9oI0FaMXx7h9/NlW DV5JBADiEbjNpDN2MMdmKz+a1UaYerY= Authentication-Results: imf17.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=TUxc7Exz; spf=pass (imf17.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com X-Stat-Signature: pra3mbdr5eh1hap611cp5z5s18d958qp X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 443A540005 X-HE-Tag: 1667601584-549379 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. One sharp edge is that PTEs that are both Write=0 and Dirty=1 are treated as shadow by the CPU, but this combination used to be created by the kernel on x86. Previous patches have changed the kernel to now avoid creating these PTEs unless they are for shadow stack memory. In case any missed corners of the kernel are still creating PTEs like this for non-shadow stack memory, and to catch any re-introductions of the logic, warn if any shadow stack PTEs (Write=0, Dirty=1) are found in non-shadow stack VMAs when they are being zapped. This won't catch transient cases but should have decent coverage. It will be compiled out when shadow stack is not configured. In order to check if a pte is shadow stack in core mm code, add default implmentations for pte_shstk() and pmd_shstk(). Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Rick Edgecombe --- v3: - New patch arch/x86/include/asm/pgtable.h | 2 ++ include/linux/pgtable.h | 14 ++++++++++++++ mm/huge_memory.c | 2 ++ mm/memory.c | 2 ++ 4 files changed, 20 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 0d18f3a4373d..051c03e59468 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -129,6 +129,7 @@ static inline bool pte_dirty(pte_t pte) return pte_flags(pte) & _PAGE_DIRTY_BITS; } +#define pte_shstk pte_shstk static inline bool pte_shstk(pte_t pte) { if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) @@ -147,6 +148,7 @@ static inline bool pmd_dirty(pmd_t pmd) return pmd_flags(pmd) & _PAGE_DIRTY_BITS; } +#define pmd_shstk pmd_shstk static inline bool pmd_shstk(pmd_t pmd) { if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 36926a207b6d..dd84af70b434 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -500,6 +500,20 @@ static inline pte_t pte_mkwrite_shstk(pte_t pte) } #endif +#ifndef pte_shstk +static inline bool pte_shstk(pte_t pte) +{ + return false; +} +#endif + +#ifndef pmd_shstk +static inline bool pmd_shstk(pmd_t pte) +{ + return false; +} +#endif + #ifndef pte_clear_savedwrite #define pte_clear_savedwrite pte_wrprotect #endif diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 7643a4db1b50..2540f0d4c8ff 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1656,6 +1656,8 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, */ orig_pmd = pmdp_huge_get_and_clear_full(vma, addr, pmd, tlb->fullmm); + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && + pmd_shstk(orig_pmd)); tlb_remove_pmd_tlb_entry(tlb, pmd, addr); if (vma_is_special_huge(vma)) { if (arch_needs_pgtable_deposit()) diff --git a/mm/memory.c b/mm/memory.c index b9bee283aad3..4331f33a02d6 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1437,6 +1437,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && + pte_shstk(ptent)); tlb_remove_tlb_entry(tlb, pte, addr); zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent);