From patchwork Sun Mar 12 11:26:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171306 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C9ABC6FA99 for ; Sun, 12 Mar 2023 11:27:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A29778E000F; Sun, 12 Mar 2023 07:27:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B2088E000E; Sun, 12 Mar 2023 07:27:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 82C428E000F; Sun, 12 Mar 2023 07:27:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 717D68E0008 for ; Sun, 12 Mar 2023 07:27:04 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4E8AF1A0806 for ; Sun, 12 Mar 2023 11:27:04 +0000 (UTC) X-FDA: 80560019568.11.4C0711C Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf17.hostedemail.com (Postfix) with ESMTP id 20A0F40006 for ; Sun, 12 Mar 2023 11:27:01 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=WEMHrPg8; spf=none (imf17.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620422; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X513hi487xr2Y5cmAhhhOp2BVxYcOzKFwOGV+7peITQ=; b=eWuaWgAkUYWn2vNL8HAxlUYZgvBC+dN1yyBJjCD46V0TLSAqDFOb1R50KwM0dnBG0BWp1z DdPnK8bAYEsDQFR6gsPCOglzXXBhteuwxkoY4i/l9mXbN0f78BCC2cBA2SLZm+LTRkU4Kn zcIx3ienQdJiy8PbY2wvu32xYDDXwzA= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=WEMHrPg8; spf=none (imf17.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620422; a=rsa-sha256; cv=none; b=mCafHxtzQm6UkAkDbT5ddWKMUE4dc9wr8POhnSWo26+wqU3EgptQX29+z9/xUwY4JXHY9b KEPG9Andly7IZBKby1qORXS2eXOlN7gTM303/h7kLbUizKYt6y+NumfJnST+pcmx1lGqbW jA2cTnb/5exKNImYtXLRQBjcGw+VRgs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620422; x=1710156422; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Tq6WtwHrazfLXD0XBMc2Q+r2SDElsHETE3zLNMxGwxY=; b=WEMHrPg8XAsq/zXeVE5iM5x9L3OS9aUd707kZyuAecHJtXvL/THrzY9p wVrcIAR+gMWelqMnYy8Pv/OIC+oBB5JUKaA6REce+d3h3jYG0XCfhOARh h7VklkIaCu9KNIbgp99OJiOCRtLJXH+1gRjVHvPIY6O+76Q7M1sJhO3Ky CtdNv3JgOBBTjsd2CCUW51lgYg0wqSN90dMRNVtft1Z6eg1pMuT40G7PI qc4ny7GA49K/uWrh8zDLp2cB/hdP8T+vLvin4emadR9dgA8Fid/2/nnlZ GXgifYiQzdkxvyJUQoCJu2YKfMxwiHZM7kV7EjEd2F6wyhvuwUSBy0VWE g==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="399577576" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="399577576" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:36 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="671580735" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="671580735" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:30 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 8FAA510D7B4; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv16 06/17] x86/uaccess: Provide untagged_addr() and remove tags before address check Date: Sun, 12 Mar 2023 14:26:01 +0300 Message-Id: <20230312112612.31869-7-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 20A0F40006 X-Stat-Signature: rh1hgq4xw156yzetp6orupkowwqbspgj X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1678620421-166114 X-HE-Meta: U2FsdGVkX1/4VctczLRe3qnXwgvZjIoFkH6Tgv898Jh26lX61SM4mKRFUWnNeygE889ImITRcJiGXKxTPi/wVVem/E0zSFRaWqb4ZvJSRdZU3DsFLt+uj8e+6mH3Xug7eaT6XYRAM1b0mSMhZgkjiXEdlDH8D6rKf2pMlwF7qxJJMwx4dCdTPmL5anolKqEVx4apLJNb0b0ZrbvFLsGaao5ShjKzbBSlb/EvMZ0LoHu3xvPedjXS1OtVh9IBXsVP7/Yzu8m1rQO8ojZSgvV4ZaZXxBlRP0m4dXZtLTLujdIt3B3A3ipMbu8xgD+qEQqIOkUvhjqmf0Xs9eBuVimEjCAKE7n9Ixx77Gtw7OTrnmFKLkBwOnaKDLRnYdI8mywmszpOltuZoIC8h39fErUlsmde0LqbGssuSOHChOAyajbOP0sKF94R7FTx2RO3/nWq55GNUwbgxv/JUSa+s8DAVcLk+n5ZYdRmTk3t6AKsAH1sUDHb+Qp28Gtpuiy4+lFq+ojH8JWCj7JMicRQqfEMNx2mkw0aFbkTW1dGCr1sUA6UT9zka/cJ9wv6LEeqxz2kj+t3j+r0lUTky4oXlhuw3nQBsLsvkPSrjKIp6sZP1cwmo+qZsOGsUMbZx9tP4TqQzWEFoIyGxLaUVRQHHOAIRs3NtOGqCQdp7+2/AHgSE/Hlp8HLQRXMeBFsSU/S7lVuvvdsb+v5h1mxxAKAOlTVsY/lBSHv9ogtZb5iO3MeLkFh6pGkSBnna2N2ilcP8g/q7qFHeKg3FuNpp1VMcYC38s6m+6oFij84p6UJFSVEC20o0m7f2/u/Rn1x8gvX/PpVW3hUntuVZSYPJDmWzG2n+evQBPmtsWbVSPg6InspIGllg+guICl6WpMfq+eshoFDb6N8WN73l9W1DMWmOXJiUzdcF8KxFzm4WtUN9Rh4gU6qmPOLZqL9yc6XcX4+A3WMya3tdgFivSlWiIHyRN0 AASK2lT4 mxIuQwLCJjY0F1N7b1rz+BUWjjzL9AfPPvnEvxTywK8Cm9auuWmWVFUsXFl2wV3lhhhvK474hRYJK8jQgOIHhzpgRJoLHgPWEvEdpUh07pwIceckO6ROFm5iGITrWfDs7Sz/c14seSHC0Fsat5QQycGvQXY85Q9bkQ7IRKQAgaShysfknvZmrytdN7aC8uiwlbL583352+I5gYs9L/JMrXV5uzNxAL8jWwltrVdnQ7hZPekUYedJ1tGb1LMxVfijsxpw/9XsZlgb4jmWcgyI4P6uJ1a/f/D3QaGfjyj1tQG1RIbiRlEmtoWDy/dTlVWyqUmiSg0yJ52NTjWAER2y8Q0wjPIchWZ5siwdBtVGvU9iFPCY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: untagged_addr() is a helper used by the core-mm to strip tag bits and get the address to the canonical shape based on rules of the current thread. It only handles userspace addresses. The untagging mask is stored in per-CPU variable and set on context switching to the task. The tags must not be included into check whether it's okay to access the userspace address. Strip tags in access_ok(). Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/x86/include/asm/mmu.h | 3 +++ arch/x86/include/asm/mmu_context.h | 11 +++++++++ arch/x86/include/asm/tlbflush.h | 10 ++++++++ arch/x86/include/asm/uaccess.h | 39 ++++++++++++++++++++++++++++-- arch/x86/kernel/process.c | 3 +++ arch/x86/mm/init.c | 5 ++++ 6 files changed, 69 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 22fc9fbf1d0a..9cac8c45a647 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -45,6 +45,9 @@ typedef struct { #ifdef CONFIG_ADDRESS_MASKING /* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ unsigned long lam_cr3_mask; + + /* Significant bits of the virtual address. Excludes tag bits. */ + u64 untag_mask; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 0295c3863db7..eb1387ac40fa 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -101,6 +101,12 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; + mm->context.untag_mask = oldmm->context.untag_mask; +} + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ + mm->context.untag_mask = -1UL; } #else @@ -113,6 +119,10 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { } + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ +} #endif #define enter_lazy_tlb enter_lazy_tlb @@ -139,6 +149,7 @@ static inline int init_new_context(struct task_struct *tsk, mm->context.execute_only_pkey = -1; } #endif + mm_reset_untag_mask(mm); init_new_context_ldt(mm); return 0; } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index e8b47f57bd4a..75bfaa421030 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -54,6 +54,15 @@ static inline void cr4_clear_bits(unsigned long mask) local_irq_restore(flags); } +#ifdef CONFIG_ADDRESS_MASKING +DECLARE_PER_CPU(u64, tlbstate_untag_mask); + +static inline u64 current_untag_mask(void) +{ + return this_cpu_read(tlbstate_untag_mask); +} +#endif + #ifndef MODULE /* * 6 because 6 should be plenty and struct tlb_state will fit in two cache @@ -380,6 +389,7 @@ static inline void set_tlbstate_lam_mode(struct mm_struct *mm) { this_cpu_write(cpu_tlbstate.lam, mm->context.lam_cr3_mask >> X86_CR3_LAM_U57_BIT); + this_cpu_write(tlbstate_untag_mask, mm->context.untag_mask); } #else diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 1cc756eafa44..c79ebdbd6356 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -7,11 +7,13 @@ #include #include #include +#include #include #include #include #include #include +#include #ifdef CONFIG_DEBUG_ATOMIC_SLEEP static inline bool pagefault_disabled(void); @@ -21,6 +23,39 @@ static inline bool pagefault_disabled(void); # define WARN_ON_IN_IRQ() #endif +#ifdef CONFIG_ADDRESS_MASKING +/* + * Mask out tag bits from the address. + * + * Magic with the 'sign' allows to untag userspace pointer without any branches + * while leaving kernel addresses intact. + */ +static inline unsigned long __untagged_addr(unsigned long addr, + unsigned long mask) +{ + long sign = addr >> 63; + + addr &= mask | sign; + return addr; +} + +#define untagged_addr(addr) ({ \ + u64 __addr = (__force u64)(addr); \ + __addr = __untagged_addr(__addr, current_untag_mask()); \ + (__force __typeof__(addr))__addr; \ +}) + +#define untagged_addr_remote(mm, addr) ({ \ + u64 __addr = (__force u64)(addr); \ + mmap_assert_locked(mm); \ + __addr = __untagged_addr(__addr, (mm)->context.untag_mask); \ + (__force __typeof__(addr))__addr; \ +}) + +#else +#define untagged_addr(addr) (addr) +#endif + /** * access_ok - Checks if a user space pointer is valid * @addr: User space pointer to start of block to check @@ -38,10 +73,10 @@ static inline bool pagefault_disabled(void); * Return: true (nonzero) if the memory block may be valid, false (zero) * if it is definitely invalid. */ -#define access_ok(addr, size) \ +#define access_ok(addr, size) \ ({ \ WARN_ON_IN_IRQ(); \ - likely(__access_ok(addr, size)); \ + likely(__access_ok(untagged_addr(addr), size)); \ }) #include diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index b650cde3f64d..bbc8c4c6e360 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -48,6 +48,7 @@ #include #include #include +#include #include "process.h" @@ -368,6 +369,8 @@ void arch_setup_new_exec(void) task_clear_spec_ssb_noexec(current); speculation_ctrl_update(read_thread_flags()); } + + mm_reset_untag_mask(current->mm); } #ifdef CONFIG_X86_IOPL_IOPERM diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index cb258f58fdc8..659b6c0f7910 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -1048,6 +1048,11 @@ __visible DEFINE_PER_CPU_ALIGNED(struct tlb_state, cpu_tlbstate) = { .cr4 = ~0UL, /* fail hard if we screw up cr4 shadow initialization */ }; +#ifdef CONFIG_ADDRESS_MASKING +DEFINE_PER_CPU(u64, tlbstate_untag_mask); +EXPORT_PER_CPU_SYMBOL(tlbstate_untag_mask); +#endif + void update_cache_mode_entry(unsigned entry, enum page_cache_mode cache) { /* entry 0 MUST be WB (hardwired to speed up translations) */