From patchwork Tue Aug 30 01:00:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12958591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B688ECAAD3 for ; Tue, 30 Aug 2022 01:02:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E0A65940010; Mon, 29 Aug 2022 21:02:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DB8B5940007; Mon, 29 Aug 2022 21:02:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C32D0940010; Mon, 29 Aug 2022 21:02:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B3D07940007 for ; Mon, 29 Aug 2022 21:02:03 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8E510C0A97 for ; Tue, 30 Aug 2022 01:02:03 +0000 (UTC) X-FDA: 79854457326.02.06B3298 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf03.hostedemail.com (Postfix) with ESMTP id F1A1B20012 for ; Tue, 30 Aug 2022 01:02:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1661821323; x=1693357323; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=T/1SON8x3I6gBc9Gilj4Fh9sNy8RCIdk2WdRlZ6DEwk=; b=NGt90BTfZHwFIbypgX+INtu6+lqILpjW9rX+OjL9o8pRCcH6hxiL3ozr LXHZ/L9CWBptEy1thfJT2lv+2SPhdXJg6h/G0GInVn+cBHD+xU+VEloXB SEr00/nGEif06qIK7QaTpNLXW+l2cAETX759A++ZZQ0CSH+CML/5ZNQFj eJ794uZuiEuUCHyZHNMm8z6dbbubTFhRbH383VO2aNmgd4XGMwjNOFit7 vndB+Hb2XAm9XLyd+Z5x9GWDaBlGRvJSFXqJqThqr8UsU2JcFprni0YY7 Q1THuK1iDKpPArsfyfmAwcOG5srFY8TXV68r4RoAT2onMm3dIG8yojroi A==; X-IronPort-AV: E=McAfee;i="6500,9779,10454"; a="321170266" X-IronPort-AV: E=Sophos;i="5.93,273,1654585200"; d="scan'208";a="321170266" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Aug 2022 18:01:36 -0700 X-IronPort-AV: E=Sophos;i="5.93,273,1654585200"; d="scan'208";a="939803136" Received: from fpalamon-mobl.amr.corp.intel.com (HELO box.shutemov.name) ([10.252.54.23]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Aug 2022 18:01:32 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 8B9E1104167; Tue, 30 Aug 2022 04:01:24 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv8 05/11] x86/uaccess: Provide untagged_addr() and remove tags before address check Date: Tue, 30 Aug 2022 04:00:58 +0300 Message-Id: <20220830010104.1282-6-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220830010104.1282-1-kirill.shutemov@linux.intel.com> References: <20220830010104.1282-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=NGt90BTf; spf=none (imf03.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=fail reason="No valid SPF" header.from=intel.com (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661821323; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+gpgieOlIPxx05fS70p4VSaJETO7zeherGqCBT59DhU=; b=0EGcM3HJB8Zsnh7iXr1kehTuTxdlAvQHa7LOrpftBOGjUPJDvGATrijoQLXCeIXCO3vnmD 4XiDfg6EUMfo+Xdlw+wmeeH6OxtK+MaUM+OecZ0bhpZ1zKaYT2NFB/m4QrlBPr7hzkh8pI sTNOQOdHo3kfA8ECAlPMiOPazCDcDjQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661821323; a=rsa-sha256; cv=none; b=6zi2XMOLJnQL0tXcUmZjEsPfXJ373ussDSnuBO+JrZOIqLMVHUW4uUo8ISvwMk+4VZr+fl ewFviK1vJm15jK1lKVlz60Wa0+UuAcOZQ4P52AWRDbKLr2SY1t/qlr7Oz0TJdqdAA1jHqx Kdrm+gmejX4JeFdCEjWHTcCxSL6g4/A= X-Stat-Signature: ohnssoau9feedma5cutyeyqg7bnno8bc X-Rspamd-Queue-Id: F1A1B20012 Authentication-Results: imf03.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=NGt90BTf; spf=none (imf03.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=fail reason="No valid SPF" header.from=intel.com (policy=none) X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1661821322-473450 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: untagged_addr() is a helper used by the core-mm to strip tag bits and get the address to the canonical shape. In only handles userspace addresses. The untagging mask is stored in mmu_context and will be set on enabling LAM for the process. The tags must not be included into check whether it's okay to access the userspace address. Strip tags in access_ok(). get_user() and put_user() don't use access_ok(), but check access against TASK_SIZE directly in assembly. Strip tags, before calling into the assembly helper. Signed-off-by: Kirill A. Shutemov Tested-by: Alexander Potapenko Acked-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/mmu.h | 3 +++ arch/x86/include/asm/mmu_context.h | 11 ++++++++ arch/x86/include/asm/uaccess.h | 42 +++++++++++++++++++++++++++--- arch/x86/kernel/process.c | 3 +++ 4 files changed, 56 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 002889ca8978..2fdb390040b5 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -43,6 +43,9 @@ typedef struct { /* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ unsigned long lam_cr3_mask; + + /* Significant bits of the virtual address. Excludes tag bits. */ + u64 untag_mask; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 69c943b2ae90..5bd3d46685dc 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -100,6 +100,12 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; + mm->context.untag_mask = oldmm->context.untag_mask; +} + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ + mm->context.untag_mask = -1UL; } #else @@ -112,6 +118,10 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { } + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ +} #endif #define enter_lazy_tlb enter_lazy_tlb @@ -138,6 +148,7 @@ static inline int init_new_context(struct task_struct *tsk, mm->context.execute_only_pkey = -1; } #endif + mm_reset_untag_mask(mm); init_new_context_ldt(mm); return 0; } diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 913e593a3b45..803241dfc473 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -6,6 +6,7 @@ */ #include #include +#include #include #include #include @@ -20,6 +21,30 @@ static inline bool pagefault_disabled(void); # define WARN_ON_IN_IRQ() #endif +#ifdef CONFIG_X86_64 +/* + * Mask out tag bits from the address. + * + * Magic with the 'sign' allows to untag userspace pointer without any branches + * while leaving kernel addresses intact. + */ +#define untagged_addr(mm, addr) ({ \ + u64 __addr = (__force u64)(addr); \ + s64 sign = (s64)__addr >> 63; \ + __addr &= (mm)->context.untag_mask | sign; \ + (__force __typeof__(addr))__addr; \ +}) + +#define untagged_ptr(mm, ptr) ({ \ + u64 __ptrval = (__force u64)(ptr); \ + __ptrval = untagged_addr(mm, __ptrval); \ + (__force __typeof__(*(ptr)) *)__ptrval; \ +}) +#else +#define untagged_addr(mm, addr) (addr) +#define untagged_ptr(mm, ptr) (ptr) +#endif + /** * access_ok - Checks if a user space pointer is valid * @addr: User space pointer to start of block to check @@ -40,7 +65,7 @@ static inline bool pagefault_disabled(void); #define access_ok(addr, size) \ ({ \ WARN_ON_IN_IRQ(); \ - likely(__access_ok(addr, size)); \ + likely(__access_ok(untagged_addr(current->mm, addr), size)); \ }) #include @@ -125,7 +150,13 @@ extern int __get_user_bad(void); * Return: zero on success, or -EFAULT on error. * On error, the variable @x is set to zero. */ -#define get_user(x,ptr) ({ might_fault(); do_get_user_call(get_user,x,ptr); }) +#define get_user(x,ptr) \ +({ \ + __typeof__(*(ptr)) __user *__ptr_clean; \ + __ptr_clean = untagged_ptr(current->mm, ptr); \ + might_fault(); \ + do_get_user_call(get_user,x,__ptr_clean); \ +}) /** * __get_user - Get a simple variable from user space, with less checking. @@ -222,7 +253,12 @@ extern void __put_user_nocheck_8(void); * * Return: zero on success, or -EFAULT on error. */ -#define put_user(x, ptr) ({ might_fault(); do_put_user_call(put_user,x,ptr); }) +#define put_user(x, ptr) ({ \ + __typeof__(*(ptr)) __user *__ptr_clean; \ + __ptr_clean = untagged_ptr(current->mm, ptr); \ + might_fault(); \ + do_put_user_call(put_user,x,__ptr_clean); \ +}) /** * __put_user - Write a simple value into user space, with less checking. diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 58a6ea472db9..b0e86fb11ffa 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -47,6 +47,7 @@ #include #include #include +#include #include "process.h" @@ -367,6 +368,8 @@ void arch_setup_new_exec(void) task_clear_spec_ssb_noexec(current); speculation_ctrl_update(read_thread_flags()); } + + mm_reset_untag_mask(current->mm); } #ifdef CONFIG_X86_IOPL_IOPERM