From patchwork Tue Dec 27 03:08:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13082042 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16D82C4332F for ; Tue, 27 Dec 2022 03:08:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 922208E0006; Mon, 26 Dec 2022 22:08:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8AB238E0001; Mon, 26 Dec 2022 22:08:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 723118E0006; Mon, 26 Dec 2022 22:08:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 593D88E0005 for ; Mon, 26 Dec 2022 22:08:53 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 32FF0801A5 for ; Tue, 27 Dec 2022 03:08:53 +0000 (UTC) X-FDA: 80286604146.15.45A3522 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf19.hostedemail.com (Postfix) with ESMTP id 2C65A1A000D for ; Tue, 27 Dec 2022 03:08:50 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=JA3+GIZO; spf=none (imf19.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.151) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672110531; a=rsa-sha256; cv=none; b=RbvJ6k+jlVBcgJdlqsTYggZ1ylcl1KNL5dO7IYfBAhxKLOpR1e0ufMQFuU1Zrs32zZPQsB /aTi0/2vXbondcE3jEp9fTkUbifq0hLozaDLKrJjeA9USGVsGVzTc9mw+IdRn0uMtFXF77 7k5RuCT8rGkp+Evsld9DROZnbECYexI= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=JA3+GIZO; spf=none (imf19.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.151) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672110531; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qmz2rANUZ4KIs+oy6HIjixRfmrtC4y0ZixDKV4/MIyc=; b=bHsoHAvjSce8EZDtv+2Bx3Kscdg73LjgECtdlCmor8fBPuow7VyRye+/TNx8fGgs7y6ETU ykDmNrKNsEzSDxDHbSUrj7cq+ugpvPsSZys10zX6Bq76w/9lQAsp+A0WhdePVMsP617xrL h8xOo0aHQmnucM9mHceeBTdUqaoFNEY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110531; x=1703646531; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=yO+iUKN0r9KVMUOqq6nIAT1LjszIJDhKiqjV1XdDapw=; b=JA3+GIZO3VAAuXhQ8KWhAYq00bbOJnJzpVGhJAxZ739pfLeDacfXahK8 XZeMdU5i2t3ULFtdoedvbYYRfRtdOmDUK4RpNaHC1t6Kr2uOZuRIRwkCC QeYYgeXwDlFE81l/MnJ/fKjpcxii1Gj9jd2VucBfvuZWHnse0qZugaAWI OgxEYER+FRAzIAzs4pnKM2xkXsyC6DY8+PhN656HT7wnFIu+K912S1tB6 EdF/VRxE4+v3NTLDCETDuCuX6jxxgXg8+2EGDzv03jtUy7ErxPxtJKzw/ LhoKFB1LKgQMYFL9sFCK7dS18CeiSmcAx8JRTE7MvLJXX1XxbyotcjPqq g==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="300994690" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="300994690" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:50 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="741605215" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="741605215" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:45 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id ABB7210BBB2; Tue, 27 Dec 2022 06:08:36 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv13 05/16] x86/uaccess: Provide untagged_addr() and remove tags before address check Date: Tue, 27 Dec 2022 06:08:18 +0300 Message-Id: <20221227030829.12508-6-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 2C65A1A000D X-Rspamd-Server: rspam01 X-Stat-Signature: jh7nf4m571kpcgm7grjg6wmeofq6i9jo X-HE-Tag: 1672110530-273273 X-HE-Meta: U2FsdGVkX18tYsqea0W9LcUphqHOANcRokbnLzSsY6DkiH9tN/bznIcKougoy2JxLY4CX/ZEMOkd+/058/xN1bgYQu23W1J+aDVgRGz9e6lv4b+up/VyUscxBXefK5OT+eR9ZehdkDbWmzE3SSlbrrmXmRBesCsUE24nTjeLkPBs3VMsU3MXlTeyCCjAsYIWqM5XLttzRPKW99FjFAMm6pcOyXa8BdzGaaIt2QzbD6/vXkEhntM6VfDTZNjtQoWE+jcozcoTHUuZnWbOCVpIhlAppj1K094QC/kV4eWWUuLAQSVhEUzlh1Moxwpf9pgKHwaVcLNFaOWeUyj9aG7aZVEOKq5GwD/XlIDfy3x7PnSEZmEFlC5t3vkaLuqdCf00dT2Gi5W9kQQKUnClbwdObDUqxN/bZUrI7tHvgv5XXG5WlakCROSo63FsCAWy/C/ZnWBkZuJMXQxDy4U0pJhaVVxsjM6o4U9IqWpZ0jvtXFVAF/zlySsv4LpdmDchEyzlS4sOziiKaRw7RuEnFRn40xVZv7bBGt5qhEzeCD+skbpTf6mQ2VCOtRJxPyKYVc58VvQ+QoxMraj0pucTgTOLkt6YpSHYMzGtFdguw4weGNz13DcMQKJcYBF+/Wb71lD8fSIH2vRIvhdzl0DBI5+YZG0HdajYFVgAk1Jc/dhWlU6R3CKM3pkzYLqdxpsz2AJcZUKCKgHrZmGxbk9+5SHGDMbBSeTsnIYSGkhB9RnrJ2V73Qw4Ky7/ZGd+EmL49bE6KeVeLLHkIo+7+XNPK8uNNBXXd2+qn24qpSs2eq7nIeAeY0ooeqwKkHKLloB61ikBQPK8wg5nbqtarF5jnt+/oss52Jy5EqbaVMP+syDhq9o3KpETb60httZqgmRhqu8E2RIIYa0cD8TBs0yCjSTPCh0lS2aQhF36ZJn1J2kCoXqipwlXVKKf774H86AMwGhFDjwoVCyod7lYqz65gaf 06ST1oiT 7LVgZLOztP28iS0OIQ8QPHXrG9k3t24Y+SJ+H X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: untagged_addr() is a helper used by the core-mm to strip tag bits and get the address to the canonical shape. In only handles userspace addresses. The untagging mask is stored in mmu_context and will be set on enabling LAM for the process. The tags must not be included into check whether it's okay to access the userspace address. Strip tags in access_ok(). get_user() and put_user() don't use access_ok(), but check access against TASK_SIZE directly in assembly. Strip tags, before calling into the assembly helper. Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) Acked-by: Andy Lutomirski Tested-by: Alexander Potapenko --- arch/x86/include/asm/mmu.h | 3 ++ arch/x86/include/asm/mmu_context.h | 11 +++++++ arch/x86/include/asm/uaccess.h | 47 +++++++++++++++++++++++++++--- arch/x86/kernel/process.c | 3 ++ 4 files changed, 60 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 9a046aacad8d..ed72fcd2292d 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -43,6 +43,9 @@ typedef struct { /* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ unsigned long lam_cr3_mask; + + /* Significant bits of the virtual address. Excludes tag bits. */ + u64 untag_mask; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 464cca41d20a..71581cb4811b 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -100,6 +100,12 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; + mm->context.untag_mask = oldmm->context.untag_mask; +} + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ + mm->context.untag_mask = -1UL; } #else @@ -112,6 +118,10 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { } + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ +} #endif #define enter_lazy_tlb enter_lazy_tlb @@ -138,6 +148,7 @@ static inline int init_new_context(struct task_struct *tsk, mm->context.execute_only_pkey = -1; } #endif + mm_reset_untag_mask(mm); init_new_context_ldt(mm); return 0; } diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 1cc756eafa44..cbb463e9344f 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -21,6 +22,37 @@ static inline bool pagefault_disabled(void); # define WARN_ON_IN_IRQ() #endif +#ifdef CONFIG_X86_64 +/* + * Mask out tag bits from the address. + * + * Magic with the 'sign' allows to untag userspace pointer without any branches + * while leaving kernel addresses intact. + */ +#define __untagged_addr(mm, addr) ({ \ + u64 __addr = (__force u64)(addr); \ + s64 sign = (s64)__addr >> 63; \ + __addr &= READ_ONCE((mm)->context.untag_mask) | sign; \ + (__force __typeof__(addr))__addr; \ +}) + +#define untagged_addr(addr) __untagged_addr(current->mm, addr) + +#define untagged_addr_remote(mm, addr) ({ \ + mmap_assert_locked(mm); \ + __untagged_addr(mm, addr); \ +}) + +#define untagged_ptr(ptr) ({ \ + u64 __ptrval = (__force u64)(ptr); \ + __ptrval = untagged_addr(__ptrval); \ + (__force __typeof__(ptr))__ptrval; \ +}) +#else +#define untagged_addr(addr) (addr) +#define untagged_ptr(ptr) (ptr) +#endif + /** * access_ok - Checks if a user space pointer is valid * @addr: User space pointer to start of block to check @@ -38,10 +70,10 @@ static inline bool pagefault_disabled(void); * Return: true (nonzero) if the memory block may be valid, false (zero) * if it is definitely invalid. */ -#define access_ok(addr, size) \ +#define access_ok(addr, size) \ ({ \ WARN_ON_IN_IRQ(); \ - likely(__access_ok(addr, size)); \ + likely(__access_ok(untagged_addr(addr), size)); \ }) #include @@ -127,7 +159,11 @@ extern int __get_user_bad(void); * Return: zero on success, or -EFAULT on error. * On error, the variable @x is set to zero. */ -#define get_user(x,ptr) ({ might_fault(); do_get_user_call(get_user,x,ptr); }) +#define get_user(x,ptr) \ +({ \ + might_fault(); \ + do_get_user_call(get_user,x,untagged_ptr(ptr)); \ +}) /** * __get_user - Get a simple variable from user space, with less checking. @@ -227,7 +263,10 @@ extern void __put_user_nocheck_8(void); * * Return: zero on success, or -EFAULT on error. */ -#define put_user(x, ptr) ({ might_fault(); do_put_user_call(put_user,x,ptr); }) +#define put_user(x, ptr) ({ \ + might_fault(); \ + do_put_user_call(put_user,x,untagged_ptr(ptr)); \ +}) /** * __put_user - Write a simple value into user space, with less checking. diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 40d156a31676..ef6bde1d40d8 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -47,6 +47,7 @@ #include #include #include +#include #include "process.h" @@ -367,6 +368,8 @@ void arch_setup_new_exec(void) task_clear_spec_ssb_noexec(current); speculation_ctrl_update(read_thread_flags()); } + + mm_reset_untag_mask(current->mm); } #ifdef CONFIG_X86_IOPL_IOPERM