From patchwork Wed May 11 02:27:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12845705 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BBA1C433EF for ; Wed, 11 May 2022 02:29:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E79516B0071; Tue, 10 May 2022 22:29:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E008C6B0072; Tue, 10 May 2022 22:29:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C7AF96B0075; Tue, 10 May 2022 22:29:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B66D16B0071 for ; Tue, 10 May 2022 22:29:41 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8D58331EA4 for ; Wed, 11 May 2022 02:29:41 +0000 (UTC) X-FDA: 79451881362.30.353BE1A Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf04.hostedemail.com (Postfix) with ESMTP id A492C400A8 for ; Wed, 11 May 2022 02:29:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652236180; x=1683772180; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=YJmOMZm2CAZNr36cdPGE67zJRph2jZlNrnhMlnzgytk=; b=RHZkyObIS1olfUz03FKB11ITQ3oGVRSxZPbKq8c8r37QeVpAN+fU+d5p PesBZjq69Dw46M65z7LmwZL2IPupkuVR4jOlSzAid0jnc7+t6j3SfkgDD v9NDwFouCGiqvYtSmcTDMif+vawFb+QGL+OJK4ANteiCpq/AGF6PrvDNk sUdMY2NuDVOjuc62es1BYYyU8NpYwkjead6pTB4mDlMMtVA7FoVwcK3fv r46a/XPx/A6EBU64V26WLsSanA+JyKh1ALutWG97zy4E3CIiXZj3HmM9e PSxoe8EIjFh1uGKVArMN1qXArXTYr8Zz9LR07JZsyJ9HbHgQQGtljIaCp A==; X-IronPort-AV: E=McAfee;i="6400,9594,10343"; a="294798589" X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="294798589" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 May 2022 19:29:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="895145507" Received: from black.fi.intel.com ([10.237.72.28]) by fmsmga005.fm.intel.com with ESMTP; 10 May 2022 19:29:36 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 1CC52530; Wed, 11 May 2022 05:28:01 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Andrey Ryabinin , Alexander Potapenko , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [RFCv2 01/10] x86/mm: Fix CR3_ADDR_MASK Date: Wed, 11 May 2022 05:27:42 +0300 Message-Id: <20220511022751.65540-3-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> References: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: jztn3urqriimzxpxxbk8uzj1ngxdsqep X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: A492C400A8 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=RHZkyObI; spf=none (imf04.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com X-Rspam-User: X-HE-Tag: 1652236170-712432 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The mask must not include bits above physical address mask. These bits are reserved and can be used for other things. Bits 61 and 62 are used for Linear Address Masking. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/processor-flags.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/include/asm/processor-flags.h b/arch/x86/include/asm/processor-flags.h index 02c2cbda4a74..a7f3d9100adb 100644 --- a/arch/x86/include/asm/processor-flags.h +++ b/arch/x86/include/asm/processor-flags.h @@ -35,7 +35,7 @@ */ #ifdef CONFIG_X86_64 /* Mask off the address space ID and SME encryption bits. */ -#define CR3_ADDR_MASK __sme_clr(0x7FFFFFFFFFFFF000ull) +#define CR3_ADDR_MASK __sme_clr(PHYSICAL_PAGE_MASK) #define CR3_PCID_MASK 0xFFFull #define CR3_NOFLUSH BIT_ULL(63) From patchwork Wed May 11 02:27:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12845709 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5603BC433FE for ; Wed, 11 May 2022 02:29:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B6C088D0001; Tue, 10 May 2022 22:29:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF61D8D0002; Tue, 10 May 2022 22:29:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 991E28D0001; Tue, 10 May 2022 22:29:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7C9806B007B for ; Tue, 10 May 2022 22:29:43 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4FBAE1D8F for ; Wed, 11 May 2022 02:29:43 +0000 (UTC) X-FDA: 79451881446.30.BFC1061 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by imf23.hostedemail.com (Postfix) with ESMTP id 4696714009B for ; Wed, 11 May 2022 02:29:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652236182; x=1683772182; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zyE/+02x9Fj+FgL1GeYtDyYutpy6orbTRtxDnBCOxV4=; b=QqjyEsgdqsP0AmMZ/w9wfk+RTFEiuMcBQEirhLEf1ltDO599fQwAjmsw 3VkZksFfs3C2KsDTmPk4qI9/27bbz2kcbbtLRLacAat1N0a8SkkEP+4gT 9yVDdmaQwWsHjj6NxBTiFzCOvcG2b2xsvLoSf5UKNp2aaRD6MhFrUDY3d U2uaP7rj5Bm7RLdrmvtdRCLk7DpmvHUqFqZe/rofU4Q2FPL2B5KpAuOD/ JZ93XYRGVFEYPXUm4sUDm8104nLYcd7ULSfHe+Bs5KWcMq0kX2LCo2a3T 0SQQ7L1/AB0gbHJxvejjrTi2bJV73JshA8Z/kUta+9SeVxQlTSH82O0wj Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10343"; a="257100662" X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="257100662" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 May 2022 19:29:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="542092629" Received: from black.fi.intel.com ([10.237.72.28]) by orsmga006.jf.intel.com with ESMTP; 10 May 2022 19:29:36 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 2444753B; Wed, 11 May 2022 05:28:01 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Andrey Ryabinin , Alexander Potapenko , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [RFCv2 02/10] x86: CPUID and CR3/CR4 flags for Linear Address Masking Date: Wed, 11 May 2022 05:27:43 +0300 Message-Id: <20220511022751.65540-4-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> References: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: 4z31s757dkx3uq66mwnpya6hg4wpdys3 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=QqjyEsgd; spf=none (imf23.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.20) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 4696714009B X-HE-Tag: 1652236169-836334 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Enumerate Linear Address Masking and provide defines for CR3 and CR4 flags. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/uapi/asm/processor-flags.h | 6 ++++++ 2 files changed, 7 insertions(+) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 73e643ae94b6..d443d1ba231a 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -299,6 +299,7 @@ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */ #define X86_FEATURE_AVX512_BF16 (12*32+ 5) /* AVX512 BFLOAT16 instructions */ +#define X86_FEATURE_LAM (12*32+26) /* Linear Address Masking */ /* AMD-defined CPU features, CPUID level 0x80000008 (EBX), word 13 */ #define X86_FEATURE_CLZERO (13*32+ 0) /* CLZERO instruction */ diff --git a/arch/x86/include/uapi/asm/processor-flags.h b/arch/x86/include/uapi/asm/processor-flags.h index c47cc7f2feeb..d898432947ff 100644 --- a/arch/x86/include/uapi/asm/processor-flags.h +++ b/arch/x86/include/uapi/asm/processor-flags.h @@ -82,6 +82,10 @@ #define X86_CR3_PCID_BITS 12 #define X86_CR3_PCID_MASK (_AC((1UL << X86_CR3_PCID_BITS) - 1, UL)) +#define X86_CR3_LAM_U57_BIT 61 /* Activate LAM for userspace, 62:57 bits masked */ +#define X86_CR3_LAM_U57 _BITULL(X86_CR3_LAM_U57_BIT) +#define X86_CR3_LAM_U48_BIT 62 /* Activate LAM for userspace, 62:48 bits masked */ +#define X86_CR3_LAM_U48 _BITULL(X86_CR3_LAM_U48_BIT) #define X86_CR3_PCID_NOFLUSH_BIT 63 /* Preserve old PCID */ #define X86_CR3_PCID_NOFLUSH _BITULL(X86_CR3_PCID_NOFLUSH_BIT) @@ -132,6 +136,8 @@ #define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT) #define X86_CR4_CET_BIT 23 /* enable Control-flow Enforcement Technology */ #define X86_CR4_CET _BITUL(X86_CR4_CET_BIT) +#define X86_CR4_LAM_SUP_BIT 28 /* LAM for supervisor pointers */ +#define X86_CR4_LAM_SUP _BITUL(X86_CR4_LAM_SUP_BIT) /* * x86-64 Task Priority Register, CR8 From patchwork Wed May 11 02:27:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12845707 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E077C433EF for ; Wed, 11 May 2022 02:29:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D19606B0072; Tue, 10 May 2022 22:29:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C7D526B0075; Tue, 10 May 2022 22:29:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA52D6B0078; Tue, 10 May 2022 22:29:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 82D616B0072 for ; Tue, 10 May 2022 22:29:42 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 5F2A531EA3 for ; Wed, 11 May 2022 02:29:42 +0000 (UTC) X-FDA: 79451881404.30.0570540 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf04.hostedemail.com (Postfix) with ESMTP id B2591400A8 for ; Wed, 11 May 2022 02:29:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652236181; x=1683772181; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=5G0RO9Jzv5sMl5l3KPmaPojaLlcXXs9/0fmVZgdGWDk=; b=hUzGlv/7nDVF92u2DWWTULLys6Jf/ckI1mEeQDLKyrIIPdizqY9KszfZ oUhUm616/8bDjCTXvfeh2dwMY+16Kt+dmMonv5/nFxytYIrFCAh1FwW5S bChzXIKWN5s1SUwu04s4PB1ty1sOCIzBogLqvOfMQcDckJsHZE9yUCni8 6pRuEya1LtcyjvqEG3kSJp+hgp5kkHjRAeZzzoDh1B6HloLvTsbuw67CA blBedDL8DT3wO3d/MpUT2RTtl1Iey7JhpfY6/0TSi9hJSQVq2y8mZJRM6 sWGIuJc88kLUI8lvNa9BOeldjPCMvGzOX8SNEMcrYHHXAz10o63vWyJHb w==; X-IronPort-AV: E=McAfee;i="6400,9594,10343"; a="294798591" X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="294798591" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 May 2022 19:29:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="895145510" Received: from black.fi.intel.com ([10.237.72.28]) by fmsmga005.fm.intel.com with ESMTP; 10 May 2022 19:29:36 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 2BB1B556; Wed, 11 May 2022 05:28:01 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Andrey Ryabinin , Alexander Potapenko , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [RFCv2 03/10] x86: Introduce userspace API to handle per-thread features Date: Wed, 11 May 2022 05:27:44 +0300 Message-Id: <20220511022751.65540-5-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> References: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: 51kmpxgaskkpb4wbpsmdaue19ew94gsd X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: B2591400A8 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="hUzGlv/7"; spf=none (imf04.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com X-Rspam-User: X-HE-Tag: 1652236171-541030 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add three new arch_prctl() handles: - ARCH_THREAD_FEATURE_ENABLE/DISABLE enables or disables the specified features. Returns what features are enabled after the operation. - ARCH_THREAD_FEATURE_LOCK prevents future disabling or enabling of the specified features. Returns the new set of locked features. The features handled per-thread and inherited over fork(2)/clone(2), but reset on exec(). This is preparation patch. It does not impelement any features. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/processor.h | 3 +++ arch/x86/include/uapi/asm/prctl.h | 5 +++++ arch/x86/kernel/process.c | 37 +++++++++++++++++++++++++++++++ 3 files changed, 45 insertions(+) diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index 91d0f93a00c7..ff0c34e18cc6 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -530,6 +530,9 @@ struct thread_struct { */ u32 pkru; + unsigned long features; + unsigned long features_locked; + /* Floating point and extended processor state */ struct fpu fpu; /* diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 500b96e71f18..67fc30d36c73 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -20,4 +20,9 @@ #define ARCH_MAP_VDSO_32 0x2002 #define ARCH_MAP_VDSO_64 0x2003 +/* Never implement 0x3001, it will confuse old glibc's */ +#define ARCH_THREAD_FEATURE_ENABLE 0x3002 +#define ARCH_THREAD_FEATURE_DISABLE 0x3003 +#define ARCH_THREAD_FEATURE_LOCK 0x3004 + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index b370767f5b19..cb8fc28f2eae 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -367,6 +367,10 @@ void arch_setup_new_exec(void) task_clear_spec_ssb_noexec(current); speculation_ctrl_update(read_thread_flags()); } + + /* Reset thread features on exec */ + current->thread.features = 0; + current->thread.features_locked = 0; } #ifdef CONFIG_X86_IOPL_IOPERM @@ -985,6 +989,35 @@ unsigned long __get_wchan(struct task_struct *p) return addr; } +static long thread_feature_prctl(struct task_struct *task, int option, + unsigned long features) +{ + const unsigned long known_features = 0; + + if (features & ~known_features) + return -EINVAL; + + if (option == ARCH_THREAD_FEATURE_LOCK) { + task->thread.features_locked |= features; + return task->thread.features_locked; + } + + /* Do not allow to change locked features */ + if (features & task->thread.features_locked) + return -EPERM; + + if (option == ARCH_THREAD_FEATURE_DISABLE) { + task->thread.features &= ~features; + goto out; + } + + /* Handle ARCH_THREAD_FEATURE_ENABLE */ + + task->thread.features |= features; +out: + return task->thread.features; +} + long do_arch_prctl_common(struct task_struct *task, int option, unsigned long arg2) { @@ -999,6 +1032,10 @@ long do_arch_prctl_common(struct task_struct *task, int option, case ARCH_GET_XCOMP_GUEST_PERM: case ARCH_REQ_XCOMP_GUEST_PERM: return fpu_xstate_prctl(task, option, arg2); + case ARCH_THREAD_FEATURE_ENABLE: + case ARCH_THREAD_FEATURE_DISABLE: + case ARCH_THREAD_FEATURE_LOCK: + return thread_feature_prctl(task, option, arg2); } return -EINVAL; From patchwork Wed May 11 02:27:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12845713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21A86C433FE for ; Wed, 11 May 2022 02:29:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C847F8D0002; Tue, 10 May 2022 22:29:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C0C6C8D0009; Tue, 10 May 2022 22:29:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 575F88D0002; Tue, 10 May 2022 22:29:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2353D8D0006 for ; Tue, 10 May 2022 22:29:46 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 0441761B91 for ; Wed, 11 May 2022 02:29:45 +0000 (UTC) X-FDA: 79451881572.16.D0ADBDE Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by imf15.hostedemail.com (Postfix) with ESMTP id 1F5F1A0076 for ; Wed, 11 May 2022 02:29:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652236185; x=1683772185; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LZG1l0QEfB5T0caqDjpXl8PdwUwKifRjuJfvSf4teKw=; b=eTRxRBwDcfIC75w8YocWz1LJze1RV0jyR3OgmLUdevnS+/aZoAy93ft0 f0QqsIzBrv1t/7VXPGUTpdhaDXonc0LZ1IlX60sKbpPCb3W8Nv8hqfhqX ubihd++3v6UuUqkJ6trNOV79Vyanzw8Mc+1mJ/QKLtSr+3DTgXnPmwJsK zdqcz+zoX6PvrfYMgAvW9vaVG59n+NtOvIwR6U39QEaRmFZ5MBHXgo01O LrF4bfVB11RVljLoZ/+PBR9QtIFN4i9jAHQfWGhhR+EXQC/iWuOcxhlGl zGkKqSfS6TZhpZcdWX9ccjYyUmeCAn5M00ro4rBR1XbHrjA/CTVnJp/LY w==; X-IronPort-AV: E=McAfee;i="6400,9594,10343"; a="332596472" X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="332596472" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 May 2022 19:29:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="636218430" Received: from black.fi.intel.com ([10.237.72.28]) by fmsmga004.fm.intel.com with ESMTP; 10 May 2022 19:29:40 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 3502357E; Wed, 11 May 2022 05:28:01 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Andrey Ryabinin , Alexander Potapenko , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [RFCv2 04/10] x86/mm: Introduce X86_THREAD_LAM_U48 and X86_THREAD_LAM_U57 Date: Wed, 11 May 2022 05:27:45 +0300 Message-Id: <20220511022751.65540-6-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> References: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 1F5F1A0076 X-Stat-Signature: thjnfz4o3h5yup95arohzc81wjdue817 Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=eTRxRBwD; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf15.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.100) smtp.mailfrom=kirill.shutemov@linux.intel.com X-Rspam-User: X-HE-Tag: 1652236172-329207 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Linear Address Masking mode for userspace pointers encoded in CR3 bits. The mode is selected per-thread. Add new thread features indicate that the thread has Linear Address Masking enabled. switch_mm_irqs_off() now respects these flags and constructs CR3 accordingly. The active LAM mode gets recorded in the tlb_state. The thread features are not yet exposed via userpsace API. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/tlbflush.h | 5 ++ arch/x86/include/uapi/asm/prctl.h | 3 + arch/x86/mm/tlb.c | 95 ++++++++++++++++++++++++++----- 3 files changed, 88 insertions(+), 15 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 98fa0a114074..77cae8623858 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -17,6 +17,10 @@ void __flush_tlb_all(void); #define TLB_FLUSH_ALL -1UL +#define LAM_NONE 0 +#define LAM_U57 1 +#define LAM_U48 2 + void cr4_update_irqsoff(unsigned long set, unsigned long clear); unsigned long cr4_read_shadow(void); @@ -88,6 +92,7 @@ struct tlb_state { u16 loaded_mm_asid; u16 next_asid; + u8 lam; /* * If set we changed the page tables in such a way that we diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 67fc30d36c73..2dd16472d078 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -25,4 +25,7 @@ #define ARCH_THREAD_FEATURE_DISABLE 0x3003 #define ARCH_THREAD_FEATURE_LOCK 0x3004 +#define X86_THREAD_LAM_U48 0x1 +#define X86_THREAD_LAM_U57 0x2 + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 6eb4d91d5365..f9fe71d1f42c 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -18,6 +18,7 @@ #include #include #include +#include #include "mm_internal.h" @@ -154,17 +155,72 @@ static inline u16 user_pcid(u16 asid) return ret; } -static inline unsigned long build_cr3(pgd_t *pgd, u16 asid) +#ifdef CONFIG_X86_64 +static inline unsigned long lam_to_cr3(u8 lam) +{ + switch (lam) { + case LAM_NONE: + return 0; + case LAM_U57: + return X86_CR3_LAM_U57; + case LAM_U48: + return X86_CR3_LAM_U48; + default: + WARN_ON_ONCE(1); + return 0; + } +} + +static inline u8 cr3_to_lam(unsigned long cr3) +{ + if (cr3 & X86_CR3_LAM_U57) + return LAM_U57; + if (cr3 & X86_CR3_LAM_U48) + return LAM_U48; + return 0; +} + +static u8 gen_lam(struct task_struct *tsk, struct mm_struct *mm) +{ + if (!tsk) + return LAM_NONE; + + if (tsk->thread.features & X86_THREAD_LAM_U57) + return LAM_U57; + if (tsk->thread.features & X86_THREAD_LAM_U48) + return LAM_U48; + return LAM_NONE; +} + +#else + +static inline unsigned long lam_to_cr3(u8 lam) +{ + return 0; +} + +static inline u8 cr3_to_lam(unsigned long cr3) +{ + return LAM_NONE; +} + +static u8 gen_lam(struct task_struct *tsk, struct mm_struct *mm) +{ + return LAM_NONE; +} +#endif + +static inline unsigned long build_cr3(pgd_t *pgd, u16 asid, u8 lam) { if (static_cpu_has(X86_FEATURE_PCID)) { - return __sme_pa(pgd) | kern_pcid(asid); + return __sme_pa(pgd) | kern_pcid(asid) | lam_to_cr3(lam); } else { VM_WARN_ON_ONCE(asid != 0); - return __sme_pa(pgd); + return __sme_pa(pgd) | lam_to_cr3(lam); } } -static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid) +static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid, u8 lam) { VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); /* @@ -173,7 +229,7 @@ static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid) * boot because all CPU's the have same capabilities: */ VM_WARN_ON_ONCE(!boot_cpu_has(X86_FEATURE_PCID)); - return __sme_pa(pgd) | kern_pcid(asid) | CR3_NOFLUSH; + return __sme_pa(pgd) | kern_pcid(asid) | CR3_NOFLUSH | lam_to_cr3(lam); } /* @@ -274,15 +330,15 @@ static inline void invalidate_user_asid(u16 asid) (unsigned long *)this_cpu_ptr(&cpu_tlbstate.user_pcid_flush_mask)); } -static void load_new_mm_cr3(pgd_t *pgdir, u16 new_asid, bool need_flush) +static void load_new_mm_cr3(pgd_t *pgdir, u16 new_asid, u8 lam, bool need_flush) { unsigned long new_mm_cr3; if (need_flush) { invalidate_user_asid(new_asid); - new_mm_cr3 = build_cr3(pgdir, new_asid); + new_mm_cr3 = build_cr3(pgdir, new_asid, lam); } else { - new_mm_cr3 = build_cr3_noflush(pgdir, new_asid); + new_mm_cr3 = build_cr3_noflush(pgdir, new_asid, lam); } /* @@ -491,6 +547,8 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, { struct mm_struct *real_prev = this_cpu_read(cpu_tlbstate.loaded_mm); u16 prev_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); + u8 prev_lam = this_cpu_read(cpu_tlbstate.lam); + u8 new_lam = gen_lam(tsk, next); bool was_lazy = this_cpu_read(cpu_tlbstate_shared.is_lazy); unsigned cpu = smp_processor_id(); u64 next_tlb_gen; @@ -504,6 +562,9 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * cpu_tlbstate.loaded_mm) matches next. * * NB: leave_mm() calls us with prev == NULL and tsk == NULL. + * + * NB: Initial LAM enabling calls us with prev == next. We must update + * CR3 if prev_lam doesn't match the new one. */ /* We don't want flush_tlb_func() to run concurrently with us. */ @@ -520,7 +581,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * isn't free. */ #ifdef CONFIG_DEBUG_VM - if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid))) { + if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid, prev_lam))) { /* * If we were to BUG here, we'd be very likely to kill * the system so hard that we don't see the call trace. @@ -551,7 +612,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * provides that full memory barrier and core serializing * instruction. */ - if (real_prev == next) { + if (real_prev == next && prev_lam == new_lam) { VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) != next->context.ctx_id); @@ -622,15 +683,16 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, barrier(); } + this_cpu_write(cpu_tlbstate.lam, new_lam); if (need_flush) { this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen); - load_new_mm_cr3(next->pgd, new_asid, true); + load_new_mm_cr3(next->pgd, new_asid, new_lam, true); trace_tlb_flush(TLB_FLUSH_ON_TASK_SWITCH, TLB_FLUSH_ALL); } else { /* The new ASID is already up to date. */ - load_new_mm_cr3(next->pgd, new_asid, false); + load_new_mm_cr3(next->pgd, new_asid, new_lam, false); trace_tlb_flush(TLB_FLUSH_ON_TASK_SWITCH, 0); } @@ -687,6 +749,7 @@ void initialize_tlbstate_and_flush(void) struct mm_struct *mm = this_cpu_read(cpu_tlbstate.loaded_mm); u64 tlb_gen = atomic64_read(&init_mm.context.tlb_gen); unsigned long cr3 = __read_cr3(); + u8 lam = cr3_to_lam(cr3); /* Assert that CR3 already references the right mm. */ WARN_ON((cr3 & CR3_ADDR_MASK) != __pa(mm->pgd)); @@ -700,7 +763,7 @@ void initialize_tlbstate_and_flush(void) !(cr4_read_shadow() & X86_CR4_PCIDE)); /* Force ASID 0 and force a TLB flush. */ - write_cr3(build_cr3(mm->pgd, 0)); + write_cr3(build_cr3(mm->pgd, 0, lam)); /* Reinitialize tlbstate. */ this_cpu_write(cpu_tlbstate.last_user_mm_spec, LAST_USER_MM_INIT); @@ -1074,8 +1137,10 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) */ unsigned long __get_current_cr3_fast(void) { - unsigned long cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, - this_cpu_read(cpu_tlbstate.loaded_mm_asid)); + unsigned long cr3 = + build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, + this_cpu_read(cpu_tlbstate.loaded_mm_asid), + this_cpu_read(cpu_tlbstate.lam)); /* For now, be very restrictive about when this can be called. */ VM_WARN_ON(in_nmi() || preemptible()); From patchwork Wed May 11 02:27:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12845710 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B86EFC433EF for ; Wed, 11 May 2022 02:29:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7EA428D0003; Tue, 10 May 2022 22:29:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7239F8D0002; Tue, 10 May 2022 22:29:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C4AC8D0003; Tue, 10 May 2022 22:29:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 34F0B8D0002 for ; Tue, 10 May 2022 22:29:45 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 09C3D21E53 for ; Wed, 11 May 2022 02:29:45 +0000 (UTC) X-FDA: 79451881530.14.54A40D5 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by imf15.hostedemail.com (Postfix) with ESMTP id 10F9BA0076 for ; Wed, 11 May 2022 02:29:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652236184; x=1683772184; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=7o08EsiH9rBqx3Xez/FIIKap6Mag0l46E+Gl2UX4hIw=; b=UzZn7s/P8RN57/5zVwYpv/HknYk7CllToKb1UfJkquUmuP+s2NXUeQ78 Nf6NwB8SYBy0x70fS4qm6um5EVHnV+ji+9H9wQ7igykxsZAB/E8/tnsut lZ/ZTCcD6L8XOstKHaBygS3wgVZA7UYTBEHXFAx0AAe7jZrG1kPF0k1kM Nfgb7V9LiMU33kNUArGE5SznQoLx1wTHGmG41cOUlQrpjyyGOcarGsagg Zhr1PJIopzmKVByJEpQOJat2NxINvcNzwDv4ql8bXCsudUPgSHksLTRUH KS+JJ/NnRAok9RqHzqQh21dTWOVlcJ21lF6Sdr36Za0xpTxpAMQ4BAf/W g==; X-IronPort-AV: E=McAfee;i="6400,9594,10343"; a="332596469" X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="332596469" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 May 2022 19:29:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="636218429" Received: from black.fi.intel.com ([10.237.72.28]) by fmsmga004.fm.intel.com with ESMTP; 10 May 2022 19:29:40 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 3AAF1512; Wed, 11 May 2022 05:28:01 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Andrey Ryabinin , Alexander Potapenko , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [RFCv2 05/10] x86/mm: Provide untagged_addr() helper Date: Wed, 11 May 2022 05:27:46 +0300 Message-Id: <20220511022751.65540-7-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> References: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 10F9BA0076 X-Stat-Signature: g64tky9ciyndw7xb435zebt8neoob3g5 Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="UzZn7s/P"; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf15.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.100) smtp.mailfrom=kirill.shutemov@linux.intel.com X-Rspam-User: X-HE-Tag: 1652236171-393927 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The helper used by the core-mm to strip tag bits and get the address to the canonical shape. In only handles userspace addresses. For LAM, the address gets sanitized according to the thread features. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/page_32.h | 3 +++ arch/x86/include/asm/page_64.h | 20 ++++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/arch/x86/include/asm/page_32.h b/arch/x86/include/asm/page_32.h index df42f8aa99e4..2d35059b90c1 100644 --- a/arch/x86/include/asm/page_32.h +++ b/arch/x86/include/asm/page_32.h @@ -15,6 +15,9 @@ extern unsigned long __phys_addr(unsigned long); #define __phys_addr_symbol(x) __phys_addr(x) #define __phys_reloc_hide(x) RELOC_HIDE((x), 0) +#define untagged_addr(addr) (addr) +#define untagged_ptr(ptr) (ptr) + #ifdef CONFIG_FLATMEM #define pfn_valid(pfn) ((pfn) < max_mapnr) #endif /* CONFIG_FLATMEM */ diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index e9c86299b835..3a40c958b24a 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -7,6 +7,7 @@ #ifndef __ASSEMBLY__ #include #include +#include /* duplicated to the one in bootmem.h */ extern unsigned long max_pfn; @@ -90,6 +91,25 @@ static __always_inline unsigned long task_size_max(void) } #endif /* CONFIG_X86_5LEVEL */ +#define __untagged_addr(addr, n) \ + ((__force __typeof__(addr))sign_extend64((__force u64)(addr), n)) + +#define untagged_addr(addr) ({ \ + u64 __addr = (__force u64)(addr); \ + if (__addr >> 63 == 0) { \ + if (current->thread.features & X86_THREAD_LAM_U57) \ + __addr &= __untagged_addr(__addr, 56); \ + else if (current->thread.features & X86_THREAD_LAM_U48) \ + __addr &= __untagged_addr(__addr, 47); \ + } \ + (__force __typeof__(addr))__addr; \ +}) + +#define untagged_ptr(ptr) ({ \ + u64 __ptrval = (__force u64)(ptr); \ + __ptrval = untagged_addr(__ptrval); \ + (__force __typeof__(*(ptr)) *)__ptrval; \ +}) #endif /* !__ASSEMBLY__ */ #ifdef CONFIG_X86_VSYSCALL_EMULATION From patchwork Wed May 11 02:27:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12845711 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C55DC433F5 for ; Wed, 11 May 2022 02:29:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6617A8D0007; Tue, 10 May 2022 22:29:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 613EE8D0006; Tue, 10 May 2022 22:29:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E08C8D0007; Tue, 10 May 2022 22:29:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F05F18D0005 for ; Tue, 10 May 2022 22:29:45 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D2F9D31B92 for ; Wed, 11 May 2022 02:29:45 +0000 (UTC) X-FDA: 79451881530.21.6899294 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf24.hostedemail.com (Postfix) with ESMTP id 0B3021800C5 for ; Wed, 11 May 2022 02:29:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652236185; x=1683772185; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tVhiT6aKeF/Rpr/dJksjmlkONreh2kMGvoLBNiaVHv4=; b=inUG4Fo9T8n2IXNzWhSN42BEKMes4XD1McmJGvVQhDu89f5nyCnYy7G3 ivPOzUq0Fag1USTbrZvv0zq8EBfxRxrtTQ6y0Kg+aSSEuubGoSCbS9l9m wFGuhZJln56ozksF+3AvXXkvi1BwEC8XAkrYn1lIXfiqJBG8mGeKqAMrE XpGZUEHq1/EU3ZaLDnb9q1qATnMVIfFYNxE+KRNUcZ/3sVm7SWAbQKHFp h/WXUFwoZL5g8ZbPJzEMIod/KVO5KDJbjr7hw8O4rsMXc6otHSYT28S+5 TGw9n/XozBOGOln9ROPZJrpOnHoe2tsttlseP+Y9yvWS4VTpTsCv0uFod A==; X-IronPort-AV: E=McAfee;i="6400,9594,10343"; a="251610749" X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="251610749" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 May 2022 19:29:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="553112425" Received: from black.fi.intel.com ([10.237.72.28]) by orsmga002.jf.intel.com with ESMTP; 10 May 2022 19:29:40 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 495565C1; Wed, 11 May 2022 05:28:01 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Andrey Ryabinin , Alexander Potapenko , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [RFCv2 06/10] x86/uaccess: Remove tags from the address before checking Date: Wed, 11 May 2022 05:27:47 +0300 Message-Id: <20220511022751.65540-8-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> References: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 0B3021800C5 X-Stat-Signature: amj6zo945i7chsyw7oyn43ratfjrajaz X-Rspam-User: Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=inUG4Fo9; spf=none (imf24.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.126) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com X-HE-Tag: 1652236176-991342 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The tags must not be included into check whether it's okay to access the userspace address. Strip tags in access_ok(). get_user() and put_user() don't use access_ok(), but check access against TASK_SIZE directly in assembly. Strip tags, before calling into the assembly helper. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/uaccess.h | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index f78e2b3501a1..0f5bf7db4ec9 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -40,7 +40,7 @@ static inline bool pagefault_disabled(void); #define access_ok(addr, size) \ ({ \ WARN_ON_IN_IRQ(); \ - likely(__access_ok(addr, size)); \ + likely(__access_ok(untagged_addr(addr), size)); \ }) #include @@ -125,7 +125,12 @@ extern int __get_user_bad(void); * Return: zero on success, or -EFAULT on error. * On error, the variable @x is set to zero. */ -#define get_user(x,ptr) ({ might_fault(); do_get_user_call(get_user,x,ptr); }) +#define get_user(x,ptr) \ +({ \ + __typeof__(*(ptr)) __user *__ptr_clean = untagged_ptr(ptr); \ + might_fault(); \ + do_get_user_call(get_user,x,__ptr_clean); \ +}) /** * __get_user - Get a simple variable from user space, with less checking. @@ -222,7 +227,11 @@ extern void __put_user_nocheck_8(void); * * Return: zero on success, or -EFAULT on error. */ -#define put_user(x, ptr) ({ might_fault(); do_put_user_call(put_user,x,ptr); }) +#define put_user(x, ptr) ({ \ + __typeof__(*(ptr)) __user *__ptr_clean = untagged_ptr(ptr); \ + might_fault(); \ + do_put_user_call(put_user,x,__ptr_clean); \ +}) /** * __put_user - Write a simple value into user space, with less checking. From patchwork Wed May 11 02:27:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12845712 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D19C1C433EF for ; Wed, 11 May 2022 02:29:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8E61E8D0005; Tue, 10 May 2022 22:29:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 83DC38D0008; Tue, 10 May 2022 22:29:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 41A9D8D0005; Tue, 10 May 2022 22:29:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E983E8D0002 for ; Tue, 10 May 2022 22:29:45 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C4AF321E53 for ; Wed, 11 May 2022 02:29:45 +0000 (UTC) X-FDA: 79451881530.23.0FEC080 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf01.hostedemail.com (Postfix) with ESMTP id 588F040003 for ; Wed, 11 May 2022 02:29:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652236185; x=1683772185; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=/zG/vU7kvADc4pg2W2FY9O2Nfewu2j+VHOpdujenCaM=; b=UKYgYJxUA9fqGDMj1ovh/I1m14zv5lP9BipMEuTq377VQGBF1mfngEgP VQoFT3ewGpIN4Tb3huTrcZ8X31b7uj0OTDGVxJxlHx11lF2lpEm4cE0Fl qXHjygJfnvVvlqP4Z/35RDj2W4mF5nAA6HHKEH5Qj0K4tA4NislvuT5n5 NvmYe9Z1bYMB0L2Bwi9M3gmfLjk8CI/WbeoWgK0cBrRiJ/WEgEonpw0nU 17ec7j/fzVE7h2dNwV0nKWsnV+lu124N5E+74YcbvziT4bpIcTaI+CEaX YVgBjKQbHWD+aWOzLyZ5mlAADAABtKjXKW5+MmS/w6DiTm38FG10UhYzp g==; X-IronPort-AV: E=McAfee;i="6400,9594,10343"; a="269695078" X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="269695078" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 May 2022 19:29:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="670166554" Received: from black.fi.intel.com ([10.237.72.28]) by fmsmga002.fm.intel.com with ESMTP; 10 May 2022 19:29:40 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 53BBA5D0; Wed, 11 May 2022 05:28:01 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Andrey Ryabinin , Alexander Potapenko , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [RFCv2 07/10] x86/mm: Handle tagged memory accesses from kernel threads Date: Wed, 11 May 2022 05:27:48 +0300 Message-Id: <20220511022751.65540-9-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> References: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 588F040003 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=UKYgYJxU; spf=none (imf01.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.115) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com X-Rspam-User: X-Stat-Signature: zpt5x4bnmkekahq5ogb4upbyn8hz8tof X-HE-Tag: 1652236173-869155 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a kernel thread performs memory access on behalf of a process (like in async I/O, io_uring, etc.) it has to respect tagging setup of the process as user addresses can include tags. Normally, LAM setup is per-thread and recorded in thread features, but for this use case kernel also tracks LAM setup per-mm. mm->context.lam would record LAM that allows the most tag bits among the threads of the mm. The info used by switch_mm_irqs_off() to construct CR3 if the task is kernel thread. Thread featrues of the kernel thread get updated according to mm->context.lam. It allows untagged_addr() to work correctly. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/mmu.h | 1 + arch/x86/mm/tlb.c | 28 ++++++++++++++++++++++++++++ 2 files changed, 29 insertions(+) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 5d7494631ea9..52f3749f14e8 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -40,6 +40,7 @@ typedef struct { #ifdef CONFIG_X86_64 unsigned short flags; + u8 lam; #endif struct mutex lock; diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index f9fe71d1f42c..b320556e1c22 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -185,6 +185,34 @@ static u8 gen_lam(struct task_struct *tsk, struct mm_struct *mm) if (!tsk) return LAM_NONE; + if (tsk->flags & PF_KTHREAD) { + /* + * For kernel thread use the most permissive LAM + * used by the mm. It's required to handle kernel thread + * memory accesses on behalf of a process. + * + * Adjust thread flags accodringly, so untagged_addr() would + * work correctly. + */ + + tsk->thread.features &= ~(X86_THREAD_LAM_U48 | + X86_THREAD_LAM_U57); + + switch (mm->context.lam) { + case LAM_NONE: + return LAM_NONE; + case LAM_U57: + tsk->thread.features |= X86_THREAD_LAM_U57; + return LAM_U57; + case LAM_U48: + tsk->thread.features |= X86_THREAD_LAM_U48; + return LAM_U48; + default: + WARN_ON_ONCE(1); + return LAM_NONE; + } + } + if (tsk->thread.features & X86_THREAD_LAM_U57) return LAM_U57; if (tsk->thread.features & X86_THREAD_LAM_U48) From patchwork Wed May 11 02:27:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12845715 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6673C433FE for ; Wed, 11 May 2022 02:29:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 83A088D0008; Tue, 10 May 2022 22:29:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7BFB18D0006; Tue, 10 May 2022 22:29:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 466868D0008; Tue, 10 May 2022 22:29:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E3DC98D0008 for ; Tue, 10 May 2022 22:29:46 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B4DAD314E3 for ; Wed, 11 May 2022 02:29:46 +0000 (UTC) X-FDA: 79451881572.26.1DBC803 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf24.hostedemail.com (Postfix) with ESMTP id 0745B1800CA for ; Wed, 11 May 2022 02:29:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652236186; x=1683772186; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=yh4/dFHOtWSE9iF+GUg2XMGu0/7JiKaWrQ/zWPdOw6Y=; b=k6a2nZL2tnpUuYiJrKMup2AnSvsPqRCdw2L/udTd7LrhmtC2l958u0Ad IT9JgqonLJzdoor0RfQDqFOOfuxLgZ/mZ75TsV+WBJ55FaJZDc3Ok6Y6K XoyXgQg+QsTA2Qt9xHjodhCvMnMTyV35QTTLhnnQTSZDg9pePIFpU7n0D gg2U+oHzqWCJ1l20hi3XUaS25CJiYEIAi38wJr8oPH1gJsYtcBWn9zju7 iJrVKKoDNdnjqAum7lVkKb2lbkWy9q6yBeQI2eEKr5sWHlmoJKH/JLZrc 1tuVxW6xrfM6cKQ6kx9/6S8KNhEGU9jUtwdalfsUC6NS8HstC+nqggVSd A==; X-IronPort-AV: E=McAfee;i="6400,9594,10343"; a="251610748" X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="251610748" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 May 2022 19:29:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="711294411" Received: from black.fi.intel.com ([10.237.72.28]) by fmsmga001.fm.intel.com with ESMTP; 10 May 2022 19:29:40 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 5EA78630; Wed, 11 May 2022 05:28:01 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Andrey Ryabinin , Alexander Potapenko , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [RFCv2 08/10] x86/mm: Make LAM_U48 and mappings above 47-bits mutually exclusive Date: Wed, 11 May 2022 05:27:49 +0300 Message-Id: <20220511022751.65540-10-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> References: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 0745B1800CA X-Stat-Signature: hb1pfpef5714a5fgzxmgg6pj6txqce47 X-Rspam-User: Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=k6a2nZL2; spf=none (imf24.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.126) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com X-HE-Tag: 1652236177-735072 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: LAM_U48 steals bits above 47-bit for tags and makes it impossible for userspace to use full address space on 5-level paging machine. Make these features mutually exclusive: whichever gets enabled first blocks the othe one. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/elf.h | 3 ++- arch/x86/include/asm/mmu_context.h | 13 +++++++++++++ arch/x86/kernel/sys_x86_64.c | 5 +++-- arch/x86/mm/hugetlbpage.c | 6 ++++-- arch/x86/mm/mmap.c | 9 ++++++++- 5 files changed, 30 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/elf.h b/arch/x86/include/asm/elf.h index 29fea180a665..53b96b0c8cc3 100644 --- a/arch/x86/include/asm/elf.h +++ b/arch/x86/include/asm/elf.h @@ -328,7 +328,8 @@ static inline int mmap_is_ia32(void) extern unsigned long task_size_32bit(void); extern unsigned long task_size_64bit(int full_addr_space); extern unsigned long get_mmap_base(int is_legacy); -extern bool mmap_address_hint_valid(unsigned long addr, unsigned long len); +extern bool mmap_address_hint_valid(struct mm_struct *mm, + unsigned long addr, unsigned long len); extern unsigned long get_sigframe_size(void); #ifdef CONFIG_X86_32 diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 27516046117a..c8a6d80dfec3 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -218,6 +218,19 @@ static inline bool arch_vma_access_permitted(struct vm_area_struct *vma, unsigned long __get_current_cr3_fast(void); +#ifdef CONFIG_X86_5LEVEL +static inline bool full_va_allowed(struct mm_struct *mm) +{ + /* LAM_U48 steals VA bits abouve 47-bit for tags */ + return mm->context.lam != LAM_U48; +} +#else +static inline bool full_va_allowed(struct mm_struct *mm) +{ + return false; +} +#endif + #include #endif /* _ASM_X86_MMU_CONTEXT_H */ diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c index 660b78827638..4526e8fadfd2 100644 --- a/arch/x86/kernel/sys_x86_64.c +++ b/arch/x86/kernel/sys_x86_64.c @@ -21,6 +21,7 @@ #include #include +#include /* * Align a virtual address to avoid aliasing in the I$ on AMD F15h. @@ -185,7 +186,7 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, /* requesting a specific address */ if (addr) { addr &= PAGE_MASK; - if (!mmap_address_hint_valid(addr, len)) + if (!mmap_address_hint_valid(mm, addr, len)) goto get_unmapped_area; vma = find_vma(mm, addr); @@ -206,7 +207,7 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, * !in_32bit_syscall() check to avoid high addresses for x32 * (and make it no op on native i386). */ - if (addr > DEFAULT_MAP_WINDOW && !in_32bit_syscall()) + if (addr > DEFAULT_MAP_WINDOW && !in_32bit_syscall() && full_va_allowed(mm)) info.high_limit += TASK_SIZE_MAX - DEFAULT_MAP_WINDOW; info.align_mask = 0; diff --git a/arch/x86/mm/hugetlbpage.c b/arch/x86/mm/hugetlbpage.c index a0d023cb4292..9fdc8db42365 100644 --- a/arch/x86/mm/hugetlbpage.c +++ b/arch/x86/mm/hugetlbpage.c @@ -18,6 +18,7 @@ #include #include #include +#include #if 0 /* This is just for testing */ struct page * @@ -103,6 +104,7 @@ static unsigned long hugetlb_get_unmapped_area_topdown(struct file *file, unsigned long pgoff, unsigned long flags) { struct hstate *h = hstate_file(file); + struct mm_struct *mm = current->mm; struct vm_unmapped_area_info info; info.flags = VM_UNMAPPED_AREA_TOPDOWN; @@ -114,7 +116,7 @@ static unsigned long hugetlb_get_unmapped_area_topdown(struct file *file, * If hint address is above DEFAULT_MAP_WINDOW, look for unmapped area * in the full address space. */ - if (addr > DEFAULT_MAP_WINDOW && !in_32bit_syscall()) + if (addr > DEFAULT_MAP_WINDOW && !in_32bit_syscall() && full_va_allowed(mm)) info.high_limit += TASK_SIZE_MAX - DEFAULT_MAP_WINDOW; info.align_mask = PAGE_MASK & ~huge_page_mask(h); @@ -161,7 +163,7 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr, if (addr) { addr &= huge_page_mask(h); - if (!mmap_address_hint_valid(addr, len)) + if (!mmap_address_hint_valid(mm, addr, len)) goto get_unmapped_area; vma = find_vma(mm, addr); diff --git a/arch/x86/mm/mmap.c b/arch/x86/mm/mmap.c index c90c20904a60..f9ca824729de 100644 --- a/arch/x86/mm/mmap.c +++ b/arch/x86/mm/mmap.c @@ -21,6 +21,7 @@ #include #include #include +#include #include "physaddr.h" @@ -35,6 +36,8 @@ unsigned long task_size_32bit(void) unsigned long task_size_64bit(int full_addr_space) { + if (!full_va_allowed(current->mm)) + return DEFAULT_MAP_WINDOW; return full_addr_space ? TASK_SIZE_MAX : DEFAULT_MAP_WINDOW; } @@ -206,11 +209,15 @@ const char *arch_vma_name(struct vm_area_struct *vma) * the failure of such a fixed mapping request, so the restriction is not * applied. */ -bool mmap_address_hint_valid(unsigned long addr, unsigned long len) +bool mmap_address_hint_valid(struct mm_struct *mm, + unsigned long addr, unsigned long len) { if (TASK_SIZE - len < addr) return false; + if (addr + len > DEFAULT_MAP_WINDOW && !full_va_allowed(mm)) + return false; + return (addr > DEFAULT_MAP_WINDOW) == (addr + len > DEFAULT_MAP_WINDOW); } From patchwork Wed May 11 02:27:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12845714 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A795C4332F for ; Wed, 11 May 2022 02:29:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4678F8D0009; Tue, 10 May 2022 22:29:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3569C8D0006; Tue, 10 May 2022 22:29:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F29FA8D000A; Tue, 10 May 2022 22:29:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B0D918D0006 for ; Tue, 10 May 2022 22:29:46 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 8931821ECA for ; Wed, 11 May 2022 02:29:46 +0000 (UTC) X-FDA: 79451881572.04.0B7AC6B Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf01.hostedemail.com (Postfix) with ESMTP id 4C9A840003 for ; Wed, 11 May 2022 02:29:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652236185; x=1683772185; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Gd+nZWXT2Vj5sId1tBKnAzjGgf10Zdvpv1KhXHK183k=; b=VPid2hcyJ6Jb6XBWXu7YpqlU4hzhpJYfTWGlWQaskucPYnKCJMNyQ2Fu SyOSvBSZKUJt9AazIHs3nJbIjUTdQIsERG2jI2fKCFuQ/L4q7mjH2B8kL KbAZlU3ncadIbUPOkf7fccuDl928b56dngFI75PS/TudDc3R+us3zJr/T u7fgiJUD7mqIN0rHM4HHgcD5EY8diXMKwOvG3AQG+9cKuphOfabV+GwpW qBpBmgZl2sqhsaKPnejd3oy4QR5JBwWpcFtm4xZ34iW//bFBqPekQhQbl ggBkcBEBchuqD7vLaLkHtR3FnQ6cB4rKh6Zwgca2yOvh0S6LRZq3Zi6/j w==; X-IronPort-AV: E=McAfee;i="6400,9594,10343"; a="269695081" X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="269695081" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 May 2022 19:29:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="670166557" Received: from black.fi.intel.com ([10.237.72.28]) by fmsmga002.fm.intel.com with ESMTP; 10 May 2022 19:29:40 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 69FD1646; Wed, 11 May 2022 05:28:01 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Andrey Ryabinin , Alexander Potapenko , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [RFCv2 09/10] x86/mm: Add userspace API to enable Linear Address Masking Date: Wed, 11 May 2022 05:27:50 +0300 Message-Id: <20220511022751.65540-11-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> References: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 4C9A840003 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=VPid2hcy; spf=none (imf01.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.115) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com X-Rspam-User: X-Stat-Signature: wryphsf5bad4znhj915ium4zgt6u4rwf X-HE-Tag: 1652236174-788169 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allow to enable Linear Address Masking via ARCH_THREAD_FEATURE_ENABLE arch_prctl(2). Signed-off-by: Kirill A. Shutemov --- arch/x86/kernel/process.c | 21 +++++++++++++++- arch/x86/kernel/process.h | 2 ++ arch/x86/kernel/process_64.c | 46 ++++++++++++++++++++++++++++++++++++ 3 files changed, 68 insertions(+), 1 deletion(-) diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index cb8fc28f2eae..911c24321312 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -46,6 +46,8 @@ #include #include #include +#include +#include #include "process.h" @@ -992,7 +994,9 @@ unsigned long __get_wchan(struct task_struct *p) static long thread_feature_prctl(struct task_struct *task, int option, unsigned long features) { - const unsigned long known_features = 0; + const unsigned long known_features = + X86_THREAD_LAM_U48 | + X86_THREAD_LAM_U57; if (features & ~known_features) return -EINVAL; @@ -1013,8 +1017,23 @@ static long thread_feature_prctl(struct task_struct *task, int option, /* Handle ARCH_THREAD_FEATURE_ENABLE */ + if (features & (X86_THREAD_LAM_U48 | X86_THREAD_LAM_U57)) { + long ret; + + /* LAM is only available in long mode */ + if (in_32bit_syscall()) + return -EINVAL; + + ret = enable_lam(task, features); + if (ret) + return ret; + } + task->thread.features |= features; out: + /* Update CR3 to get LAM active */ + switch_mm(task->mm, task->mm, task); + return task->thread.features; } diff --git a/arch/x86/kernel/process.h b/arch/x86/kernel/process.h index 76b547b83232..b8fa0e599c6e 100644 --- a/arch/x86/kernel/process.h +++ b/arch/x86/kernel/process.h @@ -4,6 +4,8 @@ #include +long enable_lam(struct task_struct *task, unsigned long features); + void __switch_to_xtra(struct task_struct *prev_p, struct task_struct *next_p); /* diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index e459253649be..a25c51da7005 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -729,6 +729,52 @@ void set_personality_ia32(bool x32) } EXPORT_SYMBOL_GPL(set_personality_ia32); +static bool lam_u48_allowed(void) +{ + struct mm_struct *mm = current->mm; + + if (!full_va_allowed(mm)) + return true; + + return find_vma(mm, DEFAULT_MAP_WINDOW) == NULL; +} + +long enable_lam(struct task_struct *task, unsigned long features) +{ + features |= task->thread.features; + + /* LAM_U48 and LAM_U57 are mutually exclusive */ + if ((features & X86_THREAD_LAM_U48) && (features & X86_THREAD_LAM_U57)) + return -EINVAL; + + if (!cpu_feature_enabled(X86_FEATURE_LAM)) + return -ENXIO; + + if (mmap_write_lock_killable(task->mm)) + return -EINTR; + + if ((features & X86_THREAD_LAM_U48) && !lam_u48_allowed()) { + mmap_write_unlock(task->mm); + return -EINVAL; + } + + /* + * Record the most permissive (allowing the widest tags) LAM + * mode to the mm context. It determinates if a mappings above + * 47 bit is allowed for the process. + * + * The mode is also used by a kernel thread when it does work + * on behalf of the process (like async I/O, io_uring, etc.) + */ + if (features & X86_THREAD_LAM_U48) + current->mm->context.lam = LAM_U48; + else if (current->mm->context.lam == LAM_NONE) + current->mm->context.lam = LAM_U57; + + mmap_write_unlock(task->mm); + return 0; +} + #ifdef CONFIG_CHECKPOINT_RESTORE static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) { From patchwork Wed May 11 02:27:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12845716 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35504C433EF for ; Wed, 11 May 2022 02:29:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B4A98D000A; Tue, 10 May 2022 22:29:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 43D2A8D0006; Tue, 10 May 2022 22:29:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26B098D000A; Tue, 10 May 2022 22:29:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 035298D0006 for ; Tue, 10 May 2022 22:29:51 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id CE2E91D8F for ; Wed, 11 May 2022 02:29:50 +0000 (UTC) X-FDA: 79451881740.25.ADF4FCD Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf02.hostedemail.com (Postfix) with ESMTP id A829F800A2 for ; Wed, 11 May 2022 02:29:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652236190; x=1683772190; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=frV6V5e6eki4023LcoU7tb6ei2BMdOM57Nm1uh2kLPA=; b=EceUoDdeKAJFsGq/SvEDSHMzIP1fvVTW5U3fhDtAGgoY+OSZ7XNEetPr nnV4tbQZuuE3+wb6yo9yeuq/CC3AzLo03wrO8hscnaOWgYoWqpbK8apYi AIOYV6yGfiR9lICnCo5Zv9ANqqdP0um9z1+ZKmcoX+O0kpbgmBlzG+xyk M8IBz+5Fd66Sq8pq0IM/1/LvzP62mLfqIyLVUHwuA9Zt92DQ+yUNt/2BN CDc2J5EDa8pc6dE3uQx2VhUYcBWyDEMCxfyMI2yKZ+8tVkLME2AHdGCGx iyCLBXuSYyRAH2w8qpYiXaLNpgxebocN6og0vSXhl/L1llsrupI6dT3/8 w==; X-IronPort-AV: E=McAfee;i="6400,9594,10343"; a="330158674" X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="330158674" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 May 2022 19:29:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,215,1647327600"; d="scan'208";a="697359194" Received: from black.fi.intel.com ([10.237.72.28]) by orsmga004.jf.intel.com with ESMTP; 10 May 2022 19:29:40 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 6FF5B590; Wed, 11 May 2022 05:28:01 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Andrey Ryabinin , Alexander Potapenko , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [RFCv2 10/10] x86: Expose thread features status in /proc/$PID/arch_status Date: Wed, 11 May 2022 05:27:51 +0300 Message-Id: <20220511022751.65540-12-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> References: <20220511022751.65540-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: A829F800A2 X-Stat-Signature: kiofn17k8gfi5wrg89w8r1yz8h6bphff X-Rspam-User: Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=EceUoDde; spf=none (imf02.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com X-Rspamd-Server: rspam09 X-HE-Tag: 1652236182-271968 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add two lines in /proc/$PID/arch_status to report enabled and locked features. Signed-off-by: Kirill A. Shutemov --- arch/x86/kernel/Makefile | 2 ++ arch/x86/kernel/fpu/xstate.c | 47 --------------------------- arch/x86/kernel/proc.c | 63 ++++++++++++++++++++++++++++++++++++ 3 files changed, 65 insertions(+), 47 deletions(-) create mode 100644 arch/x86/kernel/proc.c diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index c41ef42adbe8..19dae7a4201b 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -149,6 +149,8 @@ obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += sev.o +obj-$(CONFIG_PROC_FS) += proc.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y) diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 39e1c8626ab9..789a7a1429df 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -10,8 +10,6 @@ #include #include #include -#include -#include #include #include @@ -1730,48 +1728,3 @@ long fpu_xstate_prctl(struct task_struct *tsk, int option, unsigned long arg2) return -EINVAL; } } - -#ifdef CONFIG_PROC_PID_ARCH_STATUS -/* - * Report the amount of time elapsed in millisecond since last AVX512 - * use in the task. - */ -static void avx512_status(struct seq_file *m, struct task_struct *task) -{ - unsigned long timestamp = READ_ONCE(task->thread.fpu.avx512_timestamp); - long delta; - - if (!timestamp) { - /* - * Report -1 if no AVX512 usage - */ - delta = -1; - } else { - delta = (long)(jiffies - timestamp); - /* - * Cap to LONG_MAX if time difference > LONG_MAX - */ - if (delta < 0) - delta = LONG_MAX; - delta = jiffies_to_msecs(delta); - } - - seq_put_decimal_ll(m, "AVX512_elapsed_ms:\t", delta); - seq_putc(m, '\n'); -} - -/* - * Report architecture specific information - */ -int proc_pid_arch_status(struct seq_file *m, struct pid_namespace *ns, - struct pid *pid, struct task_struct *task) -{ - /* - * Report AVX512 state if the processor and build option supported. - */ - if (cpu_feature_enabled(X86_FEATURE_AVX512F)) - avx512_status(m, task); - - return 0; -} -#endif /* CONFIG_PROC_PID_ARCH_STATUS */ diff --git a/arch/x86/kernel/proc.c b/arch/x86/kernel/proc.c new file mode 100644 index 000000000000..7b2f39031d8a --- /dev/null +++ b/arch/x86/kernel/proc.c @@ -0,0 +1,63 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include + +/* + * Report the amount of time elapsed in millisecond since last AVX512 + * use in the task. + */ +static void avx512_status(struct seq_file *m, struct task_struct *task) +{ + unsigned long timestamp = READ_ONCE(task->thread.fpu.avx512_timestamp); + long delta; + + if (!timestamp) { + /* + * Report -1 if no AVX512 usage + */ + delta = -1; + } else { + delta = (long)(jiffies - timestamp); + /* + * Cap to LONG_MAX if time difference > LONG_MAX + */ + if (delta < 0) + delta = LONG_MAX; + delta = jiffies_to_msecs(delta); + } + + seq_put_decimal_ll(m, "AVX512_elapsed_ms:\t", delta); + seq_putc(m, '\n'); +} + +static void dump_features(struct seq_file *m, unsigned long features) +{ + if (features & X86_THREAD_LAM_U48) + seq_puts(m, "lam_u48 "); + if (features & X86_THREAD_LAM_U57) + seq_puts(m, "lam_u57 "); +} + +/* + * Report architecture specific information + */ +int proc_pid_arch_status(struct seq_file *m, struct pid_namespace *ns, + struct pid *pid, struct task_struct *task) +{ + /* + * Report AVX512 state if the processor and build option supported. + */ + if (cpu_feature_enabled(X86_FEATURE_AVX512F)) + avx512_status(m, task); + + seq_puts(m, "Thread_features:\t"); + dump_features(m, task->thread.features); + seq_putc(m, '\n'); + + seq_puts(m, "Thread_features_locked:\t"); + dump_features(m, task->thread.features_locked); + seq_putc(m, '\n'); + + return 0; +}