From patchwork Mon Jan 23 22:04:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13113074 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67D4AC25B50 for ; Mon, 23 Jan 2023 22:05:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA8A26B0078; Mon, 23 Jan 2023 17:05:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D7EAB6B007B; Mon, 23 Jan 2023 17:05:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C46DE6B007D; Mon, 23 Jan 2023 17:05:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B3F0D6B0078 for ; Mon, 23 Jan 2023 17:05:27 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 68D41C08F4 for ; Mon, 23 Jan 2023 22:05:27 +0000 (UTC) X-FDA: 80387445894.05.973122C Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by imf22.hostedemail.com (Postfix) with ESMTP id 39AE7C0016 for ; Mon, 23 Jan 2023 22:05:24 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=lkLuuT99; spf=none (imf22.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.24) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674511525; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HdRYkzOlUG43YjkCu8PsfJrpwpJ/D/N4XwLO2lELylw=; b=MeiegbExXM6im5Jp9jKuHvq3VC6PaXay817ce3u83Ef93rvyuH3kNtN/c2yIg6CpBvfEfL sb0eNNGdwpCzIN9Zww4VutZTqWBUWugzOlAtYLK9pTub68kHCM5dJ0V78zmxhfIpn4FJ/V EUvnovFFQPstWI2JMZRN6KJd+HQ7OzE= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=lkLuuT99; spf=none (imf22.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.24) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674511525; a=rsa-sha256; cv=none; b=gGFfAVWBEmJYbrjpooHF+MgzyG8MimqkOaBWnDuNSsrr+MGiHJezaCYrbesjnVjJZIBx37 vdpoAd0dMTdX+YJcFj19cxn/j+x3QojtzskFSHd3w4V3ddfMDdYUqEqcvFzVW4qGbABXtu JkaU+99pyFZd4nUR9nRousc+D6e4F7Q= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511525; x=1706047525; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=focBDCcui25Nna3zw6dN8ZwvWQ0ckdMqYvOG7S60BQo=; b=lkLuuT99kKknFhIMEi4RvK7uy2u6W18bEcr6B1DgTyYvCbxLBb1DiXzd MkntRu8HonUMPsyB0yFuATrD+vu+kzDP9RmnCmY0vO/Bn+ZyAgWVENFUr P6vmw63eSBYEDmpQ6A2fLt9ouXx72f0fOo4oAPNfTroskXioOZw+6itDs vsC47bpMSx1SFhTbOs4XZ2IS3x4Ak8M/oNSmQVyXlbylfyrxgVhHbX1F8 5O6Ei+JkWWcqG+S2yhNdywYYTKjEaLRSuW9AGek8x2IhjmiSHB49qKfzA 4GN9OxN8vWlHftHLDNxPj4XB1zYEYZ+UvbV+G9fTrasAlwmFebZxktNuQ g==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="327421937" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="327421937" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:23 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="661878108" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="661878108" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:17 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 6779A109478; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv15 07/17] x86/mm: Reduce untagged_addr() overhead for systems without LAM Date: Tue, 24 Jan 2023 01:04:50 +0300 Message-Id: <20230123220500.21077-8-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: my7t4oapuxnydjbudqi95nrj1jfdwn9s X-Rspam-User: X-Rspamd-Queue-Id: 39AE7C0016 X-Rspamd-Server: rspam06 X-HE-Tag: 1674511524-986000 X-HE-Meta: U2FsdGVkX1+pPL8DbEnUMJHdogaJxLvAOVfKoTPRT5hMBSu7CJaWwWL/rzbvMymnYadS61GsO6Noa7Wq/1R4d7KxfaQgAiJSEVpU9QfUyejQdhWUNyQXKFAqtkdx8cuV1UsryqhU0GqiPfTIYvaPewWhp1i0/Ut7xkTxo1sqayJEDkQS+ZP112Wwchco1riHx2SSYPdEIkFPtAnpiMY69Afn7Tc+geSjVCPlCW2cm6TRLpVZci8zrF3iF4emSz/Zk19E5Xq01faU19oF5LgFFIP5vU2V9raL29OLHXWuztX8Z9T6jblq8CHMKYMg61EP66jjpKYEVUsnAOKV5kjMKAEKj/U3jPNrcm8eXxLmgnHrIFVFRzEfSosS53VPWTdwzqMKP0PjYY/kg3cECSJtFJ6dQjM0ys049rPnSfAnLEwx/RfnAT+3iJSKRUcL+/lTKZB8XVLtUu+ef2agcV2jjzgKgiaow0Wn88OuKXq5vInDfsFG98EKKE5ehV3sN3EwRtlKlplMVUa1SfmVNir3ORx1UkalVjvllrxXSILKiDCdEXXujmU0qAiVW7mWV4qVv+Im6YLW5rEtV+ksrx+KmX2KlNA7RwJmO4bAGxBv8adCI3VS5S+rbb5N0TxeZ09eUY1CsdbmTqQ+CZPH8Nyr5wZq1HGZpangP+QR+IhVqiywAA5zgHD5IGTpnR0sSoEepQLrdZmftx/aC/g/stixn/3sH9gG5xKm6UJFHJDnPxeKMvCwGdMoIkmbQ1j1Lmw6aBhItPZdv1wknWzyTgcoKyCbGGmrr9W/KIRlRdeLqIQc0NqJDWJO3vVEW2qyzzBJbnAISIU+MSGvsaYyZs2O1n3rLIC6FVnax9YPllW3CoUIeZZFzwFJQot9BD9nsFcwjZyrylhPOFfTpg0CKVlA/vbdXQXe9398E4Qh9wmw6W4TYV1EfqntWYe+ssDOswzQVVYZrdHQija98TMhY7T dgSr6asA uZrZMGX19aW9NxwR/a6StDro2LTeCl8TlWouuzc0stl5BgrFLxc+lJhww3qB4bAgu9yi43EQpLrw27L9CVhAnBuVyTZej6uafYT0NJkBttoj+T8UTRtZM+wLrlndPn3P2v21krf7L3A0Xv/VIqfWmtzqHAE3jeHPhcx5cRaljhhQcZ47C4J/BAREcPYaWKaNI5YWMCWUhVPmaIpd/CbJccpmEtYp2H/8AOX8bzPiGcKwWPsBbRzF8pEYgf5BNwLa8P51onuKBmTETWpXm6fypYT5BYw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use alternatives to reduce untagged_addr() overhead. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/disabled-features.h | 8 ++++- arch/x86/include/asm/uaccess.h | 41 +++++++++++++++++------- 2 files changed, 37 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index c44b56f7ffba..3f0c31044f02 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -75,6 +75,12 @@ # define DISABLE_CALL_DEPTH_TRACKING (1 << (X86_FEATURE_CALL_DEPTH & 31)) #endif +#ifdef CONFIG_ADDRESS_MASKING +# define DISABLE_LAM 0 +#else +# define DISABLE_LAM (1 << (X86_FEATURE_LAM & 31)) +#endif + #ifdef CONFIG_INTEL_IOMMU_SVM # define DISABLE_ENQCMD 0 #else @@ -115,7 +121,7 @@ #define DISABLED_MASK10 0 #define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ DISABLE_CALL_DEPTH_TRACKING) -#define DISABLED_MASK12 0 +#define DISABLED_MASK12 (DISABLE_LAM) #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index fd9182951084..6450a2723bcd 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -30,26 +31,44 @@ static inline bool pagefault_disabled(void); * Magic with the 'sign' allows to untag userspace pointer without any branches * while leaving kernel addresses intact. */ -static inline unsigned long __untagged_addr(unsigned long addr, - unsigned long mask) +static inline unsigned long __untagged_addr(unsigned long addr) { - long sign = addr >> 63; + long sign; + + /* + * Refer tlbstate_untag_mask directly to avoid RIP-relative relocation + * in alternative instructions. The relocation gets wrong when gets + * copied to the target place. + */ + asm (ALTERNATIVE("", + "sar $63, %[sign]\n\t" /* user_ptr ? 0 : -1UL */ + "or %%gs:tlbstate_untag_mask, %[sign]\n\t" + "and %[sign], %[addr]\n\t", X86_FEATURE_LAM) + : [addr] "+r" (addr), [sign] "=r" (sign) + : "m" (tlbstate_untag_mask), "[sign]" (addr)); - addr &= mask | sign; return addr; } #define untagged_addr(addr) ({ \ - u64 __addr = (__force u64)(addr); \ - __addr = __untagged_addr(__addr, current_untag_mask()); \ - (__force __typeof__(addr))__addr; \ + unsigned long __addr = (__force unsigned long)(addr); \ + (__force __typeof__(addr))__untagged_addr(__addr); \ }) +static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, + unsigned long addr) +{ + long sign = addr >> 63; + + mmap_assert_locked(mm); + addr &= (mm)->context.untag_mask | sign; + + return addr; +} + #define untagged_addr_remote(mm, addr) ({ \ - u64 __addr = (__force u64)(addr); \ - mmap_assert_locked(mm); \ - __addr = __untagged_addr(__addr, (mm)->context.untag_mask); \ - (__force __typeof__(addr))__addr; \ + unsigned long __addr = (__force unsigned long)(addr); \ + (__force __typeof__(addr))__untagged_addr_remote(mm, __addr); \ }) #else