From patchwork Sun Mar 12 11:26:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 675B8C6FA99 for ; Sun, 12 Mar 2023 11:26:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 32CD58E0001; Sun, 12 Mar 2023 07:26:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B4308E000A; Sun, 12 Mar 2023 07:26:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E48F78E0001; Sun, 12 Mar 2023 07:26:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CB2AB8E0008 for ; Sun, 12 Mar 2023 07:26:39 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A840AAAD18 for ; Sun, 12 Mar 2023 11:26:39 +0000 (UTC) X-FDA: 80560018518.23.F7A80EC Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf25.hostedemail.com (Postfix) with ESMTP id 78B72A0008 for ; Sun, 12 Mar 2023 11:26:37 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="ae//+b+q"; spf=none (imf25.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620397; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pj/WC7utj+xn+ajKUwmk2ytcxWnqOOCcuDBlAr4sM28=; b=8GIY6h7E8chE8n+h6ZRkDB+UcouFxqf2PlNGDHmPPyKMlBg64L892pFkJGDNLyf99AAjeX nDz5rkTbALw+u1sAXQne2YvIMjRy2FBwSC2JkgMdt50ghBKNFlakJftRHZFTmuUxXLhs+H kdgBO8ju5+dxYxNXFwH5bvmABoKvM7k= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="ae//+b+q"; spf=none (imf25.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620397; a=rsa-sha256; cv=none; b=XC9OfHokVQj/X14eayq+TYiKYVZjXHGRo1S2gk9QeRwJg9XctU/LxW2XJkwuhIWAJ+Yx2J JU5tjL22gEzaK6eTRpTBLwHG9wi8cI11E/A52jxg32IW2Y0B0E44Zqgcbe2I3OhbPktFne sEKQdJzXwIkR5OG6aEolxdIQRkJSBg4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620397; x=1710156397; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=HoVvsInTn5TZ+MT8glj3IrotSxRmkydAQ9cgijR6SG0=; b=ae//+b+q2Un1jllrUsL5mv3njV0zy/zZ8Jc/TPJOg9t+5bs0G62o9iC+ II7V1pCu4cDrsBSJewmF1sM0RtfrvC9z/IhkwvySUuHpd/2BQp/HO8IHN rmUKUQa9EZh0Dayj4sDeFq0QbayzQ+eXNnHGwiH5DW9VkkYKcKHBgYOLl SV+R37K+hNz10+IMLRkov5/kdc9zr2ZrJLK5op7gxvukG3hW0M2RRAUSw rxMhNpJ2X9bXtl3IUl59BO3XKM0y1DiHVWrEuzjDKFSZ+saEnjZNpwTDv skp/ECCsM2CGWFbvayzcSvdh6ADV44jynUSsqn97vii2MTKXBRyqvsZp/ w==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="339349959" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="339349959" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="852438031" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="852438031" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:30 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 9992410D7B5; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv16 07/17] x86/mm: Reduce untagged_addr() overhead for systems without LAM Date: Sun, 12 Mar 2023 14:26:02 +0300 Message-Id: <20230312112612.31869-8-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 916aebbg5ybt9pnaz7c74b53trimzaww X-Rspamd-Queue-Id: 78B72A0008 X-HE-Tag: 1678620397-985196 X-HE-Meta: U2FsdGVkX1+cgBttI8jvz/o5359dbzgGmJHilb8fgiZmKAqXfoXUHhs5c7cU1ZKPRp7F/ZeMYt5UnMQsfk38fOHDojayKTOU0ZY75MnzX3jz3otj0ExSF8p4a/2cGEoQd48M+TH6ym9zhe0fGsMPxxuABMk7nDtJkUMFE1VDYl0TSKSHq6HZZgs6ZnY2gG/OreN6gBskVBnSDr6RL14wFUNZKhIkQOE7DXYerGXThhzSeLSMFfx0MYC4v+KmWuMpHfMKf6NfMSeS+Dv/Xp7lN2kWscb7X3UonMy5pyYAKZK5tlDKg6XPglK33kA/NUJdYke6cd34gwaMBfrn6tkzTUN7B3IxI2d341l6G9Ps7uwLD0xfuSZbnRDkym3RPrMFU+zgg6j9zY1gJijjjMliEGfJ8evrpw3yA57xz6vXli02N+zntn4RccWSECqDz0hn4ccsixQGu84sBsKEN2DjjRrmkv5Olhx2ANIcpCcatlddLqoR2wF0O0S/K8/3gt7HB0UcW4xbOpIGzDn51xbOZL1RuDSJBZ9YPTjAppINcLTr7omRK/zacDlb8W9AvL73lQHkds5Mf9HmOG+XoQWvYu8mlxrEsmvl0VMohghr3w7m8tWn3jwvn1WkICbgA77c329YXPigKAztx1/suiH4sJZIeSBSERD/dVCMf6y1rp2D/6gJLf4E6SHBaeOkulkJn/BDSVsaEO4AXXfjVYFSCfzMBJ9MGrZ0xuhfoOEfOtY/ynrHnCIuTbxLu4ldzcaHIVbMfbMCurEoY2Z4FsqPiZt+ON6GHGzhHWA7TJKqYHlkByStVAbjU7hlR6vKNzO7O3OXhSdpsepHmhjlzWoCbL+w2A4gALP1J2W+enDgC2IQoLGpqMVSw2QWRo3bXV5ePnz4TLIDYDFL7ZPeA1TXTrwQtHRi02G1hwpnPiaURzm2Uf+zMvzHZAuNtRzCn6s1JIu/2i7RsFCM0u71ZOl 5zCu6hmg kq3DDizuMPdaNn/ZZ96/+MIS1Qo2WpG5GvuOdClF1k/4QjY9nhA4K0LrJ8qEA8VL+5zGtcZRuMzCTDdoaZknS1w8FohaOX6J3g2UaGRsaDIwc7ePlSooJ2sfHRTEvpKyMCdyvG8uMM/3naARVmRPXaCfcdwbElfG6njPo/mDrhCfQMzPLEUONIq7t7NvieqjdHk19ElQmDRBdYoGR7oc84RLJn7w7o+7idYXdWVPa7ULc13PKH9W+lgIslhasRMkQNXC1hjvTHZKoYQzyaEoD5uteog== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use alternatives to reduce untagged_addr() overhead. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/disabled-features.h | 8 ++++- arch/x86/include/asm/uaccess.h | 41 +++++++++++++++++------- 2 files changed, 37 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index 5dfa4fb76f4b..fafe9be7a6f4 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -75,6 +75,12 @@ # define DISABLE_CALL_DEPTH_TRACKING (1 << (X86_FEATURE_CALL_DEPTH & 31)) #endif +#ifdef CONFIG_ADDRESS_MASKING +# define DISABLE_LAM 0 +#else +# define DISABLE_LAM (1 << (X86_FEATURE_LAM & 31)) +#endif + #ifdef CONFIG_INTEL_IOMMU_SVM # define DISABLE_ENQCMD 0 #else @@ -115,7 +121,7 @@ #define DISABLED_MASK10 0 #define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ DISABLE_CALL_DEPTH_TRACKING) -#define DISABLED_MASK12 0 +#define DISABLED_MASK12 (DISABLE_LAM) #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index c79ebdbd6356..457e814712af 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -30,26 +31,44 @@ static inline bool pagefault_disabled(void); * Magic with the 'sign' allows to untag userspace pointer without any branches * while leaving kernel addresses intact. */ -static inline unsigned long __untagged_addr(unsigned long addr, - unsigned long mask) +static inline unsigned long __untagged_addr(unsigned long addr) { - long sign = addr >> 63; + long sign; + + /* + * Refer tlbstate_untag_mask directly to avoid RIP-relative relocation + * in alternative instructions. The relocation gets wrong when gets + * copied to the target place. + */ + asm (ALTERNATIVE("", + "sar $63, %[sign]\n\t" /* user_ptr ? 0 : -1UL */ + "or %%gs:tlbstate_untag_mask, %[sign]\n\t" + "and %[sign], %[addr]\n\t", X86_FEATURE_LAM) + : [addr] "+r" (addr), [sign] "=r" (sign) + : "m" (tlbstate_untag_mask), "[sign]" (addr)); - addr &= mask | sign; return addr; } #define untagged_addr(addr) ({ \ - u64 __addr = (__force u64)(addr); \ - __addr = __untagged_addr(__addr, current_untag_mask()); \ - (__force __typeof__(addr))__addr; \ + unsigned long __addr = (__force unsigned long)(addr); \ + (__force __typeof__(addr))__untagged_addr(__addr); \ }) +static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, + unsigned long addr) +{ + long sign = addr >> 63; + + mmap_assert_locked(mm); + addr &= (mm)->context.untag_mask | sign; + + return addr; +} + #define untagged_addr_remote(mm, addr) ({ \ - u64 __addr = (__force u64)(addr); \ - mmap_assert_locked(mm); \ - __addr = __untagged_addr(__addr, (mm)->context.untag_mask); \ - (__force __typeof__(addr))__addr; \ + unsigned long __addr = (__force unsigned long)(addr); \ + (__force __typeof__(addr))__untagged_addr_remote(mm, __addr); \ }) #else