From patchwork Sun Mar 12 11:25:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DA15C6FA99 for ; Sun, 12 Mar 2023 11:26:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 93E908E0003; Sun, 12 Mar 2023 07:26:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8EF138E0001; Sun, 12 Mar 2023 07:26:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B62C8E0003; Sun, 12 Mar 2023 07:26:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6D81C8E0001 for ; Sun, 12 Mar 2023 07:26:35 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 2CE511A0625 for ; Sun, 12 Mar 2023 11:26:35 +0000 (UTC) X-FDA: 80560018350.22.55DE0CA Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf10.hostedemail.com (Postfix) with ESMTP id 2A3E1C0011 for ; Sun, 12 Mar 2023 11:26:33 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ABfu5+qt; spf=none (imf10.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620393; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=01M6Oui0yYdZJlxoRo2JZXCZONIUA8npeELpcrndGKU=; b=VX+fVNsKdqv1mrz8JNyN/yaxz9OtYsakyk3y3i3ScX2wh6CsZ4H3648+a5q6RSlwzIOro7 0pwPkmj17v9RKugx+HqrNNBgtg52Y0uQaUQfFfwjg35ihM/1I1Jf2Keu7qyCn9fUUbDlPj 1b6Fv9nhVNrHJIfsGe5J6IqL1UFJ8QQ= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ABfu5+qt; spf=none (imf10.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620393; a=rsa-sha256; cv=none; b=MtEfog35+SxCzMUwe8e7qYJIGSuvb/zZiVe+QRK7V+bm30X8eWG9SUezY1r7tiT5jJeKNm u7Ntyb0DSq5aX3gPdi0sQD2kErK8EFaFxkA8pbjY8U+ft6vohpwerWXoczoExUcfxJBYl7 Opb+895waP5iM3vP/fhqeoZQmBQRF1o= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620393; x=1710156393; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+ya4j8+hmXPftNA55cuj2jSsHwe1N+LRr5YAFmUiVLI=; b=ABfu5+qt32bypEYBQ66mKO2cPK6t5cPN0s7IsVl+T4uDM7YGXQXkAYXw NEo0zJrxJeP65lj/b4NmgwafbA3pKAkXvGHfmJLGTkWbywuuyRYQUe0UZ HrFg7igcqQ1i35d3x+VzUNv5IIZv+/JjiEIdZ8cnyCnzQYkVJs8AovKDC D56/ttCEHgXbX8G62HePJJZspMhj0VMU2mv8JwSOTwsZHOKMIpblWp8us bLrdM/SQiLOrLubvbVOlwIwRfX6KuBx3jVvNsLIFAoBMVhoQp25O2CWX8 xiTqE2FI9cgqKvVcWebxroPXFRLngcoiRgGXEsjr0BsGZBIvrXKYLYLNk Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="339349927" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="339349927" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="852438013" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="852438013" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:22 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 5C6AA10D7AF; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv16 01/17] x86/mm: Rework address range check in get_user() and put_user() Date: Sun, 12 Mar 2023 14:25:56 +0300 Message-Id: <20230312112612.31869-2-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 7swz34wnsio5bxqdr5ojg6wkcoz7yt81 X-Rspamd-Queue-Id: 2A3E1C0011 X-HE-Tag: 1678620393-388518 X-HE-Meta: U2FsdGVkX1+qtl/q+N/tUyisBnZZX4RwBodq1rgAsuuDiPcebHenVQxBq3gcWFPgw3qenPIokqahNlT30zVntLxLywnFKGH83SWeTiikEMt+L0DKyVWZvayZsbVVI2DMysWXKrrN6WsIi0rsCaYGj5/t9URR8RggS4kZCkKG1i3e6q18bIl5ao22+tSqbve7nzMO0xJGNoh9p+8KaWJaStqvvv2USZA664mDmMm5sKGDnBVHIO6q7mV5tGnPFff1KEmsfL7lZKMDNncniHJj5RXg9oGyowDAvxEAMRu7Wsg+d62fqsY8eP7Mwv1W6OBidpo1GEJmredFZX4p9n324QLN5ej9D/7HSfRYZAn+Ijn1zve8is7FD++Fx9kvkgqzbCaAF1TInIkrCd9e3FqQ0oO98Bwn1YxTVkVq4zHgH+B/gqnh4VacxHxK4WshukjolXtLSwd8T6tSZs3L7r0/RJtgKpOoO3ovKMASFxGN1bg5ZOyK3Ke7EnlzCAbDzL1tIp99nqr2F4D6oE7wWMABd6wrB7fhHfNrTIdxYDdv57pXQUEmzVdLuzfBqot6HN6ndyCCyA92KvjHUo6DZhjxGdDHUnxeP6mBcOYW4thrDIhWC4/btCtmc8omZ4Bnp8Bord5zNFBSQdodY0lfx3DsfXdvcCfh1WxuJONNzp4dO5VfHLZSWtuX09B4RVkC+Zadw8IiYoNwYteWu4NbCTrmtHnMCxuFcJYYMVyiCno4AM+mXtmMu2ATQyFtkvPNxj4ruy2U63hQOcCy3mHzn3PzzDrZZE2wTOIcNoUIJW/80qI7C58mE+5q1/KbFZzp6DVkGhLBpKrqkBM8uSErvHSvakygLIXpAtjQpXLZfVJCgiQAfyAMyUpFMx6d1Br6PiRB6C2Nf2zAVBDu9s/ojMiWyjA+82yQYzrUTVG3N1ChmGD1xFFHrUItBpm4QgJ3yzFln+voI3ySXum8f5nycBB nGLSTzm5 91BvSBeQyGEbxt3uAj72pivSdvtcHNXB8h6b4wz5wwooVtwlX27lfWva+zl38yFXwz3Bx+OrYEok8BqnYvOpGTvhAOjnB4eHV6PFf16zVviIpu80Ppzw1w/KTgS9aCoGQ8qHaqfPflDR2vUU6vOgxcrMpFJjIOA4bllPb5jUFJMCkW1O3StlYKRV18p/9xI8pXKRUPRTKiclGglCIHRLZcC4mP4Q/wS2EQ08ykr+hIo6jwbNme9POoxKHZP4jI/QJG2+yv/dLz0v0LkAJbXQhT2PucLesyVVW3fG3evmY1VSB4fDAkNPQ5dZ8/XLAiRjKL4zIciWjjxoIdxiwY/jMMLADcSqXAxcZi06K/xVrFrX0qJsGCjKp35CrKDRg7dVEngr2WKJsAo911BY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The functions get_user() and put_user() check that the target address range resides in the user space portion of the virtual address space. In order to perform this check, the functions compare the end of the range against TASK_SIZE_MAX. For kernels compiled with CONFIG_X86_5LEVEL, this process requires some additional trickery using ALTERNATIVE, as TASK_SIZE_MAX depends on the paging mode in use. Linus suggested that this check could be simplified for 64-bit kernels. It is sufficient to check bit 63 of the address to ensure that the range belongs to user space. Additionally, the use of branches can be avoided by setting the target address to all ones if bit 63 is set. There's no need to check the end of the access range as there's huge gap between end of userspace range and start of the kernel range. The gap consists of canonical hole and unused ranges on both kernel and userspace sides. If an address with bit 63 set is passed down, it will trigger a #GP exception. _ASM_EXTABLE_UA() complains about this. Replace it with plain _ASM_EXTABLE() as it is expected behaviour now. The updated get_user() and put_user() checks are also compatible with Linear Address Masking, which allows user space to encode metadata in the upper bits of pointers and eliminates the need to untag the address before handling it. Signed-off-by: Kirill A. Shutemov Suggested-by: Linus Torvalds Acked-by: Peter Zijlstra (Intel) --- arch/x86/lib/getuser.S | 83 ++++++++++++++++-------------------------- arch/x86/lib/putuser.S | 54 ++++++++++++--------------- 2 files changed, 55 insertions(+), 82 deletions(-) diff --git a/arch/x86/lib/getuser.S b/arch/x86/lib/getuser.S index b70d98d79a9d..b64a2bd1a1ef 100644 --- a/arch/x86/lib/getuser.S +++ b/arch/x86/lib/getuser.S @@ -37,22 +37,22 @@ #define ASM_BARRIER_NOSPEC ALTERNATIVE "", "lfence", X86_FEATURE_LFENCE_RDTSC -#ifdef CONFIG_X86_5LEVEL -#define LOAD_TASK_SIZE_MINUS_N(n) \ - ALTERNATIVE __stringify(mov $((1 << 47) - 4096 - (n)),%rdx), \ - __stringify(mov $((1 << 56) - 4096 - (n)),%rdx), X86_FEATURE_LA57 -#else -#define LOAD_TASK_SIZE_MINUS_N(n) \ - mov $(TASK_SIZE_MAX - (n)),%_ASM_DX -#endif +.macro check_range size:req +.if IS_ENABLED(CONFIG_X86_64) + mov %rax, %rdx + sar $63, %rdx + or %rdx, %rax +.else + cmp $TASK_SIZE_MAX-\size+1, %eax + jae .Lbad_get_user + sbb %edx, %edx /* array_index_mask_nospec() */ + and %edx, %eax +.endif +.endm .text SYM_FUNC_START(__get_user_1) - LOAD_TASK_SIZE_MINUS_N(0) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=1 ASM_STAC 1: movzbl (%_ASM_AX),%edx xor %eax,%eax @@ -62,11 +62,7 @@ SYM_FUNC_END(__get_user_1) EXPORT_SYMBOL(__get_user_1) SYM_FUNC_START(__get_user_2) - LOAD_TASK_SIZE_MINUS_N(1) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=2 ASM_STAC 2: movzwl (%_ASM_AX),%edx xor %eax,%eax @@ -76,11 +72,7 @@ SYM_FUNC_END(__get_user_2) EXPORT_SYMBOL(__get_user_2) SYM_FUNC_START(__get_user_4) - LOAD_TASK_SIZE_MINUS_N(3) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=4 ASM_STAC 3: movl (%_ASM_AX),%edx xor %eax,%eax @@ -90,30 +82,17 @@ SYM_FUNC_END(__get_user_4) EXPORT_SYMBOL(__get_user_4) SYM_FUNC_START(__get_user_8) -#ifdef CONFIG_X86_64 - LOAD_TASK_SIZE_MINUS_N(7) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=8 ASM_STAC +#ifdef CONFIG_X86_64 4: movq (%_ASM_AX),%rdx - xor %eax,%eax - ASM_CLAC - RET #else - LOAD_TASK_SIZE_MINUS_N(7) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user_8 - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX - ASM_STAC 4: movl (%_ASM_AX),%edx 5: movl 4(%_ASM_AX),%ecx +#endif xor %eax,%eax ASM_CLAC RET -#endif SYM_FUNC_END(__get_user_8) EXPORT_SYMBOL(__get_user_8) @@ -166,7 +145,7 @@ EXPORT_SYMBOL(__get_user_nocheck_8) SYM_CODE_START_LOCAL(.Lbad_get_user_clac) ASM_CLAC -bad_get_user: +.Lbad_get_user: xor %edx,%edx mov $(-EFAULT),%_ASM_AX RET @@ -184,23 +163,23 @@ SYM_CODE_END(.Lbad_get_user_8_clac) #endif /* get_user */ - _ASM_EXTABLE_UA(1b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(2b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(3b, .Lbad_get_user_clac) + _ASM_EXTABLE(1b, .Lbad_get_user_clac) + _ASM_EXTABLE(2b, .Lbad_get_user_clac) + _ASM_EXTABLE(3b, .Lbad_get_user_clac) #ifdef CONFIG_X86_64 - _ASM_EXTABLE_UA(4b, .Lbad_get_user_clac) + _ASM_EXTABLE(4b, .Lbad_get_user_clac) #else - _ASM_EXTABLE_UA(4b, .Lbad_get_user_8_clac) - _ASM_EXTABLE_UA(5b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(4b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(5b, .Lbad_get_user_8_clac) #endif /* __get_user */ - _ASM_EXTABLE_UA(6b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(7b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(8b, .Lbad_get_user_clac) + _ASM_EXTABLE(6b, .Lbad_get_user_clac) + _ASM_EXTABLE(7b, .Lbad_get_user_clac) + _ASM_EXTABLE(8b, .Lbad_get_user_clac) #ifdef CONFIG_X86_64 - _ASM_EXTABLE_UA(9b, .Lbad_get_user_clac) + _ASM_EXTABLE(9b, .Lbad_get_user_clac) #else - _ASM_EXTABLE_UA(9b, .Lbad_get_user_8_clac) - _ASM_EXTABLE_UA(10b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(9b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(10b, .Lbad_get_user_8_clac) #endif diff --git a/arch/x86/lib/putuser.S b/arch/x86/lib/putuser.S index 32125224fcca..3062d09a776d 100644 --- a/arch/x86/lib/putuser.S +++ b/arch/x86/lib/putuser.S @@ -33,20 +33,20 @@ * as they get called from within inline assembly. */ -#ifdef CONFIG_X86_5LEVEL -#define LOAD_TASK_SIZE_MINUS_N(n) \ - ALTERNATIVE __stringify(mov $((1 << 47) - 4096 - (n)),%rbx), \ - __stringify(mov $((1 << 56) - 4096 - (n)),%rbx), X86_FEATURE_LA57 -#else -#define LOAD_TASK_SIZE_MINUS_N(n) \ - mov $(TASK_SIZE_MAX - (n)),%_ASM_BX -#endif +.macro check_range size:req +.if IS_ENABLED(CONFIG_X86_64) + mov %rcx, %rbx + sar $63, %rbx + or %rbx, %rcx +.else + cmp $TASK_SIZE_MAX-\size+1, %ecx + jae .Lbad_put_user +.endif +.endm .text SYM_FUNC_START(__put_user_1) - LOAD_TASK_SIZE_MINUS_N(0) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=1 ASM_STAC 1: movb %al,(%_ASM_CX) xor %ecx,%ecx @@ -66,9 +66,7 @@ SYM_FUNC_END(__put_user_nocheck_1) EXPORT_SYMBOL(__put_user_nocheck_1) SYM_FUNC_START(__put_user_2) - LOAD_TASK_SIZE_MINUS_N(1) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=2 ASM_STAC 3: movw %ax,(%_ASM_CX) xor %ecx,%ecx @@ -88,9 +86,7 @@ SYM_FUNC_END(__put_user_nocheck_2) EXPORT_SYMBOL(__put_user_nocheck_2) SYM_FUNC_START(__put_user_4) - LOAD_TASK_SIZE_MINUS_N(3) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=4 ASM_STAC 5: movl %eax,(%_ASM_CX) xor %ecx,%ecx @@ -110,9 +106,7 @@ SYM_FUNC_END(__put_user_nocheck_4) EXPORT_SYMBOL(__put_user_nocheck_4) SYM_FUNC_START(__put_user_8) - LOAD_TASK_SIZE_MINUS_N(7) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=8 ASM_STAC 7: mov %_ASM_AX,(%_ASM_CX) #ifdef CONFIG_X86_32 @@ -144,15 +138,15 @@ SYM_CODE_START_LOCAL(.Lbad_put_user_clac) RET SYM_CODE_END(.Lbad_put_user_clac) - _ASM_EXTABLE_UA(1b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(2b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(3b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(4b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(5b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(6b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(7b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(9b, .Lbad_put_user_clac) + _ASM_EXTABLE(1b, .Lbad_put_user_clac) + _ASM_EXTABLE(2b, .Lbad_put_user_clac) + _ASM_EXTABLE(3b, .Lbad_put_user_clac) + _ASM_EXTABLE(4b, .Lbad_put_user_clac) + _ASM_EXTABLE(5b, .Lbad_put_user_clac) + _ASM_EXTABLE(6b, .Lbad_put_user_clac) + _ASM_EXTABLE(7b, .Lbad_put_user_clac) + _ASM_EXTABLE(9b, .Lbad_put_user_clac) #ifdef CONFIG_X86_32 - _ASM_EXTABLE_UA(8b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(10b, .Lbad_put_user_clac) + _ASM_EXTABLE(8b, .Lbad_put_user_clac) + _ASM_EXTABLE(10b, .Lbad_put_user_clac) #endif From patchwork Sun Mar 12 11:25:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171296 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98FA9C74A5B for ; Sun, 12 Mar 2023 11:26:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71C0D8E0005; Sun, 12 Mar 2023 07:26:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6CE2E8E0001; Sun, 12 Mar 2023 07:26:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D0A58E0005; Sun, 12 Mar 2023 07:26:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 406D38E0001 for ; Sun, 12 Mar 2023 07:26:36 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 022E140951 for ; Sun, 12 Mar 2023 11:26:35 +0000 (UTC) X-FDA: 80560018392.20.8CD16BA Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf14.hostedemail.com (Postfix) with ESMTP id EAF67100005 for ; Sun, 12 Mar 2023 11:26:33 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=aoYCQNet; spf=none (imf14.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620394; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=543GYs3qqt/4FEUXvHVYjsq+MgfRqUiu/iNq6oSVdnk=; b=TJekYhKEOU6qsKj8NrCnj3qf/BV4Sslwg3aX477R3lo8kOFPwx3bTSQXPc70PqbsMWmybX yMhpBoE53dV2etI7crKuHuEQvGtp+0x/Mm633CMYZsIDPBsGRa8+q9IStZGwMkGb+7r2WX wn3JgNGyqWKPOjhwd0aH9L/C8Hc+NxQ= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=aoYCQNet; spf=none (imf14.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620394; a=rsa-sha256; cv=none; b=28eNWjJzJ42dOva11Cn//IOTHrBAH2zSTNRh7QyjSEQB4TFSRmrUpCO5v/EEexSEYPWWge I4hCK8GHJ/PjzDksGtbWe2FSKvt8mTxirj60AkBtWNEZljRDdY77lKkAinyo5syp9hIlO/ c/7g+qvt7zOdaNeYxgFM9vhwSgcPDqU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620394; x=1710156394; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tV3JtlGM60BECBxuCA+QuZaWNCs779YV6vXF0y1DbTc=; b=aoYCQNetVIfY3+/hrjklsCbH9o5Ze6VWkg8M5bXWJ1mTNAN7yRf/n6+L 4A5ZRvIJ4b8jKnAG7sKLTAZ7NM/6wIjwN7vIhVMBncFDsFSdDdk+4m8+3 hE+yUpCqVbYlvGkLNGBhd3uaONTiGeZ8zoKVUkS/S3GyKcJUO93ZWq3Zg HMu8Ta+8SbfZxqlHJZ7YtU9fOB31Rtwece+3aQioy1jj9DZnVYhCga+Ik Cw5mBJnmUON83UgTQclH/xW9aQ7IGDGjFSYPBxODWSNf7DK2FbKNoYAlA 4GN7vXU0duh3fJfMK1hBxYNQiVF2WQ6yrl06xH02IVkaNcOxA+CKTn710 w==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="339349935" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="339349935" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="852438015" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="852438015" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:22 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 6695310D7B0; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv16 02/17] x86: Allow atomic MM_CONTEXT flags setting Date: Sun, 12 Mar 2023 14:25:57 +0300 Message-Id: <20230312112612.31869-3-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: EAF67100005 X-Rspam-User: X-Stat-Signature: dwbu8umujmqs7zkxtjes7q8hjxux9zb5 X-HE-Tag: 1678620393-52919 X-HE-Meta: U2FsdGVkX1/h5EKAnNmr1Ol5QWB7xtCA6bQNx4d4MX2rPjA6oP4dGtKzmO7mgmeoaamJjCjZx+2R/altoDzI74JcdLzxmSJH7K/Clq0vQ7y01UIGnLU/Ucuf1NT5F7ICuAnPC/EdfAALmOmHml1QTYAvMsRhRkFF1I9UfngGJmLRzq8XMONU6j9YLe3/ZCyM0bmwZrwNLCCPDIfqvuDwRwDm3sRnNK8Bwb8qnlmLtWPKFeF/JAlppqf/qVmfEWi1ODBCAqfhzZdNx9DrUae4RxWP975JM4GCD5Cv6gZ6YGmqmLZFrXUmyxacrE+N+Yj8DxnkjlCLjk9ORjQ/+XoVAoWeOp79X5F6rFVAaI/hp9VxtT2kbeQGfGUubHjnjnwU0pS01+LI+3964+jaeWUGA82cchz3SHYSEsxXsY/xVmmAodE7dtPbaAEk+LY/o4qiY+TY9NRS3vGM9VyFKg/IEB6+PqkWLZsKdaDH/syZfkdD8yI66ffLM290qy0/Hn7l2y/0skMOm07ihZggm8tZe5hGk0BabEylP1ZrUelPXK9/Pb8GdzPmNwElpwdZsUEww3d8nWBIStN8b0pnY1GFFeDl0zCqIH0UkY3pPeeoaeEuOr/8WcFLduGU6/0LuN/WuVXQKXB49fk9ZC+7lCfg+K8TfQtUcymvpEjKPbQBVyo1Y7zh2SJ1iL4JSdXORN3zyW5GBeBYsZ3TKA36C1/OykthPHgTJui2T0M6+ILL7XJwXHgrIDATQEDQcxjJBeu9wbP5F0lST2lM0lvaQAcS+Rcetup0PmGit5GpNl6p0SkwFUcHseBGj1XvfbiaT9Lzp9kC8wB+a2l90d6MiysQwZ27nE0+AGznFsz9lzJtoln7C4ymrYMlx7VCRYmOhsOf8NiL6IEsS4eCJv0QVQ6x300rMILNBfCvsUurtcI8cMmZHNI8B964tl4uPV0dsvjpT9mZZ1jlMWyy8hgHk/N LlKcmXrp LkRm9EdoPTYHn6yrYPzAkWJyRGMZYSEMYwnB4IOFD3wjLfZwhp2Iw68LkhQJQFdsD2PFwGnaBhQU7vzKulcSyez3zaXBvv4wbJ7I2WzYUm/XTuCLz4NvLicDkJUJ51ZrFgjnZO69dAi4Sh4BmVwtDPsoetIGRmwp7wQxQc1P7LhEvrUgYi5cyzxbR94QxaK6jA6OsnfyXuuc4IeNMNj2BNqXNWxHt6513aZ/8ZiUbqB0+qr8goiDA0uKSzCDDtSPjQrs0jQN0AVqafH32JlozFQ7NDMleThucisBM7X5DdvnrkV1Ff/mskVEwmjVHGmHYjNLyZoQlZcvaA1jQfcm5tevHD9uxj/j/2QeUJ+jakdQRgiI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: So far there's no need in atomic setting of MM context flags in mm_context_t::flags. The flags set early in exec and never change after that. LAM enabling requires atomic flag setting. The upcoming flag MM_CONTEXT_FORCE_TAGGED_SVA can be set much later in the process lifetime where multiple threads exist. Convert the field to unsigned long and do MM_CONTEXT_* accesses with __set_bit() and test_bit(). No functional changes. Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/x86/entry/vsyscall/vsyscall_64.c | 2 +- arch/x86/include/asm/mmu.h | 6 +++--- arch/x86/include/asm/mmu_context.h | 2 +- arch/x86/kernel/process_64.c | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/x86/entry/vsyscall/vsyscall_64.c b/arch/x86/entry/vsyscall/vsyscall_64.c index d234ca797e4a..e0ca8120aea8 100644 --- a/arch/x86/entry/vsyscall/vsyscall_64.c +++ b/arch/x86/entry/vsyscall/vsyscall_64.c @@ -317,7 +317,7 @@ static struct vm_area_struct gate_vma __ro_after_init = { struct vm_area_struct *get_gate_vma(struct mm_struct *mm) { #ifdef CONFIG_COMPAT - if (!mm || !(mm->context.flags & MM_CONTEXT_HAS_VSYSCALL)) + if (!mm || !test_bit(MM_CONTEXT_HAS_VSYSCALL, &mm->context.flags)) return NULL; #endif if (vsyscall_mode == NONE) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 5d7494631ea9..efa3eaee522c 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -9,9 +9,9 @@ #include /* Uprobes on this MM assume 32-bit code */ -#define MM_CONTEXT_UPROBE_IA32 BIT(0) +#define MM_CONTEXT_UPROBE_IA32 0 /* vsyscall page is accessible on this MM */ -#define MM_CONTEXT_HAS_VSYSCALL BIT(1) +#define MM_CONTEXT_HAS_VSYSCALL 1 /* * x86 has arch-specific MMU state beyond what lives in mm_struct. @@ -39,7 +39,7 @@ typedef struct { #endif #ifdef CONFIG_X86_64 - unsigned short flags; + unsigned long flags; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index e01aa74a6de7..b4e4a0c04304 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -182,7 +182,7 @@ static inline void arch_exit_mmap(struct mm_struct *mm) static inline bool is_64bit_mm(struct mm_struct *mm) { return !IS_ENABLED(CONFIG_IA32_EMULATION) || - !(mm->context.flags & MM_CONTEXT_UPROBE_IA32); + !test_bit(MM_CONTEXT_UPROBE_IA32, &mm->context.flags); } #else static inline bool is_64bit_mm(struct mm_struct *mm) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 4e34b3b68ebd..8b06034e8c70 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -671,7 +671,7 @@ void set_personality_64bit(void) task_pt_regs(current)->orig_ax = __NR_execve; current_thread_info()->status &= ~TS_COMPAT; if (current->mm) - current->mm->context.flags = MM_CONTEXT_HAS_VSYSCALL; + __set_bit(MM_CONTEXT_HAS_VSYSCALL, ¤t->mm->context.flags); /* TBD: overwrites user setup. Should have two bits. But 64bit processes have always behaved this way, @@ -708,7 +708,7 @@ static void __set_personality_ia32(void) * uprobes applied to this MM need to know this and * cannot use user_64bit_mode() at that time. */ - current->mm->context.flags = MM_CONTEXT_UPROBE_IA32; + __set_bit(MM_CONTEXT_UPROBE_IA32, ¤t->mm->context.flags); } current->personality |= force_personality32; From patchwork Sun Mar 12 11:25:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171295 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9255CC6FD1F for ; Sun, 12 Mar 2023 11:26:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 769BF8E0002; Sun, 12 Mar 2023 07:26:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 719EA8E0001; Sun, 12 Mar 2023 07:26:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E1E78E0002; Sun, 12 Mar 2023 07:26:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4F16A8E0001 for ; Sun, 12 Mar 2023 07:26:33 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 136711C5FB6 for ; Sun, 12 Mar 2023 11:26:33 +0000 (UTC) X-FDA: 80560018266.15.6C191C8 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf10.hostedemail.com (Postfix) with ESMTP id 3BA57C0014 for ; Sun, 12 Mar 2023 11:26:30 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=k6t5RK1d; spf=none (imf10.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620391; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3Zx9Lz2Rg/MsgXQOxPaVxC9xMKS6+wm9X6Aawuvc2lA=; b=Qjqgsc33+1RF4QwEnEPQC4WH9jIGRrgG7YNEyEmnplCVQP5k684C+N3eQ6GCJ5h6ZcZooG sAf92+IB+v436KPsh7+uR9hmmjS4t4PhR0BCEhUPtGDNFvyIgdzmV7OHJGGqOidWqGhXTa 3tXRvkccuxAEshPtqawFw39MMCYopVs= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=k6t5RK1d; spf=none (imf10.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620391; a=rsa-sha256; cv=none; b=7onvuQMdWhEsID9qOJvZslSB2x0V7guRgNgZFsDa9A85dXWmedwBTgksluMeyC4tBJTCBN lqvGDvcs+b2Ops8m2OGMO1Zqe8cBdBfLvot2lXLEvte2svS6n2IqUaYi0FXwbUgHnvOgcT u+PCtUQ+LoZAOfZFvRIBYqmRU6fSlcE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620390; x=1710156390; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2IzVo3cRxUp5tsTunaSQVvSoWjALYgrz45WTbPv6U78=; b=k6t5RK1duITUAWKJyxhq3u/Gi9emBdpdp1ecoHU10PxHbL7h7LzMUVCG PiCw7z2FgfPA8/aSmOLtlVXR5VIQtUJ7Bonf5qmGB9xwX4mIFzBK21Xis 2FIwo0jP7MW8Rka8+wQAhjIjur7PI1FmHQBeXA+9RzSyd9V8ZOZsUqHZN 7GZSJUrUAdH+lACfbBbgRYijma8XWoxo/BDdOqziLgts2iK17iKJ+rcQh gw/o6bOB4sXFrULCWuk3sU+PiKPquli1dpVXAXjp/CLQsrST9WPJtsZNh ru2H3VglwpPXwp0b9vMvNn9RrU1Nvb5CWfJgZ7H3KWo9JEUhnB7adNPQe A==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="339349920" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="339349920" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="852438012" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="852438012" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:22 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 7098610D7B1; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv16 03/17] x86: CPUID and CR3/CR4 flags for Linear Address Masking Date: Sun, 12 Mar 2023 14:25:58 +0300 Message-Id: <20230312112612.31869-4-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: k7xhsmqxcfhz3zue3t3knu77g6yiipaa X-Rspamd-Queue-Id: 3BA57C0014 X-HE-Tag: 1678620390-212078 X-HE-Meta: U2FsdGVkX19lNXAD1X7HCePjEU+e2O0QrHh7sJ1pijy7Ok8vkQ3L9tNOvSEHvMDsck6RJaNEOkDwKqlk8BIS380sQSPCP9p8qAvCV4Lwh+MT6W0TOJMi15EnFKaZoRepfqjzzb4PJxf3L1Oq6lD7iJ27AFrRRDz5f0me/iyrGE7v6isEApp392ETu8gATOf+ZbcR15uWWPliHp7vW+e0Rs++RPqSNoywPut0Sy3oes9Klpo/u4G7tYnYBWFPJr2qCEbe0s+7kEBqa00pMVLt9mds1R93OYkPsr/sj7wbnmj/ZgMjXT+bFZNU50rpp5NzlhloNb/zJa3NnChkxn6rYHAcyB/p4Txj6UBSSwwokvJ5Fgs/XlOLETwExZYTt5lHyqt4Dx/0/FDS42Lz44q/zftL94J+TU5J5ENtytgAoOjK7bq1LcfhbY45PxzlQeEO5fEk0hSVg48Np3WR2n7/0eKL/6iO7HaEeqK6O4CDxcTI0iYRmVtecFcelpX+p/tSLW02chfxEb5r0KvZVTOj+kKVsUukdUiVsg+IGYdIAjUT1cYXbESNO8BLT/tcCTaE5jBif/41udyEtzUPAre+UilT26XQty1tGObObRIkmgLLKXlaxep0SlCXe9aaEWHHm7bYrwHxWvZJ6cXbhVro7sGYHi6ZdK8av9i8ocEC4dZAulh4ljqvtl0i1UZsvzjt+rQU1MUQc4RGUtKCvu+ymO3O+L/cZdBYwsylZA2SnRp9BsWEfpfaTD3LY847AlZH3xF1OrvAb/8jeSAI6F6KELZ5GLTmVnrOBSfehJaY8nlEDH/85MoNoJ5PzKnm1jBeDe4XncvffYb65+vkBOlrL302J0EB/d+V5u87tQggmEQ1W8aXJECIUVY6xGBd6vpUE1kK4zKyrLg9pv9Fugx4YQNlg2Hu1jCvx20drUpBWIPZQRMpcfYL/mQEbnlKAp3O0WzZOxd2qCovsgBLm4I 4tqfQch8 UqXj7ke7dRLpsAg6nUhxmtTugvlWnWNcLbHnNRIu9YUNBftJjN/w2MVsVzGusQFmAyJj7vobuMVHJwcw1+WVhdoLwDQCEtHG+Wfnq9bMTQZ2txAep0c2ymGhzVFTxIo9V0Ivp+67MWvkM3SHbBIKfaJj14rIduvwkNDnS54GBKvCSIXTVO3mmBXfsNK937ED4z9IHhcqvW7jh0ljOp3SjibosP9/KkKV1Hkyp1kpoluUe+MEzCbmR0pJhEJPJRd+CkyHXG6kN2EoCEZKmes0Y7CAeAjkSK8cHQy0/e255aKJ8rjBvA8QRulPw8t4VNH6iyarMMJZoMwTKRm6H8Aalf+XFgcHJqUBdL6fzJ4SsDWilgKg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Enumerate Linear Address Masking and provide defines for CR3 and CR4 flags. The new CONFIG_ADDRESS_MASKING option enables the feature support in kernel. Signed-off-by: Kirill A. Shutemov Reviewed-by: Alexander Potapenko Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/x86/Kconfig | 11 +++++++++++ arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/processor-flags.h | 2 ++ arch/x86/include/uapi/asm/processor-flags.h | 6 ++++++ 4 files changed, 20 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index a825bf031f49..aa9f73f5d670 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -2290,6 +2290,17 @@ config RANDOMIZE_MEMORY_PHYSICAL_PADDING If unsure, leave at the default value. +config ADDRESS_MASKING + bool "Linear Address Masking support" + depends on X86_64 + help + Linear Address Masking (LAM) modifies the checking that is applied + to 64-bit linear addresses, allowing software to use of the + untranslated address bits for metadata. + + The capability can be used for efficient address sanitizers (ASAN) + implementation and for optimizations in JITs. + config HOTPLUG_CPU def_bool y depends on SMP diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 73c9672c123b..353b054812de 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -321,6 +321,7 @@ #define X86_FEATURE_LKGS (12*32+18) /* "" Load "kernel" (userspace) GS */ #define X86_FEATURE_AMX_FP16 (12*32+21) /* "" AMX fp16 Support */ #define X86_FEATURE_AVX_IFMA (12*32+23) /* "" Support for VPMADD52[H,L]UQ */ +#define X86_FEATURE_LAM (12*32+26) /* Linear Address Masking */ /* AMD-defined CPU features, CPUID level 0x80000008 (EBX), word 13 */ #define X86_FEATURE_CLZERO (13*32+ 0) /* CLZERO instruction */ diff --git a/arch/x86/include/asm/processor-flags.h b/arch/x86/include/asm/processor-flags.h index a7f3d9100adb..d8cccadc83a6 100644 --- a/arch/x86/include/asm/processor-flags.h +++ b/arch/x86/include/asm/processor-flags.h @@ -28,6 +28,8 @@ * On systems with SME, one bit (in a variable position!) is stolen to indicate * that the top-level paging structure is encrypted. * + * On systemms with LAM, bits 61 and 62 are used to indicate LAM mode. + * * All of the remaining bits indicate the physical address of the top-level * paging structure. * diff --git a/arch/x86/include/uapi/asm/processor-flags.h b/arch/x86/include/uapi/asm/processor-flags.h index c47cc7f2feeb..d898432947ff 100644 --- a/arch/x86/include/uapi/asm/processor-flags.h +++ b/arch/x86/include/uapi/asm/processor-flags.h @@ -82,6 +82,10 @@ #define X86_CR3_PCID_BITS 12 #define X86_CR3_PCID_MASK (_AC((1UL << X86_CR3_PCID_BITS) - 1, UL)) +#define X86_CR3_LAM_U57_BIT 61 /* Activate LAM for userspace, 62:57 bits masked */ +#define X86_CR3_LAM_U57 _BITULL(X86_CR3_LAM_U57_BIT) +#define X86_CR3_LAM_U48_BIT 62 /* Activate LAM for userspace, 62:48 bits masked */ +#define X86_CR3_LAM_U48 _BITULL(X86_CR3_LAM_U48_BIT) #define X86_CR3_PCID_NOFLUSH_BIT 63 /* Preserve old PCID */ #define X86_CR3_PCID_NOFLUSH _BITULL(X86_CR3_PCID_NOFLUSH_BIT) @@ -132,6 +136,8 @@ #define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT) #define X86_CR4_CET_BIT 23 /* enable Control-flow Enforcement Technology */ #define X86_CR4_CET _BITUL(X86_CR4_CET_BIT) +#define X86_CR4_LAM_SUP_BIT 28 /* LAM for supervisor pointers */ +#define X86_CR4_LAM_SUP _BITUL(X86_CR4_LAM_SUP_BIT) /* * x86-64 Task Priority Register, CR8 From patchwork Sun Mar 12 11:25:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171304 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D75DC6FA99 for ; Sun, 12 Mar 2023 11:26:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EBD608E000C; Sun, 12 Mar 2023 07:26:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E6C908E0008; Sun, 12 Mar 2023 07:26:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D0E118E000C; Sun, 12 Mar 2023 07:26:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C1E208E0008 for ; Sun, 12 Mar 2023 07:26:54 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9D4D5C09B6 for ; Sun, 12 Mar 2023 11:26:54 +0000 (UTC) X-FDA: 80560019148.09.134B4F1 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf17.hostedemail.com (Postfix) with ESMTP id 59E8540004 for ; Sun, 12 Mar 2023 11:26:52 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=nsXrJa80; spf=none (imf17.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620412; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JmeFT4XsooLL6hqolm60pP6vkTBUyGbftSfH5ntda+k=; b=vwAYbTh1OM4CIXb7m9tCsTfPGQttRCFlWTVuhRIoz+t7AwfqKs5PMthBA6pBDTWuNXi3Ah XHc06i23iLyslCXGNX5ZhG6ae1MWdJ3cgJuYDZ1hS2JKK8ze+6Jci1f8YWf99Mf9FtSKdi oYq6okmog7n01b31MROVrxbUtvfILM4= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=nsXrJa80; spf=none (imf17.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620412; a=rsa-sha256; cv=none; b=VQckuphOxBzVnuhsTTpKLPahqyMWoXHLmdsFp8cLQAiBvRf2V2sLhB+ZVWzm64AATa0Lcn esuza7+Ypgv/zXuz30tRJpkwIy9BjUc+2Pzf/ADt61Bdpm9TVWwhGoqQYhMcZmWuF9mTnc uAlHXqe7723YO/rRWfhI1AliQtVEQA4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620412; x=1710156412; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=YaUjkq9kXDyFDQlhaJ/Oy8LFGHPqcPvlEVROgb02kms=; b=nsXrJa80Jp1iQH0DZh4rr1WH/4ov8V5Exb3Fw5qgZ92q1V0oUvrbiBxq L0GUq1JAoMIHRpFEfwFxUuUbl2Y3brkVwTyonlbQdCmUubzkdIwU2owu5 JH2kktcWyeVy5JWNzxOQCZs4iAJWDDqnJ9lIpBfQl/KErawF6FUWtiL3H DQS8Odkhwjbz5uic+8I3cHO5yZ6C7nDY3e4SsJt+ntNxc1Mc/UqlxS2a2 5/GiV4VtrUa3rjbVyAGCn9QqIAOuKkffiNc9TK0/alB4K9W2lR5B634/V d7M9pntO+/ouZdugKLhL8/xBWCAlhGo/Z45QVBLkNEz/YsfJovBVzy88/ Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="399577513" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="399577513" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="671580716" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="671580716" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:22 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 7B2D110D7B2; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv16 04/17] x86/mm: Handle LAM on context switch Date: Sun, 12 Mar 2023 14:25:59 +0300 Message-Id: <20230312112612.31869-5-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 59E8540004 X-Stat-Signature: xxyu93ghsaq5f594e813qhbhe931tgdr X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1678620412-623209 X-HE-Meta: U2FsdGVkX1/l3HGDHRtvPPF6vulT4z00AXjKKjnHHntwJY8r3EK8j/sd9f4Fg9vKMCUMIS5IuETvZFnCPrFRpYjUfHAm06VBW0KsROppmGPhWqKAadKQfl1+aSnZZFeSoOklJ8PkjXmSPuBGddQLP3gjKA22vwuuglPbsjUe/O1NdZILI1piNvDt9qfyJixXlA0i6jTYP4AQ7JWvlpTj5I3yD6cUqZe3bosnW9Ml4BTQwlXOQ0rUSRwADFsG1Uzrb3yYR8av6pVYkjUihxvlYpu52euDZcd99Ve/W/V25IMw5cBXysmG29Gk2kQT7Tv5P2WF1xgDZj25yGFoeBeczVsFWi98muZ6Sn5DV3uqNtQdyno1er3v0oCcGzuXvW7PyCgjmXL+Fogk9p6PqIWMYe2KC2s+uioH/bHRJkltEp2L7atv958w5Bl1Kb6TZQjoiylCoB3BTwKv8FP0dmqQGRBcLsAA/WrZy2kM9PnhRHVYjrL16LDVUbQy4iX5N4BPajmYdElkyFnOkUVvSVFch7vVfkskGsb6Rhk5gq3ctPbuBQLJJXuKOoc1tym7+HTRE+yAzYTqMNdqaLNyuHT4f4HggAyN6aBb0LNPDAjheR6jNeS/bOgg8ay5t+cjPu1eVsS3VF3ElMM2KvGp7JUtBk8aQvtWmz7ZLvY0k/wMsaOhw5pH4726As3Zn7BoEuMJfn1+NLB7/fpj9YTo0nTHhqPQ52st5qFi/XojR9+5pGBoLGVrSn5w9WJ2UbsrXZYzI19BuChpf22Gcf4oqWQ/CZI3SWT/eSTfaBRHAyVIsfk0Z6gYdEH2oWHjO8t/VFszeCNOYPjSSaqiTTkI8cbfB9UhElx9J7Kpz7IGtoNd0XbwFmM6bGU5bTccHwzoJASoPAHi6l1CzfSKnzdTkKahG/a7fJ3/o0ioZB069gK1Z/3z4XZM40PRSxxUfAcGed8RcvQNfHYw4dA0Xd7KDEO 2hwp0lrT yYmfc9eR7UONjy4ezevtQCOkDrHETLd/mRtazhxAjlKp8jk664OMEiVziKKnSx+bg0wwFNY4O417GZYWhPI3x740CekkWtr2mCK50M3gyrnOANzB2MBPOEeWlpZSG/60Iswi5OeEfgJRKAdZ1uPwT2WQ4qTPHOdxmP9IG9/+KJuT7fcAOHuWLM+hkVp6OoHqxfX9m6FaPMhS3jYDEMCon6b+j8Sfbfimoq4I9nwyQIkOvTRPAyEYgUzQWoV+fMjvQteykQF0P+c1YdOXBMPITNw2TCucpVHzMsOK2c1tIJETdOX2AG/89SGG69Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Linear Address Masking mode for userspace pointers encoded in CR3 bits. The mode is selected per-process and stored in mm_context_t. switch_mm_irqs_off() now respects selected LAM mode and constructs CR3 accordingly. The active LAM mode gets recorded in the tlb_state. Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/x86/include/asm/mmu.h | 5 +++ arch/x86/include/asm/mmu_context.h | 24 ++++++++++++++ arch/x86/include/asm/tlbflush.h | 38 ++++++++++++++++++++- arch/x86/mm/tlb.c | 53 +++++++++++++++++++++--------- 4 files changed, 103 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index efa3eaee522c..22fc9fbf1d0a 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -42,6 +42,11 @@ typedef struct { unsigned long flags; #endif +#ifdef CONFIG_ADDRESS_MASKING + /* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ + unsigned long lam_cr3_mask; +#endif + struct mutex lock; void __user *vdso; /* vdso base address */ const struct vdso_image *vdso_image; /* vdso image in use */ diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index b4e4a0c04304..0295c3863db7 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -92,6 +92,29 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next) } #endif +#ifdef CONFIG_ADDRESS_MASKING +static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) +{ + return mm->context.lam_cr3_mask; +} + +static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) +{ + mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; +} + +#else + +static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) +{ + return 0; +} + +static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) +{ +} +#endif + #define enter_lazy_tlb enter_lazy_tlb extern void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk); @@ -169,6 +192,7 @@ static inline int arch_dup_mmap(struct mm_struct *oldmm, struct mm_struct *mm) { arch_dup_pkeys(oldmm, mm); paravirt_arch_dup_mmap(oldmm, mm); + dup_lam(oldmm, mm); return ldt_dup_context(oldmm, mm); } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index cda3118f3b27..e8b47f57bd4a 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -2,7 +2,7 @@ #ifndef _ASM_X86_TLBFLUSH_H #define _ASM_X86_TLBFLUSH_H -#include +#include #include #include @@ -12,6 +12,7 @@ #include #include #include +#include void __flush_tlb_all(void); @@ -101,6 +102,16 @@ struct tlb_state { */ bool invalidate_other; +#ifdef CONFIG_ADDRESS_MASKING + /* + * Active LAM mode. + * + * X86_CR3_LAM_U57/U48 shifted right by X86_CR3_LAM_U57_BIT or 0 if LAM + * disabled. + */ + u8 lam; +#endif + /* * Mask that contains TLB_NR_DYN_ASIDS+1 bits to indicate * the corresponding user PCID needs a flush next time we @@ -357,6 +368,31 @@ static inline bool huge_pmd_needs_flush(pmd_t oldpmd, pmd_t newpmd) } #define huge_pmd_needs_flush huge_pmd_needs_flush +#ifdef CONFIG_ADDRESS_MASKING +static inline u64 tlbstate_lam_cr3_mask(void) +{ + u64 lam = this_cpu_read(cpu_tlbstate.lam); + + return lam << X86_CR3_LAM_U57_BIT; +} + +static inline void set_tlbstate_lam_mode(struct mm_struct *mm) +{ + this_cpu_write(cpu_tlbstate.lam, + mm->context.lam_cr3_mask >> X86_CR3_LAM_U57_BIT); +} + +#else + +static inline u64 tlbstate_lam_cr3_mask(void) +{ + return 0; +} + +static inline void set_tlbstate_lam_mode(struct mm_struct *mm) +{ +} +#endif #endif /* !MODULE */ static inline void __native_tlb_flush_global(unsigned long cr4) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 92d73ccede70..724f98d0e10f 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -154,26 +154,30 @@ static inline u16 user_pcid(u16 asid) return ret; } -static inline unsigned long build_cr3(pgd_t *pgd, u16 asid) +static inline unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam) { + unsigned long cr3 = __sme_pa(pgd) | lam; + if (static_cpu_has(X86_FEATURE_PCID)) { - return __sme_pa(pgd) | kern_pcid(asid); + VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); + cr3 |= kern_pcid(asid); } else { VM_WARN_ON_ONCE(asid != 0); - return __sme_pa(pgd); } + + return cr3; } -static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid) +static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid, + unsigned long lam) { - VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); /* * Use boot_cpu_has() instead of this_cpu_has() as this function * might be called during early boot. This should work even after * boot because all CPU's the have same capabilities: */ VM_WARN_ON_ONCE(!boot_cpu_has(X86_FEATURE_PCID)); - return __sme_pa(pgd) | kern_pcid(asid) | CR3_NOFLUSH; + return build_cr3(pgd, asid, lam) | CR3_NOFLUSH; } /* @@ -274,15 +278,16 @@ static inline void invalidate_user_asid(u16 asid) (unsigned long *)this_cpu_ptr(&cpu_tlbstate.user_pcid_flush_mask)); } -static void load_new_mm_cr3(pgd_t *pgdir, u16 new_asid, bool need_flush) +static void load_new_mm_cr3(pgd_t *pgdir, u16 new_asid, unsigned long lam, + bool need_flush) { unsigned long new_mm_cr3; if (need_flush) { invalidate_user_asid(new_asid); - new_mm_cr3 = build_cr3(pgdir, new_asid); + new_mm_cr3 = build_cr3(pgdir, new_asid, lam); } else { - new_mm_cr3 = build_cr3_noflush(pgdir, new_asid); + new_mm_cr3 = build_cr3_noflush(pgdir, new_asid, lam); } /* @@ -491,6 +496,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, { struct mm_struct *real_prev = this_cpu_read(cpu_tlbstate.loaded_mm); u16 prev_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); + unsigned long new_lam = mm_lam_cr3_mask(next); bool was_lazy = this_cpu_read(cpu_tlbstate_shared.is_lazy); unsigned cpu = smp_processor_id(); u64 next_tlb_gen; @@ -520,7 +526,8 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * isn't free. */ #ifdef CONFIG_DEBUG_VM - if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid))) { + if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid, + tlbstate_lam_cr3_mask()))) { /* * If we were to BUG here, we'd be very likely to kill * the system so hard that we don't see the call trace. @@ -552,9 +559,15 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * instruction. */ if (real_prev == next) { + /* Not actually switching mm's */ VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) != next->context.ctx_id); + /* + * If this races with another thread that enables lam, 'new_lam' + * might not match tlbstate_lam_cr3_mask(). + */ + /* * Even in lazy TLB mode, the CPU should stay set in the * mm_cpumask. The TLB shootdown code can figure out from @@ -622,15 +635,16 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, barrier(); } + set_tlbstate_lam_mode(next); if (need_flush) { this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen); - load_new_mm_cr3(next->pgd, new_asid, true); + load_new_mm_cr3(next->pgd, new_asid, new_lam, true); trace_tlb_flush(TLB_FLUSH_ON_TASK_SWITCH, TLB_FLUSH_ALL); } else { /* The new ASID is already up to date. */ - load_new_mm_cr3(next->pgd, new_asid, false); + load_new_mm_cr3(next->pgd, new_asid, new_lam, false); trace_tlb_flush(TLB_FLUSH_ON_TASK_SWITCH, 0); } @@ -691,6 +705,10 @@ void initialize_tlbstate_and_flush(void) /* Assert that CR3 already references the right mm. */ WARN_ON((cr3 & CR3_ADDR_MASK) != __pa(mm->pgd)); + /* LAM expected to be disabled */ + WARN_ON(cr3 & (X86_CR3_LAM_U48 | X86_CR3_LAM_U57)); + WARN_ON(mm_lam_cr3_mask(mm)); + /* * Assert that CR4.PCIDE is set if needed. (CR4.PCIDE initialization * doesn't work like other CR4 bits because it can only be set from @@ -699,8 +717,8 @@ void initialize_tlbstate_and_flush(void) WARN_ON(boot_cpu_has(X86_FEATURE_PCID) && !(cr4_read_shadow() & X86_CR4_PCIDE)); - /* Force ASID 0 and force a TLB flush. */ - write_cr3(build_cr3(mm->pgd, 0)); + /* Disable LAM, force ASID 0 and force a TLB flush. */ + write_cr3(build_cr3(mm->pgd, 0, 0)); /* Reinitialize tlbstate. */ this_cpu_write(cpu_tlbstate.last_user_mm_spec, LAST_USER_MM_INIT); @@ -708,6 +726,7 @@ void initialize_tlbstate_and_flush(void) this_cpu_write(cpu_tlbstate.next_asid, 1); this_cpu_write(cpu_tlbstate.ctxs[0].ctx_id, mm->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[0].tlb_gen, tlb_gen); + set_tlbstate_lam_mode(mm); for (i = 1; i < TLB_NR_DYN_ASIDS; i++) this_cpu_write(cpu_tlbstate.ctxs[i].ctx_id, 0); @@ -1071,8 +1090,10 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) */ unsigned long __get_current_cr3_fast(void) { - unsigned long cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, - this_cpu_read(cpu_tlbstate.loaded_mm_asid)); + unsigned long cr3 = + build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, + this_cpu_read(cpu_tlbstate.loaded_mm_asid), + tlbstate_lam_cr3_mask()); /* For now, be very restrictive about when this can be called. */ VM_WARN_ON(in_nmi() || preemptible()); From patchwork Sun Mar 12 11:26:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36BF6C6FA99 for ; Sun, 12 Mar 2023 11:26:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C24FF8E0007; Sun, 12 Mar 2023 07:26:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BD5C78E0001; Sun, 12 Mar 2023 07:26:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A4E358E0007; Sun, 12 Mar 2023 07:26:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 981438E0001 for ; Sun, 12 Mar 2023 07:26:38 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 6AAF61C5FBC for ; Sun, 12 Mar 2023 11:26:38 +0000 (UTC) X-FDA: 80560018476.01.D669FAA Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf14.hostedemail.com (Postfix) with ESMTP id 32D9110000D for ; Sun, 12 Mar 2023 11:26:35 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=NRF0ntB8; spf=none (imf14.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620396; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+EDOkCcFvj2LTHdnZWHE+HYEtQCPudT/tOJKFx84T2I=; b=YlRrWVyF+fpPZ42il7yWeyTYsadGuJuZx2HJ4dnfT+LOW9xTKqqjJGEhZMZmQwk7mPCCCT tZ7/zfxhzI3EUtdPbw304uFVPmLLjUH2ayRc0nR5ILQKS31tkVpYo9ea1ckyh5LDH/l1Tk /nFmD0644i7kC30WGDkpH9jwTNr7Zns= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=NRF0ntB8; spf=none (imf14.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620396; a=rsa-sha256; cv=none; b=buXuVPEm9eQMt9hCWdKfOd5r7lNNnmMteKjRtvJv1Cu3EiAWZ2RBRkyjTF7sCHA00zoIyF PuKQqBVxmN1EqTePBD9JfOwMgUB8FxqINd7n2p4GR65XFGKvaZeodSWMt+1uCe8ubHJaCD rLxIuYXdiTXbF1Xi5ESTxnrB9oiC3oM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620396; x=1710156396; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jhNKWlLSFQdtM5nCCWCne2jNIxj0oFZDDW74FDfmA8g=; b=NRF0ntB8riqF1pN8aA7F6+85mTZyCoGV1WMKVLaWUQEbJZMlKEG3eMRM o24Ths9sYay86XVfpoK0dehDutGwZQ4ZVYlKR/UqJcET5BxpJDJeUpKFo 7vdSu5MlsC3gR8vD2VdvSJWhLDsuyJT8HCeC4xQ8cKLBJ7OGRRlN0OWSl TRKiUFhm0AYa4+7F3wW4Qp4kXejoPbFjzj++51Y6K+x6pDz7A7Gr0TEZC L73iaYFEWlClur0ETYlRHqe6BxASyiPrzsJ2WNqmvCR99ye0J9bhqxGFv LuzIslVCir8NwER8nriCHoRAtNYShCGCGMZgiKlQBhCgl582PhX4achoS A==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="339349954" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="339349954" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="852438029" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="852438029" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:30 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 84A2710D7B3; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv16 05/17] mm: Introduce untagged_addr_remote() Date: Sun, 12 Mar 2023 14:26:00 +0300 Message-Id: <20230312112612.31869-6-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 32D9110000D X-Rspam-User: X-Stat-Signature: 3d3yzcq8bs3ib6gkz8popifjtfqm3a3h X-HE-Tag: 1678620395-472602 X-HE-Meta: U2FsdGVkX18/4YQRJWoXWRaqX9O7hC5p6ZROD15RE/leYeaK7RN3dVNYGILbX7emS15462s+Yrcl6pvV5f/8U024zVqDGkpDNhCkgvctn1WvP/u1G/NTqqt22jMvIity8spzdBfuxi4C220ftSSBQgZEbv5zmNC+CbrCeE2nRTPAddW/gcwuuETs3xHYhQlaBQnX3z0l2RbseDOraqjfr5i6ziOHZkh8JhzPgxjZoN0ZZLSnMOkCh7wm42u/fkcb8o5zVSVAkjVUxZV8L7grhm/2dftFnYZo8pw0fJAw5PIrjKl3Okf6kX4Q6BoSHKIUetCTVuT7GQrc0r/3o6wdx5+/rHQGvfnP6aX/X5mN+mE9lRGFzipxkSADr3/VdPxtRVX78c3bSc9hkT+ZGVFKB5z+itnGZdK/wf5ZumNVKZP0eJn5WZ+S66SFrgrDF3cSl/aRQi0KJqXYNE8aKO+mFSXBvHhdCJWRT+1tEuktx2SO6HqUUr9gZ7XnV5NMuTsoEehSBIPe7utXjGrAav36At4KMZ5MeD5xmmS9DMxj6fGEcOjsRccaUhIGFLApPGKMl5OYFpcb4mI3K1dMmUmA41ZZqjhKTelYtiHMr86OFYMrOyhJPt+rxcKiPk3Xcfguo/N1UMSRyfe3NgfNSVRHdQC7HmKfUskv2s+j/W18NX7Ly+HW9FHjjrRDWK7gt2BX+lrUTqj+lvMdikcklqrfC+RqauLi8Lnv78kaGxXif2b2aHlgcQN35VN6sm5iLQc2MhV8VHB5+XYODYiGfl6Hhm2WxViyNLljQof/e79YCk/XeCiM5RXTiSEd5sS16OftoPLRnr8RewxBLaDQwwoOa3uNp+zfi7ZYPNpzVkvPyHG55v4q3oZQwtKBGN081w8xlFbGnGnB1lmj2S8b31RV3+ho0K6swM2bWV7bW1mIoqssgIG+jMWxSmX+3O46kcsEFYFm8W7+J6SJHL4ADt4 dz5lROy3 OO52DFl49UNUUEXqsE2iygZI6wNnEKSkLTZKPnX7u9BBFFxCRJov2PQXY2oy6+Er3INHPm5yKYHxPhdVmiQYzgVAvcGRJMLbjp36eLGIcYmp9w6l/Nqo4p8nBdd9rJa/oId7sdaKLUF93mbKCmuqGr5sDFNBBXTR9keQuzlvlCBy4qFOgBqSUnYvn5EgwdxgSSfQ7dRZJi6FtgAe79AQOqt6W3r01cOmsPsGZ1Ifh1WJUVBCw1abpGRgwk0rb+YwZjxIt3VagbmcTIpboTGOjewA34TvoAEkT0x3m2oRJXrMOsr1NocXAkh77fugbYofttofVRDt4Ya1IebxfEh49QBlIPNm2RJ2q8/46C24MZZNqr5M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: untagged_addr() removes tags/metadata from the address and brings it to the canonical form. The helper is implemented on arm64 and sparc. Both of them do untagging based on global rules. However, Linear Address Masking (LAM) on x86 introduces per-process settings for untagging. As a result, untagged_addr() is now only suitable for untagging addresses for the current proccess. The new helper untagged_addr_remote() has to be used when the address targets remote process. It requires the mmap lock for target mm to be taken. Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko Reviewed-by: Rick Edgecombe --- arch/sparc/include/asm/uaccess_64.h | 2 ++ drivers/vfio/vfio_iommu_type1.c | 2 +- fs/proc/task_mmu.c | 9 +++++++-- include/linux/mm.h | 11 ----------- include/linux/uaccess.h | 22 ++++++++++++++++++++++ mm/gup.c | 4 ++-- mm/madvise.c | 5 +++-- mm/migrate.c | 11 ++++++----- 8 files changed, 43 insertions(+), 23 deletions(-) diff --git a/arch/sparc/include/asm/uaccess_64.h b/arch/sparc/include/asm/uaccess_64.h index 94266a5c5b04..b825a5dd0210 100644 --- a/arch/sparc/include/asm/uaccess_64.h +++ b/arch/sparc/include/asm/uaccess_64.h @@ -8,8 +8,10 @@ #include #include +#include #include #include +#include #include #include diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 493c31de0edb..3d4dd9420c30 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -580,7 +580,7 @@ static int vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, goto done; } - vaddr = untagged_addr(vaddr); + vaddr = untagged_addr_remote(mm, vaddr); retry: vma = vma_lookup(mm, vaddr); diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 6a96e1713fd5..29fd6b1f4058 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1689,8 +1689,13 @@ static ssize_t pagemap_read(struct file *file, char __user *buf, /* watch out for wraparound */ start_vaddr = end_vaddr; - if (svpfn <= (ULONG_MAX >> PAGE_SHIFT)) - start_vaddr = untagged_addr(svpfn << PAGE_SHIFT); + if (svpfn <= (ULONG_MAX >> PAGE_SHIFT)) { + ret = mmap_read_lock_killable(mm); + if (ret) + goto out_free; + start_vaddr = untagged_addr_remote(mm, svpfn << PAGE_SHIFT); + mmap_read_unlock(mm); + } /* Ensure the address is inside the task */ if (start_vaddr > mm->task_size) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1f79667824eb..289ae4caf878 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -96,17 +96,6 @@ extern int mmap_rnd_compat_bits __read_mostly; #include #include -/* - * Architectures that support memory tagging (assigning tags to memory regions, - * embedding these tags into addresses that point to these memory regions, and - * checking that the memory and the pointer tags match on memory accesses) - * redefine this macro to strip tags from pointers. - * It's defined as noop for architectures that don't support memory tagging. - */ -#ifndef untagged_addr -#define untagged_addr(addr) (addr) -#endif - #ifndef __pa_symbol #define __pa_symbol(x) __pa(RELOC_HIDE((unsigned long)(x), 0)) #endif diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h index ab9728138ad6..3064314f4832 100644 --- a/include/linux/uaccess.h +++ b/include/linux/uaccess.h @@ -10,6 +10,28 @@ #include +/* + * Architectures that support memory tagging (assigning tags to memory regions, + * embedding these tags into addresses that point to these memory regions, and + * checking that the memory and the pointer tags match on memory accesses) + * redefine this macro to strip tags from pointers. + * + * Passing down mm_struct allows to define untagging rules on per-process + * basis. + * + * It's defined as noop for architectures that don't support memory tagging. + */ +#ifndef untagged_addr +#define untagged_addr(addr) (addr) +#endif + +#ifndef untagged_addr_remote +#define untagged_addr_remote(mm, addr) ({ \ + mmap_assert_locked(mm); \ + untagged_addr(addr); \ +}) +#endif + /* * Architectures should provide two primitives (raw_copy_{to,from}_user()) * and get rid of their private instances of copy_{to,from}_user() and diff --git a/mm/gup.c b/mm/gup.c index eab18ba045db..5ee8b682a0fe 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1085,7 +1085,7 @@ static long __get_user_pages(struct mm_struct *mm, if (!nr_pages) return 0; - start = untagged_addr(start); + start = untagged_addr_remote(mm, start); VM_BUG_ON(!!pages != !!(gup_flags & (FOLL_GET | FOLL_PIN))); @@ -1259,7 +1259,7 @@ int fixup_user_fault(struct mm_struct *mm, struct vm_area_struct *vma; vm_fault_t ret; - address = untagged_addr(address); + address = untagged_addr_remote(mm, address); if (unlocked) fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; diff --git a/mm/madvise.c b/mm/madvise.c index 340125d08c03..d4b67f36f70f 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -1402,8 +1402,6 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh size_t len; struct blk_plug plug; - start = untagged_addr(start); - if (!madvise_behavior_valid(behavior)) return -EINVAL; @@ -1435,6 +1433,9 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh mmap_read_lock(mm); } + start = untagged_addr_remote(mm, start); + end = start + len; + blk_start_plug(&plug); error = madvise_walk_vmas(mm, start, end, behavior, madvise_vma_behavior); diff --git a/mm/migrate.c b/mm/migrate.c index 98f1c11197a8..8cd11bc9208f 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2097,15 +2097,18 @@ static int do_move_pages_to_node(struct mm_struct *mm, * target node * 1 - when it has been queued */ -static int add_page_for_migration(struct mm_struct *mm, unsigned long addr, +static int add_page_for_migration(struct mm_struct *mm, const void __user *p, int node, struct list_head *pagelist, bool migrate_all) { struct vm_area_struct *vma; + unsigned long addr; struct page *page; int err; bool isolated; mmap_read_lock(mm); + addr = (unsigned long)untagged_addr_remote(mm, p); + err = -EFAULT; vma = vma_lookup(mm, addr); if (!vma || !vma_migratable(vma)) @@ -2211,7 +2214,6 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, for (i = start = 0; i < nr_pages; i++) { const void __user *p; - unsigned long addr; int node; err = -EFAULT; @@ -2219,7 +2221,6 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, goto out_flush; if (get_user(node, nodes + i)) goto out_flush; - addr = (unsigned long)untagged_addr(p); err = -ENODEV; if (node < 0 || node >= MAX_NUMNODES) @@ -2247,8 +2248,8 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, * Errors in the page lookup or isolation are not fatal and we simply * report them via status */ - err = add_page_for_migration(mm, addr, current_node, - &pagelist, flags & MPOL_MF_MOVE_ALL); + err = add_page_for_migration(mm, p, current_node, &pagelist, + flags & MPOL_MF_MOVE_ALL); if (err > 0) { /* The page is successfully queued for migration */ From patchwork Sun Mar 12 11:26:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171306 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C9ABC6FA99 for ; Sun, 12 Mar 2023 11:27:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A29778E000F; Sun, 12 Mar 2023 07:27:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B2088E000E; Sun, 12 Mar 2023 07:27:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 82C428E000F; Sun, 12 Mar 2023 07:27:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 717D68E0008 for ; Sun, 12 Mar 2023 07:27:04 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4E8AF1A0806 for ; Sun, 12 Mar 2023 11:27:04 +0000 (UTC) X-FDA: 80560019568.11.4C0711C Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf17.hostedemail.com (Postfix) with ESMTP id 20A0F40006 for ; Sun, 12 Mar 2023 11:27:01 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=WEMHrPg8; spf=none (imf17.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620422; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X513hi487xr2Y5cmAhhhOp2BVxYcOzKFwOGV+7peITQ=; b=eWuaWgAkUYWn2vNL8HAxlUYZgvBC+dN1yyBJjCD46V0TLSAqDFOb1R50KwM0dnBG0BWp1z DdPnK8bAYEsDQFR6gsPCOglzXXBhteuwxkoY4i/l9mXbN0f78BCC2cBA2SLZm+LTRkU4Kn zcIx3ienQdJiy8PbY2wvu32xYDDXwzA= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=WEMHrPg8; spf=none (imf17.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620422; a=rsa-sha256; cv=none; b=mCafHxtzQm6UkAkDbT5ddWKMUE4dc9wr8POhnSWo26+wqU3EgptQX29+z9/xUwY4JXHY9b KEPG9Andly7IZBKby1qORXS2eXOlN7gTM303/h7kLbUizKYt6y+NumfJnST+pcmx1lGqbW jA2cTnb/5exKNImYtXLRQBjcGw+VRgs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620422; x=1710156422; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Tq6WtwHrazfLXD0XBMc2Q+r2SDElsHETE3zLNMxGwxY=; b=WEMHrPg8XAsq/zXeVE5iM5x9L3OS9aUd707kZyuAecHJtXvL/THrzY9p wVrcIAR+gMWelqMnYy8Pv/OIC+oBB5JUKaA6REce+d3h3jYG0XCfhOARh h7VklkIaCu9KNIbgp99OJiOCRtLJXH+1gRjVHvPIY6O+76Q7M1sJhO3Ky CtdNv3JgOBBTjsd2CCUW51lgYg0wqSN90dMRNVtft1Z6eg1pMuT40G7PI qc4ny7GA49K/uWrh8zDLp2cB/hdP8T+vLvin4emadR9dgA8Fid/2/nnlZ GXgifYiQzdkxvyJUQoCJu2YKfMxwiHZM7kV7EjEd2F6wyhvuwUSBy0VWE g==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="399577576" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="399577576" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:36 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="671580735" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="671580735" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:30 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 8FAA510D7B4; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv16 06/17] x86/uaccess: Provide untagged_addr() and remove tags before address check Date: Sun, 12 Mar 2023 14:26:01 +0300 Message-Id: <20230312112612.31869-7-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 20A0F40006 X-Stat-Signature: rh1hgq4xw156yzetp6orupkowwqbspgj X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1678620421-166114 X-HE-Meta: U2FsdGVkX1/4VctczLRe3qnXwgvZjIoFkH6Tgv898Jh26lX61SM4mKRFUWnNeygE889ImITRcJiGXKxTPi/wVVem/E0zSFRaWqb4ZvJSRdZU3DsFLt+uj8e+6mH3Xug7eaT6XYRAM1b0mSMhZgkjiXEdlDH8D6rKf2pMlwF7qxJJMwx4dCdTPmL5anolKqEVx4apLJNb0b0ZrbvFLsGaao5ShjKzbBSlb/EvMZ0LoHu3xvPedjXS1OtVh9IBXsVP7/Yzu8m1rQO8ojZSgvV4ZaZXxBlRP0m4dXZtLTLujdIt3B3A3ipMbu8xgD+qEQqIOkUvhjqmf0Xs9eBuVimEjCAKE7n9Ixx77Gtw7OTrnmFKLkBwOnaKDLRnYdI8mywmszpOltuZoIC8h39fErUlsmde0LqbGssuSOHChOAyajbOP0sKF94R7FTx2RO3/nWq55GNUwbgxv/JUSa+s8DAVcLk+n5ZYdRmTk3t6AKsAH1sUDHb+Qp28Gtpuiy4+lFq+ojH8JWCj7JMicRQqfEMNx2mkw0aFbkTW1dGCr1sUA6UT9zka/cJ9wv6LEeqxz2kj+t3j+r0lUTky4oXlhuw3nQBsLsvkPSrjKIp6sZP1cwmo+qZsOGsUMbZx9tP4TqQzWEFoIyGxLaUVRQHHOAIRs3NtOGqCQdp7+2/AHgSE/Hlp8HLQRXMeBFsSU/S7lVuvvdsb+v5h1mxxAKAOlTVsY/lBSHv9ogtZb5iO3MeLkFh6pGkSBnna2N2ilcP8g/q7qFHeKg3FuNpp1VMcYC38s6m+6oFij84p6UJFSVEC20o0m7f2/u/Rn1x8gvX/PpVW3hUntuVZSYPJDmWzG2n+evQBPmtsWbVSPg6InspIGllg+guICl6WpMfq+eshoFDb6N8WN73l9W1DMWmOXJiUzdcF8KxFzm4WtUN9Rh4gU6qmPOLZqL9yc6XcX4+A3WMya3tdgFivSlWiIHyRN0 AASK2lT4 mxIuQwLCJjY0F1N7b1rz+BUWjjzL9AfPPvnEvxTywK8Cm9auuWmWVFUsXFl2wV3lhhhvK474hRYJK8jQgOIHhzpgRJoLHgPWEvEdpUh07pwIceckO6ROFm5iGITrWfDs7Sz/c14seSHC0Fsat5QQycGvQXY85Q9bkQ7IRKQAgaShysfknvZmrytdN7aC8uiwlbL583352+I5gYs9L/JMrXV5uzNxAL8jWwltrVdnQ7hZPekUYedJ1tGb1LMxVfijsxpw/9XsZlgb4jmWcgyI4P6uJ1a/f/D3QaGfjyj1tQG1RIbiRlEmtoWDy/dTlVWyqUmiSg0yJ52NTjWAER2y8Q0wjPIchWZ5siwdBtVGvU9iFPCY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: untagged_addr() is a helper used by the core-mm to strip tag bits and get the address to the canonical shape based on rules of the current thread. It only handles userspace addresses. The untagging mask is stored in per-CPU variable and set on context switching to the task. The tags must not be included into check whether it's okay to access the userspace address. Strip tags in access_ok(). Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/x86/include/asm/mmu.h | 3 +++ arch/x86/include/asm/mmu_context.h | 11 +++++++++ arch/x86/include/asm/tlbflush.h | 10 ++++++++ arch/x86/include/asm/uaccess.h | 39 ++++++++++++++++++++++++++++-- arch/x86/kernel/process.c | 3 +++ arch/x86/mm/init.c | 5 ++++ 6 files changed, 69 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 22fc9fbf1d0a..9cac8c45a647 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -45,6 +45,9 @@ typedef struct { #ifdef CONFIG_ADDRESS_MASKING /* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ unsigned long lam_cr3_mask; + + /* Significant bits of the virtual address. Excludes tag bits. */ + u64 untag_mask; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 0295c3863db7..eb1387ac40fa 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -101,6 +101,12 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; + mm->context.untag_mask = oldmm->context.untag_mask; +} + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ + mm->context.untag_mask = -1UL; } #else @@ -113,6 +119,10 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { } + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ +} #endif #define enter_lazy_tlb enter_lazy_tlb @@ -139,6 +149,7 @@ static inline int init_new_context(struct task_struct *tsk, mm->context.execute_only_pkey = -1; } #endif + mm_reset_untag_mask(mm); init_new_context_ldt(mm); return 0; } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index e8b47f57bd4a..75bfaa421030 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -54,6 +54,15 @@ static inline void cr4_clear_bits(unsigned long mask) local_irq_restore(flags); } +#ifdef CONFIG_ADDRESS_MASKING +DECLARE_PER_CPU(u64, tlbstate_untag_mask); + +static inline u64 current_untag_mask(void) +{ + return this_cpu_read(tlbstate_untag_mask); +} +#endif + #ifndef MODULE /* * 6 because 6 should be plenty and struct tlb_state will fit in two cache @@ -380,6 +389,7 @@ static inline void set_tlbstate_lam_mode(struct mm_struct *mm) { this_cpu_write(cpu_tlbstate.lam, mm->context.lam_cr3_mask >> X86_CR3_LAM_U57_BIT); + this_cpu_write(tlbstate_untag_mask, mm->context.untag_mask); } #else diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 1cc756eafa44..c79ebdbd6356 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -7,11 +7,13 @@ #include #include #include +#include #include #include #include #include #include +#include #ifdef CONFIG_DEBUG_ATOMIC_SLEEP static inline bool pagefault_disabled(void); @@ -21,6 +23,39 @@ static inline bool pagefault_disabled(void); # define WARN_ON_IN_IRQ() #endif +#ifdef CONFIG_ADDRESS_MASKING +/* + * Mask out tag bits from the address. + * + * Magic with the 'sign' allows to untag userspace pointer without any branches + * while leaving kernel addresses intact. + */ +static inline unsigned long __untagged_addr(unsigned long addr, + unsigned long mask) +{ + long sign = addr >> 63; + + addr &= mask | sign; + return addr; +} + +#define untagged_addr(addr) ({ \ + u64 __addr = (__force u64)(addr); \ + __addr = __untagged_addr(__addr, current_untag_mask()); \ + (__force __typeof__(addr))__addr; \ +}) + +#define untagged_addr_remote(mm, addr) ({ \ + u64 __addr = (__force u64)(addr); \ + mmap_assert_locked(mm); \ + __addr = __untagged_addr(__addr, (mm)->context.untag_mask); \ + (__force __typeof__(addr))__addr; \ +}) + +#else +#define untagged_addr(addr) (addr) +#endif + /** * access_ok - Checks if a user space pointer is valid * @addr: User space pointer to start of block to check @@ -38,10 +73,10 @@ static inline bool pagefault_disabled(void); * Return: true (nonzero) if the memory block may be valid, false (zero) * if it is definitely invalid. */ -#define access_ok(addr, size) \ +#define access_ok(addr, size) \ ({ \ WARN_ON_IN_IRQ(); \ - likely(__access_ok(addr, size)); \ + likely(__access_ok(untagged_addr(addr), size)); \ }) #include diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index b650cde3f64d..bbc8c4c6e360 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -48,6 +48,7 @@ #include #include #include +#include #include "process.h" @@ -368,6 +369,8 @@ void arch_setup_new_exec(void) task_clear_spec_ssb_noexec(current); speculation_ctrl_update(read_thread_flags()); } + + mm_reset_untag_mask(current->mm); } #ifdef CONFIG_X86_IOPL_IOPERM diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index cb258f58fdc8..659b6c0f7910 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -1048,6 +1048,11 @@ __visible DEFINE_PER_CPU_ALIGNED(struct tlb_state, cpu_tlbstate) = { .cr4 = ~0UL, /* fail hard if we screw up cr4 shadow initialization */ }; +#ifdef CONFIG_ADDRESS_MASKING +DEFINE_PER_CPU(u64, tlbstate_untag_mask); +EXPORT_PER_CPU_SYMBOL(tlbstate_untag_mask); +#endif + void update_cache_mode_entry(unsigned entry, enum page_cache_mode cache) { /* entry 0 MUST be WB (hardwired to speed up translations) */ From patchwork Sun Mar 12 11:26:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 675B8C6FA99 for ; Sun, 12 Mar 2023 11:26:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 32CD58E0001; Sun, 12 Mar 2023 07:26:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B4308E000A; Sun, 12 Mar 2023 07:26:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E48F78E0001; Sun, 12 Mar 2023 07:26:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CB2AB8E0008 for ; Sun, 12 Mar 2023 07:26:39 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A840AAAD18 for ; Sun, 12 Mar 2023 11:26:39 +0000 (UTC) X-FDA: 80560018518.23.F7A80EC Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf25.hostedemail.com (Postfix) with ESMTP id 78B72A0008 for ; Sun, 12 Mar 2023 11:26:37 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="ae//+b+q"; spf=none (imf25.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620397; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pj/WC7utj+xn+ajKUwmk2ytcxWnqOOCcuDBlAr4sM28=; b=8GIY6h7E8chE8n+h6ZRkDB+UcouFxqf2PlNGDHmPPyKMlBg64L892pFkJGDNLyf99AAjeX nDz5rkTbALw+u1sAXQne2YvIMjRy2FBwSC2JkgMdt50ghBKNFlakJftRHZFTmuUxXLhs+H kdgBO8ju5+dxYxNXFwH5bvmABoKvM7k= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="ae//+b+q"; spf=none (imf25.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620397; a=rsa-sha256; cv=none; b=XC9OfHokVQj/X14eayq+TYiKYVZjXHGRo1S2gk9QeRwJg9XctU/LxW2XJkwuhIWAJ+Yx2J JU5tjL22gEzaK6eTRpTBLwHG9wi8cI11E/A52jxg32IW2Y0B0E44Zqgcbe2I3OhbPktFne sEKQdJzXwIkR5OG6aEolxdIQRkJSBg4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620397; x=1710156397; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=HoVvsInTn5TZ+MT8glj3IrotSxRmkydAQ9cgijR6SG0=; b=ae//+b+q2Un1jllrUsL5mv3njV0zy/zZ8Jc/TPJOg9t+5bs0G62o9iC+ II7V1pCu4cDrsBSJewmF1sM0RtfrvC9z/IhkwvySUuHpd/2BQp/HO8IHN rmUKUQa9EZh0Dayj4sDeFq0QbayzQ+eXNnHGwiH5DW9VkkYKcKHBgYOLl SV+R37K+hNz10+IMLRkov5/kdc9zr2ZrJLK5op7gxvukG3hW0M2RRAUSw rxMhNpJ2X9bXtl3IUl59BO3XKM0y1DiHVWrEuzjDKFSZ+saEnjZNpwTDv skp/ECCsM2CGWFbvayzcSvdh6ADV44jynUSsqn97vii2MTKXBRyqvsZp/ w==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="339349959" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="339349959" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="852438031" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="852438031" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:30 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 9992410D7B5; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv16 07/17] x86/mm: Reduce untagged_addr() overhead for systems without LAM Date: Sun, 12 Mar 2023 14:26:02 +0300 Message-Id: <20230312112612.31869-8-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 916aebbg5ybt9pnaz7c74b53trimzaww X-Rspamd-Queue-Id: 78B72A0008 X-HE-Tag: 1678620397-985196 X-HE-Meta: U2FsdGVkX1+cgBttI8jvz/o5359dbzgGmJHilb8fgiZmKAqXfoXUHhs5c7cU1ZKPRp7F/ZeMYt5UnMQsfk38fOHDojayKTOU0ZY75MnzX3jz3otj0ExSF8p4a/2cGEoQd48M+TH6ym9zhe0fGsMPxxuABMk7nDtJkUMFE1VDYl0TSKSHq6HZZgs6ZnY2gG/OreN6gBskVBnSDr6RL14wFUNZKhIkQOE7DXYerGXThhzSeLSMFfx0MYC4v+KmWuMpHfMKf6NfMSeS+Dv/Xp7lN2kWscb7X3UonMy5pyYAKZK5tlDKg6XPglK33kA/NUJdYke6cd34gwaMBfrn6tkzTUN7B3IxI2d341l6G9Ps7uwLD0xfuSZbnRDkym3RPrMFU+zgg6j9zY1gJijjjMliEGfJ8evrpw3yA57xz6vXli02N+zntn4RccWSECqDz0hn4ccsixQGu84sBsKEN2DjjRrmkv5Olhx2ANIcpCcatlddLqoR2wF0O0S/K8/3gt7HB0UcW4xbOpIGzDn51xbOZL1RuDSJBZ9YPTjAppINcLTr7omRK/zacDlb8W9AvL73lQHkds5Mf9HmOG+XoQWvYu8mlxrEsmvl0VMohghr3w7m8tWn3jwvn1WkICbgA77c329YXPigKAztx1/suiH4sJZIeSBSERD/dVCMf6y1rp2D/6gJLf4E6SHBaeOkulkJn/BDSVsaEO4AXXfjVYFSCfzMBJ9MGrZ0xuhfoOEfOtY/ynrHnCIuTbxLu4ldzcaHIVbMfbMCurEoY2Z4FsqPiZt+ON6GHGzhHWA7TJKqYHlkByStVAbjU7hlR6vKNzO7O3OXhSdpsepHmhjlzWoCbL+w2A4gALP1J2W+enDgC2IQoLGpqMVSw2QWRo3bXV5ePnz4TLIDYDFL7ZPeA1TXTrwQtHRi02G1hwpnPiaURzm2Uf+zMvzHZAuNtRzCn6s1JIu/2i7RsFCM0u71ZOl 5zCu6hmg kq3DDizuMPdaNn/ZZ96/+MIS1Qo2WpG5GvuOdClF1k/4QjY9nhA4K0LrJ8qEA8VL+5zGtcZRuMzCTDdoaZknS1w8FohaOX6J3g2UaGRsaDIwc7ePlSooJ2sfHRTEvpKyMCdyvG8uMM/3naARVmRPXaCfcdwbElfG6njPo/mDrhCfQMzPLEUONIq7t7NvieqjdHk19ElQmDRBdYoGR7oc84RLJn7w7o+7idYXdWVPa7ULc13PKH9W+lgIslhasRMkQNXC1hjvTHZKoYQzyaEoD5uteog== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use alternatives to reduce untagged_addr() overhead. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/disabled-features.h | 8 ++++- arch/x86/include/asm/uaccess.h | 41 +++++++++++++++++------- 2 files changed, 37 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index 5dfa4fb76f4b..fafe9be7a6f4 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -75,6 +75,12 @@ # define DISABLE_CALL_DEPTH_TRACKING (1 << (X86_FEATURE_CALL_DEPTH & 31)) #endif +#ifdef CONFIG_ADDRESS_MASKING +# define DISABLE_LAM 0 +#else +# define DISABLE_LAM (1 << (X86_FEATURE_LAM & 31)) +#endif + #ifdef CONFIG_INTEL_IOMMU_SVM # define DISABLE_ENQCMD 0 #else @@ -115,7 +121,7 @@ #define DISABLED_MASK10 0 #define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ DISABLE_CALL_DEPTH_TRACKING) -#define DISABLED_MASK12 0 +#define DISABLED_MASK12 (DISABLE_LAM) #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index c79ebdbd6356..457e814712af 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -30,26 +31,44 @@ static inline bool pagefault_disabled(void); * Magic with the 'sign' allows to untag userspace pointer without any branches * while leaving kernel addresses intact. */ -static inline unsigned long __untagged_addr(unsigned long addr, - unsigned long mask) +static inline unsigned long __untagged_addr(unsigned long addr) { - long sign = addr >> 63; + long sign; + + /* + * Refer tlbstate_untag_mask directly to avoid RIP-relative relocation + * in alternative instructions. The relocation gets wrong when gets + * copied to the target place. + */ + asm (ALTERNATIVE("", + "sar $63, %[sign]\n\t" /* user_ptr ? 0 : -1UL */ + "or %%gs:tlbstate_untag_mask, %[sign]\n\t" + "and %[sign], %[addr]\n\t", X86_FEATURE_LAM) + : [addr] "+r" (addr), [sign] "=r" (sign) + : "m" (tlbstate_untag_mask), "[sign]" (addr)); - addr &= mask | sign; return addr; } #define untagged_addr(addr) ({ \ - u64 __addr = (__force u64)(addr); \ - __addr = __untagged_addr(__addr, current_untag_mask()); \ - (__force __typeof__(addr))__addr; \ + unsigned long __addr = (__force unsigned long)(addr); \ + (__force __typeof__(addr))__untagged_addr(__addr); \ }) +static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, + unsigned long addr) +{ + long sign = addr >> 63; + + mmap_assert_locked(mm); + addr &= (mm)->context.untag_mask | sign; + + return addr; +} + #define untagged_addr_remote(mm, addr) ({ \ - u64 __addr = (__force u64)(addr); \ - mmap_assert_locked(mm); \ - __addr = __untagged_addr(__addr, (mm)->context.untag_mask); \ - (__force __typeof__(addr))__addr; \ + unsigned long __addr = (__force unsigned long)(addr); \ + (__force __typeof__(addr))__untagged_addr_remote(mm, __addr); \ }) #else From patchwork Sun Mar 12 11:26:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171300 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C861EC6FD1F for ; Sun, 12 Mar 2023 11:26:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 066CC8E0009; Sun, 12 Mar 2023 07:26:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 014B68E0008; Sun, 12 Mar 2023 07:26:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DAB158E0009; Sun, 12 Mar 2023 07:26:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CA1A58E0001 for ; Sun, 12 Mar 2023 07:26:39 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 890B5AABD3 for ; Sun, 12 Mar 2023 11:26:39 +0000 (UTC) X-FDA: 80560018518.20.284F722 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf10.hostedemail.com (Postfix) with ESMTP id 7E093C0010 for ; Sun, 12 Mar 2023 11:26:37 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Va3ViAMf; spf=none (imf10.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620397; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SEQWBU+QfGDCbgeszyGr05YNz6negMxo/uUD30yHObw=; b=U6ZbMx0jALHfZEP4BulD/IeXGSrc4E7XjoFOXRBuYSxPPMEwXlhrtP9IGyBxogq/6r8vCe CqN9NQOfr6PHe0+UAtXU97shfOjLoE2pwp+R+S0y8qGCTYN8zJ00v+E/iILnPSP/AWAp9D 3UxhAdwBqJojRi5B1M9QklcitidslMk= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Va3ViAMf; spf=none (imf10.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620397; a=rsa-sha256; cv=none; b=1efKqijY87vPIBAWP4qA9lWGUymOvesEXzUFFwoSLhUsGmaYUaaMGGpzqM8ms1TtYCljDv aL77qhygYnkjUNwiqexKNOsrfGbFlU+kxrZ7Y1z/rc679ml+K3ZmVDSwwOsskPLXErelVC 0i8ko5xt1XsiiC/X10OtSSAW+gIlHW8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620397; x=1710156397; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=HmyZhvrnaOzMtHk3YHlMvWmwvxMx1gT3mopz2/NGNMw=; b=Va3ViAMfYzai7fBFMcRJ6tcOqOUOZyLzJ9r/XvNRrI9zxrAjehzQveWL lNKUyhT/0gc8Pw9tXri95P7iAnqVKqP4EJxpaD6rJ7+E4a1/bvT+bc85Q 877/qKd2MuO0uPrdGVMGPQPun6KgDdBb/UsfuzUzEgZOSRecq3BdmCpPa mJR0qHARsvryphPOcxKgDxVMDPwuK12aO0pZemniS1WhR/KcGAQmoIiZ+ O5UV/fcC38soTWXbXpgMP/1kOAFNh1KRowaHPZcz4w7rLbYlDA3YnkGQT 5q6limzoZEUSi2xWD71wr1ToOgt1f8gEoC/9c1zn8+9/Ht96jKJBCa1LN g==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="339349967" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="339349967" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="852438033" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="852438033" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:30 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id A42E010D7B6; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv16 08/17] x86/mm: Provide arch_prctl() interface for LAM Date: Sun, 12 Mar 2023 14:26:03 +0300 Message-Id: <20230312112612.31869-9-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: c3gq4t4k654ywqhbrqbejxmmzc9g9dqz X-Rspamd-Queue-Id: 7E093C0010 X-HE-Tag: 1678620397-640870 X-HE-Meta: U2FsdGVkX19v9hbT8DXMG+y/i3AQ9uk/cmwNxZGeLBHd8c+KizlCx32W/tL6qWo3mFScqS7/1bWR81JTwHcckJ1no7Zqldbqdyo2eisyyv5/qroqUilTnSgkNL24+W1fBzfK8P7xP27f7+BoY8fmF6slgoY+/KBmUC+kb9vk3+jdG5OJDMmwAXKFslY5lSb0i65+XRW4O2lcPhb5k5rMEvUIE4PHmgJ3fpGnaa4XpYcuby4iAsU9Z2PmB6lhjgU3uHKq46M71xIUMv0LTjHDKVFFABho+9xlroN0OkM2fxTQbs0U30iE9a72y7GcgktmJ2QT6BtcJc6SMDno6eeKn6q/aZ+kOLP618scidYYGjiTRa2Z/G4vq9leP67xsKHm27OGctULHr0hIkw5tF8Pk6gTxNbG8fN1z5JkA/hEcIrlL1/5rxDXrUJJDFcCeyVr+zdSN391kHc5jlgY/ls24vw5hC42oTGs04v8IiKiH+xaOEisXGME0QF/NR3BGzn8rMNAdicLiB/whVJ8iFzsigM5Io4RW+rIwQYSrO3+QMFFqMFv4tG1WK8l441JxGm0sQAi4U9LEjukbJKSovEzRhX+6n68RpJalM1ktpizntXysCNwl76rkXoIZH1BccLE//B9Jou6hq1IvmZ5OchVYWkvY14+2BTKdhYMt0I7zk5CtimnQO2YKB32GWBf1OIZ/skBRbmifP+VkoTPMbggpx5cS4tlcdasycx5NKf/Q1rP2wPRnlomCFZL7//xR67ASi+XfwqZaQMYm7RqOdAT2ffZiIxfJUDjON05o+Mqt8QztPahw9DO3WdX2fYoRxpy6PaJOZl2Y6MGS52szbZvueQ5px48oxnxk9U7PY6ZI5YQSn4BKpNwoprkrHdS2rMm7/b7ewabyMhpcqOohV8QaHVpN9/vY7S8CtPYD/ltv63YOrZFNXgbRLQpznsfhKex8s2DpzV794B65V5ZqqT +MWToqHf DaDbgjhbYsnlioyffNsG/yq9lPk17tcor666ap1P8xdHwEMuenqSpp8tkkVtqZZb7c98GtfnOIfS3ekxBSHLNb/6YqiWm1kxHiXIzGUNeaZO5upPNh7oyWeOLJd3duDbOCq36Pu83fFma5rVsWqKqYlxK4t5po74Fc+bxvaIslgNyfMRieN18M+1qecnLzO9eWkSG+F1rcRmkDY9T3n+Ty1bzuH09jpFJNVYzYsJ63wjK4Kb8Dpjd2OB9T8Krub4wA/IsqKyPS2mhmxTQxrb+744wYaylwNZXdDe3uJOQCXA/QEzyLddbZoR1rOonH/3MbThjDEWXN6FcR99RudbkpL7lIm+JHOzXt3E5MB0ZXAspF5w= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a few of arch_prctl() handles: - ARCH_ENABLE_TAGGED_ADDR enabled LAM. The argument is required number of tag bits. It is rounded up to the nearest LAM mode that can provide it. For now only LAM_U57 is supported, with 6 tag bits. - ARCH_GET_UNTAG_MASK returns untag mask. It can indicates where tag bits located in the address. - ARCH_GET_MAX_TAG_BITS returns the maximum tag bits user can request. Zero if LAM is not supported. Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/x86/include/asm/mmu.h | 2 ++ arch/x86/include/uapi/asm/prctl.h | 4 +++ arch/x86/kernel/process.c | 3 ++ arch/x86/kernel/process_64.c | 55 ++++++++++++++++++++++++++++++- 4 files changed, 63 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 9cac8c45a647..e80762e998ce 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -12,6 +12,8 @@ #define MM_CONTEXT_UPROBE_IA32 0 /* vsyscall page is accessible on this MM */ #define MM_CONTEXT_HAS_VSYSCALL 1 +/* Do not allow changing LAM mode */ +#define MM_CONTEXT_LOCK_LAM 2 /* * x86 has arch-specific MMU state beyond what lives in mm_struct. diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 500b96e71f18..a31e27b95b19 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -20,4 +20,8 @@ #define ARCH_MAP_VDSO_32 0x2002 #define ARCH_MAP_VDSO_64 0x2003 +#define ARCH_GET_UNTAG_MASK 0x4001 +#define ARCH_ENABLE_TAGGED_ADDR 0x4002 +#define ARCH_GET_MAX_TAG_BITS 0x4003 + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index bbc8c4c6e360..50d950771371 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -163,6 +163,9 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) savesegment(es, p->thread.es); savesegment(ds, p->thread.ds); + + if (p->mm && (clone_flags & (CLONE_VM | CLONE_VFORK)) == CLONE_VM) + set_bit(MM_CONTEXT_LOCK_LAM, &p->mm->context.flags); #else p->thread.sp0 = (unsigned long) (childregs + 1); savesegment(gs, p->thread.gs); diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 8b06034e8c70..88aae519c8f8 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -743,6 +743,48 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) } #endif +#ifdef CONFIG_ADDRESS_MASKING + +#define LAM_U57_BITS 6 + +static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) +{ + if (!cpu_feature_enabled(X86_FEATURE_LAM)) + return -ENODEV; + + /* PTRACE_ARCH_PRCTL */ + if (current->mm != mm) + return -EINVAL; + + if (mmap_write_lock_killable(mm)) + return -EINTR; + + if (test_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags)) { + mmap_write_unlock(mm); + return -EBUSY; + } + + if (!nr_bits) { + mmap_write_unlock(mm); + return -EINVAL; + } else if (nr_bits <= LAM_U57_BITS) { + mm->context.lam_cr3_mask = X86_CR3_LAM_U57; + mm->context.untag_mask = ~GENMASK(62, 57); + } else { + mmap_write_unlock(mm); + return -EINVAL; + } + + write_cr3(__read_cr3() | mm->context.lam_cr3_mask); + set_tlbstate_lam_mode(mm); + set_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags); + + mmap_write_unlock(mm); + + return 0; +} +#endif + long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) { int ret = 0; @@ -830,7 +872,18 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) case ARCH_MAP_VDSO_64: return prctl_map_vdso(&vdso_image_64, arg2); #endif - +#ifdef CONFIG_ADDRESS_MASKING + case ARCH_GET_UNTAG_MASK: + return put_user(task->mm->context.untag_mask, + (unsigned long __user *)arg2); + case ARCH_ENABLE_TAGGED_ADDR: + return prctl_enable_tagged_addr(task->mm, arg2); + case ARCH_GET_MAX_TAG_BITS: + if (!cpu_feature_enabled(X86_FEATURE_LAM)) + return put_user(0, (unsigned long __user *)arg2); + else + return put_user(LAM_U57_BITS, (unsigned long __user *)arg2); +#endif default: ret = -EINVAL; break; From patchwork Sun Mar 12 11:26:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A53BC6FD1F for ; Sun, 12 Mar 2023 11:27:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C4E4F8E0008; Sun, 12 Mar 2023 07:27:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BD77D8E0010; Sun, 12 Mar 2023 07:27:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A1FF8E0008; Sun, 12 Mar 2023 07:27:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7248D8E000E for ; Sun, 12 Mar 2023 07:27:04 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4D60C1C5FD1 for ; Sun, 12 Mar 2023 11:27:04 +0000 (UTC) X-FDA: 80560019568.05.3FFCC72 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf05.hostedemail.com (Postfix) with ESMTP id 2D8CB10000D for ; Sun, 12 Mar 2023 11:27:01 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=GsE6ukiM; spf=none (imf05.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620422; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+t7c0GIEcRYZsAa/7kH4wF7e/+Xbyk3x8s2toycvq8M=; b=zMTJI2IeIsn2nIbn/DqyUI8pFCLCiGFnohqLNXFtKl/BBOhKVV3hkqJtlmmxcVMsib23Lw uvzXW7Xa6vJ6UqeGPI6mcZE/f/BtCIuktn4fCffweCGgWW++Pun+C6FG5pGAaMAzjbMJFy 0FS0iP58T1oCWAwCRLy2IfEKOSDuBWc= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=GsE6ukiM; spf=none (imf05.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620422; a=rsa-sha256; cv=none; b=3JkaZC2Q7MK00mun/0i+o5ZYYA/JWlfS7WWioISGow0x2uxr6dzjBIY8x6hMarxijlYfAq fWHTR8NbEWA7bBArsUt8zbfG8+6BAIH1WulRknwnwA0tuQAk89VhmoV21TSSoiT3edhTDb Wc4KGjFMXss1cOWsIP4Yn1gxT8xP3N4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620422; x=1710156422; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dORBfkbxhzi1hEEMwZ37gZ5mNHOdLznvEXqItjRFy7I=; b=GsE6ukiM70jFK2f+Q8SXXhHan4cWoOT+Ye3o5qGzrDCbgpLdyawv7T4Y 0Wbn4DWfB97XZ9xajfZfyRsonqyz7d7qZlJYUQDLWLn09/58J3qS9hBdj VfaWbu4AMXE4WU8Bg/Wh5YgGkbRf98z7PyOfFm8O58o9IdWi9i/KpvoSC E88d/ROGZQ3tCgGTGZsGBjhBZtf2Whc/+gqikutvs3mbPUB92VEavRSKU zIqwSOwvqhIKb+4NfVoPZ9CnRx4gf2+0ADBoIXID5bSfEir3/syB9GDSW M4fG5vbXELAt4j5kVlEPiBZCK+wrnLShpRPp0rDy1mYxrSNi8FZC55L45 g==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="399577588" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="399577588" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:36 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="671580737" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="671580737" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:30 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id AF00310D7B7; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Catalin Marinas Subject: [PATCHv16 09/17] mm: Expose untagging mask in /proc/$PID/status Date: Sun, 12 Mar 2023 14:26:04 +0300 Message-Id: <20230312112612.31869-10-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: u181xteheq6fkgf3rhsrtdzrfeyyz6tt X-Rspam-User: X-Rspamd-Queue-Id: 2D8CB10000D X-Rspamd-Server: rspam06 X-HE-Tag: 1678620421-268682 X-HE-Meta: U2FsdGVkX18sdYfvwNEkjcaIj1SjWDLQp77gTxkxXq8tpiMNDtd3e9Yc6BfnO76f7zyw9OmB62LaWq6Xgx8YzBePFkiORNG7EMntKbu1HdvWf2n3/De3cY9SK6WVNGO+k/rIdt4hGaNBdKyt7f6QxBoXp2mzSlusbDpjc+64vOZ8Nptv4jD8xrh8N9bMNLWHNkhpNWhjZfcRmFlXX8I5USOVmgGd6o4iuc1/09rHMnzzFTU9zh9SpmcUqIJnvlwyY0EEEm3Ywn3vmxc+Rn5xVtYiFeB7ZeJq62EDWFFXYwFoONRalfzYPxIpGPGlfyRy2C2EfqKDhTiKpgi+lhIHk2QJU5wLEPM7dE3R/lHwRmvVBOwyAeFdNFqV9uWxEVyHXJuzc2Rjr1zebOL1J8Gt1U2HZH1rfDMtWtchTSR3owwjFPS8rxO0wW1vMbhD+hjGs7oE4PavgPMXgNT0nuh3yjInK5+CWiIRqSx/6P7mnWMt9xq+W/XpxupRt5XYIqjDI4FZBnGfMEb/MZcyClydrFe2XYnpw5WL32i5qEMcMFrxc734JOXjP/Nz3kW0ETYvyHYggjjNWx86qbACBuyYyjGYLSxWpOPiYwgjvtQyVOugBuA9vZd63OqgycmDwHzqTZAwT8lyl0gzMs/QIlbw3ni8qz2U/Q7gw3kWLDK/vI+IiKvnoh6Jr4pW5cjlHPL23ITsg55SMb3sesUP10iz2EhQYvPgmXusaBgXarByjTaaQl8WOR0cLM8VxxtPCfoxTFmJfbQ4s1rt8yJzQJdBjAf/DAWIJaVFsQQUm2tq7A/Hf0+qgypma1AtkcxByYeRoV3bOnYtT7t5rXG3nXJ3EelPfGULFOcTi7XnHRX0n0BkCbsAK4dDM3/lK4I9UDxuiPyNRAQMz5SpkcSuE7gRIdchAu/+2ZTr0+vawR3+lM6iySbLrJ59Nd5z6mkMD7/pmpsAtIQxnHdEyL6SLE+ 2ogU9LaL uekWP3FPCJAZK5WuIn1HF7VFr8XJ7QQOVadMuqmwp1YoTI9eUEptxHyOih6PIhqnuSHyRI4ZMysVEznV+kNlZjY2mrrnHZATkxpnGV6Ia0o3SBu8Kc0J7LQaarX6WW0jQtW41tWcnwQalymO8536CbJrCWhCzIQNxFc+wiU8NMjo9HbuaFOgNnpySO/mSp13lH/9IQAFuZLTOkDyITp4YlP1Y9hoDjf0yuogY6iRXv2lDM1DuUcob5kHZc1OgFLm/3IDtqqu/fqOZ/r+nuxt0tpqXIqfzXDXG7BIIQQrzx1o6kzbmx9tOkCmif4VeA1PiYj5VKRpOUFaCabr2gVGypydafZMt/6plg14gFDecK21SR4vOfNqTne7NjPU6pHJShL7FCbSR+NpSycypHyfIJj8lJZGGsD7+PCYb X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a line in /proc/$PID/status to report untag_mask. It can be used to find out LAM status of the process from the outside. It is useful for debuggers. Signed-off-by: Kirill A. Shutemov Acked-by: Catalin Marinas Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/arm64/include/asm/mmu_context.h | 6 ++++++ arch/sparc/include/asm/mmu_context_64.h | 6 ++++++ arch/x86/include/asm/mmu_context.h | 6 ++++++ fs/proc/array.c | 7 +++++++ include/linux/mmu_context.h | 7 +++++++ 5 files changed, 32 insertions(+) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 72dbd6400549..56911691bef0 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -288,6 +288,12 @@ void post_ttbr_update_workaround(void); unsigned long arm64_mm_context_get(struct mm_struct *mm); void arm64_mm_context_put(struct mm_struct *mm); +#define mm_untag_mask mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return -1UL >> 8; +} + #include #endif /* !__ASSEMBLY__ */ diff --git a/arch/sparc/include/asm/mmu_context_64.h b/arch/sparc/include/asm/mmu_context_64.h index 7a8380c63aab..799e797c5cdd 100644 --- a/arch/sparc/include/asm/mmu_context_64.h +++ b/arch/sparc/include/asm/mmu_context_64.h @@ -185,6 +185,12 @@ static inline void finish_arch_post_lock_switch(void) } } +#define mm_untag_mask mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return -1UL >> adi_nbits(); +} + #include #endif /* !(__ASSEMBLY__) */ diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index eb1387ac40fa..06eaaf75d572 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -104,6 +104,12 @@ static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) mm->context.untag_mask = oldmm->context.untag_mask; } +#define mm_untag_mask mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return mm->context.untag_mask; +} + static inline void mm_reset_untag_mask(struct mm_struct *mm) { mm->context.untag_mask = -1UL; diff --git a/fs/proc/array.c b/fs/proc/array.c index 9b0315d34c58..6daea628bc76 100644 --- a/fs/proc/array.c +++ b/fs/proc/array.c @@ -91,6 +91,7 @@ #include #include #include +#include #include #include "internal.h" @@ -423,6 +424,11 @@ static inline void task_thp_status(struct seq_file *m, struct mm_struct *mm) seq_printf(m, "THP_enabled:\t%d\n", thp_enabled); } +static inline void task_untag_mask(struct seq_file *m, struct mm_struct *mm) +{ + seq_printf(m, "untag_mask:\t%#lx\n", mm_untag_mask(mm)); +} + int proc_pid_status(struct seq_file *m, struct pid_namespace *ns, struct pid *pid, struct task_struct *task) { @@ -438,6 +444,7 @@ int proc_pid_status(struct seq_file *m, struct pid_namespace *ns, task_mem(m, mm); task_core_dumping(m, task); task_thp_status(m, mm); + task_untag_mask(m, mm); mmput(mm); } task_sig(m, task); diff --git a/include/linux/mmu_context.h b/include/linux/mmu_context.h index b9b970f7ab45..14b9c1fa05c4 100644 --- a/include/linux/mmu_context.h +++ b/include/linux/mmu_context.h @@ -28,4 +28,11 @@ static inline void leave_mm(int cpu) { } # define task_cpu_possible(cpu, p) cpumask_test_cpu((cpu), task_cpu_possible_mask(p)) #endif +#ifndef mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return -1UL; +} +#endif + #endif From patchwork Sun Mar 12 11:26:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171302 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE5F1C74A5B for ; Sun, 12 Mar 2023 11:26:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E48CB8E000A; Sun, 12 Mar 2023 07:26:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E1F608E0008; Sun, 12 Mar 2023 07:26:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC13B8E000A; Sun, 12 Mar 2023 07:26:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B916A8E0008 for ; Sun, 12 Mar 2023 07:26:40 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8B00112086E for ; Sun, 12 Mar 2023 11:26:40 +0000 (UTC) X-FDA: 80560018560.17.4EEE80C Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf14.hostedemail.com (Postfix) with ESMTP id 6C603100004 for ; Sun, 12 Mar 2023 11:26:38 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=DL+rX3xj; spf=none (imf14.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620398; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E+vgO6hMBee1lhY3UlBSn+wO/aeilTZUQ4GQAb4cApQ=; b=uRhpnT0XJSJdlJfukSEDV/DP6SKfJQXNmVpVeaj0m9kPvuTUbnfcVluimIhBDPtJuEUi3w wDSjRG4b3Ak5xG5i2RXPWEW5HM825OLoATfQ/cih1/DLROvy8NiUWaSWM4nMo/7m4N4thZ oYi+02eN4Qto7bbTnhlUyN0g7hzegbw= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=DL+rX3xj; spf=none (imf14.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620398; a=rsa-sha256; cv=none; b=4kdN110uPp1UvoR53+EoxgmaukVX0msl3tiHsV1FHIlB+1ELFHLxisk0wZ+LxQ3sCo7+NV ZeeFEscQU1i6v5/gBeAfzwAOFq7m9kO4ZORUQm8mQDgiwWSXz4ASfL2bdmVYu8zXTseSPa EaSIHVTWiVESrAaVBJtd416F+poZL1k= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620398; x=1710156398; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+iKOETahP1WXZi4zcV24NixONYfrth/yIw9MG3r4KgU=; b=DL+rX3xjk6tEfmSRwiunsyVURZjWenxeK7sMdQo4J1oORiDy+4U2ddOh HOOtQ0gnEylf18KGXU6z6CeCRoKfVjOiVNKAE6wYBXqlLcy3P2akh1FUM pIUqc32EJnABdymrusQMjpaCGaBVXrKPUxE2E9WiaIGKHiZv0pWkb24o9 MRuD2ObWP9x6oaUKUM40k5l3rWDjEBVyDdo5MptUdQAZ3eA4XVqM6Sg/E lA8xdCJGrSprFeCUzMOhAKH2zOZbxnYSRsjye8MFRbOe90yS6+ODnsSe7 rczCTgMeZWm8cNxE00Fnw/HeD7NQisTbGlxMX9vQradhbTuf/f+4A7gLF w==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="339349975" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="339349975" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="852438035" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="852438035" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:30 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id B979A10D7B8; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv16 10/17] iommu/sva: Replace pasid_valid() helper with mm_valid_pasid() Date: Sun, 12 Mar 2023 14:26:05 +0300 Message-Id: <20230312112612.31869-11-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 6C603100004 X-Rspam-User: X-Stat-Signature: sy6eaq4tuusnng74qmbkuw5u5wjp7687 X-HE-Tag: 1678620398-353804 X-HE-Meta: U2FsdGVkX19joFn2vVk5LhAvxr/Y8iUlCJSmGgbxF+BwSnuvAR8WH0JW1S0I4RsU24j8QajrNXt6St4I4CNCebupnT7Fm8qVIkrgS/YbaqswtkkxyTS1b6Cmhi30FmApv4eyRAB+QdEnxXC30pSIfVdZ6NBctXFY1eu2mzY5GKRGBkcmxVjWpYPXfEEUXZC+IztbSdfSMJG/ZQ/fDqweVtxMW+ljqVeDXOBttA9FDMqjyIfQN0/k6Ol++SDefmQHeUO4OnUKkw9sU1neP3uWlcNayyTCLRBhgDhJbgRGa4eycBRjF4g8qKwmQBV6M5xQqtwvxiXBk19Ov02D2AQTKmA2Tz+RttguOiIzo28ozF3mH0L50ZAy9C9v8lVIAIxA9BATV4cqLhqhoAaOLS9WwuhnT7CNZCtupGLSV06OWE61+wd38tDAayCaNhA8CpeAwHnUPDd+0neHN/WCf5V4xGZqSc494srUxyOR1l7W8hEdOQRyFHREdBNjD3yNfWOAl11e1rQRcC9mgwXmyywgKXJN4bv3rGcQo7F9kWUSIrjnacyCJmiIuj1w+mmDtzxywTNDO0Ntmunylw5HJ+nx+z1q5jpIc5AXr8q9/dQs9d96voVrV5pVbAI8jy4xPedORWMtnO/RUE1SjuEVKyPQ7Y/wNGQTX9KYU9azYufXNdSdQRbQVmWaM/10yXaYvI00xf/PQSmTaLy0Unf/GjLW9P1nYHx25fTRefhACaw0Sgtq+9aNCoFljy8f3Wyzt2N0mKwrlLfJIm5sT8sa/VUwdi3Vtf75nbsYMzbEneVUyf7ZHcEMVOdbEgNz3+2IL2t/+6b4lTNEVMmTPa0dAHbGD8Z07psJ6QLiKid7DabzXxUPCCH4g4KicC4cvRGSVQo8/vapD8oCWHueK1DgMW/+eWCZf+8Uvtb537J9YNulYA1WWq6kCWKz5P3SP1RXy9oI0B4OaWxYIckPZyKbUGl 1Xa2dgNl YDO/VtuV8um8biTfHpLGwqjsPOalGD5DegHjmmeTU8RWt+OYSxQMhaqgeB9KNgKBT/yqCHHNMdtkJBD7B5cxRPEl5szRS6Z3ls/1qPb4vUnNaGNm+zEftp+WTWsSxjha4//dn5p4+6iijb2QSEJBS66eivwMVXYy8CexPjTx7U3/XoOHxnP/nbWa9XRnF6WvUmpcfDmS7JH1NaHsT+qIOJBhCUUBRgOUrU5sA0tDMKxfVtGHlCDRG4EXbY5RxJIw15/QsMqxeebgJw6fnmJC0ohLvZDTwPvnRRX4yE4kFn+mzT0Fc7MtE0/vLPbV/E+cjhNNd1NpR7aLZ6ZS+AJINqMfVBA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Kernel has few users of pasid_valid() and all but one checks if the process has PASID allocated. The helper takes ioasid_t as the input. Replace the helper with mm_valid_pasid() that takes mm_struct as the argument. The only call that checks PASID that is not tied to mm_struct is open-codded now. This is preparatory patch. It helps avoid ifdeffery: no need to dereference mm->pasid in generic code to check if the process has PASID. Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) --- arch/x86/kernel/traps.c | 6 +++--- drivers/iommu/iommu-sva.c | 4 ++-- include/linux/ioasid.h | 9 --------- include/linux/sched/mm.h | 8 +++++++- 4 files changed, 12 insertions(+), 15 deletions(-) diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index d317dc3d06a3..8b83d8fbce71 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -671,15 +671,15 @@ static bool try_fixup_enqcmd_gp(void) if (!cpu_feature_enabled(X86_FEATURE_ENQCMD)) return false; - pasid = current->mm->pasid; - /* * If the mm has not been allocated a * PASID, the #GP can not be fixed up. */ - if (!pasid_valid(pasid)) + if (!mm_valid_pasid(current->mm)) return false; + pasid = current->mm->pasid; + /* * Did this thread already have its PASID activated? * If so, the #GP must be from something else. diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index 24bf9b2b58aa..4ee2929f0d7a 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -34,14 +34,14 @@ int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max) mutex_lock(&iommu_sva_lock); /* Is a PASID already associated with this mm? */ - if (pasid_valid(mm->pasid)) { + if (mm_valid_pasid(mm)) { if (mm->pasid < min || mm->pasid >= max) ret = -EOVERFLOW; goto out; } pasid = ioasid_alloc(&iommu_sva_pasid, min, max, mm); - if (!pasid_valid(pasid)) + if (pasid == INVALID_IOASID) ret = -ENOMEM; else mm_pasid_set(mm, pasid); diff --git a/include/linux/ioasid.h b/include/linux/ioasid.h index af1c9d62e642..836ae09e92c2 100644 --- a/include/linux/ioasid.h +++ b/include/linux/ioasid.h @@ -40,10 +40,6 @@ void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid, int ioasid_register_allocator(struct ioasid_allocator_ops *allocator); void ioasid_unregister_allocator(struct ioasid_allocator_ops *allocator); int ioasid_set_data(ioasid_t ioasid, void *data); -static inline bool pasid_valid(ioasid_t ioasid) -{ - return ioasid != INVALID_IOASID; -} #else /* !CONFIG_IOASID */ static inline ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, @@ -74,10 +70,5 @@ static inline int ioasid_set_data(ioasid_t ioasid, void *data) return -ENOTSUPP; } -static inline bool pasid_valid(ioasid_t ioasid) -{ - return false; -} - #endif /* CONFIG_IOASID */ #endif /* __LINUX_IOASID_H */ diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 2a243616f222..b69fe7e8c0ac 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -457,6 +457,11 @@ static inline void mm_pasid_init(struct mm_struct *mm) mm->pasid = INVALID_IOASID; } +static inline bool mm_valid_pasid(struct mm_struct *mm) +{ + return mm->pasid != INVALID_IOASID; +} + /* Associate a PASID with an mm_struct: */ static inline void mm_pasid_set(struct mm_struct *mm, u32 pasid) { @@ -465,13 +470,14 @@ static inline void mm_pasid_set(struct mm_struct *mm, u32 pasid) static inline void mm_pasid_drop(struct mm_struct *mm) { - if (pasid_valid(mm->pasid)) { + if (mm_valid_pasid(mm)) { ioasid_free(mm->pasid); mm->pasid = INVALID_IOASID; } } #else static inline void mm_pasid_init(struct mm_struct *mm) {} +static inline bool mm_valid_pasid(struct mm_struct *mm) { return false; } static inline void mm_pasid_set(struct mm_struct *mm, u32 pasid) {} static inline void mm_pasid_drop(struct mm_struct *mm) {} #endif From patchwork Sun Mar 12 11:26:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5122BC6FD1F for ; Sun, 12 Mar 2023 11:27:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E20438E000D; Sun, 12 Mar 2023 07:27:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DD0218E0008; Sun, 12 Mar 2023 07:27:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C71928E000D; Sun, 12 Mar 2023 07:27:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B8A198E0008 for ; Sun, 12 Mar 2023 07:27:01 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 8A44B140905 for ; Sun, 12 Mar 2023 11:27:01 +0000 (UTC) X-FDA: 80560019442.26.9CCAA03 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf17.hostedemail.com (Postfix) with ESMTP id 2A7AD40006 for ; Sun, 12 Mar 2023 11:26:58 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=DX0oH4A4; spf=none (imf17.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620419; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eNMbej0AjDCem+sQF9VbalutOkSMPsnwphNJOw0msfI=; b=5+XfuqyLomHUN0TW/Bb9U3AUPgPxz/vjqMKSV0w5zyNTE1OvBBDlvRu3LwA/t1p5Ie8YhG W4R1iC9HgqLsLtRB0jjRdtYG0ilEIZsnT9XaA06ixmhXuBmwYx/YhAvWap53EdqMZX8CSk z7xrqg+JIaYZeUwy6G3YrhHabLigHIg= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=DX0oH4A4; spf=none (imf17.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620419; a=rsa-sha256; cv=none; b=oBS8hjoWiZTUBBS5Vqb+6C/eCEJkIRihPNoGuyCjG8QWqvQQYlcIw0dQOddjHZ36o6LCYX dB3deW3FFoB0yqourPKKJ+I/9HjAYiubS4GELoxKANjGgzg9ozB7N1gVIDSbnVB8GwRptT 79PWugC713mvVic1E+nCdvtfhazYhLU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620419; x=1710156419; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=pXZFCCXe+PDES5Btd5MhuDoEZZ+VyMVykxFB/NLFXd8=; b=DX0oH4A4hgc/FPe/UEPIUie3vgKKxGXpzjDpQjZOBiaxp8jgCVIiQh+8 2jVIH+LWlj55pQs9ZOkAX5laH1oRTZJs6SUrfaimZPEcqFbawD+Svxofo t+KvYOXC8MvJDm/fCph3atb8+CJxhU9z4wp11q4nd+k+iPAi2CRHpMc9w NZucrD9ZXs56vapSlqDdsL9gI5shFMOLYySB3XTtjTJEFvG0OtR7r1g8a CgwCmoALIKBbcC3z7Pdd4lOC4ruREcWW0FMVg2qcj7duXbRxiPxiUUeZj 7SQmiy1UXGtxGrGBR1FjrOFZ3p5jLB1e6c4lE0AN/lh5DJuV7HADqbwbi A==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="399577565" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="399577565" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="671580733" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="671580733" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:30 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id C3FDC10D7B9; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv16 11/17] x86/mm/iommu/sva: Make LAM and SVA mutually exclusive Date: Sun, 12 Mar 2023 14:26:06 +0300 Message-Id: <20230312112612.31869-12-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2A7AD40006 X-Stat-Signature: 3pkf1jb31i3qtpekr1b7jntc868y61ty X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1678620418-475892 X-HE-Meta: U2FsdGVkX1+MbzDj2JikHjxl38U5Qf+rmhH6d9q2kRDOp2Q25rBntagv7HEQ9w5+zKw/QbhB9mlHTW4AXhhie1l76tx89zBL6kYvkri1Vmwq1dQCUHJeSQetbJSBdweYNe1KigK05RJD+c0XduNOiJm8e3FaQlEc94PQIsB6HyAku8/LQ4EAaY7a8JKDnvsJ+DP2st3rqrFqElf57YgEDzOVoTw2b8Vyl0B6VIv9FyEdXVAKlZYKUy7+lrWZrSvB2hFhtLoLFcrweQG8obDvt6tiLHfMMNtJwgJlvogL136lngfmiXnx3gT86MsaM5xllgKpsN01oX54b5jDAfn9EA8CvkO3048X0NqIKMZTRHPZ4vBFvHUCk5E1yP+sWOIwurjgqPGCqFD/AsvbSAffotz3sXR88UGk9QT8FkDSJ+UteYsBE2pmz0DNvisKH2gHHhT8SwwpiZBjfl8SVq40OpFxihVPYBnrQ9/5dSR+TarIjzu7IGRz4qYYVTBYAradf7wX7xO9+wcIqAlQlJSjT3rw4UhtW2tkFmfRntzkH4Ig31fTWjQ322pfKKMpVhm10fFp6HJzE+LHlRxSztsRgg5tH5Eb1qEhtMfyg7IcFRk2xNbXVtq3LNye3iGeu+w9X0DNJAN+C7sdAsvwZzHU8aN23jZhrEQN0YuhKWqk2LKxwZkTGIDbZKerXFy/nLjni1RmOUBrKcMXWgHtuphX+9rrY1xBxKjGH9iyTUoYT05+hCAEuk/xf9DO96rzqTqR7oyyviWZXpFanRADgoKRIBveNFQpGuT5HXWicFUUJI79pR4OqBUDvDVbXp6IpchA0Bgmdd7q0x6NSAPrsJBSqz83R6dXIWHbwa3OTSdWkG5LnGlv7d5VGjhAlK/YJyPjt7SpPKpcFaBEovEA8eEdSr5kbnE12DY23AqPR293cwoNiG1nFYXT1nkXotvUhezc2p8QNjVJlHpF9w+YoDA GZ/mhnbd HWLDEV+dMJAGQ+nAWp3R492qIuf3JveXnD9X+MRHWkuzaOGfeoWno6UXBXP9JMn/cWf5QFVUxxxqihybPIp8PqQCuZKrA0aO0TeA4Jw9GjgyzjsJWsZH+Yp7PMiEOstx6hAIsZ1SYttx+/5sfaXNmLfk2ME1AZVvc8qHuOT3nzQzN3IX0x6M4gOtd+MT9KtuGmVRsms6lt8CdKMpjyDvN3n2QJpLvhdj1SqCY0cokm0QoGSyQQsNund9AS6su5Fz2O2wTFZ5+umekpzo2J1PgHP564tBRVCMu1uqE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: IOMMU and SVA-capable devices know nothing about LAM and only expect canonical addresses. An attempt to pass down tagged pointer will lead to address translation failure. By default do not allow to enable both LAM and use SVA in the same process. The new ARCH_FORCE_TAGGED_SVA arch_prctl() overrides the limitation. By using the arch_prctl() userspace takes responsibility to never pass tagged address to the device. Signed-off-by: Kirill A. Shutemov Reviewed-by: Ashok Raj Reviewed-by: Jacob Pan Acked-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/mmu.h | 2 ++ arch/x86/include/asm/mmu_context.h | 6 ++++++ arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/process_64.c | 7 +++++++ drivers/iommu/iommu-sva.c | 4 ++++ include/linux/mmu_context.h | 7 +++++++ 6 files changed, 27 insertions(+) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index e80762e998ce..0da5c227f490 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -14,6 +14,8 @@ #define MM_CONTEXT_HAS_VSYSCALL 1 /* Do not allow changing LAM mode */ #define MM_CONTEXT_LOCK_LAM 2 +/* Allow LAM and SVA coexisting */ +#define MM_CONTEXT_FORCE_TAGGED_SVA 3 /* * x86 has arch-specific MMU state beyond what lives in mm_struct. diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 06eaaf75d572..4c396e9a384f 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -115,6 +115,12 @@ static inline void mm_reset_untag_mask(struct mm_struct *mm) mm->context.untag_mask = -1UL; } +#define arch_pgtable_dma_compat arch_pgtable_dma_compat +static inline bool arch_pgtable_dma_compat(struct mm_struct *mm) +{ + return !mm_lam_cr3_mask(mm) || + test_bit(MM_CONTEXT_FORCE_TAGGED_SVA, &mm->context.flags); +} #else static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index a31e27b95b19..eb290d89cb32 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -23,5 +23,6 @@ #define ARCH_GET_UNTAG_MASK 0x4001 #define ARCH_ENABLE_TAGGED_ADDR 0x4002 #define ARCH_GET_MAX_TAG_BITS 0x4003 +#define ARCH_FORCE_TAGGED_SVA 0x4004 #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 88aae519c8f8..eda826a956df 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -756,6 +756,10 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) if (current->mm != mm) return -EINVAL; + if (mm_valid_pasid(mm) && + !test_bit(MM_CONTEXT_FORCE_TAGGED_SVA, &mm->context.flags)) + return -EINTR; + if (mmap_write_lock_killable(mm)) return -EINTR; @@ -878,6 +882,9 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) (unsigned long __user *)arg2); case ARCH_ENABLE_TAGGED_ADDR: return prctl_enable_tagged_addr(task->mm, arg2); + case ARCH_FORCE_TAGGED_SVA: + set_bit(MM_CONTEXT_FORCE_TAGGED_SVA, &task->mm->context.flags); + return 0; case ARCH_GET_MAX_TAG_BITS: if (!cpu_feature_enabled(X86_FEATURE_LAM)) return put_user(0, (unsigned long __user *)arg2); diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index 4ee2929f0d7a..dd76a1a09cf7 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -2,6 +2,7 @@ /* * Helpers for IOMMU drivers implementing SVA */ +#include #include #include #include @@ -32,6 +33,9 @@ int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max) min == 0 || max < min) return -EINVAL; + if (!arch_pgtable_dma_compat(mm)) + return -EBUSY; + mutex_lock(&iommu_sva_lock); /* Is a PASID already associated with this mm? */ if (mm_valid_pasid(mm)) { diff --git a/include/linux/mmu_context.h b/include/linux/mmu_context.h index 14b9c1fa05c4..f2b7a3f04099 100644 --- a/include/linux/mmu_context.h +++ b/include/linux/mmu_context.h @@ -35,4 +35,11 @@ static inline unsigned long mm_untag_mask(struct mm_struct *mm) } #endif +#ifndef arch_pgtable_dma_compat +static inline bool arch_pgtable_dma_compat(struct mm_struct *mm) +{ + return true; +} +#endif + #endif From patchwork Sun Mar 12 11:26:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D5D0C7618D for ; Sun, 12 Mar 2023 11:27:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 100FC8E0011; Sun, 12 Mar 2023 07:27:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B2098E000E; Sun, 12 Mar 2023 07:27:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E45938E0011; Sun, 12 Mar 2023 07:27:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D062A8E000E for ; Sun, 12 Mar 2023 07:27:06 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A7F9C1607CB for ; Sun, 12 Mar 2023 11:27:06 +0000 (UTC) X-FDA: 80560019652.24.F895980 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf05.hostedemail.com (Postfix) with ESMTP id A08DC10000D for ; Sun, 12 Mar 2023 11:27:04 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=bUvANxVW; spf=none (imf05.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620424; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8WES2Egcbbx4sLNeBblRPHPNuF/4HH/nRA13nY+qI74=; b=A7Npn91dCyy+Zy/RQexxdcN5rB28XHHz//LcEizR2U3V16C78xj633jA1zm5pp4y/vciGJ R/jdF+r690Xw4cu/arHjsfDPOC77dlPpb2gTE8TUvK/CnAqS99A8qxKxKIteYUAKc5mtH4 xYozo+V6S0NnVJEm1lHrwP0cL6R5NWY= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=bUvANxVW; spf=none (imf05.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620424; a=rsa-sha256; cv=none; b=ZLTvqWr4l5hk/Si3iZeL5VGXAjSYwwy+v4nUP1ZzkYh/BjVEMRnk8I964P5oHu+ZKQbv0I yGaowhchVxcP+U0bmaFfp2/MAKwzk8P5tUptN3HrIIeM/vdNTCakYzn+e194lT1AMofbBi 1xTbBMZMpDnCdCXfbszraTivyB/dkzI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620424; x=1710156424; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=j1UohvtSgF7gkFQBUOjnYuywsQdG3jzVdf5xH3SvjDM=; b=bUvANxVWFW6mkpvTBzHH8umGZylnIGDMYx2/Ir8h0nEYUbNvt+bTMa9b u2wpRa4xmANw/gmdrdCYnAHgGM2A1ZvuScw1mWuo1kuzDt2MSVRCianP4 9um6Cxxy9jqyerO9Mq4t9eBn/zAmNc7ntP8PLNGgvMFo6SJS7say43Gbm /MB/0NPDCYl7AOIytX/l4snh7GHNbsk1UU7KPJh7W7d7YGzENc8huCSKM xodezp82nRR9gv/rfkrn+NoRH8MB4XYfa/6kEoE12EvQm7VuyomW1ts2u bTP/WyiP2O+6JNVU1BwayjTrYYbBVqOl1Ijz28/e1epROF+4cwLgdAEyE Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="399577606" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="399577606" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:36 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="671580741" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="671580741" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:30 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id CD4F210D7BA; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv16 12/17] selftests/x86/lam: Add malloc and tag-bits test cases for linear-address masking Date: Sun, 12 Mar 2023 14:26:07 +0300 Message-Id: <20230312112612.31869-13-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: bxjedq9disohu16xax7umofwqkakwe5x X-Rspam-User: X-Rspamd-Queue-Id: A08DC10000D X-Rspamd-Server: rspam06 X-HE-Tag: 1678620424-191690 X-HE-Meta: U2FsdGVkX18WlePWC3aIjU2xiYzFd4dAxVfcLgSrhKaxvvyJqH9vBcIXb2/1B+Nt2Tf1sv9Zr7TgsGMtH59TeLOZmCO/lHUJaXLeXErVruGKALhdt9WkdJk/NruMNXRJ/cvSfQbAYkyffciYf6/h/K1apuMqCiiapEhjD9x+zZeAf801Mr4V6Qvk1Zp77xuPrGqWtomkiW+hLoBaqZ9QMhnsRf1D5+0x1bvd5kwbC4zxihzmjPHue2ODndv1GdcwtzHBZ9DWWdTUqrx0un2dvXrbVNEuzS19+Ptvg8HM/gIrU9W+OQtvRAaUi4GXqgeYVMCXR6BdK01M2mT2aNU8JAv553iGhkAE+P2UaQWpPhzCiFNgsodZrm/sngTokr/W1Jojadv6+etfx+VzIE4D8LG+1LE7fyu17W4JYIG6LxHyD2F4ORbJO4CpgCHBI+jdTr18EpMamx/jFpG91WF2YUbLzA21QXi0td0aWAn90zZsn7otOje8hz5MM5F5RnlucuRjrPnmhDK3eukSwx3t6/f5T15RoODc8g/KMDu7YH0x+uMwaEKJIzmdZIHHty2MKMLbpCDG7wHCpiUk89FIDGOlxe1AeYiNqHoJaeLAYMNaJWclfHPxEjblyLcZycpFKkshoq01UDNhNEXBLEWVXWfhH3jSCxtfMbB7RizP/rrU5mptPTrc1zBOsArq/+uBXZO7J+pNTsTmpWp1he5hQheRrASOwzlnJ9nmL252oPQh59Lg/1eRjVem5riPKyMFiaXjhCHi7sy1cXUB7eSkEmburSR5bxO2PySF1521mZRxXBSUnL5TtnSD3O3yinpfieFUqP912yaAei904oCZLC22mvaU5u5YRpXgCGSYC0TUZgabIlvvZw3pGRKM13zeJfDEzX1Mu6y08y7GhT/RUwQIzGMtdk12a83wFU2gQ/ZkLAGKLQ5FGntXZW9I6Tus/oeJhEJRLnxhB2apupN +GN2QWkz V2z74pGSlqOx6IxrxICLc2YwxAU3n45g0JbA71LyqqO4vc01vcXduUfQ1+92cCbS0uviP1KpwiZuRuOZbnes+uJPePU+eKkCEKjQqE9TPM3rDt/6GYtpir8Fqst6QNEk2fDma/sLvsvSOjZNg/C7lHdXV6tVuvmNnMKT5lueNW+OYioUIu9/OhcQ77/mJL5L+8AjRlVtkAVQeyQKXqNg4UkB6iTRfTwqHXyhX60TdTJJbjNEmFAEYWDZAuj3txTyw0Z5w8FirsIO2nh4Dx+TzA/56K0XvjE43yrDA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Weihong Zhang LAM is supported only in 64-bit mode and applies only addresses used for data accesses. In 64-bit mode, linear address have 64 bits. LAM is applied to 64-bit linear address and allow software to use high bits for metadata. LAM supports configurations that differ regarding which pointer bits are masked and can be used for metadata. LAM includes following mode: - LAM_U57, pointer bits in positions 62:57 are masked (LAM width 6), allows bits 62:57 of a user pointer to be used as metadata. There are some arch_prctls: ARCH_ENABLE_TAGGED_ADDR: enable LAM mode, mask high bits of a user pointer. ARCH_GET_UNTAG_MASK: get current untagged mask. ARCH_GET_MAX_TAG_BITS: the maximum tag bits user can request. zero if LAM is not supported. The LAM mode is for pre-process, a process has only one chance to set LAM mode. But there is no API to disable LAM mode. So all of test cases are run under child process. Functions of this test: MALLOC - LAM_U57 masks bits 57:62 of a user pointer. Process on user space can dereference such pointers. - Disable LAM, dereference a pointer with metadata above 48 bit or 57 bit lead to trigger SIGSEGV. TAG_BITS - Max tag bits of LAM_U57 is 6. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) --- tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/lam.c | 326 +++++++++++++++++++++++++++ 2 files changed, 327 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/x86/lam.c diff --git a/tools/testing/selftests/x86/Makefile b/tools/testing/selftests/x86/Makefile index ca9374b56ead..598135d3162b 100644 --- a/tools/testing/selftests/x86/Makefile +++ b/tools/testing/selftests/x86/Makefile @@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \ - corrupt_xstate_header amx + corrupt_xstate_header amx lam # Some selftests require 32bit support enabled also on 64bit systems TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c new file mode 100644 index 000000000000..268c1d2749af --- /dev/null +++ b/tools/testing/selftests/x86/lam.c @@ -0,0 +1,326 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../kselftest.h" + +#ifndef __x86_64__ +# error This test is 64-bit only +#endif + +/* LAM modes, these definitions were copied from kernel code */ +#define LAM_NONE 0 +#define LAM_U57_BITS 6 + +#define LAM_U57_MASK (0x3fULL << 57) +/* arch prctl for LAM */ +#define ARCH_GET_UNTAG_MASK 0x4001 +#define ARCH_ENABLE_TAGGED_ADDR 0x4002 +#define ARCH_GET_MAX_TAG_BITS 0x4003 + +/* Specified test function bits */ +#define FUNC_MALLOC 0x1 +#define FUNC_BITS 0x2 + +#define TEST_MASK 0x3 + +#define MALLOC_LEN 32 + +struct testcases { + unsigned int later; + int expected; /* 2: SIGSEGV Error; 1: other errors */ + unsigned long lam; + uint64_t addr; + int (*test_func)(struct testcases *test); + const char *msg; +}; + +int tests_cnt; +jmp_buf segv_env; + +static void segv_handler(int sig) +{ + ksft_print_msg("Get segmentation fault(%d).", sig); + siglongjmp(segv_env, 1); +} + +static inline int cpu_has_lam(void) +{ + unsigned int cpuinfo[4]; + + __cpuid_count(0x7, 1, cpuinfo[0], cpuinfo[1], cpuinfo[2], cpuinfo[3]); + + return (cpuinfo[0] & (1 << 26)); +} + +/* + * Set tagged address and read back untag mask. + * check if the untagged mask is expected. + * + * @return: + * 0: Set LAM mode successfully + * others: failed to set LAM + */ +static int set_lam(unsigned long lam) +{ + int ret = 0; + uint64_t ptr = 0; + + if (lam != LAM_U57_BITS && lam != LAM_NONE) + return -1; + + /* Skip check return */ + syscall(SYS_arch_prctl, ARCH_ENABLE_TAGGED_ADDR, lam); + + /* Get untagged mask */ + syscall(SYS_arch_prctl, ARCH_GET_UNTAG_MASK, &ptr); + + /* Check mask returned is expected */ + if (lam == LAM_U57_BITS) + ret = (ptr != ~(LAM_U57_MASK)); + else if (lam == LAM_NONE) + ret = (ptr != -1ULL); + + return ret; +} + +static unsigned long get_default_tag_bits(void) +{ + pid_t pid; + int lam = LAM_NONE; + int ret = 0; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + } else if (pid == 0) { + /* Set LAM mode in child process */ + if (set_lam(LAM_U57_BITS) == 0) + lam = LAM_U57_BITS; + else + lam = LAM_NONE; + exit(lam); + } else { + wait(&ret); + lam = WEXITSTATUS(ret); + } + + return lam; +} + +/* According to LAM mode, set metadata in high bits */ +static uint64_t set_metadata(uint64_t src, unsigned long lam) +{ + uint64_t metadata; + + srand(time(NULL)); + + switch (lam) { + case LAM_U57_BITS: /* Set metadata in bits 62:57 */ + /* Get a random non-zero value as metadata */ + metadata = (rand() % ((1UL << LAM_U57_BITS) - 1) + 1) << 57; + metadata |= (src & ~(LAM_U57_MASK)); + break; + default: + metadata = src; + break; + } + + return metadata; +} + +/* + * Set metadata in user pointer, compare new pointer with original pointer. + * both pointers should point to the same address. + * + * @return: + * 0: value on the pointer with metadate and value on original are same + * 1: not same. + */ +static int handle_lam_test(void *src, unsigned int lam) +{ + char *ptr; + + strcpy((char *)src, "USER POINTER"); + + ptr = (char *)set_metadata((uint64_t)src, lam); + if (src == ptr) + return 0; + + /* Copy a string into the pointer with metadata */ + strcpy((char *)ptr, "METADATA POINTER"); + + return (!!strcmp((char *)src, (char *)ptr)); +} + + +int handle_max_bits(struct testcases *test) +{ + unsigned long exp_bits = get_default_tag_bits(); + unsigned long bits = 0; + + if (exp_bits != LAM_NONE) + exp_bits = LAM_U57_BITS; + + /* Get LAM max tag bits */ + if (syscall(SYS_arch_prctl, ARCH_GET_MAX_TAG_BITS, &bits) == -1) + return 1; + + return (exp_bits != bits); +} + +/* + * Test lam feature through dereference pointer get from malloc. + * @return 0: Pass test. 1: Get failure during test 2: Get SIGSEGV + */ +static int handle_malloc(struct testcases *test) +{ + char *ptr = NULL; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) == -1) + return 1; + + ptr = (char *)malloc(MALLOC_LEN); + if (ptr == NULL) { + perror("malloc() failure\n"); + return 1; + } + + /* Set signal handler */ + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = handle_lam_test(ptr, test->lam); + } else { + ret = 2; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) == -1 && ret == 0) + ret = 1; + + free(ptr); + + return ret; +} + +static int fork_test(struct testcases *test) +{ + int ret, child_ret; + pid_t pid; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + ret = 1; + } else if (pid == 0) { + ret = test->test_func(test); + exit(ret); + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + } + + return ret; +} + +static void run_test(struct testcases *test, int count) +{ + int i, ret = 0; + + for (i = 0; i < count; i++) { + struct testcases *t = test + i; + + /* fork a process to run test case */ + ret = fork_test(t); + if (ret != 0) + ret = (t->expected == ret); + else + ret = !(t->expected); + + tests_cnt++; + ksft_test_result(ret, t->msg); + } +} + +static struct testcases malloc_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_malloc, + .msg = "MALLOC: LAM_U57. Dereferencing pointer with metadata\n", + }, + { + .later = 1, + .expected = 2, + .lam = LAM_U57_BITS, + .test_func = handle_malloc, + .msg = "MALLOC:[Negative] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + + +static struct testcases bits_cases[] = { + { + .test_func = handle_max_bits, + .msg = "BITS: Check default tag bits\n", + }, +}; + +static void cmd_help(void) +{ + printf("usage: lam [-h] [-t test list]\n"); + printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); + printf("\t\t0x1:malloc; 0x2:max_bits;\n"); + printf("\t-h: help\n"); +} + +int main(int argc, char **argv) +{ + int c = 0; + unsigned int tests = TEST_MASK; + + tests_cnt = 0; + + if (!cpu_has_lam()) { + ksft_print_msg("Unsupported LAM feature!\n"); + return -1; + } + + while ((c = getopt(argc, argv, "ht:")) != -1) { + switch (c) { + case 't': + tests = strtoul(optarg, NULL, 16); + if (!(tests & TEST_MASK)) { + ksft_print_msg("Invalid argument!\n"); + return -1; + } + break; + case 'h': + cmd_help(); + return 0; + default: + ksft_print_msg("Invalid argument\n"); + return -1; + } + } + + if (tests & FUNC_MALLOC) + run_test(malloc_cases, ARRAY_SIZE(malloc_cases)); + + if (tests & FUNC_BITS) + run_test(bits_cases, ARRAY_SIZE(bits_cases)); + + ksft_set_plan(tests_cnt); + + return ksft_exit_pass(); +} From patchwork Sun Mar 12 11:26:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AF3BC6FA99 for ; Sun, 12 Mar 2023 11:26:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 176C48E000B; Sun, 12 Mar 2023 07:26:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 128CB8E0008; Sun, 12 Mar 2023 07:26:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E45C38E000B; Sun, 12 Mar 2023 07:26:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D1BCC8E0008 for ; Sun, 12 Mar 2023 07:26:41 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A82F9C0AC4 for ; Sun, 12 Mar 2023 11:26:41 +0000 (UTC) X-FDA: 80560018602.10.FFA2FCE Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf25.hostedemail.com (Postfix) with ESMTP id A1488A0015 for ; Sun, 12 Mar 2023 11:26:39 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=VeABaVzV; spf=none (imf25.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620399; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/1rHcPra0U9eGBs+/YWoup97Bmtdd9oXHD+y/ukSepw=; b=e0tkwYAywnwwWvv0P8bEWPJTtuCuMYePGGdxGRf9niJtwxaRfgQQdP4IsU/RJkSp19RF5a Xy/vUb0qW34SVPMM7DLnXaBtpManU0vFDekdHfZPnVP8J119jn/ITmXty/++Aeh2gGiOiZ DZfGwdGKg6fJV0RfI0qt3G6GfiPd7W4= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=VeABaVzV; spf=none (imf25.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620399; a=rsa-sha256; cv=none; b=BfQqc6877jIlQP7N8edGUdoaWkg/kmUSGxlP/rW0tcPQnq76PEbY/G5F4xflX4qPRYPnyf T592A98Mtp6LGCqJwce6zxuoZmvBN39z8oF39bh155Gy2I+PCvA/Pr0jqRmmMdeIsKykJJ 9Nmph0SAT45SN0dJeL4ZJGV21D/pSfM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620399; x=1710156399; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=P0K/8sKCzyXH2J3CrkxRYfFuqzn3N+K8CC9BjCsCzdQ=; b=VeABaVzV/biv5G1y32IL3l7UOII6I/LU+gY4VbzkynEgDKaeV2rZ/qFp 9NzMdhLgILd3XfgYvs0gzYvZc4AsNGohX6Yl86Z8YSwYt+BWxqHO1SUQ9 h0PKxsKvPxduo2JIr7G7BwXtiqzQfH8zU74BeQUonI9I6RtexmWk594K3 y8oCxvp+OiQYFvOvY7Iqlo5M/81UeiYcdu5CgPkcVEapSktKSiaeQDLyX SLUP14Utjm7sHsedOSl0vHNjXhk3/2SO5KICEJ31C5iFPag5oKif8vTIT 2tRRETHK3AsMrOMs75qU2OlhmuckZpmUggyQF2CRNR59RSGLsHunUCYKC g==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="339349983" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="339349983" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:36 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="852438037" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="852438037" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:30 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id D7FE110D7BB; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv16 13/17] selftests/x86/lam: Add mmap and SYSCALL test cases for linear-address masking Date: Sun, 12 Mar 2023 14:26:08 +0300 Message-Id: <20230312112612.31869-14-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: kgc11fzofcexmqe3e3tsb537si18m9eg X-Rspamd-Queue-Id: A1488A0015 X-HE-Tag: 1678620399-803997 X-HE-Meta: U2FsdGVkX1+GLH+qVx+hbUjAIzEth1MysukNpTyoFOGZG+APFwODiW5hpMymX2e47pVs4XeahedRTkHut9HCeXOJmvQ/MMuDPd6T/7F17eGVe4AsoEdJX8fWwfp/QAzpiyts9DMqaz5dlpynmBHTvgB11VV57ylkMSvPPg3tSiBqk6MErvxLjWie0lV5Xq+esq2tHUWwQzYv09jB50aUNtGxkIyx9+ZVERYq7Kt9JDTS8woWirg9uTt+3a3QNH7uXaxPJvrYmeU/Miwyb6oxOOhsBdwL0T8RiKgz+C9agEcw9zTxQJwTf5+HhsqJVKfGARnBjIAfuteGD20/dDi1VNJww/29isA1+aL8okY1LrUpzpfcy9CsIL1pnfMsvROv29y4P0SRBqaAk5cRQVrHGqZ75k+pDn5LCim99JFns+7SMuyRPHq1hTmtWdB4JFQpqP1P9FHqM52OwiypKaXFvqDCur15C/sAcwFmepdlKEzHdREX4bVByTCqIzNEfNP/HVqjx9INOXaSjpAKea1hYqZl2uVW+k1h0uRj+fEjdfQcObjAMGP/55PRR0BCb2CyQYii/6yAvS/ZTTQqcX11fjtoD8xpOmDrPxeS4aAjd6dM1HnGeI+FoCPVChbNGNwTf8YAMjUBaYt+/acYciHJoXPQd+MOrHPy25xi9eoqCaZWBPMenDW4AuL5m9CNt4t4yx4Estf1uqxy7eSrnLZLmRIsL8DG/d0OaLRcmG0gKIwRqeRzWgqd7rFM9kdQgsicVzbD5OgBhrtGNJRgFNlNGkzPNnrEaT24EUm4Zl5kEIUpf4eCBtvpNwic6SOJA69Xm7A/61WgjFDCAjpeWTqP5UZj7xB028XfawSagUo19nHJr8hIMSRe/d0jfcPEQWzd4QCHj62IQYcOB3wHrCxmG7BcQYbNEAgndCzDUnsOe64kmEH1rsdn//pkn2w8ZXDvSjTH0TOsb6ciuv2EDsr +HOm1rvG xA5njs1PrlyCXwZPl/oNFreyCBIReXKq679/+Ds6C0VnAwIAc8O/osW7/fZCO1EPZLwMhQBIgffapBJdsn8IQ6C6mnHvonui9qbau6V1m75CMFyueYPej1iTKuW7sMkV21BSPWsGb+rlR38+hjr7Dee+yFUrll5cPPaI07Qq0FL8qw85NF+tfNlwThaHl1sEBXdxn3cIQ4vMINDlM4OGXwKMQjEj26/PQznf9rzlm0DhAS2nC1hfEJjdKRLMq+hV1NYIWBKddg+U9zmQoI8viqQh4GR6hXySKfwlt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Weihong Zhang Add mmap and SYSCALL test cases. SYSCALL test cases: - LAM supports set metadata in high bits 62:57 (LAM_U57) of a user pointer, pass the pointer to SYSCALL, SYSCALL can dereference the pointer and return correct result. - Disable LAM, pass a pointer with metadata in high bits to SYSCALL, SYSCALL returns -1 (EFAULT). MMAP test cases: - Enable LAM_U57, MMAP with low address (below bits 47), set metadata in high bits of the address, dereference the address should be allowed. - Enable LAM_U57, MMAP with high address (above bits 47), set metadata in high bits of the address, dereference the address should be allowed. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) --- tools/testing/selftests/x86/lam.c | 144 +++++++++++++++++++++++++++++- 1 file changed, 140 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index 268c1d2749af..39ebfc511685 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -29,11 +30,18 @@ /* Specified test function bits */ #define FUNC_MALLOC 0x1 #define FUNC_BITS 0x2 +#define FUNC_MMAP 0x4 +#define FUNC_SYSCALL 0x8 -#define TEST_MASK 0x3 +#define TEST_MASK 0xf + +#define LOW_ADDR (0x1UL << 30) +#define HIGH_ADDR (0x3UL << 48) #define MALLOC_LEN 32 +#define PAGE_SIZE (4 << 10) + struct testcases { unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ @@ -49,6 +57,7 @@ jmp_buf segv_env; static void segv_handler(int sig) { ksft_print_msg("Get segmentation fault(%d).", sig); + siglongjmp(segv_env, 1); } @@ -61,6 +70,16 @@ static inline int cpu_has_lam(void) return (cpuinfo[0] & (1 << 26)); } +/* Check 5-level page table feature in CPUID.(EAX=07H, ECX=00H):ECX.[bit 16] */ +static inline int cpu_has_la57(void) +{ + unsigned int cpuinfo[4]; + + __cpuid_count(0x7, 0, cpuinfo[0], cpuinfo[1], cpuinfo[2], cpuinfo[3]); + + return (cpuinfo[2] & (1 << 16)); +} + /* * Set tagged address and read back untag mask. * check if the untagged mask is expected. @@ -213,6 +232,68 @@ static int handle_malloc(struct testcases *test) return ret; } +static int handle_mmap(struct testcases *test) +{ + void *ptr; + unsigned int flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + ptr = mmap((void *)test->addr, PAGE_SIZE, PROT_READ | PROT_WRITE, + flags, -1, 0); + if (ptr == MAP_FAILED) { + if (test->addr == HIGH_ADDR) + if (!cpu_has_la57()) + return 3; /* unsupport LA57 */ + return 1; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + ret = 1; + + if (ret == 0) { + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = handle_lam_test(ptr, test->lam); + } else { + ret = 2; + } + } + + munmap(ptr, PAGE_SIZE); + return ret; +} + +static int handle_syscall(struct testcases *test) +{ + struct utsname unme, *pu; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + pu = (struct utsname *)set_metadata((uint64_t)&unme, test->lam); + ret = uname(pu); + if (ret < 0) + ret = 1; + } else { + ret = 2; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) != -1 && ret == 0) + ret = 1; + + return ret; +} + static int fork_test(struct testcases *test) { int ret, child_ret; @@ -241,13 +322,20 @@ static void run_test(struct testcases *test, int count) struct testcases *t = test + i; /* fork a process to run test case */ + tests_cnt++; ret = fork_test(t); + + /* return 3 is not support LA57, the case should be skipped */ + if (ret == 3) { + ksft_test_result_skip(t->msg); + continue; + } + if (ret != 0) ret = (t->expected == ret); else ret = !(t->expected); - tests_cnt++; ksft_test_result(ret, t->msg); } } @@ -268,7 +356,6 @@ static struct testcases malloc_cases[] = { }, }; - static struct testcases bits_cases[] = { { .test_func = handle_max_bits, @@ -276,11 +363,54 @@ static struct testcases bits_cases[] = { }, }; +static struct testcases syscall_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_syscall, + .msg = "SYSCALL: LAM_U57. syscall with metadata\n", + }, + { + .later = 1, + .expected = 1, + .lam = LAM_U57_BITS, + .test_func = handle_syscall, + .msg = "SYSCALL:[Negative] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + +static struct testcases mmap_cases[] = { + { + .later = 1, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First mmap high address, then set LAM_U57.\n", + }, + { + .later = 0, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First LAM_U57, then High address.\n", + }, + { + .later = 0, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = LOW_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First LAM_U57, then Low address.\n", + }, +}; + static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc; 0x2:max_bits;\n"); + printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall.\n"); printf("\t-h: help\n"); } @@ -320,6 +450,12 @@ int main(int argc, char **argv) if (tests & FUNC_BITS) run_test(bits_cases, ARRAY_SIZE(bits_cases)); + if (tests & FUNC_MMAP) + run_test(mmap_cases, ARRAY_SIZE(mmap_cases)); + + if (tests & FUNC_SYSCALL) + run_test(syscall_cases, ARRAY_SIZE(syscall_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Sun Mar 12 11:26:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171308 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0428BC6FA99 for ; Sun, 12 Mar 2023 11:27:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 39C408E0010; Sun, 12 Mar 2023 07:27:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2ADF58E000E; Sun, 12 Mar 2023 07:27:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D8598E0010; Sun, 12 Mar 2023 07:27:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id F34488E000E for ; Sun, 12 Mar 2023 07:27:05 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9D50B1607C9 for ; Sun, 12 Mar 2023 11:27:05 +0000 (UTC) X-FDA: 80560019610.28.C35B1B5 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf19.hostedemail.com (Postfix) with ESMTP id 711E71A000C for ; Sun, 12 Mar 2023 11:27:03 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=fLGyrNpX; spf=none (imf19.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620423; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=x7iX9177of6b+cDB521u3bwYz9GbX1Zfo7fEoarV4D4=; b=W2EjZ3WKqJ8BfqrBGxcheIOS5CqT0Qb1d7pUY9TU44k92h2cD7+tvSp0uW1DYg1LSyY38u B1auhFjCb5ENn0LsXyXyoLhTgyaxMt6ooh+JemYGyKjrOngRWCxuO8FxQ8l64hx1h09nEN vvF46icDS+8fY8LNVP1i665fCnRRNkc= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=fLGyrNpX; spf=none (imf19.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620423; a=rsa-sha256; cv=none; b=uFrvZ+8deZatOcl2J4Y4IX2OVPJuZfavqB22i46z97PtvV9/QvqxUQ9M86oMFyZvKx6DEa zFExfiNwzZdhK/5tFLFfm7o1bEfZVGA6EfWB8b7UPjpqiC+cdHLGpysMIZU0vR0e4cQxNU nOUJvGaky3UKGH8wuwMN0Xa16xVyCzs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620423; x=1710156423; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=9L4ajpkEF2tCbqfWbypu8Ern3y+RsXjqk5niIfNGqy4=; b=fLGyrNpXDr9pq7fjCBRQoEqAiZBRGZfk1kMG7koq5zKUVWW4jJWFN2s5 NF0sLvsucm1khos8WE/vNOa0rLBbJl2yaYoahoP0jPGXrO0J0EuiGyPVm xXRUguO3+2R4L/ezLKq574YD5a7YkKkMUVkjyeIX0mPu1gnrcZQUjq9yQ +Xw4hv6it/yfDdmDHzwvD/d/LS49Uu/Kt75mNZ4QmFNjx/Ti5Jdk8qZqf RC0iONnAQVZHcfKMCkS852V70R0WpYyR7C7Ts4qJ49LwuHZ/aIKnWmjjg cyEJekignqZslqiC/C3HfqCRbx9f/JVQ+Fk+10y916fWtEBRyBsxh/4zd Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="399577598" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="399577598" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:36 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="671580739" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="671580739" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:30 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id E234310D7BC; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv16 14/17] selftests/x86/lam: Add io_uring test cases for linear-address masking Date: Sun, 12 Mar 2023 14:26:09 +0300 Message-Id: <20230312112612.31869-15-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 711E71A000C X-Stat-Signature: g6ice8wm8qun3onzwj6gox87pbxd67j3 X-Rspam-User: X-HE-Tag: 1678620423-304928 X-HE-Meta: U2FsdGVkX1+GdHONHHCsrCLinhcsU5rOCuFUgLwHHflyscJ7aYdYwdPADOAo3r+NSysOKzKAzyVnbkFkk02tyTTlsTNBnrllrQ1dk4w+QHG0H6XKO819JBHnFyNAxJLdJTL87swNNtfVIsbNv03vxC20ovj75I74YQHnoa86HFdL1lQSMMYkwmc5s4JpwNRKfPwbzwlACnzR4pK0oAsLoWPbBAxnfLqFFfepz9Um+njJkXSdhfy06PBuIemU5O4QUQLbO1hufefleQmNeT9vxjUucOtj0Ys87PEO6Di0EJCtXPvEu9NLSX7ZbsczYTJCJS8EZZYJqwn4I8jF7s3y9hCKLZ7QfHtsfFNgzZdXRkJ0jn/cICP9bNc+W5NyE8K2UCuYulqSKzQoHZgvTk32TJCWZAICnyFi31Pc6Ywu52k/X1MvbKBkHr7VrUVbZeT0CWAhCr9sIyPI+X+OhKS0/jnFBwBIoUYseFUsYZlOqPAcSFFTUjqSIJGNBnjHx3uamujr0MbM3FnGcLRyjdjU087Lag565v/0g8O3GvDcHipiSBIRsikBBGJtPWq8m/uPAhHoSDsbhzbX60TMWohhOmuxeZkN3f6QAC5me86YshOi2El5I5Bsytc/JkA9SdobioR8ozLdWUZQEhAHGNEea+YSPgwD7GXrCJsTE27KaxJ8wR8RJkp0L5U4yJEt+rt43D9O1ELrBseq7OkYnotJ843auQ2mQdkSiBe15wT9/6S3GI3PdZDABKWBBDU1aYeQW4TyoeU7volCSjUuppkKEAqSuX8VrhKU6bS6d0izvWziHF/Bz1DLiLpUWap+7oLleLL+DxGFsZrjOpT4dMB65x07rQo/1x39ZFOR/Fqo+jeCFeoOPJFQ0BV8yvjMCsZUXo4Zk055yurgFO8KUgKrkglTjYFyBzV0AAfq5KX6v1O5/V4j1RIarishv1yzrav0dnY7BudPH1s0I/yqiaS t+zZ9gNT NUtnUOQKfcpEWuOtaazqabTD56YlapRkJthbUdz8Qg9ZCUcf/1AHsjn6WqPUlETcCmB27zEK/tut9A5sxaLYYfp3bz3hqvWzzoez0Enn6fHpWzxn1XAYQEYIuj5bEzO6Xqi3wYkIlf6R6Ya4s3sgp1NsJtqkspSQ9QRmZAJEK1gb9NamCdpoQpHX4HiByP2F1Smla40/Z22SdIpedMj/wPOIc8u42P7/csI6zDQqgGfmaHm6FuaPCaLZPLHZCc7cOK5qHFEI+sBlqqYyYed3asTH6m1ALijxXifAvtgjhG9rGLP4MRUVufefc4n0cnV8t7zzW9lGxQepJX0VA6MM3Ici/JQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Weihong Zhang LAM should be supported in kernel thread, using io_uring to verify LAM feature. The test cases implement read a file through io_uring, the test cases choose an iovec array as receiving buffer, which used to receive data, according to LAM mode, set metadata in high bits of these buffer. io_uring can deal with these buffers that pointed to pointers with the metadata in high bits. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) --- tools/testing/selftests/x86/lam.c | 341 +++++++++++++++++++++++++++++- 1 file changed, 339 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index 39ebfc511685..52750ebd0887 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -9,8 +9,12 @@ #include #include #include +#include +#include #include +#include +#include #include "../kselftest.h" #ifndef __x86_64__ @@ -32,8 +36,9 @@ #define FUNC_BITS 0x2 #define FUNC_MMAP 0x4 #define FUNC_SYSCALL 0x8 +#define FUNC_URING 0x10 -#define TEST_MASK 0xf +#define TEST_MASK 0x1f #define LOW_ADDR (0x1UL << 30) #define HIGH_ADDR (0x3UL << 48) @@ -42,6 +47,13 @@ #define PAGE_SIZE (4 << 10) +#define barrier() ({ \ + __asm__ __volatile__("" : : : "memory"); \ +}) + +#define URING_QUEUE_SZ 1 +#define URING_BLOCK_SZ 2048 + struct testcases { unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ @@ -51,6 +63,33 @@ struct testcases { const char *msg; }; +/* Used by CQ of uring, source file handler and file's size */ +struct file_io { + int file_fd; + off_t file_sz; + struct iovec iovecs[]; +}; + +struct io_uring_queue { + unsigned int *head; + unsigned int *tail; + unsigned int *ring_mask; + unsigned int *ring_entries; + unsigned int *flags; + unsigned int *array; + union { + struct io_uring_cqe *cqes; + struct io_uring_sqe *sqes; + } queue; + size_t ring_sz; +}; + +struct io_ring { + int ring_fd; + struct io_uring_queue sq_ring; + struct io_uring_queue cq_ring; +}; + int tests_cnt; jmp_buf segv_env; @@ -294,6 +333,285 @@ static int handle_syscall(struct testcases *test) return ret; } +int sys_uring_setup(unsigned int entries, struct io_uring_params *p) +{ + return (int)syscall(__NR_io_uring_setup, entries, p); +} + +int sys_uring_enter(int fd, unsigned int to, unsigned int min, unsigned int flags) +{ + return (int)syscall(__NR_io_uring_enter, fd, to, min, flags, NULL, 0); +} + +/* Init submission queue and completion queue */ +int mmap_io_uring(struct io_uring_params p, struct io_ring *s) +{ + struct io_uring_queue *sring = &s->sq_ring; + struct io_uring_queue *cring = &s->cq_ring; + + sring->ring_sz = p.sq_off.array + p.sq_entries * sizeof(unsigned int); + cring->ring_sz = p.cq_off.cqes + p.cq_entries * sizeof(struct io_uring_cqe); + + if (p.features & IORING_FEAT_SINGLE_MMAP) { + if (cring->ring_sz > sring->ring_sz) + sring->ring_sz = cring->ring_sz; + + cring->ring_sz = sring->ring_sz; + } + + void *sq_ptr = mmap(0, sring->ring_sz, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, s->ring_fd, + IORING_OFF_SQ_RING); + + if (sq_ptr == MAP_FAILED) { + perror("sub-queue!"); + return 1; + } + + void *cq_ptr = sq_ptr; + + if (!(p.features & IORING_FEAT_SINGLE_MMAP)) { + cq_ptr = mmap(0, cring->ring_sz, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, s->ring_fd, + IORING_OFF_CQ_RING); + if (cq_ptr == MAP_FAILED) { + perror("cpl-queue!"); + munmap(sq_ptr, sring->ring_sz); + return 1; + } + } + + sring->head = sq_ptr + p.sq_off.head; + sring->tail = sq_ptr + p.sq_off.tail; + sring->ring_mask = sq_ptr + p.sq_off.ring_mask; + sring->ring_entries = sq_ptr + p.sq_off.ring_entries; + sring->flags = sq_ptr + p.sq_off.flags; + sring->array = sq_ptr + p.sq_off.array; + + /* Map a queue as mem map */ + s->sq_ring.queue.sqes = mmap(0, p.sq_entries * sizeof(struct io_uring_sqe), + PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, + s->ring_fd, IORING_OFF_SQES); + if (s->sq_ring.queue.sqes == MAP_FAILED) { + munmap(sq_ptr, sring->ring_sz); + if (sq_ptr != cq_ptr) { + ksft_print_msg("failed to mmap uring queue!"); + munmap(cq_ptr, cring->ring_sz); + return 1; + } + } + + cring->head = cq_ptr + p.cq_off.head; + cring->tail = cq_ptr + p.cq_off.tail; + cring->ring_mask = cq_ptr + p.cq_off.ring_mask; + cring->ring_entries = cq_ptr + p.cq_off.ring_entries; + cring->queue.cqes = cq_ptr + p.cq_off.cqes; + + return 0; +} + +/* Init io_uring queues */ +int setup_io_uring(struct io_ring *s) +{ + struct io_uring_params para; + + memset(¶, 0, sizeof(para)); + s->ring_fd = sys_uring_setup(URING_QUEUE_SZ, ¶); + if (s->ring_fd < 0) + return 1; + + return mmap_io_uring(para, s); +} + +/* + * Get data from completion queue. the data buffer saved the file data + * return 0: success; others: error; + */ +int handle_uring_cq(struct io_ring *s) +{ + struct file_io *fi = NULL; + struct io_uring_queue *cring = &s->cq_ring; + struct io_uring_cqe *cqe; + unsigned int head; + off_t len = 0; + + head = *cring->head; + + do { + barrier(); + if (head == *cring->tail) + break; + /* Get the entry */ + cqe = &cring->queue.cqes[head & *s->cq_ring.ring_mask]; + fi = (struct file_io *)cqe->user_data; + if (cqe->res < 0) + break; + + int blocks = (int)(fi->file_sz + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + for (int i = 0; i < blocks; i++) + len += fi->iovecs[i].iov_len; + + head++; + } while (1); + + *cring->head = head; + barrier(); + + return (len != fi->file_sz); +} + +/* + * Submit squeue. specify via IORING_OP_READV. + * the buffer need to be set metadata according to LAM mode + */ +int handle_uring_sq(struct io_ring *ring, struct file_io *fi, unsigned long lam) +{ + int file_fd = fi->file_fd; + struct io_uring_queue *sring = &ring->sq_ring; + unsigned int index = 0, cur_block = 0, tail = 0, next_tail = 0; + struct io_uring_sqe *sqe; + + off_t remain = fi->file_sz; + int blocks = (int)(remain + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + while (remain) { + off_t bytes = remain; + void *buf; + + if (bytes > URING_BLOCK_SZ) + bytes = URING_BLOCK_SZ; + + fi->iovecs[cur_block].iov_len = bytes; + + if (posix_memalign(&buf, URING_BLOCK_SZ, URING_BLOCK_SZ)) + return 1; + + fi->iovecs[cur_block].iov_base = (void *)set_metadata((uint64_t)buf, lam); + remain -= bytes; + cur_block++; + } + + next_tail = *sring->tail; + tail = next_tail; + next_tail++; + + barrier(); + + index = tail & *ring->sq_ring.ring_mask; + + sqe = &ring->sq_ring.queue.sqes[index]; + sqe->fd = file_fd; + sqe->flags = 0; + sqe->opcode = IORING_OP_READV; + sqe->addr = (unsigned long)fi->iovecs; + sqe->len = blocks; + sqe->off = 0; + sqe->user_data = (uint64_t)fi; + + sring->array[index] = index; + tail = next_tail; + + if (*sring->tail != tail) { + *sring->tail = tail; + barrier(); + } + + if (sys_uring_enter(ring->ring_fd, 1, 1, IORING_ENTER_GETEVENTS) < 0) + return 1; + + return 0; +} + +/* + * Test LAM in async I/O and io_uring, read current binery through io_uring + * Set metadata in pointers to iovecs buffer. + */ +int do_uring(unsigned long lam) +{ + struct io_ring *ring; + struct file_io *fi; + struct stat st; + int ret = 1; + char path[PATH_MAX]; + + /* get current process path */ + if (readlink("/proc/self/exe", path, PATH_MAX) <= 0) + return 1; + + int file_fd = open(path, O_RDONLY); + + if (file_fd < 0) + return 1; + + if (fstat(file_fd, &st) < 0) + return 1; + + off_t file_sz = st.st_size; + + int blocks = (int)(file_sz + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + fi = malloc(sizeof(*fi) + sizeof(struct iovec) * blocks); + if (!fi) + return 1; + + fi->file_sz = file_sz; + fi->file_fd = file_fd; + + ring = malloc(sizeof(*ring)); + if (!ring) + return 1; + + memset(ring, 0, sizeof(struct io_ring)); + + if (setup_io_uring(ring)) + goto out; + + if (handle_uring_sq(ring, fi, lam)) + goto out; + + ret = handle_uring_cq(ring); + +out: + free(ring); + + for (int i = 0; i < blocks; i++) { + if (fi->iovecs[i].iov_base) { + uint64_t addr = ((uint64_t)fi->iovecs[i].iov_base); + + switch (lam) { + case LAM_U57_BITS: /* Clear bits 62:57 */ + addr = (addr & ~(0x3fULL << 57)); + break; + } + free((void *)addr); + fi->iovecs[i].iov_base = NULL; + } + } + + free(fi); + + return ret; +} + +int handle_uring(struct testcases *test) +{ + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = do_uring(test->lam); + } else { + ret = 2; + } + + return ret; +} + static int fork_test(struct testcases *test) { int ret, child_ret; @@ -340,6 +658,22 @@ static void run_test(struct testcases *test, int count) } } +static struct testcases uring_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_uring, + .msg = "URING: LAM_U57. Dereferencing pointer with metadata\n", + }, + { + .later = 1, + .expected = 1, + .lam = LAM_U57_BITS, + .test_func = handle_uring, + .msg = "URING:[Negative] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + static struct testcases malloc_cases[] = { { .later = 0, @@ -410,7 +744,7 @@ static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall.\n"); + printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring.\n"); printf("\t-h: help\n"); } @@ -456,6 +790,9 @@ int main(int argc, char **argv) if (tests & FUNC_SYSCALL) run_test(syscall_cases, ARRAY_SIZE(syscall_cases)); + if (tests & FUNC_URING) + run_test(uring_cases, ARRAY_SIZE(uring_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Sun Mar 12 11:26:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171312 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CCF0C6FD1F for ; Sun, 12 Mar 2023 11:27:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F2ADD8E000E; Sun, 12 Mar 2023 07:27:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EBFA28E0015; Sun, 12 Mar 2023 07:27:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C98DE8E000E; Sun, 12 Mar 2023 07:27:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A35C58E000E for ; Sun, 12 Mar 2023 07:27:13 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 8615A1A0835 for ; Sun, 12 Mar 2023 11:27:13 +0000 (UTC) X-FDA: 80560019946.12.B11A56B Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf05.hostedemail.com (Postfix) with ESMTP id 48F1F100012 for ; Sun, 12 Mar 2023 11:27:11 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=I5PqcuJC; spf=none (imf05.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620431; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=99fwD7k8d8h6LPp7vh7F5Wrp1FzWfOzQVN1wtAJK6DE=; b=QFPPs3Fqc2aQ8jh0SeuuxhE2iH9+T7IU7JYVX3omlK878WTCMMzisOCsjcvfNgUH0w/s9j icM9rXX6HX7BM0iCWbv9IDmM3Pdpm+8e5nJLj4LtL5GGMfLDlYKmq0na70L+Xgn1s/oelq pQsVGjznFNu8bGU2fBJba9R+R0aHpoM= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=I5PqcuJC; spf=none (imf05.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620431; a=rsa-sha256; cv=none; b=pWdj1idLYFej8+MllBIspMqI14mn/iOmWvWFKXzMeNvucZsdIZ9F02fue2R+TJzNVa26Kj VVpISq+o5oarVFdUoZjaDCGRe13pJcn0asi33fy1O4wbLWJkJE9C13YX8xvm8907w3fIoW 1oz8IKCC/cmA/Lct92AA13HqpHUcS9M= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620431; x=1710156431; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=rOpAXZn5KS9JrkJNB2EKQ1dUWIxcgetG1myY1VbHNB4=; b=I5PqcuJCVb70BYIuPQUvz7HUOfqSHfoi/4ywbY6IUomjheUo1DUwjH0Z V4kLFreTlxzOAY75BtMzKWio6Lt1qhEPr4tDxs9KW4QBeD9HUkPmvANfA bs9QlkOv1GGD9AFp2GRlbsMf9EPSUzLKlo7r29h7zbO8yr/VsOH8Op3ro 6MJRoav4/5yE9FJ5ujy4wfoivbVMtb1HpD6s111g+NTdErMTn29Y0dMAj NW/GaF9sDoC1H5+AZNnh/Vxmf32rzXuRbRPPi+NMyD4qBY8lyBHKc5ENc r38prlv2/F6NvzcHY0rdbmc6CIIC2VNcY4ODGOcAmEMKKSYCTD106rIl/ A==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="399577677" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="399577677" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="671580759" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="671580759" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:38 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id ECA9D10D7BD; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv16 15/17] selftests/x86/lam: Add inherit test cases for linear-address masking Date: Sun, 12 Mar 2023 14:26:10 +0300 Message-Id: <20230312112612.31869-16-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: tagobsbx7j9ej7y8xmb59prp5oe439cb X-Rspam-User: X-Rspamd-Queue-Id: 48F1F100012 X-Rspamd-Server: rspam06 X-HE-Tag: 1678620431-343841 X-HE-Meta: U2FsdGVkX1+FNHpPP3GwsB1mhSGgKuxOi9k3hVg2QL+OYQk97VvYkDUZ2wMEuXrJlgPzEjbtaM1WEopTp1pKbHOpn2tDDKXeRLSzsk8ZGWDW6Aold29r4CNwTmbOdSZ9au0g7suf2XD2WuhfOxKOdbTrVH2Rm91ORiWwtiZQ8qRrYroTQq27MJRl0WviLhmWVTHi4Wn1BfapCt6xx1zyzCBAjQ89QS8NtE5WWIFfz7qEa9wS/wq+YoazBIrLxXzMQwY9qWokkmDvE8lDzF2y4vzk9K+HvSSet7bK3nQPO7Xd4YsDwWCYPvMOSW3Ojrijbhp8+fKPtCzQBmdn/V09c3kmC5VL6ilIo0jqezQHMr2ydq0MpHgPECrJFZUI6icMrXsVhmn1mL7QxyTZ8XgTT068jwOfOat61gt9vcIwJ/MRyjFW28fxRVDuHx1g1NVuIkaQlcjonkDBmdQu7pZixO4fyI6mXQycInB6kcKj7FsHBmyieGVFeiJ+bQi8cuYE3D3Na5/BW+UIX5/GkTAEq8QlKBdENBL0iweUOamAM4abiCafMIHTqENdNc+M0F3i5352jnADzJKPuxdFmxbI9bE75ryjqnd/4tAFVk8wwDEebvbzcJWceuc7Rse3A3PKaA4O4EaA8GmECYbopQyCtzXWyTUSVr15zvbjQVL46csfGh1XmRzSBY2vXBVk1LtzjjhvIOEhwC/hDO7l930ZbkdEqmbSeM1XXcsEHQFiax7YYlixcNM9IF66BDILWs5cP1j4pN2crg23U/50Urv+lsrUJl2/1AfHspDCKnfPlZHMDDKB8rN8NGnfk7REgPpVNzijb1TXkLgo2P8eBYBOH+R9lJKdgY60x55yoMOPKSZg+TAWaC0/mQhY3c6a1yv78zDLwuwlQiPdOJm42TnV+LBYcx4ZwFmSxL5nXxNEoaTfgCqcUt+ScV3aNdzqfW9VQVaEP6vGvJ5UbAaDzy9 NRR21Gd3 SihMc9QGEMz9mPz6mHcpiuC1cVgRpVQWwVp3sxFtzGUWBXSfW9ONLcIltL+t0ZM3aWJ946sMGLfsmZFa9LkrqHVlmBa46Qr+wwXOUUbuHpWG5nzzurV2vHN0/bwoRjY936rRpfILR8e2Vw4xvh7UK8dIPcmKILLPQxQ+7vnFWlb5FzoT+smnlf53WQqjTzm269eNyMz6oFpZLVMO+xMoxk6sk4vNoVnWbrMwgluhdDpf+TTSbugttY1Mona/uH3bX7Qmpx+FzYnhrTRnnLElaI98wBf2btPd4xZqC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Weihong Zhang LAM is enabled per-thread and gets inherited on fork(2)/clone(2). exec() reverts LAM status to the default disabled state. There are two test scenarios: - Fork test cases: These cases were used to test the inheritance of LAM for per-thread, Child process generated by fork() should inherit LAM feature from parent process, Child process can get the LAM mode same as parent process. - Execve test cases: Processes generated by execve() are different from processes generated by fork(), these processes revert LAM status to disabled status. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) --- tools/testing/selftests/x86/lam.c | 125 +++++++++++++++++++++++++++++- 1 file changed, 121 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index 52750ebd0887..ebabd4333b7d 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -37,8 +37,9 @@ #define FUNC_MMAP 0x4 #define FUNC_SYSCALL 0x8 #define FUNC_URING 0x10 +#define FUNC_INHERITE 0x20 -#define TEST_MASK 0x1f +#define TEST_MASK 0x3f #define LOW_ADDR (0x1UL << 30) #define HIGH_ADDR (0x3UL << 48) @@ -174,6 +175,28 @@ static unsigned long get_default_tag_bits(void) return lam; } +/* + * Set tagged address and read back untag mask. + * check if the untag mask is expected. + */ +static int get_lam(void) +{ + uint64_t ptr = 0; + int ret = -1; + /* Get untagged mask */ + if (syscall(SYS_arch_prctl, ARCH_GET_UNTAG_MASK, &ptr) == -1) + return -1; + + /* Check mask returned is expected */ + if (ptr == ~(LAM_U57_MASK)) + ret = LAM_U57_BITS; + else if (ptr == -1ULL) + ret = LAM_NONE; + + + return ret; +} + /* According to LAM mode, set metadata in high bits */ static uint64_t set_metadata(uint64_t src, unsigned long lam) { @@ -581,7 +604,7 @@ int do_uring(unsigned long lam) switch (lam) { case LAM_U57_BITS: /* Clear bits 62:57 */ - addr = (addr & ~(0x3fULL << 57)); + addr = (addr & ~(LAM_U57_MASK)); break; } free((void *)addr); @@ -632,6 +655,72 @@ static int fork_test(struct testcases *test) return ret; } +static int handle_execve(struct testcases *test) +{ + int ret, child_ret; + int lam = test->lam; + pid_t pid; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + ret = 1; + } else if (pid == 0) { + char path[PATH_MAX]; + + /* Set LAM mode in parent process */ + if (set_lam(lam) != 0) + return 1; + + /* Get current binary's path and the binary was run by execve */ + if (readlink("/proc/self/exe", path, PATH_MAX) <= 0) + exit(-1); + + /* run binary to get LAM mode and return to parent process */ + if (execlp(path, path, "-t 0x0", NULL) < 0) { + perror("error on exec"); + exit(-1); + } + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + if (ret != LAM_NONE) + return 1; + } + + return 0; +} + +static int handle_inheritance(struct testcases *test) +{ + int ret, child_ret; + int lam = test->lam; + pid_t pid; + + /* Set LAM mode in parent process */ + if (set_lam(lam) != 0) + return 1; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + return 1; + } else if (pid == 0) { + /* Set LAM mode in parent process */ + int child_lam = get_lam(); + + exit(child_lam); + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + + if (lam != ret) + return 1; + } + + return 0; +} + static void run_test(struct testcases *test, int count) { int i, ret = 0; @@ -740,11 +829,26 @@ static struct testcases mmap_cases[] = { }, }; +static struct testcases inheritance_cases[] = { + { + .expected = 0, + .lam = LAM_U57_BITS, + .test_func = handle_inheritance, + .msg = "FORK: LAM_U57, child process should get LAM mode same as parent\n", + }, + { + .expected = 0, + .lam = LAM_U57_BITS, + .test_func = handle_execve, + .msg = "EXECVE: LAM_U57, child process should get disabled LAM mode\n", + }, +}; + static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring.\n"); + printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring; 0x20:inherit;\n"); printf("\t-h: help\n"); } @@ -764,7 +868,7 @@ int main(int argc, char **argv) switch (c) { case 't': tests = strtoul(optarg, NULL, 16); - if (!(tests & TEST_MASK)) { + if (tests && !(tests & TEST_MASK)) { ksft_print_msg("Invalid argument!\n"); return -1; } @@ -778,6 +882,16 @@ int main(int argc, char **argv) } } + /* + * When tests is 0, it is not a real test case; + * the option used by test case(execve) to check the lam mode in + * process generated by execve, the process read back lam mode and + * check with lam mode in parent process. + */ + if (!tests) + return (get_lam()); + + /* Run test cases */ if (tests & FUNC_MALLOC) run_test(malloc_cases, ARRAY_SIZE(malloc_cases)); @@ -793,6 +907,9 @@ int main(int argc, char **argv) if (tests & FUNC_URING) run_test(uring_cases, ARRAY_SIZE(uring_cases)); + if (tests & FUNC_INHERITE) + run_test(inheritance_cases, ARRAY_SIZE(inheritance_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Sun Mar 12 11:26:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45C2AC6FA99 for ; Sun, 12 Mar 2023 11:27:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C9BD48E0013; Sun, 12 Mar 2023 07:27:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C237D8E0014; Sun, 12 Mar 2023 07:27:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 916FB8E0013; Sun, 12 Mar 2023 07:27:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8205F8E000E for ; Sun, 12 Mar 2023 07:27:13 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 61D0940967 for ; Sun, 12 Mar 2023 11:27:13 +0000 (UTC) X-FDA: 80560019946.16.F7D1368 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf23.hostedemail.com (Postfix) with ESMTP id 1FBED14000B for ; Sun, 12 Mar 2023 11:27:10 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=HYxkFnDy; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf23.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620431; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=69xvf/JgEEAl7d6NLv74/ovYVtBOKl6f6shs5hVJmMY=; b=lOWHFOzsb8gD303g5vr0RWjXw0xG5DxVtCpvrvKLxappJkNQD8vuNV5J9R+c4LWb7e0kmW geSmzoRhBE6RvTQi402Wa625fTT6QjZLPFIOh5faJOCUh7Otx2IXPswGoN1f5EGDs2PnaF kwlZVueCBZEP7UKdY0Gqn0QTZ9tvULU= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=HYxkFnDy; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf23.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620431; a=rsa-sha256; cv=none; b=UDbGCQOQGzdOuSzamP+4GQau7W3c0ZwMjZ3fQe3DqtnrVhLbUQGGUL2XZF39vzc73Kn8Z2 T4bjQAhpcKkNr2i0QA104MBOVXptly6EvEXgyXPI7+LA1Ps4oFSydxFnaxMoEC1B4k+S+M U1wD5G1clZuDrWXmzqh+qQOLEwXY1ms= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620431; x=1710156431; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=vqjgq0aq/HN6otn18FAaGaVeHvXItWk0KcS4vZFq+qs=; b=HYxkFnDyJl7ExV0+fFqfKnp0x5gjWQUOj61j5ixDXbavkjmM+tM7dyma ACRSW7jFklZ+ScaDZIwdKThEmvYyPcJbTi5RTVzADOrPhwws+hUjOHE1U 9yy6zHpg4+7DC0YrbI0GPJG5fdvNB4U10YvA1boHkKUSvvC289B1yJ3ox GLzxi5rAOduWYiloxxd9LSx69IvzAHgkzvAsUnkX+gXgXjhc0yLzknJXf LxGux8hHY0GGUXBf+vXFcaKVh59yZ+1lhlJtMK2XvBxOVF225GEAsnehB do7kB4mKG+31FGuQUQn34mBDNVrkGF9teCeZyN8rfqZvivPD/8nuFkFZB A==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="399577666" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="399577666" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="671580757" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="671580757" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:38 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 01B1710D7BE; Sun, 12 Mar 2023 14:26:20 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv16 16/17] selftests/x86/lam: Add ARCH_FORCE_TAGGED_SVA test cases for linear-address masking Date: Sun, 12 Mar 2023 14:26:11 +0300 Message-Id: <20230312112612.31869-17-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1FBED14000B X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: urujzgkgckkqotiiegkros6hpdb9nt1u X-HE-Tag: 1678620430-474388 X-HE-Meta: U2FsdGVkX1/1rYtTeblP8Mg4E5JPMHL4lbOgvb267Vrg9y10hOgMM1tqeppwbu5OPp23aLG055rzOGLWfeCPuMb6BJFJvX6VsCHrg2I/b+0OQgps6GpygmQtfBDgEPH5z3IbooreHyBVhr7n7a9ax/ikt4MNMOnk6tYzgYRptlvVhnuGKy/Ajd7j27lR30AIoAUJfi4nRfiOn/mquatyROuz8HxXM0OFm6ISuRow1H+kCZwhksmicc1vjXla37mJRR386ubaCqCGDzEiERqsuYzELbnGT1M0DevSBjd2g4KF8ngfIaHC9mefJnmOaV1cz/hAICPM7L+ZNsBPQEjnmFxVTvAGApRn49E5+De3dDhKfexp+v7XTCo12Os8kLi38tk4C+u6v1sV/ePfIg0T2PH9/MXeVVJaxQacfQklMYPtHrnUheC5kwSbv9A2hQ90Ux48b8CvWyesRY3/vk2AfywmZ0aJPNnCJ91piR8OAUlBsgDok/MopE319cQ6v7Hz/SfBwGmtdmLQNiblbykSENx7nHm4VTujp7EoU0XX51FTVKoRIEeVyVasbjr796bzIVr8E1tlve8M33UkBk/TpcfAbtsOtmuztaEZL+qajMjBVnMtYmKQI7HcpNlKAp+kFLi5gJXkG7BoypCgb5LcMP1rH1fdRkLXKYTsKG3R5Jd61shXWvCEtGgdyCpkqXvnnjXLwK/6PHoxN81RHHCyyTL0nBvJI/QKkyudnmMIN8uQVPdH7cuD+2vO1chDyoFjE5gi+DRaL+o7fwq6R72Nt5tOEoEYxvKGSHm0UK8ZdpcEaH/ZqVfiiZnN5bgVEWitLDhyoMpMIMwHhVPxqu1xRdyNmDHJt9Yc+ebEg0z6SBpQOuCtkHcZ/WBT0kax5yZ8KoY/U9t8rDoJrp4L6392pRhcEBhBzoQwutMMi9pLNcpFCZHvR7BXK9YEuK9GHCEhsngJSgtpNXUeU6BPwGf rSycElbl AMf0eMNfmR2nIQWQIKNJmf7WH/1S4yLtP1+YvfV9EoS7xe0GNrLzHDyDC5SfQh2GqDE+yd8izcvqWORF2hnlXKbE0czT01w+ca5AmBMmDfsnl4PR2XPtoZZBJUTWEP3e7MtxDixxlU3k3utmrWCon10m9e2429r0/sb06MWBLoxdLyg2Djzua5Vl5CKZwZWFRwrMKkmDDNUc9FpBl1EZvSMDIW6+4fVeCleI6Ql7AgM7J+mF73i/SJYp0o/ClJbPPjsZHb54DdsZjdX3YLoyTkse5TkaSAyJBSrC9JOBAKPnKHgrKO+eanzhIdxOwaUBT87Pf441ApHEj9sZeqwmZtydZjQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Weihong Zhang By default do not allow to enable both LAM and use SVA in the same process. The new ARCH_FORCE_TAGGED_SVA arch_prctl() overrides the limitation. Add new test cases for the new arch_prctl: Before using ARCH_FORCE_TAGGED_SVA, should not allow to enable LAM/SVA coexisting. the test cases should be negative. The test depands on idxd driver and iommu. before test, need add "intel_iommu=on,sm_on" in kernel command line and insmod idxd driver. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) --- tools/testing/selftests/x86/lam.c | 237 +++++++++++++++++++++++++++++- 1 file changed, 235 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index ebabd4333b7d..a8c91829b616 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -30,6 +30,7 @@ #define ARCH_GET_UNTAG_MASK 0x4001 #define ARCH_ENABLE_TAGGED_ADDR 0x4002 #define ARCH_GET_MAX_TAG_BITS 0x4003 +#define ARCH_FORCE_TAGGED_SVA 0x4004 /* Specified test function bits */ #define FUNC_MALLOC 0x1 @@ -38,8 +39,9 @@ #define FUNC_SYSCALL 0x8 #define FUNC_URING 0x10 #define FUNC_INHERITE 0x20 +#define FUNC_PASID 0x40 -#define TEST_MASK 0x3f +#define TEST_MASK 0x7f #define LOW_ADDR (0x1UL << 30) #define HIGH_ADDR (0x3UL << 48) @@ -55,11 +57,19 @@ #define URING_QUEUE_SZ 1 #define URING_BLOCK_SZ 2048 +/* Pasid test define */ +#define LAM_CMD_BIT 0x1 +#define PAS_CMD_BIT 0x2 +#define SVA_CMD_BIT 0x4 + +#define PAS_CMD(cmd1, cmd2, cmd3) (((cmd3) << 8) | ((cmd2) << 4) | ((cmd1) << 0)) + struct testcases { unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ unsigned long lam; uint64_t addr; + uint64_t cmd; int (*test_func)(struct testcases *test); const char *msg; }; @@ -556,7 +566,7 @@ int do_uring(unsigned long lam) struct file_io *fi; struct stat st; int ret = 1; - char path[PATH_MAX]; + char path[PATH_MAX] = {0}; /* get current process path */ if (readlink("/proc/self/exe", path, PATH_MAX) <= 0) @@ -852,6 +862,226 @@ static void cmd_help(void) printf("\t-h: help\n"); } +/* Check for file existence */ +uint8_t file_Exists(const char *fileName) +{ + struct stat buffer; + + uint8_t ret = (stat(fileName, &buffer) == 0); + + return ret; +} + +/* Sysfs idxd files */ +const char *dsa_configs[] = { + "echo 1 > /sys/bus/dsa/devices/dsa0/wq0.1/group_id", + "echo shared > /sys/bus/dsa/devices/dsa0/wq0.1/mode", + "echo 10 > /sys/bus/dsa/devices/dsa0/wq0.1/priority", + "echo 16 > /sys/bus/dsa/devices/dsa0/wq0.1/size", + "echo 15 > /sys/bus/dsa/devices/dsa0/wq0.1/threshold", + "echo user > /sys/bus/dsa/devices/dsa0/wq0.1/type", + "echo MyApp1 > /sys/bus/dsa/devices/dsa0/wq0.1/name", + "echo 1 > /sys/bus/dsa/devices/dsa0/engine0.1/group_id", + "echo dsa0 > /sys/bus/dsa/drivers/idxd/bind", + /* bind files and devices, generated a device file in /dev */ + "echo wq0.1 > /sys/bus/dsa/drivers/user/bind", +}; + +/* DSA device file */ +const char *dsaDeviceFile = "/dev/dsa/wq0.1"; +/* file for io*/ +const char *dsaPasidEnable = "/sys/bus/dsa/devices/dsa0/pasid_enabled"; + +/* + * DSA depends on kernel cmdline "intel_iommu=on,sm_on" + * return pasid_enabled (0: disable 1:enable) + */ +int Check_DSA_Kernel_Setting(void) +{ + char command[256] = ""; + char buf[256] = ""; + char *ptr; + int rv = -1; + + snprintf(command, sizeof(command) - 1, "cat %s", dsaPasidEnable); + + FILE *cmd = popen(command, "r"); + + if (cmd) { + while (fgets(buf, sizeof(buf) - 1, cmd) != NULL); + + pclose(cmd); + rv = strtol(buf, &ptr, 16); + } + + return rv; +} + +/* + * Config DSA's sysfs files as shared DSA's WQ. + * Generated a device file /dev/dsa/wq0.1 + * Return: 0 OK; 1 Failed; 3 Skip(SVA disabled). + */ +int Dsa_Init_Sysfs(void) +{ + uint len = ARRAY_SIZE(dsa_configs); + const char **p = dsa_configs; + + if (file_Exists(dsaDeviceFile) == 1) + return 0; + + /* check the idxd driver */ + if (file_Exists(dsaPasidEnable) != 1) { + printf("Please make sure idxd driver was loaded\n"); + return 3; + } + + /* Check SVA feature */ + if (Check_DSA_Kernel_Setting() != 1) { + printf("Please enable SVA.(Add intel_iommu=on,sm_on in kernel cmdline)\n"); + return 3; + } + + /* Check the idxd device file on /dev/dsa/ */ + for (int i = 0; i < len; i++) { + if (system(p[i])) + return 1; + } + + /* After config, /dev/dsa/wq0.1 should be generated */ + return (file_Exists(dsaDeviceFile) != 1); +} + +/* + * Open DSA device file, triger API: iommu_sva_alloc_pasid + */ +void *allocate_dsa_pasid(void) +{ + int fd; + void *wq; + + fd = open(dsaDeviceFile, O_RDWR); + if (fd < 0) { + perror("open"); + return MAP_FAILED; + } + + wq = mmap(NULL, 0x1000, PROT_WRITE, + MAP_SHARED | MAP_POPULATE, fd, 0); + if (wq == MAP_FAILED) + perror("mmap"); + + return wq; +} + +int set_force_svm(void) +{ + int ret = 0; + + ret = syscall(SYS_arch_prctl, ARCH_FORCE_TAGGED_SVA); + + return ret; +} + +int handle_pasid(struct testcases *test) +{ + uint tmp = test->cmd; + uint runed = 0x0; + int ret = 0; + void *wq = NULL; + + ret = Dsa_Init_Sysfs(); + if (ret != 0) + return ret; + + for (int i = 0; i < 3; i++) { + int err = 0; + + if (tmp & 0x1) { + /* run set lam mode*/ + if ((runed & 0x1) == 0) { + err = set_lam(LAM_U57_BITS); + runed = runed | 0x1; + } else + err = 1; + } else if (tmp & 0x4) { + /* run force svm */ + if ((runed & 0x4) == 0) { + err = set_force_svm(); + runed = runed | 0x4; + } else + err = 1; + } else if (tmp & 0x2) { + /* run allocate pasid */ + if ((runed & 0x2) == 0) { + runed = runed | 0x2; + wq = allocate_dsa_pasid(); + if (wq == MAP_FAILED) + err = 1; + } else + err = 1; + } + + ret = ret + err; + if (ret > 0) + break; + + tmp = tmp >> 4; + } + + if (wq != MAP_FAILED && wq != NULL) + if (munmap(wq, 0x1000)) + printf("munmap failed %d\n", errno); + + if (runed != 0x7) + ret = 1; + + return (ret != 0); +} + +/* + * Pasid test depends on idxd and SVA, kernel should enable iommu and sm. + * command line(intel_iommu=on,sm_on) + */ +static struct testcases pasid_cases[] = { + { + .expected = 1, + .cmd = PAS_CMD(LAM_CMD_BIT, PAS_CMD_BIT, SVA_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: [Negative] Execute LAM, PASID, SVA in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(LAM_CMD_BIT, SVA_CMD_BIT, PAS_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute LAM, SVA, PASID in sequence\n", + }, + { + .expected = 1, + .cmd = PAS_CMD(PAS_CMD_BIT, LAM_CMD_BIT, SVA_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: [Negative] Execute PASID, LAM, SVA in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(PAS_CMD_BIT, SVA_CMD_BIT, LAM_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute PASID, SVA, LAM in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(SVA_CMD_BIT, LAM_CMD_BIT, PAS_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute SVA, LAM, PASID in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(SVA_CMD_BIT, PAS_CMD_BIT, LAM_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute SVA, PASID, LAM in sequence\n", + }, +}; + int main(int argc, char **argv) { int c = 0; @@ -910,6 +1140,9 @@ int main(int argc, char **argv) if (tests & FUNC_INHERITE) run_test(inheritance_cases, ARRAY_SIZE(inheritance_cases)); + if (tests & FUNC_PASID) + run_test(pasid_cases, ARRAY_SIZE(pasid_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Sun Mar 12 11:26:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171310 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5B87C6FD1F for ; Sun, 12 Mar 2023 11:27:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 853C68E0012; Sun, 12 Mar 2023 07:27:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 804DA8E000E; Sun, 12 Mar 2023 07:27:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 67ED38E0012; Sun, 12 Mar 2023 07:27:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4F44B8E000E for ; Sun, 12 Mar 2023 07:27:11 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 18BA9AAD18 for ; Sun, 12 Mar 2023 11:27:11 +0000 (UTC) X-FDA: 80560019862.26.B92612C Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf05.hostedemail.com (Postfix) with ESMTP id EF90310000D for ; Sun, 12 Mar 2023 11:27:08 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=OEvsBgI2; spf=none (imf05.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620429; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OZUQfYUiRxDD4yajdfiAuzpHqN59m/bqizk0pX5fkM8=; b=O1fFvoOhTdbgTFwI+Z4Y4+w9RjHyzfegMf+5CQbIVrl3PdTgBDsk5qlAciECY5gRlJBu8i 00AAXPh2EDtPqg9sOLdPpI+jvCJAFtr1yJCS171oKeXxD0wFK8vpAVYC9o5U521aZzvCaZ hGsu6VcWsJk8a+DkSxUzjRq1lPAN7kU= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=OEvsBgI2; spf=none (imf05.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620429; a=rsa-sha256; cv=none; b=ZRX6jhzk3UdY3jbkSpbyqwLPZJicnNQYwjY6YzriU8NHO0U9KmxYIklJSyxjsagRwEo4bO 74psi1WfzCTBsGnaiU0sprtEOupTN0lCrIm8qQZxPpjyo9DBmEcxwITvWT/Mz1EZ/r0okR 01KEhQdCVMAWeYjADawKlP4sReRSDPc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620429; x=1710156429; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=oTU4sxTXYKtAIVLwPW8AIWDYdktjaL52nQvpCjw3OeY=; b=OEvsBgI2NNCX85in7SUlmH/UlqaRkrFjcjESMM/JVqHz2sFAo004V8RA tmtr6TsFWyMuiN1p/yJTshGUlxdGcejETgfC452O2Hzmj0y9dj8mONaAt 6oKPGiLIKyTMDKiud0e6onLoCaS5tTFdDCRonYIHYVt0pp1zFj9zmzfDB 6P8QSFuzLEwv34Ny6FhXWtDTlYaLbzS+hW1lOf7FQuM4YfB2FEPX98IqH 6Prplb041En3QWAxeqb2V6XFutzu1ogEQ3z1JAyDK/4s/nvRxInA4vAhT Y9jqgvtzyPZSwB8bslc2tjaGUhQxTYATiOa+QiTOEBS4zqftAW3PZOGf1 A==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="399577658" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="399577658" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="671580755" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="671580755" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:38 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 0BE2910D7BF; Sun, 12 Mar 2023 14:26:20 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv16 17/17] selftests/x86/lam: Add test cases for LAM vs thread creation Date: Sun, 12 Mar 2023 14:26:12 +0300 Message-Id: <20230312112612.31869-18-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: bfmuwowt8npqri9pfzdafs91qgmjq8uf X-Rspam-User: X-Rspamd-Queue-Id: EF90310000D X-Rspamd-Server: rspam06 X-HE-Tag: 1678620428-543920 X-HE-Meta: U2FsdGVkX19aMLaoEM6PR//XKJScuCOHFjP5WfuIrpp0a7zRZI2EWhlPHWvtPAk4OQgv/jk7HRpM5TMmBIr5Tgfsa+9Z3F6ePmOrQF3nsE8D2SwW4BVudVoOonO6ziOTx1W5CcH4/4neFnRSsuh9ghu9ehvfy2K1h4LPa4Q6WUU+ipv3KBZ7nIZ5EQeRyym7/Y5NHSIMzNPMe2S7Yf3Fqlyk63zpFEGCMiHTm8Q73qL5FS0TPXZlB8K3pS8pX7ezLsApVA4aYSG/gLKGD8+eCgIIfFcc0t2gyXQLqKgrZurTNZdqyHRYrRnQweUN+y/HyVH9NQrEPBfXWJ7BFPqLsKvot5Y/QFULulW8LX9p8myJb6xNo5EXAlFEmaChbl0pM3eaaeYhYqiS78+3lYBfbvSc59F4t/VYX01JQGFsvhxEv3sIXPP9PDeXFtvzbvpbgCxEzZcwQWpTifaY+KGrkagod8rn3uirAhZ7OBQppFvuTOPQ9rzSTUsq+5q+NQMtsW00t48QVUepWASWljs9zICtWTorZS9BaAdnWDBxQyZm4ObHbN9/A+2vQdg05TxbQ4pjSyqQAeaqxPE67QX5PdyRvt6IR79JXkhWkWrmQYSRk/ZvYdxrHBUIOsE6pMcrDrlXlfVY2Aw17AdtvMa4ZlxPbjSnZFlFzHveVD1QXD8xw6I+A7yCiW0XeUv/D4szqjU0EcDdjaZprEt8/HqruZ6F7JH/ZmJT9iy0Qi393p3ashjW6RNjZacpw/4SLxxfSvqX77YucV3oa8cyU9C4db/RoOKp9K6XNXhf+CLsRtgyMRawDiM2hTlBLADqGlOeKt4xS8oFLvp5Q6KeRlNHpssDvqpyq+e1yDk7UYby/vti7Yl7Ub7Obj1pj7a9s9/AFsBFlV9C3zOLzEYTosXETF6wukFYVRr2TBLe6TPm8dXa7PwZ7ST3b37v6ovbqaheST0nJgwcBKaiKqsPIYi v3+MUhZA JljkJRrPTV3AKUx5zUm+uQOiJKh1uKFYg5001XDEw2CGQ/bS2p3R60W4IxUvGbW20PT4zF9A2FlKa3oPujwMy2f39JjhevAh1hUjfiPEtNJu93QHcNumN7iZpNFkPJIPjQDDTIvz2REodHLKSXdrHGnzMGZ4kNw0eGM05T8AS0BiAf6J9luWQ35ltUN+CniXDXu2RmXOMjxmgvhl/YSGhu62FNrbxVOhleXEa2ZuYQbVXkd089KIqGgyzeOrPNCpAncUt35s9EmDQHFUFcwy+KiS7/SRlOE8KyHDd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: LAM enabling is only allowed when the process has single thread. LAM mode is inherited into child thread. Trying to enable LAM after spawning a thread has to fail. Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) --- tools/testing/selftests/x86/lam.c | 92 +++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index a8c91829b616..eb0e46905bf9 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -1,4 +1,5 @@ // SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE #include #include #include @@ -12,6 +13,7 @@ #include #include #include +#include #include #include @@ -50,6 +52,8 @@ #define PAGE_SIZE (4 << 10) +#define STACK_SIZE 65536 + #define barrier() ({ \ __asm__ __volatile__("" : : : "memory"); \ }) @@ -731,6 +735,75 @@ static int handle_inheritance(struct testcases *test) return 0; } +static int thread_fn_get_lam(void *arg) +{ + return get_lam(); +} + +static int thread_fn_set_lam(void *arg) +{ + struct testcases *test = arg; + + return set_lam(test->lam); +} + +static int handle_thread(struct testcases *test) +{ + char stack[STACK_SIZE]; + int ret, child_ret; + int lam = 0; + pid_t pid; + + /* Set LAM mode in parent process */ + if (!test->later) { + lam = test->lam; + if (set_lam(lam) != 0) + return 1; + } + + pid = clone(thread_fn_get_lam, stack + STACK_SIZE, + SIGCHLD | CLONE_FILES | CLONE_FS | CLONE_VM, NULL); + if (pid < 0) { + perror("Clone failed."); + return 1; + } + + waitpid(pid, &child_ret, 0); + ret = WEXITSTATUS(child_ret); + + if (lam != ret) + return 1; + + if (test->later) { + if (set_lam(test->lam) != 0) + return 1; + } + + return 0; +} + +static int handle_thread_enable(struct testcases *test) +{ + char stack[STACK_SIZE]; + int ret, child_ret; + int lam = test->lam; + pid_t pid; + + pid = clone(thread_fn_set_lam, stack + STACK_SIZE, + SIGCHLD | CLONE_FILES | CLONE_FS | CLONE_VM, test); + if (pid < 0) { + perror("Clone failed."); + return 1; + } + + waitpid(pid, &child_ret, 0); + ret = WEXITSTATUS(child_ret); + + if (lam != ret) + return 1; + + return 0; +} static void run_test(struct testcases *test, int count) { int i, ret = 0; @@ -846,6 +919,25 @@ static struct testcases inheritance_cases[] = { .test_func = handle_inheritance, .msg = "FORK: LAM_U57, child process should get LAM mode same as parent\n", }, + { + .expected = 0, + .lam = LAM_U57_BITS, + .test_func = handle_thread, + .msg = "THREAD: LAM_U57, child thread should get LAM mode same as parent\n", + }, + { + .expected = 1, + .lam = LAM_U57_BITS, + .test_func = handle_thread_enable, + .msg = "THREAD: [NEGATIVE] Enable LAM in child.\n", + }, + { + .expected = 1, + .later = 1, + .lam = LAM_U57_BITS, + .test_func = handle_thread, + .msg = "THREAD: [NEGATIVE] Enable LAM in parent after thread created.\n", + }, { .expected = 0, .lam = LAM_U57_BITS,