From patchwork Sun Mar 12 11:25:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DA15C6FA99 for ; Sun, 12 Mar 2023 11:26:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 93E908E0003; Sun, 12 Mar 2023 07:26:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8EF138E0001; Sun, 12 Mar 2023 07:26:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B62C8E0003; Sun, 12 Mar 2023 07:26:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6D81C8E0001 for ; Sun, 12 Mar 2023 07:26:35 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 2CE511A0625 for ; Sun, 12 Mar 2023 11:26:35 +0000 (UTC) X-FDA: 80560018350.22.55DE0CA Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf10.hostedemail.com (Postfix) with ESMTP id 2A3E1C0011 for ; Sun, 12 Mar 2023 11:26:33 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ABfu5+qt; spf=none (imf10.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620393; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=01M6Oui0yYdZJlxoRo2JZXCZONIUA8npeELpcrndGKU=; b=VX+fVNsKdqv1mrz8JNyN/yaxz9OtYsakyk3y3i3ScX2wh6CsZ4H3648+a5q6RSlwzIOro7 0pwPkmj17v9RKugx+HqrNNBgtg52Y0uQaUQfFfwjg35ihM/1I1Jf2Keu7qyCn9fUUbDlPj 1b6Fv9nhVNrHJIfsGe5J6IqL1UFJ8QQ= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ABfu5+qt; spf=none (imf10.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620393; a=rsa-sha256; cv=none; b=MtEfog35+SxCzMUwe8e7qYJIGSuvb/zZiVe+QRK7V+bm30X8eWG9SUezY1r7tiT5jJeKNm u7Ntyb0DSq5aX3gPdi0sQD2kErK8EFaFxkA8pbjY8U+ft6vohpwerWXoczoExUcfxJBYl7 Opb+895waP5iM3vP/fhqeoZQmBQRF1o= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620393; x=1710156393; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+ya4j8+hmXPftNA55cuj2jSsHwe1N+LRr5YAFmUiVLI=; b=ABfu5+qt32bypEYBQ66mKO2cPK6t5cPN0s7IsVl+T4uDM7YGXQXkAYXw NEo0zJrxJeP65lj/b4NmgwafbA3pKAkXvGHfmJLGTkWbywuuyRYQUe0UZ HrFg7igcqQ1i35d3x+VzUNv5IIZv+/JjiEIdZ8cnyCnzQYkVJs8AovKDC D56/ttCEHgXbX8G62HePJJZspMhj0VMU2mv8JwSOTwsZHOKMIpblWp8us bLrdM/SQiLOrLubvbVOlwIwRfX6KuBx3jVvNsLIFAoBMVhoQp25O2CWX8 xiTqE2FI9cgqKvVcWebxroPXFRLngcoiRgGXEsjr0BsGZBIvrXKYLYLNk Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="339349927" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="339349927" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="852438013" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="852438013" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:22 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 5C6AA10D7AF; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv16 01/17] x86/mm: Rework address range check in get_user() and put_user() Date: Sun, 12 Mar 2023 14:25:56 +0300 Message-Id: <20230312112612.31869-2-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 7swz34wnsio5bxqdr5ojg6wkcoz7yt81 X-Rspamd-Queue-Id: 2A3E1C0011 X-HE-Tag: 1678620393-388518 X-HE-Meta: U2FsdGVkX1+qtl/q+N/tUyisBnZZX4RwBodq1rgAsuuDiPcebHenVQxBq3gcWFPgw3qenPIokqahNlT30zVntLxLywnFKGH83SWeTiikEMt+L0DKyVWZvayZsbVVI2DMysWXKrrN6WsIi0rsCaYGj5/t9URR8RggS4kZCkKG1i3e6q18bIl5ao22+tSqbve7nzMO0xJGNoh9p+8KaWJaStqvvv2USZA664mDmMm5sKGDnBVHIO6q7mV5tGnPFff1KEmsfL7lZKMDNncniHJj5RXg9oGyowDAvxEAMRu7Wsg+d62fqsY8eP7Mwv1W6OBidpo1GEJmredFZX4p9n324QLN5ej9D/7HSfRYZAn+Ijn1zve8is7FD++Fx9kvkgqzbCaAF1TInIkrCd9e3FqQ0oO98Bwn1YxTVkVq4zHgH+B/gqnh4VacxHxK4WshukjolXtLSwd8T6tSZs3L7r0/RJtgKpOoO3ovKMASFxGN1bg5ZOyK3Ke7EnlzCAbDzL1tIp99nqr2F4D6oE7wWMABd6wrB7fhHfNrTIdxYDdv57pXQUEmzVdLuzfBqot6HN6ndyCCyA92KvjHUo6DZhjxGdDHUnxeP6mBcOYW4thrDIhWC4/btCtmc8omZ4Bnp8Bord5zNFBSQdodY0lfx3DsfXdvcCfh1WxuJONNzp4dO5VfHLZSWtuX09B4RVkC+Zadw8IiYoNwYteWu4NbCTrmtHnMCxuFcJYYMVyiCno4AM+mXtmMu2ATQyFtkvPNxj4ruy2U63hQOcCy3mHzn3PzzDrZZE2wTOIcNoUIJW/80qI7C58mE+5q1/KbFZzp6DVkGhLBpKrqkBM8uSErvHSvakygLIXpAtjQpXLZfVJCgiQAfyAMyUpFMx6d1Br6PiRB6C2Nf2zAVBDu9s/ojMiWyjA+82yQYzrUTVG3N1ChmGD1xFFHrUItBpm4QgJ3yzFln+voI3ySXum8f5nycBB nGLSTzm5 91BvSBeQyGEbxt3uAj72pivSdvtcHNXB8h6b4wz5wwooVtwlX27lfWva+zl38yFXwz3Bx+OrYEok8BqnYvOpGTvhAOjnB4eHV6PFf16zVviIpu80Ppzw1w/KTgS9aCoGQ8qHaqfPflDR2vUU6vOgxcrMpFJjIOA4bllPb5jUFJMCkW1O3StlYKRV18p/9xI8pXKRUPRTKiclGglCIHRLZcC4mP4Q/wS2EQ08ykr+hIo6jwbNme9POoxKHZP4jI/QJG2+yv/dLz0v0LkAJbXQhT2PucLesyVVW3fG3evmY1VSB4fDAkNPQ5dZ8/XLAiRjKL4zIciWjjxoIdxiwY/jMMLADcSqXAxcZi06K/xVrFrX0qJsGCjKp35CrKDRg7dVEngr2WKJsAo911BY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The functions get_user() and put_user() check that the target address range resides in the user space portion of the virtual address space. In order to perform this check, the functions compare the end of the range against TASK_SIZE_MAX. For kernels compiled with CONFIG_X86_5LEVEL, this process requires some additional trickery using ALTERNATIVE, as TASK_SIZE_MAX depends on the paging mode in use. Linus suggested that this check could be simplified for 64-bit kernels. It is sufficient to check bit 63 of the address to ensure that the range belongs to user space. Additionally, the use of branches can be avoided by setting the target address to all ones if bit 63 is set. There's no need to check the end of the access range as there's huge gap between end of userspace range and start of the kernel range. The gap consists of canonical hole and unused ranges on both kernel and userspace sides. If an address with bit 63 set is passed down, it will trigger a #GP exception. _ASM_EXTABLE_UA() complains about this. Replace it with plain _ASM_EXTABLE() as it is expected behaviour now. The updated get_user() and put_user() checks are also compatible with Linear Address Masking, which allows user space to encode metadata in the upper bits of pointers and eliminates the need to untag the address before handling it. Signed-off-by: Kirill A. Shutemov Suggested-by: Linus Torvalds Acked-by: Peter Zijlstra (Intel) --- arch/x86/lib/getuser.S | 83 ++++++++++++++++-------------------------- arch/x86/lib/putuser.S | 54 ++++++++++++--------------- 2 files changed, 55 insertions(+), 82 deletions(-) diff --git a/arch/x86/lib/getuser.S b/arch/x86/lib/getuser.S index b70d98d79a9d..b64a2bd1a1ef 100644 --- a/arch/x86/lib/getuser.S +++ b/arch/x86/lib/getuser.S @@ -37,22 +37,22 @@ #define ASM_BARRIER_NOSPEC ALTERNATIVE "", "lfence", X86_FEATURE_LFENCE_RDTSC -#ifdef CONFIG_X86_5LEVEL -#define LOAD_TASK_SIZE_MINUS_N(n) \ - ALTERNATIVE __stringify(mov $((1 << 47) - 4096 - (n)),%rdx), \ - __stringify(mov $((1 << 56) - 4096 - (n)),%rdx), X86_FEATURE_LA57 -#else -#define LOAD_TASK_SIZE_MINUS_N(n) \ - mov $(TASK_SIZE_MAX - (n)),%_ASM_DX -#endif +.macro check_range size:req +.if IS_ENABLED(CONFIG_X86_64) + mov %rax, %rdx + sar $63, %rdx + or %rdx, %rax +.else + cmp $TASK_SIZE_MAX-\size+1, %eax + jae .Lbad_get_user + sbb %edx, %edx /* array_index_mask_nospec() */ + and %edx, %eax +.endif +.endm .text SYM_FUNC_START(__get_user_1) - LOAD_TASK_SIZE_MINUS_N(0) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=1 ASM_STAC 1: movzbl (%_ASM_AX),%edx xor %eax,%eax @@ -62,11 +62,7 @@ SYM_FUNC_END(__get_user_1) EXPORT_SYMBOL(__get_user_1) SYM_FUNC_START(__get_user_2) - LOAD_TASK_SIZE_MINUS_N(1) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=2 ASM_STAC 2: movzwl (%_ASM_AX),%edx xor %eax,%eax @@ -76,11 +72,7 @@ SYM_FUNC_END(__get_user_2) EXPORT_SYMBOL(__get_user_2) SYM_FUNC_START(__get_user_4) - LOAD_TASK_SIZE_MINUS_N(3) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=4 ASM_STAC 3: movl (%_ASM_AX),%edx xor %eax,%eax @@ -90,30 +82,17 @@ SYM_FUNC_END(__get_user_4) EXPORT_SYMBOL(__get_user_4) SYM_FUNC_START(__get_user_8) -#ifdef CONFIG_X86_64 - LOAD_TASK_SIZE_MINUS_N(7) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=8 ASM_STAC +#ifdef CONFIG_X86_64 4: movq (%_ASM_AX),%rdx - xor %eax,%eax - ASM_CLAC - RET #else - LOAD_TASK_SIZE_MINUS_N(7) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user_8 - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX - ASM_STAC 4: movl (%_ASM_AX),%edx 5: movl 4(%_ASM_AX),%ecx +#endif xor %eax,%eax ASM_CLAC RET -#endif SYM_FUNC_END(__get_user_8) EXPORT_SYMBOL(__get_user_8) @@ -166,7 +145,7 @@ EXPORT_SYMBOL(__get_user_nocheck_8) SYM_CODE_START_LOCAL(.Lbad_get_user_clac) ASM_CLAC -bad_get_user: +.Lbad_get_user: xor %edx,%edx mov $(-EFAULT),%_ASM_AX RET @@ -184,23 +163,23 @@ SYM_CODE_END(.Lbad_get_user_8_clac) #endif /* get_user */ - _ASM_EXTABLE_UA(1b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(2b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(3b, .Lbad_get_user_clac) + _ASM_EXTABLE(1b, .Lbad_get_user_clac) + _ASM_EXTABLE(2b, .Lbad_get_user_clac) + _ASM_EXTABLE(3b, .Lbad_get_user_clac) #ifdef CONFIG_X86_64 - _ASM_EXTABLE_UA(4b, .Lbad_get_user_clac) + _ASM_EXTABLE(4b, .Lbad_get_user_clac) #else - _ASM_EXTABLE_UA(4b, .Lbad_get_user_8_clac) - _ASM_EXTABLE_UA(5b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(4b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(5b, .Lbad_get_user_8_clac) #endif /* __get_user */ - _ASM_EXTABLE_UA(6b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(7b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(8b, .Lbad_get_user_clac) + _ASM_EXTABLE(6b, .Lbad_get_user_clac) + _ASM_EXTABLE(7b, .Lbad_get_user_clac) + _ASM_EXTABLE(8b, .Lbad_get_user_clac) #ifdef CONFIG_X86_64 - _ASM_EXTABLE_UA(9b, .Lbad_get_user_clac) + _ASM_EXTABLE(9b, .Lbad_get_user_clac) #else - _ASM_EXTABLE_UA(9b, .Lbad_get_user_8_clac) - _ASM_EXTABLE_UA(10b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(9b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(10b, .Lbad_get_user_8_clac) #endif diff --git a/arch/x86/lib/putuser.S b/arch/x86/lib/putuser.S index 32125224fcca..3062d09a776d 100644 --- a/arch/x86/lib/putuser.S +++ b/arch/x86/lib/putuser.S @@ -33,20 +33,20 @@ * as they get called from within inline assembly. */ -#ifdef CONFIG_X86_5LEVEL -#define LOAD_TASK_SIZE_MINUS_N(n) \ - ALTERNATIVE __stringify(mov $((1 << 47) - 4096 - (n)),%rbx), \ - __stringify(mov $((1 << 56) - 4096 - (n)),%rbx), X86_FEATURE_LA57 -#else -#define LOAD_TASK_SIZE_MINUS_N(n) \ - mov $(TASK_SIZE_MAX - (n)),%_ASM_BX -#endif +.macro check_range size:req +.if IS_ENABLED(CONFIG_X86_64) + mov %rcx, %rbx + sar $63, %rbx + or %rbx, %rcx +.else + cmp $TASK_SIZE_MAX-\size+1, %ecx + jae .Lbad_put_user +.endif +.endm .text SYM_FUNC_START(__put_user_1) - LOAD_TASK_SIZE_MINUS_N(0) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=1 ASM_STAC 1: movb %al,(%_ASM_CX) xor %ecx,%ecx @@ -66,9 +66,7 @@ SYM_FUNC_END(__put_user_nocheck_1) EXPORT_SYMBOL(__put_user_nocheck_1) SYM_FUNC_START(__put_user_2) - LOAD_TASK_SIZE_MINUS_N(1) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=2 ASM_STAC 3: movw %ax,(%_ASM_CX) xor %ecx,%ecx @@ -88,9 +86,7 @@ SYM_FUNC_END(__put_user_nocheck_2) EXPORT_SYMBOL(__put_user_nocheck_2) SYM_FUNC_START(__put_user_4) - LOAD_TASK_SIZE_MINUS_N(3) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=4 ASM_STAC 5: movl %eax,(%_ASM_CX) xor %ecx,%ecx @@ -110,9 +106,7 @@ SYM_FUNC_END(__put_user_nocheck_4) EXPORT_SYMBOL(__put_user_nocheck_4) SYM_FUNC_START(__put_user_8) - LOAD_TASK_SIZE_MINUS_N(7) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=8 ASM_STAC 7: mov %_ASM_AX,(%_ASM_CX) #ifdef CONFIG_X86_32 @@ -144,15 +138,15 @@ SYM_CODE_START_LOCAL(.Lbad_put_user_clac) RET SYM_CODE_END(.Lbad_put_user_clac) - _ASM_EXTABLE_UA(1b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(2b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(3b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(4b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(5b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(6b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(7b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(9b, .Lbad_put_user_clac) + _ASM_EXTABLE(1b, .Lbad_put_user_clac) + _ASM_EXTABLE(2b, .Lbad_put_user_clac) + _ASM_EXTABLE(3b, .Lbad_put_user_clac) + _ASM_EXTABLE(4b, .Lbad_put_user_clac) + _ASM_EXTABLE(5b, .Lbad_put_user_clac) + _ASM_EXTABLE(6b, .Lbad_put_user_clac) + _ASM_EXTABLE(7b, .Lbad_put_user_clac) + _ASM_EXTABLE(9b, .Lbad_put_user_clac) #ifdef CONFIG_X86_32 - _ASM_EXTABLE_UA(8b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(10b, .Lbad_put_user_clac) + _ASM_EXTABLE(8b, .Lbad_put_user_clac) + _ASM_EXTABLE(10b, .Lbad_put_user_clac) #endif