From patchwork Thu Jun 18 00:32:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 11611087 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 93786618 for ; Thu, 18 Jun 2020 00:48:26 +0000 (UTC) Received: from ml01.01.org (ml01.01.org [198.145.21.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7B3F42168B for ; Thu, 18 Jun 2020 00:48:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7B3F42168B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-nvdimm-bounces@lists.01.org Received: from ml01.vlan13.01.org (localhost [IPv6:::1]) by ml01.01.org (Postfix) with ESMTP id 75E6110FC4F6F; Wed, 17 Jun 2020 17:48:26 -0700 (PDT) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=134.134.136.126; helo=mga18.intel.com; envelope-from=dan.j.williams@intel.com; receiver= Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id F14C410FC4AD3 for ; Wed, 17 Jun 2020 17:48:23 -0700 (PDT) IronPort-SDR: MgEMENpsk8YdZv1i5NFF/5XGG/6SrYxQRVDwf+IaY4iUHeWK2HrT2vM499tbC7iwcR2w7PnPOw 9BIFVsJan+BA== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Jun 2020 17:48:23 -0700 IronPort-SDR: 9hITMPyRDO6quo7Aov6bps1Ol/u/+yqUPQgzGOqlna9P49JkU57+a5ou2f38uC88BMudXC+hZJ 1wC+m3TMzVqA== X-IronPort-AV: E=Sophos;i="5.73,524,1583222400"; d="scan'208";a="261874129" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.16]) by fmsmga007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Jun 2020 17:48:23 -0700 Subject: [PATCH v6 2/2] x86/copy_mc: Introduce copy_mc_generic() From: Dan Williams To: akpm@linux-foundation.org, tglx@linutronix.de, mingo@redhat.com Date: Wed, 17 Jun 2020 17:32:09 -0700 Message-ID: <159244032939.1107636.15958715466456757179.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <159244031857.1107636.5054974045023236143.stgit@dwillia2-desk3.amr.corp.intel.com> References: <159244031857.1107636.5054974045023236143.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.18-3-g996c MIME-Version: 1.0 Message-ID-Hash: BBQJ2UVSIKXE7YR24MYKFQTOFLN4SFJM X-Message-ID-Hash: BBQJ2UVSIKXE7YR24MYKFQTOFLN4SFJM X-MailFrom: dan.j.williams@intel.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header CC: x86@kernel.org, stable@vger.kernel.org, Borislav Petkov , Tony Luck , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Linus Torvalds , Erwin Tsaur , linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org X-Mailman-Version: 3.1.1 Precedence: list List-Id: "Linux-nvdimm developer list." Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: The original copy_mc_fragile() implementation had negative performance implications since it did not use the fast-string instruction sequence to perform copies. For this reason copy_mc_to_kernel() fell back to plain memcpy() to preserve performance on platform that did not indicate the capability to recover from machine check exceptions. However, that capability detection was not architectural and now that some platforms can recover from fast-string consumption of memory errors the memcpy() fallback now causes these more capable platforms to fail. Introduce copy_mc_generic() as the fast default implementation of copy_mc_to_kernel() and finalize the transition of copy_mc_fragile() to be a platform quirk to indicate 'fragility'. With this in place copy_mc_to_kernel() is fast and recovery-ready by default regardless of hardware capability. Thanks to Vivek for identifying that copy_user_generic() is not suitable as the copy_mc_to_user() backend since the #MC handler explicitly checks ex_has_fault_handler(). Cc: x86@kernel.org Cc: Cc: Ingo Molnar Cc: Borislav Petkov Cc: Vivek Goyal Cc: Tony Luck Cc: "H. Peter Anvin" Cc: Andy Lutomirski Cc: Thomas Gleixner Cc: Peter Zijlstra Cc: Linus Torvalds Reported-by: Erwin Tsaur Tested-by: Erwin Tsaur Fixes: 92b0729c34ca ("x86/mm, x86/mce: Add memcpy_mcsafe()") Signed-off-by: Dan Williams --- arch/x86/include/asm/uaccess.h | 3 +++ arch/x86/lib/copy_mc.c | 12 +++++------- arch/x86/lib/copy_mc_64.S | 40 ++++++++++++++++++++++++++++++++++++++++ tools/objtool/check.c | 1 + 4 files changed, 49 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 4b2082b61e3e..b038eda58958 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -464,6 +464,9 @@ copy_mc_to_user(void *to, const void *from, unsigned len); unsigned long __must_check copy_mc_fragile(void *dst, const void *src, unsigned cnt); + +unsigned long __must_check +copy_mc_generic(void *dst, const void *src, unsigned cnt); #else static inline void enable_copy_mc_fragile(void) { diff --git a/arch/x86/lib/copy_mc.c b/arch/x86/lib/copy_mc.c index cdb8f5dc403d..9e6fac1ab72e 100644 --- a/arch/x86/lib/copy_mc.c +++ b/arch/x86/lib/copy_mc.c @@ -23,7 +23,7 @@ void enable_copy_mc_fragile(void) * * Call into the 'fragile' version on systems that have trouble * actually do machine check recovery. Everyone else can just - * use memcpy(). + * use copy_mc_generic(). * * Return 0 for success, or number of bytes not copied if there was an * exception. @@ -33,8 +33,7 @@ copy_mc_to_kernel(void *dst, const void *src, unsigned cnt) { if (static_branch_unlikely(©_mc_fragile_key)) return copy_mc_fragile(dst, src, cnt); - memcpy(dst, src, cnt); - return 0; + return copy_mc_generic(dst, src, cnt); } EXPORT_SYMBOL_GPL(copy_mc_to_kernel); @@ -56,11 +55,10 @@ copy_mc_to_user(void *to, const void *from, unsigned len) { unsigned long ret; - if (!static_branch_unlikely(©_mc_fragile_key)) - return copy_user_generic(to, from, len); - __uaccess_begin(); - ret = copy_mc_fragile(to, from, len); + if (static_branch_unlikely(©_mc_fragile_key)) + ret = copy_mc_fragile(to, from, len); + ret = copy_mc_generic(to, from, len); __uaccess_end(); return ret; } diff --git a/arch/x86/lib/copy_mc_64.S b/arch/x86/lib/copy_mc_64.S index 35a67c50890b..a08e7a4d9e28 100644 --- a/arch/x86/lib/copy_mc_64.S +++ b/arch/x86/lib/copy_mc_64.S @@ -2,7 +2,9 @@ /* Copyright(c) 2016-2020 Intel Corporation. All rights reserved. */ #include +#include #include +#include #include #include @@ -122,4 +124,42 @@ EXPORT_SYMBOL_GPL(copy_mc_fragile) _ASM_EXTABLE(.L_write_leading_bytes, .E_leading_bytes) _ASM_EXTABLE(.L_write_words, .E_write_words) _ASM_EXTABLE(.L_write_trailing_bytes, .E_trailing_bytes) + +/* + * copy_mc_generic - memory copy with exception handling + * + * Fast string copy + fault / exception handling. If the CPU does + * support machine check exception recovery, but does not support + * recovering from fast-string exceptions then this CPU needs to be + * added to the copy_mc_fragile_key set of quirks. Otherwise, absent any + * machine check recovery support this version should be no slower than + * standard memcpy. + */ +SYM_FUNC_START(copy_mc_generic) + ALTERNATIVE "jmp copy_mc_fragile", "", X86_FEATURE_ERMS + movq %rdi, %rax + movq %rdx, %rcx +.L_copy: + rep movsb + /* Copy successful. Return zero */ + xorl %eax, %eax + ret +SYM_FUNC_END(copy_mc_generic) +EXPORT_SYMBOL_GPL(copy_mc_generic) + + .section .fixup, "ax" +.E_copy: + /* + * On fault %rcx is updated such that the copy instruction could + * optionally be restarted at the fault position, i.e. it + * contains 'bytes remaining'. A non-zero return indicates error + * to copy_mc_generic() users, or indicate short transfers to + * user-copy routines. + */ + movq %rcx, %rax + ret + + .previous + + _ASM_EXTABLE_FAULT(.L_copy, .E_copy) #endif diff --git a/tools/objtool/check.c b/tools/objtool/check.c index f07adee5df9e..54c7467fa2fe 100644 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -547,6 +547,7 @@ static const char *uaccess_safe_builtin[] = { "__ubsan_handle_shift_out_of_bounds", /* misc */ "csum_partial_copy_generic", + "copy_mc_generic", "copy_mc_fragile", "copy_mc_fragile_handle_tail", "ftrace_likely_update", /* CONFIG_TRACE_BRANCH_PROFILING */