From patchwork Thu Sep 26 13:53:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chao Gao X-Patchwork-Id: 11162783 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0E25217D4 for ; Thu, 26 Sep 2019 13:51:09 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E5814207FF for ; Thu, 26 Sep 2019 13:51:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E5814207FF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iDU9N-00021R-Cb; Thu, 26 Sep 2019 13:49:41 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iDU9M-00021K-9x for xen-devel@lists.xenproject.org; Thu, 26 Sep 2019 13:49:40 +0000 X-Inumbo-ID: 7a879e70-e064-11e9-8628-bc764e2007e4 Received: from mga04.intel.com (unknown [192.55.52.120]) by localhost (Halon) with ESMTPS id 7a879e70-e064-11e9-8628-bc764e2007e4; Thu, 26 Sep 2019 13:49:37 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 26 Sep 2019 06:49:36 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,552,1559545200"; d="scan'208";a="189126011" Received: from gao-cwp.sh.intel.com ([10.239.159.26]) by fmsmga008.fm.intel.com with ESMTP; 26 Sep 2019 06:49:34 -0700 From: Chao Gao To: xen-devel@lists.xenproject.org Date: Thu, 26 Sep 2019 21:53:28 +0800 Message-Id: <1569506015-26938-1-git-send-email-chao.gao@intel.com> X-Mailer: git-send-email 1.9.1 MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v11 0/7] improve late microcode loading X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Sergey Dyasli , Ashok Raj , Wei Liu , Andrew Cooper , Jan Beulich , Boris Ostrovsky , Chao Gao , Brian Woods , Suravee Suthikulpanit , =?utf-8?q?Roger_Pau_?= =?utf-8?q?Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Changes in v11: - reject late ucode loading if any core is parked - correct the usage of microcode_mutex in microcode_update_cpu() - extend 'ucode' boot option to enable/disable ucode loading in NMI - drop the last two patches of v10 (about BDF90 and wbinvd, I haven't get an answer to opens yet). - other minor changes are described in each patch's change log Regarding changes to AMD side, I didn't do any test for them due to lack of hardware. The intention of this series is to make the late microcode loading more reliable by rendezvousing all cpus in stop_machine context. This idea comes from Ashok. I am porting his linux patch to Xen (see patch 4 more details). This series includes below changes: 1. Patch 1-3: cleanup and preparation for synchronizing ucode loading 2. Patch 4: synchronize late microcode loading 3. Patch 5: support parallel microcodes update on different cores 4. Patch 6: rendezvous CPUs in NMI handler and load ucode 5. Patch 7: reject late ucode loading if any core is parked Currently, late microcode loading does a lot of things including parsing microcode blob, checking the signature/revision and performing update. Putting all of them into stop_machine context is a bad idea because of complexity (one issue I observed is memory allocation triggered one assertion in stop_machine context). To simplify the load process, parsing microcode is moved out of the load process. Remaining parts of load process is put to stop_machine context. Previous change log: Major changes in version 10: - add back the patch to call wbinvd() conditionally - add a patch to disable late loading due to BDF90 - rendezvous CPUs in NMI handler and load ucode. But provide an option to disable this behavior. - avoid the call of self_nmi() on the control thread because it may trigger the unknown_nmi_error() in do_nmi(). - ensure ->start_update is called during system resuming from suspension Changes in version 9: - add Jan's Reviewed-by - rendevzous threads in NMI handler to disable NMI. Note that NMI can be served as usual on threads that are chosen to initiate ucode loading on each core. - avoid unnecessary memory allocation or copy when creating a microcode patch (patch 12) - rework patch 1 to avoid microcode_update_match() being used to compare two arbitrary updates. - call .end_update in early loading path. Changes in version 8: - block #NMI handling during microcode loading (Patch 16) - Don't assume that all CPUs in the system have loaded a same ucode. So when parsing a blob, we attempt to save a patch as long as it matches with current cpu signature regardless of the revision of the patch. And also for loading, we only require the patch to be loaded isn't old than the cached one. - store an update after the first successful loading on a CPU - remove the patch that calls wbinvd() unconditionally before microcode loading. It is under internal discussion. - divide two big patches into several patches to improve readability. Changes in version 7: - cache one microcode update rather than a list of it. Assuming that all CPUs (including those will be plugged in later) in the system have the same signature, one update matches with one CPU should match with others. Thus, one update is enough for microcode updating during CPU hot-plug and resuming. - To handle load failure, microcode update is cached after it is applied to avoid a broken update overriding a validated one. Unvalidated microcode updates are passed by arguments rather than another global variable, where this series slightly differs from Roger's suggestion in: https://lists.xen.org/archives/html/xen-devel/2019-03/msg00776.html - incorporate Sergey's patch (patch 10) to fix a bug: we maintain a variable to reflect current microcode revision. But in some cases, this variable isn't initialized during system boot time, which results in falsely reporting that processor is susceptible to some known vulnerabilities. - fix issues reported by Sergey: https://lists.xenproject.org/archives/html/xen-devel/2019-03/msg00901.html - Responses to Sergey/Roger/Wei/Ashok's other comments. Major changes in version 6: - run wbinvd before updating microcode (patch 10) - add an userspace tool for late microcode update (patch 1) - scale time to wait by the number of remaining CPUs to respond - remove 'cpu' parameters from some related callbacks and functins - save an ucode patch only if its supported CPU is allowed to mix with current cpu. Changes in version 5: - support parallel microcode updates for all cores (see patch 8) - Address Roger's comments on the last version. Chao Gao (7): microcode: split out apply_microcode() from cpu_request_microcode() microcode: unify ucode loading during system bootup and resuming microcode: reduce memory allocation and copy when creating a patch x86/microcode: Synchronize late microcode loading microcode: remove microcode_update_lock microcode: rendezvous CPUs in NMI handler and load ucode microcode: reject late ucode loading if any core is parked docs/misc/xen-command-line.pandoc | 6 +- xen/arch/x86/acpi/power.c | 2 +- xen/arch/x86/microcode.c | 638 ++++++++++++++++++++++++++++++++------ xen/arch/x86/microcode_amd.c | 132 ++++---- xen/arch/x86/microcode_intel.c | 99 +++--- xen/arch/x86/smpboot.c | 5 +- xen/arch/x86/traps.c | 6 +- xen/include/asm-x86/microcode.h | 5 +- xen/include/asm-x86/nmi.h | 3 + xen/include/asm-x86/processor.h | 5 +- 10 files changed, 658 insertions(+), 243 deletions(-)