From patchwork Mon Dec 27 08:15:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chao Gao X-Patchwork-Id: 12699606 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25DC1C433F5 for ; Mon, 27 Dec 2021 08:17:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235583AbhL0IRV (ORCPT ); Mon, 27 Dec 2021 03:17:21 -0500 Received: from mga09.intel.com ([134.134.136.24]:12161 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235566AbhL0IRU (ORCPT ); Mon, 27 Dec 2021 03:17:20 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1640593040; x=1672129040; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=gD95MZJuVqXHFQir/dDl6qfyNZH1jlwhc3VMpECHxWI=; b=lqVxGsIa9tEiGAvJ0Jze1xLVUdkQREgazuZmz33RVf67cCj+cV6092yi sJXDTXlXpJK8ecL5Xnq9DE5IdWOkhRCkOsWoM88RPSuMjJJ7/I5pw0oOO NHYp18OYEJ1uj26b8T9RiK8KYviQrProBU40+6KUukajZRmK956eBjx3w rEdWQ+2WzCA2g6ClmaN0ol8HeVMRT2DT5lmy+/dB9sKUUvJ/FG7V/ZTky lYabIQaAQ9EMaj6LqGtqvf6kBfWRnglGrbmhihHEmIz7MN6JRSXCWTo9C OKvqTdADr7EI21c9UDSSvVR2UsHdk93NxgouZva/ec/0yiqkZEL6qhtUg Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10209"; a="240985281" X-IronPort-AV: E=Sophos;i="5.88,237,1635231600"; d="scan'208";a="240985281" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Dec 2021 00:17:17 -0800 X-IronPort-AV: E=Sophos;i="5.88,239,1635231600"; d="scan'208";a="523208000" Received: from unknown (HELO hyperv-sh4.sh.intel.com) ([10.239.48.22]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Dec 2021 00:17:12 -0800 From: Chao Gao To: kvm@vger.kernel.org, pbonzini@redhat.com, kevin.tian@intel.com, tglx@linutronix.de Cc: Chao Gao , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , linux-kernel@vger.kernel.org Subject: [PATCH 1/6] KVM: x86: Move check_processor_compatibility from init ops to runtime ops Date: Mon, 27 Dec 2021 16:15:07 +0800 Message-Id: <20211227081515.2088920-2-chao.gao@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211227081515.2088920-1-chao.gao@intel.com> References: <20211227081515.2088920-1-chao.gao@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org so that KVM can do compatibility checks on hotplugged CPUs. Drop __init from check_processor_compatibility() and its callees. Signed-off-by: Chao Gao --- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/svm/svm.c | 4 ++-- arch/x86/kvm/vmx/evmcs.c | 2 +- arch/x86/kvm/vmx/evmcs.h | 2 +- arch/x86/kvm/vmx/vmx.c | 12 ++++++------ arch/x86/kvm/x86.c | 2 +- 6 files changed, 12 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 5d97f4adc1cb..0d5dc148090c 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1313,6 +1313,7 @@ static inline u16 kvm_lapic_irq_dest_mode(bool dest_mode_logical) struct kvm_x86_ops { const char *name; + int (*check_processor_compatibility)(void); int (*hardware_enable)(void); void (*hardware_disable)(void); void (*hardware_unsetup)(void); @@ -1513,7 +1514,6 @@ struct kvm_x86_nested_ops { struct kvm_x86_init_ops { int (*cpu_has_kvm_support)(void); int (*disabled_by_bios)(void); - int (*check_processor_compatibility)(void); int (*hardware_setup)(void); struct kvm_x86_ops *runtime_ops; diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 6cb38044a860..74c41a261633 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -3825,7 +3825,7 @@ svm_patch_hypercall(struct kvm_vcpu *vcpu, unsigned char *hypercall) hypercall[2] = 0xd9; } -static int __init svm_check_processor_compat(void) +static int svm_check_processor_compat(void) { return 0; } @@ -4374,6 +4374,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .name = "kvm_amd", .hardware_unsetup = svm_hardware_teardown, + .check_processor_compatibility = svm_check_processor_compat, .hardware_enable = svm_hardware_enable, .hardware_disable = svm_hardware_disable, .cpu_has_accelerated_tpr = svm_cpu_has_accelerated_tpr, @@ -4743,7 +4744,6 @@ static struct kvm_x86_init_ops svm_init_ops __initdata = { .cpu_has_kvm_support = has_svm, .disabled_by_bios = is_disabled, .hardware_setup = svm_hardware_setup, - .check_processor_compatibility = svm_check_processor_compat, .runtime_ops = &svm_x86_ops, }; diff --git a/arch/x86/kvm/vmx/evmcs.c b/arch/x86/kvm/vmx/evmcs.c index ba6f99f584ac..50f923e9917e 100644 --- a/arch/x86/kvm/vmx/evmcs.c +++ b/arch/x86/kvm/vmx/evmcs.c @@ -296,7 +296,7 @@ const struct evmcs_field vmcs_field_to_evmcs_1[] = { }; const unsigned int nr_evmcs_1_fields = ARRAY_SIZE(vmcs_field_to_evmcs_1); -__init void evmcs_sanitize_exec_ctrls(struct vmcs_config *vmcs_conf) +void evmcs_sanitize_exec_ctrls(struct vmcs_config *vmcs_conf) { vmcs_conf->pin_based_exec_ctrl &= ~EVMCS1_UNSUPPORTED_PINCTRL; vmcs_conf->cpu_based_2nd_exec_ctrl &= ~EVMCS1_UNSUPPORTED_2NDEXEC; diff --git a/arch/x86/kvm/vmx/evmcs.h b/arch/x86/kvm/vmx/evmcs.h index 16731d2cf231..17a7c956396b 100644 --- a/arch/x86/kvm/vmx/evmcs.h +++ b/arch/x86/kvm/vmx/evmcs.h @@ -181,7 +181,7 @@ static inline void evmcs_load(u64 phys_addr) vp_ap->enlighten_vmentry = 1; } -__init void evmcs_sanitize_exec_ctrls(struct vmcs_config *vmcs_conf); +void evmcs_sanitize_exec_ctrls(struct vmcs_config *vmcs_conf); #else /* !IS_ENABLED(CONFIG_HYPERV) */ static __always_inline void evmcs_write64(unsigned long field, u64 value) {} static inline void evmcs_write32(unsigned long field, u32 value) {} diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index fe06b02994e6..de617a7bd966 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -2373,8 +2373,8 @@ static bool cpu_has_sgx(void) return cpuid_eax(0) >= 0x12 && (cpuid_eax(0x12) & BIT(0)); } -static __init int adjust_vmx_controls(u32 ctl_min, u32 ctl_opt, - u32 msr, u32 *result) +static int adjust_vmx_controls(u32 ctl_min, u32 ctl_opt, + u32 msr, u32 *result) { u32 vmx_msr_low, vmx_msr_high; u32 ctl = ctl_min | ctl_opt; @@ -2392,8 +2392,8 @@ static __init int adjust_vmx_controls(u32 ctl_min, u32 ctl_opt, return 0; } -static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf, - struct vmx_capability *vmx_cap) +static int setup_vmcs_config(struct vmcs_config *vmcs_conf, + struct vmx_capability *vmx_cap) { u32 vmx_msr_low, vmx_msr_high; u32 min, opt, min2, opt2; @@ -6990,7 +6990,7 @@ static int vmx_vm_init(struct kvm *kvm) return 0; } -static int __init vmx_check_processor_compat(void) +static int vmx_check_processor_compat(void) { struct vmcs_config vmcs_conf; struct vmx_capability vmx_cap; @@ -7578,6 +7578,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .hardware_unsetup = hardware_unsetup, + .check_processor_compatibility = vmx_check_processor_compat, .hardware_enable = hardware_enable, .hardware_disable = hardware_disable, .cpu_has_accelerated_tpr = report_flexpriority, @@ -7911,7 +7912,6 @@ static __init int hardware_setup(void) static struct kvm_x86_init_ops vmx_init_ops __initdata = { .cpu_has_kvm_support = cpu_has_kvm_support, .disabled_by_bios = vmx_disabled_by_bios, - .check_processor_compatibility = vmx_check_processor_compat, .hardware_setup = hardware_setup, .runtime_ops = &vmx_x86_ops, diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index c194a8cbd25f..6411417b6871 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -11391,7 +11391,7 @@ int kvm_arch_check_processor_compat(void *opaque) __cr4_reserved_bits(cpu_has, &boot_cpu_data)) return -EIO; - return ops->check_processor_compatibility(); + return ops->runtime_ops->check_processor_compatibility(); } bool kvm_vcpu_is_reset_bsp(struct kvm_vcpu *vcpu) From patchwork Mon Dec 27 08:15:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chao Gao X-Patchwork-Id: 12699607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8733EC433FE for ; Mon, 27 Dec 2021 08:17:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235607AbhL0IRX (ORCPT ); Mon, 27 Dec 2021 03:17:23 -0500 Received: from mga07.intel.com ([134.134.136.100]:41613 "EHLO mga07.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235569AbhL0IRX (ORCPT ); Mon, 27 Dec 2021 03:17:23 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1640593043; x=1672129043; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6gwEeKLEMbxvYxSECEpoif+ET9leBqLI2zBcqc8zWyA=; b=oJ7V9kHKpyZHBYPhCAYfY6GuvkD52qNhQD8932g1UdSV7PZySMb1rHuZ xbsVL1w7SeJmoq0GbH8iBpA8d25V+cWs1HkeE2aq4oMYCVY/sgyKZZG1/ lZaTplBun6vJYwOczhZskpszImN9rYmdSjexOGEvfvZoVpSlmiMG6O19/ uRxeXbv3If9iJetPLGVwZtOIsQmYRT7XXXKc8l0YFuMD6urdXliuhjLpm 6R7AyDdDx9hy6doCGIR38478WU1bJ68/rV0KwqKPoWGzst6GGfJO/9zz5 J7Gox8XZsFFFvV2Pe1aKDNpUsND88NJb7U0Rlw+yyO1imyBH3+BDh8a0L g==; X-IronPort-AV: E=McAfee;i="6200,9189,10209"; a="304541632" X-IronPort-AV: E=Sophos;i="5.88,239,1635231600"; d="scan'208";a="304541632" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Dec 2021 00:17:22 -0800 X-IronPort-AV: E=Sophos;i="5.88,239,1635231600"; d="scan'208";a="523208037" Received: from unknown (HELO hyperv-sh4.sh.intel.com) ([10.239.48.22]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Dec 2021 00:17:17 -0800 From: Chao Gao To: kvm@vger.kernel.org, pbonzini@redhat.com, kevin.tian@intel.com, tglx@linutronix.de Cc: Chao Gao , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , linux-kernel@vger.kernel.org Subject: [PATCH 2/6] KVM: x86: Use kvm_x86_ops in kvm_arch_check_processor_compat Date: Mon, 27 Dec 2021 16:15:08 +0800 Message-Id: <20211227081515.2088920-3-chao.gao@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211227081515.2088920-1-chao.gao@intel.com> References: <20211227081515.2088920-1-chao.gao@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org check_processor_compatibility() is a "runtime" ops now. Use kvm_x86_ops directly such that kvm_arch_check_processor_compat can be called at runtime. Signed-off-by: Chao Gao --- arch/x86/kvm/x86.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 6411417b6871..770b68e72391 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -11383,7 +11383,6 @@ void kvm_arch_hardware_unsetup(void) int kvm_arch_check_processor_compat(void *opaque) { struct cpuinfo_x86 *c = &cpu_data(smp_processor_id()); - struct kvm_x86_init_ops *ops = opaque; WARN_ON(!irqs_disabled()); @@ -11391,7 +11390,7 @@ int kvm_arch_check_processor_compat(void *opaque) __cr4_reserved_bits(cpu_has, &boot_cpu_data)) return -EIO; - return ops->runtime_ops->check_processor_compatibility(); + return kvm_x86_ops.check_processor_compatibility(); } bool kvm_vcpu_is_reset_bsp(struct kvm_vcpu *vcpu) From patchwork Mon Dec 27 08:15:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chao Gao X-Patchwork-Id: 12699608 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB974C43219 for ; Mon, 27 Dec 2021 08:17:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235612AbhL0IRj (ORCPT ); Mon, 27 Dec 2021 03:17:39 -0500 Received: from mga12.intel.com ([192.55.52.136]:31632 "EHLO mga12.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235599AbhL0IRi (ORCPT ); Mon, 27 Dec 2021 03:17:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1640593058; x=1672129058; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=WH0GwBORKgKcWk1etFfvqiimI30RAnrK7W2Gs9Zs1Ik=; b=GklNvfLGOLR4Bu4ooSQKIy747fGZs/+OlZNHiaZBADYWfvb24impz/hf xrFGmb/FPF1CkMZIwz8qEnf1Rm6Z/cqYAMKYAv0tYkcGxNJPEOCCBcFIP /W2ATFzYQL4pEv0ZHbwQ/fw/n6t19s4gcA//E57U8bbRRoat14Bn8qVvt CDSMk3+GGPfZXeUDIEVfLSzMWnUbCq5fE9luujV4XBEg042iEVV98gqYQ Bzu5RVGloG6IRRF/VRJdN6hsTa4xxkxAnAXByPIk3gLUa3LNO2nKScg94 mVc2ZOjZTb6kJcv3lWDSzhI2MYmzOAb3cB58fGSOpoM5QZT+QnKhvEmDT Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10209"; a="221182085" X-IronPort-AV: E=Sophos;i="5.88,239,1635231600"; d="scan'208";a="221182085" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Dec 2021 00:17:38 -0800 X-IronPort-AV: E=Sophos;i="5.88,239,1635231600"; d="scan'208";a="523208109" Received: from unknown (HELO hyperv-sh4.sh.intel.com) ([10.239.48.22]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Dec 2021 00:17:24 -0800 From: Chao Gao To: kvm@vger.kernel.org, pbonzini@redhat.com, kevin.tian@intel.com, tglx@linutronix.de Cc: Chao Gao , Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , Catalin Marinas , Will Deacon , Huacai Chen , Aleksandar Markovic , Thomas Bogendoerfer , Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , Anup Patel , Atish Patra , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , David Hildenbrand , Claudio Imbrenda , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , "Maciej S. Szmigiero" , Fabiano Rosas , Ravi Bangoria , =?utf-8?q?C=C3=A9dric_Le_Goate?= =?utf-8?q?r?= , Bharata B Rao , Nicholas Piggin , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org Subject: [PATCH 3/6] KVM: Remove opaque from kvm_arch_check_processor_compat Date: Mon, 27 Dec 2021 16:15:09 +0800 Message-Id: <20211227081515.2088920-4-chao.gao@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211227081515.2088920-1-chao.gao@intel.com> References: <20211227081515.2088920-1-chao.gao@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org No arch implementation uses this opaque now. Signed-off-by: Chao Gao --- arch/arm64/kvm/arm.c | 2 +- arch/mips/kvm/mips.c | 2 +- arch/powerpc/kvm/powerpc.c | 2 +- arch/riscv/kvm/main.c | 2 +- arch/s390/kvm/kvm-s390.c | 2 +- arch/x86/kvm/x86.c | 2 +- include/linux/kvm_host.h | 2 +- virt/kvm/kvm_main.c | 14 ++------------ 8 files changed, 9 insertions(+), 19 deletions(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 14106a7c75b5..c196f005a2d3 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -73,7 +73,7 @@ int kvm_arch_hardware_setup(void *opaque) return 0; } -int kvm_arch_check_processor_compat(void *opaque) +int kvm_arch_check_processor_compat(void) { return 0; } diff --git a/arch/mips/kvm/mips.c b/arch/mips/kvm/mips.c index e59cb6246f76..c5dc4fe53bfc 100644 --- a/arch/mips/kvm/mips.c +++ b/arch/mips/kvm/mips.c @@ -140,7 +140,7 @@ int kvm_arch_hardware_setup(void *opaque) return 0; } -int kvm_arch_check_processor_compat(void *opaque) +int kvm_arch_check_processor_compat(void) { return 0; } diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c index 2ad0ccd202d5..30c817f3fa0c 100644 --- a/arch/powerpc/kvm/powerpc.c +++ b/arch/powerpc/kvm/powerpc.c @@ -423,7 +423,7 @@ int kvm_arch_hardware_setup(void *opaque) return 0; } -int kvm_arch_check_processor_compat(void *opaque) +int kvm_arch_check_processor_compat(void) { return kvmppc_core_check_processor_compat(); } diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c index 421ecf4e6360..3b0b104e443f 100644 --- a/arch/riscv/kvm/main.c +++ b/arch/riscv/kvm/main.c @@ -20,7 +20,7 @@ long kvm_arch_dev_ioctl(struct file *filp, return -EINVAL; } -int kvm_arch_check_processor_compat(void *opaque) +int kvm_arch_check_processor_compat(void) { return 0; } diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c index 9c6d45d0d345..99c70d881cb6 100644 --- a/arch/s390/kvm/kvm-s390.c +++ b/arch/s390/kvm/kvm-s390.c @@ -252,7 +252,7 @@ int kvm_arch_hardware_enable(void) return 0; } -int kvm_arch_check_processor_compat(void *opaque) +int kvm_arch_check_processor_compat(void) { return 0; } diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 770b68e72391..aa09c8792134 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -11380,7 +11380,7 @@ void kvm_arch_hardware_unsetup(void) static_call(kvm_x86_hardware_unsetup)(); } -int kvm_arch_check_processor_compat(void *opaque) +int kvm_arch_check_processor_compat(void) { struct cpuinfo_x86 *c = &cpu_data(smp_processor_id()); diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 05862176df6a..630ffd2289c6 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1307,7 +1307,7 @@ int kvm_arch_hardware_enable(void); void kvm_arch_hardware_disable(void); int kvm_arch_hardware_setup(void *opaque); void kvm_arch_hardware_unsetup(void); -int kvm_arch_check_processor_compat(void *opaque); +int kvm_arch_check_processor_compat(void); int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu); bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu); int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu); diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 168d0ab93c88..4e1e7770e984 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -5600,22 +5600,14 @@ struct kvm_vcpu * __percpu *kvm_get_running_vcpus(void) return &kvm_running_vcpu; } -struct kvm_cpu_compat_check { - void *opaque; - int *ret; -}; - static void check_processor_compat(void *data) { - struct kvm_cpu_compat_check *c = data; - - *c->ret = kvm_arch_check_processor_compat(c->opaque); + *(int *)data = kvm_arch_check_processor_compat(); } int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align, struct module *module) { - struct kvm_cpu_compat_check c; int r; int cpu; @@ -5643,10 +5635,8 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align, if (r < 0) goto out_free_1; - c.ret = &r; - c.opaque = opaque; for_each_online_cpu(cpu) { - smp_call_function_single(cpu, check_processor_compat, &c, 1); + smp_call_function_single(cpu, check_processor_compat, &r, 1); if (r < 0) goto out_free_2; } From patchwork Mon Dec 27 08:15:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chao Gao X-Patchwork-Id: 12699609 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DA5BC433F5 for ; Mon, 27 Dec 2021 08:17:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235624AbhL0IRp (ORCPT ); Mon, 27 Dec 2021 03:17:45 -0500 Received: from mga12.intel.com ([192.55.52.136]:31646 "EHLO mga12.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235622AbhL0IRo (ORCPT ); Mon, 27 Dec 2021 03:17:44 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1640593064; x=1672129064; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xYCm7W/GNTT8b6GR0Dno3E5gHGMz2WSp3wyshjf+qAM=; b=L5vYuEQNkjK00IQdp9VysCY7ENQmYxmlXieqKcm9d0VLNshlP4ehHKK5 5mp/3elMOwdTju1B5DanlY941Y3viMfaaHI7jsoqDAHcfR78VRcExhW0q /zBgDloQ80FAessN//IDWF/wxFLmFKvvBC9UmiysM5Tr6YNx3/BvirbaZ to7xbwMUfjQ/UyHGNZrqtdPaoDVQsAW1/R2dYGQC2pEkkq7ubXg1EaMXw 90hSc5k/ByCMqlhKPPPQ7aL3wEYAtx/Ww35M8HMqn4ja5CN93C2r3iSxh MVmnBjeX6R3SWxZpBkDT23hJq19QiW29sM4T1/c1MdxZMD4e2irIxWaxs A==; X-IronPort-AV: E=McAfee;i="6200,9189,10209"; a="221182097" X-IronPort-AV: E=Sophos;i="5.88,239,1635231600"; d="scan'208";a="221182097" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Dec 2021 00:17:44 -0800 X-IronPort-AV: E=Sophos;i="5.88,239,1635231600"; d="scan'208";a="523208147" Received: from unknown (HELO hyperv-sh4.sh.intel.com) ([10.239.48.22]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Dec 2021 00:17:39 -0800 From: Chao Gao To: kvm@vger.kernel.org, pbonzini@redhat.com, kevin.tian@intel.com, tglx@linutronix.de Cc: Chao Gao , John Garry , Will Deacon , Joerg Roedel , Andrew Morton , Daniel Lezcano , Thomas Richter , Shaokun Zhang , Huang Ying , Jens Axboe , linux-kernel@vger.kernel.org Subject: [PATCH 4/6] KVM: Rename and move CPUHP_AP_KVM_STARTING to ONLINE section Date: Mon, 27 Dec 2021 16:15:10 +0800 Message-Id: <20211227081515.2088920-5-chao.gao@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211227081515.2088920-1-chao.gao@intel.com> References: <20211227081515.2088920-1-chao.gao@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The CPU STARTING section doesn't allow callbacks to fail. Move KVM's hotplug callback to ONLINE section so that it can abort onlining a CPU in certain cases to avoid potentially breaking VMs running on existing CPUs. For example, when kvm fails to enable hardware virtualization on the hotplugged CPU. Place KVM's hotplug state before CPUHP_AP_SCHED_WAIT_EMPTY as it ensures when offlining a CPU, all user tasks and non-pinned kernel tasks have left the CPU, i.e. there cannot be a vCPU task around. So, it is safe for KVM's CPU offline callback to disable hardware virtualization at that point. Likewise, KVM's online callback can enable hardware virtualization before any vCPU task gets a chance to run on hotplugged CPUs. KVM's CPU hotplug callbacks are renamed as well. Suggested-by: Thomas Gleixner Signed-off-by: Chao Gao --- include/linux/cpuhotplug.h | 2 +- virt/kvm/kvm_main.c | 28 ++++++++++++++++++++-------- 2 files changed, 21 insertions(+), 9 deletions(-) diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index 773c83730906..14d354c8ce35 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -182,7 +182,6 @@ enum cpuhp_state { CPUHP_AP_CSKY_TIMER_STARTING, CPUHP_AP_TI_GP_TIMER_STARTING, CPUHP_AP_HYPERV_TIMER_STARTING, - CPUHP_AP_KVM_STARTING, CPUHP_AP_KVM_ARM_VGIC_INIT_STARTING, CPUHP_AP_KVM_ARM_VGIC_STARTING, CPUHP_AP_KVM_ARM_TIMER_STARTING, @@ -200,6 +199,7 @@ enum cpuhp_state { /* Online section invoked on the hotplugged CPU from the hotplug thread */ CPUHP_AP_ONLINE_IDLE, + CPUHP_AP_KVM_ONLINE, CPUHP_AP_SCHED_WAIT_EMPTY, CPUHP_AP_SMPBOOT_THREADS, CPUHP_AP_X86_VDSO_VMA_ONLINE, diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 4e1e7770e984..c1054604d1e8 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -4853,13 +4853,25 @@ static void hardware_enable_nolock(void *junk) } } -static int kvm_starting_cpu(unsigned int cpu) +static int kvm_online_cpu(unsigned int cpu) { + int ret = 0; + raw_spin_lock(&kvm_count_lock); - if (kvm_usage_count) + /* + * Abort the CPU online process if hardware virtualization cannot + * be enabled. Otherwise running VMs would encounter unrecoverable + * errors when scheduled to this CPU. + */ + if (kvm_usage_count) { hardware_enable_nolock(NULL); + if (atomic_read(&hardware_enable_failed)) { + ret = -EIO; + pr_info("kvm: abort onlining CPU%d", cpu); + } + } raw_spin_unlock(&kvm_count_lock); - return 0; + return ret; } static void hardware_disable_nolock(void *junk) @@ -4872,7 +4884,7 @@ static void hardware_disable_nolock(void *junk) kvm_arch_hardware_disable(); } -static int kvm_dying_cpu(unsigned int cpu) +static int kvm_offline_cpu(unsigned int cpu) { raw_spin_lock(&kvm_count_lock); if (kvm_usage_count) @@ -5641,8 +5653,8 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align, goto out_free_2; } - r = cpuhp_setup_state_nocalls(CPUHP_AP_KVM_STARTING, "kvm/cpu:starting", - kvm_starting_cpu, kvm_dying_cpu); + r = cpuhp_setup_state_nocalls(CPUHP_AP_KVM_ONLINE, "kvm/cpu:online", + kvm_online_cpu, kvm_offline_cpu); if (r) goto out_free_2; register_reboot_notifier(&kvm_reboot_notifier); @@ -5705,7 +5717,7 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align, kmem_cache_destroy(kvm_vcpu_cache); out_free_3: unregister_reboot_notifier(&kvm_reboot_notifier); - cpuhp_remove_state_nocalls(CPUHP_AP_KVM_STARTING); + cpuhp_remove_state_nocalls(CPUHP_AP_KVM_ONLINE); out_free_2: kvm_arch_hardware_unsetup(); out_free_1: @@ -5731,7 +5743,7 @@ void kvm_exit(void) kvm_async_pf_deinit(); unregister_syscore_ops(&kvm_syscore_ops); unregister_reboot_notifier(&kvm_reboot_notifier); - cpuhp_remove_state_nocalls(CPUHP_AP_KVM_STARTING); + cpuhp_remove_state_nocalls(CPUHP_AP_KVM_ONLINE); on_each_cpu(hardware_disable_nolock, NULL, 1); kvm_arch_hardware_unsetup(); kvm_arch_exit(); From patchwork Mon Dec 27 08:15:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chao Gao X-Patchwork-Id: 12699610 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AE4FC433EF for ; Mon, 27 Dec 2021 08:17:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235647AbhL0IRv (ORCPT ); Mon, 27 Dec 2021 03:17:51 -0500 Received: from mga12.intel.com ([192.55.52.136]:31646 "EHLO mga12.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235620AbhL0IRt (ORCPT ); Mon, 27 Dec 2021 03:17:49 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1640593069; x=1672129069; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=BmPh+jIgxcTIdvBnq8DhbVnkozm0iynu2KVv6M+ZfSE=; b=T71aAlRVsUiXswIEczSYUOEKffjne3laiqMe614EuiqDkGhkHeFH+ifs d17yzzJ0UGuxGav2B1MPzZo9K+m7doy8CkBSsMu2ErFeL0phAXanI1R5U c651ErLNOSg9rDwbS29GpaVJFTjFL5T+Zzpq5Z3Om3LWHaSiU1daV5EVJ KPcvPTWDzQXe/FHTgucnhdUsejjfN9pGzKz5QOro5kMnsWAYDC4nNV3uQ EDotMTl2zI5HWLywOVpxjNQDD1DCq88Ij8tgQ+uvjfh6NQua64iCY1+ub 2421VWegdUKt4gLK+L35vSLCDRSWS8EfqpCstZCB/N6+2vPakyrNH49Z3 g==; X-IronPort-AV: E=McAfee;i="6200,9189,10209"; a="221182107" X-IronPort-AV: E=Sophos;i="5.88,239,1635231600"; d="scan'208";a="221182107" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Dec 2021 00:17:49 -0800 X-IronPort-AV: E=Sophos;i="5.88,239,1635231600"; d="scan'208";a="523208173" Received: from unknown (HELO hyperv-sh4.sh.intel.com) ([10.239.48.22]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Dec 2021 00:17:44 -0800 From: Chao Gao To: kvm@vger.kernel.org, pbonzini@redhat.com, kevin.tian@intel.com, tglx@linutronix.de Cc: Chao Gao , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , linux-kernel@vger.kernel.org Subject: [PATCH 5/6] KVM: x86: Remove WARN_ON in kvm_arch_check_processor_compat Date: Mon, 27 Dec 2021 16:15:11 +0800 Message-Id: <20211227081515.2088920-6-chao.gao@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211227081515.2088920-1-chao.gao@intel.com> References: <20211227081515.2088920-1-chao.gao@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org kvm_arch_check_processor_compat() needn't be called with interrupt disabled, as it only reads some CRs/MSRs which won't be clobbered by interrupt handlers or softirq. What really needed is disabling preemption. No additional check is added because if CONFIG_DEBUG_PREEMPT is enabled, smp_processor_id() (right above the WARN_ON()) can help to detect any violation. Signed-off-by: Chao Gao --- arch/x86/kvm/x86.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index aa09c8792134..a80e3b0c11a8 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -11384,8 +11384,6 @@ int kvm_arch_check_processor_compat(void) { struct cpuinfo_x86 *c = &cpu_data(smp_processor_id()); - WARN_ON(!irqs_disabled()); - if (__cr4_reserved_bits(cpu_has, c) != __cr4_reserved_bits(cpu_has, &boot_cpu_data)) return -EIO; From patchwork Mon Dec 27 08:15:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chao Gao X-Patchwork-Id: 12699611 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 872DDC433EF for ; Mon, 27 Dec 2021 08:17:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235640AbhL0IRy (ORCPT ); Mon, 27 Dec 2021 03:17:54 -0500 Received: from mga12.intel.com ([192.55.52.136]:31646 "EHLO mga12.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235634AbhL0IRw (ORCPT ); Mon, 27 Dec 2021 03:17:52 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1640593072; x=1672129072; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=UOZ+MZumdm6HooFdYpSt0SIJwnAaNRDO5N9HSkFbq7g=; b=ANRqu2tCnIi0bZqCgXEXrxBhbm4iUdZByuWgu/pOCtwLmVDlGRKBOYbt 341lLRQ5Q81rYMHM2wjsXJEyX2lVXTptUqcwB/7CUn7SgxAgCGd1oPbYY uvm2NbN6e1CkqjWbbqpNVn7HiSTyfJhkHETsc7y2o+kWH+ScooXbHUEe4 ifMsb4fCkohL7eMj8Mf63ZJfKwhaSKYEW6GGZmjL7/7/QuJyGH0ymZX8J DePIun9lnf6Nj4bGsvCNsETL4OTvwvMuchx6VzWg7XgRrVJG5IGqoZJbE d6gEL5a5uOUa+I0KvuAMMOqJi5f8EjZfUim7V501+25QEy/3240OVyRAc g==; X-IronPort-AV: E=McAfee;i="6200,9189,10209"; a="221182111" X-IronPort-AV: E=Sophos;i="5.88,239,1635231600"; d="scan'208";a="221182111" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Dec 2021 00:17:51 -0800 X-IronPort-AV: E=Sophos;i="5.88,239,1635231600"; d="scan'208";a="523208194" Received: from unknown (HELO hyperv-sh4.sh.intel.com) ([10.239.48.22]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Dec 2021 00:17:49 -0800 From: Chao Gao To: kvm@vger.kernel.org, pbonzini@redhat.com, kevin.tian@intel.com, tglx@linutronix.de Cc: Chao Gao , linux-kernel@vger.kernel.org Subject: [PATCH 6/6] KVM: Do compatibility checks on hotplugged CPUs Date: Mon, 27 Dec 2021 16:15:12 +0800 Message-Id: <20211227081515.2088920-7-chao.gao@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211227081515.2088920-1-chao.gao@intel.com> References: <20211227081515.2088920-1-chao.gao@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org At init time, KVM does compatibility checks to ensure that all online CPUs support hardware virtualization and a common set of features. But KVM uses hotplugged CPUs without such compatibility checks. On Intel CPUs, this leads to #GP if the hotplugged CPU doesn't support VMX or vmentry failure if the hotplugged CPU doesn't meet minimal feature requirements. Do compatibility checks when onlining a CPU. If any VM is running, KVM hotplug callback returns an error to abort onlining incompatible CPUs. But if no VM is running, onlining incompatible CPUs is allowed. Instead, KVM is prohibited from creating VMs similar to the policy for init-time compatibility checks. CPU hotplug is disabled during hardware_enable_all() to prevent the corner case as shown below. A hotplugged CPU marks itself online in cpu_online_mask (1) and enables interrupt (2) before invoking callbacks registered in ONLINE section (3). So, if hardware_enable_all() is invoked on another CPU right after (2), then on_each_cpu() in hardware_enable_all() invokes hardware_enable_nolock() on the hotplugged CPU before kvm_online_cpu() is called. This makes the CPU escape from compatibility checks, which is risky. start_secondary { ... set_cpu_online(smp_processor_id(), true); <- 1 ... local_irq_enable(); <- 2 ... cpu_startup_entry(CPUHP_AP_ONLINE_IDLE); <- 3 } Keep compatibility checks at KVM init time. It can help to find incompatibility issues earlier and refuse to load arch KVM module (e.g., kvm-intel). Signed-off-by: Chao Gao --- virt/kvm/kvm_main.c | 36 ++++++++++++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index c1054604d1e8..0ff80076d48d 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -106,6 +106,8 @@ LIST_HEAD(vm_list); static cpumask_var_t cpus_hardware_enabled; static int kvm_usage_count; static atomic_t hardware_enable_failed; +/* Set if hardware becomes incompatible after CPU hotplug */ +static bool hardware_incompatible; static struct kmem_cache *kvm_vcpu_cache; @@ -4855,20 +4857,32 @@ static void hardware_enable_nolock(void *junk) static int kvm_online_cpu(unsigned int cpu) { - int ret = 0; + int ret; + ret = kvm_arch_check_processor_compat(); raw_spin_lock(&kvm_count_lock); /* * Abort the CPU online process if hardware virtualization cannot * be enabled. Otherwise running VMs would encounter unrecoverable * errors when scheduled to this CPU. */ - if (kvm_usage_count) { + if (!ret && kvm_usage_count) { hardware_enable_nolock(NULL); if (atomic_read(&hardware_enable_failed)) { ret = -EIO; pr_info("kvm: abort onlining CPU%d", cpu); } + } else if (ret && !kvm_usage_count) { + /* + * Continue onlining an incompatible CPU if no VM is + * running. KVM should reject creating any VM after this + * point. Then this CPU can be still used to run non-VM + * workload. + */ + ret = 0; + hardware_incompatible = true; + pr_info("kvm: prohibit VM creation due to incompatible CPU%d", + cpu); } raw_spin_unlock(&kvm_count_lock); return ret; @@ -4913,8 +4927,24 @@ static int hardware_enable_all(void) { int r = 0; + /* + * During onlining a CPU, cpu_online_mask is set before kvm_online_cpu() + * is called. on_each_cpu() between them includes the CPU. As a result, + * hardware_enable_nolock() may get invoked before kvm_online_cpu(). + * This would enable hardware virtualization on that cpu without + * compatibility checks, which can potentially crash system or break + * running VMs. + * + * Disable CPU hotplug to prevent this case from happening. + */ + cpus_read_lock(); raw_spin_lock(&kvm_count_lock); + if (hardware_incompatible) { + r = -EIO; + goto unlock; + } + kvm_usage_count++; if (kvm_usage_count == 1) { atomic_set(&hardware_enable_failed, 0); @@ -4926,7 +4956,9 @@ static int hardware_enable_all(void) } } +unlock: raw_spin_unlock(&kvm_count_lock); + cpus_read_unlock(); return r; }