From patchwork Sun May 21 13:09:42 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luwei Kang X-Patchwork-Id: 9739561 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id BFE3960392 for ; Mon, 22 May 2017 02:49:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A599A28622 for ; Mon, 22 May 2017 02:49:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 981FF28657; Mon, 22 May 2017 02:49:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.2 required=2.0 tests=BAYES_00, DATE_IN_PAST_12_24, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id F0CA428622 for ; Mon, 22 May 2017 02:49:18 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dCdMu-0000oz-9Y; Mon, 22 May 2017 02:46:48 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dCdMs-0000ot-SW for xen-devel@lists.xen.org; Mon, 22 May 2017 02:46:46 +0000 Received: from [85.158.143.35] by server-5.bemta-6.messagelabs.com id 0A/58-03371-61152295; Mon, 22 May 2017 02:46:46 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrMLMWRWlGSWpSXmKPExsXS1tYhoisaqBR psKNT1WLJx8UsDoweR3f/ZgpgjGLNzEvKr0hgzbj05QVzwTeFio3/n7I2MK6U7GLk5BASmM4o cf2RCYgtIcArcWTZDFYI21/icMsGZoiaUon932+A2WwC6hJb328EqxERkJa49vkyYxcjBwezQ JHE4qsRIGFhASeJzev7GEFsFgFViQ1b2sBsXgEXiRPHFjJCjJeTuHmuk3kCI/cCRoZVjOrFqU VlqUW6hnpJRZnpGSW5iZk5uoYGZnq5qcXFiempOYlJxXrJ+bmbGIGeZQCCHYw7nzsdYpTkYFI S5d3opBQpxJeUn1KZkVicEV9UmpNafIhRhoNDSYK3NQAoJ1iUmp5akZaZAwwxmLQEB4+SCO83 f6A0b3FBYm5xZjpE6hSjopQ4Lx9InwBIIqM0D64NFtaXGGWlhHkZgQ4R4ilILcrNLEGVf8Uoz sGoJMxrBjKFJzOvBG76K6DFTECLrZ/JgywuSURISTUw2mjrZnt2fbzl+FboWVGIWrKr97snMn 4e/EZemd9t5mUptmu+mlBldKvrdlWf6ye99q8lyv/PHbh7MUTYvPz9hEfv/nPMyTxVfojxucU l24wG00VO/7/v+yPaHc36Qat7kuW86AMf92dyzDSuFbm+6AOj4+eI/Fl6Fo28Et8i1sfVsgi+ 4tJQYinOSDTUYi4qTgQA2U+pn2YCAAA= X-Env-Sender: luwei.kang@intel.com X-Msg-Ref: server-6.tower-21.messagelabs.com!1495421203!46859191!1 X-Originating-IP: [134.134.136.20] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTM0LjEzNC4xMzYuMjAgPT4gMzU1MzU4\n X-StarScan-Received: X-StarScan-Version: 9.4.12; banners=-,-,- X-VirusChecked: Checked Received: (qmail 5866 invoked from network); 22 May 2017 02:46:45 -0000 Received: from mga02.intel.com (HELO mga02.intel.com) (134.134.136.20) by server-6.tower-21.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 22 May 2017 02:46:45 -0000 Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 May 2017 19:46:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.38,377,1491289200"; d="scan'208";a="264728588" Received: from vmm.bj.intel.com ([10.238.135.172]) by fmsmga004.fm.intel.com with ESMTP; 21 May 2017 19:46:40 -0700 From: Luwei Kang To: xen-devel@lists.xen.org Date: Sun, 21 May 2017 21:09:42 +0800 Message-Id: <1495372182-12928-1-git-send-email-luwei.kang@intel.com> X-Mailer: git-send-email 2.7.4 Cc: andrew.cooper3@citrix.com, boris.ostrovsky@oracle.com, Luwei Kang , jbeulich@suse.com Subject: [Xen-devel] [PATCH v3] x86/vpmu: add cpu hot unplug notifier for vpmu X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Currently, Hot unplug a physical CPU with vpmu enabled may cause system hang due to send a remote call to an offlined pCPU. This patch add a cpu hot unplug notifer to save vpmu context before cpu offline. Consider one scenario, hot unplug pCPU N with vpmu enabled. The vcpu which running on this pCPU will be switch to other online cpu. A remote call will be send to pCPU N to save the vpmu context before loading the vpmu context on this pCPU. System will hang in function on_select_cpus() because of that pCPU is offlined and can not do any respond. Signed-off-by: Luwei Kang --- v3: 1.add cpu_online() check in vpm_load() and vpmu_arch_destroy(); 2.add vpmu_ prefix. rename cpu_callback() to vpmu_cpu_callback(); v2: 1.fix some typo and coding style; 2.change "swith" to "if" in cpu_callback() because of there just have one case; 3.add VPMU_CONTEX_LOADED check before send remote call in vpmu_arch_destroy(); --- xen/arch/x86/cpu/vpmu.c | 52 +++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 46 insertions(+), 6 deletions(-) diff --git a/xen/arch/x86/cpu/vpmu.c b/xen/arch/x86/cpu/vpmu.c index 03401fd..486af12 100644 --- a/xen/arch/x86/cpu/vpmu.c +++ b/xen/arch/x86/cpu/vpmu.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -394,8 +395,11 @@ int vpmu_load(struct vcpu *v, bool_t from_guest) if ( !vpmu_is_set(vpmu, VPMU_CONTEXT_ALLOCATED) ) return 0; - /* First time this VCPU is running here */ - if ( vpmu->last_pcpu != pcpu ) + /* + * The last pCPU is still online and this is the first time this vCPU + * running here. + */ + if ( cpu_online(vpmu->last_pcpu) && vpmu->last_pcpu != pcpu ) { /* * Get the context from last pcpu that we ran on. Note that if another @@ -575,15 +579,21 @@ static void vpmu_arch_destroy(struct vcpu *v) * We will test it again in vpmu_clear_last() with interrupts * disabled to make sure we don't clear someone else. */ - if ( per_cpu(last_vcpu, vpmu->last_pcpu) == v ) + if ( cpu_online(vpmu->last_pcpu) && + per_cpu(last_vcpu, vpmu->last_pcpu) == v ) on_selected_cpus(cpumask_of(vpmu->last_pcpu), vpmu_clear_last, v, 1); if ( vpmu->arch_vpmu_ops && vpmu->arch_vpmu_ops->arch_vpmu_destroy ) { - /* Unload VPMU first. This will stop counters */ - on_selected_cpus(cpumask_of(vcpu_vpmu(v)->last_pcpu), - vpmu_save_force, v, 1); + /* + * Unload VPMU first if VPMU_CONTEXT_LOADED being set. + * This will stop counters. + */ + if ( vpmu_is_set(vpmu, VPMU_CONTEXT_LOADED) ) + on_selected_cpus(cpumask_of(vcpu_vpmu(v)->last_pcpu), + vpmu_save_force, v, 1); + vpmu->arch_vpmu_ops->arch_vpmu_destroy(v); } } @@ -835,6 +845,33 @@ long do_xenpmu_op(unsigned int op, XEN_GUEST_HANDLE_PARAM(xen_pmu_params_t) arg) return ret; } +static int vpmu_cpu_callback( + struct notifier_block *nfb, unsigned long action, void *hcpu) +{ + unsigned int cpu = (unsigned long)hcpu; + struct vcpu *vcpu = per_cpu(last_vcpu, cpu); + struct vpmu_struct *vpmu; + + if ( !vcpu ) + return NOTIFY_DONE; + + vpmu = vcpu_vpmu(vcpu); + if ( !vpmu_is_set(vpmu, VPMU_CONTEXT_ALLOCATED) ) + return NOTIFY_DONE; + + if ( action == CPU_DYING ) + { + vpmu_save_force(vcpu); + vpmu_reset(vpmu, VPMU_CONTEXT_LOADED); + } + + return NOTIFY_DONE; +} + +static struct notifier_block vpmu_cpu_nfb = { + .notifier_call = vpmu_cpu_callback +}; + static int __init vpmu_init(void) { int vendor = current_cpu_data.x86_vendor; @@ -872,8 +909,11 @@ static int __init vpmu_init(void) } if ( vpmu_mode != XENPMU_MODE_OFF ) + { + register_cpu_notifier(&vpmu_cpu_nfb); printk(XENLOG_INFO "VPMU: version " __stringify(XENPMU_VER_MAJ) "." __stringify(XENPMU_VER_MIN) "\n"); + } else opt_vpmu_enabled = 0;