From patchwork Mon May 22 20:45:40 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luwei Kang X-Patchwork-Id: 9742247 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 2BEC460380 for ; Tue, 23 May 2017 10:24:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 19DA4287A4 for ; Tue, 23 May 2017 10:24:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0E414287B8; Tue, 23 May 2017 10:24:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.2 required=2.0 tests=BAYES_00, DATE_IN_PAST_12_24, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 66EAF287A4 for ; Tue, 23 May 2017 10:24:58 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dD6xX-0000m6-Ng; Tue, 23 May 2017 10:22:35 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dD6xW-0000m0-BI for xen-devel@lists.xen.org; Tue, 23 May 2017 10:22:34 +0000 Received: from [85.158.143.35] by server-10.bemta-6.messagelabs.com id 59/CD-03613-96D04295; Tue, 23 May 2017 10:22:33 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrKLMWRWlGSWpSXmKPExsXS1tYhoZvJqxJ pcGEdl8WSj4tZHBg9ju7+zRTAGMWamZeUX5HAmvF+3kG2gn7Fih0nvjA3MH6V7GLk4hASmM4o cW7DRcYuRk4OCQFeiSPLZrBC2AESdx9cBbOFBEolZjbeZAGx2QTUJba+3wgWFxGQlrj2+TJQL wcHs0CRxOKrESBhYQEnie0bX7KB2CwCqhJPVz0HK+cVcJE4s3kBO8R4OYmb5zqZJzByL2BkWM WoUZxaVJZapGtsqJdUlJmeUZKbmJmja2hgppebWlycmJ6ak5hUrJecn7uJEehdBiDYwdi0KPA QoyQHk5Io754E5UghvqT8lMqMxOKM+KLSnNTiQ4wyHBxKErx/uVUihQSLUtNTK9Iyc4BhBpOW 4OBREuH9CZLmLS5IzC3OTIdInWJUlBLnNeUBSgiAJDJK8+DaYKF9iVFWSpiXEegQIZ6C1KLcz BJU+VeM4hyMSsK8LSBTeDLzSuCmvwJazAS02PWuMsjikkSElFQDI1toNat86VyFb8rfc2/++K j8L9hdeK2oXWgV3+er1X2Sv9I3SN0VOLIowudwt4X2QbEHaa9WPzv721hHcBHf5R6JpLV/aua 89/ENmX5a/O5Rvcv7fft834r+es9o+CY7qXvOTZWHR7dbipfGtJ8//cflSZfKkQccDVxn9rQp /jNc33gsa8erHCWW4oxEQy3mouJEANku41BoAgAA X-Env-Sender: luwei.kang@intel.com X-Msg-Ref: server-13.tower-21.messagelabs.com!1495534950!64354189!1 X-Originating-IP: [134.134.136.24] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTM0LjEzNC4xMzYuMjQgPT4gMzkwOTcx\n X-StarScan-Received: X-StarScan-Version: 9.4.12; banners=-,-,- X-VirusChecked: Checked Received: (qmail 14991 invoked from network); 23 May 2017 10:22:32 -0000 Received: from mga09.intel.com (HELO mga09.intel.com) (134.134.136.24) by server-13.tower-21.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 23 May 2017 10:22:32 -0000 Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 23 May 2017 03:22:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos; i="5.38,381,1491289200"; d="scan'208"; a="1173145725" Received: from vmm.bj.intel.com ([10.238.135.172]) by fmsmga002.fm.intel.com with ESMTP; 23 May 2017 03:22:28 -0700 From: Luwei Kang To: xen-devel@lists.xen.org Date: Tue, 23 May 2017 04:45:40 +0800 Message-Id: <1495485940-20803-1-git-send-email-luwei.kang@intel.com> X-Mailer: git-send-email 2.7.4 Cc: andrew.cooper3@citrix.com, boris.ostrovsky@oracle.com, Luwei Kang , jbeulich@suse.com Subject: [Xen-devel] [PATCH v4] x86/vpmu: add cpu hot unplug notifier for vpmu X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Currently, Hot unplug a physical CPU with vpmu enabled may cause system hang due to send a remote call to an offlined pCPU. This patch add a cpu hot unplug notifer to save vpmu context before cpu offline. Consider one scenario, hot unplug pCPU N with vpmu enabled. The vcpu which running on this pCPU will be switch to other online cpu. A remote call will be send to pCPU N to save the vpmu context before loading the vpmu context on this pCPU. System will hang in function on_select_cpus() because of that pCPU is offlined and can not do any respond. The purpose of add a VPMU_CONTEXT_LOADED check in vpmu_arch_destroy() before send a remote call to save vpmu contex is: a. when a vpmu context has been loaded in a remote pCPU, make a remote call to save the vpmu contex and stop counters is necessary. b. VPMU_CONTEXT_LOADED flag will be reset if a pCPU is offlined. this check will prevent send a remote call to an offlined pCPU. Signed-off-by: Luwei Kang Acked-by: Jan Beulich Reviewed-by: Boris Ostrovsky --- v4: 1.remove cpu_online() check in vpm_load(); 2.remove "vpmu_" prefix; 3.fix a coding style; 4.add some commit message about VPMU_CONTEXT_LOADED in vpmu_arch_destroy(); v3: 1.add cpu_online() check in vpm_load() and vpmu_arch_destroy(); 2.add vpmu_ prefix. rename cpu_callback() to vpmu_cpu_callback(); v2: 1.fix some typo and coding style; 2.change "swith" to "if" in cpu_callback() because of there just have one case; 3.add VPMU_CONTEX_LOADED check before send remote call in vpmu_arch_destroy(); --- xen/arch/x86/cpu/vpmu.c | 45 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 41 insertions(+), 4 deletions(-) diff --git a/xen/arch/x86/cpu/vpmu.c b/xen/arch/x86/cpu/vpmu.c index 03401fd..1f7830b 100644 --- a/xen/arch/x86/cpu/vpmu.c +++ b/xen/arch/x86/cpu/vpmu.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -575,15 +576,21 @@ static void vpmu_arch_destroy(struct vcpu *v) * We will test it again in vpmu_clear_last() with interrupts * disabled to make sure we don't clear someone else. */ - if ( per_cpu(last_vcpu, vpmu->last_pcpu) == v ) + if ( cpu_online(vpmu->last_pcpu) && + per_cpu(last_vcpu, vpmu->last_pcpu) == v ) on_selected_cpus(cpumask_of(vpmu->last_pcpu), vpmu_clear_last, v, 1); if ( vpmu->arch_vpmu_ops && vpmu->arch_vpmu_ops->arch_vpmu_destroy ) { - /* Unload VPMU first. This will stop counters */ - on_selected_cpus(cpumask_of(vcpu_vpmu(v)->last_pcpu), - vpmu_save_force, v, 1); + /* + * Unload VPMU first if VPMU_CONTEXT_LOADED being set. + * This will stop counters. + */ + if ( vpmu_is_set(vpmu, VPMU_CONTEXT_LOADED) ) + on_selected_cpus(cpumask_of(vcpu_vpmu(v)->last_pcpu), + vpmu_save_force, v, 1); + vpmu->arch_vpmu_ops->arch_vpmu_destroy(v); } } @@ -835,6 +842,33 @@ long do_xenpmu_op(unsigned int op, XEN_GUEST_HANDLE_PARAM(xen_pmu_params_t) arg) return ret; } +static int cpu_callback( + struct notifier_block *nfb, unsigned long action, void *hcpu) +{ + unsigned int cpu = (unsigned long)hcpu; + struct vcpu *vcpu = per_cpu(last_vcpu, cpu); + struct vpmu_struct *vpmu; + + if ( !vcpu ) + return NOTIFY_DONE; + + vpmu = vcpu_vpmu(vcpu); + if ( !vpmu_is_set(vpmu, VPMU_CONTEXT_ALLOCATED) ) + return NOTIFY_DONE; + + if ( action == CPU_DYING ) + { + vpmu_save_force(vcpu); + vpmu_reset(vpmu, VPMU_CONTEXT_LOADED); + } + + return NOTIFY_DONE; +} + +static struct notifier_block cpu_nfb = { + .notifier_call = cpu_callback +}; + static int __init vpmu_init(void) { int vendor = current_cpu_data.x86_vendor; @@ -872,8 +906,11 @@ static int __init vpmu_init(void) } if ( vpmu_mode != XENPMU_MODE_OFF ) + { + register_cpu_notifier(&cpu_nfb); printk(XENLOG_INFO "VPMU: version " __stringify(XENPMU_VER_MAJ) "." __stringify(XENPMU_VER_MIN) "\n"); + } else opt_vpmu_enabled = 0;