From patchwork Fri Nov 18 01:57:19 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wu, Feng" X-Patchwork-Id: 9435667 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 28D6760238 for ; Fri, 18 Nov 2016 02:32:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1D95C2972B for ; Fri, 18 Nov 2016 02:32:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 128B129737; Fri, 18 Nov 2016 02:32:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id C403B2972B for ; Fri, 18 Nov 2016 02:32:49 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1c7Ywd-00045T-O8; Fri, 18 Nov 2016 02:30:27 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1c7Ywc-000450-Ie for xen-devel@lists.xen.org; Fri, 18 Nov 2016 02:30:26 +0000 Received: from [193.109.254.147] by server-6.bemta-6.messagelabs.com id E9/6F-28843-1C76E285; Fri, 18 Nov 2016 02:30:25 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrHLMWRWlGSWpSXmKPExsVywNykWPdgul6 Ewbl2aYslHxezODB6HN39mymAMYo1My8pvyKBNaP32H3mgtdqFfeOTmNsYDwv18XIySEkUCnx ZOFcFhBbQoBX4siyGawQdoDEkyft7F2MXEA1fYwSs/uOM4Ik2AQUJQ5ePARWJCIgLXHt82VGk CJmgQWMEo0XjzKDJIQFIiXWrljNBGKzCKhK/Hhyka2LkYODV8BR4sv5EIgFchIbdv8Hm8kp4C RxfPVtFoiDHCX2T/3AMoGRdwEjwypGjeLUorLUIl1DI72kosz0jJLcxMwcXUMDM73c1OLixPT UnMSkYr3k/NxNjMBwYACCHYyXNwYcYpTkYFIS5W2P1osQ4kvKT6nMSCzOiC8qzUktPsQow8Gh JMH7KQ0oJ1iUmp5akZaZAwxMmLQEB4+SCO+NVKA0b3FBYm5xZjpE6hSjopQ4bw9InwBIIqM0D 64NFg2XGGWlhHkZgQ4R4ilILcrNLEGVf8UozsGoJMy7BWQKT2ZeCdz0V0CLmYAW7xHQAVlcko iQkmpgbG1i3XztpoXLmw1b7uze82NlQuKkWawTVqikqmamrnHlFtu4M1jltbnoxQs1zCZ808+ HfNj56UlYxCq+K10x1wOuxt80KW63ce4NuqgTIrOYuVFwv3unb+mffHOhSjul+1+N063OVL/n +vO+o/fQE4/9PtzRBa79xT/P1Dvt27pCRX/X1+fTlViKMxINtZiLihMBRQJ2P4ECAAA= X-Env-Sender: feng.wu@intel.com X-Msg-Ref: server-14.tower-27.messagelabs.com!1479436221!60052313!3 X-Originating-IP: [192.55.52.115] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.0.16; banners=-,-,- X-VirusChecked: Checked Received: (qmail 58799 invoked from network); 18 Nov 2016 02:30:25 -0000 Received: from mga14.intel.com (HELO mga14.intel.com) (192.55.52.115) by server-14.tower-27.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 18 Nov 2016 02:30:25 -0000 Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga103.fm.intel.com with ESMTP; 17 Nov 2016 18:30:20 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos; i="5.31,655,1473145200"; d="scan'208"; a="1070084841" Received: from unknown (HELO feng-bdw-de-pi.bj.intel.com) ([10.238.154.55]) by fmsmga001.fm.intel.com with ESMTP; 17 Nov 2016 18:30:19 -0800 From: Feng Wu To: xen-devel@lists.xen.org Date: Fri, 18 Nov 2016 09:57:19 +0800 Message-Id: <1479434244-10223-3-git-send-email-feng.wu@intel.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1479434244-10223-1-git-send-email-feng.wu@intel.com> References: <1479434244-10223-1-git-send-email-feng.wu@intel.com> Cc: kevin.tian@intel.com, Feng Wu , george.dunlap@eu.citrix.com, andrew.cooper3@citrix.com, dario.faggioli@citrix.com, jbeulich@suse.com Subject: [Xen-devel] [PATCH v8 2/7] VMX: Properly handle pi when all the assigned devices are removed X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP This patch handles some corner cases when the last assigned device is removed from the domain. In this case we should carefully handle pi descriptor and the per-cpu blocking list, to make sure: - all the PI descriptor are in the right state when next time a devices is assigned to the domain again. - No remaining vcpus of the domain in the per-cpu blocking list. Here we call vmx_pi_unblock_vcpu() to remove the vCPU from the blocking list if it is on the list. However, this could happen when vmx_vcpu_block() is being called, hence we might incorrectly add the vCPU to the blocking list while the last devcie is detached from the domain. Consider that the situation can only occur when detaching the last device from the domain and it is not a frequent operation, so we use domain_pause before that, which is considered as an clean and maintainable solution for the situation. Signed-off-by: Feng Wu Reviewed-by: Jan Beulich --- v7: - Prevent the domain from pausing itself. v6: - Comments changes - Rename vmx_pi_list_remove() to vmx_pi_unblock_vcpu() v5: - Remove a no-op wrapper v4: - Rename some functions: vmx_pi_remove_vcpu_from_blocking_list() -> vmx_pi_list_remove() vmx_pi_blocking_cleanup() -> vmx_pi_list_cleanup() - Remove the check in vmx_pi_list_cleanup() - Comments adjustment xen/arch/x86/hvm/vmx/vmx.c | 28 ++++++++++++++++++++++++---- xen/drivers/passthrough/pci.c | 14 ++++++++++++++ 2 files changed, 38 insertions(+), 4 deletions(-) diff --git a/xen/arch/x86/hvm/vmx/vmx.c b/xen/arch/x86/hvm/vmx/vmx.c index f3911f2..a8dcabe 100644 --- a/xen/arch/x86/hvm/vmx/vmx.c +++ b/xen/arch/x86/hvm/vmx/vmx.c @@ -158,14 +158,12 @@ static void vmx_pi_switch_to(struct vcpu *v) pi_clear_sn(pi_desc); } -static void vmx_pi_do_resume(struct vcpu *v) +static void vmx_pi_unblock_vcpu(struct vcpu *v) { unsigned long flags; spinlock_t *pi_blocking_list_lock; struct pi_desc *pi_desc = &v->arch.hvm_vmx.pi_desc; - ASSERT(!test_bit(_VPF_blocked, &v->pause_flags)); - /* * Set 'NV' field back to posted_intr_vector, so the * Posted-Interrupts can be delivered to the vCPU when @@ -173,12 +171,12 @@ static void vmx_pi_do_resume(struct vcpu *v) */ write_atomic(&pi_desc->nv, posted_intr_vector); - /* The vCPU is not on any blocking list. */ pi_blocking_list_lock = v->arch.hvm_vmx.pi_blocking.lock; /* Prevent the compiler from eliminating the local variable.*/ smp_rmb(); + /* The vCPU is not on any blocking list. */ if ( pi_blocking_list_lock == NULL ) return; @@ -198,6 +196,13 @@ static void vmx_pi_do_resume(struct vcpu *v) spin_unlock_irqrestore(pi_blocking_list_lock, flags); } +static void vmx_pi_do_resume(struct vcpu *v) +{ + ASSERT(!test_bit(_VPF_blocked, &v->pause_flags)); + + vmx_pi_unblock_vcpu(v); +} + /* This function is called when pcidevs_lock is held */ void vmx_pi_hooks_assign(struct domain *d) { @@ -215,11 +220,21 @@ void vmx_pi_hooks_assign(struct domain *d) /* This function is called when pcidevs_lock is held */ void vmx_pi_hooks_deassign(struct domain *d) { + struct vcpu *v; + if ( !iommu_intpost || !has_hvm_container_domain(d) ) return; ASSERT(d->arch.hvm_domain.vmx.vcpu_block); + /* + * Pausing the domain can make sure the vCPU is not + * running and hence not calling the hooks simultaneously + * when deassigning the PI hooks and removing the vCPU + * from the blocking list. + */ + domain_pause(d); + d->arch.hvm_domain.vmx.vcpu_block = NULL; d->arch.hvm_domain.vmx.pi_switch_from = NULL; d->arch.hvm_domain.vmx.pi_do_resume = NULL; @@ -230,6 +245,11 @@ void vmx_pi_hooks_deassign(struct domain *d) * assigned and "from" hook is NULL. However, it is not straightforward * to find a clear solution, so just leave it here. */ + + for_each_vcpu ( d, v ) + vmx_pi_unblock_vcpu(v); + + domain_unpause(d); } static int vmx_domain_initialise(struct domain *d) diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 8bce213..e71732f 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -1602,6 +1602,13 @@ int iommu_do_pci_domctl( break; case XEN_DOMCTL_assign_device: + /* no domain_pause() */ + if ( d == current->domain ) + { + ret = -EINVAL; + break; + } + ret = -ENODEV; if ( domctl->u.assign_device.dev != XEN_DOMCTL_DEV_PCI ) break; @@ -1642,6 +1649,13 @@ int iommu_do_pci_domctl( break; case XEN_DOMCTL_deassign_device: + /* no domain_pause() */ + if ( d == current->domain ) + { + ret = -EINVAL; + break; + } + ret = -ENODEV; if ( domctl->u.assign_device.dev != XEN_DOMCTL_DEV_PCI ) break;