From patchwork Thu Apr 21 09:38:26 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Beulich X-Patchwork-Id: 8898441 Return-Path: X-Original-To: patchwork-xen-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 0DF319F39A for ; Thu, 21 Apr 2016 09:40:33 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 06AEB202EC for ; Thu, 21 Apr 2016 09:40:32 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4CA8220295 for ; Thu, 21 Apr 2016 09:40:26 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1atB4G-0002c3-HT; Thu, 21 Apr 2016 09:38:36 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1atB4E-0002bt-Vg for xen-devel@lists.xenproject.org; Thu, 21 Apr 2016 09:38:35 +0000 Received: from [85.158.137.68] by server-16.bemta-3.messagelabs.com id F8/31-02994-A9F98175; Thu, 21 Apr 2016 09:38:34 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrAIsWRWlGSWpSXmKPExsXS6fjDS3fmfIl wg78PBCy+b5nM5MDocfjDFZYAxijWzLyk/IoE1oyZV1sZCybaVkyeM4utgfGXQRcjJ4eQQJ7E pHtfWEFsXgE7id0bXzGC2BIChhL75q9iA7FZBFQlZhw7yARiswmoS7Q92w5Uz8EhImAgce5oU hcjFwezwExGia+3fzKD1AgLOEl8OrOOCaSGV0BQ4u8OYZAwM9D47de2Mk9g5JqFkJmFJANha0 k8/HWLBcLWlli28DUzSDmzgLTE8n8cEKadxIZDjqgqQGxPid4V7UwLGDlWMaoXpxaVpRbpGus lFWWmZ5TkJmbm6BoaGOvlphYXJ6an5iQmFesl5+duYgSGHgMQ7GBs/uJ0iFGSg0lJlHdBtUS4 EF9SfkplRmJxRnxRaU5q8SFGDQ4OgW27Vl9glGLJy89LVZLgvTkPqE6wKDU9tSItMwcYHTClE hw8SiK8mSBp3uKCxNzizHSI1ClGRSlx3ukgCQGQREZpHlwbLCIvMcpKCfMyAh0lxFOQWpSbWY Iq/4pRnINRSZh3AsgUnsy8Erjpr4AWMwEt5r8rCrK4JBEhJdXAuF9a/lP/sYDpYT/8OpW+OAY KBfs0zlzmlqdqnrqupyLrkFKSwKr7x8VfR4eqqjKvXX7N3l354MoHnUs/uzot157h5Sdgbnxk UcyL3/s57XXfPNF2unHzgf2E4/GSb8VbjIt1EwsPfvxQuNs49Kya4vw/XTcuBa0oWtJz9XzQ/ 5goFymxYpeTSizFGYmGWsxFxYkA8aPEtcMCAAA= X-Env-Sender: JBeulich@suse.com X-Msg-Ref: server-11.tower-31.messagelabs.com!1461231510!9874627!1 X-Originating-IP: [137.65.248.74] X-SpamReason: No, hits=0.5 required=7.0 tests=BODY_RANDOM_LONG X-StarScan-Received: X-StarScan-Version: 8.28; banners=-,-,- X-VirusChecked: Checked Received: (qmail 6042 invoked from network); 21 Apr 2016 09:38:31 -0000 Received: from prv-mh.provo.novell.com (HELO prv-mh.provo.novell.com) (137.65.248.74) by server-11.tower-31.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 21 Apr 2016 09:38:31 -0000 Received: from INET-PRV-MTA by prv-mh.provo.novell.com with Novell_GroupWise; Thu, 21 Apr 2016 03:38:29 -0600 Message-Id: <5718BBB202000078000E4484@prv-mh.provo.novell.com> X-Mailer: Novell GroupWise Internet Agent 14.2.0 Date: Thu, 21 Apr 2016 03:38:26 -0600 From: "Jan Beulich" To: "xen-devel" Mime-Version: 1.0 Cc: Andrew Cooper , Keir Fraser , Paul Durrant , Wei Liu Subject: [Xen-devel] [PATCH] x86/vMSI-X: avoid missing first unmask of vectors X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Recent changes to Linux result in there just being a single unmask operation prior to expecting the first interrupts to arrive. However, we've had a chicken-and-egg problem here: Qemu invokes xc_domain_update_msi_irq(), ultimately leading to msixtbl_pt_register(), upon seeing that first unmask operation. Yet for msixtbl_range() to return true (in order to msixtbl_write() to get invoked at all) msixtbl_pt_register() must have completed. Deal with this by snooping suitable writes in msixtbl_range() and triggering the invocation of msix_write_completion() from msixtbl_pt_register() when that happens in the context of a still in progress vector control field write. Note that the seemingly unrelated deletion of the redundant irq_desc->msi_desc checks in msixtbl_pt_register() is to make clear to any compiler version used that the "msi_desc" local variable isn't being used uninitialized. (Doing the same in msixtbl_pt_unregister() is just for consistency reasons.) Signed-off-by: Jan Beulich Reviewed-by: Paul Durrant --- TODO: How to deal with REP MOVS to the MSI-X table (in msixtbl_range())? x86/vMSI-X: avoid missing first unmask of vectors Recent changes to Linux result in there just being a single unmask operation prior to expecting the first interrupts to arrive. However, we've had a chicken-and-egg problem here: Qemu invokes xc_domain_update_msi_irq(), ultimately leading to msixtbl_pt_register(), upon seeing that first unmask operation. Yet for msixtbl_range() to return true (in order to msixtbl_write() to get invoked at all) msixtbl_pt_register() must have completed. Deal with this by snooping suitable writes in msixtbl_range() and triggering the invocation of msix_write_completion() from msixtbl_pt_register() when that happens in the context of a still in progress vector control field write. Note that the seemingly unrelated deletion of the redundant irq_desc->msi_desc checks in msixtbl_pt_register() is to make clear to any compiler version used that the "msi_desc" local variable isn't being used uninitialized. (Doing the same in msixtbl_pt_unregister() is just for consistency reasons.) Signed-off-by: Jan Beulich --- TODO: How to deal with REP MOVS to the MSI-X table (in msixtbl_range())? --- a/xen/arch/x86/hvm/vmsi.c +++ b/xen/arch/x86/hvm/vmsi.c @@ -341,7 +352,21 @@ static int msixtbl_range(struct vcpu *v, desc = msixtbl_addr_to_desc(msixtbl_find_entry(v, addr), addr); rcu_read_unlock(&msixtbl_rcu_lock); - return !!desc; + if ( desc ) + return 1; + + if ( (addr & (PCI_MSIX_ENTRY_SIZE - 1)) == + PCI_MSIX_ENTRY_VECTOR_CTRL_OFFSET ) + { + const ioreq_t *r = &v->arch.hvm_vcpu.hvm_io.io_req; + + ASSERT(r->type == IOREQ_TYPE_COPY); + if ( r->dir == IOREQ_WRITE && r->size == 4 && !r->data_is_ptr + && !(r->data & PCI_MSIX_VECTOR_BITMASK) ) + v->arch.hvm_vcpu.hvm_io.msix_snoop_address = addr; + } + + return 0; } static const struct hvm_mmio_ops msixtbl_mmio_ops = { @@ -410,9 +434,6 @@ int msixtbl_pt_register(struct domain *d return r; } - if ( !irq_desc->msi_desc ) - goto out; - msi_desc = irq_desc->msi_desc; if ( !msi_desc ) goto out; @@ -437,6 +458,21 @@ found: out: spin_unlock_irq(&irq_desc->lock); xfree(new_entry); + + if ( !r ) + { + struct vcpu *v; + + for_each_vcpu ( d, v ) + { + if ( v->arch.hvm_vcpu.hvm_io.msix_snoop_address == + (gtable + msi_desc->msi_attrib.entry_nr * PCI_MSIX_ENTRY_SIZE + + PCI_MSIX_ENTRY_VECTOR_CTRL_OFFSET) ) + v->arch.hvm_vcpu.hvm_io.msix_unmask_address = + v->arch.hvm_vcpu.hvm_io.msix_snoop_address; + } + } + return r; } @@ -457,9 +496,6 @@ void msixtbl_pt_unregister(struct domain if ( !irq_desc ) return; - if ( !irq_desc->msi_desc ) - goto out; - msi_desc = irq_desc->msi_desc; if ( !msi_desc ) goto out; @@ -522,6 +558,8 @@ void msix_write_completion(struct vcpu * { unsigned long ctrl_address = v->arch.hvm_vcpu.hvm_io.msix_unmask_address; + v->arch.hvm_vcpu.hvm_io.msix_snoop_address = 0; + if ( !ctrl_address ) return; --- a/xen/include/asm-x86/hvm/vcpu.h +++ b/xen/include/asm-x86/hvm/vcpu.h @@ -85,6 +85,7 @@ struct hvm_vcpu_io { bool_t mmio_retry; unsigned long msix_unmask_address; + unsigned long msix_snoop_address; const struct g2m_ioport *g2m_ioport; }; --- a/xen/arch/x86/hvm/vmsi.c +++ b/xen/arch/x86/hvm/vmsi.c @@ -341,7 +352,21 @@ static int msixtbl_range(struct vcpu *v, desc = msixtbl_addr_to_desc(msixtbl_find_entry(v, addr), addr); rcu_read_unlock(&msixtbl_rcu_lock); - return !!desc; + if ( desc ) + return 1; + + if ( (addr & (PCI_MSIX_ENTRY_SIZE - 1)) == + PCI_MSIX_ENTRY_VECTOR_CTRL_OFFSET ) + { + const ioreq_t *r = &v->arch.hvm_vcpu.hvm_io.io_req; + + ASSERT(r->type == IOREQ_TYPE_COPY); + if ( r->dir == IOREQ_WRITE && r->size == 4 && !r->data_is_ptr + && !(r->data & PCI_MSIX_VECTOR_BITMASK) ) + v->arch.hvm_vcpu.hvm_io.msix_snoop_address = addr; + } + + return 0; } static const struct hvm_mmio_ops msixtbl_mmio_ops = { @@ -410,9 +434,6 @@ int msixtbl_pt_register(struct domain *d return r; } - if ( !irq_desc->msi_desc ) - goto out; - msi_desc = irq_desc->msi_desc; if ( !msi_desc ) goto out; @@ -437,6 +458,21 @@ found: out: spin_unlock_irq(&irq_desc->lock); xfree(new_entry); + + if ( !r ) + { + struct vcpu *v; + + for_each_vcpu ( d, v ) + { + if ( v->arch.hvm_vcpu.hvm_io.msix_snoop_address == + (gtable + msi_desc->msi_attrib.entry_nr * PCI_MSIX_ENTRY_SIZE + + PCI_MSIX_ENTRY_VECTOR_CTRL_OFFSET) ) + v->arch.hvm_vcpu.hvm_io.msix_unmask_address = + v->arch.hvm_vcpu.hvm_io.msix_snoop_address; + } + } + return r; } @@ -457,9 +496,6 @@ void msixtbl_pt_unregister(struct domain if ( !irq_desc ) return; - if ( !irq_desc->msi_desc ) - goto out; - msi_desc = irq_desc->msi_desc; if ( !msi_desc ) goto out; @@ -522,6 +558,8 @@ void msix_write_completion(struct vcpu * { unsigned long ctrl_address = v->arch.hvm_vcpu.hvm_io.msix_unmask_address; + v->arch.hvm_vcpu.hvm_io.msix_snoop_address = 0; + if ( !ctrl_address ) return; --- a/xen/include/asm-x86/hvm/vcpu.h +++ b/xen/include/asm-x86/hvm/vcpu.h @@ -85,6 +85,7 @@ struct hvm_vcpu_io { bool_t mmio_retry; unsigned long msix_unmask_address; + unsigned long msix_snoop_address; const struct g2m_ioport *g2m_ioport; };