From patchwork Tue Aug 15 09:55:10 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Roger_Pau_Monn=C3=A9?= X-Patchwork-Id: 9901543 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 5855260244 for ; Tue, 15 Aug 2017 09:58:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 47AA228815 for ; Tue, 15 Aug 2017 09:58:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3B18928822; Tue, 15 Aug 2017 09:58:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 953BB28815 for ; Tue, 15 Aug 2017 09:58:05 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dhYZG-0001j0-7s; Tue, 15 Aug 2017 09:55:22 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dhYZF-0001it-BR for xen-devel@lists.xen.org; Tue, 15 Aug 2017 09:55:21 +0000 Received: from [193.109.254.147] by server-2.bemta-6.messagelabs.com id 50/4F-27137-805C2995; Tue, 15 Aug 2017 09:55:20 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrAIsWRWlGSWpSXmKPExsWyU9JRQpfj6KR Ig83TuSyWfFzM4sDocXT3b6YAxijWzLyk/IoE1ow936ULdmlWnL26gbWBsV++i5GTQ0LAT2LF 2RvsIDaLgKrE/iXdrF2MHBxsAvYS079WgIRFBJQlJr66zAxiMwtEShzeOIcJxBYW8JY413CbD cTmFfCQ+HJnEQuELShxcuYTFoh6PYkbU6ewgYxkFpCWWP6PAyIsL9G8dTbYSE4BDYnnc2eDjR EVUJE4uXIN2HghAUWJ/nkP2CCuTJeY+KyHZQIj/ywkG2Yh2TALYcMsJBsWMLKsYlQvTi0qSy3 SNdVLKspMzyjJTczM0TU0MNPLTS0uTkxPzUlMKtZLzs/dxAgMSQYg2ME4/bL/IUZJDiYlUd75 UZMihfiS8lMqMxKLM+KLSnNSiw8xynBwKEnw7j4MlBMsSk1PrUjLzAFGB0xagoNHSYRX7whQm re4IDG3ODMdInWKUVFKnFcfJCEAksgozYNrg0XkJUZZKWFeRqBDhHgKUotyM0tQ5V8xinMwKg nz6oBM4cnMK4Gb/gpoMRPQ4ivtYItLEhFSUg2Mm+dMLbPaGffz2XxLzfz+hT+sJprGO5zrKbD aFCseLq4UL7718+Q2yUNNB2/XXa+wWTHvTzn3xEu7+XbttNI3LRaceHWuarTCwtB5RXOuOJpa lj45mu3GO7NIwnbVxuZpTFHb15XFzOc/7Fx7Lsg55afwVRf/EEshNhepcPZFbd33BYs634crs RRnJBpqMRcVJwIA0LwO0cMCAAA= X-Env-Sender: prvs=393075e76=roger.pau@citrix.com X-Msg-Ref: server-8.tower-27.messagelabs.com!1502790919!100949424!1 X-Originating-IP: [185.25.65.24] X-SpamReason: No, hits=0.0 required=7.0 tests=received_headers: No Received headers X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 20294 invoked from network); 15 Aug 2017 09:55:20 -0000 Received: from smtp.ctxuk.citrix.com (HELO SMTP.EU.CITRIX.COM) (185.25.65.24) by server-8.tower-27.messagelabs.com with RC4-SHA encrypted SMTP; 15 Aug 2017 09:55:20 -0000 X-IronPort-AV: E=Sophos;i="5.41,377,1498521600"; d="scan'208";a="50992808" Date: Tue, 15 Aug 2017 10:55:10 +0100 From: Roger Pau =?iso-8859-1?Q?Monn=E9?= To: Andreas Kinzler Message-ID: <20170815095306.rpxanbs7kd5m2tne@MacBook-Pro-de-Roger.local> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20170714 (1.8.3) X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) Cc: Jan Beulich , "xen-devel@lists.xen.org" Subject: Re: [Xen-devel] Regression PCI passthrough from 4.5.5 to 4.6.0-rc1 X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP On Mon, Aug 14, 2017 at 02:08:56PM +0200, Andreas Kinzler wrote: > On Mon, 14 Aug 2017 13:56:58 +0200, Roger Pau Monné > wrote: > > > > I defined XEN_PT_LOGGING_ENABLED in xen_pt.h as requested without the > > > > "hack" patch. Log is attached. Does it help? > > > It tells me that there's nothing unexpected on that side. As I think I > > > had indicated before, we really need to see both sides (qemu and > > > hypervisor), as part of the MSI-X handling lives in Xen. And for the > > > hypervisor side it is unlikely that we'll be able to get away without > > > a debugging patch. I am intending to make such available to you in > > > case you can't do so yourself, but I can't currently predict when I'll > > > get to it. > > I think the problem is that pci_msi_conf_write_intercept is failing to > > unmask the entries when MSI-X is enabled with entries already > > configured, but this will require some debugging patch as Jan said. > > Following the MSI-X code is quite complicated, this split brain > > between Xen and QEMU makes it quite hard. I can try to come up with a > > patch later. > > I can try some debug patches although my workload is very high at the > moment. It would help me quite a bit if the debug patches were suitable for > the stable 4.8 tree. Hello, Could you please try the patch below and paste the output you get on the Xen console? Jan, AFAICT (but I have to admit it's not easy to follow the code at all), the following series of events will cause the MSIX entries to not be unmasked: 1. Guest configures the MSIX table entries and unmasks each of them. 2. Guest enables MSIX. This will cause the entries to remain masked, because QEMU will only register the PIRQs and bind them when the MSI-X enable bit is set, instead of doing it for each write to the MSIX table. I guess one way to solve this would be to force QEMU to call xen_pt_msix_update_one in pci_msix_write once the entry is unmasked, even if MSIX is not enabled. I can prepare a patch for that. This doesn't happen with Linux/FreeBSD because both of them enabled MSIX first and then configure the table entries and unmask them. Roger. ---8<--- diff --git a/xen/arch/x86/hvm/vmsi.c b/xen/arch/x86/hvm/vmsi.c index d81c5d47c6..d7c64dcd90 100644 --- a/xen/arch/x86/hvm/vmsi.c +++ b/xen/arch/x86/hvm/vmsi.c @@ -330,6 +330,8 @@ static int msixtbl_write(struct vcpu *v, unsigned long address, ASSERT(msi_desc == desc->msi_desc); + printk("%smasking entry %#x\n", + (val & PCI_MSIX_VECTOR_BITMASK) ? "" : "un", nr_entry); guest_mask_msi_irq(desc, !!(val & PCI_MSIX_VECTOR_BITMASK)); unlock: @@ -431,6 +433,9 @@ static void add_msixtbl_entry(struct domain *d, entry->gtable = (unsigned long) gtable; list_add_rcu(&entry->list, &d->arch.hvm_domain.msixtbl_list); + + printk("%04x:%02x:%02x.%u added to msixtbl list\n", pdev->seg, pdev->bus, + PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn)); } static void free_msixtbl_entry(struct rcu_head *rcu) @@ -511,8 +516,12 @@ out: (gtable + msi_desc->msi_attrib.entry_nr * PCI_MSIX_ENTRY_SIZE + PCI_MSIX_ENTRY_VECTOR_CTRL_OFFSET) ) + { + printk("msixtbl_pt_register: detected attempt to write to vector ctrl (entry %#x)\n", + msi_desc->msi_attrib.entry_nr); v->arch.hvm_vcpu.hvm_io.msix_unmask_address = v->arch.hvm_vcpu.hvm_io.msix_snoop_address; + } } } @@ -621,6 +630,7 @@ void msix_write_completion(struct vcpu *v) return; v->arch.hvm_vcpu.hvm_io.msix_unmask_address = 0; + printk("Detected MSI-X unmask in write completion\n"); if ( msixtbl_write(v, ctrl_address, 4, 0) != X86EMUL_OKAY ) gdprintk(XENLOG_WARNING, "MSI-X write completion failure\n"); } diff --git a/xen/arch/x86/msi.c b/xen/arch/x86/msi.c index 2c38adb1b1..f36919d1c3 100644 --- a/xen/arch/x86/msi.c +++ b/xen/arch/x86/msi.c @@ -980,6 +980,8 @@ static int msix_capability_init(struct pci_dev *dev, list_add_tail(&entry->list, &dev->msi_list); *desc = entry; + printk("%04x:%02x:%02x.%u added entry %#x to msi_list\n", + seg, bus, slot, func, msi->entry_nr); } if ( !msix->used_entries ) @@ -1297,6 +1299,18 @@ int pci_msi_conf_write_intercept(struct pci_dev *pdev, unsigned int reg, if ( reg != msix_control_reg(pos) || size != 2 ) return -EACCES; + printk("MSIX ctrl write. Enabled: %d Maskall: %d. " + "Configured entries:\n", + !!(*data & PCI_MSIX_FLAGS_ENABLE), + !!(*data & PCI_MSIX_FLAGS_MASKALL)); + list_for_each_entry( entry, &pdev->msi_list, list ) + { + printk("%#x host_masked: %d guest_masked: %d\n", + entry->msi_attrib.entry_nr, + entry->msi_attrib.host_masked, + entry->msi_attrib.guest_masked); + } + pdev->msix->guest_maskall = !!(*data & PCI_MSIX_FLAGS_MASKALL); if ( pdev->msix->host_maskall ) *data |= PCI_MSIX_FLAGS_MASKALL;