From patchwork Mon Nov 14 19:20:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 13042744 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2D023C433FE for ; Mon, 14 Nov 2022 19:21:31 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.443551.698159 (Exim 4.92) (envelope-from ) id 1ouf1F-0003yB-6D; Mon, 14 Nov 2022 19:21:21 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 443551.698159; Mon, 14 Nov 2022 19:21:21 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ouf1F-0003y4-3D; Mon, 14 Nov 2022 19:21:21 +0000 Received: by outflank-mailman (input) for mailman id 443551; Mon, 14 Nov 2022 19:21:20 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ouf1E-0003F1-78 for xen-devel@lists.xenproject.org; Mon, 14 Nov 2022 19:21:20 +0000 Received: from wout2-smtp.messagingengine.com (wout2-smtp.messagingengine.com [64.147.123.25]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 839dd65a-6451-11ed-91b6-6bf2151ebd3b; Mon, 14 Nov 2022 20:21:19 +0100 (CET) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id 51083320096E; Mon, 14 Nov 2022 14:21:17 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Mon, 14 Nov 2022 14:21:17 -0500 Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 14 Nov 2022 14:21:15 -0500 (EST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 839dd65a-6451-11ed-91b6-6bf2151ebd3b DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= invisiblethingslab.com; h=cc:cc:content-transfer-encoding :content-type:date:date:from:from:in-reply-to:message-id :mime-version:reply-to:sender:subject:subject:to:to; s=fm1; t= 1668453676; x=1668540076; bh=nPfMqp058g7kRRkTy0PrtYx6pL+BIhReFJy JxLlMvDU=; b=adB/wBdxqn4/pY6JIFrxF6O1vOBh44yy4R4Dc2AZL08q4ws41M5 xI/6lI96evI9Zf3+P97qNAHGg5AwCEVE1ItUteABcYnD8ofDZ4G9EIakwcVYt23i qQaICE23Ct2ItuohBscfgzZ6yyN1DTSINqt//+E0pIL39xDPimxW+Y8apZi3fpYf FNBkBjgCanTFoEz7HbQYK9lrZyEDwfIA1ABy+wB3I4Hrw8DgE9KV1FiGbdgNghPc +NJ5J2Q0fM/wvH74TPNx1a4mvqI/AAPGyZ3Oc6ksm3JPgeRCzrAlZ1YPPpf8lva6 MoPezvGUMpsViTcfLaba83fEj5UFl+oPfqQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:message-id:mime-version:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; t=1668453676; x=1668540076; bh=nPfMqp058g7kR RkTy0PrtYx6pL+BIhReFJyJxLlMvDU=; b=SD2laVW9Ax8rDvDZpjGg6HEw90BFA xWiAwbnB8WGQuLHNn22HG92VuL5ipHx4+A5lCz2SL8RUcHlCuR85GmwhyWn6nqRM WzI/rqpDaDyZVb9jfkvOkTiOmjisKchSKuGZKRu3HsDXtfdgv9/pGh+m/gAF/wxR vL7i0lYbFTOE3GEAD140RLIDpyn92Dz4bc/MKXfGcHib/68S1Hj+4U6wGFq2N1jL TL7PNZmEvTL+k86rqpTNJR8TKIubbDJRPsOiplAMF+v8IpzN7WSy3bJq9UhQcMbz LmfnYwVDOcVeFvFeq5TmekV8Y0L+hFC0P4st/naRROEXfK4iBNLlJ66fw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvgedrgedvgdeliecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffogggtgfesthekredtredtjeenucfhrhhomhepofgrrhgvkhcu ofgrrhgtiiihkhhofihskhhiqdfikphrvggtkhhiuceomhgrrhhmrghrvghksehinhhvih hsihgslhgvthhhihhnghhslhgrsgdrtghomheqnecuggftrfgrthhtvghrnhepleekhfdu leetleelleetteevfeefteffkeetteejheelgfegkeelgeehhfdthedvnecuvehluhhsth gvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepmhgrrhhmrghrvghksehi nhhvihhsihgslhgvthhhihhnghhslhgrsgdrtghomh X-ME-Proxy: Feedback-ID: i1568416f:Fastmail From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Cc: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= , Jan Beulich , Andrew Cooper , =?utf-8?q?Roger_Pau_Monn=C3=A9?= , Wei Liu Subject: [PATCH 1/2] x86/msi: passthrough all MSI-X vector ctrl writes to device model Date: Mon, 14 Nov 2022 20:20:59 +0100 Message-Id: <20221114192100.1539267-1-marmarek@invisiblethingslab.com> X-Mailer: git-send-email 2.37.3 MIME-Version: 1.0 QEMU needs to know whether clearing maskbit of a vector is really clearing, or was already cleared before. Currently Xen sends only clearing that bit to the device model, but not setting it, so QEMU cannot detect it. Because of that, QEMU is working this around by checking via /dev/mem, but that isn't the proper approach. Give all necessary information to QEMU by passing all ctrl writes, including masking a vector. Signed-off-by: Marek Marczykowski-Górecki --- xen/arch/x86/hvm/vmsi.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/xen/arch/x86/hvm/vmsi.c b/xen/arch/x86/hvm/vmsi.c index 75f92885dc5e..ba4cf46dfe91 100644 --- a/xen/arch/x86/hvm/vmsi.c +++ b/xen/arch/x86/hvm/vmsi.c @@ -271,7 +271,8 @@ out: } static int msixtbl_write(struct vcpu *v, unsigned long address, - unsigned int len, unsigned long val) + unsigned int len, unsigned long val, + bool completion) { unsigned long offset; struct msixtbl_entry *entry; @@ -343,7 +344,7 @@ static int msixtbl_write(struct vcpu *v, unsigned long address, unlock: spin_unlock_irqrestore(&desc->lock, flags); - if ( len == 4 ) + if ( len == 4 && completion ) r = X86EMUL_OKAY; out: @@ -355,7 +356,7 @@ static int cf_check _msixtbl_write( const struct hvm_io_handler *handler, uint64_t address, uint32_t len, uint64_t val) { - return msixtbl_write(current, address, len, val); + return msixtbl_write(current, address, len, val, false); } static bool cf_check msixtbl_range( @@ -633,7 +634,7 @@ void msix_write_completion(struct vcpu *v) return; v->arch.hvm.hvm_io.msix_unmask_address = 0; - if ( msixtbl_write(v, ctrl_address, 4, 0) != X86EMUL_OKAY ) + if ( msixtbl_write(v, ctrl_address, 4, 0, true) != X86EMUL_OKAY ) gdprintk(XENLOG_WARNING, "MSI-X write completion failure\n"); } From patchwork Mon Nov 14 19:21:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 13042745 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B247CC433FE for ; Mon, 14 Nov 2022 19:21:33 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.443552.698170 (Exim 4.92) (envelope-from ) id 1ouf1I-0004Gj-E3; Mon, 14 Nov 2022 19:21:24 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 443552.698170; Mon, 14 Nov 2022 19:21:24 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ouf1I-0004Gb-BH; Mon, 14 Nov 2022 19:21:24 +0000 Received: by outflank-mailman (input) for mailman id 443552; Mon, 14 Nov 2022 19:21:22 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ouf1G-0002zm-ME for xen-devel@lists.xenproject.org; Mon, 14 Nov 2022 19:21:22 +0000 Received: from wout2-smtp.messagingengine.com (wout2-smtp.messagingengine.com [64.147.123.25]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 84ff66a9-6451-11ed-8fd2-01056ac49cbb; Mon, 14 Nov 2022 20:21:21 +0100 (CET) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id A67C332009FA; Mon, 14 Nov 2022 14:21:19 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Mon, 14 Nov 2022 14:21:20 -0500 Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 14 Nov 2022 14:21:18 -0500 (EST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 84ff66a9-6451-11ed-8fd2-01056ac49cbb DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= invisiblethingslab.com; h=cc:cc:content-transfer-encoding :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; t=1668453679; x=1668540079; bh=iC8thv4qLJ 7PkjWlqKqVMAcrz0XwAyc92QmmRYSHSqo=; b=DfNwMY2jrVVxMvhWOZDiHwPFCw Z6nNtugf72yWtJ9drXdixom2Gc6+QcrYmOI+XvwOxgn51o9lxuti5Qnq+GB3Pc/1 KrP9R3fylTRGXeE1HK38tBhl04RjsGtREI5N/uAaSf9WPCIEOAsDjgZnj3t7nmfZ wmhi77fZDMeO2RZVcDrZQNrVBpE7A3eHn+7/bLN3UvviJKjXpIwWf1is/7iteXWz aUOsWg28i1V9F9KdvZ801urx17Mg+nVbtrt7k6OC+c7tKW5Er7hFrUf2ibyFJzQN eFvxKimzDKYX3/YdjlQicxRJ2Q6qSk9vMZEciwBEBSRJPCD9SWTqdWT5IM1Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1668453679; x= 1668540079; bh=iC8thv4qLJ7PkjWlqKqVMAcrz0XwAyc92QmmRYSHSqo=; b=E MiT0XDkGlxu3+U2zuoZOOk7Eypuez6xQwfRO663ycRpyPNPYSkTfMABA6Vg2VJCh SlDIDclXEC43gQJt36KkziG/Yl1I9VzdfbaFFeuEKedEht5yy0JMzimGq99qctdF c0IH6keNPAluVm4GYkRAXPibsI83dIRwVmyvNCgbvuObsf+ZWQYMHbEIaIcrMBri zi0tPf62daDhGaaaoSxM0aVgL2Cz+N2sgycgLHZMhHRHuvYQlUKiec2F3sZxFT+z AvaQplrKnZnD5yofGyt22DsG5UItSarHgkF5sEStqew05q2bwS3Zvbjt7MAw1W/c B92MSy47bkakVleKTcmRg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvgedrgedvgdeliecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfgggtgfesthekredtredtjeenucfhrhhomhepofgrrhgv khcuofgrrhgtiiihkhhofihskhhiqdfikphrvggtkhhiuceomhgrrhhmrghrvghksehinh hvihhsihgslhgvthhhihhnghhslhgrsgdrtghomheqnecuggftrfgrthhtvghrnhepgfeu udehgfdvfeehhedujeehfeduveeugefhkefhheelgeevudetueeiudfggfffnecuvehluh hsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepmhgrrhhmrghrvghk sehinhhvihhsihgslhgvthhhihhnghhslhgrsgdrtghomh X-ME-Proxy: Feedback-ID: i1568416f:Fastmail From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Cc: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= , Jan Beulich , Andrew Cooper , =?utf-8?q?Roger_Pau_Monn=C3=A9?= , Wei Liu Subject: [PATCH 2/2] x86/msi: Allow writes to registers on the same page as MSI-X table Date: Mon, 14 Nov 2022 20:21:00 +0100 Message-Id: <20221114192100.1539267-2-marmarek@invisiblethingslab.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221114192100.1539267-1-marmarek@invisiblethingslab.com> References: <20221114192100.1539267-1-marmarek@invisiblethingslab.com> MIME-Version: 1.0 Some devices (notably Intel Wifi 6 AX210 card) keep auxiliary registers on the same page as MSI-X table. Device model (especially one in stubdomain) cannot really handle those, as direct writes to that page is refused (page is on mmio_ro_ranges list). Instead, add internal ioreq server that handle those writes. This may be also used to read Pending Bit Array, if it lives on the same page, making QEMU not needing /dev/mem access at all (especially helpful with lockdown enabled in dom0). If PBA lives on another page, it can be (and will be) mapped to the guest directly. If PBA lives on the same page, forbid writes. Technically, writes outside of PBA could be allowed, but at this moment the precise location of PBA isn't saved. Signed-off-by: Marek Marczykowski-Górecki --- xen/arch/x86/hvm/vmsi.c | 135 +++++++++++++++++++++++++++++++++ xen/arch/x86/include/asm/msi.h | 1 + xen/arch/x86/msi.c | 21 +++++ 3 files changed, 157 insertions(+) diff --git a/xen/arch/x86/hvm/vmsi.c b/xen/arch/x86/hvm/vmsi.c index ba4cf46dfe91..57cfcf70741e 100644 --- a/xen/arch/x86/hvm/vmsi.c +++ b/xen/arch/x86/hvm/vmsi.c @@ -428,6 +428,133 @@ static const struct hvm_io_ops msixtbl_mmio_ops = { .write = _msixtbl_write, }; +static void __iomem *msixtbl_page_handler_get_hwaddr( + const struct vcpu *v, + uint64_t address, + bool write) +{ + struct domain *d = v->domain; + struct pci_dev *pdev = NULL; + struct msixtbl_entry *entry; + void __iomem *ret = NULL; + uint64_t table_end_addr; + + rcu_read_lock(&msixtbl_rcu_lock); + /* + * Check if it's on the same page as the end of the MSI-X table, but + * outside of the table itself. + */ + list_for_each_entry( entry, &d->arch.hvm.msixtbl_list, list ) + if ( PFN_DOWN(address) == PFN_DOWN(entry->gtable + entry->table_len) && + address >= entry->gtable + entry->table_len ) + { + pdev = entry->pdev; + break; + } + rcu_read_unlock(&msixtbl_rcu_lock); + + if ( !pdev ) + return NULL; + + ASSERT( pdev->msix ); + + table_end_addr = (pdev->msix->table.first << PAGE_SHIFT) + + pdev->msix->nr_entries * PCI_MSIX_ENTRY_SIZE; + ASSERT( PFN_DOWN(table_end_addr) == pdev->msix->table.last ); + + /* If PBA lives on the same page too, forbid writes. */ + if ( write && pdev->msix->table.last == pdev->msix->pba.first ) + return NULL; + + if ( pdev->msix->last_table_page ) + ret = pdev->msix->last_table_page + (address & (PAGE_SIZE - 1)); + else + gdprintk(XENLOG_WARNING, + "MSI-X last_table_page not initialized for %04x:%02x:%02x.%u\n", + pdev->seg, + pdev->bus, + PCI_SLOT(pdev->devfn), + PCI_FUNC(pdev->devfn)); + + return ret; +} + +static bool cf_check msixtbl_page_accept( + const struct hvm_io_handler *handler, const ioreq_t *r) +{ + unsigned long addr = r->addr; + + ASSERT( r->type == IOREQ_TYPE_COPY ); + + return msixtbl_page_handler_get_hwaddr( + current, addr, r->dir == IOREQ_WRITE); +} + +static int cf_check msixtbl_page_read( + const struct hvm_io_handler *handler, + uint64_t address, uint32_t len, uint64_t *pval) +{ + void __iomem *hwaddr = msixtbl_page_handler_get_hwaddr( + current, address, false); + + if ( !hwaddr ) + return X86EMUL_UNHANDLEABLE; + + switch ( len ) { + case 1: + *pval = readb(hwaddr); + break; + case 2: + *pval = readw(hwaddr); + break; + case 4: + *pval = readl(hwaddr); + break; + case 8: + *pval = readq(hwaddr); + break; + default: + return X86EMUL_UNHANDLEABLE; + } + return X86EMUL_OKAY; +} + +static int cf_check msixtbl_page_write( + const struct hvm_io_handler *handler, + uint64_t address, uint32_t len, uint64_t val) +{ + void __iomem *hwaddr = msixtbl_page_handler_get_hwaddr( + current, address, true); + + if ( !hwaddr ) + return X86EMUL_UNHANDLEABLE; + + switch ( len ) { + case 1: + writeb(val, hwaddr); + break; + case 2: + writew(val, hwaddr); + break; + case 4: + writel(val, hwaddr); + break; + case 8: + writeq(val, hwaddr); + break; + default: + return X86EMUL_UNHANDLEABLE; + } + return X86EMUL_OKAY; + +} + +static const struct hvm_io_ops msixtbl_mmio_page_ops = { + .accept = msixtbl_page_accept, + .read = msixtbl_page_read, + .write = msixtbl_page_write, +}; + static void add_msixtbl_entry(struct domain *d, struct pci_dev *pdev, uint64_t gtable, @@ -583,6 +710,14 @@ void msixtbl_init(struct domain *d) handler->type = IOREQ_TYPE_COPY; handler->ops = &msixtbl_mmio_ops; } + + /* passthrough access to other registers on the same page */ + handler = hvm_next_io_handler(d); + if ( handler ) + { + handler->type = IOREQ_TYPE_COPY; + handler->ops = &msixtbl_mmio_page_ops; + } } void msixtbl_pt_cleanup(struct domain *d) diff --git a/xen/arch/x86/include/asm/msi.h b/xen/arch/x86/include/asm/msi.h index fe670895eed2..d4287140f04c 100644 --- a/xen/arch/x86/include/asm/msi.h +++ b/xen/arch/x86/include/asm/msi.h @@ -236,6 +236,7 @@ struct arch_msix { } table, pba; int table_refcnt[MAX_MSIX_TABLE_PAGES]; int table_idx[MAX_MSIX_TABLE_PAGES]; + void __iomem *last_table_page; spinlock_t table_lock; bool host_maskall, guest_maskall; domid_t warned; diff --git a/xen/arch/x86/msi.c b/xen/arch/x86/msi.c index d0bf63df1def..e7fe41f424d8 100644 --- a/xen/arch/x86/msi.c +++ b/xen/arch/x86/msi.c @@ -961,6 +961,21 @@ static int msix_capability_init(struct pci_dev *dev, domain_crash(d); /* XXX How to deal with existing mappings? */ } + + /* + * If the MSI-X table doesn't span full page(s), map the last page for + * passthrough accesses. + */ + if ( (msix->nr_entries * PCI_MSIX_ENTRY_SIZE) & (PAGE_SIZE - 1) ) + { + uint64_t entry_paddr = table_paddr + msix->nr_entries * PCI_MSIX_ENTRY_SIZE; + int idx = msix_get_fixmap(msix, table_paddr, entry_paddr); + + if ( idx >= 0 ) + msix->last_table_page = fix_to_virt(idx); + else + gprintk(XENLOG_ERR, "Failed to map last MSI-X table page: %d\n", idx); + } } WARN_ON(msix->table.first != (table_paddr >> PAGE_SHIFT)); ++msix->used_entries; @@ -1090,6 +1105,12 @@ static void _pci_cleanup_msix(struct arch_msix *msix) WARN(); msix->table.first = 0; msix->table.last = 0; + if ( msix->last_table_page ) + { + msix_put_fixmap(msix, + virt_to_fix((unsigned long)msix->last_table_page)); + msix->last_table_page = 0; + } if ( rangeset_remove_range(mmio_ro_ranges, msix->pba.first, msix->pba.last) )