From patchwork Thu Sep 30 07:52:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12527559 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A7A5C433EF for ; Thu, 30 Sep 2021 07:52:41 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C990B61452 for ; Thu, 30 Sep 2021 07:52:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org C990B61452 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.199499.353602 (Exim 4.92) (envelope-from ) id 1mVqrp-0001Q2-DW; Thu, 30 Sep 2021 07:52:33 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 199499.353602; Thu, 30 Sep 2021 07:52:33 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqrp-0001Pt-A7; Thu, 30 Sep 2021 07:52:33 +0000 Received: by outflank-mailman (input) for mailman id 199499; Thu, 30 Sep 2021 07:52:32 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqro-00017f-4Y for xen-devel@lists.xenproject.org; Thu, 30 Sep 2021 07:52:32 +0000 Received: from mail-lf1-x129.google.com (unknown [2a00:1450:4864:20::129]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id c0cba9af-d264-4507-8780-0fae988df9d3; Thu, 30 Sep 2021 07:52:27 +0000 (UTC) Received: by mail-lf1-x129.google.com with SMTP id j5so16802840lfg.8 for ; Thu, 30 Sep 2021 00:52:27 -0700 (PDT) Received: from localhost.localdomain (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id 13sm251099ljf.51.2021.09.30.00.52.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 00:52:25 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: c0cba9af-d264-4507-8780-0fae988df9d3 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SYPt7pXJxe63RSclQtSSHNmQ1mlJJN2741FNOcAd+O0=; b=mmTcp6QmfMwKYK8Ecm2Edk6w8zGbwZLtEjMPv51QDEdZ+/3BWqGxipsHqJioA8AD+X G3ak1zkWZvnjKpRTiNDLAzXNhxPype90V/3cZJuA1MaMck62kHkq2dZ6lFDA3DSO3GJI md3ktNg1lOpedGpEu6B9dOaK5dCwN7RysoAJHmt2URAL24egrXeHkWlYV6xvbcYgCTeI Us9GIm8dupHRsyGaojXSXOPSJe5EF5K1tXmtRxBKrpz0l+kILm93Muf6DbGQiNb52WeI MhL8Yd2Knzn0mHaB2u3fD9RkBicAmppviwtWL2u/0tEVROx2igk4cy4ftabtb5zsQmVY mK7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SYPt7pXJxe63RSclQtSSHNmQ1mlJJN2741FNOcAd+O0=; b=N0UzxToz2+Qu3odDCXTdVn1qf39LYhwBp2rtZF4zIZZEZ+PU3n4KnHQoVFStNO/S35 Sh3+mksvwjVptWM2L5xa2ewyYuW42BfMU+riSVP2LAxiSBsB6LGuV5ovNeQi4f4dlqox hPXQpaz5ay/LnwIOJLaoPkOi7NY0CKn5DeJ4dNj8gaTyWkBuugagkUPDBNUB7j16v++2 oyaAXxyDME+XboYjv/V+SmGXvDWq9zXkFSossU6hm/XefEoXE/PTInffbrD7yBFoLfkW RA9kZVem+OXwnGyovtUmzUrcGGkBG273vituH8r4ZZwvWddmRNn80K0CLYScwWp3hP0s Fqlw== X-Gm-Message-State: AOAM532AGQCzyVBez6afEDQzG0f1ePyy6KwrBkNcYS2BdEiLGiRKbRWT BzdPLQiBqeIXMm/DBO/HFfi1HGknmR0= X-Google-Smtp-Source: ABdhPJy0PnfpM6dmqJ2wGT69DJkwD/EOCR2NGBPp6wOubDrxAQpc5CiVAiUU5b60uCf9xGGlkW7xfw== X-Received: by 2002:a05:6512:1691:: with SMTP id bu17mr4290276lfb.401.1632988345818; Thu, 30 Sep 2021 00:52:25 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko , Michal Orzel Subject: [PATCH v3 01/11] vpci: Make vpci registers removal a dedicated function Date: Thu, 30 Sep 2021 10:52:13 +0300 Message-Id: <20210930075223.860329-2-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210930075223.860329-1-andr2000@gmail.com> References: <20210930075223.860329-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko This is in preparation for dynamic assignment of the vpci register handlers depending on the domain: hwdom or guest. Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Michal Orzel --- Since v1: - constify struct pci_dev where possible --- xen/drivers/vpci/vpci.c | 7 ++++++- xen/include/xen/vpci.h | 2 ++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index cbd1bac7fc33..1666402d55b8 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -35,7 +35,7 @@ extern vpci_register_init_t *const __start_vpci_array[]; extern vpci_register_init_t *const __end_vpci_array[]; #define NUM_VPCI_INIT (__end_vpci_array - __start_vpci_array) -void vpci_remove_device(struct pci_dev *pdev) +void vpci_remove_device_registers(const struct pci_dev *pdev) { spin_lock(&pdev->vpci->lock); while ( !list_empty(&pdev->vpci->handlers) ) @@ -48,6 +48,11 @@ void vpci_remove_device(struct pci_dev *pdev) xfree(r); } spin_unlock(&pdev->vpci->lock); +} + +void vpci_remove_device(struct pci_dev *pdev) +{ + vpci_remove_device_registers(pdev); xfree(pdev->vpci->msix); xfree(pdev->vpci->msi); xfree(pdev->vpci); diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 9f5b5d52e159..2e910d0b1f90 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -28,6 +28,8 @@ int __must_check vpci_add_handlers(struct pci_dev *dev); /* Remove all handlers and free vpci related structures. */ void vpci_remove_device(struct pci_dev *pdev); +/* Remove all handlers for the device given. */ +void vpci_remove_device_registers(const struct pci_dev *pdev); /* Add/remove a register handler. */ int __must_check vpci_add_register(struct vpci *vpci, From patchwork Thu Sep 30 07:52:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12527561 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1340C433FE for ; Thu, 30 Sep 2021 07:52:46 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8AE3E61452 for ; Thu, 30 Sep 2021 07:52:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 8AE3E61452 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.199501.353625 (Exim 4.92) (envelope-from ) id 1mVqrv-000233-1H; Thu, 30 Sep 2021 07:52:39 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 199501.353625; Thu, 30 Sep 2021 07:52:39 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqru-00022r-TZ; Thu, 30 Sep 2021 07:52:38 +0000 Received: by outflank-mailman (input) for mailman id 199501; Thu, 30 Sep 2021 07:52:37 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqrt-00017f-4s for xen-devel@lists.xenproject.org; Thu, 30 Sep 2021 07:52:37 +0000 Received: from mail-lf1-x129.google.com (unknown [2a00:1450:4864:20::129]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 31dbb497-6e8f-4456-bbcc-c77af33a62b1; Thu, 30 Sep 2021 07:52:28 +0000 (UTC) Received: by mail-lf1-x129.google.com with SMTP id u18so21640519lfd.12 for ; Thu, 30 Sep 2021 00:52:28 -0700 (PDT) Received: from localhost.localdomain (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id 13sm251099ljf.51.2021.09.30.00.52.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 00:52:26 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 31dbb497-6e8f-4456-bbcc-c77af33a62b1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7n1ziOkiQyIV9eus199iv91YK2oXQWN+eMmfapyz5UQ=; b=lxxe9SG314t15/TZPQBDOd4EQv7HddnV6SUX5ZEEwxxkkQT436vPQX0Xbltf0RLnvL ppnWhUbLWnELFmajNIbuSPJfPV6vK+41CSvAxQh9YrG+hmR2t8cKdbhnHfb71yauhfGq mqhx4oklREJFi9EDKdEcauypiX1t40XJ3lL7t8a9BPkB2Ub8HL2i3OIOp7vJFwNAO4G8 Be+o5fbPa3/Z1E/saDKs+dxR4aHFUpPs1HXABqnszlcs4kb5a+LiDzyMQzP2rNjGHIxl 3MSlHrlxYi3P3d49i2Wlz9Lk4cALL+4kmVo/3Fkj8FLVg2HKNFrzf/Jb9idlhCWVjwrv cgfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7n1ziOkiQyIV9eus199iv91YK2oXQWN+eMmfapyz5UQ=; b=qAIlXeF9ZnaOiG5PW8OZIR7WtPjRs8tOgFrrERI9u/LjmCwJdhEUWMgaLAsMcTlqXC Ottu/JD+kO7QGtEbnlusHy32Mp8wJIhHfOegn3n0LBLMQtLpE8qzG2xQ/JJCK+wBdwH8 UC0SJP0sY6kWOleuklc8ct/qBAebRvZy/Zj03s+ZNBamyr7Z3q4uSUGWmSbbAz+N/U+3 ifdA6PgiAPhpB1M8/Tl19mt8bLeJjJM1F+Yqg23ZmyEKBkknsZijnr6o8KXoaSG0EgZN jPgVzRn6FekDwSWOARglFHvZSZmtsUGpN5v6SoivImUi5OMa/lysNfT/Eg11IdnPEo3h /oWw== X-Gm-Message-State: AOAM533l+l04VojrA0YtzSUYiUF2a1t/gckSr9yrQBTwPhHufxDxeTct 89vS47y9cPKxMJBBsYqHdzghbIurKv4= X-Google-Smtp-Source: ABdhPJwdvPHmCq0g0ujvcf3lB0FqKYqVUjUSP/uNbltsIKz6JKcCHEqRNXIjrl7bvplAx1UBUHQHIA== X-Received: by 2002:ac2:4d10:: with SMTP id r16mr4307930lfi.309.1632988346764; Thu, 30 Sep 2021 00:52:26 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v3 02/11] vpci: Add hooks for PCI device assign/de-assign Date: Thu, 30 Sep 2021 10:52:14 +0300 Message-Id: <20210930075223.860329-3-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210930075223.860329-1-andr2000@gmail.com> References: <20210930075223.860329-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko When a PCI device gets assigned/de-assigned some work on vPCI side needs to be done for that device. Introduce a pair of hooks so vPCI can handle that. Please note, that in the current design the error path is handled by the toolstack via XEN_DOMCTL_assign_device/XEN_DOMCTL_deassign_device, so this is why it is acceptable not to de-assign devices if vPCI's assign fails, e.g. the roll back will be handled on deassign_device when it is called by the toolstack. Signed-off-by: Oleksandr Andrushchenko --- Since v2: - define CONFIG_HAS_VPCI_GUEST_SUPPORT so dead code is not compiled for x86 Since v1: - constify struct pci_dev where possible - do not open code is_system_domain() - extended the commit message --- xen/drivers/Kconfig | 4 ++++ xen/drivers/passthrough/pci.c | 9 +++++++++ xen/drivers/vpci/vpci.c | 23 +++++++++++++++++++++++ xen/include/xen/vpci.h | 20 ++++++++++++++++++++ 4 files changed, 56 insertions(+) diff --git a/xen/drivers/Kconfig b/xen/drivers/Kconfig index db94393f47a6..780490cf8e39 100644 --- a/xen/drivers/Kconfig +++ b/xen/drivers/Kconfig @@ -15,4 +15,8 @@ source "drivers/video/Kconfig" config HAS_VPCI bool +config HAS_VPCI_GUEST_SUPPORT + bool + depends on HAS_VPCI + endmenu diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 9f804a50e780..805ab86ed555 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -870,6 +870,10 @@ static int deassign_device(struct domain *d, uint16_t seg, uint8_t bus, if ( ret ) goto out; + ret = vpci_deassign_device(d, pdev); + if ( ret ) + goto out; + if ( pdev->domain == hardware_domain ) pdev->quarantine = false; @@ -1429,6 +1433,11 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag) rc = hd->platform_ops->assign_device(d, devfn, pci_to_dev(pdev), flag); } + if ( rc ) + goto done; + + rc = vpci_assign_device(d, pdev); + done: if ( rc ) printk(XENLOG_G_WARNING "%pd: assign (%pp) failed (%d)\n", diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 1666402d55b8..0fe86cb30d23 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -86,6 +86,29 @@ int __hwdom_init vpci_add_handlers(struct pci_dev *pdev) return rc; } + +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +/* Notify vPCI that device is assigned to guest. */ +int vpci_assign_device(struct domain *d, const struct pci_dev *dev) +{ + /* It only makes sense to assign for hwdom or guest domain. */ + if ( is_system_domain(d) || !has_vpci(d) ) + return 0; + + return 0; +} + +/* Notify vPCI that device is de-assigned from guest. */ +int vpci_deassign_device(struct domain *d, const struct pci_dev *dev) +{ + /* It only makes sense to de-assign from hwdom or guest domain. */ + if ( is_system_domain(d) || !has_vpci(d) ) + return 0; + + return 0; +} +#endif /* CONFIG_HAS_VPCI_GUEST_SUPPORT */ + #endif /* __XEN__ */ static int vpci_register_cmp(const struct vpci_register *r1, diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 2e910d0b1f90..ecc08f2c0f65 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -242,6 +242,26 @@ static inline bool vpci_process_pending(struct vcpu *v) } #endif +#if defined(CONFIG_HAS_VPCI) && defined(CONFIG_HAS_VPCI_GUEST_SUPPORT) +/* Notify vPCI that device is assigned/de-assigned to/from guest. */ +int __must_check vpci_assign_device(struct domain *d, + const struct pci_dev *dev); +int __must_check vpci_deassign_device(struct domain *d, + const struct pci_dev *dev); +#else +static inline int vpci_assign_device(struct domain *d, + const struct pci_dev *dev) +{ + return 0; +}; + +static inline int vpci_deassign_device(struct domain *d, + const struct pci_dev *dev) +{ + return 0; +}; +#endif + #endif /* From patchwork Thu Sep 30 07:52:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12527563 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BF12C433EF for ; Thu, 30 Sep 2021 07:52:50 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2A54061527 for ; Thu, 30 Sep 2021 07:52:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 2A54061527 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.199502.353636 (Exim 4.92) (envelope-from ) id 1mVqrz-0002V9-Ei; Thu, 30 Sep 2021 07:52:43 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 199502.353636; Thu, 30 Sep 2021 07:52:43 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqrz-0002Uq-9F; Thu, 30 Sep 2021 07:52:43 +0000 Received: by outflank-mailman (input) for mailman id 199502; Thu, 30 Sep 2021 07:52:42 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqry-00017f-5G for xen-devel@lists.xenproject.org; Thu, 30 Sep 2021 07:52:42 +0000 Received: from mail-lf1-x135.google.com (unknown [2a00:1450:4864:20::135]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id a220379f-c5e5-450e-bfa6-05edd90ae1a3; Thu, 30 Sep 2021 07:52:29 +0000 (UTC) Received: by mail-lf1-x135.google.com with SMTP id y26so21640122lfa.11 for ; Thu, 30 Sep 2021 00:52:29 -0700 (PDT) Received: from localhost.localdomain (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id 13sm251099ljf.51.2021.09.30.00.52.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 00:52:27 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: a220379f-c5e5-450e-bfa6-05edd90ae1a3 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TDiVVp/85Hu7qvLwikbqJ56SNZPH692g3qboO1XLKAU=; b=gq+EMwDmDw1J4eB8taY5f3+FOEntO+piewAuZeunn1C+ww+Gpe4FN6A4SMAuc96WSi WtQb6Bg6gpOBRKJNNFp/GdB3g/+CvNwEKOkYRPT+HN2l3zvRpuel0N2cCvaeJfo1T/k1 YFpWFuOYmjRVr2Ee5iap9oNkIwElPkIaVMFIKflEslmtC+vym0qPnktCmae2FHOiacOr 2AuMTN6SlkIBODhZSgOzQZfCJVkyKafhUqestf/egmoF+TfG5jrdIOJPPx7sqzz2VPiP TIb3PHHBsyEVScDfFPrGm0R+4TXfcCcm91GNOCLD58aT7vvr8Y1QsCDVd6eR5MDPNEwG x+sQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TDiVVp/85Hu7qvLwikbqJ56SNZPH692g3qboO1XLKAU=; b=LR1rSyifsQztEPc0TgMjEqRgA4ybpWVlQyoWxTw0gG0jLcapWJE1pT2D/+ilkQRAYn Jcd2EPU8I6lehQ43VIxt76Ywc5pvN9y0nAnusB67styG/+2RiRT06dwX/MaL4ir0VcN3 RWYK4FwtysE87GPlu4kpMolMl5aMIqyCsr2oMerDS8eBkWFWqxL3woc2PERtquOeEwuV RT3rZYRaVLLYtiZixhn3O1Sdm/tDIX11UXKa+mvP4OhnFjuf+/8jjlSw4jvIgvGFxXoD Lj+hoJ4LyT98C8vof2cUkx6TmJgCCuiHK+1kWtYhRhlssVdOKQpW1g9KuP4TJ0iI83M5 +/WQ== X-Gm-Message-State: AOAM532DC6hEznrqSCLl4bubWtZjT+3Ip5rRLViZOAugqWTvZqd5QXMn ckDHuWbzNtZ5yqMRbNuGFYm0qoC4yxQ= X-Google-Smtp-Source: ABdhPJwpuWaTde9OK4X7N00bG/MnIC2YJvlxlY2pxfCMW92/HGzZzZm71JgEX/xw/8AO8tnK5RzPZQ== X-Received: by 2002:a05:6512:31cb:: with SMTP id j11mr4500919lfe.299.1632988347903; Thu, 30 Sep 2021 00:52:27 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v3 03/11] vpci/header: Move register assignments from init_bars Date: Thu, 30 Sep 2021 10:52:15 +0300 Message-Id: <20210930075223.860329-4-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210930075223.860329-1-andr2000@gmail.com> References: <20210930075223.860329-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko This is in preparation for dynamic assignment of the vPCI register handlers depending on the domain: hwdom or guest. The need for this step is that it is easier to have all related functionality put at one place. When the subsequent patches add decisions on which handlers to install, e.g. hwdom or guest handlers, then this is easily achievable. Signed-off-by: Oleksandr Andrushchenko --- Since v1: - constify struct pci_dev where possible - extend patch description --- xen/drivers/vpci/header.c | 83 ++++++++++++++++++++++++++------------- 1 file changed, 56 insertions(+), 27 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index f8cd55e7c024..3d571356397a 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -445,6 +445,55 @@ static void rom_write(const struct pci_dev *pdev, unsigned int reg, rom->addr = val & PCI_ROM_ADDRESS_MASK; } +static int add_bar_handlers(const struct pci_dev *pdev) +{ + unsigned int i; + struct vpci_header *header = &pdev->vpci->header; + struct vpci_bar *bars = header->bars; + int rc; + + /* Setup a handler for the command register. */ + rc = vpci_add_register(pdev->vpci, vpci_hw_read16, cmd_write, PCI_COMMAND, + 2, header); + if ( rc ) + return rc; + + if ( pdev->ignore_bars ) + return 0; + + for ( i = 0; i < PCI_HEADER_NORMAL_NR_BARS + 1; i++ ) + { + if ( (bars[i].type == VPCI_BAR_IO) || (bars[i].type == VPCI_BAR_EMPTY) ) + continue; + + if ( bars[i].type == VPCI_BAR_ROM ) + { + unsigned int rom_reg; + uint8_t header_type = pci_conf_read8(pdev->sbdf, + PCI_HEADER_TYPE) & 0x7f; + if ( header_type == PCI_HEADER_TYPE_NORMAL ) + rom_reg = PCI_ROM_ADDRESS; + else + rom_reg = PCI_ROM_ADDRESS1; + rc = vpci_add_register(pdev->vpci, vpci_hw_read32, rom_write, + rom_reg, 4, &bars[i]); + if ( rc ) + return rc; + } + else + { + uint8_t reg = PCI_BASE_ADDRESS_0 + i * 4; + + /* This is either VPCI_BAR_MEM32 or VPCI_BAR_MEM64_{LO|HI}. */ + rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, + 4, &bars[i]); + if ( rc ) + return rc; + } + } + return 0; +} + static int init_bars(struct pci_dev *pdev) { uint16_t cmd; @@ -470,14 +519,8 @@ static int init_bars(struct pci_dev *pdev) return -EOPNOTSUPP; } - /* Setup a handler for the command register. */ - rc = vpci_add_register(pdev->vpci, vpci_hw_read16, cmd_write, PCI_COMMAND, - 2, header); - if ( rc ) - return rc; - if ( pdev->ignore_bars ) - return 0; + return add_bar_handlers(pdev); /* Disable memory decoding before sizing. */ cmd = pci_conf_read16(pdev->sbdf, PCI_COMMAND); @@ -492,14 +535,6 @@ static int init_bars(struct pci_dev *pdev) if ( i && bars[i - 1].type == VPCI_BAR_MEM64_LO ) { bars[i].type = VPCI_BAR_MEM64_HI; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, - 4, &bars[i]); - if ( rc ) - { - pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); - return rc; - } - continue; } @@ -532,14 +567,6 @@ static int init_bars(struct pci_dev *pdev) bars[i].addr = addr; bars[i].size = size; bars[i].prefetchable = val & PCI_BASE_ADDRESS_MEM_PREFETCH; - - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, 4, - &bars[i]); - if ( rc ) - { - pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); - return rc; - } } /* Check expansion ROM. */ @@ -553,11 +580,13 @@ static int init_bars(struct pci_dev *pdev) rom->addr = addr; header->rom_enabled = pci_conf_read32(pdev->sbdf, rom_reg) & PCI_ROM_ADDRESS_ENABLE; + } - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, rom_write, rom_reg, - 4, rom); - if ( rc ) - rom->type = VPCI_BAR_EMPTY; + rc = add_bar_handlers(pdev); + if ( rc ) + { + pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); + return rc; } return (cmd & PCI_COMMAND_MEMORY) ? modify_bars(pdev, cmd, false) : 0; From patchwork Thu Sep 30 07:52:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12527565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB001C433FE for ; Thu, 30 Sep 2021 07:52:56 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5FCBC613CE for ; Thu, 30 Sep 2021 07:52:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 5FCBC613CE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.199505.353647 (Exim 4.92) (envelope-from ) id 1mVqs4-00032P-O3; Thu, 30 Sep 2021 07:52:48 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 199505.353647; Thu, 30 Sep 2021 07:52:48 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqs4-00032G-KW; Thu, 30 Sep 2021 07:52:48 +0000 Received: by outflank-mailman (input) for mailman id 199505; Thu, 30 Sep 2021 07:52:47 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqs3-00017f-59 for xen-devel@lists.xenproject.org; Thu, 30 Sep 2021 07:52:47 +0000 Received: from mail-lf1-x12e.google.com (unknown [2a00:1450:4864:20::12e]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 85314c14-274d-4084-803c-7360f29354b5; Thu, 30 Sep 2021 07:52:30 +0000 (UTC) Received: by mail-lf1-x12e.google.com with SMTP id z24so21538335lfu.13 for ; Thu, 30 Sep 2021 00:52:30 -0700 (PDT) Received: from localhost.localdomain (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id 13sm251099ljf.51.2021.09.30.00.52.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 00:52:28 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 85314c14-274d-4084-803c-7360f29354b5 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+lvSnXHbydzAtV18JqDZDz/Q4JuLo0GbEWE975rivV4=; b=RvBF4VUM1Fk+E+WSGa+N7HuxNUkCF1yYamFAn+WCByN3h1ck/uPr2ThjCz3Htwvszv JD7MB/R4pYL7vd/scgYTgNkJQNgCjwrlCskorlXeGcafl9saDZeKdSlCK5sx/iGgPWvY FkluY894cb3vxLxpgDS+LhXM5DGQFst3R2A2zGIicFMY2V7MJBsqu2XOTQxDLsd6H5pM t1WQYQPTn9FHiTj+oWpuU/ilwnQXWAgf3VsRffW0cmEJT83rP2+h6O3y50MoDxSQrDCI C7t/ulzm7EjOmIoaSP5sIFPxPdDCdYSQ0tw1XpQfjFH7PR5oAVZvTbzkejD+3t+qJxQp d5ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+lvSnXHbydzAtV18JqDZDz/Q4JuLo0GbEWE975rivV4=; b=bbEmUZqfG+Et0o4TqJfZoETZ3j1sfaPnfQtC8JPViwOC1MmlocMDkNl1ZlV7X/92n8 0oVUpug0pvdlv85Yy2CvXwwtT7lCS7TMvRgN30XkBwZcSC/m/Aq2Z0HfJ2FdL63eoypu ipoomkjank1MhNl0n0RQijxB08e3Zb29ItFXpb4RM8X6KFBiJqcwuQyfEiNxOLmAHUcy tTfBn8z+KSV5L+uz3RMO7dbaflnCVI96QQIa3uxUA5itrnweDXLDAliYM4SNiTOjDG65 CDSFsCLh8tfzMtTCyjg7yACmThSzqgwdktchlHHmeqI+daFESFPP5yqJBtu6H865HL1Q jIgg== X-Gm-Message-State: AOAM530eR/hB3bG2uFbWDeQXOAPHE/Hn7DKEtKcHACX/RiRc1EVZ15gr s5OkVFC4/pFccrlmLyQcIsSPyQbkFGY= X-Google-Smtp-Source: ABdhPJz0Z246UwdyliQVLaHOSo2T5zlzJH0n8U8wqH0wwto1FrUvKVB6rbV1sDwYhVfQlVl9ywO6xQ== X-Received: by 2002:a05:6512:b8f:: with SMTP id b15mr4596943lfv.655.1632988348949; Thu, 30 Sep 2021 00:52:28 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v3 04/11] vpci/header: Add and remove register handlers dynamically Date: Thu, 30 Sep 2021 10:52:16 +0300 Message-Id: <20210930075223.860329-5-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210930075223.860329-1-andr2000@gmail.com> References: <20210930075223.860329-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Add relevant vpci register handlers when assigning PCI device to a domain and remove those when de-assigning. This allows having different handlers for different domains, e.g. hwdom and other guests. Use stubs for guest domains for now. Signed-off-by: Oleksandr Andrushchenko --- Since v2: - remove unneeded ifdefs for CONFIG_HAS_VPCI_GUEST_SUPPORT as more code has been eliminated from being built on x86 Since v1: - constify struct pci_dev where possible - do not open code is_system_domain() - simplify some code3. simplify - use gdprintk + error code instead of gprintk - gate vpci_bar_{add|remove}_handlers with CONFIG_HAS_VPCI_GUEST_SUPPORT, so these do not get compiled for x86 - removed unneeded is_system_domain check --- xen/drivers/vpci/header.c | 72 ++++++++++++++++++++++++++++++++++----- xen/drivers/vpci/vpci.c | 4 +-- xen/include/xen/vpci.h | 8 +++++ 3 files changed, 74 insertions(+), 10 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 3d571356397a..1ce98795fcca 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -397,6 +397,17 @@ static void bar_write(const struct pci_dev *pdev, unsigned int reg, pci_conf_write32(pdev->sbdf, reg, val); } +static void guest_bar_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t val, void *data) +{ +} + +static uint32_t guest_bar_read(const struct pci_dev *pdev, unsigned int reg, + void *data) +{ + return 0xffffffff; +} + static void rom_write(const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { @@ -445,14 +456,25 @@ static void rom_write(const struct pci_dev *pdev, unsigned int reg, rom->addr = val & PCI_ROM_ADDRESS_MASK; } -static int add_bar_handlers(const struct pci_dev *pdev) +static void guest_rom_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t val, void *data) +{ +} + +static uint32_t guest_rom_read(const struct pci_dev *pdev, unsigned int reg, + void *data) +{ + return 0xffffffff; +} + +static int add_bar_handlers(const struct pci_dev *pdev, bool is_hwdom) { unsigned int i; struct vpci_header *header = &pdev->vpci->header; struct vpci_bar *bars = header->bars; int rc; - /* Setup a handler for the command register. */ + /* Setup a handler for the command register: same for hwdom and guests. */ rc = vpci_add_register(pdev->vpci, vpci_hw_read16, cmd_write, PCI_COMMAND, 2, header); if ( rc ) @@ -475,8 +497,13 @@ static int add_bar_handlers(const struct pci_dev *pdev) rom_reg = PCI_ROM_ADDRESS; else rom_reg = PCI_ROM_ADDRESS1; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, rom_write, - rom_reg, 4, &bars[i]); + if ( is_hwdom ) + rc = vpci_add_register(pdev->vpci, vpci_hw_read32, rom_write, + rom_reg, 4, &bars[i]); + else + rc = vpci_add_register(pdev->vpci, + guest_rom_read, guest_rom_write, + rom_reg, 4, &bars[i]); if ( rc ) return rc; } @@ -485,8 +512,13 @@ static int add_bar_handlers(const struct pci_dev *pdev) uint8_t reg = PCI_BASE_ADDRESS_0 + i * 4; /* This is either VPCI_BAR_MEM32 or VPCI_BAR_MEM64_{LO|HI}. */ - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, - 4, &bars[i]); + if ( is_hwdom ) + rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, + reg, 4, &bars[i]); + else + rc = vpci_add_register(pdev->vpci, + guest_bar_read, guest_bar_write, + reg, 4, &bars[i]); if ( rc ) return rc; } @@ -520,7 +552,7 @@ static int init_bars(struct pci_dev *pdev) } if ( pdev->ignore_bars ) - return add_bar_handlers(pdev); + return add_bar_handlers(pdev, true); /* Disable memory decoding before sizing. */ cmd = pci_conf_read16(pdev->sbdf, PCI_COMMAND); @@ -582,7 +614,7 @@ static int init_bars(struct pci_dev *pdev) PCI_ROM_ADDRESS_ENABLE; } - rc = add_bar_handlers(pdev); + rc = add_bar_handlers(pdev, true); if ( rc ) { pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); @@ -593,6 +625,30 @@ static int init_bars(struct pci_dev *pdev) } REGISTER_VPCI_INIT(init_bars, VPCI_PRIORITY_MIDDLE); +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +int vpci_bar_add_handlers(const struct domain *d, const struct pci_dev *pdev) +{ + int rc; + + /* Remove previously added registers. */ + vpci_remove_device_registers(pdev); + + rc = add_bar_handlers(pdev, is_hardware_domain(d)); + if ( rc ) + gdprintk(XENLOG_ERR, + "%pp: failed to add BAR handlers for dom%pd: %d\n", + &pdev->sbdf, d, rc); + return rc; +} + +int vpci_bar_remove_handlers(const struct domain *d, const struct pci_dev *pdev) +{ + /* Remove previously added registers. */ + vpci_remove_device_registers(pdev); + return 0; +} +#endif + /* * Local variables: * mode: C diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 0fe86cb30d23..702f7b5d5dda 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -95,7 +95,7 @@ int vpci_assign_device(struct domain *d, const struct pci_dev *dev) if ( is_system_domain(d) || !has_vpci(d) ) return 0; - return 0; + return vpci_bar_add_handlers(d, dev); } /* Notify vPCI that device is de-assigned from guest. */ @@ -105,7 +105,7 @@ int vpci_deassign_device(struct domain *d, const struct pci_dev *dev) if ( is_system_domain(d) || !has_vpci(d) ) return 0; - return 0; + return vpci_bar_remove_handlers(d, dev); } #endif /* CONFIG_HAS_VPCI_GUEST_SUPPORT */ diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index ecc08f2c0f65..fd822c903af5 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -57,6 +57,14 @@ uint32_t vpci_hw_read32(const struct pci_dev *pdev, unsigned int reg, */ bool __must_check vpci_process_pending(struct vcpu *v); +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +/* Add/remove BAR handlers for a domain. */ +int vpci_bar_add_handlers(const struct domain *d, + const struct pci_dev *pdev); +int vpci_bar_remove_handlers(const struct domain *d, + const struct pci_dev *pdev); +#endif + struct vpci { /* List of vPCI handlers for a device. */ struct list_head handlers; From patchwork Thu Sep 30 07:52:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12527567 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F50CC433F5 for ; Thu, 30 Sep 2021 07:53:01 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BD90A61452 for ; Thu, 30 Sep 2021 07:53:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org BD90A61452 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.199509.353658 (Exim 4.92) (envelope-from ) id 1mVqsA-0003aI-4H; Thu, 30 Sep 2021 07:52:54 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 199509.353658; Thu, 30 Sep 2021 07:52:54 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqs9-0003a6-VA; Thu, 30 Sep 2021 07:52:53 +0000 Received: by outflank-mailman (input) for mailman id 199509; Thu, 30 Sep 2021 07:52:52 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqs8-00017f-5R for xen-devel@lists.xenproject.org; Thu, 30 Sep 2021 07:52:52 +0000 Received: from mail-lf1-x134.google.com (unknown [2a00:1450:4864:20::134]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id fac7a3cc-26c2-4c47-b457-e24184136f4b; Thu, 30 Sep 2021 07:52:31 +0000 (UTC) Received: by mail-lf1-x134.google.com with SMTP id i25so21624009lfg.6 for ; Thu, 30 Sep 2021 00:52:31 -0700 (PDT) Received: from localhost.localdomain (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id 13sm251099ljf.51.2021.09.30.00.52.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 00:52:29 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: fac7a3cc-26c2-4c47-b457-e24184136f4b DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rVtZpNhlXnSzmfbhCEL+IMfyjOh3bF01cvc201QQbmU=; b=hE2uDfRL1hRLRQVqes0DgETsiFrCAaEESlXXIrx14t2k2qtoniLHSCW3Vh+IJBx0sV jR0frYyDX02OCrbrTE0+xVaPoiG7IBmARxKBGlO5xYlEvM1AHpt0VP/5OC6FZ+KhNcMi gLauGCou1utr94NVGYrqj+DOhs2vnmXf1MkyUH27TcN8WpAPi2tgfH33I2CQAnD49Isu HsGrimm/gnCM8k41GqEEOr4htxMX71j4IjAfrqNaEroHyZf1hWcqdhMig7NbiSTdwNiI wCs6hRdLHt0TNGcleVYUxDr/4DzDcZ7eMqnNAytWzR8U8xqPWjt4FEM1wIY5Q7LmCJac EYPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rVtZpNhlXnSzmfbhCEL+IMfyjOh3bF01cvc201QQbmU=; b=glbScFWEZDA+TYr+elkg1ROQANJyYANvl8qMqP8kY3kbIpJjiVnkM73aTq34QOGHMJ 4PcyX7YNLn8xRBSlVTQCdReRF9MfiuuJUmlGQgqiTu+s9y8nfPtjhXNZwPP4Fe8a0Pjv RCdje38P/o5nwjNoTfIriQr1wVByAtKNGPo0K7Z6wdaBE8PXA5Cn+4tgcCgdjr9lHnYV aUW8ld7pwyeMJKUT+BjUXdHWpxg4K7O/cHwHhwtiiB+/YXjmB0A7pN+kbIWvwTuWg0xj gYO6y8YQqqoNaRBR6sxoi5I1ctqWlSNf/xjSDW039bImdMcXF5l/yB0XnuCH+Td/vj+f 49KQ== X-Gm-Message-State: AOAM531WkyQTPqB+nP4DVOAQis7qOhK2W1NAi9Q9/nXL1+/+k0XigMAR FUXMunkiHtfzmkdbLbsQR1qOVwkeu38= X-Google-Smtp-Source: ABdhPJy3FaYBUB50uoi6zzCEs665+xGWg3WT1AJusJrIM7oB7ma6dHvZ/NwDWDbmw2T0DPxHmRZ1Kw== X-Received: by 2002:a05:6512:12d6:: with SMTP id p22mr4595737lfg.42.1632988350031; Thu, 30 Sep 2021 00:52:30 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko , Michal Orzel Subject: [PATCH v3 05/11] vpci/header: Implement guest BAR register handlers Date: Thu, 30 Sep 2021 10:52:17 +0300 Message-Id: <20210930075223.860329-6-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210930075223.860329-1-andr2000@gmail.com> References: <20210930075223.860329-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Emulate guest BAR register values: this allows creating a guest view of the registers and emulates size and properties probe as it is done during PCI device enumeration by the guest. ROM BAR is only handled for the hardware domain and for guest domains there is a stub: at the moment PCI expansion ROM is x86 only, so it might not be used by other architectures without emulating x86. Other use-cases may include using that expansion ROM before Xen boots, hence no emulation is needed in Xen itself. Or when a guest wants to use the ROM code which seems to be rare. Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Michal Orzel Reviewed-by: Jan Beulich --- Since v1: - re-work guest read/write to be much simpler and do more work on write than read which is expected to be called more frequently - removed one too obvious comment Signed-off-by: Oleksandr Andrushchenko --- xen/drivers/vpci/header.c | 30 +++++++++++++++++++++++++++++- xen/include/xen/vpci.h | 3 +++ 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 1ce98795fcca..ec4d215f36ff 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -400,12 +400,38 @@ static void bar_write(const struct pci_dev *pdev, unsigned int reg, static void guest_bar_write(const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { + struct vpci_bar *bar = data; + bool hi = false; + + if ( bar->type == VPCI_BAR_MEM64_HI ) + { + ASSERT(reg > PCI_BASE_ADDRESS_0); + bar--; + hi = true; + } + else + { + val &= PCI_BASE_ADDRESS_MEM_MASK; + val |= bar->type == VPCI_BAR_MEM32 ? PCI_BASE_ADDRESS_MEM_TYPE_32 + : PCI_BASE_ADDRESS_MEM_TYPE_64; + val |= bar->prefetchable ? PCI_BASE_ADDRESS_MEM_PREFETCH : 0; + } + + bar->guest_addr &= ~(0xffffffffull << (hi ? 32 : 0)); + bar->guest_addr |= (uint64_t)val << (hi ? 32 : 0); + + bar->guest_addr &= ~(bar->size - 1) | ~PCI_BASE_ADDRESS_MEM_MASK; } static uint32_t guest_bar_read(const struct pci_dev *pdev, unsigned int reg, void *data) { - return 0xffffffff; + const struct vpci_bar *bar = data; + + if ( bar->type == VPCI_BAR_MEM64_HI ) + return bar->guest_addr >> 32; + + return bar->guest_addr; } static void rom_write(const struct pci_dev *pdev, unsigned int reg, @@ -522,6 +548,8 @@ static int add_bar_handlers(const struct pci_dev *pdev, bool is_hwdom) if ( rc ) return rc; } + + bars[i].guest_addr = 0; } return 0; } diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index fd822c903af5..a0320b22cb36 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -75,7 +75,10 @@ struct vpci { struct vpci_header { /* Information about the PCI BARs of this device. */ struct vpci_bar { + /* Physical view of the BAR. */ uint64_t addr; + /* Guest view of the BAR. */ + uint64_t guest_addr; uint64_t size; enum { VPCI_BAR_EMPTY, From patchwork Thu Sep 30 07:52:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12527569 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE48AC433EF for ; Thu, 30 Sep 2021 07:53:06 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A789661452 for ; Thu, 30 Sep 2021 07:53:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org A789661452 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.199512.353669 (Exim 4.92) (envelope-from ) id 1mVqsF-00049J-Df; Thu, 30 Sep 2021 07:52:59 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 199512.353669; Thu, 30 Sep 2021 07:52:59 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqsF-000497-87; Thu, 30 Sep 2021 07:52:59 +0000 Received: by outflank-mailman (input) for mailman id 199512; Thu, 30 Sep 2021 07:52:57 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqsD-00017f-5g for xen-devel@lists.xenproject.org; Thu, 30 Sep 2021 07:52:57 +0000 Received: from mail-lf1-x132.google.com (unknown [2a00:1450:4864:20::132]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id b232defa-7f1f-4f21-a814-d8a592319df9; Thu, 30 Sep 2021 07:52:32 +0000 (UTC) Received: by mail-lf1-x132.google.com with SMTP id e15so21624994lfr.10 for ; Thu, 30 Sep 2021 00:52:32 -0700 (PDT) Received: from localhost.localdomain (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id 13sm251099ljf.51.2021.09.30.00.52.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 00:52:30 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: b232defa-7f1f-4f21-a814-d8a592319df9 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UYb/ANGHKxQ/isBZ8jDr5PIwHDVyjvEKH2Wdujmg3Lc=; b=WsHQwrDr4Uem1Ds+1pFynepoQjyxeZ3K137PcLJmgDm7Mrrm1d4dj6crZNwwcehEsM rGxflMhwppxr+6Em7Ci3Czif9QiZf01ignkNZku1A2kDqHxa5dAstqUNVHxYq4DrTdAe FA6fxmriL3u0ohz55idRcIQB7Hof+Ysk65j+f8vxzl9zHHj/ESrkw3INkTBsub6Nc2uN ZgEi2Q2Y+Uqdgi7FhVG5x/sfpr1/QFBKHfNzcXPy4itnubQpBs585W8SvHSQF0HeBWNK wRF+reCZw69rw/RCSITlGRJwLzXYfOEH4H0fLtextUzkDroqcXaPYTayGMsYeLMXoG5M tfiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UYb/ANGHKxQ/isBZ8jDr5PIwHDVyjvEKH2Wdujmg3Lc=; b=4zAgsL6XanX8+iKOPhsRXjsQlYX/hUouKk6oyNSNTbs6sb8Rl5/b1TFX+WGj6CQ04t 6602IAcLo65bYVIWWmNAkBiSJwyw9Iw8o1cKQBDogATvh3dACclIxwFFCD/VyfBE2ZYI JQeBOLMvJd+T1BLFPvIA5T4FnyL1NrKbw7JRvMLA3Ng+rnSzgmNa39lgmsRuQkmm+urB 3fA9SFbay3JoPM3ifX6n4GJqSNcajW098UjhYaofQhCaFCln6yFxbIDYJDKbhaUuTpNN B7ga0FmEYwq2qfVg6C8PhFxmjjfDIZODhirwWtdoM/ZmNgnG3/2FmgCargyb6G35vGsI /9wg== X-Gm-Message-State: AOAM530O8wjJY05sKi9HCdsCe+GuNhpnXAA0V5SrKyb0WQIlTHiFx2Sf B7pducG1RTtofNuGpj3lWTJONTp0g68= X-Google-Smtp-Source: ABdhPJxtPd0OLutL4StYyqSyghgjofxpr4rhGLOzhT+xWd9gDCurFN9nBS2/pZyCBlB8otMRHCex5g== X-Received: by 2002:a05:6512:c9:: with SMTP id c9mr4363608lfp.142.1632988351070; Thu, 30 Sep 2021 00:52:31 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v3 06/11] vpci/header: Handle p2m range sets per BAR Date: Thu, 30 Sep 2021 10:52:18 +0300 Message-Id: <20210930075223.860329-7-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210930075223.860329-1-andr2000@gmail.com> References: <20210930075223.860329-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Instead of handling a single range set, that contains all the memory regions of all the BARs and ROM, have them per BAR. This is in preparation of making non-identity mappings in p2m for the MMIOs/ROM. Signed-off-by: Oleksandr Andrushchenko --- xen/drivers/vpci/header.c | 172 ++++++++++++++++++++++++++------------ xen/include/xen/vpci.h | 3 +- 2 files changed, 122 insertions(+), 53 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index ec4d215f36ff..9c603d26d302 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -131,49 +131,75 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd, bool vpci_process_pending(struct vcpu *v) { - if ( v->vpci.mem ) + if ( v->vpci.num_mem_ranges ) { struct map_data data = { .d = v->domain, .map = v->vpci.cmd & PCI_COMMAND_MEMORY, }; - int rc = rangeset_consume_ranges(v->vpci.mem, map_range, &data); + struct pci_dev *pdev = v->vpci.pdev; + struct vpci_header *header = &pdev->vpci->header; + unsigned int i; - if ( rc == -ERESTART ) - return true; + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; + int rc; - spin_lock(&v->vpci.pdev->vpci->lock); - /* Disable memory decoding unconditionally on failure. */ - modify_decoding(v->vpci.pdev, - rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : v->vpci.cmd, - !rc && v->vpci.rom_only); - spin_unlock(&v->vpci.pdev->vpci->lock); + if ( !bar->mem ) + continue; - rangeset_destroy(v->vpci.mem); - v->vpci.mem = NULL; - if ( rc ) - /* - * FIXME: in case of failure remove the device from the domain. - * Note that there might still be leftover mappings. While this is - * safe for Dom0, for DomUs the domain will likely need to be - * killed in order to avoid leaking stale p2m mappings on - * failure. - */ - vpci_remove_device(v->vpci.pdev); + rc = rangeset_consume_ranges(bar->mem, map_range, &data); + + if ( rc == -ERESTART ) + return true; + + spin_lock(&pdev->vpci->lock); + /* Disable memory decoding unconditionally on failure. */ + modify_decoding(pdev, + rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : v->vpci.cmd, + !rc && v->vpci.rom_only); + spin_unlock(&pdev->vpci->lock); + + rangeset_destroy(bar->mem); + bar->mem = NULL; + v->vpci.num_mem_ranges--; + if ( rc ) + /* + * FIXME: in case of failure remove the device from the domain. + * Note that there might still be leftover mappings. While this is + * safe for Dom0, for DomUs the domain will likely need to be + * killed in order to avoid leaking stale p2m mappings on + * failure. + */ + vpci_remove_device(pdev); + } } return false; } static int __init apply_map(struct domain *d, const struct pci_dev *pdev, - struct rangeset *mem, uint16_t cmd) + uint16_t cmd) { struct map_data data = { .d = d, .map = true }; - int rc; + struct vpci_header *header = &pdev->vpci->header; + int rc = 0; + unsigned int i; + + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; - while ( (rc = rangeset_consume_ranges(mem, map_range, &data)) == -ERESTART ) - process_pending_softirqs(); - rangeset_destroy(mem); + if ( !bar->mem ) + continue; + + while ( (rc = rangeset_consume_ranges(bar->mem, map_range, + &data)) == -ERESTART ) + process_pending_softirqs(); + rangeset_destroy(bar->mem); + bar->mem = NULL; + } if ( !rc ) modify_decoding(pdev, cmd, false); @@ -181,7 +207,7 @@ static int __init apply_map(struct domain *d, const struct pci_dev *pdev, } static void defer_map(struct domain *d, struct pci_dev *pdev, - struct rangeset *mem, uint16_t cmd, bool rom_only) + uint16_t cmd, bool rom_only, uint8_t num_mem_ranges) { struct vcpu *curr = current; @@ -192,9 +218,9 @@ static void defer_map(struct domain *d, struct pci_dev *pdev, * started for the same device if the domain is not well-behaved. */ curr->vpci.pdev = pdev; - curr->vpci.mem = mem; curr->vpci.cmd = cmd; curr->vpci.rom_only = rom_only; + curr->vpci.num_mem_ranges = num_mem_ranges; /* * Raise a scheduler softirq in order to prevent the guest from resuming * execution with pending mapping operations, to trigger the invocation @@ -206,42 +232,47 @@ static void defer_map(struct domain *d, struct pci_dev *pdev, static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) { struct vpci_header *header = &pdev->vpci->header; - struct rangeset *mem = rangeset_new(NULL, NULL, 0); struct pci_dev *tmp, *dev = NULL; const struct vpci_msix *msix = pdev->vpci->msix; - unsigned int i; + unsigned int i, j; int rc; - - if ( !mem ) - return -ENOMEM; + uint8_t num_mem_ranges; /* - * Create a rangeset that represents the current device BARs memory region + * Create a rangeset per BAR that represents the current device memory region * and compare it against all the currently active BAR memory regions. If * an overlap is found, subtract it from the region to be mapped/unmapped. * - * First fill the rangeset with all the BARs of this device or with the ROM + * First fill the rangesets with all the BARs of this device or with the ROM * BAR only, depending on whether the guest is toggling the memory decode * bit of the command register, or the enable bit of the ROM BAR register. */ for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) { - const struct vpci_bar *bar = &header->bars[i]; + struct vpci_bar *bar = &header->bars[i]; unsigned long start = PFN_DOWN(bar->addr); unsigned long end = PFN_DOWN(bar->addr + bar->size - 1); + bar->mem = NULL; + if ( !MAPPABLE_BAR(bar) || (rom_only ? bar->type != VPCI_BAR_ROM : (bar->type == VPCI_BAR_ROM && !header->rom_enabled)) ) continue; - rc = rangeset_add_range(mem, start, end); + bar->mem = rangeset_new(NULL, NULL, 0); + if ( !bar->mem ) + { + rc = -ENOMEM; + goto fail; + } + + rc = rangeset_add_range(bar->mem, start, end); if ( rc ) { printk(XENLOG_G_WARNING "Failed to add [%lx, %lx]: %d\n", start, end, rc); - rangeset_destroy(mem); - return rc; + goto fail; } } @@ -252,14 +283,21 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) unsigned long end = PFN_DOWN(vmsix_table_addr(pdev->vpci, i) + vmsix_table_size(pdev->vpci, i) - 1); - rc = rangeset_remove_range(mem, start, end); - if ( rc ) + for ( j = 0; j < ARRAY_SIZE(header->bars); j++ ) { - printk(XENLOG_G_WARNING - "Failed to remove MSIX table [%lx, %lx]: %d\n", - start, end, rc); - rangeset_destroy(mem); - return rc; + const struct vpci_bar *bar = &header->bars[j]; + + if ( !bar->mem ) + continue; + + rc = rangeset_remove_range(bar->mem, start, end); + if ( rc ) + { + printk(XENLOG_G_WARNING + "Failed to remove MSIX table [%lx, %lx]: %d\n", + start, end, rc); + goto fail; + } } } @@ -291,7 +329,8 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) unsigned long start = PFN_DOWN(bar->addr); unsigned long end = PFN_DOWN(bar->addr + bar->size - 1); - if ( !bar->enabled || !rangeset_overlaps_range(mem, start, end) || + if ( !bar->enabled || + !rangeset_overlaps_range(bar->mem, start, end) || /* * If only the ROM enable bit is toggled check against other * BARs in the same device for overlaps, but not against the @@ -300,13 +339,12 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) (rom_only && tmp == pdev && bar->type == VPCI_BAR_ROM) ) continue; - rc = rangeset_remove_range(mem, start, end); + rc = rangeset_remove_range(bar->mem, start, end); if ( rc ) { printk(XENLOG_G_WARNING "Failed to remove [%lx, %lx]: %d\n", start, end, rc); - rangeset_destroy(mem); - return rc; + goto fail; } } } @@ -324,12 +362,42 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) * will always be to establish mappings and process all the BARs. */ ASSERT((cmd & PCI_COMMAND_MEMORY) && !rom_only); - return apply_map(pdev->domain, pdev, mem, cmd); + return apply_map(pdev->domain, pdev, cmd); } - defer_map(dev->domain, dev, mem, cmd, rom_only); + /* Find out how many memory ranges has left after MSI and overlaps. */ + num_mem_ranges = 0; + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; + + if ( !rangeset_is_empty(bar->mem) ) + num_mem_ranges++; + } + + /* + * There are cases when PCI device, root port for example, has neither + * memory space nor IO. In this case PCI command register write is + * missed resulting in the underlying PCI device not functional, so: + * - if there are no regions write the command register now + * - if there are regions then defer work and write later on + */ + if ( !num_mem_ranges ) + pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); + else + defer_map(dev->domain, dev, cmd, rom_only, num_mem_ranges); return 0; + +fail: + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; + + rangeset_destroy(bar->mem); + bar->mem = NULL; + } + return rc; } static void cmd_write(const struct pci_dev *pdev, unsigned int reg, diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index a0320b22cb36..352e02d0106d 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -80,6 +80,7 @@ struct vpci { /* Guest view of the BAR. */ uint64_t guest_addr; uint64_t size; + struct rangeset *mem; enum { VPCI_BAR_EMPTY, VPCI_BAR_IO, @@ -154,9 +155,9 @@ struct vpci { struct vpci_vcpu { /* Per-vcpu structure to store state while {un}mapping of PCI BARs. */ - struct rangeset *mem; struct pci_dev *pdev; uint16_t cmd; + uint8_t num_mem_ranges; bool rom_only : 1; }; From patchwork Thu Sep 30 07:52:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12527571 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7C5AC433F5 for ; Thu, 30 Sep 2021 07:53:10 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8E25E61527 for ; Thu, 30 Sep 2021 07:53:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 8E25E61527 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.199515.353680 (Exim 4.92) (envelope-from ) id 1mVqsJ-0004kS-Su; Thu, 30 Sep 2021 07:53:03 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 199515.353680; Thu, 30 Sep 2021 07:53:03 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqsJ-0004kB-PD; Thu, 30 Sep 2021 07:53:03 +0000 Received: by outflank-mailman (input) for mailman id 199515; Thu, 30 Sep 2021 07:53:02 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqsI-00017f-5o for xen-devel@lists.xenproject.org; Thu, 30 Sep 2021 07:53:02 +0000 Received: from mail-lf1-x131.google.com (unknown [2a00:1450:4864:20::131]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id e1af39ec-0bab-4c47-980f-847a81a9d1a4; Thu, 30 Sep 2021 07:52:33 +0000 (UTC) Received: by mail-lf1-x131.google.com with SMTP id u18so21641295lfd.12 for ; Thu, 30 Sep 2021 00:52:33 -0700 (PDT) Received: from localhost.localdomain (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id 13sm251099ljf.51.2021.09.30.00.52.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 00:52:31 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: e1af39ec-0bab-4c47-980f-847a81a9d1a4 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=R0Ou982lM6u92fk8c4IDhUUqsj1bOg2PvsoDj71PTZg=; b=HsfB4Yy6HLz+R+wFcmkgwTUbBCHnW6SkBpWvFqaBK2rAHPBJrw+xeCSH9432+DJZo9 5ve06kx9L3dX7UtDZKq9DA3nlIi+bE14CIY97SbUeqz6i03Z+ohPE3k/beNTmLdolK/A oPfJd7wVYViAm5aNhgGsL+P2IVLL4FnXl0D3CFUcwIdOU/c3OfZiSxwE4Qmq4Hw18HUC U1GbiFMFKmbGs5jrbj/BMKDR61jpPpfjdkPLZdwyU7R+bsxohUhkL+RQDuKKgMlhHzVz hY3NPrrGoFrgkWyUzztwtk2tTgCkuWp9zS/RmsNONtd1nRCdlfWBLfkk1sXdN3z2Ocjj AnMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=R0Ou982lM6u92fk8c4IDhUUqsj1bOg2PvsoDj71PTZg=; b=LHSMg9E1DeReWQlODAsw4mEQRzO7Fchdkl89r8vF45fbHB7q7xCFfl2yZW5EjyNbN/ mUSOYdvK7tdqgT2VSY/T2OynokkR7/hFmItJUXWE/uFOq/hDL9kd82UkVtj0jE+tpDkQ EsHFaf5fuQaH70CjyWLyUeEwArhNQwgXfT+xxtn5VGmZ9pedIdyfulFa/gE+m+e8M4aS E3RWgqekfnkzpjtjaAy2s/GqvnhqgmCSjEpOLrx8JvF0qO3XUs1njEvh8/518hlpsDRA +ujBFrO80/76xcBSsfHAdlQrXzhOGDy5nPpqsBvz8B9MwnEfBtnBaIniSPmvjtROQDgJ alWg== X-Gm-Message-State: AOAM531nOLfC8j/QX3xh8988aPlJc3GnU51fMVFqj8qutyyOUJyDmHJR 5JOAfKKZmuycx4nqSbHmNi8y089ZvTs= X-Google-Smtp-Source: ABdhPJwNTxH3rY7fVFK+T2n7C8LTqtOBuO5NPVO/Zedb0cpTLVSylMe3TUoK8Zrs6YggTeOiaDSe7Q== X-Received: by 2002:ac2:4c81:: with SMTP id d1mr4566179lfl.340.1632988352180; Thu, 30 Sep 2021 00:52:32 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v3 07/11] vpci/header: program p2m with guest BAR view Date: Thu, 30 Sep 2021 10:52:19 +0300 Message-Id: <20210930075223.860329-8-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210930075223.860329-1-andr2000@gmail.com> References: <20210930075223.860329-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Take into account guest's BAR view and program its p2m accordingly: gfn is guest's view of the BAR and mfn is the physical BAR value as set up by the host bridge in the hardware domain. This way hardware doamin sees physical BAR values and guest sees emulated ones. Signed-off-by: Oleksandr Andrushchenko --- Since v2: - improve readability for data.start_gfn and restructure ?: construct Since v1: - s/MSI/MSI-X in comments --- xen/drivers/vpci/header.c | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 9c603d26d302..f23c956cde6c 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -30,6 +30,10 @@ struct map_data { struct domain *d; + /* Start address of the BAR as seen by the guest. */ + gfn_t start_gfn; + /* Physical start address of the BAR. */ + mfn_t start_mfn; bool map; }; @@ -37,12 +41,28 @@ static int map_range(unsigned long s, unsigned long e, void *data, unsigned long *c) { const struct map_data *map = data; + gfn_t start_gfn; int rc; for ( ; ; ) { unsigned long size = e - s + 1; + /* + * Any BAR may have holes in its memory we want to map, e.g. + * we don't want to map MSI-X regions which may be a part of that BAR, + * e.g. when a single BAR is used for both MMIO and MSI-X. + * In this case MSI-X regions are subtracted from the mapping, but + * map->start_gfn still points to the very beginning of the BAR. + * So if there is a hole present then we need to adjust start_gfn + * to reflect the fact of that substraction. + */ + start_gfn = gfn_add(map->start_gfn, s - mfn_x(map->start_mfn)); + + printk(XENLOG_G_DEBUG + "%smap [%lx, %lx] -> %#"PRI_gfn" for d%d\n", + map->map ? "" : "un", s, e, gfn_x(start_gfn), + map->d->domain_id); /* * ARM TODOs: * - On ARM whether the memory is prefetchable or not should be passed @@ -52,8 +72,10 @@ static int map_range(unsigned long s, unsigned long e, void *data, * - {un}map_mmio_regions doesn't support preemption. */ - rc = map->map ? map_mmio_regions(map->d, _gfn(s), size, _mfn(s)) - : unmap_mmio_regions(map->d, _gfn(s), size, _mfn(s)); + rc = map->map ? map_mmio_regions(map->d, start_gfn, + size, _mfn(s)) + : unmap_mmio_regions(map->d, start_gfn, + size, _mfn(s)); if ( rc == 0 ) { *c += size; @@ -69,6 +91,7 @@ static int map_range(unsigned long s, unsigned long e, void *data, ASSERT(rc < size); *c += rc; s += rc; + gfn_add(map->start_gfn, rc); if ( general_preempt_check() ) return -ERESTART; } @@ -149,6 +172,10 @@ bool vpci_process_pending(struct vcpu *v) if ( !bar->mem ) continue; + data.start_gfn = + _gfn(PFN_DOWN(is_hardware_domain(v->vpci.pdev->domain) + ? bar->addr : bar->guest_addr)); + data.start_mfn = _mfn(PFN_DOWN(bar->addr)); rc = rangeset_consume_ranges(bar->mem, map_range, &data); if ( rc == -ERESTART ) @@ -194,6 +221,9 @@ static int __init apply_map(struct domain *d, const struct pci_dev *pdev, if ( !bar->mem ) continue; + data.start_gfn = _gfn(PFN_DOWN(is_hardware_domain(d) + ? bar->addr : bar->guest_addr)); + data.start_mfn = _mfn(PFN_DOWN(bar->addr)); while ( (rc = rangeset_consume_ranges(bar->mem, map_range, &data)) == -ERESTART ) process_pending_softirqs(); From patchwork Thu Sep 30 07:52:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12527573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B094C433EF for ; Thu, 30 Sep 2021 07:53:20 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2FCF061527 for ; Thu, 30 Sep 2021 07:53:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 2FCF061527 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.199521.353691 (Exim 4.92) (envelope-from ) id 1mVqsT-0005X2-7c; Thu, 30 Sep 2021 07:53:13 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 199521.353691; Thu, 30 Sep 2021 07:53:13 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqsT-0005Wt-3j; Thu, 30 Sep 2021 07:53:13 +0000 Received: by outflank-mailman (input) for mailman id 199521; Thu, 30 Sep 2021 07:53:12 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqsS-00017f-6C for xen-devel@lists.xenproject.org; Thu, 30 Sep 2021 07:53:12 +0000 Received: from mail-lf1-x12c.google.com (unknown [2a00:1450:4864:20::12c]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 9b5aaab3-e57d-4343-9b5c-247ec58c3a4c; Thu, 30 Sep 2021 07:52:34 +0000 (UTC) Received: by mail-lf1-x12c.google.com with SMTP id j5so16804013lfg.8 for ; Thu, 30 Sep 2021 00:52:34 -0700 (PDT) Received: from localhost.localdomain (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id 13sm251099ljf.51.2021.09.30.00.52.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 00:52:32 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 9b5aaab3-e57d-4343-9b5c-247ec58c3a4c DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OR+8PvgVFP8pLzhFaQ6RjiWWTesnzHH/8qQE78jWNGs=; b=h6yENr4JdtnEf1wnG93DzqvZpXcFLZuU/1n0d6/YtV6R/vfnAzlRk88b3OdwgWTGhv IInWBJMeO4gdxQHRrESr85YYKjh60ND9+7IeV680A5/04pP2uAvMtNtSXoR1CpWWVhBj udqAiXOK/yIn2GDG8DVJ9J4/NDo+zQU18k6LOP++b/03/Bk5tsVOK2LghgJBrqM4ktmS IVFAFFPSAhmWHpLrF/GPFBiIDvne2z26lgVMN542h3uSRpSvkELVtq/IuXBPn8ndYKm7 /95BPB/BGwq574gch5vj3+XDhH22/QPCchQL1UWhBWjU+KjiACBU6bHJmWOWf/J7CsQK oEew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OR+8PvgVFP8pLzhFaQ6RjiWWTesnzHH/8qQE78jWNGs=; b=ga4IY3PqtAiqUaJfqNGsZnT5frsVlxscfrZHnuSpXdNTRom8bx611UrSS6BpLsioLo SUhWCxCUwnGhYnyh59XhWEafdauQOLEtlbLJGfenF8Knz4ybAAB++i8F/pVHs2e46zHC WcvcWEGydpOOrSI2pKst27c0YGlApLo+OwIUg02IpY+0WgPuUkiuhuvtRibs2qVO4AAi BeugOlYemxY0bxWMmGDBxDkHuOjeEw5tqvOed450JW1FZkmAbgTVlGMty4CtCN/zSied f/78JbTDB70hH8lU1HccYsWEGcsM3Ik86xRidbMs1odc8iyEVsryy5RboCChsVG+xTsX Z4Vw== X-Gm-Message-State: AOAM530LeGu8010FXSg24ogihVtuOwGKLpyWlmu4WPxG9Zg/4kXUuAWW WaoajmY1OeVDw3IIHn1crgkCliAvW6A= X-Google-Smtp-Source: ABdhPJwJzt7joepxqpjEuultT1xkb7pTstL7l3EASTsUuwntiXFMaEoCuRVVohr5ZAIfI4WqJIiFfQ== X-Received: by 2002:ac2:5c50:: with SMTP id s16mr99079lfp.605.1632988353306; Thu, 30 Sep 2021 00:52:33 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko , Michal Orzel Subject: [PATCH v3 08/11] vpci/header: Emulate PCI_COMMAND register for guests Date: Thu, 30 Sep 2021 10:52:20 +0300 Message-Id: <20210930075223.860329-9-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210930075223.860329-1-andr2000@gmail.com> References: <20210930075223.860329-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Add basic emulation support for guests. At the moment only emulate PCI_COMMAND_INTX_DISABLE bit, the rest is not emulated yet and left as TODO. Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Michal Orzel --- New in v2 --- xen/drivers/vpci/header.c | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index f23c956cde6c..754aeb5a584f 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -451,6 +451,32 @@ static void cmd_write(const struct pci_dev *pdev, unsigned int reg, pci_conf_write16(pdev->sbdf, reg, cmd); } +static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t cmd, void *data) +{ + /* TODO: Add proper emulation for all bits of the command register. */ + + if ( (cmd & PCI_COMMAND_INTX_DISABLE) == 0 ) + { + /* + * Guest wants to enable INTx. It can't be enabled if: + * - host has INTx disabled + * - MSI/MSI-X enabled + */ + if ( pdev->vpci->msi->enabled ) + cmd |= PCI_COMMAND_INTX_DISABLE; + else + { + uint16_t current_cmd = pci_conf_read16(pdev->sbdf, reg); + + if ( current_cmd & PCI_COMMAND_INTX_DISABLE ) + cmd |= PCI_COMMAND_INTX_DISABLE; + } + } + + cmd_write(pdev, reg, cmd, data); +} + static void bar_write(const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { @@ -598,9 +624,12 @@ static int add_bar_handlers(const struct pci_dev *pdev, bool is_hwdom) struct vpci_bar *bars = header->bars; int rc; - /* Setup a handler for the command register: same for hwdom and guests. */ - rc = vpci_add_register(pdev->vpci, vpci_hw_read16, cmd_write, PCI_COMMAND, - 2, header); + if ( is_hwdom ) + rc = vpci_add_register(pdev->vpci, vpci_hw_read16, cmd_write, + PCI_COMMAND, 2, header); + else + rc = vpci_add_register(pdev->vpci, vpci_hw_read16, guest_cmd_write, + PCI_COMMAND, 2, header); if ( rc ) return rc; From patchwork Thu Sep 30 07:52:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12527617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C258EC433EF for ; Thu, 30 Sep 2021 07:57:39 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 92E32615A7 for ; Thu, 30 Sep 2021 07:57:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 92E32615A7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.199561.353724 (Exim 4.92) (envelope-from ) id 1mVqwe-0008ML-Ij; Thu, 30 Sep 2021 07:57:32 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 199561.353724; Thu, 30 Sep 2021 07:57:32 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqwe-0008ME-Ee; Thu, 30 Sep 2021 07:57:32 +0000 Received: by outflank-mailman (input) for mailman id 199561; Thu, 30 Sep 2021 07:57:31 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqsX-00017f-6I for xen-devel@lists.xenproject.org; Thu, 30 Sep 2021 07:53:17 +0000 Received: from mail-lf1-x12a.google.com (unknown [2a00:1450:4864:20::12a]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 9ac7d20e-d2a5-489d-acb4-b13358f83a1a; Thu, 30 Sep 2021 07:52:35 +0000 (UTC) Received: by mail-lf1-x12a.google.com with SMTP id j5so16804175lfg.8 for ; Thu, 30 Sep 2021 00:52:35 -0700 (PDT) Received: from localhost.localdomain (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id 13sm251099ljf.51.2021.09.30.00.52.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 00:52:33 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 9ac7d20e-d2a5-489d-acb4-b13358f83a1a DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kFKr9R5/eoTpWJugUvdw0hZSjF5zvjYKXnAMVRzF8Pk=; b=cWJGSCPP+epJ29KCHDh+8crG7u2S+kudHu40Amr21tBEaYnCRj4sLpj6yJlBz5lWjY bkH0E367iOZOHaRQVdwMKNzJb+JV4+Hdpvm9oZKzkFxD726NCHoDuDDfPzLQSBUpRL3L 9gFvonYsfiA/cdTdKzmi+5Ei+EzFxrwYmM4UMvr7JF5bpk214Gygsk6sXxyi9OaJBOnD gUV2hlylrGcyAW4tN/mZrbZ2AeINau62FVNJQ+e/v/u+p9CChqku8UpSjyViK6YE5vD2 xj/9U0yHM5Xhn43L9AK2JPGa9oCy5CEHeEiudzW4kf5ZRLmKNidiK4/f1BUA6xzxzKsv 39FA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kFKr9R5/eoTpWJugUvdw0hZSjF5zvjYKXnAMVRzF8Pk=; b=r4AWQ2Ooqfpm6p4JXzLGJkWw8tW67sIWlk0D+cPk2Wicb9rDHl8iWspPvc/WqNQnWI wgNI+WNDlNEEZzyGb5lL0uQx7jdewvKVc9tss0ImmQrqXqkP9hMjnvljrkolfzJgnYtw KFQtA6AJft7Axeq7Bo5selkocKYUuzAcIKFbDjKguIsV3regHtfp8a51xGpq23vp3g0Y U36o/fmNAVJjKZIC0iR3ZIhN4nmTieiLA0zTwVNJJk8AslesdPrA2BKi9VAstXhuXCG2 ZV5c46+rcQD1rzDTFniUEmXi5TwVTUzWyDat9zCVZPuPr9ZX16AnxeVEwU+sw5pZ3VqC wOVQ== X-Gm-Message-State: AOAM53095PPna5TjcukGTE0ypoywHseoZHeELUgGc6rfgQEN6MlDOPHf Zi781nuFbzJS0VPDs0+zDdDou3gG62w= X-Google-Smtp-Source: ABdhPJzNrHs+SZ/Q+mD+Sr3QROyeTeMKrWMeDO5LFmSFYlTZ1RNOC9HAByTat96meLn6Ha/BcigpWQ== X-Received: by 2002:ac2:4f02:: with SMTP id k2mr4255807lfr.378.1632988354376; Thu, 30 Sep 2021 00:52:34 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko , Michal Orzel Subject: [PATCH v3 09/11] vpci/header: Reset the command register when adding devices Date: Thu, 30 Sep 2021 10:52:21 +0300 Message-Id: <20210930075223.860329-10-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210930075223.860329-1-andr2000@gmail.com> References: <20210930075223.860329-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Reset the command register when passing through a PCI device: it is possible that when passing through a PCI device its memory decoding bits in the command register are already set. Thus, a guest OS may not write to the command register to update memory decoding, so guest mappings (guest's view of the BARs) are left not updated. Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Michal Orzel --- Since v1: - do not write 0 to the command register, but respect host settings. --- xen/drivers/vpci/header.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 754aeb5a584f..70d911b147e1 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -451,8 +451,7 @@ static void cmd_write(const struct pci_dev *pdev, unsigned int reg, pci_conf_write16(pdev->sbdf, reg, cmd); } -static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, - uint32_t cmd, void *data) +static uint32_t emulate_cmd_reg(const struct pci_dev *pdev, uint32_t cmd) { /* TODO: Add proper emulation for all bits of the command register. */ @@ -467,14 +466,20 @@ static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, cmd |= PCI_COMMAND_INTX_DISABLE; else { - uint16_t current_cmd = pci_conf_read16(pdev->sbdf, reg); + uint16_t current_cmd = pci_conf_read16(pdev->sbdf, PCI_COMMAND); if ( current_cmd & PCI_COMMAND_INTX_DISABLE ) cmd |= PCI_COMMAND_INTX_DISABLE; } } - cmd_write(pdev, reg, cmd, data); + return cmd; +} + +static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t cmd, void *data) +{ + cmd_write(pdev, reg, emulate_cmd_reg(pdev, cmd), data); } static void bar_write(const struct pci_dev *pdev, unsigned int reg, @@ -793,6 +798,10 @@ int vpci_bar_add_handlers(const struct domain *d, const struct pci_dev *pdev) gdprintk(XENLOG_ERR, "%pp: failed to add BAR handlers for dom%pd: %d\n", &pdev->sbdf, d, rc); + + /* Reset the command register with respect to host settings. */ + pci_conf_write16(pdev->sbdf, PCI_COMMAND, emulate_cmd_reg(pdev, 0)); + return rc; } From patchwork Thu Sep 30 07:52:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12527615 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85BBDC433EF for ; Thu, 30 Sep 2021 07:57:20 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3934761555 for ; Thu, 30 Sep 2021 07:57:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 3934761555 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.199537.353713 (Exim 4.92) (envelope-from ) id 1mVqwK-0007Xi-4L; Thu, 30 Sep 2021 07:57:12 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 199537.353713; Thu, 30 Sep 2021 07:57:12 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqwK-0007Xb-1D; Thu, 30 Sep 2021 07:57:12 +0000 Received: by outflank-mailman (input) for mailman id 199537; Thu, 30 Sep 2021 07:57:10 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqsc-00017f-6R for xen-devel@lists.xenproject.org; Thu, 30 Sep 2021 07:53:22 +0000 Received: from mail-lf1-x12d.google.com (unknown [2a00:1450:4864:20::12d]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 15844ac1-baf8-4ff5-8fba-69a9900d443d; Thu, 30 Sep 2021 07:52:36 +0000 (UTC) Received: by mail-lf1-x12d.google.com with SMTP id y26so21641542lfa.11 for ; Thu, 30 Sep 2021 00:52:36 -0700 (PDT) Received: from localhost.localdomain (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id 13sm251099ljf.51.2021.09.30.00.52.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 00:52:35 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 15844ac1-baf8-4ff5-8fba-69a9900d443d DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IvOUvJEBvxetANJzBoBgBMczs7Ntkr0TVyDjFYQuZ/M=; b=VoB77MfqKEQnQtKp+lPWH1kg+gpqtdmADxdUy6EQRGw1QG6Bpabmz8yFGEsugn5XlN XAyQBFqu7TT4/qCP8RqsLPEGtUuyM53MQx37PL4X84E1UzWXD+zf3bN4anHofgQL+eoE 4tL7ybx5OYRjtT8cztLSwMusaE9J+8JXxD0e0+UtX7O2lzWOnfAYbkHwWjtKM/FRSS22 lwsQBCEKwhKyG8yWEEiT7JG4LMyllM/1Mr8/vmGhABarplV3lAQgwhOjSX+qMAM0VQtD NjO9u7+bBSv0EMamA/xDsvtBA63G2LIIM4iyBYWhKAnIairyEVCZTYZHOhRXt38yyapX TX2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IvOUvJEBvxetANJzBoBgBMczs7Ntkr0TVyDjFYQuZ/M=; b=Tuynr6w9faHH7FgeePdyYoqCMHRZIYjSwVc/L2DeZPGRhjViB56NYBAQO957QEy8I+ Qiboaxdm02HK7owR9z2n8empcW70+uI4W1tIuAt9LrQqD1H35yIMoDriN4D+oTj/QZsH 3Yq157SF145aYjVCqxEQC/QZ68CM8FJ69k/P4/WZZXc9YVxWsa0fPjNs6zaqxP6ymOcc SYOZ8Uw2f6veqYdemmbVROZ0rxrNCr90iICM0jr9yHA1d324MXOy7xWdteHw5RxTprrR o5a51RIDO4OwW3/VPZmt1/bfC34sVpt2UKVSPvuhQmgqAvQBXMfFDoQz71PtD5u4idiQ xBYw== X-Gm-Message-State: AOAM533fDSe/9837sIXbggKgQL0KGLdLttcJBEIGjaKupYd0/CFpARrL PwgRJd5UGzNT4vzX/XBSfrFdXIsVqg0= X-Google-Smtp-Source: ABdhPJzHAYn0FeIUINVCTCrJ6p1OupRcFNPn66qoht+4Rh+LnDfq3NvAUqoEvvgCWqQcD7851GTPEw== X-Received: by 2002:a19:f515:: with SMTP id j21mr4274651lfb.125.1632988355472; Thu, 30 Sep 2021 00:52:35 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v3 10/11] vpci: Add initial support for virtual PCI bus topology Date: Thu, 30 Sep 2021 10:52:22 +0300 Message-Id: <20210930075223.860329-11-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210930075223.860329-1-andr2000@gmail.com> References: <20210930075223.860329-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Assign SBDF to the PCI devices being passed through with bus 0. The resulting topology is where PCIe devices reside on the bus 0 of the root complex itself (embedded endpoints). This implementation is limited to 32 devices which are allowed on a single PCI bus. Signed-off-by: Oleksandr Andrushchenko --- Since v2: - remove casts that are (a) malformed and (b) unnecessary - add new line for better readability - remove CONFIG_HAS_VPCI_GUEST_SUPPORT ifdef's as the relevant vPCI functions are now completely gated with this config - gate common code with CONFIG_HAS_VPCI_GUEST_SUPPORT New in v2 --- xen/common/domain.c | 3 ++ xen/drivers/passthrough/pci.c | 60 +++++++++++++++++++++++++++++++++++ xen/drivers/vpci/vpci.c | 14 +++++++- xen/include/xen/pci.h | 22 +++++++++++++ xen/include/xen/sched.h | 8 +++++ 5 files changed, 106 insertions(+), 1 deletion(-) diff --git a/xen/common/domain.c b/xen/common/domain.c index 40d67ec34232..e0170087612d 100644 --- a/xen/common/domain.c +++ b/xen/common/domain.c @@ -601,6 +601,9 @@ struct domain *domain_create(domid_t domid, #ifdef CONFIG_HAS_PCI INIT_LIST_HEAD(&d->pdev_list); +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + INIT_LIST_HEAD(&d->vdev_list); +#endif #endif /* All error paths can depend on the above setup. */ diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 805ab86ed555..5b963d75d1ba 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -831,6 +831,66 @@ int pci_remove_device(u16 seg, u8 bus, u8 devfn) return ret; } +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +static struct vpci_dev *pci_find_virtual_device(const struct domain *d, + const struct pci_dev *pdev) +{ + struct vpci_dev *vdev; + + list_for_each_entry ( vdev, &d->vdev_list, list ) + if ( vdev->pdev == pdev ) + return vdev; + return NULL; +} + +int pci_add_virtual_device(struct domain *d, const struct pci_dev *pdev) +{ + struct vpci_dev *vdev; + + ASSERT(!pci_find_virtual_device(d, pdev)); + + /* Each PCI bus supports 32 devices/slots at max. */ + if ( d->vpci_dev_next > 31 ) + return -ENOSPC; + + vdev = xzalloc(struct vpci_dev); + if ( !vdev ) + return -ENOMEM; + + /* We emulate a single host bridge for the guest, so segment is always 0. */ + vdev->seg = 0; + + /* + * The bus number is set to 0, so virtual devices are seen + * as embedded endpoints behind the root complex. + */ + vdev->bus = 0; + vdev->devfn = PCI_DEVFN(d->vpci_dev_next++, 0); + + vdev->pdev = pdev; + vdev->domain = d; + + pcidevs_lock(); + list_add_tail(&vdev->list, &d->vdev_list); + pcidevs_unlock(); + + return 0; +} + +int pci_remove_virtual_device(struct domain *d, const struct pci_dev *pdev) +{ + struct vpci_dev *vdev; + + pcidevs_lock(); + vdev = pci_find_virtual_device(d, pdev); + if ( vdev ) + list_del(&vdev->list); + pcidevs_unlock(); + xfree(vdev); + return 0; +} +#endif /* CONFIG_HAS_VPCI_GUEST_SUPPORT */ + /* Caller should hold the pcidevs_lock */ static int deassign_device(struct domain *d, uint16_t seg, uint8_t bus, uint8_t devfn) diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 702f7b5d5dda..d787f13e679e 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -91,20 +91,32 @@ int __hwdom_init vpci_add_handlers(struct pci_dev *pdev) /* Notify vPCI that device is assigned to guest. */ int vpci_assign_device(struct domain *d, const struct pci_dev *dev) { + int rc; + /* It only makes sense to assign for hwdom or guest domain. */ if ( is_system_domain(d) || !has_vpci(d) ) return 0; - return vpci_bar_add_handlers(d, dev); + rc = vpci_bar_add_handlers(d, dev); + if ( rc ) + return rc; + + return pci_add_virtual_device(d, dev); } /* Notify vPCI that device is de-assigned from guest. */ int vpci_deassign_device(struct domain *d, const struct pci_dev *dev) { + int rc; + /* It only makes sense to de-assign from hwdom or guest domain. */ if ( is_system_domain(d) || !has_vpci(d) ) return 0; + rc = pci_remove_virtual_device(d, dev); + if ( rc ) + return rc; + return vpci_bar_remove_handlers(d, dev); } #endif /* CONFIG_HAS_VPCI_GUEST_SUPPORT */ diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index 43b8a0817076..33033a3a8f8d 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -137,6 +137,24 @@ struct pci_dev { struct vpci *vpci; }; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +struct vpci_dev { + struct list_head list; + /* Physical PCI device this virtual device is connected to. */ + const struct pci_dev *pdev; + /* Virtual SBDF of the device. */ + union { + struct { + uint8_t devfn; + uint8_t bus; + uint16_t seg; + }; + pci_sbdf_t sbdf; + }; + struct domain *domain; +}; +#endif + #define for_each_pdev(domain, pdev) \ list_for_each_entry(pdev, &(domain)->pdev_list, domain_list) @@ -167,6 +185,10 @@ const unsigned long *pci_get_ro_map(u16 seg); int pci_add_device(u16 seg, u8 bus, u8 devfn, const struct pci_dev_info *, nodeid_t node); int pci_remove_device(u16 seg, u8 bus, u8 devfn); +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +int pci_add_virtual_device(struct domain *d, const struct pci_dev *pdev); +int pci_remove_virtual_device(struct domain *d, const struct pci_dev *pdev); +#endif int pci_ro_device(int seg, int bus, int devfn); int pci_hide_device(unsigned int seg, unsigned int bus, unsigned int devfn); struct pci_dev *pci_get_pdev(int seg, int bus, int devfn); diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index 28146ee404e6..ecdb04b4f7fc 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -444,6 +444,14 @@ struct domain #ifdef CONFIG_HAS_PCI struct list_head pdev_list; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + struct list_head vdev_list; + /* + * Current device number used by the virtual PCI bus topology + * to assign a unique SBDF to a passed through virtual PCI device. + */ + int vpci_dev_next; +#endif #endif #ifdef CONFIG_HAS_PASSTHROUGH From patchwork Thu Sep 30 07:52:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12527613 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22018C433F5 for ; Thu, 30 Sep 2021 07:57:20 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C3CE2615A7 for ; Thu, 30 Sep 2021 07:57:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org C3CE2615A7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.199534.353702 (Exim 4.92) (envelope-from ) id 1mVqwD-0007FK-T3; Thu, 30 Sep 2021 07:57:05 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 199534.353702; Thu, 30 Sep 2021 07:57:05 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqwD-0007FD-PV; Thu, 30 Sep 2021 07:57:05 +0000 Received: by outflank-mailman (input) for mailman id 199534; Thu, 30 Sep 2021 07:57:05 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mVqsm-00017f-6u for xen-devel@lists.xenproject.org; Thu, 30 Sep 2021 07:53:32 +0000 Received: from mail-lf1-x131.google.com (unknown [2a00:1450:4864:20::131]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id dbd70223-3b19-4d81-ba12-e4167578e2fc; Thu, 30 Sep 2021 07:52:37 +0000 (UTC) Received: by mail-lf1-x131.google.com with SMTP id y26so21641792lfa.11 for ; Thu, 30 Sep 2021 00:52:37 -0700 (PDT) Received: from localhost.localdomain (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id 13sm251099ljf.51.2021.09.30.00.52.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 00:52:36 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: dbd70223-3b19-4d81-ba12-e4167578e2fc DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QXLr0ePaWCVb8hGG2KFBTiGRR67U58CE46v0AcmCx0Q=; b=oO3fIj/xL0UQThCgy+wTmcU6bDceKmLX8Qzv6Apktl1039A22n7iyzhM4ILHZQEk+b 3U8aKwZeuanr3uJ16QoBVuY6pkBoOkibx83X/0uFmy9zb3E3MZIwJG6XgDJ4ZJVGV/Bz FSDqAYzRqJlZoEtw0kluwX3tuJZyRpGFEZi3GCuGqMYZHkmeZbKAsETl1r6hJ76UYDgo nUSBxwnxwpko//NSxwLtc5bp8fA2gnmOkLNmjRhBcwRQuFOzTZdSmUEMNYonZa4nPcSy VFRCpwji+OY4ubtBrAqGLp1523Q0c8VZ1KwIMe/C43Z/VNDuT3b06DpD86EOVul+VWeZ 6GbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QXLr0ePaWCVb8hGG2KFBTiGRR67U58CE46v0AcmCx0Q=; b=0zNYfrgLZnZGtK+CWikRoPGaA3s9rK5JuQRgRKeJONfxrEgwyS+uBpp3GeoekkW7Sj JWYwcJJR/yMRftbhEPfxqTZdLckDoz4WdI/sCV/xy81nYLzuy4+x1m847NyeVLgvpchH mMQDqSt5z6dO9i8xY/yGT/gwDT+rcbT+hVt1fceiuwM6QYyHcShKBHy/5X8Z0h1k5U8b yJnsgAYOL96tGHL+OEgNdxzQYK9jsetBy0ZxY4UW8lzkQH2cw2nRwBSTVt7aw5E4VUJI 43gd2LY4ZRr6uoARph13m/KOcBpzlKfgqCL2Vhl9G5Trhl6hA5qef24FEAFxdFt5IXDP xqIQ== X-Gm-Message-State: AOAM531nbaOzMRThh4O5LLqt3CQJ40tvzxEgR9hNv87KWiuExsCP1000 TWHXZQ9APQb3H+BeY4VL0S8VQgJPNRM= X-Google-Smtp-Source: ABdhPJw6wtvHV/qjC/Fbro+8Unmd1W1nGr/gXYTIoQK09Mru1KFJP4dAtX44NeWGQfRPa6PCRGVP4Q== X-Received: by 2002:ac2:4c42:: with SMTP id o2mr4317773lfk.504.1632988356657; Thu, 30 Sep 2021 00:52:36 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v3 11/11] xen/arm: Translate virtual PCI bus topology for guests Date: Thu, 30 Sep 2021 10:52:23 +0300 Message-Id: <20210930075223.860329-12-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210930075223.860329-1-andr2000@gmail.com> References: <20210930075223.860329-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko There are three originators for the PCI configuration space access: 1. The domain that owns physical host bridge: MMIO handlers are there so we can update vPCI register handlers with the values written by the hardware domain, e.g. physical view of the registers vs guest's view on the configuration space. 2. Guest access to the passed through PCI devices: we need to properly map virtual bus topology to the physical one, e.g. pass the configuration space access to the corresponding physical devices. 3. Emulated host PCI bridge access. It doesn't exist in the physical topology, e.g. it can't be mapped to some physical host bridge. So, all access to the host bridge itself needs to be trapped and emulated. Signed-off-by: Oleksandr Andrushchenko --- Since v2: - pass struct domain instead of struct vcpu - constify arguments where possible - gate relevant code with CONFIG_HAS_VPCI_GUEST_SUPPORT New in v2 --- xen/arch/arm/domain.c | 1 + xen/arch/arm/vpci.c | 86 +++++++++++++++++++++++++++++++---- xen/arch/arm/vpci.h | 3 ++ xen/drivers/passthrough/pci.c | 25 ++++++++++ xen/include/asm-arm/pci.h | 1 + xen/include/xen/pci.h | 1 + xen/include/xen/sched.h | 2 + 7 files changed, 111 insertions(+), 8 deletions(-) diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c index fa6fcc5e467c..095671742ad8 100644 --- a/xen/arch/arm/domain.c +++ b/xen/arch/arm/domain.c @@ -797,6 +797,7 @@ void arch_domain_destroy(struct domain *d) get_order_from_bytes(d->arch.efi_acpi_len)); #endif domain_io_free(d); + domain_vpci_free(d); } void arch_domain_shutdown(struct domain *d) diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c index 5d6c29c8dcd9..26ec2fa7cf2d 100644 --- a/xen/arch/arm/vpci.c +++ b/xen/arch/arm/vpci.c @@ -17,6 +17,14 @@ #define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff) +struct vpci_mmio_priv { + /* + * Set to true if the MMIO handlers were set up for the emulated + * ECAM host PCI bridge. + */ + bool is_virt_ecam; +}; + /* Do some sanity checks. */ static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len) { @@ -38,6 +46,7 @@ static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, pci_sbdf_t sbdf; unsigned long data = ~0UL; unsigned int size = 1U << info->dabt.size; + struct vpci_mmio_priv *priv = (struct vpci_mmio_priv *)p; sbdf.sbdf = MMCFG_BDF(info->gpa); reg = REGISTER_OFFSET(info->gpa); @@ -45,6 +54,13 @@ static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, if ( !vpci_mmio_access_allowed(reg, size) ) return 0; + /* + * For the passed through devices we need to map their virtual SBDF + * to the physical PCI device being passed through. + */ + if ( priv->is_virt_ecam && !pci_translate_virtual_device(v->domain, &sbdf) ) + return 1; + data = vpci_read(sbdf, reg, min(4u, size)); if ( size == 8 ) data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32; @@ -61,6 +77,7 @@ static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info, pci_sbdf_t sbdf; unsigned long data = r; unsigned int size = 1U << info->dabt.size; + struct vpci_mmio_priv *priv = (struct vpci_mmio_priv *)p; sbdf.sbdf = MMCFG_BDF(info->gpa); reg = REGISTER_OFFSET(info->gpa); @@ -68,6 +85,13 @@ static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info, if ( !vpci_mmio_access_allowed(reg, size) ) return 0; + /* + * For the passed through devices we need to map their virtual SBDF + * to the physical PCI device being passed through. + */ + if ( priv->is_virt_ecam && !pci_translate_virtual_device(v->domain, &sbdf) ) + return 1; + vpci_write(sbdf, reg, min(4u, size), data); if ( size == 8 ) vpci_write(sbdf, reg + 4, 4, data >> 32); @@ -80,13 +104,48 @@ static const struct mmio_handler_ops vpci_mmio_handler = { .write = vpci_mmio_write, }; +/* + * There are three originators for the PCI configuration space access: + * 1. The domain that owns physical host bridge: MMIO handlers are + * there so we can update vPCI register handlers with the values + * written by the hardware domain, e.g. physical view of the registers/ + * configuration space. + * 2. Guest access to the passed through PCI devices: we need to properly + * map virtual bus topology to the physical one, e.g. pass the configuration + * space access to the corresponding physical devices. + * 3. Emulated host PCI bridge access. It doesn't exist in the physical + * topology, e.g. it can't be mapped to some physical host bridge. + * So, all access to the host bridge itself needs to be trapped and + * emulated. + */ static int vpci_setup_mmio_handler(struct domain *d, struct pci_host_bridge *bridge) { - struct pci_config_window *cfg = bridge->cfg; + struct vpci_mmio_priv *priv; + + priv = xzalloc(struct vpci_mmio_priv); + if ( !priv ) + return -ENOMEM; + + priv->is_virt_ecam = !is_hardware_domain(d); - register_mmio_handler(d, &vpci_mmio_handler, - cfg->phys_addr, cfg->size, NULL); + if ( is_hardware_domain(d) ) + { + struct pci_config_window *cfg = bridge->cfg; + + bridge->mmio_priv = priv; + register_mmio_handler(d, &vpci_mmio_handler, + cfg->phys_addr, cfg->size, + priv); + } + else + { + d->vpci_mmio_priv = priv; + /* Guest domains use what is programmed in their device tree. */ + register_mmio_handler(d, &vpci_mmio_handler, + GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, + priv); + } return 0; } @@ -95,14 +154,25 @@ int domain_vpci_init(struct domain *d) if ( !has_vpci(d) ) return 0; + return pci_host_iterate_bridges(d, vpci_setup_mmio_handler); +} + +static int domain_vpci_free_cb(struct domain *d, + struct pci_host_bridge *bridge) +{ if ( is_hardware_domain(d) ) - return pci_host_iterate_bridges(d, vpci_setup_mmio_handler); + XFREE(bridge->mmio_priv); + else + XFREE(d->vpci_mmio_priv); + return 0; +} - /* Guest domains use what is programmed in their device tree. */ - register_mmio_handler(d, &vpci_mmio_handler, - GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL); +void domain_vpci_free(struct domain *d) +{ + if ( !has_vpci(d) ) + return; - return 0; + pci_host_iterate_bridges(d, domain_vpci_free_cb); } int domain_vpci_get_num_mmio_handlers(struct domain *d) diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h index 27a2b069abd2..38e5a28c0d95 100644 --- a/xen/arch/arm/vpci.h +++ b/xen/arch/arm/vpci.h @@ -18,6 +18,7 @@ #ifdef CONFIG_HAS_VPCI int domain_vpci_init(struct domain *d); int domain_vpci_get_num_mmio_handlers(struct domain *d); +void domain_vpci_free(struct domain *d); #else static inline int domain_vpci_init(struct domain *d) { @@ -28,6 +29,8 @@ static inline int domain_vpci_get_num_mmio_handlers(struct domain *d) { return 0; } + +static inline void domain_vpci_free(struct domain *d) { } #endif #endif /* __ARCH_ARM_VPCI_H__ */ diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 5b963d75d1ba..b7dffb769cfd 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -889,6 +889,31 @@ int pci_remove_virtual_device(struct domain *d, const struct pci_dev *pdev) xfree(vdev); return 0; } + +/* + * Find the physical device which is mapped to the virtual device + * and translate virtual SBDF to the physical one. + */ +bool pci_translate_virtual_device(const struct domain *d, pci_sbdf_t *sbdf) +{ + struct vpci_dev *vdev; + bool found = false; + + pcidevs_lock(); + list_for_each_entry ( vdev, &d->vdev_list, list ) + { + if ( vdev->sbdf.sbdf == sbdf->sbdf ) + { + /* Replace virtual SBDF with the physical one. */ + *sbdf = vdev->pdev->sbdf; + found = true; + break; + } + } + pcidevs_unlock(); + + return found; +} #endif /* CONFIG_HAS_VPCI_GUEST_SUPPORT */ /* Caller should hold the pcidevs_lock */ diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h index 1bfba3da8f51..12b4bf467ad2 100644 --- a/xen/include/asm-arm/pci.h +++ b/xen/include/asm-arm/pci.h @@ -66,6 +66,7 @@ struct pci_host_bridge { uint16_t segment; /* Segment number */ struct pci_config_window* cfg; /* Pointer to the bridge config window */ struct pci_ops *ops; + void *mmio_priv; /* MMIO handler's private data. */ }; struct pci_ops { diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index 33033a3a8f8d..89cfc4853331 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -188,6 +188,7 @@ int pci_remove_device(u16 seg, u8 bus, u8 devfn); #ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT int pci_add_virtual_device(struct domain *d, const struct pci_dev *pdev); int pci_remove_virtual_device(struct domain *d, const struct pci_dev *pdev); +bool pci_translate_virtual_device(const struct domain *d, pci_sbdf_t *sbdf); #endif int pci_ro_device(int seg, int bus, int devfn); int pci_hide_device(unsigned int seg, unsigned int bus, unsigned int devfn); diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index ecdb04b4f7fc..858b4133482f 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -451,6 +451,8 @@ struct domain * to assign a unique SBDF to a passed through virtual PCI device. */ int vpci_dev_next; + /* Virtual PCI MMIO handler's private data. */ + void *vpci_mmio_priv; #endif #endif