From patchwork Thu Dec 9 07:29:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12665985 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C0884C433EF for ; Thu, 9 Dec 2021 07:29:46 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.242715.419771 (Exim 4.92) (envelope-from ) id 1mvDro-00077U-Io; Thu, 09 Dec 2021 07:29:24 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 242715.419771; Thu, 09 Dec 2021 07:29:24 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mvDro-00077M-Eo; Thu, 09 Dec 2021 07:29:24 +0000 Received: by outflank-mailman (input) for mailman id 242715; Thu, 09 Dec 2021 07:29:23 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mvDrn-0006rO-7C for xen-devel@lists.xenproject.org; Thu, 09 Dec 2021 07:29:23 +0000 Received: from mail-lf1-x131.google.com (mail-lf1-x131.google.com [2a00:1450:4864:20::131]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id ba7edfec-58c1-11ec-a831-37629979565c; Thu, 09 Dec 2021 08:29:22 +0100 (CET) Received: by mail-lf1-x131.google.com with SMTP id bu18so10413276lfb.0 for ; Wed, 08 Dec 2021 23:29:22 -0800 (PST) Received: from localhost.localdomain (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id f13sm445069lfe.297.2021.12.08.23.29.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Dec 2021 23:29:21 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: ba7edfec-58c1-11ec-a831-37629979565c DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/2Tjqi32B79jJFV5dYcHCJZi26dI5310xuMMTpxHeck=; b=XWZJBqluQ3NPKGVk25PLZ7QuEeaH1JDENGEiWOpz6crNOBOsrq8HovLQsDDhqAeml7 rvlBOyn1z2MypTZRZ52bTX2cTxsAB38TIv/wSrqjgy8KT9qu5/0cpGaObt9buUUyb1tC tjpDSnNy9UUOBtweuOaQ0pSNRJkWKTn9e/abTcFVQ55NZirn1scyS789koraHCj5rg8o vnKIr45dq84BcPYD/uS/2RwKi2vrTn3hlmV2lffhu5Sj2VwikeCUpg9S7//qDVM/jGwC 5WTjV9gkaHjzhfnlyp0WVklFwoig82Rv30jUaEyTqkWbnu4DZD6O13MLa8SzYdjOkGZq WNXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/2Tjqi32B79jJFV5dYcHCJZi26dI5310xuMMTpxHeck=; b=04nMiJe2JQiPnOmFasZSdyRYgubAP4XAFzafWVLtxVUTFpPxthudhx8VEQUBSRzKDw ULyymTeHroskv4FBG2WMjX6VDyD8WeFp3jXf2FniVSxuHvbGWAS3gTqw7T4+kTmBSdI1 3bbqNlafJyfWiZw91xEAWWFutrudUL8zUUA6sfijq1o3AgP5ho76c/U6CtVVlG/gxcOW jA2BMenj5biSIpOM6zVGE+54p92iZdUTgTlSdk0xfRwHiahap/YM7xJbwo65qbyS8CgV dVpvv/Vvxq8wiq4toRx8wombykzu6GZSUDjgVGL2Zeuhm2kpZYQ1Ukhw3TI3JgUd9ZwX z7cQ== X-Gm-Message-State: AOAM532WY4mf7vbBO079iTtY7N2x0eHS2mgcbadT3BGraHqUv9TxgN1y nrJor5B/9jTgCZRP7jWTeDujfjW5YxQODw== X-Google-Smtp-Source: ABdhPJz7R3UwMpl1RO5xA5oeeyQIc7DzXdSawwJPV1/vH2zpQkLuZTgviFe6bKKLCwE6gs6Fr1Zg7A== X-Received: by 2002:a05:6512:159d:: with SMTP id bp29mr4160681lfb.257.1639034961730; Wed, 08 Dec 2021 23:29:21 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, artem_mygaiev@epam.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko , Julien Grall Subject: [PATCH v8 1/4] xen/arm: add pci-domain for disabled devices Date: Thu, 9 Dec 2021 09:29:15 +0200 Message-Id: <20211209072918.460902-2-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211209072918.460902-1-andr2000@gmail.com> References: <20211209072918.460902-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko If a PCI host bridge device is present in the device tree, but is disabled, then its PCI host bridge driver was not instantiated. This results in the failure of the pci_get_host_bridge_segment() and the following panic during Xen start: (XEN) Device tree generation failed (-22). (XEN) (XEN) **************************************** (XEN) Panic on CPU 0: (XEN) Could not set up DOM0 guest OS (XEN) **************************************** Fix this by adding "linux,pci-domain" property for all device tree nodes which have "pci" device type, so we know which segments will be used by the guest for which bridges. Fixes: 4cfab4425d39 ("xen/arm: Add linux,pci-domain property for hwdom if not available.") Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Rahul Singh Tested-by: Rahul Singh Acked-by: Julien Grall --- Since v6: - use use_dt_domains in pci_get_new_domain_nr and return -1 if set - do not set "linux,pci-domain" if parent device is "pci" - move the code to a new helper handle_linux_pci_domain (Julien) New in v6 --- xen/arch/arm/domain_build.c | 66 +++++++++++++++++++++++------- xen/arch/arm/pci/pci-host-common.c | 8 +++- xen/include/asm-arm/pci.h | 8 ++++ 3 files changed, 66 insertions(+), 16 deletions(-) diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c index d02bacbcd1ed..6c5d22d9be1a 100644 --- a/xen/arch/arm/domain_build.c +++ b/xen/arch/arm/domain_build.c @@ -654,6 +654,55 @@ static void __init allocate_static_memory(struct domain *d, } #endif +/* + * When PCI passthrough is available we want to keep the + * "linux,pci-domain" in sync for every host bridge. + * + * Xen may not have a driver for all the host bridges. So we have + * to write an heuristic to detect whether a device node describes + * a host bridge. + * + * The current heuristic assumes that a device is a host bridge + * if the type is "pci" and then parent type is not "pci". + */ +static int handle_linux_pci_domain(struct kernel_info *kinfo, + const struct dt_device_node *node) +{ + uint16_t segment; + int res; + + if ( !is_pci_passthrough_enabled() ) + return 0; + + if ( !dt_device_type_is_equal(node, "pci") ) + return 0; + + if ( node->parent && dt_device_type_is_equal(node->parent, "pci") ) + return 0; + + if ( dt_find_property(node, "linux,pci-domain", NULL) ) + return 0; + + /* Allocate and create the linux,pci-domain */ + res = pci_get_host_bridge_segment(node, &segment); + if ( res < 0 ) + { + res = pci_get_new_domain_nr(); + if ( res < 0 ) + { + printk(XENLOG_DEBUG "Can't assign PCI segment to %s\n", + node->full_name); + return -FDT_ERR_NOTFOUND; + } + + segment = res; + printk(XENLOG_DEBUG "Assigned segment %d to %s\n", + segment, node->full_name); + } + + return fdt_property_cell(kinfo->fdt, "linux,pci-domain", segment); +} + static int __init write_properties(struct domain *d, struct kernel_info *kinfo, const struct dt_device_node *node) { @@ -755,21 +804,10 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo, return res; } - if ( is_pci_passthrough_enabled() && dt_device_type_is_equal(node, "pci") ) - { - if ( !dt_find_property(node, "linux,pci-domain", NULL) ) - { - uint16_t segment; - - res = pci_get_host_bridge_segment(node, &segment); - if ( res < 0 ) - return res; + res = handle_linux_pci_domain(kinfo, node); - res = fdt_property_cell(kinfo->fdt, "linux,pci-domain", segment); - if ( res ) - return res; - } - } + if ( res ) + return res; /* * Override the property "status" to disable the device when it's diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c index ae79a0c19b93..40e779b5d803 100644 --- a/xen/arch/arm/pci/pci-host-common.c +++ b/xen/arch/arm/pci/pci-host-common.c @@ -30,6 +30,8 @@ static LIST_HEAD(pci_host_bridges); static atomic_t domain_nr = ATOMIC_INIT(-1); +static int use_dt_domains = -1; + static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len) { return ioremap_nocache(start, len); @@ -137,14 +139,16 @@ void pci_add_host_bridge(struct pci_host_bridge *bridge) list_add_tail(&bridge->node, &pci_host_bridges); } -static int pci_get_new_domain_nr(void) +int pci_get_new_domain_nr(void) { + if ( use_dt_domains ) + return -1; + return atomic_inc_return(&domain_nr); } static int pci_bus_find_domain_nr(struct dt_device_node *dev) { - static int use_dt_domains = -1; int domain; domain = dt_get_pci_domain_nr(dev); diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h index 9736d6816da3..c313423cdcb2 100644 --- a/xen/include/asm-arm/pci.h +++ b/xen/include/asm-arm/pci.h @@ -109,6 +109,8 @@ static always_inline bool is_pci_passthrough_enabled(void) void arch_pci_init_pdev(struct pci_dev *pdev); +int pci_get_new_domain_nr(void); + #else /*!CONFIG_HAS_PCI*/ struct arch_pci_dev { }; @@ -129,5 +131,11 @@ static inline int pci_get_host_bridge_segment(const struct dt_device_node *node, return -EINVAL; } +static inline int pci_get_new_domain_nr(void) +{ + ASSERT_UNREACHABLE(); + return -1; +} + #endif /*!CONFIG_HAS_PCI*/ #endif /* __ARM_PCI_H__ */ From patchwork Thu Dec 9 07:29:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12665983 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 35308C433F5 for ; Thu, 9 Dec 2021 07:29:46 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.242716.419783 (Exim 4.92) (envelope-from ) id 1mvDrp-0007Nr-S9; Thu, 09 Dec 2021 07:29:25 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 242716.419783; Thu, 09 Dec 2021 07:29:25 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mvDrp-0007Nk-O1; Thu, 09 Dec 2021 07:29:25 +0000 Received: by outflank-mailman (input) for mailman id 242716; Thu, 09 Dec 2021 07:29:24 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mvDro-00075S-FL for xen-devel@lists.xenproject.org; Thu, 09 Dec 2021 07:29:24 +0000 Received: from mail-lj1-x234.google.com (mail-lj1-x234.google.com [2a00:1450:4864:20::234]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id bb0ca4bf-58c1-11ec-9d12-4777fae47e2b; Thu, 09 Dec 2021 08:29:23 +0100 (CET) Received: by mail-lj1-x234.google.com with SMTP id k2so7593624lji.4 for ; Wed, 08 Dec 2021 23:29:23 -0800 (PST) Received: from localhost.localdomain (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id f13sm445069lfe.297.2021.12.08.23.29.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Dec 2021 23:29:22 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: bb0ca4bf-58c1-11ec-9d12-4777fae47e2b DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XUbUdVW/JQJGJv5htHXoALs0/d8a/QR27xFCYV/hVms=; b=NB1ofDHv1bCLI5ZycXHpD1vK9WrNa6k4+O7RblnTJJuyEncVfvTRjNfEUQLEmeKNQ7 nj/4ikMG64nKUYeNSoe7q/uFXiCSLwgXcKLyiEQZuSOpA5YddU/tyzb6Ywcgswu9qfNM G5DMVaVZBCbwKpBXus+y/kvhaq5vep8nALBlQ/QSyk+Z97EHJpqynmGVIPl46b6ObAEL rAAtqBMW//87W+ZTB/k/WQ3zm+798u1+SXRVV2dsDB9Bx54YHj1gM2Lsbt3JAMPG2O/9 pUABfV/YAvD/KsmfBj5r7aGqKCYaxf/tTxiOvWZS1xBS5PxPIr4wRxNNkntybNEdeBxn IRmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XUbUdVW/JQJGJv5htHXoALs0/d8a/QR27xFCYV/hVms=; b=3+1J898qcJaJuVabMSojTUOP44W5nWJhTt5KVcpc7CCOUHDJB9YHmlUmGPrXIJAnV/ Jihbut2C4MK9I20FttUUCJPNmri6kXfETOza8+Rx2mL9DhfPscu+RXNaXExjYyfI8AOm YkyNgb8GuRa5Y+hdio1g3jh0HKPDKUtVoNWot4l0G0l9PA0sezrw0S4YQVyYmTRiNxEA 14/iLWdZEEkCfatOEBav1SgocF5ph1grJMh/CL95BT+JtD952FRY+m9OyHi15/yJOGpN P0QUzOosbFt5ZE9lD+9PwCSPCFKXb362aLV1dY+4OT+pARo5YkWnOsUPRTDCsHbf3076 ulbw== X-Gm-Message-State: AOAM5303MFbnkiG/lfmlDjWawoxyLegKDriUpe02yM2dUhiPdmZR9Cze r1kzzVlwohfkZC8jcB6W8VIvqcNGghpPQA== X-Google-Smtp-Source: ABdhPJx6MjlYrfHHU5UuBUicZWv6in5FppRcK79YaoNIBKz1cXa4FcBu7I4/oIYJgKoaeqGkn7qcWQ== X-Received: by 2002:a2e:a7c6:: with SMTP id x6mr4561545ljp.328.1639034962642; Wed, 08 Dec 2021 23:29:22 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, artem_mygaiev@epam.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v8 2/4] xen/arm: setup MMIO range trap handlers for hardware domain Date: Thu, 9 Dec 2021 09:29:16 +0200 Message-Id: <20211209072918.460902-3-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211209072918.460902-1-andr2000@gmail.com> References: <20211209072918.460902-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko In order for vPCI to work it needs to maintain guest and hardware domain's views of the configuration space. For example, BARs and COMMAND registers require emulation for guests and the guest view of the registers needs to be in sync with the real contents of the relevant registers. For that ECAM address space needs to also be trapped for the hardware domain, so we need to implement PCI host bridge specific callbacks to properly setup MMIO handlers for those ranges depending on particular host bridge implementation. Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Rahul Singh Tested-by: Rahul Singh Acked-by: Julien Grall --- Since v7: - move local variable inside the loop in pci_host_iterate_bridges_and_count and s/err/ret - fixed incorrect s/if ( err )/if ( err < 0 ) - moved the guest part in an else in domain_vpci_init - s/count/ret in domain_vpci_init as it better reflects the use of the value Since v6: - eliminate pci_host_get_num_bridges and make pci_host_iterate_bridges return the count - extend comment in domain_vpci_init - remove not yet relevant code for num MMIOs and virtual bus topology - add extra check for has_vpci in domain_vpci_get_num_mmio_handlers - remove code that fixes num MMIOs for guest domain as it doesn't belong to this patch Since v5: - add vpci_sbdf_from_gpa helper for gpa to SBDF translation - take bridge's bus start into account while calculating SBDF Since v4: - unsigned int for functions working with count - gate number of MMIO handlers needed for CONFIG_HAS_PCI_MSI and fix their number, e.g. single handler for PBA and MSI-X tables (Roger) - re-work code for assigning MMIO handlers to be simpler and account on the fact that there could multiple host bridges exist for the hwdom Since v3: - fixed comment formatting Since v2: - removed unneeded assignment (count = 0) - removed unneeded header inclusion - update commit message Since v1: - Dynamically calculate the number of MMIO handlers required for vPCI and update the total number accordingly - s/clb/cb - Do not introduce a new callback for MMIO handler setup --- xen/arch/arm/domain.c | 2 + xen/arch/arm/pci/pci-host-common.c | 19 +++++++ xen/arch/arm/vpci.c | 81 ++++++++++++++++++++++++++---- xen/arch/arm/vpci.h | 6 +++ xen/include/asm-arm/pci.h | 4 ++ 5 files changed, 102 insertions(+), 10 deletions(-) diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c index 96e1b235501d..92a6c509e5c5 100644 --- a/xen/arch/arm/domain.c +++ b/xen/arch/arm/domain.c @@ -739,6 +739,8 @@ int arch_domain_create(struct domain *d, if ( (rc = domain_vgic_register(d, &count)) != 0 ) goto fail; + count += domain_vpci_get_num_mmio_handlers(d); + if ( (rc = domain_io_init(d, count + MAX_IO_HANDLER)) != 0 ) goto fail; diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c index 40e779b5d803..84aab371cd9a 100644 --- a/xen/arch/arm/pci/pci-host-common.c +++ b/xen/arch/arm/pci/pci-host-common.c @@ -294,6 +294,25 @@ int pci_get_host_bridge_segment(const struct dt_device_node *node, return -EINVAL; } +int pci_host_iterate_bridges_and_count(struct domain *d, + int (*cb)(struct domain *d, + struct pci_host_bridge *bridge)) +{ + struct pci_host_bridge *bridge; + int count = 0; + + list_for_each_entry( bridge, &pci_host_bridges, node ) + { + int ret; + + ret = cb(d, bridge); + if ( ret < 0 ) + return ret; + count += ret; + } + return count; +} + /* * Local variables: * mode: C diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c index 23f45386f4b3..1564448c6c8d 100644 --- a/xen/arch/arm/vpci.c +++ b/xen/arch/arm/vpci.c @@ -16,16 +16,31 @@ #include +static pci_sbdf_t vpci_sbdf_from_gpa(const struct pci_host_bridge *bridge, + paddr_t gpa) +{ + pci_sbdf_t sbdf; + + if ( bridge ) + { + sbdf.sbdf = VPCI_ECAM_BDF(gpa - bridge->cfg->phys_addr); + sbdf.seg = bridge->segment; + sbdf.bus += bridge->cfg->busn_start; + } + else + sbdf.sbdf = VPCI_ECAM_BDF(gpa - GUEST_VPCI_ECAM_BASE); + + return sbdf; +} + static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, register_t *r, void *p) { - pci_sbdf_t sbdf; + struct pci_host_bridge *bridge = p; + pci_sbdf_t sbdf = vpci_sbdf_from_gpa(bridge, info->gpa); /* data is needed to prevent a pointer cast on 32bit */ unsigned long data; - /* We ignore segment part and always handle segment 0 */ - sbdf.sbdf = VPCI_ECAM_BDF(info->gpa - GUEST_VPCI_ECAM_BASE); - if ( vpci_ecam_read(sbdf, ECAM_REG_OFFSET(info->gpa), 1U << info->dabt.size, &data) ) { @@ -41,10 +56,8 @@ static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info, register_t r, void *p) { - pci_sbdf_t sbdf; - - /* We ignore segment part and always handle segment 0 */ - sbdf.sbdf = VPCI_ECAM_BDF(info->gpa - GUEST_VPCI_ECAM_BASE); + struct pci_host_bridge *bridge = p; + pci_sbdf_t sbdf = vpci_sbdf_from_gpa(bridge, info->gpa); return vpci_ecam_write(sbdf, ECAM_REG_OFFSET(info->gpa), 1U << info->dabt.size, r); @@ -55,13 +68,61 @@ static const struct mmio_handler_ops vpci_mmio_handler = { .write = vpci_mmio_write, }; +static int vpci_setup_mmio_handler_cb(struct domain *d, + struct pci_host_bridge *bridge) +{ + struct pci_config_window *cfg = bridge->cfg; + + register_mmio_handler(d, &vpci_mmio_handler, + cfg->phys_addr, cfg->size, bridge); + + /* We have registered a single MMIO handler. */ + return 1; +} + int domain_vpci_init(struct domain *d) { if ( !has_vpci(d) ) return 0; - register_mmio_handler(d, &vpci_mmio_handler, - GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL); + /* + * The hardware domain gets as many MMIOs as required by the + * physical host bridge. + * Guests get the virtual platform layout: one virtual host bridge for now. + */ + if ( is_hardware_domain(d) ) + { + int ret; + + ret = pci_host_iterate_bridges_and_count(d, vpci_setup_mmio_handler_cb); + if ( ret < 0 ) + return ret; + } + else + register_mmio_handler(d, &vpci_mmio_handler, + GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL); + + return 0; +} + +static int vpci_get_num_handlers_cb(struct domain *d, + struct pci_host_bridge *bridge) +{ + /* Each bridge has a single MMIO handler for the configuration space. */ + return 1; +} + +unsigned int domain_vpci_get_num_mmio_handlers(struct domain *d) +{ + if ( !has_vpci(d) ) + return 0; + + if ( is_hardware_domain(d) ) + { + int ret = pci_host_iterate_bridges_and_count(d, vpci_get_num_handlers_cb); + + return ret < 0 ? 0 : ret; + } return 0; } diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h index d8a7b0e3e802..3c713f3fcdb5 100644 --- a/xen/arch/arm/vpci.h +++ b/xen/arch/arm/vpci.h @@ -17,11 +17,17 @@ #ifdef CONFIG_HAS_VPCI int domain_vpci_init(struct domain *d); +unsigned int domain_vpci_get_num_mmio_handlers(struct domain *d); #else static inline int domain_vpci_init(struct domain *d) { return 0; } + +static inline unsigned int domain_vpci_get_num_mmio_handlers(struct domain *d) +{ + return 0; +} #endif #endif /* __ARCH_ARM_VPCI_H__ */ diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h index c313423cdcb2..94f003a07ca2 100644 --- a/xen/include/asm-arm/pci.h +++ b/xen/include/asm-arm/pci.h @@ -111,6 +111,10 @@ void arch_pci_init_pdev(struct pci_dev *pdev); int pci_get_new_domain_nr(void); +int pci_host_iterate_bridges_and_count(struct domain *d, + int (*cb)(struct domain *d, + struct pci_host_bridge *bridge)); + #else /*!CONFIG_HAS_PCI*/ struct arch_pci_dev { }; From patchwork Thu Dec 9 07:29:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12665989 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D5750C4332F for ; Thu, 9 Dec 2021 07:29:46 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.242717.419794 (Exim 4.92) (envelope-from ) id 1mvDrr-0007et-5f; Thu, 09 Dec 2021 07:29:27 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 242717.419794; Thu, 09 Dec 2021 07:29:27 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mvDrr-0007em-1e; Thu, 09 Dec 2021 07:29:27 +0000 Received: by outflank-mailman (input) for mailman id 242717; Thu, 09 Dec 2021 07:29:25 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mvDrp-00075S-7C for xen-devel@lists.xenproject.org; Thu, 09 Dec 2021 07:29:25 +0000 Received: from mail-lj1-x22d.google.com (mail-lj1-x22d.google.com [2a00:1450:4864:20::22d]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id bbcbace7-58c1-11ec-9d12-4777fae47e2b; Thu, 09 Dec 2021 08:29:24 +0100 (CET) Received: by mail-lj1-x22d.google.com with SMTP id u22so7572069lju.7 for ; Wed, 08 Dec 2021 23:29:24 -0800 (PST) Received: from localhost.localdomain (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id f13sm445069lfe.297.2021.12.08.23.29.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Dec 2021 23:29:23 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: bbcbace7-58c1-11ec-9d12-4777fae47e2b DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5zMEfsrDMgGvbxJE8yizewP5LPSer3MoFR6+sMloccU=; b=nwSZwsWRKgJcOzhiHDaN4norjoJ1YzrXrGfYBPtNd/KBeTeU55r2WkRIY7FcwfkxEl 3OdVmMtxVg5KGNCEiLnMMnFY8VPF3fwNuPnrxOLOWS8XTL4Wf14v2uo8XQhwS6aqzD/2 GDOa8B7wR6BHtyyF7syfE7V0VW0VrR3RdjMd81J9XarQOE49cL4eHleMJgegOwmDO3vJ QYCV3mF0yAAg/+rn6bZsllA5vmMEF/ev0MggvvLTGMO2FhZt2eYqouG6sCuclbN5sKIq 4DpEWLsl8kB/trlQ0zgMTPJN45daqmiY1prZ9pAA4PJuZWw9zqYvG28YCmVaC3PBxmVD DZRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5zMEfsrDMgGvbxJE8yizewP5LPSer3MoFR6+sMloccU=; b=KHze6+SPyXlB4YIyB5o/jn10wj2X5M2Ub8l6X9aA1nWoi1cBxochsenX/N/U7Ui/5i GruRuDikLs9nbv2xbDcRcZnrUOUamicMrTKZX9z24EHCuzx3niO03Dd/+gEEtIL6HQmw 6q6/NmXVDq0oegQrYQSBb98XbT4v3X0ZVsAYXeEJFO8aoa7E6085zCTAkcjHauyTeI2z bLKwVwEmwidSvu8/g/nHdxmUyt1epVQD+PB3R5wUbnMnIHJ0OFfSEl+qfRhH71gWo5Ml vKn+eVRDkFSRFxIrblNc34H9ylS1wbuW/YbXDC3x6SMwhPItmYMr8IKMUOSRK11vlznU 5YMQ== X-Gm-Message-State: AOAM533Jit4jLC5xiNhyBJ+XZAo/cUWMyjGA2WS0jADkmkHawiVttnyj adz2JFrnPC8nCxTxsAryxzkVpOKBl+0eHg== X-Google-Smtp-Source: ABdhPJzJNMSzJKdK5Un4XS19D0FwAW7BsnV1snLB/1oEAMUDNtrotAwvFJyvrKGJFjYrYJoKKgtucg== X-Received: by 2002:a2e:b528:: with SMTP id z8mr4648866ljm.178.1639034963970; Wed, 08 Dec 2021 23:29:23 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, artem_mygaiev@epam.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko , Julien Grall Subject: [PATCH v8 3/4] xen/arm: account IO handler for emulated PCI host bridge Date: Thu, 9 Dec 2021 09:29:17 +0200 Message-Id: <20211209072918.460902-4-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211209072918.460902-1-andr2000@gmail.com> References: <20211209072918.460902-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko At the moment, we always allocate an extra 16 slots for IO handlers (see MAX_IO_HANDLER). So while adding an IO trap handler for the emulated PCI host bridge we are not breaking anything, but we have a latent bug as the maximum number of IOs may be exceeded. Fix this by explicitly telling that we have an additional IO handler, so it is accounted. Fixes: d59168dc05a5 ("xen/arm: Enable the existing x86 virtual PCI support for ARM") Signed-off-by: Oleksandr Andrushchenko Acked-by: Julien Grall Reviewed-by: Rahul Singh Tested-by: Rahul Singh --- Since v7: - updated comment New in v7 --- xen/arch/arm/vpci.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c index 1564448c6c8d..84aaaaebd69d 100644 --- a/xen/arch/arm/vpci.c +++ b/xen/arch/arm/vpci.c @@ -124,7 +124,11 @@ unsigned int domain_vpci_get_num_mmio_handlers(struct domain *d) return ret < 0 ? 0 : ret; } - return 0; + /* + * For guests each host bridge requires one region to cover the + * configuration space. At the moment, we only expose a single host bridge. + */ + return 1; } /* From patchwork Thu Dec 9 07:29:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12665991 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5241DC43219 for ; Thu, 9 Dec 2021 07:29:47 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.242718.419804 (Exim 4.92) (envelope-from ) id 1mvDrs-0007wS-Fb; Thu, 09 Dec 2021 07:29:28 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 242718.419804; Thu, 09 Dec 2021 07:29:28 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mvDrs-0007vP-BL; Thu, 09 Dec 2021 07:29:28 +0000 Received: by outflank-mailman (input) for mailman id 242718; Thu, 09 Dec 2021 07:29:27 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mvDrq-0006rO-Uf for xen-devel@lists.xenproject.org; Thu, 09 Dec 2021 07:29:27 +0000 Received: from mail-lj1-x234.google.com (mail-lj1-x234.google.com [2a00:1450:4864:20::234]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id bc894645-58c1-11ec-a831-37629979565c; Thu, 09 Dec 2021 08:29:25 +0100 (CET) Received: by mail-lj1-x234.google.com with SMTP id i63so7604316lji.3 for ; Wed, 08 Dec 2021 23:29:25 -0800 (PST) Received: from localhost.localdomain (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id f13sm445069lfe.297.2021.12.08.23.29.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Dec 2021 23:29:24 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: bc894645-58c1-11ec-a831-37629979565c DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/ito1eEWQVNmkHwpJtruOsgKBqnwBo2/v6MgPdmvbrg=; b=Ja/TMZpxrGrQUP+ZSzBbN2hdAOjQtGvOQOUZjEOIhFVo+HC2B+L4GLe59EhkYSiyXP 164DEgQhjHVgAZh1SUiufYNfjsWAk3Opr3fSJ9adHcZPi3KXWOtgzTW1quV90uSnfUrU vssHNeqc81ur9mrPo7cw+c0TySnYaAs3yb67aRHWJ2gszg5TJP4Ch12wDJYmewulpoU6 POMwcgodrpbfIehEMTR5v7TE2HKUOdWnUpGzTd4z70aGuE1u0Sm4UOe7H9NJrX0dou74 ECRzMBIoaXCRc3utxW/U5cmYPFTlE3DJT0MQEIaz78ctktm9xIzrxUgfIztxOVZ/QG8u xCRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/ito1eEWQVNmkHwpJtruOsgKBqnwBo2/v6MgPdmvbrg=; b=S3cSC9BOv5FmSYjMLtNr5MGmwbPNH1aG8Pkwrgy6PggAkuwy9qWWljFPgM6o4CQwRy Sc7CyvC0COIJvhpzLOZ8axblxIvhnchHnfY21s6/2QgF8GiSCE078jvZ7Fn+Rbgd1lgn zd0FtQTiCUBn+5ppja+MDGsWkSIwaKCyPN36NidIZn4Dn3vHf3xRfUi9ZTwjZ9zKJzjO WAnyvn5keIfF6nZ2ySIKm4SdKOq/RnbsXrlKLgIToCVqX8Ayqzw7eEUhT6QF/c4b01jH 4z+1lse8NjDNkJqE65c409JYIaZPQimalz/LZoyzR9tdqSANYKn4toyJu7djl93CFt+m JuPQ== X-Gm-Message-State: AOAM532nH8RZWgzz4a83yr4VVxQ2CQ6XHPviInHILn0rwZ5p2ZmpNpBl sGUUnudQEEXLpXhxUdNUC2qokHWgpWgf9g== X-Google-Smtp-Source: ABdhPJyhr8+/Q9+L2OTLKFQw0Llkq/usPvEq6Lv0aWq/Bt1YFEajzpf+AHZRC18mfPCrycLzLfMxZw== X-Received: by 2002:a2e:a265:: with SMTP id k5mr4489812ljm.527.1639034965083; Wed, 08 Dec 2021 23:29:25 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, artem_mygaiev@epam.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko , Julien Grall Subject: [PATCH v8 4/4] xen/arm: do not map PCI ECAM and MMIO space to Domain-0's p2m Date: Thu, 9 Dec 2021 09:29:18 +0200 Message-Id: <20211209072918.460902-5-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211209072918.460902-1-andr2000@gmail.com> References: <20211209072918.460902-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko PCI host bridges are special devices in terms of implementing PCI passthrough. According to [1] the current implementation depends on Domain-0 to perform the initialization of the relevant PCI host bridge hardware and perform PCI device enumeration. In order to achieve that one of the required changes is to not map all the memory ranges in map_range_to_domain as we traverse the device tree on startup and perform some additional checks if the range needs to be mapped to Domain-0. The generic PCI host controller device tree binding says [2]: - ranges: As described in IEEE Std 1275-1994, but must provide at least a definition of non-prefetchable memory. One or both of prefetchable Memory and IO Space may also be provided. - reg : The Configuration Space base address and size, as accessed from the parent bus. The base address corresponds to the first bus in the "bus-range" property. If no "bus-range" is specified, this will be bus 0 (the default). From the above none of the memory ranges from the "ranges" property needs to be mapped to Domain-0 at startup as MMIO mapping is going to be handled dynamically by vPCI as we assign PCI devices, e.g. each device assigned to Domain-0/guest will have its MMIOs mapped/unmapped as needed by Xen. The "reg" property covers not only ECAM space, but may also have other then the configuration memory ranges described, for example [3]: - reg: Should contain rc_dbi, config registers location and length. - reg-names: Must include the following entries: "rc_dbi": controller configuration registers; "config": PCIe configuration space registers. This patch makes it possible to not map all the ranges from the "ranges" property and also ECAM from the "reg". All the rest from the "reg" property still needs to be mapped to Domain-0, so the PCI host bridge remains functional in Domain-0. This is done by first skipping the mappings while traversing the device tree as it is done for usual devices and then by calling a dedicated pci_host_bridge_mappings function which only maps MMIOs required by the host bridges leaving the regions, needed for vPCI traps, unmapped. [1] https://lists.xenproject.org/archives/html/xen-devel/2020-07/msg00777.html [2] https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/host-generic-pci.txt [3] https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/hisilicon-pcie.txt Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Julien Grall Reviewed-by: Rahul Singh Tested-by: Rahul Singh --- Since v7: - updates in comments and commit message Since v5: - remove some need_mapping local variables - use own_device in handle_device - add __init for pci_ecam_need_p2m_hwdom_mapping - make pci_host_bridge_mappings use p2m_mmio_direct_dev directly Since v4: - update skip_mapping comment - add comment why we need to map interrupts to Dom0 Since v3: - pass struct map_range_data to map_dt_irq_to_domain - remove redundant check from map_range_to_domain - fix handle_device's .skip_mapping Since v2: - removed check in map_range_to_domain for PCI_DEV and moved it to handle_device, so the code is simpler - s/map_pci_bridge/skip_mapping - extended comment in pci_host_bridge_mappings - minor code restructure in construct_dom0 - s/.need_p2m_mapping/.need_p2m_hwdom_mapping and related callbacks - unsigned int i; in pci_host_bridge_mappings Since v1: - Added better description of why and what needs to be mapped into Domain-0's p2m and what doesn't - Do not do any mappings for PCI devices while traversing the DT - Walk all the bridges and make required mappings in one go --- xen/arch/arm/domain_build.c | 66 +++++++++++++++++------------- xen/arch/arm/pci/ecam.c | 14 +++++++ xen/arch/arm/pci/pci-host-common.c | 50 ++++++++++++++++++++++ xen/arch/arm/pci/pci-host-zynqmp.c | 1 + xen/include/asm-arm/pci.h | 10 +++++ xen/include/asm-arm/setup.h | 13 ++++++ 6 files changed, 126 insertions(+), 28 deletions(-) diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c index 6c5d22d9be1a..6931c022a2e8 100644 --- a/xen/arch/arm/domain_build.c +++ b/xen/arch/arm/domain_build.c @@ -51,12 +51,6 @@ static int __init parse_dom0_mem(const char *s) } custom_param("dom0_mem", parse_dom0_mem); -struct map_range_data -{ - struct domain *d; - p2m_type_t p2mt; -}; - /* Override macros from asm/page.h to make them work with mfn_t */ #undef virt_to_mfn #define virt_to_mfn(va) _mfn(__virt_to_mfn(va)) @@ -1720,10 +1714,10 @@ static int __init map_dt_irq_to_domain(const struct dt_device_node *dev, const struct dt_irq *dt_irq, void *data) { - struct domain *d = data; + struct map_range_data *mr_data = data; + struct domain *d = mr_data->d; unsigned int irq = dt_irq->irq; int res; - bool need_mapping = !dt_device_for_passthrough(dev); if ( irq < NR_LOCAL_IRQS ) { @@ -1742,18 +1736,16 @@ static int __init map_dt_irq_to_domain(const struct dt_device_node *dev, return res; } - res = map_irq_to_domain(d, irq, need_mapping, dt_node_name(dev)); + res = map_irq_to_domain(d, irq, !mr_data->skip_mapping, dt_node_name(dev)); return 0; } -static int __init map_range_to_domain(const struct dt_device_node *dev, - u64 addr, u64 len, - void *data) +int __init map_range_to_domain(const struct dt_device_node *dev, + u64 addr, u64 len, void *data) { struct map_range_data *mr_data = data; struct domain *d = mr_data->d; - bool need_mapping = !dt_device_for_passthrough(dev); int res; /* @@ -1776,7 +1768,7 @@ static int __init map_range_to_domain(const struct dt_device_node *dev, } } - if ( need_mapping ) + if ( !mr_data->skip_mapping ) { res = map_regions_p2mt(d, gaddr_to_gfn(addr), @@ -1805,23 +1797,21 @@ static int __init map_range_to_domain(const struct dt_device_node *dev, * then we may need to perform additional mappings in order to make * the child resources available to domain 0. */ -static int __init map_device_children(struct domain *d, - const struct dt_device_node *dev, - p2m_type_t p2mt) +static int __init map_device_children(const struct dt_device_node *dev, + struct map_range_data *mr_data) { - struct map_range_data mr_data = { .d = d, .p2mt = p2mt }; - int ret; - if ( dt_device_type_is_equal(dev, "pci") ) { + int ret; + dt_dprintk("Mapping children of %s to guest\n", dt_node_full_name(dev)); - ret = dt_for_each_irq_map(dev, &map_dt_irq_to_domain, d); + ret = dt_for_each_irq_map(dev, &map_dt_irq_to_domain, mr_data); if ( ret < 0 ) return ret; - ret = dt_for_each_range(dev, &map_range_to_domain, &mr_data); + ret = dt_for_each_range(dev, &map_range_to_domain, mr_data); if ( ret < 0 ) return ret; } @@ -1901,14 +1891,28 @@ static int __init handle_device(struct domain *d, struct dt_device_node *dev, unsigned int i; int res; u64 addr, size; - bool need_mapping = !dt_device_for_passthrough(dev); + bool own_device = !dt_device_for_passthrough(dev); + /* + * We want to avoid mapping the MMIO in dom0 for the following cases: + * - The device is owned by dom0 (i.e. it has been flagged for + * passthrough). + * - PCI host bridges with driver in Xen. They will later be mapped by + * pci_host_bridge_mappings(). + */ + struct map_range_data mr_data = { + .d = d, + .p2mt = p2mt, + .skip_mapping = !own_device || + (is_pci_passthrough_enabled() && + (device_get_class(dev) == DEVICE_PCI_HOSTBRIDGE)) + }; naddr = dt_number_of_address(dev); dt_dprintk("%s passthrough = %d naddr = %u\n", - dt_node_full_name(dev), need_mapping, naddr); + dt_node_full_name(dev), own_device, naddr); - if ( need_mapping ) + if ( own_device ) { dt_dprintk("Check if %s is behind the IOMMU and add it\n", dt_node_full_name(dev)); @@ -1934,14 +1938,13 @@ static int __init handle_device(struct domain *d, struct dt_device_node *dev, } } - res = handle_device_interrupts(d, dev, need_mapping); + res = handle_device_interrupts(d, dev, own_device); if ( res < 0 ) return res; /* Give permission and map MMIOs */ for ( i = 0; i < naddr; i++ ) { - struct map_range_data mr_data = { .d = d, .p2mt = p2mt }; res = dt_device_get_address(dev, i, &addr, &size); if ( res ) { @@ -1955,7 +1958,7 @@ static int __init handle_device(struct domain *d, struct dt_device_node *dev, return res; } - res = map_device_children(d, dev, p2mt); + res = map_device_children(dev, &mr_data); if ( res ) return res; @@ -3114,7 +3117,14 @@ static int __init construct_dom0(struct domain *d) return rc; if ( acpi_disabled ) + { rc = prepare_dtb_hwdom(d, &kinfo); + if ( rc < 0 ) + return rc; +#ifdef CONFIG_HAS_PCI + rc = pci_host_bridge_mappings(d); +#endif + } else rc = prepare_acpi(d, &kinfo); diff --git a/xen/arch/arm/pci/ecam.c b/xen/arch/arm/pci/ecam.c index 2855dea213f4..6aeea12a68bf 100644 --- a/xen/arch/arm/pci/ecam.c +++ b/xen/arch/arm/pci/ecam.c @@ -40,6 +40,19 @@ void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge, return base + (PCI_DEVFN2(sbdf.bdf) << devfn_shift) + where; } +bool __init pci_ecam_need_p2m_hwdom_mapping(struct domain *d, + struct pci_host_bridge *bridge, + uint64_t addr) +{ + struct pci_config_window *cfg = bridge->cfg; + + /* + * We do not want ECAM address space to be mapped in Domain-0's p2m, + * so we can trap access to it. + */ + return cfg->phys_addr != addr; +} + /* ECAM ops */ const struct pci_ecam_ops pci_generic_ecam_ops = { .bus_shift = 20, @@ -47,6 +60,7 @@ const struct pci_ecam_ops pci_generic_ecam_ops = { .map_bus = pci_ecam_map_bus, .read = pci_generic_config_read, .write = pci_generic_config_write, + .need_p2m_hwdom_mapping = pci_ecam_need_p2m_hwdom_mapping, } }; diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c index 84aab371cd9a..fd8c0f837a6a 100644 --- a/xen/arch/arm/pci/pci-host-common.c +++ b/xen/arch/arm/pci/pci-host-common.c @@ -22,6 +22,8 @@ #include #include +#include + /* * List for all the pci host bridges. */ @@ -313,6 +315,54 @@ int pci_host_iterate_bridges_and_count(struct domain *d, return count; } +/* + * For each PCI host bridge we need to only map those ranges + * which are used by Domain-0 to properly initialize the bridge, + * e.g. we do not want to map ECAM configuration space which lives in + * "reg" device tree property, but we want to map other regions of + * the host bridge. The PCI aperture defined by the "ranges" device + * tree property should also be skipped. + */ +int __init pci_host_bridge_mappings(struct domain *d) +{ + struct pci_host_bridge *bridge; + struct map_range_data mr_data = { + .d = d, + .p2mt = p2m_mmio_direct_dev, + .skip_mapping = false + }; + + list_for_each_entry( bridge, &pci_host_bridges, node ) + { + const struct dt_device_node *dev = bridge->dt_node; + unsigned int i; + + for ( i = 0; i < dt_number_of_address(dev); i++ ) + { + uint64_t addr, size; + int err; + + err = dt_device_get_address(dev, i, &addr, &size); + if ( err ) + { + printk(XENLOG_ERR + "Unable to retrieve address range index=%u for %s\n", + i, dt_node_full_name(dev)); + return err; + } + + if ( bridge->ops->need_p2m_hwdom_mapping(d, bridge, addr) ) + { + err = map_range_to_domain(dev, addr, size, &mr_data); + if ( err ) + return err; + } + } + } + + return 0; +} + /* * Local variables: * mode: C diff --git a/xen/arch/arm/pci/pci-host-zynqmp.c b/xen/arch/arm/pci/pci-host-zynqmp.c index 516982bca833..101edb8593c1 100644 --- a/xen/arch/arm/pci/pci-host-zynqmp.c +++ b/xen/arch/arm/pci/pci-host-zynqmp.c @@ -34,6 +34,7 @@ const struct pci_ecam_ops nwl_pcie_ops = { .map_bus = pci_ecam_map_bus, .read = pci_generic_config_read, .write = pci_generic_config_write, + .need_p2m_hwdom_mapping = pci_ecam_need_p2m_hwdom_mapping, } }; diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h index 94f003a07ca2..7c7449d64fca 100644 --- a/xen/include/asm-arm/pci.h +++ b/xen/include/asm-arm/pci.h @@ -17,6 +17,8 @@ #ifdef CONFIG_HAS_PCI +#include + #define pci_to_dev(pcidev) (&(pcidev)->arch.dev) extern bool pci_passthrough_enabled; @@ -73,6 +75,9 @@ struct pci_ops { uint32_t reg, uint32_t len, uint32_t *value); int (*write)(struct pci_host_bridge *bridge, pci_sbdf_t sbdf, uint32_t reg, uint32_t len, uint32_t value); + bool (*need_p2m_hwdom_mapping)(struct domain *d, + struct pci_host_bridge *bridge, + uint64_t addr); }; /* @@ -97,6 +102,9 @@ int pci_generic_config_write(struct pci_host_bridge *bridge, pci_sbdf_t sbdf, uint32_t reg, uint32_t len, uint32_t value); void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge, pci_sbdf_t sbdf, uint32_t where); +bool pci_ecam_need_p2m_hwdom_mapping(struct domain *d, + struct pci_host_bridge *bridge, + uint64_t addr); struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus); struct dt_device_node *pci_find_host_bridge_node(struct device *dev); int pci_get_host_bridge_segment(const struct dt_device_node *node, @@ -115,6 +123,8 @@ int pci_host_iterate_bridges_and_count(struct domain *d, int (*cb)(struct domain *d, struct pci_host_bridge *bridge)); +int pci_host_bridge_mappings(struct domain *d); + #else /*!CONFIG_HAS_PCI*/ struct arch_pci_dev { }; diff --git a/xen/include/asm-arm/setup.h b/xen/include/asm-arm/setup.h index 95da0b7ab9cd..88d9673db817 100644 --- a/xen/include/asm-arm/setup.h +++ b/xen/include/asm-arm/setup.h @@ -2,6 +2,8 @@ #define __ARM_SETUP_H_ #include +#include +#include #define MIN_FDT_ALIGN 8 #define MAX_FDT_SIZE SZ_2M @@ -77,6 +79,14 @@ struct bootinfo { #endif }; +struct map_range_data +{ + struct domain *d; + p2m_type_t p2mt; + /* Set if mapping of the memory ranges must be skipped. */ + bool skip_mapping; +}; + extern struct bootinfo bootinfo; extern domid_t max_init_domid; @@ -124,6 +134,9 @@ void device_tree_get_reg(const __be32 **cell, u32 address_cells, u32 device_tree_get_u32(const void *fdt, int node, const char *prop_name, u32 dflt); +int map_range_to_domain(const struct dt_device_node *dev, + u64 addr, u64 len, void *data); + #endif /* * Local variables: