From patchwork Wed Sep 28 23:48:49 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Konrad Rzeszutek Wilk X-Patchwork-Id: 9355359 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 6FBD460756 for ; Wed, 28 Sep 2016 23:51:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 63C1F2964D for ; Wed, 28 Sep 2016 23:51:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5811029651; Wed, 28 Sep 2016 23:51:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id DDB9F2964D for ; Wed, 28 Sep 2016 23:51:35 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bpObD-0000sc-QK; Wed, 28 Sep 2016 23:49:15 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bpObB-0000sU-RR for xen-devel@lists.xenproject.org; Wed, 28 Sep 2016 23:49:13 +0000 Received: from [85.158.139.211] by server-10.bemta-5.messagelabs.com id 9F/9E-16745-9F65CE75; Wed, 28 Sep 2016 23:49:13 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrBLMWRWlGSWpSXmKPExsXSO6nOVfdH2Jt wg+2zxS2+b5nM5MDocfjDFZYAxijWzLyk/IoE1ox1UzvZCmaoV5xoX8HYwPhRuouRi0NIoI1J Yt/cM0wQzhdGiUWb77N1MXICORsZJXp+6kAkJjBK3L55DKiKg4NNwETizSpHkBoRAVOJ7RNPs oDUMAvcY5Q4+3MpM0iNsECaxKHvbiA1LAKqEt2/2lhBbF4BN4nzSxaygpRICMhLPLtdDxLmFH CXaH20B6xTCKjk3jI/kLCEgKHE6YfbGCcw8i1gZFjFqF6cWlSWWqRrpJdUlJmeUZKbmJmja2h gqpebWlycmJ6ak5hUrJecn7uJERgiDECwg/H7H6dDjJIcTEqivJw+b8KF+JLyUyozEosz4otK c1KLDzHKcHAoSfD+CQXKCRalpqdWpGXmAIMVJi3BwaMkwtsGkuYtLkjMLc5Mh0idYlSUEufdB 5IQAElklObBtcEi5BKjrJQwLyPQIUI8BalFuZklqPKvGMU5GJWEeQ+DTOHJzCuBm/4KaDET0O KlJ16ALC5JREhJNTD6ZrO2ukcU7fJ7d34Ow8KNU5p3pVh5RK8u555/zSHh0tnY2r4JyTWyJ9I rpj79Vnk+vOcDC/+muILe1/N7Ks5PTu5Knuv6qkvQcMbyJ0WtNcc4TXTLW47q1c0P6bhc8urQ a79ZOVJikYs0bRMMdn1wkb3yan3i7jUzg+Pe5Wzk2Cbu+tmm0EqJpTgj0VCLuag4EQBBQMs2i wIAAA== X-Env-Sender: konrad.wilk@oracle.com X-Msg-Ref: server-12.tower-206.messagelabs.com!1475106550!25755447!1 X-Originating-IP: [141.146.126.69] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTQxLjE0Ni4xMjYuNjkgPT4gMjc3MjE4\n X-StarScan-Received: X-StarScan-Version: 8.84; banners=-,-,- X-VirusChecked: Checked Received: (qmail 44611 invoked from network); 28 Sep 2016 23:49:12 -0000 Received: from aserp1040.oracle.com (HELO aserp1040.oracle.com) (141.146.126.69) by server-12.tower-206.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 28 Sep 2016 23:49:12 -0000 Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u8SNn4W3002874 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 28 Sep 2016 23:49:05 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u8SNn4b1005697 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 28 Sep 2016 23:49:04 GMT Received: from abhmp0013.oracle.com (abhmp0013.oracle.com [141.146.116.19]) by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u8SNn3Im011445; Wed, 28 Sep 2016 23:49:03 GMT Received: from localhost.us.oracle.com (/10.154.161.113) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 28 Sep 2016 16:49:03 -0700 From: Konrad Rzeszutek Wilk To: xen-devel@lists.xenproject.org, konrad@kernel.org Date: Wed, 28 Sep 2016 19:48:49 -0400 Message-Id: <1475106529-17443-3-git-send-email-konrad.wilk@oracle.com> X-Mailer: git-send-email 2.4.11 In-Reply-To: <1475106529-17443-1-git-send-email-konrad.wilk@oracle.com> References: <1475106529-17443-1-git-send-email-konrad.wilk@oracle.com> X-Source-IP: userv0021.oracle.com [156.151.31.71] Cc: Andrew Cooper , Ian Jackson , Wei Liu , Jan Beulich , Konrad Rzeszutek Wilk Subject: [Xen-devel] [PATCH v2 2/2] hvmloader, pci: Don't try to relocate memory if 64-bit BAR is bigger than ~2GB X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Where ~2GB is actually 2GB minus MMIO space used for emulated devices and minus APIC space. There is no point. We can try to balloon out the memory between hvm_info->low_mem_pgend to pci_mem_end and we will never be able have a hole big enough for 2GB MMIO. As we can't go lower than 0x80000000 and can't go above 0xF0000000 which effectively leaves us with 1792MB of MMIO space (-32MB for VGA and -16 for platform device, so in reality: 1744MB). Instead just let it go above 4GB in the 64-bit zone. Note that prior to this patch the hvmloader would relocate as much memory as it could under 4GB: Low MMIO hole not large enough for all devices, relocating some BARs to 64-bit Relocating 0xffff pages from 0e0001000 to 210000000 for lowmem MMIO hole Relocating 0xffff pages from 0d0002000 to 21ffff000 for lowmem MMIO hole Relocating 0xffff pages from 0c0003000 to 22fffe000 for lowmem MMIO hole Relocating 0xffff pages from 0b0004000 to 23fffd000 for lowmem MMIO hole Relocating 0xffff pages from 0a0005000 to 24fffc000 for lowmem MMIO hole Relocating 0xffff pages from 090006000 to 25fffb000 for lowmem MMIO hole Relocating 0xffff pages from 080007000 to 26fffa000 for lowmem MMIO hole Relocating 0x7 pages from 080000000 to 27fff9000 for lowmem MMIO hole which is completely pointless. Signed-off-by: Konrad Rzeszutek Wilk --- Cc: Jan Beulich Cc: Andrew Cooper Cc: Ian Jackson Cc: Wei Liu v1: First submission. v2: Check for up to 1744MB. - Use defines for 4GB, 2GB. - Fix style guide issues. --- tools/firmware/hvmloader/pci.c | 33 ++++++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 5 deletions(-) diff --git a/tools/firmware/hvmloader/pci.c b/tools/firmware/hvmloader/pci.c index 416829d..f6194b9 100644 --- a/tools/firmware/hvmloader/pci.c +++ b/tools/firmware/hvmloader/pci.c @@ -80,7 +80,7 @@ void pci_setup(void) { uint8_t is_64bar, using_64bar, bar64_relocate = 0; uint32_t devfn, bar_reg, cmd, bar_data, bar_data_upper; - uint64_t base, bar_sz, bar_sz_upper, mmio_total = 0; + uint64_t base, bar_sz, bar_sz_upper, mmio_total = 0, mmio_64bit_total = 0; uint32_t vga_devfn = 256; uint16_t class, vendor_id, device_id; unsigned int bar, pin, link, isa_irq; @@ -97,6 +97,7 @@ void pci_setup(void) uint32_t devfn; uint32_t bar_reg; uint64_t bar_sz; + bool above_4gb; } *bars = (struct bars *)scratch_start; unsigned int i, nr_bars = 0; uint64_t mmio_hole_size = 0; @@ -265,11 +266,30 @@ void pci_setup(void) bars[i].devfn = devfn; bars[i].bar_reg = bar_reg; bars[i].bar_sz = bar_sz; + bars[i].above_4gb = false; if ( ((bar_data & PCI_BASE_ADDRESS_SPACE) == PCI_BASE_ADDRESS_SPACE_MEMORY) || (bar_reg == PCI_ROM_ADDRESS) ) - mmio_total += bar_sz; + { + /* + * If bigger than 2GB minus emulated devices BAR space and + * APIC space, then don't try to put under 4GB. + */ + if ( is_64bar && (mmio_total >= GB(2) || bar_sz >= + (GB(2) - HVM_BELOW_4G_MMIO_LENGTH - mmio_total)) ) + { + mmio_64bit_total += bar_sz; + bars[i].above_4gb = true; + /* + * As this may not trigger now that mmio_total could be + * less than 2GB, so force it. + */ + bar64_relocate = 1; + } + else + mmio_total += bar_sz; + } nr_bars++; @@ -349,7 +369,7 @@ void pci_setup(void) pci_mem_start = hvm_info->low_mem_pgend << PAGE_SHIFT; } - if ( mmio_total > (pci_mem_end - pci_mem_start) ) + if ( mmio_total > (pci_mem_end - pci_mem_start) || bar64_relocate ) { printf("Low MMIO hole not large enough for all devices," " relocating some BARs to 64-bit\n"); @@ -431,7 +451,7 @@ void pci_setup(void) * Should either of those two conditions change, this code will break. */ using_64bar = bars[i].is_64bar && bar64_relocate - && (mmio_total > (mem_resource.max - mem_resource.base)); + && ((mmio_total + mmio_64bit_total) > (mem_resource.max - mem_resource.base)); bar_data = pci_readl(devfn, bar_reg); if ( (bar_data & PCI_BASE_ADDRESS_SPACE) == @@ -451,7 +471,10 @@ void pci_setup(void) resource = &mem_resource; bar_data &= ~PCI_BASE_ADDRESS_MEM_MASK; } - mmio_total -= bar_sz; + if ( bars[i].above_4gb ) + mmio_64bit_total -= bar_sz; + else + mmio_total -= bar_sz; } else {