From patchwork Fri Sep 14 16:14:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10601019 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A943114BD for ; Fri, 14 Sep 2018 16:21:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 92B872AD92 for ; Fri, 14 Sep 2018 16:21:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 857632BBE0; Fri, 14 Sep 2018 16:21:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0B6692B343 for ; Fri, 14 Sep 2018 16:21:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728178AbeINVgp (ORCPT ); Fri, 14 Sep 2018 17:36:45 -0400 Received: from mta-01.yadro.com ([89.207.88.251]:60244 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728236AbeINVgp (ORCPT ); Fri, 14 Sep 2018 17:36:45 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id D86E740311; Fri, 14 Sep 2018 16:14:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received:received; s=mta-01; t=1536941650; x= 1538756051; bh=UKuMLgGefKSBrwEqOyUpN4tsI+xfhbIJx+qARWmSyts=; b=l pMd3dt4vxVGPleprRIGt/6b6cpIRVVeJabXrXo46IrSltgosdRo+hOyHLw+k4BDi 932OZ7hLz3U+whn26AR3aILlqju9/tEOV+66nfIaoGYQlhUm7Rf4pwOVo3OU1TqB ndDzADYIdlZC5SRG58v/79qZhTVEYWltRSb4xmVpX0= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aW_jfnbYWg7z; Fri, 14 Sep 2018 19:14:10 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id D31BE411FC; Fri, 14 Sep 2018 19:14:10 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 14 Sep 2018 19:14:10 +0300 From: Sergey Miroshnichenko To: CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH RFC 1/4] PCI: hotplug: Add parameter to put devices to reset during rescan Date: Fri, 14 Sep 2018 19:14:01 +0300 Message-ID: <20180914161404.4685-2-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180914161404.4685-1-s.miroshnichenko@yadro.com> References: <20180914161404.4685-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Introduce a new command line option "pci=pcie_movable_bars" that indicates support of PCIe hotplug without prior reservation of memory regions by BIOS/bootloader. If a new PCIe device has been hot-plugged between two active ones, which have no (or not big enough) gap between their BARs, allocating new BARs requires to move BARs of the following working devices: 1) dev 4 | v .. | dev 3 | dev 3 | dev 5 | dev 7 | .. | BAR 0 | BAR 1 | BAR 0 | BAR 0 | 2) dev 4 | v .. | dev 3 | dev 3 | --> --> | dev 5 | dev 7 | .. | BAR 0 | BAR 1 | --> --> | BAR 0 | BAR 0 | 3) .. | dev 3 | dev 3 | dev 4 | dev 4 | dev 5 | dev 7 | .. | BAR 0 | BAR 1 | BAR 0 | BAR 1 | BAR 0 | BAR 0 | Not only BARs, but also bridge windows can be updated during a PCIe rescan, threatening all memory transactions during this procedure, so the PCI subsystem will instruct the drivers to pause by calling the reset_prepare() and reset_done() callbacks. If a device may be affected by BAR movement, the BAR changes tracking must be implemented in its driver. Signed-off-by: Sergey Miroshnichenko --- .../admin-guide/kernel-parameters.txt | 6 +++ drivers/pci/pci.c | 2 + drivers/pci/probe.c | 43 +++++++++++++++++++ include/linux/pci.h | 1 + 4 files changed, 52 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 64a3bf54b974..f8132a709061 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3311,6 +3311,12 @@ bridges without forcing it upstream. Note: this removes isolation between devices and may put more devices in an IOMMU group. + pcie_movable_bars Arrange a space at runtime for BARs of + hotplugged PCIe devices - usable if bootloader + doesn't reserve memory regions for them. Freeing + a space may require moving BARs of active devices + to higher addresses, so device drivers will be + paused during rescan. pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power Management. diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 1835f3a7aa8d..5f07a59b5924 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -6105,6 +6105,8 @@ static int __init pci_setup(char *str) pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS); } else if (!strncmp(str, "disable_acs_redir=", 18)) { disable_acs_redir_param = str + 18; + } else if (!strncmp(str, "pcie_movable_bars", 17)) { + pci_add_flags(PCI_MOVABLE_BARS); } else { printk(KERN_ERR "PCI: Unknown option `%s'\n", str); diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 201f9e5ff55c..bdaafc48dc4c 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -3138,6 +3138,45 @@ unsigned int pci_rescan_bus_bridge_resize(struct pci_dev *bridge) return max; } +/* + * Put all devices of the bus and its children to reset + */ +static void pci_bus_reset_prepare(struct pci_bus *bus) +{ + struct pci_dev *dev; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child = dev->subordinate; + + if (child) { + pci_bus_reset_prepare(child); + } else if (dev->driver && + dev->driver->err_handler && + dev->driver->err_handler->reset_prepare) { + dev->driver->err_handler->reset_prepare(dev); + } + } +} + +/* + * Complete the reset of all devices for the bus and its children + */ +static void pci_bus_reset_done(struct pci_bus *bus) +{ + struct pci_dev *dev; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child = dev->subordinate; + + if (child) { + pci_bus_reset_done(child); + } else if (dev->driver && dev->driver->err_handler && + dev->driver->err_handler->reset_done) { + dev->driver->err_handler->reset_done(dev); + } + } +} + /** * pci_rescan_bus - Scan a PCI bus for devices * @bus: PCI bus to scan @@ -3151,8 +3190,12 @@ unsigned int pci_rescan_bus(struct pci_bus *bus) { unsigned int max; + if (pci_has_flag(PCI_MOVABLE_BARS)) + pci_bus_reset_prepare(bus); max = pci_scan_child_bus(bus); pci_assign_unassigned_bus_resources(bus); + if (pci_has_flag(PCI_MOVABLE_BARS)) + pci_bus_reset_done(bus); pci_bus_add_devices(bus); return max; diff --git a/include/linux/pci.h b/include/linux/pci.h index 6925828f9f25..a8cb1a367c34 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -847,6 +847,7 @@ enum { PCI_ENABLE_PROC_DOMAINS = 0x00000010, /* Enable domains in /proc */ PCI_COMPAT_DOMAIN_0 = 0x00000020, /* ... except domain 0 */ PCI_SCAN_ALL_PCIE_DEVS = 0x00000040, /* Scan all, not just dev 0 */ + PCI_MOVABLE_BARS = 0x00000080, /* Runtime BAR reassign after hotplug */ }; /* These external functions are only available when PCI support is enabled */ From patchwork Fri Sep 14 16:14:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10601021 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 623B114BD for ; Fri, 14 Sep 2018 16:21:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 488A12AD92 for ; Fri, 14 Sep 2018 16:21:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3C6612BBE0; Fri, 14 Sep 2018 16:21:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C46912AD92 for ; Fri, 14 Sep 2018 16:21:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728236AbeINVgp (ORCPT ); Fri, 14 Sep 2018 17:36:45 -0400 Received: from mta-01.yadro.com ([89.207.88.251]:60242 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727842AbeINVgp (ORCPT ); Fri, 14 Sep 2018 17:36:45 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 91578418F9; Fri, 14 Sep 2018 16:14:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received:received; s=mta-01; t=1536941651; x= 1538756052; bh=SKrDtpNn4YSQKcsVB1OaeMOWHwjsrxY+lpoSWVb/1GQ=; b=I 40PvnhMS3hmr0D043zydVy5UN7SH9jQEkDq9OjGPoStBhzDkvBbxkrSalvkk4tQr jByUiz3+xN9vPYikbqJqrEXVZv+rmUEVfuTLZmA/rHU86N/sulh3uAt8sakx3VWl UCq9eAyTMUCub/pdCCSfBEsBGqtHImdTVow7tiGayU= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CRQUpRlf-5NM; Fri, 14 Sep 2018 19:14:11 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 0B7044122E; Fri, 14 Sep 2018 19:14:11 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 14 Sep 2018 19:14:10 +0300 From: Sergey Miroshnichenko To: CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH RFC 2/4] PCI: Release and reassign resources from the root during rescan Date: Fri, 14 Sep 2018 19:14:02 +0300 Message-ID: <20180914161404.4685-3-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180914161404.4685-1-s.miroshnichenko@yadro.com> References: <20180914161404.4685-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If assigned resources don't contain holes to fit memory for hotplugged devices, these conflicting resources must be freed, sorted and reassigned. When resources are finally allocated and written to BARs, it's time to update bridge windows with pci_setup_bridge(). Signed-off-by: Sergey Miroshnichenko --- drivers/pci/pci.h | 8 ++++++++ drivers/pci/probe.c | 35 ++++++++++++++++++++++++++++++++++- drivers/pci/setup-bus.c | 12 ++++-------- 3 files changed, 46 insertions(+), 9 deletions(-) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 6e0d1528d471..cb157630f8d7 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -224,6 +224,11 @@ enum pci_bar_type { pci_bar_mem64, /* A 64-bit memory BAR */ }; +enum release_type { + leaf_only, + whole_subtree, +}; + int pci_configure_extended_tags(struct pci_dev *dev, void *ign); bool pci_bus_read_dev_vendor_id(struct pci_bus *bus, int devfn, u32 *pl, int crs_timeout); @@ -240,6 +245,9 @@ void __pci_bus_size_bridges(struct pci_bus *bus, void __pci_bus_assign_resources(const struct pci_bus *bus, struct list_head *realloc_head, struct list_head *fail_head); +void pci_bus_release_bridge_resources(struct pci_bus *bus, + unsigned long type, + enum release_type rel_type); bool pci_bus_clip_resource(struct pci_dev *dev, int idx); void pci_reassigndev_resource_alignment(struct pci_dev *dev); diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index bdaafc48dc4c..2c2b853454c2 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -3113,6 +3113,25 @@ struct pci_bus *pci_scan_bus(int bus, struct pci_ops *ops, } EXPORT_SYMBOL(pci_scan_bus); +static void pci_setup_new_bridges(struct pci_bus *bus) +{ + struct pci_dev *dev; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child; + + if (!pci_dev_is_added(dev)) + continue; + + child = dev->subordinate; + if (child) + pci_setup_new_bridges(child); + } + + if (bus->self) + pci_setup_bridge(bus); +} + /** * pci_rescan_bus_bridge_resize - Scan a PCI bus for devices * @bridge: PCI bridge for the bus to scan @@ -3189,11 +3208,25 @@ static void pci_bus_reset_done(struct pci_bus *bus) unsigned int pci_rescan_bus(struct pci_bus *bus) { unsigned int max; + struct pci_bus *root = bus; + + while (!pci_is_root_bus(root)) + root = root->parent; if (pci_has_flag(PCI_MOVABLE_BARS)) pci_bus_reset_prepare(bus); max = pci_scan_child_bus(bus); - pci_assign_unassigned_bus_resources(bus); + if (pci_has_flag(PCI_MOVABLE_BARS)) { + pci_bus_release_bridge_resources(bus, IORESOURCE_IO, whole_subtree); + pci_bus_release_bridge_resources(bus, IORESOURCE_MEM, whole_subtree); + pci_bus_release_bridge_resources(bus, + IORESOURCE_MEM_64 | IORESOURCE_PREFETCH, + whole_subtree); + pci_assign_unassigned_root_bus_resources(root); + } else { + pci_assign_unassigned_bus_resources(bus); + } + pci_setup_new_bridges(bus); if (pci_has_flag(PCI_MOVABLE_BARS)) pci_bus_reset_done(bus); pci_bus_add_devices(bus); diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 79b1824e83b4..fd5675bb501f 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1244,7 +1244,7 @@ void __pci_bus_size_bridges(struct pci_bus *bus, struct list_head *realloc_head) case PCI_CLASS_BRIDGE_PCI: pci_bridge_check_ranges(bus); - if (bus->self->is_hotplug_bridge) { + if (bus->self->is_hotplug_bridge && !pci_has_flag(PCI_MOVABLE_BARS)) { additional_io_size = pci_hotplug_io_size; additional_mem_size = pci_hotplug_mem_size; } @@ -1582,17 +1582,13 @@ static void pci_bridge_release_resources(struct pci_bus *bus, } } -enum release_type { - leaf_only, - whole_subtree, -}; /* * try to release pci bridge resources that is from leaf bridge, * so we can allocate big new one later */ -static void pci_bus_release_bridge_resources(struct pci_bus *bus, - unsigned long type, - enum release_type rel_type) +void pci_bus_release_bridge_resources(struct pci_bus *bus, + unsigned long type, + enum release_type rel_type) { struct pci_dev *dev; bool is_leaf_bridge = true; From patchwork Fri Sep 14 16:14:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10601023 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5839015A7 for ; Fri, 14 Sep 2018 16:21:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 426782A448 for ; Fri, 14 Sep 2018 16:21:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 350A22BBE0; Fri, 14 Sep 2018 16:21:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AB57B2A448 for ; Fri, 14 Sep 2018 16:21:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728012AbeINVgp (ORCPT ); Fri, 14 Sep 2018 17:36:45 -0400 Received: from mta-01.yadro.com ([89.207.88.251]:60246 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728178AbeINVgo (ORCPT ); Fri, 14 Sep 2018 17:36:44 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id B91B64122E; Fri, 14 Sep 2018 16:14:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received:received; s=mta-01; t=1536941651; x= 1538756052; bh=HxEg0hSCMXqPYz5u/o7EAYQzRg0m5i0GBjy7NvSR2AQ=; b=B f5gjcXhMurzY+2DuKv3mi9G+7D2/1tGklo8DqrTbDNWkTqCrkYkcUBJTfff8f6KM kAp+E+NhBK2jh4I9kZRX0F3swrPNdSL/u291b+uPa6dOKQeqwFPXbY8ZP0yPOQcC tEO3sEv8c5e8SqZzEoAtnWyGd9/KrqvglzySYyp+HM= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id spDUaVe5CqYA; Fri, 14 Sep 2018 19:14:11 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 3E0E5418A4; Fri, 14 Sep 2018 19:14:11 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 14 Sep 2018 19:14:10 +0300 From: Sergey Miroshnichenko To: CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH RFC 3/4] PCI: Invalidate the released BAR resources Date: Fri, 14 Sep 2018 19:14:03 +0300 Message-ID: <20180914161404.4685-4-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180914161404.4685-1-s.miroshnichenko@yadro.com> References: <20180914161404.4685-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Otherwise after release_child_resources() there can be resources with the IORESOURCE_STARTALIGN flag remaining, but with start dropped to zero, that makes them not valid for re-assigning. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/setup-bus.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index fd5675bb501f..ec88461462b8 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1521,7 +1521,7 @@ static void __pci_bridge_assign_resources(const struct pci_dev *bridge, static void pci_bridge_release_resources(struct pci_bus *bus, unsigned long type) { - struct pci_dev *dev = bus->self; + struct pci_dev *dev = bus->self, *child_dev; struct resource *r; unsigned old_flags = 0; struct resource *b_res; @@ -1564,6 +1564,25 @@ static void pci_bridge_release_resources(struct pci_bus *bus, * all */ release_child_resources(r); + list_for_each_entry(child_dev, &bus->devices, bus_list) { + int i; + + if (child_dev->subordinate) + continue; + + for (i = 0; i <= PCI_ROM_RESOURCE; i++) { + struct resource *res = &child_dev->resource[i]; + resource_size_t size = resource_size(res); + + if (!res->flags) + continue; + + res->end = size - 1; + res->start = 0; + res->flags &= ~IORESOURCE_STARTALIGN; + res->flags |= IORESOURCE_SIZEALIGN; + } + } if (!release_resource(r)) { type = old_flags = r->flags & PCI_RES_TYPE_MASK; pci_printk(KERN_DEBUG, dev, "resource %d %pR released\n", From patchwork Fri Sep 14 16:14:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10601015 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B06E614BD for ; Fri, 14 Sep 2018 16:21:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 96DBD2B343 for ; Fri, 14 Sep 2018 16:21:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 89F952BBE0; Fri, 14 Sep 2018 16:21:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 307E82A448 for ; Fri, 14 Sep 2018 16:21:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728266AbeINVgo (ORCPT ); Fri, 14 Sep 2018 17:36:44 -0400 Received: from mta-01.yadro.com ([89.207.88.251]:60238 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726845AbeINVgo (ORCPT ); Fri, 14 Sep 2018 17:36:44 -0400 X-Greylist: delayed 441 seconds by postgrey-1.27 at vger.kernel.org; Fri, 14 Sep 2018 17:36:43 EDT Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 68C6A418FB; Fri, 14 Sep 2018 16:14:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received:received; s=mta-01; t=1536941652; x= 1538756053; bh=PdZSXUw7YG6FkWAzEo9haHTk94kYkJNqCMTtndwhZ3o=; b=q uJOKjOCndhZlW88T22tIt1X+LErWmsr8iW+CIkEzuFsRCh7sshsxusiuAfW26lGr kBk5/JhTjer3TXkQxt8qVVUqpxfo5DwHlc0SOnxK7ZC+9uKO5ZMG0IRWDk68Tqx2 e2KxcaK6s1qoIqEFmYd7WcjnS6x2rqD3cCMPhPr2Fw= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KBBTe_o-FZSg; Fri, 14 Sep 2018 19:14:12 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 66EA9418BF; Fri, 14 Sep 2018 19:14:11 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 14 Sep 2018 19:14:11 +0300 From: Sergey Miroshnichenko To: CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH RFC 4/4] PCI: Fix writing invalid BARs during pci_restore_state() Date: Fri, 14 Sep 2018 19:14:04 +0300 Message-ID: <20180914161404.4685-5-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180914161404.4685-1-s.miroshnichenko@yadro.com> References: <20180914161404.4685-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If BAR movement has happened (due to PCIe hotplug) after pci_save_state(), the saved addresses will become outdated. Restore them the most recently calculated values, not the ones stored in an arbitrary moment. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 5f07a59b5924..154130959443 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1328,7 +1328,7 @@ static void pci_restore_config_space(struct pci_dev *pdev) if (pdev->hdr_type == PCI_HEADER_TYPE_NORMAL) { pci_restore_config_space_range(pdev, 10, 15, 0); /* Restore BARs before the command register. */ - pci_restore_config_space_range(pdev, 4, 9, 10); + pci_restore_bars(pdev); pci_restore_config_space_range(pdev, 0, 3, 0); } else { pci_restore_config_space_range(pdev, 0, 15, 0);