From patchwork Fri Jan 11 15:36:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758303 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9F5351399 for ; Fri, 11 Jan 2019 15:44:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8EFEE2848B for ; Fri, 11 Jan 2019 15:44:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 83F49284E4; Fri, 11 Jan 2019 15:44:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D0629285C3 for ; Fri, 11 Jan 2019 15:43:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731890AbfAKPnz (ORCPT ); Fri, 11 Jan 2019 10:43:55 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40082 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728632AbfAKPnz (ORCPT ); Fri, 11 Jan 2019 10:43:55 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id A2CDC41998 for ; Fri, 11 Jan 2019 15:37:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221041; x=1549035442; bh=/YNOeYvN//0sDrgfxAmyZR3l8GZSRW78skv 94n8Swe0=; b=VVE75gvyH6k3rTGTawFhQihD2AxMfWzCqv91yYEsZbzYF/pjHU4 rol9qr04tLIS5RHVboxKm9HzpAgNMuuTutxwhnF06NlJcKCiCVThGrstrv1K/DBP cpYvo6X7s5JIsFShSDCA5v17rpReUyxIhI0ENqvwPlvYq/XVbAL0URHU= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AGBiW2CBahRR for ; Fri, 11 Jan 2019 18:37:21 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id C0ECF4196E for ; Fri, 11 Jan 2019 18:37:21 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:21 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 01/21] PCI: Fix writing invalid BARs during pci_restore_state() Date: Fri, 11 Jan 2019 18:36:47 +0300 Message-ID: <20190111153707.10140-2-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If BAR movement has happened (due to PCIe hotplug) after pci_save_state(), the saved addresses will become outdated. Restore them the most recently calculated values, not the ones stored in an parbitrary moment. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index c9d8e3c837de..5580ddc5b9b1 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1337,7 +1337,7 @@ static void pci_restore_config_space(struct pci_dev *pdev) if (pdev->hdr_type == PCI_HEADER_TYPE_NORMAL) { pci_restore_config_space_range(pdev, 10, 15, 0, false); /* Restore BARs before the command register. */ - pci_restore_config_space_range(pdev, 4, 9, 10, false); + pci_restore_bars(pdev); pci_restore_config_space_range(pdev, 0, 3, 0, false); } else if (pdev->hdr_type == PCI_HEADER_TYPE_BRIDGE) { pci_restore_config_space_range(pdev, 12, 15, 0, false); From patchwork Fri Jan 11 15:36:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758311 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A5D6217D2 for ; Fri, 11 Jan 2019 15:44:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 93EF92848B for ; Fri, 11 Jan 2019 15:44:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 888EF284E4; Fri, 11 Jan 2019 15:44:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 33DDD284C0 for ; Fri, 11 Jan 2019 15:44:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731859AbfAKPoA (ORCPT ); Fri, 11 Jan 2019 10:44:00 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40072 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731856AbfAKPny (ORCPT ); Fri, 11 Jan 2019 10:43:54 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 935E7419A3 for ; Fri, 11 Jan 2019 15:37:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221042; x=1549035443; bh=TZdUArR/IO2LVbq+K7IE2oXSwBkExscQpAN CTOIzSAc=; b=t74pPX8J9AkQGanT2gAYbVlBfryR2l1Vmz+ZgOfLHApkbmbIrz2 SMpLQKJEK20D059dNzSuOOA8OPa0ePyjVg8JN7/vr2JnQeHtxsAdzcf2vRniOfc2 YY1Xm9RQxLD8N0OkJtjYTHVk0JyeffFl6Ppt1KT/qifKM8hQaiFUvD9Q= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hwTmKe1IAuRr for ; Fri, 11 Jan 2019 18:37:22 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id F210C41973 for ; Fri, 11 Jan 2019 18:37:21 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:21 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 02/21] PCI: Define PCI-specific version of the release_child_resources() Date: Fri, 11 Jan 2019 18:36:48 +0300 Message-ID: <20190111153707.10140-3-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Make the released resources of a bridge valid for later re-assignment: clear the STARTALIGN flad. Resources marked with PCI_FIXED must preserve their offset and size. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/setup-bus.c | 44 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index ed960436df5e..bd728160c517 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1520,6 +1520,48 @@ static void __pci_bridge_assign_resources(const struct pci_dev *bridge, (IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\ IORESOURCE_MEM_64) +/* + * Similar to release_child_resources(), but aware of PCI_FIXED and STARTALIGN flags + */ +static void pci_release_child_resources(struct resource *r) +{ + struct resource *tmp, *p; + + if (!r) + return; + + p = r->child; + r->child = NULL; + while (p) { + resource_size_t size = resource_size(p); + + tmp = p; + p = p->sibling; + + tmp->parent = NULL; + tmp->sibling = NULL; + pci_release_child_resources(tmp); + + if (!tmp->flags) + continue; + + if (tmp->flags & IORESOURCE_PCI_FIXED) { + pr_debug("PCI: release fixed %pR (%s), keep its flags, base and size\n", + tmp, tmp->name); + continue; + } + + pr_debug("PCI: release %pR (%s)\n", tmp, tmp->name); + + /* need to restore size, and keep all the flags but STARTALIGN */ + tmp->start = 0; + tmp->end = size - 1; + + tmp->flags &= ~IORESOURCE_STARTALIGN; + tmp->flags |= IORESOURCE_SIZEALIGN; + } +} + static void pci_bridge_release_resources(struct pci_bus *bus, unsigned long type) { @@ -1565,7 +1607,7 @@ static void pci_bridge_release_resources(struct pci_bus *bus, * if there are children under that, we should release them * all */ - release_child_resources(r); + pci_release_child_resources(r); if (!release_resource(r)) { type = old_flags = r->flags & PCI_RES_TYPE_MASK; pci_printk(KERN_DEBUG, dev, "resource %d %pR released\n", From patchwork Fri Jan 11 15:36:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758281 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6AF34746 for ; Fri, 11 Jan 2019 15:43:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5A41A2848B for ; Fri, 11 Jan 2019 15:43:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4C06728574; Fri, 11 Jan 2019 15:43:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D6DC52848B for ; Fri, 11 Jan 2019 15:43:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731970AbfAKPnz (ORCPT ); Fri, 11 Jan 2019 10:43:55 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40066 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731088AbfAKPny (ORCPT ); Fri, 11 Jan 2019 10:43:54 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id C06FD419B9 for ; Fri, 11 Jan 2019 15:37:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221042; x=1549035443; bh=hCcqWUEOeHKpsKqTX40npQA0JC4L81ub3lt tuK53l5M=; b=Xh5HaAgdTGsANecJOtbq6xfVGK0xwDm7zkROkpJVD2cJ6Z04lY4 8N1z7u6TE1b/u0GDdslMyD2m0Cl7GyFacqLDzdWVrKkvFO8smGLHWe2nuVAyOrgc Z4jU8a3lCs8AvB+HBtTS4j4MuQuGDWGhZ0DeC86zy+fyDZUxryBtZqbQ= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vaKQgf0lTJwT for ; Fri, 11 Jan 2019 18:37:22 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 2B52741994 for ; Fri, 11 Jan 2019 18:37:22 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:21 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 03/21] PCI: hotplug: Add a flag for the movable BARs feature Date: Fri, 11 Jan 2019 18:36:49 +0300 Message-ID: <20190111153707.10140-4-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If a new PCIe device has been hot-plugged between the two active ones without big enough gap between their BARs, these BARs should be moved if their drivers support this feature. The drivers should be notified and paused during the procedure: 1) dev 8 (new) | v .. | dev 3 | dev 3 | dev 5 | dev 7 | .. | BAR 0 | BAR 1 | BAR 0 | BAR 0 | 2) dev 8 | v .. | dev 3 | dev 3 | --> --> | dev 5 | dev 7 | .. | BAR 0 | BAR 1 | --> --> | BAR 0 | BAR 0 | 3) .. | dev 3 | dev 3 | dev 8 | dev 8 | dev 5 | dev 7 | .. | BAR 0 | BAR 1 | BAR 0 | BAR 1 | BAR 0 | BAR 0 | Thus, prior reservation of memory regions by BIOS/bootloader/firmware is not required anymore for the PCIe hotplug. The PCI_MOVABLE_BARS flag is set by the platform is this feature is supported and tested, but can be overridden by the following command line option: pcie_movable_bars={ off | force } Signed-off-by: Sergey Miroshnichenko --- .../admin-guide/kernel-parameters.txt | 7 ++++++ drivers/pci/pci.c | 24 +++++++++++++++++++ include/linux/pci.h | 2 ++ 3 files changed, 33 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index b799bcf67d7b..2165c4b5aea6 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3387,6 +3387,13 @@ nomsi Do not use MSI for native PCIe PME signaling (this makes all PCIe root ports use INTx for all services). + pcie_movable_bars=[PCIE] + Override the movable BARs support detection: + off + Disable even if supported by the platform + force + Enable even if not explicitly declared as supported + pcmv= [HW,PCMCIA] BadgePAD 4 pd_ignore_unused diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 5580ddc5b9b1..71e056771e67 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -139,6 +139,30 @@ static int __init pcie_port_pm_setup(char *str) } __setup("pcie_port_pm=", pcie_port_pm_setup); +static bool pcie_movable_bars_off; +static bool pcie_movable_bars_force; +static int __init pcie_movable_bars_setup(char *str) +{ + if (!strcmp(str, "off")) + pcie_movable_bars_off = true; + else if (!strcmp(str, "force")) + pcie_movable_bars_force = true; + return 1; +} +__setup("pcie_movable_bars=", pcie_movable_bars_setup); + +bool pci_movable_bars_enabled(void) +{ + if (pcie_movable_bars_off) + return false; + + if (pcie_movable_bars_force) + return true; + + return pci_has_flag(PCI_MOVABLE_BARS); +} +EXPORT_SYMBOL(pci_movable_bars_enabled); + /* Time to wait after a reset for device to become responsive */ #define PCIE_RESET_READY_POLL_MS 60000 diff --git a/include/linux/pci.h b/include/linux/pci.h index 65f1d8c2f082..84751f0931cd 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -862,6 +862,7 @@ enum { PCI_ENABLE_PROC_DOMAINS = 0x00000010, /* Enable domains in /proc */ PCI_COMPAT_DOMAIN_0 = 0x00000020, /* ... except domain 0 */ PCI_SCAN_ALL_PCIE_DEVS = 0x00000040, /* Scan all, not just dev 0 */ + PCI_MOVABLE_BARS = 0x00000080, /* Runtime BAR reassign after hotplug */ }; /* These external functions are only available when PCI support is enabled */ @@ -1341,6 +1342,7 @@ unsigned char pci_bus_max_busnr(struct pci_bus *bus); void pci_setup_bridge(struct pci_bus *bus); resource_size_t pcibios_window_alignment(struct pci_bus *bus, unsigned long type); +bool pci_movable_bars_enabled(void); #define PCI_VGA_STATE_CHANGE_BRIDGE (1 << 0) #define PCI_VGA_STATE_CHANGE_DECODES (1 << 1) From patchwork Fri Jan 11 15:36:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758275 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0DCDB1874 for ; Fri, 11 Jan 2019 15:43:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F10DD284C0 for ; Fri, 11 Jan 2019 15:43:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E54BF284E4; Fri, 11 Jan 2019 15:43:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 87CE12848B for ; Fri, 11 Jan 2019 15:43:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729463AbfAKPnw (ORCPT ); Fri, 11 Jan 2019 10:43:52 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40064 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731201AbfAKPnw (ORCPT ); Fri, 11 Jan 2019 10:43:52 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 8BDE4419BE for ; Fri, 11 Jan 2019 15:37:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221043; x=1549035444; bh=tJIKNtiK4mfCf7iorpFwf9wts60UENHeARd d3iz4n8s=; b=t9i+E4xMO8ReFZHZ07vgYQ/JLAAcT6Y4PGwlXTXPpMST7wn23ro uAyxMuPG+g0qRlSBxJ/EUckrXt6UtncOpQ1wzqGOhVYFbzngi9egoa+kqUIedDxY z6eUpxF/flg/gp5zMGoNOjMvZTF8VLswa4ASgoLqplYTbCjhIvl5VQqQ= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3CikvC8qHqze for ; Fri, 11 Jan 2019 18:37:23 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 5926A41996 for ; Fri, 11 Jan 2019 18:37:22 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:22 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 04/21] PCI: Pause the devices with movable BARs during rescan Date: Fri, 11 Jan 2019 18:36:50 +0300 Message-ID: <20190111153707.10140-5-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Drivers indicate their support of movable BARs by implementing the new rescan_prepare() and rescan_done() hooks in the struct pci_driver. All device's activity must be stopped during a rescan, and iounmap() +ioremap() must be applied to every used BAR. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/probe.c | 51 +++++++++++++++++++++++++++++++++++++++++++-- include/linux/pci.h | 2 ++ 2 files changed, 51 insertions(+), 2 deletions(-) diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 257b9f6f2ebb..5243a52a4081 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -3171,6 +3171,38 @@ unsigned int pci_rescan_bus_bridge_resize(struct pci_dev *bridge) return max; } +static void pci_bus_rescan_prepare(struct pci_bus *bus) +{ + struct pci_dev *dev; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child = dev->subordinate; + + if (child) { + pci_bus_rescan_prepare(child); + } else if (dev->driver && + dev->driver->rescan_prepare) { + dev->driver->rescan_prepare(dev); + } + } +} + +static void pci_bus_rescan_done(struct pci_bus *bus) +{ + struct pci_dev *dev; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child = dev->subordinate; + + if (child) { + pci_bus_rescan_done(child); + } else if (dev->driver && + dev->driver->rescan_done) { + dev->driver->rescan_done(dev); + } + } +} + /** * pci_rescan_bus - Scan a PCI bus for devices * @bus: PCI bus to scan @@ -3184,8 +3216,23 @@ unsigned int pci_rescan_bus(struct pci_bus *bus) { unsigned int max; - max = pci_scan_child_bus(bus); - pci_assign_unassigned_bus_resources(bus); + if (pci_movable_bars_enabled()) { + struct pci_bus *root = bus; + + while (!pci_is_root_bus(root)) + root = root->parent; + + pci_bus_rescan_prepare(root); + + max = pci_scan_child_bus(root); + pci_assign_unassigned_root_bus_resources(root); + + pci_bus_rescan_done(root); + } else { + max = pci_scan_child_bus(bus); + pci_assign_unassigned_bus_resources(bus); + } + pci_bus_add_devices(bus); return max; diff --git a/include/linux/pci.h b/include/linux/pci.h index 84751f0931cd..d87bbf9e0627 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -776,6 +776,8 @@ struct pci_driver { int (*resume)(struct pci_dev *dev); /* Device woken up */ void (*shutdown)(struct pci_dev *dev); int (*sriov_configure)(struct pci_dev *dev, int num_vfs); /* On PF */ + void (*rescan_prepare)(struct pci_dev *dev); + void (*rescan_done)(struct pci_dev *dev); const struct pci_error_handlers *err_handler; const struct attribute_group **groups; struct device_driver driver; From patchwork Fri Jan 11 15:36:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758293 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DFDDB17D2 for ; Fri, 11 Jan 2019 15:43:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CECB8284E4 for ; Fri, 11 Jan 2019 15:43:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C38E328574; Fri, 11 Jan 2019 15:43:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5F3FA284C0 for ; Fri, 11 Jan 2019 15:43:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731907AbfAKPn5 (ORCPT ); Fri, 11 Jan 2019 10:43:57 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40088 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728632AbfAKPn5 (ORCPT ); Fri, 11 Jan 2019 10:43:57 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id BC393419BF for ; Fri, 11 Jan 2019 15:37:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221043; x=1549035444; bh=c4dNKThTDHLqZobO9JLvCJwC2InftpYS+lL XkZ+EQys=; b=QoWX2x52O29IQtc9CPTTDXldQtkJtp95w4rmY/asYmWMZh5mBXh 9Y95FdxtomjQWQEeGLu+zopupuBOOQ747fZeey/O24jxXFBYM1I4E8Ga/bZj23xK YTPShNE1mSI9L+QhXwqtwtBVWZTbBw0BzgDndLs9UIrTGjguQtPaJXDI= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id I_LOcJkzbbFL for ; Fri, 11 Jan 2019 18:37:23 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 9836A41997 for ; Fri, 11 Jan 2019 18:37:22 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:22 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 05/21] PCI: Release and reassign the root bridge resources during rescan Date: Fri, 11 Jan 2019 18:36:51 +0300 Message-ID: <20190111153707.10140-6-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When the movable BARs feature is enabled, don't rely on the memory gaps reserved by the BIOS/bootloader/firmware, but instead rearrange the BARs and bridge windows starting from the root. Endpoint device's BARs, after being released, are resorted and written back by the pci_assign_unassigned_root_bus_resources(). The last step of writing the recalculated windows to the bridges is done by the new pci_setup_bridges() function. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/pci.h | 1 + drivers/pci/probe.c | 22 ++++++++++++++++++++++ drivers/pci/setup-bus.c | 11 ++++++++++- 3 files changed, 33 insertions(+), 1 deletion(-) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 224d88634115..e06e8692a7b1 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -248,6 +248,7 @@ void __pci_bus_assign_resources(const struct pci_bus *bus, struct list_head *realloc_head, struct list_head *fail_head); bool pci_bus_clip_resource(struct pci_dev *dev, int idx); +void pci_bus_release_root_bridge_resources(struct pci_bus *bus); void pci_reassigndev_resource_alignment(struct pci_dev *dev); void pci_disable_bridge_window(struct pci_dev *dev); diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 5243a52a4081..964faa32e78a 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -3203,6 +3203,25 @@ static void pci_bus_rescan_done(struct pci_bus *bus) } } +static void pci_setup_bridges(struct pci_bus *bus) +{ + struct pci_dev *dev; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child; + + if (!pci_dev_is_added(dev) || pci_dev_is_ignored(dev)) + continue; + + child = dev->subordinate; + if (child) + pci_setup_bridges(child); + } + + if (bus->self) + pci_setup_bridge(bus); +} + /** * pci_rescan_bus - Scan a PCI bus for devices * @bus: PCI bus to scan @@ -3225,8 +3244,11 @@ unsigned int pci_rescan_bus(struct pci_bus *bus) pci_bus_rescan_prepare(root); max = pci_scan_child_bus(root); + + pci_bus_release_root_bridge_resources(root); pci_assign_unassigned_root_bus_resources(root); + pci_setup_bridges(root); pci_bus_rescan_done(root); } else { max = pci_scan_child_bus(bus); diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index bd728160c517..a7594c302bd9 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1613,7 +1613,7 @@ static void pci_bridge_release_resources(struct pci_bus *bus, pci_printk(KERN_DEBUG, dev, "resource %d %pR released\n", PCI_BRIDGE_RESOURCES + idx, r); /* keep the old size */ - r->end = resource_size(r) - 1; + r->end = pci_movable_bars_enabled() ? 0 : (resource_size(r) - 1); r->start = 0; r->flags = 0; @@ -1666,6 +1666,15 @@ static void pci_bus_release_bridge_resources(struct pci_bus *bus, pci_bridge_release_resources(bus, type); } +void pci_bus_release_root_bridge_resources(struct pci_bus *root_bus) +{ + pci_bus_release_bridge_resources(root_bus, IORESOURCE_IO, whole_subtree); + pci_bus_release_bridge_resources(root_bus, IORESOURCE_MEM, whole_subtree); + pci_bus_release_bridge_resources(root_bus, + IORESOURCE_MEM_64 | IORESOURCE_PREFETCH, + whole_subtree); +} + static void pci_bus_dump_res(struct pci_bus *bus) { struct resource *res; From patchwork Fri Jan 11 15:36:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758299 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6C2E917D2 for ; Fri, 11 Jan 2019 15:44:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5B4CA284C0 for ; Fri, 11 Jan 2019 15:44:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4F8BE285B0; Fri, 11 Jan 2019 15:44:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A53C8284C0 for ; Fri, 11 Jan 2019 15:43:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732022AbfAKPn6 (ORCPT ); Fri, 11 Jan 2019 10:43:58 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40086 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731971AbfAKPn5 (ORCPT ); Fri, 11 Jan 2019 10:43:57 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id D2DF0419C7 for ; Fri, 11 Jan 2019 15:37:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221044; x=1549035445; bh=qptX74TZrbdrLiox0yDoNefTa4VBTR4ylDg syy3SwBg=; b=Ro8MhW81TAaRWoFRC6aOL2bIL/P4o2JPwJJPthwtJL0eeOS2y0s e706rpH+LHL2Ez80JiturbxgLCKaoqaDoEH++3eifA2Enk8Ca5N/8VGRIOZ/+fAO sf84XLqgeir6SYmDNm+w0O5JhvqCnx9lO0Ber25bdEVcrJwEOP+meYQ8= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Y-bBhlcEefV2 for ; Fri, 11 Jan 2019 18:37:24 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id C732D41961 for ; Fri, 11 Jan 2019 18:37:22 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:22 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 06/21] PCI: Don't allow hotplugged devices to steal resources Date: Fri, 11 Jan 2019 18:36:52 +0300 Message-ID: <20190111153707.10140-7-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When movable BARs are enabled, the PCI subsystem at first releases all the bridge windows and then performs an attempt to assign new requested resources and re-assign the existing ones. If a hotplugged device gets its resources first, there could be no space left to re-assign resources of already working devices, which is unacceptable. If this happens, this patch marks one of the new devices with the new introduced flag PCI_DEV_IGNORE and retries the resource assignment. This patch adds a new res_mask bitmask to the struct pci_dev for storing the indices of assigned resources. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/bus.c | 5 ++ drivers/pci/pci.h | 11 +++++ drivers/pci/probe.c | 100 +++++++++++++++++++++++++++++++++++++++- drivers/pci/setup-bus.c | 15 ++++++ include/linux/pci.h | 1 + 5 files changed, 130 insertions(+), 2 deletions(-) diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c index 5cb40b2518f9..a9784144d6f2 100644 --- a/drivers/pci/bus.c +++ b/drivers/pci/bus.c @@ -311,6 +311,11 @@ void pci_bus_add_device(struct pci_dev *dev) { int retval; + if (pci_dev_is_ignored(dev)) { + pci_warn(dev, "%s: don't enable the ignored device\n", __func__); + return; + } + /* * Can not put in pci_device_add yet because resources * are not assigned yet for some devices. diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index e06e8692a7b1..56b905068ac5 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -366,6 +366,7 @@ static inline bool pci_dev_is_disconnected(const struct pci_dev *dev) /* pci_dev priv_flags */ #define PCI_DEV_ADDED 0 +#define PCI_DEV_IGNORE 1 static inline void pci_dev_assign_added(struct pci_dev *dev, bool added) { @@ -377,6 +378,16 @@ static inline bool pci_dev_is_added(const struct pci_dev *dev) return test_bit(PCI_DEV_ADDED, &dev->priv_flags); } +static inline void pci_dev_ignore(struct pci_dev *dev, bool ignore) +{ + assign_bit(PCI_DEV_IGNORE, &dev->priv_flags, ignore); +} + +static inline bool pci_dev_is_ignored(const struct pci_dev *dev) +{ + return test_bit(PCI_DEV_IGNORE, &dev->priv_flags); +} + #ifdef CONFIG_PCIEAER #include diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 964faa32e78a..b7d2769d4a0e 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -3171,6 +3171,23 @@ unsigned int pci_rescan_bus_bridge_resize(struct pci_dev *bridge) return max; } +static unsigned int pci_dev_res_mask(struct pci_dev *dev) +{ + unsigned int res_mask = 0; + int i; + + for (i = 0; i < PCI_BRIDGE_RESOURCES; i++) { + struct resource *r = &dev->resource[i]; + + if (!r->flags || (r->flags & IORESOURCE_UNSET) || !r->parent) + continue; + + res_mask |= (1 << i); + } + + return res_mask; +} + static void pci_bus_rescan_prepare(struct pci_bus *bus) { struct pci_dev *dev; @@ -3178,6 +3195,8 @@ static void pci_bus_rescan_prepare(struct pci_bus *bus) list_for_each_entry(dev, &bus->devices, bus_list) { struct pci_bus *child = dev->subordinate; + dev->res_mask = pci_dev_res_mask(dev); + if (child) { pci_bus_rescan_prepare(child); } else if (dev->driver && @@ -3222,6 +3241,84 @@ static void pci_setup_bridges(struct pci_bus *bus) pci_setup_bridge(bus); } +static struct pci_dev *pci_find_next_new_device(struct pci_bus *bus) +{ + struct pci_dev *dev; + + if (!bus) + return NULL; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child_bus = dev->subordinate; + + if (!pci_dev_is_added(dev) && !pci_dev_is_ignored(dev)) + return dev; + + if (child_bus) { + struct pci_dev *next_new_dev; + + next_new_dev = pci_find_next_new_device(child_bus); + if (next_new_dev) + return next_new_dev; + } + } + + return NULL; +} + +static bool pci_bus_validate_resources(struct pci_bus *bus) +{ + struct pci_dev *dev; + bool ret = true; + + if (!bus) + return false; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child = dev->subordinate; + unsigned int res_mask = pci_dev_res_mask(dev); + + if (pci_dev_is_ignored(dev)) + continue; + + if (dev->res_mask & ~res_mask) { + pci_err(dev, "%s: Non-re-enabled resources found: 0x%x -> 0x%x\n", + __func__, res_mask, dev->res_mask); + ret = false; + } + + if (child && !pci_bus_validate_resources(child)) + ret = false; + } + + return ret; +} + +static void pci_reassign_root_bus_resources(struct pci_bus *root) +{ + do { + struct pci_dev *next_new_dev; + + pci_bus_release_root_bridge_resources(root); + pci_assign_unassigned_root_bus_resources(root); + + if (pci_bus_validate_resources(root)) + break; + + next_new_dev = pci_find_next_new_device(root); + if (!next_new_dev) { + dev_err(&root->dev, "%s: failed to re-assign resources even after ignoring all the hotplugged devices\n", + __func__); + break; + } + + dev_warn(&root->dev, "%s: failed to re-assign resources, disable the next hotplugged device %s and retry\n", + __func__, dev_name(&next_new_dev->dev)); + + pci_dev_ignore(next_new_dev, true); + } while (true); +} + /** * pci_rescan_bus - Scan a PCI bus for devices * @bus: PCI bus to scan @@ -3245,8 +3342,7 @@ unsigned int pci_rescan_bus(struct pci_bus *bus) max = pci_scan_child_bus(root); - pci_bus_release_root_bridge_resources(root); - pci_assign_unassigned_root_bus_resources(root); + pci_reassign_root_bus_resources(root); pci_setup_bridges(root); pci_bus_rescan_done(root); diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index a7594c302bd9..e0c5e2c947d6 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -131,6 +131,9 @@ static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head) { int i; + if (pci_dev_is_ignored(dev)) + return; + for (i = 0; i < PCI_NUM_RESOURCES; i++) { struct resource *r; struct pci_dev_resource *dev_res, *tmp; @@ -181,6 +184,9 @@ static void __dev_sort_resources(struct pci_dev *dev, { u16 class = dev->class >> 8; + if (pci_dev_is_ignored(dev)) + return; + /* Don't touch classless devices or host bridges or ioapics. */ if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST) return; @@ -284,6 +290,9 @@ static void assign_requested_resources_sorted(struct list_head *head, int idx; list_for_each_entry(dev_res, head, list) { + if (pci_dev_is_ignored(dev_res->dev)) + continue; + res = dev_res->res; idx = res - &dev_res->dev->resource[0]; if (resource_size(res) && @@ -1028,6 +1037,9 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, list_for_each_entry(dev, &bus->devices, bus_list) { int i; + if (pci_dev_is_ignored(dev)) + continue; + for (i = 0; i < PCI_NUM_RESOURCES; i++) { struct resource *r = &dev->resource[i]; resource_size_t r_size; @@ -1385,6 +1397,9 @@ void __pci_bus_assign_resources(const struct pci_bus *bus, pbus_assign_resources_sorted(bus, realloc_head, fail_head); list_for_each_entry(dev, &bus->devices, bus_list) { + if (pci_dev_is_ignored(dev)) + continue; + pdev_assign_fixed_resources(dev); b = dev->subordinate; diff --git a/include/linux/pci.h b/include/linux/pci.h index d87bbf9e0627..5a7d2b7ff31f 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -369,6 +369,7 @@ struct pci_dev { */ unsigned int irq; struct resource resource[DEVICE_COUNT_RESOURCE]; /* I/O and memory regions + expansion ROMs */ + unsigned int res_mask; /* Bitmask of assigned resources */ bool match_driver; /* Skip attaching driver */ From patchwork Fri Jan 11 15:36:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758287 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 585081399 for ; Fri, 11 Jan 2019 15:43:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 483F92848B for ; Fri, 11 Jan 2019 15:43:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3C75828574; Fri, 11 Jan 2019 15:43:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DAEDA2848B for ; Fri, 11 Jan 2019 15:43:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731904AbfAKPn4 (ORCPT ); Fri, 11 Jan 2019 10:43:56 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40070 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731897AbfAKPnz (ORCPT ); Fri, 11 Jan 2019 10:43:55 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 9F757419C5 for ; Fri, 11 Jan 2019 15:37:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221044; x=1549035445; bh=W3NZ5OPez2hC2OkdzekDmgvN1EFGkTGcp2g a+RRv9Fc=; b=gnr5rc4JYZVmANCy83p02icXnrJxYLnQxmHskgw+TTlvbVF+6ao WFOqLmxmIumXWzj7TTrv6KSbUso75CS7CFFSXE6+ASkefL8VCP5yIShPnS1wC307 pm3mt6ySKPKot+cVPp8p8tiWhJwOEmAg+CuKMLlPVtyHtTP0vUnJTIW4= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id k-t3NWhdmkUT for ; Fri, 11 Jan 2019 18:37:24 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 0063C4199B for ; Fri, 11 Jan 2019 18:37:22 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:22 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 07/21] PCI: Mark immovable BARs with PCI_FIXED Date: Fri, 11 Jan 2019 18:36:53 +0300 Message-ID: <20190111153707.10140-8-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If a PCIe device driver doesn't yet have support for movable BARs, mark device's BARs with IORESOURCE_PCI_FIXED. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/probe.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index b7d2769d4a0e..6d272802b1ee 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -3202,6 +3202,21 @@ static void pci_bus_rescan_prepare(struct pci_bus *bus) } else if (dev->driver && dev->driver->rescan_prepare) { dev->driver->rescan_prepare(dev); + } else if (dev->driver || ((dev->class >> 8) == PCI_CLASS_DISPLAY_VGA)) { + int i; + + for (i = 0; i < PCI_NUM_RESOURCES; i++) { + struct resource *r = &dev->resource[i]; + + if (!r->flags || !r->parent || + (r->flags & IORESOURCE_UNSET) || + (r->flags & IORESOURCE_PCI_FIXED)) + continue; + + r->flags |= IORESOURCE_PCI_FIXED; + pci_warn(dev, "%s: no support for movable BARs, mark BAR %d (%pR) as fixed\n", + __func__, i, r); + } } } } From patchwork Fri Jan 11 15:36:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758269 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5C6C8746 for ; Fri, 11 Jan 2019 15:43:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 493642848B for ; Fri, 11 Jan 2019 15:43:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3999F28574; Fri, 11 Jan 2019 15:43:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D4FD32848B for ; Fri, 11 Jan 2019 15:43:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731835AbfAKPnw (ORCPT ); Fri, 11 Jan 2019 10:43:52 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40062 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731777AbfAKPnv (ORCPT ); Fri, 11 Jan 2019 10:43:51 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 82B8D419C8 for ; Fri, 11 Jan 2019 15:37:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221045; x=1549035446; bh=tpWsgAOLizXHJjAcfZNZlRhR6VTRi0YCnkM 7DrLR8d8=; b=b19kQNU2ac1LgBi6aAf0f0yGrE57aZfqX6dGl25CO5ZF8ow+n2r a0VQ4Y9JYkfnGgasEbgtgP9B6oIUSsULfZlVxR7LIoCvW8OLUpZyIDfMA3rnB/wl JEC4ZEZKiJ8ASBY49spTjZWn4vT3lhhY1rF/rQuwP8bnoAMBc4OMk6eQ= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Bx1ZDpzqhqI4 for ; Fri, 11 Jan 2019 18:37:25 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 2F8804199C for ; Fri, 11 Jan 2019 18:37:23 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:22 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 08/21] PCI: Include fixed BARs into the bus size calculating Date: Fri, 11 Jan 2019 18:36:54 +0300 Message-ID: <20190111153707.10140-9-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The only difference between the fixed and movable BARs is an offset preservation during the release+reassign procedure on PCIe rescan. When fixed BARs are included into the result of pbus_size_mem(), these BARs can be restricted: assign them to direct parents only. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/setup-bus.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index e0c5e2c947d6..99aa78668e9e 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1044,12 +1044,20 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, struct resource *r = &dev->resource[i]; resource_size_t r_size; - if (r->parent || (r->flags & IORESOURCE_PCI_FIXED) || + if (r->parent || ((r->flags & mask) != type && (r->flags & mask) != type2 && (r->flags & mask) != type3)) continue; r_size = resource_size(r); + + if (r->flags & IORESOURCE_PCI_FIXED) { + if (pci_movable_bars_enabled()) + size += r_size; + + continue; + } + #ifdef CONFIG_PCI_IOV /* put SRIOV requested res to the optional list */ if (realloc_head && i >= PCI_IOV_RESOURCES && @@ -1383,6 +1391,8 @@ static void pdev_assign_fixed_resources(struct pci_dev *dev) while (b && !r->parent) { assign_fixed_resource_on_bus(b, r); b = b->parent; + if (!r->parent && pci_movable_bars_enabled()) + break; } } } From patchwork Fri Jan 11 15:36:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758291 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6040A1399 for ; Fri, 11 Jan 2019 15:43:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4F1E62848B for ; Fri, 11 Jan 2019 15:43:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 438BC284E4; Fri, 11 Jan 2019 15:43:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E7D6E2848B for ; Fri, 11 Jan 2019 15:43:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731954AbfAKPn5 (ORCPT ); Fri, 11 Jan 2019 10:43:57 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40080 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731907AbfAKPn4 (ORCPT ); Fri, 11 Jan 2019 10:43:56 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id ACA194199C for ; Fri, 11 Jan 2019 15:37:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221045; x=1549035446; bh=w2WsEPtKzIZL+wyNvFgeoacIwtne0R6BKOf WLQ7P2EI=; b=Qq/0+dKvwiTFgztEZyLSYWzJ2yrjjdNPYlSUgzDYZcuhYOFRPcy NY5YiBKA7cg6RGq7jAy31ueWiCxM/2tNJbBMr+u0E+8CUlrTPC4mn3KTkam9XZm+ 1hSV0Kgw7LPm3Pd10RfR61ePGRSFXBUgyigOZX8Gf2HvOK0RUEs2W5GE= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dNAcFMi6xWLB for ; Fri, 11 Jan 2019 18:37:25 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 5A1DC419A0 for ; Fri, 11 Jan 2019 18:37:23 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:23 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 09/21] PCI: Don't reserve memory for hotplug when enabled movable BARs Date: Fri, 11 Jan 2019 18:36:55 +0300 Message-ID: <20190111153707.10140-10-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP pbus_size_mem() returns a precise amount of memory required to fit all the requested BARs and windows of children bridges. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/setup-bus.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 99aa78668e9e..3ed13abdf937 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1266,7 +1266,7 @@ void __pci_bus_size_bridges(struct pci_bus *bus, struct list_head *realloc_head) case PCI_CLASS_BRIDGE_PCI: pci_bridge_check_ranges(bus); - if (bus->self->is_hotplug_bridge) { + if (bus->self->is_hotplug_bridge && !pci_movable_bars_enabled()) { additional_io_size = pci_hotplug_io_size; additional_mem_size = pci_hotplug_mem_size; } From patchwork Fri Jan 11 15:36:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758273 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BA98C1399 for ; Fri, 11 Jan 2019 15:43:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AA1FF284C0 for ; Fri, 11 Jan 2019 15:43:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9EDAE284E4; Fri, 11 Jan 2019 15:43:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4DE19285B0 for ; Fri, 11 Jan 2019 15:43:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731201AbfAKPnw (ORCPT ); Fri, 11 Jan 2019 10:43:52 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40060 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731088AbfAKPnw (ORCPT ); Fri, 11 Jan 2019 10:43:52 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 680CA419C9 for ; Fri, 11 Jan 2019 15:37:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221046; x=1549035447; bh=wsOGUF0tnZahxUl8QCTD7BjM7tC0TeVABzj L0cOpQow=; b=IuwkuCfPgSGSdX14fI6MdRXHVgjvHBhpJ855hgFt+fBFBLemqHM GFO8rj0yu/gUjcysZ93jkLe030TDeLIZtVhvtAEzPrzmwRnZE0aHebEfOQnGVIP6 /U1Iju+mUyIs4f04iBFbuaK7huy85ySPuvKkytENPjKUG71tCUcPuPFg= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qdNpJCKLjHoS for ; Fri, 11 Jan 2019 18:37:26 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 91733419A2 for ; Fri, 11 Jan 2019 18:37:23 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:23 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 10/21] PCI: Allow the failed resources to be reassigned later Date: Fri, 11 Jan 2019 18:36:56 +0300 Message-ID: <20190111153707.10140-11-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Don't lose the size of the requested EP's BAR if it can't be fit in a current trial, so this can be retried. But a failed bridge window must be dropped and recalculated in the next trial. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/setup-bus.c | 3 ++- drivers/pci/setup-res.c | 14 ++++++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 3ed13abdf937..9b228a181e5c 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -309,7 +309,8 @@ static void assign_requested_resources_sorted(struct list_head *head, 0 /* don't care */, 0 /* don't care */); } - reset_resource(res); + if (!pci_movable_bars_enabled()) + reset_resource(res); } } } diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c index d8ca40a97693..d28b8ca47b91 100644 --- a/drivers/pci/setup-res.c +++ b/drivers/pci/setup-res.c @@ -298,6 +298,20 @@ static int _pci_assign_resource(struct pci_dev *dev, int resno, bus = dev->bus; while ((ret = __pci_assign_resource(bus, dev, resno, size, min_align))) { + if (pci_movable_bars_enabled()) { + if (resno >= PCI_BRIDGE_RESOURCES && + resno <= PCI_BRIDGE_RESOURCE_END) { + struct resource *res = dev->resource + resno; + + pci_err(dev, "%s: dropping the bridge window BAR %d (%s)\n", + __func__, resno, res->name); + res->start = 0; + res->end = 0; + res->flags = 0; + } + break; + } + if (!bus->parent || !bus->self->transparent) break; bus = bus->parent; From patchwork Fri Jan 11 15:36:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758297 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D7B2A1399 for ; Fri, 11 Jan 2019 15:43:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C60102848B for ; Fri, 11 Jan 2019 15:43:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BA6D6285B0; Fri, 11 Jan 2019 15:43:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 216F82848B for ; Fri, 11 Jan 2019 15:43:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728632AbfAKPn5 (ORCPT ); Fri, 11 Jan 2019 10:43:57 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40072 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731088AbfAKPn5 (ORCPT ); Fri, 11 Jan 2019 10:43:57 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id DFBEC419CA for ; Fri, 11 Jan 2019 15:37:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221046; x=1549035447; bh=EXry9Dro6vZ7MLVY6i1hJgWgs9J5E3pXm5L cGo8yVLU=; b=JHzuSyc6grDrxMSery0tptPjoSZ5I+G62fBLEdHYUeSAUXqBTb3 4zmWNupIaMEnKrYMYBqXyrXFKALgIRIl/mHwiO4AoZH9qDGwVQCxdUmwm0sZoyUa h0Y+dL6pPpXQDtJT0BK73SlAHSmdwh+x1twbE64RjDP2KX1ylxeyo8gE= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mO--JnLhIKQj for ; Fri, 11 Jan 2019 18:37:26 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id CCAB541994 for ; Fri, 11 Jan 2019 18:37:23 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:23 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 11/21] PCI: Calculate fixed areas of bridge windows based on fixed BARs Date: Fri, 11 Jan 2019 18:36:57 +0300 Message-ID: <20190111153707.10140-12-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP For every (IO, MEM, MEM64) bridge window, count the fixed resources of its children endpoints and children bridge windows: | <- BAR -> | | <- child bus fixed_range_hard -> | | <- fixed BAR -> | | <- bus's fixed_range_hard -> | | <- bus's bridge window -> | These ranges will be later used to arrange bridge windows in a way which covers every immovable BAR as well as the movable ones during hotplug. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/pci.h | 14 +++++++ drivers/pci/probe.c | 82 +++++++++++++++++++++++++++++++++++++++++ drivers/pci/setup-bus.c | 17 +++++++++ include/linux/pci.h | 6 +++ 4 files changed, 119 insertions(+) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 56b905068ac5..14e3ebe68010 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -364,6 +364,20 @@ static inline bool pci_dev_is_disconnected(const struct pci_dev *dev) return dev->error_state == pci_channel_io_perm_failure; } +static inline int pci_get_bridge_resource_idx(struct resource *r) +{ + int idx = 1; + + if (r->flags & IORESOURCE_IO) + idx = 0; + else if (!(r->flags & IORESOURCE_PREFETCH)) + idx = 1; + else if (r->flags & IORESOURCE_MEM_64) + idx = 2; + + return idx; +} + /* pci_dev priv_flags */ #define PCI_DEV_ADDED 0 #define PCI_DEV_IGNORE 1 diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 6d272802b1ee..34c769b25055 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -500,6 +500,7 @@ void pci_read_bridge_bases(struct pci_bus *child) static struct pci_bus *pci_alloc_bus(struct pci_bus *parent) { struct pci_bus *b; + int idx; b = kzalloc(sizeof(*b), GFP_KERNEL); if (!b) @@ -516,6 +517,11 @@ static struct pci_bus *pci_alloc_bus(struct pci_bus *parent) if (parent) b->domain_nr = parent->domain_nr; #endif + for (idx = 0; idx < PCI_BRIDGE_RESOURCE_NUM; ++idx) { + b->fixed_range_hard[idx].start = (resource_size_t)-1; + b->fixed_range_hard[idx].end = 0; + } + return b; } @@ -3256,6 +3262,81 @@ static void pci_setup_bridges(struct pci_bus *bus) pci_setup_bridge(bus); } +static void pci_bus_update_fixed_range_hard(struct pci_bus *bus) +{ + struct pci_dev *dev; + int idx; + resource_size_t start, end; + + for (idx = 0; idx < PCI_BRIDGE_RESOURCE_NUM; ++idx) { + bus->fixed_range_hard[idx].start = (resource_size_t)-1; + bus->fixed_range_hard[idx].end = 0; + } + + list_for_each_entry(dev, &bus->devices, bus_list) + if (dev->subordinate) + pci_bus_update_fixed_range_hard(dev->subordinate); + + list_for_each_entry(dev, &bus->devices, bus_list) { + int i; + + for (i = 0; i < PCI_BRIDGE_RESOURCES; ++i) { + struct resource *r = &dev->resource[i]; + + if (!r->flags || (r->flags & IORESOURCE_UNSET) || !r->parent) + continue; + + if (r->flags & IORESOURCE_PCI_FIXED) { + idx = pci_get_bridge_resource_idx(r); + start = bus->fixed_range_hard[idx].start; + end = bus->fixed_range_hard[idx].end; + + if (start > r->start) + start = r->start; + if (end < r->end) + end = r->end; + + if (bus->fixed_range_hard[idx].start != start || + bus->fixed_range_hard[idx].end != end) { + dev_dbg(&bus->dev, "%s: Found fixed 0x%llx-0x%llx in %s, expand the fixed bridge window %d to 0x%llx-0x%llx\n", + __func__, + (unsigned long long)r->start, + (unsigned long long)r->end, + dev_name(&dev->dev), idx, + (unsigned long long)start, + (unsigned long long)end); + bus->fixed_range_hard[idx].start = start; + bus->fixed_range_hard[idx].end = end; + } + } + } + + if (dev->subordinate) { + struct pci_bus *child = dev->subordinate; + + for (idx = 0; idx < PCI_BRIDGE_RESOURCE_NUM; ++idx) { + start = bus->fixed_range_hard[idx].start; + end = bus->fixed_range_hard[idx].end; + + if (start > child->fixed_range_hard[idx].start) + start = child->fixed_range_hard[idx].start; + if (end < child->fixed_range_hard[idx].end) + end = child->fixed_range_hard[idx].end; + + if (start < bus->fixed_range_hard[idx].start || + end > bus->fixed_range_hard[idx].end) { + dev_dbg(&bus->dev, "%s: Expand the fixed bridge window %d from %s to 0x%llx-0x%llx\n", + __func__, idx, dev_name(&child->dev), + (unsigned long long)start, + (unsigned long long)end); + bus->fixed_range_hard[idx].start = start; + bus->fixed_range_hard[idx].end = end; + } + } + } + } +} + static struct pci_dev *pci_find_next_new_device(struct pci_bus *bus) { struct pci_dev *dev; @@ -3356,6 +3437,7 @@ unsigned int pci_rescan_bus(struct pci_bus *bus) pci_bus_rescan_prepare(root); max = pci_scan_child_bus(root); + pci_bus_update_fixed_range_hard(root); pci_reassign_root_bus_resources(root); diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 9b228a181e5c..15554c80b745 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -916,9 +916,17 @@ static void pbus_size_io(struct pci_bus *bus, resource_size_t min_size, resource_size_t children_add_size = 0; resource_size_t min_align, align; + resource_size_t fixed_start = bus->fixed_range_hard[0].start; + resource_size_t fixed_end = bus->fixed_range_hard[0].end; + resource_size_t fixed_size = (fixed_start < fixed_end) ? + (fixed_end - fixed_start + 1) : 0; + if (!b_res) return; + if (min_size < fixed_size) + min_size = fixed_size; + min_align = window_alignment(bus, IORESOURCE_IO); list_for_each_entry(dev, &bus->devices, bus_list) { int i; @@ -1027,6 +1035,15 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, resource_size_t children_add_size = 0; resource_size_t children_add_align = 0; resource_size_t add_align = 0; + bool is_mem64 = (mask & IORESOURCE_MEM_64); + + resource_size_t fixed_start = bus->fixed_range_hard[is_mem64 ? 2 : 1].start; + resource_size_t fixed_end = bus->fixed_range_hard[is_mem64 ? 2 : 1].end; + resource_size_t fixed_size = (fixed_start < fixed_end) ? + (fixed_end - fixed_start + 1) : 0; + + if (min_size < fixed_size) + min_size = fixed_size; if (!b_res) return -ENOSPC; diff --git a/include/linux/pci.h b/include/linux/pci.h index 5a7d2b7ff31f..11199717e4f5 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -568,6 +568,12 @@ struct pci_bus { struct list_head resources; /* Address space routed to this bus */ struct resource busn_res; /* Bus numbers routed to this bus */ + /* + * If there are fixed resources in the bridge window, the hard range + * contains the lowest and the highest addresses of them. + */ + struct resource fixed_range_hard[PCI_BRIDGE_RESOURCE_NUM]; + struct pci_ops *ops; /* Configuration access functions */ struct msi_controller *msi; /* MSI controller */ void *sysdata; /* Hook for sys-specific extension */ From patchwork Fri Jan 11 15:36:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758271 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 84D0E17D2 for ; Fri, 11 Jan 2019 15:43:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 73F242848B for ; Fri, 11 Jan 2019 15:43:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 67EB4284E4; Fri, 11 Jan 2019 15:43:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 016FA284C0 for ; Fri, 11 Jan 2019 15:43:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731777AbfAKPnw (ORCPT ); Fri, 11 Jan 2019 10:43:52 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40056 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729463AbfAKPnw (ORCPT ); Fri, 11 Jan 2019 10:43:52 -0500 X-Greylist: delayed 388 seconds by postgrey-1.27 at vger.kernel.org; Fri, 11 Jan 2019 10:43:51 EST Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 64DDF41994 for ; Fri, 11 Jan 2019 15:37:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221047; x=1549035448; bh=xKKfQimxkbOvsMzer03bhcqjQmz53zvcXZ2 VYDeUtU0=; b=kOtBZkR3tqt8qdPSfPKe/cg2FmaD2C4y5ytpKej72oXFei8oe8R 3yZXlr7ja9Jj6xnOn+7nr2T16f4Qp13BlrSCqpNehuRhpjtHjQeHeemcfJxMzZLt nQjtpKyMIyULt8xXiqcW9iks5y56NDUwy5Fauau5Ab5avMO/4K9WTax4= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id P-1sUuxcNAok for ; Fri, 11 Jan 2019 18:37:27 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 01DE3419BA for ; Fri, 11 Jan 2019 18:37:24 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:23 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 12/21] PCI: Calculate boundaries for bridge windows Date: Fri, 11 Jan 2019 18:36:58 +0300 Message-ID: <20190111153707.10140-13-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If a bridge window contains fixed areas (there are PCIe devices with immovable BARs located on this bus), this window must be allocated within the bound memory area, limited by windows size and by address range of fixed resources, calculated as follows: | <-- bus's fixed_range_hard --> | | <-- fixed_range_hard.end - window size --> | | <-- fixed_range_hard.start + window size --> | | <-- bus's fixed_range_soft --> | Signed-off-by: Sergey Miroshnichenko --- drivers/pci/setup-bus.c | 56 +++++++++++++++++++++++++++++++++++++++++ include/linux/pci.h | 4 ++- 2 files changed, 59 insertions(+), 1 deletion(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 15554c80b745..09226a201119 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1838,6 +1838,61 @@ static enum enable_type pci_realloc_detect(struct pci_bus *bus, } #endif +static void pci_bus_update_fixed_range_soft(struct pci_bus *bus) +{ + struct pci_dev *dev; + struct pci_bus *parent = bus->parent; + int idx; + + list_for_each_entry(dev, &bus->devices, bus_list) + if (dev->subordinate) + pci_bus_update_fixed_range_soft(dev->subordinate); + + if (!parent || !bus->self) + return; + + for (idx = 0; idx < ARRAY_SIZE(bus->fixed_range_hard); ++idx) { + struct resource *r; + resource_size_t soft_start, soft_end; + resource_size_t hard_start = bus->fixed_range_hard[idx].start; + resource_size_t hard_end = bus->fixed_range_hard[idx].end; + + if (hard_start > hard_end) + continue; + + r = bus->resource[idx]; + + soft_start = hard_end - resource_size(r) + 1; + soft_end = hard_start + resource_size(r) - 1; + + if (soft_start > hard_start) + soft_start = hard_start; + + if (soft_end < hard_end) + soft_end = hard_end; + + list_for_each_entry(dev, &parent->devices, bus_list) { + struct pci_bus *sibling = dev->subordinate; + resource_size_t s_start, s_end; + + if (!sibling || sibling == bus) + continue; + + s_start = sibling->fixed_range_hard[idx].start; + s_end = sibling->fixed_range_hard[idx].end; + + if (s_start > s_end) + continue; + + if (s_end < hard_start && s_end > soft_start) + soft_start = s_end; + } + + bus->fixed_range_soft[idx].start = soft_start; + bus->fixed_range_soft[idx].end = soft_end; + } +} + /* * first try will not touch pci bridge res * second and later try will clear small leaf bridge res @@ -1876,6 +1931,7 @@ void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus) /* Depth first, calculate sizes and alignments of all subordinate buses. */ __pci_bus_size_bridges(bus, add_list); + pci_bus_update_fixed_range_soft(bus); /* Depth last, allocate resources and update the hardware. */ __pci_bus_assign_resources(bus, add_list, &fail_head); diff --git a/include/linux/pci.h b/include/linux/pci.h index 11199717e4f5..161a9b8a854e 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -570,9 +570,11 @@ struct pci_bus { /* * If there are fixed resources in the bridge window, the hard range - * contains the lowest and the highest addresses of them. + * contains the lowest and the highest addresses of them, and this + * bridge window must reside within the soft range. */ struct resource fixed_range_hard[PCI_BRIDGE_RESOURCE_NUM]; + struct resource fixed_range_soft[PCI_BRIDGE_RESOURCE_NUM]; struct pci_ops *ops; /* Configuration access functions */ struct msi_controller *msi; /* MSI controller */ From patchwork Fri Jan 11 15:36:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758295 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0407A746 for ; Fri, 11 Jan 2019 15:43:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E872A2848B for ; Fri, 11 Jan 2019 15:43:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DD97A285A9; Fri, 11 Jan 2019 15:43:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 803742848B for ; Fri, 11 Jan 2019 15:43:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731972AbfAKPn5 (ORCPT ); Fri, 11 Jan 2019 10:43:57 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40090 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731990AbfAKPn5 (ORCPT ); Fri, 11 Jan 2019 10:43:57 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id E1FA0419CB for ; Fri, 11 Jan 2019 15:37:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221047; x=1549035448; bh=1neNanWU/dFgjiE8DqEcJeoJZq1kmX/krzB weDU3jZI=; b=AsEovkVEGmDq5IkWcT5DLDc+qOPgBj7w7TCxNNWA+3Bq2yS2rJs W7m1UEENf+tUs5yMVn5h9UfDX5vUyi5FQHc1G1uptBC90TUNhHSkCPKThOCXsXrz p+y4RYUd1H92qQRvvKUJy33LV++cAhycawJ5osR+9NcpawRoHrnN7gnQ= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id klbS_ujpyEFy for ; Fri, 11 Jan 2019 18:37:27 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 2E575419BB for ; Fri, 11 Jan 2019 18:37:24 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:23 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 13/21] PCI: Make sure bridge windows include their fixed BARs Date: Fri, 11 Jan 2019 18:36:59 +0300 Message-ID: <20190111153707.10140-14-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Consider previously calculated boundaries when allocating a bridge window, setting the lowest allowed address and checking the result. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/bus.c | 2 +- drivers/pci/setup-res.c | 31 +++++++++++++++++++++++++++++-- 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c index a9784144d6f2..ce2d2aeedbd3 100644 --- a/drivers/pci/bus.c +++ b/drivers/pci/bus.c @@ -192,7 +192,7 @@ static int pci_bus_alloc_from_region(struct pci_bus *bus, struct resource *res, * this is an already-configured bridge window, its start * overrides "min". */ - if (avail.start) + if (min_used < avail.start) min_used = avail.start; max = avail.end; diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c index d28b8ca47b91..bd017bf5a800 100644 --- a/drivers/pci/setup-res.c +++ b/drivers/pci/setup-res.c @@ -248,9 +248,22 @@ static int __pci_assign_resource(struct pci_bus *bus, struct pci_dev *dev, struct resource *res = dev->resource + resno; resource_size_t min; int ret; + resource_size_t start = (resource_size_t)-1; + resource_size_t end = 0; min = (res->flags & IORESOURCE_IO) ? PCIBIOS_MIN_IO : PCIBIOS_MIN_MEM; + if (dev->subordinate && resno >= PCI_BRIDGE_RESOURCES) { + struct pci_bus *child_bus = dev->subordinate; + int b_resno = resno - PCI_BRIDGE_RESOURCES; + resource_size_t soft_start = child_bus->fixed_range_soft[b_resno].start; + + start = child_bus->fixed_range_hard[b_resno].start; + end = child_bus->fixed_range_hard[b_resno].end; + if (start < end) + min = soft_start; + } + /* * First, try exact prefetching match. Even if a 64-bit * prefetchable bridge window is below 4GB, we can't put a 32-bit @@ -262,7 +275,7 @@ static int __pci_assign_resource(struct pci_bus *bus, struct pci_dev *dev, IORESOURCE_PREFETCH | IORESOURCE_MEM_64, pcibios_align_resource, dev); if (ret == 0) - return 0; + goto check_fixed; /* * If the prefetchable window is only 32 bits wide, we can put @@ -274,7 +287,7 @@ static int __pci_assign_resource(struct pci_bus *bus, struct pci_dev *dev, IORESOURCE_PREFETCH, pcibios_align_resource, dev); if (ret == 0) - return 0; + goto check_fixed; } /* @@ -287,6 +300,20 @@ static int __pci_assign_resource(struct pci_bus *bus, struct pci_dev *dev, ret = pci_bus_alloc_resource(bus, res, size, align, min, 0, pcibios_align_resource, dev); +check_fixed: + if (ret == 0 && start < end) { + if (res->start > start || res->end < end) { + dev_err(&bus->dev, "%s: fixed area 0x%llx-0x%llx for %s doesn't fit in the allocated %pR (0x%llx-0x%llx)", + __func__, + (unsigned long long)start, (unsigned long long)end, + dev_name(&dev->dev), + res, (unsigned long long)res->start, + (unsigned long long)res->end); + release_resource(res); + return -1; + } + } + return ret; } From patchwork Fri Jan 11 15:37:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758277 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 763111399 for ; Fri, 11 Jan 2019 15:43:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6517B2848B for ; Fri, 11 Jan 2019 15:43:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 59A25284E4; Fri, 11 Jan 2019 15:43:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EB2A72848B for ; Fri, 11 Jan 2019 15:43:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731901AbfAKPnw (ORCPT ); Fri, 11 Jan 2019 10:43:52 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40058 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728632AbfAKPnw (ORCPT ); Fri, 11 Jan 2019 10:43:52 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 66092419BB for ; Fri, 11 Jan 2019 15:37:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221048; x=1549035449; bh=CYWyPQs/QFu4sPDIapOVamTOlt1BY8q50vj BXf4I488=; b=eW/tZb2KxDoL2cHJ0vCxNPe8Y5aLo9iEwvwyJWdoLlKBu7gl3A4 dsxebfRo/YkE2mLKT2yRr0kYbtIBlteshKH+Md12zvdTXvKSxCJwN6XW/XsNLhqJ zDIjnmDydvKj3u7OY9L++vn6NpP9sMtFyqOFG9xS7s4wx3a2mRSidjUY= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jEFAW9iL72k7 for ; Fri, 11 Jan 2019 18:37:28 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 607AC419BC for ; Fri, 11 Jan 2019 18:37:24 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:24 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 14/21] PCI: Prioritize fixed BAR assigning over the movable ones Date: Fri, 11 Jan 2019 18:37:00 +0300 Message-ID: <20190111153707.10140-15-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The allocated bridge windows are big enough to house all the children bridges and BARs, but the fixed resources must be assigned first, so the movable ones later divide the rest of the window. That's the assignment order: 1. Bridge windows with fixed areas; 2. The rest of bridge windows; 3. Fixed BARs of direct children EPs; 4. The rest of BARs. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/setup-bus.c | 69 ++++++++++++++++++++++++++++++++--------- 1 file changed, 55 insertions(+), 14 deletions(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 09226a201119..10ddf08b1625 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -272,31 +272,54 @@ static void reassign_resources_sorted(struct list_head *realloc_head, } } -/** - * assign_requested_resources_sorted() - satisfy resource requests - * - * @head : head of the list tracking requests for resources - * @fail_head : head of the list tracking requests that could - * not be allocated - * - * Satisfy resource requests of each element in the list. Add - * requests that could not satisfied to the failed_list. - */ -static void assign_requested_resources_sorted(struct list_head *head, - struct list_head *fail_head) +enum assign_step { + assign_fixed_bridge_windows, + assign_fixed_resources, + assign_float_resources, +}; + +static void _assign_requested_resources_sorted(struct list_head *head, + struct list_head *fail_head, + enum assign_step step) { struct resource *res; struct pci_dev_resource *dev_res; int idx; list_for_each_entry(dev_res, head, list) { + bool is_fixed; + bool is_fixed_bridge; + bool is_bridge; + if (pci_dev_is_ignored(dev_res->dev)) continue; res = dev_res->res; + if (!resource_size(res)) + continue; + idx = res - &dev_res->dev->resource[0]; - if (resource_size(res) && - pci_assign_resource(dev_res->dev, idx)) { + is_fixed = res->flags & IORESOURCE_PCI_FIXED; + is_bridge = dev_res->dev->subordinate && idx >= PCI_BRIDGE_RESOURCES; + + if (is_bridge) { + struct pci_bus *child = dev_res->dev->subordinate; + int b_res_idx = pci_get_bridge_resource_idx(res); + struct resource *fixed_res = &child->fixed_range_hard[b_res_idx]; + + is_fixed_bridge = fixed_res->start < fixed_res->end; + } else { + is_fixed_bridge = false; + } + + if (assign_fixed_bridge_windows == step && !is_fixed_bridge) + continue; + else if (assign_fixed_resources == step && (!is_fixed || is_bridge)) + continue; + else if (assign_float_resources == step && is_fixed) + continue; + + if (pci_assign_resource(dev_res->dev, idx)) { if (fail_head) { /* * if the failed res is for ROM BAR, and it will @@ -315,6 +338,24 @@ static void assign_requested_resources_sorted(struct list_head *head, } } +/** + * assign_requested_resources_sorted() - satisfy resource requests + * + * @head : head of the list tracking requests for resources + * @fail_head : head of the list tracking requests that could + * not be allocated + * + * Satisfy resource requests of each element in the list. Add + * requests that could not satisfied to the failed_list. + */ +static void assign_requested_resources_sorted(struct list_head *head, + struct list_head *fail_head) +{ + _assign_requested_resources_sorted(head, fail_head, assign_fixed_bridge_windows); + _assign_requested_resources_sorted(head, fail_head, assign_fixed_resources); + _assign_requested_resources_sorted(head, fail_head, assign_float_resources); +} + static unsigned long pci_fail_res_type_mask(struct list_head *fail_head) { struct pci_dev_resource *fail_res; From patchwork Fri Jan 11 15:37:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758283 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BD2A317D2 for ; Fri, 11 Jan 2019 15:43:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AB6702848B for ; Fri, 11 Jan 2019 15:43:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9FA3B284E4; Fri, 11 Jan 2019 15:43:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 37018284C0 for ; Fri, 11 Jan 2019 15:43:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731978AbfAKPnz (ORCPT ); Fri, 11 Jan 2019 10:43:55 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40076 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731890AbfAKPny (ORCPT ); Fri, 11 Jan 2019 10:43:54 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id F27D7419CC for ; Fri, 11 Jan 2019 15:37:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221048; x=1549035449; bh=b+JeZ61WD50Ta00T2UkPhromaBKmR9w433T eAZj3MaI=; b=f6x5FAlbVVDGnBFHmS85kQzs/N/hqvaItXeeSAQSGXrBKcYva54 WMcDlp14cpxbyTlfEb+iXTu2X/Br0SGXKmYspc+9wLXKMuqVBfZED+E0Tr207ifZ pqk+aO4tE3VWm3SyG8Xb4pBT1krcMOsKQbE32/WjFudzZ0FsJmP40/mY= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PIEC7nbw6B6J for ; Fri, 11 Jan 2019 18:37:28 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 8A017419BD for ; Fri, 11 Jan 2019 18:37:24 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:24 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 15/21] PCI: Fix race condition in pci_enable/disable_device() Date: Fri, 11 Jan 2019 18:37:01 +0300 Message-ID: <20190111153707.10140-16-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP CPU0 CPU1 pci_enable_device_mem() pci_enable_device_mem() pci_enable_bridge() pci_enable_bridge() pci_is_enabled() return false; atomic_inc_return(enable_cnt) Start actual enabling the bridge ... pci_is_enabled() ... return true; ... Start memory requests <-- FAIL ... Set the PCI_COMMAND_MEMORY bit <-- Must wait for this This patch protects the pci_enable/disable_device() and pci_enable_bridge() with mutexes. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/pci.c | 26 ++++++++++++++++++++++---- drivers/pci/probe.c | 1 + include/linux/pci.h | 1 + 3 files changed, 24 insertions(+), 4 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 71e056771e67..a5266f31b612 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1595,6 +1595,8 @@ static void pci_enable_bridge(struct pci_dev *dev) struct pci_dev *bridge; int retval; + mutex_lock(&dev->enable_mutex); + bridge = pci_upstream_bridge(dev); if (bridge) pci_enable_bridge(bridge); @@ -1602,6 +1604,7 @@ static void pci_enable_bridge(struct pci_dev *dev) if (pci_is_enabled(dev)) { if (!dev->is_busmaster) pci_set_master(dev); + mutex_unlock(&dev->enable_mutex); return; } @@ -1610,11 +1613,14 @@ static void pci_enable_bridge(struct pci_dev *dev) pci_err(dev, "Error enabling bridge (%d), continuing\n", retval); pci_set_master(dev); + mutex_unlock(&dev->enable_mutex); } static int pci_enable_device_flags(struct pci_dev *dev, unsigned long flags) { struct pci_dev *bridge; + /* Enable-locking of bridges is performed within the pci_enable_bridge() */ + bool need_lock = !dev->subordinate; int err; int i, bars = 0; @@ -1630,8 +1636,13 @@ static int pci_enable_device_flags(struct pci_dev *dev, unsigned long flags) dev->current_state = (pmcsr & PCI_PM_CTRL_STATE_MASK); } - if (atomic_inc_return(&dev->enable_cnt) > 1) + if (need_lock) + mutex_lock(&dev->enable_mutex); + if (pci_is_enabled(dev)) { + if (need_lock) + mutex_unlock(&dev->enable_mutex); return 0; /* already enabled */ + } bridge = pci_upstream_bridge(dev); if (bridge) @@ -1646,8 +1657,10 @@ static int pci_enable_device_flags(struct pci_dev *dev, unsigned long flags) bars |= (1 << i); err = do_pci_enable_device(dev, bars); - if (err < 0) - atomic_dec(&dev->enable_cnt); + if (err >= 0) + atomic_inc(&dev->enable_cnt); + if (need_lock) + mutex_unlock(&dev->enable_mutex); return err; } @@ -1890,15 +1903,20 @@ void pci_disable_device(struct pci_dev *dev) if (dr) dr->enabled = 0; + mutex_lock(&dev->enable_mutex); dev_WARN_ONCE(&dev->dev, atomic_read(&dev->enable_cnt) <= 0, "disabling already-disabled device"); - if (atomic_dec_return(&dev->enable_cnt) != 0) + if (atomic_dec_return(&dev->enable_cnt) != 0) { + mutex_unlock(&dev->enable_mutex); return; + } do_pci_disable_device(dev); dev->is_busmaster = 0; + + mutex_unlock(&dev->enable_mutex); } EXPORT_SYMBOL(pci_disable_device); diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 34c769b25055..4726049a19ad 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -2197,6 +2197,7 @@ struct pci_dev *pci_alloc_dev(struct pci_bus *bus) INIT_LIST_HEAD(&dev->bus_list); dev->dev.type = &pci_dev_type; dev->bus = pci_bus_get(bus); + mutex_init(&dev->enable_mutex); return dev; } diff --git a/include/linux/pci.h b/include/linux/pci.h index 161a9b8a854e..311d51d0a59a 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -417,6 +417,7 @@ struct pci_dev { unsigned int no_vf_scan:1; /* Don't scan for VFs after IOV enablement */ pci_dev_flags_t dev_flags; atomic_t enable_cnt; /* pci_enable_device has been called */ + struct mutex enable_mutex; u32 saved_config_space[16]; /* Config space saved at suspend time */ struct hlist_head saved_cap_space; From patchwork Fri Jan 11 15:37:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758279 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BD0CB1399 for ; Fri, 11 Jan 2019 15:43:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AB6722848B for ; Fri, 11 Jan 2019 15:43:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9E646284E4; Fri, 11 Jan 2019 15:43:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 569202848B for ; Fri, 11 Jan 2019 15:43:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731969AbfAKPny (ORCPT ); Fri, 11 Jan 2019 10:43:54 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40074 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731859AbfAKPny (ORCPT ); Fri, 11 Jan 2019 10:43:54 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 41E44419BD for ; Fri, 11 Jan 2019 15:37:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221049; x=1549035450; bh=Yj9lGaxnkQxL4mgWzyDOoIwRhCzn6tHdSBJ wM8CucLg=; b=Cjj9yDbMDOWhTjKJFPyfsMGN9o9vvgH2ByE/tuyJdMQ87l8OFCQ KEXLjfOO3DSSCIwElKUmFWgV+3JTN8UwTi9wOTWQNWTXNGSfnm0PfNqnJGVBNz6s Z5bsOFGKRo/GXCi1bHbV67fBtXIXbwxykv9POjL+Q2uOUwrCTZ8ZZuBc= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CpbUxCh5zRfl for ; Fri, 11 Jan 2019 18:37:29 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id B514841996 for ; Fri, 11 Jan 2019 18:37:24 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:24 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 16/21] PCI: Enable bridge's I/O and MEM access for hotplugged devices Date: Fri, 11 Jan 2019 18:37:02 +0300 Message-ID: <20190111153707.10140-17-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP After updating the bridge window resources, the PCI_COMMAND_IO and PCI_COMMAND_MEMORY bits of the bridge must be addressed as well. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/pci.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index a5266f31b612..633aa2570008 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1602,6 +1602,14 @@ static void pci_enable_bridge(struct pci_dev *dev) pci_enable_bridge(bridge); if (pci_is_enabled(dev)) { + int i, bars = 0; + + for (i = PCI_BRIDGE_RESOURCES; i < DEVICE_COUNT_RESOURCE; i++) { + if (dev->resource[i].flags & (IORESOURCE_MEM | IORESOURCE_IO)) + bars |= (1 << i); + } + do_pci_enable_device(dev, bars); + if (!dev->is_busmaster) pci_set_master(dev); mutex_unlock(&dev->enable_mutex); From patchwork Fri Jan 11 15:37:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758301 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 79CDC746 for ; Fri, 11 Jan 2019 15:44:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6785E2848B for ; Fri, 11 Jan 2019 15:44:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5B9CA28632; Fri, 11 Jan 2019 15:44:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 00E64284E4 for ; Fri, 11 Jan 2019 15:43:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731088AbfAKPn5 (ORCPT ); Fri, 11 Jan 2019 10:43:57 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40066 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731972AbfAKPn4 (ORCPT ); Fri, 11 Jan 2019 10:43:56 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id D36B4419C1 for ; Fri, 11 Jan 2019 15:37:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221050; x=1549035451; bh=OgLo7/CRt0OwDQd8jzrs/7oiC/sWradzhSM CAbKv7SA=; b=A+PPu3+cR6D60gXDjJTdo47XtAIOARxDcahqLPpT7mmfInfjOqD VfFq1XxYqx6zgu/3jw5lqdrIoE0WeFCBwvVnShOAu9xV1i29tW/R1IH0i/cxlp6m rDMnz4/BaGtBHUikYiBHYCgbTMKB2WDE9Fz4VBjdpkIl74qjLGlIFfSQ= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zpZnCjw0DX7F for ; Fri, 11 Jan 2019 18:37:30 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id EA232419C0 for ; Fri, 11 Jan 2019 18:37:24 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:24 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 17/21] PCI: pciehp: Add support for the movable BARs feature Date: Fri, 11 Jan 2019 18:37:03 +0300 Message-ID: <20190111153707.10140-18-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP With movable BARs, adding a hotplugged device may affect all the PCIe domain starting from the root, so use a pci_rescan_bus() function which handles the rearrangement of existing BARs and bridge windows. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/hotplug/pciehp_pci.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/drivers/pci/hotplug/pciehp_pci.c b/drivers/pci/hotplug/pciehp_pci.c index b9c1396db6fe..7c0871db5bae 100644 --- a/drivers/pci/hotplug/pciehp_pci.c +++ b/drivers/pci/hotplug/pciehp_pci.c @@ -56,12 +56,16 @@ int pciehp_configure_device(struct controller *ctrl) goto out; } - for_each_pci_bridge(dev, parent) - pci_hp_add_bridge(dev); + if (pci_movable_bars_enabled()) { + pci_rescan_bus(parent); + } else { + for_each_pci_bridge(dev, parent) + pci_hp_add_bridge(dev); - pci_assign_unassigned_bridge_resources(bridge); - pcie_bus_configure_settings(parent); - pci_bus_add_devices(parent); + pci_assign_unassigned_bridge_resources(bridge); + pcie_bus_configure_settings(parent); + pci_bus_add_devices(parent); + } out: pci_unlock_rescan_remove(); From patchwork Fri Jan 11 15:37:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758289 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AEA711869 for ; Fri, 11 Jan 2019 15:43:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9DFB12848B for ; Fri, 11 Jan 2019 15:43:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 92581284E4; Fri, 11 Jan 2019 15:43:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3752F284C0 for ; Fri, 11 Jan 2019 15:43:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731897AbfAKPn4 (ORCPT ); Fri, 11 Jan 2019 10:43:56 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40068 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731848AbfAKPnz (ORCPT ); Fri, 11 Jan 2019 10:43:55 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 3CB3B419CD for ; Fri, 11 Jan 2019 15:37:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221050; x=1549035451; bh=eYWE39w4hGy8un3UayYgMccd5gwYJbSuq1b mpqjMrAc=; b=YQXI5HeFdsEECo/c7v5arHS0VJSXBZHQNkO58t0rjvacOsnYq56 KgZsWu+VcGReezwbyS30tcckIViBRj97EBvJTRuDXrYQrhdNKMRgnDqNrxIe63lr zlf7NvNw3TeURilu1XMqnbxAkYra7to0UoO/fRaT3w3auWwRQEjIzwr4= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4t0uZFZChqAM for ; Fri, 11 Jan 2019 18:37:30 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 24C13419C1 for ; Fri, 11 Jan 2019 18:37:25 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:24 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 18/21] nvme-pci: Handle movable BARs Date: Fri, 11 Jan 2019 18:37:04 +0300 Message-ID: <20190111153707.10140-19-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hotplugged devices can affect the existing ones by moving their BARs. PCI subsystem will inform the NVME driver about this by invoking reset_prepare()+reset_done(), then iounmap()+ioremap() must be called. Signed-off-by: Sergey Miroshnichenko --- drivers/nvme/host/pci.c | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 5a0bf6a24d50..bf72f934f20c 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -113,6 +113,7 @@ struct nvme_dev { unsigned int num_vecs; int q_depth; u32 db_stride; + resource_size_t current_phys_bar; void __iomem *bar; unsigned long bar_mapped_size; struct work_struct remove_work; @@ -1667,13 +1668,16 @@ static int nvme_remap_bar(struct nvme_dev *dev, unsigned long size) { struct pci_dev *pdev = to_pci_dev(dev->dev); - if (size <= dev->bar_mapped_size) + if (dev->bar && + dev->current_phys_bar == pci_resource_start(pdev, 0) && + size <= dev->bar_mapped_size) return 0; if (size > pci_resource_len(pdev, 0)) return -ENOMEM; if (dev->bar) iounmap(dev->bar); - dev->bar = ioremap(pci_resource_start(pdev, 0), size); + dev->current_phys_bar = pci_resource_start(pdev, 0); + dev->bar = ioremap(dev->current_phys_bar, size); if (!dev->bar) { dev->bar_mapped_size = 0; return -ENOMEM; @@ -2520,6 +2524,8 @@ static void nvme_reset_work(struct work_struct *work) if (WARN_ON(dev->ctrl.state != NVME_CTRL_RESETTING)) goto out; + nvme_remap_bar(dev, db_bar_size(dev, 0)); + /* * If we're called to reset a live controller first shut it down before * moving on. @@ -2922,6 +2928,23 @@ static void nvme_error_resume(struct pci_dev *pdev) flush_work(&dev->ctrl.reset_work); } +void nvme_rescan_prepare(struct pci_dev *pdev) +{ + struct nvme_dev *dev = pci_get_drvdata(pdev); + + nvme_dev_disable(dev, false); + nvme_dev_unmap(dev); + dev->bar = NULL; +} + +void nvme_rescan_done(struct pci_dev *pdev) +{ + struct nvme_dev *dev = pci_get_drvdata(pdev); + + nvme_dev_map(dev); + nvme_reset_ctrl_sync(&dev->ctrl); +} + static const struct pci_error_handlers nvme_err_handler = { .error_detected = nvme_error_detected, .slot_reset = nvme_slot_reset, @@ -2984,6 +3007,8 @@ static struct pci_driver nvme_driver = { }, .sriov_configure = pci_sriov_configure_simple, .err_handler = &nvme_err_handler, + .rescan_prepare = nvme_rescan_prepare, + .rescan_done = nvme_rescan_done, }; static int __init nvme_init(void) From patchwork Fri Jan 11 15:37:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758307 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EB775188E for ; Fri, 11 Jan 2019 15:44:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DB2852848B for ; Fri, 11 Jan 2019 15:44:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D9A19285A9; Fri, 11 Jan 2019 15:44:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 91A22285C3 for ; Fri, 11 Jan 2019 15:44:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731948AbfAKPn7 (ORCPT ); Fri, 11 Jan 2019 10:43:59 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40076 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731975AbfAKPn4 (ORCPT ); Fri, 11 Jan 2019 10:43:56 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 104FE419C0 for ; Fri, 11 Jan 2019 15:37:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221051; x=1549035452; bh=yWOP/t3o7W/g1Yw9tCQ66cBeDXlxqnPxkpk ug1M0fOw=; b=mRSZIornQhiaww3uZg4mUgrHznq7CQ4mWU23iNSvCuQv1IDC7Ng SRKcI8q1iTJWJtauqGt/IMDIzdXm0PUp220tmrPkJ3wI7m9+4KT75RaS2FEGTM6g KsvtkF/2sZMjTvXad3ebWxWYr3Vt1Td+rnk10V9nCV45cdgpFncsDAtg= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JDZpZODN1qMv for ; Fri, 11 Jan 2019 18:37:31 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 52E04419C2 for ; Fri, 11 Jan 2019 18:37:25 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:24 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 19/21] powerpc/pci: Fix crash with enabled movable BARs Date: Fri, 11 Jan 2019 18:37:05 +0300 Message-ID: <20190111153707.10140-20-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Check a resource for the UNSET flags. Signed-off-by: Sergey Miroshnichenko --- arch/powerpc/platforms/powernv/pci-ioda.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c index 1d6406a051f1..6706dac69ee6 100644 --- a/arch/powerpc/platforms/powernv/pci-ioda.c +++ b/arch/powerpc/platforms/powernv/pci-ioda.c @@ -3037,7 +3037,8 @@ static void pnv_ioda_setup_pe_res(struct pnv_ioda_pe *pe, int index; int64_t rc; - if (!res || !res->flags || res->start > res->end) + if (!res || !res->flags || res->start > res->end || + (res->flags & IORESOURCE_UNSET)) return; if (res->flags & IORESOURCE_IO) { From patchwork Fri Jan 11 15:37:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758285 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1F6C7746 for ; Fri, 11 Jan 2019 15:43:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 104FD284C0 for ; Fri, 11 Jan 2019 15:43:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0500E284E4; Fri, 11 Jan 2019 15:43:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B21CD28574 for ; Fri, 11 Jan 2019 15:43:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732012AbfAKPnz (ORCPT ); Fri, 11 Jan 2019 10:43:55 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40078 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731904AbfAKPny (ORCPT ); Fri, 11 Jan 2019 10:43:54 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id ACEB0419CE for ; Fri, 11 Jan 2019 15:37:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221051; x=1549035452; bh=9Mgh16usuR991UNp6SjMLom44ODQbZVtZFO vTV5kUF0=; b=nqvTq8bCImar+0CUbQ3/VfMxk42VsQJj473qffeF52DhRX1iWpn N+jIS7I8XsZaMV/5wsgjOVJ1l1HP6HBpd6u7iYIAliOUYr6s62E4cpFl+melNuUb C+NU4un3PTRkYLikSKLVo+WHw9WkQfRMaX1xzShemLVERaRdYhsjFDDs= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id W8jilzyt-GXb for ; Fri, 11 Jan 2019 18:37:31 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 8A5E9419C4 for ; Fri, 11 Jan 2019 18:37:25 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:25 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 20/21] PCI: Wake up bridges during rescan when movable BARs enabled Date: Fri, 11 Jan 2019 18:37:06 +0300 Message-ID: <20190111153707.10140-21-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Use the PM runtime methods to wake up the bridges before accessing their config space. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/probe.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 4726049a19ad..a80978906ac3 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -3199,6 +3199,8 @@ static void pci_bus_rescan_prepare(struct pci_bus *bus) { struct pci_dev *dev; + pm_runtime_get_sync(&bus->dev); + list_for_each_entry(dev, &bus->devices, bus_list) { struct pci_bus *child = dev->subordinate; @@ -3242,6 +3244,8 @@ static void pci_bus_rescan_done(struct pci_bus *bus) dev->driver->rescan_done(dev); } } + + pm_runtime_put(&bus->dev); } static void pci_setup_bridges(struct pci_bus *bus) From patchwork Fri Jan 11 15:37:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 10758309 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6F2F81399 for ; Fri, 11 Jan 2019 15:44:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5FE0F2848B for ; Fri, 11 Jan 2019 15:44:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 544D728574; Fri, 11 Jan 2019 15:44:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 12A2C2848B for ; Fri, 11 Jan 2019 15:44:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731975AbfAKPn7 (ORCPT ); Fri, 11 Jan 2019 10:43:59 -0500 Received: from mta-01.yadro.com ([89.207.88.251]:40074 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731859AbfAKPnz (ORCPT ); Fri, 11 Jan 2019 10:43:55 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id D3B50419CF for ; Fri, 11 Jan 2019 15:37:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1547221052; x=1549035453; bh=oW/E446PIoYsz78RDzI3+79je16l19k7b6k lzyaq4lA=; b=kWgJYtQY+wzG2jTwB4BjDuvqxqivoQlSCpHB7Gj+jyV/go/EhoM UjBzb8NJglHAfLegLCuhabOQYvf8tHdXm88DUPHrar+L8Ld1cZnyrpZVy4TdBKg9 TXtjiCEkGJ15t3+ASn4Z2433dkR2N45r6/JJSuwebHFd2EnmppA1O3bE= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 66Im7H6_TJsn for ; Fri, 11 Jan 2019 18:37:32 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id C4694419C6 for ; Fri, 11 Jan 2019 18:37:25 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.60) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Fri, 11 Jan 2019 18:37:25 +0300 From: Sergey Miroshnichenko To: CC: , Sergey Miroshnichenko Subject: [PATCH RFC v2 21/21] x86: enable movable BARs Date: Fri, 11 Jan 2019 18:37:07 +0300 Message-ID: <20190111153707.10140-22-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190111153707.10140-1-s.miroshnichenko@yadro.com> References: <20190111153707.10140-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.60] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Enable the feature by default. Can be overridden by the command line argument "pcie_movable_bars=off". Signed-off-by: Sergey Miroshnichenko --- arch/x86/pci/init.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/x86/pci/init.c b/arch/x86/pci/init.c index 5fc617edf108..f93843f702d1 100644 --- a/arch/x86/pci/init.c +++ b/arch/x86/pci/init.c @@ -40,6 +40,8 @@ static __init int pci_arch_init(void) dmi_check_skip_isa_align(); + pci_add_flags(PCI_MOVABLE_BARS); + return 0; } arch_initcall(pci_arch_init);