From patchwork Thu Oct 24 17:11:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210415 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 09B5A139A for ; Thu, 24 Oct 2019 17:12:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DBA7721D7E for ; Thu, 24 Oct 2019 17:12:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="lcHr7bQ7" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2408006AbfJXRMn (ORCPT ); Thu, 24 Oct 2019 13:12:43 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48748 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2405917AbfJXRMm (ORCPT ); Thu, 24 Oct 2019 13:12:42 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 3FA2443A25; Thu, 24 Oct 2019 17:12:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937159; x=1573751560; bh=vnznP9M/yXFK8xAfMbW3HwbTFc8jiZDqHgo k8mI6Vk0=; b=lcHr7bQ7em0nBm6AkKQFc1v6rXOyiQS3wXP+2aMZrIH9m6NhLXA E8OUtS9XcCBv5KqtBpfZr3aeRPY9axoBt+DyURPP7iu7oO/QNzQN2RN4rGXVk7C+ vcBLbQh20S1JUNvsJYOdJFuTe+PoIVv46/Sty4NyzR+IMRlbdrDBpQgk= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tj-3jK_noj57; Thu, 24 Oct 2019 20:12:39 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 3C40042F15; Thu, 24 Oct 2019 20:12:39 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:38 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko , Srinath Mannam , Marta Rybczynska Subject: [PATCH v6 01/30] PCI: Fix race condition in pci_enable/disable_device() Date: Thu, 24 Oct 2019 20:11:59 +0300 Message-ID: <20191024171228.877974-2-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org This is a yet another approach to fix an old [1-2] concurrency issue, when: - two or more devices are being hot-added into a bridge which was initially empty; - a bridge with two or more devices is being hot-added; - during boot, if BIOS/bootloader/firmware doesn't pre-enable bridges. The problem is that a bridge is reported as enabled before the MEM/IO bits are actually written to the PCI_COMMAND register, so another driver thread starts memory requests through the not-yet-enabled bridge: CPU0 CPU1 pci_enable_device_mem() pci_enable_device_mem() pci_enable_bridge() pci_enable_bridge() pci_is_enabled() return false; atomic_inc_return(enable_cnt) Start actual enabling the bridge ... pci_is_enabled() ... return true; ... Start memory requests <-- FAIL ... Set the PCI_COMMAND_MEMORY bit <-- Must wait for this Protect the pci_enable/disable_device() and pci_enable_bridge(), which is similar to the previous solution from commit 40f11adc7cd9 ("PCI: Avoid race while enabling upstream bridges"), but adding a per-device mutexes and preventing the dev->enable_cnt from from incrementing early. CC: Srinath Mannam CC: Marta Rybczynska Signed-off-by: Sergey Miroshnichenko [1] https://lore.kernel.org/linux-pci/1501858648-22228-1-git-send-email-srinath.mannam@broadcom.com/T/#u [RFC PATCH v3] pci: Concurrency issue during pci enable bridge [2] https://lore.kernel.org/linux-pci/744877924.5841545.1521630049567.JavaMail.zimbra@kalray.eu/T/#u [RFC PATCH] nvme: avoid race-conditions when enabling devices --- drivers/pci/pci.c | 26 ++++++++++++++++++++++---- drivers/pci/probe.c | 1 + include/linux/pci.h | 1 + 3 files changed, 24 insertions(+), 4 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index a97e2571a527..44d0d12c80cf 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1643,6 +1643,8 @@ static void pci_enable_bridge(struct pci_dev *dev) struct pci_dev *bridge; int retval; + mutex_lock(&dev->enable_mutex); + bridge = pci_upstream_bridge(dev); if (bridge) pci_enable_bridge(bridge); @@ -1650,6 +1652,7 @@ static void pci_enable_bridge(struct pci_dev *dev) if (pci_is_enabled(dev)) { if (!dev->is_busmaster) pci_set_master(dev); + mutex_unlock(&dev->enable_mutex); return; } @@ -1658,11 +1661,14 @@ static void pci_enable_bridge(struct pci_dev *dev) pci_err(dev, "Error enabling bridge (%d), continuing\n", retval); pci_set_master(dev); + mutex_unlock(&dev->enable_mutex); } static int pci_enable_device_flags(struct pci_dev *dev, unsigned long flags) { struct pci_dev *bridge; + /* Enable-locking of bridges is performed within the pci_enable_bridge() */ + bool need_lock = !dev->subordinate; int err; int i, bars = 0; @@ -1678,8 +1684,13 @@ static int pci_enable_device_flags(struct pci_dev *dev, unsigned long flags) dev->current_state = (pmcsr & PCI_PM_CTRL_STATE_MASK); } - if (atomic_inc_return(&dev->enable_cnt) > 1) + if (need_lock) + mutex_lock(&dev->enable_mutex); + if (pci_is_enabled(dev)) { + if (need_lock) + mutex_unlock(&dev->enable_mutex); return 0; /* already enabled */ + } bridge = pci_upstream_bridge(dev); if (bridge) @@ -1694,8 +1705,10 @@ static int pci_enable_device_flags(struct pci_dev *dev, unsigned long flags) bars |= (1 << i); err = do_pci_enable_device(dev, bars); - if (err < 0) - atomic_dec(&dev->enable_cnt); + if (err >= 0) + atomic_inc(&dev->enable_cnt); + if (need_lock) + mutex_unlock(&dev->enable_mutex); return err; } @@ -1939,15 +1952,20 @@ void pci_disable_device(struct pci_dev *dev) if (dr) dr->enabled = 0; + mutex_lock(&dev->enable_mutex); dev_WARN_ONCE(&dev->dev, atomic_read(&dev->enable_cnt) <= 0, "disabling already-disabled device"); - if (atomic_dec_return(&dev->enable_cnt) != 0) + if (atomic_dec_return(&dev->enable_cnt) != 0) { + mutex_unlock(&dev->enable_mutex); return; + } do_pci_disable_device(dev); dev->is_busmaster = 0; + + mutex_unlock(&dev->enable_mutex); } EXPORT_SYMBOL(pci_disable_device); diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 3d5271a7a849..d4f21e413638 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -2158,6 +2158,7 @@ struct pci_dev *pci_alloc_dev(struct pci_bus *bus) INIT_LIST_HEAD(&dev->bus_list); dev->dev.type = &pci_dev_type; dev->bus = pci_bus_get(bus); + mutex_init(&dev->enable_mutex); return dev; } diff --git a/include/linux/pci.h b/include/linux/pci.h index f9088c89a534..525140e3a460 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -425,6 +425,7 @@ struct pci_dev { unsigned int no_vf_scan:1; /* Don't scan for VFs after IOV enablement */ pci_dev_flags_t dev_flags; atomic_t enable_cnt; /* pci_enable_device has been called */ + struct mutex enable_mutex; u32 saved_config_space[16]; /* Config space saved at suspend time */ struct hlist_head saved_cap_space; From patchwork Thu Oct 24 17:12:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210413 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D2054112C for ; Thu, 24 Oct 2019 17:12:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B052820650 for ; Thu, 24 Oct 2019 17:12:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="gtqnnE4R" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2405899AbfJXRMn (ORCPT ); Thu, 24 Oct 2019 13:12:43 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48758 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2405931AbfJXRMm (ORCPT ); Thu, 24 Oct 2019 13:12:42 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 9526343B0D; Thu, 24 Oct 2019 17:12:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937160; x=1573751561; bh=7LK5j1jdEBLIX2jiqRm58ICC7aVyOZzOenw 2Gw9gTSA=; b=gtqnnE4Ry6yu+htTvQwLZWM8c+2+3hPyIIy7LlPS7TK/t2ozV62 y5Ts5++ZhXMeiar6dq19hc7mvdMU3sFoODdMKuNQ85+OsYxdDupKNXfkGr2Agwpv uzAim+PBfw9VVQwKIwRqdyhLKvx5IDabVzergkOvVQPxllGa09HClX2s= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rwaLFxmMzZHi; Thu, 24 Oct 2019 20:12:40 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 8DA9843130; Thu, 24 Oct 2019 20:12:39 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:39 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 02/30] PCI: Enable bridge's I/O and MEM access for hotplugged devices Date: Thu, 24 Oct 2019 20:12:00 +0300 Message-ID: <20191024171228.877974-3-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org The PCI_COMMAND_IO and PCI_COMMAND_MEMORY bits of the bridge must be updated not only when enabling the bridge for the first time, but also if a hotplugged device requests these types of resources. Originally these bits were set by the pci_enable_device_flags() only, which exits early if the bridge is already pci_is_enabled(). So if the bridge was empty initially (an edge case), then hotplugged devices fail to IO/MEM. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/pci.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 44d0d12c80cf..e85dc63c73fd 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1650,6 +1650,14 @@ static void pci_enable_bridge(struct pci_dev *dev) pci_enable_bridge(bridge); if (pci_is_enabled(dev)) { + int i, bars = 0; + + for (i = PCI_BRIDGE_RESOURCES; i < DEVICE_COUNT_RESOURCE; i++) { + if (dev->resource[i].flags & (IORESOURCE_MEM | IORESOURCE_IO)) + bars |= (1 << i); + } + do_pci_enable_device(dev, bars); + if (!dev->is_busmaster) pci_set_master(dev); mutex_unlock(&dev->enable_mutex); From patchwork Thu Oct 24 17:12:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210427 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AB8A22D2C for ; Thu, 24 Oct 2019 17:12:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 801D721655 for ; Thu, 24 Oct 2019 17:12:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="W3BtRmtK" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2410032AbfJXRMq (ORCPT ); Thu, 24 Oct 2019 13:12:46 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48782 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2405901AbfJXRMq (ORCPT ); Thu, 24 Oct 2019 13:12:46 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 19F4A42F14; Thu, 24 Oct 2019 17:12:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937161; x=1573751562; bh=gAAEqhNNDTOwrXRTh4j6hjh9zAAjUohixCZ 5LlrKZl8=; b=W3BtRmtKxnxEi0WXFcFESDDeWDg/jjnjIlJAxNdyF74bGiYxEbM fvrs4FJLTJ8Ilr4hAfzTLz4vzVID7/b2Iiv0Irwu/N3JuI3hGYcE/T8kbiETcQ4b RylIZAySZeiapCiSJsi0OeHEfNjs6Ivb7hFI8fiBgC5MjdIl9x5HW7CE= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fpyDAiP5VViU; Thu, 24 Oct 2019 20:12:41 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 92B1E43597; Thu, 24 Oct 2019 20:12:39 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:39 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko , Sam Bobroff , Rajat Jain , Lukas Wunner , Oliver O'Halloran , David Laight Subject: [PATCH v6 03/30] PCI: hotplug: Add a flag for the movable BARs feature Date: Thu, 24 Oct 2019 20:12:01 +0300 Message-ID: <20191024171228.877974-4-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When hot-adding a device, the bridge may have windows not big enough (or fragmented too much) for newly requested BARs to fit in. And expanding these bridge windows may be impossible because blocked by "neighboring" BARs and bridge windows. Still, it may be possible to allocate a memory region for new BARs with the following procedure: 1) notify all the drivers which support movable BARs to pause and release the BARs; the rest of the drivers are guaranteed that their devices will not get BARs moved; 2) release all the bridge windows and movable BARs; 3) try to recalculate new bridge windows that will fit all the BAR types: - fixed; - immovable; - movable; - newly requested by hot-added devices; 4) if the previous step fails, disable BARs for one of the hot-added devices and retry from step 3; 5) notify the drivers, so they remap BARs and resume. If bridge calculation and BAR assignment fails with a hot-added devices, BARs of these devices will be disabled, falling back to the same amount and size of BARs as they were before the hotplug event. The kernel succeeded in assigning then, so the same algorithm will provide the same results again. This makes the prior reservation of memory by BIOS/bootloader/firmware not required anymore for the PCI hotplug. Drivers indicate their support of movable BARs by implementing the new .rescan_prepare() and .rescan_done() hooks in the struct pci_driver. All device's activity must be paused during a rescan, and iounmap()+ioremap() must be applied to every used BAR. If a device is not bound to a driver, its BARs are considered movable. For a higher probability of the successful BAR reassignment, all the BARs and bridge windows should be released before the rescan, not only those with higher addresses. One example when it is needed, BAR(I) is moved to free a gap for the new BAR(II): Before: ==================== parent bridge window =============== ---- hotplug bridge window ---- | BAR(I) | fixed BAR | fixed BAR | fixed BAR | ^ | new BAR(II) After: ==================== parent bridge window ========================= ----------- hotplug bridge window ----------- | new BAR(II) | fixed BAR | fixed BAR | fixed BAR | BAR(I) | Another example is a fragmented bridge window jammed between fixed BARs: Before: ===================== parent bridge window ======================== ---------- hotplug bridge window ---------- | fixed BAR | | BAR(I) | | BAR(II) | | BAR(III) | fixed BAR | ^ | new BAR(IV) After: ==================== parent bridge window ========================= ---------- hotplug bridge window ---------- | fixed BAR | BAR(I) | BAR(II) | BAR(III) | new BAR(IV) | fixed BAR | This patch is a preparation for future patches with actual implementation, and for now it just does the following: - declares the feature; - defines the bool pci_can_move_bars and bool pci_dev_bar_movable(dev); - invokes the .rescan_prepare() and .rescan_done() driver notifiers; - disables the feature for the powerpc/pseries. The feature is disabled by default until the final patch of the series. It can be overridden per-arch using the pci_can_move_bars=false flag or by the following command line option: pci=no_movable_bars CC: Sam Bobroff CC: Rajat Jain CC: Lukas Wunner CC: Oliver O'Halloran CC: David Laight Signed-off-by: Sergey Miroshnichenko --- .../admin-guide/kernel-parameters.txt | 1 + arch/powerpc/platforms/pseries/setup.c | 2 + drivers/pci/pci.c | 4 + drivers/pci/pci.h | 2 + drivers/pci/probe.c | 85 ++++++++++++++++++- include/linux/pci.h | 4 + 6 files changed, 96 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index a84a83f8881e..c6243aaed0c9 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3528,6 +3528,7 @@ may put more devices in an IOMMU group. force_floating [S390] Force usage of floating interrupts. nomio [S390] Do not use MIO instructions. + no_movable_bars Don't allow BARs to be moved during hotplug pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power Management. diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c index 0a40201f315f..7cd12c5a2deb 100644 --- a/arch/powerpc/platforms/pseries/setup.c +++ b/arch/powerpc/platforms/pseries/setup.c @@ -920,6 +920,8 @@ static void __init pseries_init(void) { pr_debug(" -> pseries_init()\n"); + pci_can_move_bars = false; + #ifdef CONFIG_HVC_CONSOLE if (firmware_has_feature(FW_FEATURE_LPAR)) hvc_vio_init_early(); diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index e85dc63c73fd..85014c6b2817 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -78,6 +78,8 @@ static void pci_dev_d3_sleep(struct pci_dev *dev) int pci_domains_supported = 1; #endif +bool pci_can_move_bars; + #define DEFAULT_CARDBUS_IO_SIZE (256) #define DEFAULT_CARDBUS_MEM_SIZE (64*1024*1024) /* pci=cbmemsize=nnM,cbiosize=nn can override this */ @@ -6331,6 +6333,8 @@ static int __init pci_setup(char *str) pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS); } else if (!strncmp(str, "disable_acs_redir=", 18)) { disable_acs_redir_param = str + 18; + } else if (!strncmp(str, "no_movable_bars", 15)) { + pci_can_move_bars = false; } else { pr_err("PCI: Unknown option `%s'\n", str); } diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 3f6947ee3324..19bc50597d12 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -286,6 +286,8 @@ void pci_disable_bridge_window(struct pci_dev *dev); struct pci_bus *pci_bus_get(struct pci_bus *bus); void pci_bus_put(struct pci_bus *bus); +bool pci_dev_bar_movable(struct pci_dev *dev, struct resource *res); + /* PCIe link information */ #define PCIE_SPEED2STR(speed) \ ((speed) == PCIE_SPEED_16_0GT ? "16 GT/s" : \ diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index d4f21e413638..3d8c0f653378 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -3133,6 +3133,73 @@ unsigned int pci_rescan_bus_bridge_resize(struct pci_dev *bridge) return max; } +static bool pci_dev_movable(struct pci_dev *dev, bool res_has_children) +{ + if (!pci_can_move_bars) + return false; + + if (dev->driver && dev->driver->rescan_prepare) + return true; + + if (!dev->driver && !res_has_children) + return true; + + return false; +} + +bool pci_dev_bar_movable(struct pci_dev *dev, struct resource *res) +{ + struct pci_bus_region region; + + if (res->flags & IORESOURCE_PCI_FIXED) + return false; + + /* Workaround for the legacy VGA memory 0xa0000-0xbffff */ + pcibios_resource_to_bus(dev->bus, ®ion, res); + if (region.start == 0xa0000) + return false; + + return pci_dev_movable(dev, res->child); +} + +static void pci_bus_rescan_prepare(struct pci_bus *bus) +{ + struct pci_dev *dev; + + if (bus->self) + pci_config_pm_runtime_get(bus->self); + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child = dev->subordinate; + + if (child) + pci_bus_rescan_prepare(child); + + if (dev->driver && + dev->driver->rescan_prepare) + dev->driver->rescan_prepare(dev); + } +} + +static void pci_bus_rescan_done(struct pci_bus *bus) +{ + struct pci_dev *dev; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child = dev->subordinate; + + if (dev->driver && + dev->driver->rescan_done) + dev->driver->rescan_done(dev); + + if (child) + pci_bus_rescan_done(child); + } + + if (bus->self) + pci_config_pm_runtime_put(bus->self); +} + /** * pci_rescan_bus - Scan a PCI bus for devices * @bus: PCI bus to scan @@ -3145,9 +3212,23 @@ unsigned int pci_rescan_bus_bridge_resize(struct pci_dev *bridge) unsigned int pci_rescan_bus(struct pci_bus *bus) { unsigned int max; + struct pci_bus *root = bus; + + while (!pci_is_root_bus(root)) + root = root->parent; + + if (pci_can_move_bars) { + pci_bus_rescan_prepare(root); + + max = pci_scan_child_bus(root); + pci_assign_unassigned_root_bus_resources(root); + + pci_bus_rescan_done(root); + } else { + max = pci_scan_child_bus(bus); + pci_assign_unassigned_bus_resources(bus); + } - max = pci_scan_child_bus(bus); - pci_assign_unassigned_bus_resources(bus); pci_bus_add_devices(bus); return max; diff --git a/include/linux/pci.h b/include/linux/pci.h index 525140e3a460..b981e67c8a13 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -835,6 +835,8 @@ struct pci_driver { int (*resume)(struct pci_dev *dev); /* Device woken up */ void (*shutdown)(struct pci_dev *dev); int (*sriov_configure)(struct pci_dev *dev, int num_vfs); /* On PF */ + void (*rescan_prepare)(struct pci_dev *dev); + void (*rescan_done)(struct pci_dev *dev); const struct pci_error_handlers *err_handler; const struct attribute_group **groups; struct device_driver driver; @@ -1395,6 +1397,8 @@ void pci_setup_bridge(struct pci_bus *bus); resource_size_t pcibios_window_alignment(struct pci_bus *bus, unsigned long type); +extern bool pci_can_move_bars; + #define PCI_VGA_STATE_CHANGE_BRIDGE (1 << 0) #define PCI_VGA_STATE_CHANGE_DECODES (1 << 1) From patchwork Thu Oct 24 17:12:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210419 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7EDB8139A for ; Thu, 24 Oct 2019 17:12:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5DC402166E for ; Thu, 24 Oct 2019 17:12:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="FNEv5+gi" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2409988AbfJXRMo (ORCPT ); Thu, 24 Oct 2019 13:12:44 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48770 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2405917AbfJXRMo (ORCPT ); Thu, 24 Oct 2019 13:12:44 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id C472343D79; Thu, 24 Oct 2019 17:12:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937161; x=1573751562; bh=j5t3xGbzow+1zdXT65WFVb8pwBW7/xPMS3e sfMQkuWM=; b=FNEv5+gin/VbyQV+aEv6abCeERWAz+P3v4l0p1wHCLcsAms1OSf USrkyH7WctBiKsjOFw97cBboizLOfQiKIjpS3QzMUGSIl4pTWi1GYPVGJsxomq0n iJYgH96qStYlTAu/hMv5RpVHyiQ8UgCheVQbUMtkyqHk9qSBwSWofiAY= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yVCoI2Lpkmve; Thu, 24 Oct 2019 20:12:41 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 17D4B42F14; Thu, 24 Oct 2019 20:12:41 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:39 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 04/30] PCI: Define PCI-specific version of the release_child_resources() Date: Thu, 24 Oct 2019 20:12:02 +0300 Message-ID: <20191024171228.877974-5-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org If release the bridge resources with standard release_child_resources(), it drops the .start field of children's BARs to zero, but with the STARTALIGN flag remaining set, which makes the resource invalid for reassignment. Some resources must preserve their offset and size: those marked with the PCI_FIXED and the immovable ones - which are bound by drivers without support of the movable BARs feature. Add the pci_release_child_resources() to replace release_child_resources() in handling the described PCI-specific cases. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/setup-bus.c | 54 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 53 insertions(+), 1 deletion(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index e7dbe21705ba..2c02eb1acf5d 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1482,6 +1482,54 @@ static void __pci_bridge_assign_resources(const struct pci_dev *bridge, (IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\ IORESOURCE_MEM_64) +/* + * Similar to generic release_child_resources(), but aware of immovable BARs and + * PCI_FIXED and STARTALIGN flags + */ +static void pci_release_child_resources(struct pci_bus *bus, struct resource *r) +{ + struct pci_dev *dev; + + if (!bus || !r) + return; + + if (r->flags & IORESOURCE_PCI_FIXED) + return; + + r->child = NULL; + + list_for_each_entry(dev, &bus->devices, bus_list) { + int i; + + for (i = 0; i < PCI_NUM_RESOURCES; i++) { + struct resource *tmp = &dev->resource[i]; + resource_size_t size = resource_size(tmp); + + if (!tmp->flags || tmp->parent != r) + continue; + + tmp->parent = NULL; + tmp->sibling = NULL; + + pci_release_child_resources(dev->subordinate, tmp); + + tmp->flags &= ~IORESOURCE_STARTALIGN; + tmp->flags |= IORESOURCE_SIZEALIGN; + + if (!pci_dev_bar_movable(dev, tmp)) { + pci_dbg(dev, "release immovable %pR (%s), keep its flags, base and size\n", + tmp, tmp->name); + continue; + } + + pci_dbg(dev, "release %pR (%s)\n", tmp, tmp->name); + + tmp->start = 0; + tmp->end = size - 1; + } + } +} + static void pci_bridge_release_resources(struct pci_bus *bus, unsigned long type) { @@ -1522,7 +1570,11 @@ static void pci_bridge_release_resources(struct pci_bus *bus, return; /* If there are children, release them all */ - release_child_resources(r); + if (pci_can_move_bars) + pci_release_child_resources(bus, r); + else + release_child_resources(r); + if (!release_resource(r)) { type = old_flags = r->flags & PCI_RES_TYPE_MASK; pci_info(dev, "resource %d %pR released\n", From patchwork Thu Oct 24 17:12:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210421 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 03E72112C for ; Thu, 24 Oct 2019 17:12:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D6FCA21D7E for ; Thu, 24 Oct 2019 17:12:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="HhZBdrmD" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2405917AbfJXRMp (ORCPT ); Thu, 24 Oct 2019 13:12:45 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48798 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2408021AbfJXRMp (ORCPT ); Thu, 24 Oct 2019 13:12:45 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id E266F43E07; Thu, 24 Oct 2019 17:12:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937162; x=1573751563; bh=03Gd2zXei/2YVqAgwHE1WQSBVHfa/dXqAQ0 t5M0opgg=; b=HhZBdrmDOr2TM39pzhh+ZTPVAtEDS1Dk78BQOXXT1xTBf8+6ONH ap3ZfaMg1kObCG//5WNo3RkffKpsjkeCjJgLV642ovh6OYK1Cv6zZyTonkePNJgF nIkiS/j/AxmBXx2unVtHDpdWpaL27iyspEGwIT38uTsriqqwPGdw9Xrs= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6JcBIMhHLZ5w; Thu, 24 Oct 2019 20:12:42 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 1BD46437F3; Thu, 24 Oct 2019 20:12:41 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:39 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 05/30] PCI: hotplug: movable BARs: Fix reassigning the released bridge windows Date: Thu, 24 Oct 2019 20:12:03 +0300 Message-ID: <20191024171228.877974-6-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When a bridge window is temporarily released during the rescan, its old size is not relevant anymore - it will be recreated from pbus_size_*(), so it's start value should be zero. If such window can't be reassigned, don't apply reset_resource(), so the next retry may succeed. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/setup-bus.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 2c02eb1acf5d..f2f02e6c9000 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -295,7 +295,8 @@ static void assign_requested_resources_sorted(struct list_head *head, 0 /* don't care */, 0 /* don't care */); } - reset_resource(res); + if (!pci_can_move_bars) + reset_resource(res); } } } @@ -1579,8 +1580,8 @@ static void pci_bridge_release_resources(struct pci_bus *bus, type = old_flags = r->flags & PCI_RES_TYPE_MASK; pci_info(dev, "resource %d %pR released\n", PCI_BRIDGE_RESOURCES + idx, r); - /* Keep the old size */ - r->end = resource_size(r) - 1; + /* Don't keep the old size if the bridge will be recalculated */ + r->end = pci_can_move_bars ? 0 : (resource_size(r) - 1); r->start = 0; r->flags = 0; From patchwork Thu Oct 24 17:12:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210425 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 34A871575 for ; Thu, 24 Oct 2019 17:12:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 135AC21925 for ; Thu, 24 Oct 2019 17:12:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="JufeTIOh" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2408021AbfJXRMq (ORCPT ); Thu, 24 Oct 2019 13:12:46 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48802 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2409968AbfJXRMq (ORCPT ); Thu, 24 Oct 2019 13:12:46 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 47FE543E0C; Thu, 24 Oct 2019 17:12:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937163; x=1573751564; bh=HZgKRN6tOARhBhsn62fHxtGaNd6lFwfoMrp mmOp4nHk=; b=JufeTIOh4c1paPuUJzwoxcjqL4K+SaQHIDLSG6cdHDq2FJOPBw/ PU04gXr+syBke6E5pte7aHrew35gmJ7954CCFu2CriOXcT6hNCIYvZL/T03NibvM 9t3H/1tUeszIZKjl3YjkEz/y1xeiWJ1N/HAr5MXKQCrpgZQfldMFcWPc= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ocSxCEQygCjp; Thu, 24 Oct 2019 20:12:43 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 1F8DB437F6; Thu, 24 Oct 2019 20:12:41 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:39 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 06/30] PCI: hotplug: movable BARs: Recalculate all bridge windows during rescan Date: Thu, 24 Oct 2019 20:12:04 +0300 Message-ID: <20191024171228.877974-7-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When the movable BARs feature is enabled and a rescan has been requested, release all the bridge windows and recalculate them from scratch, taking into account all kinds for BARs: fixed, immovable, movable, new. This increases the chances to find a memory space to fit BARs for newly hotplugged devices, especially if no/not enough gaps were reserved by the BIOS/bootloader/firmware. The last step of writing the recalculated windows to the bridges is done by the new pci_setup_bridges() function. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/pci.h | 1 + drivers/pci/probe.c | 22 ++++++++++++++++++++++ drivers/pci/setup-bus.c | 16 ++++++++++++++++ 3 files changed, 39 insertions(+) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 19bc50597d12..4a3f2b69285b 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -280,6 +280,7 @@ void __pci_bus_assign_resources(const struct pci_bus *bus, struct list_head *realloc_head, struct list_head *fail_head); bool pci_bus_clip_resource(struct pci_dev *dev, int idx); +void pci_bus_release_root_bridge_resources(struct pci_bus *bus); void pci_reassigndev_resource_alignment(struct pci_dev *dev); void pci_disable_bridge_window(struct pci_dev *dev); diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 3d8c0f653378..d2dbec51c4df 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -3200,6 +3200,25 @@ static void pci_bus_rescan_done(struct pci_bus *bus) pci_config_pm_runtime_put(bus->self); } +static void pci_setup_bridges(struct pci_bus *bus) +{ + struct pci_dev *dev; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child; + + if (!pci_dev_is_added(dev)) + continue; + + child = dev->subordinate; + if (child) + pci_setup_bridges(child); + } + + if (bus->self) + pci_setup_bridge(bus); +} + /** * pci_rescan_bus - Scan a PCI bus for devices * @bus: PCI bus to scan @@ -3221,8 +3240,11 @@ unsigned int pci_rescan_bus(struct pci_bus *bus) pci_bus_rescan_prepare(root); max = pci_scan_child_bus(root); + + pci_bus_release_root_bridge_resources(root); pci_assign_unassigned_root_bus_resources(root); + pci_setup_bridges(root); pci_bus_rescan_done(root); } else { max = pci_scan_child_bus(bus); diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index f2f02e6c9000..075e8185b936 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1635,6 +1635,22 @@ static void pci_bus_release_bridge_resources(struct pci_bus *bus, pci_bridge_release_resources(bus, type); } +void pci_bus_release_root_bridge_resources(struct pci_bus *root_bus) +{ + int i; + struct resource *r; + + pci_bus_release_bridge_resources(root_bus, IORESOURCE_IO, whole_subtree); + pci_bus_release_bridge_resources(root_bus, IORESOURCE_MEM, whole_subtree); + pci_bus_release_bridge_resources(root_bus, + IORESOURCE_MEM_64 | IORESOURCE_PREFETCH, + whole_subtree); + + pci_bus_for_each_resource(root_bus, r, i) { + pci_release_child_resources(root_bus, r); + } +} + static void pci_bus_dump_res(struct pci_bus *bus) { struct resource *res; From patchwork Thu Oct 24 17:12:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210423 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 09BAA139A for ; Thu, 24 Oct 2019 17:12:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DA45621925 for ; Thu, 24 Oct 2019 17:12:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="dLZ8gyDA" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2410019AbfJXRMq (ORCPT ); Thu, 24 Oct 2019 13:12:46 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48808 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2408021AbfJXRMq (ORCPT ); Thu, 24 Oct 2019 13:12:46 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 0DEE943E13; Thu, 24 Oct 2019 17:12:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937164; x=1573751565; bh=LpiNdNTxKihswl145tIG+dMOoQ28OjVfwW9 SBN7HOII=; b=dLZ8gyDAxMpxJfT7Rw5wvwqevcTB998nvQPXdWqhv2y3gIvm5fD 2Qzfhkq1AHS97bJRyiC0shAMIHLbvvBNdp7w7karyB+N/zRmVFlQ7WZI6j2uEuFb nGRnHaoXi0w2QcrSOb6FDqlCACgqjh0U+rPH6nY9xvRDK0IAeXVYLebA= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id O4Uc7COed1Sb; Thu, 24 Oct 2019 20:12:44 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 22BF9437F8; Thu, 24 Oct 2019 20:12:41 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:40 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 07/30] PCI: hotplug: movable BARs: Don't disable the released bridge windows Date: Thu, 24 Oct 2019 20:12:05 +0300 Message-ID: <20191024171228.877974-8-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org On a hotplug event with enabled BAR movement, calculating the new bridge windows takes some time. During this procedure, the structures that represent these windows are released - marked for recalculation. When new bridge windows are ready, they are written to the registers of every bridge via pci_setup_bridges(). Currently, bridge's registers are updated immediately after releasing a window to disable it. But if a driver doesn't yet support movable BARs, it doesn't stop MEM transactions during the hotplug, so disabled bridge windows will break them. Let the bridge windows remain operating after releasing, as they will be updated to the new values in the end of a hotplug event. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/setup-bus.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 075e8185b936..381ce964cb20 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1588,7 +1588,8 @@ static void pci_bridge_release_resources(struct pci_bus *bus, /* Avoiding touch the one without PREF */ if (type & IORESOURCE_PREFETCH) type = IORESOURCE_PREFETCH; - __pci_setup_bridge(bus, type); + if (!pci_can_move_bars) + __pci_setup_bridge(bus, type); /* For next child res under same bridge */ r->flags = old_flags; } From patchwork Thu Oct 24 17:12:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210435 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CE15B1575 for ; Thu, 24 Oct 2019 17:12:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id ACE1421925 for ; Thu, 24 Oct 2019 17:12:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="KrrA7SRa" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2436808AbfJXRMt (ORCPT ); Thu, 24 Oct 2019 13:12:49 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48798 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2410000AbfJXRMs (ORCPT ); Thu, 24 Oct 2019 13:12:48 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id BDC2543E1B; Thu, 24 Oct 2019 17:12:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937164; x=1573751565; bh=NvQMzjUVCOWGtfggDXlkUQRLM30JH6SR9Yn lAOvNzFQ=; b=KrrA7SRahrEMHVQTuIAXiGBPWmJ3cFBjLBkoPKW7+yuQHcsRTs0 raeH2e2wpaSWyhvTLkvlGkU2U+Wijp4Szn7fhCUzwuUBH/8a2teY3REDITdPaazD 46jLilZZ07FTRi6+0ww8BGI+GrEOMYkV/7TsMNOcLsNDLzOohai5730w= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vQVudJlUeLKR; Thu, 24 Oct 2019 20:12:44 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 273C4437FA; Thu, 24 Oct 2019 20:12:41 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:40 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 08/30] PCI: hotplug: movable BARs: Don't allow added devices to steal resources Date: Thu, 24 Oct 2019 20:12:06 +0300 Message-ID: <20191024171228.877974-9-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When movable BARs are enabled, the PCI subsystem at first releases all the bridge windows and then attempts to assign resources both to previously working devices and to the newly hotplugged ones, with the same priority. If a hotplugged device gets its BARs first, this may lead to lack of space for already working devices, which is unacceptable. If that happens, mark one of the new devices with the newly introduced flag PCI_DEV_DISABLED_BARS (if it is not yet marked) and retry the BAR recalculation. The worst case would be no BARs for hotplugged devices, while all the rest just continue working. The algorithm is simple and it doesn't retry different subsets of hot-added devices in case of a failure, e.g. if there are no space to allocate BARs for both hotplugged devices A and B, but is enough for just A, the A will be marked with PCI_DEV_DISABLED_BARS first, then (after the next failure) - B. As a result, A will not get BARs while it could. This issue is only relevant when hotplugging two and more devices simultaneously. Add a new res_mask bitmask to the struct pci_dev for storing the indices of assigned BARs. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/pci.h | 11 +++++ drivers/pci/probe.c | 102 ++++++++++++++++++++++++++++++++++++++-- drivers/pci/setup-bus.c | 15 ++++++ include/linux/pci.h | 1 + 4 files changed, 126 insertions(+), 3 deletions(-) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 4a3f2b69285b..55344f2c55bf 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -403,6 +403,7 @@ static inline bool pci_dev_is_disconnected(const struct pci_dev *dev) /* pci_dev priv_flags */ #define PCI_DEV_ADDED 0 +#define PCI_DEV_DISABLED_BARS 1 static inline void pci_dev_assign_added(struct pci_dev *dev, bool added) { @@ -414,6 +415,16 @@ static inline bool pci_dev_is_added(const struct pci_dev *dev) return test_bit(PCI_DEV_ADDED, &dev->priv_flags); } +static inline void pci_dev_disable_bars(struct pci_dev *dev) +{ + assign_bit(PCI_DEV_DISABLED_BARS, &dev->priv_flags, true); +} + +static inline bool pci_dev_bars_enabled(const struct pci_dev *dev) +{ + return !test_bit(PCI_DEV_DISABLED_BARS, &dev->priv_flags); +} + #ifdef CONFIG_PCIEAER #include diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index d2dbec51c4df..2d1157493e6a 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -3162,6 +3162,23 @@ bool pci_dev_bar_movable(struct pci_dev *dev, struct resource *res) return pci_dev_movable(dev, res->child); } +static unsigned int pci_dev_count_res_mask(struct pci_dev *dev) +{ + unsigned int res_mask = 0; + int i; + + for (i = 0; i < PCI_BRIDGE_RESOURCES; i++) { + struct resource *r = &dev->resource[i]; + + if (!r->flags || (r->flags & IORESOURCE_UNSET) || !r->parent) + continue; + + res_mask |= (1 << i); + } + + return res_mask; +} + static void pci_bus_rescan_prepare(struct pci_bus *bus) { struct pci_dev *dev; @@ -3172,6 +3189,8 @@ static void pci_bus_rescan_prepare(struct pci_bus *bus) list_for_each_entry(dev, &bus->devices, bus_list) { struct pci_bus *child = dev->subordinate; + dev->res_mask = pci_dev_count_res_mask(dev); + if (child) pci_bus_rescan_prepare(child); @@ -3207,7 +3226,7 @@ static void pci_setup_bridges(struct pci_bus *bus) list_for_each_entry(dev, &bus->devices, bus_list) { struct pci_bus *child; - if (!pci_dev_is_added(dev)) + if (!pci_dev_is_added(dev) || !pci_dev_bars_enabled(dev)) continue; child = dev->subordinate; @@ -3219,6 +3238,83 @@ static void pci_setup_bridges(struct pci_bus *bus) pci_setup_bridge(bus); } +static struct pci_dev *pci_find_next_new_device(struct pci_bus *bus) +{ + struct pci_dev *dev; + + if (!bus) + return NULL; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child_bus = dev->subordinate; + + if (!pci_dev_is_added(dev) && pci_dev_bars_enabled(dev)) + return dev; + + if (child_bus) { + struct pci_dev *next_new_dev; + + next_new_dev = pci_find_next_new_device(child_bus); + if (next_new_dev) + return next_new_dev; + } + } + + return NULL; +} + +static bool pci_bus_check_all_bars_reassigned(struct pci_bus *bus) +{ + struct pci_dev *dev; + bool ret = true; + + if (!bus) + return false; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child = dev->subordinate; + unsigned int res_mask = pci_dev_count_res_mask(dev); + + if (!pci_dev_bars_enabled(dev)) + continue; + + if (dev->res_mask & ~res_mask) { + pci_err(dev, "Non-re-enabled resources found: 0x%x -> 0x%x\n", + dev->res_mask, res_mask); + ret = false; + } + + if (child && !pci_bus_check_all_bars_reassigned(child)) + ret = false; + } + + return ret; +} + +static void pci_reassign_root_bus_resources(struct pci_bus *root) +{ + do { + struct pci_dev *next_new_dev; + + pci_assign_unassigned_root_bus_resources(root); + + if (pci_bus_check_all_bars_reassigned(root)) + break; + + next_new_dev = pci_find_next_new_device(root); + if (!next_new_dev) { + dev_err(&root->dev, "failed to re-assign resources even after ignoring all the hotplugged devices\n"); + break; + } + + dev_warn(&root->dev, "failed to re-assign resources, disable the next hotplugged device %s and retry\n", + dev_name(&next_new_dev->dev)); + + pci_dev_disable_bars(next_new_dev); + pci_bus_release_root_bridge_resources(root); + } while (true); +} + /** * pci_rescan_bus - Scan a PCI bus for devices * @bus: PCI bus to scan @@ -3238,11 +3334,11 @@ unsigned int pci_rescan_bus(struct pci_bus *bus) if (pci_can_move_bars) { pci_bus_rescan_prepare(root); + pci_bus_release_root_bridge_resources(root); max = pci_scan_child_bus(root); - pci_bus_release_root_bridge_resources(root); - pci_assign_unassigned_root_bus_resources(root); + pci_reassign_root_bus_resources(root); pci_setup_bridges(root); pci_bus_rescan_done(root); diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 381ce964cb20..4b538d132958 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -128,6 +128,9 @@ static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head) { int i; + if (!pci_dev_bars_enabled(dev)) + return; + for (i = 0; i < PCI_NUM_RESOURCES; i++) { struct resource *r; struct pci_dev_resource *dev_res, *tmp; @@ -177,6 +180,9 @@ static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head) { u16 class = dev->class >> 8; + if (!pci_dev_bars_enabled(dev)) + return; + /* Don't touch classless devices or host bridges or IOAPICs */ if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST) return; @@ -278,6 +284,9 @@ static void assign_requested_resources_sorted(struct list_head *head, int idx; list_for_each_entry(dev_res, head, list) { + if (!pci_dev_bars_enabled(dev_res->dev)) + continue; + res = dev_res->res; idx = res - &dev_res->dev->resource[0]; if (resource_size(res) && @@ -995,6 +1004,9 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, list_for_each_entry(dev, &bus->devices, bus_list) { int i; + if (!pci_dev_bars_enabled(dev)) + continue; + for (i = 0; i < PCI_NUM_RESOURCES; i++) { struct resource *r = &dev->resource[i]; resource_size_t r_size; @@ -1349,6 +1361,9 @@ void __pci_bus_assign_resources(const struct pci_bus *bus, pbus_assign_resources_sorted(bus, realloc_head, fail_head); list_for_each_entry(dev, &bus->devices, bus_list) { + if (!pci_dev_bars_enabled(dev)) + continue; + pdev_assign_fixed_resources(dev); b = dev->subordinate; diff --git a/include/linux/pci.h b/include/linux/pci.h index b981e67c8a13..7f7f704df672 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -376,6 +376,7 @@ struct pci_dev { */ unsigned int irq; struct resource resource[DEVICE_COUNT_RESOURCE]; /* I/O and memory regions + expansion ROMs */ + unsigned int res_mask; /* Bitmask of assigned resources */ bool match_driver; /* Skip attaching driver */ From patchwork Thu Oct 24 17:12:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210429 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 57C34112C for ; Thu, 24 Oct 2019 17:12:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3533C2166E for ; Thu, 24 Oct 2019 17:12:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="oxA9MLy6" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2410046AbfJXRMr (ORCPT ); Thu, 24 Oct 2019 13:12:47 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48816 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2409968AbfJXRMr (ORCPT ); Thu, 24 Oct 2019 13:12:47 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 1AF3A437FA; Thu, 24 Oct 2019 17:12:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937165; x=1573751566; bh=urtJdFbNTBYXQ8HBtx644qczAPU//SKLRkO DXd6fGkw=; b=oxA9MLy6/Z3u6KfAohuQCQVJBwhND1MhWTMPnLHibsy0gEEyUp0 ZZ3Mjaoo2O7V3pKXh6YArRhzJapbsqkSzvv5nF/gIGPZ8hc510xbIN1WNF/9lTAL yX/h8GqBEWb6dlUAI44ppcsfdzfja5tafXOhDelXq0SDI6w0bUoOfxzM= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NS9LPbWAOIig; Thu, 24 Oct 2019 20:12:45 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 2DF8F438CE; Thu, 24 Oct 2019 20:12:41 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:40 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 09/30] PCI: Include fixed and immovable BARs into the bus size calculating Date: Thu, 24 Oct 2019 20:12:07 +0300 Message-ID: <20191024171228.877974-10-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org The only difference between the fixed/immovable and movable BARs is a size and offset preservation after they are released (the corresponding struct resource* detached from a bridge window for a while during a bus rescan). Include fixed/immovable BARs into result of pbus_size_mem() and prohibit assigning them to non-direct parents. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/setup-bus.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 4b538d132958..ff33b47b1bb7 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1011,12 +1011,20 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, struct resource *r = &dev->resource[i]; resource_size_t r_size; - if (r->parent || (r->flags & IORESOURCE_PCI_FIXED) || + if (r->parent || ((r->flags & mask) != type && (r->flags & mask) != type2 && (r->flags & mask) != type3)) continue; r_size = resource_size(r); + + if (!pci_dev_bar_movable(dev, r)) { + if (pci_can_move_bars) + size += r_size; + + continue; + } + #ifdef CONFIG_PCI_IOV /* Put SRIOV requested res to the optional list */ if (realloc_head && i >= PCI_IOV_RESOURCES && From patchwork Thu Oct 24 17:12:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210431 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9E8CC1575 for ; Thu, 24 Oct 2019 17:12:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7394521D7E for ; Thu, 24 Oct 2019 17:12:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="GKuAxZVH" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2436682AbfJXRMs (ORCPT ); Thu, 24 Oct 2019 13:12:48 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48808 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2405901AbfJXRMr (ORCPT ); Thu, 24 Oct 2019 13:12:47 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id D651A438CE; Thu, 24 Oct 2019 17:12:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937165; x=1573751566; bh=Ay54PSF1F0jBkianHUfRLKAtYZp036FG1+X ImCOLkNA=; b=GKuAxZVH5Hu//GymgpYzy1uhYmT57w0I6z5koQ1Y0+KT8Q7aykk 7n1FPSPoxVD9iAMFC8LuxjBIzJcDvaFNq9zaL5jgsjU8uc/3AdAx0d+5l6GOtA/z L8fPRJQGUkCVHs5Ei16M8qg5JrWZCfLhvHyg4XzC2M+to34J0bas0hzo= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id w401pJkLQ2_F; Thu, 24 Oct 2019 20:12:45 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 31EFC438D1; Thu, 24 Oct 2019 20:12:41 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:40 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 10/30] PCI: Prohibit assigning BARs and bridge windows to non-direct parents Date: Thu, 24 Oct 2019 20:12:08 +0300 Message-ID: <20191024171228.877974-11-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When movable BARs are enabled, the feature of resource relocating from commit 2bbc6942273b5 ("PCI : ability to relocate assigned pci-resources") is not used. Instead, inability to assign a resource is used as a signal to retry BAR assignment with other configuration of bridge windows. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/setup-bus.c | 2 ++ drivers/pci/setup-res.c | 12 ++++++++++++ 2 files changed, 14 insertions(+) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index ff33b47b1bb7..cf325daae1b1 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1355,6 +1355,8 @@ static void pdev_assign_fixed_resources(struct pci_dev *dev) while (b && !r->parent) { assign_fixed_resource_on_bus(b, r); b = b->parent; + if (!r->parent && pci_can_move_bars) + break; } } } diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c index d8ca40a97693..a1657a8bf93d 100644 --- a/drivers/pci/setup-res.c +++ b/drivers/pci/setup-res.c @@ -298,6 +298,18 @@ static int _pci_assign_resource(struct pci_dev *dev, int resno, bus = dev->bus; while ((ret = __pci_assign_resource(bus, dev, resno, size, min_align))) { + if (pci_can_move_bars) { + if (resno >= PCI_BRIDGE_RESOURCES && + resno <= PCI_BRIDGE_RESOURCE_END) { + struct resource *res = dev->resource + resno; + + res->start = 0; + res->end = 0; + res->flags = 0; + } + break; + } + if (!bus->parent || !bus->self->transparent) break; bus = bus->parent; From patchwork Thu Oct 24 17:12:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210433 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A191A112C for ; Thu, 24 Oct 2019 17:12:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 808A221925 for ; Thu, 24 Oct 2019 17:12:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="sO4L9j9E" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2436807AbfJXRMt (ORCPT ); Thu, 24 Oct 2019 13:12:49 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48802 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2410042AbfJXRMs (ORCPT ); Thu, 24 Oct 2019 13:12:48 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 2C296438D1; Thu, 24 Oct 2019 17:12:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937166; x=1573751567; bh=A4K/ih2Q0DIWGW/e6i8C//zd89CA+2ig2e3 2P47HEwA=; b=sO4L9j9EgvHCcoJUaKYcmwl3BWWQAqmZ8R4eigQi0CvMnC+ST1g F1FdW5SSjx7H/2zV0uBcB75mpmgFlqFqA8c4fnKKQjPKu4vrfXaLioc+3VxMzwXV hkkb3naIubtyb49gPJ9T80auakBIQ35vffYnAC3FUEPC6hvcJmWMxkaE= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nE71faLzyPT8; Thu, 24 Oct 2019 20:12:46 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 352FE439EC; Thu, 24 Oct 2019 20:12:41 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:40 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 11/30] PCI: hotplug: movable BARs: Try to assign unassigned resources only once Date: Thu, 24 Oct 2019 20:12:09 +0300 Message-ID: <20191024171228.877974-12-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org With enabled BAR movement, BARs and bridge windows can only be assigned to their direct parents, so there can be only one variant of resource tree, thus every retry within the pci_assign_unassigned_root_bus_resources() will result in the same tree, and it is enough to try just once. In case of failures the pci_reassign_root_bus_resources() disables BARs for one of the hotplugged devices and tries the assignment again. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/setup-bus.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index cf325daae1b1..3deb1c343e89 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1819,6 +1819,13 @@ void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus) int pci_try_num = 1; enum enable_type enable_local; + if (pci_can_move_bars) { + __pci_bus_size_bridges(bus, NULL); + __pci_bus_assign_resources(bus, NULL, NULL); + + goto dump; + } + /* Don't realloc if asked to do so */ enable_local = pci_realloc_detect(bus, pci_realloc_enable); if (pci_realloc_enabled(enable_local)) { From patchwork Thu Oct 24 17:12:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210441 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9032E1920 for ; Thu, 24 Oct 2019 17:12:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 663F12166E for ; Thu, 24 Oct 2019 17:12:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="JTiHryXg" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2439814AbfJXRMu (ORCPT ); Thu, 24 Oct 2019 13:12:50 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48816 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2436722AbfJXRMu (ORCPT ); Thu, 24 Oct 2019 13:12:50 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 5C236439EC; Thu, 24 Oct 2019 17:12:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937166; x=1573751567; bh=iPggJSK3fO6BKJRkB06lv5avP4UKPr6Cm/2 rqrVmjsE=; b=JTiHryXg1fcGUiUgCc/dP5JZ6O2IWx/IbHER7i1FEnU16I30Lpq apRvl+Hopv5Ca0QbfESn6IZnckFjRplvI74SWsN67Xl4+EtLVaXIQLw8xpPuTclL 63dEyeK0FGHLI8GLzoSssR+7ilKOQmi2RA0lppFN9fQpwgZR/1ci5MmE= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NB-00Su7adSx; Thu, 24 Oct 2019 20:12:46 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 72B8642F15; Thu, 24 Oct 2019 20:12:41 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:41 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 12/30] PCI: hotplug: movable BARs: Calculate immovable parts of bridge windows Date: Thu, 24 Oct 2019 20:12:10 +0300 Message-ID: <20191024171228.877974-13-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When movable BARs are enabled, and if a bridge contains a device with fixed (IORESOURCE_PCI_FIXED) or immovable BARs, the corresponing windows can't be moved too far away from their original positions - they must still contain all the fixed/immovable BARs, like that: 1) Window position before a bus rescan: | <-- root bridge window --> | | | | | <-- bridge window --> | | | | movable BARs | **fixed BAR** | | 2) Possible valid outcome after rescan and move: | <-- root bridge window --> | | | | | <-- bridge window --> | | | | **fixed BAR** | Movable BARs | | An immovable area of a bridge (separare for IO, MEM and MEM64 window types) is a range that covers all the fixed and immovable BARs of direct children, and all the fixed area of children bridges: | <-- root bridge window --> | | | | | <-- bridge window level 1 --> | | | | ******** immovable area of this bridge window ******** | | | | | | | | **fixed BAR** | <-- bridge window level 2 --> | BARs | | | | | ***** fixed area of this bridge ***** | | | | | | | | | | | | ***fixed BAR*** | | ***fixed BAR*** | | | To store these areas, the .immovable_range field has been added to struct pci_bus. It is filled recursively from leaves to the root before a rescan. Also make pbus_size_io() and pbus_size_mem() return their usual result OR the size of an immovable range of according type, depending on which one is larger. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/pci.h | 14 +++++++ drivers/pci/probe.c | 88 +++++++++++++++++++++++++++++++++++++++++ drivers/pci/setup-bus.c | 17 ++++++++ include/linux/pci.h | 6 +++ 4 files changed, 125 insertions(+) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 55344f2c55bf..7cd108885598 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -401,6 +401,20 @@ static inline bool pci_dev_is_disconnected(const struct pci_dev *dev) return dev->error_state == pci_channel_io_perm_failure; } +static inline int pci_get_bridge_resource_idx(struct resource *r) +{ + int idx = 1; + + if (r->flags & IORESOURCE_IO) + idx = 0; + else if (!(r->flags & IORESOURCE_PREFETCH)) + idx = 1; + else if (r->flags & IORESOURCE_MEM_64) + idx = 2; + + return idx; +} + /* pci_dev priv_flags */ #define PCI_DEV_ADDED 0 #define PCI_DEV_DISABLED_BARS 1 diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 2d1157493e6a..d0d00cb3e965 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -545,6 +545,7 @@ void pci_read_bridge_bases(struct pci_bus *child) static struct pci_bus *pci_alloc_bus(struct pci_bus *parent) { struct pci_bus *b; + int idx; b = kzalloc(sizeof(*b), GFP_KERNEL); if (!b) @@ -561,6 +562,11 @@ static struct pci_bus *pci_alloc_bus(struct pci_bus *parent) if (parent) b->domain_nr = parent->domain_nr; #endif + for (idx = 0; idx < PCI_BRIDGE_RESOURCE_NUM; ++idx) { + b->immovable_range[idx].start = 0; + b->immovable_range[idx].end = 0; + } + return b; } @@ -3238,6 +3244,87 @@ static void pci_setup_bridges(struct pci_bus *bus) pci_setup_bridge(bus); } +static void pci_bus_update_immovable_range(struct pci_bus *bus) +{ + struct pci_dev *dev; + int idx; + resource_size_t start, end; + + for (idx = 0; idx < PCI_BRIDGE_RESOURCE_NUM; ++idx) { + bus->immovable_range[idx].start = 0; + bus->immovable_range[idx].end = 0; + } + + list_for_each_entry(dev, &bus->devices, bus_list) + if (dev->subordinate) + pci_bus_update_immovable_range(dev->subordinate); + + list_for_each_entry(dev, &bus->devices, bus_list) { + int i; + struct pci_bus *child = dev->subordinate; + + for (i = 0; i < PCI_BRIDGE_RESOURCES; ++i) { + struct resource *r = &dev->resource[i]; + + if (!r->flags || (r->flags & IORESOURCE_UNSET) || !r->parent) + continue; + + if (!pci_dev_bar_movable(dev, r)) { + idx = pci_get_bridge_resource_idx(r); + start = bus->immovable_range[idx].start; + end = bus->immovable_range[idx].end; + + if (!start || start > r->start) + start = r->start; + if (end < r->end) + end = r->end; + + if (bus->immovable_range[idx].start != start || + bus->immovable_range[idx].end != end) { + dev_dbg(&bus->dev, "Found fixed BAR%d 0x%llx-0x%llx in %s, expand the fixed bridge window %d to 0x%llx-0x%llx\n", + i, + (unsigned long long)r->start, + (unsigned long long)r->end, + dev_name(&dev->dev), idx, + (unsigned long long)start, + (unsigned long long)end); + bus->immovable_range[idx].start = start; + bus->immovable_range[idx].end = end; + } + } + } + + if (child) { + for (idx = 0; idx < PCI_BRIDGE_RESOURCE_NUM; ++idx) { + struct resource *child_immovable_range = + &child->immovable_range[idx]; + + if (child_immovable_range->start >= + child_immovable_range->end) + continue; + + start = bus->immovable_range[idx].start; + end = bus->immovable_range[idx].end; + + if (!start || start > child_immovable_range->start) + start = child_immovable_range->start; + if (end < child_immovable_range->end) + end = child_immovable_range->end; + + if (start < bus->immovable_range[idx].start || + end > bus->immovable_range[idx].end) { + dev_dbg(&bus->dev, "Expand the fixed bridge window %d from %s to 0x%llx-0x%llx\n", + idx, dev_name(&child->dev), + (unsigned long long)start, + (unsigned long long)end); + bus->immovable_range[idx].start = start; + bus->immovable_range[idx].end = end; + } + } + } + } +} + static struct pci_dev *pci_find_next_new_device(struct pci_bus *bus) { struct pci_dev *dev; @@ -3334,6 +3421,7 @@ unsigned int pci_rescan_bus(struct pci_bus *bus) if (pci_can_move_bars) { pci_bus_rescan_prepare(root); + pci_bus_update_immovable_range(root); pci_bus_release_root_bridge_resources(root); max = pci_scan_child_bus(root); diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 3deb1c343e89..a7546e02ea7c 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -882,9 +882,17 @@ static void pbus_size_io(struct pci_bus *bus, resource_size_t min_size, resource_size_t children_add_size = 0; resource_size_t min_align, align; + resource_size_t fixed_start = bus->immovable_range[0].start; + resource_size_t fixed_end = bus->immovable_range[0].end; + resource_size_t fixed_size = (fixed_start < fixed_end) ? + (fixed_end - fixed_start + 1) : 0; + if (!b_res) return; + if (min_size < fixed_size) + min_size = fixed_size; + min_align = window_alignment(bus, IORESOURCE_IO); list_for_each_entry(dev, &bus->devices, bus_list) { int i; @@ -993,6 +1001,15 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, resource_size_t children_add_size = 0; resource_size_t children_add_align = 0; resource_size_t add_align = 0; + bool is_mem64 = (mask & IORESOURCE_MEM_64); + + resource_size_t fixed_start = bus->immovable_range[is_mem64 ? 2 : 1].start; + resource_size_t fixed_end = bus->immovable_range[is_mem64 ? 2 : 1].end; + resource_size_t fixed_size = (fixed_start < fixed_end) ? + (fixed_end - fixed_start + 1) : 0; + + if (min_size < fixed_size) + min_size = fixed_size; if (!b_res) return -ENOSPC; diff --git a/include/linux/pci.h b/include/linux/pci.h index 7f7f704df672..ef41be0ce082 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -581,6 +581,12 @@ struct pci_bus { struct list_head resources; /* Address space routed to this bus */ struct resource busn_res; /* Bus numbers routed to this bus */ + /* + * If there are fixed or immovable resources in the bridge window, this range + * contains the lowest start address and highest end address of them. + */ + struct resource immovable_range[PCI_BRIDGE_RESOURCE_NUM]; + struct pci_ops *ops; /* Configuration access functions */ struct msi_controller *msi; /* MSI controller */ void *sysdata; /* Hook for sys-specific extension */ From patchwork Thu Oct 24 17:12:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210437 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 08422112C for ; Thu, 24 Oct 2019 17:12:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DB4A221929 for ; Thu, 24 Oct 2019 17:12:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="GNC1o+1P" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2436863AbfJXRMu (ORCPT ); Thu, 24 Oct 2019 13:12:50 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48808 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2436801AbfJXRMu (ORCPT ); Thu, 24 Oct 2019 13:12:50 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 76E9A42F15; Thu, 24 Oct 2019 17:12:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937167; x=1573751568; bh=dEtonO9WGVanX91lx4CEaG6vqwfmZCpSHvc P/eimQVM=; b=GNC1o+1PVFrShivoThk3yWRC/OSQCwMVc7QEIRUYlbyTP66iOh8 cc+mRyvYyMGPFrT2kBOz8f4PZSj25H549+b22cmVBCP5NMOtEEAQz+tMMFkGvLmo by3QJ1DOee63CqgOo/OUTzm1nc6SwQF2BMBRQXmRmdUYm3CO5aCiHvIs= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LVn_oPmCl1r7; Thu, 24 Oct 2019 20:12:47 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id A0E9E43130; Thu, 24 Oct 2019 20:12:41 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:41 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 13/30] PCI: hotplug: movable BARs: Compute limits for relocated bridge windows Date: Thu, 24 Oct 2019 20:12:11 +0300 Message-ID: <20191024171228.877974-14-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org With enabled movable BARs, bridge windows are recalculated during each pci rescan. Some of the BARs below the bridge may be fixed/immovable: these areas are represented by the .immovable_range field in struct pci_bus. If a bridge window size is equal to its immovable range, it can only be assigned to the start of this range. But if a bridge window size is larger, and this difference in size is denoted as "delta", the window can start from (immovable_range.start - delta) to (immovable_range.start), and it can end from (immovable_range.end) to (immovable_range.end + delta). This range (the new .realloc_range field in struct pci_bus) must then be compared with immovable ranges of neighbouring bridges to guarantee no intersections. This patch only calculates valid ranges for reallocated bridges during pci rescan, and the next one will make use of these values during allocation. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/setup-bus.c | 67 +++++++++++++++++++++++++++++++++++++++++ include/linux/pci.h | 6 ++++ 2 files changed, 73 insertions(+) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index a7546e02ea7c..653ba4d5f191 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1819,6 +1819,72 @@ static enum enable_type pci_realloc_detect(struct pci_bus *bus, } #endif +/* + * Calculate the address margins where the bridge windows may be allocated to fit all + * the fixed and immovable BARs beneath. + */ +static void pci_bus_update_realloc_range(struct pci_bus *bus) +{ + struct pci_dev *dev; + struct pci_bus *parent = bus->parent; + int idx; + + list_for_each_entry(dev, &bus->devices, bus_list) + if (dev->subordinate) + pci_bus_update_realloc_range(dev->subordinate); + + if (!parent || !bus->self) + return; + + for (idx = 0; idx < PCI_BRIDGE_RESOURCE_NUM; ++idx) { + struct resource *immovable_range = &bus->immovable_range[idx]; + resource_size_t window_size = resource_size(bus->resource[idx]); + resource_size_t realloc_start, realloc_end; + + bus->realloc_range[idx].start = 0; + bus->realloc_range[idx].end = 0; + + /* Check if there any immovable BARs under the bridge */ + if (immovable_range->start >= immovable_range->end) + continue; + + /* The lowest possible address where the bridge window can start */ + realloc_start = immovable_range->end - window_size + 1; + /* The highest possible address where the bridge window can end */ + realloc_end = immovable_range->start + window_size - 1; + + if (realloc_start > immovable_range->start) + realloc_start = immovable_range->start; + + if (realloc_end < immovable_range->end) + realloc_end = immovable_range->end; + + /* + * Check that realloc range doesn't intersect with hard fixed ranges + * of neighboring bridges + */ + list_for_each_entry(dev, &parent->devices, bus_list) { + struct pci_bus *neighbor = dev->subordinate; + struct resource *n_imm_range; + + if (!neighbor || neighbor == bus) + continue; + + n_imm_range = &neighbor->immovable_range[idx]; + + if (n_imm_range->start >= n_imm_range->end) + continue; + + if (n_imm_range->end < immovable_range->start && + n_imm_range->end > realloc_start) + realloc_start = n_imm_range->end; + } + + bus->realloc_range[idx].start = realloc_start; + bus->realloc_range[idx].end = realloc_end; + } +} + /* * First try will not touch PCI bridge res. * Second and later try will clear small leaf bridge res. @@ -1838,6 +1904,7 @@ void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus) if (pci_can_move_bars) { __pci_bus_size_bridges(bus, NULL); + pci_bus_update_realloc_range(bus); __pci_bus_assign_resources(bus, NULL, NULL); goto dump; diff --git a/include/linux/pci.h b/include/linux/pci.h index ef41be0ce082..e1edcb3fad31 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -587,6 +587,12 @@ struct pci_bus { */ struct resource immovable_range[PCI_BRIDGE_RESOURCE_NUM]; + /* + * Acceptable address range, where the bridge window may reside, considering its + * size, so it will cover all the fixed and immovable BARs below. + */ + struct resource realloc_range[PCI_BRIDGE_RESOURCE_NUM]; + struct pci_ops *ops; /* Configuration access functions */ struct msi_controller *msi; /* MSI controller */ void *sysdata; /* Hook for sys-specific extension */ From patchwork Thu Oct 24 17:12:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210443 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 73A29139A for ; Thu, 24 Oct 2019 17:12:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 51EEC20650 for ; Thu, 24 Oct 2019 17:12:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="anXICluz" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2439855AbfJXRMv (ORCPT ); Thu, 24 Oct 2019 13:12:51 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48802 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2436815AbfJXRMv (ORCPT ); Thu, 24 Oct 2019 13:12:51 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 8E6A343130; Thu, 24 Oct 2019 17:12:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937168; x=1573751569; bh=rr5LWboqmrfFjD3Z1bCM5jQX+PcvKf7YXEC 2Wij1RtY=; b=anXICluzLh5hXkWu3ZQ0KGWpvN72H1Pyc0+EucCRENi+MRpjEJc 25CBu0WL86RY5uyvYpV7anaye6arT6EJfPO2i1NdtKajh+AttvDsT6WPQCFNOv9m j0KD/uD9NKJYuTvqfazWULIgq/Mk4VElKI5wKlazkSGRQMHoZRMEAUtE= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 82XDHFVx3aSW; Thu, 24 Oct 2019 20:12:48 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id E8E2743B45; Thu, 24 Oct 2019 20:12:41 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:41 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 14/30] PCI: Make sure bridge windows include their fixed BARs Date: Thu, 24 Oct 2019 20:12:12 +0300 Message-ID: <20191024171228.877974-15-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When the time comes to select a start address for the bridge window during the root bus rescan, it should be not just a lowest possible address: this window must cover all the underlying fixed and immovable BARs. The lowest address that satisfies this requirement is the .realloc_range field of struct pci_bus, which is calculated during the preparation to the rescan. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/bus.c | 2 +- drivers/pci/setup-res.c | 31 +++++++++++++++++++++++++++++-- 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c index 8e40b3e6da77..a1efa87e31b9 100644 --- a/drivers/pci/bus.c +++ b/drivers/pci/bus.c @@ -192,7 +192,7 @@ static int pci_bus_alloc_from_region(struct pci_bus *bus, struct resource *res, * this is an already-configured bridge window, its start * overrides "min". */ - if (avail.start) + if (min_used < avail.start) min_used = avail.start; max = avail.end; diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c index a1657a8bf93d..1570bbd620cd 100644 --- a/drivers/pci/setup-res.c +++ b/drivers/pci/setup-res.c @@ -248,9 +248,23 @@ static int __pci_assign_resource(struct pci_bus *bus, struct pci_dev *dev, struct resource *res = dev->resource + resno; resource_size_t min; int ret; + resource_size_t start = (resource_size_t)-1; + resource_size_t end = 0; min = (res->flags & IORESOURCE_IO) ? PCIBIOS_MIN_IO : PCIBIOS_MIN_MEM; + if (dev->subordinate && resno >= PCI_BRIDGE_RESOURCES) { + struct pci_bus *child_bus = dev->subordinate; + int b_resno = resno - PCI_BRIDGE_RESOURCES; + struct resource *immovable_range = &child_bus->immovable_range[b_resno]; + + if (immovable_range->start < immovable_range->end) { + start = immovable_range->start; + end = immovable_range->end; + min = child_bus->realloc_range[b_resno].start; + } + } + /* * First, try exact prefetching match. Even if a 64-bit * prefetchable bridge window is below 4GB, we can't put a 32-bit @@ -262,7 +276,7 @@ static int __pci_assign_resource(struct pci_bus *bus, struct pci_dev *dev, IORESOURCE_PREFETCH | IORESOURCE_MEM_64, pcibios_align_resource, dev); if (ret == 0) - return 0; + goto check_fixed; /* * If the prefetchable window is only 32 bits wide, we can put @@ -274,7 +288,7 @@ static int __pci_assign_resource(struct pci_bus *bus, struct pci_dev *dev, IORESOURCE_PREFETCH, pcibios_align_resource, dev); if (ret == 0) - return 0; + goto check_fixed; } /* @@ -287,6 +301,19 @@ static int __pci_assign_resource(struct pci_bus *bus, struct pci_dev *dev, ret = pci_bus_alloc_resource(bus, res, size, align, min, 0, pcibios_align_resource, dev); +check_fixed: + if (ret == 0 && start < end) { + if (res->start > start || res->end < end) { + dev_err(&bus->dev, "fixed area 0x%llx-0x%llx for %s doesn't fit in the allocated %pR (0x%llx-0x%llx)", + (unsigned long long)start, (unsigned long long)end, + dev_name(&dev->dev), + res, (unsigned long long)res->start, + (unsigned long long)res->end); + release_resource(res); + return -1; + } + } + return ret; } From patchwork Thu Oct 24 17:12:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210439 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3E89D139A for ; Thu, 24 Oct 2019 17:12:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1270121929 for ; Thu, 24 Oct 2019 17:12:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="eMZ8xTmG" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2436801AbfJXRMu (ORCPT ); Thu, 24 Oct 2019 13:12:50 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48798 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2436814AbfJXRMu (ORCPT ); Thu, 24 Oct 2019 13:12:50 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 7EABA42F14; Thu, 24 Oct 2019 17:12:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937168; x=1573751569; bh=DbrmeTcNhYv3PGM/IbnlLPxMlO1jbQU8BpP Nc0UZMas=; b=eMZ8xTmGQYPYhkqlqcQJLz+BLrCEu8qFNTEj1tA4ukc6tFVVH0a niKDYehTl3Bez70j4Sxz1WOCmDh6sKzd9Az+Arta0VJZtUBQoC8fvpKPXxlEQ2yJ TbkniqMEo+c/TpyLPVki1HgO1DE7LytJCg0e1b3gEHTl3lrX3lMKLSTM= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NE8yWUUPu1cu; Thu, 24 Oct 2019 20:12:48 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 1C7B443B4E; Thu, 24 Oct 2019 20:12:42 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:41 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 15/30] PCI: Fix assigning the fixed prefetchable resources Date: Thu, 24 Oct 2019 20:12:13 +0300 Message-ID: <20191024171228.877974-16-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Allow matching IORESOURCE_PCI_FIXED prefetchable BARs to non-prefetchable windows, so they follow the same rules as immovable BARs. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/setup-bus.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 653ba4d5f191..c7365998fbd6 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1339,15 +1339,20 @@ static void assign_fixed_resource_on_bus(struct pci_bus *b, struct resource *r) { int i; struct resource *parent_r; - unsigned long mask = IORESOURCE_IO | IORESOURCE_MEM | - IORESOURCE_PREFETCH; + unsigned long mask = IORESOURCE_TYPE_BITS; pci_bus_for_each_resource(b, parent_r, i) { if (!parent_r) continue; - if ((r->flags & mask) == (parent_r->flags & mask) && - resource_contains(parent_r, r)) + if ((r->flags & mask) != (parent_r->flags & mask)) + continue; + + if (parent_r->flags & IORESOURCE_PREFETCH && + !(r->flags & IORESOURCE_PREFETCH)) + continue; + + if (resource_contains(parent_r, r)) request_resource(parent_r, r); } } From patchwork Thu Oct 24 17:12:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210449 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 965B61575 for ; Thu, 24 Oct 2019 17:12:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7438921D7F for ; Thu, 24 Oct 2019 17:12:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="JdjngX3l" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2436815AbfJXRMw (ORCPT ); Thu, 24 Oct 2019 13:12:52 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48798 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2436722AbfJXRMw (ORCPT ); Thu, 24 Oct 2019 13:12:52 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id C8F7E43B0D; Thu, 24 Oct 2019 17:12:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937169; x=1573751570; bh=lTVQlF3kduJM6KsoDp7P/x9aP7WF3SRvN3M l+fzwz18=; b=JdjngX3l9+/wTkWjP6QCnvCRSDyNM79OXyCLy0n7MKH89dvGnuD DGdmaK1Qo7okFH3JnlLIjxkNu/teOiTPfyWQcqrnaFq5naZmBKN8sdTMlPpMDO+E sUe/xJWAVAgfM0dz5TUAl7g79KFPsm6YNY9UwKuA3WNFV+KZc06kbQSg= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 08x2dEaeuj55; Thu, 24 Oct 2019 20:12:49 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 5485243C28; Thu, 24 Oct 2019 20:12:42 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:41 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 16/30] PCI: hotplug: movable BARs: Assign fixed and immovable BARs before others Date: Thu, 24 Oct 2019 20:12:14 +0300 Message-ID: <20191024171228.877974-17-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Reassign resources during rescan in two steps: first the fixed/immovable BARs and bridge windows that have fixed areas, so the movable ones will not steal these reserved areas; then the rest - so the movable BARs will divide the rest of the space. With this change, pci_assign_resource() is now able to assign all types of BARs, so the pdev_assign_fixed_resources() became unused and thus removed. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/pci.h | 2 ++ drivers/pci/setup-bus.c | 78 ++++++++++++++++++++++++----------------- drivers/pci/setup-res.c | 7 ++-- 3 files changed, 53 insertions(+), 34 deletions(-) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 7cd108885598..9b5164d10499 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -289,6 +289,8 @@ void pci_bus_put(struct pci_bus *bus); bool pci_dev_bar_movable(struct pci_dev *dev, struct resource *res); +int assign_fixed_resource_on_bus(struct pci_bus *b, struct resource *r); + /* PCIe link information */ #define PCIE_SPEED2STR(speed) \ ((speed) == PCIE_SPEED_16_0GT ? "16 GT/s" : \ diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index c7365998fbd6..675a612236d7 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -38,6 +38,15 @@ struct pci_dev_resource { unsigned long flags; }; +enum assign_step { + assign_fixed_resources, + assign_float_resources, +}; + +static void _assign_requested_resources_sorted(struct list_head *head, + struct list_head *fail_head, + enum assign_step step); + static void free_list(struct list_head *head) { struct pci_dev_resource *dev_res, *tmp; @@ -278,19 +287,47 @@ static void reassign_resources_sorted(struct list_head *realloc_head, */ static void assign_requested_resources_sorted(struct list_head *head, struct list_head *fail_head) +{ + _assign_requested_resources_sorted(head, fail_head, assign_fixed_resources); + _assign_requested_resources_sorted(head, fail_head, assign_float_resources); +} + +static void _assign_requested_resources_sorted(struct list_head *head, + struct list_head *fail_head, + enum assign_step step) { struct resource *res; struct pci_dev_resource *dev_res; int idx; list_for_each_entry(dev_res, head, list) { + bool is_fixed = false; + if (!pci_dev_bars_enabled(dev_res->dev)) continue; res = dev_res->res; + if (!resource_size(res)) + continue; + idx = res - &dev_res->dev->resource[0]; - if (resource_size(res) && - pci_assign_resource(dev_res->dev, idx)) { + + if (idx < PCI_BRIDGE_RESOURCES) { + is_fixed = !pci_dev_bar_movable(dev_res->dev, res); + } else { + int b_res_idx = pci_get_bridge_resource_idx(res); + struct resource *fixed_res = + &dev_res->dev->subordinate->immovable_range[b_res_idx]; + + is_fixed = (fixed_res->start < fixed_res->end); + } + + if (assign_fixed_resources == step && !is_fixed) + continue; + else if (assign_float_resources == step && is_fixed) + continue; + + if (pci_assign_resource(dev_res->dev, idx)) { if (fail_head) { /* * If the failed resource is a ROM BAR and @@ -1335,7 +1372,7 @@ void pci_bus_size_bridges(struct pci_bus *bus) } EXPORT_SYMBOL(pci_bus_size_bridges); -static void assign_fixed_resource_on_bus(struct pci_bus *b, struct resource *r) +int assign_fixed_resource_on_bus(struct pci_bus *b, struct resource *r) { int i; struct resource *parent_r; @@ -1352,35 +1389,14 @@ static void assign_fixed_resource_on_bus(struct pci_bus *b, struct resource *r) !(r->flags & IORESOURCE_PREFETCH)) continue; - if (resource_contains(parent_r, r)) - request_resource(parent_r, r); - } -} - -/* - * Try to assign any resources marked as IORESOURCE_PCI_FIXED, as they are - * skipped by pbus_assign_resources_sorted(). - */ -static void pdev_assign_fixed_resources(struct pci_dev *dev) -{ - int i; - - for (i = 0; i < PCI_NUM_RESOURCES; i++) { - struct pci_bus *b; - struct resource *r = &dev->resource[i]; - - if (r->parent || !(r->flags & IORESOURCE_PCI_FIXED) || - !(r->flags & (IORESOURCE_IO | IORESOURCE_MEM))) - continue; - - b = dev->bus; - while (b && !r->parent) { - assign_fixed_resource_on_bus(b, r); - b = b->parent; - if (!r->parent && pci_can_move_bars) - break; + if (resource_contains(parent_r, r)) { + if (!request_resource(parent_r, r)) + return 0; } } + + dev_err(&b->dev, "failed to assign immovable %pR\n", r); + return -EBUSY; } void __pci_bus_assign_resources(const struct pci_bus *bus, @@ -1396,8 +1412,6 @@ void __pci_bus_assign_resources(const struct pci_bus *bus, if (!pci_dev_bars_enabled(dev)) continue; - pdev_assign_fixed_resources(dev); - b = dev->subordinate; if (!b) continue; diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c index 1570bbd620cd..924ea8241061 100644 --- a/drivers/pci/setup-res.c +++ b/drivers/pci/setup-res.c @@ -351,8 +351,11 @@ int pci_assign_resource(struct pci_dev *dev, int resno) resource_size_t align, size; int ret; - if (res->flags & IORESOURCE_PCI_FIXED) - return 0; + if (resno < PCI_BRIDGE_RESOURCES && + !pci_dev_bar_movable(dev, res) && + res->start) { + return assign_fixed_resource_on_bus(dev->bus, res); + } res->flags |= IORESOURCE_UNSET; align = pci_resource_alignment(dev, res); From patchwork Thu Oct 24 17:12:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210445 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9EF341575 for ; Thu, 24 Oct 2019 17:12:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7D5AB21655 for ; Thu, 24 Oct 2019 17:12:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="TZavucHX" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2439943AbfJXRMw (ORCPT ); Thu, 24 Oct 2019 13:12:52 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48808 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2436814AbfJXRMv (ORCPT ); Thu, 24 Oct 2019 13:12:51 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id A116743A25; Thu, 24 Oct 2019 17:12:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937169; x=1573751570; bh=W92cqvp1KNDy4z38kjLHnZ9tQ96YMOeOsD5 kxxU2nkc=; b=TZavucHXLFabBJ740YyKwlkVKq/xL7ufdjUAwx5bl9kTrvz2qDd fmJ4mQ89nhLOGn82fOpqB5q/GLM5iSZdWAb8w4F6l1tneWEgHA8Qt2YlpAoDg/4c mxxidwytcDvjsH+ClWNFxlNsBxqPhoGfXYdvK0iPlfuUxIYTmqLtzrLE= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nG12gWa35O-g; Thu, 24 Oct 2019 20:12:49 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 8235643C73; Thu, 24 Oct 2019 20:12:42 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:42 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 17/30] PCI: hotplug: movable BARs: Don't reserve IO/mem bus space Date: Thu, 24 Oct 2019 20:12:15 +0300 Message-ID: <20191024171228.877974-18-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org A hotplugged bridge with many hotplug-capable ports may request reserving more IO space than the machine has. This could be overridden with the "hpiosize=" kernel argument though. But when BARs are movable, there are no need to reserve space anymore: new BARs are allocated not from reserved gaps, but via rearranging the existing BARs. Requesting a precise amount of space for bridge windows increases the chances of adding the new bridge successfully. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/setup-bus.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 675a612236d7..a68ec726010e 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1285,7 +1285,7 @@ void __pci_bus_size_bridges(struct pci_bus *bus, struct list_head *realloc_head) case PCI_HEADER_TYPE_BRIDGE: pci_bridge_check_ranges(bus); - if (bus->self->is_hotplug_bridge) { + if (bus->self->is_hotplug_bridge && !pci_can_move_bars) { additional_io_size = pci_hotplug_io_size; additional_mem_size = pci_hotplug_mem_size; } From patchwork Thu Oct 24 17:12:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210447 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6ACA7112C for ; Thu, 24 Oct 2019 17:12:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4561621925 for ; Thu, 24 Oct 2019 17:12:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="vYq/kGxz" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2439944AbfJXRMw (ORCPT ); Thu, 24 Oct 2019 13:12:52 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48816 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2436815AbfJXRMw (ORCPT ); Thu, 24 Oct 2019 13:12:52 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id AF3DB43B45; Thu, 24 Oct 2019 17:12:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937170; x=1573751571; bh=nwRlUMAhBu6065MIPo0iWQpzSMgerOXg0Y1 REgkLLP0=; b=vYq/kGxzNUW7colvoS3WIq6dvHWZ6VJ5yVOfTa6Pdx0fNVKhb26 gJdCgSLxY/+HC0j93JjkaRbtHp6+YrQFoaFe7mzPOHeoOdKuI3GbITOZAJLGkh1e tLLqgvje0NORcbYSqxnqMDUOQK3BzP5GgCKfyslhea3j/KU52l1YGyoI= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PF6N3UKHVgpS; Thu, 24 Oct 2019 20:12:50 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id AFBDB43D12; Thu, 24 Oct 2019 20:12:42 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:42 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 18/30] PCI: hotplug: Configure MPS for hot-added bridges during bus rescan Date: Thu, 24 Oct 2019 20:12:16 +0300 Message-ID: <20191024171228.877974-19-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Assure that MPS settings are set up for bridges which are discovered during manually triggered rescan via sysfs. This sequence of bridge init (using pci_rescan_bus()) will be used for pciehp hot-add events when BARs are movable. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/probe.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index d0d00cb3e965..94bbdf9b9dc1 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -3414,7 +3414,7 @@ static void pci_reassign_root_bus_resources(struct pci_bus *root) unsigned int pci_rescan_bus(struct pci_bus *bus) { unsigned int max; - struct pci_bus *root = bus; + struct pci_bus *root = bus, *child; while (!pci_is_root_bus(root)) root = root->parent; @@ -3435,6 +3435,9 @@ unsigned int pci_rescan_bus(struct pci_bus *bus) pci_assign_unassigned_bus_resources(bus); } + list_for_each_entry(child, &root->children, node) + pcie_bus_configure_settings(child); + pci_bus_add_devices(bus); return max; From patchwork Thu Oct 24 17:12:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210451 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C8B1A139A for ; Thu, 24 Oct 2019 17:12:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A3F50222BE for ; Thu, 24 Oct 2019 17:12:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="loKE7TRM" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2439945AbfJXRMx (ORCPT ); Thu, 24 Oct 2019 13:12:53 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48802 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2436814AbfJXRMw (ORCPT ); Thu, 24 Oct 2019 13:12:52 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id D805C43B4E; Thu, 24 Oct 2019 17:12:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937170; x=1573751571; bh=s0nvaOPonFwNp+PkAekMHzoECVHtPLrWKHG g079gQ1c=; b=loKE7TRMwq6xvQTef6FsEMihuPxcKdB0wgT//YmWyv19Z50aUai 01L0XKTLpBkjnqVdB2rp1c+BWYXeWkpCyTyaEP5w08AuprHmk4OAD16Eqjr7hLQL /Q8vm1bi7clvN5n6RW28GXRQldddTdOoW7T3o0UrE+whLa6Ls/VCH9KA= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WMely41yv-iE; Thu, 24 Oct 2019 20:12:50 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 2BD8F43597; Thu, 24 Oct 2019 20:12:43 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:42 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 19/30] PCI: hotplug: movable BARs: Ignore the MEM BAR offsets from bootloader Date: Thu, 24 Oct 2019 20:12:17 +0300 Message-ID: <20191024171228.877974-20-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org BAR allocation by BIOS/UEFI/bootloader/firmware may be non-optimal and it may even clash with the kernel's BAR assignment algorithm. For example, if no space was reserved for SR-IOV BARs, and this bridge window is packed between immovable BARs (so it is unable to extend), and if this window can't be moved, the next PCI rescan will fail, as the kernel tries to find a space for all the BARs, including SR-IOV. With this patch the kernel will use its own methods of BAR allocating when possible, increasing the chances of successful hotplug. Also add a workaround for implicitly used video BARs on x86. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/probe.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 94bbdf9b9dc1..73452aa81417 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -305,6 +305,16 @@ int __pci_read_base(struct pci_dev *dev, enum pci_bar_type type, pos, (unsigned long long)region.start); } + if (pci_can_move_bars && + !(res->flags & IORESOURCE_IO) && + (dev->class >> 8) != PCI_CLASS_DISPLAY_VGA) { + pci_warn(dev, "ignore the current offset of BAR %llx-%llx\n", + l64, l64 + sz64 - 1); + res->start = 0; + res->end = sz64 - 1; + res->flags |= IORESOURCE_SIZEALIGN; + } + goto out; From patchwork Thu Oct 24 17:12:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210453 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 11473112C for ; Thu, 24 Oct 2019 17:12:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E3C1F20650 for ; Thu, 24 Oct 2019 17:12:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="kYuLNvLm" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2439947AbfJXRMy (ORCPT ); Thu, 24 Oct 2019 13:12:54 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48808 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2436722AbfJXRMy (ORCPT ); Thu, 24 Oct 2019 13:12:54 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id BE83242F15; Thu, 24 Oct 2019 17:12:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937171; x=1573751572; bh=VeNEpSnuUTk4BsWDwgPyj1n8sam11FoBZBf qWyZyWDk=; b=kYuLNvLmsjlwggN/VzO9PtQCcYOkqXsEtlCLCmVd8AbPb+HYTNe bZSZCyhCfhVqYyi5A8KFRRY01x7U1BT7pvwctNx3Ts8N7pGYK6IQF2RtXMcbNR2S P7jFw1LDe1qBnVg6HlPGXRW4LGn8Jr0C5MplSEG4sXqmsMVFmJfLXs8c= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rC45o4Nt2fYf; Thu, 24 Oct 2019 20:12:51 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 62A0343E03; Thu, 24 Oct 2019 20:12:43 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:42 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko , Alexey Kardashevskiy , Oliver O'Halloran , Sam Bobroff Subject: [PATCH v6 20/30] powerpc/pci: Fix crash with enabled movable BARs Date: Thu, 24 Oct 2019 20:12:18 +0300 Message-ID: <20191024171228.877974-21-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Add a check for the UNSET resource flag to skip the released BARs CC: Alexey Kardashevskiy CC: Oliver O'Halloran CC: Sam Bobroff Signed-off-by: Sergey Miroshnichenko --- arch/powerpc/platforms/powernv/pci-ioda.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c index c28d0d9b7ee0..33d5ed8c258f 100644 --- a/arch/powerpc/platforms/powernv/pci-ioda.c +++ b/arch/powerpc/platforms/powernv/pci-ioda.c @@ -2976,7 +2976,8 @@ static void pnv_ioda_setup_pe_res(struct pnv_ioda_pe *pe, int index; int64_t rc; - if (!res || !res->flags || res->start > res->end) + if (!res || !res->flags || res->start > res->end || + (res->flags & IORESOURCE_UNSET)) return; if (res->flags & IORESOURCE_IO) { From patchwork Thu Oct 24 17:12:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210455 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9C7D41575 for ; Thu, 24 Oct 2019 17:12:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6F61821925 for ; Thu, 24 Oct 2019 17:12:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="h56CXPiF" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2439948AbfJXRMz (ORCPT ); Thu, 24 Oct 2019 13:12:55 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48816 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2436814AbfJXRMy (ORCPT ); Thu, 24 Oct 2019 13:12:54 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 6F4F443597; Thu, 24 Oct 2019 17:12:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937171; x=1573751572; bh=wO+0zb9Gm5RKxo969ldU/bcx3gB/Ii1YhUM AAYpjeeI=; b=h56CXPiFTI1h9W6PFvi30wD5LJI/iOTW0+GTXwp0+O/OwICGFL9 JDGMN67nzMc+BPVUeTv0g5VR/6lI8dXRqkxmtVN1vLuWcse8OOy5/GnpjzwF1cbt 5JDO7nIzB8GrF18Ukwg5PWGcI09KFVe5Nu1bz8pc4Sm9QxLZUfCVRxdM= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nszyBibWCgs4; Thu, 24 Oct 2019 20:12:51 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 9AECA43E04; Thu, 24 Oct 2019 20:12:43 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:43 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko , Oliver O'Halloran Subject: [PATCH v6 21/30] powerpc/pci: Access PCI config space directly w/o pci_dn Date: Thu, 24 Oct 2019 20:12:19 +0300 Message-ID: <20191024171228.877974-22-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org To fetch an updated DT for the newly hotplugged device, OS must explicitly request it from the firmware via the pnv_php driver. If pnv_php wasn't triggered/loaded, it is still possible to discover new devices if PCIe I/O will not stop in absence of the pci_dn structure. Reviewed-by: Oliver O'Halloran Signed-off-by: Sergey Miroshnichenko --- arch/powerpc/kernel/rtas_pci.c | 97 +++++++++++++++++++--------- arch/powerpc/platforms/powernv/pci.c | 64 ++++++++++++------ 2 files changed, 109 insertions(+), 52 deletions(-) diff --git a/arch/powerpc/kernel/rtas_pci.c b/arch/powerpc/kernel/rtas_pci.c index ae5e43eaca48..912da28b3737 100644 --- a/arch/powerpc/kernel/rtas_pci.c +++ b/arch/powerpc/kernel/rtas_pci.c @@ -42,10 +42,26 @@ static inline int config_access_valid(struct pci_dn *dn, int where) return 0; } -int rtas_read_config(struct pci_dn *pdn, int where, int size, u32 *val) +static int rtas_read_raw_config(unsigned long buid, int busno, unsigned int devfn, + int where, int size, u32 *val) { int returnval = -1; - unsigned long buid, addr; + unsigned long addr = rtas_config_addr(busno, devfn, where); + int ret; + + if (buid) { + ret = rtas_call(ibm_read_pci_config, 4, 2, &returnval, + addr, BUID_HI(buid), BUID_LO(buid), size); + } else { + ret = rtas_call(read_pci_config, 2, 2, &returnval, addr, size); + } + *val = returnval; + + return ret; +} + +int rtas_read_config(struct pci_dn *pdn, int where, int size, u32 *val) +{ int ret; if (!pdn) @@ -58,16 +74,8 @@ int rtas_read_config(struct pci_dn *pdn, int where, int size, u32 *val) return PCIBIOS_SET_FAILED; #endif - addr = rtas_config_addr(pdn->busno, pdn->devfn, where); - buid = pdn->phb->buid; - if (buid) { - ret = rtas_call(ibm_read_pci_config, 4, 2, &returnval, - addr, BUID_HI(buid), BUID_LO(buid), size); - } else { - ret = rtas_call(read_pci_config, 2, 2, &returnval, addr, size); - } - *val = returnval; - + ret = rtas_read_raw_config(pdn->phb->buid, pdn->busno, pdn->devfn, + where, size, val); if (ret) return PCIBIOS_DEVICE_NOT_FOUND; @@ -85,18 +93,44 @@ static int rtas_pci_read_config(struct pci_bus *bus, pdn = pci_get_pdn_by_devfn(bus, devfn); - /* Validity of pdn is checked in here */ - ret = rtas_read_config(pdn, where, size, val); - if (*val == EEH_IO_ERROR_VALUE(size) && - eeh_dev_check_failure(pdn_to_eeh_dev(pdn))) - return PCIBIOS_DEVICE_NOT_FOUND; + if (pdn) { + /* Validity of pdn is checked in here */ + ret = rtas_read_config(pdn, where, size, val); + + if (*val == EEH_IO_ERROR_VALUE(size) && + eeh_dev_check_failure(pdn_to_eeh_dev(pdn))) + ret = PCIBIOS_DEVICE_NOT_FOUND; + } else { + struct pci_controller *phb = pci_bus_to_host(bus); + + ret = rtas_read_raw_config(phb->buid, bus->number, devfn, + where, size, val); + } return ret; } +static int rtas_write_raw_config(unsigned long buid, int busno, unsigned int devfn, + int where, int size, u32 val) +{ + unsigned long addr = rtas_config_addr(busno, devfn, where); + int ret; + + if (buid) { + ret = rtas_call(ibm_write_pci_config, 5, 1, NULL, addr, + BUID_HI(buid), BUID_LO(buid), size, (ulong)val); + } else { + ret = rtas_call(write_pci_config, 3, 1, NULL, addr, size, (ulong)val); + } + + if (ret) + return PCIBIOS_DEVICE_NOT_FOUND; + + return PCIBIOS_SUCCESSFUL; +} + int rtas_write_config(struct pci_dn *pdn, int where, int size, u32 val) { - unsigned long buid, addr; int ret; if (!pdn) @@ -109,15 +143,8 @@ int rtas_write_config(struct pci_dn *pdn, int where, int size, u32 val) return PCIBIOS_SET_FAILED; #endif - addr = rtas_config_addr(pdn->busno, pdn->devfn, where); - buid = pdn->phb->buid; - if (buid) { - ret = rtas_call(ibm_write_pci_config, 5, 1, NULL, addr, - BUID_HI(buid), BUID_LO(buid), size, (ulong) val); - } else { - ret = rtas_call(write_pci_config, 3, 1, NULL, addr, size, (ulong)val); - } - + ret = rtas_write_raw_config(pdn->phb->buid, pdn->busno, pdn->devfn, + where, size, val); if (ret) return PCIBIOS_DEVICE_NOT_FOUND; @@ -128,12 +155,20 @@ static int rtas_pci_write_config(struct pci_bus *bus, unsigned int devfn, int where, int size, u32 val) { - struct pci_dn *pdn; + struct pci_dn *pdn = pci_get_pdn_by_devfn(bus, devfn); + int ret; - pdn = pci_get_pdn_by_devfn(bus, devfn); + if (pdn) { + /* Validity of pdn is checked in here. */ + ret = rtas_write_config(pdn, where, size, val); + } else { + struct pci_controller *phb = pci_bus_to_host(bus); - /* Validity of pdn is checked in here. */ - return rtas_write_config(pdn, where, size, val); + ret = rtas_write_raw_config(phb->buid, bus->number, devfn, + where, size, val); + } + + return ret; } static struct pci_ops rtas_pci_ops = { diff --git a/arch/powerpc/platforms/powernv/pci.c b/arch/powerpc/platforms/powernv/pci.c index 2825d004dece..ffd546cf9204 100644 --- a/arch/powerpc/platforms/powernv/pci.c +++ b/arch/powerpc/platforms/powernv/pci.c @@ -648,30 +648,29 @@ static void pnv_pci_config_check_eeh(struct pci_dn *pdn) } } -int pnv_pci_cfg_read(struct pci_dn *pdn, - int where, int size, u32 *val) +static int pnv_pci_cfg_read_raw(u64 phb_id, int busno, unsigned int devfn, + int where, int size, u32 *val) { - struct pnv_phb *phb = pdn->phb->private_data; - u32 bdfn = (pdn->busno << 8) | pdn->devfn; + u32 bdfn = (busno << 8) | devfn; s64 rc; switch (size) { case 1: { u8 v8; - rc = opal_pci_config_read_byte(phb->opal_id, bdfn, where, &v8); + rc = opal_pci_config_read_byte(phb_id, bdfn, where, &v8); *val = (rc == OPAL_SUCCESS) ? v8 : 0xff; break; } case 2: { __be16 v16; - rc = opal_pci_config_read_half_word(phb->opal_id, bdfn, where, - &v16); + rc = opal_pci_config_read_half_word(phb_id, bdfn, where, + &v16); *val = (rc == OPAL_SUCCESS) ? be16_to_cpu(v16) : 0xffff; break; } case 4: { __be32 v32; - rc = opal_pci_config_read_word(phb->opal_id, bdfn, where, &v32); + rc = opal_pci_config_read_word(phb_id, bdfn, where, &v32); *val = (rc == OPAL_SUCCESS) ? be32_to_cpu(v32) : 0xffffffff; break; } @@ -680,27 +679,28 @@ int pnv_pci_cfg_read(struct pci_dn *pdn, } pr_devel("%s: bus: %x devfn: %x +%x/%x -> %08x\n", - __func__, pdn->busno, pdn->devfn, where, size, *val); + __func__, busno, devfn, where, size, *val); + return PCIBIOS_SUCCESSFUL; } -int pnv_pci_cfg_write(struct pci_dn *pdn, - int where, int size, u32 val) +static int pnv_pci_cfg_write_raw(u64 phb_id, int busno, unsigned int devfn, + int where, int size, u32 val) { - struct pnv_phb *phb = pdn->phb->private_data; - u32 bdfn = (pdn->busno << 8) | pdn->devfn; + u32 bdfn = (busno << 8) | devfn; pr_devel("%s: bus: %x devfn: %x +%x/%x -> %08x\n", - __func__, pdn->busno, pdn->devfn, where, size, val); + __func__, busno, devfn, where, size, val); + switch (size) { case 1: - opal_pci_config_write_byte(phb->opal_id, bdfn, where, val); + opal_pci_config_write_byte(phb_id, bdfn, where, val); break; case 2: - opal_pci_config_write_half_word(phb->opal_id, bdfn, where, val); + opal_pci_config_write_half_word(phb_id, bdfn, where, val); break; case 4: - opal_pci_config_write_word(phb->opal_id, bdfn, where, val); + opal_pci_config_write_word(phb_id, bdfn, where, val); break; default: return PCIBIOS_FUNC_NOT_SUPPORTED; @@ -709,6 +709,24 @@ int pnv_pci_cfg_write(struct pci_dn *pdn, return PCIBIOS_SUCCESSFUL; } +int pnv_pci_cfg_read(struct pci_dn *pdn, + int where, int size, u32 *val) +{ + struct pnv_phb *phb = pdn->phb->private_data; + + return pnv_pci_cfg_read_raw(phb->opal_id, pdn->busno, pdn->devfn, + where, size, val); +} + +int pnv_pci_cfg_write(struct pci_dn *pdn, + int where, int size, u32 val) +{ + struct pnv_phb *phb = pdn->phb->private_data; + + return pnv_pci_cfg_write_raw(phb->opal_id, pdn->busno, pdn->devfn, + where, size, val); +} + #if CONFIG_EEH static bool pnv_pci_cfg_check(struct pci_dn *pdn) { @@ -744,13 +762,15 @@ static int pnv_pci_read_config(struct pci_bus *bus, int where, int size, u32 *val) { struct pci_dn *pdn; - struct pnv_phb *phb; + struct pci_controller *hose = pci_bus_to_host(bus); + struct pnv_phb *phb = hose->private_data; int ret; *val = 0xFFFFFFFF; pdn = pci_get_pdn_by_devfn(bus, devfn); if (!pdn) - return PCIBIOS_DEVICE_NOT_FOUND; + return pnv_pci_cfg_read_raw(phb->opal_id, bus->number, devfn, + where, size, val); if (!pnv_pci_cfg_check(pdn)) return PCIBIOS_DEVICE_NOT_FOUND; @@ -773,12 +793,14 @@ static int pnv_pci_write_config(struct pci_bus *bus, int where, int size, u32 val) { struct pci_dn *pdn; - struct pnv_phb *phb; + struct pci_controller *hose = pci_bus_to_host(bus); + struct pnv_phb *phb = hose->private_data; int ret; pdn = pci_get_pdn_by_devfn(bus, devfn); if (!pdn) - return PCIBIOS_DEVICE_NOT_FOUND; + return pnv_pci_cfg_write_raw(phb->opal_id, bus->number, devfn, + where, size, val); if (!pnv_pci_cfg_check(pdn)) return PCIBIOS_DEVICE_NOT_FOUND; From patchwork Thu Oct 24 17:12:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210461 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 12FBA2D2C for ; Thu, 24 Oct 2019 17:12:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E51232166E for ; Thu, 24 Oct 2019 17:12:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="HALqwdtR" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2439950AbfJXRM4 (ORCPT ); Thu, 24 Oct 2019 13:12:56 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48802 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2439946AbfJXRM4 (ORCPT ); Thu, 24 Oct 2019 13:12:56 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 1136042F14; Thu, 24 Oct 2019 17:12:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937172; x=1573751573; bh=PFi03fW0QCVLRAW8MdoWj14nqW075UZz9KI WOD1BWNw=; b=HALqwdtRkjbh5YFVIEa+FfhW6QNxivGBi3lXXJEK2y8tt0NtziL Juf4rLVdzvhJBueX67wO1NUuSyKjhQzXMYF6TDhga96ma5EzyLh2JKij17gdYMk9 RGuB4AlHxTr0elXVj1KObiJAOAjJXcfTHiHpMxLlXiiNLIGGVGahQbdk= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9dviRV3je1ph; Thu, 24 Oct 2019 20:12:52 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id BFE8943E06; Thu, 24 Oct 2019 20:12:43 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:43 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko , Oliver O'Halloran , Sam Bobroff Subject: [PATCH v6 22/30] powerpc/pci: Create pci_dn on demand Date: Thu, 24 Oct 2019 20:12:20 +0300 Message-ID: <20191024171228.877974-23-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org If a struct pci_dn hasn't yet been created for the PCIe device (there was no DT node for it), allocate this structure and fill with info read from the device directly. CC: Oliver O'Halloran CC: Sam Bobroff Signed-off-by: Sergey Miroshnichenko --- arch/powerpc/kernel/pci_dn.c | 88 ++++++++++++++++++++++++++++++------ 1 file changed, 74 insertions(+), 14 deletions(-) diff --git a/arch/powerpc/kernel/pci_dn.c b/arch/powerpc/kernel/pci_dn.c index 9524009ca1ae..ad0ecf48e943 100644 --- a/arch/powerpc/kernel/pci_dn.c +++ b/arch/powerpc/kernel/pci_dn.c @@ -20,6 +20,9 @@ #include #include +static struct pci_dn *pci_create_pdn_from_dev(struct pci_dev *pdev, + struct pci_dn *parent); + /* * The function is used to find the firmware data of one * specific PCI device, which is attached to the indicated @@ -52,6 +55,9 @@ static struct pci_dn *pci_bus_to_pdn(struct pci_bus *bus) dn = pci_bus_to_OF_node(pbus); pdn = dn ? PCI_DN(dn) : NULL; + if (!pdn && pbus->self) + pdn = pbus->self->dev.archdata.pci_data; + return pdn; } @@ -61,10 +67,13 @@ struct pci_dn *pci_get_pdn_by_devfn(struct pci_bus *bus, struct device_node *dn = NULL; struct pci_dn *parent, *pdn; struct pci_dev *pdev = NULL; + bool pdev_found = false; /* Fast path: fetch from PCI device */ list_for_each_entry(pdev, &bus->devices, bus_list) { if (pdev->devfn == devfn) { + pdev_found = true; + if (pdev->dev.archdata.pci_data) return pdev->dev.archdata.pci_data; @@ -73,6 +82,9 @@ struct pci_dn *pci_get_pdn_by_devfn(struct pci_bus *bus, } } + if (!pdev_found) + pdev = NULL; + /* Fast path: fetch from device node */ pdn = dn ? PCI_DN(dn) : NULL; if (pdn) @@ -85,9 +97,12 @@ struct pci_dn *pci_get_pdn_by_devfn(struct pci_bus *bus, list_for_each_entry(pdn, &parent->child_list, list) { if (pdn->busno == bus->number && - pdn->devfn == devfn) - return pdn; - } + pdn->devfn == devfn) { + if (pdev) + pdev->dev.archdata.pci_data = pdn; + return pdn; + } + } return NULL; } @@ -117,17 +132,17 @@ struct pci_dn *pci_get_pdn(struct pci_dev *pdev) list_for_each_entry(pdn, &parent->child_list, list) { if (pdn->busno == pdev->bus->number && - pdn->devfn == pdev->devfn) + pdn->devfn == pdev->devfn) { + pdev->dev.archdata.pci_data = pdn; return pdn; + } } - return NULL; + return pci_create_pdn_from_dev(pdev, parent); } -#ifdef CONFIG_PCI_IOV -static struct pci_dn *add_one_dev_pci_data(struct pci_dn *parent, - int vf_index, - int busno, int devfn) +static struct pci_dn *pci_alloc_pdn(struct pci_dn *parent, + int busno, int devfn) { struct pci_dn *pdn; @@ -143,7 +158,6 @@ static struct pci_dn *add_one_dev_pci_data(struct pci_dn *parent, pdn->parent = parent; pdn->busno = busno; pdn->devfn = devfn; - pdn->vf_index = vf_index; pdn->pe_number = IODA_INVALID_PE; INIT_LIST_HEAD(&pdn->child_list); INIT_LIST_HEAD(&pdn->list); @@ -151,7 +165,51 @@ static struct pci_dn *add_one_dev_pci_data(struct pci_dn *parent, return pdn; } -#endif + +static struct pci_dn *pci_create_pdn_from_dev(struct pci_dev *pdev, + struct pci_dn *parent) +{ + struct pci_dn *pdn = NULL; + u32 class_code; + u16 device_id; + u16 vendor_id; + + if (!parent) + return NULL; + + pdn = pci_alloc_pdn(parent, pdev->bus->busn_res.start, pdev->devfn); + pci_info(pdev, "Create a new pdn for devfn %2x\n", pdev->devfn / 8); + + if (!pdn) { + pci_err(pdev, "%s: Failed to allocate pdn\n", __func__); + return NULL; + } + + #ifdef CONFIG_EEH + if (!eeh_dev_init(pdn)) { + kfree(pdn); + pci_err(pdev, "%s: Failed to allocate edev\n", __func__); + return NULL; + } + #endif /* CONFIG_EEH */ + + pci_bus_read_config_word(pdev->bus, pdev->devfn, + PCI_VENDOR_ID, &vendor_id); + pdn->vendor_id = vendor_id; + + pci_bus_read_config_word(pdev->bus, pdev->devfn, + PCI_DEVICE_ID, &device_id); + pdn->device_id = device_id; + + pci_bus_read_config_dword(pdev->bus, pdev->devfn, + PCI_CLASS_REVISION, &class_code); + class_code >>= 8; + pdn->class_code = class_code; + + pdev->dev.archdata.pci_data = pdn; + + return pdn; +} struct pci_dn *add_dev_pci_data(struct pci_dev *pdev) { @@ -176,15 +234,17 @@ struct pci_dn *add_dev_pci_data(struct pci_dev *pdev) for (i = 0; i < pci_sriov_get_totalvfs(pdev); i++) { struct eeh_dev *edev __maybe_unused; - pdn = add_one_dev_pci_data(parent, i, - pci_iov_virtfn_bus(pdev, i), - pci_iov_virtfn_devfn(pdev, i)); + pdn = pci_alloc_pdn(parent, + pci_iov_virtfn_bus(pdev, i), + pci_iov_virtfn_devfn(pdev, i)); if (!pdn) { dev_warn(&pdev->dev, "%s: Cannot create firmware data for VF#%d\n", __func__, i); return NULL; } + pdn->vf_index = i; + #ifdef CONFIG_EEH /* Create the EEH device for the VF */ edev = eeh_dev_init(pdn); From patchwork Thu Oct 24 17:12:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210457 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 71FE9139A for ; Thu, 24 Oct 2019 17:12:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5087D21929 for ; Thu, 24 Oct 2019 17:12:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="meqQK560" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2439949AbfJXRMz (ORCPT ); Thu, 24 Oct 2019 13:12:55 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48808 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2436722AbfJXRMz (ORCPT ); Thu, 24 Oct 2019 13:12:55 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id A7F04437FA; Thu, 24 Oct 2019 17:12:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937173; x=1573751574; bh=ya0D1NvqP3VkEc07oDTPqDYb0LTTnM15hzZ ylBEZsAk=; b=meqQK560YlC9XXFWv+fYtrLniqqpbvTB6pKOW7WM3qiKUchzQsV UOaBWGG5OgO93gzOb+/z49ZFBQX8mz/+p4FFA4Hj2Cl5wVw5iMW5w54EA19FcP8K ypoupb2/6NnbHV6I3VCaIu7wqBK2hE41LhvquPGmb/CUaUpHJbaBNJO4= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OMihRmz64bwt; Thu, 24 Oct 2019 20:12:53 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id EFC6443E08; Thu, 24 Oct 2019 20:12:43 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:43 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko , Oliver O'Halloran , Sam Bobroff Subject: [PATCH v6 23/30] powerpc/pci: hotplug: Add support for movable BARs Date: Thu, 24 Oct 2019 20:12:21 +0300 Message-ID: <20191024171228.877974-24-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Add pcibios_root_bus_rescan_prepare()/_done() hooks for the powerpc, so it can reassign the PE numbers (which depend on BAR sizes and locations) and update the EEH address cache during a PCI rescan. New PE numbers are assigned during pci_setup_bridges(root) after the rescan is done. CC: Oliver O'Halloran CC: Sam Bobroff Signed-off-by: Sergey Miroshnichenko --- arch/powerpc/kernel/pci-hotplug.c | 43 +++++++++++++++++++++++++++++++ drivers/pci/probe.c | 10 +++++++ include/linux/pci.h | 3 +++ 3 files changed, 56 insertions(+) diff --git a/arch/powerpc/kernel/pci-hotplug.c b/arch/powerpc/kernel/pci-hotplug.c index fc62c4bc47b1..42847f5b0f08 100644 --- a/arch/powerpc/kernel/pci-hotplug.c +++ b/arch/powerpc/kernel/pci-hotplug.c @@ -16,6 +16,7 @@ #include #include #include +#include static struct pci_bus *find_bus_among_children(struct pci_bus *bus, struct device_node *dn) @@ -151,3 +152,45 @@ void pci_hp_add_devices(struct pci_bus *bus) pcibios_finish_adding_to_bus(bus); } EXPORT_SYMBOL_GPL(pci_hp_add_devices); + +static void pci_hp_bus_rescan_prepare(struct pci_bus *bus) +{ + struct pci_dev *dev; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child = dev->subordinate; + + if (child) + pci_hp_bus_rescan_prepare(child); + + iommu_del_device(&dev->dev); + } + + list_for_each_entry(dev, &bus->devices, bus_list) { + pcibios_release_device(dev); + } +} + +static void pci_hp_bus_rescan_done(struct pci_bus *bus) +{ + struct pci_dev *dev; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child = dev->subordinate; + + pcibios_bus_add_device(dev); + + if (child) + pci_hp_bus_rescan_done(child); + } +} + +void pcibios_root_bus_rescan_prepare(struct pci_bus *root) +{ + pci_hp_bus_rescan_prepare(root); +} + +void pcibios_root_bus_rescan_done(struct pci_bus *root) +{ + pci_hp_bus_rescan_done(root); +} diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 73452aa81417..539f5d39bb6d 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -3235,6 +3235,14 @@ static void pci_bus_rescan_done(struct pci_bus *bus) pci_config_pm_runtime_put(bus->self); } +void __weak pcibios_root_bus_rescan_prepare(struct pci_bus *root) +{ +} + +void __weak pcibios_root_bus_rescan_done(struct pci_bus *root) +{ +} + static void pci_setup_bridges(struct pci_bus *bus) { struct pci_dev *dev; @@ -3430,6 +3438,7 @@ unsigned int pci_rescan_bus(struct pci_bus *bus) root = root->parent; if (pci_can_move_bars) { + pcibios_root_bus_rescan_prepare(root); pci_bus_rescan_prepare(root); pci_bus_update_immovable_range(root); pci_bus_release_root_bridge_resources(root); @@ -3440,6 +3449,7 @@ unsigned int pci_rescan_bus(struct pci_bus *bus) pci_setup_bridges(root); pci_bus_rescan_done(root); + pcibios_root_bus_rescan_done(root); } else { max = pci_scan_child_bus(bus); pci_assign_unassigned_bus_resources(bus); diff --git a/include/linux/pci.h b/include/linux/pci.h index e1edcb3fad31..b5821134bdae 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -1275,6 +1275,9 @@ unsigned int pci_rescan_bus(struct pci_bus *bus); void pci_lock_rescan_remove(void); void pci_unlock_rescan_remove(void); +void pcibios_root_bus_rescan_prepare(struct pci_bus *root); +void pcibios_root_bus_rescan_done(struct pci_bus *root); + /* Vital Product Data routines */ ssize_t pci_read_vpd(struct pci_dev *dev, loff_t pos, size_t count, void *buf); ssize_t pci_write_vpd(struct pci_dev *dev, loff_t pos, size_t count, const void *buf); From patchwork Thu Oct 24 17:12:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210459 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CDF531575 for ; Thu, 24 Oct 2019 17:12:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id ABEDD21906 for ; Thu, 24 Oct 2019 17:12:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="Viyk7VPa" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2436722AbfJXRM4 (ORCPT ); Thu, 24 Oct 2019 13:12:56 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48816 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2436814AbfJXRMz (ORCPT ); Thu, 24 Oct 2019 13:12:55 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 1E2CB438CE; Thu, 24 Oct 2019 17:12:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937174; x=1573751575; bh=2IGyeDgGJ6qusS98yQ3CmH65HU7DzXb7ja0 tqKXs9LI=; b=Viyk7VPaJtIdDmECs2lmFodI7riWdZrmew8Q8a5QFvpMyeYKLhz NHpeYOqkGPtYEc1bSzmPzRD+jakQGVA+Ep4pWq7Q84TBPTghPumPI3jhMY4A6uMe OgHKYjXk9lACcX+ZZKABuum6dlgdBv3ltjem5bbtWY1sWa806cru+kq0= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3FXMhiCyE9oV; Thu, 24 Oct 2019 20:12:54 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 2F965437F3; Thu, 24 Oct 2019 20:12:44 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:43 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko , Oliver O'Halloran , Sam Bobroff Subject: [PATCH v6 24/30] powerpc/powernv/pci: Suppress an EEH error when reading an empty slot Date: Thu, 24 Oct 2019 20:12:22 +0300 Message-ID: <20191024171228.877974-25-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Reading an empty slot returns all ones, which triggers a false EEH error event on PowerNV. A rescan is performed after all the PEs have been unmapped, so the reserved PE index is used for unfreezing. CC: Oliver O'Halloran CC: Sam Bobroff Signed-off-by: Sergey Miroshnichenko --- arch/powerpc/platforms/powernv/pci.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/arch/powerpc/platforms/powernv/pci.c b/arch/powerpc/platforms/powernv/pci.c index ffd546cf9204..e1b45dc96474 100644 --- a/arch/powerpc/platforms/powernv/pci.c +++ b/arch/powerpc/platforms/powernv/pci.c @@ -768,9 +768,16 @@ static int pnv_pci_read_config(struct pci_bus *bus, *val = 0xFFFFFFFF; pdn = pci_get_pdn_by_devfn(bus, devfn); - if (!pdn) - return pnv_pci_cfg_read_raw(phb->opal_id, bus->number, devfn, - where, size, val); + if (!pdn) { + ret = pnv_pci_cfg_read_raw(phb->opal_id, bus->number, devfn, + where, size, val); + + if (!ret && (*val == EEH_IO_ERROR_VALUE(size)) && phb->unfreeze_pe) + phb->unfreeze_pe(phb, phb->ioda.reserved_pe_idx, + OPAL_EEH_ACTION_CLEAR_FREEZE_ALL); + + return ret; + } if (!pnv_pci_cfg_check(pdn)) return PCIBIOS_DEVICE_NOT_FOUND; From patchwork Thu Oct 24 17:12:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210463 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C1A2E139A for ; Thu, 24 Oct 2019 17:12:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A022821906 for ; Thu, 24 Oct 2019 17:12:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="FaxfcdEL" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2439952AbfJXRM5 (ORCPT ); Thu, 24 Oct 2019 13:12:57 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48986 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2436814AbfJXRM4 (ORCPT ); Thu, 24 Oct 2019 13:12:56 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id B70E9437F3; Thu, 24 Oct 2019 17:12:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937174; x=1573751575; bh=tAJ6LG7xv03hfVfcclMX90rnUJKCJ1G3FmF 9HhqNFeI=; b=FaxfcdELYvncaar7/BGagEloRrYHIq4UGgeM9O42eEmTFzNtR6W 8HBjbTt1qiI0rJeChAKYUk8Hu9/IaTRAaNahHDiGY78zyDvP65Cg0uD3W3sGtzUO +g6qrxUGzKJ9PY/n/me0EMZDwMFF1dMMjGXlkDCbtc4LqWHBEe5mRHE8= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WA_aE2_I2WcZ; Thu, 24 Oct 2019 20:12:54 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 6B2F5437F6; Thu, 24 Oct 2019 20:12:44 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:44 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko , "Rafael J . Wysocki" Subject: [PATCH v6 25/30] PNP: Don't reserve BARs for PCI when enabled movable BARs Date: Thu, 24 Oct 2019 20:12:23 +0300 Message-ID: <20191024171228.877974-26-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When the Movable BARs feature is supported, the PCI subsystem is able to distribute existing BARs and allocate the new ones itself, without need to reserve gaps by BIOS. CC: Rafael J. Wysocki Signed-off-by: Sergey Miroshnichenko --- drivers/pnp/system.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/pnp/system.c b/drivers/pnp/system.c index 6950503741eb..5977bd11f4d4 100644 --- a/drivers/pnp/system.c +++ b/drivers/pnp/system.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include @@ -58,6 +59,9 @@ static void reserve_resources_of_dev(struct pnp_dev *dev) struct resource *res; int i; + if (pci_can_move_bars) + return; + for (i = 0; (res = pnp_get_resource(dev, IORESOURCE_IO, i)); i++) { if (res->flags & IORESOURCE_DISABLED) continue; From patchwork Thu Oct 24 17:12:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210465 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EC9501575 for ; Thu, 24 Oct 2019 17:12:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CB9C321906 for ; Thu, 24 Oct 2019 17:12:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="UycNJ8PF" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2436814AbfJXRM5 (ORCPT ); Thu, 24 Oct 2019 13:12:57 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48808 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2439951AbfJXRM4 (ORCPT ); Thu, 24 Oct 2019 13:12:56 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 265C543130; Thu, 24 Oct 2019 17:12:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937175; x=1573751576; bh=o9pnKC2hPbcFQGvVIGFilenU7gIJ+8sIWVT 5vIcrWlQ=; b=UycNJ8PFCHoLxuaVVia1+xuwc/XZA8G9EfPY8kJpi4cZZaqZT0c f5yFOZAxHXUsVkvoaBA63J9DHe4cPXKD0yTyumOuMXRp5XA/hk9Hb3e27NgRp1nY cacODBfJM1AK8OT2CFjMwjG5z7nTWYYAmphWh8A/79byqe1E5ijMO9tk= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id S6S0OHwAtxJU; Thu, 24 Oct 2019 20:12:55 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 9C89043E0E; Thu, 24 Oct 2019 20:12:44 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:44 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko , Oliver O'Halloran Subject: [PATCH v6 26/30] PCI: hotplug: movable BARs: Enable the feature by default Date: Thu, 24 Oct 2019 20:12:24 +0300 Message-ID: <20191024171228.877974-27-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org This is the last patch in the series which implements the essentials of the Movable BARs feature, so it is turned by default now. Tested on: - x86_64 with "pci=realloc,pcie_bus_peer2peer" command line argument; - POWER8 PowerNV+PHB3 ppc64le with "pci=realloc,pcie_bus_peer2peer". In case of problems it is still can be overridden by the following command line option: pcie_movable_bars=off CC: Oliver O'Halloran Signed-off-by: Sergey Miroshnichenko --- drivers/pci/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 85014c6b2817..6ec1b70e4a96 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -78,7 +78,7 @@ static void pci_dev_d3_sleep(struct pci_dev *dev) int pci_domains_supported = 1; #endif -bool pci_can_move_bars; +bool pci_can_move_bars = true; #define DEFAULT_CARDBUS_IO_SIZE (256) #define DEFAULT_CARDBUS_MEM_SIZE (64*1024*1024) From patchwork Thu Oct 24 17:12:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210469 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6395A1575 for ; Thu, 24 Oct 2019 17:12:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4206B21906 for ; Thu, 24 Oct 2019 17:12:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="eG207hBf" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2439954AbfJXRM6 (ORCPT ); Thu, 24 Oct 2019 13:12:58 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48816 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2439946AbfJXRM6 (ORCPT ); Thu, 24 Oct 2019 13:12:58 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id F03F6437F6; Thu, 24 Oct 2019 17:12:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937175; x=1573751576; bh=BvsEhzhdlSaFHugg08Hbv0nvpwWr5AVr2aa 4bqlxehk=; b=eG207hBfekp7gIVP9tYX87mJmnN20Wm0vkPCRBxt1glvBS11Ef7 ubY8GIY5QMUMG1rkl5/CZi+kgPZPLZ7aR83CFYrv5w1L8rgVE18h5WjSrC47J4rN XKQ383EPrB5beTrDQ/06vBUMP6kNVmRJQKPyLD/TaEanvVhn/LJcVSYk= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id l66uyQuYIjNe; Thu, 24 Oct 2019 20:12:55 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id D5A2443E10; Thu, 24 Oct 2019 20:12:44 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:44 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko , , Christoph Hellwig Subject: [PATCH v6 27/30] nvme-pci: Handle movable BARs Date: Thu, 24 Oct 2019 20:12:25 +0300 Message-ID: <20191024171228.877974-28-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Hotplugged devices can affect the existing ones by moving their BARs. The PCI subsystem will inform the NVME driver about this by invoking the .rescan_prepare() and .rescan_done() hooks, so the BARs can by re-mapped. Tested under the "randrw" mode of the fio tool. Before the hotplugging: % sudo cat /proc/iomem ... 3fe800000000-3fe8007fffff : PCI Bus 0020:0b 3fe800000000-3fe8007fffff : PCI Bus 0020:18 3fe800000000-3fe8000fffff : 0020:18:00.0 3fe800000000-3fe8000fffff : nvme 3fe800100000-3fe80017ffff : 0020:18:00.0 ... , then another NVME drive was hot-added, so BARs of the 0020:18:00.0 are moved: % sudo cat /proc/iomem ... 3fe800000000-3fe800ffffff : PCI Bus 0020:0b 3fe800000000-3fe8007fffff : PCI Bus 0020:10 3fe800000000-3fe800003fff : 0020:10:00.0 3fe800000000-3fe800003fff : nvme 3fe800010000-3fe80001ffff : 0020:10:00.0 3fe800800000-3fe800ffffff : PCI Bus 0020:18 3fe800800000-3fe8008fffff : 0020:18:00.0 3fe800800000-3fe8008fffff : nvme 3fe800900000-3fe80097ffff : 0020:18:00.0 ... During the rescanning, both READ and WRITE speeds drop to zero for a while due to driver's pause, then restore. Also tested with an NVME as a system drive. Cc: linux-nvme@lists.infradead.org Cc: Christoph Hellwig Signed-off-by: Sergey Miroshnichenko --- drivers/nvme/host/pci.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 869f462e6b6e..5f162ea5a5f1 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1650,7 +1650,7 @@ static int nvme_remap_bar(struct nvme_dev *dev, unsigned long size) { struct pci_dev *pdev = to_pci_dev(dev->dev); - if (size <= dev->bar_mapped_size) + if (dev->bar && size <= dev->bar_mapped_size) return 0; if (size > pci_resource_len(pdev, 0)) return -ENOMEM; @@ -3059,6 +3059,23 @@ static void nvme_error_resume(struct pci_dev *pdev) flush_work(&dev->ctrl.reset_work); } +static void nvme_rescan_prepare(struct pci_dev *pdev) +{ + struct nvme_dev *dev = pci_get_drvdata(pdev); + + nvme_dev_disable(dev, false); + nvme_dev_unmap(dev); + dev->bar = NULL; +} + +static void nvme_rescan_done(struct pci_dev *pdev) +{ + struct nvme_dev *dev = pci_get_drvdata(pdev); + + nvme_dev_map(dev); + nvme_reset_ctrl_sync(&dev->ctrl); +} + static const struct pci_error_handlers nvme_err_handler = { .error_detected = nvme_error_detected, .slot_reset = nvme_slot_reset, @@ -3135,6 +3152,8 @@ static struct pci_driver nvme_driver = { #endif .sriov_configure = pci_sriov_configure_simple, .err_handler = &nvme_err_handler, + .rescan_prepare = nvme_rescan_prepare, + .rescan_done = nvme_rescan_done, }; static int __init nvme_init(void) From patchwork Thu Oct 24 17:12:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210467 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B2F01112C for ; Thu, 24 Oct 2019 17:12:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 90EC521655 for ; Thu, 24 Oct 2019 17:12:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="jn02e3pL" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2439953AbfJXRM6 (ORCPT ); Thu, 24 Oct 2019 13:12:58 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48802 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2439951AbfJXRM6 (ORCPT ); Thu, 24 Oct 2019 13:12:58 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 31136438D1; Thu, 24 Oct 2019 17:12:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937176; x=1573751577; bh=siX5+YSMeoM/DO0y3MLeeru9IHUDsIROYua KtHFaz2I=; b=jn02e3pLoFm0jLK1xraa8TmbPdNHvs6c8dKtLpynLqGJksBIPie SWETUxjd/2r78UCP2plAUYqvMT00yWtPcaxQ4K7bfFS57QiA84n2PqavcmvvdFmD moacvXo5AgLXTHPShu2kFWOrl9dihTA2Atu7GOsIKlpoIAsYv9jx7ots= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wfSg_j1etxU7; Thu, 24 Oct 2019 20:12:56 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 118A943E16; Thu, 24 Oct 2019 20:12:45 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:44 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko Subject: [PATCH v6 28/30] PCI/portdrv: Declare support of movable BARs Date: Thu, 24 Oct 2019 20:12:26 +0300 Message-ID: <20191024171228.877974-29-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Switch's BARs are not used by the portdrv driver, but they are still considered as immovable until the .rescan_prepare() and .rescan_done() hooks are added. Add these hooks to increase chances to allocate new BARs. Signed-off-by: Sergey Miroshnichenko --- drivers/pci/pcie/portdrv_pci.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/drivers/pci/pcie/portdrv_pci.c b/drivers/pci/pcie/portdrv_pci.c index 0a87091a0800..9dbddc7faaa7 100644 --- a/drivers/pci/pcie/portdrv_pci.c +++ b/drivers/pci/pcie/portdrv_pci.c @@ -197,6 +197,14 @@ static const struct pci_error_handlers pcie_portdrv_err_handler = { .resume = pcie_portdrv_err_resume, }; +static void pcie_portdrv_rescan_prepare(struct pci_dev *pdev) +{ +} + +static void pcie_portdrv_rescan_done(struct pci_dev *pdev) +{ +} + static struct pci_driver pcie_portdriver = { .name = "pcieport", .id_table = &port_pci_ids[0], @@ -207,6 +215,9 @@ static struct pci_driver pcie_portdriver = { .err_handler = &pcie_portdrv_err_handler, + .rescan_prepare = pcie_portdrv_rescan_prepare, + .rescan_done = pcie_portdrv_rescan_done, + .driver.pm = PCIE_PORTDRV_PM_OPS, }; From patchwork Thu Oct 24 17:12:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210473 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EB046112C for ; Thu, 24 Oct 2019 17:13:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C513E21906 for ; Thu, 24 Oct 2019 17:13:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="sSU16cwM" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2439956AbfJXRNE (ORCPT ); Thu, 24 Oct 2019 13:13:04 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:49038 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2439946AbfJXRNE (ORCPT ); Thu, 24 Oct 2019 13:13:04 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 2337043611; Thu, 24 Oct 2019 17:13:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937177; x=1573751578; bh=Ly1PnEFeD8eqUYeDGQz6RzFYm/+hza4Wmd7 9/emghxo=; b=sSU16cwMIvKkPcKFHziNyzbW4yXBUTk0MhmVLo4MN+CF6CHc1zs NxMJNhKC6BQt1ypZ5RIexaxkLKgsFYwLH1issHPKIToBcCEi2oTkCFzxXlE/+QGP 9/FdDUn9qFrLjHiWPYKlO459o1qhf3Iu4KYM38A4JDmvTg4qurLb1VFQ= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id adOlb3e024Bv; Thu, 24 Oct 2019 20:12:57 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 44CD1437F8; Thu, 24 Oct 2019 20:12:45 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:44 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko , Lukas Wunner Subject: [PATCH v6 29/30] PCI: pciehp: movable BARs: Trigger a domain rescan on hp events Date: Thu, 24 Oct 2019 20:12:27 +0300 Message-ID: <20191024171228.877974-30-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org With movable BARs, adding a hotplugged device is not local to its bridge anymore, but it affects the whole domain: BARs, bridge windows and bus numbers can be substantially rearranged. So instead of trying to fit the new devices into preallocated reserved gaps, initiate a full domain rescan. The pci_rescan_bus() covers all the operations of the replaced functions: - assigning new bus numbers, as the pci_hp_add_bridge() does it; - allocating BARs (pci_assign_unassigned_bridge_resources()); - cofiguring MPS settings (pcie_bus_configure_settings()); - binding devices to their drivers (pci_bus_add_devices()). CC: Lukas Wunner Signed-off-by: Sergey Miroshnichenko --- drivers/pci/hotplug/pciehp_pci.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/pci/hotplug/pciehp_pci.c b/drivers/pci/hotplug/pciehp_pci.c index d17f3bf36f70..6d4c1ef38210 100644 --- a/drivers/pci/hotplug/pciehp_pci.c +++ b/drivers/pci/hotplug/pciehp_pci.c @@ -58,6 +58,11 @@ int pciehp_configure_device(struct controller *ctrl) goto out; } + if (pci_can_move_bars) { + pci_rescan_bus(parent); + goto out; + } + for_each_pci_bridge(dev, parent) pci_hp_add_bridge(dev); From patchwork Thu Oct 24 17:12:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 11210471 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 14F56112C for ; Thu, 24 Oct 2019 17:13:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E729B2166E for ; Thu, 24 Oct 2019 17:12:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=yadro.com header.i=@yadro.com header.b="hmU0M1g4" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2439955AbfJXRM7 (ORCPT ); Thu, 24 Oct 2019 13:12:59 -0400 Received: from mta-02.yadro.com ([89.207.88.252]:48808 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2439951AbfJXRM7 (ORCPT ); Thu, 24 Oct 2019 13:12:59 -0400 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 5541043597; Thu, 24 Oct 2019 17:12:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1571937177; x=1573751578; bh=WoAGswMD0jcTKXY/it96M1aTyTP0ATGaJ8r 2Yos5H2Q=; b=hmU0M1g41x6vEfQPw5xHcLxC9h3WP7gvi3IHdFTDnisYPAjz2DD ohjsb/HzrLr/cDv6yQR/L8ZD4J1NP+/ygQCJPR9eajrPWAo6qbx6xQzX6GcHhwxs malwQtTiHiySjs4VMjl/hqOFJJP7xEhRAM+3CgSu4XWYiK5oznNeNcz8= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2GSjhsYQarjV; Thu, 24 Oct 2019 20:12:57 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 74F5843E18; Thu, 24 Oct 2019 20:12:45 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Thu, 24 Oct 2019 20:12:45 +0300 From: Sergey Miroshnichenko To: , CC: Bjorn Helgaas , , Sergey Miroshnichenko , Benjamin Herrenschmidt Subject: [PATCH v6 30/30] Revert "powerpc/powernv/pci: Work around races in PCI bridge enabling" Date: Thu, 24 Oct 2019 20:12:28 +0300 Message-ID: <20191024171228.877974-31-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191024171228.877974-1-s.miroshnichenko@yadro.com> References: <20191024171228.877974-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org This reverts commit db2173198b9513f7add8009f225afa1f1c79bcc6. The root cause of this bug is fixed by the following two commits: 1. "PCI: Fix race condition in pci_enable/disable_device()" 2. "PCI: Enable bridge's I/O and MEM access for hotplugged devices" The x86 is also affected by this bug if a PCIe bridge has been hotplugged without pre-enabling by the BIOS. CC: Benjamin Herrenschmidt Signed-off-by: Sergey Miroshnichenko --- arch/powerpc/platforms/powernv/pci-ioda.c | 37 ----------------------- 1 file changed, 37 deletions(-) diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c index 33d5ed8c258f..f12f3a49d3bb 100644 --- a/arch/powerpc/platforms/powernv/pci-ioda.c +++ b/arch/powerpc/platforms/powernv/pci-ioda.c @@ -3119,49 +3119,12 @@ static void pnv_pci_ioda_create_dbgfs(void) #endif /* CONFIG_DEBUG_FS */ } -static void pnv_pci_enable_bridge(struct pci_bus *bus) -{ - struct pci_dev *dev = bus->self; - struct pci_bus *child; - - /* Empty bus ? bail */ - if (list_empty(&bus->devices)) - return; - - /* - * If there's a bridge associated with that bus enable it. This works - * around races in the generic code if the enabling is done during - * parallel probing. This can be removed once those races have been - * fixed. - */ - if (dev) { - int rc = pci_enable_device(dev); - if (rc) - pci_err(dev, "Error enabling bridge (%d)\n", rc); - pci_set_master(dev); - } - - /* Perform the same to child busses */ - list_for_each_entry(child, &bus->children, node) - pnv_pci_enable_bridge(child); -} - -static void pnv_pci_enable_bridges(void) -{ - struct pci_controller *hose; - - list_for_each_entry(hose, &hose_list, list_node) - pnv_pci_enable_bridge(hose->bus); -} - static void pnv_pci_ioda_fixup(void) { pnv_pci_ioda_setup_PEs(); pnv_pci_ioda_setup_iommu_api(); pnv_pci_ioda_create_dbgfs(); - pnv_pci_enable_bridges(); - #ifdef CONFIG_EEH pnv_eeh_post_init(); #endif