From patchwork Fri Aug 3 11:24:31 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Takao Indoh X-Patchwork-Id: 1270401 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Original-To: patchwork-linux-pci@patchwork.kernel.org Delivered-To: patchwork-process-083081@patchwork1.kernel.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by patchwork1.kernel.org (Postfix) with ESMTP id 133C33FCFC for ; Fri, 3 Aug 2012 11:28:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753409Ab2HCL2n (ORCPT ); Fri, 3 Aug 2012 07:28:43 -0400 Received: from fgwmail6.fujitsu.co.jp ([192.51.44.36]:51577 "EHLO fgwmail6.fujitsu.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751568Ab2HCL2n (ORCPT ); Fri, 3 Aug 2012 07:28:43 -0400 Received: from m4.gw.fujitsu.co.jp (unknown [10.0.50.74]) by fgwmail6.fujitsu.co.jp (Postfix) with ESMTP id A6CCE3EE0BC; Fri, 3 Aug 2012 20:28:41 +0900 (JST) Received: from smail (m4 [127.0.0.1]) by outgoing.m4.gw.fujitsu.co.jp (Postfix) with ESMTP id 8EA4745DE50; Fri, 3 Aug 2012 20:28:41 +0900 (JST) Received: from s4.gw.fujitsu.co.jp (s4.gw.fujitsu.co.jp [10.0.50.94]) by m4.gw.fujitsu.co.jp (Postfix) with ESMTP id 78CD145DE4F; Fri, 3 Aug 2012 20:28:41 +0900 (JST) Received: from s4.gw.fujitsu.co.jp (localhost.localdomain [127.0.0.1]) by s4.gw.fujitsu.co.jp (Postfix) with ESMTP id 6A9051DB8037; Fri, 3 Aug 2012 20:28:41 +0900 (JST) Received: from ml14.s.css.fujitsu.com (ml14.s.css.fujitsu.com [10.240.81.134]) by s4.gw.fujitsu.co.jp (Postfix) with ESMTP id 0F411E08001; Fri, 3 Aug 2012 20:28:41 +0900 (JST) Received: from ml14.css.fujitsu.com (ml14 [127.0.0.1]) by ml14.s.css.fujitsu.com (Postfix) with ESMTP id D5F439F783A; Fri, 3 Aug 2012 20:28:40 +0900 (JST) Received: from [127.0.0.1] (unknown [10.124.101.83]) by ml14.s.css.fujitsu.com (Postfix) with ESMTP id 54B779F7839; Fri, 3 Aug 2012 20:28:40 +0900 (JST) X-SecurityPolicyCheck: OK by SHieldMailChecker v1.7.4 Message-ID: <501BB4EF.7080909@jp.fujitsu.com> Date: Fri, 03 Aug 2012 20:24:31 +0900 From: Takao Indoh User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:14.0) Gecko/20120713 Thunderbird/14.0 MIME-Version: 1.0 To: kexec@lists.infradead.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org CC: bhelgaas@google.com, vgoyal@redhat.com, hbabu@us.ibm.com, ishii.hironobu@jp.fujitsu.com, martin.wilck@ts.fujitsu.com Subject: [RFC][PATCH] Reset PCIe devices to address DMA problem on kdump with iommu Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Hi all, This patch adds kernel parameter "reset_pcie_devices" which resets PCIe devices at boot time to address DMA problem on kdump with iommu. When this parameter is specified, a hot reset is triggered on each PCIe root port and downstream port to reset its downstream endpoint. Background: A kdump problem about DMA has been discussed for a long time. That is, when a kernel is switched to the kdump kernel DMA derived from first kernel affects second kernel. Recently this problem surfaces when iommu is used for PCI passthrough on KVM guest. In the case of the machine I use, when intel_iommu=on is specified, DMAR error is detected in kdump kernel and PCI SERR is also detected. Finally kdump fails because some devices does not work correctly. The root cause is that ongoing DMA from first kernel causes DMAR fault because page table of DMAR is initialized while kdump kernel is booting up. Therefore to address this problem DMA needs to be stopped before DMAR is initialized at kdump kernel boot time. By this patch, PCIe devices are reset by hot reset and its DMA is stopped when reset_pcie_devices is specified. One problem of this solution is that VGA is reset and the monitor blacks out when the link between the port and VGA controller was reset. So this patch does not reset the port whose child endpoint is VGA device. Any comments would be appreciated. Signed-off-by: Takao Indoh --- Documentation/kernel-parameters.txt | 4 + drivers/pci/quirks.c | 59 ++++++++++++++++++++++++++ 2 files changed, 63 insertions(+) -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt index e714a02..e694e9f 100644 --- a/Documentation/kernel-parameters.txt +++ b/Documentation/kernel-parameters.txt @@ -2489,6 +2489,10 @@ bytes respectively. Such letter suffixes can also be entirely omitted. reset_devices [KNL] Force drivers to reset the underlying device during initialization. + reset_pcie_devices + [PCIE] Reset PCIe endpoint at boot time by sending a + hot reset to root port and downstream port + resume= [SWSUSP] Specify the partition device for software suspend Format: diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 5155317..7f7fc02 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -32,6 +32,65 @@ #include "pci.h" /* + * Reset PCIe endpoint by sending a hot reset to root port and downstream port + */ +unsigned int reset_pcie_devices; +EXPORT_SYMBOL(reset_pcie_devices); +static int __init set_reset_pcie_devices(char *str) +{ + reset_pcie_devices = 1; + return 1; +} +__setup("reset_pcie_devices", set_reset_pcie_devices); + +static void __devinit quirk_pcie_device_reset(struct pci_dev *dev) +{ + struct pci_bus *subordinate; + struct pci_dev *child; + u16 ctrl; + + if (!reset_pcie_devices || !pci_is_pcie(dev) || !dev->subordinate || + ((dev->pcie_type != PCI_EXP_TYPE_ROOT_PORT) && + (dev->pcie_type != PCI_EXP_TYPE_DOWNSTREAM))) + return; + + subordinate = dev->subordinate; + list_for_each_entry(child, &subordinate->devices, bus_list) { + if ((child->pcie_type == PCI_EXP_TYPE_UPSTREAM) || + (child->pcie_type == PCI_EXP_TYPE_PCI_BRIDGE) || + ((child->class >> 16) == PCI_BASE_CLASS_DISPLAY)) + /* Don't reset switch, bridge, VGA device */ + return; + } + + dev_info(&dev->dev, "Reset Secondary bus\n"); + + list_for_each_entry(child, &subordinate->devices, bus_list) { + dev_info(&child->dev, "save state\n"); + pci_save_state(child); + } + + /* Assert Secondary Bus Reset */ + pci_read_config_word(dev, PCI_BRIDGE_CONTROL, &ctrl); + ctrl |= PCI_BRIDGE_CTL_BUS_RESET; + pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl); + + msleep(2); + + /* De-assert Secondary Bus Reset */ + ctrl &= ~PCI_BRIDGE_CTL_BUS_RESET; + pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl); + + msleep(200); + + list_for_each_entry(child, &subordinate->devices, bus_list) { + dev_info(&child->dev, "restore state\n"); + pci_restore_state(child); + } +} +DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID, quirk_pcie_device_reset); + +/* * Decoding should be disabled for a PCI device during BAR sizing to avoid * conflict. But doing so may cause problems on host bridge and perhaps other * key system devices. For devices that need to have mmio decoding always-on,