From patchwork Fri Feb 14 21:33:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Sean V Kelley X-Patchwork-Id: 11383307 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A2CFE139A for ; Fri, 14 Feb 2020 21:33:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 821E52465D for ; Fri, 14 Feb 2020 21:33:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389206AbgBNVdY (ORCPT ); Fri, 14 Feb 2020 16:33:24 -0500 Received: from mga14.intel.com ([192.55.52.115]:64984 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388914AbgBNVdX (ORCPT ); Fri, 14 Feb 2020 16:33:23 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 14 Feb 2020 13:33:23 -0800 X-IronPort-AV: E=Sophos;i="5.70,441,1574150400"; d="scan'208";a="227742884" Received: from mravago-mobl.amr.corp.intel.com (HELO arch-ashland-svkelley.intel.com) ([10.252.135.120]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 14 Feb 2020 13:33:22 -0800 From: Sean V Kelley To: tglx@linutronix.de, bhelgaas@google.com, corbet@lwn.net, mingo@redhat.com, bp@alien8.de Cc: x86@kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kar.hin.ong@ni.com, sassmann@kpanic.de, Sean V Kelley Subject: [PATCH 1/2] pci: Add boot interrupt quirk mechanism for Xeon chipsets Date: Fri, 14 Feb 2020 13:33:12 -0800 Message-Id: <20200214213313.66622-2-sean.v.kelley@linux.intel.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200214213313.66622-1-sean.v.kelley@linux.intel.com> References: <20200214213313.66622-1-sean.v.kelley@linux.intel.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org The following was observed by Kar Hin Ong with RT patchset: Backtrace: irq 19: nobody cared (try booting with the "irqpoll" option) CPU: 0 PID: 3329 Comm: irq/34-nipalk Tainted:4.14.87-rt49 #1 Hardware name: National Instruments NI PXIe-8880/NI PXIe-8880, BIOS 2.1.5f1 01/09/2020 Call Trace: ? dump_stack+0x46/0x5e ? __report_bad_irq+0x2e/0xb0 ? note_interrupt+0x242/0x290 ? nNIKAL100_memoryRead16+0x8/0x10 [nikal] ? handle_irq_event_percpu+0x55/0x70 ? handle_irq_event+0x4f/0x80 ? handle_fasteoi_irq+0x81/0x180 ? handle_irq+0x1c/0x30 ? do_IRQ+0x41/0xd0 ? common_interrupt+0x84/0x84 ... handlers: [] irq_default_primary_handler threaded [] usb_hcd_irq Disabling IRQ #19 The problem being that this device is triggering boot interrupts due to threaded interrupt handling and masking of the IO-APIC. These boot interrupts are then forwarded on to the legacy PCH's PIRQ lines where there is no handler present for the device. Whenever a PCI device is firing interrupt (INTx) to Pin 20 of IOAPIC 2 (GSI 44); the kernel will receives 2 interrupts: 1. Interrupt from Pin 20 of IOAPIC 2 -> Expected 2. Interrupt from Pin 19 of IOAPIC 1 -> UNEXPECTED Quirks for disabling boot interrupts (preferred) or rerouting the handler exist but do not address these Xeon chipsets' mechanism: https://lore.kernel.org/lkml/12131949181903-git-send-email-sassmann@suse.de/ This patch adds a new mechanism via PCI CFG for those chipsets supporting CIPINTRC register's dis_intx_rout2ich bit. Reported-by: Kar Hin Ong Tested-by: Kar Hin Ong Signed-off-by: Sean V Kelley --- drivers/pci/quirks.c | 80 ++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 73 insertions(+), 7 deletions(-) diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 29f473ebf20f..e8ee670877d1 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -1970,26 +1970,92 @@ DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_80332_1, quirk /* * IO-APIC1 on 6300ESB generates boot interrupts, see Intel order no * 300641-004US, section 5.7.3. + * + * Core IO on Xeon E5 1600/2600/4600, see Intel order no 326509-003. + * Core IO on Xeon E5 v2, see Intel order no 329188-003. + * Core IO on Xeon E7 v2, see Intel order no 329595-002. + * Core IO on Xeon E5 v3, see Intel order no 330784-003. + * Core IO on Xeon E7 v3, see Intel order no 332315-001US. + * Core IO on Xeon E5 v4, see Intel order no 333810-002US. + * Core IO on Xeon E7 v4, see Intel order no 332315-001US. + * Core IO on Xeon D-1500, see Intel order no 332051-001. + * Core IO on Xeon Scalable, see Intel order no 610950. */ -#define INTEL_6300_IOAPIC_ABAR 0x40 +#define INTEL_6300_IOAPIC_ABAR 0x40 /* Bus 0, Dev 29, Func 5 */ #define INTEL_6300_DISABLE_BOOT_IRQ (1<<14) +#define INTEL_CIPINTRC_CFG_OFFSET 0x14C /* Bus 0, Dev 5, Func 0 */ +#define INTEL_CIPINTRC_DIS_INTX_ICH (1<<25) + static void quirk_disable_intel_boot_interrupt(struct pci_dev *dev) { u16 pci_config_word; + u32 pci_config_dword; if (noioapicquirk) return; - pci_read_config_word(dev, INTEL_6300_IOAPIC_ABAR, &pci_config_word); - pci_config_word |= INTEL_6300_DISABLE_BOOT_IRQ; - pci_write_config_word(dev, INTEL_6300_IOAPIC_ABAR, pci_config_word); - + switch (dev->device) { + case PCI_DEVICE_ID_INTEL_ESB_10: + pci_read_config_word(dev, INTEL_6300_IOAPIC_ABAR, + &pci_config_word); + pci_config_word |= INTEL_6300_DISABLE_BOOT_IRQ; + pci_write_config_word(dev, INTEL_6300_IOAPIC_ABAR, + pci_config_word); + break; + case 0x3c28: /* Xeon E5 1600/2600/4600 */ + case 0x0e28: /* Xeon E5/E7 V2 */ + case 0x2f28: /* Xeon E5/E7 V3,V4 */ + case 0x6f28: /* Xeon D-1500 */ + case 0x2034: /* Xeon Scalable Family */ + pci_read_config_dword(dev, INTEL_CIPINTRC_CFG_OFFSET, + &pci_config_dword); + pci_config_dword |= INTEL_CIPINTRC_DIS_INTX_ICH; + pci_write_config_dword(dev, INTEL_CIPINTRC_CFG_OFFSET, + pci_config_dword); + break; + default: + return; + } pci_info(dev, "disabled boot interrupts on device [%04x:%04x]\n", dev->vendor, dev->device); } -DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ESB_10, quirk_disable_intel_boot_interrupt); -DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ESB_10, quirk_disable_intel_boot_interrupt); +/* + * Device 29 Func 5 Device IDs of IOxAPIC + * containing ABAR—APIC1 Alternate Base Address Register + */ +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ESB_10, + quirk_disable_intel_boot_interrupt); +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ESB_10, + quirk_disable_intel_boot_interrupt); + +/* + * Device 5 Func 0 Device IDs of IIO or Core modules/hubs + * containing Coherent Interface Protocol Interrupt Control + * + * Device IDs obtained from volume 2 datasheets of commented + * families above. + */ +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x3c28, + quirk_disable_intel_boot_interrupt); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0e28, + quirk_disable_intel_boot_interrupt); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x2f28, + quirk_disable_intel_boot_interrupt); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x6f28, + quirk_disable_intel_boot_interrupt); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x2034, + quirk_disable_intel_boot_interrupt); +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL, 0x3c28, + quirk_disable_intel_boot_interrupt); +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL, 0x0e28, + quirk_disable_intel_boot_interrupt); +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL, 0x2f28, + quirk_disable_intel_boot_interrupt); +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL, 0x6f28, + quirk_disable_intel_boot_interrupt); +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL, 0x2034, + quirk_disable_intel_boot_interrupt); /* Disable boot interrupts on HT-1000 */ #define BC_HT1000_FEATURE_REG 0x64 From patchwork Fri Feb 14 21:33:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Sean V Kelley X-Patchwork-Id: 11383305 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 20EC2159A for ; Fri, 14 Feb 2020 21:33:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 00A29222C4 for ; Fri, 14 Feb 2020 21:33:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389316AbgBNVdZ (ORCPT ); Fri, 14 Feb 2020 16:33:25 -0500 Received: from mga14.intel.com ([192.55.52.115]:64984 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389305AbgBNVdZ (ORCPT ); Fri, 14 Feb 2020 16:33:25 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 14 Feb 2020 13:33:24 -0800 X-IronPort-AV: E=Sophos;i="5.70,441,1574150400"; d="scan'208";a="227742893" Received: from mravago-mobl.amr.corp.intel.com (HELO arch-ashland-svkelley.intel.com) ([10.252.135.120]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 14 Feb 2020 13:33:24 -0800 From: Sean V Kelley To: tglx@linutronix.de, bhelgaas@google.com, corbet@lwn.net, mingo@redhat.com, bp@alien8.de Cc: x86@kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kar.hin.ong@ni.com, sassmann@kpanic.de, Sean V Kelley Subject: [PATCH 2/2] Documentation:PCI: Add background on Boot Interrupts Date: Fri, 14 Feb 2020 13:33:13 -0800 Message-Id: <20200214213313.66622-3-sean.v.kelley@linux.intel.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200214213313.66622-1-sean.v.kelley@linux.intel.com> References: <20200214213313.66622-1-sean.v.kelley@linux.intel.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Improve understanding of the PCI quirks for this legacy PCI interrupt behavior to the benefit of developers and users alike. Signed-off-by: Sean V Kelley --- Documentation/PCI/boot-interrupts.rst | 153 ++++++++++++++++++++++++++ Documentation/PCI/index.rst | 1 + 2 files changed, 154 insertions(+) create mode 100644 Documentation/PCI/boot-interrupts.rst diff --git a/Documentation/PCI/boot-interrupts.rst b/Documentation/PCI/boot-interrupts.rst new file mode 100644 index 000000000000..632c080994fc --- /dev/null +++ b/Documentation/PCI/boot-interrupts.rst @@ -0,0 +1,153 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=============== +Boot Interrupts +=============== + +:Author: - Sean V Kelley + +Overview +======== + +On PCI Express, interrupts are represented with either MSI or inbound interrupt +messages (Assert_INTx/Deassert_INTx). The integrated IO-APIC in a given Core +IO converts the legacy interrupt messages from PCI Express to MSI interrupts. +If the IO-APIC is disabled (via the mask bits in the IO-APIC table entries), +the messages are routed to the legacy PCH. This in-band interrupt mechanism was +traditionally necessary for systems that did not support the IO-APIC and for +boot. Intel in the past has used the term "boot interrupts" to describe this +mechanism. Further, the PCI Express protocol describes this in-band legacy +wire-interrupt INTx mechanism for I/O devices to signal PCI-style level +interrupts. The subsequent paragraphs describe problems with the Core IO +handling of INTx message routing to the PCH and mitigation within BIOS and +the OS. + + +Problem +======= + +When in-band legacy INTx messages are forwarded to the PCH, they in turn +trigger a new interrupt for which the OS likely lacks a handler. When an +interrupt goes unhandled over time, they are tracked by the Linux kernel +as Spurious Interrupts. The IRQ will be disabled by the Linux kernel after +it reaches a specific count with the error "nobody cared". This disabled +IRQ now prevents valid usage by an existing interrupt which may happen to +share the IRQ line. + +irq 19: nobody cared (try booting with the "irqpoll" option) +CPU: 0 PID: 2988 Comm: irq/34-nipalk Tainted: 4.14.87-rt49-02410-g4a640ec-dirty #1 +Hardware name: National Instruments NI PXIe-8880/NI PXIe-8880, BIOS 2.1.5f1 01/09/2020 +Call Trace: + + ? dump_stack+0x46/0x5e + ? __report_bad_irq+0x2e/0xb0 + ? note_interrupt+0x242/0x290 + ? nNIKAL100_memoryRead16+0x8/0x10 [nikal] + ? handle_irq_event_percpu+0x55/0x70 + ? handle_irq_event+0x4f/0x80 + ? handle_fasteoi_irq+0x81/0x180 + ? handle_irq+0x1c/0x30 + ? do_IRQ+0x41/0xd0 + ? common_interrupt+0x84/0x84 + + +handlers: +irq_default_primary_handler threaded usb_hcd_irq +Disabling IRQ #19 + + +Conditions +========== + +The use of threaded interrupts is the most likely condition to trigger this +problem today. Threaded interrupts may not be reenabled after the IRQ handler +wakes. These "one shot" conditions mean that the threaded interrupt needs to +keep the interrupt line masked until the threaded handler has run. Especially +when dealing with high data rate interrupts, the thread needs to run to completion +otherwise some handlers will end up in stack overflows since the interrupt +of the issuing device is still active. + +Affected Chipsets +================= + +The legacy interrupt forwarding mechansim exists today in a number of devices +including but not limited to chipsets from AMD/ATI, Broadcom, and Intel. Changes +made through the mitigations below have been applied to drivers/pci/quirks.c + +Starting with ICX there are no longer any IO-APICs in the Core IO's devices. +IO-APIC is only in the PCH. Devices connected to the Core IO's PCIE Root Ports +will use native MSI/MSI-X mechanisms. + +Mitigations +=========== + +The mitigations take the form of PCI quirks. The preference has been to first +identify and make use of a means to disable the routing to the PCH. In such a +case a quirk to disable boot interrupt generation can be added.[1] + +Intel® 6300ESB I/O Controller Hub +Alternate Base Address Register: + BIE: Boot Interrupt Enable + 0 = Boot interrupt is enabled. + 1 = Boot interrupt is disabled. + +Intel® Sandy Bridge through Sky Lake based Xeon servers: +Coherent Interface Protocol Interrupt Control + dis_intx_route2pch/dis_intx_route2ich/dis_intx_route2dmi2: + When this bit is set. Local INTx messages received from the + Intel® Quick Data DMA/PCI Express ports are not routed to legacy + PCH - they are either converted into MSI via the integrated IO-APIC + (if the IO-APIC mask bit is clear in the appropriate entries) + or cause no further action (when mask bit is set) + +In the absence of a way to directly disable the routing, another approach +has been to make use of PCI Interrupt pin to INTx routing tables for purposes +of redirecting the interrupt handler to the rerouted interrupt line by default. +Therefore, on chipsets where this INTx routing cannot be disabled, the +Linux kernel will reroute the valid interrupt to its legacy interrupt. This +redirection of the handler will prevent the occurrence of the spurious +interrupt detection which would ordinarily disable the IRQ line due to +excessive unhandled counts.[2] + +The config option X86_REROUTE_FOR_BROKEN_BOOT_IRQS exists to enable +(or disable) the redirection of the interrupt handler to the PCH interrupt +line. The option can be overridden by either pci=ioapicreroute or +pci=noioapicreroute.[3] + + +More Documentation +================== + +There is an overview of the legacy interrupt handling mentioned in several +datasheets (6300ESB and 6700PXH below). While largely the same, it provides +insight into the evolution of its handling with chipsets. + +Example of disabling of the boot interrupt +------------------------------------------ + +Intel® 6300ESB I/O Controller Hub (Document # 300641-004US) + 2.15.2 PCI Express Legacy INTx Support and Boot Interrupt + https://www.intel.com/content/dam/doc/datasheet/6300esb-io-controller-hub-datasheet.pdf + +Intel® Xeon® Processor E5-1600/2400/2600/4600 v3 Product Families +Datasheet - Volume 2: Registers (Dcument # 330784-003) + 6.6.41 cipintrc Coherent Interface Protocol Interrupt Control + https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v3-datasheet-vol-2.pdf + +Example of handler rerouting +---------------------------- + +Intel® 6700PXH 64-bit PCI Hub (Document # 302628) + 2.15.2 PCI Express Legacy INTx Support and Boot Interrupt + https://www.intel.com/content/dam/doc/datasheet/6700pxh-64-bit-pci-hub-datasheet.pdf + + +If you have any legacy PCI interrupt questions that aren't answered, email me. + +Cheers, + Sean V Kelley + sean.v.kelley@linux.intel.com + +[1] https://lore.kernel.org/lkml/12131949181903-git-send-email-sassmann@suse.de/ +[2] https://lore.kernel.org/lkml/12131949182094-git-send-email-sassmann@suse.de/ +[3] https://lore.kernel.org/lkml/487C8EA7.6020205@suse.de/ diff --git a/Documentation/PCI/index.rst b/Documentation/PCI/index.rst index 6768305e4c26..8f66feaafd4f 100644 --- a/Documentation/PCI/index.rst +++ b/Documentation/PCI/index.rst @@ -16,3 +16,4 @@ Linux PCI Bus Subsystem pci-error-recovery pcieaer-howto endpoint/index + boot-interrupts