From patchwork Fri Mar 4 22:31:48 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Daney X-Patchwork-Id: 8508041 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 68B609F38C for ; Fri, 4 Mar 2016 22:34:32 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id D846720268 for ; Fri, 4 Mar 2016 22:34:30 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 688FC201FE for ; Fri, 4 Mar 2016 22:34:29 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1abyGx-0002rd-Q5; Fri, 04 Mar 2016 22:32:35 +0000 Received: from mail-pf0-x242.google.com ([2607:f8b0:400e:c00::242]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1abyGg-0002cS-Sl for linux-arm-kernel@lists.infradead.org; Fri, 04 Mar 2016 22:32:22 +0000 Received: by mail-pf0-x242.google.com with SMTP id 63so3827489pfe.0 for ; Fri, 04 Mar 2016 14:32:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=n9a9l1M6I4JHBnO+gCdNkTLZEoy+lzPoM4IYuZDVVxI=; b=f7n53xTJwqYgx+4BXYEQSLVJOI81iKmHz/2+SKC8A0rfJnOzumA4bqVhw9lOG2OT/i N3kubbH5vSW7rOKa19M7++HjGKGHG09oNt23QDswrPKJ+4Wn0j685akmFulLlkAspbZD QfTW9wCYjaD5wlIeNwp7jCPn6U5hiL4bSgjLVSzluSYZdbroKGCYA8YDp8ZQtV4GwjeY aCX4E9yPWhmUjAQRqVsWU8A30CmGIfLZDR1oza4OVY57suzw+cpb4DHr0QVxn+go+IlM 9f2Wy8S/HR5RkFF/rNyk6TQu7CVFsM9Z2oagRx8BSPsSsyd+BZAiPSsjshqP+X99nAQo BPyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=n9a9l1M6I4JHBnO+gCdNkTLZEoy+lzPoM4IYuZDVVxI=; b=VJ2br2mt5f5zqOirsxI1Wzld4Nh4axRzi9MjOL3J5nZdsxkwdUOybTUT3b9Q8N4j89 SFkNZMmQHGeJxaAA1HkD/DWeGX/T4WRuJk8Ux4wXUVU85DBpob2Jo76WXm3lgTtKY+G1 zoF+o0Ri6hf9kB3GA5GsgPqpZYMrhwzvhmGzh/21Doya1I70sE3eV7al82HgTrvfpfCY xD5ZspzeD5ZKNogMp2ekGM7doAa3tT6nWfCJmf5cLQ9WfM33cEgvSNZUbiQ2LQVULP+D gAT5F5q4+qHb3EszkoesVW80e4q62d6TLPOVK1tjf8ssTfESCqfZju3nhGcPBib9rBQc 9J3w== X-Gm-Message-State: AD7BkJJfPFOUCPCfA8H1VLNt51J1xMAJVte5uvy38coZj/f8JfX9p1dGkudOTGTxFRK4RA== X-Received: by 10.98.16.69 with SMTP id y66mr15719503pfi.86.1457130719786; Fri, 04 Mar 2016 14:31:59 -0800 (PST) Received: from dl.caveonetworks.com ([64.2.3.194]) by smtp.gmail.com with ESMTPSA id 72sm7732267pfb.64.2016.03.04.14.31.52 (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 04 Mar 2016 14:31:55 -0800 (PST) Received: from dl.caveonetworks.com (localhost.localdomain [127.0.0.1]) by dl.caveonetworks.com (8.14.5/8.14.5) with ESMTP id u24MVqDo003279; Fri, 4 Mar 2016 14:31:52 -0800 Received: (from ddaney@localhost) by dl.caveonetworks.com (8.14.5/8.14.5/Submit) id u24MVpm6003278; Fri, 4 Mar 2016 14:31:51 -0800 From: David Daney To: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, Bjorn Helgaas , Rob Herring , Pawel Moll , Mark Rutland , Ian Campbell , Kumar Gala , Will Deacon , linux-arm-kernel@lists.infradead.org, devicetree@vger.kernel.org Subject: [PATCH v7 3/3] pci, pci-thunder-ecam: Add driver for ThunderX-pass{1, 2} on-chip devices Date: Fri, 4 Mar 2016 14:31:48 -0800 Message-Id: <1457130708-3231-4-git-send-email-ddaney.cavm@gmail.com> X-Mailer: git-send-email 1.7.11.7 In-Reply-To: <1457130708-3231-1-git-send-email-ddaney.cavm@gmail.com> References: <1457130708-3231-1-git-send-email-ddaney.cavm@gmail.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20160304_143219_117490_D691420E X-CRM114-Status: GOOD ( 27.79 ) X-Spam-Score: -2.7 (--) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Dann Frazier , David Daney MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-4.1 required=5.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_MED, RP_MATCHES_RCVD, T_DKIM_INVALID, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: David Daney The cavium,pci-thunder-ecam devices are exactly ECAM based PCI root complexes. These root complexes (loosely referred to as ECAM units in the hardware manuals) are used to access the Thunder on-chips devices. They are special in that all the BARs on devices behind these root complexes are at fixed addresses. To handle this in a manner compatible with the core PCI code, we have the config access functions synthesize Enhanced Allocation (EA) capability entries for each BAR. Since this EA synthesis is needed for exactly two chip models, we can hard code some assumptions about the device topology and the layout of the config space of specific DEVFNs in the driver. Signed-off-by: David Daney Acked-by: Rob Herring --- .../devicetree/bindings/pci/pci-thunder-ecam.txt | 30 ++ drivers/pci/host/Kconfig | 7 + drivers/pci/host/Makefile | 1 + drivers/pci/host/pci-thunder-ecam.c | 407 +++++++++++++++++++++ 4 files changed, 445 insertions(+) create mode 100644 Documentation/devicetree/bindings/pci/pci-thunder-ecam.txt create mode 100644 drivers/pci/host/pci-thunder-ecam.c diff --git a/Documentation/devicetree/bindings/pci/pci-thunder-ecam.txt b/Documentation/devicetree/bindings/pci/pci-thunder-ecam.txt new file mode 100644 index 0000000..0202dd1 --- /dev/null +++ b/Documentation/devicetree/bindings/pci/pci-thunder-ecam.txt @@ -0,0 +1,30 @@ +* ThunderX PCI host controller for pass-1.x silicon + +Firmware-initialized PCI host controller to on-chip devices found on +some Cavium ThunderX processors. These devices have ECAM based config +access, but the BARs are all at fixed addresses. We handle the fixed +addresses by synthesizing Enhanced Allocation (EA) capabilities for +these devices. + +The properties and their meanings are identical to those described in +host-generic-pci.txt except as listed below. + +Properties of the host controller node that differ from +host-generic-pci.txt: + +- compatible : Must be "cavium,pci-host-thunder-ecam" + +Example: + + pcie@84b000000000 { + compatible = "cavium,pci-host-thunder-ecam"; + device_type = "pci"; + msi-parent = <&its>; + msi-map = <0 &its 0x30000 0x10000>; + bus-range = <0 31>; + #size-cells = <2>; + #address-cells = <3>; + #stream-id-cells = <1>; + reg = <0x84b0 0x00000000 0 0x02000000>; /* Configuration space */ + ranges = <0x03000000 0x8180 0x00000000 0x8180 0x00000000 0x80 0x00000000>; /* mem ranges */ + }; diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig index c642797..6a86ced 100644 --- a/drivers/pci/host/Kconfig +++ b/drivers/pci/host/Kconfig @@ -203,4 +203,11 @@ config PCI_HOST_THUNDER_PEM help Say Y here if you want PCIe support for CN88XX Cavium Thunder SoCs. +config PCI_HOST_THUNDER_ECAM + bool "Cavium Thunder ECAMe controller to on-chip devices on pass-1.x silicon" + depends on OF && ARM64 + select PCI_HOST_COMMON + help + Say Y here if you want ECAM support for CN88XX-Pass-1.x Cavium Thunder SoCs. + endmenu diff --git a/drivers/pci/host/Makefile b/drivers/pci/host/Makefile index 8903172..d6af3ba 100644 --- a/drivers/pci/host/Makefile +++ b/drivers/pci/host/Makefile @@ -23,4 +23,5 @@ obj-$(CONFIG_PCIE_ALTERA) += pcie-altera.o obj-$(CONFIG_PCIE_ALTERA_MSI) += pcie-altera-msi.o obj-$(CONFIG_PCI_HISI) += pcie-hisi.o obj-$(CONFIG_PCIE_QCOM) += pcie-qcom.o +obj-$(CONFIG_PCI_HOST_THUNDER_ECAM) += pci-thunder-ecam.o obj-$(CONFIG_PCI_HOST_THUNDER_PEM) += pci-thunder-pem.o diff --git a/drivers/pci/host/pci-thunder-ecam.c b/drivers/pci/host/pci-thunder-ecam.c new file mode 100644 index 0000000..723ce19 --- /dev/null +++ b/drivers/pci/host/pci-thunder-ecam.c @@ -0,0 +1,407 @@ +/* + * + * This file is subject to the terms and conditions of the GNU General Public + * License. See the file "COPYING" in the main directory of this archive + * for more details. + * + * Copyright (C) 2015, 2016 Cavium, Inc. + * + */ + +#include +#include +#include +#include +#include +#include + +#include "pci-host-common.h" + +/* Mapping is standard ECAM */ +static void __iomem *thunder_ecam_map_bus(struct pci_bus *bus, + unsigned int devfn, + int where) +{ + struct gen_pci *pci = bus->sysdata; + resource_size_t idx = bus->number - pci->cfg.bus_range->start; + + return pci->cfg.win[idx] + ((devfn << 12) | where); +} + +static void set_val(u32 v, int where, int size, u32 *val) +{ + int shift = (where & 3) * 8; + + pr_debug("set_val %04x: %08x\n", (unsigned)(where & ~3), v); + v >>= shift; + if (size == 1) + v &= 0xff; + else if (size == 2) + v &= 0xffff; + *val = v; +} + +static int handle_ea_bar(u32 e0, int bar, struct pci_bus *bus, + unsigned int devfn, int where, int size, u32 *val) +{ + void __iomem *addr; + u32 v; + /* + * Each entry is 16-byte aligned bits[2,3] select which word + * in the entry + */ + int where_a = where & 0xc; + + if (where_a == 0) { + set_val(e0, where, size, val); + return PCIBIOS_SUCCESSFUL; + } + if (where_a == 0x4) { + addr = bus->ops->map_bus(bus, devfn, bar); /* BAR 0 */ + if (!addr) { + *val = ~0; + return PCIBIOS_DEVICE_NOT_FOUND; + } + v = readl(addr); + v &= ~0xf; + v |= 2; /* EA entry-1. Base-L */ + set_val(v, where, size, val); + return PCIBIOS_SUCCESSFUL; + } + if (where_a == 0x8) { + u32 barl_orig; + u32 barl_rb; + + addr = bus->ops->map_bus(bus, devfn, bar); /* BAR 0 */ + if (!addr) { + *val = ~0; + return PCIBIOS_DEVICE_NOT_FOUND; + } + barl_orig = readl(addr + 0); + writel(0xffffffff, addr + 0); + barl_rb = readl(addr + 0); + writel(barl_orig, addr + 0); + /* zeros in unsettable bits. */ + v = ~barl_rb & ~3; + v |= 0xc; /* EA entry-2. Offset-L */ + set_val(v, where, size, val); + return PCIBIOS_SUCCESSFUL; + } + if (where_a == 0xc) { + addr = bus->ops->map_bus(bus, devfn, bar + 4); /* BAR 1 */ + if (!addr) { + *val = ~0; + return PCIBIOS_DEVICE_NOT_FOUND; + } + v = readl(addr); /* EA entry-3. Base-H */ + set_val(v, where, size, val); + return PCIBIOS_SUCCESSFUL; + } + return PCIBIOS_DEVICE_NOT_FOUND; +} + +static int thunder_ecam_p2_config_read(struct pci_bus *bus, unsigned int devfn, + int where, int size, u32 *val) +{ + struct gen_pci *pci = bus->sysdata; + int where_a = where & ~3; + void __iomem *addr; + u32 node_bits; + u32 v; + + /* EA Base[63:32] may be missing some bits ... */ + switch (where_a) { + case 0xa8: + case 0xbc: + case 0xd0: + case 0xe4: + break; + default: + return pci_generic_config_read(bus, devfn, where, size, val); + } + + addr = bus->ops->map_bus(bus, devfn, where_a); + if (!addr) { + *val = ~0; + return PCIBIOS_DEVICE_NOT_FOUND; + } + + v = readl(addr); + + /* + * Bit 44 of the 64-bit Base must match the same bit in + * the config space access window. Since we are working with + * the high order 32 bits, shift everything down by 32 bits. + */ + node_bits = (pci->cfg.res.start >> 32) & (1 << 12); + + v |= node_bits; + set_val(v, where, size, val); + + return PCIBIOS_SUCCESSFUL; +} + +static int thunder_ecam_config_read(struct pci_bus *bus, unsigned int devfn, + int where, int size, u32 *val) +{ + u32 v; + u32 vendor_device; + u32 class_rev; + void __iomem *addr; + int cfg_type; + int where_a = where & ~3; + + addr = bus->ops->map_bus(bus, devfn, 0xc); + if (!addr) { + *val = ~0; + return PCIBIOS_DEVICE_NOT_FOUND; + } + + v = readl(addr); + + /* Check for non type-00 header. */ + cfg_type = (v >> 16) & 0x7f; + + addr = bus->ops->map_bus(bus, devfn, 8); + if (!addr) { + *val = ~0; + return PCIBIOS_DEVICE_NOT_FOUND; + } + + class_rev = readl(addr); + if (class_rev == 0xffffffff) + goto no_emulation; + + if ((class_rev & 0xff) >= 8) { + /* Pass-2 handling */ + if (cfg_type) + goto no_emulation; + return thunder_ecam_p2_config_read(bus, devfn, where, + size, val); + } + + /* + * All BARs have fixed addresses specified by the EA + * capability, they must return zero on read. + */ + if (cfg_type == 0 && + ((where >= 0x10 && where < 0x2c) || + (where >= 0x1a4 && where < 0x1bc))) { + /* BAR or SRIOV BAR */ + *val = 0; + return PCIBIOS_SUCCESSFUL; + } + + addr = bus->ops->map_bus(bus, devfn, 0); + if (!addr) { + *val = ~0; + return PCIBIOS_DEVICE_NOT_FOUND; + } + + vendor_device = readl(addr); + if (vendor_device == 0xffffffff) + goto no_emulation; + + pr_debug("%04x:%04x - Fix pass#: %08x, where: %03x, devfn: %03x\n", + vendor_device & 0xffff, vendor_device >> 16, class_rev, + (unsigned) where, devfn); + + /* Check for non type-00 header. */ + if (cfg_type == 0) { + bool has_msix; + bool is_nic = (vendor_device == 0xa01e177d); + bool is_tns = (vendor_device == 0xa01f177d); + + addr = bus->ops->map_bus(bus, devfn, 0x70); + if (!addr) { + *val = ~0; + return PCIBIOS_DEVICE_NOT_FOUND; + } + /* E_CAP */ + v = readl(addr); + has_msix = (v & 0xff00) != 0; + + if (!has_msix && where_a == 0x70) { + v |= 0xbc00; /* next capability is EA at 0xbc */ + set_val(v, where, size, val); + return PCIBIOS_SUCCESSFUL; + } + if (where_a == 0xb0) { + addr = bus->ops->map_bus(bus, devfn, where_a); + if (!addr) { + *val = ~0; + return PCIBIOS_DEVICE_NOT_FOUND; + } + v = readl(addr); + if (v & 0xff00) + pr_err("Bad MSIX cap header: %08x\n", v); + v |= 0xbc00; /* next capability is EA at 0xbc */ + set_val(v, where, size, val); + return PCIBIOS_SUCCESSFUL; + } + if (where_a == 0xbc) { + if (is_nic) + v = 0x40014; /* EA last in chain, 4 entries. */ + else if (is_tns) + v = 0x30014; /* EA last in chain, 3 entries. */ + else if (has_msix) + v = 0x20014; /* EA last in chain, 2 entries. */ + else + v = 0x10014; /* EA last in chain, 1 entry. */ + set_val(v, where, size, val); + return PCIBIOS_SUCCESSFUL; + } + if (where_a >= 0xc0 && where_a < 0xd0) + /* EA entry-0. PP=0, BAR0 Size:3 */ + return handle_ea_bar(0x80ff0003, + 0x10, bus, devfn, where, + size, val); + if (where_a >= 0xd0 && where_a < 0xe0 && has_msix) + /* EA entry-1. PP=0, BAR4 Size:3 */ + return handle_ea_bar(0x80ff0043, + 0x20, bus, devfn, where, + size, val); + if (where_a >= 0xe0 && where_a < 0xf0 && is_tns) + /* EA entry-2. PP=0, BAR2, Size:3 */ + return handle_ea_bar(0x80ff0023, + 0x18, bus, devfn, where, + size, val); + if (where_a >= 0xe0 && where_a < 0xf0 && is_nic) + /* EA entry-2. PP=4, VF_BAR0 (9), Size:3 */ + return handle_ea_bar(0x80ff0493, + 0x1a4, bus, devfn, where, + size, val); + if (where_a >= 0xf0 && where_a < 0x100 && is_nic) + /* EA entry-3. PP=4, VF_BAR4 (d), Size:3 */ + return handle_ea_bar(0x80ff04d3, + 0x1b4, bus, devfn, where, + size, val); + } else if (cfg_type == 1) { + bool is_rsl_bridge = devfn == 0x08; + bool is_rad_bridge = devfn == 0xa0; + bool is_zip_bridge = devfn == 0xa8; + bool is_dfa_bridge = devfn == 0xb0; + bool is_nic_bridge = devfn == 0x10; + + if (where_a == 0x70) { + addr = bus->ops->map_bus(bus, devfn, where_a); + if (!addr) { + *val = ~0; + return PCIBIOS_DEVICE_NOT_FOUND; + } + v = readl(addr); + if (v & 0xff00) + pr_err("Bad PCIe cap header: %08x\n", v); + v |= 0xbc00; /* next capability is EA at 0xbc */ + set_val(v, where, size, val); + return PCIBIOS_SUCCESSFUL; + } + if (where_a == 0xbc) { + if (is_nic_bridge) + v = 0x10014; /* EA last in chain, 1 entry */ + else + v = 0x00014; /* EA last in chain, no entries */ + set_val(v, where, size, val); + return PCIBIOS_SUCCESSFUL; + } + if (where_a == 0xc0) { + if (is_rsl_bridge || is_nic_bridge) + v = 0x0101; /* subordinate:secondary = 1:1 */ + else if (is_rad_bridge) + v = 0x0202; /* subordinate:secondary = 2:2 */ + else if (is_zip_bridge) + v = 0x0303; /* subordinate:secondary = 3:3 */ + else if (is_dfa_bridge) + v = 0x0404; /* subordinate:secondary = 4:4 */ + set_val(v, where, size, val); + return PCIBIOS_SUCCESSFUL; + } + if (where_a == 0xc4 && is_nic_bridge) { + /* Enabled, not-Write, SP=ff, PP=05, BEI=6, ES=4 */ + v = 0x80ff0564; + set_val(v, where, size, val); + return PCIBIOS_SUCCESSFUL; + } + if (where_a == 0xc8 && is_nic_bridge) { + v = 0x00000002; /* Base-L 64-bit */ + set_val(v, where, size, val); + return PCIBIOS_SUCCESSFUL; + } + if (where_a == 0xcc && is_nic_bridge) { + v = 0xfffffffe; /* MaxOffset-L 64-bit */ + set_val(v, where, size, val); + return PCIBIOS_SUCCESSFUL; + } + if (where_a == 0xd0 && is_nic_bridge) { + v = 0x00008430; /* NIC Base-H */ + set_val(v, where, size, val); + return PCIBIOS_SUCCESSFUL; + } + if (where_a == 0xd4 && is_nic_bridge) { + v = 0x0000000f; /* MaxOffset-H */ + set_val(v, where, size, val); + return PCIBIOS_SUCCESSFUL; + } + } +no_emulation: + return pci_generic_config_read(bus, devfn, where, size, val); +} + +static int thunder_ecam_config_write(struct pci_bus *bus, unsigned int devfn, + int where, int size, u32 val) +{ + /* + * All BARs have fixed addresses, ignore BAR writes so they + * don't get corrupted. + */ + if ((where >= 0x10 && where < 0x2c) || + (where >= 0x1a4 && where < 0x1bc)) + /* BAR or SRIOV BAR */ + return PCIBIOS_SUCCESSFUL; + + return pci_generic_config_write(bus, devfn, where, size, val); +} + +static struct gen_pci_cfg_bus_ops thunder_ecam_bus_ops = { + .bus_shift = 20, + .ops = { + .map_bus = thunder_ecam_map_bus, + .read = thunder_ecam_config_read, + .write = thunder_ecam_config_write, + } +}; + +static const struct of_device_id thunder_ecam_of_match[] = { + { .compatible = "cavium,pci-host-thunder-ecam", + .data = &thunder_ecam_bus_ops }, + + { }, +}; +MODULE_DEVICE_TABLE(of, thunder_ecam_of_match); + +static int thunder_ecam_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + const struct of_device_id *of_id; + struct gen_pci *pci = devm_kzalloc(dev, sizeof(*pci), GFP_KERNEL); + + if (!pci) + return -ENOMEM; + + of_id = of_match_node(thunder_ecam_of_match, dev->of_node); + pci->cfg.ops = (struct gen_pci_cfg_bus_ops *)of_id->data; + + return pci_host_common_probe(pdev, pci); +} + +static struct platform_driver thunder_ecam_driver = { + .driver = { + .name = KBUILD_MODNAME, + .of_match_table = thunder_ecam_of_match, + }, + .probe = thunder_ecam_probe, +}; +module_platform_driver(thunder_ecam_driver); + +MODULE_DESCRIPTION("Thunder ECAM PCI host driver"); +MODULE_LICENSE("GPL v2");