From patchwork Wed Jun 9 16:01:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 12310645 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1AFE6C48BD1 for ; Wed, 9 Jun 2021 16:05:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0197D613B1 for ; Wed, 9 Jun 2021 16:05:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235580AbhFIQHs (ORCPT ); Wed, 9 Jun 2021 12:07:48 -0400 Received: from mga01.intel.com ([192.55.52.88]:42670 "EHLO mga01.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232900AbhFIQHp (ORCPT ); Wed, 9 Jun 2021 12:07:45 -0400 IronPort-SDR: O1bACc4YqDfFoHerifRq7P85ULT006XVq/JbHelFU3EudnZz1YVOPb0yqTGgxcCphbYTeikehM tdGZCwTRTeYg== X-IronPort-AV: E=McAfee;i="6200,9189,10010"; a="226473793" X-IronPort-AV: E=Sophos;i="5.83,261,1616482800"; d="scan'208";a="226473793" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2021 09:01:36 -0700 IronPort-SDR: C+O/KjOeq9bDFnS59XKBzEOBJugk6OagCNGLr4I808KCyPn9QKz766JLFCy+q9Zr7dnYllYdOF Of657T/Z25+w== X-IronPort-AV: E=Sophos;i="5.83,261,1616482800"; d="scan'208";a="449988641" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.25]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2021 09:01:35 -0700 Subject: [PATCH v6 1/5] cxl/acpi: Introduce the root of a cxl_port topology From: Dan Williams To: linux-cxl@vger.kernel.org Cc: Alison Schofield , "Rafael J. Wysocki" , Jonathan Cameron , linux-pci@vger.kernel.org, linux-acpi@vger.kernel.org Date: Wed, 09 Jun 2021 09:01:35 -0700 Message-ID: <162325449515.2293126.15303270193010154608.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <162325448982.2293126.16916114289970424561.stgit@dwillia2-desk3.amr.corp.intel.com> References: <162325448982.2293126.16916114289970424561.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.18-3-g996c MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org While CXL builds upon the PCI software model for enumeration and endpoint control, a static platform component is required to bootstrap the CXL memory layout. Similar to how ACPI identifies root-level PCI memory resources, ACPI data enumerates the address space and interleave configuration for CXL Memory. In addition to identifying host bridges, ACPI is responsible for enumerating the CXL memory space that can be addressed by downstream decoders. This is similar to the requirement for ACPI to publish resources via the _CRS method for PCI host bridges. Specifically, ACPI publishes a table, CXL Early Discovery Table (CEDT), which includes a list of CXL Memory resources, CXL Fixed Memory Window Structures (CFMWS). For now, introduce the core infrastructure for a cxl_port hierarchy starting with a root level anchor represented by the ACPI0017 device. Follow on changes model support for the configurable decode capabilities of cxl_port instances, i.e. CXL switch support. Co-developed-by: Alison Schofield Signed-off-by: Alison Schofield Acked-by: Rafael J. Wysocki Reviewed-by: Jonathan Cameron Signed-off-by: Dan Williams --- Documentation/ABI/testing/sysfs-bus-cxl | 20 +++ Documentation/driver-api/cxl/memory-devices.rst | 6 + drivers/cxl/Kconfig | 15 ++ drivers/cxl/Makefile | 2 drivers/cxl/acpi.c | 39 ++++++ drivers/cxl/core.c | 160 +++++++++++++++++++++++ drivers/cxl/cxl.h | 31 ++++ 7 files changed, 273 insertions(+) create mode 100644 drivers/cxl/acpi.c diff --git a/Documentation/ABI/testing/sysfs-bus-cxl b/Documentation/ABI/testing/sysfs-bus-cxl index 2fe7490ad6a8..bda2cc55cc38 100644 --- a/Documentation/ABI/testing/sysfs-bus-cxl +++ b/Documentation/ABI/testing/sysfs-bus-cxl @@ -24,3 +24,23 @@ Description: (RO) "Persistent Only Capacity" as bytes. Represents the identically named field in the Identify Memory Device Output Payload in the CXL-2.0 specification. + +What: /sys/bus/cxl/devices/*/devtype +Date: June, 2021 +KernelVersion: v5.14 +Contact: linux-cxl@vger.kernel.org +Description: + CXL device objects export the devtype attribute which mirrors + the same value communicated in the DEVTYPE environment variable + for uevents for devices on the "cxl" bus. + +What: /sys/bus/cxl/devices/portX/uport +Date: June, 2021 +KernelVersion: v5.14 +Contact: linux-cxl@vger.kernel.org +Description: + CXL port objects are enumerated from either a platform firmware + device (ACPI0017 and ACPI0016) or PCIe switch upstream port with + CXL component registers. The 'uport' symlink connects the CXL + portX object to the device that published the CXL port + capability. diff --git a/Documentation/driver-api/cxl/memory-devices.rst b/Documentation/driver-api/cxl/memory-devices.rst index 44c8ddbc8415..487ce4f41d77 100644 --- a/Documentation/driver-api/cxl/memory-devices.rst +++ b/Documentation/driver-api/cxl/memory-devices.rst @@ -30,6 +30,12 @@ CXL Memory Device CXL Core -------- +.. kernel-doc:: drivers/cxl/cxl.h + :doc: cxl objects + +.. kernel-doc:: drivers/cxl/cxl.h + :internal: + .. kernel-doc:: drivers/cxl/core.c :doc: cxl core diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig index 5483ba92b6da..d2573f6aef91 100644 --- a/drivers/cxl/Kconfig +++ b/drivers/cxl/Kconfig @@ -45,4 +45,19 @@ config CXL_MEM_RAW_COMMANDS potential impact to memory currently in use by the kernel. If developing CXL hardware or the driver say Y, otherwise say N. + +config CXL_ACPI + tristate "CXL ACPI: Platform Support" + depends on ACPI + help + Enable support for host managed device memory (HDM) resources + published by a platform's ACPI CXL memory layout description. See + Chapter 9.14.1 CXL Early Discovery Table (CEDT) in the CXL 2.0 + specification, and CXL Fixed Memory Window Structures (CEDT.CFMWS) + (https://www.computeexpresslink.org/spec-landing). The CXL core + consumes these resource to publish the root of a cxl_port decode + hierarchy to map regions that represent System RAM, or Persistent + Memory regions to be managed by LIBNVDIMM. + + If unsure say 'm'. endif diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile index d9d282dc15be..a29efb3e8ad2 100644 --- a/drivers/cxl/Makefile +++ b/drivers/cxl/Makefile @@ -1,7 +1,9 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_CXL_BUS) += cxl_core.o obj-$(CONFIG_CXL_MEM) += cxl_pci.o +obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL cxl_core-y := core.o cxl_pci-y := pci.o +cxl_acpi-y := acpi.o diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c new file mode 100644 index 000000000000..556d25ab6966 --- /dev/null +++ b/drivers/cxl/acpi.c @@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2021 Intel Corporation. All rights reserved. */ +#include +#include +#include +#include +#include +#include "cxl.h" + +static int cxl_acpi_probe(struct platform_device *pdev) +{ + struct cxl_port *root_port; + struct device *host = &pdev->dev; + + root_port = devm_cxl_add_port(host, host, CXL_RESOURCE_NONE, NULL); + if (IS_ERR(root_port)) + return PTR_ERR(root_port); + dev_dbg(host, "add: %s\n", dev_name(&root_port->dev)); + + return 0; +} + +static const struct acpi_device_id cxl_acpi_ids[] = { + { "ACPI0017", 0 }, + { "", 0 }, +}; +MODULE_DEVICE_TABLE(acpi, cxl_acpi_ids); + +static struct platform_driver cxl_acpi_driver = { + .probe = cxl_acpi_probe, + .driver = { + .name = KBUILD_MODNAME, + .acpi_match_table = cxl_acpi_ids, + }, +}; + +module_platform_driver(cxl_acpi_driver); +MODULE_LICENSE("GPL v2"); +MODULE_IMPORT_NS(CXL); diff --git a/drivers/cxl/core.c b/drivers/cxl/core.c index 853666d8a9f5..dbbb34618d7d 100644 --- a/drivers/cxl/core.c +++ b/drivers/cxl/core.c @@ -4,6 +4,8 @@ #include #include #include +#include +#include #include "cxl.h" /** @@ -13,6 +15,164 @@ * point for cross-device interleave coordination through cxl ports. */ +static DEFINE_IDA(cxl_port_ida); + +static ssize_t devtype_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%s\n", dev->type->name); +} +static DEVICE_ATTR_RO(devtype); + +static struct attribute *cxl_base_attributes[] = { + &dev_attr_devtype.attr, + NULL, +}; + +static struct attribute_group cxl_base_attribute_group = { + .attrs = cxl_base_attributes, +}; + +static void cxl_port_release(struct device *dev) +{ + struct cxl_port *port = to_cxl_port(dev); + + ida_free(&cxl_port_ida, port->id); + kfree(port); +} + +static const struct attribute_group *cxl_port_attribute_groups[] = { + &cxl_base_attribute_group, + NULL, +}; + +static const struct device_type cxl_port_type = { + .name = "cxl_port", + .release = cxl_port_release, + .groups = cxl_port_attribute_groups, +}; + +struct cxl_port *to_cxl_port(struct device *dev) +{ + if (dev_WARN_ONCE(dev, dev->type != &cxl_port_type, + "not a cxl_port device\n")) + return NULL; + return container_of(dev, struct cxl_port, dev); +} + +static void unregister_dev(void *dev) +{ + device_unregister(dev); +} + +static void cxl_unlink_uport(void *_port) +{ + struct cxl_port *port = _port; + + sysfs_remove_link(&port->dev.kobj, "uport"); +} + +static int devm_cxl_link_uport(struct device *host, struct cxl_port *port) +{ + int rc; + + rc = sysfs_create_link(&port->dev.kobj, &port->uport->kobj, "uport"); + if (rc) + return rc; + return devm_add_action_or_reset(host, cxl_unlink_uport, port); +} + +static struct cxl_port *cxl_port_alloc(struct device *uport, + resource_size_t component_reg_phys, + struct cxl_port *parent_port) +{ + struct cxl_port *port; + struct device *dev; + int rc; + + port = kzalloc(sizeof(*port), GFP_KERNEL); + if (!port) + return ERR_PTR(-ENOMEM); + + rc = ida_alloc(&cxl_port_ida, GFP_KERNEL); + if (rc < 0) + goto err; + port->id = rc; + + /* + * The top-level cxl_port "cxl_root" does not have a cxl_port as + * its parent and it does not have any corresponding component + * registers as its decode is described by a fixed platform + * description. + */ + dev = &port->dev; + if (parent_port) + dev->parent = &parent_port->dev; + else + dev->parent = uport; + + port->uport = uport; + port->component_reg_phys = component_reg_phys; + + device_initialize(dev); + device_set_pm_not_required(dev); + dev->bus = &cxl_bus_type; + dev->type = &cxl_port_type; + + return port; + +err: + kfree(port); + return ERR_PTR(rc); +} + +/** + * devm_cxl_add_port - register a cxl_port in CXL memory decode hierarchy + * @host: host device for devm operations + * @uport: "physical" device implementing this upstream port + * @component_reg_phys: (optional) for configurable cxl_port instances + * @parent_port: next hop up in the CXL memory decode hierarchy + */ +struct cxl_port *devm_cxl_add_port(struct device *host, struct device *uport, + resource_size_t component_reg_phys, + struct cxl_port *parent_port) +{ + struct cxl_port *port; + struct device *dev; + int rc; + + port = cxl_port_alloc(uport, component_reg_phys, parent_port); + if (IS_ERR(port)) + return port; + + dev = &port->dev; + if (parent_port) + rc = dev_set_name(dev, "port%d", port->id); + else + rc = dev_set_name(dev, "root%d", port->id); + if (rc) + goto err; + + rc = device_add(dev); + if (rc) + goto err; + + rc = devm_add_action_or_reset(host, unregister_dev, dev); + if (rc) + return ERR_PTR(rc); + + rc = devm_cxl_link_uport(host, port); + if (rc) + return ERR_PTR(rc); + + return port; + +err: + put_device(dev); + return ERR_PTR(rc); +} +EXPORT_SYMBOL_GPL(devm_cxl_add_port); + /** * cxl_probe_component_regs() - Detect CXL Component register blocks * @dev: Host device of the @base mapping diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 2c47e9cffd44..5651e5bb8274 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -8,6 +8,14 @@ #include #include +/** + * DOC: cxl objects + * + * The CXL core objects like ports, decoders, and regions are shared + * between the subsystem drivers cxl_acpi, cxl_pci, and core drivers + * (port-driver, region-driver, nvdimm object-drivers... etc). + */ + /* CXL 2.0 8.2.5 CXL.cache and CXL.mem Registers*/ #define CXL_CM_OFFSET 0x1000 #define CXL_CM_CAP_HDR_OFFSET 0x0 @@ -145,5 +153,28 @@ int cxl_map_device_regs(struct pci_dev *pdev, struct cxl_device_regs *regs, struct cxl_register_map *map); +#define CXL_RESOURCE_NONE ((resource_size_t) -1) + +/** + * struct cxl_port - logical collection of upstream port devices and + * downstream port devices to construct a CXL memory + * decode hierarchy. + * @dev: this port's device + * @uport: PCI or platform device implementing the upstream port capability + * @id: id for port device-name + * @component_reg_phys: component register capability base address (optional) + */ +struct cxl_port { + struct device dev; + struct device *uport; + int id; + resource_size_t component_reg_phys; +}; + +struct cxl_port *to_cxl_port(struct device *dev); +struct cxl_port *devm_cxl_add_port(struct device *host, struct device *uport, + resource_size_t component_reg_phys, + struct cxl_port *parent_port); + extern struct bus_type cxl_bus_type; #endif /* __CXL_H__ */ From patchwork Wed Jun 9 16:01:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 12310643 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7037CC48BD1 for ; Wed, 9 Jun 2021 16:05:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 58D5B613AD for ; Wed, 9 Jun 2021 16:05:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235409AbhFIQHg (ORCPT ); Wed, 9 Jun 2021 12:07:36 -0400 Received: from mga05.intel.com ([192.55.52.43]:18404 "EHLO mga05.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235109AbhFIQHf (ORCPT ); Wed, 9 Jun 2021 12:07:35 -0400 IronPort-SDR: HHjecPU5e2xlZXV7rKpZXo846vYTXZVakE0ORAQOUvetv/M+I5GCbN7KvbgbXZymcQWp38co8v lbH3ftX3OAYA== X-IronPort-AV: E=McAfee;i="6200,9189,10010"; a="290731710" X-IronPort-AV: E=Sophos;i="5.83,261,1616482800"; d="scan'208";a="290731710" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2021 09:01:42 -0700 IronPort-SDR: cuVg8ArXz3w4PVLaXf0ovWN2OTszrvNd2bK+r321zkFrId2Ycdi7zpqcZtwipbcKS7EJTNiXm+ xdGv0AZvFYyw== X-IronPort-AV: E=Sophos;i="5.83,261,1616482800"; d="scan'208";a="402486569" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.25]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2021 09:01:41 -0700 Subject: [PATCH v6 2/5] cxl/Kconfig: Default drivers to CONFIG_CXL_BUS From: Dan Williams To: linux-cxl@vger.kernel.org Cc: Ben Widawsky , Jonathan Cameron , linux-pci@vger.kernel.org, linux-acpi@vger.kernel.org Date: Wed, 09 Jun 2021 09:01:41 -0700 Message-ID: <162325450105.2293126.17046356425194082921.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <162325448982.2293126.16916114289970424561.stgit@dwillia2-desk3.amr.corp.intel.com> References: <162325448982.2293126.16916114289970424561.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.18-3-g996c MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org CONFIG_CXL_BUS is default 'n' as expected for new functionality. When that is enabled do not make the end user hunt for all the expected sub-options to enable. For example CONFIG_CXL_BUS without CONFIG_CXL_MEM is an odd/expert configuration, so is CONFIG_CXL_MEM without CONFIG_CXL_ACPI (on ACPI capable platforms). Default CONFIG_CXL_MEM and CONFIG_CXL_ACPI to CONFIG_CXL_BUS. Acked-by: Ben Widawsky Acked-by: Jonathan Cameron Signed-off-by: Dan Williams --- drivers/cxl/Kconfig | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig index d2573f6aef91..1a44b173dcbc 100644 --- a/drivers/cxl/Kconfig +++ b/drivers/cxl/Kconfig @@ -15,6 +15,7 @@ if CXL_BUS config CXL_MEM tristate "CXL.mem: Memory Devices" + default CXL_BUS help The CXL.mem protocol allows a device to act as a provider of "System RAM" and/or "Persistent Memory" that is fully coherent @@ -49,6 +50,7 @@ config CXL_MEM_RAW_COMMANDS config CXL_ACPI tristate "CXL ACPI: Platform Support" depends on ACPI + default CXL_BUS help Enable support for host managed device memory (HDM) resources published by a platform's ACPI CXL memory layout description. See From patchwork Wed Jun 9 16:01:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 12310647 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D15AEC48BDF for ; Wed, 9 Jun 2021 16:06:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BD31C613AD for ; Wed, 9 Jun 2021 16:06:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235369AbhFIQIB (ORCPT ); Wed, 9 Jun 2021 12:08:01 -0400 Received: from mga05.intel.com ([192.55.52.43]:18450 "EHLO mga05.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232900AbhFIQHv (ORCPT ); Wed, 9 Jun 2021 12:07:51 -0400 IronPort-SDR: tkVT4+FKfqZp39ix83z+GJAcc2u2lbhIZe6sCeFbVrRYYj4u77HeKfprZQ2COFomyU9N/6b3/V aFaGUSVXxhYA== X-IronPort-AV: E=McAfee;i="6200,9189,10010"; a="290731736" X-IronPort-AV: E=Sophos;i="5.83,261,1616482800"; d="scan'208";a="290731736" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2021 09:01:47 -0700 IronPort-SDR: c1EKMUj2FWnCYyWblDkreV2pXU1mdr1EgFXLMC1m7j5TbzEaqW2frvjcciwK6bt9jjH2/e4Ugb 5C/oVpdsV4nQ== X-IronPort-AV: E=Sophos;i="5.83,261,1616482800"; d="scan'208";a="552023207" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.25]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2021 09:01:46 -0700 Subject: [PATCH v6 3/5] cxl/acpi: Add downstream port data to cxl_port instances From: Dan Williams To: linux-cxl@vger.kernel.org Cc: linux-pci@vger.kernel.org, linux-acpi@vger.kernel.org Date: Wed, 09 Jun 2021 09:01:46 -0700 Message-ID: <162325450624.2293126.3533006409920271718.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <162325448982.2293126.16916114289970424561.stgit@dwillia2-desk3.amr.corp.intel.com> References: <162325448982.2293126.16916114289970424561.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.18-3-g996c MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org In preparation for infrastructure that enumerates and configures the CXL decode mechanism of an upstream port to its downstream ports, add a representation of a CXL downstream port. On ACPI systems the top-most logical downstream ports in the hierarchy are the host bridges (ACPI0016 devices) that decode the memory windows described by the CXL Early Discovery Table Fixed Memory Window Structures (CEDT.CFMWS). Signed-off-by: Dan Williams Reviewed-by: Alison Schofield Reviewed-by: Jonathan Cameron --- Documentation/ABI/testing/sysfs-bus-cxl | 13 ++++ drivers/cxl/acpi.c | 43 ++++++++++++ drivers/cxl/core.c | 107 ++++++++++++++++++++++++++++++- drivers/cxl/cxl.h | 21 ++++++ 4 files changed, 180 insertions(+), 4 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-bus-cxl b/Documentation/ABI/testing/sysfs-bus-cxl index bda2cc55cc38..f680da85fd44 100644 --- a/Documentation/ABI/testing/sysfs-bus-cxl +++ b/Documentation/ABI/testing/sysfs-bus-cxl @@ -44,3 +44,16 @@ Description: CXL component registers. The 'uport' symlink connects the CXL portX object to the device that published the CXL port capability. + +What: /sys/bus/cxl/devices/portX/dportY +Date: June, 2021 +KernelVersion: v5.14 +Contact: linux-cxl@vger.kernel.org +Description: + CXL port objects are enumerated from either a platform firmware + device (ACPI0017 and ACPI0016) or PCIe switch upstream port with + CXL component registers. The 'dportY' symlink identifies one or + more downstream ports that the upstream port may target in its + decode of CXL memory resources. The 'Y' integer reflects the + hardware port unique-id used in the hardware decoder target + list. diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c index 556d25ab6966..5eb9543c587a 100644 --- a/drivers/cxl/acpi.c +++ b/drivers/cxl/acpi.c @@ -7,17 +7,58 @@ #include #include "cxl.h" +static struct acpi_device *to_cxl_host_bridge(struct device *dev) +{ + struct acpi_device *adev = to_acpi_device(dev); + + if (strcmp(acpi_device_hid(adev), "ACPI0016") == 0) + return adev; + return NULL; +} + +static int add_host_bridge_dport(struct device *match, void *arg) +{ + int rc; + acpi_status status; + unsigned long long uid; + struct cxl_port *root_port = arg; + struct device *host = root_port->dev.parent; + struct acpi_device *bridge = to_cxl_host_bridge(match); + + if (!bridge) + return 0; + + status = acpi_evaluate_integer(bridge->handle, METHOD_NAME__UID, NULL, + &uid); + if (status != AE_OK) { + dev_err(host, "unable to retrieve _UID of %s\n", + dev_name(match)); + return -ENODEV; + } + + rc = cxl_add_dport(root_port, match, uid, CXL_RESOURCE_NONE); + if (rc) { + dev_err(host, "failed to add downstream port: %s\n", + dev_name(match)); + return rc; + } + dev_dbg(host, "add dport%llu: %s\n", uid, dev_name(match)); + return 0; +} + static int cxl_acpi_probe(struct platform_device *pdev) { struct cxl_port *root_port; struct device *host = &pdev->dev; + struct acpi_device *adev = ACPI_COMPANION(host); root_port = devm_cxl_add_port(host, host, CXL_RESOURCE_NONE, NULL); if (IS_ERR(root_port)) return PTR_ERR(root_port); dev_dbg(host, "add: %s\n", dev_name(&root_port->dev)); - return 0; + return bus_for_each_dev(adev->dev.bus, NULL, root_port, + add_host_bridge_dport); } static const struct acpi_device_id cxl_acpi_ids[] = { diff --git a/drivers/cxl/core.c b/drivers/cxl/core.c index dbbb34618d7d..8a3f3804f252 100644 --- a/drivers/cxl/core.c +++ b/drivers/cxl/core.c @@ -33,10 +33,22 @@ static struct attribute_group cxl_base_attribute_group = { .attrs = cxl_base_attributes, }; +static void cxl_dport_release(struct cxl_dport *dport) +{ + list_del(&dport->list); + put_device(dport->dport); + kfree(dport); +} + static void cxl_port_release(struct device *dev) { struct cxl_port *port = to_cxl_port(dev); + struct cxl_dport *dport, *_d; + device_lock(dev); + list_for_each_entry_safe(dport, _d, &port->dports, list) + cxl_dport_release(dport); + device_unlock(dev); ida_free(&cxl_port_ida, port->id); kfree(port); } @@ -60,9 +72,22 @@ struct cxl_port *to_cxl_port(struct device *dev) return container_of(dev, struct cxl_port, dev); } -static void unregister_dev(void *dev) +static void unregister_port(void *_port) { - device_unregister(dev); + struct cxl_port *port = _port; + struct cxl_dport *dport; + + device_lock(&port->dev); + list_for_each_entry(dport, &port->dports, list) { + char link_name[CXL_TARGET_STRLEN]; + + if (snprintf(link_name, CXL_TARGET_STRLEN, "dport%d", + dport->port_id) >= CXL_TARGET_STRLEN) + continue; + sysfs_remove_link(&port->dev.kobj, link_name); + } + device_unlock(&port->dev); + device_unregister(&port->dev); } static void cxl_unlink_uport(void *_port) @@ -113,6 +138,7 @@ static struct cxl_port *cxl_port_alloc(struct device *uport, port->uport = uport; port->component_reg_phys = component_reg_phys; + INIT_LIST_HEAD(&port->dports); device_initialize(dev); device_set_pm_not_required(dev); @@ -157,7 +183,7 @@ struct cxl_port *devm_cxl_add_port(struct device *host, struct device *uport, if (rc) goto err; - rc = devm_add_action_or_reset(host, unregister_dev, dev); + rc = devm_add_action_or_reset(host, unregister_port, port); if (rc) return ERR_PTR(rc); @@ -173,6 +199,81 @@ struct cxl_port *devm_cxl_add_port(struct device *host, struct device *uport, } EXPORT_SYMBOL_GPL(devm_cxl_add_port); +static struct cxl_dport *find_dport(struct cxl_port *port, int id) +{ + struct cxl_dport *dport; + + device_lock_assert(&port->dev); + list_for_each_entry (dport, &port->dports, list) + if (dport->port_id == id) + return dport; + return NULL; +} + +static int add_dport(struct cxl_port *port, struct cxl_dport *new) +{ + struct cxl_dport *dup; + + device_lock(&port->dev); + dup = find_dport(port, new->port_id); + if (dup) + dev_err(&port->dev, + "unable to add dport%d-%s non-unique port id (%s)\n", + new->port_id, dev_name(new->dport), + dev_name(dup->dport)); + else + list_add_tail(&new->list, &port->dports); + device_unlock(&port->dev); + + return dup ? -EEXIST : 0; +} + +/** + * cxl_add_dport - append downstream port data to a cxl_port + * @port: the cxl_port that references this dport + * @dport_dev: firmware or PCI device representing the dport + * @port_id: identifier for this dport in a decoder's target list + * @component_reg_phys: optional location of CXL component registers + * + * Note that all allocations and links are undone by cxl_port deletion + * and release. + */ +int cxl_add_dport(struct cxl_port *port, struct device *dport_dev, int port_id, + resource_size_t component_reg_phys) +{ + char link_name[CXL_TARGET_STRLEN]; + struct cxl_dport *dport; + int rc; + + if (snprintf(link_name, CXL_TARGET_STRLEN, "dport%d", port_id) >= + CXL_TARGET_STRLEN) + return -EINVAL; + + dport = kzalloc(sizeof(*dport), GFP_KERNEL); + if (!dport) + return -ENOMEM; + + INIT_LIST_HEAD(&dport->list); + dport->dport = get_device(dport_dev); + dport->port_id = port_id; + dport->component_reg_phys = component_reg_phys; + dport->port = port; + + rc = add_dport(port, dport); + if (rc) + goto err; + + rc = sysfs_create_link(&port->dev.kobj, &dport_dev->kobj, link_name); + if (rc) + goto err; + + return 0; +err: + cxl_dport_release(dport); + return rc; +} +EXPORT_SYMBOL_GPL(cxl_add_dport); + /** * cxl_probe_component_regs() - Detect CXL Component register blocks * @dev: Host device of the @base mapping diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 5651e5bb8274..dd159fd6d692 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -154,6 +154,7 @@ int cxl_map_device_regs(struct pci_dev *pdev, struct cxl_register_map *map); #define CXL_RESOURCE_NONE ((resource_size_t) -1) +#define CXL_TARGET_STRLEN 20 /** * struct cxl_port - logical collection of upstream port devices and @@ -162,19 +163,39 @@ int cxl_map_device_regs(struct pci_dev *pdev, * @dev: this port's device * @uport: PCI or platform device implementing the upstream port capability * @id: id for port device-name + * @dports: cxl_dport instances referenced by decoders * @component_reg_phys: component register capability base address (optional) */ struct cxl_port { struct device dev; struct device *uport; int id; + struct list_head dports; resource_size_t component_reg_phys; }; +/** + * struct cxl_dport - CXL downstream port + * @dport: PCI bridge or firmware device representing the downstream link + * @port_id: unique hardware identifier for dport in decoder target list + * @component_reg_phys: downstream port component registers + * @port: reference to cxl_port that contains this downstream port + * @list: node for a cxl_port's list of cxl_dport instances + */ +struct cxl_dport { + struct device *dport; + int port_id; + resource_size_t component_reg_phys; + struct cxl_port *port; + struct list_head list; +}; + struct cxl_port *to_cxl_port(struct device *dev); struct cxl_port *devm_cxl_add_port(struct device *host, struct device *uport, resource_size_t component_reg_phys, struct cxl_port *parent_port); +int cxl_add_dport(struct cxl_port *port, struct device *dport, int port_id, + resource_size_t component_reg_phys); extern struct bus_type cxl_bus_type; #endif /* __CXL_H__ */ From patchwork Wed Jun 9 16:01:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 12310649 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D3FAC48BD1 for ; Wed, 9 Jun 2021 16:06:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 658776136D for ; Wed, 9 Jun 2021 16:06:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235298AbhFIQIQ (ORCPT ); Wed, 9 Jun 2021 12:08:16 -0400 Received: from mga11.intel.com ([192.55.52.93]:32597 "EHLO mga11.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232900AbhFIQIP (ORCPT ); Wed, 9 Jun 2021 12:08:15 -0400 IronPort-SDR: JcqdW0pZvsCJ6pIwFKpK1tY/5SBEHI+dfH1IwiO9NNLBCtpMGgLJyvqHz0BHv4CBpZvzegURzq edFliC+uvsgg== X-IronPort-AV: E=McAfee;i="6200,9189,10010"; a="202077381" X-IronPort-AV: E=Sophos;i="5.83,261,1616482800"; d="scan'208";a="202077381" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2021 09:01:52 -0700 IronPort-SDR: ZB2hQTWdy7tmPjGitAyO9AXL+e7XgsGbeLyh5Lg663WEvELMuKdYn6i3SH1XkrNFAuTLNhhlhB 8olLOBQKXQAA== X-IronPort-AV: E=Sophos;i="5.83,261,1616482800"; d="scan'208";a="476999736" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.25]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2021 09:01:51 -0700 Subject: [PATCH v6 4/5] cxl/acpi: Enumerate host bridge root ports From: Dan Williams To: linux-cxl@vger.kernel.org Cc: Jonathan Cameron , linux-pci@vger.kernel.org, linux-acpi@vger.kernel.org Date: Wed, 09 Jun 2021 09:01:51 -0700 Message-ID: <162325451145.2293126.10149150938788969381.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <162325448982.2293126.16916114289970424561.stgit@dwillia2-desk3.amr.corp.intel.com> References: <162325448982.2293126.16916114289970424561.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.18-3-g996c MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org While the resources enumerated by the CEDT.CFMWS identify a cxl_port with host bridges as downstream ports, host bridges themselves are upstream ports that decode to downstream ports represented by PCIe Root Ports. Walk the PCIe Root Ports connected to a CXL Host Bridge, identified by the ACPI0016 _HID, and add each one as a cxl_dport of the host bridge cxl_port. For now, component registers are not enumerated, only the first order uport / dport relationships. Reviewed-by: Jonathan Cameron Signed-off-by: Dan Williams --- drivers/cxl/acpi.c | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 92 insertions(+), 1 deletion(-) diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c index 5eb9543c587a..1f075dffc042 100644 --- a/drivers/cxl/acpi.c +++ b/drivers/cxl/acpi.c @@ -5,8 +5,51 @@ #include #include #include +#include #include "cxl.h" +struct cxl_walk_context { + struct device *dev; + struct pci_bus *root; + struct cxl_port *port; + int error; + int count; +}; + +static int match_add_root_ports(struct pci_dev *pdev, void *data) +{ + struct cxl_walk_context *ctx = data; + struct pci_bus *root_bus = ctx->root; + struct cxl_port *port = ctx->port; + int type = pci_pcie_type(pdev); + struct device *dev = ctx->dev; + u32 lnkcap, port_num; + int rc; + + if (pdev->bus != root_bus) + return 0; + if (!pci_is_pcie(pdev)) + return 0; + if (type != PCI_EXP_TYPE_ROOT_PORT) + return 0; + if (pci_read_config_dword(pdev, pci_pcie_cap(pdev) + PCI_EXP_LNKCAP, + &lnkcap) != PCIBIOS_SUCCESSFUL) + return 0; + + /* TODO walk DVSEC to find component register base */ + port_num = FIELD_GET(PCI_EXP_LNKCAP_PN, lnkcap); + rc = cxl_add_dport(port, &pdev->dev, port_num, CXL_RESOURCE_NONE); + if (rc) { + ctx->error = rc; + return rc; + } + ctx->count++; + + dev_dbg(dev, "add dport%d: %s\n", port_num, dev_name(&pdev->dev)); + + return 0; +} + static struct acpi_device *to_cxl_host_bridge(struct device *dev) { struct acpi_device *adev = to_acpi_device(dev); @@ -16,6 +59,44 @@ static struct acpi_device *to_cxl_host_bridge(struct device *dev) return NULL; } +/* + * A host bridge is a dport to a CFMWS decode and it is a uport to the + * dport (PCIe Root Ports) in the host bridge. + */ +static int add_host_bridge_uport(struct device *match, void *arg) +{ + struct acpi_device *bridge = to_cxl_host_bridge(match); + struct cxl_port *root_port = arg; + struct device *host = root_port->dev.parent; + struct acpi_pci_root *pci_root; + struct cxl_walk_context ctx; + struct cxl_port *port; + + if (!bridge) + return 0; + + pci_root = acpi_pci_find_root(bridge->handle); + if (!pci_root) + return -ENXIO; + + /* TODO: fold in CEDT.CHBS retrieval */ + port = devm_cxl_add_port(host, match, CXL_RESOURCE_NONE, root_port); + if (IS_ERR(port)) + return PTR_ERR(port); + dev_dbg(host, "%s: add: %s\n", dev_name(match), dev_name(&port->dev)); + + ctx = (struct cxl_walk_context){ + .dev = host, + .root = pci_root->bus, + .port = port, + }; + pci_walk_bus(pci_root->bus, match_add_root_ports, &ctx); + + if (ctx.count == 0) + return -ENODEV; + return ctx.error; +} + static int add_host_bridge_dport(struct device *match, void *arg) { int rc; @@ -48,6 +129,7 @@ static int add_host_bridge_dport(struct device *match, void *arg) static int cxl_acpi_probe(struct platform_device *pdev) { + int rc; struct cxl_port *root_port; struct device *host = &pdev->dev; struct acpi_device *adev = ACPI_COMPANION(host); @@ -57,8 +139,17 @@ static int cxl_acpi_probe(struct platform_device *pdev) return PTR_ERR(root_port); dev_dbg(host, "add: %s\n", dev_name(&root_port->dev)); + rc = bus_for_each_dev(adev->dev.bus, NULL, root_port, + add_host_bridge_dport); + if (rc) + return rc; + + /* + * Root level scanned with host-bridge as dports, now scan host-bridges + * for their role as CXL uports to their CXL-capable PCIe Root Ports. + */ return bus_for_each_dev(adev->dev.bus, NULL, root_port, - add_host_bridge_dport); + add_host_bridge_uport); } static const struct acpi_device_id cxl_acpi_ids[] = { From patchwork Wed Jun 9 16:01:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 12310651 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 056F4C48BE0 for ; Wed, 9 Jun 2021 16:06:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DAE46613AD for ; Wed, 9 Jun 2021 16:06:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232900AbhFIQIi (ORCPT ); Wed, 9 Jun 2021 12:08:38 -0400 Received: from mga18.intel.com ([134.134.136.126]:57658 "EHLO mga18.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235013AbhFIQIh (ORCPT ); Wed, 9 Jun 2021 12:08:37 -0400 IronPort-SDR: DyEs3Wq3vUAl7l0FAXYB7/WeDjBq3keFUytLe+SP69mqRC7ruh+gIYHuY2EVgF97Wu+BEaTMbO zHfSkaH/TulA== X-IronPort-AV: E=McAfee;i="6200,9189,10010"; a="192417238" X-IronPort-AV: E=Sophos;i="5.83,261,1616482800"; d="scan'208";a="192417238" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2021 09:02:00 -0700 IronPort-SDR: /er0kuxVYCyZZdeZBbz8eLDjMwaRzJoO5sfwXHEgDf6pt0m6GeAsGL32EZ0BMHPWgzVQzZLC3B Rm3Ga5I/i97g== X-IronPort-AV: E=Sophos;i="5.83,261,1616482800"; d="scan'208";a="469914303" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.25]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2021 09:01:58 -0700 Subject: [PATCH v6 5/5] cxl/acpi: Introduce cxl_decoder objects From: Dan Williams To: linux-cxl@vger.kernel.org Cc: Jonathan Cameron , linux-pci@vger.kernel.org, linux-acpi@vger.kernel.org Date: Wed, 09 Jun 2021 09:01:57 -0700 Message-ID: <162325451704.2293126.8710272179264082084.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <162325448982.2293126.16916114289970424561.stgit@dwillia2-desk3.amr.corp.intel.com> References: <162325448982.2293126.16916114289970424561.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.18-3-g996c MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org A cxl_decoder is a child of a cxl_port. It represents a hardware decoder configuration of an upstream port to one or more of its downstream ports. The decoder is either represented in CXL standard HDM decoder registers (see CXL 2.0 section 8.2.5.12 CXL HDM Decoder Capability Structure), or it is a static decode configuration communicated by platform firmware (see the CXL Early Discovery Table: Fixed Memory Window Structure). The firmware described and hardware described decoders differ slightly leading to 2 different sub-types of decoders, cxl_decoder_root and cxl_decoder_switch. At the root level the decode capabilities restrict what can be mapped beneath them. Mid-level switch decoders are configured for either acclerator (type-2) or memory-expander (type-3) operation, but they are otherwise agnostic to the type of memory (volatile vs persistent) being mapped. Here is an example topology from a single-ported host-bridge environment without CFMWS decodes enumerated. /sys/bus/cxl/devices/root0 ├── devtype ├── dport0 -> ../LNXSYSTM:00/LNXSYBUS:00/ACPI0016:00 ├── port1 │   ├── decoder1.0 │   │   ├── devtype │   │   ├── end │   │   ├── locked │   │   ├── start │   │   ├── subsystem -> ../../../../bus/cxl │   │   ├── target_list │   │   ├── target_type │   │   └── uevent │   ├── devtype │   ├── dport0 -> ../../pci0000:34/0000:34:00.0 │   ├── subsystem -> ../../../bus/cxl │   ├── uevent │   └── uport -> ../../LNXSYSTM:00/LNXSYBUS:00/ACPI0016:00 ├── subsystem -> ../../bus/cxl ├── uevent └── uport -> ../platform/ACPI0017:00 Reviewed-by: Jonathan Cameron Signed-off-by: Dan Williams --- Documentation/ABI/testing/sysfs-bus-cxl | 70 ++++++++ drivers/cxl/acpi.c | 20 ++ drivers/cxl/core.c | 265 +++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 63 +++++++ 4 files changed, 417 insertions(+), 1 deletion(-) diff --git a/Documentation/ABI/testing/sysfs-bus-cxl b/Documentation/ABI/testing/sysfs-bus-cxl index f680da85fd44..0b6a2e6e8fbb 100644 --- a/Documentation/ABI/testing/sysfs-bus-cxl +++ b/Documentation/ABI/testing/sysfs-bus-cxl @@ -57,3 +57,73 @@ Description: decode of CXL memory resources. The 'Y' integer reflects the hardware port unique-id used in the hardware decoder target list. + +What: /sys/bus/cxl/devices/decoderX.Y +Date: June, 2021 +KernelVersion: v5.14 +Contact: linux-cxl@vger.kernel.org +Description: + CXL decoder objects are enumerated from either a platform + firmware description, or a CXL HDM decoder register set in a + PCIe device (see CXL 2.0 section 8.2.5.12 CXL HDM Decoder + Capability Structure). The 'X' in decoderX.Y represents the + cxl_port container of this decoder, and 'Y' represents the + instance id of a given decoder resource. + +What: /sys/bus/cxl/devices/decoderX.Y/{start,size} +Date: June, 2021 +KernelVersion: v5.14 +Contact: linux-cxl@vger.kernel.org +Description: + The 'start' and 'size' attributes together convey the physical + address base and number of bytes mapped in the decoder's decode + window. For decoders of devtype "cxl_decoder_root" the address + range is fixed. For decoders of devtype "cxl_decoder_switch" the + address is bounded by the decode range of the cxl_port ancestor + of the decoder's cxl_port, and dynamically updates based on the + active memory regions in that address space. + +What: /sys/bus/cxl/devices/decoderX.Y/locked +Date: June, 2021 +KernelVersion: v5.14 +Contact: linux-cxl@vger.kernel.org +Description: + CXL HDM decoders have the capability to lock the configuration + until the next device reset. For decoders of devtype + "cxl_decoder_root" there is no standard facility to unlock them. + For decoders of devtype "cxl_decoder_switch" a secondary bus + reset, of the PCIe bridge that provides the bus for this + decoders uport, unlocks / resets the decoder. + +What: /sys/bus/cxl/devices/decoderX.Y/target_list +Date: June, 2021 +KernelVersion: v5.14 +Contact: linux-cxl@vger.kernel.org +Description: + Display a comma separated list of the current decoder target + configuration. The list is ordered by the current configured + interleave order of the decoder's dport instances. Each entry in + the list is a dport id. + +What: /sys/bus/cxl/devices/decoderX.Y/cap_{pmem,ram,type2,type3} +Date: June, 2021 +KernelVersion: v5.14 +Contact: linux-cxl@vger.kernel.org +Description: + When a CXL decoder is of devtype "cxl_decoder_root", it + represents a fixed memory window identified by platform + firmware. A fixed window may only support a subset of memory + types. The 'cap_*' attributes indicate whether persistent + memory, volatile memory, accelerator memory, and / or expander + memory may be mapped behind this decoder's memory window. + +What: /sys/bus/cxl/devices/decoderX.Y/target_type +Date: June, 2021 +KernelVersion: v5.14 +Contact: linux-cxl@vger.kernel.org +Description: + When a CXL decoder is of devtype "cxl_decoder_switch", it can + optionally decode either accelerator memory (type-2) or expander + memory (type-3). The 'target_type' attribute indicates the + current setting which may dynamically change based on what + memory regions are activated in this decode hierarchy. diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c index 1f075dffc042..be357eea552c 100644 --- a/drivers/cxl/acpi.c +++ b/drivers/cxl/acpi.c @@ -70,6 +70,7 @@ static int add_host_bridge_uport(struct device *match, void *arg) struct device *host = root_port->dev.parent; struct acpi_pci_root *pci_root; struct cxl_walk_context ctx; + struct cxl_decoder *cxld; struct cxl_port *port; if (!bridge) @@ -94,7 +95,24 @@ static int add_host_bridge_uport(struct device *match, void *arg) if (ctx.count == 0) return -ENODEV; - return ctx.error; + if (ctx.error) + return ctx.error; + + /* TODO: Scan CHBCR for HDM Decoder resources */ + + /* + * In the single-port host-bridge case there are no HDM decoders + * in the CHBCR and a 1:1 passthrough decode is implied. + */ + if (ctx.count == 1) { + cxld = devm_cxl_add_passthrough_decoder(host, port); + if (IS_ERR(cxld)) + return PTR_ERR(cxld); + + dev_dbg(host, "add: %s\n", dev_name(&cxld->dev)); + } + + return 0; } static int add_host_bridge_dport(struct device *match, void *arg) diff --git a/drivers/cxl/core.c b/drivers/cxl/core.c index 8a3f3804f252..c47432f3ac72 100644 --- a/drivers/cxl/core.c +++ b/drivers/cxl/core.c @@ -33,6 +33,168 @@ static struct attribute_group cxl_base_attribute_group = { .attrs = cxl_base_attributes, }; +static ssize_t start_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + struct cxl_decoder *cxld = to_cxl_decoder(dev); + + return sysfs_emit(buf, "%#llx\n", cxld->range.start); +} +static DEVICE_ATTR_RO(start); + +static ssize_t size_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + struct cxl_decoder *cxld = to_cxl_decoder(dev); + + return sysfs_emit(buf, "%#llx\n", range_len(&cxld->range)); +} +static DEVICE_ATTR_RO(size); + +#define CXL_DECODER_FLAG_ATTR(name, flag) \ +static ssize_t name##_show(struct device *dev, \ + struct device_attribute *attr, char *buf) \ +{ \ + struct cxl_decoder *cxld = to_cxl_decoder(dev); \ + \ + return sysfs_emit(buf, "%s\n", \ + (cxld->flags & (flag)) ? "1" : "0"); \ +} \ +static DEVICE_ATTR_RO(name) + +CXL_DECODER_FLAG_ATTR(cap_pmem, CXL_DECODER_F_PMEM); +CXL_DECODER_FLAG_ATTR(cap_ram, CXL_DECODER_F_RAM); +CXL_DECODER_FLAG_ATTR(cap_type2, CXL_DECODER_F_TYPE2); +CXL_DECODER_FLAG_ATTR(cap_type3, CXL_DECODER_F_TYPE3); +CXL_DECODER_FLAG_ATTR(locked, CXL_DECODER_F_LOCK); + +static ssize_t target_type_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct cxl_decoder *cxld = to_cxl_decoder(dev); + + switch (cxld->target_type) { + case CXL_DECODER_ACCELERATOR: + return sysfs_emit(buf, "accelerator\n"); + case CXL_DECODER_EXPANDER: + return sysfs_emit(buf, "expander\n"); + } + return -ENXIO; +} +DEVICE_ATTR_RO(target_type); + +static ssize_t target_list_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct cxl_decoder *cxld = to_cxl_decoder(dev); + ssize_t offset = 0; + int i, rc = 0; + + device_lock(dev); + for (i = 0; i < cxld->interleave_ways; i++) { + struct cxl_dport *dport = cxld->target[i]; + struct cxl_dport *next = NULL; + + if (!dport) + break; + + if (i + 1 < cxld->interleave_ways) + next = cxld->target[i + 1]; + rc = sysfs_emit_at(buf, offset, "%d%s", dport->port_id, + next ? "," : ""); + if (rc < 0) + break; + offset += rc; + } + device_unlock(dev); + + if (rc < 0) + return rc; + + rc = sysfs_emit_at(buf, offset, "\n"); + if (rc < 0) + return rc; + + return offset + rc; +} +DEVICE_ATTR_RO(target_list); + +static struct attribute *cxl_decoder_base_attrs[] = { + &dev_attr_start.attr, + &dev_attr_size.attr, + &dev_attr_locked.attr, + &dev_attr_target_list.attr, + NULL, +}; + +static struct attribute_group cxl_decoder_base_attribute_group = { + .attrs = cxl_decoder_base_attrs, +}; + +static struct attribute *cxl_decoder_root_attrs[] = { + &dev_attr_cap_pmem.attr, + &dev_attr_cap_ram.attr, + &dev_attr_cap_type2.attr, + &dev_attr_cap_type3.attr, + NULL, +}; + +static struct attribute_group cxl_decoder_root_attribute_group = { + .attrs = cxl_decoder_root_attrs, +}; + +static const struct attribute_group *cxl_decoder_root_attribute_groups[] = { + &cxl_decoder_root_attribute_group, + &cxl_decoder_base_attribute_group, + &cxl_base_attribute_group, + NULL, +}; + +static struct attribute *cxl_decoder_switch_attrs[] = { + &dev_attr_target_type.attr, + NULL, +}; + +static struct attribute_group cxl_decoder_switch_attribute_group = { + .attrs = cxl_decoder_switch_attrs, +}; + +static const struct attribute_group *cxl_decoder_switch_attribute_groups[] = { + &cxl_decoder_switch_attribute_group, + &cxl_decoder_base_attribute_group, + &cxl_base_attribute_group, + NULL, +}; + +static void cxl_decoder_release(struct device *dev) +{ + struct cxl_decoder *cxld = to_cxl_decoder(dev); + struct cxl_port *port = to_cxl_port(dev->parent); + + ida_free(&port->decoder_ida, cxld->id); + kfree(cxld); +} + +static const struct device_type cxl_decoder_switch_type = { + .name = "cxl_decoder_switch", + .release = cxl_decoder_release, + .groups = cxl_decoder_switch_attribute_groups, +}; + +static const struct device_type cxl_decoder_root_type = { + .name = "cxl_decoder_root", + .release = cxl_decoder_release, + .groups = cxl_decoder_root_attribute_groups, +}; + +struct cxl_decoder *to_cxl_decoder(struct device *dev) +{ + if (dev_WARN_ONCE(dev, dev->type->release != cxl_decoder_release, + "not a cxl_decoder device\n")) + return NULL; + return container_of(dev, struct cxl_decoder, dev); +} + static void cxl_dport_release(struct cxl_dport *dport) { list_del(&dport->list); @@ -138,6 +300,7 @@ static struct cxl_port *cxl_port_alloc(struct device *uport, port->uport = uport; port->component_reg_phys = component_reg_phys; + ida_init(&port->decoder_ida); INIT_LIST_HEAD(&port->dports); device_initialize(dev); @@ -274,6 +437,108 @@ int cxl_add_dport(struct cxl_port *port, struct device *dport_dev, int port_id, } EXPORT_SYMBOL_GPL(cxl_add_dport); +static struct cxl_decoder * +cxl_decoder_alloc(struct cxl_port *port, int nr_targets, resource_size_t base, + resource_size_t len, int interleave_ways, + int interleave_granularity, enum cxl_decoder_type type, + unsigned long flags) +{ + struct cxl_decoder *cxld; + struct device *dev; + int rc = 0; + + if (interleave_ways < 1) + return ERR_PTR(-EINVAL); + + device_lock(&port->dev); + if (list_empty(&port->dports)) + rc = -EINVAL; + device_unlock(&port->dev); + if (rc) + return ERR_PTR(rc); + + cxld = kzalloc(struct_size(cxld, target, nr_targets), GFP_KERNEL); + if (!cxld) + return ERR_PTR(-ENOMEM); + + rc = ida_alloc(&port->decoder_ida, GFP_KERNEL); + if (rc < 0) + goto err; + + *cxld = (struct cxl_decoder) { + .id = rc, + .range = { + .start = base, + .end = base + len - 1, + }, + .flags = flags, + .interleave_ways = interleave_ways, + .interleave_granularity = interleave_granularity, + .target_type = type, + }; + + /* handle implied target_list */ + if (interleave_ways == 1) + cxld->target[0] = + list_first_entry(&port->dports, struct cxl_dport, list); + dev = &cxld->dev; + device_initialize(dev); + device_set_pm_not_required(dev); + dev->parent = &port->dev; + dev->bus = &cxl_bus_type; + + /* root ports do not have a cxl_port_type parent */ + if (port->dev.parent->type == &cxl_port_type) + dev->type = &cxl_decoder_switch_type; + else + dev->type = &cxl_decoder_root_type; + + return cxld; +err: + kfree(cxld); + return ERR_PTR(rc); +} + +static void unregister_dev(void *dev) +{ + device_unregister(dev); +} + +struct cxl_decoder * +devm_cxl_add_decoder(struct device *host, struct cxl_port *port, int nr_targets, + resource_size_t base, resource_size_t len, + int interleave_ways, int interleave_granularity, + enum cxl_decoder_type type, unsigned long flags) +{ + struct cxl_decoder *cxld; + struct device *dev; + int rc; + + cxld = cxl_decoder_alloc(port, nr_targets, base, len, interleave_ways, + interleave_granularity, type, flags); + if (IS_ERR(cxld)) + return cxld; + + dev = &cxld->dev; + rc = dev_set_name(dev, "decoder%d.%d", port->id, cxld->id); + if (rc) + goto err; + + rc = device_add(dev); + if (rc) + goto err; + + rc = devm_add_action_or_reset(host, unregister_dev, dev); + if (rc) + return ERR_PTR(rc); + return cxld; + +err: + put_device(dev); + return ERR_PTR(rc); +} +EXPORT_SYMBOL_GPL(devm_cxl_add_decoder); + /** * cxl_probe_component_regs() - Detect CXL Component register blocks * @dev: Host device of the @base mapping diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index dd159fd6d692..b988ea288f53 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -156,6 +156,45 @@ int cxl_map_device_regs(struct pci_dev *pdev, #define CXL_RESOURCE_NONE ((resource_size_t) -1) #define CXL_TARGET_STRLEN 20 +/* + * cxl_decoder flags that define the type of memory / devices this + * decoder supports as well as configuration lock status See "CXL 2.0 + * 8.2.5.12.7 CXL HDM Decoder 0 Control Register" for details. + */ +#define CXL_DECODER_F_RAM BIT(0) +#define CXL_DECODER_F_PMEM BIT(1) +#define CXL_DECODER_F_TYPE2 BIT(2) +#define CXL_DECODER_F_TYPE3 BIT(3) +#define CXL_DECODER_F_LOCK BIT(4) +#define CXL_DECODER_F_MASK GENMASK(4, 0) + +enum cxl_decoder_type { + CXL_DECODER_ACCELERATOR = 2, + CXL_DECODER_EXPANDER = 3, +}; + +/** + * struct cxl_decoder - CXL address range decode configuration + * @dev: this decoder's device + * @id: kernel device name id + * @range: address range considered by this decoder + * @interleave_ways: number of cxl_dports in this decode + * @interleave_granularity: data stride per dport + * @target_type: accelerator vs expander (type2 vs type3) selector + * @flags: memory type capabilities and locking + * @target: active ordered target list in current decoder configuration + */ +struct cxl_decoder { + struct device dev; + int id; + struct range range; + int interleave_ways; + int interleave_granularity; + enum cxl_decoder_type target_type; + unsigned long flags; + struct cxl_dport *target[]; +}; + /** * struct cxl_port - logical collection of upstream port devices and * downstream port devices to construct a CXL memory @@ -164,6 +203,7 @@ int cxl_map_device_regs(struct pci_dev *pdev, * @uport: PCI or platform device implementing the upstream port capability * @id: id for port device-name * @dports: cxl_dport instances referenced by decoders + * @decoder_ida: allocator for decoder ids * @component_reg_phys: component register capability base address (optional) */ struct cxl_port { @@ -171,6 +211,7 @@ struct cxl_port { struct device *uport; int id; struct list_head dports; + struct ida decoder_ida; resource_size_t component_reg_phys; }; @@ -197,5 +238,27 @@ struct cxl_port *devm_cxl_add_port(struct device *host, struct device *uport, int cxl_add_dport(struct cxl_port *port, struct device *dport, int port_id, resource_size_t component_reg_phys); + +struct cxl_decoder *to_cxl_decoder(struct device *dev); +struct cxl_decoder * +devm_cxl_add_decoder(struct device *host, struct cxl_port *port, int nr_targets, + resource_size_t base, resource_size_t len, + int interleave_ways, int interleave_granularity, + enum cxl_decoder_type type, unsigned long flags); + +/* + * Per the CXL specification (8.2.5.12 CXL HDM Decoder Capability Structure) + * single ported host-bridges need not publish a decoder capability when a + * passthrough decode can be assumed, i.e. all transactions that the uport sees + * are claimed and passed to the single dport. Default the range a 0-base + * 0-length until the first CXL region is activated. + */ +static inline struct cxl_decoder * +devm_cxl_add_passthrough_decoder(struct device *host, struct cxl_port *port) +{ + return devm_cxl_add_decoder(host, port, 1, 0, 0, 1, PAGE_SIZE, + CXL_DECODER_EXPANDER, 0); +} + extern struct bus_type cxl_bus_type; #endif /* __CXL_H__ */