From patchwork Wed Nov 30 23:12:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13060674 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29FB5C352A1 for ; Wed, 30 Nov 2022 23:20:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230099AbiK3XU2 (ORCPT ); Wed, 30 Nov 2022 18:20:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59876 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230262AbiK3XT5 (ORCPT ); Wed, 30 Nov 2022 18:19:57 -0500 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F395962F6 for ; Wed, 30 Nov 2022 15:12:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669849977; x=1701385977; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Z/89jLFa5/DnLVEBea5fyMtXNwt94XDl9JMNyPdln44=; b=IUteXppyCPNhxicwQBiZVMYhZbtBOoSpc2QTKgnvKokU/+S8P9fAKRcx cvO8mOdfaUBi6R8wsQEmtXB4aucXqPmlpMVyvYwWOSwy2ZgS1epZ5dfAN 8tcaFViM6GTfunSYInzBSwKJvEwUSnk1bR0sLmwZwIfFZkFBEbzlFS7sC tvDWZTTeXNcJgAi28qHdZUEqEBPe9Qee3P5V8JP2e3tFfTRSqoIXQA7i6 hjponhxE/g9vVrndFRMYWHtLTnrbDXo7Sii6rdJ8PmvzUujb0aY2IhtcP LhM/gH6owVk9DY8FMp8sj1v4NQeSD/cNMlLNlofAOj84PuYH1D/TgS/m9 A==; X-IronPort-AV: E=McAfee;i="6500,9779,10547"; a="314232854" X-IronPort-AV: E=Sophos;i="5.96,207,1665471600"; d="scan'208";a="314232854" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Nov 2022 15:12:21 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10547"; a="675206109" X-IronPort-AV: E=Sophos;i="5.96,207,1665471600"; d="scan'208";a="675206109" Received: from djiang5-desk3.ch.intel.com ([143.182.136.137]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Nov 2022 15:12:21 -0800 Subject: [RFC PATCH 1/8] cxl: break out range register decoding from cxl_hdm_decode_init() From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com, rrichter@amd.com, terry.bowman@amd.com Date: Wed, 30 Nov 2022 16:12:20 -0700 Message-ID: <166984994091.2805382.15976080608757662866.stgit@djiang5-desk3.ch.intel.com> In-Reply-To: <166984987659.2805382.17264896576424996856.stgit@djiang5-desk3.ch.intel.com> References: <166984987659.2805382.17264896576424996856.stgit@djiang5-desk3.ch.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org There are 2 scenarios that requires additional handling. 1. A device that has active ranges in DVSEC range registers (RR) but no HDM decoder register block. 2. A device that has both RR active and HDM, but the HDM decoders are not programmed. The goal is to create emulated decoder software structs based on the RR. Move the CXL DVSEC range register decoding code block from cxl_hdm_decode_init() to its own function. Refactor code in preparation for the HDM decoder emulation. There is no functionality change to the code. Name the new function to cxl_dvsec_rr_decode(). The only change is to set range->start to CXL_RESOURCE_NONE if the range is not programmed correctly or active. Signed-off-by: Dave Jiang --- drivers/cxl/core/pci.c | 55 ++++++++++++++++++++++++++++++------------------ 1 file changed, 34 insertions(+), 21 deletions(-) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 0dbbe8d39b07..d674ddfe141c 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -142,10 +142,9 @@ int cxl_await_media_ready(struct cxl_dev_state *cxlds) } EXPORT_SYMBOL_NS_GPL(cxl_await_media_ready, CXL); -static int wait_for_valid(struct cxl_dev_state *cxlds) +static int wait_for_valid(struct pci_dev *pdev, int d) { - struct pci_dev *pdev = to_pci_dev(cxlds->dev); - int d = cxlds->cxl_dvsec, rc; + int rc; u32 val; /* @@ -335,20 +334,12 @@ static bool __cxl_hdm_decode_init(struct cxl_dev_state *cxlds, return true; } -/** - * cxl_hdm_decode_init() - Setup HDM decoding for the endpoint - * @cxlds: Device state - * @cxlhdm: Mapped HDM decoder Capability - * - * Try to enable the endpoint's HDM Decoder Capability - */ -int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm) + +static int cxl_dvsec_rr_decode(struct pci_dev *pdev, int d, + struct cxl_endpoint_dvsec_info *info) { - struct pci_dev *pdev = to_pci_dev(cxlds->dev); - struct cxl_endpoint_dvsec_info info = { 0 }; int hdm_count, rc, i, ranges = 0; struct device *dev = &pdev->dev; - int d = cxlds->cxl_dvsec; u16 cap, ctrl; if (!d) { @@ -379,7 +370,7 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm) if (!hdm_count || hdm_count > 2) return -EINVAL; - rc = wait_for_valid(cxlds); + rc = wait_for_valid(pdev, d); if (rc) { dev_dbg(dev, "Failure awaiting MEM_INFO_VALID (%d)\n", rc); return rc; @@ -390,9 +381,9 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm) * disabled, and they will remain moot after the HDM Decoder * capability is enabled. */ - info.mem_enabled = FIELD_GET(CXL_DVSEC_MEM_ENABLE, ctrl); - if (!info.mem_enabled) - goto hdm_init; + info->mem_enabled = FIELD_GET(CXL_DVSEC_MEM_ENABLE, ctrl); + if (!info->mem_enabled) + return 0; for (i = 0; i < hdm_count; i++) { u64 base, size; @@ -426,22 +417,44 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm) base |= temp & CXL_DVSEC_MEM_BASE_LOW_MASK; - info.dvsec_range[i] = (struct range) { + info->dvsec_range[i] = (struct range) { .start = base, .end = base + size - 1 }; if (size) ranges++; + else + info->dvsec_range[i].start = CXL_RESOURCE_NONE; } - info.ranges = ranges; + info->ranges = ranges; + return 0; +} + +/** + * cxl_hdm_decode_init() - Setup HDM decoding for the endpoint + * @cxlds: Device state + * @cxlhdm: Mapped HDM decoder Capability + * + * Try to enable the endpoint's HDM Decoder Capability + */ +int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm) +{ + struct pci_dev *pdev = to_pci_dev(cxlds->dev); + struct cxl_endpoint_dvsec_info info = { 0 }; + int rc; + struct device *dev = &pdev->dev; + int d = cxlds->cxl_dvsec; + + rc = cxl_dvsec_rr_decode(pdev, d, &info); + if (rc < 0) + return rc; /* * If DVSEC ranges are being used instead of HDM decoder registers there * is no use in trying to manage those. */ -hdm_init: if (!__cxl_hdm_decode_init(cxlds, cxlhdm, &info)) { dev_err(dev, "Legacy range registers configuration prevents HDM operation.\n"); From patchwork Wed Nov 30 23:12:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13060675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC230C4321E for ; Wed, 30 Nov 2022 23:20:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230108AbiK3XU3 (ORCPT ); Wed, 30 Nov 2022 18:20:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230264AbiK3XT5 (ORCPT ); Wed, 30 Nov 2022 18:19:57 -0500 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2098F102F for ; Wed, 30 Nov 2022 15:12:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669849979; x=1701385979; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=c2O4WXY0pE+wjOrAJMW+0OK7FaEClFvd3QCs0X1nrIc=; b=NPa1mQTorBgDSvqFoxW1c395UmIF+E6gHwDGisbVL9xVaamkaAl324Mo IkX4L9MlivcrqWhnpbrqV1Q8XvH41NF/P533zDt8m48X1MeIrXl49nXYG dWe1uM5Xxx3jqmS04y13BG8BSI/Y0RoCQM6j0U5Cg7GF2ebcxb7UI/J7v xnGWdf80Azhd1RCMcrmGHCced6JDq3ahQ9VMEdIrikIo2Gp3zzzT9Pmxf DhPqMlfe6+aFk2s3qmdBK7YRLq2QmAQzSHsdkrHi1c6sUXx//5+e1PMtc gGI0gKhVJxRjgmqupeFZZDOtnMbwbI9hFhsfftHjkwb+JMNl8lPUHxgMD A==; X-IronPort-AV: E=McAfee;i="6500,9779,10547"; a="314232874" X-IronPort-AV: E=Sophos;i="5.96,207,1665471600"; d="scan'208";a="314232874" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Nov 2022 15:12:27 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10547"; a="675206172" X-IronPort-AV: E=Sophos;i="5.96,207,1665471600"; d="scan'208";a="675206172" Received: from djiang5-desk3.ch.intel.com ([143.182.136.137]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Nov 2022 15:12:26 -0800 Subject: [RFC PATCH 2/8] cxl: export cxl_dvsec_rr_decode() to cxl_port From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com, rrichter@amd.com, terry.bowman@amd.com Date: Wed, 30 Nov 2022 16:12:26 -0700 Message-ID: <166984994667.2805382.3598594737505566629.stgit@djiang5-desk3.ch.intel.com> In-Reply-To: <166984987659.2805382.17264896576424996856.stgit@djiang5-desk3.ch.intel.com> References: <166984987659.2805382.17264896576424996856.stgit@djiang5-desk3.ch.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Call cxl_dvsec_rr_decode() in the beginning of cxl_port_probe() and preserve the decoded information in the 'struct cxl_endpoint_dvsec_info'. This info can be passed to various functions later on in order to support the HDM decoder emulation. The invocation of cxl_dvsec_rr_decode() in cxl_hdm_decode_init() is removed and a pointer to the 'struct cxl_endpoint_dvsec_info' is passed in. Signed-off-by: Dave Jiang --- drivers/cxl/core/pci.c | 17 ++++++----------- drivers/cxl/cxl.h | 14 ++++++++++++++ drivers/cxl/cxlmem.h | 12 ------------ drivers/cxl/cxlpci.h | 3 ++- drivers/cxl/port.c | 15 +++++++++++---- 5 files changed, 33 insertions(+), 28 deletions(-) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index d674ddfe141c..7196b1fcdcfc 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -335,8 +335,8 @@ static bool __cxl_hdm_decode_init(struct cxl_dev_state *cxlds, } -static int cxl_dvsec_rr_decode(struct pci_dev *pdev, int d, - struct cxl_endpoint_dvsec_info *info) +int cxl_dvsec_rr_decode(struct pci_dev *pdev, int d, + struct cxl_endpoint_dvsec_info *info) { int hdm_count, rc, i, ranges = 0; struct device *dev = &pdev->dev; @@ -431,6 +431,7 @@ static int cxl_dvsec_rr_decode(struct pci_dev *pdev, int d, info->ranges = ranges; return 0; } +EXPORT_SYMBOL_NS_GPL(cxl_dvsec_rr_decode, CXL); /** * cxl_hdm_decode_init() - Setup HDM decoding for the endpoint @@ -439,23 +440,17 @@ static int cxl_dvsec_rr_decode(struct pci_dev *pdev, int d, * * Try to enable the endpoint's HDM Decoder Capability */ -int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm) +int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, + struct cxl_endpoint_dvsec_info *info) { struct pci_dev *pdev = to_pci_dev(cxlds->dev); - struct cxl_endpoint_dvsec_info info = { 0 }; - int rc; struct device *dev = &pdev->dev; - int d = cxlds->cxl_dvsec; - - rc = cxl_dvsec_rr_decode(pdev, d, &info); - if (rc < 0) - return rc; /* * If DVSEC ranges are being used instead of HDM decoder registers there * is no use in trying to manage those. */ - if (!__cxl_hdm_decode_init(cxlds, cxlhdm, &info)) { + if (!__cxl_hdm_decode_init(cxlds, cxlhdm, info)) { dev_err(dev, "Legacy range registers configuration prevents HDM operation.\n"); return -EBUSY; diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index d94635e43a50..b6caf1cfb3e9 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -591,10 +591,24 @@ int cxl_decoder_add_locked(struct cxl_decoder *cxld, int *target_map); int cxl_decoder_autoremove(struct device *host, struct cxl_decoder *cxld); int cxl_endpoint_autoremove(struct cxl_memdev *cxlmd, struct cxl_port *endpoint); +/** + * struct cxl_endpoint_dvsec_info - Cached DVSEC info + * @mem_enabled: cached value of mem_enabled in the DVSEC, PCIE_DEVICE + * @ranges: Number of active HDM ranges this device uses. + * @dvsec_range: cached attributes of the ranges in the DVSEC, PCIE_DEVICE + */ +struct cxl_endpoint_dvsec_info { + bool mem_enabled; + int ranges; + struct range dvsec_range[2]; +}; + struct cxl_hdm; struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port); int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm); int devm_cxl_add_passthrough_decoder(struct cxl_port *port); +int cxl_dvsec_rr_decode(struct pci_dev *pdev, int dvsec, + struct cxl_endpoint_dvsec_info *info); bool is_cxl_region(struct device *dev); diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index 35d485d041f0..fd8ed573fbf9 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -179,18 +179,6 @@ static inline int cxl_mbox_cmd_rc2errno(struct cxl_mbox_cmd *mbox_cmd) */ #define CXL_CAPACITY_MULTIPLIER SZ_256M -/** - * struct cxl_endpoint_dvsec_info - Cached DVSEC info - * @mem_enabled: cached value of mem_enabled in the DVSEC, PCIE_DEVICE - * @ranges: Number of active HDM ranges this device uses. - * @dvsec_range: cached attributes of the ranges in the DVSEC, PCIE_DEVICE - */ -struct cxl_endpoint_dvsec_info { - bool mem_enabled; - int ranges; - struct range dvsec_range[2]; -}; - /** * struct cxl_dev_state - The driver device state * diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h index eec597dbe763..02a39a2609b7 100644 --- a/drivers/cxl/cxlpci.h +++ b/drivers/cxl/cxlpci.h @@ -73,6 +73,7 @@ static inline resource_size_t cxl_regmap_to_base(struct pci_dev *pdev, int devm_cxl_port_enumerate_dports(struct cxl_port *port); struct cxl_dev_state; -int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm); +int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, + struct cxl_endpoint_dvsec_info *info); void read_cdat_data(struct cxl_port *port); #endif /* __CXL_PCI_H__ */ diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index 5453771bf330..f899d62a91af 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -32,7 +32,10 @@ static void schedule_detach(void *cxlmd) static int cxl_port_probe(struct device *dev) { + struct cxl_endpoint_dvsec_info info = { 0 }; struct cxl_port *port = to_cxl_port(dev); + struct cxl_dev_state *cxlds; + struct cxl_memdev *cxlmd; struct cxl_hdm *cxlhdm; int rc; @@ -43,6 +46,13 @@ static int cxl_port_probe(struct device *dev) return rc; if (rc == 1) return devm_cxl_add_passthrough_decoder(port); + } else { + cxlmd = to_cxl_memdev(port->uport); + cxlds = cxlmd->cxlds; + rc = cxl_dvsec_rr_decode(to_pci_dev(cxlds->dev), + cxlds->cxl_dvsec, &info); + if (rc < 0) + return rc; } cxlhdm = devm_cxl_setup_hdm(port); @@ -50,9 +60,6 @@ static int cxl_port_probe(struct device *dev) return PTR_ERR(cxlhdm); if (is_cxl_endpoint(port)) { - struct cxl_memdev *cxlmd = to_cxl_memdev(port->uport); - struct cxl_dev_state *cxlds = cxlmd->cxlds; - /* Cache the data early to ensure is_visible() works */ read_cdat_data(port); @@ -61,7 +68,7 @@ static int cxl_port_probe(struct device *dev) if (rc) return rc; - rc = cxl_hdm_decode_init(cxlds, cxlhdm); + rc = cxl_hdm_decode_init(cxlds, cxlhdm, &info); if (rc) return rc; From patchwork Wed Nov 30 23:12:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13060677 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C73EC47088 for ; Wed, 30 Nov 2022 23:20:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229712AbiK3XUo (ORCPT ); Wed, 30 Nov 2022 18:20:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229821AbiK3XUM (ORCPT ); Wed, 30 Nov 2022 18:20:12 -0500 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 898C798005 for ; Wed, 30 Nov 2022 15:13:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669849985; x=1701385985; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=mJDyeITSCeC3z03AedLhyOfaQo2CXqiIxZuxZ1oRsS8=; b=JWQmmwIuKX0+bpEwlJS7SLR+/0FHO2QKRpu7vUl3T/WxMvzAZW7clGiP SKEdLWtqBRwbgPSVAOsQhoh4sDy4uiWUhSnsBlU3nXYlCr45NvvKLj3wX sUbr4m3WEC9iG++TQQL1ctyljt56gN3FJdy1BE3r2UrRzWlU3ChI0Vc8y Jd4ME0JyPAVNwe3EIG7cOLNEfQ2SNo1GgVE3irHWAACzgmDPKMr42zDj+ wzsVNXlg2Joqpl6A6phT/5n4odgzAXVJkmPfezbyDkmU7MT2LI9/KyGML SCszrnEUeyBGCXbPeeRa3a/DZxmcBufX0+CBkm2OL7xXeTEa2bj79CifT A==; X-IronPort-AV: E=McAfee;i="6500,9779,10547"; a="379811984" X-IronPort-AV: E=Sophos;i="5.96,207,1665471600"; d="scan'208";a="379811984" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Nov 2022 15:12:33 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10547"; a="733171014" X-IronPort-AV: E=Sophos;i="5.96,207,1665471600"; d="scan'208";a="733171014" Received: from djiang5-desk3.ch.intel.com ([143.182.136.137]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Nov 2022 15:12:32 -0800 Subject: [RFC PATCH 3/8] cxl: refactor cxl_hdm_decode_init() From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com, rrichter@amd.com, terry.bowman@amd.com Date: Wed, 30 Nov 2022 16:12:32 -0700 Message-ID: <166984995249.2805382.1339697898311247332.stgit@djiang5-desk3.ch.intel.com> In-Reply-To: <166984987659.2805382.17264896576424996856.stgit@djiang5-desk3.ch.intel.com> References: <166984987659.2805382.17264896576424996856.stgit@djiang5-desk3.ch.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org With the previous refactoring of DVSEC range registers out of cxl_hdm_decode_init(), it basically becomes a skeleton function. Squash __cxl_hdm_decode_init() with cxl_hdm_decode_init() to simplify the code. Signed-off-by: Dave Jiang --- drivers/cxl/core/pci.c | 138 ++++++++++++++++++++---------------------------- 1 file changed, 57 insertions(+), 81 deletions(-) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 7196b1fcdcfc..385dbe9bd5f2 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -260,81 +260,6 @@ static int devm_cxl_enable_hdm(struct device *host, struct cxl_hdm *cxlhdm) return devm_add_action_or_reset(host, disable_hdm, cxlhdm); } -static bool __cxl_hdm_decode_init(struct cxl_dev_state *cxlds, - struct cxl_hdm *cxlhdm, - struct cxl_endpoint_dvsec_info *info) -{ - void __iomem *hdm = cxlhdm->regs.hdm_decoder; - struct cxl_port *port = cxlhdm->port; - struct device *dev = cxlds->dev; - struct cxl_port *root; - int i, rc, allowed; - u32 global_ctrl; - - global_ctrl = readl(hdm + CXL_HDM_DECODER_CTRL_OFFSET); - - /* - * If the HDM Decoder Capability is already enabled then assume - * that some other agent like platform firmware set it up. - */ - if (global_ctrl & CXL_HDM_DECODER_ENABLE) { - rc = devm_cxl_enable_mem(&port->dev, cxlds); - if (rc) - return false; - return true; - } - - root = to_cxl_port(port->dev.parent); - while (!is_cxl_root(root) && is_cxl_port(root->dev.parent)) - root = to_cxl_port(root->dev.parent); - if (!is_cxl_root(root)) { - dev_err(dev, "Failed to acquire root port for HDM enable\n"); - return false; - } - - for (i = 0, allowed = 0; info->mem_enabled && i < info->ranges; i++) { - struct device *cxld_dev; - - cxld_dev = device_find_child(&root->dev, &info->dvsec_range[i], - dvsec_range_allowed); - if (!cxld_dev) { - dev_dbg(dev, "DVSEC Range%d denied by platform\n", i); - continue; - } - dev_dbg(dev, "DVSEC Range%d allowed by platform\n", i); - put_device(cxld_dev); - allowed++; - } - - if (!allowed) { - cxl_set_mem_enable(cxlds, 0); - info->mem_enabled = 0; - } - - /* - * Per CXL 2.0 Section 8.1.3.8.3 and 8.1.3.8.4 DVSEC CXL Range 1 Base - * [High,Low] when HDM operation is enabled the range register values - * are ignored by the device, but the spec also recommends matching the - * DVSEC Range 1,2 to HDM Decoder Range 0,1. So, non-zero info->ranges - * are expected even though Linux does not require or maintain that - * match. If at least one DVSEC range is enabled and allowed, skip HDM - * Decoder Capability Enable. - */ - if (info->mem_enabled) - return false; - - rc = devm_cxl_enable_hdm(&port->dev, cxlhdm); - if (rc) - return false; - - rc = devm_cxl_enable_mem(&port->dev, cxlds); - if (rc) - return false; - - return true; -} - - int cxl_dvsec_rr_decode(struct pci_dev *pdev, int d, struct cxl_endpoint_dvsec_info *info) { @@ -445,17 +370,68 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, { struct pci_dev *pdev = to_pci_dev(cxlds->dev); struct device *dev = &pdev->dev; + void __iomem *hdm = cxlhdm->regs.hdm_decoder; + struct cxl_port *port = cxlhdm->port; + struct cxl_port *root; + int i, rc, allowed; + u32 global_ctrl; + + global_ctrl = readl(hdm + CXL_HDM_DECODER_CTRL_OFFSET); /* - * If DVSEC ranges are being used instead of HDM decoder registers there - * is no use in trying to manage those. + * If the HDM Decoder Capability is already enabled then assume + * that some other agent like platform firmware set it up. */ - if (!__cxl_hdm_decode_init(cxlds, cxlhdm, info)) { - dev_err(dev, - "Legacy range registers configuration prevents HDM operation.\n"); - return -EBUSY; + if (global_ctrl & CXL_HDM_DECODER_ENABLE) + return devm_cxl_enable_mem(&port->dev, cxlds); + + root = to_cxl_port(port->dev.parent); + while (!is_cxl_root(root) && is_cxl_port(root->dev.parent)) + root = to_cxl_port(root->dev.parent); + if (!is_cxl_root(root)) { + dev_err(dev, "Failed to acquire root port for HDM enable\n"); + return -ENODEV; } + for (i = 0, allowed = 0; info->mem_enabled && i < info->ranges; i++) { + struct device *cxld_dev; + + cxld_dev = device_find_child(&root->dev, &info->dvsec_range[i], + dvsec_range_allowed); + if (!cxld_dev) { + dev_dbg(dev, "DVSEC Range%d denied by platform\n", i); + continue; + } + dev_dbg(dev, "DVSEC Range%d allowed by platform\n", i); + put_device(cxld_dev); + allowed++; + } + + if (!allowed) { + cxl_set_mem_enable(cxlds, 0); + info->mem_enabled = 0; + } + + /* + * Per CXL 2.0 Section 8.1.3.8.3 and 8.1.3.8.4 DVSEC CXL Range 1 Base + * [High,Low] when HDM operation is enabled the range register values + * are ignored by the device, but the spec also recommends matching the + * DVSEC Range 1,2 to HDM Decoder Range 0,1. So, non-zero info->ranges + * are expected even though Linux does not require or maintain that + * match. If at least one DVSEC range is enabled and allowed, skip HDM + * Decoder Capability Enable. + */ + if (info->mem_enabled) + return -EBUSY; + + rc = devm_cxl_enable_hdm(&port->dev, cxlhdm); + if (rc) + return rc; + + rc = devm_cxl_enable_mem(&port->dev, cxlds); + if (rc) + return rc; + return 0; } EXPORT_SYMBOL_NS_GPL(cxl_hdm_decode_init, CXL); From patchwork Wed Nov 30 23:12:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13060676 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 484A6C4321E for ; Wed, 30 Nov 2022 23:20:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229772AbiK3XUn (ORCPT ); Wed, 30 Nov 2022 18:20:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59712 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229918AbiK3XUP (ORCPT ); Wed, 30 Nov 2022 18:20:15 -0500 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80D3FA0568 for ; Wed, 30 Nov 2022 15:13:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669849989; x=1701385989; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jAn1SeLPPxEKDZ8cws4fwvhUj1/nKqsbUdc5a4YG8O4=; b=RX/em8qktzjIeDOSa4SYnExfs/rmuV96fHPYIxkAdcvdj+0uKu8qZ82o d1VAijF92nOOC6x+cwHt4V4PomqvIwCbYIHkqlVR6aVwMWH4qi2ta9MsV SQ32a+6NeTiEAlfJEUTUvmIsKQr5AxyBL5BSeY8crV1YT2PRomx5TA44N Ai8GCmCI+POcKecTMXtna5t7pK21B5ubUNTSHPN2WS9lwVNKwuuTDiRiF RGeyASRuvlKLrxzx7Drc52scDIoDvkJKbrexWtf+crqdd8Lh/kcplvc3I JT5i9Cki3+wtC7Qj+JKbNE98DmLv18+hZa7tLB4RqbzIO1X4gpyaAGrz5 Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10547"; a="379812000" X-IronPort-AV: E=Sophos;i="5.96,207,1665471600"; d="scan'208";a="379812000" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Nov 2022 15:12:39 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10547"; a="733171035" X-IronPort-AV: E=Sophos;i="5.96,207,1665471600"; d="scan'208";a="733171035" Received: from djiang5-desk3.ch.intel.com ([143.182.136.137]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Nov 2022 15:12:38 -0800 Subject: [RFC PATCH 4/8] cxl: emulate HDM decoder from DVSEC range registers From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com, rrichter@amd.com, terry.bowman@amd.com Date: Wed, 30 Nov 2022 16:12:38 -0700 Message-ID: <166984995817.2805382.13166060288973761642.stgit@djiang5-desk3.ch.intel.com> In-Reply-To: <166984987659.2805382.17264896576424996856.stgit@djiang5-desk3.ch.intel.com> References: <166984987659.2805382.17264896576424996856.stgit@djiang5-desk3.ch.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org In the case where HDM decoder register block exists but is not programmed and at the same time the DVSEC range register range is active, populate the CXL decoder object 'cxl_decoder' with info from DVSEC range registers. Signed-off-by: Dave Jiang --- drivers/cxl/core/hdm.c | 33 ++++++++++++++++++++++++++++++--- drivers/cxl/core/pci.c | 12 ------------ drivers/cxl/cxl.h | 3 ++- drivers/cxl/port.c | 2 +- 4 files changed, 33 insertions(+), 17 deletions(-) diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c index d1d2caea5c62..9773a5efaefd 100644 --- a/drivers/cxl/core/hdm.c +++ b/drivers/cxl/core/hdm.c @@ -674,9 +674,31 @@ static int cxl_decoder_reset(struct cxl_decoder *cxld) return 0; } +static int cxl_setup_hdm_decoder_from_dvsec(struct cxl_port *port, + struct cxl_decoder *cxld, int which, + struct cxl_endpoint_dvsec_info *info) +{ + if (!is_cxl_endpoint(port)) + return -EOPNOTSUPP; + + if (info->dvsec_range[which].start == CXL_RESOURCE_NONE) + return -ENXIO; + + cxld->target_type = CXL_DECODER_ACCELERATOR; + + cxld->hpa_range = (struct range) { + .start = info->dvsec_range[which].start, + .end = info->dvsec_range[which].end, + }; + + cxld->flags |= CXL_DECODER_F_ENABLE | CXL_DECODER_F_LOCK; + port->commit_end = cxld->id; + return 0; +} + static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, int *target_map, void __iomem *hdm, int which, - u64 *dpa_base) + u64 *dpa_base, struct cxl_endpoint_dvsec_info *info) { struct cxl_endpoint_decoder *cxled = NULL; u64 size, base, skip, dpa_size; @@ -712,6 +734,10 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, .end = base + size - 1, }; + if (cxled && !committed && + info->dvsec_range[which].start != CXL_RESOURCE_NONE) + return cxl_setup_hdm_decoder_from_dvsec(port, cxld, which, info); + /* decoders are enabled if committed */ if (committed) { cxld->flags |= CXL_DECODER_F_ENABLE; @@ -785,7 +811,8 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, * devm_cxl_enumerate_decoders - add decoder objects per HDM register set * @cxlhdm: Structure to populate with HDM capabilities */ -int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm) +int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm, + struct cxl_endpoint_dvsec_info *info) { void __iomem *hdm = cxlhdm->regs.hdm_decoder; struct cxl_port *port = cxlhdm->port; @@ -837,7 +864,7 @@ int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm) cxld = &cxlsd->cxld; } - rc = init_hdm_decoder(port, cxld, target_map, hdm, i, &dpa_base); + rc = init_hdm_decoder(port, cxld, target_map, hdm, i, &dpa_base, info); if (rc) { put_device(&cxld->dev); return rc; diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 385dbe9bd5f2..5e44fe23fa76 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -412,18 +412,6 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, info->mem_enabled = 0; } - /* - * Per CXL 2.0 Section 8.1.3.8.3 and 8.1.3.8.4 DVSEC CXL Range 1 Base - * [High,Low] when HDM operation is enabled the range register values - * are ignored by the device, but the spec also recommends matching the - * DVSEC Range 1,2 to HDM Decoder Range 0,1. So, non-zero info->ranges - * are expected even though Linux does not require or maintain that - * match. If at least one DVSEC range is enabled and allowed, skip HDM - * Decoder Capability Enable. - */ - if (info->mem_enabled) - return -EBUSY; - rc = devm_cxl_enable_hdm(&port->dev, cxlhdm); if (rc) return rc; diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index b6caf1cfb3e9..3fe1043a7796 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -605,7 +605,8 @@ struct cxl_endpoint_dvsec_info { struct cxl_hdm; struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port); -int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm); +int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm, + struct cxl_endpoint_dvsec_info *info); int devm_cxl_add_passthrough_decoder(struct cxl_port *port); int cxl_dvsec_rr_decode(struct pci_dev *pdev, int dvsec, struct cxl_endpoint_dvsec_info *info); diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index f899d62a91af..3dedf3dd534a 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -79,7 +79,7 @@ static int cxl_port_probe(struct device *dev) } } - rc = devm_cxl_enumerate_decoders(cxlhdm); + rc = devm_cxl_enumerate_decoders(cxlhdm, &info); if (rc) { dev_err(dev, "Couldn't enumerate decoders (%d)\n", rc); return rc; From patchwork Wed Nov 30 23:12:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13060678 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3A26C4321E for ; Wed, 30 Nov 2022 23:20:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229854AbiK3XUq (ORCPT ); Wed, 30 Nov 2022 18:20:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230078AbiK3XU1 (ORCPT ); Wed, 30 Nov 2022 18:20:27 -0500 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 786C3A0571 for ; Wed, 30 Nov 2022 15:13:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669849992; x=1701385992; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=QJnR5quGMjk9DtK9wME+6whYjdrDoTUPqDidMK2nkB8=; b=Fc/unhQIBnG98pruba0SA+Lbdj62EyWo7u0806nnfmdkyblnU3aISPjy F5iYahVHiTV2n3GH9FKaagF7mMlbHwlKKNoSwLWaly2f0Q/syHJ26uF6m QM1erBng1+nGKSPCahfx9j+GJYvfwyei75uTocEOonJTVLc9fJDLfBIYJ /5Urvw4FfdHZI9Itdcqmyolzmv/CfUaHAIOfVycaAQXjHwfSfHhZso4Qk ZBfbCHIENRU8Un5VWoIfXJ/xZDYXso9H6YaIeVCHVAe4ugWPdVK6SEEpa VLCwzq7IR8+irlZF3vDsiJoUFKwHBmteE51lvmmwUVR8zeipQ5Ep7G+ML w==; X-IronPort-AV: E=McAfee;i="6500,9779,10547"; a="379812030" X-IronPort-AV: E=Sophos;i="5.96,207,1665471600"; d="scan'208";a="379812030" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Nov 2022 15:12:44 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10547"; a="733171063" X-IronPort-AV: E=Sophos;i="5.96,207,1665471600"; d="scan'208";a="733171063" Received: from djiang5-desk3.ch.intel.com ([143.182.136.137]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Nov 2022 15:12:44 -0800 Subject: [RFC PATCH 5/8] cxl: create emulated cxl_hdm for devices that do not have HDM decoders From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com, rrichter@amd.com, terry.bowman@amd.com Date: Wed, 30 Nov 2022 16:12:43 -0700 Message-ID: <166984996390.2805382.1358232383903012041.stgit@djiang5-desk3.ch.intel.com> In-Reply-To: <166984987659.2805382.17264896576424996856.stgit@djiang5-desk3.ch.intel.com> References: <166984987659.2805382.17264896576424996856.stgit@djiang5-desk3.ch.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org CXL rev3 spec 8.1.3 RCDs may not have HDM register blocks. Create a fake HDM with information from the CXL PCIe DVSEC registers. The decoder count will be set to the HDM count retrieved from the DVSEC cap register. Signed-off-by: Dave Jiang --- drivers/cxl/core/hdm.c | 27 ++++++++++++++++++++++++++- drivers/cxl/core/pci.c | 9 ++++++--- drivers/cxl/cxl.h | 3 ++- drivers/cxl/cxlmem.h | 1 + drivers/cxl/port.c | 2 +- 5 files changed, 36 insertions(+), 6 deletions(-) diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c index 9773a5efaefd..3a9e9b854587 100644 --- a/drivers/cxl/core/hdm.c +++ b/drivers/cxl/core/hdm.c @@ -96,11 +96,31 @@ static void __iomem *map_hdm_decoder_regs(struct cxl_port *port, return crb + map.hdm_decoder.offset; } +static struct cxl_hdm *devm_cxl_setup_emulated_hdm(struct cxl_port *port, + struct cxl_endpoint_dvsec_info *info) +{ + struct device *dev = &port->dev; + struct cxl_hdm *cxlhdm; + + if (!info->mem_enabled) + return ERR_PTR(-ENODEV); + + cxlhdm = devm_kzalloc(dev, sizeof(*cxlhdm), GFP_KERNEL); + if (!cxlhdm) + return ERR_PTR(-ENOMEM); + + cxlhdm->port = port; + cxlhdm->decoder_count = info->ranges; + dev_set_drvdata(&port->dev, cxlhdm); + return cxlhdm; +} + /** * devm_cxl_setup_hdm - map HDM decoder component registers * @port: cxl_port to map */ -struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port) +struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port, + struct cxl_endpoint_dvsec_info *info) { struct device *dev = &port->dev; void __iomem *crb, *hdm; @@ -111,9 +131,14 @@ struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port) return ERR_PTR(-ENOMEM); cxlhdm->port = port; + if (port->parent_dport->rch) + cxlhdm->rcd = true; crb = devm_cxl_iomap_block(dev, port->component_reg_phys, CXL_COMPONENT_REG_BLOCK_SIZE); if (!crb) { + if (info->mem_enabled) + return devm_cxl_setup_emulated_hdm(port, info); + dev_err(dev, "No component registers mapped\n"); return ERR_PTR(-ENXIO); } diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 5e44fe23fa76..009c11b43303 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -374,16 +374,19 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, struct cxl_port *port = cxlhdm->port; struct cxl_port *root; int i, rc, allowed; - u32 global_ctrl; + u32 global_ctrl = 0; - global_ctrl = readl(hdm + CXL_HDM_DECODER_CTRL_OFFSET); + if (hdm) + global_ctrl = readl(hdm + CXL_HDM_DECODER_CTRL_OFFSET); /* * If the HDM Decoder Capability is already enabled then assume * that some other agent like platform firmware set it up. */ - if (global_ctrl & CXL_HDM_DECODER_ENABLE) + if (global_ctrl & CXL_HDM_DECODER_ENABLE || (!hdm && info->mem_enabled)) return devm_cxl_enable_mem(&port->dev, cxlds); + else if (!hdm) + return -ENODEV; root = to_cxl_port(port->dev.parent); while (!is_cxl_root(root) && is_cxl_port(root->dev.parent)) diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 3fe1043a7796..b2eea921bd5f 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -604,7 +604,8 @@ struct cxl_endpoint_dvsec_info { }; struct cxl_hdm; -struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port); +struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port, + struct cxl_endpoint_dvsec_info *info); int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm, struct cxl_endpoint_dvsec_info *info); int devm_cxl_add_passthrough_decoder(struct cxl_port *port); diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index fd8ed573fbf9..6d477590559c 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -402,6 +402,7 @@ struct cxl_hdm { unsigned int target_count; unsigned int interleave_mask; struct cxl_port *port; + bool rcd; }; struct seq_file; diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index 3dedf3dd534a..910ebfc19a45 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -55,7 +55,7 @@ static int cxl_port_probe(struct device *dev) return rc; } - cxlhdm = devm_cxl_setup_hdm(port); + cxlhdm = devm_cxl_setup_hdm(port, &info); if (IS_ERR(cxlhdm)) return PTR_ERR(cxlhdm); From patchwork Wed Nov 30 23:12:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13060679 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5C58C4321E for ; Wed, 30 Nov 2022 23:21:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229828AbiK3XVA (ORCPT ); Wed, 30 Nov 2022 18:21:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42404 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230227AbiK3XUm (ORCPT ); Wed, 30 Nov 2022 18:20:42 -0500 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA810B0328 for ; Wed, 30 Nov 2022 15:13:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669850005; x=1701386005; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=er5rcaLuDpbnT7fs/EuI+OwZZrxSFUterqYzVlsiiJc=; b=CddgDNSQwwFWzrV8GFo5zfbnH9fc9Q8sqyQwnyQKqnEsCKEMcy+5DrEC NMGtouo/wmj02z9WDSNC/8wKDLVfyXClecnrf8WCsXSUQrjAI0nI4eVXp yiT7WikK2qRCFF+LdEtk/T9khxRJ17ijuPGmShgARXpwuaNbvRX8utnV/ TFgz+lKH0v4XSlXnauv+BOWEFGY8MWgG9DNWdvn+4gAGYmUuHv2X3M8rB XdBthF/1utvZTlf0hBOQs82UFJ4ak8XSZh1ERLqtQ+6Xb676z0ysZJiAJ mNvo9Y5q/CmFYTBNTH2sU8dZ3/CA01FBFo3PLvQUrPtLpc+FoHZG+mllH g==; X-IronPort-AV: E=McAfee;i="6500,9779,10547"; a="379812064" X-IronPort-AV: E=Sophos;i="5.96,207,1665471600"; d="scan'208";a="379812064" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Nov 2022 15:12:50 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10547"; a="733171098" X-IronPort-AV: E=Sophos;i="5.96,207,1665471600"; d="scan'208";a="733171098" Received: from djiang5-desk3.ch.intel.com ([143.182.136.137]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Nov 2022 15:12:49 -0800 Subject: [RFC PATCH 6/8] cxl: create emulated decoders for devices without HDM decoders From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com, rrichter@amd.com, terry.bowman@amd.com Date: Wed, 30 Nov 2022 16:12:49 -0700 Message-ID: <166984996963.2805382.2390204746333079984.stgit@djiang5-desk3.ch.intel.com> In-Reply-To: <166984987659.2805382.17264896576424996856.stgit@djiang5-desk3.ch.intel.com> References: <166984987659.2805382.17264896576424996856.stgit@djiang5-desk3.ch.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org CXL rev3.0 spec 8.1.3 RCDs may not have HDM register blocks. Create fake decoders based on CXL PCIe DVSEC registers. The DVSEC Range Regiters provide the memory range for these decoder structs. For the RCD, there can be up to 2 decoders depending on the DVSEC Capability register HDM_count. Signed-off-by: Dave Jiang --- drivers/cxl/core/hdm.c | 59 +++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 50 insertions(+), 9 deletions(-) diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c index 3a9e9b854587..60b6c141f514 100644 --- a/drivers/cxl/core/hdm.c +++ b/drivers/cxl/core/hdm.c @@ -721,6 +721,29 @@ static int cxl_setup_hdm_decoder_from_dvsec(struct cxl_port *port, return 0; } +static int init_emulated_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, + struct cxl_endpoint_dvsec_info *info, int which) +{ + if (info->dvsec_range[which].start == CXL_RESOURCE_NONE) + return -ENODEV; + + cxld->hpa_range = (struct range) { + .start = info->dvsec_range[which].start, + .end = info->dvsec_range[which].end, + }; + + cxld->flags = CXL_DECODER_F_ENABLE | CXL_DECODER_F_LOCK; + cxld->target_type = CXL_DECODER_ACCELERATOR; + if (cxld->id != port->commit_end + 1) { + dev_warn(&port->dev, + "decoder%d.%d: Committed out of order\n", + port->id, cxld->id); + return -ENXIO; + } + port->commit_end = cxld->id; + return 0; +} + static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, int *target_map, void __iomem *hdm, int which, u64 *dpa_base, struct cxl_endpoint_dvsec_info *info) @@ -739,6 +762,13 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, if (is_endpoint_decoder(&cxld->dev)) cxled = to_cxl_endpoint_decoder(&cxld->dev); + if (!hdm) { + if (cxled) + return init_emulated_hdm_decoder(port, cxld, info, which); + else + return -EINVAL; + } + ctrl = readl(hdm + CXL_HDM_DECODER0_CTRL_OFFSET(which)); base = ioread64_hi_lo(hdm + CXL_HDM_DECODER0_BASE_LOW_OFFSET(which)); size = ioread64_hi_lo(hdm + CXL_HDM_DECODER0_SIZE_LOW_OFFSET(which)); @@ -832,19 +862,15 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, return 0; } -/** - * devm_cxl_enumerate_decoders - add decoder objects per HDM register set - * @cxlhdm: Structure to populate with HDM capabilities - */ -int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm, - struct cxl_endpoint_dvsec_info *info) +static void cxl_settle_decoders(struct cxl_hdm *cxlhdm) { void __iomem *hdm = cxlhdm->regs.hdm_decoder; - struct cxl_port *port = cxlhdm->port; - int i, committed; - u64 dpa_base = 0; + int committed, i; u32 ctrl; + if (!hdm) + return; + /* * Since the register resource was recently claimed via request_region() * be careful about trusting the "not-committed" status until the commit @@ -861,6 +887,21 @@ int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm, /* ensure that future checks of committed can be trusted */ if (committed != cxlhdm->decoder_count) msleep(20); +} + +/** + * devm_cxl_enumerate_decoders - add decoder objects per HDM register set + * @cxlhdm: Structure to populate with HDM capabilities + */ +int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm, + struct cxl_endpoint_dvsec_info *info) +{ + void __iomem *hdm = cxlhdm->regs.hdm_decoder; + struct cxl_port *port = cxlhdm->port; + int i; + u64 dpa_base = 0; + + cxl_settle_decoders(cxlhdm); for (i = 0; i < cxlhdm->decoder_count; i++) { int target_map[CXL_DECODER_MAX_INTERLEAVE] = { 0 }; From patchwork Wed Nov 30 23:12:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13060680 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43F9CC47088 for ; Wed, 30 Nov 2022 23:21:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229744AbiK3XVB (ORCPT ); Wed, 30 Nov 2022 18:21:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229770AbiK3XUn (ORCPT ); Wed, 30 Nov 2022 18:20:43 -0500 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8BAEBB0309 for ; Wed, 30 Nov 2022 15:13:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669850008; x=1701386008; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=3+37oeQ0PtgjSuwYZMHdr1Unz7CY6Ph996OtgGAlrBo=; b=Lzbj3nuonzo4K7nS/XmjytYR8ANQ3G9q7H5EVZCorTKP3Nq3Lc3zo/fd F+LkRINmc+EOHVu37g1cdnCMoVlSCBb96DxHtkd0RuC81vWFWSLVsCZGH lRr+oFAWCcaZiGVEJS8zpm6R8DajMcd1dWRd/XMPf1d6i2hYO+cfP4vh1 FxVj0ALE2qs2yBVraswxoLSFfGdYGVjwjTgEsZSHCYgg5GQ8Vu9JfAaS3 2ij+cOCSPrjkpdXBV0xlzk6tmIAvPTRfuCPNYPy5T6PV+W9KFQrILBxVt StKEaxRe0JqYXbZ06dvTstCm323Qgf2FLRBhQgjLsO5TihU1IIpfEJd3I w==; X-IronPort-AV: E=McAfee;i="6500,9779,10547"; a="379812083" X-IronPort-AV: E=Sophos;i="5.96,207,1665471600"; d="scan'208";a="379812083" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Nov 2022 15:12:56 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10547"; a="733171128" X-IronPort-AV: E=Sophos;i="5.96,207,1665471600"; d="scan'208";a="733171128" Received: from djiang5-desk3.ch.intel.com ([143.182.136.137]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Nov 2022 15:12:55 -0800 Subject: [RFC PATCH 7/8] cxl: suppress component register discovery failure warning for RCD From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com, rrichter@amd.com, terry.bowman@amd.com Date: Wed, 30 Nov 2022 16:12:55 -0700 Message-ID: <166984997536.2805382.16376618769262720267.stgit@djiang5-desk3.ch.intel.com> In-Reply-To: <166984987659.2805382.17264896576424996856.stgit@djiang5-desk3.ch.intel.com> References: <166984987659.2805382.17264896576424996856.stgit@djiang5-desk3.ch.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org For an RCD, it is expected that component register won't be discovered by cxl_pci. Suppress warning if RCD. Signed-off-by: Dave Jiang --- drivers/cxl/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 73ff6c33a0c0..d808da838909 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -486,7 +486,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) */ cxlds->component_reg_phys = CXL_RESOURCE_NONE; rc = cxl_setup_regs(pdev, CXL_REGLOC_RBI_COMPONENT, &map); - if (rc) + if (rc && !cxlds->rcd) dev_warn(&pdev->dev, "No component registers (%d)\n", rc); cxlds->component_reg_phys = cxl_regmap_to_base(pdev, &map); From patchwork Wed Nov 30 23:13:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13060681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2E7FC4321E for ; Wed, 30 Nov 2022 23:21:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229943AbiK3XVG (ORCPT ); Wed, 30 Nov 2022 18:21:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60066 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229918AbiK3XUs (ORCPT ); Wed, 30 Nov 2022 18:20:48 -0500 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F602B033C for ; Wed, 30 Nov 2022 15:13:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669850010; x=1701386010; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=GK2UGDGHO1BsuakgcQiTXroGCuH/wCMhZZtPFwFmKPg=; b=AB3A57K6JYChqpu0gk5nu3ouOvichYQSL1sKPSXNrf2O5xeMh7DDrm+R 95Z4iUiKJBtp5bWmDxbeKV4Tf+UUt29/ycaDUuwKiy12p6JpwPwQXoW4E 35nyogzzZBySTXTri3R2y53PxbxpMvPNO5gg51rBPLAQgc00hwAPsoRjy 173bNp3+UBWctmeNh6ExoU4iHjYl6VPAoC406Z7A0DwQuny8Gc1OGDFKb 1ggvfa0Ol8pUh/iZ0D+Hmj+m+8NYBFgvbMxp0WD4JuCXoxD94Kk/1lpoA y4kV26O1o34BziPWXIx2dq8mX24PSFUsQcjn2QPJDpsXok8ixbuNvE9Ur A==; X-IronPort-AV: E=McAfee;i="6500,9779,10547"; a="379812112" X-IronPort-AV: E=Sophos;i="5.96,207,1665471600"; d="scan'208";a="379812112" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Nov 2022 15:13:01 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10547"; a="733171169" X-IronPort-AV: E=Sophos;i="5.96,207,1665471600"; d="scan'208";a="733171169" Received: from djiang5-desk3.ch.intel.com ([143.182.136.137]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Nov 2022 15:13:01 -0800 Subject: [RFC PATCH 8/8] cxl: remove locked check for dvsec_range_allowed() From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com, rrichter@amd.com, terry.bowman@amd.com Date: Wed, 30 Nov 2022 16:13:01 -0700 Message-ID: <166984998110.2805382.13413879869123517058.stgit@djiang5-desk3.ch.intel.com> In-Reply-To: <166984987659.2805382.17264896576424996856.stgit@djiang5-desk3.ch.intel.com> References: <166984987659.2805382.17264896576424996856.stgit@djiang5-desk3.ch.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org There is no reason that the CFMWS will always set the "Fixed Device Configuration" bit in the "Window Restrictions" field. Remove the CXL_DECODER_F_LOCK check. Signed-off-by: Dave Jiang --- drivers/cxl/core/pci.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 009c11b43303..d92fb757e495 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -229,8 +229,6 @@ static int dvsec_range_allowed(struct device *dev, void *arg) cxld = to_cxl_decoder(dev); - if (!(cxld->flags & CXL_DECODER_F_LOCK)) - return 0; if (!(cxld->flags & CXL_DECODER_F_RAM)) return 0;