From patchwork Thu Aug 11 23:56:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 12941817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E09FCC25B06 for ; Thu, 11 Aug 2022 23:56:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229594AbiHKX46 (ORCPT ); Thu, 11 Aug 2022 19:56:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55368 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234910AbiHKX45 (ORCPT ); Thu, 11 Aug 2022 19:56:57 -0400 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 52D7B9FABA for ; Thu, 11 Aug 2022 16:56:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1660262216; x=1691798216; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=veC3xzGemXB2uVKBGQIWOkScK2u1aIifkusXF1Q1Dl4=; b=mZLikFXhyL+YykOZpt5ZglxK6y0HT0Bj9DwGL0xnhr74cKDBDlybwnhS c1XrYnV47AlTAAfETEd5m9pLX9Np04i4j3sNmAgK/f/fjgM/fqRA0jeGM pyhYXKrjKX8igmPq+H0oiQI0LzDGwTSbuLuIu3M4HeqDiHhHVFMvES5oH BQn/EkUl1BRtJDq3xG6O7tpKidk3C7JTZl4MQE2oFOw/mIc8elAUhha3c owFG+Xdsyq13dpTcByxccd9TwW1ZNHr4ap/c+TORmGCf3GfPtWmUQ+TpI Ccr+0lzTPQKdPi1BQhWsdGAhI+AGg0Fvg4lFpxtfj621K/gFXVtr4yzCS w==; X-IronPort-AV: E=McAfee;i="6400,9594,10436"; a="290241412" X-IronPort-AV: E=Sophos;i="5.93,231,1654585200"; d="scan'208";a="290241412" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Aug 2022 16:56:55 -0700 X-IronPort-AV: E=Sophos;i="5.93,231,1654585200"; d="scan'208";a="556329216" Received: from djiang5-desk4.jf.intel.com ([10.165.157.96]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Aug 2022 16:56:55 -0700 Subject: [PATCH v2 1/3] cxl: Add check for result of interleave ways plus granularity combo From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Thu, 11 Aug 2022 16:56:55 -0700 Message-ID: <166026221529.1454405.7141583168966440657.stgit@djiang5-desk4.jf.intel.com> In-Reply-To: <166026184968.1454405.494690416353090765.stgit@djiang5-desk4.jf.intel.com> References: <166026184968.1454405.494690416353090765.stgit@djiang5-desk4.jf.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Add a helper function to check the combination of interleave ways and interleave granularity together is sane against the interleave mask from the HDM decoder. Add the check to cxl_region_attach() to make sure the region config is sane. Add the check to cxl_port_setup_targets() to make sure the port setup config is also sane. Calculation refers to CXL spec v3 8.2.4.19.13 implementation note #3. Signed-off-by: Dave Jiang Reviewed-by: Dan Williams --- drivers/cxl/core/region.c | 11 ++++++++++- drivers/cxl/cxlmem.h | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+), 1 deletion(-) diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index cf5d5811fe4c..6f77ad22d47d 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -1081,6 +1081,10 @@ static int cxl_port_setup_targets(struct cxl_port *port, return rc; } + rc = cxl_interleave_capable(port, &cxlr->dev, iw, ig); + if (rc) + return rc; + cxld->interleave_ways = iw; cxld->interleave_granularity = ig; dev_dbg(&cxlr->dev, "%s:%s iw: %d ig: %d\n", dev_name(port->uport), @@ -1218,6 +1222,12 @@ static int cxl_region_attach(struct cxl_region *cxlr, return -EBUSY; } + ep_port = cxled_to_port(cxled); + rc = cxl_interleave_capable(ep_port, &cxlr->dev, p->interleave_ways, + p->interleave_granularity); + if (rc) + return rc; + for (i = 0; i < p->interleave_ways; i++) { struct cxl_endpoint_decoder *cxled_target; struct cxl_memdev *cxlmd_target; @@ -1236,7 +1246,6 @@ static int cxl_region_attach(struct cxl_region *cxlr, } } - ep_port = cxled_to_port(cxled); root_port = cxlrd_to_port(cxlrd); dport = cxl_find_dport_by_dev(root_port, ep_port->host_bridge); if (!dport) { diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index 88e3a8e54b6a..625fce0b6c2c 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -401,6 +401,43 @@ struct cxl_hdm { struct cxl_port *port; }; +static inline int cxl_interleave_capable(struct cxl_port *port, + struct device *dev, + int ways, int granularity) +{ + struct cxl_hdm *cxlhdm = dev_get_drvdata(&port->dev); + unsigned int addr_mask; + u16 eig; + u8 eiw; + int rc; + + rc = granularity_to_cxl(granularity, &eig); + if (rc) + return rc; + + rc = ways_to_cxl(ways, &eiw); + if (rc) + return rc; + + if (eiw == 0) + return 0; + + if (is_power_of_2(eiw)) + addr_mask = GENMASK(eig + 8 + eiw - 1, eig + 8); + else + addr_mask = GENMASK((eig + eiw) / 3 - 1, eig + 8); + + if (~cxlhdm->interleave_mask & addr_mask) { + dev_warn(dev, + "%s:%s interleave (eig: %d eiw: %d mask: %#x) exceed cap (mask: %#x)\n", + dev_name(port->uport), dev_name(&port->dev), eig, eiw, + cxlhdm->interleave_mask, addr_mask); + return -EINVAL; + } + + return 0; +} + struct seq_file; struct dentry *cxl_debugfs_create_dir(const char *dir); void cxl_dpa_debug(struct seq_file *file, struct cxl_dev_state *cxlds); From patchwork Thu Aug 11 23:57:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 12941818 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A02AC25B06 for ; Thu, 11 Aug 2022 23:57:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235762AbiHKX5C (ORCPT ); Thu, 11 Aug 2022 19:57:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234910AbiHKX5C (ORCPT ); Thu, 11 Aug 2022 19:57:02 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45D5F9FABA for ; Thu, 11 Aug 2022 16:57:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1660262221; x=1691798221; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=m435kbwUzK8QRzpLpPX1eATLQiS+A/dnSffsGUfRhRc=; b=AOkZbCvDchkPid05mbqdH3F6+2QDRoOOjl2Y4oZEENv/8ZKh0BIQp2Mf /AcdPcgN3wphmTVdVL8q+R7Jz2yCV83iajhP7TeOXmOvFPMDyp8cBS1rx V7pieAAuUtMLAxUBqNv00WPWr3/fxM48+eAzwkDQi9YOErB/6wAjA+Jnl AbmUSKdYg2AwFKIzDTjzbu6zKLiP2DGh4ppPKIckkjOeZd2H6LcN3PMZU JmfdprAhM+k2V/77EA3xXxc9ZqPixGfRk7xn/iSeT9VmJlbWzlJpGxoTU q/B3dbHcTVcalR+RGODNwqn4pOZ3eGONhVKNV94JWjtS4tPd340gdrO5i Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10436"; a="291487411" X-IronPort-AV: E=Sophos;i="5.93,231,1654585200"; d="scan'208";a="291487411" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Aug 2022 16:57:00 -0700 X-IronPort-AV: E=Sophos;i="5.93,231,1654585200"; d="scan'208";a="634441700" Received: from djiang5-desk4.jf.intel.com ([10.165.157.96]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Aug 2022 16:57:00 -0700 Subject: [PATCH v2 2/3] cxl: Add CXL spec v3.0 interleave support From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Thu, 11 Aug 2022 16:57:00 -0700 Message-ID: <166026222055.1454405.5853343066246585992.stgit@djiang5-desk4.jf.intel.com> In-Reply-To: <166026184968.1454405.494690416353090765.stgit@djiang5-desk4.jf.intel.com> References: <166026184968.1454405.494690416353090765.stgit@djiang5-desk4.jf.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org CXL spec v3.0 added 2 CAP bits to the CXL HDM Decoder Capability Register. CXL spec v3.0 8.2.4.19.1. Bit 11 indicates that 3, 6, and 12 way interleave is capable. Bit 12 indicates that 16 way interleave is capable. Add code to parse_hdm_decoder_caps() to cache those new bits. Add check in cxl_interleave_verify() call to make sure those CAP bits matches the passed in interleave value. Signed-off-by: Dave Jiang Reviewed-by: Dan Williams --- drivers/cxl/core/hdm.c | 6 ++++++ drivers/cxl/cxl.h | 2 ++ drivers/cxl/cxlmem.h | 13 +++++++++++++ 3 files changed, 21 insertions(+) diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c index 8143e2615957..0baf3c4820a5 100644 --- a/drivers/cxl/core/hdm.c +++ b/drivers/cxl/core/hdm.c @@ -80,6 +80,12 @@ static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm) cxlhdm->interleave_mask |= GENMASK(11, 8); if (FIELD_GET(CXL_HDM_DECODER_INTERLEAVE_14_12, hdm_cap)) cxlhdm->interleave_mask |= GENMASK(14, 12); + + cxlhdm->interleave_cap = CXL_HDM_INTERLEAVE_CAP_DEFAULT; + if (FIELD_GET(CXL_HDM_DECODER_INTERLEAVE_3_6_12_WAY, hdm_cap)) + cxlhdm->interleave_cap |= CXL_HDM_INTERLEAVE_CAP_3_6_12; + if (FIELD_GET(CXL_HDM_DECODER_INTERLEAVE_16_WAY, hdm_cap)) + cxlhdm->interleave_cap |= CXL_HDM_INTERLEAVE_CAP_16; } static void __iomem *map_hdm_decoder_regs(struct cxl_port *port, diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index bc604b7e44fb..105d814941e7 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -42,6 +42,8 @@ #define CXL_HDM_DECODER_TARGET_COUNT_MASK GENMASK(7, 4) #define CXL_HDM_DECODER_INTERLEAVE_11_8 BIT(8) #define CXL_HDM_DECODER_INTERLEAVE_14_12 BIT(9) +#define CXL_HDM_DECODER_INTERLEAVE_3_6_12_WAY BIT(11) +#define CXL_HDM_DECODER_INTERLEAVE_16_WAY BIT(12) #define CXL_HDM_DECODER_CTRL_OFFSET 0x4 #define CXL_HDM_DECODER_ENABLE BIT(1) #define CXL_HDM_DECODER0_BASE_LOW_OFFSET(i) (0x20 * (i) + 0x10) diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index 625fce0b6c2c..ebd645d8bbb4 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -393,14 +393,24 @@ static inline void cxl_mem_active_dec(void) } #endif +#define CXL_HDM_INTERLEAVE_CAP_DEFAULT BIT(1) | BIT(2) | BIT(4) | BIT(8) +#define CXL_HDM_INTERLEAVE_CAP_3_6_12 BIT(3) | BIT(6) | BIT(12) +#define CXL_HDM_INTERLEAVE_CAP_16 BIT(16) + struct cxl_hdm { struct cxl_component_regs regs; unsigned int decoder_count; unsigned int target_count; unsigned int interleave_mask; + unsigned long interleave_cap; struct cxl_port *port; }; +static inline bool valid_interleave_ways(struct cxl_hdm *cxlhdm, u8 iw) +{ + return test_bit(iw, &cxlhdm->interleave_cap); +} + static inline int cxl_interleave_capable(struct cxl_port *port, struct device *dev, int ways, int granularity) @@ -422,6 +432,9 @@ static inline int cxl_interleave_capable(struct cxl_port *port, if (eiw == 0) return 0; + if (!valid_interleave_ways(cxlhdm, ways)) + return -EINVAL; + if (is_power_of_2(eiw)) addr_mask = GENMASK(eig + 8 + eiw - 1, eig + 8); else From patchwork Thu Aug 11 23:57:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 12941819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D479C25B06 for ; Thu, 11 Aug 2022 23:57:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235980AbiHKX5H (ORCPT ); Thu, 11 Aug 2022 19:57:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234910AbiHKX5G (ORCPT ); Thu, 11 Aug 2022 19:57:06 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2A0479E2D9 for ; Thu, 11 Aug 2022 16:57:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1660262226; x=1691798226; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=sckOKfrLsCaMpezkbRORhs3sScWR3PH/m3WaHEilYCs=; b=ZZNLDfP3KoAyJD84751FhmnHc6zETzN06pkzGe0Hv557So7XTNkERWzj koV2JX9hQuBAAoykhxXUObMRGFiruRLh8CSYCC8KYUbzVx0JnG59oUGrJ eKBneZ8lJbYMQE75lELREnfqfdYjfnXpA+zVOVMya3K7FzJlkfgcjQyfo YaMSEWUfcP/JjTVZ4svW2gKFsy+lzQj1tjp9WnMiYLQSpoSmNfYMqFwnI NKa3ft3oxc+bMz3wnf9rDIdk7sjVOK4l4IC/58VpvjPIVdcBgnG4mM0Fy NciKAUQfT1ShheN9kviLa8uCJBvbOuKImaVAq+jJh21b5IUtSZ+UdtcYC g==; X-IronPort-AV: E=McAfee;i="6400,9594,10436"; a="271260955" X-IronPort-AV: E=Sophos;i="5.93,231,1654585200"; d="scan'208";a="271260955" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Aug 2022 16:57:05 -0700 X-IronPort-AV: E=Sophos;i="5.93,231,1654585200"; d="scan'208";a="638714328" Received: from djiang5-desk4.jf.intel.com ([10.165.157.96]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Aug 2022 16:57:05 -0700 Subject: [PATCH v2 3/3] tools/testing/cxl: Add interleave check support to mock cxl port device From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Thu, 11 Aug 2022 16:57:05 -0700 Message-ID: <166026222564.1454405.200833694547817544.stgit@djiang5-desk4.jf.intel.com> In-Reply-To: <166026184968.1454405.494690416353090765.stgit@djiang5-desk4.jf.intel.com> References: <166026184968.1454405.494690416353090765.stgit@djiang5-desk4.jf.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Attach the cxl mock hdm to the port device to allow cxl_interleave_verify() to check the interleave configuration. Set the interleave_mask as well to support the new verification code. Signed-off-by: Dave Jiang Reviewed-by: Dan Williams --- tools/testing/cxl/test/cxl.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tools/testing/cxl/test/cxl.c b/tools/testing/cxl/test/cxl.c index a072b2d3e726..3ce353a20b80 100644 --- a/tools/testing/cxl/test/cxl.c +++ b/tools/testing/cxl/test/cxl.c @@ -398,6 +398,9 @@ static struct cxl_hdm *mock_cxl_setup_hdm(struct cxl_port *port) return ERR_PTR(-ENOMEM); cxlhdm->port = port; + cxlhdm->interleave_mask = GENMASK(14, 8); + cxlhdm->interleave_cap = CXL_HDM_INTERLEAVE_CAP_DEFAULT; + dev_set_drvdata(&port->dev, cxlhdm); return cxlhdm; }