From patchwork Tue Sep 17 01:39:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Honggang LI X-Patchwork-Id: 11147985 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 78BAE1599 for ; Tue, 17 Sep 2019 01:39:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 52D6320880 for ; Tue, 17 Sep 2019 01:39:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730648AbfIQBjh (ORCPT ); Mon, 16 Sep 2019 21:39:37 -0400 Received: from mx1.redhat.com ([209.132.183.28]:51272 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728878AbfIQBjh (ORCPT ); Mon, 16 Sep 2019 21:39:37 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 26C9CC057867; Tue, 17 Sep 2019 01:39:37 +0000 (UTC) Received: from dhcp-128-227.nay.redhat.com (unknown [10.66.128.227]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3F0FD5C1D6; Tue, 17 Sep 2019 01:39:34 +0000 (UTC) From: Honggang LI To: bvanassche@acm.org, dledford@redhat.com, jgg@ziepe.ca, linux-rdma@vger.kernel.org Cc: Honggang Li Subject: [PATCH 1/2] IB/srp: Add parse function for maximum initiator to target IU size Date: Tue, 17 Sep 2019 09:39:04 +0800 Message-Id: <20190917013905.2600-1-honli@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Tue, 17 Sep 2019 01:39:37 +0000 (UTC) Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Honggang Li According to SRP specifications 'srp-r16a' and 'srp2r06', IOControllerProfile attributes for SRP target port include the maximum initiator to target IU size. SRP connection daemons, such as srp_daemon, can get the value from subnet manager. The SRP connection daemon can pass this value to kernel. This patch add parse function for it. Upstream commit [1] enables the kernel parameter, 'use_imm_data', by default. [1] also use (8 * 1024) as the default value for kernel parameter 'max_imm_data'. With those default values, the maximum initiator to target IU size will be 8260. In case the SRPT modules, which include the in-tree 'ib_srpt.ko' module, do not support SRP-2 'immediate data' feature, the default maximum initiator to target IU size is significantly samller than 8260. For 'ib_srpt.ko' module, which built from source before [2], the default maximum initiator to target IU is 2116. [1] introduces a regression issue for old srp target with default kernel parameters, as the connection will be reject because of too large maximum initiator to target IU size. [1] 882981f4a411 RDMA/srp: Add support for immediate data [2] 5dabcd0456d7 RDMA/srpt: Add support for immediate data Signed-off-by: Honggang Li --- drivers/infiniband/ulp/srp/ib_srp.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c index b5960351bec0..2eadb038b257 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.c +++ b/drivers/infiniband/ulp/srp/ib_srp.c @@ -75,6 +75,7 @@ static bool prefer_fr = true; static bool register_always = true; static bool never_register; static int topspin_workarounds = 1; +static uint32_t srp_opt_max_it_iu_size; module_param(srp_sg_tablesize, uint, 0444); MODULE_PARM_DESC(srp_sg_tablesize, "Deprecated name for cmd_sg_entries"); @@ -3411,6 +3412,7 @@ enum { SRP_OPT_IP_SRC = 1 << 15, SRP_OPT_IP_DEST = 1 << 16, SRP_OPT_TARGET_CAN_QUEUE= 1 << 17, + SRP_OPT_MAX_IT_IU_SIZE = 1 << 18, }; static unsigned int srp_opt_mandatory[] = { @@ -3443,6 +3445,7 @@ static const match_table_t srp_opt_tokens = { { SRP_OPT_QUEUE_SIZE, "queue_size=%d" }, { SRP_OPT_IP_SRC, "src=%s" }, { SRP_OPT_IP_DEST, "dest=%s" }, + { SRP_OPT_MAX_IT_IU_SIZE, "max_it_iu_size=%d" }, { SRP_OPT_ERR, NULL } }; @@ -3736,6 +3739,14 @@ static int srp_parse_options(struct net *net, const char *buf, target->tl_retry_count = token; break; + case SRP_OPT_MAX_IT_IU_SIZE: + if (match_int(args, &token) || token < 0) { + pr_warn("bad maximum initiator to target IU size '%s'\n", p); + goto out; + } + srp_opt_max_it_iu_size = token; + break; + default: pr_warn("unknown parameter or missing value '%s' in target creation request\n", p); From patchwork Tue Sep 17 01:39:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Honggang LI X-Patchwork-Id: 11147987 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EA2CB1599 for ; Tue, 17 Sep 2019 01:39:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D1FB020880 for ; Tue, 17 Sep 2019 01:39:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728878AbfIQBjn (ORCPT ); Mon, 16 Sep 2019 21:39:43 -0400 Received: from mx1.redhat.com ([209.132.183.28]:53860 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728842AbfIQBjn (ORCPT ); Mon, 16 Sep 2019 21:39:43 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0A5D37BDA1; Tue, 17 Sep 2019 01:39:43 +0000 (UTC) Received: from dhcp-128-227.nay.redhat.com (unknown [10.66.128.227]) by smtp.corp.redhat.com (Postfix) with ESMTP id 30A3C5C1D6; Tue, 17 Sep 2019 01:39:40 +0000 (UTC) From: Honggang LI To: bvanassche@acm.org, dledford@redhat.com, jgg@ziepe.ca, linux-rdma@vger.kernel.org Cc: Honggang Li Subject: [PATCH 2/2] RDMA/SRP: calculate max_it_iu_size if remote max_it_iu length available Date: Tue, 17 Sep 2019 09:39:05 +0800 Message-Id: <20190917013905.2600-2-honli@redhat.com> In-Reply-To: <20190917013905.2600-1-honli@redhat.com> References: <20190917013905.2600-1-honli@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Tue, 17 Sep 2019 01:39:43 +0000 (UTC) Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Honggang Li The default maximum immediate size is too big for old srp clients, which without immediate data support. According to the SRP and SRP-2 specifications, the IOControllerProfile attributes for SRP target ports contains the maximum initiator to target iu length. The maximum initiator to target iu length can be get from the subnet manager, such as opensm and opafm. We should calculate the max_it_iu_size instead of the default value, when remote maximum initiator to target iu length available. Signed-off-by: Honggang Li --- drivers/infiniband/ulp/srp/ib_srp.c | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c index 2eadb038b257..3b0ca663c6cf 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.c +++ b/drivers/infiniband/ulp/srp/ib_srp.c @@ -76,6 +76,7 @@ static bool register_always = true; static bool never_register; static int topspin_workarounds = 1; static uint32_t srp_opt_max_it_iu_size; +static unsigned int srp_max_imm_data; module_param(srp_sg_tablesize, uint, 0444); MODULE_PARM_DESC(srp_sg_tablesize, "Deprecated name for cmd_sg_entries"); @@ -138,9 +139,9 @@ module_param_named(use_imm_data, srp_use_imm_data, bool, 0644); MODULE_PARM_DESC(use_imm_data, "Whether or not to request permission to use immediate data during SRP login."); -static unsigned int srp_max_imm_data = 8 * 1024; -module_param_named(max_imm_data, srp_max_imm_data, uint, 0644); -MODULE_PARM_DESC(max_imm_data, "Maximum immediate data size."); +static unsigned int srp_default_max_imm_data = 8 * 1024; +module_param_named(max_imm_data, srp_default_max_imm_data, uint, 0644); +MODULE_PARM_DESC(max_imm_data, "Default maximum immediate data size."); static unsigned ch_count; module_param(ch_count, uint, 0444); @@ -1369,9 +1370,20 @@ static uint32_t srp_max_it_iu_len(int cmd_sg_cnt, bool use_imm_data) sizeof(struct srp_indirect_buf) + cmd_sg_cnt * sizeof(struct srp_direct_buf); - if (use_imm_data) - max_iu_len = max(max_iu_len, SRP_IMM_DATA_OFFSET + - srp_max_imm_data); + if (use_imm_data) { + if (srp_opt_max_it_iu_size == 0) { + srp_max_imm_data = srp_default_max_imm_data; + max_iu_len = max(max_iu_len, + SRP_IMM_DATA_OFFSET + srp_max_imm_data); + } + else { + srp_max_imm_data = + srp_opt_max_it_iu_size - SRP_IMM_DATA_OFFSET; + max_iu_len = srp_opt_max_it_iu_size; + } + pr_debug("srp_max_imm_data = %d, max_iu_len = %d\n", + srp_max_imm_data, max_iu_len); + } return max_iu_len; } @@ -3829,6 +3841,8 @@ static ssize_t srp_create_target(struct device *dev, if (ret < 0) goto put; + srp_opt_max_it_iu_size = 0; + ret = srp_parse_options(target->net, buf, target); if (ret) goto out;