From patchwork Fri May 22 21:33:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 11566199 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 650301391 for ; Fri, 22 May 2020 21:33:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 580FC20757 for ; Fri, 22 May 2020 21:33:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731051AbgEVVdv (ORCPT ); Fri, 22 May 2020 17:33:51 -0400 Received: from mail-pj1-f65.google.com ([209.85.216.65]:38405 "EHLO mail-pj1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731033AbgEVVdv (ORCPT ); Fri, 22 May 2020 17:33:51 -0400 Received: by mail-pj1-f65.google.com with SMTP id t8so3488924pju.3 for ; Fri, 22 May 2020 14:33:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xcZXxuFmCrGCBtaA4BKkKSU9REt6gfh1XPUB7cdu8HU=; b=W6TvuHrgADdDFbQ9Ub6cAI4UyReBhY6OgECvGhAodVqGbXA77fUnToTEEdOVguAKTC W4f9b0YN2hI8KTr1lQ99bkZSwqAPOz9F2NDJOSbqrtTxnobuU0RdHh/Uc3vyyfoHJlix xSKSZXtlKVuMS7KHLhUHyCDIrniqdclgBvvO5QEaHDp181EeN2/fxuYvlbLW4tcpHSOi vG+hDA+agt4YqauyozTbclaMgXfYqh/0ViOT4hsorqtg5VeUVAsy0hDMEO2D+kAzLVm0 8Cw7UqSDnDg9IVv0QjI6KqKdHNLt88je7MY3Qg+ohYfxkaWYapiaT20bVxkK7+8uzPRh J0SQ== X-Gm-Message-State: AOAM532rNTj5mPEnyhICTWdO63zEBlomH1etjeLPUEfB+Daihv9LVCUZ 8qJXMl/PYQTnldlW5cAAMw8= X-Google-Smtp-Source: ABdhPJwsK0Krmpts479lx8JiktRDvTnyCGIS0HuEYlI4FQdJHc6WV4hrtPgGU4LepEhm4wVWvy/HZA== X-Received: by 2002:a17:90a:20cf:: with SMTP id f73mr6962954pjg.86.1590183229662; Fri, 22 May 2020 14:33:49 -0700 (PDT) Received: from localhost.localdomain ([2601:647:4000:d7:b874:274b:7df6:e61b]) by smtp.gmail.com with ESMTPSA id mn19sm7480755pjb.8.2020.05.22.14.33.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 May 2020 14:33:48 -0700 (PDT) From: Bart Van Assche To: Jason Gunthorpe Cc: Leon Romanovsky , Doug Ledford , linux-rdma@vger.kernel.org, Bart Van Assche , Laurence Oberman , Kamal Heib Subject: [PATCH 1/4] RDMA/srp: Make the channel count configurable per target Date: Fri, 22 May 2020 14:33:38 -0700 Message-Id: <20200522213341.16341-2-bvanassche@acm.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200522213341.16341-1-bvanassche@acm.org> References: <20200522213341.16341-1-bvanassche@acm.org> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Increase the flexibility of the SRP initiator driver by making the channel count configurable per target instead of only providing a kernel module parameter for configuring the channel count. Cc: Laurence Oberman Cc: Kamal Heib Signed-off-by: Bart Van Assche --- drivers/infiniband/ulp/srp/ib_srp.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c index 00b4f88b113e..d686c39710c0 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.c +++ b/drivers/infiniband/ulp/srp/ib_srp.c @@ -3424,6 +3424,7 @@ enum { SRP_OPT_IP_DEST = 1 << 16, SRP_OPT_TARGET_CAN_QUEUE= 1 << 17, SRP_OPT_MAX_IT_IU_SIZE = 1 << 18, + SRP_OPT_CH_COUNT = 1 << 19, }; static unsigned int srp_opt_mandatory[] = { @@ -3457,6 +3458,7 @@ static const match_table_t srp_opt_tokens = { { SRP_OPT_IP_SRC, "src=%s" }, { SRP_OPT_IP_DEST, "dest=%s" }, { SRP_OPT_MAX_IT_IU_SIZE, "max_it_iu_size=%d" }, + { SRP_OPT_CH_COUNT, "ch_count=%d", }, { SRP_OPT_ERR, NULL } }; @@ -3758,6 +3760,14 @@ static int srp_parse_options(struct net *net, const char *buf, target->max_it_iu_size = token; break; + case SRP_OPT_CH_COUNT: + if (match_int(args, &token) || token < 1) { + pr_warn("bad channel count %s\n", p); + goto out; + } + target->ch_count = token; + break; + default: pr_warn("unknown parameter or missing value '%s' in target creation request\n", p); @@ -3921,11 +3931,12 @@ static ssize_t srp_create_target(struct device *dev, goto out; ret = -ENOMEM; - target->ch_count = max_t(unsigned, num_online_nodes(), - min(ch_count ? : - min(4 * num_online_nodes(), - ibdev->num_comp_vectors), - num_online_cpus())); + if (target->ch_count == 0) + target->ch_count = max_t(unsigned, num_online_nodes(), + min(ch_count ? : + min(4 * num_online_nodes(), + ibdev->num_comp_vectors), + num_online_cpus())); target->ch = kcalloc(target->ch_count, sizeof(*target->ch), GFP_KERNEL); if (!target->ch) From patchwork Fri May 22 21:33:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 11566201 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2CE6B1391 for ; Fri, 22 May 2020 21:33:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2070A2078D for ; Fri, 22 May 2020 21:33:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731123AbgEVVdx (ORCPT ); Fri, 22 May 2020 17:33:53 -0400 Received: from mail-pj1-f65.google.com ([209.85.216.65]:54724 "EHLO mail-pj1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731033AbgEVVdw (ORCPT ); Fri, 22 May 2020 17:33:52 -0400 Received: by mail-pj1-f65.google.com with SMTP id s69so5537750pjb.4 for ; Fri, 22 May 2020 14:33:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4K9rHHICE3+inYoR2BYCilENf4wIRIaOf+lyZ8wD/kY=; b=TtoHtuDmnWNeWTX1udhbDJeqs1XnLz5WbCwaKo+Nv+c0JgnO7YhYeRtIUhWhUAlxhK ZwfQiZB8gaCkU25kVcYQsumlqcY4rE8RIZEyOj00Y/pTHm8crufOUWrK5y+H4oK/Jjum BOY7gMsfCBjlPBmYoiPoN16BJJTU5bBnRE2Go0r9Hy39TlSe9A3kDfqZ5SjRZ3q8OP78 gtUqJjgfitcHd+tvh0uhxzicvekD+HEr84ASQzXiUcejLM/ZrxR5SPTPAcwCYAV9UVuK xm34RPMWJe93TR+23EKoXcZV/SmAqhtuEH0dBC+/qs8S9cVnuLC618Vck89rfIKgZgqm 37XQ== X-Gm-Message-State: AOAM531UAQepnf5gbY9SvCk8PyuCaNew/QqUT8ogv59ym7goHR6nju+q Wzl0MR/pgyDOdkjktzT5YHYn7a1i X-Google-Smtp-Source: ABdhPJxQLRdOvif77tOPMoeTTIELVGGxIrGCaJEU4vz67I/5f36v6hjZie4ANkuYdE5vk7m5L4oOfA== X-Received: by 2002:a17:90a:1485:: with SMTP id k5mr2140769pja.108.1590183231897; Fri, 22 May 2020 14:33:51 -0700 (PDT) Received: from localhost.localdomain ([2601:647:4000:d7:b874:274b:7df6:e61b]) by smtp.gmail.com with ESMTPSA id mn19sm7480755pjb.8.2020.05.22.14.33.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 May 2020 14:33:51 -0700 (PDT) From: Bart Van Assche To: Jason Gunthorpe Cc: Leon Romanovsky , Doug Ledford , linux-rdma@vger.kernel.org, Bart Van Assche , Laurence Oberman , Kamal Heib Subject: [PATCH 2/4] RDMA/srpt: Make debug output more detailed Date: Fri, 22 May 2020 14:33:39 -0700 Message-Id: <20200522213341.16341-3-bvanassche@acm.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200522213341.16341-1-bvanassche@acm.org> References: <20200522213341.16341-1-bvanassche@acm.org> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Since the session name by itself is not sufficient to uniquely identify a queue pair, include the queue pair number. Show the ASCII channel state name instead of the numeric value. This change makes the ib_srpt debug output more consistent. Cc: Laurence Oberman Cc: Kamal Heib Signed-off-by: Bart Van Assche --- drivers/infiniband/ulp/srpt/ib_srpt.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c index a294630f2100..a39ad9fc4224 100644 --- a/drivers/infiniband/ulp/srpt/ib_srpt.c +++ b/drivers/infiniband/ulp/srpt/ib_srpt.c @@ -214,8 +214,9 @@ static const char *get_ch_state_name(enum rdma_ch_state s) */ static void srpt_qp_event(struct ib_event *event, struct srpt_rdma_ch *ch) { - pr_debug("QP event %d on ch=%p sess_name=%s state=%d\n", - event->event, ch, ch->sess_name, ch->state); + pr_debug("QP event %d on ch=%p sess_name=%s-%d state=%s\n", + event->event, ch, ch->sess_name, ch->qp->qp_num, + get_ch_state_name(ch->state)); switch (event->event) { case IB_EVENT_COMM_EST: @@ -1985,8 +1986,8 @@ static void __srpt_close_all_ch(struct srpt_port *sport) list_for_each_entry(nexus, &sport->nexus_list, entry) { list_for_each_entry(ch, &nexus->ch_list, list) { if (srpt_disconnect_ch(ch) >= 0) - pr_info("Closing channel %s because target %s_%d has been disabled\n", - ch->sess_name, + pr_info("Closing channel %s-%d because target %s_%d has been disabled\n", + ch->sess_name, ch->qp->qp_num, dev_name(&sport->sdev->device->dev), sport->port); srpt_close_ch(ch); From patchwork Fri May 22 21:33:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 11566203 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0D84F1392 for ; Fri, 22 May 2020 21:33:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EB1A92078D for ; Fri, 22 May 2020 21:33:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731140AbgEVVdz (ORCPT ); Fri, 22 May 2020 17:33:55 -0400 Received: from mail-pj1-f68.google.com ([209.85.216.68]:52721 "EHLO mail-pj1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731130AbgEVVdy (ORCPT ); Fri, 22 May 2020 17:33:54 -0400 Received: by mail-pj1-f68.google.com with SMTP id a5so5545795pjh.2 for ; Fri, 22 May 2020 14:33:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=f98PTCIHfsRe/PVHZ+ofAlaoYmlI4YmDVK0u2K32q6M=; b=LNGasCXvsn/kpY1IyKfyY3JOneMXjrrqrRD3OnRhrsz1htcrjBGDuARIfjrNwN+GJh 07WI0B41SD6ayvmh0iM5ZBvsaPO3xE36rDRY9EetI1lSUT0lJyEJcODN3cmR3ATh5JwQ fkFJWdp5HUqr7j8xy0a6gL7lsadnqLFuJXrcD5jY0Srm/c2vs1rb/HbkyHQlVEKS5jri TWRDR7Vi2Y+49vZ6mKhiykPYXqJ/F6x1vRCW+1MuiCcw+QRdc+DtpD+VAXEjVJ9SoDKN D2Kbpb+H27/Gjsr/7KMCBooLb41tJU/IP9vhycleDepE9IuzZUtVE6Iw87BZTTu3iA54 mooA== X-Gm-Message-State: AOAM531h59nlsNOnuh0b4nmGXDbYzTaephcX3PNanFdS6EGlOfogRMY/ M9itzxAPUWHDvCMjdoRMbyg= X-Google-Smtp-Source: ABdhPJzE89u5Qfv0v5w20f7SkdOnhCSUCvX5h9m2hOEHRDsRVBkeebGxkYyJqMLlsI3etrkqjnveIg== X-Received: by 2002:a17:90a:36c9:: with SMTP id t67mr3586190pjb.17.1590183233610; Fri, 22 May 2020 14:33:53 -0700 (PDT) Received: from localhost.localdomain ([2601:647:4000:d7:b874:274b:7df6:e61b]) by smtp.gmail.com with ESMTPSA id mn19sm7480755pjb.8.2020.05.22.14.33.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 May 2020 14:33:52 -0700 (PDT) From: Bart Van Assche To: Jason Gunthorpe Cc: Leon Romanovsky , Doug Ledford , linux-rdma@vger.kernel.org, Bart Van Assche , Laurence Oberman , Kamal Heib Subject: [PATCH 3/4] RDMA/srpt: Reduce max_recv_sge to 1 Date: Fri, 22 May 2020 14:33:40 -0700 Message-Id: <20200522213341.16341-4-bvanassche@acm.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200522213341.16341-1-bvanassche@acm.org> References: <20200522213341.16341-1-bvanassche@acm.org> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Since srpt_post_recv() always sets num_sge to 1, reduce the max_recv_sge parameter that is used at queue pair allocation time to 1. Cc: Laurence Oberman Cc: Kamal Heib Signed-off-by: Bart Van Assche --- drivers/infiniband/ulp/srpt/ib_srpt.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c index a39ad9fc4224..1ad3cc7c553a 100644 --- a/drivers/infiniband/ulp/srpt/ib_srpt.c +++ b/drivers/infiniband/ulp/srpt/ib_srpt.c @@ -1818,16 +1818,12 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch) qp_init->cap.max_rdma_ctxs = sq_size / 2; qp_init->cap.max_send_sge = min(attrs->max_send_sge, SRPT_MAX_SG_PER_WQE); - qp_init->cap.max_recv_sge = min(attrs->max_recv_sge, - SRPT_MAX_SG_PER_WQE); + qp_init->cap.max_recv_sge = 1; qp_init->port_num = ch->sport->port; - if (sdev->use_srq) { + if (sdev->use_srq) qp_init->srq = sdev->srq; - } else { + else qp_init->cap.max_recv_wr = ch->rq_size; - qp_init->cap.max_recv_sge = min(attrs->max_recv_sge, - SRPT_MAX_SG_PER_WQE); - } if (ch->using_rdma_cm) { ret = rdma_create_qp(ch->rdma_cm.cm_id, sdev->pd, qp_init); From patchwork Fri May 22 21:33:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 11566205 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 585E71392 for ; Fri, 22 May 2020 21:33:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4AC8D20757 for ; Fri, 22 May 2020 21:33:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731124AbgEVVd4 (ORCPT ); Fri, 22 May 2020 17:33:56 -0400 Received: from mail-pl1-f196.google.com ([209.85.214.196]:46086 "EHLO mail-pl1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731033AbgEVVd4 (ORCPT ); Fri, 22 May 2020 17:33:56 -0400 Received: by mail-pl1-f196.google.com with SMTP id b12so4914033plz.13 for ; Fri, 22 May 2020 14:33:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cMKI/PI9BVrcCOggrtr7KyDylvCF+aXKoXPAnRVfFPw=; b=LTQV0wOuuQPHbiD5J2kFnpPbQLgmd6pWoF6CDq3Piz2yDHbjvVzwatE9Vko10YUOD2 9BjyCLsqxv7OHr4a/2vN2ISYVaAdeTsa6OC5mEqrnXnVvji2tie68R8Z4WTpB8mJqxs7 V+w7oTz2W1OlVNYkU3WrJ0pR832Bbyrn0VMonfsUEr8Rpc2OXzWY7IAtytxdYxk5TVZp sJl73MOv0I5KzTXFxVro1UB2vE/1wZyBpbwL1phNAW59ijVrVyb/dZxAUJLH1HeQ60d1 ObawQEIpE5aFu8+DWhY88VtoMi8u7cjoAvsDqquGjsdt8kcztbR9ISX6AnyJ3ZlgddrC hbDg== X-Gm-Message-State: AOAM530VGO/hDDBIsZOfciv2jVocxP8eDJqEfg0khiJQwyCk1sxq6sNO thzrSLzy/SKzkkWD0FhjuBc= X-Google-Smtp-Source: ABdhPJx8IdquqlD300TqaOPaTCxb8/BET2Zx2TwRiVnHk//IAKhUY8nSFHR5XBUjxNsWGqROLRdnhg== X-Received: by 2002:a17:90a:2a4a:: with SMTP id d10mr6866578pjg.32.1590183235221; Fri, 22 May 2020 14:33:55 -0700 (PDT) Received: from localhost.localdomain ([2601:647:4000:d7:b874:274b:7df6:e61b]) by smtp.gmail.com with ESMTPSA id mn19sm7480755pjb.8.2020.05.22.14.33.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 May 2020 14:33:54 -0700 (PDT) From: Bart Van Assche To: Jason Gunthorpe Cc: Leon Romanovsky , Doug Ledford , linux-rdma@vger.kernel.org, Bart Van Assche , Laurence Oberman , Kamal Heib Subject: [PATCH 4/4] RDMA/srpt: Increase max_send_sge Date: Fri, 22 May 2020 14:33:41 -0700 Message-Id: <20200522213341.16341-5-bvanassche@acm.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200522213341.16341-1-bvanassche@acm.org> References: <20200522213341.16341-1-bvanassche@acm.org> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The ib_srpt driver limits max_send_sge to 16. Since that is a workaround for an mlx4 bug that has been fixed, increase max_send_sge. For mlx4, do not use the value advertised by the driver (32) since that causes QP's to transition to the error status. See also commit f95ccffc715b ("IB/mlx4: Use 4K pages for kernel QP's WQE buffer"). Cc: Laurence Oberman Cc: Kamal Heib Signed-off-by: Bart Van Assche --- drivers/infiniband/ulp/srpt/ib_srpt.c | 25 +++++++++++++++++++++++-- drivers/infiniband/ulp/srpt/ib_srpt.h | 5 ----- 2 files changed, 23 insertions(+), 7 deletions(-) diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c index 1ad3cc7c553a..8d9b8664ea75 100644 --- a/drivers/infiniband/ulp/srpt/ib_srpt.c +++ b/drivers/infiniband/ulp/srpt/ib_srpt.c @@ -1781,8 +1781,30 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch) struct srpt_device *sdev = sport->sdev; const struct ib_device_attr *attrs = &sdev->device->attrs; int sq_size = sport->port_attrib.srp_sq_size; + unsigned int max_sge_delta = 0; int i, ret; + switch (sdev->device->ops.driver_id) { + case RDMA_DRIVER_MLX4: + /* + * The smallest max_sge_delta value that works with + * ConnectX-3 firmware version 2.42.5000. + */ + max_sge_delta = 2; + break; + case RDMA_DRIVER_MTHCA: + /* + * From the OFED release notes: In mem-free devices, RC + * QPs can be created with a maximum of (max_sge - 1) + * entries only. See also + * https://git.openfabrics.org/?p=compat-rdma/docs.git;a=blob;f=release_notes/mthca_release_notes.txt;h=40f3c4ea77a07fe5ded888b8417530471e89d87b. + */ + max_sge_delta = 1; + break; + default: + break; + } + WARN_ON(ch->rq_size < 1); ret = -ENOMEM; @@ -1816,8 +1838,7 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch) */ qp_init->cap.max_send_wr = min(sq_size / 2, attrs->max_qp_wr); qp_init->cap.max_rdma_ctxs = sq_size / 2; - qp_init->cap.max_send_sge = min(attrs->max_send_sge, - SRPT_MAX_SG_PER_WQE); + qp_init->cap.max_send_sge = attrs->max_send_sge - max_sge_delta; qp_init->cap.max_recv_sge = 1; qp_init->port_num = ch->sport->port; if (sdev->use_srq) diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.h b/drivers/infiniband/ulp/srpt/ib_srpt.h index 2e1a69840857..f31c349d07a1 100644 --- a/drivers/infiniband/ulp/srpt/ib_srpt.h +++ b/drivers/infiniband/ulp/srpt/ib_srpt.h @@ -105,11 +105,6 @@ enum { SRP_CMD_ACA = 0x4, SRPT_DEF_SG_TABLESIZE = 128, - /* - * An experimentally determined value that avoids that QP creation - * fails due to "swiotlb buffer is full" on systems using the swiotlb. - */ - SRPT_MAX_SG_PER_WQE = 16, MIN_SRPT_SQ_SIZE = 16, DEF_SRPT_SQ_SIZE = 4096,