From patchwork Sat Feb 20 11:30:02 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Santosh Shilimkar X-Patchwork-Id: 8366271 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id D59C3C0553 for ; Sat, 20 Feb 2016 11:31:59 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id EAB382051A for ; Sat, 20 Feb 2016 11:31:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CF0D620504 for ; Sat, 20 Feb 2016 11:31:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2994391AbcBTLbn (ORCPT ); Sat, 20 Feb 2016 06:31:43 -0500 Received: from userp1040.oracle.com ([156.151.31.81]:21351 "EHLO userp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2994118AbcBTLa4 (ORCPT ); Sat, 20 Feb 2016 06:30:56 -0500 Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u1KBUrVa020718 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Sat, 20 Feb 2016 11:30:54 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u1KBUqJX025036 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Sat, 20 Feb 2016 11:30:53 GMT Received: from abhmp0018.oracle.com (abhmp0018.oracle.com [141.146.116.24]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id u1KBUpxD003561; Sat, 20 Feb 2016 11:30:52 GMT Received: from localhost.localdomain (/10.159.225.68) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Sat, 20 Feb 2016 03:30:48 -0800 From: Santosh Shilimkar To: netdev@vger.kernel.org, davem@davemloft.net Cc: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Santosh Shilimkar Subject: [net-next][PATCH 11/13] RDS: IB: add Fastreg MR (FRMR) detection support Date: Sat, 20 Feb 2016 03:30:02 -0800 Message-Id: <1455967804-27644-12-git-send-email-santosh.shilimkar@oracle.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1455967804-27644-1-git-send-email-santosh.shilimkar@oracle.com> References: <1455967804-27644-1-git-send-email-santosh.shilimkar@oracle.com> X-Source-IP: aserv0022.oracle.com [141.146.126.234] Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Discovere Fast Memmory Registration support using IB device IB_DEVICE_MEM_MGT_EXTENSIONS. Certain HCA might support just FRMR or FMR or both FMR and FRWR. In case both mr type are supported, default FMR is used. Using module parameter 'prefer_frmr', user can choose its preferred MR method for RDS. Ofcourse the module parameter has no effect if the HCA support only FRMR or only FRMR. Default MR is still kept as FMR against what everyone else is following. Default will be changed to FRMR once the RDS performance with FRMR is comparable with FMR. The work is in progress for the same. Signed-off-by: Santosh Shilimkar Signed-off-by: Santosh Shilimkar --- net/rds/ib.c | 14 ++++++++++++++ net/rds/ib.h | 4 ++++ net/rds/ib_mr.h | 1 + 3 files changed, 19 insertions(+) diff --git a/net/rds/ib.c b/net/rds/ib.c index bb32cb9..68c94b0 100644 --- a/net/rds/ib.c +++ b/net/rds/ib.c @@ -47,6 +47,7 @@ unsigned int rds_ib_mr_1m_pool_size = RDS_MR_1M_POOL_SIZE; unsigned int rds_ib_mr_8k_pool_size = RDS_MR_8K_POOL_SIZE; unsigned int rds_ib_retry_count = RDS_IB_DEFAULT_RETRY_COUNT; +bool prefer_frmr; module_param(rds_ib_mr_1m_pool_size, int, 0444); MODULE_PARM_DESC(rds_ib_mr_1m_pool_size, " Max number of 1M mr per HCA"); @@ -54,6 +55,8 @@ module_param(rds_ib_mr_8k_pool_size, int, 0444); MODULE_PARM_DESC(rds_ib_mr_8k_pool_size, " Max number of 8K mr per HCA"); module_param(rds_ib_retry_count, int, 0444); MODULE_PARM_DESC(rds_ib_retry_count, " Number of hw retries before reporting an error"); +module_param(prefer_frmr, bool, 0444); +MODULE_PARM_DESC(prefer_frmr, "Preferred MR method if both FMR and FRMR supported"); /* * we have a clumsy combination of RCU and a rwsem protecting this list @@ -140,6 +143,13 @@ static void rds_ib_add_one(struct ib_device *device) rds_ibdev->max_wrs = device->attrs.max_qp_wr; rds_ibdev->max_sge = min(device->attrs.max_sge, RDS_IB_MAX_SGE); + rds_ibdev->has_fr = (device->attrs.device_cap_flags & + IB_DEVICE_MEM_MGT_EXTENSIONS); + rds_ibdev->has_fmr = (device->alloc_fmr && device->dealloc_fmr && + device->map_phys_fmr && device->unmap_fmr); + rds_ibdev->use_fastreg = (rds_ibdev->has_fr && + (!rds_ibdev->has_fmr || prefer_frmr)); + rds_ibdev->fmr_max_remaps = device->attrs.max_map_per_fmr?: 32; rds_ibdev->max_1m_mrs = device->attrs.max_mr ? min_t(unsigned int, (device->attrs.max_mr / 2), @@ -178,6 +188,10 @@ static void rds_ib_add_one(struct ib_device *device) rds_ibdev->fmr_max_remaps, rds_ibdev->max_1m_mrs, rds_ibdev->max_8k_mrs); + pr_info("RDS/IB: %s: %s supported and preferred\n", + device->name, + rds_ibdev->use_fastreg ? "FRMR" : "FMR"); + INIT_LIST_HEAD(&rds_ibdev->ipaddr_list); INIT_LIST_HEAD(&rds_ibdev->conn_list); diff --git a/net/rds/ib.h b/net/rds/ib.h index 62fe7d5..c5eddc2 100644 --- a/net/rds/ib.h +++ b/net/rds/ib.h @@ -200,6 +200,10 @@ struct rds_ib_device { struct list_head conn_list; struct ib_device *dev; struct ib_pd *pd; + bool has_fmr; + bool has_fr; + bool use_fastreg; + unsigned int max_mrs; struct rds_ib_mr_pool *mr_1m_pool; struct rds_ib_mr_pool *mr_8k_pool; diff --git a/net/rds/ib_mr.h b/net/rds/ib_mr.h index add7725..2f9b9c3 100644 --- a/net/rds/ib_mr.h +++ b/net/rds/ib_mr.h @@ -93,6 +93,7 @@ struct rds_ib_mr_pool { extern struct workqueue_struct *rds_ib_mr_wq; extern unsigned int rds_ib_mr_1m_pool_size; extern unsigned int rds_ib_mr_8k_pool_size; +extern bool prefer_frmr; struct rds_ib_mr_pool *rds_ib_create_mr_pool(struct rds_ib_device *rds_dev, int npages);