From patchwork Mon May 3 06:48:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devesh Sharma X-Patchwork-Id: 12235727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05323C43460 for ; Mon, 3 May 2021 06:48:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D561D611AD for ; Mon, 3 May 2021 06:48:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230364AbhECGtS (ORCPT ); Mon, 3 May 2021 02:49:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231605AbhECGtR (ORCPT ); Mon, 3 May 2021 02:49:17 -0400 Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96EA6C061756 for ; Sun, 2 May 2021 23:48:23 -0700 (PDT) Received: by mail-pl1-x644.google.com with SMTP id h20so2294574plr.4 for ; Sun, 02 May 2021 23:48:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version; bh=COVqszlNJard7bUu4+ks4yO1w9lOMnS489/lJ5qSl8s=; b=elx7maC22nnoisv8YS39tZ93cfHKqaOyjRcOQpx1HM99ERkdnG/O3fI1hobu47AU0C y61Pk8992mU6eh/7P98KBBzd/xxRtiNyyh2Uvi4vW3VpNwrPzJHwUV3I6C5yvDK6fXLR gv4L79yuXy8bsWu7ES1sMR0LxZ3uhxdZ/a/Jo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version; bh=COVqszlNJard7bUu4+ks4yO1w9lOMnS489/lJ5qSl8s=; b=eeQVB9QcNNLvlIGacij99yxHgKlDMZliqQRavZH+0kjDnSoVeaXwWBjNGxJFO8Ki0Z LvAXMRMEBLh2pAQyBAAjwK2JwtMc1FB6Yy0GDOrSTPwYbaK9DYqhbw4dXmWX4/DB6xYT tRmlHP5TAj+vT6IU9Y8HXPsvD05yrCKVXqigkCwwB1h1qBglhYK4Tx9M1QArRqgEk1D2 Ufkp92UFe3RqjBKSAvLEtcBbO2r19PgLMsjssX9/OxWVYZfKaDHDOBKFUYZwMkm9Y2qw m/JvI4BPUaKTzOrKubyP9IAr0Bu7lVsnLh9mYvJg/ipOE5FHn5aWdcsGIiwoia/1V5a+ I04w== X-Gm-Message-State: AOAM533nNVph+SQzt7VN3MQbQTU2K7XzhAeemgMi2xQU//Y0G2fmGhMA 3DDmoZR3yzTfRz4/oPDLmPNTqjgon9/LB54ZfzCAW2yzYu8ft4QjgE8bVDpiUK3cHGMz25jbgmA WyYq/15nJEnmGP7tfkEPEQgbiJLGfXs/ZjMN9qgXo/BU5orkwzJ5jrqSoDGuGmqjmCq9vc47Aqv uy4WepTCkT X-Google-Smtp-Source: ABdhPJxbZcCsodQZ7EcGhzhhncC1vhcEaGee235AN+XzRLJ8f4Ryv3ZSztWyn0lWsMA80MQBbLqOkA== X-Received: by 2002:a17:90a:5806:: with SMTP id h6mr19476504pji.14.1620024502475; Sun, 02 May 2021 23:48:22 -0700 (PDT) Received: from dev01.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id i9sm19585389pjh.9.2021.05.02.23.48.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 02 May 2021 23:48:22 -0700 (PDT) From: Devesh Sharma To: linux-rdma@vger.kernel.org Cc: Devesh Sharma Subject: [rdma-core 4/4] bnxt_re/lib: let bnxt_re_dev store device attrs Date: Mon, 3 May 2021 12:18:02 +0530 Message-Id: <20210503064802.457482-5-devesh.sharma@broadcom.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210503064802.457482-1-devesh.sharma@broadcom.com> References: <20210503064802.457482-1-devesh.sharma@broadcom.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Making a change to let bnxt_re_dev strucutre store the device attributes once for entire life of device context. This change saves a firmware call per QP creation and speed it up. Signed-off-by: Devesh Sharma --- providers/bnxt_re/main.c | 31 ++++++++++++++++++------------- providers/bnxt_re/main.h | 2 ++ providers/bnxt_re/verbs.c | 25 +++++++++++-------------- 3 files changed, 31 insertions(+), 27 deletions(-) diff --git a/providers/bnxt_re/main.c b/providers/bnxt_re/main.c index a78e6b98..1779e1ec 100644 --- a/providers/bnxt_re/main.c +++ b/providers/bnxt_re/main.c @@ -129,10 +129,11 @@ static struct verbs_context *bnxt_re_alloc_context(struct ibv_device *vdev, int cmd_fd, void *private_data) { - struct ibv_get_context cmd; + struct bnxt_re_dev *rdev = to_bnxt_re_dev(vdev); struct ubnxt_re_cntx_resp resp; - struct bnxt_re_dev *dev = to_bnxt_re_dev(vdev); struct bnxt_re_context *cntx; + struct ibv_get_context cmd; + int ret; cntx = verbs_init_and_alloc_context(vdev, cmd_fd, cntx, ibvctx, RDMA_DRIVER_BNXT_RE); @@ -146,9 +147,9 @@ static struct verbs_context *bnxt_re_alloc_context(struct ibv_device *vdev, cntx->dev_id = resp.dev_id; cntx->max_qp = resp.max_qp; - dev->pg_size = resp.pg_size; - dev->cqe_size = resp.cqe_sz; - dev->max_cq_depth = resp.max_cqd; + rdev->pg_size = resp.pg_size; + rdev->cqe_size = resp.cqe_sz; + rdev->max_cq_depth = resp.max_cqd; if (resp.comp_mask & BNXT_RE_UCNTX_CMASK_HAVE_CCTX) { cntx->cctx.chip_num = resp.chip_id0 & 0xFFFF; cntx->cctx.chip_rev = (resp.chip_id0 >> @@ -159,7 +160,7 @@ static struct verbs_context *bnxt_re_alloc_context(struct ibv_device *vdev, } pthread_spin_init(&cntx->fqlock, PTHREAD_PROCESS_PRIVATE); /* mmap shared page. */ - cntx->shpg = mmap(NULL, dev->pg_size, PROT_READ | PROT_WRITE, + cntx->shpg = mmap(NULL, rdev->pg_size, PROT_READ | PROT_WRITE, MAP_SHARED, cmd_fd, 0); if (cntx->shpg == MAP_FAILED) { cntx->shpg = NULL; @@ -168,6 +169,10 @@ static struct verbs_context *bnxt_re_alloc_context(struct ibv_device *vdev, pthread_mutex_init(&cntx->shlock, NULL); verbs_set_ops(&cntx->ibvctx, &bnxt_re_cntx_ops); + cntx->rdev = rdev; + ret = ibv_query_device(&cntx->ibvctx.context, &rdev->devattr); + if (ret) + goto failed; return &cntx->ibvctx; @@ -180,19 +185,19 @@ failed: static void bnxt_re_free_context(struct ibv_context *ibvctx) { struct bnxt_re_context *cntx = to_bnxt_re_context(ibvctx); - struct bnxt_re_dev *dev = to_bnxt_re_dev(ibvctx->device); + struct bnxt_re_dev *rdev = to_bnxt_re_dev(ibvctx->device); /* Unmap if anything device specific was mapped in init_context. */ pthread_mutex_destroy(&cntx->shlock); if (cntx->shpg) - munmap(cntx->shpg, dev->pg_size); + munmap(cntx->shpg, rdev->pg_size); pthread_spin_destroy(&cntx->fqlock); /* Un-map DPI only for the first PD that was * allocated in this context. */ if (cntx->udpi.dbpage && cntx->udpi.dbpage != MAP_FAILED) { - munmap(cntx->udpi.dbpage, dev->pg_size); + munmap(cntx->udpi.dbpage, rdev->pg_size); cntx->udpi.dbpage = NULL; } @@ -203,13 +208,13 @@ static void bnxt_re_free_context(struct ibv_context *ibvctx) static struct verbs_device * bnxt_re_device_alloc(struct verbs_sysfs_dev *sysfs_dev) { - struct bnxt_re_dev *dev; + struct bnxt_re_dev *rdev; - dev = calloc(1, sizeof(*dev)); - if (!dev) + rdev = calloc(1, sizeof(*rdev)); + if (!rdev) return NULL; - return &dev->vdev; + return &rdev->vdev; } static const struct verbs_device_ops bnxt_re_dev_ops = { diff --git a/providers/bnxt_re/main.h b/providers/bnxt_re/main.h index d470e30a..a63719e8 100644 --- a/providers/bnxt_re/main.h +++ b/providers/bnxt_re/main.h @@ -166,10 +166,12 @@ struct bnxt_re_dev { uint32_t cqe_size; uint32_t max_cq_depth; + struct ibv_device_attr devattr; }; struct bnxt_re_context { struct verbs_context ibvctx; + struct bnxt_re_dev *rdev; uint32_t dev_id; uint32_t max_qp; struct bnxt_re_chip_ctx cctx; diff --git a/providers/bnxt_re/verbs.c b/providers/bnxt_re/verbs.c index 4344b3dd..7f98c260 100644 --- a/providers/bnxt_re/verbs.c +++ b/providers/bnxt_re/verbs.c @@ -777,25 +777,22 @@ int bnxt_re_arm_cq(struct ibv_cq *ibvcq, int flags) static int bnxt_re_check_qp_limits(struct bnxt_re_context *cntx, struct ibv_qp_init_attr *attr) { - struct ibv_device_attr devattr; - int ret; + struct ibv_device_attr *devattr; + struct bnxt_re_dev *rdev; - ret = bnxt_re_query_device( - &cntx->ibvctx.context, NULL, - container_of(&devattr, struct ibv_device_attr_ex, orig_attr), - sizeof(devattr)); - if (ret) - return ret; - if (attr->cap.max_send_sge > devattr.max_sge) + rdev = cntx->rdev; + devattr = &rdev->devattr; + + if (attr->cap.max_send_sge > devattr->max_sge) return EINVAL; - if (attr->cap.max_recv_sge > devattr.max_sge) + if (attr->cap.max_recv_sge > devattr->max_sge) return EINVAL; if (attr->cap.max_inline_data > BNXT_RE_MAX_INLINE_SIZE) return EINVAL; - if (attr->cap.max_send_wr > devattr.max_qp_wr) - attr->cap.max_send_wr = devattr.max_qp_wr; - if (attr->cap.max_recv_wr > devattr.max_qp_wr) - attr->cap.max_recv_wr = devattr.max_qp_wr; + if (attr->cap.max_send_wr > devattr->max_qp_wr) + attr->cap.max_send_wr = devattr->max_qp_wr; + if (attr->cap.max_recv_wr > devattr->max_qp_wr) + attr->cap.max_recv_wr = devattr->max_qp_wr; return 0; }