From patchwork Wed May 5 17:10:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devesh Sharma X-Patchwork-Id: 12240781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E06EC43461 for ; Wed, 5 May 2021 17:40:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2EB9661073 for ; Wed, 5 May 2021 17:40:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234641AbhEERlp (ORCPT ); Wed, 5 May 2021 13:41:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51812 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234964AbhEERiq (ORCPT ); Wed, 5 May 2021 13:38:46 -0400 Received: from mail-pj1-x1043.google.com (mail-pj1-x1043.google.com [IPv6:2607:f8b0:4864:20::1043]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 14A0DC061242 for ; Wed, 5 May 2021 10:11:20 -0700 (PDT) Received: by mail-pj1-x1043.google.com with SMTP id lp4so1167597pjb.1 for ; Wed, 05 May 2021 10:11:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version; bh=xovdW97TbBC+dsfTcVdKGImohu2qf4Rmll/wB6h4QwI=; b=a7Vs/VAja7YLpyZU6hwslhprxT6QIRxYeBzhguWDFR4hO/TKpVA3L8ckOjegIST7Fb gUbYV+zpKCxAjmUi5f1q9mBEzgv7dnZ0RsLdB56tJQClglc4rtxoSX9Zfurc6LiGkNqp KpfrSVg8ATsENNoduvHUjsgwCRzuSzlze7ic0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version; bh=xovdW97TbBC+dsfTcVdKGImohu2qf4Rmll/wB6h4QwI=; b=Gp8vadw398bZNKKRedBXjaUX6MrnNaJG67fm3toyWWZasi6DewWKrCNBnXfDQKTqJ3 kQNPUWTTIsEiYhXUzRpPdnTOTuKvdP4SOFnXhDzgjupbMFRy59/RUNSKlNYCwFbyP973 739SnX/lSmY7nm6nqUdXYd6JeVp5IUvzYH1iGwrP9yd7oZRR+QNLWjBZA0kjUc8l7p1c g7Y2IjjXCToI8CAnBACPq4e+7ObEDXWfeOyRhUKr2PrLg1pzi0m1KlpsQMQ61xmdUQEK 7J9g9zLSi7TcbGfShWzrlOw6khHc4+u5OVcUtUrtzUpfio+FcJr8mXleSMTywV6ZuSgt U6sQ== X-Gm-Message-State: AOAM530MI2IPMroGjRg1bQhwPE6DFi0aFbgkjjh3QYaNjbLJSYWU7Kda Zj8BhFMkt3GOeYapg7kwrGDoZLkCYuS+tN1PRY0n+nKbh3PvW7srgqYIcfsLVMMEravFbFgKNPS eeLKUJmPSod9Ic9N3H/idHIpmokk0hO4Ijz74smg0Mnoya/YXiiUELMxSujCvJiprl4k7YSN5h+ y3XqgU+99l X-Google-Smtp-Source: ABdhPJzYpvR9zvAtAgO+/JTiY0WHjW8v37aH7HEXEeMHgoZH5dZ2ZLqtgRW9ONMsd+xjxdH36Gx3gg== X-Received: by 2002:a17:90a:414a:: with SMTP id m10mr12127085pjg.63.1620234678914; Wed, 05 May 2021 10:11:18 -0700 (PDT) Received: from dev01.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id u21sm15381614pfm.89.2021.05.05.10.11.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 May 2021 10:11:18 -0700 (PDT) From: Devesh Sharma To: linux-rdma@vger.kernel.org Cc: Devesh Sharma Subject: [V2 rdma-core 4/4] bnxt_re/lib: query device attributes only once and store Date: Wed, 5 May 2021 22:40:56 +0530 Message-Id: <20210505171056.514204-5-devesh.sharma@broadcom.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210505171056.514204-1-devesh.sharma@broadcom.com> References: <20210505171056.514204-1-devesh.sharma@broadcom.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Making a change to query device attributes only once during context initialization. Context structure would store the attributes for future reference. This avoids multiple user to kernel context switch during QP creation. Fixes: d2745fe2ab86 ("Add support for posting and polling") Signed-off-by: Devesh Sharma --- providers/bnxt_re/main.c | 31 ++++++++++++++++++------------- providers/bnxt_re/main.h | 2 ++ providers/bnxt_re/verbs.c | 25 +++++++++++-------------- 3 files changed, 31 insertions(+), 27 deletions(-) diff --git a/providers/bnxt_re/main.c b/providers/bnxt_re/main.c index a78e6b98..1779e1ec 100644 --- a/providers/bnxt_re/main.c +++ b/providers/bnxt_re/main.c @@ -129,10 +129,11 @@ static struct verbs_context *bnxt_re_alloc_context(struct ibv_device *vdev, int cmd_fd, void *private_data) { - struct ibv_get_context cmd; + struct bnxt_re_dev *rdev = to_bnxt_re_dev(vdev); struct ubnxt_re_cntx_resp resp; - struct bnxt_re_dev *dev = to_bnxt_re_dev(vdev); struct bnxt_re_context *cntx; + struct ibv_get_context cmd; + int ret; cntx = verbs_init_and_alloc_context(vdev, cmd_fd, cntx, ibvctx, RDMA_DRIVER_BNXT_RE); @@ -146,9 +147,9 @@ static struct verbs_context *bnxt_re_alloc_context(struct ibv_device *vdev, cntx->dev_id = resp.dev_id; cntx->max_qp = resp.max_qp; - dev->pg_size = resp.pg_size; - dev->cqe_size = resp.cqe_sz; - dev->max_cq_depth = resp.max_cqd; + rdev->pg_size = resp.pg_size; + rdev->cqe_size = resp.cqe_sz; + rdev->max_cq_depth = resp.max_cqd; if (resp.comp_mask & BNXT_RE_UCNTX_CMASK_HAVE_CCTX) { cntx->cctx.chip_num = resp.chip_id0 & 0xFFFF; cntx->cctx.chip_rev = (resp.chip_id0 >> @@ -159,7 +160,7 @@ static struct verbs_context *bnxt_re_alloc_context(struct ibv_device *vdev, } pthread_spin_init(&cntx->fqlock, PTHREAD_PROCESS_PRIVATE); /* mmap shared page. */ - cntx->shpg = mmap(NULL, dev->pg_size, PROT_READ | PROT_WRITE, + cntx->shpg = mmap(NULL, rdev->pg_size, PROT_READ | PROT_WRITE, MAP_SHARED, cmd_fd, 0); if (cntx->shpg == MAP_FAILED) { cntx->shpg = NULL; @@ -168,6 +169,10 @@ static struct verbs_context *bnxt_re_alloc_context(struct ibv_device *vdev, pthread_mutex_init(&cntx->shlock, NULL); verbs_set_ops(&cntx->ibvctx, &bnxt_re_cntx_ops); + cntx->rdev = rdev; + ret = ibv_query_device(&cntx->ibvctx.context, &rdev->devattr); + if (ret) + goto failed; return &cntx->ibvctx; @@ -180,19 +185,19 @@ failed: static void bnxt_re_free_context(struct ibv_context *ibvctx) { struct bnxt_re_context *cntx = to_bnxt_re_context(ibvctx); - struct bnxt_re_dev *dev = to_bnxt_re_dev(ibvctx->device); + struct bnxt_re_dev *rdev = to_bnxt_re_dev(ibvctx->device); /* Unmap if anything device specific was mapped in init_context. */ pthread_mutex_destroy(&cntx->shlock); if (cntx->shpg) - munmap(cntx->shpg, dev->pg_size); + munmap(cntx->shpg, rdev->pg_size); pthread_spin_destroy(&cntx->fqlock); /* Un-map DPI only for the first PD that was * allocated in this context. */ if (cntx->udpi.dbpage && cntx->udpi.dbpage != MAP_FAILED) { - munmap(cntx->udpi.dbpage, dev->pg_size); + munmap(cntx->udpi.dbpage, rdev->pg_size); cntx->udpi.dbpage = NULL; } @@ -203,13 +208,13 @@ static void bnxt_re_free_context(struct ibv_context *ibvctx) static struct verbs_device * bnxt_re_device_alloc(struct verbs_sysfs_dev *sysfs_dev) { - struct bnxt_re_dev *dev; + struct bnxt_re_dev *rdev; - dev = calloc(1, sizeof(*dev)); - if (!dev) + rdev = calloc(1, sizeof(*rdev)); + if (!rdev) return NULL; - return &dev->vdev; + return &rdev->vdev; } static const struct verbs_device_ops bnxt_re_dev_ops = { diff --git a/providers/bnxt_re/main.h b/providers/bnxt_re/main.h index d470e30a..a63719e8 100644 --- a/providers/bnxt_re/main.h +++ b/providers/bnxt_re/main.h @@ -166,10 +166,12 @@ struct bnxt_re_dev { uint32_t cqe_size; uint32_t max_cq_depth; + struct ibv_device_attr devattr; }; struct bnxt_re_context { struct verbs_context ibvctx; + struct bnxt_re_dev *rdev; uint32_t dev_id; uint32_t max_qp; struct bnxt_re_chip_ctx cctx; diff --git a/providers/bnxt_re/verbs.c b/providers/bnxt_re/verbs.c index 59a57f72..fb2cf5ac 100644 --- a/providers/bnxt_re/verbs.c +++ b/providers/bnxt_re/verbs.c @@ -777,25 +777,22 @@ int bnxt_re_arm_cq(struct ibv_cq *ibvcq, int flags) static int bnxt_re_check_qp_limits(struct bnxt_re_context *cntx, struct ibv_qp_init_attr *attr) { - struct ibv_device_attr devattr; - int ret; + struct ibv_device_attr *devattr; + struct bnxt_re_dev *rdev; - ret = bnxt_re_query_device( - &cntx->ibvctx.context, NULL, - container_of(&devattr, struct ibv_device_attr_ex, orig_attr), - sizeof(devattr)); - if (ret) - return ret; - if (attr->cap.max_send_sge > devattr.max_sge) + rdev = cntx->rdev; + devattr = &rdev->devattr; + + if (attr->cap.max_send_sge > devattr->max_sge) return EINVAL; - if (attr->cap.max_recv_sge > devattr.max_sge) + if (attr->cap.max_recv_sge > devattr->max_sge) return EINVAL; if (attr->cap.max_inline_data > BNXT_RE_MAX_INLINE_SIZE) return EINVAL; - if (attr->cap.max_send_wr > devattr.max_qp_wr) - attr->cap.max_send_wr = devattr.max_qp_wr; - if (attr->cap.max_recv_wr > devattr.max_qp_wr) - attr->cap.max_recv_wr = devattr.max_qp_wr; + if (attr->cap.max_send_wr > devattr->max_qp_wr) + attr->cap.max_send_wr = devattr->max_qp_wr; + if (attr->cap.max_recv_wr > devattr->max_qp_wr) + attr->cap.max_recv_wr = devattr->max_qp_wr; return 0; }