From patchwork Mon Nov 16 20:23:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11910761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB0FCC63777 for ; Mon, 16 Nov 2020 20:23:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6475220781 for ; Mon, 16 Nov 2020 20:23:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="aM1md6eq" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732273AbgKPUXU (ORCPT ); Mon, 16 Nov 2020 15:23:20 -0500 Received: from nat-hk.nvidia.com ([203.18.50.4]:43069 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732296AbgKPUXU (ORCPT ); Mon, 16 Nov 2020 15:23:20 -0500 Received: from HKMAIL104.nvidia.com (Not Verified[10.18.92.77]) by nat-hk.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 17 Nov 2020 04:23:19 +0800 Received: from HKMAIL103.nvidia.com (10.18.16.12) by HKMAIL104.nvidia.com (10.18.16.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Mon, 16 Nov 2020 20:23:19 +0000 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.172) by HKMAIL103.nvidia.com (10.18.16.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Mon, 16 Nov 2020 20:23:18 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SZ8GRjlUmg3htAfvorM0/hC8Z+22jxN5ZKmhA8k/SdY6zXv7IQj4eeTQd1wpV1EhcS/elliIaAtzZWvbxCBefB+hqd8PuJGZ1LhQ9RLOg7DHMwuE2jHaDI2vNrXdAFEt1FCr9eqMtF3anaFC7Ykmh+UyVRfbkAshV6zaPIeV/L7BHQ86Fvyr2UpFOGiqGQ3WRQEHKAARljz2nhA1YMadBApT8BwT2+FiAohOuQumyv4em62G4/SiiTnrvbFY2Tn7xUfUSQlVPbDqEZ5pbksIZX1LEwsb3C0xhBmLGCmgMFB7fcyZRydRmQSgFNCa0mL8TcsQnUA9B61UOiLkM2A8Sw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/EnnM0es8CU1PxAsd9gvqh2VIRy8netKPwRkme9VR7c=; b=iwajj08ZUFDpNhnz83xnqeNbstLSg1fqQ1GbAdNiPT6qCPtwybZopHPHBRewi9XXWiT70PzJE/Zi1BuMQ9ZhoBThmZNqe2THfxsb8Z29lQ6UuJeAEYBWvuHS48bUpyqZuMKB4Ihzl2hekoLHXZPrjx8N4cmtSTTR9t22uVtHaIK+1ARxDgwPtHWopKJsavky+cEMnMCvdNvJ0C1rnsKAY6LmoMUJC5q1A3MvV8weSLUbPuoqCJFjv/RXEh6m7DBlBrQ5Q21fClAhjoUpB9QQXJKK8vjeODgKZwir/NOdCdFF/Y5hv4tCwXlyO6PQ+6vDumoglboHKpyRmoZermwGQQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Received: from DM6PR12MB3834.namprd12.prod.outlook.com (2603:10b6:5:14a::12) by DM6PR12MB3305.namprd12.prod.outlook.com (2603:10b6:5:189::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3564.28; Mon, 16 Nov 2020 20:23:16 +0000 Received: from DM6PR12MB3834.namprd12.prod.outlook.com ([fe80::e40c:730c:156c:2ef9]) by DM6PR12MB3834.namprd12.prod.outlook.com ([fe80::e40c:730c:156c:2ef9%7]) with mapi id 15.20.3564.028; Mon, 16 Nov 2020 20:23:16 +0000 From: Jason Gunthorpe To: , Bob Pearson CC: Gal Pressman Subject: [PATCH 4/9] efa: Move the context intialization out of efa_query_device_ex() Date: Mon, 16 Nov 2020 16:23:05 -0400 Message-ID: <4-v1-34e141ddf17e+89-query_device_ex_jgg@nvidia.com> In-Reply-To: <0-v1-34e141ddf17e+89-query_device_ex_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR19CA0008.namprd19.prod.outlook.com (2603:10b6:208:178::21) To DM6PR12MB3834.namprd12.prod.outlook.com (2603:10b6:5:14a::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by MN2PR19CA0008.namprd19.prod.outlook.com (2603:10b6:208:178::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3564.25 via Frontend Transport; Mon, 16 Nov 2020 20:23:13 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kel1q-006l7z-Q0; Mon, 16 Nov 2020 16:23:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1605558199; bh=MvlJR4z3rc0QfhINWKcWhq0XqLO/+ljYEfZwMSN/l9Q=; h=ARC-Seal:ARC-Message-Signature:ARC-Authentication-Results:From:To: CC:Subject:Date:Message-ID:In-Reply-To:References: Content-Transfer-Encoding:Content-Type:X-ClientProxiedBy: MIME-Version:X-MS-Exchange-MessageSentRepresentingType; b=aM1md6eqVbkI012Z2x8XTbPqVbKLS6Q8P9xkpP8p/jZCIni9/glTeoq3QwFrpjZdr cLVX+lafbnjb9Fv22DCYfTDMllFB+pQ1Ptp2Ha/cvWZ+LVFjO/2+qRbjp/h1taYPao QTpNthNTY9e/g+R9JwRNT6+wLdGut8j0HQAH6Y5wYL2SZemVgKM2h6hltd35rTvcBm mb87FXfYkwkNGEQ6lsrhO1KLmhUUZ/PwzRtuZ0WT+VKVA0ITtjEWLDqiuRFPcXx0Jh C3cPULw8PoMuV25MmR9RYAyBXRgYl9sw3UzZvgDx6Wr+DFYBvd+8EejHNvBOOI9kLr Y68m6AoElnkNw== Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org When the user calls efa_query_device_ex() it should not cause the context values to be mutated, only the attribute shuld be returned. Move this code to a dedicated function that is only called during context setup. Cc: Gal Pressman Signed-off-by: Jason Gunthorpe --- providers/efa/efa.c | 14 +------------ providers/efa/verbs.c | 46 +++++++++++++++++++++++++++++++++++-------- providers/efa/verbs.h | 1 + 3 files changed, 40 insertions(+), 21 deletions(-) diff --git a/providers/efa/efa.c b/providers/efa/efa.c index 35f9b246a711ec..b24c14f7fa1fe1 100644 --- a/providers/efa/efa.c +++ b/providers/efa/efa.c @@ -54,10 +54,7 @@ static struct verbs_context *efa_alloc_context(struct ibv_device *vdev, { struct efa_alloc_ucontext_resp resp = {}; struct efa_alloc_ucontext cmd = {}; - struct ibv_device_attr_ex attr; - unsigned int qp_table_sz; struct efa_context *ctx; - int err; cmd.comp_mask |= EFA_ALLOC_UCONTEXT_CMD_COMP_TX_BATCH; cmd.comp_mask |= EFA_ALLOC_UCONTEXT_CMD_COMP_MIN_SQ_WR; @@ -86,17 +83,8 @@ static struct verbs_context *efa_alloc_context(struct ibv_device *vdev, verbs_set_ops(&ctx->ibvctx, &efa_ctx_ops); - err = efa_query_device_ex(&ctx->ibvctx.context, NULL, &attr, - sizeof(attr)); - if (err) + if (!efa_query_device_ctx(ctx)) goto err_free_spinlock; - - qp_table_sz = roundup_pow_of_two(attr.orig_attr.max_qp); - ctx->qp_table_sz_m1 = qp_table_sz - 1; - ctx->qp_table = calloc(qp_table_sz, sizeof(*ctx->qp_table)); - if (!ctx->qp_table) - goto err_free_spinlock; - return &ctx->ibvctx; err_free_spinlock: diff --git a/providers/efa/verbs.c b/providers/efa/verbs.c index 1a9633155c62f8..52d6285f1f409c 100644 --- a/providers/efa/verbs.c +++ b/providers/efa/verbs.c @@ -106,14 +106,6 @@ int efa_query_device_ex(struct ibv_context *context, if (err) return err; - ctx->device_caps = resp.device_caps; - ctx->max_sq_wr = resp.max_sq_wr; - ctx->max_rq_wr = resp.max_rq_wr; - ctx->max_sq_sge = resp.max_sq_sge; - ctx->max_rq_sge = resp.max_rq_sge; - ctx->max_rdma_size = resp.max_rdma_size; - ctx->max_wr_rdma_sge = a->max_sge_rd; - a->max_qp_wr = min_t(int, a->max_qp_wr, ctx->max_llq_size / sizeof(struct efa_io_tx_wqe)); snprintf(a->fw_ver, sizeof(a->fw_ver), "%u.%u.%u.%u", @@ -122,6 +114,44 @@ int efa_query_device_ex(struct ibv_context *context, return 0; } +int efa_query_device_ctx(struct efa_context *ctx) +{ + struct ibv_device_attr_ex attr; + struct efa_query_device_ex_resp resp; + size_t resp_size = sizeof(resp); + unsigned int qp_table_sz; + int err; + + if (ctx->cmds_supp_udata_mask & EFA_USER_CMDS_SUPP_UDATA_QUERY_DEVICE) { + err = ibv_cmd_query_device_any(&ctx->ibvctx.context, NULL, + &attr, sizeof(attr), + &resp.ibv_resp, &resp_size); + if (err) + return err; + + ctx->device_caps = resp.device_caps; + ctx->max_sq_wr = resp.max_sq_wr; + ctx->max_rq_wr = resp.max_rq_wr; + ctx->max_sq_sge = resp.max_sq_sge; + ctx->max_rq_sge = resp.max_rq_sge; + ctx->max_rdma_size = resp.max_rdma_size; + ctx->max_wr_rdma_sge = attr.orig_attr.max_sge_rd; + } else { + err = ibv_cmd_query_device_any(&ctx->ibvctx.context, NULL, + &attr, sizeof(attr.orig_attr), + NULL, NULL); + if (err) + return err; + } + + qp_table_sz = roundup_pow_of_two(attr.orig_attr.max_qp); + ctx->qp_table_sz_m1 = qp_table_sz - 1; + ctx->qp_table = calloc(qp_table_sz, sizeof(*ctx->qp_table)); + if (!ctx->qp_table) + return ENOMEM; + return 0; +} + int efadv_query_device(struct ibv_context *ibvctx, struct efadv_device_attr *attr, uint32_t inlen) diff --git a/providers/efa/verbs.h b/providers/efa/verbs.h index da022e615af064..3b0e4e0d498761 100644 --- a/providers/efa/verbs.h +++ b/providers/efa/verbs.h @@ -9,6 +9,7 @@ #include #include +int efa_query_device_ctx(struct efa_context *ctx); int efa_query_device(struct ibv_context *uctx, struct ibv_device_attr *attr); int efa_query_port(struct ibv_context *uctx, uint8_t port, struct ibv_port_attr *attr);