From patchwork Mon Dec 25 13:56:55 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 10132575 X-Patchwork-Delegate: leon@leon.nu Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 214F460766 for ; Mon, 25 Dec 2017 13:57:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1414F2F135 for ; Mon, 25 Dec 2017 13:57:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 092F62F13C; Mon, 25 Dec 2017 13:57:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9FB392F140 for ; Mon, 25 Dec 2017 13:57:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752561AbdLYN5g (ORCPT ); Mon, 25 Dec 2017 08:57:36 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:37284 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752348AbdLYN50 (ORCPT ); Mon, 25 Dec 2017 08:57:26 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 25 Dec 2017 15:57:18 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id vBPDvIcv016264; Mon, 25 Dec 2017 15:57:18 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id vBPDvIIG010651; Mon, 25 Dec 2017 15:57:18 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id vBPDvI99010650; Mon, 25 Dec 2017 15:57:18 +0200 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, Alexr@mellanox.com, jgg@mellanox.com, majd@mellanox.com Subject: [PATCH rdma-core 3/9] mlx5: Add support for the ibv_td object and its related verbs Date: Mon, 25 Dec 2017 15:56:55 +0200 Message-Id: <1514210221-10466-4-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1514210221-10466-1-git-send-email-yishaih@mellanox.com> References: <1514210221-10466-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch introduces the initial implementation of the ibv_alloc/dealloc_td verbs. Upon TD creation a dedicated BF (blue-flame) register should be attached to its object and upon destruction the BF will be detached. The motivation behind this, is to enable an application to create QP(s) with a given TD (by using the ibv_parent_domain) and then benefit from dropping the lock around the internal BF register upon post send. Downstream patches in this series add the above functionality. Signed-off-by: Yishai Hadas --- providers/mlx5/mlx5.c | 2 ++ providers/mlx5/mlx5.h | 14 ++++++++++++++ providers/mlx5/verbs.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 66 insertions(+) diff --git a/providers/mlx5/mlx5.c b/providers/mlx5/mlx5.c index 3648621..d078548 100644 --- a/providers/mlx5/mlx5.c +++ b/providers/mlx5/mlx5.c @@ -1026,6 +1026,8 @@ static int mlx5_init_context(struct verbs_device *vdev, v_ctx->destroy_rwq_ind_table = mlx5_destroy_rwq_ind_table; v_ctx->post_srq_ops = mlx5_post_srq_ops; v_ctx->modify_cq = mlx5_modify_cq; + v_ctx->alloc_td = mlx5_alloc_td; + v_ctx->dealloc_td = mlx5_dealloc_td; memset(&device_attr, 0, sizeof(device_attr)); if (!mlx5_query_device_ex(ctx, NULL, &device_attr, diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h index 5a1f7c0..4d4c583 100644 --- a/providers/mlx5/mlx5.h +++ b/providers/mlx5/mlx5.h @@ -320,6 +320,12 @@ struct mlx5_buf { enum mlx5_alloc_type type; }; +struct mlx5_td { + struct ibv_td ibv_td; + struct mlx5_bf *bf; + atomic_int refcount; +}; + struct mlx5_pd { struct ibv_pd ibv_pd; uint32_t pdn; @@ -568,6 +574,11 @@ static inline struct mlx5_srq *to_msrq(struct ibv_srq *ibsrq) return container_of(vsrq, struct mlx5_srq, vsrq); } +static inline struct mlx5_td *to_mtd(struct ibv_td *ibtd) +{ + return to_mxxx(td, td); +} + static inline struct mlx5_qp *to_mqp(struct ibv_qp *ibqp) { struct verbs_qp *vqp = (struct verbs_qp *)ibqp; @@ -753,6 +764,9 @@ int mlx5_post_srq_ops(struct ibv_srq *srq, struct ibv_ops_wr *wr, struct ibv_ops_wr **bad_wr); +struct ibv_td *mlx5_alloc_td(struct ibv_context *context, struct ibv_td_init_attr *init_attr); +int mlx5_dealloc_td(struct ibv_td *td); + static inline void *mlx5_find_uidx(struct mlx5_context *ctx, uint32_t uidx) { int tind = uidx >> MLX5_UIDX_TABLE_SHIFT; diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c index a3323fd..43db501 100644 --- a/providers/mlx5/verbs.c +++ b/providers/mlx5/verbs.c @@ -153,6 +153,56 @@ struct ibv_pd *mlx5_alloc_pd(struct ibv_context *context) return &pd->ibv_pd; } +static struct mlx5_bf *mlx5_attach_dedicated_bf(struct ibv_context *context) +{ + return NULL; +} + +static void mlx5_detach_dedicated_bf(struct ibv_context *context, struct mlx5_bf *bf) +{ +} + +struct ibv_td *mlx5_alloc_td(struct ibv_context *context, struct ibv_td_init_attr *init_attr) +{ + struct mlx5_td *td; + + if (init_attr->comp_mask) { + errno = EINVAL; + return NULL; + } + + td = calloc(1, sizeof(*td)); + if (!td) { + errno = ENOMEM; + return NULL; + } + + td->bf = mlx5_attach_dedicated_bf(context); + if (!td->bf) { + free(td); + return NULL; + } + + td->ibv_td.context = context; + atomic_init(&td->refcount, 1); + + return &td->ibv_td; +} + +int mlx5_dealloc_td(struct ibv_td *ib_td) +{ + struct mlx5_td *td; + + td = to_mtd(ib_td); + if (atomic_load(&td->refcount) > 1) + return EBUSY; + + mlx5_detach_dedicated_bf(ib_td->context, td->bf); + free(td); + + return 0; +} + int mlx5_free_pd(struct ibv_pd *pd) { int ret;