From patchwork Tue Mar 20 21:18:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 10298015 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id BF3DA602B3 for ; Tue, 20 Mar 2018 21:55:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9FFF12966D for ; Tue, 20 Mar 2018 21:55:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9259D29659; Tue, 20 Mar 2018 21:55:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A78F629659 for ; Tue, 20 Mar 2018 21:55:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751343AbeCTVzu (ORCPT ); Tue, 20 Mar 2018 17:55:50 -0400 Received: from mail-it0-f66.google.com ([209.85.214.66]:53601 "EHLO mail-it0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751064AbeCTVzt (ORCPT ); Tue, 20 Mar 2018 17:55:49 -0400 Received: by mail-it0-f66.google.com with SMTP id b136-v6so4400626iti.3 for ; Tue, 20 Mar 2018 14:55:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=aBx3SoMPoFK7/paDQh5rm5iqyZGtGeS6CBds1nhDn58=; b=i850YYgVRxCPjNZOsKu3rkk2VkpUsPoQwKg0tom02yCuNGnmqZLXg5eEj0S4gNS1YL nUDeYWqcA4KZQpYqGkLrB7yHWMPZRgs8k5Jr5aweZYMUBfbbMLHm5b7H0UJ8B15i8KY9 KNnTn9EEDyTdJaJ4LTAMl4IgsKDGJqPYCm5n59KYPIu0SYmvk3CRyZ3qIJm5btCAFBp6 FP4KG7wL5rmKZNWYwmnbBZ3F3BqEEyK7qEn7JaJvs1lNm824bYij1+7Q1omiPa8vpgr0 Rt0bZuYXAXZQCbNnADZ1qvm6aR6MFkE0f1J1J7putMoCHs1QEhaax77/idadRnXn6dgU ciaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=aBx3SoMPoFK7/paDQh5rm5iqyZGtGeS6CBds1nhDn58=; b=EORrtf5BrR9COmrslmu0P7KcMv/os0OW3IineKrxfgnZWF02WkXhaFN6WnDDIWySgL 0eKkipcM6sjsG6lbi2f1DLdNcsfwCpJAyuSNl6crnH1McD8eWcXcSJ+PV2BKoNHltyGq vE/xFVgW7VnqXkN0/1OooI4v6kObAdfCxh1zdf4IHqo7iRiyXpT7dkpqbv7wamCVm2lr i9zWNJY8Sbj0VdTxQzLuArLNCEA5rqMxII6xe/JQ/Gny8/s5XZ9PHr3C+yE8qptIapcU /mOPTyiMzFEVzCSK2oXhdxGbAragKEkmzDnrFAwCPKWpmEEpfz+eh5JOARp/WPr+Phrt qvNQ== X-Gm-Message-State: AElRT7EyMz0bYOlP8acDDV3yg/Pq63V/VvV0GCoJqcXdnPpjUt24Rknl pGmujl3prnMxq2Ah4mWy8X5wFj3KJV8= X-Google-Smtp-Source: AG47ELttT6sJEwXzkp0BEefKZDkcTia6uvCuMzPZLaQbFiexdZMlDusNTQaiICJpFXYtxuSjGMBSVw== X-Received: by 2002:a24:95d6:: with SMTP id m205-v6mr1471831itd.77.1521582948631; Tue, 20 Mar 2018 14:55:48 -0700 (PDT) Received: from ziepe.ca (S010614cc2056d97f.ed.shawcable.net. [174.3.196.123]) by smtp.gmail.com with ESMTPSA id o189sm1628613iof.87.2018.03.20.14.55.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 20 Mar 2018 14:55:47 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.86_2) (envelope-from ) id 1eyOen-0002gV-VJ; Tue, 20 Mar 2018 15:18:57 -0600 From: Jason Gunthorpe To: linux-rdma@vger.kernel.org Cc: Jason Gunthorpe , Devesh Sharma , Yishai Hadas , Matan Barak Subject: [PATCH rdma-core 18/20] bnxt: Move bnxt to exclusively use the kernel uapi headers Date: Tue, 20 Mar 2018 15:18:48 -0600 Message-Id: <20180320211850.10090-19-jgg@ziepe.ca> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180320211850.10090-1-jgg@ziepe.ca> References: <20180320211850.10090-1-jgg@ziepe.ca> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe Using the driver support macros we can now directly create the required driver structs automatically. There is one exciting thing here, bnxt_re has a problem with alignment of the alloc_pd driver_data, due to a bug in the core code. bnxt worked around it with some very ugly pointer math, but we have to solve it properly as the common macros detect the problem. Signed-off-by: Jason Gunthorpe --- buildlib/make_abi_structs.py | 6 ++--- kernel-headers/CMakeLists.txt | 1 + providers/bnxt_re/bnxt_re-abi.h | 60 +++++++++-------------------------------- providers/bnxt_re/main.c | 4 +-- providers/bnxt_re/verbs.c | 20 +++++++------- 5 files changed, 29 insertions(+), 62 deletions(-) diff --git a/buildlib/make_abi_structs.py b/buildlib/make_abi_structs.py index 029f10428ad300..0817735ebe17d1 100644 --- a/buildlib/make_abi_structs.py +++ b/buildlib/make_abi_structs.py @@ -24,8 +24,8 @@ def in_struct(ln,FO,nesting=0): """Copy a top level structure over to the #define output, keeping track of nested structures.""" if nesting == 0: - if ln == "};": - FO.write("}\n\n"); + if re.match(r"(}.*);",ln): + FO.write(ln[:-1] + "\n\n"); return find_struct; FO.write(ln + " \\\n"); @@ -33,7 +33,7 @@ def in_struct(ln,FO,nesting=0): if ln == "struct {" or ln == "union {": return functools.partial(in_struct,nesting=nesting+1); - if re.match(r"}.*;",ln): + if re.match(r"}.*;",ln): return functools.partial(in_struct,nesting=nesting-1); return functools.partial(in_struct,nesting=nesting); diff --git a/kernel-headers/CMakeLists.txt b/kernel-headers/CMakeLists.txt index 8f41b6e98b54f5..2c062f8d17a67b 100644 --- a/kernel-headers/CMakeLists.txt +++ b/kernel-headers/CMakeLists.txt @@ -55,6 +55,7 @@ endfunction() # Transform the kernel ABIs used by the providers rdma_kernel_provider_abi( + rdma/bnxt_re-abi.h rdma/cxgb4-abi.h rdma/hns-abi.h rdma/ib_user_verbs.h diff --git a/providers/bnxt_re/bnxt_re-abi.h b/providers/bnxt_re/bnxt_re-abi.h index a9cd44c72e77df..02d1d5774ab3fe 100644 --- a/providers/bnxt_re/bnxt_re-abi.h +++ b/providers/bnxt_re/bnxt_re-abi.h @@ -41,11 +41,23 @@ #include #include +#include #define BNXT_RE_ABI_VERSION 1 #define BNXT_RE_FULL_FLAG_DELTA 0x80 +DECLARE_DRV_CMD(ubnxt_re_pd, IB_USER_VERBS_CMD_ALLOC_PD, + empty, bnxt_re_pd_resp); +DECLARE_DRV_CMD(ubnxt_re_cq, IB_USER_VERBS_CMD_CREATE_CQ, + bnxt_re_cq_req, bnxt_re_cq_resp); +DECLARE_DRV_CMD(ubnxt_re_qp, IB_USER_VERBS_CMD_CREATE_QP, + bnxt_re_qp_req, bnxt_re_qp_resp); +DECLARE_DRV_CMD(ubnxt_re_cntx, IB_USER_VERBS_CMD_GET_CONTEXT, + empty, bnxt_re_uctx_resp); +DECLARE_DRV_CMD(ubnxt_re_mr, IB_USER_VERBS_CMD_REG_MR, + empty, empty); + enum bnxt_re_wr_opcode { BNXT_RE_WR_OPCD_SEND = 0x00, BNXT_RE_WR_OPCD_SEND_IMM = 0x01, @@ -189,41 +201,6 @@ struct bnxt_re_db_hdr { __le32 typ_qid; /* typ: 4, qid:20*/ }; -struct ubnxt_re_cntx_resp { - struct ib_uverbs_get_context_resp resp; - __u32 dev_id; - __u32 max_qp; /* To allocate qp-table */ - __u32 pg_size; - __u32 cqe_size; - __u32 max_cqd; - __u32 rsvd; -}; - -struct ubnxt_re_pd_resp { - struct ib_uverbs_alloc_pd_resp resp; - __u32 pdid; - __u32 dpi; - __u64 dbr; -}; - -struct ubnxt_re_mr_resp { - struct ib_uverbs_reg_mr_resp resp; -}; - -struct ubnxt_re_cq_req { - struct ibv_create_cq cmd; - __u64 cq_va; - __u64 cq_handle; -}; - -struct ubnxt_re_cq_resp { - struct ib_uverbs_create_cq_resp resp; - __u32 cqid; - __u32 tail; - __u32 phase; - __u32 rsvd; -}; - struct bnxt_re_bcqe { __le32 flg_st_typ_ph; __le32 qphi_rwrid; @@ -257,19 +234,6 @@ struct bnxt_re_term_cqe { __le64 rsvd1; }; -struct ubnxt_re_qp_req { - struct ibv_create_qp cmd; - __u64 qpsva; - __u64 qprva; - __u64 qp_handle; -}; - -struct ubnxt_re_qp_resp { - struct ib_uverbs_create_qp_resp resp; - __u32 qpid; - __u32 rsvd; -}; - struct bnxt_re_bsqe { __le32 rsv_ws_fl_wt; __le32 key_immd; diff --git a/providers/bnxt_re/main.c b/providers/bnxt_re/main.c index 32ad59832e66af..54e3cc31cb3adc 100644 --- a/providers/bnxt_re/main.c +++ b/providers/bnxt_re/main.c @@ -124,13 +124,13 @@ static struct verbs_context *bnxt_re_alloc_context(struct ibv_device *vdev, memset(&resp, 0, sizeof(resp)); if (ibv_cmd_get_context(&cntx->ibvctx, &cmd, sizeof(cmd), - &resp.resp, sizeof(resp))) + &resp.ibv_resp, sizeof(resp))) goto failed; cntx->dev_id = resp.dev_id; cntx->max_qp = resp.max_qp; dev->pg_size = resp.pg_size; - dev->cqe_size = resp.cqe_size; + dev->cqe_size = resp.cqe_sz; dev->max_cq_depth = resp.max_cqd; pthread_spin_init(&cntx->fqlock, PTHREAD_PROCESS_PRIVATE); /* mmap shared page. */ diff --git a/providers/bnxt_re/verbs.c b/providers/bnxt_re/verbs.c index d4232b6224e40c..c90581d6041b64 100644 --- a/providers/bnxt_re/verbs.c +++ b/providers/bnxt_re/verbs.c @@ -93,11 +93,13 @@ struct ibv_pd *bnxt_re_alloc_pd(struct ibv_context *ibvctx) memset(&resp, 0, sizeof(resp)); if (ibv_cmd_alloc_pd(ibvctx, &pd->ibvpd, &cmd, sizeof(cmd), - &resp.resp, sizeof(resp))) + &resp.ibv_resp, sizeof(resp))) goto out; pd->pdid = resp.pdid; - dbr = *(uint64_t *)((uint32_t *)&resp + 3); + dbr = resp.dbr; + static_assert(offsetof(struct ubnxt_re_pd_resp, dbr) == 4 * 3, + "Bad dbr placement"); /* Map DB page now. */ if (!cntx->udpi.dbpage) { @@ -144,7 +146,7 @@ struct ibv_mr *bnxt_re_reg_mr(struct ibv_pd *ibvpd, void *sva, size_t len, return NULL; if (ibv_cmd_reg_mr(ibvpd, sva, len, (uintptr_t)sva, access, &mr->ibvmr, - &cmd, sizeof(cmd), &resp.resp, sizeof(resp))) { + &cmd, sizeof(cmd), &resp.ibv_resp, sizeof(resp))) { free(mr); return NULL; } @@ -169,7 +171,7 @@ struct ibv_cq *bnxt_re_create_cq(struct ibv_context *ibvctx, int ncqe, struct ibv_comp_channel *channel, int vec) { struct bnxt_re_cq *cq; - struct ubnxt_re_cq_req cmd; + struct ubnxt_re_cq cmd; struct ubnxt_re_cq_resp resp; struct bnxt_re_context *cntx = to_bnxt_re_context(ibvctx); @@ -196,8 +198,8 @@ struct ibv_cq *bnxt_re_create_cq(struct ibv_context *ibvctx, int ncqe, memset(&resp, 0, sizeof(resp)); if (ibv_cmd_create_cq(ibvctx, ncqe, channel, vec, - &cq->ibvcq, &cmd.cmd, sizeof(cmd), - &resp.resp, sizeof(resp))) + &cq->ibvcq, &cmd.ibv_cmd, sizeof(cmd), + &resp.ibv_resp, sizeof(resp))) goto cmdfail; cq->cqid = resp.cqid; @@ -873,7 +875,7 @@ struct ibv_qp *bnxt_re_create_qp(struct ibv_pd *ibvpd, struct ibv_qp_init_attr *attr) { struct bnxt_re_qp *qp; - struct ubnxt_re_qp_req req; + struct ubnxt_re_qp req; struct ubnxt_re_qp_resp resp; struct bnxt_re_qpcap *cap; @@ -898,8 +900,8 @@ struct ibv_qp *bnxt_re_create_qp(struct ibv_pd *ibvpd, req.qprva = qp->rqq ? (uintptr_t)qp->rqq->va : 0; req.qp_handle = (uintptr_t)qp; - if (ibv_cmd_create_qp(ibvpd, &qp->ibvqp, attr, &req.cmd, sizeof(req), - &resp.resp, sizeof(resp))) { + if (ibv_cmd_create_qp(ibvpd, &qp->ibvqp, attr, &req.ibv_cmd, sizeof(req), + &resp.ibv_resp, sizeof(resp))) { goto failcmd; }