From patchwork Wed Dec 12 15:45:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Arumugam, Kamenee" X-Patchwork-Id: 10726601 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B2853174F for ; Wed, 12 Dec 2018 15:45:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A1A3C2AAA1 for ; Wed, 12 Dec 2018 15:45:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9604F2AC78; Wed, 12 Dec 2018 15:45:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 137122AAA1 for ; Wed, 12 Dec 2018 15:45:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726922AbeLLPpm (ORCPT ); Wed, 12 Dec 2018 10:45:42 -0500 Received: from mga06.intel.com ([134.134.136.31]:32477 "EHLO mga06.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726281AbeLLPpm (ORCPT ); Wed, 12 Dec 2018 10:45:42 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 12 Dec 2018 07:45:41 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,344,1539673200"; d="scan'208";a="258915062" Received: from scymds02.sc.intel.com ([10.82.195.37]) by orsmga004.jf.intel.com with ESMTP; 12 Dec 2018 07:45:41 -0800 Received: from scvm10.sc.intel.com (scvm10.sc.intel.com [10.82.195.27]) by scymds02.sc.intel.com with ESMTP id wBCFjf6t029079 for ; Wed, 12 Dec 2018 07:45:41 -0800 Received: from scvm10.sc.intel.com (localhost [127.0.0.1]) by scvm10.sc.intel.com with ESMTP id wBCFjfMi029429 for ; Wed, 12 Dec 2018 07:45:41 -0800 Subject: [PATCH] hfi1verbs: Update rvt cq headers in rdma_core hfi1 provider To: linux-rdma@vger.kernel.org From: Kamenee Arumugam Date: Wed, 12 Dec 2018 07:45:41 -0800 Message-ID: <20181212154541.29403.28684.stgit@scvm10.sc.intel.com> User-Agent: StGit/0.16 MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP hfi1_wc struct is the same struct as ib_uverbs_wc and hfi1_cq_wc is not in the uapi header. Remove hfi1_wc and hfi1_cq_wc struct from providers. Include rvt-abi.h header that have rvt_cq_wc struct definition into kernel-headers. The member ib_uverbs_wc arrray in rvt_cq_wc is replaced with one in rvt-abi. Therefore,plus one to the number of cq entries. Reviewed-by: Dennis Dalessandro Reviewed-by: Mike Marciniszyn Signed-off-by: Kamenee Arumugam --- kernel-headers/CMakeLists.txt | 1 + providers/hfi1verbs/hfiverbs.h | 37 +++++-------------------------------- providers/hfi1verbs/verbs.c | 23 +++++++++++++---------- 3 files changed, 19 insertions(+), 42 deletions(-) diff --git a/kernel-headers/CMakeLists.txt b/kernel-headers/CMakeLists.txt index e00eadb..e6b4d6e 100644 --- a/kernel-headers/CMakeLists.txt +++ b/kernel-headers/CMakeLists.txt @@ -25,6 +25,7 @@ publish_internal_headers(rdma rdma/rdma_user_rxe.h rdma/rvt-abi.h rdma/vmw_pvrdma-abi.h + rdma/rvt-abi.h ) publish_internal_headers(rdma/hfi diff --git a/providers/hfi1verbs/hfiverbs.h b/providers/hfi1verbs/hfiverbs.h index 070a01c..7ce27c8 100644 --- a/providers/hfi1verbs/hfiverbs.h +++ b/providers/hfi1verbs/hfiverbs.h @@ -65,7 +65,8 @@ #include #include - +#define RDMA_ATOMIC_UAPI(_type, _name) _Atomic(_type) _name +#include "rdma/rvt-abi.h" #define PFX "hfi1: " struct hfi1_device { @@ -77,39 +78,11 @@ struct hfi1_context { struct verbs_context ibv_ctx; }; -/* - * This structure needs to have the same size and offsets as - * the kernel's ib_wc structure since it is memory mapped. - */ -struct hfi1_wc { - uint64_t wr_id; - enum ibv_wc_status status; - enum ibv_wc_opcode opcode; - uint32_t vendor_err; - uint32_t byte_len; - uint32_t imm_data; /* in network byte order */ - uint32_t qp_num; - uint32_t src_qp; - enum ibv_wc_flags wc_flags; - uint16_t pkey_index; - uint16_t slid; - uint8_t sl; - uint8_t dlid_path_bits; - uint8_t port_num; -}; - -struct hfi1_cq_wc { - _Atomic(uint32_t) head; - _Atomic(uint32_t) tail; - struct hfi1_wc queue[1]; -}; - struct hfi1_cq { - struct ibv_cq ibv_cq; - struct hfi1_cq_wc *queue; - pthread_spinlock_t lock; + struct ibv_cq ibv_cq; + struct rvt_cq_wc *queue; + pthread_spinlock_t lock; }; - /* * Receive work request queue entry. * The size of the sg_list is determined when the QP is created and stored diff --git a/providers/hfi1verbs/verbs.c b/providers/hfi1verbs/verbs.c index ff001f6..dcf6714 100644 --- a/providers/hfi1verbs/verbs.c +++ b/providers/hfi1verbs/verbs.c @@ -63,10 +63,15 @@ #include #include #include - #include "hfiverbs.h" #include "hfi-abi.h" +static size_t hfi1_cq_size(int cqe) +{ + return sizeof(struct rvt_cq_wc) + + sizeof(struct ib_uverbs_wc) * (cqe + 1); +} + int hfi1_query_device(struct ibv_context *context, struct ibv_device_attr *attr) { @@ -186,7 +191,7 @@ struct ibv_cq *hfi1_create_cq(struct ibv_context *context, int cqe, return NULL; } - size = sizeof(struct hfi1_cq_wc) + sizeof(struct hfi1_wc) * cqe; + size = hfi1_cq_size(cqe); cq->queue = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, context->cmd_fd, resp.offset); if ((void *) cq->queue == MAP_FAILED) { @@ -231,8 +236,7 @@ int hfi1_resize_cq(struct ibv_cq *ibcq, int cqe) memset(&resp, 0, sizeof(resp)); pthread_spin_lock(&cq->lock); /* Save the old size so we can unmmap the queue. */ - size = sizeof(struct hfi1_cq_wc) + - (sizeof(struct hfi1_wc) * cq->ibv_cq.cqe); + size = hfi1_cq_size(cq->ibv_cq.cqe); ret = ibv_cmd_resize_cq(ibcq, cqe, &cmd, sizeof cmd, &resp.ibv_resp, sizeof resp); if (ret) { @@ -240,8 +244,7 @@ int hfi1_resize_cq(struct ibv_cq *ibcq, int cqe) return ret; } (void) munmap(cq->queue, size); - size = sizeof(struct hfi1_cq_wc) + - (sizeof(struct hfi1_wc) * cq->ibv_cq.cqe); + size = hfi1_cq_size(cqe); cq->queue = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, ibcq->context->cmd_fd, resp.offset); ret = errno; @@ -269,8 +272,8 @@ int hfi1_destroy_cq(struct ibv_cq *ibcq) if (ret) return ret; - (void) munmap(cq->queue, sizeof(struct hfi1_cq_wc) + - (sizeof(struct hfi1_wc) * cq->ibv_cq.cqe)); + (void) munmap(cq->queue, hfi1_cq_size(cq->ibv_cq.cqe)); + free(cq); return 0; } @@ -288,7 +291,7 @@ int hfi1_destroy_cq_v1(struct ibv_cq *ibcq) int hfi1_poll_cq(struct ibv_cq *ibcq, int ne, struct ibv_wc *wc) { struct hfi1_cq *cq = to_icq(ibcq); - struct hfi1_cq_wc *q; + struct rvt_cq_wc *q; int npolled; uint32_t tail; @@ -300,7 +303,7 @@ int hfi1_poll_cq(struct ibv_cq *ibcq, int ne, struct ibv_wc *wc) break; /* Make sure entry is read after head index is read. */ atomic_thread_fence(memory_order_acquire); - memcpy(wc, &q->queue[tail], sizeof(*wc)); + memcpy(wc, &q->uqueue[tail], sizeof(*wc)); if (tail == cq->ibv_cq.cqe) tail = 0; else