From patchwork Wed Apr 3 11:33:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kamal Heib X-Patchwork-Id: 10883503 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AD66C17E0 for ; Wed, 3 Apr 2019 11:35:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 953ED289AF for ; Wed, 3 Apr 2019 11:35:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 89897289CB; Wed, 3 Apr 2019 11:35:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id F347F289AF for ; Wed, 3 Apr 2019 11:35:35 +0000 (UTC) Received: from localhost ([127.0.0.1]:46112 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hBeB5-0003Un-9A for patchwork-qemu-devel@patchwork.kernel.org; Wed, 03 Apr 2019 07:35:35 -0400 Received: from eggs.gnu.org ([209.51.188.92]:53684) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hBe9h-0002Y8-IV for qemu-devel@nongnu.org; Wed, 03 Apr 2019 07:34:10 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hBe9f-0006NJ-6e for qemu-devel@nongnu.org; Wed, 03 Apr 2019 07:34:09 -0400 Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]:37738) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hBe9Z-0006DU-Fg for qemu-devel@nongnu.org; Wed, 03 Apr 2019 07:34:05 -0400 Received: by mail-wr1-x442.google.com with SMTP id w10so20834680wrm.4 for ; Wed, 03 Apr 2019 04:34:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LerJD0JSpo5l+BG9wB09n8fUNkVBNgzdjGgv3NRBx7c=; b=MD+IXk+enUjK2CGcjfd049ywTBh0LuwCOZ5WgEZ1pgUktFudJtOQJjRkQwk8tCChyN LeWGrS2nvLca5XzoPlDRjKO1/MA6vX9tJKQVlLVxi5Z2MmBS7AVDxUv4rCaUZZ4N05sW oH7uGfhZp6HfckJuYoCxL0t6d37GQR8qf8zC/vHH726v6sImVNfGxGxaySuiwaQAlPDb e6p7kRL758JVw8vKsbwhZFGp1ryfSevHh5f8477MdI6daJK3e13CgoMIW2FAK8w61w9T HM/qpagGuZo5C561WoK6oKz8V1tqpKYubjDBDYYQE9iBcQVCss6Mc+icZS64e1YoZn++ WlHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LerJD0JSpo5l+BG9wB09n8fUNkVBNgzdjGgv3NRBx7c=; b=DSXcXjNxCbFNr9q6onKTj0yeKafkd0f6+AUlUfoy8LO+Qb7qotPPtByS1ec46gYrfh Wcc77mjf6Wr8k+33i/MV7vaOfxOEv+gr1A8Yr7ECPLi2jIAwNA8C0rapdsFa3cBHLEQ5 MPJeK3+Jx02eT9/XS1LjvOuaMOjEWLnPWL2+3vsCWyQm5YD1feq62/ODYyThSyyMWmVH JwOk53mfSuPl7cDafUtgdIY/uiHsnx4yaCTBWnJcCCycHLlmsQAyPmnkFCokbhP28iL9 fipcOuLyOHwvy9Jq6xGIbf9BN89VLZDz8dGy5upv3N2s2g99BKflakFi9NxWZ6ezkFvO asLw== X-Gm-Message-State: APjAAAUBVHZNu1cg6Bbfkc+gGzFfGBs4NxwFj/svsj9m+AIvx4Vku52t 7oxVT7gRNWUOiWrtOGC4LKT2GJkl X-Google-Smtp-Source: APXvYqzULPkMf0yLmHxEqgFCsFH+79kvTqM/Lo5VZJTVw/Jd7GzsHLWYZ6uwDg+4YCX8cN9qv7902g== X-Received: by 2002:adf:e610:: with SMTP id p16mr2510195wrm.64.1554291239484; Wed, 03 Apr 2019 04:33:59 -0700 (PDT) Received: from kheib-workstation.redhat.com ([192.116.94.220]) by smtp.gmail.com with ESMTPSA id u189sm8946635wme.25.2019.04.03.04.33.58 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 03 Apr 2019 04:33:58 -0700 (PDT) From: Kamal Heib To: qemu-devel@nongnu.org Date: Wed, 3 Apr 2019 14:33:41 +0300 Message-Id: <20190403113343.26384-3-kamalheib1@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190403113343.26384-1-kamalheib1@gmail.com> References: <20190403113343.26384-1-kamalheib1@gmail.com> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::442 Subject: [Qemu-devel] [PATCH 2/4] hw/rdma: Add support for managing SRQ resource X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kamal Heib , Yuval Shaia Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Adding the required functions and definitions for support managing the shared receive queues (SRQs). Signed-off-by: Kamal Heib Reviewed-by: Yuval Shaia --- hw/rdma/rdma_rm.c | 93 ++++++++++++++++++++++++++++++++++++++++++ hw/rdma/rdma_rm.h | 10 +++++ hw/rdma/rdma_rm_defs.h | 8 ++++ 3 files changed, 111 insertions(+) diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index b683506b8616..c4fb140dcd96 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -544,6 +544,96 @@ void rdma_rm_dealloc_qp(RdmaDeviceResources *dev_res, uint32_t qp_handle) rdma_res_tbl_dealloc(&dev_res->qp_tbl, qp->qpn); } +RdmaRmSRQ *rdma_rm_get_srq(RdmaDeviceResources *dev_res, uint32_t srq_handle) +{ + return rdma_res_tbl_get(&dev_res->srq_tbl, srq_handle); +} + +int rdma_rm_alloc_srq(RdmaDeviceResources *dev_res, uint32_t pd_handle, + uint32_t max_wr, uint32_t max_sge, uint32_t srq_limit, + uint32_t *srq_handle, void *opaque) +{ + RdmaRmSRQ *srq; + RdmaRmPD *pd; + int rc; + + pd = rdma_rm_get_pd(dev_res, pd_handle); + if (!pd) { + return -EINVAL; + } + + srq = rdma_res_tbl_alloc(&dev_res->srq_tbl, srq_handle); + if (!srq) { + return -ENOMEM; + } + + rc = rdma_backend_create_srq(&srq->backend_srq, &pd->backend_pd, + max_wr, max_sge, srq_limit); + if (rc) { + rc = -EIO; + goto out_dealloc_srq; + } + + srq->opaque = opaque; + + return 0; + +out_dealloc_srq: + rdma_res_tbl_dealloc(&dev_res->srq_tbl, *srq_handle); + + return rc; +} + +int rdma_rm_query_srq(RdmaDeviceResources *dev_res, uint32_t srq_handle, + struct ibv_srq_attr *srq_attr) +{ + RdmaRmSRQ *srq; + + srq = rdma_rm_get_srq(dev_res, srq_handle); + if (!srq) { + return -EINVAL; + } + + return rdma_backend_query_srq(&srq->backend_srq, srq_attr); +} + +int rdma_rm_modify_srq(RdmaDeviceResources *dev_res, uint32_t srq_handle, + struct ibv_srq_attr *srq_attr, int srq_attr_mask) +{ + RdmaRmSRQ *srq; + + srq = rdma_rm_get_srq(dev_res, srq_handle); + if (!srq) { + return -EINVAL; + } + + if ((srq_attr_mask & IBV_SRQ_LIMIT) && + (srq_attr->srq_limit == 0)) { + return -EINVAL; + } + + if ((srq_attr_mask & IBV_SRQ_MAX_WR) && + (srq_attr->max_wr == 0)) { + return -EINVAL; + } + + return rdma_backend_modify_srq(&srq->backend_srq, srq_attr, + srq_attr_mask); +} + +void rdma_rm_dealloc_srq(RdmaDeviceResources *dev_res, uint32_t srq_handle) +{ + RdmaRmSRQ *srq; + + srq = rdma_rm_get_srq(dev_res, srq_handle); + if (!srq) { + return; + } + + rdma_backend_destroy_srq(&srq->backend_srq, dev_res); + rdma_res_tbl_dealloc(&dev_res->srq_tbl, srq_handle); +} + void *rdma_rm_get_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t cqe_ctx_id) { void **cqe_ctx; @@ -673,6 +763,8 @@ int rdma_rm_init(RdmaDeviceResources *dev_res, struct ibv_device_attr *dev_attr) res_tbl_init("CQE_CTX", &dev_res->cqe_ctx_tbl, dev_attr->max_qp * dev_attr->max_qp_wr, sizeof(void *)); res_tbl_init("UC", &dev_res->uc_tbl, MAX_UCS, sizeof(RdmaRmUC)); + res_tbl_init("SRQ", &dev_res->srq_tbl, dev_attr->max_srq, + sizeof(RdmaRmSRQ)); init_ports(dev_res); @@ -691,6 +783,7 @@ void rdma_rm_fini(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev, fini_ports(dev_res, backend_dev, ifname); + res_tbl_free(&dev_res->srq_tbl); res_tbl_free(&dev_res->uc_tbl); res_tbl_free(&dev_res->cqe_ctx_tbl); res_tbl_free(&dev_res->qp_tbl); diff --git a/hw/rdma/rdma_rm.h b/hw/rdma/rdma_rm.h index 4f03f9b8c5f1..e88ab95e264b 100644 --- a/hw/rdma/rdma_rm.h +++ b/hw/rdma/rdma_rm.h @@ -65,6 +65,16 @@ int rdma_rm_query_qp(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev, int attr_mask, struct ibv_qp_init_attr *init_attr); void rdma_rm_dealloc_qp(RdmaDeviceResources *dev_res, uint32_t qp_handle); +RdmaRmSRQ *rdma_rm_get_srq(RdmaDeviceResources *dev_res, uint32_t srq_handle); +int rdma_rm_alloc_srq(RdmaDeviceResources *dev_res, uint32_t pd_handle, + uint32_t max_wr, uint32_t max_sge, uint32_t srq_limit, + uint32_t *srq_handle, void *opaque); +int rdma_rm_query_srq(RdmaDeviceResources *dev_res, uint32_t srq_handle, + struct ibv_srq_attr *srq_attr); +int rdma_rm_modify_srq(RdmaDeviceResources *dev_res, uint32_t srq_handle, + struct ibv_srq_attr *srq_attr, int srq_attr_mask); +void rdma_rm_dealloc_srq(RdmaDeviceResources *dev_res, uint32_t srq_handle); + int rdma_rm_alloc_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t *cqe_ctx_id, void *ctx); void *rdma_rm_get_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t cqe_ctx_id); diff --git a/hw/rdma/rdma_rm_defs.h b/hw/rdma/rdma_rm_defs.h index e774af528022..7bdd9f291f9f 100644 --- a/hw/rdma/rdma_rm_defs.h +++ b/hw/rdma/rdma_rm_defs.h @@ -33,6 +33,7 @@ #define MAX_QP_RD_ATOM 16 #define MAX_QP_INIT_RD_ATOM 16 #define MAX_AH 64 +#define MAX_SRQ 512 #define MAX_RM_TBL_NAME 16 #define MAX_CONSEQ_EMPTY_POLL_CQ 4096 /* considered as error above this */ @@ -89,6 +90,12 @@ typedef struct RdmaRmQP { enum ibv_qp_state qp_state; } RdmaRmQP; +typedef struct RdmaRmSRQ { + RdmaBackendSRQ backend_srq; + uint32_t recv_cq_handle; + void *opaque; +} RdmaRmSRQ; + typedef struct RdmaRmGid { union ibv_gid gid; int backend_gid_index; @@ -129,6 +136,7 @@ struct RdmaDeviceResources { RdmaRmResTbl qp_tbl; RdmaRmResTbl cq_tbl; RdmaRmResTbl cqe_ctx_tbl; + RdmaRmResTbl srq_tbl; GHashTable *qp_hash; /* Keeps mapping between real and emulated */ QemuMutex lock; RdmaRmStats stats;