From patchwork Fri Apr 21 17:30:17 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devesh Sharma X-Patchwork-Id: 9693295 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 2D3BC601D4 for ; Fri, 21 Apr 2017 17:59:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 239BE28663 for ; Fri, 21 Apr 2017 17:59:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1879028668; Fri, 21 Apr 2017 17:59:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AAC9328663 for ; Fri, 21 Apr 2017 17:58:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1041572AbdDURnZ (ORCPT ); Fri, 21 Apr 2017 13:43:25 -0400 Received: from mail-io0-f182.google.com ([209.85.223.182]:33972 "EHLO mail-io0-f182.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1034482AbdDURkf (ORCPT ); Fri, 21 Apr 2017 13:40:35 -0400 Received: by mail-io0-f182.google.com with SMTP id a103so128375819ioj.1 for ; Fri, 21 Apr 2017 10:40:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=lhzTHIcfr0lx6ba5FY9ddbG/TXPhoDlfTYzVDfYpgsM=; b=Q7B3z2RpDXvGnprxrHpzNNpbASl6FBOy7aWSawTU6++0Xl36kqYhVYKk6/0KhxkqS6 RM0ca28TVdP8A5yqRd++3OvrK3wqA4IfYhrJWzBd4hctcC1mdnHlHbt7rmJkTFA5gLod hmqzqsiUhD+f4LzF5SKLr5M28YSc3TdXVxG10= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=lhzTHIcfr0lx6ba5FY9ddbG/TXPhoDlfTYzVDfYpgsM=; b=YzyRVcO71epYnNbqdFQNUYr/MS9qDeCbabLabsqsbnV7gZXlxnptrCXxM4BZwnx7W6 1D3QJDx/9yh3drgo7DALGb3qZMesLCsd7A5EBC26UvmkKjxbxg97OZ3ZWrEv6ImwoQO7 7G6Syr2r3YIzv097PN+NX/VHikqTyj0Ro0839clC4yctEuwrxIGXw/m4KmvXZWgnXKgD PkTm1Hi2TuWtLqQsMy5gSJkYmk0NNTvKTWwoJ54RgPVpVN4bYl1+kuDtMIFDU4kqeNXX 2OIB1OiE+b4Ba3hIZt8zIc7XjVO/ycqI4JawFnApEJM0sNz6yDzSQrRbcT6kJg31q6n+ B9nw== X-Gm-Message-State: AN3rC/4gsx9pIv3/qB1U4zsgO+xXz6EXkksmXU1xh7RthuvcibXARA2I 8BEcwR7+RLvPPh1DUzM= X-Received: by 10.99.56.91 with SMTP id h27mr13346581pgn.97.1492795861495; Fri, 21 Apr 2017 10:31:01 -0700 (PDT) Received: from neo00-el73.iig.avagotech.net ([192.19.239.250]) by smtp.gmail.com with ESMTPSA id q1sm17217355pfl.89.2017.04.21.10.30.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Apr 2017 10:31:00 -0700 (PDT) From: Devesh Sharma To: linux-rdma@vger.kernel.org Subject: [rdma-core v4 7/8] libbnxt_re: Enable polling for UD completions Date: Fri, 21 Apr 2017 13:30:17 -0400 Message-Id: <1492795818-29618-8-git-send-email-devesh.sharma@broadcom.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1492795818-29618-1-git-send-email-devesh.sharma@broadcom.com> References: <1492795818-29618-1-git-send-email-devesh.sharma@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch adds support to allow polling of send and recv completions for a UD qp. Signed-off-by: Sriharsha Basavapatna Signed-off-by: Somnath Kotur Signed-off-by: Selvin Xavier Signed-off-by: Devesh Sharma --- providers/bnxt_re/bnxt_re-abi.h | 6 ++++++ providers/bnxt_re/verbs.c | 43 +++++++++++++++++++++++++++-------------- 2 files changed, 35 insertions(+), 14 deletions(-) diff --git a/providers/bnxt_re/bnxt_re-abi.h b/providers/bnxt_re/bnxt_re-abi.h index 36b74cd..db11322 100644 --- a/providers/bnxt_re/bnxt_re-abi.h +++ b/providers/bnxt_re/bnxt_re-abi.h @@ -174,6 +174,12 @@ enum bnxt_re_ud_flags_mask { BNXT_RE_UD_FLAGS_ROCE_IPV6 = 0x03 }; +enum bnxt_re_ud_cqe_mask { + BNXT_RE_UD_CQE_MAC_MASK = 0xFFFFFFFFFFFFULL, + BNXT_RE_UD_CQE_SRCQPLO_MASK = 0xFFFF, + BNXT_RE_UD_CQE_SRCQPLO_SHIFT = 0x30 +}; + enum bnxt_re_shpg_offt { BNXT_RE_SHPG_BEG_RESV_OFFT = 0x00, BNXT_RE_SHPG_AVID_OFFT = 0x10, diff --git a/providers/bnxt_re/verbs.c b/providers/bnxt_re/verbs.c index 54c8906..1f8ff8e 100644 --- a/providers/bnxt_re/verbs.c +++ b/providers/bnxt_re/verbs.c @@ -338,10 +338,8 @@ static uint8_t bnxt_re_poll_scqe(struct bnxt_re_qp *qp, struct ibv_wc *ibvwc, return pcqe; } -static int bnxt_re_poll_err_rcqe(struct bnxt_re_qp *qp, - struct ibv_wc *ibvwc, - struct bnxt_re_bcqe *hdr, - struct bnxt_re_rc_cqe *rcqe) +static int bnxt_re_poll_err_rcqe(struct bnxt_re_qp *qp, struct ibv_wc *ibvwc, + struct bnxt_re_bcqe *hdr, void *cqe) { struct bnxt_re_queue *rq = qp->rqq; struct bnxt_re_wrid *rwrid; @@ -366,6 +364,10 @@ static int bnxt_re_poll_err_rcqe(struct bnxt_re_qp *qp, ibvwc->qp_num = qp->qpid; ibvwc->opcode = IBV_WC_RECV; ibvwc->byte_len = 0; + ibvwc->wc_flags = 0; + if (qp->qptyp == IBV_QPT_UD) + ibvwc->src_qp = 0; + bnxt_re_incr_head(qp->rqq); if (qp->qpst != IBV_QPS_ERR) qp->qpst = IBV_QPS_ERR; @@ -377,16 +379,32 @@ static int bnxt_re_poll_err_rcqe(struct bnxt_re_qp *qp, return 1; } +static void bnxt_re_fill_ud_cqe(struct ibv_wc *ibvwc, + struct bnxt_re_bcqe *hdr, void *cqe) +{ + struct bnxt_re_ud_cqe *ucqe = cqe; + uint32_t qpid; + + qpid = ((le32toh(hdr->qphi_rwrid) >> BNXT_RE_BCQE_SRCQP_SHIFT) & + BNXT_RE_BCQE_SRCQP_SHIFT) << 0x10; /* higher 8 bits of 24 */ + qpid |= (le64toh(ucqe->qplo_mac) >> BNXT_RE_UD_CQE_SRCQPLO_SHIFT) & + BNXT_RE_UD_CQE_SRCQPLO_MASK; /*lower 16 of 24 */ + ibvwc->src_qp = qpid; + ibvwc->wc_flags |= IBV_WC_GRH; + /*IB-stack ABI in user do not ask for MAC to be reported. */ +} + static void bnxt_re_poll_success_rcqe(struct bnxt_re_qp *qp, struct ibv_wc *ibvwc, - struct bnxt_re_bcqe *hdr, - struct bnxt_re_rc_cqe *rcqe) + struct bnxt_re_bcqe *hdr, void *cqe) { struct bnxt_re_queue *rq = qp->rqq; struct bnxt_re_wrid *rwrid; + struct bnxt_re_rc_cqe *rcqe; uint32_t head = rq->head; uint8_t flags, is_imm, is_rdma; + rcqe = cqe; rwrid = &qp->rwrid[head]; ibvwc->status = IBV_WC_SUCCESS; @@ -409,6 +427,9 @@ static void bnxt_re_poll_success_rcqe(struct bnxt_re_qp *qp, ibvwc->opcode = IBV_WC_RECV_RDMA_WITH_IMM; } + if (qp->qptyp == IBV_QPT_UD) + bnxt_re_fill_ud_cqe(ibvwc, hdr, cqe); + bnxt_re_incr_head(rq); } @@ -416,19 +437,17 @@ static uint8_t bnxt_re_poll_rcqe(struct bnxt_re_qp *qp, struct ibv_wc *ibvwc, void *cqe, int *cnt) { struct bnxt_re_bcqe *hdr; - struct bnxt_re_rc_cqe *rcqe; uint8_t status, pcqe = false; - rcqe = cqe; hdr = cqe + sizeof(struct bnxt_re_rc_cqe); status = (le32toh(hdr->flg_st_typ_ph) >> BNXT_RE_BCQE_STATUS_SHIFT) & BNXT_RE_BCQE_STATUS_MASK; *cnt = 1; if (status == BNXT_RE_RSP_ST_OK) - bnxt_re_poll_success_rcqe(qp, ibvwc, hdr, rcqe); + bnxt_re_poll_success_rcqe(qp, ibvwc, hdr, cqe); else - *cnt = bnxt_re_poll_err_rcqe(qp, ibvwc, hdr, rcqe); + *cnt = bnxt_re_poll_err_rcqe(qp, ibvwc, hdr, cqe); return pcqe; } @@ -486,9 +505,6 @@ static int bnxt_re_poll_one(struct bnxt_re_cq *cq, int nwc, struct ibv_wc *wc) (uintptr_t)le64toh(scqe->qp_handle); if (!qp) break; /*stale cqe. should be rung.*/ - if (qp->qptyp == IBV_QPT_UD) - goto bail; /* TODO: Add UD poll */ - pcqe = bnxt_re_poll_scqe(qp, wc, cqe, &cnt); break; case BNXT_RE_WC_TYPE_RECV_RC: @@ -503,7 +519,6 @@ static int bnxt_re_poll_one(struct bnxt_re_cq *cq, int nwc, struct ibv_wc *wc) goto bail; /*TODO: Add SRQ poll */ pcqe = bnxt_re_poll_rcqe(qp, wc, cqe, &cnt); - /* TODO: Process UD rcqe */ break; case BNXT_RE_WC_TYPE_RECV_RAW: break;