From patchwork Fri May 12 10:24:22 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devesh Sharma X-Patchwork-Id: 9723867 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 4F197600CB for ; Fri, 12 May 2017 10:24:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 436A3287F3 for ; Fri, 12 May 2017 10:24:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3846128801; Fri, 12 May 2017 10:24:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C4DB3287F3 for ; Fri, 12 May 2017 10:24:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756902AbdELKYj (ORCPT ); Fri, 12 May 2017 06:24:39 -0400 Received: from mail-pg0-f52.google.com ([74.125.83.52]:33883 "EHLO mail-pg0-f52.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756901AbdELKYi (ORCPT ); Fri, 12 May 2017 06:24:38 -0400 Received: by mail-pg0-f52.google.com with SMTP id u28so28409118pgn.1 for ; Fri, 12 May 2017 03:24:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=k3fP1lX/EU5Dr+uBCV46LhXC+AM2O8Kr5jIIY7tqUHA=; b=Teev3Jx+A2s24F9fA18/ZpWsx31t8yTefzK8dnvCdWqRm3zYvL0JFdaSUMsM2fwGGB ISGmeb1EMo740toWnePu9+vajSYqmvExdXReE+bk2cTpqQ5BS5/nLD+wm0NluEtTUOb0 8z0DikCyHFVjPt7ryKeXl4QqpEA582FEgClHU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=k3fP1lX/EU5Dr+uBCV46LhXC+AM2O8Kr5jIIY7tqUHA=; b=RBlTF3AH68WycrgRY4mAfAJRo6gC2KtWKAKgRM7N37fWZD9ioecQP9l3nSE4uGYL5m fVYUB9a5SQERoZKQniMrni6+zQ2Od1WlsED7MNenHB8mD5GF0E3K6dQpoa8T3IWJUA/D OvBvwCendt4E2hzUHe5lpUyvtcmo81sZoNfjfFfsJcG17rz3V37Ab2izo9eclZG3+yyb ehjaPncV/VJ3jMHmR++J+dbICq8+obICiJCfSHVyRzssnz46C7XtDpx3cUKTIzBkKTW9 Wc2a9HsrXTjW1nj2AAdUMXsKfFPvXEeTjUz+jG8yGSP0sWvhfYC6bxBfptTzRfkup6y4 b9GA== X-Gm-Message-State: AODbwcBUDwNwYEw3rJ9xDZmEckE4RcFaTLo4BLv1ZQSWxx9xu8O1PYnF ruuBbZJRTDvq3cIK6ybjgA== X-Received: by 10.98.108.68 with SMTP id h65mr3575098pfc.98.1494584676894; Fri, 12 May 2017 03:24:36 -0700 (PDT) Received: from neo00-el73.iig.avagotech.net ([192.19.239.250]) by smtp.gmail.com with ESMTPSA id c12sm4901872pfl.79.2017.05.12.03.24.35 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 12 May 2017 03:24:36 -0700 (PDT) From: Devesh Sharma To: linux-rdma@vger.kernel.org Subject: [PATCH 1/5] libbnxt_re: sq needs to be augmented by 128B Date: Fri, 12 May 2017 06:24:22 -0400 Message-Id: <1494584666-11064-2-git-send-email-devesh.sharma@broadcom.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1494584666-11064-1-git-send-email-devesh.sharma@broadcom.com> References: <1494584666-11064-1-git-send-email-devesh.sharma@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Somnath Kotur In order to avoid out of order completions there is a requirement to augment the SQ of any QP by 128B. This patch adds (128 + 1)B delta while SQ allocation and updates the queue-full determining logic. bnxt_re_query_qp is relying on whatever attributes driver returns. Thus, dropping to update the max_send_wr and max_recv_wr caps during bnxt_re_create_qp. Signed-off-by: Somnath Kotur Signed-off-by: Devesh Sharma --- providers/bnxt_re/bnxt_re-abi.h | 2 ++ providers/bnxt_re/memory.h | 10 +++++++++- providers/bnxt_re/verbs.c | 9 ++++++--- 3 files changed, 17 insertions(+), 4 deletions(-) diff --git a/providers/bnxt_re/bnxt_re-abi.h b/providers/bnxt_re/bnxt_re-abi.h index 205d8c4..73beef5 100644 --- a/providers/bnxt_re/bnxt_re-abi.h +++ b/providers/bnxt_re/bnxt_re-abi.h @@ -43,6 +43,8 @@ #define BNXT_RE_ABI_VERSION 1 +#define BNXT_RE_FULL_FLAG_DELTA 0x80 + enum bnxt_re_wr_opcode { BNXT_RE_WR_OPCD_SEND = 0x00, BNXT_RE_WR_OPCD_SEND_IMM = 0x01, diff --git a/providers/bnxt_re/memory.h b/providers/bnxt_re/memory.h index 0150d80..aac0ff3 100644 --- a/providers/bnxt_re/memory.h +++ b/providers/bnxt_re/memory.h @@ -49,6 +49,14 @@ struct bnxt_re_queue { uint32_t head; uint32_t tail; uint32_t stride; + /* Represents the difference between the real queue depth allocated in + * HW and the user requested queue depth and is used to correctly flag + * queue full condition based on user supplied queue depth. + * This value can vary depending on the type of queue and any HW + * requirements that mandate keeping a fixed gap between the producer + * and the consumer indices in the queue + */ + uint32_t diff; pthread_spinlock_t qlock; }; @@ -86,7 +94,7 @@ static inline void iowrite32(__u32 *dst, __le32 *src) /* Basic queue operation */ static inline uint32_t bnxt_re_is_que_full(struct bnxt_re_queue *que) { - return (((que->tail + 1) & (que->depth - 1)) == que->head); + return (((que->diff + que->tail) & (que->depth - 1)) == que->head); } static inline uint32_t bnxt_re_is_que_empty(struct bnxt_re_queue *que) diff --git a/providers/bnxt_re/verbs.c b/providers/bnxt_re/verbs.c index 2768a56..35eb12d 100644 --- a/providers/bnxt_re/verbs.c +++ b/providers/bnxt_re/verbs.c @@ -793,7 +793,11 @@ static int bnxt_re_alloc_queues(struct bnxt_re_qp *qp, que = qp->sqq; que->stride = bnxt_re_get_sqe_sz(); - que->depth = roundup_pow_of_two(attr->cap.max_send_wr + 1); + /* 8916 adjustment */ + que->depth = roundup_pow_of_two(attr->cap.max_send_wr + 1 + + BNXT_RE_FULL_FLAG_DELTA); + que->diff = que->depth - attr->cap.max_send_wr; + /* psn_depth extra entries of size que->stride */ psn_depth = (que->depth * sizeof(struct bnxt_re_psns)) / que->stride; @@ -828,6 +832,7 @@ static int bnxt_re_alloc_queues(struct bnxt_re_qp *qp, que = qp->rqq; que->stride = bnxt_re_get_rqe_sz(); que->depth = roundup_pow_of_two(attr->cap.max_recv_wr + 1); + que->diff = que->depth - attr->cap.max_recv_wr; ret = bnxt_re_alloc_aligned(qp->rqq, pg_size); if (ret) goto fail; @@ -888,9 +893,7 @@ struct ibv_qp *bnxt_re_create_qp(struct ibv_pd *ibvpd, qp->rcq = to_bnxt_re_cq(attr->recv_cq); qp->udpi = &cntx->udpi; /* Save/return the altered Caps. */ - attr->cap.max_send_wr = cap->max_swr; cap->max_ssge = attr->cap.max_send_sge; - attr->cap.max_recv_wr = cap->max_rwr; cap->max_rsge = attr->cap.max_recv_sge; cap->max_inline = attr->cap.max_inline_data; cap->sqsig = attr->sq_sig_all;