From patchwork Wed May 10 10:45:31 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 9719657 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id CEDC960236 for ; Wed, 10 May 2017 10:46:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C53392857C for ; Wed, 10 May 2017 10:46:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BA1E52857F; Wed, 10 May 2017 10:46:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3F43D28576 for ; Wed, 10 May 2017 10:46:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752589AbdEJKqO (ORCPT ); Wed, 10 May 2017 06:46:14 -0400 Received: from mail-qk0-f169.google.com ([209.85.220.169]:35827 "EHLO mail-qk0-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752285AbdEJKqN (ORCPT ); Wed, 10 May 2017 06:46:13 -0400 Received: by mail-qk0-f169.google.com with SMTP id a72so24556021qkj.2 for ; Wed, 10 May 2017 03:46:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=0PHd4fxYbB+QXJrdGKlxN59AJcYwU2oCdvH9/sPFTbk=; b=dpplw5mOLWZKrDpsejPobaPb+00ZpFPPjYO/a440oDxJEzbrDWv5QZXQMmbK6tOjdO ZqtYFiVewGmsyuR2cWTy0n0VTCtIM09BwZjNG4DqZhn9C6cqjT1AyiPCPHPVIHkJkA9L qDi6ndQ6TjCRMjIZozu5t0EqjGaeaYXDDxlQY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=0PHd4fxYbB+QXJrdGKlxN59AJcYwU2oCdvH9/sPFTbk=; b=EpaZ8IbXeaF2fCGCDQkTbwZYvW+IKNc4UFEjdDIVlU/DmYMWUgrDIC8Yf/syrg6fHg y5l5bEWN4/iG6hOcyWAI+nBgXbDU9Z/icYEBPut27j0zBXJaeUcG6BIt79xawQKcewFM gk/UT7TN3d6EsfcYsA1DcvNidLkOxtUzQFwVpX7TUpGrv7/UANuYWAd4iiVvN8S9D2P+ MPkC4/QMRY4BpEQ39UWFrIQyXbFay6hj/XkvIKCKtmtDiv9wkxavwKiO+jMW6vb5lYw6 a37Jnpp7PJYqq1EL7SVRfQwrBnd+uWIN5unu7i+nF1EewcAE5jih2F+eqnrlSdumeP7O ZfIw== X-Gm-Message-State: AODbwcDLQNWzp5lx2ZSS2vsrGdvpVxuZ2aZUck5cvF8R/Hu/xRXP5Lwz E6q6ZQwdWyKnNdOx X-Received: by 10.55.198.197 with SMTP id s66mr5034230qkl.179.1494413172701; Wed, 10 May 2017 03:46:12 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net ([192.19.239.250]) by smtp.gmail.com with ESMTPSA id k10sm1868531qke.3.2017.05.10.03.46.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 10 May 2017 03:46:12 -0700 (PDT) From: Selvin Xavier To: dledford@redhat.com, linux-rdma@vger.kernel.org Cc: Somnath Kotur , Selvin Xavier Subject: [PATCH for-next 06/14] RDMA/bnxt_re: Add HW workaround for avoiding stall for UD QPs Date: Wed, 10 May 2017 03:45:31 -0700 Message-Id: <1494413139-11883-7-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1494413139-11883-1-git-send-email-selvin.xavier@broadcom.com> References: <1494413139-11883-1-git-send-email-selvin.xavier@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Somnath Kotur HW stalls out after 0x800000 WQEs are posted for UD QPs. To workaround this problem, driver will send a modify_qp cmd to the HW at around the halfway mark(0x400000) so that BONO can accordingly modify the QP context in the HW to prevent this stall. This workaround needs to be done for UD, QP1 and Raw Ethertype packets. Added a counter to keep track of WQEs posted during post_send. Signed-off-by: Somnath Kotur Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 2 ++ drivers/infiniband/hw/bnxt_re/ib_verbs.c | 18 ++++++++++++++++++ drivers/infiniband/hw/bnxt_re/qplib_fp.c | 3 +++ drivers/infiniband/hw/bnxt_re/qplib_fp.h | 1 + 4 files changed, 24 insertions(+) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index fad04b2..277c2da 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -56,6 +56,8 @@ #define BNXT_RE_MAX_SRQC_COUNT (64 * 1024) #define BNXT_RE_MAX_CQ_COUNT (64 * 1024) +#define BNXT_RE_UD_QP_HW_STALL 0x400000 + struct bnxt_re_work { struct work_struct work; unsigned long event; diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index eabc97e..61703f3 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -2048,6 +2048,22 @@ static int bnxt_re_copy_wr_payload(struct bnxt_re_dev *rdev, return payload_sz; } +static void bnxt_ud_qp_hw_stall_workaround(struct bnxt_re_qp *qp) +{ + if ((qp->ib_qp.qp_type == IB_QPT_UD || + qp->ib_qp.qp_type == IB_QPT_GSI || + qp->ib_qp.qp_type == IB_QPT_RAW_ETHERTYPE) && + qp->qplib_qp.wqe_cnt == BNXT_RE_UD_QP_HW_STALL) { + int qp_attr_mask; + struct ib_qp_attr qp_attr; + + qp_attr_mask = IB_QP_STATE; + qp_attr.qp_state = IB_QPS_RTS; + bnxt_re_modify_qp(&qp->ib_qp, &qp_attr, qp_attr_mask, NULL); + qp->qplib_qp.wqe_cnt = 0; + } +} + static int bnxt_re_post_send_shadow_qp(struct bnxt_re_dev *rdev, struct bnxt_re_qp *qp, struct ib_send_wr *wr) @@ -2093,6 +2109,7 @@ static int bnxt_re_post_send_shadow_qp(struct bnxt_re_dev *rdev, wr = wr->next; } bnxt_qplib_post_send_db(&qp->qplib_qp); + bnxt_ud_qp_hw_stall_workaround(qp); spin_unlock_irqrestore(&qp->sq_lock, flags); return rc; } @@ -2189,6 +2206,7 @@ int bnxt_re_post_send(struct ib_qp *ib_qp, struct ib_send_wr *wr, wr = wr->next; } bnxt_qplib_post_send_db(&qp->qplib_qp); + bnxt_ud_qp_hw_stall_workaround(qp); spin_unlock_irqrestore(&qp->sq_lock, flags); return rc; diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.c b/drivers/infiniband/hw/bnxt_re/qplib_fp.c index 66abec0..0716298 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.c @@ -1298,6 +1298,9 @@ int bnxt_qplib_post_send(struct bnxt_qplib_qp *qp, } sq->hwq.prod++; + + qp->wqe_cnt++; + done: return rc; } diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.h b/drivers/infiniband/hw/bnxt_re/qplib_fp.h index 71539ea..36b7b7d 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.h @@ -250,6 +250,7 @@ struct bnxt_qplib_qp { u8 timeout; u8 retry_cnt; u8 rnr_retry; + u64 wqe_cnt; u32 min_rnr_timer; u32 max_rd_atomic; u32 max_dest_rd_atomic;