From patchwork Sun Oct 14 18:55:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 10640789 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3230A13AD for ; Sun, 14 Oct 2018 18:55:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DE69928B9F for ; Sun, 14 Oct 2018 18:55:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D2B75291F2; Sun, 14 Oct 2018 18:55:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from pdx1-mailman02.dreamhost.com (pdx1-mailman02.dreamhost.com [64.90.62.194]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 822E928B9F for ; Sun, 14 Oct 2018 18:55:58 +0000 (UTC) Received: from pdx1-mailman02.dreamhost.com (localhost [IPv6:::1]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id C7B4C21FA74; Sun, 14 Oct 2018 11:55:46 -0700 (PDT) X-Original-To: lustre-devel@lists.lustre.org Delivered-To: lustre-devel-lustre.org@pdx1-mailman02.dreamhost.com Received: from smtp3.ccs.ornl.gov (smtp3.ccs.ornl.gov [160.91.203.39]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id CA02621F42E for ; Sun, 14 Oct 2018 11:55:38 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp3.ccs.ornl.gov (Postfix) with ESMTP id DA69E223C; Sun, 14 Oct 2018 14:55:34 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id D89532A5; Sun, 14 Oct 2018 14:55:34 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Sun, 14 Oct 2018 14:55:30 -0400 Message-Id: <1539543332-28679-9-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1539543332-28679-1-git-send-email-jsimmons@infradead.org> References: <1539543332-28679-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 08/10] lustre: lnd: correct WR fast reg accounting X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alexey Lyashkov , Amir Shehata , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Amir Shehata Ensure that enough WRs are allocated for the fast reg case which needs two additional WRs per transfer: the first for memory window registration and the second for memory window invalidation. Failure to allocate these causes the following problem: mlx5_warn:mlx5_0:begin_wqe:4085(pid 9590): work queue overflow Signed-off-by: Amir Shehata Signed-off-by: Alexey Lyashkov WC-bug-id: https://jira.whamcloud.com/browse/LU-9943 Reviewed-on: https://review.whamcloud.com/30311 Reviewed-by: Alexey Lyashkov Reviewed-by: Dmitry Eremin Reviewed-by: Doug Oucharek Reviewed-by: James Simmons Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c index 43266d8..66aa45f 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c @@ -656,8 +656,13 @@ static unsigned int kiblnd_send_wrs(struct kib_conn *conn) * One WR for the LNet message * And ibc_max_frags for the transfer WRs */ + u32 dev_caps = conn->ibc_hdev->ibh_dev->ibd_dev_caps; unsigned int ret = 1 + conn->ibc_max_frags; + /* FastReg needs two extra WRs for map and invalidate */ + if (dev_caps & IBLND_DEV_CAPS_FASTREG_ENABLED) + ret += 2; + /* account for a maximum of ibc_queue_depth in-flight transfers */ ret *= conn->ibc_queue_depth; return ret;