From patchwork Thu Jun 30 19:04:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12902080 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB708CCA473 for ; Thu, 30 Jun 2022 19:05:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236741AbiF3TFM (ORCPT ); Thu, 30 Jun 2022 15:05:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42900 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236812AbiF3TFJ (ORCPT ); Thu, 30 Jun 2022 15:05:09 -0400 Received: from mail-oa1-x29.google.com (mail-oa1-x29.google.com [IPv6:2001:4860:4864:20::29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B73EF37034 for ; Thu, 30 Jun 2022 12:05:08 -0700 (PDT) Received: by mail-oa1-x29.google.com with SMTP id 586e51a60fabf-101b4f9e825so505319fac.5 for ; Thu, 30 Jun 2022 12:05:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=z1aodIcPkHUv4RoA42hxgdLDPNogfGmfi8vqfC5gmTM=; b=cMP0tGo4ocax7yvPbl8X9J1N4o2MDQyRurLnPS5NE++saHeFvO0vyVpNhh78Ca0vaU xKsS72TG25EEjDzB3O9p62eKTsgzS4MPUZgmjbUMhITcatMSPmemjXurHPvmn1cjjf1R xxT3rp6a5zjuGsn/CsoSBTZjCgdwoIP9lmT6oA/hNSRBJLI24h0A/O91QyCp5SEY1QEL zU/ys+k0ui6t693gvV6Kf8Gj6cPQWzvGLbOT7AtDV6wY/25w0HlLqROcwj4+yVUFM7u5 cgc/bx+A2Nq7WDQkKX8qA4WuTBJiwpU7TK3W+tdq2xFX3nH96l+EaGegqc+kAWpbcGr9 aaUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=z1aodIcPkHUv4RoA42hxgdLDPNogfGmfi8vqfC5gmTM=; b=3h1vZlkjY+L2pd33nXwqpOGzaG0+3lLvyAXS5xygF0CaZU2iYdzQZ4T/bo26nVGPED 7y/20Xn+/5XthWbJBuTKvTG/aW1Pa+wK5e5ftuf+srgaMPXd9yxAuK9w18bUyiFLct0e vI/hYnw0ffbJs++o5yIygNe8IGlfGfqg6Muj99GFTfgpG3YAs39jgo3jGnvNJ1PNmSDE Rje62cj3iUrx4f1zQiXq7fgDk8RDnW+mYGlOmvaTkTUi3ZqmN6w73KOE0x9Jq4N1aF9S ZrNqiNikYwnef1hii34toEX3pBjiqiBPeTb3/Oy4lymwC8hV3yPNf5CtUQYUxJ6isjDS 4DSw== X-Gm-Message-State: AJIora9lymaNYRRz2+sLzZJy0bW4pwVCQ9RrVm9QUcnf0bfY0G9BSCy5 CWT8oG0sX8P0GmGi1rRGqY7tG0Jz88+/Yw== X-Google-Smtp-Source: AGRyM1tAu6tuqTgaNepZRPATEpKnplieihid/lv4UDbsOQ07kzeDy3j5kbnDDWFsFTckCbNas7Tunw== X-Received: by 2002:a05:6870:5819:b0:101:f651:99a2 with SMTP id r25-20020a056870581900b00101f65199a2mr6100840oap.167.1656615908048; Thu, 30 Jun 2022 12:05:08 -0700 (PDT) Received: from u-22.tx.rr.com (097-099-248-255.res.spectrum.com. [97.99.248.255]) by smtp.googlemail.com with ESMTPSA id o4-20020a9d4104000000b0060bfb4e4033sm11756442ote.9.2022.06.30.12.05.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jun 2022 12:05:07 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v2 1/9] RDMA/rxe: Add rxe_is_fenced() subroutine Date: Thu, 30 Jun 2022 14:04:18 -0500 Message-Id: <20220630190425.2251-2-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220630190425.2251-1-rpearsonhpe@gmail.com> References: <20220630190425.2251-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The code thc that decides whether to defer execution of a wqe in rxe_requester.c is isolated into a subroutine rxe_is_fenced() and removed from the call to req_next_wqe(). The condition whether a wqe should be fenced is changed to comply with the IBA. Currently an operation is fenced if the fence bit is set in the wqe flags and the last wqe has not completed. For normal operations the IBA actually only requires that the last read or atomic operation is complete. Signed-off-by: Bob Pearson Reviewed-by: Li Zhijian --- v2 replaces "RDMA/rxe: Fix incorrect fencing" drivers/infiniband/sw/rxe/rxe_req.c | 37 ++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 9d98237389cf..e8a1664a40eb 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -161,16 +161,36 @@ static struct rxe_send_wqe *req_next_wqe(struct rxe_qp *qp) (wqe->state != wqe_state_processing))) return NULL; - if (unlikely((wqe->wr.send_flags & IB_SEND_FENCE) && - (index != cons))) { - qp->req.wait_fence = 1; - return NULL; - } - wqe->mask = wr_opcode_mask(wqe->wr.opcode, qp); return wqe; } +/** + * rxe_wqe_is_fenced - check if next wqe is fenced + * @qp: the queue pair + * @wqe: the next wqe + * + * Returns: 1 if wqe needs to wait + * 0 if wqe is ready to go + */ +static int rxe_wqe_is_fenced(struct rxe_qp *qp, struct rxe_send_wqe *wqe) +{ + /* Local invalidate fence (LIF) see IBA 10.6.5.1 + * Requires ALL previous operations on the send queue + * are complete. Make mandatory for the rxe driver. + */ + if (wqe->wr.opcode == IB_WR_LOCAL_INV) + return qp->req.wqe_index != queue_get_consumer(qp->sq.queue, + QUEUE_TYPE_FROM_CLIENT); + + /* Fence see IBA 10.8.3.3 + * Requires that all previous read and atomic operations + * are complete. + */ + return (wqe->wr.send_flags & IB_SEND_FENCE) && + atomic_read(&qp->req.rd_atomic) != qp->attr.max_rd_atomic; +} + static int next_opcode_rc(struct rxe_qp *qp, u32 opcode, int fits) { switch (opcode) { @@ -632,6 +652,11 @@ int rxe_requester(void *arg) if (unlikely(!wqe)) goto exit; + if (rxe_wqe_is_fenced(qp, wqe)) { + qp->req.wait_fence = 1; + goto exit; + } + if (wqe->mask & WR_LOCAL_OP_MASK) { ret = rxe_do_local_ops(qp, wqe); if (unlikely(ret)) From patchwork Thu Jun 30 19:04:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12902081 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 571A8CCA47B for ; Thu, 30 Jun 2022 19:05:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236812AbiF3TFM (ORCPT ); Thu, 30 Jun 2022 15:05:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42936 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236817AbiF3TFL (ORCPT ); Thu, 30 Jun 2022 15:05:11 -0400 Received: from mail-oa1-x34.google.com (mail-oa1-x34.google.com [IPv6:2001:4860:4864:20::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B8023AA79 for ; Thu, 30 Jun 2022 12:05:10 -0700 (PDT) Received: by mail-oa1-x34.google.com with SMTP id 586e51a60fabf-10bab338f70so492922fac.7 for ; Thu, 30 Jun 2022 12:05:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DyjLX6ic4CHAm5vSjTY4RzmAKn5FQWP0TWCHfZ2cDvE=; b=pZRzuL+uqrxtoSqTFjyswvJnYHzQihNtfuUuzT8lTreRoyfT0ofAMg0Hz75nEbgefU 22QVri6ph0d5xl5vuoyMqGiTlHLmsIoix9TOnGXAfx748FUH3YZDK72cXl7ZAapVyYVY paVaDzrLT/r929TAHDbIFXR6HVLvXghLLozoNawT/TIcmbB/bAKGsmiLJ7zuPcUtVrDQ p9wl50ddJbT+4f8L2YeFUk+WRd0J5K+GP6yotYiAbYbRLxHPBzT+Gu9gZYyJy3ZdFEgr 56Y0U7Jta3YvKKQRxqziOw9NMBSUryxFXRbXKgHSKvHETjOa6r7TEjmfadVH8F6IeQ63 sYjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DyjLX6ic4CHAm5vSjTY4RzmAKn5FQWP0TWCHfZ2cDvE=; b=bIFk1gQSo43ctn4xQA/rs4Ru4UTgdV7/2EeK5S45Sp9XIX00+vZcxRkc9lXg4isIyj JYQN5gUl2YtTlHG+SiOLGsre1hm1Hhv8KXrP+OVfy4UJljy62a7ZdatGPMhxiCX6Kyz/ F6rbmFCspjB1BKu9Wc1TBs7AFIhqlwKtlKagc0pRCPS4DFZYdgOWK+LXkhE5yC3x8UgH grLzOPu4sU0g+EDVtPOs4OZuoB2tcx4ahLBPVSg5PH2zAq1ygONGwqvAoCos14k5SaTT Ys+a0ebz/lq/DtcII4+RkAYi7u2I5iLG0ejux5mz0VKjOUfef7f4rc4du8jAEZ09VZnk RCpg== X-Gm-Message-State: AJIora/xOCXh30zbfifa48Kd3K4iDjD2z6h5C/R1h/n2aegKQBSUfqXr 4UdsIUyhG06kfVtF+nYspss= X-Google-Smtp-Source: AGRyM1smD0Z/WzwqAvg1Kf6obfh/lGaOQt/txZDOmvkR3LMqjBb+fkUelhrxiP5LPOODXaFPN9qGYw== X-Received: by 2002:a05:6871:4705:b0:108:7537:d1ad with SMTP id py5-20020a056871470500b001087537d1admr6280910oab.5.1656615909183; Thu, 30 Jun 2022 12:05:09 -0700 (PDT) Received: from u-22.tx.rr.com (097-099-248-255.res.spectrum.com. [97.99.248.255]) by smtp.googlemail.com with ESMTPSA id o4-20020a9d4104000000b0060bfb4e4033sm11756442ote.9.2022.06.30.12.05.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jun 2022 12:05:08 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v2 2/9] RDMA/rxe: Convert pr_warn/err to pr_debug in pyverbs Date: Thu, 30 Jun 2022 14:04:19 -0500 Message-Id: <20220630190425.2251-3-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220630190425.2251-1-rpearsonhpe@gmail.com> References: <20220630190425.2251-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The pyverbs test suite generates a few dmesg traces from intentional error tests. This patch replaces those messages with pr_debug() calls which improves the usefullness of the tests. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_cq.c | 8 ++++---- drivers/infiniband/sw/rxe/rxe_resp.c | 8 +++++--- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_cq.c b/drivers/infiniband/sw/rxe/rxe_cq.c index 642b52539ac3..b1a0ab3cd4bd 100644 --- a/drivers/infiniband/sw/rxe/rxe_cq.c +++ b/drivers/infiniband/sw/rxe/rxe_cq.c @@ -19,16 +19,16 @@ int rxe_cq_chk_attr(struct rxe_dev *rxe, struct rxe_cq *cq, } if (cqe > rxe->attr.max_cqe) { - pr_warn("cqe(%d) > max_cqe(%d)\n", - cqe, rxe->attr.max_cqe); + pr_debug("cqe(%d) > max_cqe(%d)\n", + cqe, rxe->attr.max_cqe); goto err1; } if (cq) { count = queue_count(cq->queue, QUEUE_TYPE_TO_CLIENT); if (cqe < count) { - pr_warn("cqe(%d) < current # elements in queue (%d)", - cqe, count); + pr_debug("cqe(%d) < current # elements in queue (%d)", + cqe, count); goto err1; } } diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index c45c9d954931..7aea5f06d061 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -448,7 +448,8 @@ static enum resp_states check_rkey(struct rxe_qp *qp, if (rkey_is_mw(rkey)) { mw = rxe_lookup_mw(qp, access, rkey); if (!mw) { - pr_err("%s: no MW matches rkey %#x\n", __func__, rkey); + pr_debug("%s: no MW matches rkey %#x\n", + __func__, rkey); state = RESPST_ERR_RKEY_VIOLATION; goto err; } @@ -468,7 +469,8 @@ static enum resp_states check_rkey(struct rxe_qp *qp, } else { mr = lookup_mr(qp->pd, access, rkey, RXE_LOOKUP_REMOTE); if (!mr) { - pr_err("%s: no MR matches rkey %#x\n", __func__, rkey); + pr_debug("%s: no MR matches rkey %#x\n", + __func__, rkey); state = RESPST_ERR_RKEY_VIOLATION; goto err; } @@ -1434,7 +1436,7 @@ int rxe_responder(void *arg) case RESPST_ERROR: qp->resp.goto_error = 0; - pr_warn("qp#%d moved to error state\n", qp_num(qp)); + pr_debug("qp#%d moved to error state\n", qp_num(qp)); rxe_qp_error(qp); goto exit; From patchwork Thu Jun 30 19:04:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12902083 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9AB82CCA47F for ; Thu, 30 Jun 2022 19:05:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236813AbiF3TFM (ORCPT ); Thu, 30 Jun 2022 15:05:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42948 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236355AbiF3TFL (ORCPT ); Thu, 30 Jun 2022 15:05:11 -0400 Received: from mail-oa1-x36.google.com (mail-oa1-x36.google.com [IPv6:2001:4860:4864:20::36]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EAA4337A32 for ; Thu, 30 Jun 2022 12:05:10 -0700 (PDT) Received: by mail-oa1-x36.google.com with SMTP id 586e51a60fabf-101d2e81bceso585824fac.0 for ; Thu, 30 Jun 2022 12:05:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=z/SBzaqpErJoT/wcV9MTdujz+diRyCzAJur4KDGnY3A=; b=oob0klT/ryd4u3YiApFMmOM+r/5WBrGXchJROWcyGnN8F7V5cuVLEqiL7nT4BttzLl O9s202sP3xtCK/alpewWKFHJLTGbcR/uml02Tw6tfan/LFi3ShSyaERWyiuDIFc3+bOc bcsGSvYjTkWjhtI+Pj93TMGTJKfQ0M+7Rzl7l5mY1h0o8mXluMR+WrFuQ0MT53sjsKAZ 4c8pPOkyWgNhXQvGpk96HfQGVHsnx/vNYEaX5G03aI1il0fO0G9jQcVla4DesHHY6k6B yMnK7PTmJHIHuHHA8RFMg4w/q0qq7Nf3Ug/P3tqU/tJUHzCNcZPaPLzFASwJLNvbIFWm n+SQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=z/SBzaqpErJoT/wcV9MTdujz+diRyCzAJur4KDGnY3A=; b=dKNRUp51PtPt1GuypCzYdwKcYoW1oE3uuyILcALPtgs0ClASPI1wBa3vp+A+JV+acg PeAGX/jFqjKrkgkjibFvNzzGOrQCq7diIOf71FlUvGJJrrlVKohXybsVLPDZTWbYDU0c hZESomeWcHFeqF82VIjdkNCdaH2XqVOX3RmoWdDzzL/qvAl0aShTAPJa31rAJtxeWVKm N8Z+0DjW9iD06ts4rGeCOzVt7ae/plGi754uPP6Qlv02s0s7ghKcVzhxAnbIX5Uc3rcX OXTA57WLVMKDriPAHQAT5nDPk4ya4ZR+gzuA+51Zmmir8Dh5mDxLHh/GWtqLvfhMGbYZ rNdg== X-Gm-Message-State: AJIora92Vku/E9t1vvbMPV62pkBF7xulzTUTbRv/wXFVT6U9Wh81M4OF yP/o39Wp5Ys+S4silH9l5tk= X-Google-Smtp-Source: AGRyM1urucKaa1wWBFTOHaSmmu41hkfNE6OcuVMjFchdfgR59b2B9fcN9atxxzfGDGlGwFi8LHOoTA== X-Received: by 2002:a05:6870:65ab:b0:101:acac:7ba7 with SMTP id fp43-20020a05687065ab00b00101acac7ba7mr6429482oab.29.1656615910298; Thu, 30 Jun 2022 12:05:10 -0700 (PDT) Received: from u-22.tx.rr.com (097-099-248-255.res.spectrum.com. [97.99.248.255]) by smtp.googlemail.com with ESMTPSA id o4-20020a9d4104000000b0060bfb4e4033sm11756442ote.9.2022.06.30.12.05.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jun 2022 12:05:09 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson , Frank Zago Subject: [PATCH for-next v2 3/9] RDMA/rxe: Remove unnecessary include statement Date: Thu, 30 Jun 2022 14:04:20 -0500 Message-Id: <20220630190425.2251-4-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220630190425.2251-1-rpearsonhpe@gmail.com> References: <20220630190425.2251-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org rxe_verbs.h includes the file . It should have been , however, it is not used and not required in this file. This patch removes the include statement. Reported-by: Frank Zago Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_verbs.h | 1 - 1 file changed, 1 deletion(-) diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index ac464e68c923..ec9a70aecc1e 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -9,7 +9,6 @@ #include #include -#include #include "rxe_pool.h" #include "rxe_task.h" #include "rxe_hw_counters.h" From patchwork Thu Jun 30 19:04:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12902082 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C9D5C433EF for ; Thu, 30 Jun 2022 19:05:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235540AbiF3TFN (ORCPT ); Thu, 30 Jun 2022 15:05:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42976 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236818AbiF3TFM (ORCPT ); Thu, 30 Jun 2022 15:05:12 -0400 Received: from mail-oa1-x32.google.com (mail-oa1-x32.google.com [IPv6:2001:4860:4864:20::32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5B7737A32 for ; Thu, 30 Jun 2022 12:05:11 -0700 (PDT) Received: by mail-oa1-x32.google.com with SMTP id 586e51a60fabf-101ab23ff3fso536157fac.1 for ; Thu, 30 Jun 2022 12:05:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ln8t+yJCM/V15DKVa+5JWZsTKpmaN58ItH631HbFTxU=; b=FLoIotsT4S8bjvgtudqUF4rM05QSKqDaGkJfObLvbjsjqjbwevK1bAyGqjYi7HRSSw 7ax16CMx0TiknF5BUK3EiMsaXugu7/Us5fiNXC3S9vN9jFA5a8aDFkVkGo9mT1qhqy+3 COvUaXPd9JqtslwDtEGHmkhnIGNatsHU41dcFsloMtbiGCAwP8M3gEsqnsphPqH1e9b8 H+pfldgIfLMLD/M5vt49VZ5/2RwoUTYfA5hByJxEHnqBYOtViTzqqLeHYxQUWgT5VQ4x 4j7TP7EsoqD+wrHQVhnhw17lJd/OHyE7o47KF4yvuwN1ZjPS3k50UrRh4t+YXVwsNrzP 2AVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ln8t+yJCM/V15DKVa+5JWZsTKpmaN58ItH631HbFTxU=; b=dud82qTMlQq7YrYal8cLyhHJaOuHNmVCaEVGs5QDYqthp9Pabmcy2zrauOxuGKu3Yb qKVqHYS1anaonbPMzTAU3f/JFiPR9J+Btf3IsCy5pboVKkAd7/FCk71WnecwMlEgiTCA 2pvmjYqgxQ3MiWOVvelLymae96A53edQTx6wTE3IAKqICCMTBTMXKfwFM6h/1Wm8U6P2 dBlkOxCtCEX3y2gDYaJeF5Dr+aWDyGY0AxIfof0/bZp+/hTcspy8FjqxbvDx+a/fVaEP xX9C3Ix01ff5YyaIphUg2NGdtFiePQGO2uMwbxXYyvpJs+oovnD5LEZw4O+UMs8qWu7Z kANA== X-Gm-Message-State: AJIora/FjhZPPdysy0J9pzgY+XkawaEy6a6O/SOgUGdVKaowvExA90EK aevLJIse1LCJH7BcL1snA1M= X-Google-Smtp-Source: AGRyM1tMckgj/TShfko7ncxCXDNKN2RAPy0/9TSdG3lsQ9/HJzuX3TWxQLFnN8hUsZoEuun4v3ibcg== X-Received: by 2002:a05:6870:51cb:b0:fb:5c97:bd1b with SMTP id b11-20020a05687051cb00b000fb5c97bd1bmr6178120oaj.104.1656615911493; Thu, 30 Jun 2022 12:05:11 -0700 (PDT) Received: from u-22.tx.rr.com (097-099-248-255.res.spectrum.com. [97.99.248.255]) by smtp.googlemail.com with ESMTPSA id o4-20020a9d4104000000b0060bfb4e4033sm11756442ote.9.2022.06.30.12.05.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jun 2022 12:05:10 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson , Frank Zago Subject: [PATCH for-next v2 4/9] RDMA/rxe: Replace include statement Date: Thu, 30 Jun 2022 14:04:21 -0500 Message-Id: <20220630190425.2251-5-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220630190425.2251-1-rpearsonhpe@gmail.com> References: <20220630190425.2251-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org rxe_queue.h currently includes for a definition of struct rxe_queue_buf. But it is only used as a pointer so the definition is not needed. This patch replaces the include statement with the declaration struct rxe_queue_buf; Reported-by: Frank Zago Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_queue.h | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_queue.h b/drivers/infiniband/sw/rxe/rxe_queue.h index 6227112ef7a2..ed44042782fa 100644 --- a/drivers/infiniband/sw/rxe/rxe_queue.h +++ b/drivers/infiniband/sw/rxe/rxe_queue.h @@ -7,9 +7,6 @@ #ifndef RXE_QUEUE_H #define RXE_QUEUE_H -/* for definition of shared struct rxe_queue_buf */ -#include - /* Implements a simple circular buffer that is shared between user * and the driver and can be resized. The requested element size is * rounded up to a power of 2 and the number of elements in the buffer @@ -53,6 +50,8 @@ enum queue_type { QUEUE_TYPE_FROM_DRIVER, }; +struct rxe_queue_buf; + struct rxe_queue { struct rxe_dev *rxe; struct rxe_queue_buf *buf; From patchwork Thu Jun 30 19:04:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12902084 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57B85C43334 for ; Thu, 30 Jun 2022 19:05:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236816AbiF3TFO (ORCPT ); Thu, 30 Jun 2022 15:05:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236355AbiF3TFN (ORCPT ); Thu, 30 Jun 2022 15:05:13 -0400 Received: from mail-oa1-x36.google.com (mail-oa1-x36.google.com [IPv6:2001:4860:4864:20::36]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1245937A06 for ; Thu, 30 Jun 2022 12:05:13 -0700 (PDT) Received: by mail-oa1-x36.google.com with SMTP id 586e51a60fabf-101d2e81bceso585999fac.0 for ; Thu, 30 Jun 2022 12:05:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Xyz7156MFriJeXhwB/oxcEWhAksS88t3ZJa/MRrEWnc=; b=W+wBKdJurIF0GyLI+gOhLaUaY+Qv5PT6oas9Ua/f4kdIbvKdB78bV9SNzgZOUZjHGg QDojitB9uZ2VpqbmnCo3n9oYtGNmiqIQa/qHVLO+ZnZTpk57MMQ1ebIs+7MHiuawnsfX +Hey53jed8N7rAOjUZG/QJzEkxkIdis9ROPKqUv+APeQMBdTpCHyExlXYuHLAIs3QbdS GwYclnNUdEZokWHV9U4RFP/Foz2F8Q1PdKV8UMlkFinnnskb3iJQhv7bw/GMJWn59F1g +DVXbfZljLCizdBewSz3SxEPc/v+gCq1OIDavVep7NmSoh3eCZNdaExiqycnIc00EpfK Ny2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Xyz7156MFriJeXhwB/oxcEWhAksS88t3ZJa/MRrEWnc=; b=0cR9rOEWRQalyNfCF3mMwMkXTJkaXsu7wlpOhNsLVMoB1vK/G40gSggPEuvVUR+sSN lgBayBfXhpxe0GJN7A+PHOlnfu0o4G8rkPyO+KDjLdAFhDf/9fMMSmYQWOVeJqWVPdew GYuOZfZ2fK4Fozn04rzpQqFwm36D5lbiPd3QrWqzTWDm2uPjczHInuYN4Civ6/mMEtzp 4ZfTwn/lHTqdwrLtrnQNllONvHw0njrUkvDEQ7ZwiopxyzE+pV03Fxy+OkX5VXDe7avg t9XlBINEY4WZ23qZLQmMPD2vO6PFDHZYksKcHDlu5ZLBu2YreggTW+MckI91rUhldRYW Viww== X-Gm-Message-State: AJIora/IhS4MEas0lv+krIRGxL+v1IG9q6iNEYqhP2IVC02J/2q1rokG 3tOF61cELYbP7x7ZgYfPFzY= X-Google-Smtp-Source: AGRyM1sVGkcwrxDjF6czfXHNKzNXnFYibUdsAocBybm3hCtdoIQZFnmM+RQezlPzdHNIPYX5yQoHJA== X-Received: by 2002:a05:6870:331f:b0:10b:af85:e15 with SMTP id x31-20020a056870331f00b0010baf850e15mr1074864oae.124.1656615912698; Thu, 30 Jun 2022 12:05:12 -0700 (PDT) Received: from u-22.tx.rr.com (097-099-248-255.res.spectrum.com. [97.99.248.255]) by smtp.googlemail.com with ESMTPSA id o4-20020a9d4104000000b0060bfb4e4033sm11756442ote.9.2022.06.30.12.05.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jun 2022 12:05:12 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v2 5/9] RDMA/rxe: Fix rnr retry behavior Date: Thu, 30 Jun 2022 14:04:22 -0500 Message-Id: <20220630190425.2251-6-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220630190425.2251-1-rpearsonhpe@gmail.com> References: <20220630190425.2251-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Currently the completer tasklet when retransmit timer or the rnr timer fires the same flag (qp->req.need_retry) is set so that if either timer fires it will attempt to perform a retry flow on the send queue. This has the effect of responding to an RNR NAK at the first retransmit timer event which might not allow the requested rnr timeout. This patch adds a new flag (qp->req.wait_for_rnr_timer) which, if set, prevents a retry flow until the rnr nak timer fires. This patch fixes rnr retry errors which can be observed by running the pyverbs test_rdmacm_async_traffic_external_qp multiple times. With this patch applied they do not occur. Link: https://lore.kernel.org/linux-rdma/a8287823-1408-4273-bc22-99a0678db640@gmail.com/ Link: https://lore.kernel.org/linux-rdma/2bafda9e-2bb6-186d-12a1-179e8f6a2678@talpey.com/ Fixes: 8700e3e7c485 ("Soft RoCE (RXE) - The software RoCE driver") Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_comp.c | 8 +++++++- drivers/infiniband/sw/rxe/rxe_qp.c | 1 + drivers/infiniband/sw/rxe/rxe_req.c | 15 +++++++++++++-- drivers/infiniband/sw/rxe/rxe_verbs.h | 1 + 4 files changed, 22 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c index da3a398053b8..4fc31bb7eee6 100644 --- a/drivers/infiniband/sw/rxe/rxe_comp.c +++ b/drivers/infiniband/sw/rxe/rxe_comp.c @@ -114,6 +114,8 @@ void retransmit_timer(struct timer_list *t) { struct rxe_qp *qp = from_timer(qp, t, retrans_timer); + pr_debug("%s: fired for qp#%d\n", __func__, qp->elem.index); + if (qp->valid) { qp->comp.timeout = 1; rxe_run_task(&qp->comp.task, 1); @@ -730,11 +732,15 @@ int rxe_completer(void *arg) break; case COMPST_RNR_RETRY: + /* we come here if we received an RNR NAK */ if (qp->comp.rnr_retry > 0) { if (qp->comp.rnr_retry != 7) qp->comp.rnr_retry--; - qp->req.need_retry = 1; + /* don't start a retry flow until the + * rnr timer has fired + */ + qp->req.wait_for_rnr_timer = 1; pr_debug("qp#%d set rnr nak timer\n", qp_num(qp)); mod_timer(&qp->rnr_nak_timer, diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index 22e9b85344c3..edc0c34feb05 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -507,6 +507,7 @@ static void rxe_qp_reset(struct rxe_qp *qp) atomic_set(&qp->ssn, 0); qp->req.opcode = -1; qp->req.need_retry = 0; + qp->req.wait_for_rnr_timer = 0; qp->req.noack_pkts = 0; qp->resp.msn = 0; qp->resp.opcode = -1; diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index e8a1664a40eb..4d92f929d269 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -101,7 +101,11 @@ void rnr_nak_timer(struct timer_list *t) { struct rxe_qp *qp = from_timer(qp, t, rnr_nak_timer); - pr_debug("qp#%d rnr nak timer fired\n", qp_num(qp)); + pr_debug("%s: fired for qp#%d\n", __func__, qp_num(qp)); + + /* request a send queue retry */ + qp->req.need_retry = 1; + qp->req.wait_for_rnr_timer = 0; rxe_run_task(&qp->req.task, 1); } @@ -640,10 +644,17 @@ int rxe_requester(void *arg) qp->req.need_rd_atomic = 0; qp->req.wait_psn = 0; qp->req.need_retry = 0; + qp->req.wait_for_rnr_timer = 0; goto exit; } - if (unlikely(qp->req.need_retry)) { + /* we come here if the retransmot timer has fired + * or if the rnr timer has fired. If the retransmit + * timer fires while we are processing an RNR NAK wait + * until the rnr timer has fired before starting the + * retry flow + */ + if (unlikely(qp->req.need_retry && !qp->req.wait_for_rnr_timer)) { req_retry(qp); qp->req.need_retry = 0; } diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index ec9a70aecc1e..d9f01af4050d 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -123,6 +123,7 @@ struct rxe_req_info { int need_rd_atomic; int wait_psn; int need_retry; + int wait_for_rnr_timer; int noack_pkts; struct rxe_task task; }; From patchwork Thu Jun 30 19:04:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12902085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85940C433EF for ; Thu, 30 Jun 2022 19:05:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236355AbiF3TFQ (ORCPT ); Thu, 30 Jun 2022 15:05:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43020 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236818AbiF3TFP (ORCPT ); Thu, 30 Jun 2022 15:05:15 -0400 Received: from mail-oa1-x32.google.com (mail-oa1-x32.google.com [IPv6:2001:4860:4864:20::32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B97F237034 for ; Thu, 30 Jun 2022 12:05:14 -0700 (PDT) Received: by mail-oa1-x32.google.com with SMTP id 586e51a60fabf-f2a4c51c45so479840fac.9 for ; Thu, 30 Jun 2022 12:05:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Bv7fZnS0RrFaJh9bc0HOC2R7xKFbMSNqXsVuNFHrE64=; b=GZVQ2umwV/OdhejQewnw1s3+U2mNlbyxN8wwk0Bma8FDRWTf3lQSPoHVpn35850mKh fp8Ve9epZwLendTgh0NsOwfH/0S20hNOJGVy/GeFnVx/YpcXta9RRgs7USdksYOYycny ZL3RfkLp+0RCrbaO11AZsO+8ga32zKB1y6lBZbteHzM0cpGby/70+ID2OW0xxOwGRav8 n7kCefH4rxeLUEsDq13o/13h5qUb9bnAtLKZrN7Kde4t+/VSM297aGl5PpFXqm24IOOC ndqwoTmcXxtA5V6BzMGkghTPzsfYPgPclVzn2REBLHTi4Za779NG15XJHwz8QVCw3BMU OTwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Bv7fZnS0RrFaJh9bc0HOC2R7xKFbMSNqXsVuNFHrE64=; b=ijjXNsIPB4RqXbg2L9tHCwZP6BCsV1KnneL5/y3N5v16/blR4Kn9ihOrAWY0UBQkHX pPsmLDtwjgJyHX1tOwTgiqCtgzSh7MhMOc3370OT3+9dkrlC3P9aTMrzSxTaki5ygq/m hMMU2nIsPc9IrYaAHNeuZinA4SYSl7x1agD7q2/0dZXXgxnUdF7kZ8nGVDTVj+ok2yf+ 17fAzWrP4HO8rMA5e53D28rXxqsbJdfwQpk18i+fizcPsj9l8lo2AaIctC9SI+UpVAEX 10qlyIpCNwq4izZZ89XUrgf/PFpKHzUhtjpAu9BBtdNPps72qX62xmsZBpC//dp1rnKd M9uA== X-Gm-Message-State: AJIora9bdkTSZgdHJ87vfrIG+rjmkOlUMpJWXAikHn1CPk/xoNr7Vwv/ fzJcxQy2X384IuYolPI7k0o= X-Google-Smtp-Source: AGRyM1v8xB/PSf1VGFSepX0s7ljyLRserDD607fVVrV7Vv5KtnITbR5JI2tKSSt+emsxm9X0+jtVPw== X-Received: by 2002:a05:6870:8914:b0:106:9d06:fe63 with SMTP id i20-20020a056870891400b001069d06fe63mr6839475oao.103.1656615914081; Thu, 30 Jun 2022 12:05:14 -0700 (PDT) Received: from u-22.tx.rr.com (097-099-248-255.res.spectrum.com. [97.99.248.255]) by smtp.googlemail.com with ESMTPSA id o4-20020a9d4104000000b0060bfb4e4033sm11756442ote.9.2022.06.30.12.05.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jun 2022 12:05:13 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson , Jenny Hack Subject: [PATCH for-next v2 6/9] RDMA/rxe: Fix deadlock in rxe_do_local_ops() Date: Thu, 30 Jun 2022 14:04:23 -0500 Message-Id: <20220630190425.2251-7-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220630190425.2251-1-rpearsonhpe@gmail.com> References: <20220630190425.2251-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org When a local operation (invalidate mr, reg mr, bind mw) is finished there will be no ack packet coming from a responder to cause the wqe to be completed. This may happen anyway if a subsequent wqe performs IO. Currently if the wqe is signalled the completer tasklet is scheduled immediately but not otherwise. This leads to a deadlock if the next wqe has the fence bit set in send flags and the operation is not signalled. This patch removes the condition that the wqe must be signalled in order to schedule the completer tasklet which is the simplest fix for this deadlock and is fairly low cost. This is the analog for local operations of always setting the ackreq bit in all last or only request packets even if the operation is not signalled. Reported-by: Jenny Hack Fixes: c1a411268a4b1 ("RDMA/rxe: Move local ops to subroutine") Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_req.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 4d92f929d269..81eca57b04b8 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -605,9 +605,11 @@ static int rxe_do_local_ops(struct rxe_qp *qp, struct rxe_send_wqe *wqe) wqe->status = IB_WC_SUCCESS; qp->req.wqe_index = queue_next_index(qp->sq.queue, qp->req.wqe_index); - if ((wqe->wr.send_flags & IB_SEND_SIGNALED) || - qp->sq_sig_type == IB_SIGNAL_ALL_WR) - rxe_run_task(&qp->comp.task, 1); + /* There is no ack coming for local work requests + * which can lead to a deadlock. So go ahead and complete + * it now. + */ + rxe_run_task(&qp->comp.task, 1); return 0; } From patchwork Thu Jun 30 19:04:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12902086 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41D00C43334 for ; Thu, 30 Jun 2022 19:05:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236819AbiF3TFR (ORCPT ); Thu, 30 Jun 2022 15:05:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43034 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236817AbiF3TFQ (ORCPT ); Thu, 30 Jun 2022 15:05:16 -0400 Received: from mail-oi1-x235.google.com (mail-oi1-x235.google.com [IPv6:2607:f8b0:4864:20::235]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E8EA937A06 for ; Thu, 30 Jun 2022 12:05:15 -0700 (PDT) Received: by mail-oi1-x235.google.com with SMTP id w83so538367oiw.1 for ; Thu, 30 Jun 2022 12:05:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LWdD9BfJ+fRfDfw5ycyY7chG7VG1cND2Lz8zAHRmU/s=; b=XdHapiG5BgngkGIhPDwBRZnO6ITy/Ukz57FfleX2ed+Q7G6nK4dEkR56IhJQXwDfNd CGiRC2Ea+JOxYXQ5PnKWvfB9E5XpZiH+/RtOSYZR97Dq6ePB3YmSliTNpfl83hJqkVIu 1yePi6AJoejhOlJETjotn3YY5kXTqpwrRaOkvceyHHQ37gGJ0Y93qnmh0tZ66zeqKDfK COXSI1QMgek98yYkzqfqdovUbwukvp1cWBevs25uYUTlUiLRlz8EktKUV47/dpQtt0As OgLrhi66VNF4Qa+09GhSa4EUUDgTOHQFeEUqwBglV1vC9MIXBC+TP9LwcNsoN/5pBcEj +HTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LWdD9BfJ+fRfDfw5ycyY7chG7VG1cND2Lz8zAHRmU/s=; b=mYSr2jzjdq+Zs6hwSzk9UADUY2dpnN2kA8RSF9ScnQ2fE0V2+kCw+PglHuLjhSPunA 8EdzfcOiMXIr1T7XLVbULf3YJ59/mSU4k6DtGNQ6IBCJS3GrEeuD2YVivZiZPHW67v7D 5y0g1+8Nx66sHaduAR3yS+X1lFA3TKzG11x5qLyIBrk8kr/YSQ0yNLcKesrCaxxjYjI+ TAiKzci2k3wO5gG2CrmWsGZFofLhI3QkmPEUqdDowSolDmhmRttVQjlwP7Mwke00yDYw lcZrX+c/w63nPLK7Kg2tLiiuuCEvGY4efazkBN4zR9ovJZemHs7dJW/sF8Qps8DQDuOT qWUg== X-Gm-Message-State: AJIora/OgWlnRpmJklig+QNmSH1j+VP/2QSr4delJrRGuTXbr6w+XBP5 e2LCHcBW4qt50BIrJcRUhd7P9ZU1N0alUw== X-Google-Smtp-Source: AGRyM1tM1hxt7PvSdh+PmFcSmy3tIxZnJDW700lwb/q9zMzpxzL/TtzFhVA3m9cHHKCYPyzA40s5cg== X-Received: by 2002:a05:6808:191b:b0:335:82bb:afd4 with SMTP id bf27-20020a056808191b00b0033582bbafd4mr6430795oib.5.1656615915178; Thu, 30 Jun 2022 12:05:15 -0700 (PDT) Received: from u-22.tx.rr.com (097-099-248-255.res.spectrum.com. [97.99.248.255]) by smtp.googlemail.com with ESMTPSA id o4-20020a9d4104000000b0060bfb4e4033sm11756442ote.9.2022.06.30.12.05.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jun 2022 12:05:14 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v2 7/9] RDMA/rxe: Make the tasklet exits the same Date: Thu, 30 Jun 2022 14:04:24 -0500 Message-Id: <20220630190425.2251-8-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220630190425.2251-1-rpearsonhpe@gmail.com> References: <20220630190425.2251-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Make changes to the three tasklets so that the exit logic from each is the same. This makes the code easier to understand. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_comp.c | 35 +++++++++--------- drivers/infiniband/sw/rxe/rxe_req.c | 54 ++++++++++++++++------------ drivers/infiniband/sw/rxe/rxe_resp.c | 17 +++++---- 3 files changed, 60 insertions(+), 46 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c index 4fc31bb7eee6..bc53cad077aa 100644 --- a/drivers/infiniband/sw/rxe/rxe_comp.c +++ b/drivers/infiniband/sw/rxe/rxe_comp.c @@ -562,7 +562,7 @@ int rxe_completer(void *arg) struct sk_buff *skb = NULL; struct rxe_pkt_info *pkt = NULL; enum comp_state state; - int ret = 0; + int ret; if (!rxe_get(qp)) return -EAGAIN; @@ -571,8 +571,7 @@ int rxe_completer(void *arg) qp->req.state == QP_STATE_RESET) { rxe_drain_resp_pkts(qp, qp->valid && qp->req.state == QP_STATE_ERROR); - ret = -EAGAIN; - goto done; + goto exit; } if (qp->comp.timeout) { @@ -582,10 +581,8 @@ int rxe_completer(void *arg) qp->comp.timeout_retry = 0; } - if (qp->req.need_retry) { - ret = -EAGAIN; - goto done; - } + if (qp->req.need_retry) + goto exit; state = COMPST_GET_ACK; @@ -678,8 +675,7 @@ int rxe_completer(void *arg) qp->qp_timeout_jiffies) mod_timer(&qp->retrans_timer, jiffies + qp->qp_timeout_jiffies); - ret = -EAGAIN; - goto done; + goto exit; case COMPST_ERROR_RETRY: /* we come here if the retry timer fired and we did @@ -691,10 +687,8 @@ int rxe_completer(void *arg) */ /* there is nothing to retry in this case */ - if (!wqe || (wqe->state == wqe_state_posted)) { - ret = -EAGAIN; - goto done; - } + if (!wqe || (wqe->state == wqe_state_posted)) + goto exit; /* if we've started a retry, don't start another * retry sequence, unless this is a timeout. @@ -746,8 +740,7 @@ int rxe_completer(void *arg) mod_timer(&qp->rnr_nak_timer, jiffies + rnrnak_jiffies(aeth_syn(pkt) & ~AETH_TYPE_MASK)); - ret = -EAGAIN; - goto done; + goto exit; } else { rxe_counter_inc(rxe, RXE_CNT_RNR_RETRY_EXCEEDED); @@ -760,12 +753,20 @@ int rxe_completer(void *arg) WARN_ON_ONCE(wqe->status == IB_WC_SUCCESS); do_complete(qp, wqe); rxe_qp_error(qp); - ret = -EAGAIN; - goto done; + goto exit; } } + /* A non-zero return value will cause rxe_do_task to + * exit its loop and end the tasklet. A zero return + * will continue looping and return to rxe_completer + */ done: + ret = 0; + goto out; +exit: + ret = -EAGAIN; +out: if (pkt) free_pkt(pkt); rxe_put(qp); diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 81eca57b04b8..008ae51c7560 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -625,6 +625,7 @@ int rxe_requester(void *arg) u32 payload; int mtu; int opcode; + int err; int ret; struct rxe_send_wqe rollback_wqe; u32 rollback_psn; @@ -635,7 +636,6 @@ int rxe_requester(void *arg) if (!rxe_get(qp)) return -EAGAIN; -next_wqe: if (unlikely(!qp->valid || qp->req.state == QP_STATE_ERROR)) goto exit; @@ -650,7 +650,7 @@ int rxe_requester(void *arg) goto exit; } - /* we come here if the retransmot timer has fired + /* we come here if the retransmit timer has fired * or if the rnr timer has fired. If the retransmit * timer fires while we are processing an RNR NAK wait * until the rnr timer has fired before starting the @@ -671,11 +671,11 @@ int rxe_requester(void *arg) } if (wqe->mask & WR_LOCAL_OP_MASK) { - ret = rxe_do_local_ops(qp, wqe); - if (unlikely(ret)) + err = rxe_do_local_ops(qp, wqe); + if (unlikely(err)) goto err; else - goto next_wqe; + goto done; } if (unlikely(qp_type(qp) == IB_QPT_RC && @@ -724,8 +724,7 @@ int rxe_requester(void *arg) wqe->state = wqe_state_done; wqe->status = IB_WC_SUCCESS; __rxe_do_task(&qp->comp.task); - rxe_put(qp); - return 0; + goto done; } payload = mtu; } @@ -741,25 +740,29 @@ int rxe_requester(void *arg) if (unlikely(!av)) { pr_err("qp#%d Failed no address vector\n", qp_num(qp)); wqe->status = IB_WC_LOC_QP_OP_ERR; - goto err_drop_ah; + goto err; } skb = init_req_packet(qp, av, wqe, opcode, payload, &pkt); if (unlikely(!skb)) { pr_err("qp#%d Failed allocating skb\n", qp_num(qp)); wqe->status = IB_WC_LOC_QP_OP_ERR; - goto err_drop_ah; + if (ah) + rxe_put(ah); + goto err; } - ret = finish_packet(qp, av, wqe, &pkt, skb, payload); - if (unlikely(ret)) { + err = finish_packet(qp, av, wqe, &pkt, skb, payload); + if (unlikely(err)) { pr_debug("qp#%d Error during finish packet\n", qp_num(qp)); - if (ret == -EFAULT) + if (err == -EFAULT) wqe->status = IB_WC_LOC_PROT_ERR; else wqe->status = IB_WC_LOC_QP_OP_ERR; kfree_skb(skb); - goto err_drop_ah; + if (ah) + rxe_put(ah); + goto err; } if (ah) @@ -774,13 +777,14 @@ int rxe_requester(void *arg) save_state(wqe, qp, &rollback_wqe, &rollback_psn); update_wqe_state(qp, wqe, &pkt); update_wqe_psn(qp, wqe, &pkt, payload); - ret = rxe_xmit_packet(qp, &pkt, skb); - if (ret) { + + err = rxe_xmit_packet(qp, &pkt, skb); + if (err) { qp->need_req_skb = 1; rollback_state(wqe, qp, &rollback_wqe, rollback_psn); - if (ret == -EAGAIN) { + if (err == -EAGAIN) { rxe_run_task(&qp->req.task, 1); goto exit; } @@ -791,16 +795,20 @@ int rxe_requester(void *arg) update_state(qp, &pkt); - goto next_wqe; - -err_drop_ah: - if (ah) - rxe_put(ah); + /* A non-zero return value will cause rxe_do_task to + * exit its loop and end the tasklet. A zero return + * will continue looping and return to rxe_requester + */ +done: + ret = 0; + goto out; err: wqe->state = wqe_state_error; __rxe_do_task(&qp->comp.task); - exit: + ret = -EAGAIN; +out: rxe_put(qp); - return -EAGAIN; + + return ret; } diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index 7aea5f06d061..a45b1d22bc77 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -1258,17 +1258,15 @@ int rxe_responder(void *arg) struct rxe_dev *rxe = to_rdev(qp->ibqp.device); enum resp_states state; struct rxe_pkt_info *pkt = NULL; - int ret = 0; + int ret; if (!rxe_get(qp)) return -EAGAIN; qp->resp.aeth_syndrome = AETH_ACK_UNLIMITED; - if (!qp->valid) { - ret = -EINVAL; - goto done; - } + if (!qp->valid) + goto exit; switch (qp->resp.state) { case QP_STATE_RESET: @@ -1445,9 +1443,16 @@ int rxe_responder(void *arg) } } + /* A non-zero return value will cause rxe_do_task to + * exit its loop and end the tasklet. A zero return + * will continue looping and return to rxe_responder + */ +done: + ret = 0; + goto out; exit: ret = -EAGAIN; -done: +out: rxe_put(qp); return ret; } From patchwork Thu Jun 30 19:04:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12902087 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57809CCA473 for ; Thu, 30 Jun 2022 19:05:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236820AbiF3TFS (ORCPT ); Thu, 30 Jun 2022 15:05:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236817AbiF3TFR (ORCPT ); Thu, 30 Jun 2022 15:05:17 -0400 Received: from mail-oi1-x235.google.com (mail-oi1-x235.google.com [IPv6:2607:f8b0:4864:20::235]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DCD9E37A29 for ; Thu, 30 Jun 2022 12:05:16 -0700 (PDT) Received: by mail-oi1-x235.google.com with SMTP id r82so531248oig.2 for ; Thu, 30 Jun 2022 12:05:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9D+sLMq8vkOoIkXhT2rT04WVWXiLtRahkQVBbSnUaUM=; b=ISyfCIYb8RhOkVb/p/raX/ebmNFv5EicWqI+6fZcxiW10SD3ZEpy76p5Q4pa2Dvq8j rEhxyw6GSwcJA2za4iXLufvlnPihpWBktrASp2JD5TKx18qjbuFJvDfGxUoYqfXitmCc hBlqRyf52MIkxiQ2YGqBF9znCyZv7UMIZ1o8/pXqMENd/wy6ke9XlRJXnDbBz6Kt5tSz OFjU1VYWMz3fIoHgoWHtKx9tI0ttzyhdYgaQk5RFKOHA06/mMvVX4cMnkVN5+k6tUNdE zmc5OIi6B/rHnoJPBMo28O0tC/zcz6nPBXOHSRizkBCCoKWuMrRMo9fByZnmA0aeweyY uPpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9D+sLMq8vkOoIkXhT2rT04WVWXiLtRahkQVBbSnUaUM=; b=Bid5m1olG9TxFnsKtOisqkIULksKh9Ls/W2HKQfoI/qcrBJ/64vMtH+fMLi+Nl3Jj3 Xsg42ni8xIAXqNPcOAYWHfeKsSeIrN3ejVpvJrR/R4qvd+dvoCYqCldyQOlkokm9iMTX lVGwziMltzKtUyUWLHH+sHV3iVT3agbdklq57/flsT7osoq59R6TtSObtvLfqf0xiUpJ a6dftF6cOsNWg0zJXuEuEi0f1jySU8dYqJmOaP8Intp2BcXzdyDh0cCn47c8/d4Tj5vK bIeKMGyIvUxQUfZYWbeI1MMx84fiCxXHQ0fzDJ/b9G4YDKWgwt8IHrUaKUpVCNzjaTMR KNlA== X-Gm-Message-State: AJIora/45OeT3Gbz2MBTHos60yv1OREvMHPUwl4QS/L6TBgrwE9IEWkw eWL8dDKVSX4+fJBaSJLTB90= X-Google-Smtp-Source: AGRyM1vbTKERmjY8jOCAXvzb2OUiVRru/QJvXS0OWgc0e3DuL6VR2toMFqOt/LfQIDoKIJ/WTYgPew== X-Received: by 2002:a05:6808:2395:b0:335:93c2:5804 with SMTP id bp21-20020a056808239500b0033593c25804mr7829689oib.2.1656615916200; Thu, 30 Jun 2022 12:05:16 -0700 (PDT) Received: from u-22.tx.rr.com (097-099-248-255.res.spectrum.com. [97.99.248.255]) by smtp.googlemail.com with ESMTPSA id o4-20020a9d4104000000b0060bfb4e4033sm11756442ote.9.2022.06.30.12.05.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jun 2022 12:05:15 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v2 8/9] RDMA/rxe: Limit the number of calls to each tasklet Date: Thu, 30 Jun 2022 14:04:25 -0500 Message-Id: <20220630190425.2251-9-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220630190425.2251-1-rpearsonhpe@gmail.com> References: <20220630190425.2251-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Limit the maximum number of calls to each tasklet from rxe_do_task() before yielding the cpu. When the limit is reached reschedule the tasklet and exit the calling loop. This patch prevents one tasklet from consuming 100% of a cpu core and causing a deadlock or soft lockup. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_param.h | 6 ++++++ drivers/infiniband/sw/rxe/rxe_task.c | 16 ++++++++++++---- 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_param.h b/drivers/infiniband/sw/rxe/rxe_param.h index 568a7cbd13d4..9f3b036e6bde 100644 --- a/drivers/infiniband/sw/rxe/rxe_param.h +++ b/drivers/infiniband/sw/rxe/rxe_param.h @@ -105,6 +105,12 @@ enum rxe_device_param { RXE_INFLIGHT_SKBS_PER_QP_HIGH = 64, RXE_INFLIGHT_SKBS_PER_QP_LOW = 16, + /* Max number of interations of each tasklet + * before yielding the cpu to let other + * work make progress + */ + RXE_MAX_ITERATIONS = 1024, + /* Delay before calling arbiter timer */ RXE_NSEC_ARB_TIMER_DELAY = 200, diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index 0c4db5bb17d7..2248cf33d776 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -8,7 +8,7 @@ #include #include -#include "rxe_task.h" +#include "rxe.h" int __rxe_do_task(struct rxe_task *task) @@ -33,6 +33,7 @@ void rxe_do_task(struct tasklet_struct *t) int cont; int ret; struct rxe_task *task = from_tasklet(task, t, tasklet); + unsigned int iterations = RXE_MAX_ITERATIONS; spin_lock_bh(&task->state_lock); switch (task->state) { @@ -61,13 +62,20 @@ void rxe_do_task(struct tasklet_struct *t) spin_lock_bh(&task->state_lock); switch (task->state) { case TASK_STATE_BUSY: - if (ret) + if (ret) { task->state = TASK_STATE_START; - else + } else if (iterations--) { cont = 1; + } else { + /* reschedule the tasklet and exit + * the loop to give up the cpu + */ + tasklet_schedule(&task->tasklet); + task->state = TASK_STATE_START; + } break; - /* soneone tried to run the task since the last time we called + /* someone tried to run the task since the last time we called * func, so we will call one more time regardless of the * return value */ From patchwork Thu Jun 30 19:04:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12902088 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72895C43334 for ; Thu, 30 Jun 2022 19:05:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236817AbiF3TFU (ORCPT ); Thu, 30 Jun 2022 15:05:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43086 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236818AbiF3TFT (ORCPT ); Thu, 30 Jun 2022 15:05:19 -0400 Received: from mail-oa1-x29.google.com (mail-oa1-x29.google.com [IPv6:2001:4860:4864:20::29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F2EB137A29 for ; Thu, 30 Jun 2022 12:05:17 -0700 (PDT) Received: by mail-oa1-x29.google.com with SMTP id 586e51a60fabf-101b4f9e825so505890fac.5 for ; Thu, 30 Jun 2022 12:05:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fPzDkaS+L/Ev2tFB61N27jlt6NfPFid4f2EAXcBDn2E=; b=ELTTZuLpnz7JdNf0hEUSLe5fVyi95b5IcjaVZs+dDNcMKByGbpqK3tTUWNU1xl2FWY v/ZfgRIKC2HsAivbOhQo59VCNQcF0H7UDJbSv3iHZKvZYLfO9RkA/HAjjqDtnIEvQyQK tmH3sRD47tM8Fi1lrx/2ST4VHBFT2nmUXonaEsloL0mPJG9zb2i405hFZnNYPhtCSkTr R3WovGQ78qRs+EbTLjYlfjr7AxGf0J5nHrxPjNJMiCC6NbrFY1ViX8VqNWtoGtreILcD fcaWLPbyAu43ZG8qOzH7oGkV3nEGBhzFaMAhkh3X7oTAM0gyYqCRM+F86QSGf5f7w5gc Pg3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fPzDkaS+L/Ev2tFB61N27jlt6NfPFid4f2EAXcBDn2E=; b=luXdW6NlGgO/6rtg9FKAYC3Eil0OocBTfThVPVbnsc6d+fYeGQkeCvStT0gW9J6lCW ZjUxe/QY0wkfWV/2Cln0kjtzTNQekgeOr0aFY1g+DQ8GjaolkiU6lWTrKu0N1QjbUK60 vgfLQwQjL0SZW2g7Tq20iDvgO6CeqBX1AoPwT0+9hiEMzAvp+mwYRZfESGnoruG4P/Uy ScBX8e3KfH57gz8JRMGxFBdGuFtpcsd53N8cKSLLwTQHdXZii+n1LjqM3D7QvHVeDv7u nuu7Ey2hD5jJgCJZf+C57YoIlNIZASJHfmVA11dQDA2D13XOENhHF5Vtu1BNYPf+p4V7 FaVA== X-Gm-Message-State: AJIora9wJj/ia6byd0NZ11SV+W7Jg8MilGyydgA/NkR5lE6+UMjOJdxa 6iid0uks6Wi49Z81U+IClwVflPnNPrjCPQ== X-Google-Smtp-Source: AGRyM1v1tY7hcIt0rtxvjKNYHZiEms6w69ar2wJs4HfQj8gDGEauFE+n5UEZsQQPMhDESupGCU0jCQ== X-Received: by 2002:a05:6871:4705:b0:108:7537:d1ad with SMTP id py5-20020a056871470500b001087537d1admr6281319oab.5.1656615917672; Thu, 30 Jun 2022 12:05:17 -0700 (PDT) Received: from u-22.tx.rr.com (097-099-248-255.res.spectrum.com. [97.99.248.255]) by smtp.googlemail.com with ESMTPSA id o4-20020a9d4104000000b0060bfb4e4033sm11756442ote.9.2022.06.30.12.05.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jun 2022 12:05:16 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v2 9/9] RDMA/rxe: Replace __rxe_do_task by rxe_run_task Date: Thu, 30 Jun 2022 14:04:26 -0500 Message-Id: <20220630190425.2251-10-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220630190425.2251-1-rpearsonhpe@gmail.com> References: <20220630190425.2251-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org In rxe_req.c replace calls to __rxe_do_task() by calls to rxe_run_task(.., 0). Using __rxe_do_task is an error because the completer tasklet is not designed to be re-entrant and __rxe_do_task() should only be called when it is clear that no one else could be calling the completer tasklet as is the case in rxe_qp.c where this call is used in safe environments. Signed-off-by: Bob Pearson Reviewed-by: Li Zhijian --- drivers/infiniband/sw/rxe/rxe_req.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 008ae51c7560..58b9f170b18b 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -723,7 +723,7 @@ int rxe_requester(void *arg) qp->req.wqe_index); wqe->state = wqe_state_done; wqe->status = IB_WC_SUCCESS; - __rxe_do_task(&qp->comp.task); + rxe_run_task(&qp->comp.task, 0); goto done; } payload = mtu; @@ -804,7 +804,7 @@ int rxe_requester(void *arg) goto out; err: wqe->state = wqe_state_error; - __rxe_do_task(&qp->comp.task); + rxe_run_task(&qp->comp.task, 0); exit: ret = -EAGAIN; out: