From patchwork Tue Oct 18 04:33:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13009874 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B58DC4332F for ; Tue, 18 Oct 2022 04:36:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229456AbiJREgR (ORCPT ); Tue, 18 Oct 2022 00:36:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50988 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229687AbiJREgQ (ORCPT ); Tue, 18 Oct 2022 00:36:16 -0400 Received: from mail-oi1-x232.google.com (mail-oi1-x232.google.com [IPv6:2607:f8b0:4864:20::232]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7CCBFA0243 for ; Mon, 17 Oct 2022 21:36:15 -0700 (PDT) Received: by mail-oi1-x232.google.com with SMTP id g10so14395522oif.10 for ; Mon, 17 Oct 2022 21:36:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Qm5mGTa3bhYcwi5JlLG1oAfRXa/BUdi8iIjw7y2ov7A=; b=pgd8TvXcV6XaoJ6v+EjLZHNh0zb1gtKebySZJ5GU5w4qXHR4c4+CpEHe/7fmmacwEJ E9OBdCPE6taJHjMRdDfRGrSt1i1vcGd7wBVaEHvV5qG8FpV5SGdzlNL6+64rFl3+HaQL RzrnYRHbrqlATJOLzpTwDmU6/PGL1sVWq6gmem2j1onAqIQOBjobRrrp4DjEZcX02stp fAZhjcbtm6ASpo0Z4hCo3EqzCL+6LOTUxo3sTyFVP4vxcyxbcVhjnpL9DEK6IDT9UeUE ugFwY9fuX1EGvTJdxk0GRdT0e68gfdGhj29UGKobcNlh+Dl40CRbRyPNv0GcBWO/ORry HUYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Qm5mGTa3bhYcwi5JlLG1oAfRXa/BUdi8iIjw7y2ov7A=; b=He7fYjUwmJc0WaVbSzBTrpv4RZg23OfHiKjqeIkB3k/fUhEFZdarvFP5StuuPJh+Vd VbpyOj495XHXm4qoMnqpx0q2xo4zWEasF1sGLtL2nT9lKITGdSOOSg35l/r5CBF45SIH 1mY8aEWgG3qMJhOCy1cTa4RWeD0jWhTcFR3ywjqnVFSH5D1okrmyeT4aZGX/tNuLjh4b ajfGNJcUbb0kkKg+vNWeer4xolO+6+IfCCLsMn4YKWAhyhtP/Cz9vBzSm725SV6f16Uh z5dUX0Rgj90q/URAXheau3FPt6Nfrpj0zKappgF22EinlT5niUf9EDV5gophXuTcW7Aq i1Yg== X-Gm-Message-State: ACrzQf1bbUzyKyajlRkh+r6UjGCwFTsseKN3PLBZNeFuxYL4DToJcEgF UPX5bqWmksAGa4HSTksRq74= X-Google-Smtp-Source: AMsMyM7qOsf+Y/ZkO41OO7iE052m73EKGDFe8Z2jvWBMfcissJwFaSt4dk1rx6KgyYYGvsOCJ+91HQ== X-Received: by 2002:a05:6808:46:b0:354:943f:bcc5 with SMTP id v6-20020a056808004600b00354943fbcc5mr572024oic.104.1666067774820; Mon, 17 Oct 2022 21:36:14 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-290b-8972-ce76-602c.res6.spectrum.com. [2603:8081:140c:1a00:290b:8972:ce76:602c]) by smtp.googlemail.com with ESMTPSA id e96-20020a9d01e9000000b006618ca5caa0sm5480333ote.78.2022.10.17.21.36.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Oct 2022 21:36:14 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, matsuda-daisuke@fujitsu.com, lizhijian@fujitsu.com, leon@kernel.org, linux-rdma@vger.kernel.org, jenny.hack@hpe.com, ian.ziemba@hpe.com Cc: Bob Pearson Subject: [PATCH for-next 01/16] RDMA/rxe: Remove init of task locks from rxe_qp.c Date: Mon, 17 Oct 2022 23:33:32 -0500 Message-Id: <20221018043345.4033-2-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221018043345.4033-1-rpearsonhpe@gmail.com> References: <20221018043345.4033-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The calls to spin_lock_init() for the tasklet spinlocks in rxe_qp_init_misc() are redundant since they are intiialized in rxe_init_task(). This patch removes them. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_qp.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index a62bab88415c..57c3f05ad15b 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -172,10 +172,6 @@ static void rxe_qp_init_misc(struct rxe_dev *rxe, struct rxe_qp *qp, spin_lock_init(&qp->state_lock); - spin_lock_init(&qp->req.task.state_lock); - spin_lock_init(&qp->resp.task.state_lock); - spin_lock_init(&qp->comp.task.state_lock); - spin_lock_init(&qp->sq.sq_lock); spin_lock_init(&qp->rq.producer_lock); spin_lock_init(&qp->rq.consumer_lock); From patchwork Tue Oct 18 04:33:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13009875 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62438C433FE for ; Tue, 18 Oct 2022 04:36:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229687AbiJREgS (ORCPT ); Tue, 18 Oct 2022 00:36:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50994 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229648AbiJREgR (ORCPT ); Tue, 18 Oct 2022 00:36:17 -0400 Received: from mail-oa1-x2b.google.com (mail-oa1-x2b.google.com [IPv6:2001:4860:4864:20::2b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C3E76A0241 for ; Mon, 17 Oct 2022 21:36:16 -0700 (PDT) Received: by mail-oa1-x2b.google.com with SMTP id 586e51a60fabf-136b5dd6655so15617010fac.3 for ; Mon, 17 Oct 2022 21:36:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6QtHbx7j5MLEIzMsh4ikjKbKP5OllTIf9TzcOjs/vnc=; b=SQYozpljiYG6dY8y6PwLB/BLOuWmwHvRLZq9KK5GWxqwXY+0bYmqn5t9UgGB4MlFq7 9SlvlMttq7wKWFXz45iqayjizG71hDJlYt10LZWLPRFBrBjssHLGYGStBAefSyuiVHhW qh/K6/w4r5bL+aT7CnTx71Su023vsmDrcTb5T78XaNUOjyihf6Cjqn/hEpQ0IbjFta9A M+0VYOhgGib4+yXOMCvER1QuGX++OgAlbnlidYk8dvUDYWIH97v1UlqDPsOzf8wyrhsW cmwChBjXrn8IBGpD83hmTELwnqN9XfUn6sRLsj7Aa9sjftr2BEyIOzSJrdEBg2VOl7w4 uCvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6QtHbx7j5MLEIzMsh4ikjKbKP5OllTIf9TzcOjs/vnc=; b=57aJ91iW2lSN344lVGrXk57G9OfxxpqZBAQir9D70cv4inPFjfML9ALzGO8g/WT5IP vt6PNGecpizYQ1fqmcpfLUe60DzANKLNphd5/HURsEp6onknqCyt3ytzQoZWuk4riro6 xNB0MOCt7MqCa93e7kbxPmTDaPwc6id5cSnmLD/Y5VrHo3KSCb3MWIImo9RMCuEWf807 rtm0eJojOBoJtiIggqXvxxT303lXRRPd4bxsIX3Z9G2Rscd9bHXAVTyxvbCeTwNnIDhp W59gkjQPu3a9hbPkxkFdZea4iSukE9n2mrQead+BHWUX85hJv7ArRwSOBzmSd0V6x1D0 iXhA== X-Gm-Message-State: ACrzQf3I/NdiiljUFpYHUk+b9NnL2Wy0b4YkPjT/Lqc+7jdikKTW6rOc 3IOo7cQT3UCHfbJX+rIu7vE= X-Google-Smtp-Source: AMsMyM47iAB44hchj3MpvreNUmAnztMoQCgHa4anjpqiLFAD9Trdg1Xfig+n1n4hg543QbMM+FCBDw== X-Received: by 2002:a05:6870:b150:b0:131:c7a9:5d15 with SMTP id a16-20020a056870b15000b00131c7a95d15mr17180011oal.183.1666067776135; Mon, 17 Oct 2022 21:36:16 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-290b-8972-ce76-602c.res6.spectrum.com. [2603:8081:140c:1a00:290b:8972:ce76:602c]) by smtp.googlemail.com with ESMTPSA id e96-20020a9d01e9000000b006618ca5caa0sm5480333ote.78.2022.10.17.21.36.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Oct 2022 21:36:15 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, matsuda-daisuke@fujitsu.com, lizhijian@fujitsu.com, leon@kernel.org, linux-rdma@vger.kernel.org, jenny.hack@hpe.com, ian.ziemba@hpe.com Cc: Bob Pearson Subject: [PATCH for-next 02/16] RDMA/rxe: Removed unused name from rxe_task struct Date: Mon, 17 Oct 2022 23:33:33 -0500 Message-Id: <20221018043345.4033-3-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221018043345.4033-1-rpearsonhpe@gmail.com> References: <20221018043345.4033-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The name field in struct rxe_task is never used. This patch removes it. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_qp.c | 9 +++------ drivers/infiniband/sw/rxe/rxe_task.c | 4 +--- drivers/infiniband/sw/rxe/rxe_task.h | 4 +--- 3 files changed, 5 insertions(+), 12 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index 57c3f05ad15b..03bd9f3e9956 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -238,10 +238,8 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp, skb_queue_head_init(&qp->req_pkts); - rxe_init_task(&qp->req.task, qp, - rxe_requester, "req"); - rxe_init_task(&qp->comp.task, qp, - rxe_completer, "comp"); + rxe_init_task(&qp->req.task, qp, rxe_requester); + rxe_init_task(&qp->comp.task, qp, rxe_completer); qp->qp_timeout_jiffies = 0; /* Can't be set for UD/UC in modify_qp */ if (init->qp_type == IB_QPT_RC) { @@ -288,8 +286,7 @@ static int rxe_qp_init_resp(struct rxe_dev *rxe, struct rxe_qp *qp, skb_queue_head_init(&qp->resp_pkts); - rxe_init_task(&qp->resp.task, qp, - rxe_responder, "resp"); + rxe_init_task(&qp->resp.task, qp, rxe_responder); qp->resp.opcode = OPCODE_NONE; qp->resp.msn = 0; diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index ec2b7de1c497..182d0532a8ab 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -94,12 +94,10 @@ void rxe_do_task(struct tasklet_struct *t) task->ret = ret; } -int rxe_init_task(struct rxe_task *task, - void *arg, int (*func)(void *), char *name) +int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *)) { task->arg = arg; task->func = func; - snprintf(task->name, sizeof(task->name), "%s", name); task->destroyed = false; tasklet_setup(&task->tasklet, rxe_do_task); diff --git a/drivers/infiniband/sw/rxe/rxe_task.h b/drivers/infiniband/sw/rxe/rxe_task.h index 7f612a1c68a7..b3dfd970d1dc 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.h +++ b/drivers/infiniband/sw/rxe/rxe_task.h @@ -25,7 +25,6 @@ struct rxe_task { void *arg; int (*func)(void *arg); int ret; - char name[16]; bool destroyed; }; @@ -34,8 +33,7 @@ struct rxe_task { * arg => parameter to pass to fcn * func => function to call until it returns != 0 */ -int rxe_init_task(struct rxe_task *task, - void *arg, int (*func)(void *), char *name); +int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *)); /* cleanup task */ void rxe_cleanup_task(struct rxe_task *task); From patchwork Tue Oct 18 04:33:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13009876 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 887B9C433FE for ; Tue, 18 Oct 2022 04:36:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229919AbiJREgX (ORCPT ); Tue, 18 Oct 2022 00:36:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229648AbiJREgU (ORCPT ); Tue, 18 Oct 2022 00:36:20 -0400 Received: from mail-ot1-x32a.google.com (mail-ot1-x32a.google.com [IPv6:2607:f8b0:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D63DA0244 for ; Mon, 17 Oct 2022 21:36:18 -0700 (PDT) Received: by mail-ot1-x32a.google.com with SMTP id a16-20020a056830101000b006619dba7fd4so6981175otp.12 for ; Mon, 17 Oct 2022 21:36:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=k536cdm23n4LIka7KeODZwz+qCdnz3UYNGSBkjqH9DY=; b=qvK+46MFcRdV9YREJK/MMODf+L07v1Js+7dQ9EqMxu5T02BjOY7MqQGJx9ebxCWb0P X3pXB9I7w4RvmUsTc45xNO4HYJTm1YUJr50TBOSy20WZf3oC9WqP190LkonnHofK1UoJ JRS7hI5rWgf3HuP++oj3ZF6zyE2rGwbgwM8+Uq7W/OAm0acnxdewMSGhTt0xtrI/izdF wuFyywXjJo4FJt60gMBbujJIuWzt9SjDcoZMWlTaBVfZ9Hu5lPwj/yLh7Gtsk7YjeQwM Fw9EBnLrsu/RCPlYXCJCvLV5ygD61HnnAfvS3GDOONiz5/bfwTjYebV7OqSERGWJI7W2 38Ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=k536cdm23n4LIka7KeODZwz+qCdnz3UYNGSBkjqH9DY=; b=LB8f+O7B/BYCXZJTw1SH3TQMcNxsY0wmXTgr2d5JB4fipgZjK0RbFvrhyHY1rNS4CZ CIlmXx1qgphD6Iwps4nHA6xdQ1I+4IxD9fBh+BFs31mpXeo622ubgi65JP27G/4C+YCW MR5UnYsuvbucbULRL35UHBxYWJqlVYkYLl2HuW+aRxhDsSecDYGTK/6fypyxqyXrRWeM arguVfuk7YHY5P+MWGEoUqA866z8RAhCUYYjSw1/ixPlqzc+hrZ4ZLncEzYLnnpezVfg aRCcAPY/sIGOzTdLyD2ugFYd6Pc6wHdQpIQOZSDEvUYSvHocQxw4P55ti6A+Tewg7+mO cHnw== X-Gm-Message-State: ACrzQf17eB/nqvKfJVI5Odq5nihuVI/+0eferakDEyrmxHVIzyGYCQf5 ypjGFXkeF1ajzLYs9adIykQ= X-Google-Smtp-Source: AMsMyM6vV2rhxJVKG2K+CP3LOAPTPfxSFzZXwYB3x3L6xX6Nm2MlbrKY08jpQrYzivn2CZLiMWZVfA== X-Received: by 2002:a05:6830:6002:b0:660:7be2:59b2 with SMTP id bx2-20020a056830600200b006607be259b2mr512323otb.183.1666067777337; Mon, 17 Oct 2022 21:36:17 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-290b-8972-ce76-602c.res6.spectrum.com. [2603:8081:140c:1a00:290b:8972:ce76:602c]) by smtp.googlemail.com with ESMTPSA id e96-20020a9d01e9000000b006618ca5caa0sm5480333ote.78.2022.10.17.21.36.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Oct 2022 21:36:16 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, matsuda-daisuke@fujitsu.com, lizhijian@fujitsu.com, leon@kernel.org, linux-rdma@vger.kernel.org, jenny.hack@hpe.com, ian.ziemba@hpe.com Cc: Bob Pearson Subject: [PATCH for-next 03/16] RDMA/rxe: Split rxe_run_task() into two subroutines Date: Mon, 17 Oct 2022 23:33:34 -0500 Message-Id: <20221018043345.4033-4-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221018043345.4033-1-rpearsonhpe@gmail.com> References: <20221018043345.4033-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Split rxe_run_task(task, sched) into rxe_run_task(task) and rxe_sched_task(task). Signed-off-by: Ian Ziemba Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_comp.c | 19 +++++++++++-------- drivers/infiniband/sw/rxe/rxe_net.c | 4 ++-- drivers/infiniband/sw/rxe/rxe_qp.c | 10 +++++----- drivers/infiniband/sw/rxe/rxe_req.c | 10 +++++----- drivers/infiniband/sw/rxe/rxe_resp.c | 5 ++++- drivers/infiniband/sw/rxe/rxe_task.c | 15 ++++++++++----- drivers/infiniband/sw/rxe/rxe_task.h | 7 +++---- drivers/infiniband/sw/rxe/rxe_verbs.c | 8 ++++---- 8 files changed, 44 insertions(+), 34 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c index fb0c008af78c..d2a250123617 100644 --- a/drivers/infiniband/sw/rxe/rxe_comp.c +++ b/drivers/infiniband/sw/rxe/rxe_comp.c @@ -118,7 +118,7 @@ void retransmit_timer(struct timer_list *t) if (qp->valid) { qp->comp.timeout = 1; - rxe_run_task(&qp->comp.task, 1); + rxe_sched_task(&qp->comp.task); } } @@ -132,7 +132,10 @@ void rxe_comp_queue_pkt(struct rxe_qp *qp, struct sk_buff *skb) if (must_sched != 0) rxe_counter_inc(SKB_TO_PKT(skb)->rxe, RXE_CNT_COMPLETER_SCHED); - rxe_run_task(&qp->comp.task, must_sched); + if (must_sched) + rxe_sched_task(&qp->comp.task); + else + rxe_run_task(&qp->comp.task); } static inline enum comp_state get_wqe(struct rxe_qp *qp, @@ -305,7 +308,7 @@ static inline enum comp_state check_ack(struct rxe_qp *qp, qp->comp.psn = pkt->psn; if (qp->req.wait_psn) { qp->req.wait_psn = 0; - rxe_run_task(&qp->req.task, 0); + rxe_run_task(&qp->req.task); } } return COMPST_ERROR_RETRY; @@ -452,7 +455,7 @@ static void do_complete(struct rxe_qp *qp, struct rxe_send_wqe *wqe) */ if (qp->req.wait_fence) { qp->req.wait_fence = 0; - rxe_run_task(&qp->req.task, 0); + rxe_run_task(&qp->req.task); } } @@ -466,7 +469,7 @@ static inline enum comp_state complete_ack(struct rxe_qp *qp, if (qp->req.need_rd_atomic) { qp->comp.timeout_retry = 0; qp->req.need_rd_atomic = 0; - rxe_run_task(&qp->req.task, 0); + rxe_run_task(&qp->req.task); } } @@ -512,7 +515,7 @@ static inline enum comp_state complete_wqe(struct rxe_qp *qp, if (qp->req.wait_psn) { qp->req.wait_psn = 0; - rxe_run_task(&qp->req.task, 1); + rxe_sched_task(&qp->req.task); } } @@ -646,7 +649,7 @@ int rxe_completer(void *arg) if (qp->req.wait_psn) { qp->req.wait_psn = 0; - rxe_run_task(&qp->req.task, 1); + rxe_sched_task(&qp->req.task); } state = COMPST_DONE; @@ -714,7 +717,7 @@ int rxe_completer(void *arg) RXE_CNT_COMP_RETRY); qp->req.need_retry = 1; qp->comp.started_retry = 1; - rxe_run_task(&qp->req.task, 0); + rxe_run_task(&qp->req.task); } goto done; diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c index 35f327b9d4b8..c36cad9c7a66 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.c +++ b/drivers/infiniband/sw/rxe/rxe_net.c @@ -345,7 +345,7 @@ static void rxe_skb_tx_dtor(struct sk_buff *skb) if (unlikely(qp->need_req_skb && skb_out < RXE_INFLIGHT_SKBS_PER_QP_LOW)) - rxe_run_task(&qp->req.task, 1); + rxe_sched_task(&qp->req.task); rxe_put(qp); } @@ -429,7 +429,7 @@ int rxe_xmit_packet(struct rxe_qp *qp, struct rxe_pkt_info *pkt, if ((qp_type(qp) != IB_QPT_RC) && (pkt->mask & RXE_END_MASK)) { pkt->wqe->state = wqe_state_done; - rxe_run_task(&qp->comp.task, 1); + rxe_sched_task(&qp->comp.task); } rxe_counter_inc(rxe, RXE_CNT_SENT_PKTS); diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index 03bd9f3e9956..3f6d62a80bea 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -536,10 +536,10 @@ static void rxe_qp_drain(struct rxe_qp *qp) if (qp->req.state != QP_STATE_DRAINED) { qp->req.state = QP_STATE_DRAIN; if (qp_type(qp) == IB_QPT_RC) - rxe_run_task(&qp->comp.task, 1); + rxe_sched_task(&qp->comp.task); else __rxe_do_task(&qp->comp.task); - rxe_run_task(&qp->req.task, 1); + rxe_sched_task(&qp->req.task); } } } @@ -553,13 +553,13 @@ void rxe_qp_error(struct rxe_qp *qp) qp->attr.qp_state = IB_QPS_ERR; /* drain work and packet queues */ - rxe_run_task(&qp->resp.task, 1); + rxe_sched_task(&qp->resp.task); if (qp_type(qp) == IB_QPT_RC) - rxe_run_task(&qp->comp.task, 1); + rxe_sched_task(&qp->comp.task); else __rxe_do_task(&qp->comp.task); - rxe_run_task(&qp->req.task, 1); + rxe_sched_task(&qp->req.task); } /* called by the modify qp verb */ diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index f63771207970..41f1d84f0acb 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -105,7 +105,7 @@ void rnr_nak_timer(struct timer_list *t) /* request a send queue retry */ qp->req.need_retry = 1; qp->req.wait_for_rnr_timer = 0; - rxe_run_task(&qp->req.task, 1); + rxe_sched_task(&qp->req.task); } static struct rxe_send_wqe *req_next_wqe(struct rxe_qp *qp) @@ -608,7 +608,7 @@ static int rxe_do_local_ops(struct rxe_qp *qp, struct rxe_send_wqe *wqe) * which can lead to a deadlock. So go ahead and complete * it now. */ - rxe_run_task(&qp->comp.task, 1); + rxe_sched_task(&qp->comp.task); return 0; } @@ -733,7 +733,7 @@ int rxe_requester(void *arg) qp->req.wqe_index); wqe->state = wqe_state_done; wqe->status = IB_WC_SUCCESS; - rxe_run_task(&qp->comp.task, 0); + rxe_run_task(&qp->comp.task); goto done; } payload = mtu; @@ -795,7 +795,7 @@ int rxe_requester(void *arg) rollback_state(wqe, qp, &rollback_wqe, rollback_psn); if (err == -EAGAIN) { - rxe_run_task(&qp->req.task, 1); + rxe_sched_task(&qp->req.task); goto exit; } @@ -817,7 +817,7 @@ int rxe_requester(void *arg) qp->req.wqe_index = queue_next_index(qp->sq.queue, qp->req.wqe_index); wqe->state = wqe_state_error; qp->req.state = QP_STATE_ERROR; - rxe_run_task(&qp->comp.task, 0); + rxe_run_task(&qp->comp.task); exit: ret = -EAGAIN; out: diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index ed5a09e86417..2a2a50de51b2 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -91,7 +91,10 @@ void rxe_resp_queue_pkt(struct rxe_qp *qp, struct sk_buff *skb) must_sched = (pkt->opcode == IB_OPCODE_RC_RDMA_READ_REQUEST) || (skb_queue_len(&qp->req_pkts) > 1); - rxe_run_task(&qp->resp.task, must_sched); + if (must_sched) + rxe_sched_task(&qp->resp.task); + else + rxe_run_task(&qp->resp.task); } static inline enum resp_states get_req(struct rxe_qp *qp, diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index 182d0532a8ab..446ee2c3d381 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -127,15 +127,20 @@ void rxe_cleanup_task(struct rxe_task *task) tasklet_kill(&task->tasklet); } -void rxe_run_task(struct rxe_task *task, int sched) +void rxe_run_task(struct rxe_task *task) { if (task->destroyed) return; - if (sched) - tasklet_schedule(&task->tasklet); - else - rxe_do_task(&task->tasklet); + rxe_do_task(&task->tasklet); +} + +void rxe_sched_task(struct rxe_task *task) +{ + if (task->destroyed) + return; + + tasklet_schedule(&task->tasklet); } void rxe_disable_task(struct rxe_task *task) diff --git a/drivers/infiniband/sw/rxe/rxe_task.h b/drivers/infiniband/sw/rxe/rxe_task.h index b3dfd970d1dc..590b1c1d7e7c 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.h +++ b/drivers/infiniband/sw/rxe/rxe_task.h @@ -52,10 +52,9 @@ int __rxe_do_task(struct rxe_task *task); */ void rxe_do_task(struct tasklet_struct *t); -/* run a task, else schedule it to run as a tasklet, The decision - * to run or schedule tasklet is based on the parameter sched. - */ -void rxe_run_task(struct rxe_task *task, int sched); +void rxe_run_task(struct rxe_task *task); + +void rxe_sched_task(struct rxe_task *task); /* keep a task from scheduling */ void rxe_disable_task(struct rxe_task *task); diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 88825edc7dce..f2f82efbaf6d 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -695,9 +695,9 @@ static int rxe_post_send_kernel(struct rxe_qp *qp, const struct ib_send_wr *wr, wr = next; } - rxe_run_task(&qp->req.task, 1); + rxe_sched_task(&qp->req.task); if (unlikely(qp->req.state == QP_STATE_ERROR)) - rxe_run_task(&qp->comp.task, 1); + rxe_sched_task(&qp->comp.task); return err; } @@ -719,7 +719,7 @@ static int rxe_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr, if (qp->is_user) { /* Utilize process context to do protocol processing */ - rxe_run_task(&qp->req.task, 0); + rxe_run_task(&qp->req.task); return 0; } else return rxe_post_send_kernel(qp, wr, bad_wr); @@ -759,7 +759,7 @@ static int rxe_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr, spin_unlock_irqrestore(&rq->producer_lock, flags); if (qp->resp.state == QP_STATE_ERROR) - rxe_run_task(&qp->resp.task, 1); + rxe_sched_task(&qp->resp.task); err1: return err; From patchwork Tue Oct 18 04:33:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13009877 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66755C4332F for ; Tue, 18 Oct 2022 04:36:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229648AbiJREgZ (ORCPT ); Tue, 18 Oct 2022 00:36:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51042 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229711AbiJREgU (ORCPT ); Tue, 18 Oct 2022 00:36:20 -0400 Received: from mail-oa1-x34.google.com (mail-oa1-x34.google.com [IPv6:2001:4860:4864:20::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3384AA0249 for ; Mon, 17 Oct 2022 21:36:19 -0700 (PDT) Received: by mail-oa1-x34.google.com with SMTP id 586e51a60fabf-1322d768ba7so15598768fac.5 for ; Mon, 17 Oct 2022 21:36:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=n+pQLfFTLvpppau6pAQopOyapj3Gnfc23EXBLPi5pxo=; b=VT84l+wYFbT4YBdyOxK2ciUyNVKwdGBsNQ2liETRe3LVtztMxM041SJ2smY4Xfh8NF CoVY+z97moIjFVNZh4z6iM1Sj2S/yKYB7HOavGM8QGOKlH/3G3CmbMgA4gGzNnKNYyY9 ZyaoZKaEEcAC/ddCk5Ztb1RDIBhRN8PBa/7tOMs+O2ZGg5SYxjGgKoldmZETJQ/gvmfs I7XE2MIIKeFz2JPUqy+OYWx9ObGSsaHcY6bJgVcBijEkEqKEIvYDhx6dJz6cG3kJw0LB SsdVoPw/4f1kRr/lnFiSpzS0747Vy8cO6QoKH+My4+bp7m+UK8epy/54ZiPHaApCcFlV K/mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=n+pQLfFTLvpppau6pAQopOyapj3Gnfc23EXBLPi5pxo=; b=xRxko4FpDZus2vVL/cc2aKapcMreLjvxI0m3/JBwu3W40g5dTIMzgofN84dhkWRkkB FipYmro0MW7vF4cMO0RZttgsrBNivXIOoqUdSudlnYBhZJiWHlEzr0wVKuAhpsk4Wg0O f+Mz6roCe9TrjKzYklNAPa37tp/IrrS3JE6QrPiQfEGpsgYIyfbImXm5xjyVgZ3HQ/qW Hk1s8Tip+t5tXItlXk8tIntR0a9VONzaDQ19fplp4ks/sxsSqewQBpcu5ekB3hRb/Nr5 37l7fICGitQvmpKiDbn8rmRAgJ+jo1N9fDa1xpaORCSJ39c0L5Hg0Ao7fzhF3qBBAYy+ JEPw== X-Gm-Message-State: ACrzQf2SxiQ7EL4XF/c44d2I+uaCNy2GXjum4F0KAMrdbbAyVGdg2WwQ Zr8Tpl+gMkEdYB0weypZS0Y= X-Google-Smtp-Source: AMsMyM7/oyTYASi+zIBlVgluB0ir0kbEWaqWwkeZ1V0fTlUCk6J1NlXAllyxOvd1CHTPcQ+83riYMg== X-Received: by 2002:a05:6871:28e:b0:131:91b0:6a1 with SMTP id i14-20020a056871028e00b0013191b006a1mr604118oae.200.1666067778557; Mon, 17 Oct 2022 21:36:18 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-290b-8972-ce76-602c.res6.spectrum.com. [2603:8081:140c:1a00:290b:8972:ce76:602c]) by smtp.googlemail.com with ESMTPSA id e96-20020a9d01e9000000b006618ca5caa0sm5480333ote.78.2022.10.17.21.36.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Oct 2022 21:36:18 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, matsuda-daisuke@fujitsu.com, lizhijian@fujitsu.com, leon@kernel.org, linux-rdma@vger.kernel.org, jenny.hack@hpe.com, ian.ziemba@hpe.com Cc: Bob Pearson Subject: [PATCH 04/16] for-next RDMA/rxe: Make rxe_do_task static Date: Mon, 17 Oct 2022 23:33:35 -0500 Message-Id: <20221018043345.4033-5-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221018043345.4033-1-rpearsonhpe@gmail.com> References: <20221018043345.4033-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The subroutine rxe_do_task() is only called in rxe_task.c. This patch makes it static and renames it do_task(). Signed-off-by: Ian Ziemba Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_task.c | 2 +- drivers/infiniband/sw/rxe/rxe_task.h | 8 -------- 2 files changed, 1 insertion(+), 9 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index 446ee2c3d381..097ddb16c230 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -28,7 +28,7 @@ int __rxe_do_task(struct rxe_task *task) * a second caller finds the task already running * but looks just after the last call to func */ -void rxe_do_task(struct tasklet_struct *t) +static void rxe_do_task(struct tasklet_struct *t) { int cont; int ret; diff --git a/drivers/infiniband/sw/rxe/rxe_task.h b/drivers/infiniband/sw/rxe/rxe_task.h index 590b1c1d7e7c..99e0173e5c46 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.h +++ b/drivers/infiniband/sw/rxe/rxe_task.h @@ -44,14 +44,6 @@ void rxe_cleanup_task(struct rxe_task *task); */ int __rxe_do_task(struct rxe_task *task); -/* - * common function called by any of the main tasklets - * If there is any chance that there is additional - * work to do someone must reschedule the task before - * leaving - */ -void rxe_do_task(struct tasklet_struct *t); - void rxe_run_task(struct rxe_task *task); void rxe_sched_task(struct rxe_task *task); From patchwork Tue Oct 18 04:33:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13009878 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E94CC433FE for ; Tue, 18 Oct 2022 04:36:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230152AbiJREgb (ORCPT ); Tue, 18 Oct 2022 00:36:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51106 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229725AbiJREg0 (ORCPT ); Tue, 18 Oct 2022 00:36:26 -0400 Received: from mail-oa1-x31.google.com (mail-oa1-x31.google.com [IPv6:2001:4860:4864:20::31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4E1AA025F for ; Mon, 17 Oct 2022 21:36:21 -0700 (PDT) Received: by mail-oa1-x31.google.com with SMTP id 586e51a60fabf-132b8f6f1b2so15555509fac.11 for ; Mon, 17 Oct 2022 21:36:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ts7wj/6p6IhLkdtFxIR72LxXhWu5bCOcfIH/Wo6L9d8=; b=BJ2WGWpDbLi8Rm7OCO3jzs1c8yCgRIw8ZU1pBhWa+OH+rrzXd7qvzkq/S8k9IIWfPh XOnUcIXk+mLBcYrfTzBGYGlwU23WTx5S2JhRgKVgFv5pbzBegIvr349EyfqWXwTISxTR a0EV3/COCSyDH/oYyWwnTFks1cYAGAj+7RrdTcygPoJsIzLrfJr4qdw8yHlp7JwGu5VM U0AEBo7fqweddze6+a9+ymPASb7zaJ6JpTQgMNu2jBp/tyCZ6Sc8fpe1pE0wRGgAC4t2 K6iL136Wp5HZmwF6/iA5GwXo/9DM8xWWP8+SV3a3hhhkHmtwOP2dW34TluJsh7j3Nczl KOkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ts7wj/6p6IhLkdtFxIR72LxXhWu5bCOcfIH/Wo6L9d8=; b=zJ4uHLESAfF+VfmEi0tcsF6iE+on6Z73srUyCyyPEXkKVdOczL1+Gw2zNQjUwwwHOD GC7rwfbrmYxSYl12bZ7DBR7u2cqZQVeRtQl3oDT4UjXc8+IM++BsFZ2PBcjmFdg5MdxJ cSe/6fRWDMOSr9K+zRPpnq3o3EJUAC7L0zEvxybMXY9EMoQ8Sth79ClhCJKNV7xknlA+ r7VNVdP8Rdq1L/feEEgMqPMCK7ugW6MksMIQgfXae9SHFvx9RR1s58leMdKB8+222+A9 WMz6xLZi5bB/Nvpq+IHWh7lf8VFa5S2XmWPCmzXm6U2be/KlQIoHrItea45ckBBbRMEn mDeA== X-Gm-Message-State: ACrzQf2sCovJ8S8/o4gsKY52xDz9NCuCCAIMeMde8VudVp6k5hNhglOv 4KstcqwE+vfMUWI9MDc4eoc= X-Google-Smtp-Source: AMsMyM4Jqm7Phy8dpRyJs58NYWCsHGUlZGcB5dVbUr142Z7f62AZS53+oFN/NDPqL3GiMVNpqWD/wg== X-Received: by 2002:a05:6871:588:b0:132:8901:5d4c with SMTP id u8-20020a056871058800b0013289015d4cmr17199565oan.106.1666067779663; Mon, 17 Oct 2022 21:36:19 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-290b-8972-ce76-602c.res6.spectrum.com. [2603:8081:140c:1a00:290b:8972:ce76:602c]) by smtp.googlemail.com with ESMTPSA id e96-20020a9d01e9000000b006618ca5caa0sm5480333ote.78.2022.10.17.21.36.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Oct 2022 21:36:19 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, matsuda-daisuke@fujitsu.com, lizhijian@fujitsu.com, leon@kernel.org, linux-rdma@vger.kernel.org, jenny.hack@hpe.com, ian.ziemba@hpe.com Cc: Bob Pearson Subject: [PATCH for-next 05/16] RDMA/rxe: Rename task->state_lock to task->lock Date: Mon, 17 Oct 2022 23:33:36 -0500 Message-Id: <20221018043345.4033-6-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221018043345.4033-1-rpearsonhpe@gmail.com> References: <20221018043345.4033-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Rename task-state_lock to task->lock Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_task.c | 18 +++++++++--------- drivers/infiniband/sw/rxe/rxe_task.h | 2 +- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index 097ddb16c230..42442ede99e8 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -35,22 +35,22 @@ static void do_task(struct tasklet_struct *t) struct rxe_task *task = from_tasklet(task, t, tasklet); unsigned int iterations = RXE_MAX_ITERATIONS; - spin_lock_bh(&task->state_lock); + spin_lock_bh(&task->lock); switch (task->state) { case TASK_STATE_START: task->state = TASK_STATE_BUSY; - spin_unlock_bh(&task->state_lock); + spin_unlock_bh(&task->lock); break; case TASK_STATE_BUSY: task->state = TASK_STATE_ARMED; fallthrough; case TASK_STATE_ARMED: - spin_unlock_bh(&task->state_lock); + spin_unlock_bh(&task->lock); return; default: - spin_unlock_bh(&task->state_lock); + spin_unlock_bh(&task->lock); pr_warn("%s failed with bad state %d\n", __func__, task->state); return; } @@ -59,7 +59,7 @@ static void do_task(struct tasklet_struct *t) cont = 0; ret = task->func(task->arg); - spin_lock_bh(&task->state_lock); + spin_lock_bh(&task->lock); switch (task->state) { case TASK_STATE_BUSY: if (ret) { @@ -88,7 +88,7 @@ static void do_task(struct tasklet_struct *t) pr_warn("%s failed with bad state %d\n", __func__, task->state); } - spin_unlock_bh(&task->state_lock); + spin_unlock_bh(&task->lock); } while (cont); task->ret = ret; @@ -103,7 +103,7 @@ int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *)) tasklet_setup(&task->tasklet, do_task); task->state = TASK_STATE_START; - spin_lock_init(&task->state_lock); + spin_lock_init(&task->lock); return 0; } @@ -119,9 +119,9 @@ void rxe_cleanup_task(struct rxe_task *task) task->destroyed = true; do { - spin_lock_bh(&task->state_lock); + spin_lock_bh(&task->lock); idle = (task->state == TASK_STATE_START); - spin_unlock_bh(&task->state_lock); + spin_unlock_bh(&task->lock); } while (!idle); tasklet_kill(&task->tasklet); diff --git a/drivers/infiniband/sw/rxe/rxe_task.h b/drivers/infiniband/sw/rxe/rxe_task.h index 99e0173e5c46..7b88129702ac 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.h +++ b/drivers/infiniband/sw/rxe/rxe_task.h @@ -21,7 +21,7 @@ enum { struct rxe_task { struct tasklet_struct tasklet; int state; - spinlock_t state_lock; /* spinlock for task state */ + spinlock_t lock; void *arg; int (*func)(void *arg); int ret; From patchwork Tue Oct 18 04:33:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13009880 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04317C4332F for ; Tue, 18 Oct 2022 04:36:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230178AbiJREgc (ORCPT ); Tue, 18 Oct 2022 00:36:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51128 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230018AbiJREg1 (ORCPT ); Tue, 18 Oct 2022 00:36:27 -0400 Received: from mail-oa1-x30.google.com (mail-oa1-x30.google.com [IPv6:2001:4860:4864:20::30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F1820A0261 for ; Mon, 17 Oct 2022 21:36:21 -0700 (PDT) Received: by mail-oa1-x30.google.com with SMTP id 586e51a60fabf-1321a1e94b3so15615249fac.1 for ; Mon, 17 Oct 2022 21:36:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CuLsuXZ7cplicFlCz9EKGPq647h3L7v9Bx7lFVIXiJw=; b=HHzeNrrWRNhnD2rVy4d4ha7mz7a9F2rXJca+GI1+2tCSRhOFYSwoaUeOme1XLqM9jH 1uTpYuZ5skMZKSKu6ZonsM0oAJ96QVFonFGgo+Yd54FvmRHQldbCB0jcZG6NyzTpzCQF Sg4PLnObNzvSH6G4HtALgzYsRe99Yg7pbdwZdX0A9jRsQgxFh92V+HDJ8jnI8/zvSouk qZhn0I1rOK1RZwrlg2s7uhdS4OOY+/wBZBAnk0S+WvbDpEVNM8H2rFYGbCBdGheyostd 7H/lvGHgPSfYB3CMKNWoCygmQaE2+uDAuhgSngvXq0AiLI4MoJsrnILxbE3SDIb4tI3q 0/sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CuLsuXZ7cplicFlCz9EKGPq647h3L7v9Bx7lFVIXiJw=; b=k1iz6KRPF/jBUZp1BEmcbRLUwhsS/c+EaF1eZr+lEDgM49rSo1ngVBoDstUOmnKFA0 v0eO2v4bMsnt8DasXF2bge8hU1Dg838RX9bKl07eUmhdXZ8S/WSRn1TpiyeDScGiKfJI eSy6VT1ataaGZJHH9zcCK+ik+ETGWQ4+5siXyKz8Vk5EOxWaIUs5/A5/7sD7BG2flR9U hW8KA5PLYHX2DwfOsOglxR8FZmGh9r28kq+kQwHz9Gh8Jj75u2oIKtQxl47yHeTBjxZx M1AsEAJyoDXl1DXNLjyO2CxmffzLrWP9oiFa94uZhrVqlFNsUtgQqie2KUS2gHipKnag QqRQ== X-Gm-Message-State: ACrzQf3Qsl72gK98/pExD8MCwIEOZ4Dux8WYCKxWnm69kkz+tXHQknEO 7nz8SqszyTWBh6ERpP8Fjf4= X-Google-Smtp-Source: AMsMyM5ighnX/nITZob4nJE19YSD6LkjzSpZxQVK2q1Mg3CNGcwrpomf4iLtrfBfZjWWc/fkn6ndiA== X-Received: by 2002:a05:6870:601e:b0:132:7ced:b20e with SMTP id t30-20020a056870601e00b001327cedb20emr591363oaa.124.1666067780693; Mon, 17 Oct 2022 21:36:20 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-290b-8972-ce76-602c.res6.spectrum.com. [2603:8081:140c:1a00:290b:8972:ce76:602c]) by smtp.googlemail.com with ESMTPSA id e96-20020a9d01e9000000b006618ca5caa0sm5480333ote.78.2022.10.17.21.36.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Oct 2022 21:36:20 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, matsuda-daisuke@fujitsu.com, lizhijian@fujitsu.com, leon@kernel.org, linux-rdma@vger.kernel.org, jenny.hack@hpe.com, ian.ziemba@hpe.com Cc: Bob Pearson Subject: [PATCH for-next 06/16] RDMA/rxe: Make task interface pluggable Date: Mon, 17 Oct 2022 23:33:37 -0500 Message-Id: <20221018043345.4033-7-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221018043345.4033-1-rpearsonhpe@gmail.com> References: <20221018043345.4033-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Make the internal interface to the task operations pluggable and add a new 'inline' type. Signed-off-by: Ian Ziemba Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_qp.c | 8 +- drivers/infiniband/sw/rxe/rxe_task.c | 160 ++++++++++++++++++++++----- drivers/infiniband/sw/rxe/rxe_task.h | 44 +++++--- 3 files changed, 165 insertions(+), 47 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index 3f6d62a80bea..b5e108794aa1 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -238,8 +238,10 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp, skb_queue_head_init(&qp->req_pkts); - rxe_init_task(&qp->req.task, qp, rxe_requester); - rxe_init_task(&qp->comp.task, qp, rxe_completer); + rxe_init_task(&qp->req.task, qp, rxe_requester, RXE_TASK_TYPE_TASKLET); + rxe_init_task(&qp->comp.task, qp, rxe_completer, + (qp_type(qp) == IB_QPT_RC) ? RXE_TASK_TYPE_TASKLET : + RXE_TASK_TYPE_INLINE); qp->qp_timeout_jiffies = 0; /* Can't be set for UD/UC in modify_qp */ if (init->qp_type == IB_QPT_RC) { @@ -286,7 +288,7 @@ static int rxe_qp_init_resp(struct rxe_dev *rxe, struct rxe_qp *qp, skb_queue_head_init(&qp->resp_pkts); - rxe_init_task(&qp->resp.task, qp, rxe_responder); + rxe_init_task(&qp->resp.task, qp, rxe_responder, RXE_TASK_TYPE_TASKLET); qp->resp.opcode = OPCODE_NONE; qp->resp.msn = 0; diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index 42442ede99e8..fcd87114949f 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -28,12 +28,11 @@ int __rxe_do_task(struct rxe_task *task) * a second caller finds the task already running * but looks just after the last call to func */ -static void do_task(struct tasklet_struct *t) +static void do_task(struct rxe_task *task) { + unsigned int iterations = RXE_MAX_ITERATIONS; int cont; int ret; - struct rxe_task *task = from_tasklet(task, t, tasklet); - unsigned int iterations = RXE_MAX_ITERATIONS; spin_lock_bh(&task->lock); switch (task->state) { @@ -94,28 +93,21 @@ static void do_task(struct tasklet_struct *t) task->ret = ret; } -int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *)) +static void disable_task(struct rxe_task *task) { - task->arg = arg; - task->func = func; - task->destroyed = false; - - tasklet_setup(&task->tasklet, do_task); - - task->state = TASK_STATE_START; - spin_lock_init(&task->lock); + /* todo */ +} - return 0; +static void enable_task(struct rxe_task *task) +{ + /* todo */ } -void rxe_cleanup_task(struct rxe_task *task) +/* busy wait until any previous tasks are done */ +static void cleanup_task(struct rxe_task *task) { bool idle; - /* - * Mark the task, then wait for it to finish. It might be - * running in a non-tasklet (direct call) context. - */ task->destroyed = true; do { @@ -123,32 +115,144 @@ void rxe_cleanup_task(struct rxe_task *task) idle = (task->state == TASK_STATE_START); spin_unlock_bh(&task->lock); } while (!idle); +} - tasklet_kill(&task->tasklet); +/* silently treat schedule as inline for inline tasks */ +static void inline_sched(struct rxe_task *task) +{ + do_task(task); } -void rxe_run_task(struct rxe_task *task) +static void inline_run(struct rxe_task *task) { - if (task->destroyed) - return; + do_task(task); +} - do_task(&task->tasklet); +static void inline_disable(struct rxe_task *task) +{ + disable_task(task); } -void rxe_sched_task(struct rxe_task *task) +static void inline_enable(struct rxe_task *task) { - if (task->destroyed) - return; + enable_task(task); +} + +static void inline_cleanup(struct rxe_task *task) +{ + cleanup_task(task); +} + +static const struct rxe_task_ops inline_ops = { + .sched = inline_sched, + .run = inline_run, + .enable = inline_enable, + .disable = inline_disable, + .cleanup = inline_cleanup, +}; +static void inline_init(struct rxe_task *task) +{ + task->ops = &inline_ops; +} + +/* use tsklet_xxx to avoid name collisions with tasklet_xxx */ +static void tsklet_sched(struct rxe_task *task) +{ tasklet_schedule(&task->tasklet); } -void rxe_disable_task(struct rxe_task *task) +static void tsklet_do_task(struct tasklet_struct *tasklet) { + struct rxe_task *task = container_of(tasklet, typeof(*task), tasklet); + + do_task(task); +} + +static void tsklet_run(struct rxe_task *task) +{ + do_task(task); +} + +static void tsklet_disable(struct rxe_task *task) +{ + disable_task(task); tasklet_disable(&task->tasklet); } -void rxe_enable_task(struct rxe_task *task) +static void tsklet_enable(struct rxe_task *task) { tasklet_enable(&task->tasklet); + enable_task(task); +} + +static void tsklet_cleanup(struct rxe_task *task) +{ + cleanup_task(task); + tasklet_kill(&task->tasklet); +} + +static const struct rxe_task_ops tsklet_ops = { + .sched = tsklet_sched, + .run = tsklet_run, + .enable = tsklet_enable, + .disable = tsklet_disable, + .cleanup = tsklet_cleanup, +}; + +static void tsklet_init(struct rxe_task *task) +{ + tasklet_setup(&task->tasklet, tsklet_do_task); + task->ops = &tsklet_ops; +} + +int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *), + enum rxe_task_type type) +{ + task->arg = arg; + task->func = func; + task->destroyed = false; + task->type = type; + task->state = TASK_STATE_START; + + spin_lock_init(&task->lock); + + switch (type) { + case RXE_TASK_TYPE_INLINE: + inline_init(task); + break; + case RXE_TASK_TYPE_TASKLET: + tsklet_init(task); + break; + default: + pr_debug("%s: invalid task type = %d\n", __func__, type); + return -EINVAL; + } + + return 0; +} + +void rxe_sched_task(struct rxe_task *task) +{ + task->ops->sched(task); +} + +void rxe_run_task(struct rxe_task *task) +{ + task->ops->run(task); +} + +void rxe_enable_task(struct rxe_task *task) +{ + task->ops->enable(task); +} + +void rxe_disable_task(struct rxe_task *task) +{ + task->ops->disable(task); +} + +void rxe_cleanup_task(struct rxe_task *task) +{ + task->ops->cleanup(task); } diff --git a/drivers/infiniband/sw/rxe/rxe_task.h b/drivers/infiniband/sw/rxe/rxe_task.h index 7b88129702ac..31963129ff7a 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.h +++ b/drivers/infiniband/sw/rxe/rxe_task.h @@ -7,6 +7,21 @@ #ifndef RXE_TASK_H #define RXE_TASK_H +struct rxe_task; + +struct rxe_task_ops { + void (*sched)(struct rxe_task *task); + void (*run)(struct rxe_task *task); + void (*disable)(struct rxe_task *task); + void (*enable)(struct rxe_task *task); + void (*cleanup)(struct rxe_task *task); +}; + +enum rxe_task_type { + RXE_TASK_TYPE_INLINE = 0, + RXE_TASK_TYPE_TASKLET = 1, +}; + enum { TASK_STATE_START = 0, TASK_STATE_BUSY = 1, @@ -19,24 +34,19 @@ enum { * called again. */ struct rxe_task { - struct tasklet_struct tasklet; - int state; - spinlock_t lock; - void *arg; - int (*func)(void *arg); - int ret; - bool destroyed; + struct tasklet_struct tasklet; + int state; + spinlock_t lock; + void *arg; + int (*func)(void *arg); + int ret; + bool destroyed; + const struct rxe_task_ops *ops; + enum rxe_task_type type; }; -/* - * init rxe_task structure - * arg => parameter to pass to fcn - * func => function to call until it returns != 0 - */ -int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *)); - -/* cleanup task */ -void rxe_cleanup_task(struct rxe_task *task); +int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *), + enum rxe_task_type type); /* * raw call to func in loop without any checking @@ -54,4 +64,6 @@ void rxe_disable_task(struct rxe_task *task); /* allow task to run */ void rxe_enable_task(struct rxe_task *task); +void rxe_cleanup_task(struct rxe_task *task); + #endif /* RXE_TASK_H */ From patchwork Tue Oct 18 04:33:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13009879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C393C4332F for ; Tue, 18 Oct 2022 04:36:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230161AbiJREgb (ORCPT ); Tue, 18 Oct 2022 00:36:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51108 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230010AbiJREg1 (ORCPT ); Tue, 18 Oct 2022 00:36:27 -0400 Received: from mail-oo1-xc2b.google.com (mail-oo1-xc2b.google.com [IPv6:2607:f8b0:4864:20::c2b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BF531A0244 for ; Mon, 17 Oct 2022 21:36:22 -0700 (PDT) Received: by mail-oo1-xc2b.google.com with SMTP id g15-20020a4a894f000000b0047f8e899623so2927698ooi.5 for ; Mon, 17 Oct 2022 21:36:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=24SPsUkbVLd5ye8D67mHnRGjR1k+hcsP3X79sb7BR3w=; b=AMwE9dRaytTK20YWqrkFwDv4gSlPQpm7fW98AqGzGfATyg0p/jus8mX0/DKAWymhQs E97ULfgs0i72LK3+t50su/ubVO8hnj5u4HHzAcqhFeoQZGTFcgT05qI7rZA7J69v9HJs sfecUnayPt/x8aDuDl5pVZZwJm3UAe197L3XQPxNfPE0PxLGZ6sJHGE+eWirYJnTL/oR xsgJKCalQKQSyStFh6t8NPMmhro0821R64ZQOQQvCZ4GvCqSL+1Sy1JzYYkrqgKepYYy muQjRjJ/U85uR0QMSaFAWQ723jmwmPgDDSWQTQZzE2GdnPeDut3QdDCnd/kFRkeIsqOB NxmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=24SPsUkbVLd5ye8D67mHnRGjR1k+hcsP3X79sb7BR3w=; b=gEAjNCzHkwaxLIxTwHNnkBSNvktHkUGCOQEbZLxzCjj7gJFFBVxFV8MVwW9SRiXIEP PxsOtYh75Oj5css07kvWfL8ytSPI7fZ0t9A5Wow2H70NRynCVXD3rLIA6CT4r1V/bs9D fbTlJhg1P9HR5WCK3JQi4BFz8oeGwgV8ZplkWZA5ygoigltxo5R8rISllJZ7u2l3Ua+G fIWHDRn3eDapjK2agrJgdRQuKDFGeVHSTHYZUOIR5CyWo8kHjlh+3lzk3+hT2jIQfewb iFgRK/QLUrqm2GBuoa640TCHjBPZm5nYW6by8A5GsF/Cu6z+6N+DtMVPNehj/oUW0M/x V+xw== X-Gm-Message-State: ACrzQf1AVa9546mFjPhqVa6ApjxrDjeaIsli7i+DY26ruYvLGraKT6+t /91x4YoukpkHg843z83n8iU= X-Google-Smtp-Source: AMsMyM4eswEJxUXP7y86BbDDUbAbGQWPf6KYOI9drcySKpS4y1UiIthoi+XeCC5jnIBcefP1tDUJeA== X-Received: by 2002:a4a:4847:0:b0:443:347d:6617 with SMTP id p68-20020a4a4847000000b00443347d6617mr535267ooa.94.1666067781936; Mon, 17 Oct 2022 21:36:21 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-290b-8972-ce76-602c.res6.spectrum.com. [2603:8081:140c:1a00:290b:8972:ce76:602c]) by smtp.googlemail.com with ESMTPSA id e96-20020a9d01e9000000b006618ca5caa0sm5480333ote.78.2022.10.17.21.36.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Oct 2022 21:36:21 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, matsuda-daisuke@fujitsu.com, lizhijian@fujitsu.com, leon@kernel.org, linux-rdma@vger.kernel.org, jenny.hack@hpe.com, ian.ziemba@hpe.com Cc: Bob Pearson Subject: [PATCH for-next 07/16] RDMA/rxe: Simplify reset state handling in rxe_resp.c Date: Mon, 17 Oct 2022 23:33:38 -0500 Message-Id: <20221018043345.4033-8-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221018043345.4033-1-rpearsonhpe@gmail.com> References: <20221018043345.4033-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Make rxe_responder() more like rxe_completer() and take qp reset handling out of the state machine. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_resp.c | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index 2a2a50de51b2..dd11dea70bbf 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -40,7 +40,6 @@ enum resp_states { RESPST_ERR_LENGTH, RESPST_ERR_CQ_OVERFLOW, RESPST_ERROR, - RESPST_RESET, RESPST_DONE, RESPST_EXIT, }; @@ -75,7 +74,6 @@ static char *resp_state_name[] = { [RESPST_ERR_LENGTH] = "ERR_LENGTH", [RESPST_ERR_CQ_OVERFLOW] = "ERR_CQ_OVERFLOW", [RESPST_ERROR] = "ERROR", - [RESPST_RESET] = "RESET", [RESPST_DONE] = "DONE", [RESPST_EXIT] = "EXIT", }; @@ -1278,8 +1276,9 @@ int rxe_responder(void *arg) switch (qp->resp.state) { case QP_STATE_RESET: - state = RESPST_RESET; - break; + rxe_drain_req_pkts(qp, false); + qp->resp.wqe = NULL; + goto exit; default: state = RESPST_GET_REQ; @@ -1438,11 +1437,6 @@ int rxe_responder(void *arg) goto exit; - case RESPST_RESET: - rxe_drain_req_pkts(qp, false); - qp->resp.wqe = NULL; - goto exit; - case RESPST_ERROR: qp->resp.goto_error = 0; pr_debug("qp#%d moved to error state\n", qp_num(qp)); From patchwork Tue Oct 18 04:33:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13009881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4808FC433FE for ; Tue, 18 Oct 2022 04:36:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230188AbiJREge (ORCPT ); Tue, 18 Oct 2022 00:36:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51126 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229917AbiJREg1 (ORCPT ); Tue, 18 Oct 2022 00:36:27 -0400 Received: from mail-oi1-x232.google.com (mail-oi1-x232.google.com [IPv6:2607:f8b0:4864:20::232]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB15DA0248 for ; Mon, 17 Oct 2022 21:36:23 -0700 (PDT) Received: by mail-oi1-x232.google.com with SMTP id l5so14398942oif.7 for ; Mon, 17 Oct 2022 21:36:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yW+DMhH4EeyZPJ/sfP09rQT+gnK+v+z7PE/z/pWlIgo=; b=RbnjK9O36NQy/mXBNXpXUx8qJ3GaX5X/vEzsaAETqdCYL9zhP/P4baTfAAIU+xK7za dUivgmoNJf7COEKwQ9WUX8NRlcDGqd37R3G08U8aiUmcZYO5XMC1MMptN4L31odCNj1V T7XUhxjEnfN2oOUZv3jSsaqW3pootEb08En9D7hDL3z/OtoRHgIFt4nACiwTubHY/vKX 18zSTn0QUJdtJBis5kQ4KRzBy+9Sb4G2NDiyS/CVatzupzBsE+H2QhLYoVhoo64j4q+z i1BmXaJ28qYufmPQJxPBZ0N4b4SpUhrDpaznQybU9HvXoMRWHotkKFPlRmswKnQqsBTU loLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yW+DMhH4EeyZPJ/sfP09rQT+gnK+v+z7PE/z/pWlIgo=; b=XA8yLNNZktaZWP0tLvDw5lwuSxL396f8tKyVxy1kiucBsEMdEe80Bw9WHuXfX7VUpU yEaGT8w+h1ghxgXbeaOd6bl2l/SkVabbE3uDt+ybt+qrtyvij1f9uOlLPIS5+VMoxbZY mzZQ9+VNEqTNTWLfmsdndU8gtR185Gd8x55nF1RWu9mLO/KzhTJ9ActezFAy2WMaHRrM jOGFM5cc6MVL8xyqiEDBoOBwiepubdoDslZBYpEfsFAMedjjRyddGM0a3DeZP0LqwfmL FXrfqDHHnT4haaol/R7wuZ1JiTOT3nY9npWhKiaR72/95sJ4ZRpTNnBO3JFgYg3LOZ0d l8VA== X-Gm-Message-State: ACrzQf0Znp7HJ/4ANX2HhaSoas31F2fIgakYdeisCBCVfTEuACpnY9zT d5QFFqw/bRdxwXtHpgT7/ny2DzktNFUh4w== X-Google-Smtp-Source: AMsMyM7rpBYpdZwYbXZ636zha/4eLEoHZMMux5MyGf50WInCWrQawKisMnDSxt0Qv2Ct6DB4Ax1DJA== X-Received: by 2002:a05:6808:1894:b0:355:2235:31f7 with SMTP id bi20-20020a056808189400b00355223531f7mr8282173oib.248.1666067783221; Mon, 17 Oct 2022 21:36:23 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-290b-8972-ce76-602c.res6.spectrum.com. [2603:8081:140c:1a00:290b:8972:ce76:602c]) by smtp.googlemail.com with ESMTPSA id e96-20020a9d01e9000000b006618ca5caa0sm5480333ote.78.2022.10.17.21.36.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Oct 2022 21:36:22 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, matsuda-daisuke@fujitsu.com, lizhijian@fujitsu.com, leon@kernel.org, linux-rdma@vger.kernel.org, jenny.hack@hpe.com, ian.ziemba@hpe.com Cc: Bob Pearson Subject: [PATCH for-next 08/16] RDMA/rxe: Split rxe_drain_resp_pkts() Date: Mon, 17 Oct 2022 23:33:39 -0500 Message-Id: <20221018043345.4033-9-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221018043345.4033-1-rpearsonhpe@gmail.com> References: <20221018043345.4033-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Split rxe_drain_resp_pkts() into two subroutines which perform separate functions. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_comp.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c index d2a250123617..5d434cce2b69 100644 --- a/drivers/infiniband/sw/rxe/rxe_comp.c +++ b/drivers/infiniband/sw/rxe/rxe_comp.c @@ -524,17 +524,21 @@ static inline enum comp_state complete_wqe(struct rxe_qp *qp, return COMPST_GET_WQE; } -static void rxe_drain_resp_pkts(struct rxe_qp *qp, bool notify) +static void rxe_drain_resp_pkts(struct rxe_qp *qp) { struct sk_buff *skb; - struct rxe_send_wqe *wqe; - struct rxe_queue *q = qp->sq.queue; while ((skb = skb_dequeue(&qp->resp_pkts))) { rxe_put(qp); kfree_skb(skb); ib_device_put(qp->ibqp.device); } +} + +static void rxe_drain_send_queue(struct rxe_qp *qp, bool notify) +{ + struct rxe_send_wqe *wqe; + struct rxe_queue *q = qp->sq.queue; while ((wqe = queue_head(q, q->type))) { if (notify) { @@ -565,6 +569,7 @@ int rxe_completer(void *arg) struct sk_buff *skb = NULL; struct rxe_pkt_info *pkt = NULL; enum comp_state state; + bool notify; int ret; if (!rxe_get(qp)) @@ -572,8 +577,9 @@ int rxe_completer(void *arg) if (!qp->valid || qp->comp.state == QP_STATE_ERROR || qp->comp.state == QP_STATE_RESET) { - rxe_drain_resp_pkts(qp, qp->valid && - qp->comp.state == QP_STATE_ERROR); + notify = qp->valid && (qp->comp.state == QP_STATE_ERROR); + rxe_drain_resp_pkts(qp); + rxe_drain_send_queue(qp, notify); goto exit; } From patchwork Tue Oct 18 04:33:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13009882 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEAB2C4332F for ; Tue, 18 Oct 2022 04:36:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229978AbiJREgh (ORCPT ); Tue, 18 Oct 2022 00:36:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229756AbiJREg2 (ORCPT ); Tue, 18 Oct 2022 00:36:28 -0400 Received: from mail-oa1-x2b.google.com (mail-oa1-x2b.google.com [IPv6:2001:4860:4864:20::2b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91A7BA0249 for ; Mon, 17 Oct 2022 21:36:24 -0700 (PDT) Received: by mail-oa1-x2b.google.com with SMTP id 586e51a60fabf-136b5dd6655so15617205fac.3 for ; Mon, 17 Oct 2022 21:36:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tSv2XfJeImnY9SqiatADTftY/YAZIrV/g9P7mzLsx/0=; b=lm9ga4DMZocsvBrgDeeH09EJdAqQGUq8vs6htY7hv3/iZ5+3OIgUXXZp9lxbc61bfK vPvv99eCjKFVDOs0owrGunmFZ4QP8HsTo7YIXNPhzaX3gXlCn5SK3i2hlbRG/O5yoE5L LBgrNYG+7OwVq2gB6O9iRuVmXAXAdbNV0jX5m0Ak5MMKdzLiKAZ3/pRUWoYLUP3v/2Sm fQVdXriGzL1UMrD8413L7OuGevuApZc2Ck6vAx3Mw0gsHwiQVw2B26ZQ2+QcHtB7jQ8f BsMKqn0P1hIF731b4APCm8QCutfrf0zPzO5W3B08E7aLgwKEW2VBHlTQuOtXVARFpt2e HeNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tSv2XfJeImnY9SqiatADTftY/YAZIrV/g9P7mzLsx/0=; b=aim1Pct/JQLYPd2zQkstTmU6LWxYT41z0vN9JZiPoCjio7ZGpP6RSZtdhhrvTF9wdR Mv2F7Xm2DiNFHfl66Ex8c9uTDQA/MWKZ2ByyGLxFHqfQ4yCDN1uwIYCtd7H0s0DgCAMt RuU27CuXKMwUi1pkMv+8sxkFGKytPq1XcJS4eUZwcqn+xfbXDBMapOufmq8K07MxoIVq /CyMgNscOSpd/axncQMJ6ZulLiPChCDexGSjHUCEi4M/Ky1qcS9pSUCOPByERtuqHIjW fPNLTfvv+L1KhQP22fZuWkc0d3kcwLuquFAbMBXqyIfAgD0lXCuZmX8yFhO2HzIwzKUb ygcg== X-Gm-Message-State: ACrzQf1FqupIWb/XTjmbHg5HGSjqhlnZZT6Gixk8lf9nnaoFJm8dg8ze qGy6BNV7wSuvP6yy/QXNP00= X-Google-Smtp-Source: AMsMyM53KGjfRC7Jn0MdwO19FEa6wR//mS/QHTnU3hOD9wDM1EuA4DbA/blR+N7ids0rgaJTlF8llQ== X-Received: by 2002:a05:6870:f622:b0:130:8d6:cbdb with SMTP id ek34-20020a056870f62200b0013008d6cbdbmr16512907oab.265.1666067784330; Mon, 17 Oct 2022 21:36:24 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-290b-8972-ce76-602c.res6.spectrum.com. [2603:8081:140c:1a00:290b:8972:ce76:602c]) by smtp.googlemail.com with ESMTPSA id e96-20020a9d01e9000000b006618ca5caa0sm5480333ote.78.2022.10.17.21.36.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Oct 2022 21:36:23 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, matsuda-daisuke@fujitsu.com, lizhijian@fujitsu.com, leon@kernel.org, linux-rdma@vger.kernel.org, jenny.hack@hpe.com, ian.ziemba@hpe.com Cc: Bob Pearson Subject: [PATCH for-next 09/16] RDMA/rxe: Handle qp error in rxe_resp.c Date: Mon, 17 Oct 2022 23:33:40 -0500 Message-Id: <20221018043345.4033-10-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221018043345.4033-1-rpearsonhpe@gmail.com> References: <20221018043345.4033-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Split rxe_drain_req_pkts() into two subroutines which perform separate tasks. Change qp error and reset states and !qp->valid in the same way as rxe_comp.c. FLush recv wqes for qp in error. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_resp.c | 64 ++++++++++++++++++++-------- 1 file changed, 47 insertions(+), 17 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index dd11dea70bbf..0bcdd1154641 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -1025,7 +1025,6 @@ static enum resp_states do_complete(struct rxe_qp *qp, return RESPST_CLEANUP; } - static int send_common_ack(struct rxe_qp *qp, u8 syndrome, u32 psn, int opcode, const char *msg) { @@ -1240,22 +1239,56 @@ static enum resp_states do_class_d1e_error(struct rxe_qp *qp) } } -static void rxe_drain_req_pkts(struct rxe_qp *qp, bool notify) +static void rxe_drain_req_pkts(struct rxe_qp *qp) { struct sk_buff *skb; - struct rxe_queue *q = qp->rq.queue; while ((skb = skb_dequeue(&qp->req_pkts))) { rxe_put(qp); kfree_skb(skb); ib_device_put(qp->ibqp.device); } +} + +int complete_flush(struct rxe_qp *qp, struct rxe_recv_wqe *wqe) +{ + struct rxe_cqe cqe; + struct ib_wc *wc = &cqe.ibwc; + struct ib_uverbs_wc *uwc = &cqe.uibwc; + + memset(&cqe, 0, sizeof(cqe)); - if (notify) - return; + if (qp->rcq->is_user) { + uwc->status = IB_WC_WR_FLUSH_ERR; + uwc->qp_num = qp->ibqp.qp_num; + uwc->wr_id = wqe->wr_id; + } else { + wc->status = IB_WC_WR_FLUSH_ERR; + wc->qp = &qp->ibqp; + wc->wr_id = wqe->wr_id; + } - while (!qp->srq && q && queue_head(q, q->type)) + if (rxe_cq_post(qp->rcq, &cqe, 0)) + return -ENOMEM; + + return 0; +} + +/* drain the receive queue. Complete each wqe with a flush error + * if notify is true or until a cq overflow occurs. + */ +static void rxe_drain_recv_queue(struct rxe_qp *qp, bool notify) +{ + struct rxe_recv_wqe *wqe; + struct rxe_queue *q = qp->rq.queue; + + while ((wqe = queue_head(q, q->type))) { + if (notify && complete_flush(qp, wqe)) + notify = 0; queue_advance_consumer(q, q->type); + } + + qp->resp.wqe = NULL; } int rxe_responder(void *arg) @@ -1264,6 +1297,7 @@ int rxe_responder(void *arg) struct rxe_dev *rxe = to_rdev(qp->ibqp.device); enum resp_states state; struct rxe_pkt_info *pkt = NULL; + bool notify; int ret; if (!rxe_get(qp)) @@ -1271,20 +1305,16 @@ int rxe_responder(void *arg) qp->resp.aeth_syndrome = AETH_ACK_UNLIMITED; - if (!qp->valid) - goto exit; - - switch (qp->resp.state) { - case QP_STATE_RESET: - rxe_drain_req_pkts(qp, false); - qp->resp.wqe = NULL; + if (!qp->valid || qp->resp.state == QP_STATE_ERROR || + qp->resp.state == QP_STATE_RESET) { + notify = qp->valid && (qp->resp.state == QP_STATE_ERROR); + rxe_drain_req_pkts(qp); + rxe_drain_recv_queue(qp, notify); goto exit; - - default: - state = RESPST_GET_REQ; - break; } + state = RESPST_GET_REQ; + while (1) { pr_debug("qp#%d state = %s\n", qp_num(qp), resp_state_name[state]); From patchwork Tue Oct 18 04:33:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13009883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87D67C433FE for ; Tue, 18 Oct 2022 04:36:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229702AbiJREgi (ORCPT ); Tue, 18 Oct 2022 00:36:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230086AbiJREg2 (ORCPT ); Tue, 18 Oct 2022 00:36:28 -0400 Received: from mail-oi1-x231.google.com (mail-oi1-x231.google.com [IPv6:2607:f8b0:4864:20::231]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 240AFA0268 for ; Mon, 17 Oct 2022 21:36:26 -0700 (PDT) Received: by mail-oi1-x231.google.com with SMTP id u15so14417495oie.2 for ; Mon, 17 Oct 2022 21:36:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gtHZM1yGvnB6A7szWnXegeRUkqTXVVuK9djjikzFkEc=; b=FykQRsmYjl3I9s4ChJjaC9ADQgwuxN64p4ra/AH+bdD6tI61CfxsbJ3MwFdOIy9rgy +qmQIl/jTXzuWKvQbipSbiygD6Vt54Lks6gkbprmRV1eTFNK7WicBcbuDbC5nxT2KdYU l0T/BWXJZdkLFR7KzAwT4pknpa9QnoHA4bxpAeXcaU5WC7bBKSo8r5l1p4S5gpGa+MMi IkT93/D8kDge7ZihPSUSl6ab8GWvQzqzfC2lCgV6wp6646VnHoYgAO22AgG8VelUOydD t7uoYklHcC7kAgwmcTqwj1wPoGWtZJ7ZaQnIokJ4sFzbpZ9nlKnwnCW6uEYkz+LScWCb 6A7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gtHZM1yGvnB6A7szWnXegeRUkqTXVVuK9djjikzFkEc=; b=4oca178J1+BvQNoln2di1uupCgQOwuVi4HMBLWcdf0EO2wRT7q5drEZBRQZiTXdw2F 0iHedC295bOZBeI1lg34uxDii2ZiaWEG0TFjS1mhYGnaIjw4beEwT13cXPPzsvkiLqcP KqVHg4Vr0/EeUmYL4VsAi+0Zo2zpvQ7P8PQn1XbwH1/BB4XugNXT3J0nvJJ5OHxdM3sl Pd0OuPXFSy5/Pr8THpHZZiAX9s/GQOQ6ZwRD192lsjU3EWDY8TN63G7h9sWj1jzBcG7z UegAv8MwTnWG6rYxNbk2Rf6QhdLeJIrXWKDTAPwJ++GQ7McCy7rOHt/dogq1u20aC0mQ ioww== X-Gm-Message-State: ACrzQf2j7/WUAWO17hp5y4vMjTcGKRCxDS0uld91cZoTRJr4KmkpIxJM X1faYIDJXzChd8JGcyAC1d0= X-Google-Smtp-Source: AMsMyM7c60wdr/okxEurSk5kyAIfuimd/S/G3O9PwUhUM2QnXzFFLXYtSvJXaCV8vQu1kGNAHbc7eQ== X-Received: by 2002:a05:6808:1507:b0:355:3ec:3109 with SMTP id u7-20020a056808150700b0035503ec3109mr576251oiw.263.1666067785544; Mon, 17 Oct 2022 21:36:25 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-290b-8972-ce76-602c.res6.spectrum.com. [2603:8081:140c:1a00:290b:8972:ce76:602c]) by smtp.googlemail.com with ESMTPSA id e96-20020a9d01e9000000b006618ca5caa0sm5480333ote.78.2022.10.17.21.36.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Oct 2022 21:36:25 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, matsuda-daisuke@fujitsu.com, lizhijian@fujitsu.com, leon@kernel.org, linux-rdma@vger.kernel.org, jenny.hack@hpe.com, ian.ziemba@hpe.com Cc: Bob Pearson Subject: [PATCH for-next 10/16] RDMA/rxe: Cleanup comp tasks in rxe_qp.c Date: Mon, 17 Oct 2022 23:33:41 -0500 Message-Id: <20221018043345.4033-11-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221018043345.4033-1-rpearsonhpe@gmail.com> References: <20221018043345.4033-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Take advantage of inline task behavior to cleanup code in rxe_qp.c for completer tasks. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_qp.c | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index b5e108794aa1..3691eb97c576 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -480,8 +480,7 @@ static void rxe_qp_reset(struct rxe_qp *qp) /* stop request/comp */ if (qp->sq.queue) { - if (qp_type(qp) == IB_QPT_RC) - rxe_disable_task(&qp->comp.task); + rxe_disable_task(&qp->comp.task); rxe_disable_task(&qp->req.task); } @@ -524,9 +523,7 @@ static void rxe_qp_reset(struct rxe_qp *qp) rxe_enable_task(&qp->resp.task); if (qp->sq.queue) { - if (qp_type(qp) == IB_QPT_RC) - rxe_enable_task(&qp->comp.task); - + rxe_enable_task(&qp->comp.task); rxe_enable_task(&qp->req.task); } } @@ -537,10 +534,7 @@ static void rxe_qp_drain(struct rxe_qp *qp) if (qp->sq.queue) { if (qp->req.state != QP_STATE_DRAINED) { qp->req.state = QP_STATE_DRAIN; - if (qp_type(qp) == IB_QPT_RC) - rxe_sched_task(&qp->comp.task); - else - __rxe_do_task(&qp->comp.task); + rxe_sched_task(&qp->comp.task); rxe_sched_task(&qp->req.task); } } @@ -556,11 +550,7 @@ void rxe_qp_error(struct rxe_qp *qp) /* drain work and packet queues */ rxe_sched_task(&qp->resp.task); - - if (qp_type(qp) == IB_QPT_RC) - rxe_sched_task(&qp->comp.task); - else - __rxe_do_task(&qp->comp.task); + rxe_sched_task(&qp->comp.task); rxe_sched_task(&qp->req.task); } From patchwork Tue Oct 18 04:33:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13009884 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D0F1C43219 for ; Tue, 18 Oct 2022 04:36:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229756AbiJREgj (ORCPT ); Tue, 18 Oct 2022 00:36:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51246 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230103AbiJREg3 (ORCPT ); Tue, 18 Oct 2022 00:36:29 -0400 Received: from mail-oa1-x2e.google.com (mail-oa1-x2e.google.com [IPv6:2001:4860:4864:20::2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F33CA024F for ; Mon, 17 Oct 2022 21:36:27 -0700 (PDT) Received: by mail-oa1-x2e.google.com with SMTP id 586e51a60fabf-1370acb6588so15580194fac.9 for ; Mon, 17 Oct 2022 21:36:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hwENGQaZsNNSx3Gv9rNHoDvR95XXZpwgm4cJviWnxVc=; b=jy4RbXnciS20q63uUOErGXeZGZadN/zoBxlBocsAMmETBq+pu19M+0Fv/jhBiYasfB FCexFk17aFLmYEiPE09rC87L6Ce0/Ma9lSCiI4jDfihrW3Ajpq3XPIC1wq0RMXVwbIYF oa7yH8RSP47/nTt35qSE6UfFr/bc3AgKhMPOaiDF9xlUO38bqXD18ia3jJUteW6iJlN6 uIzlbRzwN+TigI1mmA6XsspxYMOAJO0n+QzHlzigMEl573YojIEI8YaAL38TEMYixr6J AqWWRDL4Ze0Cgsly1E1Ibp3R1FjGaGjzIAWuBfataHw1KTR8DlhE2G2IYiEXOnp8vTjQ semQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hwENGQaZsNNSx3Gv9rNHoDvR95XXZpwgm4cJviWnxVc=; b=JlPTnkjXyJ3WgrMKmtN4tqdTrBZKrlnTsDJgYugzDe5DmI1Rz7f6gFmxWap7p3BQBu kU9v4Dl3ntu6k00Wgk29wBrpiQDDxVJBphuugMmQnd7lkfgiUSpWFkVS3tB4SVEbeNQk /C+fH5RH6h7dyHgDrkG/XqNyO3BMPdaXYUxmLjO+MXN6Gv5PAKzC6FKlJJ+j4xEd9h65 66Rr/blEdXw7xc3TCrAw/eDckw1S+6BlQ10Rk4FIkaIpmF2mFteuN6mB6D7QxL+QNrqo 2g1UT2RRxmg9SkwlLabdI33Lfsdmxcp89WfuaoXfujJKMF78mx2/ZE+thJelxp2LxrBm D+HQ== X-Gm-Message-State: ACrzQf3BBQfhjecVAgaSsxHDQDwsifsSvu5r8kcyJJ9uE06mZ+u0776e Sv1a5uz4/WGVkg4a/EKagbIU2v/2uL0pJQ== X-Google-Smtp-Source: AMsMyM4y9Ur1WFMq5tHtxjLbRvEqoOJZUR36hNE1EQlOzKlpXX7TzoZUGMArkswKq5Pc5zoAc/ZFQw== X-Received: by 2002:a05:6870:2052:b0:132:7b2:2fe6 with SMTP id l18-20020a056870205200b0013207b22fe6mr642634oad.98.1666067786636; Mon, 17 Oct 2022 21:36:26 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-290b-8972-ce76-602c.res6.spectrum.com. [2603:8081:140c:1a00:290b:8972:ce76:602c]) by smtp.googlemail.com with ESMTPSA id e96-20020a9d01e9000000b006618ca5caa0sm5480333ote.78.2022.10.17.21.36.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Oct 2022 21:36:26 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, matsuda-daisuke@fujitsu.com, lizhijian@fujitsu.com, leon@kernel.org, linux-rdma@vger.kernel.org, jenny.hack@hpe.com, ian.ziemba@hpe.com Cc: Bob Pearson Subject: [PATCH for-next 11/16] RDMA/rxe: Remove __rxe_do_task() Date: Mon, 17 Oct 2022 23:33:42 -0500 Message-Id: <20221018043345.4033-12-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221018043345.4033-1-rpearsonhpe@gmail.com> References: <20221018043345.4033-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The subroutine __rxe_do_task is not thread safe. It is only used in the rxe_qp_reset() and rxe_qp_do_cleanup() routines. After changes in error handling in the tasklet functions the queues can be drained by calling them once outside of the tasklet code. This allows __rxe_do_task() to be removed. Signed-off-by: Ian Ziemba Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_qp.c | 60 ++++++++++++++-------------- drivers/infiniband/sw/rxe/rxe_task.c | 13 ------ drivers/infiniband/sw/rxe/rxe_task.h | 6 --- 3 files changed, 29 insertions(+), 50 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index 3691eb97c576..50f6b8b8ad9d 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -477,28 +477,23 @@ static void rxe_qp_reset(struct rxe_qp *qp) { /* stop tasks from running */ rxe_disable_task(&qp->resp.task); - - /* stop request/comp */ - if (qp->sq.queue) { - rxe_disable_task(&qp->comp.task); - rxe_disable_task(&qp->req.task); - } + rxe_disable_task(&qp->comp.task); + rxe_disable_task(&qp->req.task); /* move qp to the reset state */ qp->req.state = QP_STATE_RESET; qp->comp.state = QP_STATE_RESET; qp->resp.state = QP_STATE_RESET; - /* let state machines reset themselves drain work and packet queues - * etc. - */ - __rxe_do_task(&qp->resp.task); + /* drain work and packet queues */ + rxe_responder(qp); + rxe_completer(qp); + rxe_requester(qp); - if (qp->sq.queue) { - __rxe_do_task(&qp->comp.task); - __rxe_do_task(&qp->req.task); + if (qp->rq.queue) + rxe_queue_reset(qp->rq.queue); + if (qp->sq.queue) rxe_queue_reset(qp->sq.queue); - } /* cleanup attributes */ atomic_set(&qp->ssn, 0); @@ -521,11 +516,8 @@ static void rxe_qp_reset(struct rxe_qp *qp) /* reenable tasks */ rxe_enable_task(&qp->resp.task); - - if (qp->sq.queue) { - rxe_enable_task(&qp->comp.task); - rxe_enable_task(&qp->req.task); - } + rxe_enable_task(&qp->comp.task); + rxe_enable_task(&qp->req.task); } /* drain the send queue */ @@ -543,15 +535,25 @@ static void rxe_qp_drain(struct rxe_qp *qp) /* move the qp to the error state */ void rxe_qp_error(struct rxe_qp *qp) { + /* stop tasks from running */ + rxe_disable_task(&qp->resp.task); + rxe_disable_task(&qp->comp.task); + rxe_disable_task(&qp->req.task); + qp->req.state = QP_STATE_ERROR; qp->resp.state = QP_STATE_ERROR; qp->comp.state = QP_STATE_ERROR; qp->attr.qp_state = IB_QPS_ERR; /* drain work and packet queues */ - rxe_sched_task(&qp->resp.task); - rxe_sched_task(&qp->comp.task); - rxe_sched_task(&qp->req.task); + rxe_responder(qp); + rxe_completer(qp); + rxe_requester(qp); + + /* reenable tasks */ + rxe_enable_task(&qp->resp.task); + rxe_enable_task(&qp->comp.task); + rxe_enable_task(&qp->req.task); } /* called by the modify qp verb */ @@ -770,24 +772,20 @@ static void rxe_qp_do_cleanup(struct work_struct *work) qp->valid = 0; qp->qp_timeout_jiffies = 0; - rxe_cleanup_task(&qp->resp.task); if (qp_type(qp) == IB_QPT_RC) { del_timer_sync(&qp->retrans_timer); del_timer_sync(&qp->rnr_nak_timer); } + rxe_cleanup_task(&qp->resp.task); rxe_cleanup_task(&qp->req.task); rxe_cleanup_task(&qp->comp.task); - /* flush out any receive wr's or pending requests */ - if (qp->req.task.func) - __rxe_do_task(&qp->req.task); - - if (qp->sq.queue) { - __rxe_do_task(&qp->comp.task); - __rxe_do_task(&qp->req.task); - } + /* drain any receive wr's or pending requests */ + rxe_responder(qp); + rxe_completer(qp); + rxe_requester(qp); if (qp->sq.queue) rxe_queue_cleanup(qp->sq.queue); diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index fcd87114949f..75ec195f4176 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -10,19 +10,6 @@ #include "rxe.h" -int __rxe_do_task(struct rxe_task *task) - -{ - int ret; - - while ((ret = task->func(task->arg)) == 0) - ; - - task->ret = ret; - - return ret; -} - /* * this locking is due to a potential race where * a second caller finds the task already running diff --git a/drivers/infiniband/sw/rxe/rxe_task.h b/drivers/infiniband/sw/rxe/rxe_task.h index 31963129ff7a..d594468fcf56 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.h +++ b/drivers/infiniband/sw/rxe/rxe_task.h @@ -48,12 +48,6 @@ struct rxe_task { int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *), enum rxe_task_type type); -/* - * raw call to func in loop without any checking - * can call when tasklets are disabled - */ -int __rxe_do_task(struct rxe_task *task); - void rxe_run_task(struct rxe_task *task); void rxe_sched_task(struct rxe_task *task); From patchwork Tue Oct 18 04:33:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13009885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2B00C433FE for ; Tue, 18 Oct 2022 04:36:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230014AbiJREgk (ORCPT ); Tue, 18 Oct 2022 00:36:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51128 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229711AbiJREga (ORCPT ); Tue, 18 Oct 2022 00:36:30 -0400 Received: from mail-oi1-x234.google.com (mail-oi1-x234.google.com [IPv6:2607:f8b0:4864:20::234]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC09CA0270 for ; Mon, 17 Oct 2022 21:36:28 -0700 (PDT) Received: by mail-oi1-x234.google.com with SMTP id w196so14384233oiw.8 for ; Mon, 17 Oct 2022 21:36:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=moCS8bMrE9y/ygj3tPnfai4xVg2bxmPG46HvIHu3Rcc=; b=I5jXWSMDLgl+WNPTDFSY4uizQA+6CzjOFBqrPrr8VgwdQGIa+H/Zm+eNFZ1j5tb3cD G4b3BCJ8JAOOkfSB4MGvf3v4/ko1XKxddqIac2mjqyXcMA4iWjzjg/ZzJ2O+ON/fsYMN jF1xe0fsDQjGEJEsWsaceFo40FyzdPseoYZ5CpdWL4OAkoVuR2qvWw04lq0uyDMS69wH byynAPYPHsXBhWrcHDy7NWiSCz3X5T/F0kOG4ZWzsagwU1xL8S89KYojfCAM4I32j40x 810yUvt1E3wGZlpnK47iu+5856xk+hn2pUyYzP4csDs3eNzJe5239oW0sKHvos/WuteP /mTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=moCS8bMrE9y/ygj3tPnfai4xVg2bxmPG46HvIHu3Rcc=; b=b+w5Ei8sXl2lTAgsVBA4blbbydHuZu3W3tPGAltUfmFGhXEDyjQQLtKa9JdIRIipfA Ee2aS/qMb1vqY1YCbmMVKaWDvzKH3ulSIci3SrZtg0PG/BGpz87YpDH4RCgVTiQK8LQp 73gmdcAFDdhEvbzppbMHYh2So7Q+VP7VldqVvsPw+ciQHXIOLv0R6GmxH9m0Vu+k2f72 StmkgFLqn7LGSQukfXIY/wEPqvNqCA1WiYAQ1u6Ccf9m4gCihLWvV7nu4PwgMW8bT3qL MGVAICAjO4r/WPYR6farTXecB/UVhXDBHq5jBUTJ9gRCHSboa81AXHnqarAtqlZQFQeQ ucjA== X-Gm-Message-State: ACrzQf3B1ANFQc/CYjN02Ud5YXwZwBDIEqu4jkGPhYjKhLPqpCM21wL2 skzj9IFh3TD2C1iuNMZzwVo= X-Google-Smtp-Source: AMsMyM4yH2VC5/4+gB49TYIYz1XoncbT3t6bHTTv8KITRQBACSTSkPHvjBJjbl6+wTA6+ZHdDH3Ytw== X-Received: by 2002:a05:6808:3085:b0:355:956:6fe4 with SMTP id bl5-20020a056808308500b0035509566fe4mr10925737oib.43.1666067788026; Mon, 17 Oct 2022 21:36:28 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-290b-8972-ce76-602c.res6.spectrum.com. [2603:8081:140c:1a00:290b:8972:ce76:602c]) by smtp.googlemail.com with ESMTPSA id e96-20020a9d01e9000000b006618ca5caa0sm5480333ote.78.2022.10.17.21.36.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Oct 2022 21:36:27 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, matsuda-daisuke@fujitsu.com, lizhijian@fujitsu.com, leon@kernel.org, linux-rdma@vger.kernel.org, jenny.hack@hpe.com, ian.ziemba@hpe.com Cc: Bob Pearson Subject: [PATCH for-next 12/16] RDMA/rxe: Make tasks schedule each other Date: Mon, 17 Oct 2022 23:33:43 -0500 Message-Id: <20221018043345.4033-13-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221018043345.4033-1-rpearsonhpe@gmail.com> References: <20221018043345.4033-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Replace rxe_run_task() by rxe_sched_task() when tasks call each other. These are not performance critical and mainly involve error paths but they run the risk of causing deadlocks. Signed-off-by: Ian Ziemba Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_comp.c | 8 ++++---- drivers/infiniband/sw/rxe/rxe_req.c | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c index 5d434cce2b69..6c15c9307660 100644 --- a/drivers/infiniband/sw/rxe/rxe_comp.c +++ b/drivers/infiniband/sw/rxe/rxe_comp.c @@ -308,7 +308,7 @@ static inline enum comp_state check_ack(struct rxe_qp *qp, qp->comp.psn = pkt->psn; if (qp->req.wait_psn) { qp->req.wait_psn = 0; - rxe_run_task(&qp->req.task); + rxe_sched_task(&qp->req.task); } } return COMPST_ERROR_RETRY; @@ -455,7 +455,7 @@ static void do_complete(struct rxe_qp *qp, struct rxe_send_wqe *wqe) */ if (qp->req.wait_fence) { qp->req.wait_fence = 0; - rxe_run_task(&qp->req.task); + rxe_sched_task(&qp->req.task); } } @@ -469,7 +469,7 @@ static inline enum comp_state complete_ack(struct rxe_qp *qp, if (qp->req.need_rd_atomic) { qp->comp.timeout_retry = 0; qp->req.need_rd_atomic = 0; - rxe_run_task(&qp->req.task); + rxe_sched_task(&qp->req.task); } } @@ -723,7 +723,7 @@ int rxe_completer(void *arg) RXE_CNT_COMP_RETRY); qp->req.need_retry = 1; qp->comp.started_retry = 1; - rxe_run_task(&qp->req.task); + rxe_sched_task(&qp->req.task); } goto done; diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 41f1d84f0acb..fba7572e1d0c 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -733,7 +733,7 @@ int rxe_requester(void *arg) qp->req.wqe_index); wqe->state = wqe_state_done; wqe->status = IB_WC_SUCCESS; - rxe_run_task(&qp->comp.task); + rxe_sched_task(&qp->comp.task); goto done; } payload = mtu; @@ -817,7 +817,7 @@ int rxe_requester(void *arg) qp->req.wqe_index = queue_next_index(qp->sq.queue, qp->req.wqe_index); wqe->state = wqe_state_error; qp->req.state = QP_STATE_ERROR; - rxe_run_task(&qp->comp.task); + rxe_sched_task(&qp->comp.task); exit: ret = -EAGAIN; out: From patchwork Tue Oct 18 04:33:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13009886 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B29F3C4332F for ; Tue, 18 Oct 2022 04:36:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229711AbiJREgl (ORCPT ); Tue, 18 Oct 2022 00:36:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229725AbiJREgb (ORCPT ); Tue, 18 Oct 2022 00:36:31 -0400 Received: from mail-oa1-x2b.google.com (mail-oa1-x2b.google.com [IPv6:2001:4860:4864:20::2b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4A03A0244 for ; Mon, 17 Oct 2022 21:36:30 -0700 (PDT) Received: by mail-oa1-x2b.google.com with SMTP id 586e51a60fabf-1324e7a1284so15590969fac.10 for ; Mon, 17 Oct 2022 21:36:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ebsTGF8eEEinSFVudIsRB2osA85CvNiD6HhTY4AJp0I=; b=m5dSK5P0E8omzcj5t6+vurYAVn3rUoX/oZvBCrJKiYlrqdHOleU6xZHFyiMFB3DzLB nFQWKR7u4jMRabz8J8Cp/UDFZ/AEIZLZBvq4JDqMQWS97bzs6dVMFoH9VKSUfvPS4Bb+ ez0nunRgn8iSoUZ3C/z6F3rDVUkJIJwLFkGQdfArKOG6SemM/0/T4VIt4/Kt1b6FdE1V ktXKrBDgJkZVcsdrICZDB7fqxjJ5NLMcFTC8zxmgnwhTE5QIJ506qleljC6O7dXlx/sm OEbSyHhb9O4SnT1jNFDIWD7wsLglW0Eh1ceR+uMc6ajEHFKqbJWISui4OqXh69W84hxD URZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ebsTGF8eEEinSFVudIsRB2osA85CvNiD6HhTY4AJp0I=; b=LyBmeBAqOxr6L5tzN86RssOLApfp03jaLmDc32KSsFecriXQFVnFQwkZ/eYhJQttLa gxuAiNIH77xBpIZKgCcGX7BRcED7ryjZVyvoA7foZdPbicw870yzhSjcnhU/vVPMkWgG GI9uCp28bhes0KNfhq8cQR/16qWSpcm70z6EdK4c77ZlRhS1SI6/vJI7O0vnFCNpdgpC Y/lqjpd+2qExbgboZ1ByZfC7cmVEhYH+Ncq6MlDb0REHRzW/oVNp9f2JeBcIuz4TvSYG 6ziKHD/b1LzLELq3j4z5ciQjoEYnldDMMo4+cDFzGquPfkVbgyaSa+B+Kg6YKDqj31Yg EZZg== X-Gm-Message-State: ACrzQf389CxILCaL/LK6CrWBwAYVZCppoW+hZqGi8eAAa8vlu6ydikUI hTdGnrY5kVKdbBolGMagWoY= X-Google-Smtp-Source: AMsMyM5Ik9CrWIJ4VpV8ElnUlGGGUz2t3P5vEeOyx9M7t4NGMDwGjsUUywn5yEsalD9H1qtzivPHAg== X-Received: by 2002:a05:6871:79e:b0:132:e1f1:c75f with SMTP id o30-20020a056871079e00b00132e1f1c75fmr17726866oap.234.1666067789264; Mon, 17 Oct 2022 21:36:29 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-290b-8972-ce76-602c.res6.spectrum.com. [2603:8081:140c:1a00:290b:8972:ce76:602c]) by smtp.googlemail.com with ESMTPSA id e96-20020a9d01e9000000b006618ca5caa0sm5480333ote.78.2022.10.17.21.36.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Oct 2022 21:36:28 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, matsuda-daisuke@fujitsu.com, lizhijian@fujitsu.com, leon@kernel.org, linux-rdma@vger.kernel.org, jenny.hack@hpe.com, ian.ziemba@hpe.com Cc: Bob Pearson Subject: [PATCH for-next 13/16] RDMA/rxe: Implement disable/enable_task() Date: Mon, 17 Oct 2022 23:33:44 -0500 Message-Id: <20221018043345.4033-14-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221018043345.4033-1-rpearsonhpe@gmail.com> References: <20221018043345.4033-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Implement common disable_task() and enable_task() routines by adding a new PAUSED state to the do_task() state machine. These replace tasklet_disable and tasklet_enable with code that can be shared with all the task types. Move rxe_sched_task to re-schedule the task outside of the locks to avoid a deadlock. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_task.c | 107 ++++++++++++++++----------- drivers/infiniband/sw/rxe/rxe_task.h | 1 + 2 files changed, 66 insertions(+), 42 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index 75ec195f4176..a9eb66d69cb7 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -10,36 +10,46 @@ #include "rxe.h" -/* - * this locking is due to a potential race where - * a second caller finds the task already running - * but looks just after the last call to func - */ -static void do_task(struct rxe_task *task) +static bool task_is_idle(struct rxe_task *task) { - unsigned int iterations = RXE_MAX_ITERATIONS; - int cont; - int ret; + if (task->destroyed) + return false; spin_lock_bh(&task->lock); switch (task->state) { case TASK_STATE_START: task->state = TASK_STATE_BUSY; spin_unlock_bh(&task->lock); - break; - + return true; case TASK_STATE_BUSY: task->state = TASK_STATE_ARMED; fallthrough; case TASK_STATE_ARMED: - spin_unlock_bh(&task->lock); - return; - + case TASK_STATE_PAUSED: + break; default: + WARN_ON(1); + break; + } + spin_unlock_bh(&task->lock); + + return false; +} + +static void do_task(struct rxe_task *task) +{ + unsigned int iterations = RXE_MAX_ITERATIONS; + bool resched = false; + int cont; + int ret; + + /* flush out pending tasks */ + spin_lock_bh(&task->lock); + if (task->state == TASK_STATE_PAUSED) { spin_unlock_bh(&task->lock); - pr_warn("%s failed with bad state %d\n", __func__, task->state); return; } + spin_unlock_bh(&task->lock); do { cont = 0; @@ -47,47 +57,52 @@ static void do_task(struct rxe_task *task) spin_lock_bh(&task->lock); switch (task->state) { + case TASK_STATE_START: case TASK_STATE_BUSY: if (ret) { task->state = TASK_STATE_START; - } else if (iterations--) { + } else if (task->type == RXE_TASK_TYPE_INLINE || + iterations--) { cont = 1; } else { - /* reschedule the tasklet and exit - * the loop to give up the cpu - */ - tasklet_schedule(&task->tasklet); task->state = TASK_STATE_START; + resched = true; } break; - - /* someone tried to run the task since the last time we called - * func, so we will call one more time regardless of the - * return value - */ case TASK_STATE_ARMED: task->state = TASK_STATE_BUSY; cont = 1; break; - + case TASK_STATE_PAUSED: + break; default: - pr_warn("%s failed with bad state %d\n", __func__, - task->state); + WARN_ON(1); + break; } spin_unlock_bh(&task->lock); } while (cont); + if (resched) + rxe_sched_task(task); + task->ret = ret; } static void disable_task(struct rxe_task *task) { - /* todo */ + spin_lock_bh(&task->lock); + task->state = TASK_STATE_PAUSED; + spin_unlock_bh(&task->lock); } static void enable_task(struct rxe_task *task) { - /* todo */ + spin_lock_bh(&task->lock); + task->state = TASK_STATE_START; + spin_unlock_bh(&task->lock); + + /* restart task in case */ + rxe_run_task(task); } /* busy wait until any previous tasks are done */ @@ -99,7 +114,8 @@ static void cleanup_task(struct rxe_task *task) do { spin_lock_bh(&task->lock); - idle = (task->state == TASK_STATE_START); + idle = (task->state == TASK_STATE_START || + task->state == TASK_STATE_PAUSED); spin_unlock_bh(&task->lock); } while (!idle); } @@ -107,22 +123,26 @@ static void cleanup_task(struct rxe_task *task) /* silently treat schedule as inline for inline tasks */ static void inline_sched(struct rxe_task *task) { - do_task(task); + if (task_is_idle(task)) + do_task(task); } static void inline_run(struct rxe_task *task) { - do_task(task); + if (task_is_idle(task)) + do_task(task); } static void inline_disable(struct rxe_task *task) { - disable_task(task); + if (!task->destroyed) + disable_task(task); } static void inline_enable(struct rxe_task *task) { - enable_task(task); + if (!task->destroyed) + enable_task(task); } static void inline_cleanup(struct rxe_task *task) @@ -146,31 +166,34 @@ static void inline_init(struct rxe_task *task) /* use tsklet_xxx to avoid name collisions with tasklet_xxx */ static void tsklet_sched(struct rxe_task *task) { - tasklet_schedule(&task->tasklet); + if (task_is_idle(task)) + tasklet_schedule(&task->tasklet); } static void tsklet_do_task(struct tasklet_struct *tasklet) { struct rxe_task *task = container_of(tasklet, typeof(*task), tasklet); - do_task(task); + if (!task->destroyed) + do_task(task); } static void tsklet_run(struct rxe_task *task) { - do_task(task); + if (task_is_idle(task)) + do_task(task); } static void tsklet_disable(struct rxe_task *task) { - disable_task(task); - tasklet_disable(&task->tasklet); + if (!task->destroyed) + disable_task(task); } static void tsklet_enable(struct rxe_task *task) { - tasklet_enable(&task->tasklet); - enable_task(task); + if (!task->destroyed) + enable_task(task); } static void tsklet_cleanup(struct rxe_task *task) diff --git a/drivers/infiniband/sw/rxe/rxe_task.h b/drivers/infiniband/sw/rxe/rxe_task.h index d594468fcf56..792832786456 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.h +++ b/drivers/infiniband/sw/rxe/rxe_task.h @@ -26,6 +26,7 @@ enum { TASK_STATE_START = 0, TASK_STATE_BUSY = 1, TASK_STATE_ARMED = 2, + TASK_STATE_PAUSED = 3, }; /* From patchwork Tue Oct 18 04:33:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13009887 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7ADAC43217 for ; Tue, 18 Oct 2022 04:36:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229837AbiJREgn (ORCPT ); Tue, 18 Oct 2022 00:36:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51130 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230010AbiJREgc (ORCPT ); Tue, 18 Oct 2022 00:36:32 -0400 Received: from mail-oo1-xc35.google.com (mail-oo1-xc35.google.com [IPv6:2607:f8b0:4864:20::c35]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2A11EA0248 for ; Mon, 17 Oct 2022 21:36:31 -0700 (PDT) Received: by mail-oo1-xc35.google.com with SMTP id c17-20020a4aa4d1000000b0047653e7c5f3so2943162oom.1 for ; Mon, 17 Oct 2022 21:36:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hLdRG9YSPMV2q7W0Y/cMM2IC9csSbc4KwhgV7nSPUMw=; b=i+lLYZJdPF+8JZl0ehJ+puus/Ngr40vRQnSM6DBTTvUlydH5Y/zbwFJ2hF8pwOvbyW +O6dEnlHWQzF1vcrt6r0WAlZ5t6sYPS7cpc5EnCaWdkNsVhXLSMd9ZjmP1yAfRmedP7G asy18nbz2RbC1m8QPpl2L7EI/956QH1nSkhTIAGbW+tgY0SZk9TrCH5RHH2sMDr7MnI0 poXGWL4a33S2x/sCTssEq9LRIS+jy68rQ464fJwtifWs5NK3b+B7lnMuqRdcWPPFlxvr wZYgGEJeM/yrMwhtCJiS98dKi6fduYJOAUa5mgmTNWYq6X+B+0d7vC9DkWf17e/hmnu6 KBTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hLdRG9YSPMV2q7W0Y/cMM2IC9csSbc4KwhgV7nSPUMw=; b=HqJaasRYO0UA6o6u/x6mghyBB0tjk1cbEWq76TK8oijpB79qqTUNeKLBfarfQ62thg /kksZ5BmjBS/ujq4SKTXDPY4OgEj52vpkWFHfFU1nMssksAzyisThlvn13OC+1/phyEo efX0A7m8VqNqnXzeZld/pGqlMOkMc5ecFz8XM5xqCppZiBZBUiyEfShAFdgPfMDRqYw1 FjvwWpn7U+phM4GFEp38kKBSTG0u3+qwuVRzc55x1Mmrmy4zwwHyBa+LfZPFWyZMiIFc 2S13Di9hQMBmKlirujADzKHxdLT/jYe2wxbVhhcLeMeWc/HOy6XfXOgYO/J49rXO2qQ+ Ve2g== X-Gm-Message-State: ACrzQf1N2vSrmUaiJk+isNYAjS+ROS9LMeCjCHRKEbywOUXwT1AaN0Vp XzNKZ637zxWbdFohi+jZzl4IvOnfWb9kMw== X-Google-Smtp-Source: AMsMyM4BIIdRKpr2HEfRbpGmbU4PoLiDBekM7S3m3fDuNBIy1WjWHDCMZLhzX+JLKf2TGHNcPFUbXQ== X-Received: by 2002:a4a:8d57:0:b0:476:99ae:46a5 with SMTP id x23-20020a4a8d57000000b0047699ae46a5mr538016ook.51.1666067790469; Mon, 17 Oct 2022 21:36:30 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-290b-8972-ce76-602c.res6.spectrum.com. [2603:8081:140c:1a00:290b:8972:ce76:602c]) by smtp.googlemail.com with ESMTPSA id e96-20020a9d01e9000000b006618ca5caa0sm5480333ote.78.2022.10.17.21.36.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Oct 2022 21:36:29 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, matsuda-daisuke@fujitsu.com, lizhijian@fujitsu.com, leon@kernel.org, linux-rdma@vger.kernel.org, jenny.hack@hpe.com, ian.ziemba@hpe.com Cc: Bob Pearson Subject: [PATCH for-next 14/16] RDMA/rxe: Replace TASK_STATE_START by TASK_STATE_IDLE Date: Mon, 17 Oct 2022 23:33:45 -0500 Message-Id: <20221018043345.4033-15-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221018043345.4033-1-rpearsonhpe@gmail.com> References: <20221018043345.4033-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Replace the enum TASK_STATE_START by TASK_STATE_IDLE. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_task.c | 37 ++++++++++++++-------------- drivers/infiniband/sw/rxe/rxe_task.h | 4 +-- 2 files changed, 21 insertions(+), 20 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index a9eb66d69cb7..a2c58ce66c8f 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -12,12 +12,12 @@ static bool task_is_idle(struct rxe_task *task) { - if (task->destroyed) + if (!task->valid) return false; spin_lock_bh(&task->lock); switch (task->state) { - case TASK_STATE_START: + case TASK_STATE_IDLE: task->state = TASK_STATE_BUSY; spin_unlock_bh(&task->lock); return true; @@ -57,15 +57,15 @@ static void do_task(struct rxe_task *task) spin_lock_bh(&task->lock); switch (task->state) { - case TASK_STATE_START: + case TASK_STATE_IDLE: case TASK_STATE_BUSY: if (ret) { - task->state = TASK_STATE_START; + task->state = TASK_STATE_IDLE; } else if (task->type == RXE_TASK_TYPE_INLINE || iterations--) { cont = 1; } else { - task->state = TASK_STATE_START; + task->state = TASK_STATE_IDLE; resched = true; } break; @@ -98,7 +98,7 @@ static void disable_task(struct rxe_task *task) static void enable_task(struct rxe_task *task) { spin_lock_bh(&task->lock); - task->state = TASK_STATE_START; + task->state = TASK_STATE_IDLE; spin_unlock_bh(&task->lock); /* restart task in case */ @@ -110,11 +110,11 @@ static void cleanup_task(struct rxe_task *task) { bool idle; - task->destroyed = true; + task->valid = false; do { spin_lock_bh(&task->lock); - idle = (task->state == TASK_STATE_START || + idle = (task->state == TASK_STATE_IDLE || task->state == TASK_STATE_PAUSED); spin_unlock_bh(&task->lock); } while (!idle); @@ -135,13 +135,13 @@ static void inline_run(struct rxe_task *task) static void inline_disable(struct rxe_task *task) { - if (!task->destroyed) + if (task->valid) disable_task(task); } static void inline_enable(struct rxe_task *task) { - if (!task->destroyed) + if (task->valid) enable_task(task); } @@ -174,7 +174,7 @@ static void tsklet_do_task(struct tasklet_struct *tasklet) { struct rxe_task *task = container_of(tasklet, typeof(*task), tasklet); - if (!task->destroyed) + if (task->valid) do_task(task); } @@ -186,13 +186,13 @@ static void tsklet_run(struct rxe_task *task) static void tsklet_disable(struct rxe_task *task) { - if (!task->destroyed) + if (task->valid) disable_task(task); } static void tsklet_enable(struct rxe_task *task) { - if (!task->destroyed) + if (task->valid) enable_task(task); } @@ -219,11 +219,10 @@ static void tsklet_init(struct rxe_task *task) int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *), enum rxe_task_type type) { - task->arg = arg; - task->func = func; - task->destroyed = false; - task->type = type; - task->state = TASK_STATE_START; + task->arg = arg; + task->func = func; + task->type = type; + task->state = TASK_STATE_IDLE; spin_lock_init(&task->lock); @@ -239,6 +238,8 @@ int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *), return -EINVAL; } + task->valid = true; + return 0; } diff --git a/drivers/infiniband/sw/rxe/rxe_task.h b/drivers/infiniband/sw/rxe/rxe_task.h index 792832786456..c81947e88629 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.h +++ b/drivers/infiniband/sw/rxe/rxe_task.h @@ -23,7 +23,7 @@ enum rxe_task_type { }; enum { - TASK_STATE_START = 0, + TASK_STATE_IDLE = 0, TASK_STATE_BUSY = 1, TASK_STATE_ARMED = 2, TASK_STATE_PAUSED = 3, @@ -41,7 +41,7 @@ struct rxe_task { void *arg; int (*func)(void *arg); int ret; - bool destroyed; + bool valid; const struct rxe_task_ops *ops; enum rxe_task_type type; }; From patchwork Tue Oct 18 04:33:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13009888 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64A34C433FE for ; Tue, 18 Oct 2022 04:36:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229917AbiJREgo (ORCPT ); Tue, 18 Oct 2022 00:36:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51156 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230181AbiJREgd (ORCPT ); Tue, 18 Oct 2022 00:36:33 -0400 Received: from mail-oa1-x2a.google.com (mail-oa1-x2a.google.com [IPv6:2001:4860:4864:20::2a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5570AA0249 for ; Mon, 17 Oct 2022 21:36:32 -0700 (PDT) Received: by mail-oa1-x2a.google.com with SMTP id 586e51a60fabf-12c8312131fso15610833fac.4 for ; Mon, 17 Oct 2022 21:36:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AH0HuGzyqYEVe4u3MM7m2JilO3VaD70MzOFOyhxiih4=; b=d1JZbh82jANFWBCFVxuvUoiDvQETiNAw18gbXuZAM+le+gb0ZqkWMW4G7X6I2k8Snt dS8utI1TJqKY97T0Xadrb2viu+QzxmErm91+Q57NUWxEdDmNDx0Rmf21rnJD9+9mBLqb A6WX6zXY8afBdxewR0c2Mm7Jvr9RohNCt66BVzi7vsGntoNaNwZkiI0jjWNt4VSzxXCU ENxhfX1S9KZedu/XiNbU4SDcBTYPtzH+E6hN7LJNhP6tDSUeigriy1LZwVwf70d3GOFh Aa138Th6CwCpwxZICBdJypxIjxbhhssdhbM/u8pDGpT2AzzrAcsAHZA1DGHgS+QfYBfY 4VhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AH0HuGzyqYEVe4u3MM7m2JilO3VaD70MzOFOyhxiih4=; b=OM9FCR/codFItEUw6mvVOfdv0bHt6M8SNG4twU3CmIZPt4WRXi79L+v7rNEfc6J5XD ttDzUS3XqzDKGJqdGtg43fKHnKqFBGJUNLSFKQObu6XxejEy2SO0OdhhVPF+p7rxB/HK TbElpG9EZfg+ObBJJ+rZj6HyACxxLRUv2VlxboqUjv2EG05znjfPwiuP8ybFk8ULY5+1 Tp7VVpPsyM9Bps7JqX34bbcM4pUIUNLnAHL+m461aBgvHkeLyuU1xeapBTx/NXE72nJn EFeRXa7hGE9d5dPIOM2uLhf57knfHU833y0BoFUOGXULBjaLiweDtaMB6LFptJw6cg9d hdgQ== X-Gm-Message-State: ACrzQf3Bxy84xBNWPhtP4SfDUtRjU22ymIRccbj7Mfhd04o4edenit8h YBmQIeG8mNSZWXPpMRVJQxk= X-Google-Smtp-Source: AMsMyM7B/9qL6oAIIakt1io7o3eCWAG3+YjeVZ3tr1vr4iEGyFi/lXYMpRIZlTT4pJgdq9sWOsCf8w== X-Received: by 2002:a05:6870:960b:b0:132:e9d6:ea39 with SMTP id d11-20020a056870960b00b00132e9d6ea39mr617905oaq.257.1666067791690; Mon, 17 Oct 2022 21:36:31 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-290b-8972-ce76-602c.res6.spectrum.com. [2603:8081:140c:1a00:290b:8972:ce76:602c]) by smtp.googlemail.com with ESMTPSA id e96-20020a9d01e9000000b006618ca5caa0sm5480333ote.78.2022.10.17.21.36.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Oct 2022 21:36:31 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, matsuda-daisuke@fujitsu.com, lizhijian@fujitsu.com, leon@kernel.org, linux-rdma@vger.kernel.org, jenny.hack@hpe.com, ian.ziemba@hpe.com Cc: Bob Pearson Subject: [PATCH for-next 15/16] RDMA/rxe: Add workqueue support for tasks Date: Mon, 17 Oct 2022 23:33:46 -0500 Message-Id: <20221018043345.4033-16-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221018043345.4033-1-rpearsonhpe@gmail.com> References: <20221018043345.4033-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add a third task type RXE_TASK_TYPE_WORKQUEUE to rxe_task.c. Signed-off-by: Ian Ziemba Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe.c | 9 ++- drivers/infiniband/sw/rxe/rxe_task.c | 84 ++++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_task.h | 10 +++- 3 files changed, 101 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c index 51daac5c4feb..6d80218334ca 100644 --- a/drivers/infiniband/sw/rxe/rxe.c +++ b/drivers/infiniband/sw/rxe/rxe.c @@ -210,10 +210,16 @@ static int __init rxe_module_init(void) { int err; - err = rxe_net_init(); + err = rxe_alloc_wq(); if (err) return err; + err = rxe_net_init(); + if (err) { + rxe_destroy_wq(); + return err; + } + rdma_link_register(&rxe_link_ops); pr_info("loaded\n"); return 0; @@ -224,6 +230,7 @@ static void __exit rxe_module_exit(void) rdma_link_unregister(&rxe_link_ops); ib_unregister_driver(RDMA_DRIVER_RXE); rxe_net_exit(); + rxe_destroy_wq(); pr_info("unloaded\n"); } diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index a2c58ce66c8f..ea33ea3bc0b1 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -10,6 +10,25 @@ #include "rxe.h" +static struct workqueue_struct *rxe_wq; + +int rxe_alloc_wq(void) +{ + rxe_wq = alloc_workqueue("rxe_wq", WQ_MEM_RECLAIM | + WQ_HIGHPRI | WQ_CPU_INTENSIVE | + WQ_SYSFS, WQ_MAX_ACTIVE); + + if (!rxe_wq) + return -ENOMEM; + + return 0; +} + +void rxe_destroy_wq(void) +{ + destroy_workqueue(rxe_wq); +} + static bool task_is_idle(struct rxe_task *task) { if (!task->valid) @@ -216,6 +235,68 @@ static void tsklet_init(struct rxe_task *task) task->ops = &tsklet_ops; } +static void work_sched(struct rxe_task *task) +{ + if (!task->valid) + return; + + queue_work(rxe_wq, &task->work); +} + +static void work_do_task(struct work_struct *work) +{ + struct rxe_task *task = container_of(work, typeof(*task), work); + + if (!task->valid) + return; + + do_task(task); +} + +static void work_run(struct rxe_task *task) +{ + if (!task->valid) + return; + + do_task(task); +} + +static void work_enable(struct rxe_task *task) +{ + if (!task->valid) + return; + + enable_task(task); +} + +static void work_disable(struct rxe_task *task) +{ + if (!task->valid) + return; + + disable_task(task); + flush_workqueue(rxe_wq); +} + +static void work_cleanup(struct rxe_task *task) +{ + cleanup_task(task); +} + +static const struct rxe_task_ops work_ops = { + .sched = work_sched, + .run = work_run, + .enable = work_enable, + .disable = work_disable, + .cleanup = work_cleanup, +}; + +static void work_init(struct rxe_task *task) +{ + INIT_WORK(&task->work, work_do_task); + task->ops = &work_ops; +} + int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *), enum rxe_task_type type) { @@ -233,6 +314,9 @@ int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *), case RXE_TASK_TYPE_TASKLET: tsklet_init(task); break; + case RXE_TASK_TYPE_WORKQUEUE: + work_init(task); + break; default: pr_debug("%s: invalid task type = %d\n", __func__, type); return -EINVAL; diff --git a/drivers/infiniband/sw/rxe/rxe_task.h b/drivers/infiniband/sw/rxe/rxe_task.h index c81947e88629..4887ca566769 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.h +++ b/drivers/infiniband/sw/rxe/rxe_task.h @@ -20,6 +20,7 @@ struct rxe_task_ops { enum rxe_task_type { RXE_TASK_TYPE_INLINE = 0, RXE_TASK_TYPE_TASKLET = 1, + RXE_TASK_TYPE_WORKQUEUE = 2, }; enum { @@ -35,7 +36,10 @@ enum { * called again. */ struct rxe_task { - struct tasklet_struct tasklet; + union { + struct tasklet_struct tasklet; + struct work_struct work; + }; int state; spinlock_t lock; void *arg; @@ -46,6 +50,10 @@ struct rxe_task { enum rxe_task_type type; }; +int rxe_alloc_wq(void); + +void rxe_destroy_wq(void); + int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *), enum rxe_task_type type); From patchwork Tue Oct 18 04:33:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13009889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D20FC4332F for ; Tue, 18 Oct 2022 04:36:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230104AbiJREgo (ORCPT ); Tue, 18 Oct 2022 00:36:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230203AbiJREgf (ORCPT ); Tue, 18 Oct 2022 00:36:35 -0400 Received: from mail-oi1-x22d.google.com (mail-oi1-x22d.google.com [IPv6:2607:f8b0:4864:20::22d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 73A77A0265 for ; Mon, 17 Oct 2022 21:36:33 -0700 (PDT) Received: by mail-oi1-x22d.google.com with SMTP id n83so14389821oif.11 for ; Mon, 17 Oct 2022 21:36:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Kr6A3rEE+c3RZe1NR4tKdOx5mJcIGy59rKaiFSlw9bQ=; b=kFZzU5HGxVF4SJ255RlThKej+w6gEYO4mJ8ohsChuxyluCrdhrbyGVrl22jjvtJ/Wy AfA4Cy9IBW/IduftZmNnSq9WVwylWDaLdd6fIbgfxd0s4gX1Xt5nXxEF97evFlTYwdc+ CFlTL+CMJrGJ2ER4J1FpPHvCrBAVjPhtVno84js5BoJ87DWt9vzTFzO1m1+skWmC8zhd yvzie5Ukn/khA+MZUhoHVJVRbsHk44atH+p8hT85aLAw9xT55Rp30ygNIeczHrAnaTGR UBXggapp6WKlttrebjrsVTMnUv67MWJJ0elmir/6J0XaBVV+TOfcMBtl8zRoj53ClIA6 Xf/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Kr6A3rEE+c3RZe1NR4tKdOx5mJcIGy59rKaiFSlw9bQ=; b=wDEzQwaMPkIO5LyfhDPECZcxDb+RUdMYMjaH11DRMWGU4mAK1ffF3mFV6h+zmm09FK dusd9xzUg+ZHioTSGzqDbeF4cOlXGuwZ0meqAuJpArijhiwIJjpbg9UXnjnR2C9Lj1Jg IYJUsV2waWXaDP2wQaqFhkmuKsZGpLH3M86Dh7K+pAdvI5ATGpRdTBM2pJAlhMGfFKj8 lkDaosf0exU1k9qduT8nwO9ZudsKNhHmxYJFeaXG+5UACZ0COnUAQsibVmrDtkpP5oyM qH+ee3/C5XgRoFsNZwgrRgn6hJQnokX4/xHihSN59Rh6xjDg+E5EJeFzP35/AiZnXqL3 bQ2g== X-Gm-Message-State: ACrzQf1TuaMMZX4R7PkCGJxkELeGR1CdW/CXBC7BzOskhhM2Lp/3rzpu qh5novYNA4QLvZMms0+HPdzlTFOFENUD2g== X-Google-Smtp-Source: AMsMyM5ff7oZTHUzC2Elu+pIsqiJhvxPXjYdZKzYPlt1pTToEpwRy3tmdgpvTk55l8Fm6S+E3eNvLQ== X-Received: by 2002:aca:f1c5:0:b0:355:3d98:8c09 with SMTP id p188-20020acaf1c5000000b003553d988c09mr3154774oih.203.1666067792803; Mon, 17 Oct 2022 21:36:32 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-290b-8972-ce76-602c.res6.spectrum.com. [2603:8081:140c:1a00:290b:8972:ce76:602c]) by smtp.googlemail.com with ESMTPSA id e96-20020a9d01e9000000b006618ca5caa0sm5480333ote.78.2022.10.17.21.36.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Oct 2022 21:36:32 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, matsuda-daisuke@fujitsu.com, lizhijian@fujitsu.com, leon@kernel.org, linux-rdma@vger.kernel.org, jenny.hack@hpe.com, ian.ziemba@hpe.com Cc: Bob Pearson Subject: [PATCH for-next 16/16] RDMA/rxe: Add parameters to control task type Date: Mon, 17 Oct 2022 23:33:47 -0500 Message-Id: <20221018043345.4033-17-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221018043345.4033-1-rpearsonhpe@gmail.com> References: <20221018043345.4033-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add modparams to control the task types for req, comp, and resp tasks. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_qp.c | 6 +++--- drivers/infiniband/sw/rxe/rxe_task.c | 8 ++++++++ drivers/infiniband/sw/rxe/rxe_task.h | 4 ++++ 3 files changed, 15 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index 50f6b8b8ad9d..673d52271062 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -238,9 +238,9 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp, skb_queue_head_init(&qp->req_pkts); - rxe_init_task(&qp->req.task, qp, rxe_requester, RXE_TASK_TYPE_TASKLET); + rxe_init_task(&qp->req.task, qp, rxe_requester, rxe_req_task_type); rxe_init_task(&qp->comp.task, qp, rxe_completer, - (qp_type(qp) == IB_QPT_RC) ? RXE_TASK_TYPE_TASKLET : + (qp_type(qp) == IB_QPT_RC) ? rxe_comp_task_type : RXE_TASK_TYPE_INLINE); qp->qp_timeout_jiffies = 0; /* Can't be set for UD/UC in modify_qp */ @@ -288,7 +288,7 @@ static int rxe_qp_init_resp(struct rxe_dev *rxe, struct rxe_qp *qp, skb_queue_head_init(&qp->resp_pkts); - rxe_init_task(&qp->resp.task, qp, rxe_responder, RXE_TASK_TYPE_TASKLET); + rxe_init_task(&qp->resp.task, qp, rxe_responder, rxe_resp_task_type); qp->resp.opcode = OPCODE_NONE; qp->resp.msn = 0; diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index ea33ea3bc0b1..350b033e6e59 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -10,6 +10,14 @@ #include "rxe.h" +int rxe_req_task_type = RXE_TASK_TYPE_TASKLET; +int rxe_comp_task_type = RXE_TASK_TYPE_TASKLET; +int rxe_resp_task_type = RXE_TASK_TYPE_TASKLET; + +module_param_named(req_task_type, rxe_req_task_type, int, 0664); +module_param_named(comp_task_type, rxe_comp_task_type, int, 0664); +module_param_named(resp_task_type, rxe_resp_task_type, int, 0664); + static struct workqueue_struct *rxe_wq; int rxe_alloc_wq(void) diff --git a/drivers/infiniband/sw/rxe/rxe_task.h b/drivers/infiniband/sw/rxe/rxe_task.h index 4887ca566769..4c37b7c47a60 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.h +++ b/drivers/infiniband/sw/rxe/rxe_task.h @@ -7,6 +7,10 @@ #ifndef RXE_TASK_H #define RXE_TASK_H +extern int rxe_req_task_type; +extern int rxe_comp_task_type; +extern int rxe_resp_task_type; + struct rxe_task; struct rxe_task_ops {