From patchwork Sat Oct 29 03:09:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13024484 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5184FA3743 for ; Sat, 29 Oct 2022 03:10:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229916AbiJ2DKt (ORCPT ); Fri, 28 Oct 2022 23:10:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33602 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229926AbiJ2DKp (ORCPT ); Fri, 28 Oct 2022 23:10:45 -0400 Received: from mail-ot1-x334.google.com (mail-ot1-x334.google.com [IPv6:2607:f8b0:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31BEB2A40E for ; Fri, 28 Oct 2022 20:10:19 -0700 (PDT) Received: by mail-ot1-x334.google.com with SMTP id m5-20020a9d73c5000000b0066738ce4f12so4003779otk.12 for ; Fri, 28 Oct 2022 20:10:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EU9/rBYN2hXS/v49sgDZV0QWnssuEz3K4CW9gz24luE=; b=Vp86E2FRAefpgsOo9rPP5iHZquewIBiuG0TlFre5tMozafwOmpeb3KG1hEv/rwiJc1 ZP4+zbehdytLt8+TloQBTgUM7VzeyRcMsi+1TrRa0WvD3bm1tMiL3tWH8ObYyhYgCEDI JYTzypZkeGUCalFdULgQZGmP+C+3A9UD/lJMlIliwD3Bz8BUQBPhgRkI5qTU19wV4kvA OV4EeGKzzD8WSpVsjBc1otDfl2xWF5luwyuAEEYk1s/jMnK/vO4zkpFU8QNTjP7NlmNu BjBnfX/X/QcHJFMwHQNyQzlERNeFESX9dCDv+b32uG1zA3gldAaI8/fdNt78aw6XWmod cKIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EU9/rBYN2hXS/v49sgDZV0QWnssuEz3K4CW9gz24luE=; b=NbY64xGDtWHd9OqpCJ6DewJt8Z/zx+gRZoo85BCsgVZdDGub8fdN4vb14VQsaVou2h DZuOeJaB18tLxn91L/fQL2t338TjGFvYoGFONXNCSZhQYugoAnQXhlf7rDx/eZU6EQ9K /5F7tVr5OAjzEBILTOHhzx04fx4/Eu9s/ytH2AxNs5OmF6Uop0Lt92LQoegX7G5XKPik biEkV3vlc6fj1fkrHYAoQLfZz6OTROyvInU3xiVrDvC89RR47F8mvXgaUI5tIdAP+iUb MzR262ymAWTNvirY+QGJSmPBMUON5seYqVdGGFaYJcpDEV9PUwMQI5ENMmRzvPXUG84S IIQA== X-Gm-Message-State: ACrzQf3i5ZxsMUomUzKtzcVXn7Xj4WptCr5Dz9FK/1T8PBuVYW6nbwTF q9ARz+IRdxXN/qUAPg+BavI= X-Google-Smtp-Source: AMsMyM79ZhR9iZGQYymXFfEAOiKP+hl/hBIFcNF/VNCczcc5PsOT2T46R+g3mrFXIgXgV2lL2ietzQ== X-Received: by 2002:a05:6830:4486:b0:661:9f14:9b45 with SMTP id r6-20020a056830448600b006619f149b45mr1188221otv.227.1667013018958; Fri, 28 Oct 2022 20:10:18 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-5d1e-45f3-e9b5-d771.res6.spectrum.com. [2603:8081:140c:1a00:5d1e:45f3:e9b5:d771]) by smtp.googlemail.com with ESMTPSA id p3-20020a0568301d4300b006391adb6034sm162493oth.72.2022.10.28.20.10.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 20:10:18 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, leon@kernel.org, zyjzyj2000@gmail.com, jhack@hpe.com, linux-rdma@vger.kernel.org Cc: Bob Pearson , Ian Ziemba Subject: [PATCH for-next v3 01/13] RDMA/rxe: Make task interface pluggable Date: Fri, 28 Oct 2022 22:09:58 -0500 Message-Id: <20221029031009.64467-2-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221029031009.64467-1-rpearsonhpe@gmail.com> References: <20221029031009.64467-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Make the internal interface to the task operations pluggable and add a new 'inline' type. Signed-off-by: Ian Ziemba Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_qp.c | 8 +- drivers/infiniband/sw/rxe/rxe_task.c | 160 ++++++++++++++++++++++----- drivers/infiniband/sw/rxe/rxe_task.h | 44 +++++--- 3 files changed, 165 insertions(+), 47 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index 3f6d62a80bea..b5e108794aa1 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -238,8 +238,10 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp, skb_queue_head_init(&qp->req_pkts); - rxe_init_task(&qp->req.task, qp, rxe_requester); - rxe_init_task(&qp->comp.task, qp, rxe_completer); + rxe_init_task(&qp->req.task, qp, rxe_requester, RXE_TASK_TYPE_TASKLET); + rxe_init_task(&qp->comp.task, qp, rxe_completer, + (qp_type(qp) == IB_QPT_RC) ? RXE_TASK_TYPE_TASKLET : + RXE_TASK_TYPE_INLINE); qp->qp_timeout_jiffies = 0; /* Can't be set for UD/UC in modify_qp */ if (init->qp_type == IB_QPT_RC) { @@ -286,7 +288,7 @@ static int rxe_qp_init_resp(struct rxe_dev *rxe, struct rxe_qp *qp, skb_queue_head_init(&qp->resp_pkts); - rxe_init_task(&qp->resp.task, qp, rxe_responder); + rxe_init_task(&qp->resp.task, qp, rxe_responder, RXE_TASK_TYPE_TASKLET); qp->resp.opcode = OPCODE_NONE; qp->resp.msn = 0; diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index 0208d833a41b..8dfbfa164eff 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -24,12 +24,11 @@ int __rxe_do_task(struct rxe_task *task) * a second caller finds the task already running * but looks just after the last call to func */ -static void do_task(struct tasklet_struct *t) +static void do_task(struct rxe_task *task) { + unsigned int iterations = RXE_MAX_ITERATIONS; int cont; int ret; - struct rxe_task *task = from_tasklet(task, t, tasklet); - unsigned int iterations = RXE_MAX_ITERATIONS; spin_lock_bh(&task->lock); switch (task->state) { @@ -90,28 +89,21 @@ static void do_task(struct tasklet_struct *t) task->ret = ret; } -int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *)) +static void disable_task(struct rxe_task *task) { - task->arg = arg; - task->func = func; - task->destroyed = false; - - tasklet_setup(&task->tasklet, do_task); - - task->state = TASK_STATE_START; - spin_lock_init(&task->lock); + /* todo */ +} - return 0; +static void enable_task(struct rxe_task *task) +{ + /* todo */ } -void rxe_cleanup_task(struct rxe_task *task) +/* busy wait until any previous tasks are done */ +static void cleanup_task(struct rxe_task *task) { bool idle; - /* - * Mark the task, then wait for it to finish. It might be - * running in a non-tasklet (direct call) context. - */ task->destroyed = true; do { @@ -119,32 +111,144 @@ void rxe_cleanup_task(struct rxe_task *task) idle = (task->state == TASK_STATE_START); spin_unlock_bh(&task->lock); } while (!idle); +} - tasklet_kill(&task->tasklet); +/* silently treat schedule as inline for inline tasks */ +static void inline_sched(struct rxe_task *task) +{ + do_task(task); } -void rxe_run_task(struct rxe_task *task) +static void inline_run(struct rxe_task *task) { - if (task->destroyed) - return; + do_task(task); +} - do_task(&task->tasklet); +static void inline_disable(struct rxe_task *task) +{ + disable_task(task); } -void rxe_sched_task(struct rxe_task *task) +static void inline_enable(struct rxe_task *task) { - if (task->destroyed) - return; + enable_task(task); +} + +static void inline_cleanup(struct rxe_task *task) +{ + cleanup_task(task); +} + +static const struct rxe_task_ops inline_ops = { + .sched = inline_sched, + .run = inline_run, + .enable = inline_enable, + .disable = inline_disable, + .cleanup = inline_cleanup, +}; +static void inline_init(struct rxe_task *task) +{ + task->ops = &inline_ops; +} + +/* use tsklet_xxx to avoid name collisions with tasklet_xxx */ +static void tsklet_sched(struct rxe_task *task) +{ tasklet_schedule(&task->tasklet); } -void rxe_disable_task(struct rxe_task *task) +static void tsklet_do_task(struct tasklet_struct *tasklet) { + struct rxe_task *task = container_of(tasklet, typeof(*task), tasklet); + + do_task(task); +} + +static void tsklet_run(struct rxe_task *task) +{ + do_task(task); +} + +static void tsklet_disable(struct rxe_task *task) +{ + disable_task(task); tasklet_disable(&task->tasklet); } -void rxe_enable_task(struct rxe_task *task) +static void tsklet_enable(struct rxe_task *task) { tasklet_enable(&task->tasklet); + enable_task(task); +} + +static void tsklet_cleanup(struct rxe_task *task) +{ + cleanup_task(task); + tasklet_kill(&task->tasklet); +} + +static const struct rxe_task_ops tsklet_ops = { + .sched = tsklet_sched, + .run = tsklet_run, + .enable = tsklet_enable, + .disable = tsklet_disable, + .cleanup = tsklet_cleanup, +}; + +static void tsklet_init(struct rxe_task *task) +{ + tasklet_setup(&task->tasklet, tsklet_do_task); + task->ops = &tsklet_ops; +} + +int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *), + enum rxe_task_type type) +{ + task->arg = arg; + task->func = func; + task->destroyed = false; + task->type = type; + task->state = TASK_STATE_START; + + spin_lock_init(&task->lock); + + switch (type) { + case RXE_TASK_TYPE_INLINE: + inline_init(task); + break; + case RXE_TASK_TYPE_TASKLET: + tsklet_init(task); + break; + default: + pr_debug("%s: invalid task type = %d\n", __func__, type); + return -EINVAL; + } + + return 0; +} + +void rxe_sched_task(struct rxe_task *task) +{ + task->ops->sched(task); +} + +void rxe_run_task(struct rxe_task *task) +{ + task->ops->run(task); +} + +void rxe_enable_task(struct rxe_task *task) +{ + task->ops->enable(task); +} + +void rxe_disable_task(struct rxe_task *task) +{ + task->ops->disable(task); +} + +void rxe_cleanup_task(struct rxe_task *task) +{ + task->ops->cleanup(task); } diff --git a/drivers/infiniband/sw/rxe/rxe_task.h b/drivers/infiniband/sw/rxe/rxe_task.h index 7b88129702ac..31963129ff7a 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.h +++ b/drivers/infiniband/sw/rxe/rxe_task.h @@ -7,6 +7,21 @@ #ifndef RXE_TASK_H #define RXE_TASK_H +struct rxe_task; + +struct rxe_task_ops { + void (*sched)(struct rxe_task *task); + void (*run)(struct rxe_task *task); + void (*disable)(struct rxe_task *task); + void (*enable)(struct rxe_task *task); + void (*cleanup)(struct rxe_task *task); +}; + +enum rxe_task_type { + RXE_TASK_TYPE_INLINE = 0, + RXE_TASK_TYPE_TASKLET = 1, +}; + enum { TASK_STATE_START = 0, TASK_STATE_BUSY = 1, @@ -19,24 +34,19 @@ enum { * called again. */ struct rxe_task { - struct tasklet_struct tasklet; - int state; - spinlock_t lock; - void *arg; - int (*func)(void *arg); - int ret; - bool destroyed; + struct tasklet_struct tasklet; + int state; + spinlock_t lock; + void *arg; + int (*func)(void *arg); + int ret; + bool destroyed; + const struct rxe_task_ops *ops; + enum rxe_task_type type; }; -/* - * init rxe_task structure - * arg => parameter to pass to fcn - * func => function to call until it returns != 0 - */ -int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *)); - -/* cleanup task */ -void rxe_cleanup_task(struct rxe_task *task); +int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *), + enum rxe_task_type type); /* * raw call to func in loop without any checking @@ -54,4 +64,6 @@ void rxe_disable_task(struct rxe_task *task); /* allow task to run */ void rxe_enable_task(struct rxe_task *task); +void rxe_cleanup_task(struct rxe_task *task); + #endif /* RXE_TASK_H */ From patchwork Sat Oct 29 03:09:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13024485 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19E2FFA3741 for ; Sat, 29 Oct 2022 03:10:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229918AbiJ2DKu (ORCPT ); Fri, 28 Oct 2022 23:10:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33600 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229928AbiJ2DKp (ORCPT ); Fri, 28 Oct 2022 23:10:45 -0400 Received: from mail-oo1-xc31.google.com (mail-oo1-xc31.google.com [IPv6:2607:f8b0:4864:20::c31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3188329CA4 for ; Fri, 28 Oct 2022 20:10:20 -0700 (PDT) Received: by mail-oo1-xc31.google.com with SMTP id g15-20020a4a894f000000b0047f8e899623so1014720ooi.5 for ; Fri, 28 Oct 2022 20:10:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KxVc8Xs0oLXoDtXiq0E1glHusFjw0NUoNGpkz6t1r38=; b=bb1bZmn3KXcIPFKxL9i3CBl+a256ZqgrrrEn/Mmi78nIh8j0b/U4HiMzk6k4zDRdw9 N7np7qid2oMCGGSMsjGrMebT9bPQ6POM/keXyvhvQn+7JXrb9QyXiL/0lkY7ZR+x5guM s7IvMyQ8J9dm5ooQCQOg44nSvMlix9zjMqbKH51z6RDBBb/Q9SH30QTnGErOgW+CkDxr nGhlj/0/kwQHQ5fZ+qwK7JnUAyiLwxBMhI1VgpAsbJ7bbnqn74HzzZMF+TWkleTbLUGK obHVwZ7sfRxvwMPdXPa45MB3/oa2FX7H/uef6hkfEjkLCPqNFm075uNe80V+yt8SVpPC prcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KxVc8Xs0oLXoDtXiq0E1glHusFjw0NUoNGpkz6t1r38=; b=kJvFhgb9V6GZC0y69geUeGsZZUGlr1Z/VD7qnAAN1vvOmT/LH0XZ0Aa+buttk4EwZo KcNT+y0myXvrgw7r9u+obIHl8dnATQ3a4jAlcbwnMeNPjDwi5wwAhzeFHU0qdswX1tUk 7S5xmNTzjpGp0KL2HqSr9sUSbOpv0Vsh1L78SqhGwGDQ/HdKkzZ5qmE5TxN21b3EcaTI lSTqdHLKEeOXRl55WUsaqR0owVMBgg9vaevdRMC9vr50Asrl15YKdSKCh+p92yCRcBgA bdaQCQw4DJJ+cTdkGHYmKNRcf1siGDPihZj0NmtrZn6Lqs/X4pVphF0KDU8z7ofmWXDh LkWQ== X-Gm-Message-State: ACrzQf2Z6iE8XaABe43+VMTTlQUwGE6bJiWNkHOegGV4j1niiBJOTedn wx+Gb3m0MTGEWo30+fK+d/k= X-Google-Smtp-Source: AMsMyM5KHNdHti1EdQkjfwfGRQBfKq2WyJ14kQCL3xqUwavkfTa5hGyzEXcOjjF6AfroLsWzjs2lJg== X-Received: by 2002:a4a:8e81:0:b0:475:811f:3f9e with SMTP id p1-20020a4a8e81000000b00475811f3f9emr1082867ook.35.1667013020042; Fri, 28 Oct 2022 20:10:20 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-5d1e-45f3-e9b5-d771.res6.spectrum.com. [2603:8081:140c:1a00:5d1e:45f3:e9b5:d771]) by smtp.googlemail.com with ESMTPSA id p3-20020a0568301d4300b006391adb6034sm162493oth.72.2022.10.28.20.10.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 20:10:19 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, leon@kernel.org, zyjzyj2000@gmail.com, jhack@hpe.com, linux-rdma@vger.kernel.org Cc: Bob Pearson , Ian Ziemba Subject: [PATCH for-next v3 02/13] RDMA/rxe: Split rxe_drain_resp_pkts() Date: Fri, 28 Oct 2022 22:09:59 -0500 Message-Id: <20221029031009.64467-3-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221029031009.64467-1-rpearsonhpe@gmail.com> References: <20221029031009.64467-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Split rxe_drain_resp_pkts() into two subroutines which perform separate functions. Signed-off-by: Ian Ziemba Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_comp.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c index 66f392810c86..76dc0a4702fd 100644 --- a/drivers/infiniband/sw/rxe/rxe_comp.c +++ b/drivers/infiniband/sw/rxe/rxe_comp.c @@ -532,17 +532,21 @@ static inline enum comp_state complete_wqe(struct rxe_qp *qp, return COMPST_GET_WQE; } -static void rxe_drain_resp_pkts(struct rxe_qp *qp, bool notify) +static void rxe_drain_resp_pkts(struct rxe_qp *qp) { struct sk_buff *skb; - struct rxe_send_wqe *wqe; - struct rxe_queue *q = qp->sq.queue; while ((skb = skb_dequeue(&qp->resp_pkts))) { rxe_put(qp); kfree_skb(skb); ib_device_put(qp->ibqp.device); } +} + +static void rxe_drain_send_queue(struct rxe_qp *qp, bool notify) +{ + struct rxe_send_wqe *wqe; + struct rxe_queue *q = qp->sq.queue; while ((wqe = queue_head(q, q->type))) { if (notify) { @@ -573,6 +577,7 @@ int rxe_completer(void *arg) struct sk_buff *skb = NULL; struct rxe_pkt_info *pkt = NULL; enum comp_state state; + bool notify; int ret; if (!rxe_get(qp)) @@ -580,8 +585,9 @@ int rxe_completer(void *arg) if (!qp->valid || qp->comp.state == QP_STATE_ERROR || qp->comp.state == QP_STATE_RESET) { - rxe_drain_resp_pkts(qp, qp->valid && - qp->comp.state == QP_STATE_ERROR); + notify = qp->valid && (qp->comp.state == QP_STATE_ERROR); + rxe_drain_resp_pkts(qp); + rxe_drain_send_queue(qp, notify); goto exit; } From patchwork Sat Oct 29 03:10:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13024490 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 884BCFA3741 for ; Sat, 29 Oct 2022 03:10:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229613AbiJ2DKy (ORCPT ); Fri, 28 Oct 2022 23:10:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33692 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229932AbiJ2DKp (ORCPT ); Fri, 28 Oct 2022 23:10:45 -0400 Received: from mail-ot1-x330.google.com (mail-ot1-x330.google.com [IPv6:2607:f8b0:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5E2722AC53 for ; Fri, 28 Oct 2022 20:10:21 -0700 (PDT) Received: by mail-ot1-x330.google.com with SMTP id v40-20020a056830092800b00661e37421c2so4023095ott.3 for ; Fri, 28 Oct 2022 20:10:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tiDP5+mzppeVixN1QhIBC8v4aiDRCLnh9K76vVm1dow=; b=JI4xsDSyclYeqRZW3zbZuOo8fg6JmdwSz9WbA+0xw5aEIuYYLiUELws0Rug7n3A9gU 8Nmb5jFyQ15qkqVSwqbG3jemKQmKn8vszRXv4bchR9V2Lo7Rrjr+jXxb0uWlA7QuIB7P tKf131m/DPzJOz09FBxlWbLaJoxaEc5WXViGg8SECOh+uL5046phXaidUL+Nvg2b7L8I u0MLJqErd9xRtGBK7Z3XfcRzgBqOEu6vITcwA0D2l/fg6yTlJHACu83nbWDCLq1Wer9Z 5nupRM9eJMtzeRn75PZqHMYq/K/05H5XAzqcF4444JTtgLFDCXvMyrJFzJv8DHIBU1c+ iypg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tiDP5+mzppeVixN1QhIBC8v4aiDRCLnh9K76vVm1dow=; b=iPxoWSORmeKHQ0uz39eCEDGIVoI0B0dqMJKAr50OFWwaj4btTVRPksdPY1zLmfjN8R QpJDFY9SuN5Lko0CzKAAlwrEdogwzO8AgIc8dRElxjLlkgcR0UvaEe/F+uFMgkcwVMt1 Z1MPlPFhZqRSlbspYkx4/tY7k/Gr/lgMAJ1k/WygPoCxqF+j3ZdPdJgg75sTAcmospUH wKIzXiFya2EA2GQX0rDAjuxV0h2BbK9cvbZY3iZB1XS7F7ZmPFmMHiY5CcnpZ4ICdQq3 qmwiU9bJTJkMOBQkO2yOmZscZ3IywGOChkgnKjIxlXjIabwKpQGDECf5tyaSuuMpsW0o KIRw== X-Gm-Message-State: ACrzQf2yw82LQqSDs7Iekz6/mb6YLtqLIxRn7I83htrpRx+kColFAT7P 5OouRrnzGvonv0bQr8nBWD4= X-Google-Smtp-Source: AMsMyM6x4piTdc19Q1c5E1T6XuYekKpuzQGhBPU2v6bBcZX+nX3VVCYEvwNY3dWA9FnGnj11WvJ0kA== X-Received: by 2002:a05:6830:3703:b0:65f:c2ff:c526 with SMTP id bl3-20020a056830370300b0065fc2ffc526mr1193128otb.302.1667013021015; Fri, 28 Oct 2022 20:10:21 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-5d1e-45f3-e9b5-d771.res6.spectrum.com. [2603:8081:140c:1a00:5d1e:45f3:e9b5:d771]) by smtp.googlemail.com with ESMTPSA id p3-20020a0568301d4300b006391adb6034sm162493oth.72.2022.10.28.20.10.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 20:10:20 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, leon@kernel.org, zyjzyj2000@gmail.com, jhack@hpe.com, linux-rdma@vger.kernel.org Cc: Bob Pearson , Ian Ziemba Subject: [PATCH for-next v3 03/13] RDMA/rxe: Simplify reset state handling in rxe_resp.c Date: Fri, 28 Oct 2022 22:10:00 -0500 Message-Id: <20221029031009.64467-4-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221029031009.64467-1-rpearsonhpe@gmail.com> References: <20221029031009.64467-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Make rxe_responder() more like rxe_completer() and take qp reset handling out of the state machine. Signed-off-by: Ian Ziemba Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_resp.c | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index c32bc12cc82f..c4f365449aa5 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -40,7 +40,6 @@ enum resp_states { RESPST_ERR_LENGTH, RESPST_ERR_CQ_OVERFLOW, RESPST_ERROR, - RESPST_RESET, RESPST_DONE, RESPST_EXIT, }; @@ -75,7 +74,6 @@ static char *resp_state_name[] = { [RESPST_ERR_LENGTH] = "ERR_LENGTH", [RESPST_ERR_CQ_OVERFLOW] = "ERR_CQ_OVERFLOW", [RESPST_ERROR] = "ERROR", - [RESPST_RESET] = "RESET", [RESPST_DONE] = "DONE", [RESPST_EXIT] = "EXIT", }; @@ -1281,8 +1279,9 @@ int rxe_responder(void *arg) switch (qp->resp.state) { case QP_STATE_RESET: - state = RESPST_RESET; - break; + rxe_drain_req_pkts(qp, false); + qp->resp.wqe = NULL; + goto exit; default: state = RESPST_GET_REQ; @@ -1441,11 +1440,6 @@ int rxe_responder(void *arg) goto exit; - case RESPST_RESET: - rxe_drain_req_pkts(qp, false); - qp->resp.wqe = NULL; - goto exit; - case RESPST_ERROR: qp->resp.goto_error = 0; pr_debug("qp#%d moved to error state\n", qp_num(qp)); From patchwork Sat Oct 29 03:10:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13024481 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BE25FA3741 for ; Sat, 29 Oct 2022 03:10:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229597AbiJ2DKs (ORCPT ); Fri, 28 Oct 2022 23:10:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33514 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229923AbiJ2DKn (ORCPT ); Fri, 28 Oct 2022 23:10:43 -0400 Received: from mail-oa1-x31.google.com (mail-oa1-x31.google.com [IPv6:2001:4860:4864:20::31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3171929818 for ; Fri, 28 Oct 2022 20:10:22 -0700 (PDT) Received: by mail-oa1-x31.google.com with SMTP id 586e51a60fabf-13ae8117023so8327945fac.9 for ; Fri, 28 Oct 2022 20:10:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3BkUIhKPeo9V7NcaZnYbiCv3hvi8SppsEDDvvMCB51U=; b=WQNZdsUj23WRZneckh8QEPvzmubw37hTFeDmbx+z7MGE45YmllCx4L3xg/ihV9ZCev r0bwhNiCqq+Xkjk0fl502DD3Wm6hM5QVbTb+26ZMFpo9k1pSAXRy41pLiZpW7JBK09ox eWzlFpck7/lp53KhLs0SP/eoxwjE8TBwwPgk2Yc8rcEoYn7G1UD2gzvo7FMi+VX3udy3 kOB1uzlwfgTrZiTO3c4ImE8M5+SeySLNd5vtBYn4CQXOSGtUhX38h0udnt+pr7auzR97 mwyE4Cs6LF+c6K6LqTG7EDiPjhQb73v4pLirmLWpDhAXT6U8+pZ98jYvNKs9+VG3E8s7 Nr1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3BkUIhKPeo9V7NcaZnYbiCv3hvi8SppsEDDvvMCB51U=; b=dp6jAeKG4JQgN0rB3Sh41qfcUUl4B1ski5C80qcgkyTSyG3k0MDM+noY5gqOSyqNTU wGAPTZXMKUm7HV5vl5qpMEU7fR40ZqFVNlc8q5u6gzVbVW2gBNmVFYE9qdWUQsxL7Rny 0ySt6t6fKZPfu8hsrJRoBi4fSjlv6rKtVxssEAndjYNq8zjhjzmpwJ1Z6YAhFPKLHgvU wnt7q1mi6wvE221n0Fe0fPpBAAw+cm6pz/Wn7ykSYzDgQWnuba2cRWmWZRZ6W9AYweUx ElZLRZm+UVHXalQQnyyORSVSDe15SZU4gft/mrhPVHKAna1LoofcBHinTXEuhlxxCCLX sphg== X-Gm-Message-State: ACrzQf0lCslnKmJWNm+0bVASXPRHFbivU3tKWdVg18wYp6doA8YxmVXt qEPWYP34LAE9YOcGvMdbsl4= X-Google-Smtp-Source: AMsMyM5BS/zP58FO2zCwkTOVXD3MXZ8P+y3y4lXbxuVzPG+frWVEj40FRWMZEX0kQ/iObaAOVOtwrA== X-Received: by 2002:a05:6870:14c5:b0:13c:2432:6b0 with SMTP id l5-20020a05687014c500b0013c243206b0mr1294626oab.185.1667013021986; Fri, 28 Oct 2022 20:10:21 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-5d1e-45f3-e9b5-d771.res6.spectrum.com. [2603:8081:140c:1a00:5d1e:45f3:e9b5:d771]) by smtp.googlemail.com with ESMTPSA id p3-20020a0568301d4300b006391adb6034sm162493oth.72.2022.10.28.20.10.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 20:10:21 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, leon@kernel.org, zyjzyj2000@gmail.com, jhack@hpe.com, linux-rdma@vger.kernel.org Cc: Bob Pearson , Ian Ziemba Subject: [PATCH for-next v3 04/13] RDMA/rxe: Handle qp error in rxe_resp.c Date: Fri, 28 Oct 2022 22:10:01 -0500 Message-Id: <20221029031009.64467-5-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221029031009.64467-1-rpearsonhpe@gmail.com> References: <20221029031009.64467-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Split rxe_drain_req_pkts() into two subroutines which perform separate tasks. Change qp error and reset states and !qp->valid in the same way as rxe_comp.c. Flush recv wqes for qp in error. Signed-off-by: Ian Ziemba Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_resp.c | 64 ++++++++++++++++++++-------- 1 file changed, 47 insertions(+), 17 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index c4f365449aa5..16298d88a9d7 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -1028,7 +1028,6 @@ static enum resp_states do_complete(struct rxe_qp *qp, return RESPST_CLEANUP; } - static int send_common_ack(struct rxe_qp *qp, u8 syndrome, u32 psn, int opcode, const char *msg) { @@ -1243,22 +1242,56 @@ static enum resp_states do_class_d1e_error(struct rxe_qp *qp) } } -static void rxe_drain_req_pkts(struct rxe_qp *qp, bool notify) +static void rxe_drain_req_pkts(struct rxe_qp *qp) { struct sk_buff *skb; - struct rxe_queue *q = qp->rq.queue; while ((skb = skb_dequeue(&qp->req_pkts))) { rxe_put(qp); kfree_skb(skb); ib_device_put(qp->ibqp.device); } +} + +static int complete_flush(struct rxe_qp *qp, struct rxe_recv_wqe *wqe) +{ + struct rxe_cqe cqe; + struct ib_wc *wc = &cqe.ibwc; + struct ib_uverbs_wc *uwc = &cqe.uibwc; + + memset(&cqe, 0, sizeof(cqe)); - if (notify) - return; + if (qp->rcq->is_user) { + uwc->status = IB_WC_WR_FLUSH_ERR; + uwc->qp_num = qp->ibqp.qp_num; + uwc->wr_id = wqe->wr_id; + } else { + wc->status = IB_WC_WR_FLUSH_ERR; + wc->qp = &qp->ibqp; + wc->wr_id = wqe->wr_id; + } - while (!qp->srq && q && queue_head(q, q->type)) + if (rxe_cq_post(qp->rcq, &cqe, 0)) + return -ENOMEM; + + return 0; +} + +/* drain the receive queue. Complete each wqe with a flush error + * if notify is true or until a cq overflow occurs. + */ +static void rxe_drain_recv_queue(struct rxe_qp *qp, bool notify) +{ + struct rxe_recv_wqe *wqe; + struct rxe_queue *q = qp->rq.queue; + + while ((wqe = queue_head(q, q->type))) { + if (notify && complete_flush(qp, wqe)) + notify = 0; queue_advance_consumer(q, q->type); + } + + qp->resp.wqe = NULL; } int rxe_responder(void *arg) @@ -1267,6 +1300,7 @@ int rxe_responder(void *arg) struct rxe_dev *rxe = to_rdev(qp->ibqp.device); enum resp_states state; struct rxe_pkt_info *pkt = NULL; + bool notify; int ret; if (!rxe_get(qp)) @@ -1274,20 +1308,16 @@ int rxe_responder(void *arg) qp->resp.aeth_syndrome = AETH_ACK_UNLIMITED; - if (!qp->valid) - goto exit; - - switch (qp->resp.state) { - case QP_STATE_RESET: - rxe_drain_req_pkts(qp, false); - qp->resp.wqe = NULL; + if (!qp->valid || qp->resp.state == QP_STATE_ERROR || + qp->resp.state == QP_STATE_RESET) { + notify = qp->valid && (qp->resp.state == QP_STATE_ERROR); + rxe_drain_req_pkts(qp); + rxe_drain_recv_queue(qp, notify); goto exit; - - default: - state = RESPST_GET_REQ; - break; } + state = RESPST_GET_REQ; + while (1) { pr_debug("qp#%d state = %s\n", qp_num(qp), resp_state_name[state]); From patchwork Sat Oct 29 03:10:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13024483 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71756FA3742 for ; Sat, 29 Oct 2022 03:10:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229456AbiJ2DKu (ORCPT ); Fri, 28 Oct 2022 23:10:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33280 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229930AbiJ2DKp (ORCPT ); Fri, 28 Oct 2022 23:10:45 -0400 Received: from mail-oa1-x2c.google.com (mail-oa1-x2c.google.com [IPv6:2001:4860:4864:20::2c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31A732A403 for ; Fri, 28 Oct 2022 20:10:24 -0700 (PDT) Received: by mail-oa1-x2c.google.com with SMTP id 586e51a60fabf-13bd2aea61bso8411540fac.0 for ; Fri, 28 Oct 2022 20:10:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jywcoq6WfpSz2CinBO3sC4x3ilzoShyeAxGkJ7NOWiY=; b=K8dNoJB9WCoiwr7HKUpVI2zBB9bFC/sGcQJA6obJ0LhCQTWQdkRPnLsXlZ3ay7evW6 Ht5v00+/kxAUfM/f5bo4fuy55Gc5ArpeCDYzT6ZGhIdEgT8eSo8K0NxJR7Y72LJ2rvPa xGrn4tOIT1t5+veyRyMdTJjgmgMGCf7CWEHutzQQxUM9T60CcRPUtUXv1WeuqwDNB/tq mB7C+NNORdY2rimNzacJ0wsZik/fXlk+sobvvWs82GdQ570TUKs+Kq8iEfIYL2xNg2Kr 5A+K9Y/u/NicW5soGJWsYEMC7ZT6XCldkPWK9Uq+SRVYTszPR/Bi+jTDTyF5jX6KxXZk MOPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jywcoq6WfpSz2CinBO3sC4x3ilzoShyeAxGkJ7NOWiY=; b=0/8rdFm9/Kv6813FGGoczK91g85/aZ4fkVemJpMpQ8Mn+zupcheQ/4DYyResfCJYM4 j9ifJKTX3t3WZU321Y3VGNs+2hi13tZ1swD6/y6Mtmg57r7CrhldDisRdQrKf6Hj3OVT zoV6MD3jEQeeI4B3nj/KzmPrcg+w+hn02zGbg1XQcsMxqlZT445qLDe39+PwOIKrWbaV MYCmY0M2m3e5KGSb8U43ttOB6YfySMYJZKpT2IPWVZ4SmCScRa2FvGwsaFjzmW9Qv+9y LsGba5SWlmRrrW4aexOtUxY01dFG1y6YFThcIszWsNXe0BLDN1SCw3IU85hhYmCQFPWJ IhzA== X-Gm-Message-State: ACrzQf1rEr77km/fu0NrXicQg525OHCMjlZ6FLY2CUJfyYx0Os8A8aMW a/VuqEiE14HYNlXk/h8d0d0= X-Google-Smtp-Source: AMsMyM7vsl034KhRlWp2MzTvz0KfLObK3DiLcMClWHQJx/Hu4OX67zKeQVDUjLL4e7PDr4FKaoEXjw== X-Received: by 2002:a05:6870:a188:b0:131:ca36:7f9b with SMTP id a8-20020a056870a18800b00131ca367f9bmr11334187oaf.86.1667013023139; Fri, 28 Oct 2022 20:10:23 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-5d1e-45f3-e9b5-d771.res6.spectrum.com. [2603:8081:140c:1a00:5d1e:45f3:e9b5:d771]) by smtp.googlemail.com with ESMTPSA id p3-20020a0568301d4300b006391adb6034sm162493oth.72.2022.10.28.20.10.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 20:10:22 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, leon@kernel.org, zyjzyj2000@gmail.com, jhack@hpe.com, linux-rdma@vger.kernel.org Cc: Bob Pearson , Ian Ziemba Subject: [PATCH v3 05/13] RDMA/rxe: Cleanup comp tasks in rxe_qp.c Date: Fri, 28 Oct 2022 22:10:02 -0500 Message-Id: <20221029031009.64467-6-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221029031009.64467-1-rpearsonhpe@gmail.com> References: <20221029031009.64467-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Take advantage of inline task behavior to cleanup code in rxe_qp.c for completer tasks. Signed-off-by: Ian Ziemba Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_qp.c | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index b5e108794aa1..3691eb97c576 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -480,8 +480,7 @@ static void rxe_qp_reset(struct rxe_qp *qp) /* stop request/comp */ if (qp->sq.queue) { - if (qp_type(qp) == IB_QPT_RC) - rxe_disable_task(&qp->comp.task); + rxe_disable_task(&qp->comp.task); rxe_disable_task(&qp->req.task); } @@ -524,9 +523,7 @@ static void rxe_qp_reset(struct rxe_qp *qp) rxe_enable_task(&qp->resp.task); if (qp->sq.queue) { - if (qp_type(qp) == IB_QPT_RC) - rxe_enable_task(&qp->comp.task); - + rxe_enable_task(&qp->comp.task); rxe_enable_task(&qp->req.task); } } @@ -537,10 +534,7 @@ static void rxe_qp_drain(struct rxe_qp *qp) if (qp->sq.queue) { if (qp->req.state != QP_STATE_DRAINED) { qp->req.state = QP_STATE_DRAIN; - if (qp_type(qp) == IB_QPT_RC) - rxe_sched_task(&qp->comp.task); - else - __rxe_do_task(&qp->comp.task); + rxe_sched_task(&qp->comp.task); rxe_sched_task(&qp->req.task); } } @@ -556,11 +550,7 @@ void rxe_qp_error(struct rxe_qp *qp) /* drain work and packet queues */ rxe_sched_task(&qp->resp.task); - - if (qp_type(qp) == IB_QPT_RC) - rxe_sched_task(&qp->comp.task); - else - __rxe_do_task(&qp->comp.task); + rxe_sched_task(&qp->comp.task); rxe_sched_task(&qp->req.task); } From patchwork Sat Oct 29 03:10:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13024488 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65737FA3746 for ; Sat, 29 Oct 2022 03:10:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229494AbiJ2DKx (ORCPT ); Fri, 28 Oct 2022 23:10:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33686 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229520AbiJ2DKp (ORCPT ); Fri, 28 Oct 2022 23:10:45 -0400 Received: from mail-oa1-x32.google.com (mail-oa1-x32.google.com [IPv6:2001:4860:4864:20::32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3AB142A954 for ; Fri, 28 Oct 2022 20:10:24 -0700 (PDT) Received: by mail-oa1-x32.google.com with SMTP id 586e51a60fabf-13bd19c3b68so8325115fac.7 for ; Fri, 28 Oct 2022 20:10:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NLhGIriOec+yW5+uAGQBdRbRCWaKcwXXxn/8Qw9nAnA=; b=I6W4EZaaO9pWLvcMyoWS4epBDWmaPNv/BcCL05mBWFks2Alo0AEgXXAI6kinDxRcWA p2XHKxmK1WupW0jyhosikoQHly7if5dQs28rXOYmfHMFk11oSOyJVNbfRWkUPe9wZzo0 RsvvXoLW/353HxOpZAKqwI2QooflMeVqyQTzwZ4GYsPzztXO6DJKOY6pQgO5BqTPBNTV mqziF8nqVr/p3LKU2GyQ4TfGw+cgxqFSe0yJ6/iYTmEOcDN+Sif67AAaaLH00pR1xYKv 6T6XBTNrc2h60srGFM5ba2VPe08PqaGnV5MPJbe4R2A/xpJsIYbszUzwGRnVwXDaLmE2 UBVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NLhGIriOec+yW5+uAGQBdRbRCWaKcwXXxn/8Qw9nAnA=; b=0axVREa00YO0WrMUtklaVXhgwe2hGwcA1d3pws9pfJ0QM3AbDK7vyjzyo4Gwf0m7gK S6xwL91C4S3kYyHFXatoD2P4ga6b2u04B3sJhvS7TsmsxesKa6PyBkLEDYLvVOsOpXgS DbdUPA8XGNs5w/lcsJ/TGRAYXmuXxtMTewRYm/UR+hGreLQ5jC5eRQ7SnDnkTdnAUWR8 93qJntXX8SSHXSU+x/+VeefRlcqmoHUFXRt+Dxqi5b914fcbFE1M+k2zKIJoRC/hctup qzl10rs0xtOPggNVYhzBYl+7JRTXoP6VtLJdjpxA4AbR5CGD+iOSOwvFvUDqtsj7DLaA ZZRA== X-Gm-Message-State: ACrzQf0mt8idb53Qa2hVJrl4eUS2V6ERU+iF2uatsC36czK/a94r8Ei7 /MXuuAQ9gKlAg5DUczQQtrZUMbYZNA0= X-Google-Smtp-Source: AMsMyM579DeKc0tz/XzXneC5fUk3rmyBahmrmqPfG7yd8w7CdCsX84dJZu63yqRgRT2GUAn9ArE2VQ== X-Received: by 2002:a05:6871:54d:b0:13b:ff2:caab with SMTP id t13-20020a056871054d00b0013b0ff2caabmr1376282oal.108.1667013024073; Fri, 28 Oct 2022 20:10:24 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-5d1e-45f3-e9b5-d771.res6.spectrum.com. [2603:8081:140c:1a00:5d1e:45f3:e9b5:d771]) by smtp.googlemail.com with ESMTPSA id p3-20020a0568301d4300b006391adb6034sm162493oth.72.2022.10.28.20.10.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 20:10:23 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, leon@kernel.org, zyjzyj2000@gmail.com, jhack@hpe.com, linux-rdma@vger.kernel.org Cc: Bob Pearson , Ian Ziemba Subject: [PATCH for-next v3 06/13] RDMA/rxe: Remove __rxe_do_task() Date: Fri, 28 Oct 2022 22:10:03 -0500 Message-Id: <20221029031009.64467-7-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221029031009.64467-1-rpearsonhpe@gmail.com> References: <20221029031009.64467-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The subroutine __rxe_do_task is not thread safe. It is only used in the rxe_qp_reset() and rxe_qp_do_cleanup() routines. After changes in error handling in the tasklet functions the queues can be drained by calling them once outside of the tasklet code. This allows __rxe_do_task() to be removed. Signed-off-by: Ian Ziemba Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_qp.c | 60 ++++++++++++++-------------- drivers/infiniband/sw/rxe/rxe_task.c | 13 ------ drivers/infiniband/sw/rxe/rxe_task.h | 6 --- 3 files changed, 29 insertions(+), 50 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index 3691eb97c576..50f6b8b8ad9d 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -477,28 +477,23 @@ static void rxe_qp_reset(struct rxe_qp *qp) { /* stop tasks from running */ rxe_disable_task(&qp->resp.task); - - /* stop request/comp */ - if (qp->sq.queue) { - rxe_disable_task(&qp->comp.task); - rxe_disable_task(&qp->req.task); - } + rxe_disable_task(&qp->comp.task); + rxe_disable_task(&qp->req.task); /* move qp to the reset state */ qp->req.state = QP_STATE_RESET; qp->comp.state = QP_STATE_RESET; qp->resp.state = QP_STATE_RESET; - /* let state machines reset themselves drain work and packet queues - * etc. - */ - __rxe_do_task(&qp->resp.task); + /* drain work and packet queues */ + rxe_responder(qp); + rxe_completer(qp); + rxe_requester(qp); - if (qp->sq.queue) { - __rxe_do_task(&qp->comp.task); - __rxe_do_task(&qp->req.task); + if (qp->rq.queue) + rxe_queue_reset(qp->rq.queue); + if (qp->sq.queue) rxe_queue_reset(qp->sq.queue); - } /* cleanup attributes */ atomic_set(&qp->ssn, 0); @@ -521,11 +516,8 @@ static void rxe_qp_reset(struct rxe_qp *qp) /* reenable tasks */ rxe_enable_task(&qp->resp.task); - - if (qp->sq.queue) { - rxe_enable_task(&qp->comp.task); - rxe_enable_task(&qp->req.task); - } + rxe_enable_task(&qp->comp.task); + rxe_enable_task(&qp->req.task); } /* drain the send queue */ @@ -543,15 +535,25 @@ static void rxe_qp_drain(struct rxe_qp *qp) /* move the qp to the error state */ void rxe_qp_error(struct rxe_qp *qp) { + /* stop tasks from running */ + rxe_disable_task(&qp->resp.task); + rxe_disable_task(&qp->comp.task); + rxe_disable_task(&qp->req.task); + qp->req.state = QP_STATE_ERROR; qp->resp.state = QP_STATE_ERROR; qp->comp.state = QP_STATE_ERROR; qp->attr.qp_state = IB_QPS_ERR; /* drain work and packet queues */ - rxe_sched_task(&qp->resp.task); - rxe_sched_task(&qp->comp.task); - rxe_sched_task(&qp->req.task); + rxe_responder(qp); + rxe_completer(qp); + rxe_requester(qp); + + /* reenable tasks */ + rxe_enable_task(&qp->resp.task); + rxe_enable_task(&qp->comp.task); + rxe_enable_task(&qp->req.task); } /* called by the modify qp verb */ @@ -770,24 +772,20 @@ static void rxe_qp_do_cleanup(struct work_struct *work) qp->valid = 0; qp->qp_timeout_jiffies = 0; - rxe_cleanup_task(&qp->resp.task); if (qp_type(qp) == IB_QPT_RC) { del_timer_sync(&qp->retrans_timer); del_timer_sync(&qp->rnr_nak_timer); } + rxe_cleanup_task(&qp->resp.task); rxe_cleanup_task(&qp->req.task); rxe_cleanup_task(&qp->comp.task); - /* flush out any receive wr's or pending requests */ - if (qp->req.task.func) - __rxe_do_task(&qp->req.task); - - if (qp->sq.queue) { - __rxe_do_task(&qp->comp.task); - __rxe_do_task(&qp->req.task); - } + /* drain any receive wr's or pending requests */ + rxe_responder(qp); + rxe_completer(qp); + rxe_requester(qp); if (qp->sq.queue) rxe_queue_cleanup(qp->sq.queue); diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index 8dfbfa164eff..120693c9a795 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -6,19 +6,6 @@ #include "rxe.h" -int __rxe_do_task(struct rxe_task *task) - -{ - int ret; - - while ((ret = task->func(task->arg)) == 0) - ; - - task->ret = ret; - - return ret; -} - /* * this locking is due to a potential race where * a second caller finds the task already running diff --git a/drivers/infiniband/sw/rxe/rxe_task.h b/drivers/infiniband/sw/rxe/rxe_task.h index 31963129ff7a..d594468fcf56 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.h +++ b/drivers/infiniband/sw/rxe/rxe_task.h @@ -48,12 +48,6 @@ struct rxe_task { int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *), enum rxe_task_type type); -/* - * raw call to func in loop without any checking - * can call when tasklets are disabled - */ -int __rxe_do_task(struct rxe_task *task); - void rxe_run_task(struct rxe_task *task); void rxe_sched_task(struct rxe_task *task); From patchwork Sat Oct 29 03:10:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13024489 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 213EDC38A02 for ; Sat, 29 Oct 2022 03:10:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229520AbiJ2DKx (ORCPT ); Fri, 28 Oct 2022 23:10:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33272 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229912AbiJ2DKp (ORCPT ); Fri, 28 Oct 2022 23:10:45 -0400 Received: from mail-oa1-x32.google.com (mail-oa1-x32.google.com [IPv6:2001:4860:4864:20::32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 494142A964 for ; Fri, 28 Oct 2022 20:10:25 -0700 (PDT) Received: by mail-oa1-x32.google.com with SMTP id 586e51a60fabf-13bef14ea06so8364180fac.3 for ; Fri, 28 Oct 2022 20:10:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GUSQl+n9eOMiYHJBVEgK0D6CPdr/mZLR3WgDvJg/GxE=; b=Kp1HlXbePzZw3g4IicJDCVeMZQvKaAPKo7VZhX9Lpj3Npjbgo9KkFJHBiNKK+KuTgq EkMu4vMrwRln9zhh/SmyD9l3/sEc0lx14vPtu6OQm1uIC2eOc8i7db22DA2qn2yaZYLA TQYxkMS5H0hJO9hn+CTwszqPXlTJYV7nDGS4zQFfWPjJKX1umHDdYrAVGfIH3UPX5VKK V27hQ3ezrAWziZCqna5F1n468aAv8AKIfNQCLJW+59/JIPox5/AGMqbaAGJeKeHaC5qD 8gB3AMlJjKR8j9rmcbKmFpqsn7itm9LpBwBJCGCTF+CqlqqTA4TJippbVorCAS4/UD5g 7EGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GUSQl+n9eOMiYHJBVEgK0D6CPdr/mZLR3WgDvJg/GxE=; b=ReTQYXtmGiC09kLwmK9ygn/dnCjDa/EvGm11mqCsaje/0JTankgZrbTDFNjo2JV5xa 3FMLu2/zc4+XGJ7LWRcwN3sWSak/F7XEktf6nkLh5wO2KtQ/jeXBn67xtI+LzDCcbVS/ dW2jYDtipq0wR7LmU3PtBTv1TOXwSFxWWAWJKyuJ0+Zqax+wrVN5mdVXktVhbWOqYf2m K/4vk/ZWeZQ6RfxPPTgJqvb4wTxdcLb8GYmQgJ1a1ojrjrBnnHtmjfpsN3B1AQFLae84 jjQnNyRqFHvRdPOk6gJbIzC06IAx1qZDjQd/iEPfp+8FP24pLWXSP8WTVggaHbKX9b7Y mygg== X-Gm-Message-State: ACrzQf2EVUmyrcLw/XLe0QoGvGcErdZBaQuG7zm0S9jPXCHdgKNqVJ4H ZRhcISy/5R8mnsN+wUiVQLg= X-Google-Smtp-Source: AMsMyM40ZA4wCI0Rm6XDR5AcD16LrQ0Pew7qlYdVDeuckjln5UQFQzpxob5wOTjMCr+Qn2sQvsPpGw== X-Received: by 2002:a05:6870:e88a:b0:13b:6e13:a9a5 with SMTP id q10-20020a056870e88a00b0013b6e13a9a5mr1324180oan.264.1667013025060; Fri, 28 Oct 2022 20:10:25 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-5d1e-45f3-e9b5-d771.res6.spectrum.com. [2603:8081:140c:1a00:5d1e:45f3:e9b5:d771]) by smtp.googlemail.com with ESMTPSA id p3-20020a0568301d4300b006391adb6034sm162493oth.72.2022.10.28.20.10.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 20:10:24 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, leon@kernel.org, zyjzyj2000@gmail.com, jhack@hpe.com, linux-rdma@vger.kernel.org Cc: Bob Pearson , Ian Ziemba Subject: [PATCH for-next v3 07/13] RDMA/rxe: Make tasks schedule each other Date: Fri, 28 Oct 2022 22:10:04 -0500 Message-Id: <20221029031009.64467-8-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221029031009.64467-1-rpearsonhpe@gmail.com> References: <20221029031009.64467-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Replace rxe_run_task() by rxe_sched_task() when tasks call each other. These are not performance critical and mainly involve error paths but they run the risk of causing deadlocks. Signed-off-by: Ian Ziemba Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_comp.c | 8 ++++---- drivers/infiniband/sw/rxe/rxe_req.c | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c index 76dc0a4702fd..f2256f67edbf 100644 --- a/drivers/infiniband/sw/rxe/rxe_comp.c +++ b/drivers/infiniband/sw/rxe/rxe_comp.c @@ -316,7 +316,7 @@ static inline enum comp_state check_ack(struct rxe_qp *qp, qp->comp.psn = pkt->psn; if (qp->req.wait_psn) { qp->req.wait_psn = 0; - rxe_run_task(&qp->req.task); + rxe_sched_task(&qp->req.task); } } return COMPST_ERROR_RETRY; @@ -463,7 +463,7 @@ static void do_complete(struct rxe_qp *qp, struct rxe_send_wqe *wqe) */ if (qp->req.wait_fence) { qp->req.wait_fence = 0; - rxe_run_task(&qp->req.task); + rxe_sched_task(&qp->req.task); } } @@ -477,7 +477,7 @@ static inline enum comp_state complete_ack(struct rxe_qp *qp, if (qp->req.need_rd_atomic) { qp->comp.timeout_retry = 0; qp->req.need_rd_atomic = 0; - rxe_run_task(&qp->req.task); + rxe_sched_task(&qp->req.task); } } @@ -731,7 +731,7 @@ int rxe_completer(void *arg) RXE_CNT_COMP_RETRY); qp->req.need_retry = 1; qp->comp.started_retry = 1; - rxe_run_task(&qp->req.task); + rxe_sched_task(&qp->req.task); } goto done; diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 41f1d84f0acb..fba7572e1d0c 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -733,7 +733,7 @@ int rxe_requester(void *arg) qp->req.wqe_index); wqe->state = wqe_state_done; wqe->status = IB_WC_SUCCESS; - rxe_run_task(&qp->comp.task); + rxe_sched_task(&qp->comp.task); goto done; } payload = mtu; @@ -817,7 +817,7 @@ int rxe_requester(void *arg) qp->req.wqe_index = queue_next_index(qp->sq.queue, qp->req.wqe_index); wqe->state = wqe_state_error; qp->req.state = QP_STATE_ERROR; - rxe_run_task(&qp->comp.task); + rxe_sched_task(&qp->comp.task); exit: ret = -EAGAIN; out: From patchwork Sat Oct 29 03:10:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13024482 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF23EC38A02 for ; Sat, 29 Oct 2022 03:10:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229915AbiJ2DKs (ORCPT ); Fri, 28 Oct 2022 23:10:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33274 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229925AbiJ2DKn (ORCPT ); Fri, 28 Oct 2022 23:10:43 -0400 Received: from mail-oa1-x31.google.com (mail-oa1-x31.google.com [IPv6:2001:4860:4864:20::31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 315BB29814 for ; Fri, 28 Oct 2022 20:10:26 -0700 (PDT) Received: by mail-oa1-x31.google.com with SMTP id 586e51a60fabf-13ba9a4430cso8323973fac.11 for ; Fri, 28 Oct 2022 20:10:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=p3kPzmPDjjNhDNGku1w9RH09kAppbd1qvrZUMK/RXkU=; b=IKn7NiEp4Dy+EDfqswhqzMwlT9wMe4wzl53jA+cBUZ8qY+YKc2rYCGXnlXDPflac91 Zw9L3d6BUvtHTA40foDwHRLByK6+MNVf43jYcsGlZ88c7+mTRnZaBN1JNN1WW86tn/ns SSn1MXKpPqyKVstM7+UqboV8Lqy13yxAv8IyNj8vodrMS73sxO4zyWKO8aC4jQYJe2Og huu0kTWMpt2laPF/CXUCZhvXKU09aQhhuUaEHafoNxpEi9Ovu1fMvZunDKmiL0niEvbq ETEQspaoV5j9Fgv2iyJfZxfI1t2raAEL8KpTha+IMz3K3YaGjXqfNe6RgOYkt+qFk/vZ gOIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=p3kPzmPDjjNhDNGku1w9RH09kAppbd1qvrZUMK/RXkU=; b=JoXzjG2/m9xP0BhmEIGxkBzg9k2pCu/CL/xi0VCmrkxDI+Yk83qCek5q899Ap1HPNx CQ45UaNjOqh2Vr8crSoiyWhZN5IgvW20lDF9FgRs6+rdJ3EbHlQUV9fz2rt0X4uiTeVE 9zPmPdt1n9JMX8L+Caqk6+F8R9+jKjdbwjLV7grfc6sFV9+XDUXK7hiVaekFZmvdyzzT tAPKGNfnCCKrOAO/ezt5aN6pUqUQxZi98evPmSIrhiY3QHwVemzbUYimeNh02818QfAH OHPk1sKamFcOlu3YdmI54w66KvRphInLlC+4UjT8aiaF3SYIpDCghIA6H6sICR3kR3F5 Dbgw== X-Gm-Message-State: ACrzQf3Z99T3cTXA1qcTR47l+PFPBJAPayJ/AB+FrMaWQpY3cky+j5kJ 5Bqw7uN/LKFolMi4s+TGVFo= X-Google-Smtp-Source: AMsMyM6USB781BkoCuOQWjg/fgqRSE/QQcmmPHpOypb6zTT2y5a86i/VRFVe3p+ddkxKjwhqJ6GDlg== X-Received: by 2002:a05:6870:c082:b0:12b:542c:71cf with SMTP id c2-20020a056870c08200b0012b542c71cfmr10666966oad.45.1667013026037; Fri, 28 Oct 2022 20:10:26 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-5d1e-45f3-e9b5-d771.res6.spectrum.com. [2603:8081:140c:1a00:5d1e:45f3:e9b5:d771]) by smtp.googlemail.com with ESMTPSA id p3-20020a0568301d4300b006391adb6034sm162493oth.72.2022.10.28.20.10.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 20:10:25 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, leon@kernel.org, zyjzyj2000@gmail.com, jhack@hpe.com, linux-rdma@vger.kernel.org Cc: Bob Pearson , Ian Ziemba Subject: [PATCH for-next v3 08/13] RDMA/rxe: Implement disable/enable_task() Date: Fri, 28 Oct 2022 22:10:05 -0500 Message-Id: <20221029031009.64467-9-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221029031009.64467-1-rpearsonhpe@gmail.com> References: <20221029031009.64467-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Implement common disable_task() and enable_task() routines by adding a new PAUSED state to the do_task() state machine. These replace tasklet_disable and tasklet_enable with code that can be shared with all the task types. Move rxe_sched_task to re-schedule the task outside of the locks to avoid a deadlock. Signed-off-by: Ian Ziemba Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_task.c | 107 ++++++++++++++++----------- drivers/infiniband/sw/rxe/rxe_task.h | 1 + 2 files changed, 66 insertions(+), 42 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index 120693c9a795..d824de82f2ae 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -6,36 +6,46 @@ #include "rxe.h" -/* - * this locking is due to a potential race where - * a second caller finds the task already running - * but looks just after the last call to func - */ -static void do_task(struct rxe_task *task) +static bool task_is_idle(struct rxe_task *task) { - unsigned int iterations = RXE_MAX_ITERATIONS; - int cont; - int ret; + if (task->destroyed) + return false; spin_lock_bh(&task->lock); switch (task->state) { case TASK_STATE_START: task->state = TASK_STATE_BUSY; spin_unlock_bh(&task->lock); - break; - + return true; case TASK_STATE_BUSY: task->state = TASK_STATE_ARMED; fallthrough; case TASK_STATE_ARMED: - spin_unlock_bh(&task->lock); - return; - + case TASK_STATE_PAUSED: + break; default: + WARN_ON(1); + break; + } + spin_unlock_bh(&task->lock); + + return false; +} + +static void do_task(struct rxe_task *task) +{ + unsigned int iterations = RXE_MAX_ITERATIONS; + bool resched = false; + int cont; + int ret; + + /* flush out pending tasks */ + spin_lock_bh(&task->lock); + if (task->state == TASK_STATE_PAUSED) { spin_unlock_bh(&task->lock); - pr_warn("%s failed with bad state %d\n", __func__, task->state); return; } + spin_unlock_bh(&task->lock); do { cont = 0; @@ -43,47 +53,52 @@ static void do_task(struct rxe_task *task) spin_lock_bh(&task->lock); switch (task->state) { + case TASK_STATE_START: case TASK_STATE_BUSY: if (ret) { task->state = TASK_STATE_START; - } else if (iterations--) { + } else if (task->type == RXE_TASK_TYPE_INLINE || + iterations--) { cont = 1; } else { - /* reschedule the tasklet and exit - * the loop to give up the cpu - */ - tasklet_schedule(&task->tasklet); task->state = TASK_STATE_START; + resched = true; } break; - - /* someone tried to run the task since the last time we called - * func, so we will call one more time regardless of the - * return value - */ case TASK_STATE_ARMED: task->state = TASK_STATE_BUSY; cont = 1; break; - + case TASK_STATE_PAUSED: + break; default: - pr_warn("%s failed with bad state %d\n", __func__, - task->state); + WARN_ON(1); + break; } spin_unlock_bh(&task->lock); } while (cont); + if (resched) + rxe_sched_task(task); + task->ret = ret; } static void disable_task(struct rxe_task *task) { - /* todo */ + spin_lock_bh(&task->lock); + task->state = TASK_STATE_PAUSED; + spin_unlock_bh(&task->lock); } static void enable_task(struct rxe_task *task) { - /* todo */ + spin_lock_bh(&task->lock); + task->state = TASK_STATE_START; + spin_unlock_bh(&task->lock); + + /* restart task in case */ + rxe_run_task(task); } /* busy wait until any previous tasks are done */ @@ -95,7 +110,8 @@ static void cleanup_task(struct rxe_task *task) do { spin_lock_bh(&task->lock); - idle = (task->state == TASK_STATE_START); + idle = (task->state == TASK_STATE_START || + task->state == TASK_STATE_PAUSED); spin_unlock_bh(&task->lock); } while (!idle); } @@ -103,22 +119,26 @@ static void cleanup_task(struct rxe_task *task) /* silently treat schedule as inline for inline tasks */ static void inline_sched(struct rxe_task *task) { - do_task(task); + if (task_is_idle(task)) + do_task(task); } static void inline_run(struct rxe_task *task) { - do_task(task); + if (task_is_idle(task)) + do_task(task); } static void inline_disable(struct rxe_task *task) { - disable_task(task); + if (!task->destroyed) + disable_task(task); } static void inline_enable(struct rxe_task *task) { - enable_task(task); + if (!task->destroyed) + enable_task(task); } static void inline_cleanup(struct rxe_task *task) @@ -142,31 +162,34 @@ static void inline_init(struct rxe_task *task) /* use tsklet_xxx to avoid name collisions with tasklet_xxx */ static void tsklet_sched(struct rxe_task *task) { - tasklet_schedule(&task->tasklet); + if (task_is_idle(task)) + tasklet_schedule(&task->tasklet); } static void tsklet_do_task(struct tasklet_struct *tasklet) { struct rxe_task *task = container_of(tasklet, typeof(*task), tasklet); - do_task(task); + if (!task->destroyed) + do_task(task); } static void tsklet_run(struct rxe_task *task) { - do_task(task); + if (task_is_idle(task)) + do_task(task); } static void tsklet_disable(struct rxe_task *task) { - disable_task(task); - tasklet_disable(&task->tasklet); + if (!task->destroyed) + disable_task(task); } static void tsklet_enable(struct rxe_task *task) { - tasklet_enable(&task->tasklet); - enable_task(task); + if (!task->destroyed) + enable_task(task); } static void tsklet_cleanup(struct rxe_task *task) diff --git a/drivers/infiniband/sw/rxe/rxe_task.h b/drivers/infiniband/sw/rxe/rxe_task.h index d594468fcf56..792832786456 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.h +++ b/drivers/infiniband/sw/rxe/rxe_task.h @@ -26,6 +26,7 @@ enum { TASK_STATE_START = 0, TASK_STATE_BUSY = 1, TASK_STATE_ARMED = 2, + TASK_STATE_PAUSED = 3, }; /* From patchwork Sat Oct 29 03:10:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13024493 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7190AFA3741 for ; Sat, 29 Oct 2022 03:11:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229913AbiJ2DK7 (ORCPT ); Fri, 28 Oct 2022 23:10:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33276 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229934AbiJ2DKq (ORCPT ); Fri, 28 Oct 2022 23:10:46 -0400 Received: from mail-ot1-x32b.google.com (mail-ot1-x32b.google.com [IPv6:2607:f8b0:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 655612AC65 for ; Fri, 28 Oct 2022 20:10:27 -0700 (PDT) Received: by mail-ot1-x32b.google.com with SMTP id d26-20020a05683018fa00b0066ab705617aso3999426otf.13 for ; Fri, 28 Oct 2022 20:10:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IgZtbe5QbJIdt0GKspEyQP1OPD6Un66ARePr7nadvDw=; b=QpXWXucHfuEvPLpd2lq2j0mQmgQUf4EM/5n/C/e5uMXOE3LmLjtES5Mq3gvg3gS6NN GTAWrFsO4DMFhXPD4S8B3SBiAg5DVNPOTYa0pnNH4o6nMnKsaKQ+ZvWYUZvOXNDHRJkI I7VmFOO08ltpB5MQXB1My4Ja4XcirjYm+NXcJsWUn/Twjr/6TupZzSvYBpOGjo65SRj0 Kj1eWBHFItEM8jKTwxCRpEWJmdEh/QHYGoEMTRAjNR3dluXt23zfm/pvCF0L0w9YCdYn QvN4SJQXbFd2XNgvL3GdcbW215qmQUUfYgazLUC1lAfkey7/dKv5IjhxgIcyNXqT6n3E jfIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IgZtbe5QbJIdt0GKspEyQP1OPD6Un66ARePr7nadvDw=; b=gihvQ9+Q4FpoA6fpgOGqdhj3Uh/mZfB7hA7UyFDTlndfbA/oS4wZcedlqAxVyRlJHI y966gJBxJpVslYkyfXDoyc56jaRBafSl8LNz36dEyHNL5NYgAFgZSLZr94+MHinFRHDq dOSuHDUHHI7ryyRusmfhNP1x3nep8M8DUty2saJ7UQk/mjMph1DemyycNc0cS3RwlmWw JLimGdpJNXiIq1LA7IT8h0+37mHF8kX1d8jtzxWm8nhwN4CX2GOePVU/MFwz+5/3TR9c F6tHzsRLA51G7b5LUsG4wdLsLDkWR08NIHuB2EJ5kmZEoAPM3uHx2XR6XouScXnLX9UB zBfw== X-Gm-Message-State: ACrzQf3NBbQC7izU04Tm4cYPisajEH8FxhxjrmzhPxcGumbuHYNLbNEk gQdDfh8Q6SWcBSqFVlGCuMk= X-Google-Smtp-Source: AMsMyM7XMxd+nMxzP9eN2eN9cxRpuV+mKgnSQjZrD5hsh4Zhh6FMPFXKbZjsp30k7mUSnxb6yYmbtQ== X-Received: by 2002:a05:6830:204f:b0:663:d1ef:9c0a with SMTP id f15-20020a056830204f00b00663d1ef9c0amr1245458otp.10.1667013027135; Fri, 28 Oct 2022 20:10:27 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-5d1e-45f3-e9b5-d771.res6.spectrum.com. [2603:8081:140c:1a00:5d1e:45f3:e9b5:d771]) by smtp.googlemail.com with ESMTPSA id p3-20020a0568301d4300b006391adb6034sm162493oth.72.2022.10.28.20.10.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 20:10:26 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, leon@kernel.org, zyjzyj2000@gmail.com, jhack@hpe.com, linux-rdma@vger.kernel.org Cc: Bob Pearson , Ian Ziemba Subject: [PATCH for-next v3 09/13] RDMA/rxe: Replace TASK_STATE_START by TASK_STATE_IDLE Date: Fri, 28 Oct 2022 22:10:06 -0500 Message-Id: <20221029031009.64467-10-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221029031009.64467-1-rpearsonhpe@gmail.com> References: <20221029031009.64467-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Replace the enum TASK_STATE_START by TASK_STATE_IDLE. Signed-off-by: Ian Ziemba Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_task.c | 14 +++++++------- drivers/infiniband/sw/rxe/rxe_task.h | 2 +- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index d824de82f2ae..0fd0d97e8272 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -13,7 +13,7 @@ static bool task_is_idle(struct rxe_task *task) spin_lock_bh(&task->lock); switch (task->state) { - case TASK_STATE_START: + case TASK_STATE_IDLE: task->state = TASK_STATE_BUSY; spin_unlock_bh(&task->lock); return true; @@ -53,15 +53,15 @@ static void do_task(struct rxe_task *task) spin_lock_bh(&task->lock); switch (task->state) { - case TASK_STATE_START: + case TASK_STATE_IDLE: case TASK_STATE_BUSY: if (ret) { - task->state = TASK_STATE_START; + task->state = TASK_STATE_IDLE; } else if (task->type == RXE_TASK_TYPE_INLINE || iterations--) { cont = 1; } else { - task->state = TASK_STATE_START; + task->state = TASK_STATE_IDLE; resched = true; } break; @@ -94,7 +94,7 @@ static void disable_task(struct rxe_task *task) static void enable_task(struct rxe_task *task) { spin_lock_bh(&task->lock); - task->state = TASK_STATE_START; + task->state = TASK_STATE_IDLE; spin_unlock_bh(&task->lock); /* restart task in case */ @@ -110,7 +110,7 @@ static void cleanup_task(struct rxe_task *task) do { spin_lock_bh(&task->lock); - idle = (task->state == TASK_STATE_START || + idle = (task->state == TASK_STATE_IDLE || task->state == TASK_STATE_PAUSED); spin_unlock_bh(&task->lock); } while (!idle); @@ -219,7 +219,7 @@ int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *), task->func = func; task->destroyed = false; task->type = type; - task->state = TASK_STATE_START; + task->state = TASK_STATE_IDLE; spin_lock_init(&task->lock); diff --git a/drivers/infiniband/sw/rxe/rxe_task.h b/drivers/infiniband/sw/rxe/rxe_task.h index 792832786456..0146307fc517 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.h +++ b/drivers/infiniband/sw/rxe/rxe_task.h @@ -23,7 +23,7 @@ enum rxe_task_type { }; enum { - TASK_STATE_START = 0, + TASK_STATE_IDLE = 0, TASK_STATE_BUSY = 1, TASK_STATE_ARMED = 2, TASK_STATE_PAUSED = 3, From patchwork Sat Oct 29 03:10:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13024487 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0F54FA3745 for ; Sat, 29 Oct 2022 03:10:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229635AbiJ2DKw (ORCPT ); Fri, 28 Oct 2022 23:10:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33384 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229494AbiJ2DKp (ORCPT ); Fri, 28 Oct 2022 23:10:45 -0400 Received: from mail-oa1-x32.google.com (mail-oa1-x32.google.com [IPv6:2001:4860:4864:20::32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31D9E2A411 for ; Fri, 28 Oct 2022 20:10:28 -0700 (PDT) Received: by mail-oa1-x32.google.com with SMTP id 586e51a60fabf-13b103a3e5dso8355326fac.2 for ; Fri, 28 Oct 2022 20:10:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6Z4qyWIMoBUjYRI1MMwuXnIkq79/rYZQuSMRXsWdTRk=; b=qD42AWLtOmlILqRP5T0YeDkfw5HSWAAlLgkLPVQtmiLRgJuKDjdkMambHrgndwAdMJ Gjh2rTxr8w/khKrorBtYXMzjkzWgyIpMA83dUgn9JAaO+YQZW/BCXapjXvL9XsPxHnmA f2ws6KoJ4TQE3H1aNbF2GaPZ5nFglqaRRMvCzMcrT9rnouCKBd5Dg5y0Ux2giaJ0ADjQ WlHyusdckG9ATZ7xPN9D7jwzX4j/yjE7mNc0AfhKwXSM/IBY7Of6qED9/P3liOvAoSBJ zb+FBmzanebsSX1fQx78S4kol+B855bIy2AZCE7izs+XAfpT3psjzxr9LfCtaoD4+L36 VVDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6Z4qyWIMoBUjYRI1MMwuXnIkq79/rYZQuSMRXsWdTRk=; b=P7JaFUQ4Z49hzdQkcpPpYnp7VI9OGE2vkjCxsdQVsqltdrl8t3Xrn22hG8BM1CsoJf cAX6NiVsw7zOWDZws2i9ugzDCp3ou3Z8vOvkrsS4Lc8Jk8hByBrZh1cn77iO0L8NJVJH kDHGCYXq5obP9ZUaNJHI0KsX75QiVVOKxLnXkixgwdXIgjD3xopMqW8H6MKy4rbnGKqu k2A2KLBhf9lgmHKpR2zvKJbSBOhNIX1QP3XLXn6OT4vNzc6Zmqq+JuX+lJOswGCWeNIf dD4eOt4zLX2r7DeikFa2HjycXshegtQF0L2vxJscajhq4lpmG51CY0HzlgsWAyR8rgw6 anzQ== X-Gm-Message-State: ACrzQf3oV7KlhsDfL651lBWI5tBp4U8JYfYv5vbz1V0z08f+8aohnB98 gR2DdCA7rP7aJn58B7zXUV73vHJUK2g= X-Google-Smtp-Source: AMsMyM7Qwpe3H+nxEdO9v14M5MiRG4bdHLnNAYUhACl+6jFUdLulRQw1yBv67mziLpsV4jttreq5eg== X-Received: by 2002:a05:6870:f201:b0:13b:2f1f:865b with SMTP id t1-20020a056870f20100b0013b2f1f865bmr11399373oao.205.1667013028132; Fri, 28 Oct 2022 20:10:28 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-5d1e-45f3-e9b5-d771.res6.spectrum.com. [2603:8081:140c:1a00:5d1e:45f3:e9b5:d771]) by smtp.googlemail.com with ESMTPSA id p3-20020a0568301d4300b006391adb6034sm162493oth.72.2022.10.28.20.10.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 20:10:27 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, leon@kernel.org, zyjzyj2000@gmail.com, jhack@hpe.com, linux-rdma@vger.kernel.org Cc: Bob Pearson , Ian Ziemba Subject: [PATCH for-next v3 10/13] RDMA/rxe: Replace task->destroyed by task state INVALID. Date: Fri, 28 Oct 2022 22:10:07 -0500 Message-Id: <20221029031009.64467-11-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221029031009.64467-1-rpearsonhpe@gmail.com> References: <20221029031009.64467-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add a new state TASK_STATE_INVALID to replace the flag task->destroyed. Make changes to task->state proteted by task->lock now including TASK_STATE_INVALID. Signed-off-by: Ian Ziemba Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_task.c | 49 ++++++++++------------------ drivers/infiniband/sw/rxe/rxe_task.h | 3 +- 2 files changed, 19 insertions(+), 33 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index 0fd0d97e8272..da175f2a0dbf 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -8,9 +8,6 @@ static bool task_is_idle(struct rxe_task *task) { - if (task->destroyed) - return false; - spin_lock_bh(&task->lock); switch (task->state) { case TASK_STATE_IDLE: @@ -19,12 +16,8 @@ static bool task_is_idle(struct rxe_task *task) return true; case TASK_STATE_BUSY: task->state = TASK_STATE_ARMED; - fallthrough; - case TASK_STATE_ARMED: - case TASK_STATE_PAUSED: break; default: - WARN_ON(1); break; } spin_unlock_bh(&task->lock); @@ -41,7 +34,8 @@ static void do_task(struct rxe_task *task) /* flush out pending tasks */ spin_lock_bh(&task->lock); - if (task->state == TASK_STATE_PAUSED) { + if (task->state == TASK_STATE_PAUSED || + task->state == TASK_STATE_INVALID) { spin_unlock_bh(&task->lock); return; } @@ -69,10 +63,7 @@ static void do_task(struct rxe_task *task) task->state = TASK_STATE_BUSY; cont = 1; break; - case TASK_STATE_PAUSED: - break; default: - WARN_ON(1); break; } spin_unlock_bh(&task->lock); @@ -87,14 +78,16 @@ static void do_task(struct rxe_task *task) static void disable_task(struct rxe_task *task) { spin_lock_bh(&task->lock); - task->state = TASK_STATE_PAUSED; + if (task->state != TASK_STATE_INVALID) + task->state = TASK_STATE_PAUSED; spin_unlock_bh(&task->lock); } static void enable_task(struct rxe_task *task) { spin_lock_bh(&task->lock); - task->state = TASK_STATE_IDLE; + if (task->state != TASK_STATE_INVALID) + task->state = TASK_STATE_IDLE; spin_unlock_bh(&task->lock); /* restart task in case */ @@ -104,16 +97,16 @@ static void enable_task(struct rxe_task *task) /* busy wait until any previous tasks are done */ static void cleanup_task(struct rxe_task *task) { - bool idle; - - task->destroyed = true; + bool busy; do { spin_lock_bh(&task->lock); - idle = (task->state == TASK_STATE_IDLE || - task->state == TASK_STATE_PAUSED); + busy = (task->state == TASK_STATE_BUSY || + task->state == TASK_STATE_ARMED); + if (!busy) + task->state = TASK_STATE_INVALID; spin_unlock_bh(&task->lock); - } while (!idle); + } while (busy); } /* silently treat schedule as inline for inline tasks */ @@ -131,14 +124,12 @@ static void inline_run(struct rxe_task *task) static void inline_disable(struct rxe_task *task) { - if (!task->destroyed) - disable_task(task); + disable_task(task); } static void inline_enable(struct rxe_task *task) { - if (!task->destroyed) - enable_task(task); + enable_task(task); } static void inline_cleanup(struct rxe_task *task) @@ -168,10 +159,7 @@ static void tsklet_sched(struct rxe_task *task) static void tsklet_do_task(struct tasklet_struct *tasklet) { - struct rxe_task *task = container_of(tasklet, typeof(*task), tasklet); - - if (!task->destroyed) - do_task(task); + do_task(container_of(tasklet, struct rxe_task, tasklet)); } static void tsklet_run(struct rxe_task *task) @@ -182,14 +170,12 @@ static void tsklet_run(struct rxe_task *task) static void tsklet_disable(struct rxe_task *task) { - if (!task->destroyed) - disable_task(task); + disable_task(task); } static void tsklet_enable(struct rxe_task *task) { - if (!task->destroyed) - enable_task(task); + enable_task(task); } static void tsklet_cleanup(struct rxe_task *task) @@ -217,7 +203,6 @@ int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *), { task->arg = arg; task->func = func; - task->destroyed = false; task->type = type; task->state = TASK_STATE_IDLE; diff --git a/drivers/infiniband/sw/rxe/rxe_task.h b/drivers/infiniband/sw/rxe/rxe_task.h index 0146307fc517..2c4ef4d339f1 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.h +++ b/drivers/infiniband/sw/rxe/rxe_task.h @@ -27,6 +27,7 @@ enum { TASK_STATE_BUSY = 1, TASK_STATE_ARMED = 2, TASK_STATE_PAUSED = 3, + TASK_STATE_INVALID = 4, }; /* @@ -41,7 +42,7 @@ struct rxe_task { void *arg; int (*func)(void *arg); int ret; - bool destroyed; + bool invalid; const struct rxe_task_ops *ops; enum rxe_task_type type; }; From patchwork Sat Oct 29 03:10:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13024492 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 981B0C38A02 for ; Sat, 29 Oct 2022 03:10:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229877AbiJ2DK5 (ORCPT ); Fri, 28 Oct 2022 23:10:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33694 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229933AbiJ2DKp (ORCPT ); Fri, 28 Oct 2022 23:10:45 -0400 Received: from mail-oa1-x2c.google.com (mail-oa1-x2c.google.com [IPv6:2001:4860:4864:20::2c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 571BD2A96A for ; Fri, 28 Oct 2022 20:10:29 -0700 (PDT) Received: by mail-oa1-x2c.google.com with SMTP id 586e51a60fabf-13b6c1c89bdso8303695fac.13 for ; Fri, 28 Oct 2022 20:10:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=soz/F6qoTZgUnUzOYqKvjKeAcULP68mCGNrH+7qHydk=; b=ZMAR4uz82O8YYa71qADcL9CZVS1BmRsnxryByrsWV3hUsfLZrYhNqC3SLNhWuSzWbB RHeN8U5ersX+lruBCxVTLyInJgpzOjqnyT+Y04bA4dReX5kWUz/JQkoDxDXlszOec9h6 qhgZdOVVmTjl3HFXGaHBD8lXKldJV9VDtxoYHD39fxUF+UyrsJy0dOg9z1zpD43lisyW Cyms/dPbIqsJShStzdnI/zRWhR58WV5sZohtexMsmIwakXrI3+fYHfEeOXK45pqFKuUI D2OwXeScd2413BXevawJqu7ZA2cV5NaWTchIuXY58CTeLfu8IQ6HYxnx6tYYzIwQnWwr J5Wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=soz/F6qoTZgUnUzOYqKvjKeAcULP68mCGNrH+7qHydk=; b=RmCy/f7iWjYdsf9tg8rdMOslfhxLQc1MRu7HpI7ICrIgncqYj53YbwzPobT9cp26hV YFKIPAVZMBT+tlP/RqPyQ5LkTUYMMB+ypgelKy8aIsspnwXIVOvmJQyI/7iW+z6obMKL OhmEFv6veY50ccW3X1z5LbGXuJCCocfJV5XtiokwX5590Oh/GD58wIbZoAUCyt/32cYB pChRQZGr5AxcH+8c75h44U86yuoSEZd86FaeriNB7ZFEPJsvKV8PHcD69eRy1udLCiqI hbuugEmIDvF0v8Z2QIEphqO7q/VxUIRfjjo4BTrWA3BaDyLoYJdSHf25jG+L1siS2h40 CoGw== X-Gm-Message-State: ACrzQf1Swnh63JHQzR5L7rlnP2WXnjOriy1kBlqS0FNqXXJ5Lm/OxVj/ wfknWQvKaePpizT74b/7bIg= X-Google-Smtp-Source: AMsMyM5TMjLgrq7+C1+F2ZVElBA1teWXl3TMyg6kD21nGMvYMEYjBCDqK58/AbtEFAqFRPIVlh1Tdg== X-Received: by 2002:a05:6870:3120:b0:132:7d08:7059 with SMTP id v32-20020a056870312000b001327d087059mr1336547oaa.226.1667013029213; Fri, 28 Oct 2022 20:10:29 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-5d1e-45f3-e9b5-d771.res6.spectrum.com. [2603:8081:140c:1a00:5d1e:45f3:e9b5:d771]) by smtp.googlemail.com with ESMTPSA id p3-20020a0568301d4300b006391adb6034sm162493oth.72.2022.10.28.20.10.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 20:10:28 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, leon@kernel.org, zyjzyj2000@gmail.com, jhack@hpe.com, linux-rdma@vger.kernel.org Cc: Bob Pearson , Ian Ziemba Subject: [PATCH for-next v3 11/13] RDMA/rxe: Add workqueue support for tasks Date: Fri, 28 Oct 2022 22:10:08 -0500 Message-Id: <20221029031009.64467-12-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221029031009.64467-1-rpearsonhpe@gmail.com> References: <20221029031009.64467-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add a third task type RXE_TASK_TYPE_WORKQUEUE to rxe_task.c. Signed-off-by: Ian Ziemba Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe.c | 9 +++- drivers/infiniband/sw/rxe/rxe_task.c | 66 ++++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_task.h | 10 ++++- 3 files changed, 83 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c index 51daac5c4feb..6d80218334ca 100644 --- a/drivers/infiniband/sw/rxe/rxe.c +++ b/drivers/infiniband/sw/rxe/rxe.c @@ -210,10 +210,16 @@ static int __init rxe_module_init(void) { int err; - err = rxe_net_init(); + err = rxe_alloc_wq(); if (err) return err; + err = rxe_net_init(); + if (err) { + rxe_destroy_wq(); + return err; + } + rdma_link_register(&rxe_link_ops); pr_info("loaded\n"); return 0; @@ -224,6 +230,7 @@ static void __exit rxe_module_exit(void) rdma_link_unregister(&rxe_link_ops); ib_unregister_driver(RDMA_DRIVER_RXE); rxe_net_exit(); + rxe_destroy_wq(); pr_info("unloaded\n"); } diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index da175f2a0dbf..c1177752088d 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -6,6 +6,22 @@ #include "rxe.h" +static struct workqueue_struct *rxe_wq; + +int rxe_alloc_wq(void) +{ + rxe_wq = alloc_workqueue("rxe_wq", WQ_CPU_INTENSIVE, WQ_MAX_ACTIVE); + if (!rxe_wq) + return -ENOMEM; + + return 0; +} + +void rxe_destroy_wq(void) +{ + destroy_workqueue(rxe_wq); +} + static bool task_is_idle(struct rxe_task *task) { spin_lock_bh(&task->lock); @@ -198,6 +214,53 @@ static void tsklet_init(struct rxe_task *task) task->ops = &tsklet_ops; } +static void work_sched(struct rxe_task *task) +{ + if (task_is_idle(task)) + queue_work(rxe_wq, &task->work); +} + +static void work_do_task(struct work_struct *work) +{ + do_task(container_of(work, struct rxe_task, work)); +} + +static void work_run(struct rxe_task *task) +{ + if (task_is_idle(task)) + do_task(task); +} + +static void work_enable(struct rxe_task *task) +{ + enable_task(task); +} + +static void work_disable(struct rxe_task *task) +{ + disable_task(task); + flush_workqueue(rxe_wq); +} + +static void work_cleanup(struct rxe_task *task) +{ + cleanup_task(task); +} + +static const struct rxe_task_ops work_ops = { + .sched = work_sched, + .run = work_run, + .enable = work_enable, + .disable = work_disable, + .cleanup = work_cleanup, +}; + +static void work_init(struct rxe_task *task) +{ + INIT_WORK(&task->work, work_do_task); + task->ops = &work_ops; +} + int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *), enum rxe_task_type type) { @@ -215,6 +278,9 @@ int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *), case RXE_TASK_TYPE_TASKLET: tsklet_init(task); break; + case RXE_TASK_TYPE_WORKQUEUE: + work_init(task); + break; default: pr_debug("%s: invalid task type = %d\n", __func__, type); return -EINVAL; diff --git a/drivers/infiniband/sw/rxe/rxe_task.h b/drivers/infiniband/sw/rxe/rxe_task.h index 2c4ef4d339f1..d1156b935635 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.h +++ b/drivers/infiniband/sw/rxe/rxe_task.h @@ -20,6 +20,7 @@ struct rxe_task_ops { enum rxe_task_type { RXE_TASK_TYPE_INLINE = 0, RXE_TASK_TYPE_TASKLET = 1, + RXE_TASK_TYPE_WORKQUEUE = 2, }; enum { @@ -36,7 +37,10 @@ enum { * called again. */ struct rxe_task { - struct tasklet_struct tasklet; + union { + struct tasklet_struct tasklet; + struct work_struct work; + }; int state; spinlock_t lock; void *arg; @@ -47,6 +51,10 @@ struct rxe_task { enum rxe_task_type type; }; +int rxe_alloc_wq(void); + +void rxe_destroy_wq(void); + int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *), enum rxe_task_type type); From patchwork Sat Oct 29 03:10:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13024486 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EADD6C38A02 for ; Sat, 29 Oct 2022 03:10:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229865AbiJ2DKv (ORCPT ); Fri, 28 Oct 2022 23:10:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229635AbiJ2DKp (ORCPT ); Fri, 28 Oct 2022 23:10:45 -0400 Received: from mail-ot1-x32b.google.com (mail-ot1-x32b.google.com [IPv6:2607:f8b0:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 435E32A959 for ; Fri, 28 Oct 2022 20:10:30 -0700 (PDT) Received: by mail-ot1-x32b.google.com with SMTP id r13-20020a056830418d00b0065601df69c0so4014515otu.7 for ; Fri, 28 Oct 2022 20:10:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oLMAqDdDT7iDgI/SIqz57S3USSjHBvktEV3soMVXD0U=; b=LHOaSQwBUdnerTMbM0+P9aP7rp9JMFBFmUxgtQYLdBHupmT/j8fq2rpI/8RX8io5Fb fqy8URakPS2YQPAxyBdG0rJ/lVa8zElyv2QllLLVIdXQfhUOhLUwPvHlQiFIPkMSiMWl ybzdXBP51y7uO9VZNfrzY/04NuljHmjAfpO4SzZgH0cAh6isYJkHxIB1Qrc3jhOJ2GKe UB58lhCvqvpyuFX4nx7/uPhsNnPXpbBL7Nk2JseMBoOfycls205ufovm1uTIdQWf6jYQ ikohhoVAviDG1R4SFy2gI3Zs7X/w9MEWgrPhewicRwSOz+NxtOg9tt7Umz8aazXYSvJB IGgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oLMAqDdDT7iDgI/SIqz57S3USSjHBvktEV3soMVXD0U=; b=HZpZsOOjw/sNtmz+k0ShKlmcJQlBvQEnCMlAtfQjxAj6404zmgrOtUcR3TcC/3hKEA yWAxe8DQGGVsGWHue27oz6eZRAq2UTx9XabJr3GPhT+CAs6cATFLsGwk+gT3d/ImFjkL 5GbK2ZsRj7Uxnej4vD9o/2peHtleUj6PLxKt0opWuHYL+ahaGGrK/3syEwV8FQFhlpN3 PrAmHGWx9Be1vYw1ufpu5pvtYL7OTnMFrnbrTnpB9hhCsNmOJnA5eu6CgWbxC5QE23wh oROj2gqriywOBjrPQVYjTja0nVAB+lo4q98yhd25t0q4rhJWuaCMTnMp/0dtrYsffhqQ pMlg== X-Gm-Message-State: ACrzQf2XYeOjx6mHyq4N239v67Qt60mpW8pJacNQFsy3YHo+IJzqoN5G Os6Snto38rXQSVINR66Ar4c= X-Google-Smtp-Source: AMsMyM7xOftsJn6dfS6U0EG2TLnEIwzs29MoROezgPA81Hh/YjHfWCXIP29Jo2aKTPpOxi4KYX1NYA== X-Received: by 2002:a05:6830:2e3:b0:661:9db6:223e with SMTP id r3-20020a05683002e300b006619db6223emr1235308ote.338.1667013030351; Fri, 28 Oct 2022 20:10:30 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-5d1e-45f3-e9b5-d771.res6.spectrum.com. [2603:8081:140c:1a00:5d1e:45f3:e9b5:d771]) by smtp.googlemail.com with ESMTPSA id p3-20020a0568301d4300b006391adb6034sm162493oth.72.2022.10.28.20.10.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 20:10:29 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, leon@kernel.org, zyjzyj2000@gmail.com, jhack@hpe.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v3 12/13] RDMA/rxe: Make WORKQUEUE default for RC tasks Date: Fri, 28 Oct 2022 22:10:09 -0500 Message-Id: <20221029031009.64467-13-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221029031009.64467-1-rpearsonhpe@gmail.com> References: <20221029031009.64467-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Change RXE_TASK_TYPE_TASKLET to RXE_TASK_TYPE_WORKQUEUE in rxe_qp.c. This makes work queues the default for tasks except for UD completion tasks. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_qp.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index 50f6b8b8ad9d..ca467d8991a9 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -238,9 +238,10 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp, skb_queue_head_init(&qp->req_pkts); - rxe_init_task(&qp->req.task, qp, rxe_requester, RXE_TASK_TYPE_TASKLET); + rxe_init_task(&qp->req.task, qp, rxe_requester, + RXE_TASK_TYPE_WORKQUEUE); rxe_init_task(&qp->comp.task, qp, rxe_completer, - (qp_type(qp) == IB_QPT_RC) ? RXE_TASK_TYPE_TASKLET : + (qp_type(qp) == IB_QPT_RC) ? RXE_TASK_TYPE_WORKQUEUE : RXE_TASK_TYPE_INLINE); qp->qp_timeout_jiffies = 0; /* Can't be set for UD/UC in modify_qp */ @@ -288,7 +289,8 @@ static int rxe_qp_init_resp(struct rxe_dev *rxe, struct rxe_qp *qp, skb_queue_head_init(&qp->resp_pkts); - rxe_init_task(&qp->resp.task, qp, rxe_responder, RXE_TASK_TYPE_TASKLET); + rxe_init_task(&qp->resp.task, qp, rxe_responder, + RXE_TASK_TYPE_WORKQUEUE); qp->resp.opcode = OPCODE_NONE; qp->resp.msn = 0; From patchwork Sat Oct 29 03:10:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13024491 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7EACFA3742 for ; Sat, 29 Oct 2022 03:10:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229789AbiJ2DKz (ORCPT ); Fri, 28 Oct 2022 23:10:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33386 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229889AbiJ2DKp (ORCPT ); Fri, 28 Oct 2022 23:10:45 -0400 Received: from mail-oo1-xc2c.google.com (mail-oo1-xc2c.google.com [IPv6:2607:f8b0:4864:20::c2c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 506702A967 for ; Fri, 28 Oct 2022 20:10:32 -0700 (PDT) Received: by mail-oo1-xc2c.google.com with SMTP id r15-20020a4abf0f000000b004761c7e6be1so1011500oop.9 for ; Fri, 28 Oct 2022 20:10:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=q/7LZaHNgDRVDQQv8FaYutnEvbje8YPWsOt8sPGR6AM=; b=pAHL8t3HHmtfugM+KyxWPEPbIMHHpmIF2rCBs0mTTqHjGuYthib3yOTt0liupOk0oZ +v0aGUxyQBF2AQg33CjypmC+9R70G46fa52hAyp39N1ZDxhJua2+YlkCygOCZnpxlkBA T460lIZcpgB7Faxg729nAbFrli7oNybiRuGgeuiFAb4R6cLMJE8OwsJGifvxmgs/inns ZgwTotR8j6m9isVw57usYCKtA8GlMX1zJQcj9YJOLam/meQGEDBlD/9eZkh0icuInq9x tX+YmeaBq+WgpZxf7TDK2x4ypoGTemFWfCqvefE0sxg/b/8P3SvsU5DBO16by81pj3Fa oJgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=q/7LZaHNgDRVDQQv8FaYutnEvbje8YPWsOt8sPGR6AM=; b=12sMd9mZDfb5+qZMIBmfenXwS9bIjRjCLYIDtUdUyFM2jJVe854e287fen4DiJJfrz 28jFaY5jLZCjoz1pZSVUQqOpR4PzSY3r6vangPfb1qenNW9UQeYIpNRkIC0sSluZ0LI7 /eyVbNaVTSImTUzW5g8ig1q62tBY/jPzqwdi8q3E6fHdvNy73Y8LLKApyrQO84Ul5bmI BDzvwTZc5rxRPrtoBssbnr6zAl3Q68AQ0yLLxw8WnrlG3H+OHJLMBmJZBtfMjLjW2l5x EYmhrAPanVbCOhYCUi8yMOfpXhRcpI4AbonKMQEC+k6ZZj9rgeV+fVgvTenlht7lkpoq 19og== X-Gm-Message-State: ACrzQf3Y9IJcXQMHr2A0E/8WkqVfEy0boBqUPNwAMpT3wg78z6i6x/Ep Tm8eW1itQhtxdqfysylvEhw= X-Google-Smtp-Source: AMsMyM6cL8e8PL32hmf8h4oKW0wMc7Rrb+8Yv5f8I9nsNPncGAeMXW4PT+E80gGg2UTo4wPrQYpW7Q== X-Received: by 2002:a4a:d4c1:0:b0:494:87d8:426a with SMTP id r1-20020a4ad4c1000000b0049487d8426amr1106239oos.70.1667013031433; Fri, 28 Oct 2022 20:10:31 -0700 (PDT) Received: from ubuntu-22.tx.rr.com (2603-8081-140c-1a00-5d1e-45f3-e9b5-d771.res6.spectrum.com. [2603:8081:140c:1a00:5d1e:45f3:e9b5:d771]) by smtp.googlemail.com with ESMTPSA id p3-20020a0568301d4300b006391adb6034sm162493oth.72.2022.10.28.20.10.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 20:10:31 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, leon@kernel.org, zyjzyj2000@gmail.com, jhack@hpe.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v3 13/13] RDMA/rxe: Remove tasklets from rxe_task.c Date: Fri, 28 Oct 2022 22:10:10 -0500 Message-Id: <20221029031009.64467-14-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221029031009.64467-1-rpearsonhpe@gmail.com> References: <20221029031009.64467-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Remove the option to select tasklets(). Maintain the option to have a pluggable interface for future expansion. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_task.c | 51 ---------------------------- drivers/infiniband/sw/rxe/rxe_task.h | 8 ++--- 2 files changed, 2 insertions(+), 57 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_task.c b/drivers/infiniband/sw/rxe/rxe_task.c index c1177752088d..d9c4ab2e58c8 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.c +++ b/drivers/infiniband/sw/rxe/rxe_task.c @@ -166,54 +166,6 @@ static void inline_init(struct rxe_task *task) task->ops = &inline_ops; } -/* use tsklet_xxx to avoid name collisions with tasklet_xxx */ -static void tsklet_sched(struct rxe_task *task) -{ - if (task_is_idle(task)) - tasklet_schedule(&task->tasklet); -} - -static void tsklet_do_task(struct tasklet_struct *tasklet) -{ - do_task(container_of(tasklet, struct rxe_task, tasklet)); -} - -static void tsklet_run(struct rxe_task *task) -{ - if (task_is_idle(task)) - do_task(task); -} - -static void tsklet_disable(struct rxe_task *task) -{ - disable_task(task); -} - -static void tsklet_enable(struct rxe_task *task) -{ - enable_task(task); -} - -static void tsklet_cleanup(struct rxe_task *task) -{ - cleanup_task(task); - tasklet_kill(&task->tasklet); -} - -static const struct rxe_task_ops tsklet_ops = { - .sched = tsklet_sched, - .run = tsklet_run, - .enable = tsklet_enable, - .disable = tsklet_disable, - .cleanup = tsklet_cleanup, -}; - -static void tsklet_init(struct rxe_task *task) -{ - tasklet_setup(&task->tasklet, tsklet_do_task); - task->ops = &tsklet_ops; -} - static void work_sched(struct rxe_task *task) { if (task_is_idle(task)) @@ -275,9 +227,6 @@ int rxe_init_task(struct rxe_task *task, void *arg, int (*func)(void *), case RXE_TASK_TYPE_INLINE: inline_init(task); break; - case RXE_TASK_TYPE_TASKLET: - tsklet_init(task); - break; case RXE_TASK_TYPE_WORKQUEUE: work_init(task); break; diff --git a/drivers/infiniband/sw/rxe/rxe_task.h b/drivers/infiniband/sw/rxe/rxe_task.h index d1156b935635..fbc7e2bf4e5a 100644 --- a/drivers/infiniband/sw/rxe/rxe_task.h +++ b/drivers/infiniband/sw/rxe/rxe_task.h @@ -19,8 +19,7 @@ struct rxe_task_ops { enum rxe_task_type { RXE_TASK_TYPE_INLINE = 0, - RXE_TASK_TYPE_TASKLET = 1, - RXE_TASK_TYPE_WORKQUEUE = 2, + RXE_TASK_TYPE_WORKQUEUE = 1, }; enum { @@ -37,10 +36,7 @@ enum { * called again. */ struct rxe_task { - union { - struct tasklet_struct tasklet; - struct work_struct work; - }; + struct work_struct work; int state; spinlock_t lock; void *arg;