From patchwork Wed Oct 9 12:51:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13828397 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 51EB8198A01; Wed, 9 Oct 2024 12:51:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478308; cv=none; b=puYU3mSn6ZPmw9sMmqPo/dVkSgrlgCa8qC5egrtOI+PLLKKZuMhfutFN9iLgXiM+93Z6yh0kA9kCHTDTkuma4yvjtXKDSGHXMgjbmqW49yO2XqeoFJ2BBlT4TSiGdrYdkLtKMOkOon+5ua2z9fPW/sk+0nuKWX3H+2Mt8XdIQLM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478308; c=relaxed/simple; bh=rnKdynjqKm/mQK0nDRtTLsJ5yELo/xHl8PWVPVBr0B4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=E2edssElZ6lWbvuq0ehhS5VvBjcwmtIIa7AOqyziBpUAHDMkrFjk1XFJFs49Tm7sdQoDiDYp1jVAn0TVZyIwjM9l6m17oj7n+rgnqJEjWHbcgv5kmUj8LC12cRSl6sg1zYmlr7yknct/3ps0r+k0xM7/fP0BDihaz32RdKOicsk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=iwEBHHGD; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="iwEBHHGD" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B3F95C4CECD; Wed, 9 Oct 2024 12:51:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1728478307; bh=rnKdynjqKm/mQK0nDRtTLsJ5yELo/xHl8PWVPVBr0B4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iwEBHHGDYbUoag2h+zmYdpPC6ps6kVvptJ0fL+KFKT2jqPRO8hfLYUAFKCAdC5SoS u2sXqEXnNYLVbQWcm3zElBs7UIvzN5IQQqQN2j6F+ZxAftAn2PB6A0NXleX4+XSGod 1BlEKavWbtnxa+pH0XTf6Je9n4CpPVY9CCeoXJgbYgJHWh75GQxyNDAbSLZEtm+nGV 0AnqA9NuR4azXvKG/3gxF4zrPOptwbTJciF9HbAydI9QSm1p3qbScCO4wiAdHfVCNO BHB4n8GEoxPolSGX2XUwrlkc756g8+vnmD1WmJ/QIjCXKvTes5Gog2xWW4uBskXWqc Ss+hm5CXtSKqw== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, paulmck@kernel.org, joel@joelfernandes.org, frederic@kernel.org, boqun.feng@gmail.com, urezki@gmail.com, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, qiang.zhang1211@gmail.com, peterz@infradead.org, neeraj.upadhyay@amd.com, Neeraj Upadhyay Subject: [PATCH v2 01/10] rcu: Change rdp arg to cpu number for rcu_watching_snap_stopped_since() Date: Wed, 9 Oct 2024 18:21:18 +0530 Message-Id: <20241009125127.18902-2-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> References: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Neeraj Upadhyay In preparation of making rcu_watching_snap_stopped_since() available for RCU code outside of tree.c, change the rdp argument to cpu number. Signed-off-by: Neeraj Upadhyay --- kernel/rcu/tree.c | 14 +++++++------- kernel/rcu/tree_exp.h | 2 +- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index a60616e69b66..ea17dd2d0344 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -312,16 +312,16 @@ static bool rcu_watching_snap_in_eqs(int snap) * rcu_watching_snap_stopped_since() - Has RCU stopped watching a given CPU * since the specified @snap? * - * @rdp: The rcu_data corresponding to the CPU for which to check EQS. + * @cpu: The CPU for which to check EQS. * @snap: rcu_watching snapshot taken when the CPU wasn't in an EQS. * - * Returns true if the CPU corresponding to @rdp has spent some time in an - * extended quiescent state since @snap. Note that this doesn't check if it - * /still/ is in an EQS, just that it went through one since @snap. + * Returns true if the CPU has spent some time in an extended quiescent state + * since @snap. Note that this doesn't check if it /still/ is in an EQS, just + * that it went through one since @snap. * * This is meant to be used in a loop waiting for a CPU to go through an EQS. */ -static bool rcu_watching_snap_stopped_since(struct rcu_data *rdp, int snap) +static bool rcu_watching_snap_stopped_since(int cpu, int snap) { /* * The first failing snapshot is already ordered against the accesses @@ -334,7 +334,7 @@ static bool rcu_watching_snap_stopped_since(struct rcu_data *rdp, int snap) if (WARN_ON_ONCE(rcu_watching_snap_in_eqs(snap))) return true; - return snap != ct_rcu_watching_cpu_acquire(rdp->cpu); + return snap != ct_rcu_watching_cpu_acquire(cpu); } /* @@ -826,7 +826,7 @@ static int rcu_watching_snap_recheck(struct rcu_data *rdp) * read-side critical section that started before the beginning * of the current RCU grace period. */ - if (rcu_watching_snap_stopped_since(rdp, rdp->watching_snap)) { + if (rcu_watching_snap_stopped_since(rdp->cpu, rdp->watching_snap)) { trace_rcu_fqs(rcu_state.name, rdp->gp_seq, rdp->cpu, TPS("dti")); rcu_gpnum_ovf(rnp, rdp); return 1; diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index fb664d3a01c9..e3bd4c18a852 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -401,7 +401,7 @@ static void __sync_rcu_exp_select_node_cpus(struct rcu_exp_work *rewp) unsigned long mask = rdp->grpmask; retry_ipi: - if (rcu_watching_snap_stopped_since(rdp, rdp->exp_watching_snap)) { + if (rcu_watching_snap_stopped_since(rdp->cpu, rdp->exp_watching_snap)) { mask_ofl_test |= mask; continue; } From patchwork Wed Oct 9 12:51:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13828398 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 493DC19D075; Wed, 9 Oct 2024 12:51:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478313; cv=none; b=mkMDNztwM/kJMw74fbwbVHg3Nb3GEij55Bqyr8T/K5Cml1N+u8Cl23hf2Zaa0UKDsXBrJn3n1bghQu0hk5m4GO1LE8OULQb52fOQAg36TYEIrx4YkCkqaGOPKSPU62fXrnb5wrH1R2R79M+w3qfxtN/HkE2ZF4x8p+la3r2Ehlk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478313; c=relaxed/simple; bh=8qSnsgM8LhMwsfqVBK36j4HZgFG548kSvr6lIkRispo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pmgnEksddVpX49mh4cKBH5bVnv9yguKcjgHLvaD6mYfMe1n0kxS/KpQbjUScxGQNP+aH2pT56tSF14evKB+XXc+KTyZGckBdywwrPGcdKzRm6d+nIvBTF83mFBctXdcvJ8T9Vj/FrtNL+C55xgVRjM3/Gin9yBtxRI5P4IkYkEM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=OEEd2bL6; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="OEEd2bL6" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8227CC4CEC5; Wed, 9 Oct 2024 12:51:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1728478312; bh=8qSnsgM8LhMwsfqVBK36j4HZgFG548kSvr6lIkRispo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OEEd2bL6J6Yh6YBsyCq5m+LGHBJwL1tXjnrk7zssZzMyP59806bkeUfg2fH6FJ1nd aONa9hiDlTdYipD61Gc3YoEou7VDEd9L5b8nD9R8eUi0BkcnTDjBDcX5x8e3oRK5uw MEye2dbo1oq3MWZlGUpE5NJwCeTMghGq2+D7R0R+UozOEXr6PXCi7hnLlEDlIhryBC 99SafEdlZpV1azA9O9dOZxGImxH5GSf5IN7mdGf6sR9mgmhRaNJxWR4ZoWA8nDEo4J rO1xSxAOIkzDS66GODK3oVyP+EWfM55tjYmlubGKLoZ+r0oK+JvpB9TZFuFXog26lU /uObIQnhEidIg== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, paulmck@kernel.org, joel@joelfernandes.org, frederic@kernel.org, boqun.feng@gmail.com, urezki@gmail.com, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, qiang.zhang1211@gmail.com, peterz@infradead.org, neeraj.upadhyay@amd.com, Neeraj Upadhyay Subject: [PATCH v2 02/10] rcu: Make some rcu_watching_* functions global Date: Wed, 9 Oct 2024 18:21:19 +0530 Message-Id: <20241009125127.18902-3-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> References: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Neeraj Upadhyay rcu_watching_snap_in_eqs() and rcu_watching_snap_stopped_since() will be used in subsequent commits by RCU-tasks to check rcu_watching state of idle tasks. So, make these functions global. Signed-off-by: Neeraj Upadhyay --- kernel/rcu/rcu.h | 4 ++++ kernel/rcu/tree.c | 4 ++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index feb3ac1dc5d5..5fec1b52039c 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -609,6 +609,8 @@ void srcutorture_get_gp_data(struct srcu_struct *sp, int *flags, #ifdef CONFIG_TINY_RCU static inline bool rcu_watching_zero_in_eqs(int cpu, int *vp) { return false; } +static inline bool rcu_watching_snap_in_eqs(int snap) { return false; } +static inline bool rcu_watching_snap_stopped_since(int cpu, int snap) { return false; } static inline unsigned long rcu_get_gp_seq(void) { return 0; } static inline unsigned long rcu_exp_batches_completed(void) { return 0; } static inline unsigned long @@ -622,6 +624,8 @@ static inline void rcu_gp_slow_register(atomic_t *rgssp) { } static inline void rcu_gp_slow_unregister(atomic_t *rgssp) { } #else /* #ifdef CONFIG_TINY_RCU */ bool rcu_watching_zero_in_eqs(int cpu, int *vp); +bool rcu_watching_snap_in_eqs(int snap); +bool rcu_watching_snap_stopped_since(int cpu, int snap); unsigned long rcu_get_gp_seq(void); unsigned long rcu_exp_batches_completed(void); unsigned long srcu_batches_completed(struct srcu_struct *sp); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index ea17dd2d0344..5ecbf85f157d 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -303,7 +303,7 @@ static void rcu_watching_online(void) * Return true if the snapshot returned from ct_rcu_watching() * indicates that RCU is in an extended quiescent state. */ -static bool rcu_watching_snap_in_eqs(int snap) +bool rcu_watching_snap_in_eqs(int snap) { return !(snap & CT_RCU_WATCHING); } @@ -321,7 +321,7 @@ static bool rcu_watching_snap_in_eqs(int snap) * * This is meant to be used in a loop waiting for a CPU to go through an EQS. */ -static bool rcu_watching_snap_stopped_since(int cpu, int snap) +bool rcu_watching_snap_stopped_since(int cpu, int snap) { /* * The first failing snapshot is already ordered against the accesses From patchwork Wed Oct 9 12:51:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13828399 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 289FC1A0BDC; Wed, 9 Oct 2024 12:51:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478318; cv=none; b=roafxk3xGg8aYlABTxrSNvLpFSo0QcQ43rYQgnT63Xl7cwyn5emj6yxfG6Me8wyM817+XkscU9R7V4pruH0+0K+A4kymeaAj0zPD2eN7kmTEnndgujPJyHHqtS2Sy5YbMTOiTcQrlWECZnpnIXCQqh0qcihvXoUV6vuNnVOOpH8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478318; c=relaxed/simple; bh=Vse4+IeX1I6gTLyGFA7QmaJV7gC8q/+woRsbl3quBDk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=MBkstxjvvQypiu2znLSbG/jqllVUo0PSK3pDNUyzubB9dJfNxaFQH+CkvUUMc8Etv3OG/pdTG2UoidNrXL5cySfcxUJfrkgMvRnIHOCfkkONuGRtO91GyK3XoZJQDq2NkY64yKwo6ee/o4nUodRvWF4nPrCeWqUM/U5DzYOQiks= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QYHxrEWr; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QYHxrEWr" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7C564C4CEC5; Wed, 9 Oct 2024 12:51:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1728478317; bh=Vse4+IeX1I6gTLyGFA7QmaJV7gC8q/+woRsbl3quBDk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QYHxrEWr62dprPX4XvwAQTZUdX4/fNJcf1toXJPwHQn1g4WNllmPG0EMHLQAniRwF 7f2/kDOHJXvoWv21izfGpPPctjsKRsZsObUb9EW7EFzHW1ATxqudgk0kcGdmdfNtbN 9hSF61E2NdcBqONDdbrF57GbaLPKpxuD1rUO1SAslRf3MNxcAUkPUUeakOcmKyjLR2 j/WPSekI7TP6AqUpLf/fh6PUowSi2t4LDisTF9xnRPAyCTs1SMGlPnIbAYoZAKGUH3 ptudWgjmFLXOFXSE3rX9/NAG0dLBn9GIs59GBcXCVjmPPJuTrm1qYT9+6ovXI8NU51 90pQuUmrp444A== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, paulmck@kernel.org, joel@joelfernandes.org, frederic@kernel.org, boqun.feng@gmail.com, urezki@gmail.com, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, qiang.zhang1211@gmail.com, peterz@infradead.org, neeraj.upadhyay@amd.com, Neeraj Upadhyay Subject: [PATCH v2 03/10] rcu/tasks: Move holdout checks for idle task to a separate function Date: Wed, 9 Oct 2024 18:21:20 +0530 Message-Id: <20241009125127.18902-4-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> References: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Neeraj Upadhyay Move checks for an idle task being a holdout task for RCU-tasks to a separate function - rcu_idle_task_is_holdout(). This function will be used in subsequent commits to add additional checks for idle task. No functional change intended. Suggested-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- kernel/rcu/tasks.h | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 6333f4ccf024..56015ced3f37 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -976,6 +976,15 @@ static void rcu_tasks_pregp_step(struct list_head *hop) synchronize_rcu(); } +static bool rcu_idle_task_is_holdout(struct task_struct *t, int cpu) +{ + /* Idle tasks on offline CPUs are RCU-tasks quiescent states. */ + if (!rcu_cpu_online(cpu)) + return false; + + return true; +} + /* Check for quiescent states since the pregp's synchronize_rcu() */ static bool rcu_tasks_is_holdout(struct task_struct *t) { @@ -995,9 +1004,8 @@ static bool rcu_tasks_is_holdout(struct task_struct *t) cpu = task_cpu(t); - /* Idle tasks on offline CPUs are RCU-tasks quiescent states. */ - if (t == idle_task(cpu) && !rcu_cpu_online(cpu)) - return false; + if (t == idle_task(cpu)) + return rcu_idle_task_is_holdout(t, cpu); return true; } From patchwork Wed Oct 9 12:51:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13828400 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB9F21A2540; Wed, 9 Oct 2024 12:52:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478322; cv=none; b=AcWy5sz1/nc0+afsLxNER1Nrdcg/0AxgOvyG5/VBPRdu8WaGfoc7UMJAcUSd+JJwbkOtEzRh6IeAwDm+HWZjsdcUc5BzTooV+vI4X+OXBgn9d/k5vZcofS7F6ZWHd3P1gPornGYtf6RgDp5F06bkzVGFXD9Wvle6lzfWxei/F7s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478322; c=relaxed/simple; bh=+KNwfAeGfopxST2YcyDd++R2YqnV7Z//j5Y+S4cdFGk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=bDIi15QzUgpQgpVjRmxJtaClvmUeq5DDBhSk7GGBkD5pwKQjo6T6tHHnsvgdzYxR6t02s16rLohJB9Rf/vlLq/+NzAe5/U5KpaTAyvWefg3XMdNtg/VgRX7UU4ZwGZ10VkVbTDATTzVQ6lM0nX/ZL08wvnXzME7V9R5kulgddy4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=XKfsG7XJ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="XKfsG7XJ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5F3C2C4CEC5; Wed, 9 Oct 2024 12:51:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1728478322; bh=+KNwfAeGfopxST2YcyDd++R2YqnV7Z//j5Y+S4cdFGk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XKfsG7XJH/N7e+l+FLANgLva7Ppn0kXak1ESbL/pKgXvCLqlIuMmvYlQvCxPtyqzK rin8pYgxFASnk/9259KLAmMUct/Q9DsBu1/SiQg/a5nIlm1pgpSdGSguSZsgw0yab7 RuGSLGT6HtQLrCwR+Vya9mWDUgOzE6c1uiuOZRw6mUQEIuYIp2zGKM4omLr+l6yhCd h15QBDL0FUd2gVmjyM26Yy1cADtpVpNYEbbnbBlxlvWubaDcyPJSp+5ozz71iPScYJ XyT01xs5i8ADF9KhnPmV3KkC1d7KtMkOPWHj6n4bueLMW1eK5DD72Gr3bLjZhP9fNm hTbTvWxHWWpjw== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, paulmck@kernel.org, joel@joelfernandes.org, frederic@kernel.org, boqun.feng@gmail.com, urezki@gmail.com, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, qiang.zhang1211@gmail.com, peterz@infradead.org, neeraj.upadhyay@amd.com, Neeraj Upadhyay Subject: [PATCH v2 04/10] rcu/tasks: Create rcu_idle_task_is_holdout() definition for !SMP Date: Wed, 9 Oct 2024 18:21:21 +0530 Message-Id: <20241009125127.18902-5-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> References: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Neeraj Upadhyay rcu_idle_task_is_holdout() is called in rcu_tasks_kthread() context. As idle tasks cannot be non-voluntary preempted, non-running idle tasks are not in RCU-tasks critical section. So, idle task is not a RCU-tasks holdout task on !SMP (which also covers TINY_RCU). Signed-off-by: Neeraj Upadhyay --- kernel/rcu/tasks.h | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 56015ced3f37..b794deeaf6d8 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -976,6 +976,7 @@ static void rcu_tasks_pregp_step(struct list_head *hop) synchronize_rcu(); } +#ifdef CONFIG_SMP static bool rcu_idle_task_is_holdout(struct task_struct *t, int cpu) { /* Idle tasks on offline CPUs are RCU-tasks quiescent states. */ @@ -984,6 +985,17 @@ static bool rcu_idle_task_is_holdout(struct task_struct *t, int cpu) return true; } +#else /* #ifdef CONFIG_SMP */ +static inline bool rcu_idle_task_is_holdout(struct task_struct *t, int cpu) +{ + /* + * rcu_idle_task_is_holdout() is called in rcu_tasks_kthread() + * context. Idle thread would have done a voluntary context + * switch. + */ + return false; +} +#endif /* Check for quiescent states since the pregp's synchronize_rcu() */ static bool rcu_tasks_is_holdout(struct task_struct *t) From patchwork Wed Oct 9 12:51:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13828401 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB0421E1A1B; Wed, 9 Oct 2024 12:52:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478327; cv=none; b=LY5SYw5yAkJP/psIWb322BKn9A0IKG5V2hKI21wd17t2IzetbgW4M7LAz0KP4b7qLAbQcuwfoGyDEVeiAEFQ9or9RRisU5cYJuJHCrSgJ8FAOTsPgGfR4b7zi4UEa/mvuh9Y5eIDXH7kOSDKxhnTjPYQBLekDIyPAUJmWFBQ0EM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478327; c=relaxed/simple; bh=A1yHtMIzh/btwkSjs1f15zt7Ryx3LfuM9GWls63if/o=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iLuFwzk9tLYuLTclmmHZQ9qo6rQQcOkWtPdW+u3zeSy/5MilF27+NpT1jwwe83Ie+0yKjeTWEgw1pKGrY7+1bS54oLcVi8l8Ua+KHEpchTqI4TkUKYBYP7BFW+cgA1k7K6iHrYhPil27CLAY65RQDkK9HdvcMbBXQihaY1NdTfE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=gUXoabhT; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gUXoabhT" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 391B3C4CEC5; Wed, 9 Oct 2024 12:52:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1728478327; bh=A1yHtMIzh/btwkSjs1f15zt7Ryx3LfuM9GWls63if/o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gUXoabhTLLWNdcfYOPlIRFhUvh8ZOscsUNJ2KZ28Otiv1LprsJwHDIUGz8J+egJ8Z t43OVZztbRs+RRQhUizvS34DIb7Ey5vyr8C5QFWr3dqg+DQVmtM5wpyEmx1zQ15PwD XqRQrhiruR49Wy9skGtt0R7hJIQjJ23snTWvWg4PAShWuhca4zjDGe7o5P2qZgLyso Fiw1q647bIt6BNNtkeVimnt0IxjU8jnSyFBnz+ie087ba6h+1BLPl3bnJsZmWJRqVR mac+ZgzUtwdwMLtBAVhdMWLQf9SjxjDgumnEkH6fbYxr72USe6JK3CEhYzEZFWRTWv 086WhOrIoEb4g== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, paulmck@kernel.org, joel@joelfernandes.org, frederic@kernel.org, boqun.feng@gmail.com, urezki@gmail.com, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, qiang.zhang1211@gmail.com, peterz@infradead.org, neeraj.upadhyay@amd.com, Neeraj Upadhyay Subject: [PATCH v2 05/10] rcu/tasks: Consider idle tasks not running on CPU as non-holdouts Date: Wed, 9 Oct 2024 18:21:22 +0530 Message-Id: <20241009125127.18902-6-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> References: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Neeraj Upadhyay As idle tasks cannot be non-voluntary preempted, idle tasks which are not running on CPU are not in RCU-tasks read side critical section. So, remove them for holdout tasks for RCU-tasks. Signed-off-by: Neeraj Upadhyay --- kernel/rcu/tasks.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index b794deeaf6d8..9523aff6cdae 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -983,6 +983,15 @@ static bool rcu_idle_task_is_holdout(struct task_struct *t, int cpu) if (!rcu_cpu_online(cpu)) return false; + /* + * As idle tasks cannot be involuntary preempted, non-running idle tasks + * are not in RCU-tasks critical section. + * synchronize_rcu() calls in rcu_tasks_pregp_step() and rcu_tasks_postgp() + * ensure that all ->on_cpu transitions are complete. + */ + if (!t->on_cpu) + return false; + return true; } #else /* #ifdef CONFIG_SMP */ From patchwork Wed Oct 9 12:51:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13828402 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D743C19340C; Wed, 9 Oct 2024 12:52:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478332; cv=none; b=VOBSBK/zqXXaS1jQO2RjiinOp0ragzCdMjpzhlr1MDy57baefkIUFN+bZNRj1at2ELVMzrojQu9ohd7sbXw3Wq+YFYHxNz9dSKbs/VyEpbHpGcOP51lNnYMIOpGT54W7Cwo00APzdSKy5kIVmF57C9ntPZB004OP1gwEcCehRQ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478332; c=relaxed/simple; bh=3OB6pgXC0VJmU+IWwSUEhbW9bzPbw6tk0P+9End7n00=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OoeVP++IF0n6mjOGcaohACy7wRCBP0asoR5BqAgSmMhScggrzJSe7MA/2tyCMdivuHn4Xr61kiNX63E7akVb17osZbtlVBy5oa26TMoMvSA+/UTxO9L0o3NfAWa81QYMfQh/JpE8YnloShgF/GraZ4pjHujOi2d5rGzmMXgbFGY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kImqHSvJ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kImqHSvJ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 256B5C4CEC5; Wed, 9 Oct 2024 12:52:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1728478332; bh=3OB6pgXC0VJmU+IWwSUEhbW9bzPbw6tk0P+9End7n00=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kImqHSvJ1PoiZEKcKs8q2xRocxvYhexe2uq702SwleNQks1fQbcyjrffJIHmXYsWT eKoY5wOdWCvbx9rYEvUGSLbTMeAoPjISNjyQAxKB/Jt72Yr/6B+FYYzhZnrRqz5WdS qNkVr8l30RdI/9dyjSHEt3XuPocjV+V7BmaeY7jrDX2XctJM07wRgcEYDcntVPcwLf OSRYBOWRrgURVggEMuuMEq4y8tdTJkZnfmUdWtirjPBf7986mvuUimm2whIe78QSm2 0k9cHtV7t7Dea1GO9FYFD8r2KtBh7ABck34MiOaYulJNjZhotYKnDUBiei3Hox+e7S lNflP2X/in2cw== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, paulmck@kernel.org, joel@joelfernandes.org, frederic@kernel.org, boqun.feng@gmail.com, urezki@gmail.com, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, qiang.zhang1211@gmail.com, peterz@infradead.org, neeraj.upadhyay@amd.com, Neeraj Upadhyay Subject: [PATCH v2 06/10] rcu/tasks: Check RCU watching state for holdout idle tasks Date: Wed, 9 Oct 2024 18:21:23 +0530 Message-Id: <20241009125127.18902-7-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> References: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Neeraj Upadhyay Use RCU watching state of a CPU to check whether RCU-tasks GP need to wait for idle task on that CPU. Idle tasks which are in deep-idle states where RCU is not watching or which have transitioned to/from deep-idle state do not block RCU-tasks grace period. Signed-off-by: Neeraj Upadhyay --- kernel/rcu/tasks.h | 36 +++++++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 9523aff6cdae..d8506d2e6f54 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -36,6 +36,8 @@ typedef void (*postgp_func_t)(struct rcu_tasks *rtp); * @cpu: CPU number corresponding to this entry. * @index: Index of this CPU in rtpcp_array of the rcu_tasks structure. * @rtpp: Pointer to the rcu_tasks structure. + * @rcu_watching_snap: Per-GP RCU-watching snapshot for idle tasks. + * @rcu_watching_snap_rec: RCU-watching snapshot recorded for idle task. */ struct rcu_tasks_percpu { struct rcu_segcblist cblist; @@ -52,6 +54,8 @@ struct rcu_tasks_percpu { int cpu; int index; struct rcu_tasks *rtpp; + int rcu_watching_snap; + bool rcu_watching_snap_rec; }; /** @@ -957,9 +961,14 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) // rcu_tasks_pregp_step() and by the scheduler's locks and interrupt // disabling. +void call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func); +DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_wait_gp, call_rcu_tasks, "RCU Tasks"); + /* Pre-grace-period preparation. */ static void rcu_tasks_pregp_step(struct list_head *hop) { + int cpu; + /* * Wait for all pre-existing t->on_rq and t->nvcsw transitions * to complete. Invoking synchronize_rcu() suffices because all @@ -974,11 +983,20 @@ static void rcu_tasks_pregp_step(struct list_head *hop) * grace period. */ synchronize_rcu(); + + /* Initialize watching snapshots for this GP */ + for_each_possible_cpu(cpu) { + struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu); + + rtpcp->rcu_watching_snap_rec = false; + } } #ifdef CONFIG_SMP static bool rcu_idle_task_is_holdout(struct task_struct *t, int cpu) { + struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu); + /* Idle tasks on offline CPUs are RCU-tasks quiescent states. */ if (!rcu_cpu_online(cpu)) return false; @@ -992,6 +1010,21 @@ static bool rcu_idle_task_is_holdout(struct task_struct *t, int cpu) if (!t->on_cpu) return false; + if (!rtpcp->rcu_watching_snap_rec) { + /* + * Do plain access. Ordering between remote CPU's pre idle accesses + * and post rcu-tasks grace period is provided by synchronize_rcu() + * in rcu_tasks_postgp(). + */ + rtpcp->rcu_watching_snap = ct_rcu_watching_cpu(cpu); + rtpcp->rcu_watching_snap_rec = true; + /* RCU-idle contexts are RCU-tasks quiescent state for idle tasks. */ + if (rcu_watching_snap_in_eqs(rtpcp->rcu_watching_snap)) + return false; + } else if (rcu_watching_snap_stopped_since(cpu, rtpcp->rcu_watching_snap)) { + return false; + } + return true; } #else /* #ifdef CONFIG_SMP */ @@ -1042,9 +1075,6 @@ static void rcu_tasks_pertask(struct task_struct *t, struct list_head *hop) } } -void call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func); -DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_wait_gp, call_rcu_tasks, "RCU Tasks"); - /* Processing between scanning taskslist and draining the holdout list. */ static void rcu_tasks_postscan(struct list_head *hop) { From patchwork Wed Oct 9 12:51:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13828403 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A0CFB1E22E0; Wed, 9 Oct 2024 12:52:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478337; cv=none; b=bTUfLBWjvvkHP5XJFp2UcbfJffECzZKBIl2oyeEL1CyoYfd1gE/d45PrnXxrvkg1VOq+BPtCcjyRSOJRbomRQSIuuT6ZDesNSxY9NK7oE74CyPTy+tP0JyD+q56GbvBU/Ve8xka7lnZ3/rXVcNAck2AzczGIvwj796N2nIDjm94= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478337; c=relaxed/simple; bh=7YdM26j1jMpKxr6XHqzYpzezpp/C8Ve3E5DloVJy/gU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fdLsRC+d78Ums+ExjhLhpGZgrpfgEa9lQJjc3wnJauzidpr+yR+qOqQBcohFYnD600Es8ppoGYOWcilsjJMW0ezcFr1jw2DXi7+6foZTU/oRyUeakSpi/MDtLVYSFXGDUJNnzu2YSeKen5rsc+hg740vVH8cngp6bCbV58cNfVo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=LsQfYygc; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="LsQfYygc" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F2912C4CECD; Wed, 9 Oct 2024 12:52:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1728478337; bh=7YdM26j1jMpKxr6XHqzYpzezpp/C8Ve3E5DloVJy/gU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LsQfYygcSmoCwuA694YgKY2dSbaETofQTA9rENoTfNJfSQScWWElGpPKsPl3OwWXh rDfFghdnHcF3wY6GTRk1LATMXyq18G2VcK0EhuyZ+GjQtcewLJWfMMGiHAAd708xYt aQGctj6p+EbKNxKhEVag0iGIFjk9/ufr6dfPvfuM8QbMH7j+lghTthCslEc3Lr4dsV SayLRihHihgbiLqkuwDHDgSM8FlF/ZsyYWxyPaoLVWGxZieqsa+kDesMdtgQdeb1Ws OvszKRpL1DPmkklFEeH33cxs2vHV5MtviN81iYDRV1voR1KVPmH35SFgGSQTqylW2G FsIdok45w/dvA== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, paulmck@kernel.org, joel@joelfernandes.org, frederic@kernel.org, boqun.feng@gmail.com, urezki@gmail.com, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, qiang.zhang1211@gmail.com, peterz@infradead.org, neeraj.upadhyay@amd.com, Neeraj Upadhyay Subject: [PATCH v2 07/10] rcu/tasks: Check RCU watching state for holdout idle injection tasks Date: Wed, 9 Oct 2024 18:21:24 +0530 Message-Id: <20241009125127.18902-8-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> References: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Neeraj Upadhyay Use RCU watching state of a CPU to check whether RCU-tasks GP need to wait for idle injection task on that CPU. Idle injection tasks which are in deep-idle states where RCU is not watching or which have transitioned to/from deep-idle state do not block RCU-tasks grace period. Signed-off-by: Neeraj Upadhyay --- kernel/rcu/tasks.h | 63 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 15 deletions(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index d8506d2e6f54..1947f9b6346d 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -38,6 +38,8 @@ typedef void (*postgp_func_t)(struct rcu_tasks *rtp); * @rtpp: Pointer to the rcu_tasks structure. * @rcu_watching_snap: Per-GP RCU-watching snapshot for idle tasks. * @rcu_watching_snap_rec: RCU-watching snapshot recorded for idle task. + * @rcu_watching_idle_inj_snap: Per-GP RCU-watching snapshot for idle inject task. + * @rcu_watching_idle_inj_rec: RCU-watching snapshot recorded for idle inject task. */ struct rcu_tasks_percpu { struct rcu_segcblist cblist; @@ -56,6 +58,8 @@ struct rcu_tasks_percpu { struct rcu_tasks *rtpp; int rcu_watching_snap; bool rcu_watching_snap_rec; + int rcu_watching_idle_inj_snap; + bool rcu_watching_idle_inj_rec; }; /** @@ -989,10 +993,34 @@ static void rcu_tasks_pregp_step(struct list_head *hop) struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu); rtpcp->rcu_watching_snap_rec = false; + rtpcp->rcu_watching_idle_inj_rec = false; } } #ifdef CONFIG_SMP +static bool rcu_idle_check_rcu_watching(int *rcu_watching_snap, bool *rcu_watching_rec, int cpu) +{ + if (!*rcu_watching_rec) { + /* + * Do plain access. Ordering between remote CPU's pre idle accesses + * and post rcu-tasks grace period is provided by synchronize_rcu() + * in rcu_tasks_postgp(). + */ + *rcu_watching_snap = ct_rcu_watching_cpu(cpu); + *rcu_watching_rec = true; + if (rcu_watching_snap_in_eqs(*rcu_watching_snap)) + /* + * RCU-idle contexts are RCU-tasks quiescent state for idle + * (and idle injection) tasks. + */ + return false; + } else if (rcu_watching_snap_stopped_since(cpu, *rcu_watching_snap)) { + return false; + } + + return true; +} + static bool rcu_idle_task_is_holdout(struct task_struct *t, int cpu) { struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu); @@ -1010,22 +1038,16 @@ static bool rcu_idle_task_is_holdout(struct task_struct *t, int cpu) if (!t->on_cpu) return false; - if (!rtpcp->rcu_watching_snap_rec) { - /* - * Do plain access. Ordering between remote CPU's pre idle accesses - * and post rcu-tasks grace period is provided by synchronize_rcu() - * in rcu_tasks_postgp(). - */ - rtpcp->rcu_watching_snap = ct_rcu_watching_cpu(cpu); - rtpcp->rcu_watching_snap_rec = true; - /* RCU-idle contexts are RCU-tasks quiescent state for idle tasks. */ - if (rcu_watching_snap_in_eqs(rtpcp->rcu_watching_snap)) - return false; - } else if (rcu_watching_snap_stopped_since(cpu, rtpcp->rcu_watching_snap)) { - return false; - } + return rcu_idle_check_rcu_watching(&rtpcp->rcu_watching_snap, + &rtpcp->rcu_watching_snap_rec, cpu); +} - return true; +static bool rcu_idle_inj_is_holdout(struct task_struct *t, int cpu) +{ + struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu); + + return rcu_idle_check_rcu_watching(&rtpcp->rcu_watching_idle_inj_snap, + &rtpcp->rcu_watching_idle_inj_rec, cpu); } #else /* #ifdef CONFIG_SMP */ static inline bool rcu_idle_task_is_holdout(struct task_struct *t, int cpu) @@ -1037,6 +1059,15 @@ static inline bool rcu_idle_task_is_holdout(struct task_struct *t, int cpu) */ return false; } + +static inline bool rcu_idle_inj_is_holdout(struct task_struct *t, int cpu) +{ + /* + * Idle injection tasks are PF_IDLE within preempt disabled + * region. So, we should not enter this call for !SMP. + */ + return false; +} #endif /* Check for quiescent states since the pregp's synchronize_rcu() */ @@ -1060,6 +1091,8 @@ static bool rcu_tasks_is_holdout(struct task_struct *t) if (t == idle_task(cpu)) return rcu_idle_task_is_holdout(t, cpu); + else if (is_idle_task(t)) + return rcu_idle_inj_is_holdout(t, cpu); return true; } From patchwork Wed Oct 9 12:51:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13828404 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 803861E25E5; Wed, 9 Oct 2024 12:52:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478342; cv=none; b=T64z/vE6xyca2K8zJBSMjnkq75Rsui5MewutdWMYbK4RljFzw4DNshDjoRyXk5IAI6LdDqxl4N9HY+PwABUIWUelOWs3+e4LyMlWyKqQstrfenvj6lJTzOhgPvgjU5LnIEye8Fy0uNfLu6L/36H1xSOKMbJMVghaiLCYkXtL/C4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478342; c=relaxed/simple; bh=5GHN3TQydO4+DYblsifPfmO2C1JXUc1tJrJ11vf8hJw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=IsSfT9vRz8dv1H83dt29pvVRj59b54PvUaiemPSi+Fz1MCbswkuM9gdxnBKNWbAvgmtzbVtzpKWhJSIWPL97qFEe3C7wCn65FahPdQRz2JKEB3NE0z7sOTNbJV+Kpm9NFwZQHXt4XNLZyiCsKqNNC5eF8PsA07b8H1TymbybBm0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=uaGKqEeF; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="uaGKqEeF" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CB86DC4CED1; Wed, 9 Oct 2024 12:52:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1728478342; bh=5GHN3TQydO4+DYblsifPfmO2C1JXUc1tJrJ11vf8hJw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uaGKqEeFN0PwyFg/+8hX9ylsmGNk289GmEKjA1KNrlB6ZRKGO+vWPuHHyqzJmaCf0 OrxeZjwjhLoEibjLP0JKyTanXAxNvbMadz6cd6FVH5eNkqDdxrqdNF5Gw56qsvHDkO k3fVN0xnaNdBJ24a5+GDLuiu0n6ItV8ayBKcae7GJNilks2W1xlIgXPSTfWbRedF7C lWxoxSkTL0EpXubTWHdVhqBUWoNn8G82jEY4Yi/RbZa1F9qnvreNuasqZzUFLalvU2 vSYOVrWgyY/Oj2lHQ62EKMwaKroSQcN/5cHMNbPj9b75Q2vn4NeoF58k11QiASISe9 RPjK9XhHmVfxQ== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, paulmck@kernel.org, joel@joelfernandes.org, frederic@kernel.org, boqun.feng@gmail.com, urezki@gmail.com, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, qiang.zhang1211@gmail.com, peterz@infradead.org, neeraj.upadhyay@amd.com, Neeraj Upadhyay Subject: [PATCH v2 08/10] rcu/tasks: Make RCU-tasks pay attention to idle tasks Date: Wed, 9 Oct 2024 18:21:25 +0530 Message-Id: <20241009125127.18902-9-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> References: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Neeraj Upadhyay Currently, idle tasks are ignored by RCU-tasks. Change this to start paying attention to idle tasks except in deep-idle functions where RCU is not watching. With this, for architectures where kernel entry/exit and deep-idle functions have been properly tagged noinstr, Tasks Rude RCU can be disabled. [ neeraj.upadhyay: Frederic Weisbecker and Paul E. McKenney feedback. ] Signed-off-by: Neeraj Upadhyay --- .../RCU/Design/Requirements/Requirements.rst | 12 +++--- kernel/rcu/tasks.h | 41 ++++++++----------- 2 files changed, 24 insertions(+), 29 deletions(-) diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Documentation/RCU/Design/Requirements/Requirements.rst index 6125e7068d2c..5016b85d53d7 100644 --- a/Documentation/RCU/Design/Requirements/Requirements.rst +++ b/Documentation/RCU/Design/Requirements/Requirements.rst @@ -2611,8 +2611,8 @@ critical sections that are delimited by voluntary context switches, that is, calls to schedule(), cond_resched(), and synchronize_rcu_tasks(). In addition, transitions to and from userspace execution also delimit tasks-RCU read-side critical sections. -Idle tasks are ignored by Tasks RCU, and Tasks Rude RCU may be used to -interact with them. +Idle tasks which are idle from RCU's perspective are ignored by Tasks RCU, +and Tasks Rude RCU may be used to interact with them. Note well that involuntary context switches are *not* Tasks-RCU quiescent states. After all, in preemptible kernels, a task executing code in a @@ -2643,10 +2643,10 @@ moniker. And this operation is considered to be quite rude by real-time workloads that don't want their ``nohz_full`` CPUs receiving IPIs and by battery-powered systems that don't want their idle CPUs to be awakened. -Once kernel entry/exit and deep-idle functions have been properly tagged -``noinstr``, Tasks RCU can start paying attention to idle tasks (except -those that are idle from RCU's perspective) and then Tasks Rude RCU can -be removed from the kernel. +As Tasks RCU now pays attention to idle tasks (except those that are idle +from RCU's perspective), once kernel entry/exit and deep-idle functions have +been properly tagged ``noinstr``, Tasks Rude RCU can be removed from the +kernel. The tasks-rude-RCU API is also reader-marking-free and thus quite compact, consisting solely of synchronize_rcu_tasks_rude(). diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 1947f9b6346d..72dc0d0a4a8f 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -912,14 +912,15 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) //////////////////////////////////////////////////////////////////////// // // Simple variant of RCU whose quiescent states are voluntary context -// switch, cond_resched_tasks_rcu_qs(), user-space execution, and idle. -// As such, grace periods can take one good long time. There are no -// read-side primitives similar to rcu_read_lock() and rcu_read_unlock() -// because this implementation is intended to get the system into a safe -// state for some of the manipulations involved in tracing and the like. -// Finally, this implementation does not support high call_rcu_tasks() -// rates from multiple CPUs. If this is required, per-CPU callback lists -// will be needed. +// switch, cond_resched_tasks_rcu_qs(), user-space execution, and idle +// tasks which are in RCU-idle context. As such, grace periods can take +// one good long time. There are no read-side primitives similar to +// rcu_read_lock() and rcu_read_unlock() because this implementation is +// intended to get the system into a safe state for some of the +// manipulations involved in tracing and the like. Finally, this +// implementation does not support high call_rcu_tasks() rates from +// multiple CPUs. If this is required, per-CPU callback lists will be +// needed. // // The implementation uses rcu_tasks_wait_gp(), which relies on function // pointers in the rcu_tasks structure. The rcu_spawn_tasks_kthread() @@ -1079,14 +1080,6 @@ static bool rcu_tasks_is_holdout(struct task_struct *t) if (!READ_ONCE(t->on_rq)) return false; - /* - * Idle tasks (or idle injection) within the idle loop are RCU-tasks - * quiescent states. But CPU boot code performed by the idle task - * isn't a quiescent state. - */ - if (is_idle_task(t)) - return false; - cpu = task_cpu(t); if (t == idle_task(cpu)) @@ -1265,11 +1258,12 @@ static void tasks_rcu_exit_srcu_stall(struct timer_list *unused) * period elapses, in other words after all currently executing RCU * read-side critical sections have completed. call_rcu_tasks() assumes * that the read-side critical sections end at a voluntary context - * switch (not a preemption!), cond_resched_tasks_rcu_qs(), entry into idle, - * or transition to usermode execution. As such, there are no read-side - * primitives analogous to rcu_read_lock() and rcu_read_unlock() because - * this primitive is intended to determine that all tasks have passed - * through a safe state, not so much for data-structure synchronization. + * switch (not a preemption!), cond_resched_tasks_rcu_qs(), entry into + * RCU-idle context or transition to usermode execution. As such, there + * are no read-side primitives analogous to rcu_read_lock() and + * rcu_read_unlock() because this primitive is intended to determine + * that all tasks have passed through a safe state, not so much for + * data-structure synchronization. * * See the description of call_rcu() for more detailed information on * memory ordering guarantees. @@ -1287,8 +1281,9 @@ EXPORT_SYMBOL_GPL(call_rcu_tasks); * grace period has elapsed, in other words after all currently * executing rcu-tasks read-side critical sections have elapsed. These * read-side critical sections are delimited by calls to schedule(), - * cond_resched_tasks_rcu_qs(), idle execution, userspace execution, calls - * to synchronize_rcu_tasks(), and (in theory, anyway) cond_resched(). + * cond_resched_tasks_rcu_qs(), idle execution within RCU-idle context, + * userspace execution, calls to synchronize_rcu_tasks(), and (in theory, + * anyway) cond_resched(). * * This is a very specialized primitive, intended only for a few uses in * tracing and other situations requiring manipulation of function From patchwork Wed Oct 9 12:51:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13828405 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 278D01A2C19; Wed, 9 Oct 2024 12:52:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478347; cv=none; b=m7rQj2jeSCQJ+8zcoRH3uqxW7k8FpEh51MehQCnEExYhpk8HJjF5F4/QPx+1lwrW/k/bjbsffFDmK9tw8E9r8TMIUMFuIdWN4yLtXVSsKXJ3ikoEddQX/3M7TtWWYu38Z7XnW9KfVOPOZKr8pZ5WihvMv8S/VZrSDEggAjvA5Mo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478347; c=relaxed/simple; bh=lOXgRfOyLIl7elkNZsxeg0qekVV+z+V29DqLUPqFN78=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Wkanq1kYIY1cQHh5qkw7+5Qxmk0w3I8yCQhumXfeg73wA4vsE1d4VVc7Gr2YcbggtRy7ZayO+vFpsd+Hr5SJdna/Lqze3S8g4esPom1uBE1l0KiGi8MHnS4YRXcQoN9cYOgrAspt63LuVngb0xSQcMsKfGN+jXAvU8xl/Nw7gpg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=K2ZxVGxf; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="K2ZxVGxf" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A9566C4CEC5; Wed, 9 Oct 2024 12:52:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1728478346; bh=lOXgRfOyLIl7elkNZsxeg0qekVV+z+V29DqLUPqFN78=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=K2ZxVGxfr/4AK7Agl9MI+e69zrPCZb4KtzRhkDgzTrYKQ3K5Z1h9lZNmmfZx08Feo Vm49v6S2GgwaMl/SHf9zmG218tjoeJiodloI8duuCmI2Z55Lj00eZCl6L4w5hD3eMO /wEzKBGJmkg5vCNpvJN2yZY6UG4yOYcAfr0WJl7q9FzwBP2XDNRK2jly2PU0Ewfu8E GY/ss0KIeBs1We3lEfNwkuiaRLUEg6axGD+kSNsBwQyKBhG9vEum7C8iQcUTPrjA81 Klhth8uZ+zPjjBQeewcRFR/la0H2KZxtXjV1USkbXZyhtpuoLSgeWME8epe893eLj2 AbXNuCDN62hkw== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, paulmck@kernel.org, joel@joelfernandes.org, frederic@kernel.org, boqun.feng@gmail.com, urezki@gmail.com, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, qiang.zhang1211@gmail.com, peterz@infradead.org, neeraj.upadhyay@amd.com, Neeraj Upadhyay Subject: [PATCH v2 09/10] context_tracking: Invoke RCU-tasks enter/exit for NMI context Date: Wed, 9 Oct 2024 18:21:26 +0530 Message-Id: <20241009125127.18902-10-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> References: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Neeraj Upadhyay rcu_task_enter() and rcu_task_exit() are not called on NMI entry and exit. So, Tasks-RCU-Rude grace period wait is required to ensure that NMI handlers have entered/exited into Tasks-RCU eqs. For architectures which do not require Tasks-RCU-Rude (as the code sections where RCU is not watching are marked as noinstr), when those architectures switch to not using Tasks-RCU-Rude, NMI handlers task exit to eqs will need to be handled correctly for Tasks-RCU holdout tasks running on nohz_full CPUs. As it is safe to call these two functions from NMI context, remove the in_nmi() check. This ensures that RCU-tasks entry/exit is marked correctly for NMI handlers. With this check removed, all callers of ct_kernel_exit_state() and ct_kernel_enter_state() now also call rcu_task_exit() and rcu_task_enter() respectively. So, fold rcu_task_exit() and rcu_task_entry() calls into ct_kernel_exit_state() and ct_kernel_enter_state(). Reported-by: Frederic Weisbecker Suggested-by: Frederic Weisbecker Suggested-by: "Paul E. McKenney" Reviewed-by: Paul E. McKenney Signed-off-by: Neeraj Upadhyay --- kernel/context_tracking.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 938c48952d26..85ced563af23 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -91,6 +91,7 @@ static noinstr void ct_kernel_exit_state(int offset) seq = ct_state_inc(offset); // RCU is no longer watching. Better be in extended quiescent state! WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && (seq & CT_RCU_WATCHING)); + rcu_task_exit(); } /* @@ -102,6 +103,8 @@ static noinstr void ct_kernel_enter_state(int offset) { int seq; + rcu_task_enter(); + /* * CPUs seeing atomic_add_return() must see prior idle sojourns, * and we also must force ordering with the next RCU read-side @@ -149,7 +152,6 @@ static void noinstr ct_kernel_exit(bool user, int offset) // RCU is watching here ... ct_kernel_exit_state(offset); // ... but is no longer watching here. - rcu_task_exit(); } /* @@ -173,7 +175,6 @@ static void noinstr ct_kernel_enter(bool user, int offset) ct->nesting++; return; } - rcu_task_enter(); // RCU is not watching here ... ct_kernel_enter_state(offset); // ... but is watching here. @@ -238,9 +239,6 @@ void noinstr ct_nmi_exit(void) // RCU is watching here ... ct_kernel_exit_state(CT_RCU_WATCHING); // ... but is no longer watching here. - - if (!in_nmi()) - rcu_task_exit(); } /** @@ -273,9 +271,6 @@ void noinstr ct_nmi_enter(void) */ if (!rcu_is_watching_curr_cpu()) { - if (!in_nmi()) - rcu_task_enter(); - // RCU is not watching here ... ct_kernel_enter_state(CT_RCU_WATCHING); // ... but is watching here. From patchwork Wed Oct 9 12:51:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13828406 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D597B198E77; Wed, 9 Oct 2024 12:52:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478351; cv=none; b=FuPwUkzR2akrVa9oUxcxTbczpZCcCN3XyiuVSYtUKTwMymftbRngB64bIso5J+TtNCmOgOzbEPFYABVBPX+OBjKEtXPRR2S3GBNPVwjLxhJ0Y6WULZf0tXfVNpnCe1ndsqTRBGhSI/VWtUV6uXwuNSMMRWIFVdFti6TCYMGUF+0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728478351; c=relaxed/simple; bh=PZBlSy5cYNFCTG1gkW22h6MA1qut/IZhj71pdF1EkhM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=olf2DVoLxsrDumPmEEnPgPLZka6JgjjkCSgwfES3Koe//328Bp71PytCh74EPEmFKhTV1jXKgfKgWfyZUZUjoLbOOeqYfOcaZTyPqCJSLuM/z6w4qNygQgScCwEEYcYFx2fhoQccMILxlOjPDQAwQVkJlXNfmDggNQT0lFJLnwk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=nbj//0A6; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="nbj//0A6" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5B308C4CEC5; Wed, 9 Oct 2024 12:52:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1728478351; bh=PZBlSy5cYNFCTG1gkW22h6MA1qut/IZhj71pdF1EkhM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nbj//0A6W8qw2wR2SD8dkda/79mSFO+6Kll+nXGcS4nfWS27I6DMMTV4cHlUZFc+V NEd+osPaQJsEYkQgJ7xsr0sZzksNiKwp0wls4gagar7ryhqmPRSb7sPn9wHhPrCGtf pU/Z0s/PwLdwkgxf/1Kq6kiQY+f6RB7/MXfuicbJ1Hp0f+F2KgqFCtsd5w9i1yZ28U CGI+E8iU3iHwLe+1PaFSqMZKTetXsOsB+JjVz2uWzHjDIZb8TvqsO0XYD9ipv/Xar9 /UbdpcokvvoGlgiNZ/Ae7C++1dh1dxfbLFrQW9wSVb5pXi1QoMMnjAyr/FlMEOPIxx tpk8WjT5Q9qug== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, paulmck@kernel.org, joel@joelfernandes.org, frederic@kernel.org, boqun.feng@gmail.com, urezki@gmail.com, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, qiang.zhang1211@gmail.com, peterz@infradead.org, neeraj.upadhyay@amd.com, Neeraj Upadhyay Subject: [PATCH v2 10/10] rcu: Allow short-circuiting of synchronize_rcu_tasks_rude() Date: Wed, 9 Oct 2024 18:21:27 +0530 Message-Id: <20241009125127.18902-11-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> References: <20241009125127.18902-1-neeraj.upadhyay@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" There are now architectures for which all deep-idle and entry-exit functions are properly inlined or marked noinstr. Such architectures do not need synchronize_rcu_tasks_rude(), or will not once RCU Tasks has been modified to pay attention to idle tasks. This commit therefore allows a CONFIG_ARCH_HAS_NOINSTR_MARKINGS Kconfig option to turn synchronize_rcu_tasks_rude() into a no-op. To facilitate testing, kernels built by rcutorture scripting will enable RCU Tasks Trace even on systems that do not need it. [ paulmck: Apply Peter Zijlstra feedback. ] Signed-off-by: Paul E. McKenney Cc: Peter Zijlstra Signed-off-by: Neeraj Upadhyay --- kernel/rcu/tasks.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 72dc0d0a4a8f..ef9de6b91a3d 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1485,7 +1485,8 @@ static void call_rcu_tasks_rude(struct rcu_head *rhp, rcu_callback_t func) */ void synchronize_rcu_tasks_rude(void) { - synchronize_rcu_tasks_generic(&rcu_tasks_rude); + if (!IS_ENABLED(CONFIG_ARCH_WANTS_NO_INSTR) || IS_ENABLED(CONFIG_FORCE_TASKS_RUDE_RCU)) + synchronize_rcu_tasks_generic(&rcu_tasks_rude); } EXPORT_SYMBOL_GPL(synchronize_rcu_tasks_rude);