From patchwork Tue Jun 4 22:26:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13685928 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 62B6112DDAF; Tue, 4 Jun 2024 22:26:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717540014; cv=none; b=efTsUEbi8UYw8zo3UvyKKJ+Wpte4R1cN+qbfLPCEteWEBcqZYoh/uSKKzgWFlFaKkd+rSdLoD5Kg7Zekdzpu/52bEv2DIsRkF7l5bjP0pfMYlZLEE+dnNRNtxmDjWLWqje609CPZhBPMLdQdHlEWbRxwbSNkIMmO0MjEa9WRV54= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717540014; c=relaxed/simple; bh=qkbOPPNcOof4xO9uLKIhmjR/JL9cMPc3uYhJKaAInxI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NMefKAOQGOCd9rJHyUFDptR7g4OQ7WuriZR1YMSWzVNXy67PY2SfPxP2RdUitgDrtQC4+cuaGnhYukkabG8ft1UTqeG6FRc/2c4/79Sk3TpqfnmXkNfag8umV8+BtDYXy2sJeCDGvdd97Ykdwu8nASGQMW6ngrqjRxHsqHxIL0k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=JcUfXmE8; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="JcUfXmE8" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3C3D8C2BBFC; Tue, 4 Jun 2024 22:26:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717540014; bh=qkbOPPNcOof4xO9uLKIhmjR/JL9cMPc3uYhJKaAInxI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JcUfXmE8dG7jeKn0l9eO+lD7Um6W3MenxheD2/QGTki7aKasEey1tETS9C2OXQ6Ln p4Vs2hNkNj/I8IJoAf9W/PbbqplwZRc5M7ZqK4vwgFyzhFcmj6RgsU449jZNlaK1Vv N+u87Bfkz7Orz01PdXfVYJUxuv6guSuDHbQpnv3y9Uo9+p7EEivA4wavrDd6NY5D0b Uxl+0GzrvVn+Gz4nefTybZzNbc1cCTYHY6QaeS1eYBVQb0v74ZfxuhLdURUDkt4syn /Zxf1dqWUN6lRmoD5eCRhfzSrk8fbU1soBY9BbLPpGnHziKndbY7XVqRlUFonCCm0E Xw6UmtUgYTAxQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id EB63FCE3ED6; Tue, 4 Jun 2024 15:26:53 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH rcu 1/6] rcu: Remove full ordering on second EQS snapshot Date: Tue, 4 Jun 2024 15:26:47 -0700 Message-Id: <20240604222652.2370998-1-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <81f4e616-8125-4934-a8e2-fd5beae90995@paulmck-laptop> References: <81f4e616-8125-4934-a8e2-fd5beae90995@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Frederic Weisbecker When the grace period kthread checks the extended quiescent state counter of a CPU, full ordering is necessary to ensure that either: * If the GP kthread observes the remote target in an extended quiescent state, then that target must observe all accesses prior to the current grace period, including the current grace period sequence number, once it exits that extended quiescent state. Also the GP kthread must observe all accesses performed by the target prior it entering in EQS. or: * If the GP kthread observes the remote target NOT in an extended quiescent state, then the target further entering in an extended quiescent state must observe all accesses prior to the current grace period, including the current grace period sequence number, once it enters that extended quiescent state. Also the GP kthread later observing that EQS must also observe all accesses performed by the target prior it entering in EQS. This ordering is explicitly performed both on the first EQS snapshot and on the second one as well through the combination of a preceding full barrier followed by an acquire read. However the second snapshot's full memory barrier is redundant and not needed to enforce the above guarantees: GP kthread Remote target ---- ----- // Access prior GP WRITE_ONCE(A, 1) // first snapshot smp_mb() x = smp_load_acquire(EQS) // Access prior GP WRITE_ONCE(B, 1) // EQS enter // implied full barrier by atomic_add_return() atomic_add_return(RCU_DYNTICKS_IDX, EQS) // implied full barrier by atomic_add_return() READ_ONCE(A) // second snapshot y = smp_load_acquire(EQS) z = READ_ONCE(B) If the GP kthread above fails to observe the remote target in EQS (x not in EQS), the remote target will observe A == 1 after further entering in EQS. Then the second snapshot taken by the GP kthread only need to be an acquire read in order to observe z == 1. Therefore remove the needless full memory barrier on second snapshot. Signed-off-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 28c7031711a3f..f07b8bff4621b 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -321,7 +321,7 @@ static bool rcu_dynticks_in_eqs(int snap) */ static bool rcu_dynticks_in_eqs_since(struct rcu_data *rdp, int snap) { - return snap != rcu_dynticks_snap(rdp->cpu); + return snap != ct_dynticks_cpu_acquire(rdp->cpu); } /* From patchwork Tue Jun 4 22:26:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13685929 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6F1714B07B; Tue, 4 Jun 2024 22:26:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717540014; cv=none; b=MwZ1GWg/gf1ZrfphJ0ceSlsRVa6L/CcldBZc1P6LIpqLuuOry37T5YLbPL+2rpvvBuIaioYqidTMgPg0v+0MHQBPPMjGzfs180zx2syg0U/d7LrUm7S5H297bG2g7p0LWCO8NiA9mQbywtVeDf/WgfwLFzSifWvMvIkxt5vAdq8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717540014; c=relaxed/simple; bh=eTYCEt4S09cbBIvwTWJRKuNdHno4GV8Da6nEtMiRHjQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Z5/FdJT2z31+NJ53focm491XWppSkokGQjKEmWXjZhi0ReGHvBLtdoelMTyigzcTOjvwUKvNvXH4FXe0fLg7ick8D+HEqpiceYrKa9yniVe63Q9b9G2O1Mt9j0B2bnzGbvoNWIM59zHcv/2RLvhCc1484Hbe8MacDJQlZ3lyEhM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=uAWo6mlb; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="uAWo6mlb" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4BED1C3277B; Tue, 4 Jun 2024 22:26:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717540014; bh=eTYCEt4S09cbBIvwTWJRKuNdHno4GV8Da6nEtMiRHjQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uAWo6mlbDhfzDWNpWWScNcWSR5t5VJjBGdLPbUTLhzKQPUB6xgnFv3Pkw2fjANkGf KBnEd95v9S+5oS4dmvUbGK6pRJh7Ud6ULW95udpU7SNUzrK7Tg3DQna6pG50RxRnr8 TVMLO7k9kCM5EJAI/zZZsgEFoAv60N+lFufHoJ7Gftcuqas5i+Wf0wK/pwpruv7v9+ n5naJIYMOBb+xMVqkVn1bSnsjDcpeH0/d1Y3C7HL/B1x482eGpkomdnV2KZiO/aHtb 5AosWwwxV+cs3avkEbCD1V2J5vwJ1yEDLeyVN5b3e+q6whrkWPzIA1ND1mu9gvo2rg 2rMFN2HCt1AXQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id EE455CE3F0F; Tue, 4 Jun 2024 15:26:53 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH rcu 2/6] rcu: Remove superfluous full memory barrier upon first EQS snapshot Date: Tue, 4 Jun 2024 15:26:48 -0700 Message-Id: <20240604222652.2370998-2-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <81f4e616-8125-4934-a8e2-fd5beae90995@paulmck-laptop> References: <81f4e616-8125-4934-a8e2-fd5beae90995@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Frederic Weisbecker When the grace period kthread checks the extended quiescent state counter of a CPU, full ordering is necessary to ensure that either: * If the GP kthread observes the remote target in an extended quiescent state, then that target must observe all accesses prior to the current grace period, including the current grace period sequence number, once it exits that extended quiescent state. or: * If the GP kthread observes the remote target NOT in an extended quiescent state, then the target further entering in an extended quiescent state must observe all accesses prior to the current grace period, including the current grace period sequence number, once it enters that extended quiescent state. This ordering is enforced through a full memory barrier placed right before taking the first EQS snapshot. However this is superfluous because the snapshot is taken while holding the target's rnp lock which provides the necessary ordering through its chain of smp_mb__after_unlock_lock(). Remove the needless explicit barrier before the snapshot and put a comment about the implicit barrier newly relied upon here. Signed-off-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- .../Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst | 6 +++--- kernel/rcu/tree.c | 7 ++++++- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst b/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst index 5750f125361b0..728b1e690c646 100644 --- a/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst +++ b/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst @@ -149,9 +149,9 @@ This case is handled by calls to the strongly ordered ``atomic_add_return()`` read-modify-write atomic operation that is invoked within ``rcu_dynticks_eqs_enter()`` at idle-entry time and within ``rcu_dynticks_eqs_exit()`` at idle-exit time. -The grace-period kthread invokes ``rcu_dynticks_snap()`` and -``rcu_dynticks_in_eqs_since()`` (both of which invoke -an ``atomic_add_return()`` of zero) to detect idle CPUs. +The grace-period kthread invokes first ``ct_dynticks_cpu_acquire()`` +(preceded by a full memory barrier) and ``rcu_dynticks_in_eqs_since()`` +(both of which rely on acquire semantics) to detect idle CPUs. +-----------------------------------------------------------------------+ | **Quick Quiz**: | diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index f07b8bff4621b..1a6ef9c5c949e 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -769,7 +769,12 @@ static void rcu_gpnum_ovf(struct rcu_node *rnp, struct rcu_data *rdp) */ static int dyntick_save_progress_counter(struct rcu_data *rdp) { - rdp->dynticks_snap = rcu_dynticks_snap(rdp->cpu); + /* + * Full ordering against accesses prior current GP and also against + * current GP sequence number is enforced by current rnp locking + * with chained smp_mb__after_unlock_lock(). + */ + rdp->dynticks_snap = ct_dynticks_cpu_acquire(rdp->cpu); if (rcu_dynticks_in_eqs(rdp->dynticks_snap)) { trace_rcu_fqs(rcu_state.name, rdp->gp_seq, rdp->cpu, TPS("dti")); rcu_gpnum_ovf(rdp->mynode, rdp); From patchwork Tue Jun 4 22:26:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13685930 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6F6414B092; Tue, 4 Jun 2024 22:26:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717540014; cv=none; b=nzVjG/877BLyfo6Y6U7w3KVgGFR/Dl9OOwirqxEtZDjMJRfXxRCwmsqanLontfIYJFJm6PyV9YtQu2THlBFxfN8uhUnXB2UgC+O9tEpVhrCtWbFtRdGNeDZYRejq1WjGj4kTm+iUOjpj1aUUs2C57AonrV047EzI0Pz5/fHdZCg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717540014; c=relaxed/simple; bh=TioFm82Snd8j9pH8jLZE3PHQ5qk2yV0osr+xyGd7g+c=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=atJgD3z4OlRkCA1NYNDOGafuHOtIjFTj3QCKMF9aOwmFf5LhTk7H93IpIOkw21RvXYvnOd/ZKTSCO7qYv4GJMMA7jhwsV8pzejG8jYiv2alKejhaBayzNrK33xtctaciiogbQ6rrwGM4CwEWiD6j407pFd52ym+U0vMr+j3dCkY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=sGai/H0a; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="sGai/H0a" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5631BC4AF08; Tue, 4 Jun 2024 22:26:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717540014; bh=TioFm82Snd8j9pH8jLZE3PHQ5qk2yV0osr+xyGd7g+c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sGai/H0ahb0euEOElAvwYeBDGe9Jz07a85uy4SPB+l6ilTdewzdtN5Ae4hLz4OI+j I2GqRslKomd29+Tmkzt6rrykXqrevKlx5MJnvHL5bOROOoBT8k4kBeGGv8FCJgF3OG YBFxN5on/nWq/stc5hGRnSuOhvyl1YQDKE/VcgYOIbxfqbWQIN+kYk51IYMcCYvkg3 Y8k4H6jedHeqI3RYTeXCltZM6pwzXrla3xfHIo8jHnCjgCrAsnHc+Kg5A1i6tNJ5jw zNoGl1Gk27gYNYooerRsbrmx909YHi1YeiXdB994wWE8uY3VqchImLi18fEtViVyUG TodZ+5mr+0Ixg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id F0EB8CE3F26; Tue, 4 Jun 2024 15:26:53 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH rcu 3/6] rcu/exp: Remove superfluous full memory barrier upon first EQS snapshot Date: Tue, 4 Jun 2024 15:26:49 -0700 Message-Id: <20240604222652.2370998-3-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <81f4e616-8125-4934-a8e2-fd5beae90995@paulmck-laptop> References: <81f4e616-8125-4934-a8e2-fd5beae90995@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Frederic Weisbecker When the grace period kthread checks the extended quiescent state counter of a CPU, full ordering is necessary to ensure that either: * If the GP kthread observes the remote target in an extended quiescent state, then that target must observe all accesses prior to the current grace period, including the current grace period sequence number, once it exits that extended quiescent state. or: * If the GP kthread observes the remote target NOT in an extended quiescent state, then the target further entering in an extended quiescent state must observe all accesses prior to the current grace period, including the current grace period sequence number, once it enters that extended quiescent state. This ordering is enforced through a full memory barrier placed right before taking the first EQS snapshot. However this is superfluous because the snapshot is taken while holding the target's rnp lock which provides the necessary ordering through its chain of smp_mb__after_unlock_lock(). Remove the needless explicit barrier before the snapshot and put a comment about the implicit barrier newly relied upon here. Signed-off-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_exp.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 8a1d9c8bd9f74..bec24ea6777e8 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -357,7 +357,13 @@ static void __sync_rcu_exp_select_node_cpus(struct rcu_exp_work *rewp) !(rnp->qsmaskinitnext & mask)) { mask_ofl_test |= mask; } else { - snap = rcu_dynticks_snap(cpu); + /* + * Full ordering against accesses prior current GP and + * also against current GP sequence number is enforced + * by current rnp locking with chained + * smp_mb__after_unlock_lock(). + */ + snap = ct_dynticks_cpu_acquire(cpu); if (rcu_dynticks_in_eqs(snap)) mask_ofl_test |= mask; else From patchwork Tue Jun 4 22:26:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13685932 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AAD1B14C5AE; Tue, 4 Jun 2024 22:26:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717540014; cv=none; b=BEdccXzWt5vkLQqCLAklctOeNOFNGZVBPyP+I/fMr+/XFqcL+XGwUTbbYv57noxkd3NC21Z4ovfCQQeT65vSYHQS1L4GT7KweTQpX0fr0qXsJuo3JyO4ZULSa3PH0ulj5GmbM6AH7TfZVaJAPldL7mXs8FQ7QybSo/oFIAXQ6lE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717540014; c=relaxed/simple; bh=i403K4lRwrOOtPyPhCFF+YfWbFsrii2TF1tAnWab5XE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tv0AAebLQllHb69+UwbQf/HJkBr7fYh56jN24luCmUxr901aa/198GRELebD+Qyqqx8jSy/lNHqz6Q+6txpmso9TCvr65plGfreyp707hOZfSSCBt50c1/bxRXm75GHCRaosctiYk0fpxG7aiYqxZobJIiqABfGQSS9mwzgzoJ8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=dPd4guF0; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="dPd4guF0" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 60B62C4AF0B; Tue, 4 Jun 2024 22:26:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717540014; bh=i403K4lRwrOOtPyPhCFF+YfWbFsrii2TF1tAnWab5XE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dPd4guF0RRmMpV+EL05Bi66m27gUk5t/z4GZ848oqnXCHYRdc+dql67eEOFW51XD9 AphaFbKSiXDlK15IsJmsO39QzweI4OvKMKuC/5OQjHKGndgmoSIfDxurzR514D4d22 7pF9kApYJvuB0KU3aF9LaiqxrLeg+dQ94DxARkj7o4rvZ49Zv22uYAB2krlr6rN/b/ qPex9HHIAxZ8gv2383gypqCwenKxslp5H711gIrz9sy8T+39h8Q5Q5Y6j3EVJPVXm7 strtTgJfbToIak0svQZ7iXtyCfZ98XIHfIetbWsmlboBX0Q0Ma9MazQavAB3gPkd8N 0TiPsaBd48Z3w== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id F35B1CE3F27; Tue, 4 Jun 2024 15:26:53 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH rcu 4/6] rcu: Remove full memory barrier on boot time eqs sanity check Date: Tue, 4 Jun 2024 15:26:50 -0700 Message-Id: <20240604222652.2370998-4-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <81f4e616-8125-4934-a8e2-fd5beae90995@paulmck-laptop> References: <81f4e616-8125-4934-a8e2-fd5beae90995@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Frederic Weisbecker When the boot CPU initializes the per-CPU data on behalf of all possible CPUs, a sanity check is performed on each of them to make sure none is initialized in an extended quiescent state. This check involves a full memory barrier which is useless at this early boot stage. Do a plain access instead. Signed-off-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 1a6ef9c5c949e..9fa6d2b557d6c 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4772,7 +4772,7 @@ rcu_boot_init_percpu_data(int cpu) rdp->grpmask = leaf_node_cpu_bit(rdp->mynode, cpu); INIT_WORK(&rdp->strict_work, strict_work_handler); WARN_ON_ONCE(ct->dynticks_nesting != 1); - WARN_ON_ONCE(rcu_dynticks_in_eqs(rcu_dynticks_snap(cpu))); + WARN_ON_ONCE(rcu_dynticks_in_eqs(ct_dynticks_cpu(cpu))); rdp->barrier_seq_snap = rcu_state.barrier_sequence; rdp->rcu_ofl_gp_seq = rcu_state.gp_seq; rdp->rcu_ofl_gp_state = RCU_GP_CLEANED; From patchwork Tue Jun 4 22:26:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13685931 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AACDC14C5AA; Tue, 4 Jun 2024 22:26:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717540014; cv=none; b=SIaUXBEBghiO62S+nhqkoVVKcVxt7g2wjB8IYyJZsDco9/Qufx7yTe+vNCAA0C46vQ3rM6nvHlVdQakZnaH1KpWpMgBb4s+USnF8l3U9f7+jPDJI9qy9xZG2y8ZAMEe2V8Ek5EMGh3+Q/Ef3u8FXoeXW859hXwXjBdWXPcFFtgk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717540014; c=relaxed/simple; bh=d/J2eAs8W8ieRKshFNI5ALv15WcIcn5ewASod+OGMGA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=TtXcpDQUcmC4VZvCYJog7jqYxZAyipOUbmT3ubvHyG/hkPPxLmM8klMma6QblUVcOCBATloJad2413RZi2BPvfrnvvEVCgyvYhI30ijNU80r7kRjK4A43EQ71Q6SK0KW4iFSBbeuroCp1PvgtN33/ZbuwkEQR2unEkamLoJZgbY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=f9JSUeq9; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="f9JSUeq9" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6781CC4AF0C; Tue, 4 Jun 2024 22:26:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717540014; bh=d/J2eAs8W8ieRKshFNI5ALv15WcIcn5ewASod+OGMGA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=f9JSUeq9wRJSr8RmfHHb9fWhqMY2YPWhkA8E/eH1+KJVOHnbxiPj1Ek6WIk0V5cpR WNLWhbSEGI4wWoxUeRpeP3/vemriELGexloi1rnC0VFq/IgMxf9BJb8dskLbjBQBaS sitqP6m3kH2MDNL2OE4XaL7wJSUPbovbWFjYhzi205elMvZBnqVgZZ8j0zA4jdr8s2 FrZdteoodqUbbQYyJcDGwBBUAdYv9nRPLoSf0Ke0VSA1V1PoLgMaStW7YVmpG1Piiz mcMU3cB8SOQf/0NbOGp0cr3hrFSkUGPkjCotD4z5Hde7nuJne0RqwDbpW5b6LQdUqG vJel83o6uMfuw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 01FC7CE3F2C; Tue, 4 Jun 2024 15:26:54 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH rcu 5/6] rcu: Remove full memory barrier on RCU stall printout Date: Tue, 4 Jun 2024 15:26:51 -0700 Message-Id: <20240604222652.2370998-5-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <81f4e616-8125-4934-a8e2-fd5beae90995@paulmck-laptop> References: <81f4e616-8125-4934-a8e2-fd5beae90995@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Frederic Weisbecker RCU stall printout fetches the EQS state of a CPU with a preceding full memory barrier. However there is nothing to order this read against at this debugging stage. It is inherently racy when performed remotely. Do a plain read instead. This was the last user of rcu_dynticks_snap(). Signed-off-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 10 ---------- kernel/rcu/tree_stall.h | 4 ++-- 2 files changed, 2 insertions(+), 12 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 9fa6d2b557d6c..db9ca6c0c9b30 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -295,16 +295,6 @@ static void rcu_dynticks_eqs_online(void) ct_state_inc(RCU_DYNTICKS_IDX); } -/* - * Snapshot the ->dynticks counter with full ordering so as to allow - * stable comparison of this counter with past and future snapshots. - */ -static int rcu_dynticks_snap(int cpu) -{ - smp_mb(); // Fundamental RCU ordering guarantee. - return ct_dynticks_cpu_acquire(cpu); -} - /* * Return true if the snapshot returned from rcu_dynticks_snap() * indicates that RCU is in an extended quiescent state. diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h index 460efecd077be..4b0e9d7c4c68e 100644 --- a/kernel/rcu/tree_stall.h +++ b/kernel/rcu/tree_stall.h @@ -501,7 +501,7 @@ static void print_cpu_stall_info(int cpu) } delta = rcu_seq_ctr(rdp->mynode->gp_seq - rdp->rcu_iw_gp_seq); falsepositive = rcu_is_gp_kthread_starving(NULL) && - rcu_dynticks_in_eqs(rcu_dynticks_snap(cpu)); + rcu_dynticks_in_eqs(ct_dynticks_cpu(cpu)); rcuc_starved = rcu_is_rcuc_kthread_starving(rdp, &j); if (rcuc_starved) // Print signed value, as negative values indicate a probable bug. @@ -515,7 +515,7 @@ static void print_cpu_stall_info(int cpu) rdp->rcu_iw_pending ? (int)min(delta, 9UL) + '0' : "!."[!delta], ticks_value, ticks_title, - rcu_dynticks_snap(cpu) & 0xffff, + ct_dynticks_cpu(cpu) & 0xffff, ct_dynticks_nesting_cpu(cpu), ct_dynticks_nmi_nesting_cpu(cpu), rdp->softirq_snap, kstat_softirqs_cpu(RCU_SOFTIRQ, cpu), data_race(rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart, From patchwork Tue Jun 4 22:26:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13685933 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B828014B09F; Tue, 4 Jun 2024 22:26:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717540014; cv=none; b=mqTaxE8FeBDYL1yulrMNwEkRI/J12gw0BGrzHsFWVPOML+6P4FB1nMCorQYjWZN/hbozezG16P49BXq5cswD6/aj9jcE8Txh5URHMR2gqGcYTZUCMCNadAUVVp+9d0UaIUaE3k4e3fAcrT5QdISC0G0K/1uh6blX/eltEA1sIZ8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717540014; c=relaxed/simple; bh=EJa2/B/2JEA6GDpzIc1X8nYf8o99b9S13mmhOSyFJjQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=IZ+C4DT8jm9/uyyni0IaclAzlu6r4P/2gjhEeaaohDZQ1ROZCmKJQLWl4XtO24COuCsZkc07M87B9rBoaZ7sNH6lnMt2QSftC055b2COHH6ifOOUe09AKAlNj/LfvO+DiEVHLA7ozryefBsLp/7+fF6r2INx5nBgoJJtNLTYD3w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=WNP9oZK0; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="WNP9oZK0" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 94CAFC4AF11; Tue, 4 Jun 2024 22:26:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717540014; bh=EJa2/B/2JEA6GDpzIc1X8nYf8o99b9S13mmhOSyFJjQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WNP9oZK04DbR4/EDzSV05mXsdh1GSeulV1RssYxdMCv9AvT/L+hDvS71a8+nWwi/S yXKf9kwMM0CeLDPh5c1QbTH8Og14ds02FRyeRWMBvyw6zGfWAeRzykf4GvH1fxvZyY 6DGyzmqtITU3Y7Ctpn8cVHvyJVJyt8bcVomaW/azeDS3qTMGtG1USQWAsIVREd2aOC msNRKXeStwciMR8rUy58BB2wfnY7zGlZZAx6Z92DFG1OiXIXtQkZ4Haz1Ipdjq2ool rzIzSxUuA2EdBL3+kR21nLvIqcz5q+PfLLv62AzeUkd4LXo3eSZJyfDGpqqBDuBO0B HJn9QzTSzSjwg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 0487ECE3F30; Tue, 4 Jun 2024 15:26:54 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH rcu 6/6] rcu/exp: Remove redundant full memory barrier at the end of GP Date: Tue, 4 Jun 2024 15:26:52 -0700 Message-Id: <20240604222652.2370998-6-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <81f4e616-8125-4934-a8e2-fd5beae90995@paulmck-laptop> References: <81f4e616-8125-4934-a8e2-fd5beae90995@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Frederic Weisbecker A full memory barrier is necessary at the end of the expedited grace period to order: 1) The grace period completion (pictured by the GP sequence number) with all preceding accesses. This pairs with rcu_seq_end() performed by the concurrent kworker. 2) The grace period completion and subsequent post-GP update side accesses. Pairs again against rcu_seq_end(). This full barrier is already provided by the final sync_exp_work_done() test, making the subsequent explicit one redundant. Remove it and improve comments. Signed-off-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_exp.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index bec24ea6777e8..721cb93b1fece 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -265,7 +265,12 @@ static bool sync_exp_work_done(unsigned long s) { if (rcu_exp_gp_seq_done(s)) { trace_rcu_exp_grace_period(rcu_state.name, s, TPS("done")); - smp_mb(); /* Ensure test happens before caller kfree(). */ + /* + * Order GP completion with preceding accesses. Order also GP + * completion with post GP update side accesses. Pairs with + * rcu_seq_end(). + */ + smp_mb(); return true; } return false; @@ -959,7 +964,6 @@ void synchronize_rcu_expedited(void) rnp = rcu_get_root(); wait_event(rnp->exp_wq[rcu_seq_ctr(s) & 0x3], sync_exp_work_done(s)); - smp_mb(); /* Work actions happen before return. */ /* Let the next expedited grace period start. */ mutex_unlock(&rcu_state.exp_mutex);