From patchwork Thu Jan 16 20:05:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13942193 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B2FC722CBEE; Thu, 16 Jan 2025 20:05:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737057945; cv=none; b=uqYET2YO54QNICWohQcxp9oZDZb+S3EVmWHQlAx1xFi3WpbZO1wef7ptSApf86l7opZP+PYVqwf2MF18ND8Pe1+X9L1MSsN7JLdcOOWk14xH9fB4AezrfxOwnx6Fbg+arurM9lFZhD1kcS/80PvIqJLNFY+m2KzSDagAhoZhii4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737057945; c=relaxed/simple; bh=WMQ7DP9OiOxGto+2UoZj3IavynbQoyjALlXlLrmJ9XU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=K66eAaaL+yYN8jwGRwW85KFhEtFG73NqpTPxz/tS34ENcnb8fwXqRAoJsPXGQ8NH6lqXgCp+A9rFUXHQXSriSga+g05d3ou36Q7X426dEbljzlmQ/QhwQjS7UC+QAkwES2D3TFr5LajtUI7yDKdAmkgXVaeQ3l4gFBqdF3ExIDQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=dfGbICgQ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="dfGbICgQ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 46D72C4CED6; Thu, 16 Jan 2025 20:05:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1737057945; bh=WMQ7DP9OiOxGto+2UoZj3IavynbQoyjALlXlLrmJ9XU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dfGbICgQLsRbw3ftKiz4V5vKHFBzrCDmivdd4/DsNTsomLjOUIfi0LiyOnOJERlo+ 0cFPbQNnEpiIXsaAEt23SjWkrueXCg6QNlfzgI2wQld4i+cPJ9d8NZnHKz7a9xVps4 3C9Ze7iqo1IDIARxP2ts8WuWeshquPJ+7REABNlLwIiEUF4NlxmSkxG7RAW/HxfaN7 Ev1oBq+cxsPKNNnDDO4o1PiqVw3QDDV2YAa5kz/uVipwUP59Ekv2gAOBcwvXHhCrPM LpSEXJjul4ddw2s5DtheJzTFMT7+9xH5hCMHFUka1YV/4Rl0q5ccx5aa4T7xuQw7tZ 0Sj/cqHaqTrmw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 02437CE10F9; Thu, 16 Jan 2025 12:05:45 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Frederic Weisbecker Subject: [PATCH rcu 1/5] rcu: Split rcu_report_exp_cpu_mult() mask parameter and use for tracing Date: Thu, 16 Jan 2025 12:05:39 -0800 Message-Id: <20250116200543.3782539-1-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <7b24a850-7d56-4386-8544-be5e643b6b9f@paulmck-laptop> References: <7b24a850-7d56-4386-8544-be5e643b6b9f@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This commit renames the rcu_report_exp_cpu_mult() function from "mask" to "mask_in" and introduced a "mask" local variable to better support upcoming event-tracing additions. Signed-off-by: Paul E. McKenney Cc: Frederic Weisbecker --- kernel/rcu/tree_exp.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 77efed89c79e3..8d4895c854c59 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -230,17 +230,19 @@ static void __maybe_unused rcu_report_exp_rnp(struct rcu_node *rnp, bool wake) * specified leaf rcu_node structure, which is acquired by the caller. */ static void rcu_report_exp_cpu_mult(struct rcu_node *rnp, unsigned long flags, - unsigned long mask, bool wake) + unsigned long mask_in, bool wake) __releases(rnp->lock) { int cpu; + unsigned long mask; struct rcu_data *rdp; raw_lockdep_assert_held_rcu_node(rnp); - if (!(rnp->expmask & mask)) { + if (!(rnp->expmask & mask_in)) { raw_spin_unlock_irqrestore_rcu_node(rnp, flags); return; } + mask = mask_in & rnp->expmask; WRITE_ONCE(rnp->expmask, rnp->expmask & ~mask); for_each_leaf_node_cpu_mask(rnp, cpu, mask) { rdp = per_cpu_ptr(&rcu_data, cpu); From patchwork Thu Jan 16 20:05:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13942195 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B308622FDF6; Thu, 16 Jan 2025 20:05:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737057945; cv=none; b=r7i4p9/vrSwlhxeBDcTFXD28nlIQwHwo/ZqLsM3ZoMl3ec1g8U8Zh/cZHYrNT6thmcKe+c7NX1lsheM1vqQPrHEHCnpuj+fom79/woGI1aVeMw5QJEeWw0D+egAuUBKmOV5VM2gEJf0p8hJDHgWxse+ZBAjKlAa/7FkBXsG1Nfc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737057945; c=relaxed/simple; bh=xMr7OA3HqbZQZeURX+525+rWHcKLudpdVYa2A5sHf0s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=oPw9iG3O6TI1qEjpErOgttmtAnkG3p4Tay2U0c9jx0Y6xdWXj3CxAfyoi4MvduqJbTsyKDWLOHC/NxnyeI4QspSuxJdGx3GVo+2wsOTa+7m/qUflMBDB0tjFuEdAZ6dUlmG4NX1b05A0fUuCdLCmHcvt8oBYwQHwwcy3X55IxCM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=bs5l+RAZ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="bs5l+RAZ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 59DABC4CEDD; Thu, 16 Jan 2025 20:05:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1737057945; bh=xMr7OA3HqbZQZeURX+525+rWHcKLudpdVYa2A5sHf0s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bs5l+RAZopTZpiC8d8/4xfY0KXCLGWWWcFOIqBcdbbxsgh1KKo8yD1m1TzXo0nvvD 1ScqVlVqW0YitdL3AM9WWcUm+htiq5CZ+HpISKowRQFa3HKzpxvJ2WCdBh3WczYYq5 GQ9jQSg6s62rARYiBNEczUly1KFKOllLGiHT2JQRc9W9ndIavmqlt4uDLQhcCnG1g9 t2XOlimUS+e2Dqj1MDQeXdN64ufKai0xaPLgQbUD4eHequWbtEtItc1ZCrECbeE/s7 7mkGKy5FlZWT9Nsq2JAaoiWTwNHC+gy4urtT5mGzknQE90fiamXxgOJCFxJ2kniThw +rpRT2RwHHAJw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 04816CE12BE; Thu, 16 Jan 2025 12:05:45 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Zilin Guan , "Paul E . McKenney" Subject: [PATCH rcu 2/5] rcu: Remove READ_ONCE() for rdp->gpwrap access in __note_gp_changes() Date: Thu, 16 Jan 2025 12:05:40 -0800 Message-Id: <20250116200543.3782539-2-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <7b24a850-7d56-4386-8544-be5e643b6b9f@paulmck-laptop> References: <7b24a850-7d56-4386-8544-be5e643b6b9f@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Zilin Guan There is one access to the per-CPU rdp->gpwrap field in the __note_gp_changes() function that does not use READ_ONCE(), but all other accesses do use READ_ONCE(). When using the 8*TREE03 and CONFIG_NR_CPUS=8 configuration, KCSAN found no data races at that point. This is because all calls to __note_gp_changes() hold rnp->lock, which excludes writes to the rdp->gpwrap fields for all CPUs associated with that same leaf rcu_node structure. This commit therefore removes READ_ONCE() from rdp->gpwrap accesses within the __note_gp_changes() function. Signed-off-by: Zilin Guan Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 229f427b8c821..e49bcb86b6d3c 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1275,7 +1275,7 @@ static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp) /* Handle the ends of any preceding grace periods first. */ if (rcu_seq_completed_gp(rdp->gp_seq, rnp->gp_seq) || - unlikely(READ_ONCE(rdp->gpwrap))) { + unlikely(rdp->gpwrap)) { if (!offloaded) ret = rcu_advance_cbs(rnp, rdp); /* Advance CBs. */ rdp->core_needs_qs = false; @@ -1289,7 +1289,7 @@ static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp) /* Now handle the beginnings of any new-to-this-CPU grace periods. */ if (rcu_seq_new_gp(rdp->gp_seq, rnp->gp_seq) || - unlikely(READ_ONCE(rdp->gpwrap))) { + unlikely(rdp->gpwrap)) { /* * If the current grace period is waiting for this CPU, * set up to detect a quiescent state, otherwise don't @@ -1304,7 +1304,7 @@ static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp) rdp->gp_seq = rnp->gp_seq; /* Remember new grace-period state. */ if (ULONG_CMP_LT(rdp->gp_seq_needed, rnp->gp_seq_needed) || rdp->gpwrap) WRITE_ONCE(rdp->gp_seq_needed, rnp->gp_seq_needed); - if (IS_ENABLED(CONFIG_PROVE_RCU) && READ_ONCE(rdp->gpwrap)) + if (IS_ENABLED(CONFIG_PROVE_RCU) && rdp->gpwrap) WRITE_ONCE(rdp->last_sched_clock, jiffies); WRITE_ONCE(rdp->gpwrap, false); rcu_gpnum_ovf(rnp, rdp); From patchwork Thu Jan 16 20:05:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13942194 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B302622E41C; Thu, 16 Jan 2025 20:05:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737057945; cv=none; b=H7fH0O5gJiItI11edKGTHIuFnoBrQRpLBQESeBgtAgMzzzF+SLh1w9DGIZ4KiNxxnyObYFmfSA4lnn1Qh0QaVdh1oGX9bMhiDjtm/acW/0FOaULElIKviQslPucZeBl6HDeUrUF/0f0VNzR/w1qYkbPi1KK9zZyzJeWdkC0Hq+o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737057945; c=relaxed/simple; bh=B7Xp1bdgAsRzNNN/TtnikhjmR/mJsYXDdd2BddykECM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=LO9oqjAnycFyEVbfRBbqke6oH44AD3da7jILtguruwVqn6p2CYpsu5NEA6j2TG5/Sw+QBtYbK2QZ66cHace/gGncDiz7FJVdvZzl+0U7D4tQD8W3frroRb5YagZTuHLi84WxyHCT361Vjo8F3RDS8ZcNF7sGWcEgvLMBQaKAJB0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=M+uy3Sqq; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="M+uy3Sqq" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5ED40C4CEE4; Thu, 16 Jan 2025 20:05:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1737057945; bh=B7Xp1bdgAsRzNNN/TtnikhjmR/mJsYXDdd2BddykECM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=M+uy3SqqFDLIkAbpe7uM5EuPTFgd/0drRsWzqtEBIG0ph62kY8Uv8SUeLZkaAyBw+ mTaKsZU7PMLE7ojAO3yLS5qtFGEo0KStb1fEFdVIDII4NWf3BlRz0vuH902+inAVxa P6+hjfT17uW4m8vUKZxvFHrHtA7tHu9znR5vAlWIBiFMTWCUBzutklTC11BZoUC8Ym cJ9DkVjQDWlxqjTSwaBnNVYg/Rl9r+Rk7H+1U4D4cLIaWrmtBELEp9G+JrdBA6zB0d GH7FzE2slq+3fzcCVGDxHR97L4DqpdyDeLUz36lD6CvTmI+6SNgENeE1ncCjIM33sL r3DTKTeWV4hVQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 0747CCE37B4; Thu, 16 Jan 2025 12:05:45 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 3/5] rcu: Fix get_state_synchronize_rcu_full() GP-start detection Date: Thu, 16 Jan 2025 12:05:41 -0800 Message-Id: <20250116200543.3782539-3-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <7b24a850-7d56-4386-8544-be5e643b6b9f@paulmck-laptop> References: <7b24a850-7d56-4386-8544-be5e643b6b9f@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The get_state_synchronize_rcu_full() and poll_state_synchronize_rcu_full() functions use the root rcu_node structure's ->gp_seq field to detect the beginnings and ends of grace periods, respectively. This choice is necessary for the poll_state_synchronize_rcu_full() function because (give or take counter wrap), the following sequence is guaranteed not to trigger: get_state_synchronize_rcu_full(&rgos); synchronize_rcu(); WARN_ON_ONCE(!poll_state_synchronize_rcu_full(&rgos)); The RCU callbacks that awaken synchronize_rcu() instances are guaranteed not to be invoked before the root rcu_node structure's ->gp_seq field is updated to indicate the end of the grace period. However, these callbacks might start being invoked immediately thereafter, in particular, before rcu_state.gp_seq has been updated. Therefore, poll_state_synchronize_rcu_full() must refer to the root rcu_node structure's ->gp_seq field. Because this field is updated under this structure's ->lock, any code following a call to poll_state_synchronize_rcu_full() will be fully ordered after the full grace-period computation, as is required by RCU's memory-ordering semantics. By symmetry, the get_state_synchronize_rcu_full() function should also use this same root rcu_node structure's ->gp_seq field. But it turns out that symmetry is profoundly (though extremely infrequently) destructive in this case. To see this, consider the following sequence of events: 1. CPU 0 starts a new grace period, and updates rcu_state.gp_seq accordingly. 2. As its first step of grace-period initialization, CPU 0 examines the current CPU hotplug state and decides that it need not wait for CPU 1, which is currently offline. 3. CPU 1 comes online, and updates its state. But this does not affect the current grace period, but rather the one after that. After all, CPU 1 was offline when the current grace period started, so all pre-existing RCU readers on CPU 1 must have completed or been preempted before it last went offline. The current grace period therefore has nothing it needs to wait for on CPU 1. 4. CPU 1 switches to an rcutorture kthread which is running rcutorture's rcu_torture_reader() function, which starts a new RCU reader. 5. CPU 2 is running rcutorture's rcu_torture_writer() function and collects a new polled grace-period "cookie" using get_state_synchronize_rcu_full(). Because the newly started grace period has not completed initialization, the root rcu_node structure's ->gp_seq field has not yet been updated to indicate that this new grace period has already started. This cookie is therefore set up for the end of the current grace period (rather than the end of the following grace period). 6. CPU 0 finishes grace-period initialization. 7. If CPU 1’s rcutorture reader is preempted, it will be added to the ->blkd_tasks list, but because CPU 1’s ->qsmask bit is not set in CPU 1's leaf rcu_node structure, the ->gp_tasks pointer will not be updated.  Thus, this grace period will not wait on it.  Which is only fair, given that the CPU did not come online until after the grace period officially started. 8. CPUs 0 and 2 then detect the new grace period and then report a quiescent state to the RCU core. 9. Because CPU 1 was offline at the start of the current grace period, CPUs 0 and 2 are the only CPUs that this grace period needs to wait on. So the grace period ends and post-grace-period cleanup starts. In particular, the root rcu_node structure's ->gp_seq field is updated to indicate that this grace period has now ended. 10. CPU 2 continues running rcu_torture_writer() and sees that, from the viewpoint of the root rcu_node structure consulted by the poll_state_synchronize_rcu_full() function, the grace period has ended.  It therefore updates state accordingly. 11. CPU 1 is still running the same RCU reader, which notices this update and thus complains about the too-short grace period. The fix is for the get_state_synchronize_rcu_full() function to use rcu_state.gp_seq instead of the the root rcu_node structure's ->gp_seq field. With this change in place, if step 5's cookie indicates that the grace period has not yet started, then any prior code executed by CPU 2 must have happened before CPU 1 came online. This will in turn prevent CPU 1's code in steps 3 and 11 from spanning CPU 2's grace-period wait, thus preventing CPU 1 from being subjected to a too-short grace period. This commit therefore makes this change. Note that there is no change to the poll_state_synchronize_rcu_full() function, which as noted above, must continue to use the root rcu_node structure's ->gp_seq field. This is of course an asymmetry between these two functions, but is an asymmetry that is absolutely required for correct operation. It is a common human tendency to greatly value symmetry, and sometimes symmetry is a wonderful thing. Other times, symmetry results in poor performance. But in this case, symmetry is just plain wrong. Nevertheless, the asymmetry does require an additional adjustment. It is possible for get_state_synchronize_rcu_full() to see a given grace period as having started, but for an immediately following poll_state_synchronize_rcu_full() to see it as having not yet started. Given the current rcu_seq_done_exact() implementation, this will result in a false-positive indication that the grace period is done from poll_state_synchronize_rcu_full(). This is dealt with by making rcu_seq_done_exact() reach back three grace periods rather than just two of them. Although this fixes 91a967fd6934 ("rcu: Add full-sized polling for get_completed*() and poll_state*()"), it is not clear that it is worth backporting this commit. First, it took me many weeks to convince rcutorture to reproduce this more frequently than once per year. Second, this cannot be reproduced at all without frequent CPU-hotplug operations, as in waiting all of 50 milliseconds from the end of the previous operation until starting the next one. Third, the TREE03.boot settings cause multi-millisecond delays during RCU grace-period initialization, which greatly increase the probability of the above sequence of events. (Don't do this in production workloads!) Fourth, the TREE03 rcutorture scenario was modified to use four-CPU guest OSes, to have a single-rcu_node combining tree, no testing of RCU priority boosting, and no random preemption, and these modifications were necessary to reproduce this issue in a reasonable timeframe. Fifth, extremely heavy use of get_state_synchronize_rcu_full() and/or poll_state_synchronize_rcu_full() is required to reproduce this, and as of v6.12, only kfree_rcu() uses it, and even then not particularly heavily. Signed-off-by: Paul E. McKenney --- kernel/rcu/rcu.h | 2 +- kernel/rcu/tree.c | 9 ++++++--- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index feb3ac1dc5d59..f87c9d6d36fcb 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -162,7 +162,7 @@ static inline bool rcu_seq_done_exact(unsigned long *sp, unsigned long s) { unsigned long cur_s = READ_ONCE(*sp); - return ULONG_CMP_GE(cur_s, s) || ULONG_CMP_LT(cur_s, s - (2 * RCU_SEQ_STATE_MASK + 1)); + return ULONG_CMP_GE(cur_s, s) || ULONG_CMP_LT(cur_s, s - (3 * RCU_SEQ_STATE_MASK + 1)); } /* diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index e49bcb86b6d3c..9ba34206e20d2 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4197,14 +4197,17 @@ EXPORT_SYMBOL_GPL(get_state_synchronize_rcu); */ void get_state_synchronize_rcu_full(struct rcu_gp_oldstate *rgosp) { - struct rcu_node *rnp = rcu_get_root(); - /* * Any prior manipulation of RCU-protected data must happen * before the loads from ->gp_seq and ->expedited_sequence. */ smp_mb(); /* ^^^ */ - rgosp->rgos_norm = rcu_seq_snap(&rnp->gp_seq); + + // Yes, rcu_state.gp_seq, not rnp_root->gp_seq, the latter's use + // in poll_state_synchronize_rcu_full() notwithstanding. Use of + // the latter here would result in too-short grace periods due to + // interactions with newly onlined CPUs. + rgosp->rgos_norm = rcu_seq_snap(&rcu_state.gp_seq); rgosp->rgos_exp = rcu_seq_snap(&rcu_state.expedited_sequence); } EXPORT_SYMBOL_GPL(get_state_synchronize_rcu_full); From patchwork Thu Jan 16 20:05:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13942196 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA8BB234D00; Thu, 16 Jan 2025 20:05:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737057946; cv=none; b=tSZ096OAhX7oiZNBL3PzXwxWNH0YDVFFJcTi5j/ah5uCebyQ/KX/5Od8S3wikDkvv4ae32MIeFk0yV8+1+iYcuV+F3mKZ1V/WiWIFNCIXDijijVoafdFxt5n+t51YgLBzeHJ/gREojI0wkt2csenNZNb+zx6t8nY89zq6y+9RCE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737057946; c=relaxed/simple; bh=yHSEYo1yMi2XoWvkgiqu3Uvnki/MK7OFLB78k2z+u9U=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=F5SfBDTgNzJUOJHjUT4xgCfP1ip3sqfi73pjQbkigH0gnIcAOQvMkvpfO26dW0tJU7XNu4Tnnky/r7AYMtdEpGVekWy3nFyMfopKKSDzcNCLPQS+iIxXKPM4Sl5xQO75dj/GovTP1vxLaome64cFaP6oJcOsYmSJtWXWoCGDzpw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=bPVNSuvJ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="bPVNSuvJ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 672EAC4AF0B; Thu, 16 Jan 2025 20:05:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1737057945; bh=yHSEYo1yMi2XoWvkgiqu3Uvnki/MK7OFLB78k2z+u9U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bPVNSuvJfNNQoXOHUhovEMHQJXkn2TomzR2GxXijjKLFHVxZ9FJf36cUZQ8LQvdDx 3k9PrKWiN/X6lBxclYD+qRDfCByWv1TCcvXF01LQOFfDw2A2fwM96/ajCSbEno9aoA +sBxU0HxNXrgBK49GJyWvAz1ywul3XAMgZJPAw7CCliSKF2RRgOqMhhiW0ntAMYH3G AUI4I/EjYRjC8w+uRwmvHdYRrCnfX9ug07NG7Ne9KvSuDxq3ryXusrqpg+sPUCH4Du XvpkOGaB/KiE2VHV+wg2G5bBktZlzEztyjWYUaE5fpsAJstWMTfbLK+z0pEun5/vHL tDJZWpzYloxAw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 09E48CE37B6; Thu, 16 Jan 2025 12:05:45 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Sebastian Andrzej Siewior , Frederic Weisbecker , Thomas Gleixner , Alexei Starovoitov , Andrii Nakryiko , Mathieu Desnoyers , Masami Hiramatsu , linux-trace-kernel@vger.kernel.org Subject: [PATCH rcu 4/5] rcu-tasks: Move RCU Tasks self-tests to core_initcall() Date: Thu, 16 Jan 2025 12:05:42 -0800 Message-Id: <20250116200543.3782539-4-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <7b24a850-7d56-4386-8544-be5e643b6b9f@paulmck-laptop> References: <7b24a850-7d56-4386-8544-be5e643b6b9f@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The timer and hrtimer softirq processing has moved to dedicated threads for kernels built with CONFIG_IRQ_FORCED_THREADING=y. This results in timers not expiring until later in early boot, which in turn causes the RCU Tasks self-tests to hang in kernels built with CONFIG_PROVE_RCU=y, which further causes the entire kernel to hang. One fix would be to make timers work during this time, but there are no known users of RCU Tasks grace periods during that time, so no justification for the added complexity. Not yet, anyway. This commit therefore moves the call to rcu_init_tasks_generic() from kernel_init_freeable() to a core_initcall(). This works because the timer and hrtimer kthreads are created at early_initcall() time. Signed-off-by: Paul E. McKenney Cc: Sebastian Andrzej Siewior Cc: Frederic Weisbecker Cc: Thomas Gleixner Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Steven Rostedt Cc: Mathieu Desnoyers Cc: Masami Hiramatsu Cc: --- include/linux/rcupdate.h | 6 ------ init/main.c | 1 - kernel/rcu/tasks.h | 5 ++++- 3 files changed, 4 insertions(+), 8 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 3bb554723074d..c6c4aee47df2c 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -121,12 +121,6 @@ void rcu_init(void); extern int rcu_scheduler_active; void rcu_sched_clock_irq(int user); -#ifdef CONFIG_TASKS_RCU_GENERIC -void rcu_init_tasks_generic(void); -#else -static inline void rcu_init_tasks_generic(void) { } -#endif - #ifdef CONFIG_RCU_STALL_COMMON void rcu_sysrq_start(void); void rcu_sysrq_end(void); diff --git a/init/main.c b/init/main.c index 00fac1170294f..fb880b6fb524b 100644 --- a/init/main.c +++ b/init/main.c @@ -1562,7 +1562,6 @@ static noinline void __init kernel_init_freeable(void) init_mm_internals(); - rcu_init_tasks_generic(); do_pre_smp_initcalls(); lockup_detector_init(); diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 59314da5eb604..466668eb4fadf 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -2256,7 +2256,7 @@ void __init tasks_cblist_init_generic(void) #endif } -void __init rcu_init_tasks_generic(void) +static int __init rcu_init_tasks_generic(void) { #ifdef CONFIG_TASKS_RCU rcu_spawn_tasks_kthread(); @@ -2272,7 +2272,10 @@ void __init rcu_init_tasks_generic(void) // Run the self-tests. rcu_tasks_initiate_self_tests(); + + return 0; } +core_initcall(rcu_init_tasks_generic); #else /* #ifdef CONFIG_TASKS_RCU_GENERIC */ static inline void rcu_tasks_bootup_oddness(void) {} From patchwork Thu Jan 16 20:05:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13942192 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C59914AD2B; Thu, 16 Jan 2025 20:05:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737057945; cv=none; b=VBjbTwPEg/sSwH4DHUsYh0PbqpZqtyorPw0zGw70xlod8pS2XeEl3HCGXUnWs69hbmrQ5VKd6vfqLRmS/JVscY5Wknu0Iw7wbWgr2LL3F6/xPkBrcuIIvYp+ji3yIleblqaGISICalFJ+6afmjO7AhXPYTJQT7thGb3HyV/COaY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737057945; c=relaxed/simple; bh=NP/CK3qlbbTrzpBV1+3fFxhbMdYgJcUACRadRqSIOlU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=it+/WIMTxIR/Ia6UDCm5YBZSka0UUbgKjKL85E7/k+l8HdBTrDWyl1Aasj8gLdfbsI4iCTDN+hda2VZBrvDGR+B/VSCDecQgXcpmipOoTh60rLXmVg9JPeBiIpG888wrbz1h2GXMW35enbZqsh0oH3WP7KAxXH7kuAHqQKl056E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=foXRttxb; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="foXRttxb" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7199CC4CEE6; Thu, 16 Jan 2025 20:05:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1737057945; bh=NP/CK3qlbbTrzpBV1+3fFxhbMdYgJcUACRadRqSIOlU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=foXRttxbGiSK8jAUDaijACmdlwrAwCe1qQ/OZkefWBPlanZgtZYE0SyNObw+JTPks XMNzeRXcGt/4/2zGdJkoiN/5d+WUzVAlBhbXKm5v2JKmbcv4rxkDb5CoWb/CLKe6Xl GzapcHq/F1NppAaEA1X5BgTxW5ebO6/BNAidJ/vRzgDEPp1flS6k9XGX4uQiBA4xIU 0QHvgpXjpTMx7G0XE5d9MBjMLQnwV8aR3ZFS72Itn8O++QRmVowbfP78+Sf6G45M2u L2o42QfTh67lazQcH6VDlGyhwGE/N44I3j2vxw/fRRqH5hCUSFzevKw892DNxMUARw Sfd/+ymlhmhdg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 0C072CE37C1; Thu, 16 Jan 2025 12:05:45 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 5/5] rcu/nocb: Print segment lengths in show_rcu_nocb_gp_state() Date: Thu, 16 Jan 2025 12:05:43 -0800 Message-Id: <20250116200543.3782539-5-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <7b24a850-7d56-4386-8544-be5e643b6b9f@paulmck-laptop> References: <7b24a850-7d56-4386-8544-be5e643b6b9f@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Analysis of an rcutorture callback-based forward-progress test failure was hampered by the lack of ->cblist segment lengths. This commit therefore adds this information, so that what would have been ".W85620.N." (there are some callbacks waiting for grace period sequence number 85620 and some number more that have not yet been assigned to a grace period) now prints as ".W2(85620).N6." (there are 2 callbacks waiting for grace period 85620 and 6 not yet assigned to a grace period). Note that "D" (done), "N" (next and not yet assigned to a grace period, and "B" (bypass, also not yet assigned to a grace period) have just the number of callbacks without the parenthesized grace-period sequence number. In contrast, "W" (waiting for the current grace period) and "R" (ready to wait for the next grace period to start) both have parenthesized grace-period sequence numbers. Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 2605dd234a13c..5ff3bc56ff515 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1557,8 +1557,11 @@ static void show_rcu_nocb_gp_state(struct rcu_data *rdp) /* Dump out nocb kthread state for the specified rcu_data structure. */ static void show_rcu_nocb_state(struct rcu_data *rdp) { - char bufw[20]; - char bufr[20]; + char bufd[22]; + char bufw[45]; + char bufr[45]; + char bufn[22]; + char bufb[22]; struct rcu_data *nocb_next_rdp; struct rcu_segcblist *rsclp = &rdp->cblist; bool waslocked; @@ -1572,9 +1575,13 @@ static void show_rcu_nocb_state(struct rcu_data *rdp) typeof(*rdp), nocb_entry_rdp); - sprintf(bufw, "%ld", rsclp->gp_seq[RCU_WAIT_TAIL]); - sprintf(bufr, "%ld", rsclp->gp_seq[RCU_NEXT_READY_TAIL]); - pr_info(" CB %d^%d->%d %c%c%c%c%c F%ld L%ld C%d %c%c%s%c%s%c%c q%ld %c CPU %d%s\n", + sprintf(bufd, "%ld", rsclp->seglen[RCU_DONE_TAIL]); + sprintf(bufw, "%ld(%ld)", rsclp->seglen[RCU_WAIT_TAIL], rsclp->gp_seq[RCU_WAIT_TAIL]); + sprintf(bufr, "%ld(%ld)", rsclp->seglen[RCU_NEXT_READY_TAIL], + rsclp->gp_seq[RCU_NEXT_READY_TAIL]); + sprintf(bufn, "%ld", rsclp->seglen[RCU_NEXT_TAIL]); + sprintf(bufb, "%ld", rcu_cblist_n_cbs(&rdp->nocb_bypass)); + pr_info(" CB %d^%d->%d %c%c%c%c%c F%ld L%ld C%d %c%s%c%s%c%s%c%s%c%s q%ld %c CPU %d%s\n", rdp->cpu, rdp->nocb_gp_rdp->cpu, nocb_next_rdp ? nocb_next_rdp->cpu : -1, "kK"[!!rdp->nocb_cb_kthread], @@ -1586,12 +1593,15 @@ static void show_rcu_nocb_state(struct rcu_data *rdp) jiffies - rdp->nocb_nobypass_last, rdp->nocb_nobypass_count, ".D"[rcu_segcblist_ready_cbs(rsclp)], + rcu_segcblist_segempty(rsclp, RCU_DONE_TAIL) ? "" : bufd, ".W"[!rcu_segcblist_segempty(rsclp, RCU_WAIT_TAIL)], rcu_segcblist_segempty(rsclp, RCU_WAIT_TAIL) ? "" : bufw, ".R"[!rcu_segcblist_segempty(rsclp, RCU_NEXT_READY_TAIL)], rcu_segcblist_segempty(rsclp, RCU_NEXT_READY_TAIL) ? "" : bufr, ".N"[!rcu_segcblist_segempty(rsclp, RCU_NEXT_TAIL)], + rcu_segcblist_segempty(rsclp, RCU_NEXT_TAIL) ? "" : bufn, ".B"[!!rcu_cblist_n_cbs(&rdp->nocb_bypass)], + !rcu_cblist_n_cbs(&rdp->nocb_bypass) ? "" : bufb, rcu_segcblist_n_cbs(&rdp->cblist), rdp->nocb_cb_kthread ? task_state_to_char(rdp->nocb_cb_kthread) : '.', rdp->nocb_cb_kthread ? (int)task_cpu(rdp->nocb_cb_kthread) : -1,