From patchwork Wed May 15 12:53:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13665203 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A9D228564F; Wed, 15 May 2024 12:53:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777620; cv=none; b=pjzismJS9l0aciAu9zfqDmfvUARTcckJNDdblwkuOQb5NJpOSKlGYB3cwvuNy+YLwIZmv3e87cwqQ+jSbJixsbBB+eDcKdyTw1JVzp9eRIzquF62R7pkcSXzp0iZgvxMVpEhHsjpkgPMc9zUJu8Yn2Q95IkSvLUQFFgCWVGz3II= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777620; c=relaxed/simple; bh=+Ijc3TFzSdS3MOMdPr/UmaBupSlN/ypWN9jGVIpDD2c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dqxKpwEuJsBOzXLHe3VI4njsjro1BZ8xerWHRaLLfyUTqOfYHGkLh1eF1X81MWe+i1eYgCH8Lh+er55lo5dSbXED80xmNwiWXkmJtHv2dRyf3oVO/Tjn+rNym2y0yEamVMT5ntSyb4ROR+2mZhZ/lUr6C+9lHlVFfSFmBjhi/t8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=qsDVcVQw; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="qsDVcVQw" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 535D4C4AF07; Wed, 15 May 2024 12:53:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715777620; bh=+Ijc3TFzSdS3MOMdPr/UmaBupSlN/ypWN9jGVIpDD2c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qsDVcVQwZYfLnq6Woi6YVeDig7BPG3NdiNj2USEC43sJM01981goiAoG5mhmMySPa vWYoUpvYeW3hHtGcQaO95p4DJUfOrsU+9fgzRd/ayLqSRCiu4CQSEprWrLbvXU8F7Y Gith46EPn1Yv8INVoMTuoqSpksIQgblNAWAR5yC4T+g6GaQA+i/awNxw1ejdwUkKpL OhrcazveQAOXr2wbFaMCu4e02REJpI/djlBt40LSHiuFlMl+IREJPAl3gjQQesAAnU QSPBSZ+jhuKZ5ba9NCfXccomLb4jNiAYOALiWfwn1mWMwpB7k6LUa56XQFlrNoqTCR NnhAHhoDoTmUQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Valentin Schneider , "Paul E . McKenney" , Boqun Feng , Joel Fernandes , Neeraj Upadhyay , Uladzislau Rezki , Zqiang , rcu Subject: [PATCH 1/6] rcu: Remove full ordering on second EQS snapshot Date: Wed, 15 May 2024 14:53:27 +0200 Message-ID: <20240515125332.9306-2-frederic@kernel.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240515125332.9306-1-frederic@kernel.org> References: <20240515125332.9306-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 When the grace period kthread checks the extended quiescent state counter of a CPU, full ordering is necessary to ensure that either: * If the GP kthread observes the remote target in an extended quiescent state, then that target must observe all accesses prior to the current grace period, including the current grace period sequence number, once it exits that extended quiescent state. Also the GP kthread must observe all accesses performed by the target prior it entering in EQS. or: * If the GP kthread observes the remote target NOT in an extended quiescent state, then the target further entering in an extended quiescent state must observe all accesses prior to the current grace period, including the current grace period sequence number, once it enters that extended quiescent state. Also the GP kthread later observing that EQS must also observe all accesses performed by the target prior it entering in EQS. This ordering is explicitly performed both on the first EQS snapshot and on the second one as well through the combination of a preceding full barrier followed by an acquire read. However the second snapshot's full memory barrier is redundant and not needed to enforce the above guarantees: GP kthread Remote target ---- ----- // Access prior GP WRITE_ONCE(A, 1) // first snapshot smp_mb() x = smp_load_acquire(EQS) // Access prior GP WRITE_ONCE(B, 1) // EQS enter // implied full barrier by atomic_add_return() atomic_add_return(RCU_DYNTICKS_IDX, EQS) // implied full barrier by atomic_add_return() READ_ONCE(A) // second snapshot y = smp_load_acquire(EQS) z = READ_ONCE(B) If the GP kthread above fails to observe the remote target in EQS (x not in EQS), the remote target will observe A == 1 after further entering in EQS. Then the second snapshot taken by the GP kthread only need to be an acquire read in order to observe z == 1. Therefore remove the needless full memory barrier on second snapshot. Signed-off-by: Frederic Weisbecker Reviewed-by: Valentin Schneider --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 5e6828132007..58415cdc54f8 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -325,7 +325,7 @@ static bool rcu_dynticks_in_eqs(int snap) */ static bool rcu_dynticks_in_eqs_since(struct rcu_data *rdp, int snap) { - return snap != rcu_dynticks_snap(rdp->cpu); + return snap != ct_dynticks_cpu_acquire(rdp->cpu); } /* From patchwork Wed May 15 12:53:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13665204 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 280A486130; Wed, 15 May 2024 12:53:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777623; cv=none; b=dBPPhN+4Pg6g3F6QK8HcJQqBtSRqN70hGun0dhUd+/SLZT2TemZiGbi7BwJHHhseD/Iv6ymWITzv+FOmmYJ+IhpKJ3Bp2cT4z7tn7c7rmbN+OF/CPdiYuRQAb9b3/34fFpIgaIgG+F51VsnjnekU1oy3+qrASKaWs09B3fvivs0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777623; c=relaxed/simple; bh=lDuQXnRkttWQtYPH873R34M0qG7UvBXw4mUYnyTd/2M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aVykN3UU2+JCMclWh8aTCbluILsAGaBs4pEdVDhxjBDZURzvpu89DC1wDlgq7twsc4XFKKO9fxhQz8nDen2b0tJILa0B5Fz3qt6wPnPBblbuaK0E7SPNvcnFh2wIFKuPuqM+k/34r6s1pIU4J8m6hBmcJITBnPcQCvJGHRzbMA0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=qGdE5R9f; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="qGdE5R9f" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B1016C32782; Wed, 15 May 2024 12:53:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715777622; bh=lDuQXnRkttWQtYPH873R34M0qG7UvBXw4mUYnyTd/2M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qGdE5R9fS+NcqEx+GxPfGDnFmztS213YlOX81/dlCr76aJTfxenjBAp31IB6YIDw8 OHzWNQRfiw70AKZBfwsi0eNX2xHECKjEIGd6N4XgOhvE4cxyrTHvzgMCkYcMzG7H/Y F2I0mZr07e/0zRRNjituereUAhp6/eHffzK0ipP9hx/N0qCu+evtYDCYSUFIlQRx59 bZz3JLF2jUZMhIx/Kc5TI3lVD7KuKofP5vRik8VSU9BlOIwrnL6f538Xjmg8O3mzal gzSiiNzrnsd30QN9NyVxRFR10tZ36NIZFocCPOeWMi28apbCqnxCkI0hkaM3hNiq35 C9uqaxDvEbdxg== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Valentin Schneider , "Paul E . McKenney" , Boqun Feng , Joel Fernandes , Neeraj Upadhyay , Uladzislau Rezki , Zqiang , rcu Subject: [PATCH 2/6] rcu: Remove superfluous full memory barrier upon first EQS snapshot Date: Wed, 15 May 2024 14:53:28 +0200 Message-ID: <20240515125332.9306-3-frederic@kernel.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240515125332.9306-1-frederic@kernel.org> References: <20240515125332.9306-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 When the grace period kthread checks the extended quiescent state counter of a CPU, full ordering is necessary to ensure that either: * If the GP kthread observes the remote target in an extended quiescent state, then that target must observe all accesses prior to the current grace period, including the current grace period sequence number, once it exits that extended quiescent state. or: * If the GP kthread observes the remote target NOT in an extended quiescent state, then the target further entering in an extended quiescent state must observe all accesses prior to the current grace period, including the current grace period sequence number, once it enters that extended quiescent state. This ordering is enforced through a full memory barrier placed right before taking the first EQS snapshot. However this is superfluous because the snapshot is taken while holding the target's rnp lock which provides the necessary ordering through its chain of smp_mb__after_unlock_lock(). Remove the needless explicit barrier before the snapshot and put a comment about the implicit barrier newly relied upon here. Signed-off-by: Frederic Weisbecker --- .../Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst | 6 +++--- kernel/rcu/tree.c | 7 ++++++- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst b/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst index 5750f125361b..728b1e690c64 100644 --- a/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst +++ b/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst @@ -149,9 +149,9 @@ This case is handled by calls to the strongly ordered ``atomic_add_return()`` read-modify-write atomic operation that is invoked within ``rcu_dynticks_eqs_enter()`` at idle-entry time and within ``rcu_dynticks_eqs_exit()`` at idle-exit time. -The grace-period kthread invokes ``rcu_dynticks_snap()`` and -``rcu_dynticks_in_eqs_since()`` (both of which invoke -an ``atomic_add_return()`` of zero) to detect idle CPUs. +The grace-period kthread invokes first ``ct_dynticks_cpu_acquire()`` +(preceded by a full memory barrier) and ``rcu_dynticks_in_eqs_since()`` +(both of which rely on acquire semantics) to detect idle CPUs. +-----------------------------------------------------------------------+ | **Quick Quiz**: | diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 58415cdc54f8..f5354de5644b 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -773,7 +773,12 @@ static void rcu_gpnum_ovf(struct rcu_node *rnp, struct rcu_data *rdp) */ static int dyntick_save_progress_counter(struct rcu_data *rdp) { - rdp->dynticks_snap = rcu_dynticks_snap(rdp->cpu); + /* + * Full ordering against accesses prior current GP and also against + * current GP sequence number is enforced by current rnp locking + * with chained smp_mb__after_unlock_lock(). + */ + rdp->dynticks_snap = ct_dynticks_cpu_acquire(rdp->cpu); if (rcu_dynticks_in_eqs(rdp->dynticks_snap)) { trace_rcu_fqs(rcu_state.name, rdp->gp_seq, rdp->cpu, TPS("dti")); rcu_gpnum_ovf(rdp->mynode, rdp); From patchwork Wed May 15 12:53:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13665205 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 41BD8127E0A; Wed, 15 May 2024 12:53:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777625; cv=none; b=Tp3BweMb8zXfHzB2/EpwMURWqkRA0Fy+YJdXw2UoSDApynJJoE7SspawQoGX2T7DJdiHPzfOGvKAU8sbNeWNIC13A4hBJBhvVW8v+WelGH36rzWWdce1EE/EmLJalcU8rhLDXTbSHEUWKuza8w0Cda0w3BQRTVNyNzue12iVohU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777625; c=relaxed/simple; bh=MdzjiUsPb9xUfFMEziaGcXIRd8h9jhd5Aag1va9ra28=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=P7Va3kEgGQ2U8Y/wpdBsLKuuoMnz7YBLLyAUh/9eci1yAaF9PJv0RR7COS9gOpveFdmYPmOvOO1L/sZGDO3eaO2EhEd0hMOlsk5Ytiw/ghebDsDzMCGwsoS524GP/ypf0CrnekOkc5lRxMeUmEhOBW2ei4Zinw42pSnThXgqcKU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=hiR9/Pvd; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="hiR9/Pvd" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1CBA8C4AF08; Wed, 15 May 2024 12:53:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715777625; bh=MdzjiUsPb9xUfFMEziaGcXIRd8h9jhd5Aag1va9ra28=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hiR9/Pvdqc4Tv3LWXkd9/FPw/NyVasazyt/C8l04pt5IEFNIcE9kblgboXGyADNHx laiAj2ftzFrwtpybS7ZhcmFn56DfJX2oTOMMKhiRtAAm8gMfXj3fK2Ob05y/UZ5lRC TiUb7YuAWMUuYbA+9HqE2ayqhsE/Ji16HygG705JkVCtX9BJvERPQRLFcPKLyBkeg9 ejBzhY6sxdZaXJbZtoXXgnEe+BePqz2HbQixxwRM3C92ch/5fviLXq8g17u0Rq2j9q Pegyyd2bO9o5oUWpJP70RcDw3LRdcf3dx6GYmtF/JMAi6OckEsCB/09Ka4IHeAQKqq kbceTK3tNv6iQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Valentin Schneider , "Paul E . McKenney" , Boqun Feng , Joel Fernandes , Neeraj Upadhyay , Uladzislau Rezki , Zqiang , rcu Subject: [PATCH 3/6] rcu/exp: Remove superfluous full memory barrier upon first EQS snapshot Date: Wed, 15 May 2024 14:53:29 +0200 Message-ID: <20240515125332.9306-4-frederic@kernel.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240515125332.9306-1-frederic@kernel.org> References: <20240515125332.9306-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 When the grace period kthread checks the extended quiescent state counter of a CPU, full ordering is necessary to ensure that either: * If the GP kthread observes the remote target in an extended quiescent state, then that target must observe all accesses prior to the current grace period, including the current grace period sequence number, once it exits that extended quiescent state. or: * If the GP kthread observes the remote target NOT in an extended quiescent state, then the target further entering in an extended quiescent state must observe all accesses prior to the current grace period, including the current grace period sequence number, once it enters that extended quiescent state. This ordering is enforced through a full memory barrier placed right before taking the first EQS snapshot. However this is superfluous because the snapshot is taken while holding the target's rnp lock which provides the necessary ordering through its chain of smp_mb__after_unlock_lock(). Remove the needless explicit barrier before the snapshot and put a comment about the implicit barrier newly relied upon here. Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree_exp.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 8a1d9c8bd9f7..bec24ea6777e 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -357,7 +357,13 @@ static void __sync_rcu_exp_select_node_cpus(struct rcu_exp_work *rewp) !(rnp->qsmaskinitnext & mask)) { mask_ofl_test |= mask; } else { - snap = rcu_dynticks_snap(cpu); + /* + * Full ordering against accesses prior current GP and + * also against current GP sequence number is enforced + * by current rnp locking with chained + * smp_mb__after_unlock_lock(). + */ + snap = ct_dynticks_cpu_acquire(cpu); if (rcu_dynticks_in_eqs(snap)) mask_ofl_test |= mask; else From patchwork Wed May 15 12:53:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13665206 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D451812880A; Wed, 15 May 2024 12:53:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777627; cv=none; b=r1tF+6t3Qkg8Ju179BjxMSNbfcwQuDy42L6SOHn4cPy2TQogV8PLF40lk7812NpgZpUrhLVCME2EwaNhBcaKgC1OVFV4cUZMBg20QaUQFN37P8ojupw2Xd9FccvWRqw/7qwdjfV6SAG9wHwOm9Va83WKjlaCgMDJ0J7NDaCepPc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777627; c=relaxed/simple; bh=Cwf7KVnfd1lPdYZIzVZjhOBDZCNVDm3+eiHPehIQsUM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MD0sMQXzC04gbrhwwLAPmIG4+3JCcwaJadff0ipzVaaN0jeHQCiHtgJ2Uxyo7CA23FikpQszrzkKR4r1J12LsXrAKIwiYRORHbSl/rXCKwCFZ3rdi4vio/dlGpf3GSYCKwJPw3vrsIFE0+/ggdNeo2ovsQaEIquIeUyGSqAlEPM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Jk/IArrO; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Jk/IArrO" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7A9D6C116B1; Wed, 15 May 2024 12:53:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715777627; bh=Cwf7KVnfd1lPdYZIzVZjhOBDZCNVDm3+eiHPehIQsUM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Jk/IArrO6/Ocivft4HDWBiiKfGZW17EWtQmnaEY6IvqGqfzXtA67ivHh4gIxMnIu9 rmeRvHRY20SaosNvaMYQK2TDaKrwoHmbRw3veWnYz4JVRJzSWU5XBlxpN52ALGZXG8 ySlFelKvcW5WE4wYzANVet9cLfTwNb/H58xl3TFMkMo6/bqfN3+UCfauQT2gBgUT8G NhJQtR3S9Q6+b5tGKHulGmML5Fcuc1F0zydSyayp71ZEgmr8KvHwh8Ve/En0gtilYT 4zC/NpYuWhh54xMrmEyGqRfIFbjxf94J0xM1Q/a0LKcvsvrVHWmzqPYEv2LD/DCNQF Moq/KJ+/cwo/g== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Valentin Schneider , "Paul E . McKenney" , Boqun Feng , Joel Fernandes , Neeraj Upadhyay , Uladzislau Rezki , Zqiang , rcu Subject: [PATCH 4/6] rcu: Remove full memory barrier on boot time eqs sanity check Date: Wed, 15 May 2024 14:53:30 +0200 Message-ID: <20240515125332.9306-5-frederic@kernel.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240515125332.9306-1-frederic@kernel.org> References: <20240515125332.9306-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 When the boot CPU initializes the per-CPU data on behalf of all possible CPUs, a sanity check is performed on each of them to make sure none is initialized in an extended quiescent state. This check involves a full memory barrier which is useless at this early boot stage. Do a plain access instead. Signed-off-by: Frederic Weisbecker Reviewed-by: Valentin Schneider --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index f5354de5644b..02f6f3483482 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4812,7 +4812,7 @@ rcu_boot_init_percpu_data(int cpu) rdp->grpmask = leaf_node_cpu_bit(rdp->mynode, cpu); INIT_WORK(&rdp->strict_work, strict_work_handler); WARN_ON_ONCE(ct->dynticks_nesting != 1); - WARN_ON_ONCE(rcu_dynticks_in_eqs(rcu_dynticks_snap(cpu))); + WARN_ON_ONCE(rcu_dynticks_in_eqs(ct_dynticks_cpu(cpu))); rdp->barrier_seq_snap = rcu_state.barrier_sequence; rdp->rcu_ofl_gp_seq = rcu_state.gp_seq; rdp->rcu_ofl_gp_state = RCU_GP_CLEANED; From patchwork Wed May 15 12:53:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13665207 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 01C3D129E94; Wed, 15 May 2024 12:53:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777630; cv=none; b=hdZVnz4m72XBGr+kYTVJeC62+hk7BCRti17glPgJmgL1O9CgssekZSahaxxonZP3QRRUdMxdKb6tTVoHWPaBYVu4AJT3tQGlZQDu4u1AO+pgr5SuEeu9t7zMzlyNvDv/1/sfwuucilOVqGZMVqARuHx8PKzXzlgWSw5TKVLN+Rc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777630; c=relaxed/simple; bh=ZmJO5xL9+txZPsu8NViNS/0QQGVmcC4U2pZPeVVIfOw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QxWDE2zAN92O2FSQAwSGTXPTKqOsrBsBpKdQXVZ9CEOsxew4zag0rmTN6fO9P/9JJpAFx+cpsJTgA0Q1AMMkcxxEDqQg9fMu0SvZ6vDOqZvKZMG1aikLUKvLkhnXceyxxdcJYS9dMgAMbBojS9WSEVC+3ho9ehrBIswet7U0mkA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IGVsu+3a; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IGVsu+3a" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D55B4C32786; Wed, 15 May 2024 12:53:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715777629; bh=ZmJO5xL9+txZPsu8NViNS/0QQGVmcC4U2pZPeVVIfOw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IGVsu+3aCsyIiOIlpkiLdzOCbGYk+LDkdxQARmQngQjluzIT1ZYkSkojMP/KrlHNi ednrGvwKygBLy6+AAo0p+q0a5cIo6yr19B8aUa7b5WTohhobDRxa/HKdDWhCdH2J/y W+4kNqnCVb+qhRReQbnrQb+sgKaqz4JknqWHSODNdpFWMnm2B7vmMJ2YecUfrapFyX 3fwqs28mthSsiytsrBtGGsBcEOXGDRw3dc+MogW/UNNGgJQ9vZLrHUfen5P6MFI2Kd yWT3Kh+IbQ/v0C5CbxygM/taJdJgy2XNt6HP/OK/W9CdzknjbMdoTpx2yhoTHTh7O9 Stl2xck+TGX3A== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Valentin Schneider , "Paul E . McKenney" , Boqun Feng , Joel Fernandes , Neeraj Upadhyay , Uladzislau Rezki , Zqiang , rcu Subject: [PATCH 5/6] rcu: Remove full memory barrier on RCU stall printout Date: Wed, 15 May 2024 14:53:31 +0200 Message-ID: <20240515125332.9306-6-frederic@kernel.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240515125332.9306-1-frederic@kernel.org> References: <20240515125332.9306-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 RCU stall printout fetches the EQS state of a CPU with a preceding full memory barrier. However there is nothing to order this read against at this debugging stage. It is inherently racy when performed remotely. Do a plain read instead. This was the last user of rcu_dynticks_snap(). Signed-off-by: Frederic Weisbecker Reviewed-by: Valentin Schneider --- kernel/rcu/tree.c | 10 ---------- kernel/rcu/tree_stall.h | 4 ++-- 2 files changed, 2 insertions(+), 12 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 02f6f3483482..04dde7473613 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -299,16 +299,6 @@ static void rcu_dynticks_eqs_online(void) ct_state_inc(RCU_DYNTICKS_IDX); } -/* - * Snapshot the ->dynticks counter with full ordering so as to allow - * stable comparison of this counter with past and future snapshots. - */ -static int rcu_dynticks_snap(int cpu) -{ - smp_mb(); // Fundamental RCU ordering guarantee. - return ct_dynticks_cpu_acquire(cpu); -} - /* * Return true if the snapshot returned from rcu_dynticks_snap() * indicates that RCU is in an extended quiescent state. diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h index 460efecd077b..4b0e9d7c4c68 100644 --- a/kernel/rcu/tree_stall.h +++ b/kernel/rcu/tree_stall.h @@ -501,7 +501,7 @@ static void print_cpu_stall_info(int cpu) } delta = rcu_seq_ctr(rdp->mynode->gp_seq - rdp->rcu_iw_gp_seq); falsepositive = rcu_is_gp_kthread_starving(NULL) && - rcu_dynticks_in_eqs(rcu_dynticks_snap(cpu)); + rcu_dynticks_in_eqs(ct_dynticks_cpu(cpu)); rcuc_starved = rcu_is_rcuc_kthread_starving(rdp, &j); if (rcuc_starved) // Print signed value, as negative values indicate a probable bug. @@ -515,7 +515,7 @@ static void print_cpu_stall_info(int cpu) rdp->rcu_iw_pending ? (int)min(delta, 9UL) + '0' : "!."[!delta], ticks_value, ticks_title, - rcu_dynticks_snap(cpu) & 0xffff, + ct_dynticks_cpu(cpu) & 0xffff, ct_dynticks_nesting_cpu(cpu), ct_dynticks_nmi_nesting_cpu(cpu), rdp->softirq_snap, kstat_softirqs_cpu(RCU_SOFTIRQ, cpu), data_race(rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart, From patchwork Wed May 15 12:53:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13665208 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E45212AAD3; Wed, 15 May 2024 12:53:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777632; cv=none; b=YYwV/A3FgOGabnHZ9/1PlyFtxK/pMaAoaJUuVOki5ToqwGJKWJSHeiHqBXYVF9wPBRugvrHjkrawfO/YELj877t7+I1PkPHrECI/h3TLnLZYLWErLRmLv4YXrOZ7jUE/MBvYZbPFL0d4FY0fKbdZvosteHXhI8zfLl90D848DfM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777632; c=relaxed/simple; bh=61n4BvWS+RKXfIV6Crmix3uFCPd3dUP3naHmOjOk0/U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uXkjGAX4+I7udrtJJyFk/6HeSZCKt1sYF0J9sGW0+fuDaASWn0WNZuyO2bqZeuhCQ4tvvj9+vyV8Y3biAJw978lyE2dLr/rM/MNlawFccTLlJ1JjnTtA7IBgOx7wyMycKwHGdVcqnoLuCA8SoeTdrH8DcmX+3jgCp/omW4uq5gk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=sOP210ma; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="sOP210ma" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3DB15C4AF07; Wed, 15 May 2024 12:53:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715777632; bh=61n4BvWS+RKXfIV6Crmix3uFCPd3dUP3naHmOjOk0/U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sOP210ma84aHUIqjdbTezTc6vStV8NonefdGSCfvpWLsE+X/49qBJg38k8b6KCPPC lafQb+1GJQ1YHBEHgkD5uSdi+Oify/E7brJF2l/rkO0jBjb65iFO+8mx++JcJW31p1 MQDBjBGo+4HIF+BnHwBN0IU3QH/Zbva8io9toYEj6gWMuAcRP2O07erl1db3Ke++Z+ U4U5WPdsBJeMysx67cBHP/UeS6pZ3+lK7zL8nIZNYN8mueyTCOgahEkkpMlgPbgBlp nBeVVCmzMdesWIkdMF1ODbtYpt62MqamVbmYM941LWarhvXdEEoH2a4atEx9KUNiwU gvJk320hR2TCw== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Valentin Schneider , "Paul E . McKenney" , Boqun Feng , Joel Fernandes , Neeraj Upadhyay , Uladzislau Rezki , Zqiang , rcu Subject: [PATCH 6/6] rcu/exp: Remove redundant full memory barrier at the end of GP Date: Wed, 15 May 2024 14:53:32 +0200 Message-ID: <20240515125332.9306-7-frederic@kernel.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240515125332.9306-1-frederic@kernel.org> References: <20240515125332.9306-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 A full memory barrier is necessary at the end of the expedited grace period to order: 1) The grace period completion (pictured by the GP sequence number) with all preceding accesses. This pairs with rcu_seq_end() performed by the concurrent kworker. 2) The grace period completion and subsequent post-GP update side accesses. Pairs again against rcu_seq_end(). This full barrier is already provided by the final sync_exp_work_done() test, making the subsequent explicit one redundant. Remove it and improve comments. Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree_exp.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index bec24ea6777e..721cb93b1fec 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -265,7 +265,12 @@ static bool sync_exp_work_done(unsigned long s) { if (rcu_exp_gp_seq_done(s)) { trace_rcu_exp_grace_period(rcu_state.name, s, TPS("done")); - smp_mb(); /* Ensure test happens before caller kfree(). */ + /* + * Order GP completion with preceding accesses. Order also GP + * completion with post GP update side accesses. Pairs with + * rcu_seq_end(). + */ + smp_mb(); return true; } return false; @@ -959,7 +964,6 @@ void synchronize_rcu_expedited(void) rnp = rcu_get_root(); wait_event(rnp->exp_wq[rcu_seq_ctr(s) & 0x3], sync_exp_work_done(s)); - smp_mb(); /* Work actions happen before return. */ /* Let the next expedited grace period start. */ mutex_unlock(&rcu_state.exp_mutex);