From patchwork Tue Oct 3 23:28:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13408091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F648E8FDC0 for ; Tue, 3 Oct 2023 23:29:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236153AbjJCX3S (ORCPT ); Tue, 3 Oct 2023 19:29:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52980 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236052AbjJCX3R (ORCPT ); Tue, 3 Oct 2023 19:29:17 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E77E1BD; Tue, 3 Oct 2023 16:29:13 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3D2B6C433C7; Tue, 3 Oct 2023 23:29:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1696375753; bh=SeJHCTpKmvGZu4Eri7N06Cc3UjYAkpJhCi+GE0MfaBE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tAVXdvQDOshnwP7RiEpZBQNaMyg87EVmQYfybY4RJsaRDcsnWJ0doRAHE02jHwjzl x3Gak5bzB5M4vfjOcV9bhQ5DZw4tXvNyH9fEY+4XL/HEL8l3H4K2g5WZ8ZIKnoDbs9 Vj+Xl7yNFdnDplAhdnKNV/tjbz1cnh3wIzpbihaK1eqDlOtjlXNVR9bIXNyvGfypEm ygXG2wWRF18xWrJPKAh97N1+jik/gdlmbs3Kt1t8ExrO7xDeeoJKQIwoFL44MXou5C zpDszM2snNCzdj2YqmbVEF14BN1oTysf2DnF4wNhCN7pGd5AfktnK6wgEjnJaObOJ/ xsxTD0qxTBfAw== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Yong He , Neeraj upadhyay , Joel Fernandes , Zhouyi Zhou , Boqun Feng , Uladzislau Rezki , RCU Subject: [PATCH 1/5] srcu: Fix callbacks acceleration mishandling Date: Wed, 4 Oct 2023 01:28:59 +0200 Message-ID: <20231003232903.7109-2-frederic@kernel.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231003232903.7109-1-frederic@kernel.org> References: <20231003232903.7109-1-frederic@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org SRCU callbacks acceleration might fail if the preceding callbacks advance also fails. This can happen when the following steps are met: 1) The RCU_WAIT_TAIL segment has callbacks (say for gp_num 8) and the RCU_NEXT_READY_TAIL also has callbacks (say for gp_num 12). 2) The grace period for RCU_WAIT_TAIL is observed as started but not yet completed so rcu_seq_current() returns 4 + SRCU_STATE_SCAN1 = 5. 3) This value is passed to rcu_segcblist_advance() which can't move any segment forward and fails. 4) srcu_gp_start_if_needed() still proceeds with callback acceleration. But then the call to rcu_seq_snap() observes the grace period for the RCU_WAIT_TAIL segment (gp_num 8) as completed and the subsequent one for the RCU_NEXT_READY_TAIL segment as started (ie: 8 + SRCU_STATE_SCAN1 = 9) so it returns a snapshot of the next grace period, which is 16. 5) The value of 16 is passed to rcu_segcblist_accelerate() but the freshly enqueued callback in RCU_NEXT_TAIL can't move to RCU_NEXT_READY_TAIL which already has callbacks for a previous grace period (gp_num = 12). So acceleration fails. 6) Note in all these steps, srcu_invoke_callbacks() hadn't had a chance to run srcu_invoke_callbacks(). Then some very bad outcome may happen if the following happens: 7) Some other CPU races and starts the grace period number 16 before the CPU handling previous steps had a chance. Therefore srcu_gp_start() isn't called on the latter sdp to fix the acceleration leak from previous steps with a new pair of call to advance/accelerate. 8) The grace period 16 completes and srcu_invoke_callbacks() is finally called. All the callbacks from previous grace periods (8 and 12) are correctly advanced and executed but callbacks in RCU_NEXT_READY_TAIL still remain. Then rcu_segcblist_accelerate() is called with a snaphot of 20. 9) Since nothing started the grace period number 20, callbacks stay unhandled. This has been reported in real load: [3144162.608392] INFO: task kworker/136:12:252684 blocked for more than 122 seconds. [3144162.615986] Tainted: G O K 5.4.203-1-tlinux4-0011.1 #1 [3144162.623053] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [3144162.631162] kworker/136:12 D 0 252684 2 0x90004000 [3144162.631189] Workqueue: kvm-irqfd-cleanup irqfd_shutdown [kvm] [3144162.631192] Call Trace: [3144162.631202] __schedule+0x2ee/0x660 [3144162.631206] schedule+0x33/0xa0 [3144162.631209] schedule_timeout+0x1c4/0x340 [3144162.631214] ? update_load_avg+0x82/0x660 [3144162.631217] ? raw_spin_rq_lock_nested+0x1f/0x30 [3144162.631218] wait_for_completion+0x119/0x180 [3144162.631220] ? wake_up_q+0x80/0x80 [3144162.631224] __synchronize_srcu.part.19+0x81/0xb0 [3144162.631226] ? __bpf_trace_rcu_utilization+0x10/0x10 [3144162.631227] synchronize_srcu+0x5f/0xc0 [3144162.631236] irqfd_shutdown+0x3c/0xb0 [kvm] [3144162.631239] ? __schedule+0x2f6/0x660 [3144162.631243] process_one_work+0x19a/0x3a0 [3144162.631244] worker_thread+0x37/0x3a0 [3144162.631247] kthread+0x117/0x140 [3144162.631247] ? process_one_work+0x3a0/0x3a0 [3144162.631248] ? __kthread_cancel_work+0x40/0x40 [3144162.631250] ret_from_fork+0x1f/0x30 Fix this with taking the snapshot for acceleration _before_ the read of the current grace period number. The only side effect of this solution is that callbacks advancing happen then _after_ the full barrier in rcu_seq_snap(). This is not a problem because that barrier only cares about: 1) Ordering accesses of the update side before call_srcu() so they don't bleed. 2) See all the accesses prior to the grace period of the current gp_num The only things callbacks advancing need to be ordered against are carried by snp locking. Reported-by: Yong He Co-developed-by: Yong He Co-developed-by: Joel Fernandes Co-developed-by: Neeraj upadhyay Link: http://lore.kernel.org/CANZk6aR+CqZaqmMWrC2eRRPY12qAZnDZLwLnHZbNi=xXMB401g@mail.gmail.com Fixes: da915ad5cf25 ("srcu: Parallelize callback handling") Signed-off-by: Frederic Weisbecker --- kernel/rcu/srcutree.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 5602042856b1..9fab9ac36996 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1244,10 +1244,37 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp, spin_lock_irqsave_sdp_contention(sdp, &flags); if (rhp) rcu_segcblist_enqueue(&sdp->srcu_cblist, rhp); - rcu_segcblist_advance(&sdp->srcu_cblist, - rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); + /* + * The snapshot for acceleration must be taken _before_ the read of the + * current gp sequence used for advancing, otherwise advancing may fail + * and acceleration may then fail too. + * + * This could happen if: + * + * 1) The RCU_WAIT_TAIL segment has callbacks (gp_num = X + 4) and the + * RCU_NEXT_READY_TAIL also has callbacks (gp_num = X + 8). + * + * 2) The grace period for RCU_WAIT_TAIL is seen as started but not + * completed so rcu_seq_current() returns X + SRCU_STATE_SCAN1. + * + * 3) This value is passed to rcu_segcblist_advance() which can't move + * any segment forward and fails. + * + * 4) srcu_gp_start_if_needed() still proceeds with callback acceleration. + * But then the call to rcu_seq_snap() observes the grace period for the + * RCU_WAIT_TAIL segment as completed and the subsequent one for the + * RCU_NEXT_READY_TAIL segment as started (ie: X + 4 + SRCU_STATE_SCAN1) + * so it returns a snapshot of the next grace period, which is X + 12. + * + * 5) The value of X + 12 is passed to rcu_segcblist_accelerate() but the + * freshly enqueued callback in RCU_NEXT_TAIL can't move to + * RCU_NEXT_READY_TAIL which already has callbacks for a previous grace + * period (gp_num = X + 8). So acceleration fails. + */ s = rcu_seq_snap(&ssp->srcu_sup->srcu_gp_seq); - (void)rcu_segcblist_accelerate(&sdp->srcu_cblist, s); + rcu_segcblist_advance(&sdp->srcu_cblist, + rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); + WARN_ON_ONCE(!rcu_segcblist_accelerate(&sdp->srcu_cblist, s) && rhp); if (ULONG_CMP_LT(sdp->srcu_gp_seq_needed, s)) { sdp->srcu_gp_seq_needed = s; needgp = true; From patchwork Tue Oct 3 23:29:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13408092 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3899E8FDBF for ; Tue, 3 Oct 2023 23:29:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236243AbjJCX3U (ORCPT ); Tue, 3 Oct 2023 19:29:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42918 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236052AbjJCX3U (ORCPT ); Tue, 3 Oct 2023 19:29:20 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98382AF; Tue, 3 Oct 2023 16:29:16 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0AB0DC433CB; Tue, 3 Oct 2023 23:29:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1696375756; bh=CgxY0LqJVEeiQkh649qa4IrU9jtDS9lKPNw7zJo199M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GdXvGbF/AWxAYuxckUhWjwbmC2g8wVhiZA+GsiwvL2w7wW+3ZfahRlEMz5eqlQiuZ A8IJPuFHWxv1q/wlKOLNIJY1WtvDfMnk8i3N7Q48BOaAgWNkNDhGw1mujdUdI2cw3y 81C8Ty5CpnNlhTT9W1J4Ucp/RoAQYbKGdM0M9gRpiDwWR9pFGqCdZPBn7WzFKFbdBX Y8SWXIfR5NbltrTaLhYEjbCq/H50C4OQPp+WNM5isy2ztzmkDsomYm/ok6XTUImG1a milVgIIcWpGmrvc69skdFZ9uWD0ycflSvBh02qpWAjAkibTQBcGs+PfDUT//aZRWL3 E5H2XdgVc5U5g== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Yong He , Neeraj upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , RCU Subject: [PATCH 2/5] srcu: Only accelerate on enqueue time Date: Wed, 4 Oct 2023 01:29:00 +0200 Message-ID: <20231003232903.7109-3-frederic@kernel.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231003232903.7109-1-frederic@kernel.org> References: <20231003232903.7109-1-frederic@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org Acceleration in SRCU happens on enqueue time for each new callback. This operation is expected not to fail and therefore any similar attempt from other places shouldn't find any remaining callbacks to accelerate. Moreover accelerations performed beyond enqueue time are error prone because rcu_seq_snap() then may return the snapshot for a new grace period that is not going to be started. Remove these dangerous and needless accelerations and introduce instead assertions reporting leaking unaccelerated callbacks beyond enqueue time. Co-developed-by: Yong He Co-developed-by: Joel Fernandes Co-developed-by: Neeraj upadhyay Signed-off-by: Frederic Weisbecker Reviewed-by: Like Xu --- kernel/rcu/srcutree.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 9fab9ac36996..560e99ec5333 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -784,8 +784,7 @@ static void srcu_gp_start(struct srcu_struct *ssp) spin_lock_rcu_node(sdp); /* Interrupts already disabled. */ rcu_segcblist_advance(&sdp->srcu_cblist, rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); - (void)rcu_segcblist_accelerate(&sdp->srcu_cblist, - rcu_seq_snap(&ssp->srcu_sup->srcu_gp_seq)); + WARN_ON_ONCE(!rcu_segcblist_segempty(&sdp->srcu_cblist, RCU_NEXT_TAIL)); spin_unlock_rcu_node(sdp); /* Interrupts remain disabled. */ WRITE_ONCE(ssp->srcu_sup->srcu_gp_start, jiffies); WRITE_ONCE(ssp->srcu_sup->srcu_n_exp_nodelay, 0); @@ -1721,6 +1720,7 @@ static void srcu_invoke_callbacks(struct work_struct *work) ssp = sdp->ssp; rcu_cblist_init(&ready_cbs); spin_lock_irq_rcu_node(sdp); + WARN_ON_ONCE(!rcu_segcblist_segempty(&sdp->srcu_cblist, RCU_NEXT_TAIL)); rcu_segcblist_advance(&sdp->srcu_cblist, rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); if (sdp->srcu_cblist_invoking || @@ -1750,8 +1750,6 @@ static void srcu_invoke_callbacks(struct work_struct *work) */ spin_lock_irq_rcu_node(sdp); rcu_segcblist_add_len(&sdp->srcu_cblist, -len); - (void)rcu_segcblist_accelerate(&sdp->srcu_cblist, - rcu_seq_snap(&ssp->srcu_sup->srcu_gp_seq)); sdp->srcu_cblist_invoking = false; more = rcu_segcblist_ready_cbs(&sdp->srcu_cblist); spin_unlock_irq_rcu_node(sdp); From patchwork Tue Oct 3 23:29:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13408093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66F3AE8FDC0 for ; Tue, 3 Oct 2023 23:29:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236052AbjJCX3Z (ORCPT ); Tue, 3 Oct 2023 19:29:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42984 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236651AbjJCX3X (ORCPT ); Tue, 3 Oct 2023 19:29:23 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3E766DC; Tue, 3 Oct 2023 16:29:19 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A0123C433C7; Tue, 3 Oct 2023 23:29:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1696375758; bh=v3XWzp5y3b223EZI0Adeq7leNILWY4Ny2s3VXNJdVDs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rxBFBO9fexrgAIW8L6p/0lDFR+U7WAcxIocLnY5i/UnkELy4bW/aP796oq73qPyDe GgqjDQK5xFLEp4d9OxZ7X1QP5pGMLkxZDlpb4v7PNi+0s+guASOFbwUYqeCBCMaUFC lk7UIp12bc6DjYNl2ccy5N5xrHAPtnGqFQDiXFEjxGN8IEKvA9qPfwRgQzBL7HGvIx AgMOTbk04C7loWaFhVWUwVft01hR9G4WyrkiM3bnFKNyFDSkH+4bdVgsMUedtGVf7o 1PLlBmv41w4svke1Z8RSZd1rj1N0UBikaJTJq2i8hidOcZuLTXbNwmJ2gKPM2Va0Bk 4Ja4UJsw9BAKQ== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Yong He , Neeraj upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , RCU Subject: [PATCH 3/5] srcu: Remove superfluous callbacks advancing from srcu_start_gp() Date: Wed, 4 Oct 2023 01:29:01 +0200 Message-ID: <20231003232903.7109-4-frederic@kernel.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231003232903.7109-1-frederic@kernel.org> References: <20231003232903.7109-1-frederic@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org Callbacks advancing on SRCU must be performed on two specific places: 1) On enqueue time in order to make room for the acceleration of the new callback. 2) On invocation time in order to move the callbacks ready to invoke. Any other callback advancing callsite is needless. Remove the remaining one in srcu_gp_start(). Co-developed-by: Yong He Co-developed-by: Joel Fernandes Co-developed-by: Neeraj upadhyay Signed-off-by: Frederic Weisbecker --- kernel/rcu/srcutree.c | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 560e99ec5333..e9356a103626 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -772,20 +772,10 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock_nmisafe); */ static void srcu_gp_start(struct srcu_struct *ssp) { - struct srcu_data *sdp; int state; - if (smp_load_acquire(&ssp->srcu_sup->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) - sdp = per_cpu_ptr(ssp->sda, get_boot_cpu_id()); - else - sdp = this_cpu_ptr(ssp->sda); lockdep_assert_held(&ACCESS_PRIVATE(ssp->srcu_sup, lock)); WARN_ON_ONCE(ULONG_CMP_GE(ssp->srcu_sup->srcu_gp_seq, ssp->srcu_sup->srcu_gp_seq_needed)); - spin_lock_rcu_node(sdp); /* Interrupts already disabled. */ - rcu_segcblist_advance(&sdp->srcu_cblist, - rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); - WARN_ON_ONCE(!rcu_segcblist_segempty(&sdp->srcu_cblist, RCU_NEXT_TAIL)); - spin_unlock_rcu_node(sdp); /* Interrupts remain disabled. */ WRITE_ONCE(ssp->srcu_sup->srcu_gp_start, jiffies); WRITE_ONCE(ssp->srcu_sup->srcu_n_exp_nodelay, 0); smp_mb(); /* Order prior store to ->srcu_gp_seq_needed vs. GP start. */ From patchwork Tue Oct 3 23:29:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13408094 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CF8FE8FDBF for ; Tue, 3 Oct 2023 23:29:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236248AbjJCX30 (ORCPT ); Tue, 3 Oct 2023 19:29:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43032 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236317AbjJCX3Z (ORCPT ); Tue, 3 Oct 2023 19:29:25 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0116890; Tue, 3 Oct 2023 16:29:21 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 827CEC433CC; Tue, 3 Oct 2023 23:29:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1696375761; bh=AYpsqjGYQerYpEpgb3/3JuONsnqpe9k5KpicXU2iC5A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Vb8yt7rj1+eASMqdgd8uOd7YztQbD2LY9C8D6UKrRfCDQEWlcvcvXP+pYsO1KYl3F wKo7Ipua6EfHoRWZg6Gh9t3205hor55zobDIFfZQK+/0sBOlmNyXAzW5kyHOveRedJ tlpo2xc6XMDQkLBC/d/cAioxpFZ/Tb40pP0QU+M4nhFeRad5DIaZ7oQZLxwrmVzEvr BH2bP6bdKh3ey9tW6/I+GMMIihWm9SNc5TC0i9t83zpy4b+Ge+x4qt/Sohw3GLHgN+ LUFyxegCWNp/NlJAT09G8e1UXQu8Xg/FZnZ79+rrMLxk6hfFfzi3QJZ0bKBz97ZPSI a7srEntCpnslg== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Yong He , Neeraj upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , RCU Subject: [PATCH 4/5] srcu: No need to advance/accelerate if no callback enqueued Date: Wed, 4 Oct 2023 01:29:02 +0200 Message-ID: <20231003232903.7109-5-frederic@kernel.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231003232903.7109-1-frederic@kernel.org> References: <20231003232903.7109-1-frederic@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org While in grace period start, there is nothing to accelerate and therefore no need to advance the callbacks either if no callback is to be enqueued. Spare these needless operations in this case. Signed-off-by: Frederic Weisbecker --- kernel/rcu/srcutree.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index e9356a103626..2bfc8ed1eed2 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1261,9 +1261,11 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp, * period (gp_num = X + 8). So acceleration fails. */ s = rcu_seq_snap(&ssp->srcu_sup->srcu_gp_seq); - rcu_segcblist_advance(&sdp->srcu_cblist, - rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); - WARN_ON_ONCE(!rcu_segcblist_accelerate(&sdp->srcu_cblist, s) && rhp); + if (rhp) { + rcu_segcblist_advance(&sdp->srcu_cblist, + rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); + WARN_ON_ONCE(!rcu_segcblist_accelerate(&sdp->srcu_cblist, s)); + } if (ULONG_CMP_LT(sdp->srcu_gp_seq_needed, s)) { sdp->srcu_gp_seq_needed = s; needgp = true; From patchwork Tue Oct 3 23:29:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13408095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D43B2E8FDC0 for ; Tue, 3 Oct 2023 23:29:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236839AbjJCX3e (ORCPT ); Tue, 3 Oct 2023 19:29:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53746 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236767AbjJCX33 (ORCPT ); Tue, 3 Oct 2023 19:29:29 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D0F92F0; Tue, 3 Oct 2023 16:29:24 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 33129C433CA; Tue, 3 Oct 2023 23:29:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1696375764; bh=0rRo8FAmb5oxl6EZiJPhw9pHIJx5I2TobFmAd6Jx3js=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ckZ84jL+5znrqWKHYEA+9b1Rw3RT0Gi+UXmpmDTGlZhoPinert8WUNxoDLU4o56nk dmedaAxQJKrj7ntNQVKbPtnO2xlcoeh9TptXuh3uHCcLUzchAgQQSAxWftjDL7dHI4 F0GlfzowXeBcgPjhVYmPfr5Vtqv1pKYzqUVAEbkdOOLw4rADsiMRppbrszlTsiYIXp najipfPoK5n//X9Kg/cvv3L1npieqZqr+QdozUZ9ALZVuzsIdeTaN32AS/NkaSNHQV vrd4UXNW8B3aGZGyz0IN4a6Eb7zGdywILKMBiwNxLEtYz+WsYVe5USMhK4ClYS+sxK PV7LRnOBhW8iw== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Yong He , Neeraj upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , RCU Subject: [PATCH 5/5] srcu: Explain why callbacks invocations can't run concurrently Date: Wed, 4 Oct 2023 01:29:03 +0200 Message-ID: <20231003232903.7109-6-frederic@kernel.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231003232903.7109-1-frederic@kernel.org> References: <20231003232903.7109-1-frederic@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org If an SRCU barrier is queued while callbacks are running and a new callbacks invocator for the same sdp were to run concurrently, the RCU barrier might execute too early. As this requirement is non-obvious, make sure to keep a record. Signed-off-by: Frederic Weisbecker --- kernel/rcu/srcutree.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 2bfc8ed1eed2..0351a4e83529 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1715,6 +1715,11 @@ static void srcu_invoke_callbacks(struct work_struct *work) WARN_ON_ONCE(!rcu_segcblist_segempty(&sdp->srcu_cblist, RCU_NEXT_TAIL)); rcu_segcblist_advance(&sdp->srcu_cblist, rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); + /* + * Although this function is theoretically re-entrant, concurrent + * callbacks invocation is disallowed to avoid executing an SRCU barrier + * too early. + */ if (sdp->srcu_cblist_invoking || !rcu_segcblist_ready_cbs(&sdp->srcu_cblist)) { spin_unlock_irq_rcu_node(sdp); @@ -1745,6 +1750,7 @@ static void srcu_invoke_callbacks(struct work_struct *work) sdp->srcu_cblist_invoking = false; more = rcu_segcblist_ready_cbs(&sdp->srcu_cblist); spin_unlock_irq_rcu_node(sdp); + /* An SRCU barrier or callbacks from previous nesting work pending */ if (more) srcu_schedule_cbs_sdp(sdp, 0); }