From patchwork Wed Mar 15 19:43:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13176613 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAD9BC6FD1D for ; Wed, 15 Mar 2023 19:44:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231802AbjCOToe (ORCPT ); Wed, 15 Mar 2023 15:44:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47370 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229546AbjCOTod (ORCPT ); Wed, 15 Mar 2023 15:44:33 -0400 Received: from mail-qt1-x82f.google.com (mail-qt1-x82f.google.com [IPv6:2607:f8b0:4864:20::82f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98396136FB for ; Wed, 15 Mar 2023 12:44:30 -0700 (PDT) Received: by mail-qt1-x82f.google.com with SMTP id y10so17531412qtj.2 for ; Wed, 15 Mar 2023 12:44:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1678909469; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=jqFrNqITGGDLlhPYnkJjrrLRlwOYf2ibTxsThXGpk8A=; b=VEyLp9OjLmJF5YzG1BBVxZJwxAFxMj+a4hkb4juSICDtdMWn8iT2nSGV1mAMg+JGAf usO9i856r9iQFDut5RGxvV/cwJqOKC6yQiQqrFSVmBrJqPxApDUBvZ0+dfZ3iMEtsCLO XfvJmsZnSgkJMKtqKzf1Wmk2eEKQgYX45tu9c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678909469; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=jqFrNqITGGDLlhPYnkJjrrLRlwOYf2ibTxsThXGpk8A=; b=j0O0VQJqIHpxHY/kS4vpwbLRDO3h1s2fl9f+lA5phjtvjOGcBmV2eNS/6mEDox912c bIUqugzKMvjXehdVr8Tx54c8QzAIBK+vi/IIqPdvXag7zrdpYEznXyaSgrGDYH4QZvpq OiEtpAbaevqNjGH4VFPoaLctS0ic2QJ23s2l+8l8+Y5WHklDXI+5tlO7d7ME4k9+pUF8 4BfIAfU+C9OReElakJpXthurnWUih+6LeZkw9urfPpRAtc5XeZpHFY7JhUxF0++XW81e 59iXRhf3htP3Sm/zjjbR732CAKpSdrFai/zitIjsjCNMpe6pa7eylZwvvvCdenQIqWTB 8k5Q== X-Gm-Message-State: AO0yUKUHaZrVb7dQQYw6+ScOikFFRTERDjRooHrG10/jRZSzr6h7WFLY WxouwaZlGmY1+pF5NAClDnsCxvmlUeBiJf7ULt8= X-Google-Smtp-Source: AK7set9dfK434dXDr0yIat6SPxVU8XoMS5ZcP+FDLE+fW7L96CzOkOycO/RXcxTtUb42myef7phsBQ== X-Received: by 2002:a05:622a:60f:b0:3b7:ec70:30af with SMTP id z15-20020a05622a060f00b003b7ec7030afmr1753785qta.46.1678909468933; Wed, 15 Mar 2023 12:44:28 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id c15-20020ac8660f000000b003b86b088755sm4346666qtp.15.2023.03.15.12.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 12:44:28 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes Cc: Zqiang , rcu@vger.kernel.org Subject: [PATCH 1/9] rcu: Fix set/clear TICK_DEP_BIT_RCU_EXP bitmask race Date: Wed, 15 Mar 2023 19:43:41 +0000 Message-Id: <20230315194349.10798-1-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang For kernels built with CONFIG_NO_HZ_FULL=y, the following scenario can result in the scheduling-clock interrupt remaining enabled on a holdout CPU after its quiescent state has been reported: CPU1 CPU2 rcu_report_exp_cpu_mult synchronize_rcu_expedited_wait acquires rnp->lock mask = rnp->expmask; for_each_leaf_node_cpu_mask(rnp, cpu, mask) rnp->expmask = rnp->expmask & ~mask; rdp = per_cpu_ptr(&rcu_data, cpu1); for_each_leaf_node_cpu_mask(rnp, cpu, mask) rdp = per_cpu_ptr(&rcu_data, cpu1); if (!rdp->rcu_forced_tick_exp) continue; rdp->rcu_forced_tick_exp = true; tick_dep_set_cpu(cpu1, TICK_DEP_BIT_RCU_EXP); The problem is that CPU2's sampling of rnp->expmask is obsolete by the time it invokes tick_dep_set_cpu(), and CPU1 is not guaranteed to see CPU2's store to ->rcu_forced_tick_exp in time to clear it. And even if CPU1 does see that store, it might invoke tick_dep_clear_cpu() before CPU2 got around to executing its tick_dep_set_cpu(), which would still leave the victim CPU with its scheduler-clock tick running. Either way, an nohz_full real-time application running on the victim CPU would have its latency needlessly degraded. Note that expedited RCU grace periods look at context-tracking information, and so if the CPU is executing in nohz_full usermode throughout, that CPU cannot be victimized in this manner. This commit therefore causes synchronize_rcu_expedited_wait to hold the rcu_node structure's ->lock when checking for holdout CPUs, setting TICK_DEP_BIT_RCU_EXP, and invoking tick_dep_set_cpu(), thus preventing this race. Signed-off-by: Zqiang Reviewed-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_exp.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 249c2967d9e6..7cc4856da081 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -594,6 +594,7 @@ static void synchronize_rcu_expedited_wait(void) struct rcu_data *rdp; struct rcu_node *rnp; struct rcu_node *rnp_root = rcu_get_root(); + unsigned long flags; trace_rcu_exp_grace_period(rcu_state.name, rcu_exp_gp_seq_endval(), TPS("startwait")); jiffies_stall = rcu_exp_jiffies_till_stall_check(); @@ -602,17 +603,17 @@ static void synchronize_rcu_expedited_wait(void) if (synchronize_rcu_expedited_wait_once(1)) return; rcu_for_each_leaf_node(rnp) { + raw_spin_lock_irqsave_rcu_node(rnp, flags); mask = READ_ONCE(rnp->expmask); for_each_leaf_node_cpu_mask(rnp, cpu, mask) { rdp = per_cpu_ptr(&rcu_data, cpu); if (rdp->rcu_forced_tick_exp) continue; rdp->rcu_forced_tick_exp = true; - preempt_disable(); if (cpu_online(cpu)) tick_dep_set_cpu(cpu, TICK_DEP_BIT_RCU_EXP); - preempt_enable(); } + raw_spin_unlock_irqrestore_rcu_node(rnp, flags); } j = READ_ONCE(jiffies_till_first_fqs); if (synchronize_rcu_expedited_wait_once(j + HZ)) From patchwork Wed Mar 15 19:43:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13176614 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 864DBC6FD1D for ; Wed, 15 Mar 2023 19:45:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232171AbjCOTpI (ORCPT ); Wed, 15 Mar 2023 15:45:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47824 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232511AbjCOTok (ORCPT ); Wed, 15 Mar 2023 15:44:40 -0400 Received: from mail-qt1-x829.google.com (mail-qt1-x829.google.com [IPv6:2607:f8b0:4864:20::829]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9C48426B2 for ; Wed, 15 Mar 2023 12:44:33 -0700 (PDT) Received: by mail-qt1-x829.google.com with SMTP id l13so17525090qtv.3 for ; Wed, 15 Mar 2023 12:44:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1678909473; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VzLowNCggdMuHVa7t9ihdZAmx6P+qhTQOwYOWCMVniY=; b=X5E0FVsOr52URNj8IKnNJAMyRNN6iAb0W+Te/IP/Rkl5Hpf1U/QqPijvWm3XlbaicC 3mlCYV4ql1Go4r8wPQCmoR9J9NpghgNsyUoeZ8oRouPfvr2gPKGhgdMmlX/lLdQcMHMI R6nYUPAL6WViNKApXzNGke+SYrLWQYJ/Zfe8Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678909473; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VzLowNCggdMuHVa7t9ihdZAmx6P+qhTQOwYOWCMVniY=; b=NouRu5p90Rf/y8jkb2ZrNZDqqk1LqDw3N+inAPyJBzglwGVMiHbNbQwDpMAu8U/Q6j Bp3hNDC4VGHggZ9a+Maxq1PftBh/zN45Xzsx0+C/GaP6UrP+RMzEORRTsEjAa2Z+nL+e SsyhDeS76t2Z33n0TQ8htqExLCQ9KKlw24VG/GjJyZEY+FjIG1UTIWI60r4cMPcLau5o QNMLnMkxpOC8f92SC13kXdX8qV2BlyaXOOwS+WjBiR54faq7vbZW2Fe9wucHww1Dl0Q1 NjKXpJvzWddOI64tpfGoa14a7CVu0ZTXQmBTtkWNXQ8g4XSnV+fbL2ooq/KhBYSm8yfa YKAA== X-Gm-Message-State: AO0yUKVWLZemVJ1X5oYtjYzEBVdCy97yjwjTShLs8IxHlphazUOEK16k PAlThGoalOVhnlMdTWCTbm6mYw== X-Google-Smtp-Source: AK7set9UP2Vz06UydD43DMA5XM+4jGd4zT8Fi3xXIs0gA1il9wwSDKIsH1/iOjul9KKptIOJ9gLXqQ== X-Received: by 2002:ac8:5886:0:b0:3bd:16cf:2f33 with SMTP id t6-20020ac85886000000b003bd16cf2f33mr1586127qta.63.1678909472162; Wed, 15 Mar 2023 12:44:32 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id c15-20020ac8660f000000b003b86b088755sm4346666qtp.15.2023.03.15.12.44.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 12:44:31 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes Cc: Zqiang , rcu@vger.kernel.org Subject: [PATCH 4/9] rcu: Register rcu-lazy shrinker only for CONFIG_RCU_LAZY=y kernels Date: Wed, 15 Mar 2023 19:43:44 +0000 Message-Id: <20230315194349.10798-4-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog In-Reply-To: <20230315194349.10798-1-joel@joelfernandes.org> References: <20230315194349.10798-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang The lazy_rcu_shrink_count() shrinker function is registered even in kernels built with CONFIG_RCU_LAZY=n, in which case this function uselessly consumes cycles learning that no CPU has any lazy callbacks queued. This commit therefore registers this shrinker function only in the kernels built with CONFIG_RCU_LAZY=y, where it might actually do something useful. Signed-off-by: Zqiang Reviewed-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) Reviewed-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 9e1c8caec5ce..f2280616f9d5 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1312,6 +1312,7 @@ int rcu_nocb_cpu_offload(int cpu) } EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload); +#ifdef CONFIG_RCU_LAZY static unsigned long lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) { @@ -1360,6 +1361,7 @@ static struct shrinker lazy_rcu_shrinker = { .batch = 0, .seeks = DEFAULT_SEEKS, }; +#endif // #ifdef CONFIG_RCU_LAZY void __init rcu_init_nohz(void) { @@ -1391,8 +1393,10 @@ void __init rcu_init_nohz(void) if (!rcu_state.nocb_is_setup) return; +#ifdef CONFIG_RCU_LAZY if (register_shrinker(&lazy_rcu_shrinker, "rcu-lazy")) pr_err("Failed to register lazy_rcu shrinker!\n"); +#endif // #ifdef CONFIG_RCU_LAZY if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) { pr_info("\tNote: kernel parameter 'rcu_nocbs=', 'nohz_full', or 'isolcpus=' contains nonexistent CPUs.\n"); From patchwork Wed Mar 15 19:43:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13176615 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A387C7618A for ; Wed, 15 Mar 2023 19:45:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232375AbjCOTpJ (ORCPT ); Wed, 15 Mar 2023 15:45:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47952 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232345AbjCOTom (ORCPT ); Wed, 15 Mar 2023 15:44:42 -0400 Received: from mail-qt1-x82f.google.com (mail-qt1-x82f.google.com [IPv6:2607:f8b0:4864:20::82f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 839CA6EA7 for ; Wed, 15 Mar 2023 12:44:34 -0700 (PDT) Received: by mail-qt1-x82f.google.com with SMTP id t9so1349773qtx.8 for ; Wed, 15 Mar 2023 12:44:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1678909473; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ThU1XJtO1aLExHCNuOZtJgSY8HbvsH7BE8/boByrYno=; b=ag5X7WW6rfx5+7t1/eBsvLbRkB0y7Vb/fHHyhSxrRthTHHyDI7UjKzwlHHipX9cFtF Kka1hk8oKQDU69luA4VDWIBakHNnDiGrOW8TUl6Tbt1TJYOVUHXpB2E2TdW5uov5uc98 nrZnUhAVT3BLC9hxVe2eeMxuyFGzvMETCnPyU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678909473; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ThU1XJtO1aLExHCNuOZtJgSY8HbvsH7BE8/boByrYno=; b=RAe37afrysbkmNfVwzBAykJJxcEQ+XWYYGNpCZBD78ktWDayWlJN7ZF3OelkeJqO3A LclM8dthn6ZhWxq2XRsG4xgPa9uYJoWF3C82OJudLIFyIpW0orIO+1YPeL6FN3cevP50 gDRAfrgdy0snw+PUUAqTfqpS5LlDPATp1ccAgrgiXDuhODXWql+7PfG8Fu+K8Blvw1PL +A0qtLfffSacyn7qqE92nEM89G2pzWPZMW/JALvFHTXwkIGbvYkfXj8svV5Y/RCyMCVh 2bcC8zYCrauo5uL6Sq6okB+ox27643LHyZCsMY39fONBJ+tSAH/j+jKY1oG030fr/Ois qPDg== X-Gm-Message-State: AO0yUKUV0CwicexTzy9dirv6uWm6kQb7BvvejsjNpEtKaifiKGlB4o0W gMOPcxCQNBxfYpgHGlO69n1FAw== X-Google-Smtp-Source: AK7set9eHeATCHiRK0lgFQHYvnJM3VGTCsQZJTXU87E+PPKBbb5RQF3SZBaL8sWIrBT9HAPSQX7+ug== X-Received: by 2002:a05:622a:1902:b0:3b6:93fe:443c with SMTP id w2-20020a05622a190200b003b693fe443cmr1567792qtc.32.1678909473597; Wed, 15 Mar 2023 12:44:33 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id c15-20020ac8660f000000b003b86b088755sm4346666qtp.15.2023.03.15.12.44.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 12:44:33 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes Cc: Zqiang , rcu@vger.kernel.org Subject: [PATCH 5/9] rcu: Remove never-set needwake assignment from rcu_report_qs_rdp() Date: Wed, 15 Mar 2023 19:43:45 +0000 Message-Id: <20230315194349.10798-5-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog In-Reply-To: <20230315194349.10798-1-joel@joelfernandes.org> References: <20230315194349.10798-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang The rcu_accelerate_cbs() function is invoked by rcu_report_qs_rdp() only if there is a grace period in progress that is still blocked by at least one CPU on this rcu_node structure. This means that rcu_accelerate_cbs() should never return the value true, and thus that this function should never set the needwake variable and in turn never invoke rcu_gp_kthread_wake(). This commit therefore removes the needwake variable and the invocation of rcu_gp_kthread_wake() in favor of a WARN_ON_ONCE() on the call to rcu_accelerate_cbs(). The purpose of this new WARN_ON_ONCE() is to detect situations where the system's opinion differs from ours. Signed-off-by: Zqiang Reviewed-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 8e880c09ab59..e80e8f128c57 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1955,7 +1955,6 @@ rcu_report_qs_rdp(struct rcu_data *rdp) { unsigned long flags; unsigned long mask; - bool needwake = false; bool needacc = false; struct rcu_node *rnp; @@ -1987,7 +1986,12 @@ rcu_report_qs_rdp(struct rcu_data *rdp) * NOCB kthreads have their own way to deal with that... */ if (!rcu_rdp_is_offloaded(rdp)) { - needwake = rcu_accelerate_cbs(rnp, rdp); + /* + * The current GP has not yet ended, so it + * should not be possible for rcu_accelerate_cbs() + * to return true. So complain, but don't awaken. + */ + WARN_ON_ONCE(rcu_accelerate_cbs(rnp, rdp)); } else if (!rcu_segcblist_completely_offloaded(&rdp->cblist)) { /* * ...but NOCB kthreads may miss or delay callbacks acceleration @@ -1999,8 +2003,6 @@ rcu_report_qs_rdp(struct rcu_data *rdp) rcu_disable_urgency_upon_qs(rdp); rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags); /* ^^^ Released rnp->lock */ - if (needwake) - rcu_gp_kthread_wake(); if (needacc) { rcu_nocb_lock_irqsave(rdp, flags); From patchwork Wed Mar 15 19:43:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13176616 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEABCC7618D for ; Wed, 15 Mar 2023 19:45:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232498AbjCOTpK (ORCPT ); Wed, 15 Mar 2023 15:45:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232664AbjCOToq (ORCPT ); Wed, 15 Mar 2023 15:44:46 -0400 Received: from mail-qt1-x82e.google.com (mail-qt1-x82e.google.com [IPv6:2607:f8b0:4864:20::82e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A9DC02BEDF for ; Wed, 15 Mar 2023 12:44:35 -0700 (PDT) Received: by mail-qt1-x82e.google.com with SMTP id r16so17474578qtx.9 for ; Wed, 15 Mar 2023 12:44:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1678909475; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3K8lekfNTHeJ4C0iv99fjGghn93IIX905cIt06EpTfQ=; b=cKuWC+aHDyxATa6HN+YqE2E5Okxl3eR7i8zhoQiYjbzAOf2vN+cKz3FWTds5n8vFyZ 4FHsCNQavFITJZv8en/rMnQhzx8Bu1Ufn4X/caDxoFZldJHM/1+37+83SF2L6Q6HM4me bh0PyPNf3ZJ5AH49Aq7N5gzNJEuxy5pZTENp0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678909475; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3K8lekfNTHeJ4C0iv99fjGghn93IIX905cIt06EpTfQ=; b=Y0bIvJ/oHSBfykJiVgmqVCjV2H+lddNzViuijyJny6qfk3WMI0v7EecpDXsYY+vKpZ l+BAIjDawm+fxG8bMhDEZ45ViDA4RH6E9t3de4KEro4eRMvOLp1aDdMi/SBhsGQruTa0 +WS/qT1rDXLJ9HMn0WkI+ZDe4wxnvwj6Hwb3XEZCUX6rGeLYkIw1R8N/qexBfdoRodKN tvhXoy7XnSI50a0cx9O9TM9GXcw6CXiHXrkM5kLrmgJ0VfLG74jiTpwZqMWOZTyvvOKj XnScZ9Y9Y3Y5gnLpGWIWZh7RXrjXXR0wBUYGDBsd4JPHVgaLBZxAggId4K65fnFvVqiU BeAg== X-Gm-Message-State: AO0yUKWlFoFcA03IvYntTEKgEdMWj11bNQhkcGtrdf+6RBYdXlHFQna6 1em7VQ58VlEPWnSgc3stQmXtng== X-Google-Smtp-Source: AK7set+LT+EegIShwTd4ywC4A9b7O5vXSYEk4EuLY9iyTgGFwjNnq/dG6dMyVv6nragGJ8LRGB3skQ== X-Received: by 2002:a05:622a:30d:b0:3bf:c407:10ca with SMTP id q13-20020a05622a030d00b003bfc40710camr1981909qtw.10.1678909474756; Wed, 15 Mar 2023 12:44:34 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id c15-20020ac8660f000000b003b86b088755sm4346666qtp.15.2023.03.15.12.44.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 12:44:34 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes Cc: Zqiang , rcu@vger.kernel.org Subject: [PATCH 6/9] rcu: Permit start_poll_synchronize_rcu_expedited() to be invoked early Date: Wed, 15 Mar 2023 19:43:46 +0000 Message-Id: <20230315194349.10798-6-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog In-Reply-To: <20230315194349.10798-1-joel@joelfernandes.org> References: <20230315194349.10798-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang According to the commit log of the patch that added it to the kernel, start_poll_synchronize_rcu_expedited() can be invoked very early, as in long before rcu_init() has been invoked. But before rcu_init(), the rcu_data structure's ->mynode field has not yet been initialized. This means that the start_poll_synchronize_rcu_expedited() function's attempt to set the CPU's leaf rcu_node structure's ->exp_seq_poll_rq field will result in a segmentation fault. This commit therefore causes start_poll_synchronize_rcu_expedited() to set ->exp_seq_poll_rq only after rcu_init() has initialized all CPUs' rcu_data structures' ->mynode fields. It also removes the check from the rcu_init() function so that start_poll_synchronize_rcu_expedited( is unconditionally invoked. Yes, this might result in an unnecessary boot-time grace period, but this is down in the noise. Signed-off-by: Zqiang Reviewed-by: Frederic Weisbecker Reviewed-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 5 ++--- kernel/rcu/tree_exp.h | 5 +++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index e80e8f128c57..90d54571126a 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4942,9 +4942,8 @@ void __init rcu_init(void) else qovld_calc = qovld; - // Kick-start any polled grace periods that started early. - if (!(per_cpu_ptr(&rcu_data, cpu)->mynode->exp_seq_poll_rq & 0x1)) - (void)start_poll_synchronize_rcu_expedited(); + // Kick-start in case any polled grace periods started early. + (void)start_poll_synchronize_rcu_expedited(); rcu_test_sync_prims(); } diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 7cc4856da081..5343f32e7d67 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -1066,9 +1066,10 @@ unsigned long start_poll_synchronize_rcu_expedited(void) if (rcu_init_invoked()) raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags); if (!poll_state_synchronize_rcu(s)) { - rnp->exp_seq_poll_rq = s; - if (rcu_init_invoked()) + if (rcu_init_invoked()) { + rnp->exp_seq_poll_rq = s; queue_work(rcu_gp_wq, &rnp->exp_poll_wq); + } } if (rcu_init_invoked()) raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags); From patchwork Wed Mar 15 19:43:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13176619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A769EC61DA4 for ; Wed, 15 Mar 2023 19:45:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232186AbjCOTpM (ORCPT ); Wed, 15 Mar 2023 15:45:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47776 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232702AbjCOTor (ORCPT ); Wed, 15 Mar 2023 15:44:47 -0400 Received: from mail-qt1-x830.google.com (mail-qt1-x830.google.com [IPv6:2607:f8b0:4864:20::830]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D25EF4FAB0 for ; Wed, 15 Mar 2023 12:44:36 -0700 (PDT) Received: by mail-qt1-x830.google.com with SMTP id cf14so17462352qtb.10 for ; Wed, 15 Mar 2023 12:44:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1678909476; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=208GxfDZm7oSyOpPWnvVrrbR0LSr9U2B1RRikPcTIHU=; b=SwfZcZz22GQ86AfZbWhrkMmC0APeen3E/TYOlJxwfPQj5qa3k998D+cUVTsr+OrgyK WvbfHiF17cuaKhsWGLmCE3S/m6YnKZ9M/k74AU81Siisp2uBs9tI/TxBZD5TyjWVi8wS x5b4+KHAhLFQQO1DjIaf7rO6WNKuOgp35UykA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678909476; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=208GxfDZm7oSyOpPWnvVrrbR0LSr9U2B1RRikPcTIHU=; b=OFs/2PDsddXHPDirfK+hOUJoidhQABAosgJoK36DKXVpgwQ0QmtHj+SAfrPHZdv7gc lNyflHgbHp76r51Cj7SMgDnjoBTJDkh6ufFtGNyyUx96PoFMI7X28x0zcOoScDT48wn9 K/zQ7roC94I3iMmKFmxZsDyqaC+imsQbYWfjvg2rLT+vMG8ICtaZ/Y6V+xkfYQ2l8cqu N1kDYwgvRl1rdtyegw3D73sLCGz7MptbWYD0cQXusqdvZTlaNH0XcfOMd0qzghoA6Q63 tJUMHCbZTZ7QiTOLXaFZizZ/eKLtNmJ4gJo05dhGNfTJ/RVL1hAgvR1xCQ/JacQeqR+X NHsg== X-Gm-Message-State: AO0yUKUk2kQVK7v9zM02HqKDlGespCj0N7r2x74Fv5X+wpyBmdeyC5BQ eSwTYRFUFTCbMmFHPb287KYoKg== X-Google-Smtp-Source: AK7set+CAI/fwx2l399eAuZ9xV5Jlp4O6BubsvjdMkMS6+QB/t270bnUo23l5y+5gqaF6IPk8+QwXw== X-Received: by 2002:ac8:5c13:0:b0:3d4:8ce9:cef7 with SMTP id i19-20020ac85c13000000b003d48ce9cef7mr2038670qti.8.1678909475786; Wed, 15 Mar 2023 12:44:35 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id c15-20020ac8660f000000b003b86b088755sm4346666qtp.15.2023.03.15.12.44.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 12:44:35 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes Cc: Mark Brown , rcu@vger.kernel.org Subject: [PATCH 7/9] rcu-tasks: Report stalls during synchronize_srcu() in rcu_tasks_postscan() Date: Wed, 15 Mar 2023 19:43:47 +0000 Message-Id: <20230315194349.10798-7-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog In-Reply-To: <20230315194349.10798-1-joel@joelfernandes.org> References: <20230315194349.10798-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Neeraj Upadhyay The call to synchronize_srcu() from rcu_tasks_postscan() can be stalled by a task getting stuck in do_exit() between that function's calls to exit_tasks_rcu_start() and exit_tasks_rcu_finish(). To ease diagnosis of this situation, print a stall warning message every rcu_task_stall_info period when rcu_tasks_postscan() is stalled. [ paulmck: Adjust to handle CONFIG_SMP=n. ] Reported-by: Mark Brown Link: https://lore.kernel.org/rcu/20230111212736.GA1062057@paulmck-ThinkPad-P17-Gen-1/ Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tasks.h | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index bfb5e1549f2b..baf7ec178155 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -139,6 +139,12 @@ static struct rcu_tasks rt_name = \ /* Track exiting tasks in order to allow them to be waited for. */ DEFINE_STATIC_SRCU(tasks_rcu_exit_srcu); +#ifdef CONFIG_TASKS_RCU +/* Report delay in synchronize_srcu() completion in rcu_tasks_postscan(). */ +static void tasks_rcu_exit_srcu_stall(struct timer_list *unused); +static DEFINE_TIMER(tasks_rcu_exit_srcu_stall_timer, tasks_rcu_exit_srcu_stall); +#endif + /* Avoid IPIing CPUs early in the grace period. */ #define RCU_TASK_IPI_DELAY (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB) ? HZ / 2 : 0) static int rcu_task_ipi_delay __read_mostly = RCU_TASK_IPI_DELAY; @@ -830,6 +836,13 @@ static void rcu_tasks_pertask(struct task_struct *t, struct list_head *hop) /* Processing between scanning taskslist and draining the holdout list. */ static void rcu_tasks_postscan(struct list_head *hop) { + int rtsi = READ_ONCE(rcu_task_stall_info); + + if (!IS_ENABLED(CONFIG_TINY_RCU)) { + tasks_rcu_exit_srcu_stall_timer.expires = jiffies + rtsi; + add_timer(&tasks_rcu_exit_srcu_stall_timer); + } + /* * Exiting tasks may escape the tasklist scan. Those are vulnerable * until their final schedule() with TASK_DEAD state. To cope with @@ -848,6 +861,9 @@ static void rcu_tasks_postscan(struct list_head *hop) * call to synchronize_rcu(). */ synchronize_srcu(&tasks_rcu_exit_srcu); + + if (!IS_ENABLED(CONFIG_TINY_RCU)) + del_timer_sync(&tasks_rcu_exit_srcu_stall_timer); } /* See if tasks are still holding out, complain if so. */ @@ -923,6 +939,21 @@ static void rcu_tasks_postgp(struct rcu_tasks *rtp) void call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func); DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_wait_gp, call_rcu_tasks, "RCU Tasks"); +static void tasks_rcu_exit_srcu_stall(struct timer_list *unused) +{ +#ifndef CONFIG_TINY_RCU + int rtsi; + + rtsi = READ_ONCE(rcu_task_stall_info); + pr_info("%s: %s grace period number %lu (since boot) gp_state: %s is %lu jiffies old.\n", + __func__, rcu_tasks.kname, rcu_tasks.tasks_gp_seq, + tasks_gp_state_getname(&rcu_tasks), jiffies - rcu_tasks.gp_jiffies); + pr_info("Please check any exiting tasks stuck between calls to exit_tasks_rcu_start() and exit_tasks_rcu_finish()\n"); + tasks_rcu_exit_srcu_stall_timer.expires = jiffies + rtsi; + add_timer(&tasks_rcu_exit_srcu_stall_timer); +#endif // #ifndef CONFIG_TINY_RCU +} + /** * call_rcu_tasks() - Queue an RCU for invocation task-based grace period * @rhp: structure to be used for queueing the RCU updates. From patchwork Wed Mar 15 19:43:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13176618 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A96F0C7619A for ; Wed, 15 Mar 2023 19:45:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232132AbjCOTpM (ORCPT ); Wed, 15 Mar 2023 15:45:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48166 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232727AbjCOTor (ORCPT ); Wed, 15 Mar 2023 15:44:47 -0400 Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com [IPv6:2607:f8b0:4864:20::835]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E78EB6B95E for ; Wed, 15 Mar 2023 12:44:37 -0700 (PDT) Received: by mail-qt1-x835.google.com with SMTP id h19so1348516qtn.1 for ; Wed, 15 Mar 2023 12:44:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1678909477; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kmL3NaGdkJr0Bc+bh875hISga550hRk4RtUTBtwwn0w=; b=bnBx2KLF4/8ayT1hsKRH/YELY31dEXlopSuwsSIBqqg+vaXM/EzMzHk3VEelU5122J jwQb+81WA0fyC2BsjfFRu6dch564OgwoXcO8sX+yjQIZbugkKoJ2HG34dTE3ocYhrCar 8S/A3URcPENhyHM0kLeY152ccpK6HGiSkJAnw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678909477; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kmL3NaGdkJr0Bc+bh875hISga550hRk4RtUTBtwwn0w=; b=NqkvVoR/GilSa+4IzRJ6/7bGHbccm5ud3/ZsPCFlsY2ZxtD0zSEV9ae7/O9AT6xV2C vEkOEp1i03prw+kQJjS8likWL4uZpBDHjcDOuR9TLMRl20s7xb2kn5EBE0KM5fCYhPzP xYtHdSDlaEJgNgBOCYoMLsmk8/bSGfZ0DPsdz5Ha4QcmXVD1WzjtAAnfeRQ861rcEb+q 8bjgIkq/vA3iauZke17PUX9uPWM4BR1StCpoOgTFt1DuHqLs8BZ7DL6l6vwsdhvwjC0I vtxZWyOULL64W6VxFxekP13Gc+02n70lh3JfTcij7shNsRa3chxN1cmD0f0rtutBxCJG 3k0w== X-Gm-Message-State: AO0yUKUIY5Z7y3xwLqAsZisdx1hio5wQPQzJker/SpkHojAyZmXTT6z6 EVqBTCqpPvIHJrp4ymXbo/ZcxQ== X-Google-Smtp-Source: AK7set/ClKfiibjlVpcqax3G+t0GuqIKsox+SDD/jjB1WY+e+c9YqxkMfF3ISk2g3yWXShpXuSq1zg== X-Received: by 2002:ac8:5989:0:b0:3bf:c5a7:595f with SMTP id e9-20020ac85989000000b003bfc5a7595fmr1903191qte.21.1678909476912; Wed, 15 Mar 2023 12:44:36 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id c15-20020ac8660f000000b003b86b088755sm4346666qtp.15.2023.03.15.12.44.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 12:44:36 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes , Ingo Molnar , Thomas Gleixner Cc: Zheng Yejian , stable@vger.kernel.org, rcu@vger.kernel.org Subject: [PATCH 8/9] rcu: Avoid stack overflow due to __rcu_irq_enter_check_tick() being kprobe-ed Date: Wed, 15 Mar 2023 19:43:48 +0000 Message-Id: <20230315194349.10798-8-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog In-Reply-To: <20230315194349.10798-1-joel@joelfernandes.org> References: <20230315194349.10798-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zheng Yejian Registering a kprobe on __rcu_irq_enter_check_tick() can cause kernel stack overflow as shown below. This issue can be reproduced by enabling CONFIG_NO_HZ_FULL and booting the kernel with argument "nohz_full=", and then giving the following commands at the shell prompt: # cd /sys/kernel/tracing/ # echo 'p:mp1 __rcu_irq_enter_check_tick' >> kprobe_events # echo 1 > events/kprobes/enable This commit therefore adds __rcu_irq_enter_check_tick() to the kprobes blacklist using NOKPROBE_SYMBOL(). Insufficient stack space to handle exception! ESR: 0x00000000f2000004 -- BRK (AArch64) FAR: 0x0000ffffccf3e510 Task stack: [0xffff80000ad30000..0xffff80000ad38000] IRQ stack: [0xffff800008050000..0xffff800008058000] Overflow stack: [0xffff089c36f9f310..0xffff089c36fa0310] CPU: 5 PID: 190 Comm: bash Not tainted 6.2.0-rc2-00320-g1f5abbd77e2c #19 Hardware name: linux,dummy-virt (DT) pstate: 400003c5 (nZcv DAIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : __rcu_irq_enter_check_tick+0x0/0x1b8 lr : ct_nmi_enter+0x11c/0x138 sp : ffff80000ad30080 x29: ffff80000ad30080 x28: ffff089c82e20000 x27: 0000000000000000 x26: 0000000000000000 x25: ffff089c02a8d100 x24: 0000000000000000 x23: 00000000400003c5 x22: 0000ffffccf3e510 x21: ffff089c36fae148 x20: ffff80000ad30120 x19: ffffa8da8fcce148 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: ffffa8da8e44ea6c x14: ffffa8da8e44e968 x13: ffffa8da8e03136c x12: 1fffe113804d6809 x11: ffff6113804d6809 x10: 0000000000000a60 x9 : dfff800000000000 x8 : ffff089c026b404f x7 : 00009eec7fb297f7 x6 : 0000000000000001 x5 : ffff80000ad30120 x4 : dfff800000000000 x3 : ffffa8da8e3016f4 x2 : 0000000000000003 x1 : 0000000000000000 x0 : 0000000000000000 Kernel panic - not syncing: kernel stack overflow CPU: 5 PID: 190 Comm: bash Not tainted 6.2.0-rc2-00320-g1f5abbd77e2c #19 Hardware name: linux,dummy-virt (DT) Call trace: dump_backtrace+0xf8/0x108 show_stack+0x20/0x30 dump_stack_lvl+0x68/0x84 dump_stack+0x1c/0x38 panic+0x214/0x404 add_taint+0x0/0xf8 panic_bad_stack+0x144/0x160 handle_bad_stack+0x38/0x58 __bad_stack+0x78/0x7c __rcu_irq_enter_check_tick+0x0/0x1b8 arm64_enter_el1_dbg.isra.0+0x14/0x20 el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 arm64_enter_el1_dbg.isra.0+0x14/0x20 el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 arm64_enter_el1_dbg.isra.0+0x14/0x20 el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 [...] el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 arm64_enter_el1_dbg.isra.0+0x14/0x20 el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 arm64_enter_el1_dbg.isra.0+0x14/0x20 el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 el1_interrupt+0x28/0x60 el1h_64_irq_handler+0x18/0x28 el1h_64_irq+0x64/0x68 __ftrace_set_clr_event_nolock+0x98/0x198 __ftrace_set_clr_event+0x58/0x80 system_enable_write+0x144/0x178 vfs_write+0x174/0x738 ksys_write+0xd0/0x188 __arm64_sys_write+0x4c/0x60 invoke_syscall+0x64/0x180 el0_svc_common.constprop.0+0x84/0x160 do_el0_svc+0x48/0xe8 el0_svc+0x34/0xd0 el0t_64_sync_handler+0xb8/0xc0 el0t_64_sync+0x190/0x194 SMP: stopping secondary CPUs Kernel Offset: 0x28da86000000 from 0xffff800008000000 PHYS_OFFSET: 0xfffff76600000000 CPU features: 0x00000,01a00100,0000421b Memory Limit: none Link: https://lore.kernel.org/all/20221119040049.795065-1-zhengyejian1@huawei.com/ Fixes: aaf2bc50df1f ("rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()") Signed-off-by: Zheng Yejian Cc: stable@vger.kernel.org Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 90d54571126a..ee27a03d7576 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -640,6 +640,7 @@ void __rcu_irq_enter_check_tick(void) } raw_spin_unlock_rcu_node(rdp->mynode); } +NOKPROBE_SYMBOL(__rcu_irq_enter_check_tick); #endif /* CONFIG_NO_HZ_FULL */ /* From patchwork Wed Mar 15 19:43:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13176617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A8C2C6FD1D for ; Wed, 15 Mar 2023 19:45:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232593AbjCOTpK (ORCPT ); Wed, 15 Mar 2023 15:45:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232754AbjCOTor (ORCPT ); Wed, 15 Mar 2023 15:44:47 -0400 Received: from mail-qt1-x82f.google.com (mail-qt1-x82f.google.com [IPv6:2607:f8b0:4864:20::82f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 712646C1AD for ; Wed, 15 Mar 2023 12:44:38 -0700 (PDT) Received: by mail-qt1-x82f.google.com with SMTP id y10so17531906qtj.2 for ; Wed, 15 Mar 2023 12:44:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1678909478; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4B9TRjdncmXbZW5HDxJoPgY1Lgs+0sxOKkTAY/VF78g=; b=dqOfXnxHTLjoCGihltc9NkJBBAVuV0+9aZqiLbjL4hII42e0QglBr/nfEAbRs2bOkJ GJMPOjeYCEpzI+Scjv/06JFOBqgaiyJfIHVZH/2XR0pVlXRzCzTggamJVGUci2HlXgXm tuGi6ULhvXLuP1gdGRXg0ylLcoiFMYJdJDCmk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678909478; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4B9TRjdncmXbZW5HDxJoPgY1Lgs+0sxOKkTAY/VF78g=; b=CebZFsC2oj0Fk8wH+SCflDmBPz8Wg5R0dgMn4eDWq7YuYZ2I5E3Bt5a4pv15N89nUt WyUXrJ081q2kl2Wd00fzw/AXnBKPKNfAETvic9KKGEqEwQYTe4TpBjKkyW1N2fjmv5/8 mqUfy0VLDmCcaIn7nxLVK58lj2v24zl/kMqGMLP/6JpRKKOpJyTbqgpndD558Ydi1lP6 aYa7AR/fMvx4mA3yPFFI62f9EIKz4YrRPISSFBTcez85nqJ4kDGDto4gfNNX8P6sNVv0 7QmSaJT9sAu5dIcTK1onGINjAuBvt6DCvSvzla0NgAC9jIAtZGVyo0nvhZTOIpZ3oseq 8hTg== X-Gm-Message-State: AO0yUKVeoi8gTSKNroSfnqHJgr7BnqwIwadHbO5hHKK4CCJcrBg18dYo hc3EpcKO0YDt3CAnWJQHSNNIQg== X-Google-Smtp-Source: AK7set/vWDGY32rblTovY6ftZOnuWh8zkM0Wz+Ch7gWDOOg0NQXco/uUREG10n6H7kQ6zngLdzVBHg== X-Received: by 2002:a05:622a:44f:b0:3b8:340b:1aab with SMTP id o15-20020a05622a044f00b003b8340b1aabmr1820256qtx.25.1678909477913; Wed, 15 Mar 2023 12:44:37 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id c15-20020ac8660f000000b003b86b088755sm4346666qtp.15.2023.03.15.12.44.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 12:44:37 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes Cc: Zqiang , rcu@vger.kernel.org Subject: [PATCH 9/9] rcu: Protect rcu_print_task_exp_stall() ->exp_tasks access Date: Wed, 15 Mar 2023 19:43:49 +0000 Message-Id: <20230315194349.10798-9-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog In-Reply-To: <20230315194349.10798-1-joel@joelfernandes.org> References: <20230315194349.10798-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang For kernels built with CONFIG_PREEMPT_RCU=y, the following scenario can result in a NULL-pointer dereference: CPU1 CPU2 rcu_preempt_deferred_qs_irqrestore rcu_print_task_exp_stall if (special.b.blocked) READ_ONCE(rnp->exp_tasks) != NULL raw_spin_lock_rcu_node np = rcu_next_node_entry(t, rnp) if (&t->rcu_node_entry == rnp->exp_tasks) WRITE_ONCE(rnp->exp_tasks, np) .... raw_spin_unlock_irqrestore_rcu_node raw_spin_lock_irqsave_rcu_node t = list_entry(rnp->exp_tasks->prev, struct task_struct, rcu_node_entry) (if rnp->exp_tasks is NULL, this will dereference a NULL pointer) The problem is that CPU2 accesses the rcu_node structure's->exp_tasks field without holding the rcu_node structure's ->lock and CPU2 did not observe CPU1's change to rcu_node structure's ->exp_tasks in time. Therefore, if CPU1 sets rcu_node structure's->exp_tasks pointer to NULL, then CPU2 might dereference that NULL pointer. This commit therefore holds the rcu_node structure's ->lock while accessing that structure's->exp_tasks field. [ paulmck: Apply Frederic Weisbecker feedback. ] Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_exp.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 5343f32e7d67..3b7abb58157d 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -803,9 +803,11 @@ static int rcu_print_task_exp_stall(struct rcu_node *rnp) int ndetected = 0; struct task_struct *t; - if (!READ_ONCE(rnp->exp_tasks)) - return 0; raw_spin_lock_irqsave_rcu_node(rnp, flags); + if (!rnp->exp_tasks) { + raw_spin_unlock_irqrestore_rcu_node(rnp, flags); + return 0; + } t = list_entry(rnp->exp_tasks->prev, struct task_struct, rcu_node_entry); list_for_each_entry_continue(t, &rnp->blkd_tasks, rcu_node_entry) {