From patchwork Thu Sep 15 05:58:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12976821 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D29ECECAAD3 for ; Thu, 15 Sep 2022 05:58:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229471AbiIOF6t (ORCPT ); Thu, 15 Sep 2022 01:58:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53282 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229452AbiIOF6s (ORCPT ); Thu, 15 Sep 2022 01:58:48 -0400 Received: from mail-pg1-x534.google.com (mail-pg1-x534.google.com [IPv6:2607:f8b0:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A1C1E90199 for ; Wed, 14 Sep 2022 22:58:47 -0700 (PDT) Received: by mail-pg1-x534.google.com with SMTP id g4so16398133pgc.0 for ; Wed, 14 Sep 2022 22:58:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=eGzhxNHfxgZG+HUCB9A/U66lF7BlvYesE74FyPkk6Rk=; b=oSfzNplE9jROtQE+W9Ioy6jG8Dwhn360g1wue0YJiinyz2YwoJfySMbkF16p4x/9kD A7lkEeVmW/YMX2OOaptC8zjVMI2kzmLsJ7F5oje5pTP6pafkD0cYBrcKTCnkCw0z1pvH jmuNHIfl6rWRVWPNMt1X7PJ8fOdQiRP8JF5/LmOScd4rMgavDVA9KWJdGuFyFAXWGjhi 4UZd43xAH2P3aUCPs5AjDYCfp9XOalDxO0qUXBqcoSCi1gkPGWQuXC2gQMb1D1NPnlLs lyBr1PNPJpOhcZTDJDM+qkYoqqwuBTkH8QFKOUR5qXUlO6KzlthNIYmbkzqT8uJPjJa9 dP2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=eGzhxNHfxgZG+HUCB9A/U66lF7BlvYesE74FyPkk6Rk=; b=nObcw9GjUJc3IItJ1/g2TtHBJWD1WHoE4O5HvAPBJeRBdQ6mXHcmEc2uJzVHJtZsSO i2iccHNQfWn2+YebjoPesFpbcwnto4qQEKVVYl8adp+bYc5DMyW2sYlSq4X222AmG4su 8I5ZDiOZxAUyfFhLwar4ur02e2MPD6cP7hT3aoMiJ9BkkGChiiEiLWOp0+78pYo9d9iJ FWp+QUDwYM0RndwMHXh1c/fRcJfKPG5vsLORTtfklNESS0xFpTKKrP1a1/bZyoWIFxAL 1YeNlXrL+zimq9bRkyRk44pOot/FmrY7/UJRQXgfHT0GVWC5eM6Fap1dY7FjvNBs1Vky LwIg== X-Gm-Message-State: ACgBeo1m4CsC1Rg4ElVU774fPim3JBVLRGx29ecsMN00uvB/unhggP1V dOqyjBqQRqaKzqdBL7CurcinvWZ0fA== X-Google-Smtp-Source: AA6agR6mMUMxMV4NGkonr8rqsuGBesgWg1Qgt/WxSY83l1NQgUl41SQor7YgQEspCIdqBVr8nHx9Gw== X-Received: by 2002:a05:6a00:1943:b0:53a:8602:6758 with SMTP id s3-20020a056a00194300b0053a86026758mr40803724pfk.47.1663221526719; Wed, 14 Sep 2022 22:58:46 -0700 (PDT) Received: from piliu.users.ipa.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id z8-20020a170903018800b001768452d4d7sm11983386plg.14.2022.09.14.22.58.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Sep 2022 22:58:46 -0700 (PDT) From: Pingfan Liu To: rcu@vger.kernel.org Cc: Pingfan Liu , "Paul E. McKenney" , David Woodhouse , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes , "Jason A. Donenfeld" Subject: [PATCHv2 1/3] rcu: Keep qsmaskinitnext fresh when rcutree_online_cpu() Date: Thu, 15 Sep 2022 13:58:23 +0800 Message-Id: <20220915055825.21525-2-kernelfans@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220915055825.21525-1-kernelfans@gmail.com> References: <20220915055825.21525-1-kernelfans@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org rcutree_online_cpu() is concurrent, so there is the following race scene: CPU 1 CPU2 mask_old = rcu_rnp_online_cpus(rnp); ... mask_new = rcu_rnp_online_cpus(rnp); ... set_cpus_allowed_ptr(t, cm); set_cpus_allowed_ptr(t, cm); Consequently, the old mask will overwrite the new mask in the task's cpus_ptr. Since there is a mutex ->boost_kthread_mutex, using it to build an order, then the latest ->qsmaskinitnext will be fetched for updating cpus_ptr. But for concurrent offlining, ->qsmaskinitnext is not reilable when rcutree_offline_cpu(). That is another story and comes in the following patch. Signed-off-by: Pingfan Liu Cc: "Paul E. McKenney" Cc: David Woodhouse Cc: Frederic Weisbecker Cc: Neeraj Upadhyay Cc: Josh Triplett Cc: Steven Rostedt Cc: Mathieu Desnoyers Cc: Lai Jiangshan Cc: Joel Fernandes Cc: "Jason A. Donenfeld" To: rcu@vger.kernel.org --- kernel/rcu/tree_plugin.h | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 438ecae6bd7e..ef6d3ae239b9 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -1224,7 +1224,7 @@ static void rcu_spawn_one_boost_kthread(struct rcu_node *rnp) static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu) { struct task_struct *t = rnp->boost_kthread_task; - unsigned long mask = rcu_rnp_online_cpus(rnp); + unsigned long mask; cpumask_var_t cm; int cpu; @@ -1233,6 +1233,11 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu) if (!zalloc_cpumask_var(&cm, GFP_KERNEL)) return; mutex_lock(&rnp->boost_kthread_mutex); + /* + * Relying on the lock to serialize, so when onlining, the latest + * qsmaskinitnext is for cpus_ptr. + */ + mask = rcu_rnp_online_cpus(rnp); for_each_leaf_node_possible_cpu(rnp, cpu) if ((mask & leaf_node_cpu_bit(rnp, cpu)) && cpu != outgoingcpu) From patchwork Thu Sep 15 05:58:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12976822 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62145ECAAA1 for ; Thu, 15 Sep 2022 05:58:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229460AbiIOF6z (ORCPT ); Thu, 15 Sep 2022 01:58:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53302 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229452AbiIOF6y (ORCPT ); Thu, 15 Sep 2022 01:58:54 -0400 Received: from mail-pg1-x529.google.com (mail-pg1-x529.google.com [IPv6:2607:f8b0:4864:20::529]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 05F6790199 for ; Wed, 14 Sep 2022 22:58:54 -0700 (PDT) Received: by mail-pg1-x529.google.com with SMTP id h188so16334580pgc.12 for ; Wed, 14 Sep 2022 22:58:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=LLHYN7LHMfNXPe81ykcurF/+WoxshOqmmf34A1mIGv8=; b=JKPnb9Nk7w/Ri8Fsc3ZZ6kv/3ySZSfvr/cZcjTfZZc6Gt8PiyTeyq5Xo0cBrXU9qbe 2h5vd7obJfefxDmzOsrNCmBK6W5iEIlmi6PfpOEqoprDbm6wuMpV7l9COPwlze18hfND Fbq37rsbfzTfaC9/IUGwZCyb/sOsbn5lBeF4OzmeGVHQghSB9/sZQ7RBOP5qMzZSPYPs REXYsUzEfPLOQaqPUn/cTXAl5mDY/EEpyr+KKPRL8L35PA2B6im6GKH5azw7C/1ugcph j9GAbRIOPO0u4RAYL/jKGch8dmR1ifBpb6sa9wTwcSopOYz1vx419/VpDLeZuvOckpTK YzrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=LLHYN7LHMfNXPe81ykcurF/+WoxshOqmmf34A1mIGv8=; b=YTDuNIP22FrB0h0OB+heue3IrCf2tJNu2AXySCylGblAxgU6Lr7PVti30PIA+OV/PK L6yuwm4pYP9WPTJp2rQXGCfHADFQwpPVFiPqXv/SwFTkKjbgYMX/DE0UVAxST2IlJYGD 6m5u0kG6vd0XKq3fEZ0bvQLvmWEJx+CXmBl7zg3qpswRSfF0ghW5u8YC7lSEheetOFPL dDhwMBFWc5H+O4nvu0to7uBgYtRe9Ut75McY9KHiYfsai5fRJdc410whkYk308Y7Grwq 24QPzYyt/rmz6EE8hLq25PYLB6t+CCrzCWuI3WAgDE6mpNNvbROF3OsjUj1tCWcDKHuG 5o3w== X-Gm-Message-State: ACgBeo1hWd96g3ppNV+4nd8kFfjvXzL/ZnI6KP2Qqh3dYeXaU3Wo22Jc DgA9g3uw7CWXkiiadS+m0Jh8daCatQ== X-Google-Smtp-Source: AA6agR4sNzdf2IJ7GtDQBMcr07zBfOEMD9r/xyp41fZNdXJT57R8qeDXqC3QVkGADxaIlApG97Y3TQ== X-Received: by 2002:aa7:9717:0:b0:53e:84e4:dceb with SMTP id a23-20020aa79717000000b0053e84e4dcebmr39026695pfg.48.1663221533072; Wed, 14 Sep 2022 22:58:53 -0700 (PDT) Received: from piliu.users.ipa.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id z8-20020a170903018800b001768452d4d7sm11983386plg.14.2022.09.14.22.58.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Sep 2022 22:58:52 -0700 (PDT) From: Pingfan Liu To: rcu@vger.kernel.org Cc: Pingfan Liu , "Paul E. McKenney" , David Woodhouse , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes , "Jason A. Donenfeld" Subject: [PATCHv2 2/3] rcu: Resort to cpu_dying_mask for affinity when offlining Date: Thu, 15 Sep 2022 13:58:24 +0800 Message-Id: <20220915055825.21525-3-kernelfans@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220915055825.21525-1-kernelfans@gmail.com> References: <20220915055825.21525-1-kernelfans@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org During offlining, the concurrent rcutree_offline_cpu() can not be aware of each other through ->qsmaskinitnext. But cpu_dying_mask carries such information at that point and can be utilized. Besides, a trivial change which removes the redudant call to rcu_boost_kthread_setaffinity() in rcutree_dead_cpu() since rcutree_offline_cpu() can fully serve that purpose. Signed-off-by: Pingfan Liu Cc: "Paul E. McKenney" Cc: David Woodhouse Cc: Frederic Weisbecker Cc: Neeraj Upadhyay Cc: Josh Triplett Cc: Steven Rostedt Cc: Mathieu Desnoyers Cc: Lai Jiangshan Cc: Joel Fernandes Cc: "Jason A. Donenfeld" To: rcu@vger.kernel.org --- kernel/rcu/tree.c | 2 -- kernel/rcu/tree_plugin.h | 6 ++++++ 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 79aea7df4345..8a829b64f5b2 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2169,8 +2169,6 @@ int rcutree_dead_cpu(unsigned int cpu) return 0; WRITE_ONCE(rcu_state.n_online_cpus, rcu_state.n_online_cpus - 1); - /* Adjust any no-longer-needed kthreads. */ - rcu_boost_kthread_setaffinity(rnp, -1); // Stop-machine done, so allow nohz_full to disable tick. tick_dep_clear(TICK_DEP_BIT_RCU); return 0; diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index ef6d3ae239b9..e5afc63bd97f 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -1243,6 +1243,12 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu) cpu != outgoingcpu) cpumask_set_cpu(cpu, cm); cpumask_and(cm, cm, housekeeping_cpumask(HK_TYPE_RCU)); + /* + * For concurrent offlining, bit of qsmaskinitnext is not cleared yet. + * So resort to cpu_dying_mask, whose changes has already been visible. + */ + if (outgoingcpu != -1) + cpumask_andnot(cm, cm, cpu_dying_mask); if (cpumask_empty(cm)) cpumask_copy(cm, housekeeping_cpumask(HK_TYPE_RCU)); set_cpus_allowed_ptr(t, cm); From patchwork Thu Sep 15 05:58:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12976823 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D553ECAAA1 for ; Thu, 15 Sep 2022 05:59:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229483AbiIOF7C (ORCPT ); Thu, 15 Sep 2022 01:59:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53320 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229452AbiIOF7B (ORCPT ); Thu, 15 Sep 2022 01:59:01 -0400 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A9CC890199 for ; Wed, 14 Sep 2022 22:59:00 -0700 (PDT) Received: by mail-pj1-x1032.google.com with SMTP id q15-20020a17090a304f00b002002ac83485so16541651pjl.0 for ; Wed, 14 Sep 2022 22:59:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=92owxcP8FsQYEgA9nCUUN+ZZEwVzMqZqaSjqiZXq0Co=; b=e9NJ1th2a1Xz9ysAot9r4SldrLzXY/TeA1dQehbT0bhVU/WVOQ2TOs28XsSbDZqeOO 7PQ8mFK4NLpxhrYNbeTlS1cKPz4scGZivWsmitEkELb6ap5Bs0U1eWjnMXoZ6+P8CrST xX6hg9phfJIoxheNikGzhgNEziaCh0sEhXfDYjkK2INVzAXVv5cZ8GsQBl6hFmLsg7Rd 9hC7cAHu1jDLpZtzEPwCjPjI+wU5tUq//i7Hl6V2X5FidT4ezC7Eyti0odS8fJ4AZo+n PFjhmu7qER5+EA//fNQOmr2afziVkmYxW6L+wrJhlheZ5BtcK01jmjLEipgxtAGHBuMS x0Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=92owxcP8FsQYEgA9nCUUN+ZZEwVzMqZqaSjqiZXq0Co=; b=Gz5T2sHkt2QRIxqVEzc1vl8VRHv1p9xu1HfIu7ZmcgPkG3vlp6wpc6MxIR+zvm7YHr MDRhtAhFblEo+eIeuv5lE7qNw7vHFx4MK4/b+4hGgCaz/tMT5cqQe+Xg70vWyQ18C2ES DSfxMJmBclyFpLe3oCpteu3aElR3XdoJgPiuNZmSfRUdm77fK4WaQ33jaXOLemeTPvDw 3kHKgHp8X4WF4xwaimrPqa/xB8inaaMVsziihp3AQBnxZZibPrNYllGoNJ3EhojPX21S o71iWhRQZq9aPfv7sXAsl5cH2nzKn8k35fr8Cyu+Er4lpJXaz6iC/egG/Y54vm6caoIC 8NfQ== X-Gm-Message-State: ACrzQf3IidK3zOtZs4gXyMqPK+VQr7cIRpjRIo+6q1/FH7w0AJcBVwnf Z1qRdgfPwqLplTF75OLxcVwZkcQnOQ== X-Google-Smtp-Source: AMsMyM7X+WXO1r/aVIhvEvhzTl2fK7jdYGx2z4hjtMZ0h6cpyrKSknUXHgTN1rME/wnqewSUEBoYbw== X-Received: by 2002:a17:903:41c8:b0:178:3128:b58a with SMTP id u8-20020a17090341c800b001783128b58amr2736955ple.118.1663221539317; Wed, 14 Sep 2022 22:58:59 -0700 (PDT) Received: from piliu.users.ipa.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id z8-20020a170903018800b001768452d4d7sm11983386plg.14.2022.09.14.22.58.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Sep 2022 22:58:58 -0700 (PDT) From: Pingfan Liu To: rcu@vger.kernel.org Cc: Pingfan Liu , "Paul E. McKenney" , David Woodhouse , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes , "Jason A. Donenfeld" Subject: [PATCHv2 3/3] rcu: coordinate tick dependency during concurrent offlining Date: Thu, 15 Sep 2022 13:58:25 +0800 Message-Id: <20220915055825.21525-4-kernelfans@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220915055825.21525-1-kernelfans@gmail.com> References: <20220915055825.21525-1-kernelfans@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org As Paul pointed out "The tick_dep_clear() is SMP-safe because it uses atomic operations, but the problem is that if there are multiple nohz_full CPUs going offline concurrently, the first CPU to invoke rcutree_dead_cpu() will turn the tick off. This might require an atomically manipulated counter to mediate the calls to rcutree_dead_cpu(). " This patch introduces a new member ->dying to rcu_node, which reflects the number of concurrent offlining cpu. TICK_DEP_BIT_RCU is set by the first entrance and cleared by the last. Note: now, tick_dep_set() is put under the rnp->lock, but since it takes no lock, no extra locking order is introduced. Suggested-by: "Paul E. McKenney" Signed-off-by: Pingfan Liu Cc: "Paul E. McKenney" Cc: David Woodhouse Cc: Frederic Weisbecker Cc: Neeraj Upadhyay Cc: Josh Triplett Cc: Steven Rostedt Cc: Mathieu Desnoyers Cc: Lai Jiangshan Cc: Joel Fernandes Cc: "Jason A. Donenfeld" To: rcu@vger.kernel.org --- kernel/rcu/tree.c | 19 ++++++++++++++----- kernel/rcu/tree.h | 1 + 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 8a829b64f5b2..f8bd0fc5fd2f 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2164,13 +2164,19 @@ int rcutree_dead_cpu(unsigned int cpu) { struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); struct rcu_node *rnp = rdp->mynode; /* Outgoing CPU's rdp & rnp. */ + unsigned long flags; + u8 dying; if (!IS_ENABLED(CONFIG_HOTPLUG_CPU)) return 0; WRITE_ONCE(rcu_state.n_online_cpus, rcu_state.n_online_cpus - 1); - // Stop-machine done, so allow nohz_full to disable tick. - tick_dep_clear(TICK_DEP_BIT_RCU); + raw_spin_lock_irqsave_rcu_node(rnp, flags); + dying = --rnp->dying; + if (!dying) + // Stop-machine done, so allow nohz_full to disable tick. + tick_dep_clear(TICK_DEP_BIT_RCU); + raw_spin_lock_irqsave_rcu_node(rnp, flags); return 0; } @@ -4020,17 +4026,20 @@ int rcutree_offline_cpu(unsigned int cpu) unsigned long flags; struct rcu_data *rdp; struct rcu_node *rnp; + u8 dying; rdp = per_cpu_ptr(&rcu_data, cpu); rnp = rdp->mynode; raw_spin_lock_irqsave_rcu_node(rnp, flags); rnp->ffmask &= ~rdp->grpmask; + /* Let rcutree_dead_cpu() know a new offlining. */ + dying = rnp->dying++; + if (!dying) + // nohz_full CPUs need the tick for stop-machine to work quickly + tick_dep_set(TICK_DEP_BIT_RCU); raw_spin_unlock_irqrestore_rcu_node(rnp, flags); rcutree_affinity_setting(cpu, cpu); - - // nohz_full CPUs need the tick for stop-machine to work quickly - tick_dep_set(TICK_DEP_BIT_RCU); return 0; } diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index d4a97e40ea9c..b508a12ac953 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -81,6 +81,7 @@ struct rcu_node { int grphi; /* highest-numbered CPU here. */ u8 grpnum; /* group number for next level up. */ u8 level; /* root is at level 0. */ + u8 dying; /* num of concurrent rdp offlining */ bool wait_blkd_tasks;/* Necessary to wait for blocked tasks to */ /* exit RCU read-side critical sections */ /* before propagating offline up the */