From patchwork Fri Jan 12 23:36:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Zhou X-Patchwork-Id: 13518769 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25FA2C4707C for ; Fri, 12 Jan 2024 23:37:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9017E8D0001; Fri, 12 Jan 2024 18:37:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 88A3E6B0095; Fri, 12 Jan 2024 18:37:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6DC648D0001; Fri, 12 Jan 2024 18:37:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4DBC06B0093 for ; Fri, 12 Jan 2024 18:37:19 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 209D5A0AF9 for ; Fri, 12 Jan 2024 23:37:19 +0000 (UTC) X-FDA: 81672272598.20.B0586EC Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by imf29.hostedemail.com (Postfix) with ESMTP id 570D712000A for ; Fri, 12 Jan 2024 23:37:17 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=kernel.org (policy=none); spf=pass (imf29.hostedemail.com: domain of dennisszhou@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=dennisszhou@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705102637; a=rsa-sha256; cv=none; b=xdqqaHqKvHSGfjFuzzJoupIl2uW4WSNQOQPyvQeT9yooaTmV4jTQsRdtYPPUdA9mxdLbfK YD8vi2Qy0sqYBCxqN3wazzeEs+P36KAi+23S8tE6VcB5Mo66RfssWydfrW/8whSKIlJ0pl TXSNK0bQvF3egSrLqIzc2m9KirdgXK4= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=kernel.org (policy=none); spf=pass (imf29.hostedemail.com: domain of dennisszhou@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=dennisszhou@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705102637; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ifDoLpiNdOYTSQ47Pl1LromjALktxWcq8aIySdxMVro=; b=fRbH6QrNdXmJAdaRaWYC7eQ2IM2m20rnnRFnVUbyUtaoHqsc3fqjbLRaVsB+S24NrrbEpN Yg3SyD3sbpVrHLeT6UeWGxeQO5Lugzu311iY0yGRkmIiYR3f5FVqjNhJ8DVYXbTBgxZcvq Kp4NTdMSeM++YpgdmhhuumIEP/YKcJI= Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-6d9344f30caso4792812b3a.1 for ; Fri, 12 Jan 2024 15:37:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705102636; x=1705707436; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ifDoLpiNdOYTSQ47Pl1LromjALktxWcq8aIySdxMVro=; b=wnSFUERrQDPXAG1oVJJP66HBePW4L8rbmMaNlmX6uN4ZZqa5sIXXkL7t6b3665rFGf ovm4QRa5oEA6LepbfMLq0sVp6MZnwns8dWhybcgv9ngp0jCAw9NDgV9fc4YMqymDbVbW 7Y/gGuAZHWJA+G3IBUZEJ5FnQ709r8ocGym+mZew5jxbIUVAbZW+qoV//pV0lhIXpGKo MAQdFNpn5nJeYDGvs+z8bhuoEmfocD7XaXUe6jwDYHcY+lMDB1M8sluQNIVhjs/v7NI0 vtr3z6uKs4f6YhzmVa3yI6lviXE8WGXQatTipWWWYGYXU/EAxMWpz2JEambqSgUl8Zxu AZUg== X-Gm-Message-State: AOJu0Yw3MxUgQY/Kz7kYs23ByI6pYZOrfH9R8ThZ6Lcrb59jxO6Uypep k8kZPV2EgLJVDkIGZHxTuSODbY6tz1k82A== X-Google-Smtp-Source: AGHT+IETOzry7t2QrGyMY0qVE9zN8YF+Pj/G2WT9Tg51W3pV2fUxP0YtB8ryO5YxL+RNGUaJMSnd/Q== X-Received: by 2002:a05:6a00:1822:b0:6da:56c:1019 with SMTP id y34-20020a056a00182200b006da056c1019mr2564523pfa.3.1705102636149; Fri, 12 Jan 2024 15:37:16 -0800 (PST) Received: from snowbird.lan ([136.25.84.107]) by smtp.gmail.com with ESMTPSA id x5-20020aa793a5000000b006d9b4171b20sm3676811pff.112.2024.01.12.15.37.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 Jan 2024 15:37:15 -0800 (PST) From: Dennis Zhou To: Tejun Heo , Christoph Lameter , Thomas Gleixner Cc: Peter Zijlstra , Valentin Schneider , Dave Chinner , Yury Norov , Andy Shevchenko , Rasmus Villemoes , Ye Bin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dennis Zhou Subject: [PATCH 3/3] cpu/hotplug: Get rid of cpu_dying_mask Date: Fri, 12 Jan 2024 15:36:48 -0800 Message-Id: <4aeddaa133df7c0b7795b7774d2222efedc3aa12.1705101789.git.dennis@kernel.org> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 570D712000A X-Stat-Signature: 7xgegczxckm9krwqw5kx1d9wpwoetngr X-Rspam-User: X-HE-Tag: 1705102637-363646 X-HE-Meta: U2FsdGVkX185xsw+2/+9RyBT4y2rJsf4rjIwVjWSqAyHn3thCRHgwMlQ+I/G78yIK3DwAu9lXNh84fTPFwLQoP2CXnM4Bo2jhMxPpDSbdAE4qyv9o9MrXm+5eavfFaOyWehX2JoFB5PA3t9uAibp9zQWI3UxNKCLZjzUlLYCssj/bVSGc9+tAE1Ww8YgpeQf3xAj0Z3IgFLYfAvs5zje8+mWOnQ4lMs4tA21pxvVNNTEmjFQ5Mv27rVgdivh5GoR6OMJwuQmnUnGuHwn+x1NI+mABmhpKrTkyH3cfmrOveWnV4EIkqqnMQdNfILZ0AlkggzZ0zr6TGSfr4ZSY+hJnFGC20GJfDb7tvpNUamVlG/1VlGseb2mwz3Mp4zaLiWUVOJGb5IdPcsjcilozFD0H0r+ljzKDTeTcXTZurRna/1CIVIYs1DmF9Tsu2P6eZoRFCmFtt+rKLfSeo3Q9mZ7s/80Dts+iOf7VguYOJ74jwzqUFPd3DNw78RqSEaiOr3/UmE22X2MysUKBqQS++BprdAii/Y7hzfNuDEJSwm/+wuNscWmaUoYaj0YIgpwAJHLQ9ko9xMASMzFyFwvsUILnIZ5JWFhVqtwn0QPv6I3yzYWxK6azlW63bK6nUA/d5SvbQV4R9COtno0K5wXp0XssXCv0/xLsCFfElxITUXoq5v1Q/YL/XUOgSd/xsWQBxTuXueNlzHcFvbGRmWy6khKxGvfPijoFB6IPB0PX/lfDHCeF28Vv4I1irINY83NEmEzlF0/ylqigax3MnOvzYAh8O3CMfi1ys5P5XW7QMJEiCKNdAIg+ZMeEO7ZdJdfKZ9uvDJVoyuGxvTO+kHBx2gmYKF4zeJ4gnoor68ZdJ3QHygiqnw/XJeekrA1uRbgQ3pHmEajRC2ilBcCXddxTpeRuQl6fRd4tmeVgEqZt1hvmDZdiZDLEpKAxKIBen8tic1Ue91DFmlSM3xsAqa1tYF If9Ks0UU T2EM4+pKWYd6jPRegvCmujQzkImRyR+yQusayX4JRLB/Vk8kVkpA7jKqfuQRLuPZlSU+yGdETk8+Ry7Xw80vjCsyW3lszx2SPWMTvxeiH9JplwkT7xvkYrF09TaXZs/EHSvXBo0bAAQXxcriigVEh+4TpjqlrNUZSem4Goae23LLo9J2NCP+P1EalrniP+CONCiJbZLWgjNOTPgw4YtsXSwEfrhFTwzrHVTPcTzCraYfKAWYC2Jm0gSLXpR3gKNWYkVgsVyXXU7wx5yiaot/XuMfud8Qxi9Hf+9TUN86NDGZjoQGgFICMXBNXFjBqVSPpd54rhUMdZY7V98tBV9f/gTrzcv/SYtmKG/u3Ckno1omhp8ufIMH2jYnpEWN8LGQtS0iqftW7+eRNpwniX9lzKMap6dkmHundLjrEJhUzGysUug0tUF1IHoNLyfoU8nEi3gQ/H+VxrdBgTqry6ItsKJ2PLxYvTVrYqyEIZP/QxZTKYVnAnGUo+fas8zDPA5m/OPhWzO4IoETIN6e1aYyMmGAfWC8S5VfD1/s7omw/gItNTxpY2WAcCn3UZUgRWt4u0jg6QkqweLcZf+eajaiIntf9zlp77AyZBih+eqf74bHZAztGiuda6AT8CWlJOVWsTjvpEeEC7xU2too= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The cpu_dying_mask is not only undocumented but also to some extent a misnomer. It's purpose is to capture the last direction of a cpu_up() or cpu_down() operation taking eventual rollback operations into account. The name and the lack of documentation lured already someone to use it in the wrong way. The initial user is the scheduler code which needs to keep the decision correct whether to schedule tasks on a CPU, which is between the CPUHP_ONLINE and the CPUHP_ACTIVE state and has the balance_push() hook installed. cpu_dying mask is not really useful for general consumption. The cpu_dying_mask bits are sticky even after cpu_up() or cpu_down() completes. It might be argued, that the cpu_dying_mask bit could be cleared when cpu_down() completes, but that's not possible under all circumstances. Especially not with partial hotplug operations. In that case the bit must be sticky in order to keep the initial user, i.e. the scheduler correct. Replace the cpumask completely by: - recording the direction internally in the CPU hotplug core state - exposing that state via a documented function to the scheduler After that cpu_dying_mask is not longer in use and removed before the next user trips over it. Signed-off-by: Thomas Gleixner [Dennis: ported to v6.7-rc4, delete in cpumask.h didn't apply cleanly] Signed-off-by: Dennis Zhou --- include/linux/cpumask.h | 21 -------------------- kernel/cpu.c | 43 +++++++++++++++++++++++++++++++++++------ kernel/sched/core.c | 4 ++-- kernel/smpboot.h | 2 ++ 4 files changed, 41 insertions(+), 29 deletions(-) diff --git a/include/linux/cpumask.h b/include/linux/cpumask.h index cfb545841a2c..b19b6fd29a0d 100644 --- a/include/linux/cpumask.h +++ b/include/linux/cpumask.h @@ -126,12 +126,10 @@ extern struct cpumask __cpu_possible_mask; extern struct cpumask __cpu_online_mask; extern struct cpumask __cpu_present_mask; extern struct cpumask __cpu_active_mask; -extern struct cpumask __cpu_dying_mask; #define cpu_possible_mask ((const struct cpumask *)&__cpu_possible_mask) #define cpu_online_mask ((const struct cpumask *)&__cpu_online_mask) #define cpu_present_mask ((const struct cpumask *)&__cpu_present_mask) #define cpu_active_mask ((const struct cpumask *)&__cpu_active_mask) -#define cpu_dying_mask ((const struct cpumask *)&__cpu_dying_mask) extern atomic_t __num_online_cpus; @@ -1035,15 +1033,6 @@ set_cpu_active(unsigned int cpu, bool active) cpumask_clear_cpu(cpu, &__cpu_active_mask); } -static inline void -set_cpu_dying(unsigned int cpu, bool dying) -{ - if (dying) - cpumask_set_cpu(cpu, &__cpu_dying_mask); - else - cpumask_clear_cpu(cpu, &__cpu_dying_mask); -} - /** * to_cpumask - convert a NR_CPUS bitmap to a struct cpumask * * @bitmap: the bitmap @@ -1119,11 +1108,6 @@ static inline bool cpu_active(unsigned int cpu) return cpumask_test_cpu(cpu, cpu_active_mask); } -static inline bool cpu_dying(unsigned int cpu) -{ - return cpumask_test_cpu(cpu, cpu_dying_mask); -} - #else #define num_online_cpus() 1U @@ -1151,11 +1135,6 @@ static inline bool cpu_active(unsigned int cpu) return cpu == 0; } -static inline bool cpu_dying(unsigned int cpu) -{ - return false; -} - #endif /* NR_CPUS > 1 */ #define cpu_is_offline(cpu) unlikely(!cpu_online(cpu)) diff --git a/kernel/cpu.c b/kernel/cpu.c index c4929e9cd9be..ce78757b7535 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -54,6 +54,9 @@ * @rollback: Perform a rollback * @single: Single callback invocation * @bringup: Single callback bringup or teardown selector + * @goes_down: Indicator for direction of cpu_up()/cpu_down() operations + * including eventual rollbacks. Not affected by state or + * instance add/remove operations. See cpuhp_cpu_goes_down(). * @cpu: CPU number * @node: Remote CPU node; for multi-instance, do a * single entry callback for install/remove @@ -74,6 +77,7 @@ struct cpuhp_cpu_state { bool rollback; bool single; bool bringup; + bool goes_down; struct hlist_node *node; struct hlist_node *last; enum cpuhp_state cb_state; @@ -474,6 +478,37 @@ void cpu_maps_update_done(void) mutex_unlock(&cpu_add_remove_lock); } +/** + * cpuhp_cpu_goes_down - Query the current/last CPU hotplug direction of a CPU + * @cpu: The CPU to query + * + * The direction indicator is modified by the hotplug core on + * cpu_up()/cpu_down() operations including eventual rollback operations. + * The indicator is not affected by state or instance install/remove + * operations. + * + * The indicator is sticky after the hotplug operation completes, whether + * the operation was a full up/down or just a partial bringup/teardown. + * + * goes_down + * cpu_up(target) enter -> False + * rollback on fail -> True + * cpu_up(target) exit Last state + * + * cpu_down(target) enter -> True + * rollback on fail -> False + * cpu_down(target) exit Last state + * + * The return value is a racy snapshot and not protected against concurrent + * CPU hotplug operations which modify the indicator. + * + * Returns: True if cached direction is down, false otherwise + */ +bool cpuhp_cpu_goes_down(unsigned int cpu) +{ + return data_race(per_cpu(cpuhp_state.goes_down, cpu)); +} + /* * If set, cpu_up and cpu_down will return -EBUSY and do nothing. * Should always be manipulated under cpu_add_remove_lock @@ -708,8 +743,7 @@ cpuhp_set_state(int cpu, struct cpuhp_cpu_state *st, enum cpuhp_state target) st->target = target; st->single = false; st->bringup = bringup; - if (cpu_dying(cpu) != !bringup) - set_cpu_dying(cpu, !bringup); + st->goes_down = !bringup; return prev_state; } @@ -743,8 +777,7 @@ cpuhp_reset_state(int cpu, struct cpuhp_cpu_state *st, } st->bringup = bringup; - if (cpu_dying(cpu) != !bringup) - set_cpu_dying(cpu, !bringup); + st->goes_down = !bringup; } /* Regular hotplug invocation of the AP hotplug thread */ @@ -3127,8 +3160,6 @@ EXPORT_SYMBOL(__cpu_present_mask); struct cpumask __cpu_active_mask __read_mostly; -struct cpumask __cpu_dying_mask __read_mostly; - atomic_t __num_online_cpus __read_mostly; EXPORT_SYMBOL(__num_online_cpus); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index a708d225c28e..6d4f0cdad446 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2468,7 +2468,7 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu) return cpu_online(cpu); /* Regular kernel threads don't get to stay during offline. */ - if (cpu_dying(cpu)) + if (cpuhp_cpu_goes_down(cpu)) return false; /* But are allowed during online. */ @@ -9434,7 +9434,7 @@ static void balance_push(struct rq *rq) * Only active while going offline and when invoked on the outgoing * CPU. */ - if (!cpu_dying(rq->cpu) || rq != this_rq()) + if (!cpuhp_cpu_goes_down(rq->cpu) || rq != this_rq()) return; /* diff --git a/kernel/smpboot.h b/kernel/smpboot.h index 34dd3d7ba40b..9d3b4d554411 100644 --- a/kernel/smpboot.h +++ b/kernel/smpboot.h @@ -20,4 +20,6 @@ int smpboot_unpark_threads(unsigned int cpu); void __init cpuhp_threads_init(void); +bool cpuhp_cpu_goes_down(unsigned int cpu); + #endif