From patchwork Fri Apr 14 23:45:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 13212235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 49003C77B76 for ; Sat, 15 Apr 2023 00:46:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Date:MIME-Version:References:Subject:Cc :To:From:Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: List-Owner; bh=3IFflc1y3ZCV278tmS88jzI132ri+GNBkJ47jylVmS0=; b=H2BiI8u5tzIY1Y Ce5UCvKjanLzkUw5k4FZRhlhcHqHfmDIbWUHYwtv7pEkjzplEv4ypw5pvOBrNy92PSBiObzfcWyP+ S/6b7qq525mlIX1tcsNNbwLVN5if1Nsg/lK3D+wyFxDe5AUbck3T/cT+Dnq38yb+ZUEvuqVLG1jQC qzaQheaK7CSPJHZeB25erhJw4Wa6B9Ol1NgnNoubAOdnEfdnsJLCkVUOPCVu84cehsnXRLM7IvDS7 n8KE2o8uhRxjgOGI/YftDzD9HtLqThXkahqzjUGEqqPNX+ZPaTR6EBhf2h5pwD/7qNa8XAcxylX9f XK+ziKs0HoIcP+mjed9Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pnU2s-00AvJH-2n; Sat, 15 Apr 2023 00:45:38 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pnU2k-00AvGs-1e; Sat, 15 Apr 2023 00:45:30 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Date:Content-Type:MIME-Version: References:Subject:Cc:To:From:Message-ID:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:In-Reply-To; bh=OOycMYPFVi5YhixhBoVMbIx7KiHrtAXAV72ZpNnWZL8=; b=UNTa/iQzkAuLC1oyYMlUuvNAwr Wky4xUCLCzNZ01KlMnTxz81oTmIcP7Wo/dO3PPLQwaFXGnZWHrZ8Hy2pH9fcBATfzVKvErMm9ZezH +istI4rrUHViJgYPTQidiN1IOFZUYDPoXf3gvM8n/2RH0JPo6Hin6V8P5x3kyildpZFLW1DmKncCE zfUGfkK5YkgSR/r/yCZJGVyu/GTW4YSR6uXq8TD7uxBljgRPfgAQQ5yFrU3F2UyK82JRlY3kWFkFk rDjYVwP6UaXYUIcPt/fpsXw8RqyUbh53rpoS++ofc1/N7VAoxVJYNfCQcalTLwUyAXsdvs0zzRDv7 p8tJ5kLw==; Received: from galois.linutronix.de ([193.142.43.55]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pnT6Y-00FjKa-0e; Fri, 14 Apr 2023 23:45:26 +0000 Message-ID: <20230414232311.192114505@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1681515906; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=OOycMYPFVi5YhixhBoVMbIx7KiHrtAXAV72ZpNnWZL8=; b=hJ/cJ/LsDaASOUqASLPPkSC6Ps6RxfdsA7gQYRBMpPaZEuge6FBpr2f6vVc+8HZFkB/rmb RJy5f49D9C3GZ5iAOC7Zko7KcGkJJr2aVQq4d0I8MHxwSrgTL3A0ePu6a/KfQsXuVA4Xbu ajazkAeZDlEyf5WTyO4+9yNz86aTWvAhKkL+wFnm59Pu29bb/grFar2EnhIA3IYkZTtEDm 0zbDi0Vg6WqjBsnDoMhcNfUK9CtbuR2h5QYljCFrrrsyKUW96SL8J44WHxTF6P75LRzQZI y5u0fp80f4IGejFZO38GlIfCPV4fIBaOEn1NXHQkG/2n87JafGV7laJAQBArhQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1681515906; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=OOycMYPFVi5YhixhBoVMbIx7KiHrtAXAV72ZpNnWZL8=; b=3elqLNqpb58Nl0QV7Y5mpEmw+yXYy7Jx8aSLhWQ4/jwrJTvN020KSkN4jR6yL3KGS/kbNN oIimmB7TZWnYdBDQ== From: Thomas Gleixner To: LKML Cc: x86@kernel.org, David Woodhouse , Andrew Cooper , Brian Gerst , "Arjan van de Veen" , Paolo Bonzini , Paul McKenney , Tom Lendacky , Sean Christopherson , Oleksandr Natalenko , Paul Menzel , "Guilherme G. Piccoli" , Piotr Gorski , David Woodhouse , Usama Arif , Juergen Gross , Boris Ostrovsky , xen-devel@lists.xenproject.org, Russell King , Arnd Bergmann , linux-arm-kernel@lists.infradead.org, Catalin Marinas , Will Deacon , Guo Ren , linux-csky@vger.kernel.org, Thomas Bogendoerfer , linux-mips@vger.kernel.org, "James E.J. Bottomley" , Helge Deller , linux-parisc@vger.kernel.org, Paul Walmsley , Palmer Dabbelt , linux-riscv@lists.infradead.org, Mark Rutland , Sabin Rapan Subject: [patch 32/37] cpu/hotplug: Allow "parallel" bringup up to CPUHP_BP_KICK_AP_STATE References: <20230414225551.858160935@linutronix.de> MIME-Version: 1.0 Date: Sat, 15 Apr 2023 01:45:05 +0200 (CEST) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230415_004522_782012_08659A7B X-CRM114-Status: GOOD ( 23.76 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org There is often significant latency in the early stages of CPU bringup, and time is wasted by waking each CPU (e.g. with SIPI/INIT/INIT on x86) and then waiting for it to respond before moving on to the next. Allow a platform to enable parallel setup which brings all to be onlined CPUs up to the CPUHP_BP_KICK_AP state. While this state advancement on the control CPU (BP) is single-threaded the important part is the last state CPUHP_BP_KICK_AP which wakes the to be onlined CPUs up. This allows the CPUs to run up to the first sychronization point cpuhp_ap_sync_alive() where they wait for the control CPU to release them one by one for the full onlining procedure. This parallelism depends on the CPU hotplug core sync mechanism which ensures that the parallel brought up CPUs wait for release before touching any state which would make the CPU visible to anything outside the hotplug control mechanism. To handle the SMT constraints of X86 correctly the bringup happens in two iterations when CONFIG_HOTPLUG_SMT is enabled. The control CPU brings up the primary SMT threads of each core first, which can load the microcode without the need to rendevouz with the thread siblings. Once that's completed it brings up the secondary SMT threads. Co-developed-by: David Woodhouse Signed-off-by: David Woodhouse Signed-off-by: Thomas Gleixner --- Documentation/admin-guide/kernel-parameters.txt | 6 + arch/Kconfig | 4 include/linux/cpuhotplug.h | 1 kernel/cpu.c | 103 ++++++++++++++++++++++-- 4 files changed, 109 insertions(+), 5 deletions(-) --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -815,6 +815,12 @@ on every CPU online, such as boot, and resume from suspend. Default: 10000 + cpuhp.parallel= + [SMP] Enable/disable parallel bringup of secondary CPUs + Format: + Default is enabled if CONFIG_HOTPLUG_PARALLEL=y. Otherwise + the parameter has no effect. + crash_kexec_post_notifiers Run kdump after running panic-notifiers and dumping kmsg. This only for the users who doubt kdump always --- a/arch/Kconfig +++ b/arch/Kconfig @@ -53,6 +53,10 @@ config HOTPLUG_SPLIT_STARTUP bool select HOTPLUG_CORE_SYNC_FULL +config HOTPLUG_PARALLEL + bool + select HOTPLUG_SPLIT_STARTUP + config GENERIC_ENTRY bool --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -526,6 +526,7 @@ void cpuhp_ap_sync_alive(void); void arch_cpuhp_sync_state_poll(void); void arch_cpuhp_cleanup_kick_cpu(unsigned int cpu); int arch_cpuhp_kick_ap_alive(unsigned int cpu, struct task_struct *tidle); +bool arch_cpuhp_init_parallel_bringup(void); #ifdef CONFIG_HOTPLUG_CORE_SYNC_DEAD void cpuhp_ap_report_dead(void); --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -649,8 +649,23 @@ bool cpu_smt_possible(void) cpu_smt_control != CPU_SMT_NOT_SUPPORTED; } EXPORT_SYMBOL_GPL(cpu_smt_possible); + +static inline bool cpuhp_smt_aware(void) +{ + return topology_smt_supported(); +} + +static inline const struct cpumask *cpuhp_get_primary_thread_mask(void) +{ + return cpu_primary_thread_mask; +} #else static inline bool cpu_smt_allowed(unsigned int cpu) { return true; } +static inline bool cpuhp_smt_aware(void) { return false; } +static inline const struct cpumask *cpuhp_get_primary_thread_mask(void) +{ + return cpu_present_mask; +} #endif static inline enum cpuhp_state @@ -1743,16 +1758,94 @@ int bringup_hibernate_cpu(unsigned int s return 0; } -void __init bringup_nonboot_cpus(unsigned int setup_max_cpus) +static void __init cpuhp_bringup_mask(const struct cpumask *mask, unsigned int ncpus, + enum cpuhp_state target) { unsigned int cpu; - for_each_present_cpu(cpu) { - if (num_online_cpus() >= setup_max_cpus) + for_each_cpu(cpu, mask) { + struct cpuhp_cpu_state *st = per_cpu_ptr(&cpuhp_state, cpu); + + if (!--ncpus) break; - if (!cpu_online(cpu)) - cpu_up(cpu, CPUHP_ONLINE); + + if (cpu_up(cpu, target) && can_rollback_cpu(st)) { + /* + * If this failed then cpu_up() might have only + * rolled back to CPUHP_BP_KICK_AP for the final + * online. Clean it up. NOOP if already rolled back. + */ + WARN_ON(cpuhp_invoke_callback_range(false, cpu, st, CPUHP_OFFLINE)); + } + } +} + +#ifdef CONFIG_HOTPLUG_PARALLEL +static bool __cpuhp_parallel_bringup __ro_after_init = true; + +static int __init parallel_bringup_parse_param(char *arg) +{ + return kstrtobool(arg, &__cpuhp_parallel_bringup); +} +early_param("cpuhp.parallel", parallel_bringup_parse_param); + +/* + * On architectures which have enabled parallel bringup this invokes all BP + * prepare states for each of the to be onlined APs first. The last state + * sends the startup IPI to the APs. The APs proceed through the low level + * bringup code in parallel and then wait for the control CPU to release + * them one by one for the final onlining procedure. + * + * This avoids waiting for each AP to respond to the startup IPI in + * CPUHP_BRINGUP_CPU. + */ +static bool __init cpuhp_bringup_cpus_parallel(unsigned int ncpus) +{ + const struct cpumask *mask = cpu_present_mask; + + if (__cpuhp_parallel_bringup) + __cpuhp_parallel_bringup = arch_cpuhp_init_parallel_bringup(); + if (!__cpuhp_parallel_bringup) + return false; + + if (cpuhp_smt_aware()) { + const struct cpumask *pmask = cpuhp_get_primary_thread_mask(); + static struct cpumask tmp_mask __initdata; + + /* + * X86 requires to prevent that SMT siblings stopped while + * the primary thread does a microcode update for various + * reasons. Bring the primary threads up first. + */ + cpumask_and(&tmp_mask, mask, pmask); + cpuhp_bringup_mask(&tmp_mask, ncpus, CPUHP_BP_KICK_AP); + cpuhp_bringup_mask(&tmp_mask, ncpus, CPUHP_ONLINE); + /* Account for the online CPUs */ + ncpus -= num_online_cpus(); + if (!ncpus) + return true; + /* Create the mask for secondary CPUs */ + cpumask_andnot(&tmp_mask, mask, pmask); + mask = &tmp_mask; } + + /* Bring the not-yet started CPUs up */ + cpuhp_bringup_mask(mask, ncpus, CPUHP_BP_KICK_AP); + cpuhp_bringup_mask(mask, ncpus, CPUHP_ONLINE); + return true; +} +#else +static inline bool cpuhp_bringup_cpus_parallel(unsigned int ncpus) { return false; } +#endif /* CONFIG_HOTPLUG_PARALLEL */ + +void __init bringup_nonboot_cpus(unsigned int setup_max_cpus) +{ + /* Try parallel bringup optimization if enabled */ + if (cpuhp_bringup_cpus_parallel(setup_max_cpus)) + return; + + /* Full per CPU serialized bringup */ + cpuhp_bringup_mask(cpu_present_mask, setup_max_cpus, CPUHP_ONLINE); } #ifdef CONFIG_PM_SLEEP_SMP