From patchwork Tue Dec 15 11:16:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 11974563 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC5F4C4361B for ; Tue, 15 Dec 2020 11:18:50 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4F4D422255 for ; Tue, 15 Dec 2020 11:18:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4F4D422255 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=9JGZY7qWNJD9IdtPOaWxmoSdtj2jVzGKO/0KDCIU2aQ=; b=uMK7pjZuKDIeX8RfG3m7HIIrG UnVAqEeK0g0F0DCuvHBr1F3Uwq51SKZ5id3gk4AhanTAZIlcHqExDZjryQsuKU/bKWpcJkLgbxXMu jb8cwXWyYMIuSrEYX9U6XohlT1iroNTskgrAnzVG1yElXlnmGS/IsM18hHXk7d9d1tmjBygqVoj5I u3YqgHy11vtScm7IA2hG4fohah5RMzj7Y+IHO8JQ4pJu2/MPl92hPm+PWPGA6sAcPg+wJ6UqKvgWo gHNNkpZtXR2ExiHZbKcOFH3ArAAC9lYWRH9MZBGzKqvlxj52oYNt4YtAvgSjDo0e561xvQAQtJFNR DPD8LjMbw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kp8Km-0005NV-Q1; Tue, 15 Dec 2020 11:17:36 +0000 Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kp8Ki-0005MM-MW for linux-arm-kernel@lists.infradead.org; Tue, 15 Dec 2020 11:17:34 +0000 Received: by mail-pl1-x630.google.com with SMTP id bj5so10853892plb.4 for ; Tue, 15 Dec 2020 03:17:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nclCHANFNijvEDnymHTZhqBOvD0PdZUMm8AQp2xAWLI=; b=uOj10VFTGCZlHtTLgzshL9J7I+7HFxXnslBQ6WH1pbh+rQukw+h8S0dpORerJ5H2oH T3vI1t+J8y5Rz1gKkhQMn5mrBOLRsaHcHHsFijW1dK63wgnDdjxbPA5TbhfDyJmhmQGE ABHxvNaIvNAI43e9owLbZs/ZicJHyZVjkGaIG+L/HARYb8+jpe1HhScjAGi6CeQtqg5Y mWCFVrcqFbq+e+KHJPPY/Uz/FsbUkj6aDf7ZiCs27CwXG7hu6p9feXPhpkfxGhTHecZQ TPSXzDazZAj+eAQsUNwddaSsBVnOYV2DJhWfVNCLB7EJ+qMjMqNnt023j2w0fDq+CUNO ppRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nclCHANFNijvEDnymHTZhqBOvD0PdZUMm8AQp2xAWLI=; b=H4RVG9HPIhlDRrmSJ6D5CG3gUX9cv7eeCMmjwh49s816xBIRILc8Zxf+6/R7eoFR3N jFOcB6K45/DbVM8dheHo7CK+YKdCJ4Z8KqKUHnoXdWLCG+sPdkCeroh8G/B1TANtelBr 1E4EfcRsVFQyt54f/L6dSotqMhg4+97TnG/4xRUSRwfnuHfEP4HUIEPHABlUgPNQJlCn CelEktHJPDAKbN9YxcmGVC47Is0kNzm3FHavebHtgHTF2J9g2Ki1b8jPU0EYsV7nytAE yCABdNE44Mqn+qFJfZM6iSVP9BDYmNGadPje7WJYyF1kuZEyIXhgI8TqbEiJSH7OtNLi V1zw== X-Gm-Message-State: AOAM532mw/LVIMfnGF5YmAtmMJbrVbyTbSwwHCu0yNoyj8uA5N1yhS9g PSRGOhKQ7lDZUpNGLtqwaCZ/3w== X-Google-Smtp-Source: ABdhPJycj3p9qs4bJA1A2K4DuS1ScRWDc3mE4oOc+jvseAOncXmuTkTMszfNTV6wm5uUs/E585De9A== X-Received: by 2002:a17:90a:ea02:: with SMTP id w2mr29527143pjy.99.1608031049626; Tue, 15 Dec 2020 03:17:29 -0800 (PST) Received: from localhost ([122.172.20.109]) by smtp.gmail.com with ESMTPSA id x7sm22694450pfb.96.2020.12.15.03.17.28 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Dec 2020 03:17:29 -0800 (PST) From: Viresh Kumar To: Rafael Wysocki , Ionela Voinescu Subject: [RFC V2 1/2] topology: Allow multiple entities to provide sched_freq_tick() callback Date: Tue, 15 Dec 2020 16:46:35 +0530 Message-Id: <28ade070dd80f6f22d6e8fce5db5f0142b428fa9.1608030508.git.viresh.kumar@linaro.org> X-Mailer: git-send-email 2.25.0.rc1.19.g042ed3e048af In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201215_061732_770220_B88D32EB X-CRM114-Status: GOOD ( 21.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Vincent Guittot , linux-pm@vger.kernel.org, Viresh Kumar , linux-kernel@vger.kernel.org, Peter Puhov , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This patch attempts to make it generic enough so other parts of the kernel can also provide their own implementation of scale_freq_tick() callback, which is called by the scheduler periodically to update the per-cpu freq_scale variable. The implementations now need to provide struct scale_freq_tick_data for the CPUs for which they have hardware counters available, and a callback gets registered for each possible CPU in a per-cpu variable. The AMU counters are updated to adapt to this and they take the highest priority if they are available, i.e. they will be used instead of CPPC based counters for example. Signed-off-by: Viresh Kumar --- arch/arm64/include/asm/topology.h | 8 +-- arch/arm64/kernel/topology.c | 89 ++++++++++++++----------------- drivers/base/arch_topology.c | 56 +++++++++++++++++-- include/linux/arch_topology.h | 13 ++++- 4 files changed, 104 insertions(+), 62 deletions(-) diff --git a/arch/arm64/include/asm/topology.h b/arch/arm64/include/asm/topology.h index 3b8dca4eb08d..be6a53ba3e2d 100644 --- a/arch/arm64/include/asm/topology.h +++ b/arch/arm64/include/asm/topology.h @@ -17,15 +17,9 @@ int pcibus_to_node(struct pci_bus *bus); #include void update_freq_counters_refs(void); -void topology_scale_freq_tick(void); -#ifdef CONFIG_ARM64_AMU_EXTN -/* - * Replace task scheduler's default counter-based - * frequency-invariance scale factor setting. - */ +/* Replace task scheduler's default frequency-invariance scale factor setting */ #define arch_scale_freq_tick topology_scale_freq_tick -#endif /* CONFIG_ARM64_AMU_EXTN */ /* Replace task scheduler's default frequency-invariant accounting */ #define arch_set_freq_scale topology_set_freq_scale diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c index 0fc2b32ddb45..22f056821585 100644 --- a/arch/arm64/kernel/topology.c +++ b/arch/arm64/kernel/topology.c @@ -199,8 +199,44 @@ static int freq_inv_set_max_ratio(int cpu, u64 max_rate, u64 ref_rate) return 0; } -static DEFINE_STATIC_KEY_FALSE(amu_fie_key); -#define amu_freq_invariant() static_branch_unlikely(&amu_fie_key) +static void amu_scale_freq_tick(void) +{ + u64 prev_core_cnt, prev_const_cnt; + u64 core_cnt, const_cnt, scale; + + prev_const_cnt = this_cpu_read(arch_const_cycles_prev); + prev_core_cnt = this_cpu_read(arch_core_cycles_prev); + + update_freq_counters_refs(); + + const_cnt = this_cpu_read(arch_const_cycles_prev); + core_cnt = this_cpu_read(arch_core_cycles_prev); + + if (unlikely(core_cnt <= prev_core_cnt || + const_cnt <= prev_const_cnt)) + return; + + /* + * /\core arch_max_freq_scale + * scale = ------- * -------------------- + * /\const SCHED_CAPACITY_SCALE + * + * See validate_cpu_freq_invariance_counters() for details on + * arch_max_freq_scale and the use of SCHED_CAPACITY_SHIFT. + */ + scale = core_cnt - prev_core_cnt; + scale *= this_cpu_read(arch_max_freq_scale); + scale = div64_u64(scale >> SCHED_CAPACITY_SHIFT, + const_cnt - prev_const_cnt); + + scale = min_t(unsigned long, scale, SCHED_CAPACITY_SCALE); + this_cpu_write(freq_scale, (unsigned long)scale); +} + +static struct scale_freq_tick_data amu_sftd = { + .source = SCALE_FREQ_SOURCE_AMU, + .scale_freq = amu_scale_freq_tick, +}; static void amu_fie_setup(const struct cpumask *cpus) { @@ -227,7 +263,7 @@ static void amu_fie_setup(const struct cpumask *cpus) if (!invariant && !cpumask_equal(amu_fie_cpus, cpu_present_mask)) return; - static_branch_enable(&amu_fie_key); + topology_set_scale_freq_tick(&amu_sftd, amu_fie_cpus); pr_info("CPUs[%*pbl]: counters will be used for FIE.", cpumask_pr_args(amu_fie_cpus)); @@ -283,53 +319,6 @@ static int __init init_amu_fie(void) } core_initcall(init_amu_fie); -bool arch_freq_counters_available(const struct cpumask *cpus) -{ - return amu_freq_invariant() && - cpumask_subset(cpus, amu_fie_cpus); -} - -void topology_scale_freq_tick(void) -{ - u64 prev_core_cnt, prev_const_cnt; - u64 core_cnt, const_cnt, scale; - int cpu = smp_processor_id(); - - if (!amu_freq_invariant()) - return; - - if (!cpumask_test_cpu(cpu, amu_fie_cpus)) - return; - - prev_const_cnt = this_cpu_read(arch_const_cycles_prev); - prev_core_cnt = this_cpu_read(arch_core_cycles_prev); - - update_freq_counters_refs(); - - const_cnt = this_cpu_read(arch_const_cycles_prev); - core_cnt = this_cpu_read(arch_core_cycles_prev); - - if (unlikely(core_cnt <= prev_core_cnt || - const_cnt <= prev_const_cnt)) - return; - - /* - * /\core arch_max_freq_scale - * scale = ------- * -------------------- - * /\const SCHED_CAPACITY_SCALE - * - * See validate_cpu_freq_invariance_counters() for details on - * arch_max_freq_scale and the use of SCHED_CAPACITY_SHIFT. - */ - scale = core_cnt - prev_core_cnt; - scale *= this_cpu_read(arch_max_freq_scale); - scale = div64_u64(scale >> SCHED_CAPACITY_SHIFT, - const_cnt - prev_const_cnt); - - scale = min_t(unsigned long, scale, SCHED_CAPACITY_SCALE); - this_cpu_write(freq_scale, (unsigned long)scale); -} - #ifdef CONFIG_ACPI_CPPC_LIB #include diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c index de8587cc119e..07774fb47bc4 100644 --- a/drivers/base/arch_topology.c +++ b/drivers/base/arch_topology.c @@ -21,17 +21,65 @@ #include #include +static DEFINE_PER_CPU(struct scale_freq_tick_data *, sft_data); +static struct cpumask scale_freq_tick_mask; + +static bool supports_scale_freq_tick(const struct cpumask *cpus) +{ + return cpumask_subset(cpus, &scale_freq_tick_mask); +} + bool topology_scale_freq_invariant(void) { return cpufreq_supports_freq_invariance() || - arch_freq_counters_available(cpu_online_mask); + supports_scale_freq_tick(cpu_online_mask); +} + +void topology_set_scale_freq_tick(struct scale_freq_tick_data *data, + const struct cpumask *cpus) +{ + struct scale_freq_tick_data *sftd; + int cpu; + + for_each_cpu(cpu, cpus) { + sftd = per_cpu(sft_data, cpu); + + /* Use AMU counters whenever possible */ + if (!sftd || sftd->source != SCALE_FREQ_SOURCE_AMU) { + per_cpu(sft_data, cpu) = data; + cpumask_set_cpu(cpu, &scale_freq_tick_mask); + } + } } +EXPORT_SYMBOL_GPL(topology_set_scale_freq_tick); -__weak bool arch_freq_counters_available(const struct cpumask *cpus) +void topology_clear_scale_freq_tick(enum scale_freq_tick_source source, + const struct cpumask *cpus) { - return false; + struct scale_freq_tick_data *sftd; + int cpu; + + for_each_cpu(cpu, cpus) { + sftd = per_cpu(sft_data, cpu); + + if (sftd && sftd->source == source) { + per_cpu(sft_data, cpu) = NULL; + cpumask_clear_cpu(cpu, &scale_freq_tick_mask); + } + } } +EXPORT_SYMBOL_GPL(topology_clear_scale_freq_tick); + +void topology_scale_freq_tick(void) +{ + struct scale_freq_tick_data *sftd = *this_cpu_ptr(&sft_data); + + if (sftd) + sftd->scale_freq(); +} + DEFINE_PER_CPU(unsigned long, freq_scale) = SCHED_CAPACITY_SCALE; +EXPORT_SYMBOL_GPL(freq_scale); void topology_set_freq_scale(const struct cpumask *cpus, unsigned long cur_freq, unsigned long max_freq) @@ -47,7 +95,7 @@ void topology_set_freq_scale(const struct cpumask *cpus, unsigned long cur_freq, * want to update the scale factor with information from CPUFREQ. * Instead the scale factor will be updated from arch_scale_freq_tick. */ - if (arch_freq_counters_available(cpus)) + if (supports_scale_freq_tick(cpus)) return; scale = (cur_freq << SCHED_CAPACITY_SHIFT) / max_freq; diff --git a/include/linux/arch_topology.h b/include/linux/arch_topology.h index 0f6cd6b73a61..b2422ebef2dd 100644 --- a/include/linux/arch_topology.h +++ b/include/linux/arch_topology.h @@ -34,7 +34,18 @@ void topology_set_freq_scale(const struct cpumask *cpus, unsigned long cur_freq, unsigned long max_freq); bool topology_scale_freq_invariant(void); -bool arch_freq_counters_available(const struct cpumask *cpus); +enum scale_freq_tick_source { + SCALE_FREQ_SOURCE_AMU, +}; + +struct scale_freq_tick_data { + enum scale_freq_tick_source source; + void (*scale_freq)(void); +}; + +void topology_scale_freq_tick(void); +void topology_set_scale_freq_tick(struct scale_freq_tick_data *data, const struct cpumask *cpus); +void topology_clear_scale_freq_tick(enum scale_freq_tick_source source, const struct cpumask *cpus); DECLARE_PER_CPU(unsigned long, thermal_pressure);