From patchwork Thu Jan 13 23:39:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Oskolkov X-Patchwork-Id: 12713190 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB30FC4332F for ; Thu, 13 Jan 2022 23:40:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3087B6B0073; Thu, 13 Jan 2022 18:40:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2910B6B0074; Thu, 13 Jan 2022 18:40:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 133936B0075; Thu, 13 Jan 2022 18:40:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.a.hostedemail.com [64.99.140.24]) by kanga.kvack.org (Postfix) with ESMTP id 027706B0073 for ; Thu, 13 Jan 2022 18:40:06 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id BC9BE234B3 for ; Thu, 13 Jan 2022 23:40:05 +0000 (UTC) X-FDA: 79026884370.01.65EACA1 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf31.hostedemail.com (Postfix) with ESMTP id EC9A420004 for ; Thu, 13 Jan 2022 23:40:04 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id v66-20020a256145000000b006115377709aso14765457ybb.3 for ; Thu, 13 Jan 2022 15:40:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=FMGfyZNGg16kQgWmzvaNooZwk13glosUs6aVPqZ/nrc=; b=dPjeDOi3O6ASy2ii197CAuCJWdyt8duA1D7+djoQotRqDWEhrzMmCDnirA/nQ9hobQ b0jwGR5DttHhc7TO0zCq9P83NNX8HIuOu7C3PuUi1naYFMUSXJLqh/bj7/0/HUMide1a CQ7+ws5aRRjDBPW3sVeq/GIKcXNaoWg9GlsmsDhG0RZFV53D9DAEMUYsppjTQW/+r9pl hPifaqFSqER+Dfj0ncmOx3oxCYdMKfxadHa0FxG4lAh+xyj5K3yQN35aVqZ7Hq8hNZDe mLUDfq3b+sMRtPRShVuPrMejCh12/LAjHxZAbOMHfdK+fhggZQjwS9ZxWf50K3mVt1lr 7F/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=FMGfyZNGg16kQgWmzvaNooZwk13glosUs6aVPqZ/nrc=; b=1RRsg7DEe+o6FQkyT9OOsGcEIRKK+fS4gDoBp8gBd+1LU7N98dW1YTw51qeD5YNQzA hGh3oYhEXa/nH6zGEnJtWhXllAgSvK6SqVLqaEzgvSd7jNVB0cRNcHMr44bJT7hdeHO2 zsOWk8bGyQ1cEQ86DIDEHjVOes3yOpidBjlnTvDAl9eD6u5URpwmFmFtBSr6lzx2mS8M voTYRmR3aKv4i3HFYEVG2z+6cyfSCmFp9pFALrrIPZMx3zr9AYxjpnmofEpIj28R+C/x rILzJt0i0dq7084cZ+hXmJPR+lysdccojaNemV7jMTA1n4nh1ZQdKOpuew9day+cwEM8 l3Iw== X-Gm-Message-State: AOAM5318lFYwE8Qg18ks+h04B1jyBd8eo5pPIcZwATbVJyUbta+pMXe/ to5vkKP7btxJoFmHxUhPYDyYlZAj X-Google-Smtp-Source: ABdhPJz5otcxfEhE6MWpPlPMKjfGbxs3qBpBtoEpYWyLi1d9W5GUhm/RULyKZYaWhqebI5v0hyA4l39c X-Received: from posk.svl.corp.google.com ([2620:15c:2cd:202:c548:e79f:8954:121f]) (user=posk job=sendgmr) by 2002:a25:9a02:: with SMTP id x2mr9180866ybn.701.1642117204312; Thu, 13 Jan 2022 15:40:04 -0800 (PST) Date: Thu, 13 Jan 2022 15:39:36 -0800 In-Reply-To: <20220113233940.3608440-1-posk@google.com> Message-Id: <20220113233940.3608440-2-posk@google.com> Mime-Version: 1.0 References: <20220113233940.3608440-1-posk@google.com> X-Mailer: git-send-email 2.34.1.703.g22d0c6ccf7-goog Subject: [RFC PATCH v2 1/5] sched/umcg: add WF_CURRENT_CPU and externise ttwu From: Peter Oskolkov To: Peter Zijlstra , mingo@redhat.com, tglx@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-api@vger.kernel.org, x86@kernel.org, pjt@google.com, posk@google.com, avagin@google.com, jannh@google.com, tdelisle@uwaterloo.ca, posk@posk.io X-Rspamd-Queue-Id: EC9A420004 X-Stat-Signature: te11r6k7zkjdj47dfxmjdokgdpxcs5oi Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=dPjeDOi3; spf=pass (imf31.hostedemail.com: domain of 3VLjgYQQKCF0KJNFBJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--posk.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3VLjgYQQKCF0KJNFBJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--posk.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam07 X-HE-Tag: 1642117204-867058 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Peter Oskolkov Add WF_CURRENT_CPU wake flag that advices the scheduler to move the wakee to the current CPU. This is useful for fast on-CPU context switching use cases such as UMCG. In addition, make ttwu external rather than static so that the flag could be passed to it from outside of sched/core.c. Signed-off-by: Peter Oskolkov Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20211122211327.5931-2-posk@google.com --- kernel/sched/core.c | 3 +-- kernel/sched/fair.c | 4 ++++ kernel/sched/sched.h | 15 +++++++++------ 3 files changed, 14 insertions(+), 8 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 83872f95a1ea..04525933de94 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3980,8 +3980,7 @@ bool ttwu_state_match(struct task_struct *p, unsigned int state, int *success) * Return: %true if @p->state changes (an actual wakeup was done), * %false otherwise. */ -static int -try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) +int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) { unsigned long flags; int cpu, success = 0; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 095b0aa378df..4b70cf8f1ec3 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6838,6 +6838,10 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags) if (wake_flags & WF_TTWU) { record_wakee(p); + if ((wake_flags & WF_CURRENT_CPU) && + cpumask_test_cpu(cpu, p->cpus_ptr)) + return cpu; + if (sched_energy_enabled()) { new_cpu = find_energy_efficient_cpu(p, prev_cpu); if (new_cpu >= 0) diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index de53be905739..77f67d09b946 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2052,13 +2052,14 @@ static inline int task_on_rq_migrating(struct task_struct *p) } /* Wake flags. The first three directly map to some SD flag value */ -#define WF_EXEC 0x02 /* Wakeup after exec; maps to SD_BALANCE_EXEC */ -#define WF_FORK 0x04 /* Wakeup after fork; maps to SD_BALANCE_FORK */ -#define WF_TTWU 0x08 /* Wakeup; maps to SD_BALANCE_WAKE */ +#define WF_EXEC 0x02 /* Wakeup after exec; maps to SD_BALANCE_EXEC */ +#define WF_FORK 0x04 /* Wakeup after fork; maps to SD_BALANCE_FORK */ +#define WF_TTWU 0x08 /* Wakeup; maps to SD_BALANCE_WAKE */ -#define WF_SYNC 0x10 /* Waker goes to sleep after wakeup */ -#define WF_MIGRATED 0x20 /* Internal use, task got migrated */ -#define WF_ON_CPU 0x40 /* Wakee is on_cpu */ +#define WF_SYNC 0x10 /* Waker goes to sleep after wakeup */ +#define WF_MIGRATED 0x20 /* Internal use, task got migrated */ +#define WF_ON_CPU 0x40 /* Wakee is on_cpu */ +#define WF_CURRENT_CPU 0x80 /* Prefer to move the wakee to the current CPU. */ #ifdef CONFIG_SMP static_assert(WF_EXEC == SD_BALANCE_EXEC); @@ -3112,6 +3113,8 @@ static inline bool is_per_cpu_kthread(struct task_struct *p) extern void swake_up_all_locked(struct swait_queue_head *q); extern void __prepare_to_swait(struct swait_queue_head *q, struct swait_queue *wait); +extern int try_to_wake_up(struct task_struct *tsk, unsigned int state, int wake_flags); + #ifdef CONFIG_PREEMPT_DYNAMIC extern int preempt_dynamic_mode; extern int sched_dynamic_mode(const char *str);