From patchwork Thu Apr 10 15:23:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 14046698 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5825BC369A5 for ; Thu, 10 Apr 2025 15:23:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA64C280113; Thu, 10 Apr 2025 11:23:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CB52B28010C; Thu, 10 Apr 2025 11:23:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B3133280113; Thu, 10 Apr 2025 11:23:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9363E28010C for ; Thu, 10 Apr 2025 11:23:52 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4EA5EBAD42 for ; Thu, 10 Apr 2025 15:23:53 +0000 (UTC) X-FDA: 83318504346.19.3C2CB2A Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf27.hostedemail.com (Postfix) with ESMTP id BB5E74000C for ; Thu, 10 Apr 2025 15:23:51 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=cBZB64Tv; spf=pass (imf27.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744298631; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xjd6QDnItT1Te77IumJblFaXH/YVD3VP9R5dDyQ+3D0=; b=IYacnstqpQ7/kBLwPIbb67phWQ2m/6HihXgfCTm2KHs08DM4N23PcbD6nBy9yO0DHn7ajd IJgKLyAfAEYJWtEE92K95JsFL4rOqBxAi+MJDiFYvn6eiayCQ11ZHB+bihXrFcDvcrACs+ u2UMJ+g3RW8SKqIrNelIBFbIViR3XgE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744298631; a=rsa-sha256; cv=none; b=k4v6NYXUahD8nfVG2vA8aos2HmByTYAA32qqTnvOkBskbT3c4EV20aMbf1/MNKvimjliEe qBoeWNeozm723QouACrt1HMvTfg0kI5cYI2mhtnF9GiNezhv0ZmtFe8qiI2i6vo6KljFwJ SPC7CrZ0Ida5fkPmBJFFR4RQX6B1d8A= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=cBZB64Tv; spf=pass (imf27.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id DEC555C5D16; Thu, 10 Apr 2025 15:21:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 272C5C4CEDD; Thu, 10 Apr 2025 15:23:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744298630; bh=uy0c2oFcabCgEI0cFKHtoYjpr7+77MB0L7hpksyKfHQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cBZB64TvnjyyfMbQs693eWSSDRKWnwPO1MetMG7eHb3iXXOJZtEfa824H25luTplL GepSJ+c/iNoAm6Dnc5W5ClEx7Abyoyp19dBkJFK4YXwJh1mgflHpvVwkluUr3NpBAr 9cVqcWIE1c2Iog8Dye5Mt999H4pFNFd7mkknWxblbA80ecio3iXYjHs3erzd5czkJh A64lYdbYyb/3lv6cN4Pn60IC0KRcOQmo+6JNXZ6s8eLcmUQC45twqa4V4O7zgqEk0g ekInDXHd4YWoMTGkyxGdhaV2Z4Rpurn9zxmBCO/9fQ0eehosPDGlOZosiWqymcqNXm uDROZVeA9Lrtw== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Andrew Morton , Ingo Molnar , Marcelo Tosatti , Michal Hocko , Oleg Nesterov , Peter Zijlstra , Thomas Gleixner , Valentin Schneider , Vlastimil Babka , linux-mm@kvack.org Subject: [PATCH 5/6] sched/isolation: Introduce isolated task work Date: Thu, 10 Apr 2025 17:23:26 +0200 Message-ID: <20250410152327.24504-6-frederic@kernel.org> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250410152327.24504-1-frederic@kernel.org> References: <20250410152327.24504-1-frederic@kernel.org> MIME-Version: 1.0 X-Stat-Signature: 1fjuc134ja4eg3e89b6uxcsqopgwq3qt X-Rspam-User: X-Rspamd-Queue-Id: BB5E74000C X-Rspamd-Server: rspam08 X-HE-Tag: 1744298631-977062 X-HE-Meta: U2FsdGVkX19WiCzO0bLr0XC/sCN0nduHh2ax6AGsz3w0ThkSaagpXl1K1DVFE8EKomVlYUUeZ2Bvc8yU7KY4rDGvJaOgsq0AGraizFHP2EZPPrCF0rFlUUAMxezQvPu/qpDf7E9IYKXMbHbe0qnmHpfTBVt6W5d+QLeZN9c44raDbO+RZwwYk5+EzXypzBV8d4JdhDAJFQqhWsmDDjQg9njiYiIH3ikMxDcHxudNG7dRwY9rLuCJgCQJL47iqhf7hJVxld+whs1E03zVtnCd17UWJP/hvNtNERL4IO0iZZV7HpHCMR7lxM8gc5QPjm3281MlpgU6wKZ/c3oPtVXoVF0Cw7eKdT0HAUsinOeEJsPItuajMydqVHmNDWQBdpd3MDWX7LqC0zyXm7XqQhzFt/dbEuWqkZzNuyBwxqjaifE2xnKdeMzU12KMF9kxWIFy54GmdEXP/Z2Hb6mFj1UDDZKaazbpVRDrBk5xbcXc0OAuoDQ/1PzSo3mbGMzwwflHZXtfrTVFD2Maet1IoI0l/d0gzElHT6Z1qwuOISntRkiwkRveU9BH1U2Y9Xcu4ozb6CRM6d8eIMP99VRoOwOua3LJAR0aV3k4GmR7vtXkNGKl0uoiDBn0lDWeCH/w3VmbPq786ILMHEqEoF0Zr9h1aquTxDzjGA4vDAOVBzeTalSoRde67pMayQwTLH1GGOQ2rtQU7bg6bAjvJli3hpzRdsgzACEE/x8KyfRGIZ+aC/Hc6drBYZK3jIpygQaPBN6IPWygM9tTowUZvST61fAIBHUC0Pa+MNchZIWT8M1UUT05Z4hlYPxwqQjPUR2DJME0dnvk84mwcUFGNn+YltizQ5RWYZPfy+hPIt+V6qTJuZq2IzAxgjgS9E5pR0HK3rgeocjljeGV18iDmoBQ4jcYvlG+WtQdVBmqxuyfLNl1SpA5+8zUvFRKDA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Some asynchronous kernel work may be pending upon resume to userspace and execute later on. On isolated workload this becomes problematic once the process is done with preparatory work involving syscalls and wants to run in userspace without being interrupted. Provide an infrastructure to queue a work to be executed from the current isolated task context right before resuming to userspace. This goes with the assumption that isolated tasks are pinned to a single nohz_full CPU. Signed-off-by: Frederic Weisbecker --- include/linux/sched.h | 1 + include/linux/sched/isolation.h | 17 +++++++++++++++++ kernel/sched/core.c | 1 + kernel/sched/isolation.c | 31 +++++++++++++++++++++++++++++++ kernel/sched/sched.h | 1 + 5 files changed, 51 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index b5ce76db6d75..4d764eb96e3e 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1437,6 +1437,7 @@ struct task_struct { #endif #ifdef CONFIG_NO_HZ_FULL + struct callback_head nohz_full_work; atomic_t tick_dep_mask; #endif diff --git a/include/linux/sched/isolation.h b/include/linux/sched/isolation.h index d8501f4709b5..74da4324b984 100644 --- a/include/linux/sched/isolation.h +++ b/include/linux/sched/isolation.h @@ -77,4 +77,21 @@ static inline bool cpu_is_isolated(int cpu) cpuset_cpu_is_isolated(cpu); } +#if defined(CONFIG_NO_HZ_FULL) +extern int __isolated_task_work_queue(void); + +static inline int isolated_task_work_queue(void) +{ + if (!housekeeping_cpu(raw_smp_processor_id(), HK_TYPE_KERNEL_NOISE)) + return -ENOTSUPP; + + return __isolated_task_work_queue(); +} + +extern void isolated_task_work_init(struct task_struct *tsk); +#else +static inline int isolated_task_work_queue(void) { return -ENOTSUPP; } +static inline void isolated_task_work_init(struct task_struct *tsk) { } +#endif /* CONFIG_NO_HZ_FULL */ + #endif /* _LINUX_SCHED_ISOLATION_H */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index add41254b6e5..c8b8b61ac3a6 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4524,6 +4524,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->migration_pending = NULL; #endif init_sched_mm_cid(p); + isolated_task_work_init(p); } DEFINE_STATIC_KEY_FALSE(sched_numa_balancing); diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c index 81bc8b329ef1..e246287de9fa 100644 --- a/kernel/sched/isolation.c +++ b/kernel/sched/isolation.c @@ -249,3 +249,34 @@ static int __init housekeeping_isolcpus_setup(char *str) return housekeeping_setup(str, flags); } __setup("isolcpus=", housekeeping_isolcpus_setup); + +#if defined(CONFIG_NO_HZ_FULL) +static void isolated_task_work(struct callback_head *head) +{ +} + +int __isolated_task_work_queue(void) +{ + unsigned long flags; + int ret; + + if (current->flags & PF_KTHREAD) + return -EINVAL; + + local_irq_save(flags); + if (task_work_queued(¤t->nohz_full_work)) { + ret = 0; + goto out; + } + + ret = task_work_add(current, ¤t->nohz_full_work, TWA_RESUME); +out: + local_irq_restore(flags); + return ret; +} + +void isolated_task_work_init(struct task_struct *tsk) +{ + init_task_work(&tsk->nohz_full_work, isolated_task_work); +} +#endif /* CONFIG_NO_HZ_FULL */ diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 47972f34ea70..e7dc4ae5ccc1 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -60,6 +60,7 @@ #include #include #include +#include #include #include #include