From patchwork Sun Feb 9 22:30:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13967158 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DF45C0219B for ; Sun, 9 Feb 2025 22:30:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 891D0280002; Sun, 9 Feb 2025 17:30:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 841206B0096; Sun, 9 Feb 2025 17:30:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6BCA8280002; Sun, 9 Feb 2025 17:30:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4A9C16B0095 for ; Sun, 9 Feb 2025 17:30:31 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id F0159140B98 for ; Sun, 9 Feb 2025 22:30:30 +0000 (UTC) X-FDA: 83101851420.27.7AA8D3B Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf20.hostedemail.com (Postfix) with ESMTP id 4D5DF1C0013 for ; Sun, 9 Feb 2025 22:30:29 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=DZCOzR8n; spf=pass (imf20.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739140229; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=speXJxf6MRThMCo7hPIqARwtF16DCSTlU2Q9hXtFiPY=; b=lC7KqvK8aSsMDA/oAHvcXxsjxRceRSwfHRNIdatPqdoWFDxADuRcdYTxlYV4HA42ZVxt6K EqduWjkvDyi3hTPz6PJ0V8DT8NYIjj9G60ctAs/NY/Qq+sfMNkMwOXmK4bkg2xquZMLiPN AgVdR6dqLJJcmidlKpctGXLu0Zz1Z2o= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=DZCOzR8n; spf=pass (imf20.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739140229; a=rsa-sha256; cv=none; b=YgqCr3ia1hYXTKPKQSjeOOKtTA0AN9wcKPzy6yJ3yVOwniFIItgCTNF5Dh4jR5dXcwIBlX +D2jaEv9D+3HFRe/t2uA1zBbYtIpHjdJDRB+rumUm9PBuG4vX/f/tmdnqb1KX5GGlYi67Q xj8JUWC6KmiekGJ59Xb1Pujiw2djQno= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id E76785C353D; Sun, 9 Feb 2025 22:29:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 99927C4CEE7; Sun, 9 Feb 2025 22:30:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739140228; bh=hpX5Isxts6HZwFzDaFwdkHMytRYO5OiHbV2gKBTOr1s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DZCOzR8nKDpcLHHG+pUeg6XDkn23fPlzAQdzNGIId4ORHo4fyoa7vHVWO+9GkTzkQ rOAL8y9B5yI2fOD/bhpjqte9Ww4uACLQTLAGJ7r1DQtJK++Vu0v3ysxhrN0ae+AgmQ YOyoQmsrC1+06KU7jI2g+OOWGuThvARFCN82GCvxhURurB94W68F854JlfHd+gjI/a 0/Stpzb4hrHV3Z5A+rddkQ5H4Tgmv46ELAZEuc240GNCyulpPTbhMuHJ6crLPOE0lC nHBhpPNlvMIz9X+ja6w5uCnxjlAC7wDmN5gdKhX0MbzYNyOsJ0RJfSxRqIBmfDVKDm lGvlFEzyz5cCw== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov , linux-mm@kvack.org Subject: [PATCH 5/6 v2] sched/isolation: Introduce isolated task work Date: Sun, 9 Feb 2025 23:30:03 +0100 Message-ID: <20250209223005.11519-6-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250209223005.11519-1-frederic@kernel.org> References: <20250209223005.11519-1-frederic@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 4D5DF1C0013 X-Stat-Signature: ps8oyuja6auza8swcwpmbf8eebdiucz5 X-HE-Tag: 1739140229-859382 X-HE-Meta: U2FsdGVkX1+Od0sb7mnX7JvMUXeheyz6iAE92Rt4QD/LUtNXlPtLFZQn5cc7KnJxbdIFwVrAWD0CUm/OdH/xM5im9LPe2JlGQ7OegmL8qLLFCf3P9hKxH+v3OIK0BoKTWpYEGZ3mb7+Zu1ZP7DCf2LZv1VrWdaJ7hOMI3nDSxbHUQF8aA349HnZ9OZNY4XjEZ3E2yGdWyrLdqU4f4dXt+pQaETSOh429J4FMMY+fhpMWDtc4KUoycR6j+++VCvS74lHrO/zi443FSmr5bFzgmj1gWPLw/cFHd6L3ZIimwEqtrR103L2gj1YgWeJe9hYVnMHkIPa0FdUvyG3DYHpWBvNwUPe7fjRyrqiikWYvmCE/7iptoF98YiWPcfX14TW3IWVHCtLAzoUkelL1lNpPfQRy8MufQMXdQOJIvRMBK0f9govUTZ2zUYYbsRmlLVqzlw2XxKMh2WZPzpG/3g3ROPELR/fw7xmLXSHGrGyHV/Hn33M/7hI8u3QNqe2bzX/VAsjdRYglCvU3Et7XfTtWat8X8Wbbq9wC2htYpFcs07wnGADfb97+0nxECZhvoKVbNkY9N2WKUnGtgH4dmkGknunBNAcQTZ8IIGQyo2qCahFy80RFtSlIMyS9NLQ7uGMGIalSA+7np74dTMz8iVrIO9WpV6uEzMfJ8YQ7ZZ82lwg71re5/VNd5LL6WnjIXGA11DOIk90BZ3ktUfdk35juVrzZhoz3dSM9flNd3vYcivqAvKBh4L4Iqs8FCtM3PBRBBl7ujSY3xffy4AcypEUlS/OJ99WcDGdX9JBfJaYwLK9UhSxe6Xmw/B+nPhHEA6pWs+eVMLMBQBf5nMQVpyoQ3kIPLnfcYS1i4iVPrV+i8lKX+Pqu8kJKWn9SToVhfYEuVDnh+gDNoJ6s3JxiTr1VOXdfAaUgZAyYBdFczGEqKqR4Vs7867GawOkrUkMMmXTJWoXyePmwN7ig1NCT+of DVQxTMLn okiVz/aDefXGduoxcK/2/hrlPzX+hs2gNmwpHYsgenlU3HbQ7usguZ5e+Z+S8KRKml+tN/8xuUOnqUnzgCD8OGyaLsJsbu2qE42qRZWB/2uqt7dpEV5S1jO2Avki0kcgYlB3IxU7lOHT77jW2iH7D7oS6Kq9BYGdYFH2nn7vHp2n2rr02S75HDrj0mGFNdj6SUyvbPHKB+5QPXPC2ZrROl6C3WGV7YdewjWCnCMi186XcAYY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Some asynchronous kernel work may be pending upon resume to userspace and execute later on. On isolated workload this becomes problematic once the process is done with preparatory work involving syscalls and wants to run in userspace without being interrupted. Provide an infrastructure to queue a work to be executed from the current isolated task context right before resuming to userspace. This goes with the assumption that isolated tasks are pinned to a single nohz_full CPU. Signed-off-by: Frederic Weisbecker --- include/linux/sched.h | 1 + include/linux/sched/isolation.h | 17 +++++++++++++++++ kernel/sched/core.c | 1 + kernel/sched/isolation.c | 31 +++++++++++++++++++++++++++++++ kernel/sched/sched.h | 1 + 5 files changed, 51 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index 10a9aa41b43a..82827f962745 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1410,6 +1410,7 @@ struct task_struct { #endif #ifdef CONFIG_NO_HZ_FULL + struct callback_head nohz_full_work; atomic_t tick_dep_mask; #endif diff --git a/include/linux/sched/isolation.h b/include/linux/sched/isolation.h index d8501f4709b5..74da4324b984 100644 --- a/include/linux/sched/isolation.h +++ b/include/linux/sched/isolation.h @@ -77,4 +77,21 @@ static inline bool cpu_is_isolated(int cpu) cpuset_cpu_is_isolated(cpu); } +#if defined(CONFIG_NO_HZ_FULL) +extern int __isolated_task_work_queue(void); + +static inline int isolated_task_work_queue(void) +{ + if (!housekeeping_cpu(raw_smp_processor_id(), HK_TYPE_KERNEL_NOISE)) + return -ENOTSUPP; + + return __isolated_task_work_queue(); +} + +extern void isolated_task_work_init(struct task_struct *tsk); +#else +static inline int isolated_task_work_queue(void) { return -ENOTSUPP; } +static inline void isolated_task_work_init(struct task_struct *tsk) { } +#endif /* CONFIG_NO_HZ_FULL */ + #endif /* _LINUX_SCHED_ISOLATION_H */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 606f596a6e0d..78b4b996f85d 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4525,6 +4525,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->migration_pending = NULL; #endif init_sched_mm_cid(p); + isolated_task_work_init(p); } DEFINE_STATIC_KEY_FALSE(sched_numa_balancing); diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c index 81bc8b329ef1..f25a5cb33c0d 100644 --- a/kernel/sched/isolation.c +++ b/kernel/sched/isolation.c @@ -249,3 +249,34 @@ static int __init housekeeping_isolcpus_setup(char *str) return housekeeping_setup(str, flags); } __setup("isolcpus=", housekeeping_isolcpus_setup); + +#if defined(CONFIG_NO_HZ_FULL) +static void isolated_task_work(struct callback_head *head) +{ +} + +int __isolated_task_work_queue(void) +{ + unsigned long flags; + int ret; + + if (current->flags & PF_KTHREAD) + return 0; + + local_irq_save(flags); + if (task_work_queued(¤t->nohz_full_work)) { + ret = 0; + goto out; + } + + ret = task_work_add(current, ¤t->nohz_full_work, TWA_RESUME); +out: + local_irq_restore(flags); + return ret; +} + +void isolated_task_work_init(struct task_struct *tsk) +{ + init_task_work(&tsk->nohz_full_work, isolated_task_work); +} +#endif /* CONFIG_NO_HZ_FULL */ diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 38e0e323dda2..f80dc1cad219 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -60,6 +60,7 @@ #include #include #include +#include #include #include #include