From patchwork Sun Feb 9 22:29:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13967154 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED4C4C021A1 for ; Sun, 9 Feb 2025 22:30:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 55BE06B0085; Sun, 9 Feb 2025 17:30:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E5586B0088; Sun, 9 Feb 2025 17:30:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3AB606B0089; Sun, 9 Feb 2025 17:30:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1DBB36B0085 for ; Sun, 9 Feb 2025 17:30:19 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D12501A0E54 for ; Sun, 9 Feb 2025 22:30:18 +0000 (UTC) X-FDA: 83101850916.11.7DC3D9E Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf23.hostedemail.com (Postfix) with ESMTP id 353E8140005 for ; Sun, 9 Feb 2025 22:30:17 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=KAPDlktu; spf=pass (imf23.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739140217; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Wwcm63/SXqoYo+OkHy67Yhb8UnsnykgZyNY9QjtMMiU=; b=LMPZPPjWUCaEUS7EK+XHBDd5eVFRTnGAudpT4REAaHouSEbWmkLHQ40kJvR48GxmsuUKVl 2y+Uy64ofFeaI6JEbT2ZNzKfYvv8ZOaax98n7zS9Az5+PnSYlTi/yG3Lq+hhP2VKcxZLE5 cvwOT8jQOyUDaSkE0TMrA1Aii0MELd8= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=KAPDlktu; spf=pass (imf23.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739140217; a=rsa-sha256; cv=none; b=FSm0Mtvq/Zs8mtbxOVutc3PsXHcrpKkOOdi3NNVVHufg930E2NorHoW6SjtCH5zpD5Rfkx 4gifSDbvR00BEj7g0Ek8568hiI3zZshNZbKJMMA8cJnYag7PfOL90phaYp+6ciosSP/r9N cwOdXoabvOft5Ek+MwRM1yh7y7MSiwE= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id D738B5C5A21; Sun, 9 Feb 2025 22:29:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A12FBC4CEDF; Sun, 9 Feb 2025 22:30:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739140216; bh=Wqntl1ggAaL74aa0p3y2a++/5r9dcdOK0C0/zF8B/OM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KAPDlktuV4x4lOY7dt1MX379eFc7iOHZYEvyz50M+V1vMYLZ7fvi/DNn8BHL0uWC0 cXq25hvD0Trn40F1pWIwz9IczyCePognvXVQCGaIAKDvM2xUvzI6+hvU03b5Cy6676 LrZm7F5c4pvdz6EE7y7SE2gv1dj+bDk7jWASzAw/FMf9AVm6+e4zikrFJvFmLs4lOd 5iO3Oxaf7du0MevTy+0WWhJdedrBIABnQJ9GZ2QAeADelMF09h3j9edOOY4Ltj8gEI 9LAaHzsaa8DCVFuBbzU19CGwd0mRHpbQgpwddFBVCMCemNmiHRz8OEsri+qzxovymK LVdtWCPkeLQxA== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov , linux-mm@kvack.org Subject: [PATCH 1/6 v2] task_work: Provide means to check if a work is queued Date: Sun, 9 Feb 2025 23:29:59 +0100 Message-ID: <20250209223005.11519-2-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250209223005.11519-1-frederic@kernel.org> References: <20250209223005.11519-1-frederic@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: y7etra196i9poxwsptf448koj16k1rur X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 353E8140005 X-HE-Tag: 1739140217-433472 X-HE-Meta: U2FsdGVkX19RdehvKWDfY/ib4fDtLcJWx+VnAz8BvrRqpHA2EmfDet9hY7rMr79ExAnV6sw7r8moSvcvizAbiYn/wGxcMot8KJ7vanGkHEldiCkHSgo1TbEmUrsTj6tw2ojBvVjkq7wc08lXdegiSmCT3tDXTrXA4grakuJW6HIfv1J57gjJdazWMugJNuErky6YCd2ql9wXL4CUqsbfyhj2eeIMddF+s53UnhTiA+jPQcK65/ssFDorxaOCcC+bmA4MSoF8bdcX+bgX/nMXfnOn4/NO0dmKQ5yZUuKm2URco1xIn6nic+GyKKyOJAOu8kAFUlp27g+BItQDTsfD0KXd4f3RBGB9Qegg/56Rkcnehvj6rVSr+DsqcdBrS0sZJJ94u3aAj15qj5T5qtNp+pY7TzWWJaDdlTvfTTrDiEKl9wxYJNR38/jTdh+mh+jPEbTXi6/yQGc/bXWsJBAkJ8Gm92vYNXZRZdNTDeFfdoHhX7fU5nys2vNktjDuIjhbNgAvWYyv9pi7WTAaFCx+OILg3bQ7c4p/+ZnQwd7CpiBBH/BZdbpvxPoStRRD8Zz2fGK+gFbBjBziRQ1xxmGKU14A4ju9ayY29FiqwCRgsSSvqL/sEa59/zNMfkTk8fZnJpOIRoLuT7yKryAt0LG0wAnEOFPqvAxcLknFMYL0bG1BUcxl2/TAQk6RqzjvZKukaF3Y3LFrRlLvgBjO5OJdkOrTUOSuj0x3uxp8eu5faNIuZXKNgptjt73XQP76d8zgVJk4cLA964ewGpUTl4G0FlGwvvH/2WS/5tWCLoKDDD7L3ghRfd2RwRkeupD14aOBUxNshWLcA+f4Aze0vwKAasV0Y5eWW/aL/pKNa9rOEtmS3AF5Kb5+G0Sjki+RWDdXTLzzbuk+3k5Bcy9qa5036bxshB/e7mEPbj98rHY378LYfssocJ5wCpgdZPBKe7/LS3vIajlAf3MkIRfjcdN mnahSVfT OWhZxQ0WJ9gHLXoAHnvw6FFJJkofYZUId++hevg6e3LcCpHXH8dyaq9vZEtV+giEZPuVhzTiEzcHUIjA9FAmYzPa2bwhynlVvTEUEF+qoXdvxgQEYZNtC0Ozy+FCzZgUfa/HIAb5vyoTBzv5COn4/Fs7yKTGV/dUxTSeUIWtyervxodr4HHmU5f8pBaJModHIm/IrTutHJLkHcdNhQaXskkZ5rlbgORIUH/awZ4Zlyt+5KbJnsblsiIkbZxUj/bIZrC9jz3lZNQy4kTAlgUkx2lrSThU8L7PhSP8pa/v1VnHFay/tKV9fgymlRQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Some task work users implement their own ways to know if a callback is already queued on the current task while fiddling with the callback head internals. Provide instead a consolidated API to serve this very purpose. Signed-off-by: Frederic Weisbecker Reviewed-by: Oleg Nesterov --- include/linux/task_work.h | 12 ++++++++++++ kernel/task_work.c | 11 +++++++++-- 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/include/linux/task_work.h b/include/linux/task_work.h index 0646804860ff..31caf12c1313 100644 --- a/include/linux/task_work.h +++ b/include/linux/task_work.h @@ -5,12 +5,15 @@ #include #include +#define TASK_WORK_DEQUEUED ((void *) -1UL) + typedef void (*task_work_func_t)(struct callback_head *); static inline void init_task_work(struct callback_head *twork, task_work_func_t func) { twork->func = func; + twork->next = TASK_WORK_DEQUEUED; } enum task_work_notify_mode { @@ -26,6 +29,15 @@ static inline bool task_work_pending(struct task_struct *task) return READ_ONCE(task->task_works); } +/* + * Check if a work is queued. Beware: this is inherently racy if the work can + * be queued elsewhere than the current task. + */ +static inline bool task_work_queued(struct callback_head *twork) +{ + return twork->next != TASK_WORK_DEQUEUED; +} + int task_work_add(struct task_struct *task, struct callback_head *twork, enum task_work_notify_mode mode); diff --git a/kernel/task_work.c b/kernel/task_work.c index d1efec571a4a..0d7b04095753 100644 --- a/kernel/task_work.c +++ b/kernel/task_work.c @@ -56,6 +56,8 @@ int task_work_add(struct task_struct *task, struct callback_head *work, { struct callback_head *head; + work->next = TASK_WORK_DEQUEUED; + if (notify == TWA_NMI_CURRENT) { if (WARN_ON_ONCE(task != current)) return -EINVAL; @@ -67,8 +69,10 @@ int task_work_add(struct task_struct *task, struct callback_head *work, head = READ_ONCE(task->task_works); do { - if (unlikely(head == &work_exited)) + if (unlikely(head == &work_exited)) { + work->next = TASK_WORK_DEQUEUED; return -ESRCH; + } work->next = head; } while (!try_cmpxchg(&task->task_works, &head, work)); @@ -129,8 +133,10 @@ task_work_cancel_match(struct task_struct *task, if (!match(work, data)) { pprev = &work->next; work = READ_ONCE(*pprev); - } else if (try_cmpxchg(pprev, &work, work->next)) + } else if (try_cmpxchg(pprev, &work, work->next)) { + work->next = TASK_WORK_DEQUEUED; break; + } } raw_spin_unlock_irqrestore(&task->pi_lock, flags); @@ -224,6 +230,7 @@ void task_work_run(void) do { next = work->next; + work->next = TASK_WORK_DEQUEUED; work->func(work); work = next; cond_resched(); From patchwork Sun Feb 9 22:30:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13967155 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA376C0219B for ; Sun, 9 Feb 2025 22:30:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 647156B008C; Sun, 9 Feb 2025 17:30:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F5E96B008A; Sun, 9 Feb 2025 17:30:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D4056B008C; Sun, 9 Feb 2025 17:30:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 290D16B0089 for ; Sun, 9 Feb 2025 17:30:22 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A0D69B2B4A for ; Sun, 9 Feb 2025 22:30:21 +0000 (UTC) X-FDA: 83101851042.08.856DA8B Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf28.hostedemail.com (Postfix) with ESMTP id 0A8CBC0003 for ; Sun, 9 Feb 2025 22:30:19 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="EwO/kDlD"; spf=pass (imf28.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739140220; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sbVVcl65vVigDl2Jx5EfhHJZQbi3xtIqgRGat45nlfM=; b=i279osf5/Cxb0IIDZRrdUn9DcN9FHskfG0wpKBuchSsYLNagZuFd31HeOf5gKu9ZvVx16r G6LhvVae/n1y8Z6YfvIfWBGprDS5LUTpgDwTmEXa4GNKyZbaIVB1o97GmKVwsTxeujsfkW XVaqOj5GIxFzpCuhyCqLXiA/C0sY55w= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="EwO/kDlD"; spf=pass (imf28.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739140220; a=rsa-sha256; cv=none; b=6/2ltJ4cqG3uWfJCPYGOCd/zb8Tr/AubQP+QJ5H7GKK6dHgVmIjASi8l6AZHnXLkCaXGf5 dAJ/chl72sG/4UZWd1JplXH1cp9DYxGiKI7qRl8shR4c58HoQcN4HFvKUllscPhM0gii0w f7zxWjQTfVwkWnkjqF72UpIdKcan90s= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id A81EF5C5AA8; Sun, 9 Feb 2025 22:29:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 960E4C4CEE5; Sun, 9 Feb 2025 22:30:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739140219; bh=ZSwI4jjw4GwkwAYOaWT3SF4KhLf9zm4bvh5Jlfu1Q2E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EwO/kDlDiWWnwTYOvc8Wn8BZKybBl8uluDwXw+J8N/hg8bPkRKhsgmNiGp96wemhW pK9lDV47kMS4n9Q9dVw7HhKnYg5h6s+YZTZw9Cfo23iUx0wgHGiMeUmOm/kLmSjmCH EtXXDe6Oyuvv8wTXW2OEfMWsrIiCM0JlEpKMRyoc5ERxfiBW/wZQ31kenSphOApFmF 96QgMhFEQuNq0gGlKvCZauQLWQVQZd9rwHiRQdoUwM2sdD+gY+1H+O44p8J7VokeEt jDbHxzNwUMnuFEm9LfXRfs4n5e9xWmozs7DdkyXijWLGxrhjbp0fX0Rax8A5Y9SpFn wYbMkbB7Vm0Bw== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov , linux-mm@kvack.org Subject: [PATCH 2/6 v2] sched/fair: Use task_work_queued() on numa_work Date: Sun, 9 Feb 2025 23:30:00 +0100 Message-ID: <20250209223005.11519-3-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250209223005.11519-1-frederic@kernel.org> References: <20250209223005.11519-1-frederic@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 0A8CBC0003 X-Stat-Signature: ghe4x5jz4cxy5ciybx6rs1tj5ipx6axx X-HE-Tag: 1739140219-961675 X-HE-Meta: U2FsdGVkX18eZeBxq5YEk3mw1Hy6vA413nwS9pwN4ENiOlwbv+WsgdHO8jgVGJM2r6C8TqNyFrLHwp+rbL5QqfJVcElCyydfxRt84IwrSicQXVTMboTdzrX2sN0lIUYQ1vbloVYhJscUGQUqGQbnZu3dDcuhjU+PjSHxtdRmnH2NVAeePZHacLVCAprlFx5Wmz/PJif8cUGlTIUUg2qHeoW4cOiTE9mUFALJOFN9ABQblc/4h58Nh/Q6rMCgwKy0lDYMgRnTpKgSvwP6R/UxHGK+nNYD3PO0i7KK/tbBgHsc8Sv/UBtKrQKc9Utg2Yqo3NfOxqKmaLcAX9s9XT+v5r3Zk6Xn1XUGlXdKBx5aHRIWCZnNhDtiEb7MpzXdAQ7KfF+P0Ts+dhIhfkXT6g23ta/Y4K0Naw5WCf+MqLf4h3mk/98vuX1VYHUeM4Zbr5RtUdWlSn8ycQpfAvCoSk1KdTU0uyol155ZHTV3+DXCwlusHScXS0F5N6TWbpuzPg4V0jkfOnS/BVgN2kmiKsWsdZ2fpeif26uBCa26dPOnsckYubDeI/B8IGsg/T2+oPYzVF2mNT04C13Y27jU22UF+I4x6qj0PRhpO+/JXluEKvw9nnJwcKfDb9lLWkzlZ6FrzsxhTspr6E1i9L/XSRuMDMmIZZkwqboYZVz5DoHsSJQrc7K/u99Sw1lKVWjxkwNG5JJdbJgoy/wSMb3+mlUUpB23hPdD4+YR1TP1+M786Vy2Qd8fGf299ViXXSh68+S6CKlRrSBqca1IBqnyPHW81cwW3x3fjsKto/XPH6fyuICY67BmAkvHgrzqAGm5fJXEG1WUDo5eJ28ZEa3GsQI9hd0GOCUFNlCDCetBIoHtQERkY3rLhhFJ36HSoLmqT2Y3IO1hWQg0JGxqspP79qhAPfLAKlMvukNyvogRhxjPvtnV1OAbmPvmYFFvKIZtWbVgzwtuzBhHnjl+FBkuHpI a9A63bUu cGfztpK0ootl0OQkfZO5DbymJFFfCopDj78ZoLzfRUkqQWzgHGrOR7xemfC7bhj0M2kbJzg4b1KLD0tdGHh1yxoc3O2RmxHGBy8LDDfEF5PEWt34tsIKtQT4YlhX0hfAuMJFJkA0u1jg1PGwbMU7Y51QnCxFUv4RbWL1cbuwP0PUOSdvmAVqdNf5WBbXJX3f+Jvjyvn+NHAQ26pGr5L/j7vNpOWygA4DwfW2KuV7vddh/YsLs0l3ox72BJpdOTAOlV+F7y/x8xPsQ7vk2yUW9WnE4agM4wkGZJjQD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Remove the ad-hoc implementation of task_work_queued(). Signed-off-by: Frederic Weisbecker Reviewed-by: Oleg Nesterov --- kernel/sched/fair.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index ce2e94ccad0c..6a2f45821dc0 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -3303,7 +3303,6 @@ static void task_numa_work(struct callback_head *work) SCHED_WARN_ON(p != container_of(work, struct task_struct, numa_work)); - work->next = work; /* * Who cares about NUMA placement when they're dying. * @@ -3551,8 +3550,6 @@ void init_numa_balancing(unsigned long clone_flags, struct task_struct *p) p->numa_scan_seq = mm ? mm->numa_scan_seq : 0; p->numa_scan_period = sysctl_numa_balancing_scan_delay; p->numa_migrate_retry = 0; - /* Protect against double add, see task_tick_numa and task_numa_work */ - p->numa_work.next = &p->numa_work; p->numa_faults = NULL; p->numa_pages_migrated = 0; p->total_numa_faults = 0; @@ -3593,7 +3590,7 @@ static void task_tick_numa(struct rq *rq, struct task_struct *curr) /* * We don't care about NUMA placement if we don't have memory. */ - if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || work->next != work) + if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || task_work_queued(work)) return; /* From patchwork Sun Feb 9 22:30:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13967156 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7E3DC021A1 for ; Sun, 9 Feb 2025 22:30:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 312E66B0089; Sun, 9 Feb 2025 17:30:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2C4406B008A; Sun, 9 Feb 2025 17:30:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13B026B0092; Sun, 9 Feb 2025 17:30:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E9C5C6B0089 for ; Sun, 9 Feb 2025 17:30:24 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 8BAA01A0F42 for ; Sun, 9 Feb 2025 22:30:24 +0000 (UTC) X-FDA: 83101851168.28.773AAA3 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf29.hostedemail.com (Postfix) with ESMTP id 070F212000F for ; Sun, 9 Feb 2025 22:30:22 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=i3yc1Tcl; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf29.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739140223; a=rsa-sha256; cv=none; b=uuEKsX7kNjR0mIllKPKCbD621H76eQC+ZqQotQMmTuYipjrN5bSPj56+cYBSt1y12yJ8y3 jI9b9aARZLBAiPGFDM1fQH/zrrgnID/Yfk/4n3JhVuw1g6NwO4G2WXQqwoLb8HRtc04uk1 dJGo7lSph9JEC3jLiHSDzp2F8rrQMTs= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=i3yc1Tcl; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf29.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739140223; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=a7uGpIMKlk6nj4vQQW/vGZhG72gVz8uknW/7qvfGUMY=; b=TZtqEQ+PzB4IMcvgwjlHNGNo3vIOC6NuhU93vd53wjEuVx38ayUOeo1bDKkCQF496XT4Pt LH3Fxt+CO2UjMZHPm2C7KzI6hpvGKTHJiZttb71afhdB1lSSqlZfC0426kmwJjNIv9H6p9 DcqoO8wj/41DTlVMVVYZzBDkYtoVKYw= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id B978E5C5A66; Sun, 9 Feb 2025 22:29:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7DBEFC4CEDF; Sun, 9 Feb 2025 22:30:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739140222; bh=FqNJ6UqFojjpII8sBMn0id8eG6MSR2NFISi/uXoXOtU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=i3yc1TclMFUBV7la89jwOBPVVcbWq9j9foLUc+l8S0j49dsWRkglTHczvWZ8vSUbr wD8rxcG6eePf4kB/0zJICVTMcSta23HMamwJHE6fo8Fp0v0A/ZkcOUtU9XTzHwlNZy +Gyep4bt1xDGZTNwnAkA2n28MRQnXc6BcSBuVaqslF+JGVWoaypjRynftsYAbPdiKf C7J0qpFDBdeJpw12H1SK8BUhLmUyc5msm80v3jZ7Ryi0H+ejgxjDYhgQGw+HwxKjQo x7TmecEPgKUGc1ZAqYnJ6g+Y9xrKsHJG2JEsdQx1RPMLftIKiFUFxpU+TT64Gd/iyV 6KMEvGDKLmi5Q== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov , linux-mm@kvack.org Subject: [PATCH 3/6 v2] sched: Use task_work_queued() on cid_work Date: Sun, 9 Feb 2025 23:30:01 +0100 Message-ID: <20250209223005.11519-4-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250209223005.11519-1-frederic@kernel.org> References: <20250209223005.11519-1-frederic@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 070F212000F X-Stat-Signature: zmdi6cgwf1gqbmdsw7hjbjkyxix8zjyr X-Rspam-User: X-HE-Tag: 1739140222-325244 X-HE-Meta: U2FsdGVkX1+GeZ5SmfbG0PPW2so0w5W8G4NRDMYs8upNUgkgkp5qR6XqqPeNIqN8n/IMB6h/xdPSlI7oHwIrfgCBkIUabX617gXFgqZvaPndohbBbpqvBF6GtyJrPbqJBuDO7vMo8hTy6Jed7ZxfHnENhKBEOMziVYKjg6qxY2o5RJbmoBkKTh8CLyfof9uWZRgqAHj0h1dlSHS5u2OqU9bOm75G6bpSkoOIrepUMLaos0eMUU2K6khKP66lRm1w+MBeK4qATKQ6tk0JIdKQsclqiuiAl3JcqLcepWUEmXU2G0PV55EIkS36NjAl2voSkAKXbaJaLn7oFCCraK20XJ/Mb8iBzC3/OQt8dWBLZKxeCsK01yEH4vRgvwivcMiJL1NIS38M9Gh9shI0jHy6piR0eco+8SvtWwglf4jM7495RgR7XVCGKeVzrDvN2M61chaS3FWz7KmKiwJyOpf6POL14vM0kjQJ2H8kQfX45JHR3D3rC0ycxCU0e2qXWvKpzX0OuoF3MIWJq/1vXvWcTZzozdPtjW5ltvNSIGPhLPbdfcQ7i/OHAx58LvoAdPPvgOckmdhbi13Sku+887sNDY8COkFA4eb6uTlVwk8jMo6ZI5TL5i+/5ZVQrolwOHCplmnjTytCOKkBncXZ8N2t/3szjO53hu2OKl6S0tuqMIldcdHDZKnWkTySt+FdjZJqmlGFXSwABo9s8DcmVfSqftP0ByWejxlBSeRY5vL7zdR+h7OsNSvg4xaG1ZO8NqWF0uzPbjLNWw8KZTNHbIweRosu621mgVw7vHa73eZYlDXB/GHd+KZwqtFFCwSDJ7fKgvDJl2xOrQD/qy0nZdHW+TM3ZZyCsPjDiRjv4OMsmHR13+eo92ExCIG7qUMFb+0vMdANlUgDsLtU/vsQGCyoCF9yap06neXW3LhFuCrH3imrPgRRyJCav5TDKflRrDsIRDBLr11tSYE1+gvRrQG S2lT662N 83j2FnpCaF9YK2GNpSFqXVODRRxOvm7pnesqt0ntTHw5a6bxMHxXqaAJZdRyjI2aVK/tLzoAw/260Vm3EQxjSSqnZJWSa5BiNF3QEGZkvN+T7L1XqZije9lI9GY+6DtfiuBKjgpa6dbZJUU/cXcBkBRmDTlU6q2f7Len84W4qmNhojMgeQVVR4++Q+zyVujmcvddoziTWt9EJM9DjuXibwUjU0KHWaCrz9gthR+XNiP7Ynw7PCxWJs+qVnbyIBci8uZGg9Y/jhpOlTS4I2ttKtnOOcN3kYL3UdYxc3EwtN/Kd2UZgYNfTQIHZyA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Remove the ad-hoc implementation of task_work_queued() Reviewed-by: Valentin Schneider Signed-off-by: Frederic Weisbecker Reviewed-by: Oleg Nesterov --- kernel/sched/core.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 165c90ba64ea..606f596a6e0d 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -10538,7 +10538,6 @@ static void task_mm_cid_work(struct callback_head *work) SCHED_WARN_ON(t != container_of(work, struct task_struct, cid_work)); - work->next = work; /* Prevent double-add */ if (t->flags & PF_EXITING) return; mm = t->mm; @@ -10582,7 +10581,6 @@ void init_sched_mm_cid(struct task_struct *t) if (mm_users == 1) mm->mm_cid_next_scan = jiffies + msecs_to_jiffies(MM_CID_SCAN_DELAY); } - t->cid_work.next = &t->cid_work; /* Protect against double add */ init_task_work(&t->cid_work, task_mm_cid_work); } @@ -10591,8 +10589,7 @@ void task_tick_mm_cid(struct rq *rq, struct task_struct *curr) struct callback_head *work = &curr->cid_work; unsigned long now = jiffies; - if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || - work->next != work) + if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || task_work_queued(work)) return; if (time_before(now, READ_ONCE(curr->mm->mm_cid_next_scan))) return; From patchwork Sun Feb 9 22:30:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13967157 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D39E1C0219D for ; Sun, 9 Feb 2025 22:30:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 66E19280001; Sun, 9 Feb 2025 17:30:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 61F2B6B0095; Sun, 9 Feb 2025 17:30:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 470DD280001; Sun, 9 Feb 2025 17:30:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2BF216B0092 for ; Sun, 9 Feb 2025 17:30:28 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A331D1A0F83 for ; Sun, 9 Feb 2025 22:30:27 +0000 (UTC) X-FDA: 83101851294.26.1D9F2E1 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf24.hostedemail.com (Postfix) with ESMTP id 251A9180007 for ; Sun, 9 Feb 2025 22:30:26 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="GZIidSr/"; spf=pass (imf24.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739140226; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0GRnoQstWRF5YH6CvkAyI7a+IxCpygSOTbnLHR+rnTs=; b=cd/fwjcM3w1Vvlo/fLBnwdQN2Bjf+VXPSbkaxHEPvWjy/hVpXLQTRAMFU08RE2/8lp3l/e xYFV40pIIlZFRK0Jz+xxr94x3o6D0DFOz7W2fl5a36e76dXlF552DucuriSgGOhmcqu728 A1UPtxAMKKBjojbeBnOgSfgfKCEE1+E= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739140226; a=rsa-sha256; cv=none; b=Uk7asyz9qMwCQW5nLejGuwg4gMmqQW6vegkU/Yh2V67JXFKQKCmUoh8qVw42S+BHdb52om u0cvFuAVRnFSj1l+HqXsb8OanIX5TXdU5wl2rsoIpyN3WxM/NqtjhwV7lrsgF7MDtxYwIU xOqRByJ0AdSPdunck+0nm49d7O8Wx/A= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="GZIidSr/"; spf=pass (imf24.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id D09465C5AFF; Sun, 9 Feb 2025 22:29:45 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8DE7BC4CEDD; Sun, 9 Feb 2025 22:30:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739140225; bh=TXaR3PXscB0IR2zu2ykA6NDQs55O4UuGlAU5yWSXoiE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GZIidSr/P6waR4fDq5bCpnMDJ/9CJnNUWMNJm2VsqNQoj5A+V+IecPyfvvCnFJedQ uWCQySMec2pbxlzt6O3nszier5u2Q+MhW9zH3TeiMwbTiCDWRfw/XwHw3bNvSV07nT jv1IUqSyzNgavbpRXwQVNK3kpXQJlEM2E1O6k7UsmMzeAXDxEdDooxhJtVZdCBp5e3 75BhwkH7VaYwBbSZneZEvagRZTWxuJ1rgjsyXEWTEDxRKmBNuM+GfxL+ON9wYrEroQ jrp4RdT8Zf3ab8qNiYvPZRVUQ/ghWDBWrSO4qEyw8aF8L/Nf9r2hdzQlcn7LgoI4Ih YxlqbBf8xueMw== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov , linux-mm@kvack.org Subject: [PATCH 4/6 v2] tick/nohz: Move nohz_full related fields out of hot task struct's places Date: Sun, 9 Feb 2025 23:30:02 +0100 Message-ID: <20250209223005.11519-5-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250209223005.11519-1-frederic@kernel.org> References: <20250209223005.11519-1-frederic@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 251A9180007 X-Rspamd-Server: rspam07 X-Stat-Signature: mkhbx5fj49ib59fpwcepbuhjk9rdxn85 X-HE-Tag: 1739140226-662757 X-HE-Meta: U2FsdGVkX18otl/xrwfW6rM6IuYuQ2OlZXt547CovNFs9ZGpbC4i8bKR6OqUmkvixe2ntZNQWG/+/TPLYpCEzBbQz9x1Az49fNxUuRifp1HezjU/v9jPKRZS5dmbYk3rYJx7jHTqI2S4B6I1zL8mPFzvVLuKGfjI0RLv9qrFKjZFE1T2FmLyw2SgIwQ/lK4qmFimzNHJaVz8vyf0pTLcCoIX+XLDZwoz8qWa2/5OOj+v3MpsbK/UM5jDqdkjHSFwq5AUxHkpR0tgQh/dm0/Mxy6XCEW3+8JvD/+XiWZXixGCJ+gtD4YDfOd6IWZDM+0MfjBeDk0VtD838sWkBQpYEAd4SG6pWq/93ONxxhjhqeHRHclHCQ2RSmMPr/ng3j+WD3TqgM3r8keK7d2oEwHvkgcxNfnIaJiADiMg8ccsv9Rth+/SNT8JwX533dEs31zmIfq/R2NBFFcAYmBYvnzYqsX5YYIvBWIp89BuNcmIqelW66gbz3EentwkvOoxSokE8X/XEZbEjaij4WV7NOGbeD6BoCEX+G6Na+Zxiwj3D5OezXAkHm9yfWxPYQsUblvu9hRQ3yIxkJ5ZqRP5rrgWCO6ad9UNvRAywJ32nHW9kKDglHAvumAqFw0aLeG918urq6zXlWtZRLt0NQeTkQH/xrQMaywYSipvI7nfTCkG7DiFDgmzL9naYwWZJPVlwjGXH0TirndgHEbwIyzOU2BE3AzH0Zix1aCxq5L6CpekXwS3gp0iD/7rsAgMm2i2gObJ81l4j3csZCdy+YAmS2BJasDg5W1dfLYcMIUilQQG3X7el+pUZvfq3grQVX2u0fpxmTEUOGmLQX4YWFemevJQK+NZKGDcQ78rv2oGRdkUSr6Wn/Un8GZM1a1TKyfBCKdEu9G08k0USTz3HFAx0lqFR87C+JWCcwucaK5P5xBPF4d7lbiexl43Pf6bJYEUwg5OoxM3I+LkeGh+xDbf0Gw vGakI8E+ fYwKx3DjpkMKi9kgJbAY9W8mqUx+moQfxLPXTQ6FegNnmul+yj6Vr/OohgEXhriYvnUg4ey4xM6hwE6gBu+EfJnx/IaLobsnupz1z40YIq9GabnUeAGkvoSldlApXYyd8vEiHvY4oxTDmHjs7NpXOkxzh3cbAQWKNJNj+T2TAFHzRCcaMR5S1V1EsdEIHj3I6/uQLn7d639x9cSVbfDXvbJuPNkcFjk6bKJfdgjCxwbEpt0L54lKGHazhsRQGZlYInLJ8rBi2Nz4DDHgSUP8Salp0u+PW/O/4qaZR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: nohz_full is a feature that only fits into rare and very corner cases. Yet distros enable it by default and therefore the related fields are always reserved in the task struct. Those task fields are stored in the middle of cacheline hot places such as cputime accounting and context switch counting, which doesn't make any sense for a feature that is disabled most of the time. Move the nohz_full storage to colder places. Signed-off-by: Frederic Weisbecker --- include/linux/sched.h | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 9632e3318e0d..10a9aa41b43a 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1088,13 +1088,7 @@ struct task_struct { #endif u64 gtime; struct prev_cputime prev_cputime; -#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN - struct vtime vtime; -#endif -#ifdef CONFIG_NO_HZ_FULL - atomic_t tick_dep_mask; -#endif /* Context switch counts: */ unsigned long nvcsw; unsigned long nivcsw; @@ -1411,6 +1405,14 @@ struct task_struct { struct task_delay_info *delays; #endif +#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN + struct vtime vtime; +#endif + +#ifdef CONFIG_NO_HZ_FULL + atomic_t tick_dep_mask; +#endif + #ifdef CONFIG_FAULT_INJECTION int make_it_fail; unsigned int fail_nth; From patchwork Sun Feb 9 22:30:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13967158 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DF45C0219B for ; Sun, 9 Feb 2025 22:30:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 891D0280002; Sun, 9 Feb 2025 17:30:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 841206B0096; Sun, 9 Feb 2025 17:30:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6BCA8280002; Sun, 9 Feb 2025 17:30:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4A9C16B0095 for ; Sun, 9 Feb 2025 17:30:31 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id F0159140B98 for ; Sun, 9 Feb 2025 22:30:30 +0000 (UTC) X-FDA: 83101851420.27.7AA8D3B Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf20.hostedemail.com (Postfix) with ESMTP id 4D5DF1C0013 for ; Sun, 9 Feb 2025 22:30:29 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=DZCOzR8n; spf=pass (imf20.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739140229; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=speXJxf6MRThMCo7hPIqARwtF16DCSTlU2Q9hXtFiPY=; b=lC7KqvK8aSsMDA/oAHvcXxsjxRceRSwfHRNIdatPqdoWFDxADuRcdYTxlYV4HA42ZVxt6K EqduWjkvDyi3hTPz6PJ0V8DT8NYIjj9G60ctAs/NY/Qq+sfMNkMwOXmK4bkg2xquZMLiPN AgVdR6dqLJJcmidlKpctGXLu0Zz1Z2o= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=DZCOzR8n; spf=pass (imf20.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739140229; a=rsa-sha256; cv=none; b=YgqCr3ia1hYXTKPKQSjeOOKtTA0AN9wcKPzy6yJ3yVOwniFIItgCTNF5Dh4jR5dXcwIBlX +D2jaEv9D+3HFRe/t2uA1zBbYtIpHjdJDRB+rumUm9PBuG4vX/f/tmdnqb1KX5GGlYi67Q xj8JUWC6KmiekGJ59Xb1Pujiw2djQno= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id E76785C353D; Sun, 9 Feb 2025 22:29:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 99927C4CEE7; Sun, 9 Feb 2025 22:30:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739140228; bh=hpX5Isxts6HZwFzDaFwdkHMytRYO5OiHbV2gKBTOr1s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DZCOzR8nKDpcLHHG+pUeg6XDkn23fPlzAQdzNGIId4ORHo4fyoa7vHVWO+9GkTzkQ rOAL8y9B5yI2fOD/bhpjqte9Ww4uACLQTLAGJ7r1DQtJK++Vu0v3ysxhrN0ae+AgmQ YOyoQmsrC1+06KU7jI2g+OOWGuThvARFCN82GCvxhURurB94W68F854JlfHd+gjI/a 0/Stpzb4hrHV3Z5A+rddkQ5H4Tgmv46ELAZEuc240GNCyulpPTbhMuHJ6crLPOE0lC nHBhpPNlvMIz9X+ja6w5uCnxjlAC7wDmN5gdKhX0MbzYNyOsJ0RJfSxRqIBmfDVKDm lGvlFEzyz5cCw== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov , linux-mm@kvack.org Subject: [PATCH 5/6 v2] sched/isolation: Introduce isolated task work Date: Sun, 9 Feb 2025 23:30:03 +0100 Message-ID: <20250209223005.11519-6-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250209223005.11519-1-frederic@kernel.org> References: <20250209223005.11519-1-frederic@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 4D5DF1C0013 X-Stat-Signature: ps8oyuja6auza8swcwpmbf8eebdiucz5 X-HE-Tag: 1739140229-859382 X-HE-Meta: U2FsdGVkX1+Od0sb7mnX7JvMUXeheyz6iAE92Rt4QD/LUtNXlPtLFZQn5cc7KnJxbdIFwVrAWD0CUm/OdH/xM5im9LPe2JlGQ7OegmL8qLLFCf3P9hKxH+v3OIK0BoKTWpYEGZ3mb7+Zu1ZP7DCf2LZv1VrWdaJ7hOMI3nDSxbHUQF8aA349HnZ9OZNY4XjEZ3E2yGdWyrLdqU4f4dXt+pQaETSOh429J4FMMY+fhpMWDtc4KUoycR6j+++VCvS74lHrO/zi443FSmr5bFzgmj1gWPLw/cFHd6L3ZIimwEqtrR103L2gj1YgWeJe9hYVnMHkIPa0FdUvyG3DYHpWBvNwUPe7fjRyrqiikWYvmCE/7iptoF98YiWPcfX14TW3IWVHCtLAzoUkelL1lNpPfQRy8MufQMXdQOJIvRMBK0f9govUTZ2zUYYbsRmlLVqzlw2XxKMh2WZPzpG/3g3ROPELR/fw7xmLXSHGrGyHV/Hn33M/7hI8u3QNqe2bzX/VAsjdRYglCvU3Et7XfTtWat8X8Wbbq9wC2htYpFcs07wnGADfb97+0nxECZhvoKVbNkY9N2WKUnGtgH4dmkGknunBNAcQTZ8IIGQyo2qCahFy80RFtSlIMyS9NLQ7uGMGIalSA+7np74dTMz8iVrIO9WpV6uEzMfJ8YQ7ZZ82lwg71re5/VNd5LL6WnjIXGA11DOIk90BZ3ktUfdk35juVrzZhoz3dSM9flNd3vYcivqAvKBh4L4Iqs8FCtM3PBRBBl7ujSY3xffy4AcypEUlS/OJ99WcDGdX9JBfJaYwLK9UhSxe6Xmw/B+nPhHEA6pWs+eVMLMBQBf5nMQVpyoQ3kIPLnfcYS1i4iVPrV+i8lKX+Pqu8kJKWn9SToVhfYEuVDnh+gDNoJ6s3JxiTr1VOXdfAaUgZAyYBdFczGEqKqR4Vs7867GawOkrUkMMmXTJWoXyePmwN7ig1NCT+of DVQxTMLn okiVz/aDefXGduoxcK/2/hrlPzX+hs2gNmwpHYsgenlU3HbQ7usguZ5e+Z+S8KRKml+tN/8xuUOnqUnzgCD8OGyaLsJsbu2qE42qRZWB/2uqt7dpEV5S1jO2Avki0kcgYlB3IxU7lOHT77jW2iH7D7oS6Kq9BYGdYFH2nn7vHp2n2rr02S75HDrj0mGFNdj6SUyvbPHKB+5QPXPC2ZrROl6C3WGV7YdewjWCnCMi186XcAYY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Some asynchronous kernel work may be pending upon resume to userspace and execute later on. On isolated workload this becomes problematic once the process is done with preparatory work involving syscalls and wants to run in userspace without being interrupted. Provide an infrastructure to queue a work to be executed from the current isolated task context right before resuming to userspace. This goes with the assumption that isolated tasks are pinned to a single nohz_full CPU. Signed-off-by: Frederic Weisbecker --- include/linux/sched.h | 1 + include/linux/sched/isolation.h | 17 +++++++++++++++++ kernel/sched/core.c | 1 + kernel/sched/isolation.c | 31 +++++++++++++++++++++++++++++++ kernel/sched/sched.h | 1 + 5 files changed, 51 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index 10a9aa41b43a..82827f962745 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1410,6 +1410,7 @@ struct task_struct { #endif #ifdef CONFIG_NO_HZ_FULL + struct callback_head nohz_full_work; atomic_t tick_dep_mask; #endif diff --git a/include/linux/sched/isolation.h b/include/linux/sched/isolation.h index d8501f4709b5..74da4324b984 100644 --- a/include/linux/sched/isolation.h +++ b/include/linux/sched/isolation.h @@ -77,4 +77,21 @@ static inline bool cpu_is_isolated(int cpu) cpuset_cpu_is_isolated(cpu); } +#if defined(CONFIG_NO_HZ_FULL) +extern int __isolated_task_work_queue(void); + +static inline int isolated_task_work_queue(void) +{ + if (!housekeeping_cpu(raw_smp_processor_id(), HK_TYPE_KERNEL_NOISE)) + return -ENOTSUPP; + + return __isolated_task_work_queue(); +} + +extern void isolated_task_work_init(struct task_struct *tsk); +#else +static inline int isolated_task_work_queue(void) { return -ENOTSUPP; } +static inline void isolated_task_work_init(struct task_struct *tsk) { } +#endif /* CONFIG_NO_HZ_FULL */ + #endif /* _LINUX_SCHED_ISOLATION_H */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 606f596a6e0d..78b4b996f85d 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4525,6 +4525,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->migration_pending = NULL; #endif init_sched_mm_cid(p); + isolated_task_work_init(p); } DEFINE_STATIC_KEY_FALSE(sched_numa_balancing); diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c index 81bc8b329ef1..f25a5cb33c0d 100644 --- a/kernel/sched/isolation.c +++ b/kernel/sched/isolation.c @@ -249,3 +249,34 @@ static int __init housekeeping_isolcpus_setup(char *str) return housekeeping_setup(str, flags); } __setup("isolcpus=", housekeeping_isolcpus_setup); + +#if defined(CONFIG_NO_HZ_FULL) +static void isolated_task_work(struct callback_head *head) +{ +} + +int __isolated_task_work_queue(void) +{ + unsigned long flags; + int ret; + + if (current->flags & PF_KTHREAD) + return 0; + + local_irq_save(flags); + if (task_work_queued(¤t->nohz_full_work)) { + ret = 0; + goto out; + } + + ret = task_work_add(current, ¤t->nohz_full_work, TWA_RESUME); +out: + local_irq_restore(flags); + return ret; +} + +void isolated_task_work_init(struct task_struct *tsk) +{ + init_task_work(&tsk->nohz_full_work, isolated_task_work); +} +#endif /* CONFIG_NO_HZ_FULL */ diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 38e0e323dda2..f80dc1cad219 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -60,6 +60,7 @@ #include #include #include +#include #include #include #include From patchwork Sun Feb 9 22:30:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13967159 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9363FC021A1 for ; Sun, 9 Feb 2025 22:30:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2500C280003; Sun, 9 Feb 2025 17:30:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D8626B0096; Sun, 9 Feb 2025 17:30:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F202C280003; Sun, 9 Feb 2025 17:30:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D0E306B0095 for ; Sun, 9 Feb 2025 17:30:33 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 881E4120D1F for ; Sun, 9 Feb 2025 22:30:33 +0000 (UTC) X-FDA: 83101851546.19.EB9707F Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf07.hostedemail.com (Postfix) with ESMTP id EB83340006 for ; Sun, 9 Feb 2025 22:30:31 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=OiznD3kz; spf=pass (imf07.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739140232; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=b1s6u44EkwbMOEY9WgoVy3zZsrG3py7NrPuyt60qlMg=; b=2GvaickNu1Dff2t6W2FaubmXNhX/dmZSu1hJcDPsFz6Apk6GgSv1+8lJXEe6s5vEK9Enxh IqS0DmfQP+yZK+YmkPOmdCmkGChEdIfTVm7uOSPYbxxWEsXsZSe1yGd+n7OdyU0wVPQ40C 8CL7WY+f8pjkZAdQ2ohzP+EHLkkXXGU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739140232; a=rsa-sha256; cv=none; b=Wo4UTGnm6M0mO3BKkaLOo2QhsjIBT5KeL6y5pU5ZuX6ZIEk0HyGnHf8DfQumg/xREjhNt8 uO1qLhcRO/KIMjTt8279EPOtZpdMVY0gNX1K5+BlznHyj6xBQgZJCcDJZvzmdO4e3A9B+Q 1lBHZ2IUCUcLbOXKLeATLaUif4gLMdo= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=OiznD3kz; spf=pass (imf07.hostedemail.com: domain of frederic@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=frederic@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 9B2FF5C5978; Sun, 9 Feb 2025 22:29:51 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ADF1BC4CEDD; Sun, 9 Feb 2025 22:30:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739140231; bh=jJC9e1iQUws78AISrOzytLZgqafLyxgOcIGGwKvAOvw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OiznD3kzJiFVXddt6RYzafH2Caz7uSr1dddZCrIu/u3/8mczDP7ZEKGEDjOVW/9yB btHfhsaIgyvr6xWqi13Em7VXQZrpIGjktay/irh2ACH0rVTRyVFuM/cwBj9P3YzoS2 HMYFygOMnWMB+fi00pTr3JpElh/O2dNQg8UHryyWUEt5erxMlSLx3vAWejp1Gl10L3 xA5YNFPcDwqlP5fBFm2Bie4loz+E0Sn1Zn5Yqu8KeIyjGU8ahnvAufLyvVx2ooEKYm kmKPw2Hf1FhX5g2//fFphPUfZoIcVTp+CopdTmBHa2lkSa3i/yegVGmp+lerJO4ZiS B3jytGQj0Lgew== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov , linux-mm@kvack.org Subject: [PATCH 6/6 v2] mm: Drain LRUs upon resume to userspace on nohz_full CPUs Date: Sun, 9 Feb 2025 23:30:04 +0100 Message-ID: <20250209223005.11519-7-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250209223005.11519-1-frederic@kernel.org> References: <20250209223005.11519-1-frederic@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: EB83340006 X-Rspamd-Server: rspam07 X-Stat-Signature: nt335gk6gagkzh6gbinhrbey33g455ko X-HE-Tag: 1739140231-718244 X-HE-Meta: U2FsdGVkX1+uIBOlYGl7I7itZL7piOd/Wk58OfNgU1QJZbFQGCd8tTplo7R+0LMg/VglKXsXMut2TrUVP1N1zaSYbyyuQ+o/vPdrlOAcXftVsV3k8HzsrUpg6RnPmaBZGOyhhvJBiToouDQkpXpXcovUwdip8Dq1Ihvj/K/nTRRuACZQw0K2eLyQVKYgaqaTNUjr+8aCuGqccPgx6ttqTMegZ8kz7qVrMqP1SUd5v1+UuujHRLJIH/Y88axj5H9r3A3Yks982RmxUW3EX6665Bb4SZKi/uYvUd64SIdYTsSi4xzaARO0B6KfBfYaUQAlFhVxdZ8OfZTigLG3e9CN+f/uAllSENFJwekJNc80EQc8ysGjFnwL/JbCybdOEwBguRwMUnGp2+aPpxCsQRKD3R98jN9KU6PIkbfI5YWhEXvY/PYHO1FcNaj6AvscOTH1s7tZnBSW8f0gC7Rf2u51UFWfdmo4Bd7xrkyHGfhs+85XUoU6Y31IfCPK40eE4novCcgIoV/qsPkORPnnR4RuEhgfQOc1ZpYPhIglcXjv6PVFO+EV/HhE5+Kl//tkZBBhB9s5BjEFqUwUZCj/aXBN7aZCy5rCMeRPUTHydvbuyVpqTZcmVIF//6se2SwWeSXGd4+4tLM+B0/Padf9SrxvXWtLtYXPfbfmZQP5Ja4OlVp+XZ2TCHOIdfnsnkGpWSYiD/N/KodMqylsce1vFiK1ZULGPXMKwuLo4GzcDF4ed76BXiewCCugIqQR6WIbmjZe1RrHD2jyD2KR12rnSw7ABlDLE8ftZaeFQpC5uYSUscAmxi+OOwWH0E85KbCm14Ss9nppdiUVeZJTDeQo4Yp1aZbMG1AZxvhc4nM+JkfACurBb7wQ9jYhzRxLuDh/PiwMwVGD7XZbVmXdzw5g+Kci9JhyGJtpI+/cl/kRsAVnv7wEB+CCPnh2ayEUAXl8OKP4e+zhiY+YWm0bZIYRIDq b7XDNjTw SJmlNgOCzodP0wy1R+wDgewBWVzpXO8LHWV/jbIq1HdmOzhi+WtKQawLWlAZii/4++MYy1h91ERDEDFdpqxFtmFShfq7IMGn8HIzQFgdZ7h/2hjqLQcuWjDeBP6RZmNuera9vJO6jsJKn65f3Z55n5AL71Et/GjO3v3AAF16slGb3+2shpSikqL9NvtK/RdziGOS9Xpzs7toknF60taEQomNIthAbt+DZv2TzaUJO0KQ3tkbCvIg4YDM0OqInUy1wiHfefULppSUIKu5EttWWafM8NpkJhTJDxVPWqBjmAPGL9PBPrhByIgjIFn8HbDW3lTUu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: LRUs can be drained through several ways. One of them may add disturbances to isolated workloads while queuing a work at any time to any target, whether running in nohz_full mode or not. Prevent from that on isolated tasks with defering LRUs drains upon resuming to userspace using the isolated task work framework. Signed-off-by: Frederic Weisbecker --- include/linux/swap.h | 1 + kernel/sched/isolation.c | 3 +++ mm/swap.c | 8 +++++++- 3 files changed, 11 insertions(+), 1 deletion(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index b13b72645db3..a6fdcc04403e 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -406,6 +406,7 @@ extern void lru_add_drain(void); extern void lru_add_drain_cpu(int cpu); extern void lru_add_drain_cpu_zone(struct zone *zone); extern void lru_add_drain_all(void); +extern void lru_add_and_bh_lrus_drain(void); void folio_deactivate(struct folio *folio); void folio_mark_lazyfree(struct folio *folio); extern void swap_setup(void); diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c index f25a5cb33c0d..1f9ec201864c 100644 --- a/kernel/sched/isolation.c +++ b/kernel/sched/isolation.c @@ -8,6 +8,8 @@ * */ +#include + enum hk_flags { HK_FLAG_DOMAIN = BIT(HK_TYPE_DOMAIN), HK_FLAG_MANAGED_IRQ = BIT(HK_TYPE_MANAGED_IRQ), @@ -253,6 +255,7 @@ __setup("isolcpus=", housekeeping_isolcpus_setup); #if defined(CONFIG_NO_HZ_FULL) static void isolated_task_work(struct callback_head *head) { + lru_add_and_bh_lrus_drain(); } int __isolated_task_work_queue(void) diff --git a/mm/swap.c b/mm/swap.c index fc8281ef4241..da1e569ee3ce 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -37,6 +37,7 @@ #include #include #include +#include #include "internal.h" @@ -376,6 +377,8 @@ static void __lru_cache_activate_folio(struct folio *folio) } local_unlock(&cpu_fbatches.lock); + + isolated_task_work_queue(); } #ifdef CONFIG_LRU_GEN @@ -738,7 +741,7 @@ void lru_add_drain(void) * the same cpu. It shouldn't be a problem in !SMP case since * the core is only one and the locks will disable preemption. */ -static void lru_add_and_bh_lrus_drain(void) +void lru_add_and_bh_lrus_drain(void) { local_lock(&cpu_fbatches.lock); lru_add_drain_cpu(smp_processor_id()); @@ -769,6 +772,9 @@ static bool cpu_needs_drain(unsigned int cpu) { struct cpu_fbatches *fbatches = &per_cpu(cpu_fbatches, cpu); + if (!housekeeping_cpu(cpu, HK_TYPE_KERNEL_NOISE)) + return false; + /* Check these in order of likelihood that they're not zero */ return folio_batch_count(&fbatches->lru_add) || folio_batch_count(&fbatches->lru_move_tail) ||