From patchwork Mon Apr 15 15:21:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Desnoyers X-Patchwork-Id: 13630219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8561C00A94 for ; Mon, 15 Apr 2024 15:21:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5FEAE6B009F; Mon, 15 Apr 2024 11:21:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 539DE6B00A0; Mon, 15 Apr 2024 11:21:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B36A6B00A1; Mon, 15 Apr 2024 11:21:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 20AD66B009F for ; Mon, 15 Apr 2024 11:21:22 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E5CB94063C for ; Mon, 15 Apr 2024 15:21:21 +0000 (UTC) X-FDA: 82012129962.01.B7140A7 Received: from smtpout.efficios.com (smtpout.efficios.com [167.114.26.122]) by imf21.hostedemail.com (Postfix) with ESMTP id 56DA71C000A for ; Mon, 15 Apr 2024 15:21:19 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b=YIpnWYr3; spf=pass (imf21.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 167.114.26.122 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com; dmarc=pass (policy=none) header.from=efficios.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713194479; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jqT0VPYbD9D0oVnT2Ukuebptjv54pkYlVzbpc73bxpQ=; b=kzv+JifbHMmPXKLc9iViT7v4yPo+VWZpDUVtyMzdMtLjtlwmpO66QjGSJYNuk3YkuFUtsa jXwOsl5hLOhPQxtFAOV5iDtm8Z49pQreqvHpuH7mfhciqqKTB0uK5LVISLCoygvRl8Q5oe 3JzFVXFR7F7Ox2AI80S/bCligg1mUfc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713194479; a=rsa-sha256; cv=none; b=oKApfwWjfAJKJGfSOC+JIVgY01aeO2QZnvMqy+90eLXH39J67EOtdBi/asi76wL8gpSsjy 4wylCbrkkaV0TzgXfpBb3wsYd6tRojUoX2srW3kN4FceGPqR/bPbAVKEvuoje1hnSB0/w5 VM+dsd7XQBcepr1wVlFy8jNITJIDnS0= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b=YIpnWYr3; spf=pass (imf21.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 167.114.26.122 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com; dmarc=pass (policy=none) header.from=efficios.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1713194478; bh=osgSadJZvQn1wrb//TYw1z/uIXekqzai4TYartnsTww=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YIpnWYr3OBNaVcOtObbIoF+WFBDFoEQ/+FXjlJGb5vZB+HrgGpmxY8qVUM3yt7rbK pqGEs1MaSgpWZeTEHe7l9yu17qAlGcj5gxBTDhZZ4LLh/htXEt0TMDTv+04thWG6KA e/rZHM8+gRKKta041QgyUJBZ6PCTJyuRk0rI3Kscx8JaOGTMnlVyCCYLAEcsjbY+8Z PXJtu1qOkAqYXDYv+HAUIA+d+DjAeGzD+fSsD6slc5Lqnsq+fP0NRGuMC6qvhllBEx kFI0glyI2mjasoAdAkLHJ5EbBqgqvHT3QtrW/rDNII+MhcDO03FlLwq8JNwPDHaFpp K1ISv5JuAFfPg== Received: from thinkos.internal.efficios.com (192-222-143-198.qc.cable.ebox.net [192.222.143.198]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4VJ9qf0ksTzvSD; Mon, 15 Apr 2024 11:21:18 -0400 (EDT) From: Mathieu Desnoyers To: Ingo Molnar , Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers , Steven Rostedt , Vincent Guittot , Juri Lelli , Dietmar Eggemann , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , "levi . yun" , Catalin Marinas , Mark Rutland , Will Deacon , Aaron Lu , Thomas Gleixner , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Arnd Bergmann , Andrew Morton , linux-arch@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org, stable@vger.kernel.org Subject: [PATCH 1/2] sched: Add missing memory barrier in switch_mm_cid Date: Mon, 15 Apr 2024 11:21:13 -0400 Message-Id: <20240415152114.59122-2-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240415152114.59122-1-mathieu.desnoyers@efficios.com> References: <20240415152114.59122-1-mathieu.desnoyers@efficios.com> MIME-Version: 1.0 X-Stat-Signature: feen9586equyrfsw68mmhcahekrdwqae X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 56DA71C000A X-Rspam-User: X-HE-Tag: 1713194479-136159 X-HE-Meta: U2FsdGVkX1+3AwA23SyXSzTRApXtEllRhBmC4qxmbfN4r4c35VTwCRi9MkFAMtBzaFsk9XHxOx4bxXoTY+RhFWcmxcD3Ueb+VS9TWpinY0njG7tGijPQGulIOO0+SqttP+9eexM4b/27NE/Pa9qsz+OytN+XwDXedJZAlk4R9QJ/YSuO2rQGBAki+J25zUGOHKH+IDl3lxiXkKdHK1AY+NnqIaav8U/QV6xsrP5CspgCOI0SbGEO5/E06YEivZdT16zAO9Y2Xfvqz5G2U5++W6WfUbyRI0miDHdFsYawrZ34nH6w8IXyJCvuujuxefnKzV+G2TPFiiZJib+y/BG6XmfEezuIX4WvG9CxVnoL6cjrLztZy1hWciFccdBaqu6loJUTvUjOTY65Qy7oWxR6PrGgTAqmgh5OG3Xgi5NPIMs9eqU4xodCCIFZN33+jmTfJNk39WtCzm5npzSeOqOZNtWlepixHo4YVLDoUEuLLm9XdY68BWpkzOGeHLnMRkquCqaRR7WARYHSpEiZq4LsqXsqgm3i1hkWIZ8X+tqt++UZQWIa/HdM5mMWjrCddhhGRwfU4NVd0JMZQWgX32VJD1Yi10BOqlTlg6FjAZPa8pdVcmlI6BlDzQX8KQ69VrHbrGLCUXBvXvgM3AXjC2Z1kdLDMjRoSt3ub8/ledvMMD2M7iEroRABPp4Os5m+5tYnpGeLCZsj3+BSQBb96mPc7C1GW9I03ukPHOOXS2CeBcZNmn2W92B2OgsRUt/7oXLTSKmxOvCXLGn7lybSypYw7bXMQooqEbxBEItl+LKixX4/yEA+nXw1Oo0CeJjVFrQRNdjW938cCWiXJX8ahWDuZ/2BltQlVOVfAHn/eRUPp34apzIj2jZzg0E5XsJ2HWzcau1LzI+MicFQ8ekqGuao9AyBQ7Shy0dkiCnXsl7pPk+7RzkD/hZZbC9oDc4kxWtE5VvaChXrxr+07R486n+ K1/3ilhC vowePhyvRvbYR7geNLwOtH/KdZB5RuKpjocRVdD5t16PdWBRo6p9SheK6LCcVQKL3Z8qibfuEaXJ0067X5CJHxpGbFOAJpdXqOFsk09/Uier8RU3PBAH9XIWgo0WnTfoYpq3k41lxKwHTGtiFXn7nyGpdUiyXkTbb7XWCxv00MJ7z+/J22kRBWMlKw5QsLejRq+JG1opqyWPwKrgLDXzOMiI+ZwSpMe11si25MrusJHqsHvDCaj419vrfEMWOh/Qgyc3ELnN/rsLh0zv+zZNW3rc7Nt99b1BpCqd6Ul/kxMces6RaSEGS0p/l2tsvp2EcxL0P5kGPR35Vjgns26RjkYjwZiazqRIcwplUhEbzf3fBWHRV7TbqWj5RrMjt+p7VjcBZgxXtrs1UUEmGX6OFSVCJ9Gvi0jqZj/cJQv25fwXBG54HEBDbAnUabU4ImQp8GVc4u32L/HCOf6VGdWvzA8l9rjvwIAprPIDTgoSVDUZKmVwucBuM8x1kSbBA+RjiOGuHvKx0r9wo248RE4q6lqZ+II2vb68V1CovYS6eBDKwrS/rjIipANofam/X1jeIWOQnZ/YtJGCcxEc62Bj7RRAojPDLuUQ74hSvt1Pw29Dxt+IgJ1YtQ0RgCd+LWoIX+akQSaY0luuHZnbjyTuQ8VjdrOFPoog9Sd+MNJUpUVTj6vYN9EPTXzdLLGegQdodmiGsWekHEKfHJ4MQwIxJiMp9nr1oMU0aIbtxqLS+AhAKFhangbZypYkIzGQpaI/Lm5GKsGw+mx9NC7sRj/z6hTrDljEEiFe3HoyRXdsDLkaRKPbvI/NCOUBwJkgU6MESo375G+wasPIPKcOKitPt7xW/rQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Many architectures' switch_mm() (e.g. arm64) do not have an smp_mb() which the core scheduler code has depended upon since commit: commit 223baf9d17f25 ("sched: Fix performance regression introduced by mm_cid") If switch_mm() doesn't call smp_mb(), sched_mm_cid_remote_clear() can unset the actively used cid when it fails to observe active task after it sets lazy_put. There *is* a memory barrier between storing to rq->curr and _return to userspace_ (as required by membarrier), but the rseq mm_cid has stricter requirements: the barrier needs to be issued between store to rq->curr and switch_mm_cid(), which happens earlier than: - spin_unlock(), - switch_to(). So it's fine when the architecture switch_mm() happens to have that barrier already, but less so when the architecture only provides the full barrier in switch_to() or spin_unlock(). It is a bug in the rseq switch_mm_cid() implementation. All architectures that don't have memory barriers in switch_mm(), but rather have the full barrier either in finish_lock_switch() or switch_to() have them too late for the needs of switch_mm_cid(). Introduce a new smp_mb__after_switch_mm(), defined as smp_mb() in the generic barrier.h header, and use it in switch_mm_cid() for scheduler transitions where switch_mm() is expected to provide a memory barrier. Architectures can override smp_mb__after_switch_mm() if their switch_mm() implementation provides an implicit memory barrier. Override it with a no-op on x86 which implicitly provide this memory barrier by writing to CR3. Link: https://lore.kernel.org/lkml/20240305145335.2696125-1-yeoreum.yun@arm.com/ Reported-by: levi.yun Signed-off-by: Mathieu Desnoyers Reviewed-by: Catalin Marinas # for arm64 Acked-by: Dave Hansen # for x86 Fixes: 223baf9d17f2 ("sched: Fix performance regression introduced by mm_cid") Cc: # 6.4.x Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Steven Rostedt Cc: Vincent Guittot Cc: Juri Lelli Cc: Dietmar Eggemann Cc: Ben Segall Cc: Mel Gorman Cc: Daniel Bristot de Oliveira Cc: Valentin Schneider Cc: levi.yun Cc: Mathieu Desnoyers Cc: Catalin Marinas Cc: Mark Rutland Cc: Will Deacon Cc: Aaron Lu Cc: Thomas Gleixner Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Arnd Bergmann Cc: Andrew Morton Cc: linux-arch@vger.kernel.org Cc: linux-mm@kvack.org Cc: x86@kernel.org --- arch/x86/include/asm/barrier.h | 3 +++ include/asm-generic/barrier.h | 8 ++++++++ kernel/sched/sched.h | 20 ++++++++++++++------ 3 files changed, 25 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/barrier.h b/arch/x86/include/asm/barrier.h index fe1e7e3cc844..63bdc6b85219 100644 --- a/arch/x86/include/asm/barrier.h +++ b/arch/x86/include/asm/barrier.h @@ -79,6 +79,9 @@ do { \ #define __smp_mb__before_atomic() do { } while (0) #define __smp_mb__after_atomic() do { } while (0) +/* Writing to CR3 provides a full memory barrier in switch_mm(). */ +#define smp_mb__after_switch_mm() do { } while (0) + #include #endif /* _ASM_X86_BARRIER_H */ diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h index 0c0695763bea..dc32b96140c1 100644 --- a/include/asm-generic/barrier.h +++ b/include/asm-generic/barrier.h @@ -294,5 +294,13 @@ do { \ #define io_stop_wc() do { } while (0) #endif +/* + * Architectures that guarantee an implicit smp_mb() in switch_mm() + * can override smp_mb__after_switch_mm. + */ +#ifndef smp_mb__after_switch_mm +#define smp_mb__after_switch_mm() smp_mb() +#endif + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_GENERIC_BARRIER_H */ diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index d2242679239e..d2895d264196 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -79,6 +79,8 @@ # include #endif +#include + #include "cpupri.h" #include "cpudeadline.h" @@ -3445,13 +3447,19 @@ static inline void switch_mm_cid(struct rq *rq, * between rq->curr store and load of {prev,next}->mm->pcpu_cid[cpu]. * Provide it here. */ - if (!prev->mm) // from kernel + if (!prev->mm) { // from kernel smp_mb(); - /* - * user -> user transition guarantees a memory barrier through - * switch_mm() when current->mm changes. If current->mm is - * unchanged, no barrier is needed. - */ + } else { // from user + /* + * user -> user transition relies on an implicit + * memory barrier in switch_mm() when + * current->mm changes. If the architecture + * switch_mm() does not have an implicit memory + * barrier, it is emitted here. If current->mm + * is unchanged, no barrier is needed. + */ + smp_mb__after_switch_mm(); + } } if (prev->mm_cid_active) { mm_cid_snapshot_time(rq, prev->mm); From patchwork Mon Apr 15 15:21:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Desnoyers X-Patchwork-Id: 13630220 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D6DBC4345F for ; Mon, 15 Apr 2024 15:21:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA4B26B00A0; Mon, 15 Apr 2024 11:21:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B54606B00A1; Mon, 15 Apr 2024 11:21:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90A196B00A2; Mon, 15 Apr 2024 11:21:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5A8AE6B00A1 for ; Mon, 15 Apr 2024 11:21:22 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id EE4371C0A86 for ; Mon, 15 Apr 2024 15:21:21 +0000 (UTC) X-FDA: 82012129962.26.7B0837B Received: from smtpout.efficios.com (smtpout.efficios.com [167.114.26.122]) by imf14.hostedemail.com (Postfix) with ESMTP id 3D661100015 for ; Mon, 15 Apr 2024 15:21:20 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b=GAH6fifi; dmarc=pass (policy=none) header.from=efficios.com; spf=pass (imf14.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 167.114.26.122 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713194480; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EdqK+KP1QZw77hd2TSIgV12/zV7R3tfaMik2pcHlQFw=; b=cI5b75GaFbRBPwrpEfYLQpLYitloGRRIBpsmEbwHzf+EPQm7W7utJFqyktkBSRrJJckMO6 y+NbmhOXFcRFQFtZ4/69IH102QdxMNxCXeO3V/ZMCcvn/DXda7rIFbaAnlji2AjbXQ1cXD PkMapt503WyvJemxv4TxZNDUIZvn9zc= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b=GAH6fifi; dmarc=pass (policy=none) header.from=efficios.com; spf=pass (imf14.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 167.114.26.122 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713194480; a=rsa-sha256; cv=none; b=Cr2uHEg/Nvp8bGHTAXq0psoZQPFbkdmAt66tbVdcnw5sGK6t9I4leTRtehsN6n+gqVsp7C 90YTcJPdjSK7JsEMBSfIv68gHfWehCDszcEf3j6I8a6gqoqUZcM5D54qyDgS6feSFvhbsw 8PsIz+7gZfbgmEU4qodCwixPWemzlZM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1713194479; bh=b9XYtPZFX8rSd40j1Mr6Vwh3YnqZdJjiiGq5LiMjyew=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GAH6fifiACkpTiicD55tC+a3o5n0ZYZ30clKFOQtUStm/R76Z7z1a8VjoOwJjhSgi Fs/Cu+APyYsWpZhFyaew8O6V1UaN9NmfRNd4qcy/xkBurPaQaB+d358cHbOQ/ic7qw zYuQvci4eFSFDacKvCy73c9Tiiby91oyECNg7v+EXDuXIKKW7SLO5TDyHLjv9llmfK vypkSTLXi5ilnF89qYrVekej1zgyAmQcW4gwFhAnXiJbBkd5cLyLrzY05tTOuuRsBU s+1/9VeUNldo7AGbPIJ1SC8N+k+IDc0tMIQDj/gSwZCtUmtAySZS60llmuUIA9lo/5 UYmJQYn/+P5pg== Received: from thinkos.internal.efficios.com (192-222-143-198.qc.cable.ebox.net [192.222.143.198]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4VJ9qf4p5kzvWF; Mon, 15 Apr 2024 11:21:18 -0400 (EDT) From: Mathieu Desnoyers To: Ingo Molnar , Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers , Steven Rostedt , Vincent Guittot , Juri Lelli , Dietmar Eggemann , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , "levi . yun" , Catalin Marinas , Mark Rutland , Will Deacon , Aaron Lu , Thomas Gleixner , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Arnd Bergmann , Andrew Morton , linux-arch@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Subject: [PATCH 2/2] sched: Move mm_cid code from sched.h to core.c Date: Mon, 15 Apr 2024 11:21:14 -0400 Message-Id: <20240415152114.59122-3-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240415152114.59122-1-mathieu.desnoyers@efficios.com> References: <20240415152114.59122-1-mathieu.desnoyers@efficios.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 3D661100015 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 4c7gekzz71qeryyhofokuwy5diaiemph X-HE-Tag: 1713194480-18356 X-HE-Meta: U2FsdGVkX1/i3GMh9Rs9U89p9m//MELcTwWsg2zUA+DrKzluvgPySQCAt4c5uoqqrER6LurII2tzZQlXJctXE7NcgkPyqJPSxC6lUgYb7t/d+VychizJwHHq5j3ewVjZN0DdToGO2YpZMRHq/xMTwI9yzF9DwKpKKsTRprG2UvqlMvOZt386g08r/VWyVXFPFJL5pK71rNIfO0DqIN3MaHE40N24ob4v9l7Cc8230ZyFsIEYcG8/YlJQGhaTfoB8HzlUhphzAB2ryEGH2WElukXj22cvtiN3V1Nz+UyZFrUPeVI4aIYaXRR2XXGmSmP6uI0nDaNn2ZRmMxgL9WYwoFTKnBfxxDc4gkiqCVXOpq1VnFkIANUyPfCi2f9Si/YhZJbQOx3IkcUkhqF/08wztWj6alcqQW2CZjfOkjfcHdIiA5lkmSo9PKbEreahlA4DigQjchvHWwDO4Sn1HsF/uDBfaKArlx+81y3A6CA3PyrScriPIQISrrozKTYTMeADxdDwiDMfTHoWlk+d0UEEA6OnPUhe0ua+nDCUUHUczDCZp+gQH77zSfLzn+ORie/HKFsCdVJcUydndy+pQvpJs7IBeXnlkmG4wD0UB3GywdfqIKzowch0Xp9DAzMkmxIZo42lExcanMSg5xiHmAjVHygGlvjAy5PD/gowxmdIlOSiIcgGlxJ5K8X8c9DT8vHbwiw3djnJaAzPivSFi2H4jNFrcGJqnYQKWdHpd2z2qggCK5tQ1F+iyu0E30RMEczSVCtnlbjwbn0hmKU415bAIFllnonnzUKA/kH3fG5QEp7bqXKY+iEcTk30w6EoiG4LZv+XjFC8kg45u2Sf6saHQlQIVjJRxeZNR5NkHFuP1ZnbyU9CxDiwXw13bQdrxGkHvUiyWklFQF2YrD1EfvPlQPGbZfOQX5sEqqchPAy4WIJCa0flZ+PicEY0F0tp6g8dktDlb1D3D47LSoca9EB fpGFsw6B NyKQMEYGuhRB0bshC5XQ2ttQgiM5npvbf4Y11jI/k0mqSWHqCo721TYPZwg9oYJyXTBsHJcRpnPKu51fT79uYDfbD+fLTxp6UZOlQZc4a0loPOVHU93IJsy/25eZv5fq9wa9qXS4p5g9xf4eR++DzA8sgmUNHdbbTxAkViaRENPBzzfcsW/BYU74A46D0WFD0+khh5B34QkzIjJq0LHebRd5hmE4HL/Zfu1a/IcF/IgmCHf6H6ziMXE01lPxbcrL5PFsDkRiYrV9AY8FGMw4EpM3LhxJAD+nOqd0Z16TjHeR1rnPJUx/YoYjAIpy3jeDbgSgomWAS5UU/wWM9NMqZ69MxEkUfhr1S5zaFpzKPTgLWU3ROcwilwpLNeYKqU9QDqrEI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The mm_cid code in sched/sched.h is only used from sched/core.c. Move it to the compile unit where it belongs. While reviewing mm_cid functions which were already in sched/core.c, I noticed that a few of them are non-static even though they are only used from core.c. Make those functions static inline. For sake of keeping things consistent, mm_cid functions only marked "static" are now marked "static inline". The variables cid_lock and use_cid_lock are only used from core.c, mark them as static. Moving from non-static to static inline for: - sched_mm_cid_migrate_from - init_sched_mm_cid - task_tick_mm_cid And the forced inlining of: - __sched_mm_cid_migrate_from_fetch_cid - __sched_mm_cid_migrate_from_try_steal_cid - sched_mm_cid_migrate_to - sched_mm_cid_remote_clear - sched_mm_cid_remote_clear_old - sched_mm_cid_remote_clear_weight slightly improves the size of sched/core.o on x86-64 (in bytes): text data before: 192261 58677 after: 191629 58641 ----------------------------- delta: -632 -36 Signed-off-by: Mathieu Desnoyers Cc: Ingo Molnar Cc: Peter Zijlstra --- kernel/sched/core.c | 277 +++++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 241 ------------------------------------- 2 files changed, 257 insertions(+), 261 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 7019a40457a6..57b03d874530 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -457,6 +457,22 @@ sched_core_dequeue(struct rq *rq, struct task_struct *p, int flags) { } #endif /* CONFIG_SCHED_CORE */ +#ifdef CONFIG_SCHED_MM_CID +static inline void switch_mm_cid(struct rq *rq, struct task_struct *prev, + struct task_struct *next); +static inline void sched_mm_cid_migrate_from(struct task_struct *t); +static inline void sched_mm_cid_migrate_to(struct rq *dst_rq, struct task_struct *t); +static inline void task_tick_mm_cid(struct rq *rq, struct task_struct *curr); +static inline void init_sched_mm_cid(struct task_struct *t); +#else +static inline void switch_mm_cid(struct rq *rq, struct task_struct *prev, + struct task_struct *next) { } +static inline void sched_mm_cid_migrate_from(struct task_struct *t) { } +static inline void sched_mm_cid_migrate_to(struct rq *dst_rq, struct task_struct *t) { } +static inline void task_tick_mm_cid(struct rq *rq, struct task_struct *curr) { } +static inline void init_sched_mm_cid(struct task_struct *t) { } +#endif + /* * Serialization rules: * @@ -11551,6 +11567,9 @@ void call_trace_sched_update_nr_running(struct rq *rq, int count) #ifdef CONFIG_SCHED_MM_CID +#define SCHED_MM_CID_PERIOD_NS (100ULL * 1000000) /* 100ms */ +#define MM_CID_SCAN_DELAY 100 /* 100ms */ + /* * @cid_lock: Guarantee forward-progress of cid allocation. * @@ -11558,7 +11577,7 @@ void call_trace_sched_update_nr_running(struct rq *rq, int count) * is only used when contention is detected by the lock-free allocation so * forward progress can be guaranteed. */ -DEFINE_RAW_SPINLOCK(cid_lock); +static DEFINE_RAW_SPINLOCK(cid_lock); /* * @use_cid_lock: Select cid allocation behavior: lock-free vs spinlock. @@ -11569,7 +11588,7 @@ DEFINE_RAW_SPINLOCK(cid_lock); * completes and sets @use_cid_lock back to 0. This guarantees forward progress * of a cid allocation. */ -int use_cid_lock; +static int use_cid_lock; /* * mm_cid remote-clear implements a lock-free algorithm to clear per-mm/cpu cid @@ -11659,15 +11678,233 @@ int use_cid_lock; * because this would UNSET a cid which is actively used. */ -void sched_mm_cid_migrate_from(struct task_struct *t) +static inline void __mm_cid_put(struct mm_struct *mm, int cid) +{ + if (cid < 0) + return; + cpumask_clear_cpu(cid, mm_cidmask(mm)); +} + +/* + * The per-mm/cpu cid can have the MM_CID_LAZY_PUT flag set or transition to + * the MM_CID_UNSET state without holding the rq lock, but the rq lock needs to + * be held to transition to other states. + * + * State transitions synchronized with cmpxchg or try_cmpxchg need to be + * consistent across cpus, which prevents use of this_cpu_cmpxchg. + */ +static inline void mm_cid_put_lazy(struct task_struct *t) +{ + struct mm_struct *mm = t->mm; + struct mm_cid __percpu *pcpu_cid = mm->pcpu_cid; + int cid; + + lockdep_assert_irqs_disabled(); + cid = __this_cpu_read(pcpu_cid->cid); + if (!mm_cid_is_lazy_put(cid) || + !try_cmpxchg(&this_cpu_ptr(pcpu_cid)->cid, &cid, MM_CID_UNSET)) + return; + __mm_cid_put(mm, mm_cid_clear_lazy_put(cid)); +} + +static inline int mm_cid_pcpu_unset(struct mm_struct *mm) +{ + struct mm_cid __percpu *pcpu_cid = mm->pcpu_cid; + int cid, res; + + lockdep_assert_irqs_disabled(); + cid = __this_cpu_read(pcpu_cid->cid); + for (;;) { + if (mm_cid_is_unset(cid)) + return MM_CID_UNSET; + /* + * Attempt transition from valid or lazy-put to unset. + */ + res = cmpxchg(&this_cpu_ptr(pcpu_cid)->cid, cid, MM_CID_UNSET); + if (res == cid) + break; + cid = res; + } + return cid; +} + +static inline void mm_cid_put(struct mm_struct *mm) +{ + int cid; + + lockdep_assert_irqs_disabled(); + cid = mm_cid_pcpu_unset(mm); + if (cid == MM_CID_UNSET) + return; + __mm_cid_put(mm, mm_cid_clear_lazy_put(cid)); +} + +static inline int __mm_cid_try_get(struct mm_struct *mm) +{ + struct cpumask *cpumask; + int cid; + + cpumask = mm_cidmask(mm); + /* + * Retry finding first zero bit if the mask is temporarily + * filled. This only happens during concurrent remote-clear + * which owns a cid without holding a rq lock. + */ + for (;;) { + cid = cpumask_first_zero(cpumask); + if (cid < nr_cpu_ids) + break; + cpu_relax(); + } + if (cpumask_test_and_set_cpu(cid, cpumask)) + return -1; + return cid; +} + +/* + * Save a snapshot of the current runqueue time of this cpu + * with the per-cpu cid value, allowing to estimate how recently it was used. + */ +static inline void mm_cid_snapshot_time(struct rq *rq, struct mm_struct *mm) +{ + struct mm_cid *pcpu_cid = per_cpu_ptr(mm->pcpu_cid, cpu_of(rq)); + + lockdep_assert_rq_held(rq); + WRITE_ONCE(pcpu_cid->time, rq->clock); +} + +static inline int __mm_cid_get(struct rq *rq, struct mm_struct *mm) +{ + int cid; + + /* + * All allocations (even those using the cid_lock) are lock-free. If + * use_cid_lock is set, hold the cid_lock to perform cid allocation to + * guarantee forward progress. + */ + if (!READ_ONCE(use_cid_lock)) { + cid = __mm_cid_try_get(mm); + if (cid >= 0) + goto end; + raw_spin_lock(&cid_lock); + } else { + raw_spin_lock(&cid_lock); + cid = __mm_cid_try_get(mm); + if (cid >= 0) + goto unlock; + } + + /* + * cid concurrently allocated. Retry while forcing following + * allocations to use the cid_lock to ensure forward progress. + */ + WRITE_ONCE(use_cid_lock, 1); + /* + * Set use_cid_lock before allocation. Only care about program order + * because this is only required for forward progress. + */ + barrier(); + /* + * Retry until it succeeds. It is guaranteed to eventually succeed once + * all newcoming allocations observe the use_cid_lock flag set. + */ + do { + cid = __mm_cid_try_get(mm); + cpu_relax(); + } while (cid < 0); + /* + * Allocate before clearing use_cid_lock. Only care about + * program order because this is for forward progress. + */ + barrier(); + WRITE_ONCE(use_cid_lock, 0); +unlock: + raw_spin_unlock(&cid_lock); +end: + mm_cid_snapshot_time(rq, mm); + return cid; +} + +static inline int mm_cid_get(struct rq *rq, struct mm_struct *mm) +{ + struct mm_cid __percpu *pcpu_cid = mm->pcpu_cid; + struct cpumask *cpumask; + int cid; + + lockdep_assert_rq_held(rq); + cpumask = mm_cidmask(mm); + cid = __this_cpu_read(pcpu_cid->cid); + if (mm_cid_is_valid(cid)) { + mm_cid_snapshot_time(rq, mm); + return cid; + } + if (mm_cid_is_lazy_put(cid)) { + if (try_cmpxchg(&this_cpu_ptr(pcpu_cid)->cid, &cid, MM_CID_UNSET)) + __mm_cid_put(mm, mm_cid_clear_lazy_put(cid)); + } + cid = __mm_cid_get(rq, mm); + __this_cpu_write(pcpu_cid->cid, cid); + return cid; +} + +static inline void switch_mm_cid(struct rq *rq, struct task_struct *prev, + struct task_struct *next) +{ + /* + * Provide a memory barrier between rq->curr store and load of + * {prev,next}->mm->pcpu_cid[cpu] on rq->curr->mm transition. + * + * Should be adapted if context_switch() is modified. + */ + if (!next->mm) { // to kernel + /* + * user -> kernel transition does not guarantee a barrier, but + * we can use the fact that it performs an atomic operation in + * mmgrab(). + */ + if (prev->mm) // from user + smp_mb__after_mmgrab(); + /* + * kernel -> kernel transition does not change rq->curr->mm + * state. It stays NULL. + */ + } else { // to user + /* + * kernel -> user transition does not provide a barrier + * between rq->curr store and load of {prev,next}->mm->pcpu_cid[cpu]. + * Provide it here. + */ + if (!prev->mm) { // from kernel + smp_mb(); + } else { // from user + /* + * user -> user transition relies on an implicit + * memory barrier in switch_mm() when + * current->mm changes. If the architecture + * switch_mm() does not have an implicit memory + * barrier, it is emitted here. If current->mm + * is unchanged, no barrier is needed. + */ + smp_mb__after_switch_mm(); + } + } + if (prev->mm_cid_active) { + mm_cid_snapshot_time(rq, prev->mm); + mm_cid_put_lazy(prev); + prev->mm_cid = -1; + } + if (next->mm_cid_active) + next->last_mm_cid = next->mm_cid = mm_cid_get(rq, next->mm); +} + +static inline void sched_mm_cid_migrate_from(struct task_struct *t) { t->migrate_from_cpu = task_cpu(t); } -static -int __sched_mm_cid_migrate_from_fetch_cid(struct rq *src_rq, - struct task_struct *t, - struct mm_cid *src_pcpu_cid) +static inline int __sched_mm_cid_migrate_from_fetch_cid(struct rq *src_rq, + struct task_struct *t, + struct mm_cid *src_pcpu_cid) { struct mm_struct *mm = t->mm; struct task_struct *src_task; @@ -11703,11 +11940,10 @@ int __sched_mm_cid_migrate_from_fetch_cid(struct rq *src_rq, return src_cid; } -static -int __sched_mm_cid_migrate_from_try_steal_cid(struct rq *src_rq, - struct task_struct *t, - struct mm_cid *src_pcpu_cid, - int src_cid) +static inline int __sched_mm_cid_migrate_from_try_steal_cid(struct rq *src_rq, + struct task_struct *t, + struct mm_cid *src_pcpu_cid, + int src_cid) { struct task_struct *src_task; struct mm_struct *mm = t->mm; @@ -11767,7 +12003,7 @@ int __sched_mm_cid_migrate_from_try_steal_cid(struct rq *src_rq, * Interrupts are disabled, which keeps the window of cid ownership without the * source rq lock held small. */ -void sched_mm_cid_migrate_to(struct rq *dst_rq, struct task_struct *t) +static inline void sched_mm_cid_migrate_to(struct rq *dst_rq, struct task_struct *t) { struct mm_cid *src_pcpu_cid, *dst_pcpu_cid; struct mm_struct *mm = t->mm; @@ -11820,8 +12056,9 @@ void sched_mm_cid_migrate_to(struct rq *dst_rq, struct task_struct *t) WRITE_ONCE(dst_pcpu_cid->cid, src_cid); } -static void sched_mm_cid_remote_clear(struct mm_struct *mm, struct mm_cid *pcpu_cid, - int cpu) +static inline void sched_mm_cid_remote_clear(struct mm_struct *mm, + struct mm_cid *pcpu_cid, + int cpu) { struct rq *rq = cpu_rq(cpu); struct task_struct *t; @@ -11876,7 +12113,7 @@ static void sched_mm_cid_remote_clear(struct mm_struct *mm, struct mm_cid *pcpu_ } } -static void sched_mm_cid_remote_clear_old(struct mm_struct *mm, int cpu) +static inline void sched_mm_cid_remote_clear_old(struct mm_struct *mm, int cpu) { struct rq *rq = cpu_rq(cpu); struct mm_cid *pcpu_cid; @@ -11908,8 +12145,8 @@ static void sched_mm_cid_remote_clear_old(struct mm_struct *mm, int cpu) sched_mm_cid_remote_clear(mm, pcpu_cid, cpu); } -static void sched_mm_cid_remote_clear_weight(struct mm_struct *mm, int cpu, - int weight) +static inline void sched_mm_cid_remote_clear_weight(struct mm_struct *mm, int cpu, + int weight) { struct mm_cid *pcpu_cid; int cid; @@ -11965,7 +12202,7 @@ static void task_mm_cid_work(struct callback_head *work) sched_mm_cid_remote_clear_weight(mm, cpu, weight); } -void init_sched_mm_cid(struct task_struct *t) +static inline void init_sched_mm_cid(struct task_struct *t) { struct mm_struct *mm = t->mm; int mm_users = 0; @@ -11979,7 +12216,7 @@ void init_sched_mm_cid(struct task_struct *t) init_task_work(&t->cid_work, task_mm_cid_work); } -void task_tick_mm_cid(struct rq *rq, struct task_struct *curr) +static inline void task_tick_mm_cid(struct rq *rq, struct task_struct *curr) { struct callback_head *work = &curr->cid_work; unsigned long now = jiffies; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index d2895d264196..1b8e3e23ef40 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -3237,247 +3237,6 @@ extern int sched_dynamic_mode(const char *str); extern void sched_dynamic_update(int mode); #endif -#ifdef CONFIG_SCHED_MM_CID - -#define SCHED_MM_CID_PERIOD_NS (100ULL * 1000000) /* 100ms */ -#define MM_CID_SCAN_DELAY 100 /* 100ms */ - -extern raw_spinlock_t cid_lock; -extern int use_cid_lock; - -extern void sched_mm_cid_migrate_from(struct task_struct *t); -extern void sched_mm_cid_migrate_to(struct rq *dst_rq, struct task_struct *t); -extern void task_tick_mm_cid(struct rq *rq, struct task_struct *curr); -extern void init_sched_mm_cid(struct task_struct *t); - -static inline void __mm_cid_put(struct mm_struct *mm, int cid) -{ - if (cid < 0) - return; - cpumask_clear_cpu(cid, mm_cidmask(mm)); -} - -/* - * The per-mm/cpu cid can have the MM_CID_LAZY_PUT flag set or transition to - * the MM_CID_UNSET state without holding the rq lock, but the rq lock needs to - * be held to transition to other states. - * - * State transitions synchronized with cmpxchg or try_cmpxchg need to be - * consistent across cpus, which prevents use of this_cpu_cmpxchg. - */ -static inline void mm_cid_put_lazy(struct task_struct *t) -{ - struct mm_struct *mm = t->mm; - struct mm_cid __percpu *pcpu_cid = mm->pcpu_cid; - int cid; - - lockdep_assert_irqs_disabled(); - cid = __this_cpu_read(pcpu_cid->cid); - if (!mm_cid_is_lazy_put(cid) || - !try_cmpxchg(&this_cpu_ptr(pcpu_cid)->cid, &cid, MM_CID_UNSET)) - return; - __mm_cid_put(mm, mm_cid_clear_lazy_put(cid)); -} - -static inline int mm_cid_pcpu_unset(struct mm_struct *mm) -{ - struct mm_cid __percpu *pcpu_cid = mm->pcpu_cid; - int cid, res; - - lockdep_assert_irqs_disabled(); - cid = __this_cpu_read(pcpu_cid->cid); - for (;;) { - if (mm_cid_is_unset(cid)) - return MM_CID_UNSET; - /* - * Attempt transition from valid or lazy-put to unset. - */ - res = cmpxchg(&this_cpu_ptr(pcpu_cid)->cid, cid, MM_CID_UNSET); - if (res == cid) - break; - cid = res; - } - return cid; -} - -static inline void mm_cid_put(struct mm_struct *mm) -{ - int cid; - - lockdep_assert_irqs_disabled(); - cid = mm_cid_pcpu_unset(mm); - if (cid == MM_CID_UNSET) - return; - __mm_cid_put(mm, mm_cid_clear_lazy_put(cid)); -} - -static inline int __mm_cid_try_get(struct mm_struct *mm) -{ - struct cpumask *cpumask; - int cid; - - cpumask = mm_cidmask(mm); - /* - * Retry finding first zero bit if the mask is temporarily - * filled. This only happens during concurrent remote-clear - * which owns a cid without holding a rq lock. - */ - for (;;) { - cid = cpumask_first_zero(cpumask); - if (cid < nr_cpu_ids) - break; - cpu_relax(); - } - if (cpumask_test_and_set_cpu(cid, cpumask)) - return -1; - return cid; -} - -/* - * Save a snapshot of the current runqueue time of this cpu - * with the per-cpu cid value, allowing to estimate how recently it was used. - */ -static inline void mm_cid_snapshot_time(struct rq *rq, struct mm_struct *mm) -{ - struct mm_cid *pcpu_cid = per_cpu_ptr(mm->pcpu_cid, cpu_of(rq)); - - lockdep_assert_rq_held(rq); - WRITE_ONCE(pcpu_cid->time, rq->clock); -} - -static inline int __mm_cid_get(struct rq *rq, struct mm_struct *mm) -{ - int cid; - - /* - * All allocations (even those using the cid_lock) are lock-free. If - * use_cid_lock is set, hold the cid_lock to perform cid allocation to - * guarantee forward progress. - */ - if (!READ_ONCE(use_cid_lock)) { - cid = __mm_cid_try_get(mm); - if (cid >= 0) - goto end; - raw_spin_lock(&cid_lock); - } else { - raw_spin_lock(&cid_lock); - cid = __mm_cid_try_get(mm); - if (cid >= 0) - goto unlock; - } - - /* - * cid concurrently allocated. Retry while forcing following - * allocations to use the cid_lock to ensure forward progress. - */ - WRITE_ONCE(use_cid_lock, 1); - /* - * Set use_cid_lock before allocation. Only care about program order - * because this is only required for forward progress. - */ - barrier(); - /* - * Retry until it succeeds. It is guaranteed to eventually succeed once - * all newcoming allocations observe the use_cid_lock flag set. - */ - do { - cid = __mm_cid_try_get(mm); - cpu_relax(); - } while (cid < 0); - /* - * Allocate before clearing use_cid_lock. Only care about - * program order because this is for forward progress. - */ - barrier(); - WRITE_ONCE(use_cid_lock, 0); -unlock: - raw_spin_unlock(&cid_lock); -end: - mm_cid_snapshot_time(rq, mm); - return cid; -} - -static inline int mm_cid_get(struct rq *rq, struct mm_struct *mm) -{ - struct mm_cid __percpu *pcpu_cid = mm->pcpu_cid; - struct cpumask *cpumask; - int cid; - - lockdep_assert_rq_held(rq); - cpumask = mm_cidmask(mm); - cid = __this_cpu_read(pcpu_cid->cid); - if (mm_cid_is_valid(cid)) { - mm_cid_snapshot_time(rq, mm); - return cid; - } - if (mm_cid_is_lazy_put(cid)) { - if (try_cmpxchg(&this_cpu_ptr(pcpu_cid)->cid, &cid, MM_CID_UNSET)) - __mm_cid_put(mm, mm_cid_clear_lazy_put(cid)); - } - cid = __mm_cid_get(rq, mm); - __this_cpu_write(pcpu_cid->cid, cid); - return cid; -} - -static inline void switch_mm_cid(struct rq *rq, - struct task_struct *prev, - struct task_struct *next) -{ - /* - * Provide a memory barrier between rq->curr store and load of - * {prev,next}->mm->pcpu_cid[cpu] on rq->curr->mm transition. - * - * Should be adapted if context_switch() is modified. - */ - if (!next->mm) { // to kernel - /* - * user -> kernel transition does not guarantee a barrier, but - * we can use the fact that it performs an atomic operation in - * mmgrab(). - */ - if (prev->mm) // from user - smp_mb__after_mmgrab(); - /* - * kernel -> kernel transition does not change rq->curr->mm - * state. It stays NULL. - */ - } else { // to user - /* - * kernel -> user transition does not provide a barrier - * between rq->curr store and load of {prev,next}->mm->pcpu_cid[cpu]. - * Provide it here. - */ - if (!prev->mm) { // from kernel - smp_mb(); - } else { // from user - /* - * user -> user transition relies on an implicit - * memory barrier in switch_mm() when - * current->mm changes. If the architecture - * switch_mm() does not have an implicit memory - * barrier, it is emitted here. If current->mm - * is unchanged, no barrier is needed. - */ - smp_mb__after_switch_mm(); - } - } - if (prev->mm_cid_active) { - mm_cid_snapshot_time(rq, prev->mm); - mm_cid_put_lazy(prev); - prev->mm_cid = -1; - } - if (next->mm_cid_active) - next->last_mm_cid = next->mm_cid = mm_cid_get(rq, next->mm); -} - -#else -static inline void switch_mm_cid(struct rq *rq, struct task_struct *prev, struct task_struct *next) { } -static inline void sched_mm_cid_migrate_from(struct task_struct *t) { } -static inline void sched_mm_cid_migrate_to(struct rq *dst_rq, struct task_struct *t) { } -static inline void task_tick_mm_cid(struct rq *rq, struct task_struct *curr) { } -static inline void init_sched_mm_cid(struct task_struct *t) { } -#endif - extern u64 avg_vruntime(struct cfs_rq *cfs_rq); extern int entity_eligible(struct cfs_rq *cfs_rq, struct sched_entity *se);