From patchwork Thu Apr 11 17:43:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Desnoyers X-Patchwork-Id: 13626497 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 359B3C4345F for ; Thu, 11 Apr 2024 17:46:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A97186B0092; Thu, 11 Apr 2024 13:46:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A47776B0093; Thu, 11 Apr 2024 13:46:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90EAB6B0096; Thu, 11 Apr 2024 13:46:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 73ED86B0092 for ; Thu, 11 Apr 2024 13:46:07 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 24BFE120C41 for ; Thu, 11 Apr 2024 17:46:07 +0000 (UTC) X-FDA: 81997979574.24.5C90B2C Received: from smtpout.efficios.com (smtpout.efficios.com [167.114.26.122]) by imf29.hostedemail.com (Postfix) with ESMTP id 90D84120012 for ; Thu, 11 Apr 2024 17:46:04 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b=cw926R2Z; dmarc=pass (policy=none) header.from=efficios.com; spf=pass (imf29.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 167.114.26.122 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712857564; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=rw9zYGyHFDhZYwrQeqSGkQCpgh8g21m51ySM2La/EY4=; b=KVQu/LqFr9o78Crl00FQTb/9C9z2uojHDIqt2+JPXzWah8V+o02PZtlD6e8ckWJS6LUJ9a 3qO3nMmNf5KdwGxotWzGCGfmAv5QxiSWet6CH4/Sk0tqJFLF9ETo9R7D8Fd0HxRTSgB/tJ zSZBOGGcni5p63wxHT0/eUrlVjcj/3A= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b=cw926R2Z; dmarc=pass (policy=none) header.from=efficios.com; spf=pass (imf29.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 167.114.26.122 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712857564; a=rsa-sha256; cv=none; b=BI9EeABldBWUBIcg0HX0isfFePCVNOawPvD9PyJTFwFi1sbJ3KJKz2mbiatlp+KwP/KBzr 5AnEvAnpZm362rcVV+t5fY79AnjETmdah5xnPXsjA2K4qUCHuraQOnfUVlIjsqsvFf08q3 suf00/PzbEOnlppAX7VuIweqa1F/+Kw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1712857563; bh=OKxXIh/WXhatcgx1Ou+NvsrOJcDB3Ihf5QKnsGO8RQI=; h=From:To:Cc:Subject:Date:From; b=cw926R2ZfZ1029Vbp53hck1uPrzSwQh8tRDZhRSYvJRpmGGx6eXEKOMBajPcXeAiM 62lGtEXeEuE7Bh79gMY/YrQ7Ix11qWduAI5cuBM2MJ4HF8Bdm/pJJQQgI5zzAWOGl8 g7v/GJekpVD2eEh4gTp5Hw5IkehtGrQZg4LsgB96VecOZWTPRevSsyrrudfmx+Rf7h EqugCQIzkPP17GYgRErQJh3FIntGNYOPODSjK5ZZ1SXX40e41jWtIxAk4zbNOA+bNj jbRhNSO0j8FJ1E8xg/thTSJbuVcEGgtjj9+BsqRGKJSpKpkOZvs4xCRDsS5JpF4viv dbGRGq6E0YUmA== Received: from localhost.localdomain (192-222-143-198.qc.cable.ebox.net [192.222.143.198]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4VFnDV4cYWzsHQ; Thu, 11 Apr 2024 13:46:02 -0400 (EDT) From: Mathieu Desnoyers To: Ingo Molnar , Peter Zijlstra , Andrew Morton Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers , "levi . yun" , Catalin Marinas , Dave Hansen , stable@vger.kernel.org, Steven Rostedt , Vincent Guittot , Juri Lelli , Dietmar Eggemann , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Mark Rutland , Will Deacon , Aaron Lu , Thomas Gleixner , Borislav Petkov , "H. Peter Anvin" , Arnd Bergmann , linux-arch@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Subject: [PATCH] sched: Add missing memory barrier in switch_mm_cid Date: Thu, 11 Apr 2024 13:43:02 -0400 Message-Id: <20240411174302.353889-1-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 90D84120012 X-Stat-Signature: 4ad4s4iud7giekeybwjawnd6rq99dxg5 X-Rspam-User: X-HE-Tag: 1712857564-412783 X-HE-Meta: U2FsdGVkX19NRpUZonnBqD1p79tioC6CJoCia2sI98mTD7xa20xOdGGBhJDnuOjvLgzbeeyarc4RxkYrlVapF/2//dTqPJU9cwnAZEFMbpwjYekjUaSaMmz+ChC3kMWubYLi5Jj3EqFVHiwl9RJ9biYU2QTt9mWVVw9o/g1OQBdREYkasnwy5gr1EfjCOkVeIf2Dl31cejMlUMiP0clqrjye9v3Sem2V91PCsbmdrKT/JmodATcEw/a+2BRyMk4FXtjuYVABR9gRB2x/HbPxbJSpOYXo58eEKZ/SAYKjS+k2bOVvkqO3+L5sNfaA+eXbGfRciuSqYIfpkQaaSSZqy91hqmQ70f41nRqSO1XW7JmGqtA3N66LsSd7GWc8hUxXvSSeNFDEaEse6JDIVGsNZnOTxfg2S6maCUBFhD8sKgmMU4qhO6HGMbhfWXnyadFXbktrg6lam1R1zhRGuRM0jW8s5yogrsFLmulceIZQwQORQvV7KfyEL3sCWprE047aQG1U7fl3iiHDBIUtt5jiwlRhEFLwOAUMerelffauqwiDJUfUt+/H0mJnilyeX83l6EN+XHOMSPSNKdv2vI21yqenNiGBgfqlyWni2YPEgK36otmE83/DHnOJz/YgM6UmA0qHylHvvux4ZIn4VBq7i+yp6cYoU+xS/qw+8MrtA1+i1pQ9ifOCODGB/qw9ljErtfCniIhJBaflhuC1Gp1AcDzSD39HPBBrBiNi03zxS4yak9thpxCkXswvU0ez5HM76qdyTfGcRBAN7MtD0DTtroHPaRWAcLkv1wTCYrSQxUhsGWDc1rSvHPk6YC4E8DBJ5N18tCs6u+HQiI6OdI1yXZkgcKYXdo449/ob86oym/gSsAtMVO/Q2/5POGrY1EdlVJHlCRb7j8wT0VUMFTDmU2Pjw0NUKd34IYrZM9fkpT6gnbPxkV4WnDQv+4TnclWfiWJBlPGSecuvuEnlEL6 uae9OH67 OWGfxJz0DM3y7rUU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Many architectures' switch_mm() (e.g. arm64) do not have an smp_mb() which the core scheduler code has depended upon since commit: commit 223baf9d17f25 ("sched: Fix performance regression introduced by mm_cid") If switch_mm() doesn't call smp_mb(), sched_mm_cid_remote_clear() can unset the actively used cid when it fails to observe active task after it sets lazy_put. There *is* a memory barrier between storing to rq->curr and _return to userspace_ (as required by membarrier), but the rseq mm_cid has stricter requirements: the barrier needs to be issued between store to rq->curr and switch_mm_cid(), which happens earlier than: - spin_unlock(), - switch_to(). So it's fine when the architecture switch_mm() happens to have that barrier already, but less so when the architecture only provides the full barrier in switch_to() or spin_unlock(). It is a bug in the rseq switch_mm_cid() implementation. All architectures that don't have memory barriers in switch_mm(), but rather have the full barrier either in finish_lock_switch() or switch_to() have them too late for the needs of switch_mm_cid(). Introduce a new smp_mb__after_switch_mm(), defined as smp_mb() in the generic barrier.h header, and use it in switch_mm_cid() for scheduler transitions where switch_mm() is expected to provide a memory barrier. Architectures can override smp_mb__after_switch_mm() if their switch_mm() implementation provides an implicit memory barrier. Override it with a no-op on x86 which implicitly provide this memory barrier by writing to CR3. Link: https://lore.kernel.org/lkml/20240305145335.2696125-1-yeoreum.yun@arm.com/ Reported-by: levi.yun Signed-off-by: Mathieu Desnoyers Reviewed-by: Catalin Marinas # for arm64 Acked-by: Dave Hansen # for x86 Fixes: 223baf9d17f2 ("sched: Fix performance regression introduced by mm_cid") Cc: # 6.4.x Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Steven Rostedt Cc: Vincent Guittot Cc: Juri Lelli Cc: Dietmar Eggemann Cc: Ben Segall Cc: Mel Gorman Cc: Daniel Bristot de Oliveira Cc: Valentin Schneider Cc: levi.yun Cc: Mathieu Desnoyers Cc: Catalin Marinas Cc: Mark Rutland Cc: Will Deacon Cc: Aaron Lu Cc: Thomas Gleixner Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Arnd Bergmann Cc: Andrew Morton Cc: linux-arch@vger.kernel.org Cc: linux-mm@kvack.org Cc: x86@kernel.org Signed-off-by: Mathieu Desnoyers --- arch/x86/include/asm/barrier.h | 3 +++ include/asm-generic/barrier.h | 8 ++++++++ kernel/sched/sched.h | 20 ++++++++++++++------ 3 files changed, 25 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/barrier.h b/arch/x86/include/asm/barrier.h index 0216f63a366b..d0795b5fab46 100644 --- a/arch/x86/include/asm/barrier.h +++ b/arch/x86/include/asm/barrier.h @@ -79,6 +79,9 @@ do { \ #define __smp_mb__before_atomic() do { } while (0) #define __smp_mb__after_atomic() do { } while (0) +/* Writing to CR3 provides a full memory barrier in switch_mm(). */ +#define smp_mb__after_switch_mm() do { } while (0) + #include #endif /* _ASM_X86_BARRIER_H */ diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h index 961f4d88f9ef..5a6c94d7a598 100644 --- a/include/asm-generic/barrier.h +++ b/include/asm-generic/barrier.h @@ -296,5 +296,13 @@ do { \ #define io_stop_wc() do { } while (0) #endif +/* + * Architectures that guarantee an implicit smp_mb() in switch_mm() + * can override smp_mb__after_switch_mm. + */ +#ifndef smp_mb__after_switch_mm +#define smp_mb__after_switch_mm() smp_mb() +#endif + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_GENERIC_BARRIER_H */ diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 001fe047bd5d..35717359d3ca 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -79,6 +79,8 @@ # include #endif +#include + #include "cpupri.h" #include "cpudeadline.h" @@ -3445,13 +3447,19 @@ static inline void switch_mm_cid(struct rq *rq, * between rq->curr store and load of {prev,next}->mm->pcpu_cid[cpu]. * Provide it here. */ - if (!prev->mm) // from kernel + if (!prev->mm) { // from kernel smp_mb(); - /* - * user -> user transition guarantees a memory barrier through - * switch_mm() when current->mm changes. If current->mm is - * unchanged, no barrier is needed. - */ + } else { // from user + /* + * user -> user transition relies on an implicit + * memory barrier in switch_mm() when + * current->mm changes. If the architecture + * switch_mm() does not have an implicit memory + * barrier, it is emitted here. If current->mm + * is unchanged, no barrier is needed. + */ + smp_mb__after_switch_mm(); + } } if (prev->mm_cid_active) { mm_cid_snapshot_time(rq, prev->mm);