From patchwork Wed Aug 17 16:26:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 12946243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1D87C3F6B0 for ; Wed, 17 Aug 2022 16:27:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D29218D000A; Wed, 17 Aug 2022 12:27:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CD4968D0005; Wed, 17 Aug 2022 12:27:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B74428D000A; Wed, 17 Aug 2022 12:27:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A72268D0005 for ; Wed, 17 Aug 2022 12:27:40 -0400 (EDT) Received: from smtpin31.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 830FAA16E8 for ; Wed, 17 Aug 2022 16:27:40 +0000 (UTC) X-FDA: 79809615480.31.29805E7 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf31.hostedemail.com (Postfix) with ESMTP id 729D5201D6 for ; Wed, 17 Aug 2022 16:27:38 +0000 (UTC) From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1660753656; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wiRzqGd4tF+H6l5Mpr46iPb7Iqjo75nyCRDmhTMfrBU=; b=q8KvGO/BxuB0m+9l8pbSRGxWaSIsfwgo/OI7ry39vBOkoqPWERrWsH75Dn5BlgsdwY6cYO USAWE72Uet8ZT6/Tjk0HKWLJeR/E6R9gKCE7rW1B3MjGDdb4AOQGCZEFAGN8la5QrYoadO rhVnmDufo2AuPuHxz6uMHkWS01BxpOk8h/yBQEVvFk8FZXfJlZy/UgjV/mNfuPUskjVQpl ot+Z+JAKtbnOGw7mqxpNJWdjPIiUkHpACE8ds73O6jcVweHre7dgtXg/97kPK5OUcU5Sjj ETY24DRJy65Sul1qjGAA6u/3AL//cw8pj4pTFOMU2MMaiBETekdC+uDmLCCzcQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1660753656; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wiRzqGd4tF+H6l5Mpr46iPb7Iqjo75nyCRDmhTMfrBU=; b=W/30GvMvGlOlQIUvbaTWlnADOD2PKDbdwCvsDvpT8PIE9hDi9tNJue2Jum01uAmt4IMFBc sMHHIVmCW86MpoAQ== To: linux-kernel@vger.kernel.org Cc: Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Linus Torvalds , Matthew Wilcox , Andrew Morton , Christoph Lameter , David Rientjes , Joonsoo Kim , Pekka Enberg , Vlastimil Babka , linux-mm@kvack.org, Sebastian Andrzej Siewior Subject: [PATCH 1/9] slub: Make PREEMPT_RT support less convoluted Date: Wed, 17 Aug 2022 18:26:55 +0200 Message-Id: <20220817162703.728679-2-bigeasy@linutronix.de> In-Reply-To: <20220817162703.728679-1-bigeasy@linutronix.de> References: <20220817162703.728679-1-bigeasy@linutronix.de> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b="q8KvGO/B"; dkim=pass header.d=linutronix.de header.s=2020e header.b="W/30GvMv"; spf=pass (imf31.hostedemail.com: domain of bigeasy@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de; dmarc=pass (policy=none) header.from=linutronix.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660753659; a=rsa-sha256; cv=none; b=OXvpgQHPG9KnRJ+68WuKwXcGLkaCjgzIcFkwJc+eeU6W0vnE6/qdNpnWVkP3d3tlBpTJg0 ZOpxi+ncrIym33J/sG4aCNP1MKJ28Yxstk0VElLXBkHJOpIE2jU3dP135/9frIeQRPGiBA seRuE0HxnJGolOf7U55n2f7f+xAUAbo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660753659; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wiRzqGd4tF+H6l5Mpr46iPb7Iqjo75nyCRDmhTMfrBU=; b=7NvMk5lPmljiF0CtxQX3v7ZHFPlwGBqKoVsISqntriKoF4NRAfAcDhot/ItPJUiIrO2/NT Knnd3jM+gIsZOqoQ+5lepL9a6jN9aMOBoAsghD2Lt19vR9ArCzaTDnojsICPXpYG7xmbMN KXG5QuodoZbLxL+RvIY9p2DIjp/WNzc= Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b="q8KvGO/B"; dkim=pass header.d=linutronix.de header.s=2020e header.b="W/30GvMv"; spf=pass (imf31.hostedemail.com: domain of bigeasy@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de; dmarc=pass (policy=none) header.from=linutronix.de X-Rspam-User: X-Rspamd-Server: rspam12 X-Stat-Signature: beueg8tr3qrb1r3j3c97y5jr514kgk3q X-Rspamd-Queue-Id: 729D5201D6 X-HE-Tag: 1660753658-3577 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Thomas Gleixner The slub code already has a few helpers depending on PREEMPT_RT. Add a few more and get rid of the CONFIG_PREEMPT_RT conditionals all over the place. No functional change. Signed-off-by: Thomas Gleixner Cc: Andrew Morton Cc: Christoph Lameter Cc: David Rientjes Cc: Joonsoo Kim Cc: Pekka Enberg Cc: Vlastimil Babka Cc: linux-mm@kvack.org Signed-off-by: Sebastian Andrzej Siewior --- mm/slub.c | 66 +++++++++++++++++++++++++------------------------------ 1 file changed, 30 insertions(+), 36 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 862dbd9af4f52..5f7c5b5bd49f9 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -100,9 +100,11 @@ * except the stat counters. This is a percpu structure manipulated only by * the local cpu, so the lock protects against being preempted or interrupted * by an irq. Fast path operations rely on lockless operations instead. - * On PREEMPT_RT, the local lock does not actually disable irqs (and thus - * prevent the lockless operations), so fastpath operations also need to take - * the lock and are no longer lockless. + * + * On PREEMPT_RT, the local lock neither disables interrupts nor preemption + * which means the lockless fastpath cannot be used as it might interfere with + * an in-progress slow path operations. In this case the local lock is always + * taken but it still utilizes the freelist for the common operations. * * lockless fastpaths * @@ -163,8 +165,11 @@ * function call even on !PREEMPT_RT, use inline preempt_disable() there. */ #ifndef CONFIG_PREEMPT_RT -#define slub_get_cpu_ptr(var) get_cpu_ptr(var) -#define slub_put_cpu_ptr(var) put_cpu_ptr(var) +#define slub_get_cpu_ptr(var) get_cpu_ptr(var) +#define slub_put_cpu_ptr(var) put_cpu_ptr(var) +#define use_lockless_fast_path() (true) +#define slub_local_irq_save(flags) local_irq_save(flags) +#define slub_local_irq_restore(flags) local_irq_restore(flags) #else #define slub_get_cpu_ptr(var) \ ({ \ @@ -176,6 +181,9 @@ do { \ (void)(var); \ migrate_enable(); \ } while (0) +#define use_lockless_fast_path() (false) +#define slub_local_irq_save(flags) do { } while (0) +#define slub_local_irq_restore(flags) do { } while (0) #endif #ifdef CONFIG_SLUB_DEBUG @@ -460,16 +468,14 @@ static __always_inline void __slab_unlock(struct slab *slab) static __always_inline void slab_lock(struct slab *slab, unsigned long *flags) { - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - local_irq_save(*flags); + slub_local_irq_save(*flags); __slab_lock(slab); } static __always_inline void slab_unlock(struct slab *slab, unsigned long *flags) { __slab_unlock(slab); - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - local_irq_restore(*flags); + slub_local_irq_restore(*flags); } /* @@ -482,7 +488,7 @@ static inline bool __cmpxchg_double_slab(struct kmem_cache *s, struct slab *slab void *freelist_new, unsigned long counters_new, const char *n) { - if (!IS_ENABLED(CONFIG_PREEMPT_RT)) + if (use_lockless_fast_path()) lockdep_assert_irqs_disabled(); #if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \ defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) @@ -3197,14 +3203,8 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, struct list_l object = c->freelist; slab = c->slab; - /* - * We cannot use the lockless fastpath on PREEMPT_RT because if a - * slowpath has taken the local_lock_irqsave(), it is not protected - * against a fast path operation in an irq handler. So we need to take - * the slow path which uses local_lock. It is still relatively fast if - * there is a suitable cpu freelist. - */ - if (IS_ENABLED(CONFIG_PREEMPT_RT) || + + if (!use_lockless_fast_path() || unlikely(!object || !slab || !node_match(slab, node))) { object = __slab_alloc(s, gfpflags, node, addr, c); } else { @@ -3463,6 +3463,7 @@ static __always_inline void do_slab_free(struct kmem_cache *s, void *tail_obj = tail ? : head; struct kmem_cache_cpu *c; unsigned long tid; + void **freelist; redo: /* @@ -3477,9 +3478,13 @@ static __always_inline void do_slab_free(struct kmem_cache *s, /* Same with comment on barrier() in slab_alloc_node() */ barrier(); - if (likely(slab == c->slab)) { -#ifndef CONFIG_PREEMPT_RT - void **freelist = READ_ONCE(c->freelist); + if (unlikely(slab != c->slab)) { + __slab_free(s, slab, head, tail_obj, cnt, addr); + return; + } + + if (use_lockless_fast_path()) { + freelist = READ_ONCE(c->freelist); set_freepointer(s, tail_obj, freelist); @@ -3491,16 +3496,8 @@ static __always_inline void do_slab_free(struct kmem_cache *s, note_cmpxchg_failure("slab_free", s, tid); goto redo; } -#else /* CONFIG_PREEMPT_RT */ - /* - * We cannot use the lockless fastpath on PREEMPT_RT because if - * a slowpath has taken the local_lock_irqsave(), it is not - * protected against a fast path operation in an irq handler. So - * we need to take the local_lock. We shouldn't simply defer to - * __slab_free() as that wouldn't use the cpu freelist at all. - */ - void **freelist; - + } else { + /* Update the free list under the local lock */ local_lock(&s->cpu_slab->lock); c = this_cpu_ptr(s->cpu_slab); if (unlikely(slab != c->slab)) { @@ -3515,11 +3512,8 @@ static __always_inline void do_slab_free(struct kmem_cache *s, c->tid = next_tid(tid); local_unlock(&s->cpu_slab->lock); -#endif - stat(s, FREE_FASTPATH); - } else - __slab_free(s, slab, head, tail_obj, cnt, addr); - + } + stat(s, FREE_FASTPATH); } static __always_inline void slab_free(struct kmem_cache *s, struct slab *slab,