From patchwork Thu Aug 25 07:51:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 12954313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82965C04AA5 for ; Thu, 25 Aug 2022 07:51:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 21AD3940007; Thu, 25 Aug 2022 03:51:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A4366B0075; Thu, 25 Aug 2022 03:51:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 01E0E940007; Thu, 25 Aug 2022 03:51:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E4CC56B0074 for ; Thu, 25 Aug 2022 03:51:40 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id BD9FC1C4B80 for ; Thu, 25 Aug 2022 07:51:40 +0000 (UTC) X-FDA: 79837345560.08.9FE0EAF Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf22.hostedemail.com (Postfix) with ESMTP id 2FE87C0002 for ; Thu, 25 Aug 2022 07:51:40 +0000 (UTC) Date: Thu, 25 Aug 2022 09:51:36 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1661413897; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=nQIEg4T59jEH7CtskgPVxtoZ3qrIbcaq3GNOPUVBBiI=; b=wg0l3If88bTfLBmeq/1o7x0m6kS14mmvDg/gl1cnfjCaGnqzlDGeCfhyth8120le0zbe4Z lE9h+HY8+tjzLKXqavzZwlVxerT4WjHUDkFsHImfEWzKOjRD1YW16mHLywtST0zhe2TODc 2rq1gmCLLZKYq3n8Lr4XiWKlc64XMWX2YX5PE9tRwJBfJJM+xR7BgVu619GgZJcfoxtoU0 2GHYWXSQ3TQUaErP+nsPEd9wQSWZco+W+prHNijUqDH2kIpKXf8//afJnnKGi/7V/WQjKo fsFop4zir32M7j8Cc2uhgcK4ScVB71XOj9zDaBJbMmabk4i01D3fRUK9kjW3xQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1661413897; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=nQIEg4T59jEH7CtskgPVxtoZ3qrIbcaq3GNOPUVBBiI=; b=jaVKpxy44iXyulF+AyEkU5/ShE8TfUfYUgIzbDZdq65aSBC1ZCp8k4E3zkGh4GeJlBSpVJ 8uZwLyhBr3/dSZDA== From: Sebastian Andrzej Siewior To: Vlastimil Babka Cc: Rongwei Wang , Christoph Lameter , Joonsoo Kim , David Rientjes , Pekka Enberg , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , linux-mm@kvack.org, Thomas Gleixner , Mike Galbraith , Andrew Morton Subject: [PATCH 6/5] slub: Make PREEMPT_RT support less convoluted Message-ID: References: <20220823170400.26546-1-vbabka@suse.cz> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20220823170400.26546-1-vbabka@suse.cz> ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661413900; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nQIEg4T59jEH7CtskgPVxtoZ3qrIbcaq3GNOPUVBBiI=; b=6MyoDcsJWyexqGSZMfRAbdLodv1NomGfjsK4c0hlxVtaKcvTurnRiQIKOyvqfncSG2fa0N X09HdrekiiD7FARQQz68gG4tNOllOhfKAZEd9uK6Oh1OkVkMpD+VRqH3lABBdITdA2yZc/ nktSPX7s3XNdKCm10NO37G3VAGPJb2c= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=wg0l3If8; dkim=pass header.d=linutronix.de header.s=2020e header.b=jaVKpxy4; spf=pass (imf22.hostedemail.com: domain of bigeasy@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de; dmarc=pass (policy=none) header.from=linutronix.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661413900; a=rsa-sha256; cv=none; b=Wr5iIdNvo+/o1sTKbyUjUaWXHqpj4NZ5vUA/j5WFNg9jfJz64ByWCDxuI9iwOugg7tA7rL NxS2B4/DMvMbkvStiCgaSThH0aR76uQjaZDc9eYmw+7/YFyGfzXIpXYos8RQJDMIDXpYLo jUOI0yew7L2vRaAlND+NznpS6JMglb4= Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=wg0l3If8; dkim=pass header.d=linutronix.de header.s=2020e header.b=jaVKpxy4; spf=pass (imf22.hostedemail.com: domain of bigeasy@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de; dmarc=pass (policy=none) header.from=linutronix.de X-Rspam-User: X-Stat-Signature: d8imte75bt4iag47w41z3amd1qfk5nb7 X-Rspamd-Queue-Id: 2FE87C0002 X-Rspamd-Server: rspam03 X-HE-Tag: 1661413900-563170 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Thomas Gleixner The slub code already has a few helpers depending on PREEMPT_RT. Add a few more and get rid of the CONFIG_PREEMPT_RT conditionals all over the place. No functional change. Signed-off-by: Thomas Gleixner Cc: Andrew Morton Cc: Christoph Lameter Cc: David Rientjes Cc: Joonsoo Kim Cc: Pekka Enberg Cc: Vlastimil Babka Cc: linux-mm@kvack.org Signed-off-by: Sebastian Andrzej Siewior Acked-by: Peter Zijlstra (Intel) Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- Vlastimil, does it work for you to include this patch in your series? It depends now on your series :) It has this USE_LOCKLESS_FAST_PATH() Linus asked about so we should be good. mm/slub.c | 56 ++++++++++++++++++++++++-------------------------------- 1 file changed, 24 insertions(+), 32 deletions(-) --- a/mm/slub.c +++ b/mm/slub.c @@ -104,9 +104,11 @@ * except the stat counters. This is a percpu structure manipulated only by * the local cpu, so the lock protects against being preempted or interrupted * by an irq. Fast path operations rely on lockless operations instead. - * On PREEMPT_RT, the local lock does not actually disable irqs (and thus - * prevent the lockless operations), so fastpath operations also need to take - * the lock and are no longer lockless. + * + * On PREEMPT_RT, the local lock neither disables interrupts nor preemption + * which means the lockless fastpath cannot be used as it might interfere with + * an in-progress slow path operations. In this case the local lock is always + * taken but it still utilizes the freelist for the common operations. * * lockless fastpaths * @@ -167,8 +169,9 @@ * function call even on !PREEMPT_RT, use inline preempt_disable() there. */ #ifndef CONFIG_PREEMPT_RT -#define slub_get_cpu_ptr(var) get_cpu_ptr(var) -#define slub_put_cpu_ptr(var) put_cpu_ptr(var) +#define slub_get_cpu_ptr(var) get_cpu_ptr(var) +#define slub_put_cpu_ptr(var) put_cpu_ptr(var) +#define USE_LOCKLESS_FAST_PATH() (true) #else #define slub_get_cpu_ptr(var) \ ({ \ @@ -180,6 +183,7 @@ do { \ (void)(var); \ migrate_enable(); \ } while (0) +#define USE_LOCKLESS_FAST_PATH() (false) #endif #ifdef CONFIG_SLUB_DEBUG @@ -474,7 +478,7 @@ static inline bool __cmpxchg_double_slab void *freelist_new, unsigned long counters_new, const char *n) { - if (!IS_ENABLED(CONFIG_PREEMPT_RT)) + if (USE_LOCKLESS_FAST_PATH()) lockdep_assert_irqs_disabled(); #if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \ defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) @@ -3287,14 +3291,8 @@ static __always_inline void *slab_alloc_ object = c->freelist; slab = c->slab; - /* - * We cannot use the lockless fastpath on PREEMPT_RT because if a - * slowpath has taken the local_lock_irqsave(), it is not protected - * against a fast path operation in an irq handler. So we need to take - * the slow path which uses local_lock. It is still relatively fast if - * there is a suitable cpu freelist. - */ - if (IS_ENABLED(CONFIG_PREEMPT_RT) || + + if (!USE_LOCKLESS_FAST_PATH() || unlikely(!object || !slab || !node_match(slab, node))) { object = __slab_alloc(s, gfpflags, node, addr, c); } else { @@ -3554,6 +3552,7 @@ static __always_inline void do_slab_free void *tail_obj = tail ? : head; struct kmem_cache_cpu *c; unsigned long tid; + void **freelist; redo: /* @@ -3568,9 +3567,13 @@ static __always_inline void do_slab_free /* Same with comment on barrier() in slab_alloc_node() */ barrier(); - if (likely(slab == c->slab)) { -#ifndef CONFIG_PREEMPT_RT - void **freelist = READ_ONCE(c->freelist); + if (unlikely(slab != c->slab)) { + __slab_free(s, slab, head, tail_obj, cnt, addr); + return; + } + + if (USE_LOCKLESS_FAST_PATH()) { + freelist = READ_ONCE(c->freelist); set_freepointer(s, tail_obj, freelist); @@ -3582,16 +3585,8 @@ static __always_inline void do_slab_free note_cmpxchg_failure("slab_free", s, tid); goto redo; } -#else /* CONFIG_PREEMPT_RT */ - /* - * We cannot use the lockless fastpath on PREEMPT_RT because if - * a slowpath has taken the local_lock_irqsave(), it is not - * protected against a fast path operation in an irq handler. So - * we need to take the local_lock. We shouldn't simply defer to - * __slab_free() as that wouldn't use the cpu freelist at all. - */ - void **freelist; - + } else { + /* Update the free list under the local lock */ local_lock(&s->cpu_slab->lock); c = this_cpu_ptr(s->cpu_slab); if (unlikely(slab != c->slab)) { @@ -3606,11 +3601,8 @@ static __always_inline void do_slab_free c->tid = next_tid(tid); local_unlock(&s->cpu_slab->lock); -#endif - stat(s, FREE_FASTPATH); - } else - __slab_free(s, slab, head, tail_obj, cnt, addr); - + } + stat(s, FREE_FASTPATH); } static __always_inline void slab_free(struct kmem_cache *s, struct slab *slab,