From patchwork Fri Feb 14 16:27:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13975226 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61598C021A6 for ; Fri, 14 Feb 2025 16:28:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 629B36B0092; Fri, 14 Feb 2025 11:28:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5B2B3280003; Fri, 14 Feb 2025 11:28:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 31A54280002; Fri, 14 Feb 2025 11:28:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E7BAE6B0092 for ; Fri, 14 Feb 2025 11:28:01 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7E1961206C5 for ; Fri, 14 Feb 2025 16:27:46 +0000 (UTC) X-FDA: 83119081332.10.324EFB7 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf06.hostedemail.com (Postfix) with ESMTP id 3AF05180002 for ; Fri, 14 Feb 2025 16:27:44 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=none; spf=pass (imf06.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739550464; a=rsa-sha256; cv=none; b=jiuAh+0MEVejXOs+ASkpQeriTrYHVwFjG9X529W38SjL6rUlCSkhrgDLWO2WtoWT4C8Yqp YoI5v8Lo1QVvueSvnDTPlLTH177zMlJTgD1hDv9IW4s+B6oqtaziSPyAUNZkwbkox7+vcD UxhDmUBOPMoqwohwCUTloO2ShIMQogo= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; spf=pass (imf06.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739550464; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CUWP95JP941w8JGTSAzIZmOj8I1HZaIRyrH+fUtL7DY=; b=3d1LQfvYQy/3ydOszGHqTxrxvvishcG1vyOMdqfDn2w5Pno9G3z3Tk+/UPc8WDeGEKeP77 UFYaNjl1ZVWnmGYXSJIhVGb7Ht55vghSr8FVDlHrvKZJ846SGt6qk6D4W+VYpVAG3aIa+o U3plYs3jqj4yjeaH2ykKdPIdWGpa7Is= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id D0C8B1F391; Fri, 14 Feb 2025 16:27:42 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id B7FB813AF0; Fri, 14 Feb 2025 16:27:42 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id +NqlLP5ur2eHSAAAD6G6ig (envelope-from ); Fri, 14 Feb 2025 16:27:42 +0000 From: Vlastimil Babka Date: Fri, 14 Feb 2025 17:27:41 +0100 Subject: [PATCH RFC v2 05/10] slab: switch percpu sheaves locking to localtry_lock MIME-Version: 1.0 Message-Id: <20250214-slub-percpu-caches-v2-5-88592ee0966a@suse.cz> References: <20250214-slub-percpu-caches-v2-0-88592ee0966a@suse.cz> In-Reply-To: <20250214-slub-percpu-caches-v2-0-88592ee0966a@suse.cz> To: Suren Baghdasaryan , "Liam R. Howlett" , Christoph Lameter , David Rientjes Cc: Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Uladzislau Rezki , linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, maple-tree@lists.infradead.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Stat-Signature: zo7mjfo9nh979pykcgwc7a6q1e3x1z34 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 3AF05180002 X-Rspam-User: X-HE-Tag: 1739550464-536729 X-HE-Meta: U2FsdGVkX1/qXcK+nWMRqVxxW4zBLz3aNDnWwWXQG41PGrolf40Xrxvag1LRkDDVNWdAMsz4cHG8J6uzUlyFJualTeDK8QrcRmCu9N+zCHgDMTR36M1K+YiRdsEnZtoZ+FH4gMc9+Px05k5k2OQTKJ8VYjYAqR2BF1RHlpSj4sVMXBHh3FNw1r6ktnY2tPT7IgDcumS1vMApi2wooHI5T/oQe9DiINMi+Zoy/lHtPhZxcPVBufR5FFxUukLIHG6rPeUKzfInd8MVFTcQJOU3I0Cv2N7GAitP5BRPJMYhYYhsgcy82hSchNqtGf8/raMDFLdL1C9M8o3K9IPUrFpcZt0+022X2Jz7JzZN/sEJlhmMED6P99LtE/A9F/4wqXBqu15SyVlcaiNYojqE0XAVoIbYYVo4ZnFrZkEs1aSKB9kt+irlSeL5a5yScVKUHMrfqybutCfCw3wbFqo7b6xo2UK/UMsQEc7nuBL9ohRUBCY3i+XTl1GhLfxTYxSSUfpHa2n4N70LygGNRHHsHcMTuYJt2qRZlfbvweoLU/yw+0TIcs2J6bhuHYLrsxeg0AgeJPhwrN5uMBi4IWB1pjpWXD0sYS5g6WUZG0u5tQXhpOsdJjfQGZRxRDtYCejDsa52850Vqlk8oJUSUTfutSPyxGr+LltHn4BahgQEXz/r53MPQogu52QZkV70wy0HLiYY2mDwhx5jukPwyQVZJcvRdu+AWB2xhu+dYY2aIE0NhRBZRgZ6UbIuW4/CEsB11+1oPAJpXrthPBU7Ry8ULz31PlM6DKWwlHiFMno5l1Z6KJclkVBBwc2vVX7tpweWqnjdj6Taq+PB/UBPn39iAQW0GgJYCjoFhjFRFtTEzp6am1ycPHQk5OSI9ncMDvR64B/HYvmOTaX/KOJm8/Z4//X3wyvBsuAdV4yLJQW/gIVGkllIOUJ9iRcGrHy275t/iKGE0M58ztPKXQKyYGLMR64 xbQJkYdb Cn1KcHmjMl9hUm6eiPLlcy/BdTLnYo11mGKBXVQgmnJM8+HYSFcD9wBipcUKPkQff0xXsO6al5AjY7n8QilRsocemdeqMywal8f5m1rgNkMxrfoHPJIuEeu830aAK8CFf24/SCZ1H4Ws9vyg/GUAz19n2muUZqsH6db/+2vD37L9Dd2029gieWVhyS99hE/+1UJfRL5CmulZB8KwYeDw8wNW9FEu6jNAuSRXZWiIBLDfuBvY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Instead of local_lock_irqsave(), use localtry_trylock() when potential callers include irq context, and localtry_lock() otherwise (such as when we already know the gfp flags allow blocking). This should reduce the locking (due to irq disabling/enabling) overhead. Failing to use percpu sheaves in an irq due to preempting an already locked user of sheaves should be rare so it's a favorable tradeoff. Signed-off-by: Vlastimil Babka --- mm/slub.c | 122 ++++++++++++++++++++++++++++++++++++++------------------------ 1 file changed, 76 insertions(+), 46 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 40175747212fefb27137309b27571abe8d0966e2..3d7345e7e938d53950ed0d6abe8eb0e93cf8f5b1 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -450,7 +450,7 @@ struct slab_sheaf { }; struct slub_percpu_sheaves { - local_lock_t lock; + localtry_lock_t lock; struct slab_sheaf *main; /* never NULL when unlocked */ struct slab_sheaf *spare; /* empty or full, may be NULL */ struct slab_sheaf *rcu_free; @@ -2529,16 +2529,19 @@ static struct slab_sheaf *alloc_full_sheaf(struct kmem_cache *s, gfp_t gfp) static void __kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p); -static void sheaf_flush_main(struct kmem_cache *s) +/* returns true if at least partially flushed */ +static bool sheaf_flush_main(struct kmem_cache *s) { struct slub_percpu_sheaves *pcs; unsigned int batch, remaining; void *objects[PCS_BATCH_MAX]; struct slab_sheaf *sheaf; - unsigned long flags; + bool ret = false; next_batch: - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + if (!localtry_trylock(&s->cpu_sheaves->lock)) + return ret; + pcs = this_cpu_ptr(s->cpu_sheaves); sheaf = pcs->main; @@ -2549,14 +2552,18 @@ static void sheaf_flush_main(struct kmem_cache *s) remaining = sheaf->size; - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); __kmem_cache_free_bulk(s, batch, &objects[0]); stat_add(s, SHEAF_FLUSH_MAIN, batch); + ret = true; + if (remaining) goto next_batch; + + return ret; } static void sheaf_flush(struct kmem_cache *s, struct slab_sheaf *sheaf) @@ -2593,6 +2600,8 @@ static void rcu_free_sheaf_nobarn(struct rcu_head *head) * Caller needs to make sure migration is disabled in order to fully flush * single cpu's sheaves * + * must not be called from an irq + * * flushing operations are rare so let's keep it simple and flush to slabs * directly, skipping the barn */ @@ -2600,9 +2609,8 @@ static void pcs_flush_all(struct kmem_cache *s) { struct slub_percpu_sheaves *pcs; struct slab_sheaf *spare, *rcu_free; - unsigned long flags; - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + localtry_lock(&s->cpu_sheaves->lock); pcs = this_cpu_ptr(s->cpu_sheaves); spare = pcs->spare; @@ -2611,7 +2619,7 @@ static void pcs_flush_all(struct kmem_cache *s) rcu_free = pcs->rcu_free; pcs->rcu_free = NULL; - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); if (spare) { sheaf_flush(s, spare); @@ -4554,10 +4562,11 @@ static __fastpath_inline void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp) { struct slub_percpu_sheaves *pcs; - unsigned long flags; void *object; - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + if (!localtry_trylock(&s->cpu_sheaves->lock)) + return NULL; + pcs = this_cpu_ptr(s->cpu_sheaves); if (unlikely(pcs->main->size == 0)) { @@ -4590,7 +4599,7 @@ void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp) } } - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); if (!can_alloc) return NULL; @@ -4612,7 +4621,11 @@ void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp) if (!full) return NULL; - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + /* + * we can reach here only when gfpflags_allow_blocking + * so this must not be an irq + */ + localtry_lock(&s->cpu_sheaves->lock); pcs = this_cpu_ptr(s->cpu_sheaves); /* @@ -4646,7 +4659,7 @@ void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp) do_alloc: object = pcs->main->objects[--pcs->main->size]; - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); stat(s, ALLOC_PCS); @@ -4658,12 +4671,13 @@ unsigned int alloc_from_pcs_bulk(struct kmem_cache *s, size_t size, void **p) { struct slub_percpu_sheaves *pcs; struct slab_sheaf *main; - unsigned long flags; unsigned int allocated = 0; unsigned int batch; next_batch: - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + if (!localtry_trylock(&s->cpu_sheaves->lock)) + return allocated; + pcs = this_cpu_ptr(s->cpu_sheaves); if (unlikely(pcs->main->size == 0)) { @@ -4683,7 +4697,7 @@ unsigned int alloc_from_pcs_bulk(struct kmem_cache *s, size_t size, void **p) goto do_alloc; } - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); /* * Once full sheaves in barn are depleted, let the bulk @@ -4701,7 +4715,7 @@ unsigned int alloc_from_pcs_bulk(struct kmem_cache *s, size_t size, void **p) main->size -= batch; memcpy(p, main->objects + main->size, batch * sizeof(void *)); - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); stat_add(s, ALLOC_PCS, batch); @@ -5121,13 +5135,14 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, * The object is expected to have passed slab_free_hook() already. */ static __fastpath_inline -void free_to_pcs(struct kmem_cache *s, void *object) +bool free_to_pcs(struct kmem_cache *s, void *object) { struct slub_percpu_sheaves *pcs; - unsigned long flags; restart: - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + if (!localtry_trylock(&s->cpu_sheaves->lock)) + return false; + pcs = this_cpu_ptr(s->cpu_sheaves); if (unlikely(pcs->main->size == s->sheaf_capacity)) { @@ -5162,7 +5177,7 @@ void free_to_pcs(struct kmem_cache *s, void *object) struct slab_sheaf *to_flush = pcs->spare; pcs->spare = NULL; - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); sheaf_flush(s, to_flush); empty = to_flush; @@ -5170,17 +5185,27 @@ void free_to_pcs(struct kmem_cache *s, void *object) } alloc_empty: - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); empty = alloc_empty_sheaf(s, GFP_NOWAIT); if (!empty) { - sheaf_flush_main(s); - goto restart; + if (sheaf_flush_main(s)) + goto restart; + else + return false; } got_empty: - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + if (!localtry_trylock(&s->cpu_sheaves->lock)) { + struct node_barn *barn; + + barn = get_node(s, numa_mem_id())->barn; + + barn_put_empty_sheaf(barn, empty, true); + return false; + } + pcs = this_cpu_ptr(s->cpu_sheaves); /* @@ -5209,9 +5234,11 @@ void free_to_pcs(struct kmem_cache *s, void *object) do_free: pcs->main->objects[pcs->main->size++] = object; - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); stat(s, FREE_PCS); + + return true; } static void __rcu_free_sheaf_prepare(struct kmem_cache *s, @@ -5270,9 +5297,10 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj) { struct slub_percpu_sheaves *pcs; struct slab_sheaf *rcu_sheaf; - unsigned long flags; - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + if (!localtry_trylock(&s->cpu_sheaves->lock)) + goto fail; + pcs = this_cpu_ptr(s->cpu_sheaves); if (unlikely(!pcs->rcu_free)) { @@ -5286,16 +5314,16 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj) goto do_free; } - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); empty = alloc_empty_sheaf(s, GFP_NOWAIT); - if (!empty) { - stat(s, FREE_RCU_SHEAF_FAIL); - return false; - } + if (!empty) + goto fail; + + if (!localtry_trylock(&s->cpu_sheaves->lock)) + goto fail; - local_lock_irqsave(&s->cpu_sheaves->lock, flags); pcs = this_cpu_ptr(s->cpu_sheaves); if (unlikely(pcs->rcu_free)) @@ -5311,19 +5339,22 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj) rcu_sheaf->objects[rcu_sheaf->size++] = obj; if (likely(rcu_sheaf->size < s->sheaf_capacity)) { - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); stat(s, FREE_RCU_SHEAF); return true; } pcs->rcu_free = NULL; - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf); stat(s, FREE_RCU_SHEAF); - return true; + +fail: + stat(s, FREE_RCU_SHEAF_FAIL); + return false; } /* @@ -5335,7 +5366,6 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p) { struct slub_percpu_sheaves *pcs; struct slab_sheaf *main; - unsigned long flags; unsigned int batch, i = 0; bool init; @@ -5358,7 +5388,9 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p) } next_batch: - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + if (!localtry_trylock(&s->cpu_sheaves->lock)) + goto fallback; + pcs = this_cpu_ptr(s->cpu_sheaves); if (unlikely(pcs->main->size == s->sheaf_capacity)) { @@ -5389,13 +5421,13 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p) } no_empty: - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); /* * if we depleted all empty sheaves in the barn or there are too * many full sheaves, free the rest to slab pages */ - +fallback: __kmem_cache_free_bulk(s, size, p); return; } @@ -5407,7 +5439,7 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p) memcpy(main->objects + main->size, p, batch * sizeof(void *)); main->size += batch; - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); stat_add(s, FREE_PCS, batch); @@ -5507,9 +5539,7 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object, if (unlikely(!slab_free_hook(s, object, slab_want_init_on_free(s), false))) return; - if (s->cpu_sheaves) - free_to_pcs(s, object); - else + if (!s->cpu_sheaves || !free_to_pcs(s, object)) do_slab_free(s, slab, object, object, 1, addr); } @@ -6288,7 +6318,7 @@ static int init_percpu_sheaves(struct kmem_cache *s) pcs = per_cpu_ptr(s->cpu_sheaves, cpu); - local_lock_init(&pcs->lock); + localtry_lock_init(&pcs->lock); nid = cpu_to_mem(cpu);