From patchwork Fri Feb 14 16:27:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13975223 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACA18C021A6 for ; Fri, 14 Feb 2025 16:27:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 256F86B0085; Fri, 14 Feb 2025 11:27:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 207F66B0088; Fri, 14 Feb 2025 11:27:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A8006B0089; Fri, 14 Feb 2025 11:27:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E21416B0085 for ; Fri, 14 Feb 2025 11:27:55 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 72BE9C0674 for ; Fri, 14 Feb 2025 16:27:52 +0000 (UTC) X-FDA: 83119081584.05.F00BBB6 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf07.hostedemail.com (Postfix) with ESMTP id 4DCDF40007 for ; Fri, 14 Feb 2025 16:27:50 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf07.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739550470; a=rsa-sha256; cv=none; b=kNIgu74rAbnReOX3fMb82KJL1YoKTEe48AMowdO0WJauV/QesIfGoDdk99ozCYuQW1hmmA z3OG+ai+1esZcn2xkhjDr7JhiexqeH+al9an/bXhtga7Xk76T7vb4aaCcARyMBkxC+E6jy sJH/5aPG9DqHappg+iu7YUEKKb3pF3A= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf07.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739550470; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QEa7rQvQJybxs8PWOc++9/bwy/3U8sG6HhgmTzdZL1s=; b=El9ucKdjXluQsBbRqaml1InOjgWzwjTLx1aZN50K6PJk7Eh/cDvodadbeQFQ8S6CDCmAZ7 l3nhfPh3YUKSTB+icyaqlWz/2eSrnHMwysLFs71XsokjYrDsAMlwdM9Ne74MSfcaSCZ8LR BPecf0DQ3NqELLZXB+u/VWCOapydndE= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 931FD2117C; Fri, 14 Feb 2025 16:27:42 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 768B213AF0; Fri, 14 Feb 2025 16:27:42 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id OMmlHP5ur2eHSAAAD6G6ig (envelope-from ); Fri, 14 Feb 2025 16:27:42 +0000 From: Vlastimil Babka Date: Fri, 14 Feb 2025 17:27:38 +0100 Subject: [PATCH RFC v2 02/10] slab: add sheaf support for batching kfree_rcu() operations MIME-Version: 1.0 Message-Id: <20250214-slub-percpu-caches-v2-2-88592ee0966a@suse.cz> References: <20250214-slub-percpu-caches-v2-0-88592ee0966a@suse.cz> In-Reply-To: <20250214-slub-percpu-caches-v2-0-88592ee0966a@suse.cz> To: Suren Baghdasaryan , "Liam R. Howlett" , Christoph Lameter , David Rientjes Cc: Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Uladzislau Rezki , linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, maple-tree@lists.infradead.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Rspam-User: X-Rspamd-Queue-Id: 4DCDF40007 X-Rspamd-Server: rspam12 X-Stat-Signature: 9hjkyn88pm4ok7c6gdyiwn1cau4mrrjs X-HE-Tag: 1739550470-696135 X-HE-Meta: U2FsdGVkX1/Sy5CL6vD/wTzH0izcii1cvzA0C0NcGYbqNXOoEGY3eVsOCcEiRQbexDccLgl3c2ntVGHLHxYGL8WeQwUc1S2bm+Qxviy7ePoewrfx6Zc0xgf/kjhP+u+EpXvEdScT3TxDtnw+jDmVNJl/CNihQfnTRjUGwuJjNI4ifIwocnjV9JOTd9Bwb0Z2bZ2cO9CeOeas4pYDqCSQ+K4hJJvfWSadbaeGJEYlfeldNUkVbJeEeIuWWpAmfim3ii3QquMqwNiaz7Z/2fatAzPVKKZ+kPNrAAZCMdIORyD6PDQLEhgKVuVs5qHrMJwet8fOLqu9VIz7su5vk/cpaPrVERV5iG0hPUpR5lbXyFn9mL2PU16dLpuoa8jhoI68WT6qlF1r2x1+/10wSN9ftVlAOUbNj51s8X7vjcCFvhlfJl75OV03NmwyANw/8oXA3fxTZcKVBQHm//VWh3T7egbnaEG87y6hEg1jHcBhVkcsIPfjB95sKhh6iSLYjRhGjczUb5o9ITG4/hX5dFWtz4aE/x+5L7wrhio5NHdc6au/pKulRWUawFq3hwXq/mxMMHTwuCta4nq8MUpUeUU2uhJSr3JnpHRBV6HkLsvn3xsj2SiiBB2g01hUgbXqczhFntsb62CLBK675kmkp/Ks6iNDVfQStUfrzv28tGejRJkOKhQtcM8V5hWV4OnIhUqANWngpd2l5tJxQ2oVD5uHm1L5mtR8wWg0+woiP+q1fjlVufFbJaAxIUsFtJqzsJ1UxJ2dX2QmCmqO/bWjJVhoAhAoLqB/WPXBMEuF/PDJYJuuw8IyuIs96xrEr0UsHfwLHelmkjdNUytl+y1tuWlG+oCDH5O78y+PWMU+NgzpbD0+WRuggWMYqIjkWmSnpxCuKrxZqBLVqvVt6FOT7qS/vWkxOjWsnY3vSedexu1ZHNYIIskVDi0WspbYCG4Ft6Q7n0ObYPa+WARNb+3rdjs 5y1/k0G/ gwzJgKHeGhAvplc10obJa70H+lHIxjJxYAeW8Qh327qfDjUsfmuJ7FYX+Wt+ppxE8vKrl9fXlBajBMEbs9KXLZ6LSejLOB3rLHuRjBLX99JrCl0xTAETmKUKnLjsaLw0eUAGNDGJalE4CNCe1wRdGR3jRSB5mz9+a51GOLr5Rzm5bHBtPO0JQtn8np0ZNGnbq+asgEAbhOZk6WIDSyqdDW4nJG++B9ZeqDxtcEdbdKOeIyB0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Extend the sheaf infrastructure for more efficient kfree_rcu() handling. For caches with sheaves, on each cpu maintain a rcu_free sheaf in addition to main and spare sheaves. kfree_rcu() operations will try to put objects on this sheaf. Once full, the sheaf is detached and submitted to call_rcu() with a handler that will try to put in in the barn, or flush to slab pages using bulk free, when the barn is full. Then a new empty sheaf must be obtained to put more objects there. It's possible that no free sheaves are available to use for a new rcu_free sheaf, and the allocation in kfree_rcu() context can only use GFP_NOWAIT and thus may fail. In that case, fall back to the existing kfree_rcu() machinery. Expected advantages: - batching the kfree_rcu() operations, that could eventually replace the existing batching - sheaves can be reused for allocations via barn instead of being flushed to slabs, which is more efficient - this includes cases where only some cpus are allowed to process rcu callbacks (Android) Possible disadvantage: - objects might be waiting for more than their grace period (it is determined by the last object freed into the sheaf), increasing memory usage - but the existing batching does that too? Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny implementation favors smaller memory footprint over performance. Signed-off-by: Vlastimil Babka --- mm/slab.h | 2 + mm/slab_common.c | 21 ++++++++ mm/slub.c | 151 +++++++++++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 170 insertions(+), 4 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 8daaec53b6ecfc44171191d421adb12e5cba2c58..94e9959e1aefa350d3d74e3f5309fde7a5cf2ec8 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -459,6 +459,8 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s) return !(s->flags & (SLAB_CACHE_DMA|SLAB_ACCOUNT|SLAB_RECLAIM_ACCOUNT)); } +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj); + /* Legal flag mask for kmem_cache_create(), for various configurations */ #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \ SLAB_CACHE_DMA32 | SLAB_PANIC | \ diff --git a/mm/slab_common.c b/mm/slab_common.c index ceeefb287899a82f30ad79b403556001c1860311..c6853450ed74160cfcb497c09f92c1f9f7b12629 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1613,6 +1613,24 @@ static void kfree_rcu_work(struct work_struct *work) kvfree_rcu_list(head); } +static bool kfree_rcu_sheaf(void *obj) +{ + struct kmem_cache *s; + struct folio *folio; + struct slab *slab; + + folio = virt_to_folio(obj); + if (unlikely(!folio_test_slab(folio))) + return false; + + slab = folio_slab(folio); + s = slab->slab_cache; + if (s->cpu_sheaves) + return __kfree_rcu_sheaf(s, obj); + + return false; +} + static bool need_offload_krc(struct kfree_rcu_cpu *krcp) { @@ -1957,6 +1975,9 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr) if (!head) might_sleep(); + if (kfree_rcu_sheaf(ptr)) + return; + // Queue the object but don't yet schedule the batch. if (debug_rcu_head_queue(ptr)) { // Probable double kfree_rcu(), just leak. diff --git a/mm/slub.c b/mm/slub.c index c06734912972b799f537359f7fe6a750918ffe9e..40175747212fefb27137309b27571abe8d0966e2 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -350,6 +350,8 @@ enum stat_item { ALLOC_FASTPATH, /* Allocation from cpu slab */ ALLOC_SLOWPATH, /* Allocation by getting a new cpu slab */ FREE_PCS, /* Free to percpu sheaf */ + FREE_RCU_SHEAF, /* Free to rcu_free sheaf */ + FREE_RCU_SHEAF_FAIL, /* Failed to free to a rcu_free sheaf */ FREE_FASTPATH, /* Free to cpu slab */ FREE_SLOWPATH, /* Freeing not to cpu slab */ FREE_FROZEN, /* Freeing to frozen slab */ @@ -2569,6 +2571,24 @@ static void sheaf_flush(struct kmem_cache *s, struct slab_sheaf *sheaf) sheaf->size = 0; } +static void __rcu_free_sheaf_prepare(struct kmem_cache *s, + struct slab_sheaf *sheaf); + +static void rcu_free_sheaf_nobarn(struct rcu_head *head) +{ + struct slab_sheaf *sheaf; + struct kmem_cache *s; + + sheaf = container_of(head, struct slab_sheaf, rcu_head); + s = sheaf->cache; + + __rcu_free_sheaf_prepare(s, sheaf); + + sheaf_flush(s, sheaf); + + free_empty_sheaf(s, sheaf); +} + /* * Caller needs to make sure migration is disabled in order to fully flush * single cpu's sheaves @@ -2598,8 +2618,8 @@ static void pcs_flush_all(struct kmem_cache *s) free_empty_sheaf(s, spare); } - // TODO: handle rcu_free - BUG_ON(rcu_free); + if (rcu_free) + call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn); sheaf_flush_main(s); } @@ -2616,8 +2636,10 @@ static void __pcs_flush_all_cpu(struct kmem_cache *s, unsigned int cpu) pcs->spare = NULL; } - // TODO: handle rcu_free - BUG_ON(pcs->rcu_free); + if (pcs->rcu_free) { + call_rcu(&pcs->rcu_free->rcu_head, rcu_free_sheaf_nobarn); + pcs->rcu_free = NULL; + } sheaf_flush_main(s); } @@ -5192,6 +5214,118 @@ void free_to_pcs(struct kmem_cache *s, void *object) stat(s, FREE_PCS); } +static void __rcu_free_sheaf_prepare(struct kmem_cache *s, + struct slab_sheaf *sheaf) +{ + bool init = slab_want_init_on_free(s); + void **p = &sheaf->objects[0]; + unsigned int i = 0; + + while (i < sheaf->size) { + struct slab *slab = virt_to_slab(p[i]); + + memcg_slab_free_hook(s, slab, p + i, 1); + alloc_tagging_slab_free_hook(s, slab, p + i, 1); + + if (unlikely(!slab_free_hook(s, p[i], init, false))) { + p[i] = p[--sheaf->size]; + continue; + } + + i++; + } +} + +static void rcu_free_sheaf(struct rcu_head *head) +{ + struct slab_sheaf *sheaf; + struct node_barn *barn; + struct kmem_cache *s; + + sheaf = container_of(head, struct slab_sheaf, rcu_head); + + s = sheaf->cache; + + __rcu_free_sheaf_prepare(s, sheaf); + + barn = get_node(s, numa_mem_id())->barn; + + /* due to slab_free_hook() */ + if (unlikely(sheaf->size == 0)) + goto empty; + + if (!barn_put_full_sheaf(barn, sheaf, false)) + return; + + sheaf_flush(s, sheaf); + +empty: + if (!barn_put_empty_sheaf(barn, sheaf, false)) + return; + + free_empty_sheaf(s, sheaf); +} + +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj) +{ + struct slub_percpu_sheaves *pcs; + struct slab_sheaf *rcu_sheaf; + unsigned long flags; + + local_lock_irqsave(&s->cpu_sheaves->lock, flags); + pcs = this_cpu_ptr(s->cpu_sheaves); + + if (unlikely(!pcs->rcu_free)) { + + struct slab_sheaf *empty; + + empty = barn_get_empty_sheaf(pcs->barn); + + if (empty) { + pcs->rcu_free = empty; + goto do_free; + } + + local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + + empty = alloc_empty_sheaf(s, GFP_NOWAIT); + + if (!empty) { + stat(s, FREE_RCU_SHEAF_FAIL); + return false; + } + + local_lock_irqsave(&s->cpu_sheaves->lock, flags); + pcs = this_cpu_ptr(s->cpu_sheaves); + + if (unlikely(pcs->rcu_free)) + barn_put_empty_sheaf(pcs->barn, empty, true); + else + pcs->rcu_free = empty; + } + +do_free: + + rcu_sheaf = pcs->rcu_free; + + rcu_sheaf->objects[rcu_sheaf->size++] = obj; + + if (likely(rcu_sheaf->size < s->sheaf_capacity)) { + local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + stat(s, FREE_RCU_SHEAF); + return true; + } + + pcs->rcu_free = NULL; + local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + + call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf); + + stat(s, FREE_RCU_SHEAF); + + return true; +} + /* * Bulk free objects to the percpu sheaves. * Unlike free_to_pcs() this includes the calls to all necessary hooks @@ -6522,6 +6656,11 @@ int __kmem_cache_shutdown(struct kmem_cache *s) struct kmem_cache_node *n; flush_all_cpus_locked(s); + + /* we might have rcu sheaves in flight */ + if (s->cpu_sheaves) + rcu_barrier(); + /* Attempt to free all objects */ for_each_kmem_cache_node(s, node, n) { if (n->barn) @@ -7927,6 +8066,8 @@ STAT_ATTR(ALLOC_PCS, alloc_cpu_sheaf); STAT_ATTR(ALLOC_FASTPATH, alloc_fastpath); STAT_ATTR(ALLOC_SLOWPATH, alloc_slowpath); STAT_ATTR(FREE_PCS, free_cpu_sheaf); +STAT_ATTR(FREE_RCU_SHEAF, free_rcu_sheaf); +STAT_ATTR(FREE_RCU_SHEAF_FAIL, free_rcu_sheaf_fail); STAT_ATTR(FREE_FASTPATH, free_fastpath); STAT_ATTR(FREE_SLOWPATH, free_slowpath); STAT_ATTR(FREE_FROZEN, free_frozen); @@ -8022,6 +8163,8 @@ static struct attribute *slab_attrs[] = { &alloc_fastpath_attr.attr, &alloc_slowpath_attr.attr, &free_cpu_sheaf_attr.attr, + &free_rcu_sheaf_attr.attr, + &free_rcu_sheaf_fail_attr.attr, &free_fastpath_attr.attr, &free_slowpath_attr.attr, &free_frozen_attr.attr,