From patchwork Tue Nov 12 16:38:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13872511 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F51DD42BAB for ; Tue, 12 Nov 2024 16:40:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4235C6B00FD; Tue, 12 Nov 2024 11:39:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 10F126B0105; Tue, 12 Nov 2024 11:39:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CEB226B00FD; Tue, 12 Nov 2024 11:39:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 203936B0096 for ; Tue, 12 Nov 2024 11:39:52 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id CCDEA803F8 for ; Tue, 12 Nov 2024 16:39:51 +0000 (UTC) X-FDA: 82778004204.28.5A9CED4 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf29.hostedemail.com (Postfix) with ESMTP id A3D3C120003 for ; Tue, 12 Nov 2024 16:38:52 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; spf=pass (imf29.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731429396; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S/1END9d9BHjTJ+7Rf7H45tc8S6VpQYl1A6JL4iS4e8=; b=F+MEXPeslRoXONOi6fX6xCazLAJDkmVRvuY58WSWvRq3jYytR7RS3O/zdGlm+OLk2rORmc CTGlC1YpI0fggjkNYsakBNVO3pDckDsoxZDR81O6IJqevBG+oLWvk0A1QGT4vvECMx4al7 nJYSyyySoraH8iSu+qZyKkRs1GCkWAE= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; spf=pass (imf29.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731429396; a=rsa-sha256; cv=none; b=3zJ6i8gmIqJGqIcQd5RbXvS9kAwiP8zEFQPazQe8gdIgg1oH+f+s/tJCSX/s6vvnXxxCfG VIR/2gZjWlIcvf3NizABf71iGOkDMG099mCdcng5HcJX4SfW4IjfwpMvPHJw0v47zUx3si Doc4qxlF7WpcfOQcszhzUvkoUZYb2Hk= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id CF7711F749; Tue, 12 Nov 2024 16:39:47 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id B0F5F13AA1; Tue, 12 Nov 2024 16:39:47 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 4KDlKtOEM2e6IwAAD6G6ig (envelope-from ); Tue, 12 Nov 2024 16:39:47 +0000 From: Vlastimil Babka Date: Tue, 12 Nov 2024 17:38:50 +0100 Subject: [PATCH RFC 6/6] mm, slub: sheaf prefilling for guaranteed allocations MIME-Version: 1.0 Message-Id: <20241112-slub-percpu-caches-v1-6-ddc0bdc27e05@suse.cz> References: <20241112-slub-percpu-caches-v1-0-ddc0bdc27e05@suse.cz> In-Reply-To: <20241112-slub-percpu-caches-v1-0-ddc0bdc27e05@suse.cz> To: Suren Baghdasaryan , "Liam R. Howlett" , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, "Paul E. McKenney" , Lorenzo Stoakes , Matthew Wilcox , Boqun Feng , Uladzislau Rezki , linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, maple-tree@lists.infradead.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=6843; i=vbabka@suse.cz; h=from:subject:message-id; bh=6nE4JXhtpPa9Ksw5Gnb4/9dN7CeJDc1XrhY/1U7BLXM=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBnM4TQAMvaemFosL6x4yQCCpE61LymMdBMw/tef FneZ10kYRyJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZzOE0AAKCRC74LB10kWI mrqtB/0Y1K6PeRzfOZSmJ81PnW9hb1dhLtFTaI0qEdXsbvIsyomX+bah7XtYhtN10o0/VWvsMbS yqlNe52NVyvhVgMqoEgwxFHryXf7Cl21sh60HUFKowamWJpYrXo1TuIBiUTr/GOfnl1kRzdoOyo 8Fx4blrdqQ9op2CBxEqosOGC5CnKk/SKwycaQPXDFaQPJuBpjVMLNzrc091SMUbXb6947+Z7rZp FbibQPWnvNxH/v1n4AfVDEg5LWv/ob64p2hPCESPfN6GA76ownzMxl7vUZz481tSR1Y2XfYMaOf MwBsFdMwq0p9wbn14cfAbl3RYZq4yAGwWqGGe0WFs+eH7zhO X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A3D3C120003 X-Stat-Signature: sbxaghhb11bkg5dhye6x1npcnq8paugn X-Rspam-User: X-HE-Tag: 1731429532-4813 X-HE-Meta: U2FsdGVkX19dQJgt8ajFzl7UR49znK7tS8v3hoNU83tTlBMuqy1ZDvJQegkdwoBBuVxJe07U5ExP0OHvu7dME1G+W6cYVBp5LyTNUbvmCTtHAuOQ3kgR2JhNxfXdmMiZNOAsMrJ2n5w6n+TBiqh8GJjN6qFODXp/6BX7G9rGJdr6wg5x4vgYDQza46qzunLwZlwgzMnuNoIgYTQoFp7TYjUmpfahw6WEJeUMs2warK+WUNcnNpAZ1AN0PyAGs2lNnE2ctkWXv590OfAqC8YY0/l2HokTZRchxjvC8VpLiKrXiwDbcbXaPgpwMiS4+KAhfaI6ficA2tFh7we6fytLjldiEk4+/qmw0/KJKc0KozhEJ4nTY41nGlfXt7Mo+DWTR+hay7E5vm34qJriwydEHDfIGDK+fDNZEYtW99yUgYQ6/Zg5yxggkQA/sQovx23M4Zrsf0ZfhqL58UfJ3CSk+WFrosnG75eUIe7tsMGl6RBaeVVWwsK3Fnw8LI0yjiD6V+BSvOW9yXGryroCc9l8rf3zJLnQGGj7EJc3a8WFrThLkeCKFzljIoWCNSLOSAubf/77bCEImaUIoBmqw6/olsNSUOEaKawQg3mEmmEsWSlnhf488S8TSN3uAVYeCFeCnmIH3rcqcy+GJH5wu6MOlk2agwleAlPsagwEagFtAwhhjGhNI4C0+mXywbo+CZcsg8gONOcPpIdwJDjFJKcABU63h8mBrM6OYiAvc+643+JbWQhDVllnGGhVmjIS0TrhqgoXISKlDlaJMF8v+vPB5aFQJpq7dIHN+oM27RY/6kJ4gxPsajS1qS2c3emtuVxTu+/UEVQFgrtfqwQZsz2+2+1mU5cllyXOWbcxs/vA2wHpRRLp5PRMu3Kyl08dPZzhqcGmQCqcCUHyPCyqSo7/BfOOLOjG/gQECyeIgFbNqlJMwopVBTxic7QS4kqm2c3w9vVn8Wc1D6Hdrg2shnh JRrf44aF LRmh0NSruy9YbmjPwENnS2turYPDJbGAUES+u7IXay59PBlqR9CE5H3s54+3oVT5xo0LELxHpIOIIvZmmoyNR3YpKOiVG2LuJ6kPxZr9GzdFRFcFbobBFUOVgCjFNYxyOFs/ogD6qLVD7Tp1Rc/ijyLQZIQZrkpnRa+9zS7jS6bBMJclituqL7ZRNlqAgQ48SBeZB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add three functions for efficient guaranteed allocations in a critical section (that cannot sleep) when the exact number of allocations is not known beforehand, but an upper limit can be calculated. kmem_cache_prefill_sheaf() returns a sheaf containing at least given number of objects. kmem_cache_alloc_from_sheaf() will allocate an object from the sheaf and is guaranteed not to fail until depleted. kmem_cache_return_sheaf() is for giving the sheaf back to the slab allocator after the critical section. This will also attempt to refill it to cache's sheaf capacity for better efficiency of sheaves handling, but it's not stricly necessary to succeed. TODO: the current implementation is limited to cache's sheaf_capacity Signed-off-by: Vlastimil Babka --- include/linux/slab.h | 11 ++++ mm/slub.c | 149 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 160 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index 23904321992ad2eeb9389d0883cf4d5d5d71d896..a87dc3c6392fe235de2eabe1792df86d40c3bbf9 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -820,6 +820,17 @@ void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment __malloc; #define kmem_cache_alloc_node(...) alloc_hooks(kmem_cache_alloc_node_noprof(__VA_ARGS__)) +struct slab_sheaf * +kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int count); + +void kmem_cache_return_sheaf(struct kmem_cache *s, gfp_t gfp, + struct slab_sheaf *sheaf); + +void *kmem_cache_alloc_from_sheaf_noprof(struct kmem_cache *cachep, gfp_t gfp, + struct slab_sheaf *sheaf) __assume_slab_alignment __malloc; +#define kmem_cache_alloc_from_sheaf(...) \ + alloc_hooks(kmem_cache_alloc_from_sheaf_noprof(__VA_ARGS__)) + /* * These macros allow declaring a kmem_buckets * parameter alongside size, which * can be compiled out with CONFIG_SLAB_BUCKETS=n so that a large number of call diff --git a/mm/slub.c b/mm/slub.c index 1900afa6153ca6d88f9df7db3ce84d98629489e7..a0e2cb7dfb5173f39f36bea1eb9760c3c1b99dd7 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -444,6 +444,7 @@ struct slab_sheaf { union { struct rcu_head rcu_head; struct list_head barn_list; + bool oversize; }; struct kmem_cache *cache; unsigned int size; @@ -2819,6 +2820,30 @@ static int barn_put_full_sheaf(struct node_barn *barn, struct slab_sheaf *sheaf, return ret; } +static struct slab_sheaf *barn_get_full_or_empty_sheaf(struct node_barn *barn) +{ + struct slab_sheaf *sheaf = NULL; + unsigned long flags; + + spin_lock_irqsave(&barn->lock, flags); + + if (barn->nr_empty) { + sheaf = list_first_entry(&barn->sheaves_empty, + struct slab_sheaf, barn_list); + list_del(&sheaf->barn_list); + barn->nr_empty--; + } else if (barn->nr_full) { + sheaf = list_first_entry(&barn->sheaves_full, struct slab_sheaf, + barn_list); + list_del(&sheaf->barn_list); + barn->nr_full--; + } + + spin_unlock_irqrestore(&barn->lock, flags); + + return sheaf; +} + /* * If a full sheaf is available, return it and put the supplied empty one to * barn. We ignore the limit on empty sheaves as the number of sheaves doesn't @@ -4893,6 +4918,130 @@ void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t gfpflags, int nod } EXPORT_SYMBOL(kmem_cache_alloc_node_noprof); + +/* + * returns a sheaf that has least the given count of objects + * when prefilling is needed, do so with given gfp flags + * + * return NULL if prefilling failed, or when the requested count is + * above cache's sheaf_capacity (TODO: lift this limitation) + */ +struct slab_sheaf * +kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int count) +{ + struct slub_percpu_sheaves *pcs; + struct slab_sheaf *sheaf = NULL; + + //TODO: handle via oversize sheaf + if (count > s->sheaf_capacity) + return NULL; + + pcs = cpu_sheaves_lock(s->cpu_sheaves); + + if (pcs->spare && pcs->spare->size > 0) { + sheaf = pcs->spare; + pcs->spare = NULL; + } + + if (!sheaf) + sheaf = barn_get_full_or_empty_sheaf(pcs->barn); + + cpu_sheaves_unlock(s->cpu_sheaves); + + if (!sheaf) + sheaf = alloc_empty_sheaf(s, gfp); + + if (sheaf && sheaf->size < count) { + if (refill_sheaf(s, sheaf, gfp)) { + sheaf_flush(s, sheaf); + free_empty_sheaf(s, sheaf); + sheaf = NULL; + } + } + + return sheaf; +} + +/* + * Use this to return a sheaf obtained by kmem_cache_prefill_sheaf() + * It tries to refill the sheaf back to the cache's sheaf_capacity + * to avoid handling partially full sheaves. + * + * If the refill fails because gfp is e.g. GFP_NOWAIT, the sheaf is + * instead dissolved + */ +void kmem_cache_return_sheaf(struct kmem_cache *s, gfp_t gfp, + struct slab_sheaf *sheaf) +{ + struct slub_percpu_sheaves *pcs; + bool refill = false; + struct node_barn *barn; + + //TODO: handle oversize sheaf + + pcs = cpu_sheaves_lock(s->cpu_sheaves); + + if (!pcs->spare) { + pcs->spare = sheaf; + sheaf = NULL; + } + + /* racy check */ + if (!sheaf && pcs->barn->nr_full >= MAX_FULL_SHEAVES) { + barn = pcs->barn; + refill = true; + } + + cpu_sheaves_unlock(s->cpu_sheaves); + + if (!sheaf) + return; + + /* + * if the barn is full of full sheaves or we fail to refill the sheaf, + * simply flush and free it + */ + if (!refill || refill_sheaf(s, sheaf, gfp)) { + sheaf_flush(s, sheaf); + free_empty_sheaf(s, sheaf); + return; + } + + /* we racily determined the sheaf would fit, so now force it */ + barn_put_full_sheaf(barn, sheaf, true); +} + +/* + * Allocate from a sheaf obtained by kmem_cache_prefill_sheaf() + * + * Guaranteed not to fail as many allocations as was the requested count. + * After the sheaf is emptied, it fails - no fallback to the slab cache itself. + * + * The gfp parameter is meant only to specify __GFP_ZERO or __GFP_ACCOUNT + * memcg charging is forced over limit if necessary, to avoid failure. + */ +void * +kmem_cache_alloc_from_sheaf_noprof(struct kmem_cache *s, gfp_t gfp, + struct slab_sheaf *sheaf) +{ + void *ret = NULL; + bool init; + + if (sheaf->size == 0) + goto out; + + ret = sheaf->objects[--sheaf->size]; + + init = slab_want_init_on_alloc(gfp, s); + + /* add __GFP_NOFAIL to force successful memcg charging */ + slab_post_alloc_hook(s, NULL, gfp | __GFP_NOFAIL, 1, &ret, init, s->object_size); +out: + trace_kmem_cache_alloc(_RET_IP_, ret, s, gfp, NUMA_NO_NODE); + + return ret; +} + /* * To avoid unnecessary overhead, we pass through large allocation requests * directly to the page allocator. We use __GFP_COMP, because we will need to