From patchwork Fri Mar 4 06:34:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12768532 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25008C433F5 for ; Fri, 4 Mar 2022 06:34:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A0D008D0003; Fri, 4 Mar 2022 01:34:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 996198D0001; Fri, 4 Mar 2022 01:34:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 80F558D0003; Fri, 4 Mar 2022 01:34:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0140.hostedemail.com [216.40.44.140]) by kanga.kvack.org (Postfix) with ESMTP id 6D0348D0001 for ; Fri, 4 Mar 2022 01:34:46 -0500 (EST) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 0DEFF74D05 for ; Fri, 4 Mar 2022 06:34:46 +0000 (UTC) X-FDA: 79205740572.30.600976D Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf17.hostedemail.com (Postfix) with ESMTP id 90B254000F for ; Fri, 4 Mar 2022 06:34:45 +0000 (UTC) Received: by mail-pl1-f173.google.com with SMTP id e2so6901093pls.10 for ; Thu, 03 Mar 2022 22:34:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TVdYQpXFu0DWrIT8aY3Z6KXTWb1Uch1rx+HdjmDRFcE=; b=LsASEabx8/b3sjb/G+RHgJLKsqZzcmnsQpo7M0GFhPCRRcuFrPFU7DM1g1X9MwHM21 PCXAVQsHx+MY8cnF0ZNt6zIq0IoAZxJ+YDgCqdORaHNDX7Fz8BwvxFmSI3Z+xQzxLh33 BUyMjlWWgmy5H4BIxoTuTqf3UE5ZxGD7UwaXXV0XkkZ/t7lFPk5JmQnFpv5BflK6/4Xa rKVoS4CylYDxBNLDH1+HfxM1pKl0ZoCw1xFctB8g1uqKLagtbseDlW1WFBCZiHTwGO2N mZrCNFxqr5JLCxO3V2BqG1pVkczUxsM4IWh7nFxGKarULxMOmvPb/3v9bRDemWr79jBL YoEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TVdYQpXFu0DWrIT8aY3Z6KXTWb1Uch1rx+HdjmDRFcE=; b=pQ5VQgYK8eocv3+8Ht8FFjG4scJp6L1BaUGhJ+83iAZUWEA+O8u9c6C6ZejWKAv7es Ux4CDZ0+XsVX+OHL6MV8XYQvKOhLucPEllLRb1V3S2GW9j+zVXnDu9FDjfI3QAZK9ADA 8bVJe/FZb8Rdmt1aX216VRPdr/SGljxbiPQyufs06s0eUiHFk7qeWyhuZNN6PhAsDL9x RJFWYJ7131mb228avWuegyxP+8nrFOG79/S5imbnikMylyJp/vNSoeu/VpVt3Kb9z7LZ AbeSL8F3Y5UCkfKaTDzstw8Bwz3h5Zyrq542mlHsevLJ/k8l0j25w6QD4xqRSjBWTb8s MEhw== X-Gm-Message-State: AOAM532cNMEL2Isk8udQcrkzuGhxwUla73OW1F+DDvgc+r88uYX21/fv hOU63B2cYpHQRFyaxrcWHl9ndqL1hr1PXQ== X-Google-Smtp-Source: ABdhPJwrb7HJV9HhwNvNZ4oa0OmTF8RoFPyx59jmPxqApPtPntlDOJEcyLgJkbpsXbKA494+Pn+H8w== X-Received: by 2002:a17:90a:9306:b0:1bc:9256:5477 with SMTP id p6-20020a17090a930600b001bc92565477mr9339370pjo.170.1646375684474; Thu, 03 Mar 2022 22:34:44 -0800 (PST) Received: from ip-172-31-19-208.ap-northeast-1.compute.internal (ec2-18-181-137-102.ap-northeast-1.compute.amazonaws.com. [18.181.137.102]) by smtp.gmail.com with ESMTPSA id v10-20020a056a00148a00b004e0f420dd90sm4900007pfu.40.2022.03.03.22.34.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Mar 2022 22:34:44 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: linux-mm@kvack.org Cc: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Marco Elver , Matthew WilCox , Roman Gushchin , linux-kernel@vger.kernel.org, 42.hyeyoo@gmail.com Subject: [PATCH v2 1/5] mm/slab: kmalloc: pass requests larger than order-1 page to page allocator Date: Fri, 4 Mar 2022 06:34:23 +0000 Message-Id: <20220304063427.372145-2-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20220304063427.372145-1-42.hyeyoo@gmail.com> References: <20220304063427.372145-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 90B254000F X-Stat-Signature: 3sepggf6tkk4ac1m39h54a361tfntyns Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=LsASEabx; spf=pass (imf17.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-HE-Tag: 1646375685-79528 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is not much benefit for serving large objects in kmalloc(). Let's pass large requests to page allocator like SLUB for better maintenance of common code. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- include/linux/slab.h | 35 ++++++++++++++++------------------- mm/slab.c | 31 +++++++++++++++++++++++++++---- mm/slab.h | 19 +++++++++++++++++++ mm/slub.c | 19 ------------------- 4 files changed, 62 insertions(+), 42 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 37bde99b74af..e7b3330db4f3 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -224,29 +224,19 @@ void kmem_dump_obj(void *object); * Kmalloc array related definitions */ -#ifdef CONFIG_SLAB /* - * The largest kmalloc size supported by the SLAB allocators is - * 32 megabyte (2^25) or the maximum allocatable page order if that is - * less than 32 MB. - * - * WARNING: Its not easy to increase this value since the allocators have - * to do various tricks to work around compiler limitations in order to - * ensure proper constant folding. + * SLAB and SLUB directly allocates requests fitting in to an order-1 page + * (PAGE_SIZE*2). Larger requests are passed to the page allocator. */ -#define KMALLOC_SHIFT_HIGH ((MAX_ORDER + PAGE_SHIFT - 1) <= 25 ? \ - (MAX_ORDER + PAGE_SHIFT - 1) : 25) -#define KMALLOC_SHIFT_MAX KMALLOC_SHIFT_HIGH +#ifdef CONFIG_SLAB +#define KMALLOC_SHIFT_HIGH (PAGE_SHIFT + 1) +#define KMALLOC_SHIFT_MAX (MAX_ORDER + PAGE_SHIFT - 1) #ifndef KMALLOC_SHIFT_LOW #define KMALLOC_SHIFT_LOW 5 #endif #endif #ifdef CONFIG_SLUB -/* - * SLUB directly allocates requests fitting in to an order-1 page - * (PAGE_SIZE*2). Larger requests are passed to the page allocator. - */ #define KMALLOC_SHIFT_HIGH (PAGE_SHIFT + 1) #define KMALLOC_SHIFT_MAX (MAX_ORDER + PAGE_SHIFT - 1) #ifndef KMALLOC_SHIFT_LOW @@ -564,15 +554,15 @@ static __always_inline __alloc_size(1) void *kmalloc_large(size_t size, gfp_t fl * Try really hard to succeed the allocation but fail * eventually. */ +#ifndef CONFIG_SLOB static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) { if (__builtin_constant_p(size)) { -#ifndef CONFIG_SLOB unsigned int index; -#endif + if (size > KMALLOC_MAX_CACHE_SIZE) return kmalloc_large(size, flags); -#ifndef CONFIG_SLOB + index = kmalloc_index(size); if (!index) @@ -581,10 +571,17 @@ static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) return kmem_cache_alloc_trace( kmalloc_caches[kmalloc_type(flags)][index], flags, size); -#endif } return __kmalloc(size, flags); } +#else +static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) +{ + if (__builtin_constant_p(size) && size > KMALLOC_MAX_CACHE_SIZE) + return kmalloc_large(size, flags); + return __kmalloc(size, flags); +} +#endif static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node) { diff --git a/mm/slab.c b/mm/slab.c index ddf5737c63d9..570af6dc3478 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3624,7 +3624,8 @@ __do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller) void *ret; if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) - return NULL; + return kmalloc_large(size, flags); + cachep = kmalloc_slab(size, flags); if (unlikely(ZERO_OR_NULL_PTR(cachep))) return cachep; @@ -3685,7 +3686,8 @@ static __always_inline void *__do_kmalloc(size_t size, gfp_t flags, void *ret; if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) - return NULL; + return kmalloc_large(size, flags); + cachep = kmalloc_slab(size, flags); if (unlikely(ZERO_OR_NULL_PTR(cachep))) return cachep; @@ -3739,14 +3741,21 @@ void kmem_cache_free_bulk(struct kmem_cache *orig_s, size_t size, void **p) { struct kmem_cache *s; size_t i; + struct folio *folio; local_irq_disable(); for (i = 0; i < size; i++) { void *objp = p[i]; - if (!orig_s) /* called via kfree_bulk */ + if (!orig_s) { + /* called via kfree_bulk */ + folio = virt_to_folio(objp); + if (unlikely(!folio_test_slab(folio))) { + free_large_kmalloc(folio, objp); + continue; + } s = virt_to_cache(objp); - else + } else s = cache_from_obj(orig_s, objp); if (!s) continue; @@ -3776,11 +3785,20 @@ void kfree(const void *objp) { struct kmem_cache *c; unsigned long flags; + struct folio *folio; + void *object = (void *) objp; trace_kfree(_RET_IP_, objp); if (unlikely(ZERO_OR_NULL_PTR(objp))) return; + + folio = virt_to_folio(objp); + if (unlikely(!folio_test_slab(folio))) { + free_large_kmalloc(folio, object); + return; + } + local_irq_save(flags); kfree_debugcheck(objp); c = virt_to_cache(objp); @@ -4211,12 +4229,17 @@ void __check_heap_object(const void *ptr, unsigned long n, size_t __ksize(const void *objp) { struct kmem_cache *c; + struct folio *folio; size_t size; BUG_ON(!objp); if (unlikely(objp == ZERO_SIZE_PTR)) return 0; + folio = virt_to_folio(objp); + if (!folio_test_slab(folio)) + return folio_size(folio); + c = virt_to_cache(objp); size = c ? c->object_size : 0; diff --git a/mm/slab.h b/mm/slab.h index c7f2abc2b154..31e98beb47a3 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -664,6 +664,25 @@ static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) print_tracking(cachep, x); return cachep; } + +static __always_inline void kfree_hook(void *x) +{ + kmemleak_free(x); + kasan_kfree_large(x); +} + +static inline void free_large_kmalloc(struct folio *folio, void *object) +{ + unsigned int order = folio_order(folio); + + if (WARN_ON_ONCE(order == 0)) + pr_warn_once("object pointer: 0x%p\n", object); + + kfree_hook(object); + mod_lruvec_page_state(folio_page(folio, 0), NR_SLAB_UNRECLAIMABLE_B, + -(PAGE_SIZE << order)); + __free_pages(folio_page(folio, 0), order); +} #endif /* CONFIG_SLOB */ static inline size_t slab_ksize(const struct kmem_cache *s) diff --git a/mm/slub.c b/mm/slub.c index 261474092e43..04fd084f4709 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1686,12 +1686,6 @@ static inline void *kmalloc_large_node_hook(void *ptr, size_t size, gfp_t flags) return ptr; } -static __always_inline void kfree_hook(void *x) -{ - kmemleak_free(x); - kasan_kfree_large(x); -} - static __always_inline bool slab_free_hook(struct kmem_cache *s, void *x, bool init) { @@ -3535,19 +3529,6 @@ struct detached_freelist { struct kmem_cache *s; }; -static inline void free_large_kmalloc(struct folio *folio, void *object) -{ - unsigned int order = folio_order(folio); - - if (WARN_ON_ONCE(order == 0)) - pr_warn_once("object pointer: 0x%p\n", object); - - kfree_hook(object); - mod_lruvec_page_state(folio_page(folio, 0), NR_SLAB_UNRECLAIMABLE_B, - -(PAGE_SIZE << order)); - __free_pages(folio_page(folio, 0), order); -} - /* * This function progressively scans the array with free objects (with * a limited look ahead) and extract objects belonging to the same