From patchwork Tue Oct 25 20:52:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13019897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 63FE0C38A2D for ; Tue, 25 Oct 2022 20:54:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=osD7OgH672Ig2zBGyjrCkUTdY2rsT25jraPKZCKuL8M=; b=5DcyRy2YjarDWk WUBDOGJYgL0Hgl4GIfsq6TJ9oxlwJNMW8HeUljVJDzRqGexkkwm0blTgA+4Yp2UmAnl3wYi73/5kX t4AsKUNnhyyZVjioX9qXGn4nT3SLoHUoDs3JlPdioj7gVFmw8NBDKn0U7qh6Nk736AGJgo6J3S0IK PhJnV5z6gncyHKz+BKYoaAtwKp6okq4sP/rRKRJMgymMupuMn+QF6Agfp6y4J6BpAX+bcHj6Es2b6 D+KrLNpDVueU8/XERC+wk9oJIFUJClPv3y8LgKVfCuhgrz6gf9AmLCx0lr7zLb6ry6hjZnmBYp7Ni a3qIn/T0w6bPDHSD9irw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1onQuy-0078J0-4v; Tue, 25 Oct 2022 20:53:00 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1onQuu-0078HE-Dj for linux-arm-kernel@lists.infradead.org; Tue, 25 Oct 2022 20:52:58 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B26CB61B7F; Tue, 25 Oct 2022 20:52:55 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A0860C4347C; Tue, 25 Oct 2022 20:52:52 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann Cc: Will Deacon , Marc Zyngier , Greg Kroah-Hartman , Andrew Morton , Herbert Xu , Ard Biesheuvel , Christoph Hellwig , Isaac Manjarres , Saravana Kannan , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Subject: [PATCH v2 1/2] mm: slab: Introduce __GFP_PACKED for smaller kmalloc() alignments Date: Tue, 25 Oct 2022 21:52:46 +0100 Message-Id: <20221025205247.3264568-2-catalin.marinas@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221025205247.3264568-1-catalin.marinas@arm.com> References: <20221025205247.3264568-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221025_135256_564517_5C179910 X-CRM114-Status: GOOD ( 21.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org By default kmalloc() returns objects aligned to ARCH_KMALLOC_MINALIGN. This can be somewhat large on architectures defining ARCH_DMA_MINALIGN (e.g. 128 on arm64) and significant memory is wasted through small kmalloc() allocations. Reduce the minimum alignment for kmalloc() to the default KMALLOC_MIN_SIZE (8 for slub, 32 for slab) but align the requested size to the bigger ARCH_KMALLOC_MINALIGN unless a newly added __GFP_PACKED flag is passed. With this gfp flag, the alignment is reduced to KMALLOC_PACKED_ALIGN, at least sizeof(unsigned long long). There's no slob support. Signed-off-by: Catalin Marinas --- include/linux/gfp_types.h | 10 ++++++++-- include/linux/slab.h | 22 ++++++++++++++++++---- mm/slab_common.c | 3 ++- 3 files changed, 28 insertions(+), 7 deletions(-) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index d88c46ca82e1..305cb8cb6f8b 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -55,8 +55,9 @@ typedef unsigned int __bitwise gfp_t; #define ___GFP_SKIP_KASAN_UNPOISON 0 #define ___GFP_SKIP_KASAN_POISON 0 #endif +#define ___GFP_PACKED 0x8000000u #ifdef CONFIG_LOCKDEP -#define ___GFP_NOLOCKDEP 0x8000000u +#define ___GFP_NOLOCKDEP 0x10000000u #else #define ___GFP_NOLOCKDEP 0 #endif @@ -243,6 +244,10 @@ typedef unsigned int __bitwise gfp_t; * * %__GFP_SKIP_KASAN_POISON makes KASAN skip poisoning on page deallocation. * Typically, used for userspace pages. Only effective in HW_TAGS mode. + * + * %__GFP_PACKED returns a pointer aligned to the possibly smaller + * KMALLOC_PACKED_ALIGN rather than ARCH_KMALLOC_MINALIGN. Useful for small + * object allocation on architectures that define large ARCH_DMA_MINALIGN. */ #define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN) #define __GFP_COMP ((__force gfp_t)___GFP_COMP) @@ -251,12 +256,13 @@ typedef unsigned int __bitwise gfp_t; #define __GFP_SKIP_ZERO ((__force gfp_t)___GFP_SKIP_ZERO) #define __GFP_SKIP_KASAN_UNPOISON ((__force gfp_t)___GFP_SKIP_KASAN_UNPOISON) #define __GFP_SKIP_KASAN_POISON ((__force gfp_t)___GFP_SKIP_KASAN_POISON) +#define __GFP_PACKED ((__force gfp_t)___GFP_PACKED) /* Disable lockdep for GFP context tracking */ #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) /* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT (27 + IS_ENABLED(CONFIG_LOCKDEP)) +#define __GFP_BITS_SHIFT (28 + IS_ENABLED(CONFIG_LOCKDEP)) #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) /** diff --git a/include/linux/slab.h b/include/linux/slab.h index 90877fcde70b..0f59585b5fbf 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -223,8 +223,6 @@ void kmem_dump_obj(void *object); */ #if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8 #define ARCH_KMALLOC_MINALIGN ARCH_DMA_MINALIGN -#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN -#define KMALLOC_SHIFT_LOW ilog2(ARCH_DMA_MINALIGN) #else #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long) #endif @@ -310,6 +308,11 @@ static inline unsigned int arch_slab_minalign(void) #define KMALLOC_MIN_SIZE (1 << KMALLOC_SHIFT_LOW) #endif +/* + * This alignment should be at least sizeof(unsigned long long). + */ +#define KMALLOC_PACKED_ALIGN (KMALLOC_MIN_SIZE) + /* * This restriction comes from byte sized index implementation. * Page size is normally 2^12 bytes and, in this case, if we want to use @@ -382,6 +385,17 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags) return KMALLOC_CGROUP; } +/* + * Align the size to ARCH_KMALLOC_MINALIGN unless __GFP_PACKED is passed. + */ +static __always_inline size_t kmalloc_size_align(size_t size, gfp_t flags) +{ + if (ARCH_KMALLOC_MINALIGN > KMALLOC_PACKED_ALIGN && + !(flags & __GFP_PACKED)) + size = ALIGN(size, ARCH_KMALLOC_MINALIGN); + return size; +} + /* * Figure out which kmalloc slab an allocation of a certain size * belongs to. @@ -568,7 +582,7 @@ static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) if (size > KMALLOC_MAX_CACHE_SIZE) return kmalloc_large(size, flags); #ifndef CONFIG_SLOB - index = kmalloc_index(size); + index = kmalloc_index(kmalloc_size_align(size, flags)); if (!index) return ZERO_SIZE_PTR; @@ -590,7 +604,7 @@ static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t fla if (size > KMALLOC_MAX_CACHE_SIZE) return kmalloc_large_node(size, flags, node); - index = kmalloc_index(size); + index = kmalloc_index(kmalloc_size_align(size, flags)); if (!index) return ZERO_SIZE_PTR; diff --git a/mm/slab_common.c b/mm/slab_common.c index 33b1886b06eb..0e4ea396cd4f 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -627,7 +627,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name, unsigned int useroffset, unsigned int usersize) { int err; - unsigned int align = ARCH_KMALLOC_MINALIGN; + unsigned int align = KMALLOC_PACKED_ALIGN; s->name = name; s->size = s->object_size = size; @@ -720,6 +720,7 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags) { unsigned int index; + size = kmalloc_size_align(size, flags); if (size <= 192) { if (!size) return ZERO_SIZE_PTR;