From patchwork Tue Apr 5 13:57:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 12801561 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8D05DC38A05 for ; Tue, 5 Apr 2022 14:06:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9HX6llSBOidNqqVCfj7Q4Vo/5eqIsZ9TQrW4jri+ZIs=; b=xpEa24te0EKUwJ zIy+ODF7L73lMQUn3X+J+X9GdW8bSL8B8c3b/nfyxMWi7FrYX+HaLnjzUVLIIg1ACccl9ZdYkzRz8 QScokahBW/FaGgNPlxJDSyUQokxoDvr7syKh8ypM8ACg5CvXpnIlJPkyFpVX1xHXcMq8rYga1bOAt 5ui1Y3krUjhu7zRJLlniebydmhdxnJ1JOo4GHvr2JbJAvqDbk0AEqrqVHTSCU6lwIbMC3I2+jpqmO q3m0v9lVd7sGVd5p1WyBXc5tZDHIDbpmeR7ITckzQOADo9rOzCmVzFyEIsVOT59z+cJbJ41hFiqG8 2nZZk5ndGFjSyX+fe8Tw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nbjng-001JAA-U7; Tue, 05 Apr 2022 14:04:53 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nbjhS-001G0c-FR for linux-arm-kernel@lists.infradead.org; Tue, 05 Apr 2022 13:58:30 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 01D99B81BA9; Tue, 5 Apr 2022 13:58:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D18DAC385A4; Tue, 5 Apr 2022 13:58:21 +0000 (UTC) From: Catalin Marinas To: Will Deacon , Marc Zyngier , Arnd Bergmann , Greg Kroah-Hartman , Andrew Morton , Linus Torvalds Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 08/10] mm/slab: Allow dynamic kmalloc() minimum alignment Date: Tue, 5 Apr 2022 14:57:56 +0100 Message-Id: <20220405135758.774016-9-catalin.marinas@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220405135758.774016-1-catalin.marinas@arm.com> References: <20220405135758.774016-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220405_065826_882992_4843EB35 X-CRM114-Status: GOOD ( 18.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org ARCH_KMALLOC_MINALIGN represents the minimum guaranteed kmalloc() alignment but an architecture may require a larger run-time alignment. Do not create kmalloc caches smaller than arch_kmalloc_minalign(). Signed-off-by: Catalin Marinas Cc: Andrew Morton Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- include/linux/slab.h | 2 ++ mm/slab.c | 6 +----- mm/slab.h | 2 ++ mm/slab_common.c | 33 +++++++++++++++++++++++---------- 4 files changed, 28 insertions(+), 15 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index d58211bdeceb..2137dba85691 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -332,6 +332,8 @@ enum kmalloc_cache_type { extern struct kmem_cache * kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]; +unsigned int arch_kmalloc_minalign(void); + /* * Define gfp bits that should not be set for KMALLOC_NORMAL. */ diff --git a/mm/slab.c b/mm/slab.c index b04e40078bdf..4aaeeb9c994d 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1256,11 +1256,7 @@ void __init kmem_cache_init(void) * Initialize the caches that provide memory for the kmem_cache_node * structures first. Without this, further allocations will bug. */ - kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE] = create_kmalloc_cache( - kmalloc_info[INDEX_NODE].name[KMALLOC_NORMAL], - kmalloc_info[INDEX_NODE].size, - ARCH_KMALLOC_FLAGS, 0, - kmalloc_info[INDEX_NODE].size); + new_kmalloc_cache(INDEX_NODE, KMALLOC_NORMAL, ARCH_KMALLOC_FLAGS); slab_state = PARTIAL_NODE; setup_kmalloc_cache_index_table(); diff --git a/mm/slab.h b/mm/slab.h index fd7ae2024897..e9238406602a 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -283,6 +283,8 @@ int __kmem_cache_create(struct kmem_cache *, slab_flags_t flags); struct kmem_cache *create_kmalloc_cache(const char *name, unsigned int size, slab_flags_t flags, unsigned int useroffset, unsigned int usersize); +void __init new_kmalloc_cache(int idx, enum kmalloc_cache_type type, + slab_flags_t flags); extern void create_boot_cache(struct kmem_cache *, const char *name, unsigned int size, slab_flags_t flags, unsigned int useroffset, unsigned int usersize); diff --git a/mm/slab_common.c b/mm/slab_common.c index 6ee64d6208b3..594d8a8a68d0 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -838,9 +838,18 @@ void __init setup_kmalloc_cache_index_table(void) } } -static void __init +unsigned int __weak arch_kmalloc_minalign(void) +{ + return ARCH_KMALLOC_MINALIGN; +} + +void __init new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags) { + unsigned int minalign = arch_kmalloc_minalign(); + unsigned int aligned_size = kmalloc_info[idx].size; + int aligned_idx = idx; + if (type == KMALLOC_RECLAIM) { flags |= SLAB_RECLAIM_ACCOUNT; } else if (IS_ENABLED(CONFIG_MEMCG_KMEM) && (type == KMALLOC_CGROUP)) { @@ -851,10 +860,17 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags) flags |= SLAB_ACCOUNT; } - kmalloc_caches[type][idx] = create_kmalloc_cache( - kmalloc_info[idx].name[type], - kmalloc_info[idx].size, flags, 0, - kmalloc_info[idx].size); + if (minalign > ARCH_KMALLOC_MINALIGN) { + aligned_size = ALIGN(aligned_size, minalign); + aligned_idx = __kmalloc_index(aligned_size, false); + } + + if (!kmalloc_caches[type][aligned_idx]) + kmalloc_caches[type][aligned_idx] = create_kmalloc_cache( + kmalloc_info[aligned_idx].name[type], + aligned_size, flags, 0, aligned_size); + if (idx != aligned_idx) + kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx]; /* * If CONFIG_MEMCG_KMEM is enabled, disable cache merging for @@ -904,11 +920,8 @@ void __init create_kmalloc_caches(slab_flags_t flags) struct kmem_cache *s = kmalloc_caches[KMALLOC_NORMAL][i]; if (s) { - kmalloc_caches[KMALLOC_DMA][i] = create_kmalloc_cache( - kmalloc_info[i].name[KMALLOC_DMA], - kmalloc_info[i].size, - SLAB_CACHE_DMA | flags, 0, - kmalloc_info[i].size); + new_kmalloc_cache(i, KMALLOC_DMA, + SLAB_CACHE_DMA | flags); } } #endif