From patchwork Fri May 31 19:14:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13682017 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80425C25B7C for ; Fri, 31 May 2024 19:15:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 61D256B00AF; Fri, 31 May 2024 15:15:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5CB316B00B0; Fri, 31 May 2024 15:15:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 382C66B00B1; Fri, 31 May 2024 15:15:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0F7DD6B00AF for ; Fri, 31 May 2024 15:15:04 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id C2EA4C186C for ; Fri, 31 May 2024 19:15:03 +0000 (UTC) X-FDA: 82179643686.17.48FE66E Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf28.hostedemail.com (Postfix) with ESMTP id 19580C0006 for ; Fri, 31 May 2024 19:15:00 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=o2OPMeYn; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf28.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717182901; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/h1Bmcftz6GEcVlDRbChnUTa/LRBKcnGJrrY7sSbaMk=; b=psDIT4GyYaoGxNg+K/ytlZ9jmESh0DCW/xR+c4EHfwDJua4061faOwO0yk9FR1PfhAHLpz vWLvFGj1BAEFJ6Kj7oY8HjIJtFKxaaw5VsCgiT7gWot5A31YggDhibJZ8ESglGUq7NI2+T O9kPTmYvx1ab1gDW5Vtxxn07PAbAiIU= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=o2OPMeYn; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf28.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717182901; a=rsa-sha256; cv=none; b=v2ZfAQk+oT6/7hpX10uAKS+ZmIVnykMx1l0n7RftRfaPWqS5oA+j2kbC/4Dd40/xcsr+i+ NXMUJsOBupxwCRNCKhVcoIwNzh71CIAf+Q4MS1i1obCQ0Plwe2Cb/KMcVLw5A/8t7zSK0L T/AUjc5+Ssc7JrsKSXuana/VFkHJ3Tg= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 1E60362DEA; Fri, 31 May 2024 19:15:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5FDCAC4AF0B; Fri, 31 May 2024 19:14:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717182899; bh=SEppnjIqapMPAdV3gOtlWmc+MgaX+XeR2G6jWLuYp7Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=o2OPMeYnvF/RFYKaf/zp7Z8RM2X/F+vCy9GQY8I0Xac+aLT6YtbqxYij5Ro2zWi+a MN9VGaIrehneZvdwFtmr4oRoqYDRm2KSiJtIsuhOObVx1I4PNbOB+Ouoa3tqEvWFDA HSO5HevkfkfN6er7AJp9RlHES2dgJa2zpAhykipseObxwNudHFe0HalIiGVTKVHnmi AqpKV4f8T8pSoz5kt0RKKbLDLOr9OupC6Q7Y7HM7+cbnWHXwXOBdD6b53spXB6kriD 6iSMpHTPLoSPE13x9RlwVt49lrgdHsnOwM4iDFAdAiH52QtrgFm7zUF5eHwg63Ugcm gwHZ+zd6FxRTw== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v4 3/6] mm/slab: Introduce kvmalloc_buckets_node() that can take kmem_buckets argument Date: Fri, 31 May 2024 12:14:55 -0700 Message-Id: <20240531191458.987345-3-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240531191304.it.853-kees@kernel.org> References: <20240531191304.it.853-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5214; i=kees@kernel.org; h=from:subject; bh=SEppnjIqapMPAdV3gOtlWmc+MgaX+XeR2G6jWLuYp7Y=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmWiGxmv39xi7SHjeU5Qh9TUaZ7uKF3EFTYjEdf tCKJ2uSBP+JAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZlohsQAKCRCJcvTf3G3A Jr4cEACEhuBwLvPpMEMM6sggZmXcHry4TE03jPbQd9Mxmy5fFK55DUXqOjb7y21q5sv4rW+FYr+ cxS5RUtfUdTgJDLUlLXIFYuGfkjbVWyf0/UO1wjWEFajzvLZHhctDIxhDXOmz1eNdgiETeCuZy+ 0tK4LLZ7icrZDV64sHkxFdLp+QjgcG9dcn9ddPD+UMyzaehfifqCmbKPTSGQpVlWPDUz/lcu1L9 TpM1bUWcFosoEEoyPwiwzFFGSiB1K2FYqkZduo+6FElV+chVVesLAlL07MVW08zhnM5amAHBtX2 miRaeDPfZ7xIMoweuL4Vvg48VVwgnvfGJH9uu2AXACHy+sUPeUkzpqIeAUbr8pdxPHMbJ7R5p6N IOfEMRRT3t5Sja0RVGPoetNdjckFemnbuYPHydOPHxLEvx2yijFMqhzA21K1v6cp/zULxeNu6/a kAbSOq4OofTYQyu++5F1OmY4bOOeEcEWkLHmC/ZNSYxJI+PudtmIO8QYJSZcWQhhEFbGz2KivPz cdEOaIglVTuJPLq1Jwd6gg19ZqOdrKnyU3QWpKIbkR7YOHpHoc1iDMMIz+Rlg6pXcrnWBlrQeAX ggr1unJKL/WqIBTqG9zmyEyMw39wFI8PexEG8KeNryf3AdYLbv1kgoz2EMLNsURDKu2MjhCcplH x/xz3swVnXdautw= = X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 19580C0006 X-Stat-Signature: wpxzbqjaxcpws6zhay345dykdrra9o9n X-HE-Tag: 1717182900-966247 X-HE-Meta: U2FsdGVkX182JpJengqgjUw1BRP4SV5SgAEOBeFdO2V1K/qw9Y/8HEKGSiGSQbK7zmljs2OpZZGFVqwbTLZKfHqrMP7whzPK+Jzi0erYWB4YJSjCc7I84KIceNWad9KN+jcz+aRWNOXNBo/HoXW6+AYU5sYQ3rYT1Rw5Wli89yb1C51by10VCFDdJcMxlnaaWGSEdfG/YCJGk1CWn+2pcoO0L41Gt5K8TzupLe1COtPiBRbCyvEs12zAmaE32XIs3ng8ZoCfmsHuyakGU8KGtvNc6c3zO+XJhv1RfHpXiqYbZLbPiNTF+CjBJcTHaGOyC2an03IxpJP7AxGW/i+pZDdiSfos8YsAxhUgIjzX3IIjdzsv8myRvAFFP1lSrTRW9NVMP9+deJ9FPQWpfIzjOCbw7u8uPskZmQBCFWSnuoKCrDlXXO1IN+uNrdNcOfaCcuIjYXCzuXcu6JuU4AOWdU7fKUdnOcy2vI6Afryz9MPf629cxuhgZoKb/IfQRrlpj6Vovnz/hahkwFJa8vddM7FnWwLsQR2zudwrHIcOLsFuTl/crjequIZBbogj+Y+Q9PvVIQ0Uw1ax4yAToYFbbYg1s5vwXbLfzwTeu/VNY9ggi16UkK8x5ZvoTsS1KFfT7dr4RX1P5NMgVJSPyqmqZLkZVJAdqv0n8fN+97o4pccs54SAJYyRoW3sHfZqG65JYhuPUptrMVp8HOS4zaTRpirpkyoKYkib6oneDE+QiWx66qJKvpPj+9jtdS9q5S4ElF7zXtgaNteYkbWpaMzAIyuQHUi+7Z7Z/r6oohC9a78mZWc6paKeYQZizADmJSuc0SCTeD4hV/tkBKmmT0goULfOxczk1wWytolNNvb/3zI2a7jax8+9TGQpaWx7CF7jCx4v7Hh/I5tYhJ780ZEt8swoXLRGwWd/Bdkb+a2xuwA+xuph8See7GbTA8QTcCjwgSThZVSMtRTb55AJ3pw i3+H7uSM uy88nROpfk7/alKdn6TwseEKzCFmnP4OcuDCdbUmAZoj2+XWwb4tVX6mZiw4uaL7UTOU2qDPn/HwaebFC9rFdbf4daIwLJPgGGOIDcQqOeUSRa5u8g6AsgF69LjRAIu3lIFURciO61DJwiDhjDbtJFm1XXLknyneg3QpKK7yO7qOrWZanR5kIpRNc8P1LJbkka+CzBDspHeVN61ZFVTEXDksB1nBMCTPSAxH3BsWk3aocI/S2X6comnkP7jSQCXw8nIG6Akj1SXChH3+GUzWRa2QcrZrkRyRiRJchEk8E78kvUjtleIthGVNgqK459VdfBf2qdDLkqEedoUj6dySzfSjV+b8qXV2cgRaQeuXhfXb69PyYopWCUCjNoWxk+kR3RQ2oTPFTNRX/6zmcTcess5Pc9zmSe5+TZTl6wHbQROJDOcYQVx5To5UU2G0Mzh6Pdku8jyaiUKcU9QnaQ8yVs34Ggz4YaGEv5bbR0PDmhJhKOWS9qKR4C7A2l2Af//NV05qwlibFPjNk/G52MCSqOrenG1WhPEBK6cjmw71NEPKzXfi5mZiAm130livwUlalSM1gOmyW1n9WHWnHTMM2NQBzgMMJDLjlL9RdXPkpLR+gCaA4r0C4cCuIfjKWzcWXraKy3Lyz21OMiW8ckwVwIE//9tlMGQR54PnYzgb0VemCEsoJ3TTtR38xmrXAs4bXeUeE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Plumb kmem_buckets arguments through kvmalloc_node_noprof() so it is possible to provide an API to perform kvmalloc-style allocations with a particular set of buckets. Introduce kvmalloc_buckets_node() that takes a kmem_buckets argument. Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: jvoisin Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 19 +++++++++++++++---- lib/rhashtable.c | 2 +- mm/util.c | 13 +++++++++---- 3 files changed, 25 insertions(+), 9 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index b1165b22cc6f..8853c6eb20b4 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -799,11 +799,22 @@ static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags) #define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__)) #define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node) -extern void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) __alloc_size(1); -#define kvmalloc_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) +#ifdef CONFIG_SLAB_BUCKETS +extern void *kvmalloc_buckets_node_noprof(kmem_buckets *b, size_t size, gfp_t flags, int node) + __alloc_size(2); +# define kvmalloc_node_noprof(b, size, flags, node) \ + kvmalloc_buckets_node_noprof(b, size, flags, node) +#else +extern void *kvmalloc_buckets_node_noprof(size_t size, gfp_t flags, int node) + __alloc_size(1); +# define kvmalloc_node_noprof(b, size, flags, node) \ + kvmalloc_buckets_node_noprof(size, flags, node) +#endif +#define kvmalloc_buckets_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) +#define kvmalloc_node(...) kvmalloc_buckets_node(NULL, __VA_ARGS__) #define kvmalloc(_size, _flags) kvmalloc_node(_size, _flags, NUMA_NO_NODE) -#define kvmalloc_noprof(_size, _flags) kvmalloc_node_noprof(_size, _flags, NUMA_NO_NODE) +#define kvmalloc_noprof(_size, _flags) kvmalloc_node_noprof(NULL, _size, _flags, NUMA_NO_NODE) #define kvzalloc(_size, _flags) kvmalloc(_size, (_flags)|__GFP_ZERO) #define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, (_flags)|__GFP_ZERO, _node) @@ -816,7 +827,7 @@ kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node) if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; - return kvmalloc_node_noprof(bytes, flags, node); + return kvmalloc_node_noprof(NULL, bytes, flags, node); } #define kvmalloc_array_noprof(...) kvmalloc_array_node_noprof(__VA_ARGS__, NUMA_NO_NODE) diff --git a/lib/rhashtable.c b/lib/rhashtable.c index dbbed19f8fff..ef0f496e4aed 100644 --- a/lib/rhashtable.c +++ b/lib/rhashtable.c @@ -184,7 +184,7 @@ static struct bucket_table *bucket_table_alloc(struct rhashtable *ht, static struct lock_class_key __key; tbl = alloc_hooks_tag(ht->alloc_tag, - kvmalloc_node_noprof(struct_size(tbl, buckets, nbuckets), + kvmalloc_node_noprof(NULL, struct_size(tbl, buckets, nbuckets), gfp|__GFP_ZERO, NUMA_NO_NODE)); size = nbuckets; diff --git a/mm/util.c b/mm/util.c index 80430e5ba981..53f7fc5912bd 100644 --- a/mm/util.c +++ b/mm/util.c @@ -593,9 +593,11 @@ unsigned long vm_mmap(struct file *file, unsigned long addr, } EXPORT_SYMBOL(vm_mmap); +#ifdef CONFIG_SLAB_BUCKETS /** - * kvmalloc_node - attempt to allocate physically contiguous memory, but upon + * kvmalloc_buckets_node_noprof - attempt to allocate physically contiguous memory, but upon * failure, fall back to non-contiguous (vmalloc) allocation. + * @b: which set of kmalloc buckets to allocate from. * @size: size of the request. * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL. * @node: numa node to allocate from @@ -609,7 +611,10 @@ EXPORT_SYMBOL(vm_mmap); * * Return: pointer to the allocated memory of %NULL in case of failure */ -void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) +void *kvmalloc_buckets_node_noprof(kmem_buckets *b, size_t size, gfp_t flags, int node) +#else +void *kvmalloc_buckets_node_noprof(size_t size, gfp_t flags, int node) +#endif { gfp_t kmalloc_flags = flags; void *ret; @@ -631,7 +636,7 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) kmalloc_flags &= ~__GFP_NOFAIL; } - ret = kmalloc_node_noprof(size, kmalloc_flags, node); + ret = __kmalloc_node_noprof(b, size, kmalloc_flags, node); /* * It doesn't really make sense to fallback to vmalloc for sub page @@ -660,7 +665,7 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(kvmalloc_node_noprof); +EXPORT_SYMBOL(kvmalloc_buckets_node_noprof); /** * kvfree() - Free memory.