From patchwork Thu Sep 5 07:56:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13791863 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 450B8194A48 for ; Thu, 5 Sep 2024 07:57:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523028; cv=none; b=ZoYxbRUbFbIebk+3VgNaMpUCim/AzPf+dRvCbn6ixOzteeUCBt1IJumwnUDDbRJD2wp5bOyxxnabz/S2s9h1OIGg54hRWevn7gdRRqVZEeU8w9US2CO7XCej0ME9YE4te3vAzzB7jsQuB/zLq7q5CHXlKEJ40zA/vbQShXGBMGQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523028; c=relaxed/simple; bh=rY6kC2WiEYeQ/Zth3l+6kLSSBQxvb1BV7zU7+V4xp9g=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=BwDMs3Sff8cIn8ODp+F+5hKruqDrrC2KBwvips8bgK7cZG8EJWJXcdXZe2Zr6UyBU9izIHiTMT/SNlXk/O8eJ2zAFRVayvqaq0BSfGIE6RlbUXmsoQZkFD+mq8bRrIsv1vH6L3fdmuFfOa3044b2NK+MUQKakaTkVDK+eI4JjL8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=PV7TCy+x; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="PV7TCy+x" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 62592C4CEC4; Thu, 5 Sep 2024 07:57:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725523027; bh=rY6kC2WiEYeQ/Zth3l+6kLSSBQxvb1BV7zU7+V4xp9g=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=PV7TCy+xlc5pai2T6OdADRXzqDCI7RS8Bew/YoEZlFm+l72tOIN19MbKrPF6i5xBM gOSOOQaJQe465NMdIBUFo+hbYDaRhYpDBiPbcDjqtRlv4DoS2DtOzj+tQv6cfdWBNb q0/i6dtuVk/rAoczLEeDpKbjTKcXRRWRLJDQAjn70rnQxzBUnrBpk0NSHVvzdefNjO RB4FHrnCps+DNwlrGxUABHRx3MBhNmetxyiNvz5dkhcGbVTYjn3mTBI+qXz7EvI+WC Y5xY+PYXXi78sd/hs7aUOS4++QZ8sdSPHGUlCFu1/5fer/b2UN8aXI5F+OehXTDhhW fKT+xPJB0wmuQ== From: Christian Brauner Date: Thu, 05 Sep 2024 09:56:44 +0200 Subject: [PATCH v4 01/17] slab: s/__kmem_cache_create/do_kmem_cache_create/g Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240905-work-kmem_cache_args-v4-1-ed45d5380679@kernel.org> References: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> In-Reply-To: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=2107; i=brauner@kernel.org; h=from:subject:message-id; bh=rY6kC2WiEYeQ/Zth3l+6kLSSBQxvb1BV7zU7+V4xp9g=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdTPEK4X7zku9Z8Eq+IAvWO5pVE032Bs03YHF2aUuIm s81MSG5o5SFQYyLQVZMkcWh3SRcbjlPxWajTA2YOaxMIEMYuDgFYCKFpxn+yoRdZjFudkjftsb2 yvpJ/zeGidzekMn8YGWqmMvE35bsexn+mTQ0TNHW08p6kHv/WRij7csO74MdNnfOrmJ8xKLl3sr MAwA= X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Free up reusing the double-underscore variant for follow-up patches. Reviewed-by: Kees Cook Reviewed-by: Jens Axboe Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Vlastimil Babka Signed-off-by: Christian Brauner Reviewed-by: Roman Gushchin --- mm/slab.h | 2 +- mm/slab_common.c | 4 ++-- mm/slub.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index a6051385186e..684bb48c4f39 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -424,7 +424,7 @@ kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags, unsigned long caller) gfp_t kmalloc_fix_flags(gfp_t flags); /* Functions provided by the slab allocators */ -int __kmem_cache_create(struct kmem_cache *, slab_flags_t flags); +int do_kmem_cache_create(struct kmem_cache *, slab_flags_t flags); void __init kmem_cache_init(void); extern void create_boot_cache(struct kmem_cache *, const char *name, diff --git a/mm/slab_common.c b/mm/slab_common.c index 95db3702f8d6..91e0e36e4379 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -234,7 +234,7 @@ static struct kmem_cache *create_cache(const char *name, s->useroffset = useroffset; s->usersize = usersize; #endif - err = __kmem_cache_create(s, flags); + err = do_kmem_cache_create(s, flags); if (err) goto out_free_cache; @@ -778,7 +778,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name, s->usersize = usersize; #endif - err = __kmem_cache_create(s, flags); + err = do_kmem_cache_create(s, flags); if (err) panic("Creation of kmalloc slab %s size=%u failed. Reason %d\n", diff --git a/mm/slub.c b/mm/slub.c index 9aa5da1e8e27..23d9d783ff26 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -5902,7 +5902,7 @@ __kmem_cache_alias(const char *name, unsigned int size, unsigned int align, return s; } -int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) +int do_kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) { int err; From patchwork Thu Sep 5 07:56:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13791864 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 41133193422 for ; Thu, 5 Sep 2024 07:57:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523032; cv=none; b=Uanp1ed7OwbUBMOvUijsgKHlHmlHGE7x+g2GAHMOHiPzjBPkmQJAk4WvATmuJ1BgnHT8Ykoy6u8k6+wZ3JSs7y3hGJZwYr1fZ81cAe7V98WYgcB4FlOTJMu1KvGSCZlqcVIXd/We+w2/Pm9Y1FZDZHdR4AGa1TTueYoOmVc+REI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523032; c=relaxed/simple; bh=YrF0YLTWmhzO7ohK2/AAEF++N1Zrk2GGMPwzT5f/roQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=MV4zB1H9mkPk0Ox4VOJkPtnktDW8zu1y+eWpI3t43bWZlrO+sYoymWBhoqy1aZ2RBmNRbP9WjPzvMq2BunjKKtAGV9dGWnSCaZfbN4lOPjVFNLYIktIbV3BW1oBVrJEdeH3ZOlFHww9oXkIBtxUI+AdBXXe9EAekQp84hZmkFsQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RlCvUk1a; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RlCvUk1a" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 639DFC4CEC6; Thu, 5 Sep 2024 07:57:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725523031; bh=YrF0YLTWmhzO7ohK2/AAEF++N1Zrk2GGMPwzT5f/roQ=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=RlCvUk1aAW1A2PTD5pA5BLm8yKdG5h9Nt+M5LC6cKKQZe2I8dRntcu8FyPewu3x5J bPbBpiEdveTRgdl2rDXE02u69/PKcGxp4H2uFl4dCWQx0k4W96+0wwJ7e2MpqqErjU jiLHLqE0gCi9Iivb/u2ycl8NEWOIb4HS+0v/dgFWitb1THmhvyhD6XJXIOa/u9H+tv oLqtkwQfqy4mhDvTn2hhvtXkhRKGd4M5lIJIBbRIB7cH1wYLawsCDoihg623L76PHv 0ze59QHEhOpVT6y4uckiaW2M9CYoVqFlWcG1cYyb/+yuXiGNA2MakauPnqCJhQ3vg9 v/eKzN6tnn1Xw== From: Christian Brauner Date: Thu, 05 Sep 2024 09:56:45 +0200 Subject: [PATCH v4 02/17] slab: add struct kmem_cache_args Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240905-work-kmem_cache_args-v4-2-ed45d5380679@kernel.org> References: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> In-Reply-To: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=5895; i=brauner@kernel.org; h=from:subject:message-id; bh=YrF0YLTWmhzO7ohK2/AAEF++N1Zrk2GGMPwzT5f/roQ=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdTPFSL7jxYpd08eYXgeIGe5bWCPKWVF16ZvHq2XwLa 363/g6djlIWBjEuBlkxRRaHdpNwueU8FZuNMjVg5rAygQxh4OIUgIlozWb4zbI7r2Llq6MVPNum sxuH8z9azFfQ4R4//xLr78mf5jW9s2Vk+LRC82SZM8PUuc8EzDzOe63d90G9/pLulPkctmbFkdN 7+QE= X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Currently we have multiple kmem_cache_create*() variants that take up to seven separate parameters with one of the functions having to grow an eigth parameter in the future to handle both usercopy and a custom freelist pointer. Add a struct kmem_cache_args structure and move less common parameters into it. Core parameters such as name, object size, and flags continue to be passed separately. Add a new function __kmem_cache_create_args() that takes a struct kmem_cache_args pointer and port do_kmem_cache_create_usercopy() over to it. In follow-up patches we will port the other kmem_cache_create*() variants over to it as well. Reviewed-by: Kees Cook Reviewed-by: Jens Axboe Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Vlastimil Babka Signed-off-by: Christian Brauner Reviewed-by: Roman Gushchin --- include/linux/slab.h | 21 ++++++++++++++++ mm/slab_common.c | 67 +++++++++++++++++++++++++++++++++++++++------------- 2 files changed, 72 insertions(+), 16 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 5b2da2cf31a8..79d8c8bca4a4 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -240,6 +240,27 @@ struct mem_cgroup; */ bool slab_is_available(void); +/** + * @align: The required alignment for the objects. + * @useroffset: Usercopy region offset + * @usersize: Usercopy region size + * @freeptr_offset: Custom offset for the free pointer in RCU caches + * @use_freeptr_offset: Whether a @freeptr_offset is used + * @ctor: A constructor for the objects. + */ +struct kmem_cache_args { + unsigned int align; + unsigned int useroffset; + unsigned int usersize; + unsigned int freeptr_offset; + bool use_freeptr_offset; + void (*ctor)(void *); +}; + +struct kmem_cache *__kmem_cache_create_args(const char *name, + unsigned int object_size, + struct kmem_cache_args *args, + slab_flags_t flags); struct kmem_cache *kmem_cache_create(const char *name, unsigned int size, unsigned int align, slab_flags_t flags, void (*ctor)(void *)); diff --git a/mm/slab_common.c b/mm/slab_common.c index 91e0e36e4379..0f13c045b8d1 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -248,14 +248,24 @@ static struct kmem_cache *create_cache(const char *name, return ERR_PTR(err); } -static struct kmem_cache * -do_kmem_cache_create_usercopy(const char *name, - unsigned int size, unsigned int freeptr_offset, - unsigned int align, slab_flags_t flags, - unsigned int useroffset, unsigned int usersize, - void (*ctor)(void *)) +/** + * __kmem_cache_create_args - Create a kmem cache + * @name: A string which is used in /proc/slabinfo to identify this cache. + * @object_size: The size of objects to be created in this cache. + * @args: Arguments for the cache creation (see struct kmem_cache_args). + * @flags: See %SLAB_* flags for an explanation of individual @flags. + * + * Cannot be called within a interrupt, but can be interrupted. + * + * Return: a pointer to the cache on success, NULL on failure. + */ +struct kmem_cache *__kmem_cache_create_args(const char *name, + unsigned int object_size, + struct kmem_cache_args *args, + slab_flags_t flags) { struct kmem_cache *s = NULL; + unsigned int freeptr_offset = UINT_MAX; const char *cache_name; int err; @@ -275,7 +285,7 @@ do_kmem_cache_create_usercopy(const char *name, mutex_lock(&slab_mutex); - err = kmem_cache_sanity_check(name, size); + err = kmem_cache_sanity_check(name, object_size); if (err) { goto out_unlock; } @@ -296,12 +306,14 @@ do_kmem_cache_create_usercopy(const char *name, /* Fail closed on bad usersize of useroffset values. */ if (!IS_ENABLED(CONFIG_HARDENED_USERCOPY) || - WARN_ON(!usersize && useroffset) || - WARN_ON(size < usersize || size - usersize < useroffset)) - usersize = useroffset = 0; - - if (!usersize) - s = __kmem_cache_alias(name, size, align, flags, ctor); + WARN_ON(!args->usersize && args->useroffset) || + WARN_ON(object_size < args->usersize || + object_size - args->usersize < args->useroffset)) + args->usersize = args->useroffset = 0; + + if (!args->usersize) + s = __kmem_cache_alias(name, object_size, args->align, flags, + args->ctor); if (s) goto out_unlock; @@ -311,9 +323,11 @@ do_kmem_cache_create_usercopy(const char *name, goto out_unlock; } - s = create_cache(cache_name, size, freeptr_offset, - calculate_alignment(flags, align, size), - flags, useroffset, usersize, ctor); + if (args->use_freeptr_offset) + freeptr_offset = args->freeptr_offset; + s = create_cache(cache_name, object_size, freeptr_offset, + calculate_alignment(flags, args->align, object_size), + flags, args->useroffset, args->usersize, args->ctor); if (IS_ERR(s)) { err = PTR_ERR(s); kfree_const(cache_name); @@ -335,6 +349,27 @@ do_kmem_cache_create_usercopy(const char *name, } return s; } +EXPORT_SYMBOL(__kmem_cache_create_args); + +static struct kmem_cache * +do_kmem_cache_create_usercopy(const char *name, + unsigned int size, unsigned int freeptr_offset, + unsigned int align, slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)) +{ + struct kmem_cache_args kmem_args = { + .align = align, + .use_freeptr_offset = freeptr_offset != UINT_MAX, + .freeptr_offset = freeptr_offset, + .useroffset = useroffset, + .usersize = usersize, + .ctor = ctor, + }; + + return __kmem_cache_create_args(name, size, &kmem_args, flags); +} + /** * kmem_cache_create_usercopy - Create a cache with a region suitable From patchwork Thu Sep 5 07:56:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13791865 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3C99F194089 for ; Thu, 5 Sep 2024 07:57:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523036; cv=none; b=TtSgGp5VupeRX+kOIPy0TkVojVTz5A8lihe9hq6zLV7H0HU6kvUKcm7jgoT0ISt09lzfhU/GIgG7AK+VN51Y5c+cJPC8NvU3OOJkJu+lqjHdWbbd9IPtvjO43RhL9qY5Q57S0O8ZAfPsn49Iw3omwk7voNgXwHfhPFH+T4U+7C8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523036; c=relaxed/simple; bh=GREA3sYaJRn8q3FSMpLjOkkur0kW09aZN/fas4uylbs=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=MrRrIKu1qkTD876aSa7O4Bhx4FZiX92PnQZM+YLcDSKKpeC6JDQ3IKGXR9W5m6Ho8idP73Iy31WN/crtlFF/qQ3wvRMHOPiCAQLPElMe+syHy6Lr6sEJO+7TDvP5qZtzPDgypVhptsVcnK9+wHt71N9Qlj7XJO/fyJAGdYM05uY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=n1yEAPYO; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="n1yEAPYO" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 624BBC4CEC3; Thu, 5 Sep 2024 07:57:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725523035; bh=GREA3sYaJRn8q3FSMpLjOkkur0kW09aZN/fas4uylbs=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=n1yEAPYOkhFGaueY8PxQNM4lK424wC4LLcA4VZoAvb5sRDnTnUifIO/WWNvg15+c/ 96l0X4SE43ZRjAj/nvkbLg0iz5LROt+spWqSPPq2Txpfg7cSNT/Xz2E2WqOpJZdw5D lFRk0NfhYGoH7+UQIyDyVZc9fWNIfgPJaBG7ofLXjCVb0EVzcm7SSJ42nymshXMfe+ /D4Wk6cfDKl6Bd4QkWanB6UptwyqN34Wy2BO7glxcb4C0nXfVsAoM6q1Qo8ZLO2pJP ODo9PzM48pFQfM+m3oM7RCNDYwGgXrjv8vh7tPt4xxUrBY34iT/H0MdLerLxsImKvW oa/+9x27JXu5A== From: Christian Brauner Date: Thu, 05 Sep 2024 09:56:46 +0200 Subject: [PATCH v4 03/17] slab: port kmem_cache_create() to struct kmem_cache_args Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240905-work-kmem_cache_args-v4-3-ed45d5380679@kernel.org> References: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> In-Reply-To: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=1010; i=brauner@kernel.org; h=from:subject:message-id; bh=GREA3sYaJRn8q3FSMpLjOkkur0kW09aZN/fas4uylbs=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdTPHa0lwdyv1ebHdoTLLK7L1hdrPXNizcXFIZcJw/9 OjNO+2cHaUsDGJcDLJiiiwO7Sbhcst5KjYbZWrAzGFlAhnCwMUpABP5+oLhv2d7+4ofqTcu5jDm sq7rXHiimsc2YkPJs6o1ZgeKwyOC5BkZPlRMu3Ap7vgjqcV1UbX2r95Oyfi5J+0L61L/Hq6MW9z 9vAA= X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Port kmem_cache_create() to struct kmem_cache_args. Reviewed-by: Kees Cook Reviewed-by: Jens Axboe Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Vlastimil Babka Signed-off-by: Christian Brauner Reviewed-by: Roman Gushchin --- mm/slab_common.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 0f13c045b8d1..ac0832dac01e 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -439,8 +439,12 @@ struct kmem_cache * kmem_cache_create(const char *name, unsigned int size, unsigned int align, slab_flags_t flags, void (*ctor)(void *)) { - return do_kmem_cache_create_usercopy(name, size, UINT_MAX, align, flags, - 0, 0, ctor); + struct kmem_cache_args kmem_args = { + .align = align, + .ctor = ctor, + }; + + return __kmem_cache_create_args(name, size, &kmem_args, flags); } EXPORT_SYMBOL(kmem_cache_create); From patchwork Thu Sep 5 07:56:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13791866 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2C4751940A1 for ; Thu, 5 Sep 2024 07:57:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523040; cv=none; b=I2Hg7ko7D2ZttYjQ/YebKdNLhCIEYUZ+/XX+OtnV8AUkzElgSFO47Rpe/MMYvtdIfRVZmWiTWcZYchCXbHCOpFiWgRv5Maxa6Hdqf0sm21T+G1A6ks5YS2+GfwMr/31VUL6gl/e7ZG2dKPlNlIlJVpjxjE9XqGF41nKBlOpq62o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523040; c=relaxed/simple; bh=yFwPV9RImMNE3Jd/xX3WyJKu7K+3lEbT+HKSODA016Y=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=rrQ7hI8h8wgKywWx6R4YLllNw8FUQ0as0kFf+O9gnxKd50J/gDFGx9S02/dy97xSoyTyA7iUiIsyiE7JWwtbwRNf1eqCjPc3Q3fLXWUYQRPglQ2SeYXI98cW83Z9MO2V6N/N261bup321235M55D8uQx8p/i+bdNKvjltpLzRiQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=pZmgX8C5; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="pZmgX8C5" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3F658C4CEC4; Thu, 5 Sep 2024 07:57:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725523039; bh=yFwPV9RImMNE3Jd/xX3WyJKu7K+3lEbT+HKSODA016Y=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=pZmgX8C5yhojHGzSpwIYVN1O8utd8eTeLfdJ+KCv8Vq2ZO5b52/05Pm9H3KqyU9Sq KvxnTBeWV9EhLSJMPdcmuc8CcLt8l1dDGMfDb7IKtrjwwFUfYWiPmPCaC3zwNGQLVv 1UychvUaOYnSlbFnpF0RBzzuiI6FDN4e2JsiHn4W6UvPYT1xRC7ZYyltk5yrG9OmPl XskKQWIaXsjwZ4rxWuBORBXonKDQC7cmwAb2l7a61JT1WYOE0KwjZKZh7N7HpgXgWb 4nTvRSeFZgNbpVuX7XD3mTpt4gLCfZGcoTm1paEOrUvgjg0Ve5wFXFHy0idr+l5LP4 5uSsgI5j5WxUQ== From: Christian Brauner Date: Thu, 05 Sep 2024 09:56:47 +0200 Subject: [PATCH v4 04/17] slab: port kmem_cache_create_rcu() to struct kmem_cache_args Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240905-work-kmem_cache_args-v4-4-ed45d5380679@kernel.org> References: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> In-Reply-To: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=1121; i=brauner@kernel.org; h=from:subject:message-id; bh=yFwPV9RImMNE3Jd/xX3WyJKu7K+3lEbT+HKSODA016Y=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdTPEqyfVkZ/Lj+7FDdPbsN+wabw4veVAVw1Dp/L6yh 1tta6p2RykLgxgXg6yYIotDu0m43HKeis1GmRowc1iZQIYwcHEKwETebWZkWJdyJGSWRv2/JNut Rf8X2y3aqBSctErO9RKXB7feo4PxlxgZrp64x/5lh85xJQ5hG0E+iSWFzVsOqEx4/f12rLuN5oX 5XAA= X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Port kmem_cache_create_rcu() to struct kmem_cache_args. Reviewed-by: Kees Cook Reviewed-by: Jens Axboe Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Vlastimil Babka Signed-off-by: Christian Brauner Reviewed-by: Roman Gushchin --- mm/slab_common.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index ac0832dac01e..da62ed30f95d 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -481,9 +481,13 @@ struct kmem_cache *kmem_cache_create_rcu(const char *name, unsigned int size, unsigned int freeptr_offset, slab_flags_t flags) { - return do_kmem_cache_create_usercopy(name, size, freeptr_offset, 0, - flags | SLAB_TYPESAFE_BY_RCU, 0, 0, - NULL); + struct kmem_cache_args kmem_args = { + .freeptr_offset = freeptr_offset, + .use_freeptr_offset = true, + }; + + return __kmem_cache_create_args(name, size, &kmem_args, + flags | SLAB_TYPESAFE_BY_RCU); } EXPORT_SYMBOL(kmem_cache_create_rcu); From patchwork Thu Sep 5 07:56:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13791867 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 38C39194C6B for ; Thu, 5 Sep 2024 07:57:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523044; cv=none; b=RiOX4MKZizLSVabFTZOgdrCULVXuAbKpv6zEUs7Qo+kyOpDR+3qlzl9qrhF22iLxF+DsqL4aS9HbBTHaByzv/p3mIjpM8gUtv2DSBqPgpt1T1/8QGFBI+klbR3WzRL70EA4cn06JiTuFVnV5jrsFfTV0jhdlY3wzWDsePoWKXIU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523044; c=relaxed/simple; bh=6Lwr5vukmiiT/FN2uceSot6K8Ni/9MrZgxWKfDi+Qrw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=RYP3Z7SYj19Iqi6Zhv/oGR6Ds842cN+a8uarezwxuTq06ZnpHbD2FgdH6qJaktuNJKPiVXlgbYjejqaI0BQWpg+hd/SjUQTSXSXuZ27QbdYwmapZEvj+INR29RyaireNGZQe72Y+4k26CYNBeMhklLE8guIjDfZGzzuS50fgqbc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=R/D1P6TD; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="R/D1P6TD" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 45DEDC4CEC6; Thu, 5 Sep 2024 07:57:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725523043; bh=6Lwr5vukmiiT/FN2uceSot6K8Ni/9MrZgxWKfDi+Qrw=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=R/D1P6TD7i3UY0YoNJUwmFJ6UyYHcjDyPANa2Wlb5o90hiQ26zKS5AR/jkc4f4SEy YceSaYFbEZHm/C5PYEd5MAowgwnVp6kdP/beQa/V8vHmLlhdst0Xz1xWgKs0FkuH1+ kRbclNIQTlz5eexEo1D/n1agRtf/mXV9cXck0Yzr6cRT+LB+zwIebWhvkBdbQee3Q2 SnFAEKpZUub3Ye8AUkfS/JNiKoWP7Vh+Yrrg3dCuoORvb55B+KAXFtGKaXoZYcqaRg GErCg5kgnpSiSOWZql/CqFt4+pB8A7esVmSzAcsfb0+uGVxtIuu6szTo6w26XFuiOx 2HsMsH/RvcV9A== From: Christian Brauner Date: Thu, 05 Sep 2024 09:56:48 +0200 Subject: [PATCH v4 05/17] slab: port kmem_cache_create_usercopy() to struct kmem_cache_args Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240905-work-kmem_cache_args-v4-5-ed45d5380679@kernel.org> References: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> In-Reply-To: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=2077; i=brauner@kernel.org; h=from:subject:message-id; bh=6Lwr5vukmiiT/FN2uceSot6K8Ni/9MrZgxWKfDi+Qrw=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdTPGKsfyzU2iy6oqJdrum/37ZLHLyxDqPwI66x69/G dy/cZWns6OUhUGMi0FWTJHFod0kXG45T8Vmo0wNmDmsTCBDGLg4BeAmr2b4X3hDN+grt9yT67Il Ng1Zx+Lux4SrOv7fxsC34I5BRsWGYEaG9/ZhR62uTH6+RsrF4JPes3+/Ns2q3ugaI+X3Ysf/D67 XOAA= X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Port kmem_cache_create_usercopy() to struct kmem_cache_args and remove the now unused do_kmem_cache_create_usercopy() helper. Reviewed-by: Kees Cook Reviewed-by: Jens Axboe Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Vlastimil Babka Signed-off-by: Christian Brauner Reviewed-by: Roman Gushchin --- mm/slab_common.c | 30 ++++++++---------------------- 1 file changed, 8 insertions(+), 22 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index da62ed30f95d..16c36a946135 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -351,26 +351,6 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, } EXPORT_SYMBOL(__kmem_cache_create_args); -static struct kmem_cache * -do_kmem_cache_create_usercopy(const char *name, - unsigned int size, unsigned int freeptr_offset, - unsigned int align, slab_flags_t flags, - unsigned int useroffset, unsigned int usersize, - void (*ctor)(void *)) -{ - struct kmem_cache_args kmem_args = { - .align = align, - .use_freeptr_offset = freeptr_offset != UINT_MAX, - .freeptr_offset = freeptr_offset, - .useroffset = useroffset, - .usersize = usersize, - .ctor = ctor, - }; - - return __kmem_cache_create_args(name, size, &kmem_args, flags); -} - - /** * kmem_cache_create_usercopy - Create a cache with a region suitable * for copying to userspace @@ -405,8 +385,14 @@ kmem_cache_create_usercopy(const char *name, unsigned int size, unsigned int useroffset, unsigned int usersize, void (*ctor)(void *)) { - return do_kmem_cache_create_usercopy(name, size, UINT_MAX, align, flags, - useroffset, usersize, ctor); + struct kmem_cache_args kmem_args = { + .align = align, + .ctor = ctor, + .useroffset = useroffset, + .usersize = usersize, + }; + + return __kmem_cache_create_args(name, size, &kmem_args, flags); } EXPORT_SYMBOL(kmem_cache_create_usercopy); From patchwork Thu Sep 5 07:56:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13791868 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0FC3C1940BC for ; Thu, 5 Sep 2024 07:57:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523048; cv=none; b=gdJZpheQOwLZqd5neob8FA3ZbeXQefq7xGJKmkTLhjrv5eq6VfrzXI+Sd/dPAHOlHzporK++SVLvsNLvlkaUUpRmRIEUpvNm3fJljUA8ymLkQwbK2LnevRnoC5W6Hbh7b6k3kFhGhHWdjShkZXMn+5TiDdGazlo1NX8QfW1YLVs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523048; c=relaxed/simple; bh=9LaPebyH3fPMvl0DXSpF9bydw1rDuiO92393crLdD6M=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=enfWhnlZhx5EeOox9kblF98EfIZiP19sCgbqfApNCLpnnVlyLd/SeFr2vnwmLpgSiEH6GoPwcDSNtoFhUKPjGDLlYCFJ7LGJKXpZMr9SngRrZZPyIgZglaQAU82nd5a/WYZGOKYB0UPYydmbAn82axLhAmcDjUmqynXIrkwX2Wk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=rMBdS4oJ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="rMBdS4oJ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2E314C4CEC9; Thu, 5 Sep 2024 07:57:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725523047; bh=9LaPebyH3fPMvl0DXSpF9bydw1rDuiO92393crLdD6M=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=rMBdS4oJXDrqT8tGREWQUTOIfMJ/VpD19pMK7GmE6v97b4Pb2Y4v9LbtFL42rZbSy 4QLWYUCZnCESDH/lYq7gVT5xSfchSm9rP1Ofd5fEPA7M7Wjc9+OPwFMuXcpMa7ljRv GUDOr8RdXdjIa0ysmqgYVYyGSdZxY3Z9pMKAihH1fXR6lcUc95iKW61EhSVTEjOyds gES6UEM4ZLmYRj3x+nBKmtXAjTT6EXs3Z0nP61/sxeR7GFRX+B9EcWB6o2EttWLYgN RNpJMAHt4PanxiTjwkHRzn2t8D/GfFIUrR6crlqou9TzXGGzhaPB6DRa061E56J/dM 7Tsat6ATNJ5/g== From: Christian Brauner Date: Thu, 05 Sep 2024 09:56:49 +0200 Subject: [PATCH v4 06/17] slab: pass struct kmem_cache_args to create_cache() Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240905-work-kmem_cache_args-v4-6-ed45d5380679@kernel.org> References: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> In-Reply-To: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=3238; i=brauner@kernel.org; h=from:subject:message-id; bh=9LaPebyH3fPMvl0DXSpF9bydw1rDuiO92393crLdD6M=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdTPFKWG33zVhlwrtNldGc23ao7HF5fPuMy6fDemo3L XaX5PbN7yhlYRDjYpAVU2RxaDcJl1vOU7HZKFMDZg4rE8gQBi5OAZjI5PMM/zNOmF/zC1hxZfra wDv9x4K3Xj5x8ceq6VGXD3d4/53qLBvEyNDsvTPy8Bf1+vmHhXb4LUrTW6rm9nr9y7yK7SaaFuY zX3IBAA== X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Pass struct kmem_cache_args to create_cache() so that we can later simplify further helpers. Reviewed-by: Kees Cook Reviewed-by: Jens Axboe Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Vlastimil Babka Signed-off-by: Christian Brauner Reviewed-by: Roman Gushchin --- mm/slab_common.c | 39 +++++++++++++++++++-------------------- 1 file changed, 19 insertions(+), 20 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 16c36a946135..9baa61c9c670 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -202,22 +202,22 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align, } static struct kmem_cache *create_cache(const char *name, - unsigned int object_size, unsigned int freeptr_offset, - unsigned int align, slab_flags_t flags, - unsigned int useroffset, unsigned int usersize, - void (*ctor)(void *)) + unsigned int object_size, + struct kmem_cache_args *args, + slab_flags_t flags) { struct kmem_cache *s; int err; - if (WARN_ON(useroffset + usersize > object_size)) - useroffset = usersize = 0; + if (WARN_ON(args->useroffset + args->usersize > object_size)) + args->useroffset = args->usersize = 0; /* If a custom freelist pointer is requested make sure it's sane. */ err = -EINVAL; - if (freeptr_offset != UINT_MAX && - (freeptr_offset >= object_size || !(flags & SLAB_TYPESAFE_BY_RCU) || - !IS_ALIGNED(freeptr_offset, sizeof(freeptr_t)))) + if (args->use_freeptr_offset && + (args->freeptr_offset >= object_size || + !(flags & SLAB_TYPESAFE_BY_RCU) || + !IS_ALIGNED(args->freeptr_offset, sizeof(freeptr_t)))) goto out; err = -ENOMEM; @@ -227,12 +227,15 @@ static struct kmem_cache *create_cache(const char *name, s->name = name; s->size = s->object_size = object_size; - s->rcu_freeptr_offset = freeptr_offset; - s->align = align; - s->ctor = ctor; + if (args->use_freeptr_offset) + s->rcu_freeptr_offset = args->freeptr_offset; + else + s->rcu_freeptr_offset = UINT_MAX; + s->align = args->align; + s->ctor = args->ctor; #ifdef CONFIG_HARDENED_USERCOPY - s->useroffset = useroffset; - s->usersize = usersize; + s->useroffset = args->useroffset; + s->usersize = args->usersize; #endif err = do_kmem_cache_create(s, flags); if (err) @@ -265,7 +268,6 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, slab_flags_t flags) { struct kmem_cache *s = NULL; - unsigned int freeptr_offset = UINT_MAX; const char *cache_name; int err; @@ -323,11 +325,8 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, goto out_unlock; } - if (args->use_freeptr_offset) - freeptr_offset = args->freeptr_offset; - s = create_cache(cache_name, object_size, freeptr_offset, - calculate_alignment(flags, args->align, object_size), - flags, args->useroffset, args->usersize, args->ctor); + args->align = calculate_alignment(flags, args->align, object_size); + s = create_cache(cache_name, object_size, args, flags); if (IS_ERR(s)) { err = PTR_ERR(s); kfree_const(cache_name); From patchwork Thu Sep 5 07:56:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13791869 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B6D9F194123 for ; Thu, 5 Sep 2024 07:57:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523051; cv=none; b=OfY0cfHm6LlUo2BSzLsXyvtKJILSMvsfqFDrrcigwjVoh6bPyRVl14Vl5KoD1X+33MWjiBARkr5xATmZNyXqvFpbxDwMmtnVAaRYaLq6AvUDH9igXW6moOXIgAc88K3CJq3Lz7aazJ8X8I0pCna3SBD63yMo5XQT4Kd8Ro1epWo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523051; c=relaxed/simple; bh=F4tMDDfenpBgeWRetu8ug4Xik9BM3oJX2qts2ZpcdrM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=poyqvW/jkChNX5CftNYHLmeLgkejyFomNZJB8S1rt2uhZn8Gexidc6JW3xeky3yHbkbPO1szUVz/LDqe+jPcDf/dhspPFLuyB1j2XKrA1BhqIgchJ8/+QjF1jAPbdPCEMm3KANI0ew0hwFcUO0hlLGnziVWvD5BJyerw07tQ3QQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ObpeGRhp; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ObpeGRhp" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 29AD4C4CEC6; Thu, 5 Sep 2024 07:57:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725523051; bh=F4tMDDfenpBgeWRetu8ug4Xik9BM3oJX2qts2ZpcdrM=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=ObpeGRhpLdQ/xuBN6oc6xnWemMqwQAuRAUt5YcF7NoCkWvjhAwlxQWyj0YMXbjEnA xL+aJg6yRld7k9ZRH+ok+weCGJzWrsHl08suV/gG2HeJ3/bwtGoGhpv6BivMo7Gh/Y UVXJXgjKC/yBQHHnVSSqhpPpIc46SKUmR3BUjov2sSc1Mi+CfEyraqEvSHni4F3tkZ MzrzFsfHl0cO15XuhnUg81sW2N32QewmdDV6gkcmZXKczJ+sn1glfWxTCxgd4k3LwZ vyBL3URCpolzi5OPCOlGEedhX0Hvc+I3WaG+oV2yYW7weQgP4k4TnhHF4l50gtempi NTlN8VzUHGzvw== From: Christian Brauner Date: Thu, 05 Sep 2024 09:56:50 +0200 Subject: [PATCH v4 07/17] slab: pull kmem_cache_open() into do_kmem_cache_create() Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240905-work-kmem_cache_args-v4-7-ed45d5380679@kernel.org> References: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> In-Reply-To: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=4372; i=brauner@kernel.org; h=from:subject:message-id; bh=F4tMDDfenpBgeWRetu8ug4Xik9BM3oJX2qts2ZpcdrM=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdTPF+/LIxODE3/l71sjm8Yc+2flq/cFrAmkffmULnH w0LqTr5paOUhUGMi0FWTJHFod0kXG45T8Vmo0wNmDmsTCBDGLg4BWAiDBEM/4sL0/4K6YoFeClI L/9byMhqxCZcLfdt52Kv+x5Zwh/nCzMybM5ddf7X++DrBstaBVet+fJQ5/yETMadStuc6jZ9r9f dwgIA X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 do_kmem_cache_create() is the only caller and we're going to pass down struct kmem_cache_args in a follow-up patch. Reviewed-by: Kees Cook Reviewed-by: Jens Axboe Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Vlastimil Babka Signed-off-by: Christian Brauner Reviewed-by: Roman Gushchin --- mm/slub.c | 132 +++++++++++++++++++++++++++++--------------------------------- 1 file changed, 62 insertions(+), 70 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 23d9d783ff26..30f4ca6335c7 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -5290,65 +5290,6 @@ static int calculate_sizes(struct kmem_cache *s) return !!oo_objects(s->oo); } -static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) -{ - s->flags = kmem_cache_flags(flags, s->name); -#ifdef CONFIG_SLAB_FREELIST_HARDENED - s->random = get_random_long(); -#endif - - if (!calculate_sizes(s)) - goto error; - if (disable_higher_order_debug) { - /* - * Disable debugging flags that store metadata if the min slab - * order increased. - */ - if (get_order(s->size) > get_order(s->object_size)) { - s->flags &= ~DEBUG_METADATA_FLAGS; - s->offset = 0; - if (!calculate_sizes(s)) - goto error; - } - } - -#ifdef system_has_freelist_aba - if (system_has_freelist_aba() && !(s->flags & SLAB_NO_CMPXCHG)) { - /* Enable fast mode */ - s->flags |= __CMPXCHG_DOUBLE; - } -#endif - - /* - * The larger the object size is, the more slabs we want on the partial - * list to avoid pounding the page allocator excessively. - */ - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); - - set_cpu_partial(s); - -#ifdef CONFIG_NUMA - s->remote_node_defrag_ratio = 1000; -#endif - - /* Initialize the pre-computed randomized freelist if slab is up */ - if (slab_state >= UP) { - if (init_cache_random_seq(s)) - goto error; - } - - if (!init_kmem_cache_nodes(s)) - goto error; - - if (alloc_kmem_cache_cpus(s)) - return 0; - -error: - __kmem_cache_release(s); - return -EINVAL; -} - static void list_slab_objects(struct kmem_cache *s, struct slab *slab, const char *text) { @@ -5904,26 +5845,77 @@ __kmem_cache_alias(const char *name, unsigned int size, unsigned int align, int do_kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) { - int err; + int err = -EINVAL; - err = kmem_cache_open(s, flags); - if (err) - return err; + s->flags = kmem_cache_flags(flags, s->name); +#ifdef CONFIG_SLAB_FREELIST_HARDENED + s->random = get_random_long(); +#endif + + if (!calculate_sizes(s)) + goto out; + if (disable_higher_order_debug) { + /* + * Disable debugging flags that store metadata if the min slab + * order increased. + */ + if (get_order(s->size) > get_order(s->object_size)) { + s->flags &= ~DEBUG_METADATA_FLAGS; + s->offset = 0; + if (!calculate_sizes(s)) + goto out; + } + } + +#ifdef system_has_freelist_aba + if (system_has_freelist_aba() && !(s->flags & SLAB_NO_CMPXCHG)) { + /* Enable fast mode */ + s->flags |= __CMPXCHG_DOUBLE; + } +#endif + + /* + * The larger the object size is, the more slabs we want on the partial + * list to avoid pounding the page allocator excessively. + */ + s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); + s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); + + set_cpu_partial(s); + +#ifdef CONFIG_NUMA + s->remote_node_defrag_ratio = 1000; +#endif + + /* Initialize the pre-computed randomized freelist if slab is up */ + if (slab_state >= UP) { + if (init_cache_random_seq(s)) + goto out; + } + + if (!init_kmem_cache_nodes(s)) + goto out; + + if (!alloc_kmem_cache_cpus(s)) + goto out; /* Mutex is not taken during early boot */ - if (slab_state <= UP) - return 0; + if (slab_state <= UP) { + err = 0; + goto out; + } err = sysfs_slab_add(s); - if (err) { - __kmem_cache_release(s); - return err; - } + if (err) + goto out; if (s->flags & SLAB_STORE_USER) debugfs_slab_add(s); - return 0; +out: + if (err) + __kmem_cache_release(s); + return err; } #ifdef SLAB_SUPPORTS_SYSFS From patchwork Thu Sep 5 07:56:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13791870 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B02EC1940BC for ; Thu, 5 Sep 2024 07:57:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523055; cv=none; b=SF37XR1oSqip3fsZw7skTSroNNN8iybAcuUSShjoXjJ4vWKInyoBjtpVrAXtQnjOMe9y8VBmrwzj2EPJC3UEbAEpzP6EcqfE4ZLlnZJmxG8p14cgjq7JdYEjHcHpi6dc5tEKrrJhpLdWLkysye3ACHmx0XhODMdiqV24FUJGqNY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523055; c=relaxed/simple; bh=68ks3+RK7luxREXLt0gNfsWbQ+8I8NfI8y27/qM8GTo=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=HGqnz/J9zTWoqRYAHr6iumLlhWtR2vo/RYrj4Vg4fIUY8a0m25kjSbOdvpsqvM1rzPjb5TglfrA098JWKFkgcnwvhON37ps2/z0TZT1RXUGsa8OqoQzquykDnUOW5+lfBtsfSsUDCP3txUraAN48VsKt5GwfN5mxtzSFicdszl4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=lCMJJsA2; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="lCMJJsA2" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 28B9FC4CEC3; Thu, 5 Sep 2024 07:57:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725523055; bh=68ks3+RK7luxREXLt0gNfsWbQ+8I8NfI8y27/qM8GTo=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=lCMJJsA2FYQgZ241Y/sVm4oECKak0msryTpe6Sxy0xhPqrJ4bz+KhoUhxL14XWa1c 7E+aemB9oTGTfc5vf7ZjV6+34ksKJNjfyrwdWGGVBINGAM5bGQEHyfHw0aanRFgDfr 95fU5OABVDgKWclGHd20xyuZDsq+ffWmV7zDnzjhyo5D2a+9PoWQZhHl9tt8+dkT7Q UJVo/ISOhmxyzB8f/ZerRvzWyNLDq9UE4WkJcK7GI/AN2FVNQ96k1gBj2pqV6egzIm iGbxOz4BJ5tXpM1QhItXBzVswa4b1k2vJkjww2yuXsrYFWrJ7pQlGvB8rpGUrNO8TA TsQJqs2fi09KQ== From: Christian Brauner Date: Thu, 05 Sep 2024 09:56:51 +0200 Subject: [PATCH v4 08/17] slab: pass struct kmem_cache_args to do_kmem_cache_create() Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240905-work-kmem_cache_args-v4-8-ed45d5380679@kernel.org> References: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> In-Reply-To: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=4004; i=brauner@kernel.org; h=from:subject:message-id; bh=68ks3+RK7luxREXLt0gNfsWbQ+8I8NfI8y27/qM8GTo=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdTPHe0MpoGSi7d66U3P/akqWzzBdPLC8ol1nexlAr+ ncn67pdHaUsDGJcDLJiiiwO7Sbhcst5KjYbZWrAzGFlAhnCwMUpABOZeoPhf8l6tcjWzRWT1WbF 3H4W5XN1Ert36Ee5VbJvw3KvM/h5v2P4xbzkMcfhuDniFefderU3aERvNT+tKWEXWZq1K9iyqaC YBwA= X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 and initialize most things in do_kmem_cache_create(). In a follow-up patch we'll remove rcu_freeptr_offset from struct kmem_cache. Reviewed-by: Kees Cook Reviewed-by: Jens Axboe Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Vlastimil Babka Signed-off-by: Christian Brauner Reviewed-by: Roman Gushchin --- mm/slab.h | 4 +++- mm/slab_common.c | 27 ++++++--------------------- mm/slub.c | 17 ++++++++++++++++- 3 files changed, 25 insertions(+), 23 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 684bb48c4f39..c7a4e0fc3cf1 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -424,7 +424,9 @@ kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags, unsigned long caller) gfp_t kmalloc_fix_flags(gfp_t flags); /* Functions provided by the slab allocators */ -int do_kmem_cache_create(struct kmem_cache *, slab_flags_t flags); +int do_kmem_cache_create(struct kmem_cache *s, const char *name, + unsigned int size, struct kmem_cache_args *args, + slab_flags_t flags); void __init kmem_cache_init(void); extern void create_boot_cache(struct kmem_cache *, const char *name, diff --git a/mm/slab_common.c b/mm/slab_common.c index 9baa61c9c670..19ae3dd6e36f 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -224,20 +224,7 @@ static struct kmem_cache *create_cache(const char *name, s = kmem_cache_zalloc(kmem_cache, GFP_KERNEL); if (!s) goto out; - - s->name = name; - s->size = s->object_size = object_size; - if (args->use_freeptr_offset) - s->rcu_freeptr_offset = args->freeptr_offset; - else - s->rcu_freeptr_offset = UINT_MAX; - s->align = args->align; - s->ctor = args->ctor; -#ifdef CONFIG_HARDENED_USERCOPY - s->useroffset = args->useroffset; - s->usersize = args->usersize; -#endif - err = do_kmem_cache_create(s, flags); + err = do_kmem_cache_create(s, name, object_size, args, flags); if (err) goto out_free_cache; @@ -788,9 +775,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name, { int err; unsigned int align = ARCH_KMALLOC_MINALIGN; - - s->name = name; - s->size = s->object_size = size; + struct kmem_cache_args kmem_args = {}; /* * kmalloc caches guarantee alignment of at least the largest @@ -799,14 +784,14 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name, */ if (flags & SLAB_KMALLOC) align = max(align, 1U << (ffs(size) - 1)); - s->align = calculate_alignment(flags, align, size); + kmem_args.align = calculate_alignment(flags, align, size); #ifdef CONFIG_HARDENED_USERCOPY - s->useroffset = useroffset; - s->usersize = usersize; + kmem_args.useroffset = useroffset; + kmem_args.usersize = usersize; #endif - err = do_kmem_cache_create(s, flags); + err = do_kmem_cache_create(s, name, size, &kmem_args, flags); if (err) panic("Creation of kmalloc slab %s size=%u failed. Reason %d\n", diff --git a/mm/slub.c b/mm/slub.c index 30f4ca6335c7..4719b60215b8 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -5843,14 +5843,29 @@ __kmem_cache_alias(const char *name, unsigned int size, unsigned int align, return s; } -int do_kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) +int do_kmem_cache_create(struct kmem_cache *s, const char *name, + unsigned int size, struct kmem_cache_args *args, + slab_flags_t flags) { int err = -EINVAL; + s->name = name; + s->size = s->object_size = size; + s->flags = kmem_cache_flags(flags, s->name); #ifdef CONFIG_SLAB_FREELIST_HARDENED s->random = get_random_long(); #endif + if (args->use_freeptr_offset) + s->rcu_freeptr_offset = args->freeptr_offset; + else + s->rcu_freeptr_offset = UINT_MAX; + s->align = args->align; + s->ctor = args->ctor; +#ifdef CONFIG_HARDENED_USERCOPY + s->useroffset = args->useroffset; + s->usersize = args->usersize; +#endif if (!calculate_sizes(s)) goto out; From patchwork Thu Sep 5 07:56:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13791871 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D9A12194123 for ; Thu, 5 Sep 2024 07:57:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523059; cv=none; b=SIeAahyWWP3mO5IfwosioiggKZAV27v4s8h5TyXbdR313NQyCCAAicEwqLYs03sMmGsGrvt+nPVM2QFTFTcB+amIuC2UoZDaP5qlif9gr+ggO/GzH685bLPJrdtzvm+CPLvUjsnJoFHmo0REMeOEnZjrF7eSMGd0spHcH7pCVs8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523059; c=relaxed/simple; bh=Zf6O0R24GKVW0QtPgh/qra6wApA/id2hJZ9TPx/ME9U=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=h38/Hwh//3ZqLiqLCUuTvyp8u8Yh9pdUO7nqxzmOTqQ+aQFJ2DJRSYHOikD6cOPo+v8CBR1cMPfnu4eJdFDIPbMKOJc0hLNEtA+e6n7gH5krTSssB+S0aq4lJ9ynqsi2NgBrDV1f9WP7VNSuYzPTPjO9vYwYsU8sfalhLaD6lcI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=pT0XIJG5; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="pT0XIJG5" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0E0B0C4CEC3; Thu, 5 Sep 2024 07:57:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725523059; bh=Zf6O0R24GKVW0QtPgh/qra6wApA/id2hJZ9TPx/ME9U=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=pT0XIJG5jf6UweHnX3PnusUh7KGghRnwnSRbHnn3xA4ShIU+NmamAtaK6vBrYk2mi H6So88w+HbyvPAA6PDRnnFNIX0mSLfAAN1ytfggZbn5Czz9MLNbo0wEr/26NW+zctS NbjwmB9y12/NwWbSHwQ95npLdiS3NL9NjuEc79Da+qVyQ3KrGqzzXIuDfMpyu6Z60L 90wpyQFvLYUI15pVE3ZdqZtstQx6165hHWkrJaTTk9Y1v70sc8xzly8D/iwtpDHXlq FrL8ueGzjJSOMor9BAJg0BpmFwFpTKugigsDRLEcIUjr0aax7uYvEm1co7kpgV+KT8 qFafnPFKwZwwQ== From: Christian Brauner Date: Thu, 05 Sep 2024 09:56:52 +0200 Subject: [PATCH v4 09/17] slab: remove rcu_freeptr_offset from struct kmem_cache Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240905-work-kmem_cache_args-v4-9-ed45d5380679@kernel.org> References: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> In-Reply-To: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=3996; i=brauner@kernel.org; h=from:subject:message-id; bh=Zf6O0R24GKVW0QtPgh/qra6wApA/id2hJZ9TPx/ME9U=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdTPG+VWf2/MDpjftOT9Dq6BX8s4SRRaTC3vkMS+kit vQu1Ze/OkpZGMS4GGTFFFkc2k3C5ZbzVGw2ytSAmcPKBDKEgYtTACbCEsjwh2d513G2xGq1PKvE d3ETuZ0Wbma+rL342ZUPPB2sn12X2jD8L0n7/Zo7ePHZ5tUKqYdmx3dnZx0WiJ1mnrW6d0fuGlM dbgA= X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Pass down struct kmem_cache_args to calculate_sizes() so we can use args->{use}_freeptr_offset directly. This allows us to remove ->rcu_freeptr_offset from struct kmem_cache. Reviewed-by: Kees Cook Reviewed-by: Jens Axboe Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Vlastimil Babka Signed-off-by: Christian Brauner Reviewed-by: Roman Gushchin --- mm/slab.h | 2 -- mm/slub.c | 25 +++++++------------------ 2 files changed, 7 insertions(+), 20 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index c7a4e0fc3cf1..36ac38e21fcb 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -261,8 +261,6 @@ struct kmem_cache { unsigned int object_size; /* Object size without metadata */ struct reciprocal_value reciprocal_size; unsigned int offset; /* Free pointer offset */ - /* Specific free pointer requested (if not UINT_MAX) */ - unsigned int rcu_freeptr_offset; #ifdef CONFIG_SLUB_CPU_PARTIAL /* Number of per cpu partial objects to keep around */ unsigned int cpu_partial; diff --git a/mm/slub.c b/mm/slub.c index 4719b60215b8..a23c7036cd61 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3916,8 +3916,7 @@ static void *__slab_alloc_node(struct kmem_cache *s, * If the object has been wiped upon free, make sure it's fully initialized by * zeroing out freelist pointer. * - * Note that we also wipe custom freelist pointers specified via - * s->rcu_freeptr_offset. + * Note that we also wipe custom freelist pointers. */ static __always_inline void maybe_wipe_obj_freeptr(struct kmem_cache *s, void *obj) @@ -5141,17 +5140,11 @@ static void set_cpu_partial(struct kmem_cache *s) #endif } -/* Was a valid freeptr offset requested? */ -static inline bool has_freeptr_offset(const struct kmem_cache *s) -{ - return s->rcu_freeptr_offset != UINT_MAX; -} - /* * calculate_sizes() determines the order and the distribution of data within * a slab object. */ -static int calculate_sizes(struct kmem_cache *s) +static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s) { slab_flags_t flags = s->flags; unsigned int size = s->object_size; @@ -5192,7 +5185,7 @@ static int calculate_sizes(struct kmem_cache *s) */ s->inuse = size; - if (((flags & SLAB_TYPESAFE_BY_RCU) && !has_freeptr_offset(s)) || + if (((flags & SLAB_TYPESAFE_BY_RCU) && !args->use_freeptr_offset) || (flags & SLAB_POISON) || s->ctor || ((flags & SLAB_RED_ZONE) && (s->object_size < sizeof(void *) || slub_debug_orig_size(s)))) { @@ -5214,8 +5207,8 @@ static int calculate_sizes(struct kmem_cache *s) */ s->offset = size; size += sizeof(void *); - } else if ((flags & SLAB_TYPESAFE_BY_RCU) && has_freeptr_offset(s)) { - s->offset = s->rcu_freeptr_offset; + } else if ((flags & SLAB_TYPESAFE_BY_RCU) && args->use_freeptr_offset) { + s->offset = args->freeptr_offset; } else { /* * Store freelist pointer near middle of object to keep @@ -5856,10 +5849,6 @@ int do_kmem_cache_create(struct kmem_cache *s, const char *name, #ifdef CONFIG_SLAB_FREELIST_HARDENED s->random = get_random_long(); #endif - if (args->use_freeptr_offset) - s->rcu_freeptr_offset = args->freeptr_offset; - else - s->rcu_freeptr_offset = UINT_MAX; s->align = args->align; s->ctor = args->ctor; #ifdef CONFIG_HARDENED_USERCOPY @@ -5867,7 +5856,7 @@ int do_kmem_cache_create(struct kmem_cache *s, const char *name, s->usersize = args->usersize; #endif - if (!calculate_sizes(s)) + if (!calculate_sizes(args, s)) goto out; if (disable_higher_order_debug) { /* @@ -5877,7 +5866,7 @@ int do_kmem_cache_create(struct kmem_cache *s, const char *name, if (get_order(s->size) > get_order(s->object_size)) { s->flags &= ~DEBUG_METADATA_FLAGS; s->offset = 0; - if (!calculate_sizes(s)) + if (!calculate_sizes(args, s)) goto out; } } From patchwork Thu Sep 5 07:56:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13791872 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A9F061940BC for ; Thu, 5 Sep 2024 07:57:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523063; cv=none; b=hc4hvP9wB1zNV6p4fijp649cKqCaM2GDVvAcu22CxyBmP8t/sqerQtp7yqgLqGpsYRv0UPM/VR0wQ/yc0miO99MVtpC55HsrN/XLibTQ2O43OUQ/Zv5RAzYT+cGJjZN+EHCjUjLvqlQ8spNL7I/UTvZVuUmnPLFUvMAlnKam8QE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523063; c=relaxed/simple; bh=R+pPpbtSd7TpTn7/geWedM/ETd5BwUD/V+znBniHb4Y=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=O124tdf6MlRgOdzr/9NKlH2+sFC3+0J+YqSiOAcJ8XT0reQT58j7mOhS1agmb6DKNqbr7CrIksA9F7ZdZxa6DPdy7KNT8ZyPCNuyuR/HKgiuKWcMy0VNBDGBZcQpIQ8eYWmG3OnK6UrIpb1rHuW8Ajoajo6H0m9wOGxhXLoKLvQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KYwRNlCa; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KYwRNlCa" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CE3E7C4CEC9; Thu, 5 Sep 2024 07:57:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725523063; bh=R+pPpbtSd7TpTn7/geWedM/ETd5BwUD/V+znBniHb4Y=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=KYwRNlCarPHqMLpahTJxt9QurLzF3yW5roIRbxWkYPvzJJeQxMBhK7l4eQPQ+zUht M35sBmieTcbswH4nzib0mhw8RkFBbOqAVbwmmHU/OoHbSo+qB5RbqoTlg5LgtpiJe2 ZxpUT6My1K+7df/sHXZRyMCZD2v3O1xaGGVp+MqlJnCgWYQ/wIJGJMqKmACLK3cJ+u G4sRZDQEHpyvN7XmyRv1T/t4BPteuRlJefS4jEDwUV2FT73yONODrCuVq7wpcJM1ys YdyCNYyN9RhyV2tdcMgFXEMBTQ52q883ZbY4PYRf8JCyL/s/t7Lrf31llAmmhfYMSL dBWL4YbTmXV/w== From: Christian Brauner Date: Thu, 05 Sep 2024 09:56:53 +0200 Subject: [PATCH v4 10/17] slab: port KMEM_CACHE() to struct kmem_cache_args Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240905-work-kmem_cache_args-v4-10-ed45d5380679@kernel.org> References: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> In-Reply-To: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=1255; i=brauner@kernel.org; h=from:subject:message-id; bh=R+pPpbtSd7TpTn7/geWedM/ETd5BwUD/V+znBniHb4Y=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdTPEOPOFxrsVP6W3YHb7n2U915Ges537TFGG2fOPb8 1fEzcPNOkpZGMS4GGTFFFkc2k3C5ZbzVGw2ytSAmcPKBDKEgYtTACbSac/wP3tipJtDqcTMiBy3 Uz/2yU1fupp34Z1bnKeOmxTYdm70vsPIMF3joN7yQj2NZT5p5Yv6JDZw2WS0GUv95Lu0YtZkqcX izAA= X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Make KMEM_CACHE() use struct kmem_cache_args. Reviewed-by: Kees Cook Reviewed-by: Jens Axboe Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Vlastimil Babka Signed-off-by: Christian Brauner Reviewed-by: Roman Gushchin --- include/linux/slab.h | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 79d8c8bca4a4..97117a2fcf34 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -283,9 +283,11 @@ int kmem_cache_shrink(struct kmem_cache *s); * f.e. add ____cacheline_aligned_in_smp to the struct declaration * then the objects will be properly aligned in SMP configurations. */ -#define KMEM_CACHE(__struct, __flags) \ - kmem_cache_create(#__struct, sizeof(struct __struct), \ - __alignof__(struct __struct), (__flags), NULL) +#define KMEM_CACHE(__struct, __flags) \ + __kmem_cache_create_args(#__struct, sizeof(struct __struct), \ + &(struct kmem_cache_args) { \ + .align = __alignof__(struct __struct), \ + }, (__flags)) /* * To whitelist a single field for copying to/from usercopy, use this From patchwork Thu Sep 5 07:56:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13791873 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 496441946B8 for ; Thu, 5 Sep 2024 07:57:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523067; cv=none; b=C8W1+W+GFjL4UnLtn69i36yvcVwbPyxTqtZwTdxRw0w6ehyrf/NYrXcCkjyJdgqVC6yb4tDcOqpym5S/RfmkZxHsaEbEKlnKAjDRQH1U0oRB/yDSTcODfPvDuDwuz3/jvCW/WR3GZhWL5AIS5KIGC2KixbgEeNQ2T4C2jh5h7Eg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523067; c=relaxed/simple; bh=TRtSxtRqOM5QvqM+zpubd3lGKKoppNfgu7eGyWI4SSc=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Sc7Q+MPFvK1Sw8hJSKk/8eQAMxYe64T/cI4Ga4RZ63b2zjsEYYlzTVKvXNHEtx2iLwnKhb3wjYzKz6mgttoGTE5kgTQPKI8OpKAusr6eA9LugYN6bgSd1R3oxyXwT/qkI81MqSEG5KSbBvm8bmeNNUcEjO3qDZoSqrd1ofxCc/4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IV4vVsui; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IV4vVsui" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9D250C4CECB; Thu, 5 Sep 2024 07:57:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725523067; bh=TRtSxtRqOM5QvqM+zpubd3lGKKoppNfgu7eGyWI4SSc=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=IV4vVsuiF90rCnPrP9+D3fjrKdB7nkiU71sfUPuDqDHt7Nt84djcBwn0SvpdezNkT s8NRunt/BRsUKMWxfeqJdGZ5sAM/JrbGl0ilv7KFyPEyhaUi5sd5nRZ4cPW+cQJq73 JJkYqO2Mtk26MOvw28Lmx3oy5lZvOmIoJXIiEMF5tqIBS/DTl7f9F7xs6Z3We7/Upo D5loFUNGZEKdB5hKGPQnbYAUUhJXpd09ZCRWVf29oPY3USCLy09Cxm/mcjcHrbJ1fB O5KIxLW1Q0Djvh+7P7Aqw1aef+KotOa5TUewsPpw/5bgrWZoF0vpC5ZDwaPZXB/kG1 GYM2dMWIdk11Q== From: Christian Brauner Date: Thu, 05 Sep 2024 09:56:54 +0200 Subject: [PATCH v4 11/17] slab: port KMEM_CACHE_USERCOPY() to struct kmem_cache_args Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240905-work-kmem_cache_args-v4-11-ed45d5380679@kernel.org> References: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> In-Reply-To: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=1473; i=brauner@kernel.org; h=from:subject:message-id; bh=TRtSxtRqOM5QvqM+zpubd3lGKKoppNfgu7eGyWI4SSc=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdTPF+IvJk4uvP0VEreI5OXvLo5XUJHsXH71v3RZ+fs yVSb5OmSEcpC4MYF4OsmCKLQ7tJuNxynorNRpkaMHNYmUCGMHBxCsBE/PkYGV59mvLjgeraKQv/ f/+7OO7NgfUHju9nC2DMN3MzPGZndGIew/+UV9sDP31frXnR7BW7xOSZJduq1dj8zRNWWcZGH/6 fY8kPAA== X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Make KMEM_CACHE_USERCOPY() use struct kmem_cache_args. Reviewed-by: Kees Cook Reviewed-by: Jens Axboe Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Vlastimil Babka Signed-off-by: Christian Brauner Reviewed-by: Roman Gushchin --- include/linux/slab.h | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 97117a2fcf34..cb264dded324 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -293,12 +293,13 @@ int kmem_cache_shrink(struct kmem_cache *s); * To whitelist a single field for copying to/from usercopy, use this * macro instead for KMEM_CACHE() above. */ -#define KMEM_CACHE_USERCOPY(__struct, __flags, __field) \ - kmem_cache_create_usercopy(#__struct, \ - sizeof(struct __struct), \ - __alignof__(struct __struct), (__flags), \ - offsetof(struct __struct, __field), \ - sizeof_field(struct __struct, __field), NULL) +#define KMEM_CACHE_USERCOPY(__struct, __flags, __field) \ + __kmem_cache_create_args(#__struct, sizeof(struct __struct), \ + &(struct kmem_cache_args) { \ + .align = __alignof__(struct __struct), \ + .useroffset = offsetof(struct __struct, __field), \ + .usersize = sizeof_field(struct __struct, __field), \ + }, (__flags)) /* * Common kmalloc functions provided by all allocators From patchwork Thu Sep 5 07:56:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13791874 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4AC8D1946B8 for ; Thu, 5 Sep 2024 07:57:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523071; cv=none; b=hCzAvfOlqUgf5PMq2Xhu3RuOn9RhDFHSWfXylS6r4OWTv/GsnhUd6yujvzuu7vp7BHxIzXRr8nmeUPKYVzf0uChzvxWOuSLG2r2+zwTEK1aA7Ua/4gYRyzDk8NBtoejx6U1Icl7gfAOkOR6z6+04BQL48EcVm1ZfoVjiENM1yzw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523071; c=relaxed/simple; bh=xO3vcgDDnoCbF7ELZ5lnFORM5yLOC/pUkBkeRwkJlfs=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Jr/H3ySyiBvoFCLl1ISrFey+MUaOrLb3WjXJV/75gDxGbMGx9sIPho6Hantr9pbALjW5+CdDEoZ61DNccie21dslCHZY35o2Au+TlQHjK59ofJ2P2eW/cTZu8KKmcm9xAoCUlX95tZEccqJ5+JCnZ55o/u3ej6Cq2NXUkRJXVj8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=I7HvquUc; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="I7HvquUc" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A0D83C4CEC9; Thu, 5 Sep 2024 07:57:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725523071; bh=xO3vcgDDnoCbF7ELZ5lnFORM5yLOC/pUkBkeRwkJlfs=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=I7HvquUcEms6uaml8hK37xscwhQS72RhSKTnUPyh0vN66t2ZzapSWd1Gvi8uO6rhw umvVp1zmu8b5gVdoEmQ/RkZVPFUZFdUddhPh6Z/DD38rDAsP8xSVXqXx6ZTX1QkApz 89BQ+AZbPcJBU/XOw9PXkxOKGsgt/flpFUZvv+hJzOMDTzNxLwcjMGeBnU+fRS/+qG YGuLmL7irHDeC4iYwA9A8x36GNAJ7vj6gYJ3uhr6CIif750S1YMIKkrAzdal+WxfzE gA542w9e/qVTZIpPda7NJzSMIT52pzB4GuW4/Ijqve1H4S0Xg5R030ZVEkEuL8kt0h QZX8zNQI101Mw== From: Christian Brauner Date: Thu, 05 Sep 2024 09:56:55 +0200 Subject: [PATCH v4 12/17] slab: create kmem_cache_create() compatibility layer Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240905-work-kmem_cache_args-v4-12-ed45d5380679@kernel.org> References: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> In-Reply-To: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=3953; i=brauner@kernel.org; h=from:subject:message-id; bh=xO3vcgDDnoCbF7ELZ5lnFORM5yLOC/pUkBkeRwkJlfs=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdTPGWnv7hec00zY7uWcHZa36fWJ7VU33/TDKP6lqLz PW/14vGd5SyMIhxMciKKbI4tJuEyy3nqdhslKkBM4eVCWQIAxenAExk1kJGhmWHXumx3lsayv60 +XC6uci3R/ZlmQHCfw+In94cNDlrqxQjw/v1V2ZwbjStnHKhNWCVFYNj/4XntZ8kj/4XWR3ftMu pggcA X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Use _Generic() to create a compatibility layer that type switches on the third argument to either call __kmem_cache_create() or __kmem_cache_create_args(). If NULL is passed for the struct kmem_cache_args argument use default args making porting for callers that don't care about additional arguments easy. Reviewed-by: Kees Cook Reviewed-by: Jens Axboe Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Roman Gushchin --- include/linux/slab.h | 29 ++++++++++++++++++++++++++--- mm/slab_common.c | 10 +++++----- 2 files changed, 31 insertions(+), 8 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index cb264dded324..f74ceb788ac1 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -261,9 +261,10 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, unsigned int object_size, struct kmem_cache_args *args, slab_flags_t flags); -struct kmem_cache *kmem_cache_create(const char *name, unsigned int size, - unsigned int align, slab_flags_t flags, - void (*ctor)(void *)); + +struct kmem_cache *__kmem_cache_create(const char *name, unsigned int size, + unsigned int align, slab_flags_t flags, + void (*ctor)(void *)); struct kmem_cache *kmem_cache_create_usercopy(const char *name, unsigned int size, unsigned int align, slab_flags_t flags, @@ -272,6 +273,28 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name, struct kmem_cache *kmem_cache_create_rcu(const char *name, unsigned int size, unsigned int freeptr_offset, slab_flags_t flags); + +/* If NULL is passed for @args, use this variant with default arguments. */ +static inline struct kmem_cache * +__kmem_cache_default_args(const char *name, unsigned int size, + struct kmem_cache_args *args, + slab_flags_t flags) +{ + struct kmem_cache_args kmem_default_args = {}; + + /* Make sure we don't get passed garbage. */ + if (WARN_ON_ONCE(args)) + return NULL; + + return __kmem_cache_create_args(name, size, &kmem_default_args, flags); +} + +#define kmem_cache_create(__name, __object_size, __args, ...) \ + _Generic((__args), \ + struct kmem_cache_args *: __kmem_cache_create_args, \ + void *: __kmem_cache_default_args, \ + default: __kmem_cache_create)(__name, __object_size, __args, __VA_ARGS__) + void kmem_cache_destroy(struct kmem_cache *s); int kmem_cache_shrink(struct kmem_cache *s); diff --git a/mm/slab_common.c b/mm/slab_common.c index 19ae3dd6e36f..418459927670 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -383,7 +383,7 @@ kmem_cache_create_usercopy(const char *name, unsigned int size, EXPORT_SYMBOL(kmem_cache_create_usercopy); /** - * kmem_cache_create - Create a cache. + * __kmem_cache_create - Create a cache. * @name: A string which is used in /proc/slabinfo to identify this cache. * @size: The size of objects to be created in this cache. * @align: The required alignment for the objects. @@ -407,9 +407,9 @@ EXPORT_SYMBOL(kmem_cache_create_usercopy); * * Return: a pointer to the cache on success, NULL on failure. */ -struct kmem_cache * -kmem_cache_create(const char *name, unsigned int size, unsigned int align, - slab_flags_t flags, void (*ctor)(void *)) +struct kmem_cache *__kmem_cache_create(const char *name, unsigned int size, + unsigned int align, slab_flags_t flags, + void (*ctor)(void *)) { struct kmem_cache_args kmem_args = { .align = align, @@ -418,7 +418,7 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align, return __kmem_cache_create_args(name, size, &kmem_args, flags); } -EXPORT_SYMBOL(kmem_cache_create); +EXPORT_SYMBOL(__kmem_cache_create); /** * kmem_cache_create_rcu - Create a SLAB_TYPESAFE_BY_RCU cache. From patchwork Thu Sep 5 07:56:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13791875 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 468CD1946C3 for ; Thu, 5 Sep 2024 07:57:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523075; cv=none; b=qCgSgS8TK14F5K30YzcBnnBMSSaLzr2UiqQk4s+XF6HlZsoBj76bdLD0ZbMoiWakvD75gxXsVZkjwQm1xyjwBAJ89uUeMYZhlWhLFb+aZiGdTX2xItBNShRdJ4Wcw+az3tcRZuKEPGt5a9xrNBl63Q7VtgZOaHTjMjItQs7ILdA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523075; c=relaxed/simple; bh=i+qfsAgWogADWXKIX6Ox+SamBJe0NAxzXt2tozWQS4Y=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=r/n+EY552TcYuluq7oTE6G/TJXz+kHxBvIsuQ8YEngwNRQ2MaTX6ukzt5zI9Ka7Y1+2Kuxz26NIz3LS4HqNhP8fMS/GyaH6J22nxYUDlU8YQfxXZG8z/dKl7CG7SD7x+pcJSsTCSrE+izTRPsz8PddkXd/+iHSh7E5srfSYpmcE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mYkXOr7x; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mYkXOr7x" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9B1F7C4AF09; Thu, 5 Sep 2024 07:57:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725523075; bh=i+qfsAgWogADWXKIX6Ox+SamBJe0NAxzXt2tozWQS4Y=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=mYkXOr7xS4Z4mkCbI0v9BXGFcdBL//failIOy4yeoCV0iFHDbwEGNpZ5lUT96/2rM FzuwjOabbXeq4zn+r6w1OuIBW8p/DnVBT7j6rukPS3OyCyC6AN3aWf49VSYSpI0VML 2UJwPFaz4/GJtXg4EXkPaof4zAh7spKsPz8nMcqahR/WVWwLm8l8FFx+JZP5Lh/HoQ 24x+U2rzZbmqfLeIKcCBfI3DZenROsICqGWZKO5nWoZIuP2vAt3nRO1ffbjo2ZXPYA MR/QICqC6xGnAAmxZyKiOcMTZ+C0BBDqJXU9I69e63Eh/30cpfHZvxcjNz43rgheQ3 CgE25VlJ6vPAQ== From: Christian Brauner Date: Thu, 05 Sep 2024 09:56:56 +0200 Subject: [PATCH v4 13/17] file: port to struct kmem_cache_args Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240905-work-kmem_cache_args-v4-13-ed45d5380679@kernel.org> References: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> In-Reply-To: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=1128; i=brauner@kernel.org; h=from:subject:message-id; bh=i+qfsAgWogADWXKIX6Ox+SamBJe0NAxzXt2tozWQS4Y=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdTPFeMkFxdeKVKz5uTZv//oozC6j/eSV1QtD8yNSbF 3dazFPS6ihlYRDjYpAVU2RxaDcJl1vOU7HZKFMDZg4rE8gQBi5OAZiIawrDPzX+vdEdqptkD+78 32r3fWVw3YzH7oZLj54zjVh1yzvumwnDP+UO2TWtxg38Rrvuz43yCxZ9w/I3XOsE35Oppb7yIhl /+QE= X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Port filp_cache to struct kmem_cache_args. Reviewed-by: Kees Cook Reviewed-by: Jens Axboe Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Vlastimil Babka Signed-off-by: Christian Brauner Reviewed-by: Roman Gushchin --- fs/file_table.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/fs/file_table.c b/fs/file_table.c index 3ef558f27a1c..861c03608e83 100644 --- a/fs/file_table.c +++ b/fs/file_table.c @@ -511,9 +511,14 @@ EXPORT_SYMBOL(__fput_sync); void __init files_init(void) { - filp_cachep = kmem_cache_create_rcu("filp", sizeof(struct file), - offsetof(struct file, f_freeptr), - SLAB_HWCACHE_ALIGN | SLAB_PANIC | SLAB_ACCOUNT); + struct kmem_cache_args args = { + .use_freeptr_offset = true, + .freeptr_offset = offsetof(struct file, f_freeptr), + }; + + filp_cachep = kmem_cache_create("filp", sizeof(struct file), &args, + SLAB_HWCACHE_ALIGN | SLAB_PANIC | + SLAB_ACCOUNT | SLAB_TYPESAFE_BY_RCU); percpu_counter_init(&nr_files, 0, GFP_KERNEL); } From patchwork Thu Sep 5 07:56:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13791876 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A0976193422 for ; Thu, 5 Sep 2024 07:57:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523079; cv=none; b=aDCxXSOw8E4TQqw5yoaO95SH6MfBLv63RbwqJFZvbnPq32TEZ9qWcfKrXnVEUaW0VfMkgKi2B98jMQxeF6xGFCDJQfD5Nxu+ym7BDo7S5B+20+B9oNAHguYXfhtBVvWoX4sgB9fonA6nRrrSWbFslrjQklHyye0rjH7uUmz1mDA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523079; c=relaxed/simple; bh=c9WyXR3UutI5YuOZDzru6g9A7doJ0To6Z/yCENECssA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=HnYXOZOhTS6+C3FffjFXG22S+K1gwU27j5uB5yyGc9zt4gQ9LON16PUmro3J6d8OIhQAMtXEyMxSf1rOfB9YUjqj7WslNa03nxGLe5/9qpFBiBekiiX6qX4PNoqSHYyO+WmLS9mxonLRLt3DQb7F0t5lVDLF8ea5CbJwjDw8CT8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Cm9Sd2Di; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Cm9Sd2Di" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9920BC4CEC4; Thu, 5 Sep 2024 07:57:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725523079; bh=c9WyXR3UutI5YuOZDzru6g9A7doJ0To6Z/yCENECssA=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=Cm9Sd2DiCmnN0M4VfGZk7Ktm5XhL7pstZXSAV2/tXTU0vjUTxs17RLf9hjLGtOvjr aJuzjiNbaXAw0LLzOGR4Kkckwn0fnE09r3ZmDZ/kHmLqzpeCr1buh5ukhX4ArhnWrg PUQIGMuRq8rnFXLrp2CEa8HmM52xfYv66EJKiCyh1q3wy8/FOiQXi7vs9Tah2zsODe CosxMcX8TUjUa8lTOHalsD5psOIeXV3QgYlKjByOBrBU7LF4R/gPR00gARG+pwrohn DDHKwylYuoaxckibZsHQvhMGDCp7+cnBl1N65RZblKCkMhS8juxVlCdR9cqGkOs/5Y AZuImLS0sp9dg== From: Christian Brauner Date: Thu, 05 Sep 2024 09:56:57 +0200 Subject: [PATCH v4 14/17] slab: remove kmem_cache_create_rcu() Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240905-work-kmem_cache_args-v4-14-ed45d5380679@kernel.org> References: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> In-Reply-To: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=3200; i=brauner@kernel.org; h=from:subject:message-id; bh=c9WyXR3UutI5YuOZDzru6g9A7doJ0To6Z/yCENECssA=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdTPG+F8+58dldnj9qzNtvnnm8aYfemt7vfz66bb2oG iHMce3Hwo5SFgYxLgZZMUUWh3aTcLnlPBWbjTI1YOawMoEMYeDiFICJxGsy/M/5INce7OTJUvjF Z33SiZQN9YV1DbtmHLIulz2wyily3XOGf/aFNUbt6q4vZqyd+vP26xvmEqsurtf8Lms8z/OtwNX yg+wA X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Now that we have ported all users of kmem_cache_create_rcu() to struct kmem_cache_args the function is unused and can be removed. Reviewed-by: Kees Cook Reviewed-by: Jens Axboe Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Vlastimil Babka Signed-off-by: Christian Brauner Reviewed-by: Roman Gushchin --- include/linux/slab.h | 3 --- mm/slab_common.c | 43 ------------------------------------------- 2 files changed, 46 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index f74ceb788ac1..8246f9b28f43 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -270,9 +270,6 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name, slab_flags_t flags, unsigned int useroffset, unsigned int usersize, void (*ctor)(void *)); -struct kmem_cache *kmem_cache_create_rcu(const char *name, unsigned int size, - unsigned int freeptr_offset, - slab_flags_t flags); /* If NULL is passed for @args, use this variant with default arguments. */ static inline struct kmem_cache * diff --git a/mm/slab_common.c b/mm/slab_common.c index 418459927670..9133b9fafcb1 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -420,49 +420,6 @@ struct kmem_cache *__kmem_cache_create(const char *name, unsigned int size, } EXPORT_SYMBOL(__kmem_cache_create); -/** - * kmem_cache_create_rcu - Create a SLAB_TYPESAFE_BY_RCU cache. - * @name: A string which is used in /proc/slabinfo to identify this cache. - * @size: The size of objects to be created in this cache. - * @freeptr_offset: The offset into the memory to the free pointer - * @flags: SLAB flags - * - * Cannot be called within an interrupt, but can be interrupted. - * - * See kmem_cache_create() for an explanation of possible @flags. - * - * By default SLAB_TYPESAFE_BY_RCU caches place the free pointer outside - * of the object. This might cause the object to grow in size. Callers - * that have a reason to avoid this can specify a custom free pointer - * offset in their struct where the free pointer will be placed. - * - * Note that placing the free pointer inside the object requires the - * caller to ensure that no fields are invalidated that are required to - * guard against object recycling (See SLAB_TYPESAFE_BY_RCU for - * details.). - * - * Using zero as a value for @freeptr_offset is valid. To request no - * offset UINT_MAX must be specified. - * - * Note that @ctor isn't supported with custom free pointers as a @ctor - * requires an external free pointer. - * - * Return: a pointer to the cache on success, NULL on failure. - */ -struct kmem_cache *kmem_cache_create_rcu(const char *name, unsigned int size, - unsigned int freeptr_offset, - slab_flags_t flags) -{ - struct kmem_cache_args kmem_args = { - .freeptr_offset = freeptr_offset, - .use_freeptr_offset = true, - }; - - return __kmem_cache_create_args(name, size, &kmem_args, - flags | SLAB_TYPESAFE_BY_RCU); -} -EXPORT_SYMBOL(kmem_cache_create_rcu); - static struct kmem_cache *kmem_buckets_cache __ro_after_init; /** From patchwork Thu Sep 5 07:56:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13791877 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2BE7819408C for ; Thu, 5 Sep 2024 07:58:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523083; cv=none; b=VPNzJzf7EVn4kdmuj5Bp60YY+rs0KWl1L96977w4WKW2shQmTcSL2UPQKyDkLrV4QFApmuMPSYaufVqZllZOTDOYDyZnvD7z/BR9+0f636utUB/QK7IJxdgaBtBAgHd7ft7glIGyJDXnXoFQmbRumVsuwQHIhTjgDv6dlXd2kJc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523083; c=relaxed/simple; bh=8MPF6aXbdUIaNsM6rCGkaQsu5CsscD0F2Lvkbk763C4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=JmXX1d0s8lRLq642Wg2vGsOEhx5Q54bai1USFywhTMPE0A1d7ZUSiepHHFhqSPL62crIbi9yajojKRXNM5PF0bG9LAUWWabdSDkn3uXDXPuRALClZJNLy6Zj00bUX51RM1qG5LIJ1Pqksx8Z2KtCMcFVJYU9bqXveQsu4Y4sdGw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=jCEn44ZW; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="jCEn44ZW" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 97A91C4CEC7; Thu, 5 Sep 2024 07:57:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725523083; bh=8MPF6aXbdUIaNsM6rCGkaQsu5CsscD0F2Lvkbk763C4=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=jCEn44ZWRCSA9HVoaikbAlgBU3BAXgX6znj1B8SQc1aXfEfaCf+J3Q4fxzbhZP5W6 rtBpLbe0X4Ib1EkGzQv0ekZvpN+Vm5TRW1iv0InpFeutCESIk0Oq2Iv3yKZ6p0qIl6 QSVsvTyShitYfBic6Btst0IgZwZnx7Vm99PJpan+xTk8Bls4i+lPzBOAM5x+yiHbh8 jXTpHM8wbwV/yThzpFhtxakWzjPis+aVldz5ERTyEslB6axT1eLYBPs9uVIcuBvkF5 VZtYBHZKRSMFz4439dzA3m9KwZl7r/7o9RkDdIIAKGVCkGGI69dWBHpykvc80J9e98 D7Fe9jhM5aZqg== From: Christian Brauner Date: Thu, 05 Sep 2024 09:56:58 +0200 Subject: [PATCH v4 15/17] slab: make kmem_cache_create_usercopy() static inline Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240905-work-kmem_cache_args-v4-15-ed45d5380679@kernel.org> References: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> In-Reply-To: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=4607; i=brauner@kernel.org; h=from:subject:message-id; bh=8MPF6aXbdUIaNsM6rCGkaQsu5CsscD0F2Lvkbk763C4=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdTPHOYd487fKTZzb1T4RE484+Ejn26RbHH7HqspR89 X8N+T0bOkpZGMS4GGTFFFkc2k3C5ZbzVGw2ytSAmcPKBDKEgYtTACbC2crw39fm9y5vhWt7lc81 rOgSP+6hsrlmx0nlI8KaUU9rYqKlLzL8M8xa73bneNjG24EpL+wmRj02nvFzPZfN95w/rQ8/mJs 9ZAYA X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Make kmem_cache_create_usercopy() a static inline function. Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Roman Gushchin --- include/linux/slab.h | 49 ++++++++++++++++++++++++++++++++++++++++++++----- mm/slab_common.c | 45 --------------------------------------------- 2 files changed, 44 insertions(+), 50 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 8246f9b28f43..d744397aa46f 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -265,11 +265,50 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, struct kmem_cache *__kmem_cache_create(const char *name, unsigned int size, unsigned int align, slab_flags_t flags, void (*ctor)(void *)); -struct kmem_cache *kmem_cache_create_usercopy(const char *name, - unsigned int size, unsigned int align, - slab_flags_t flags, - unsigned int useroffset, unsigned int usersize, - void (*ctor)(void *)); + +/** + * kmem_cache_create_usercopy - Create a cache with a region suitable + * for copying to userspace + * @name: A string which is used in /proc/slabinfo to identify this cache. + * @size: The size of objects to be created in this cache. + * @align: The required alignment for the objects. + * @flags: SLAB flags + * @useroffset: Usercopy region offset + * @usersize: Usercopy region size + * @ctor: A constructor for the objects. + * + * Cannot be called within a interrupt, but can be interrupted. + * The @ctor is run when new pages are allocated by the cache. + * + * The flags are + * + * %SLAB_POISON - Poison the slab with a known test pattern (a5a5a5a5) + * to catch references to uninitialised memory. + * + * %SLAB_RED_ZONE - Insert `Red` zones around the allocated memory to check + * for buffer overruns. + * + * %SLAB_HWCACHE_ALIGN - Align the objects in this cache to a hardware + * cacheline. This can be beneficial if you're counting cycles as closely + * as davem. + * + * Return: a pointer to the cache on success, NULL on failure. + */ +static inline struct kmem_cache * +kmem_cache_create_usercopy(const char *name, unsigned int size, + unsigned int align, slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)) +{ + struct kmem_cache_args kmem_args = { + .align = align, + .ctor = ctor, + .useroffset = useroffset, + .usersize = usersize, + }; + + return __kmem_cache_create_args(name, size, &kmem_args, flags); +} /* If NULL is passed for @args, use this variant with default arguments. */ static inline struct kmem_cache * diff --git a/mm/slab_common.c b/mm/slab_common.c index 9133b9fafcb1..3477a3918afd 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -337,51 +337,6 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, } EXPORT_SYMBOL(__kmem_cache_create_args); -/** - * kmem_cache_create_usercopy - Create a cache with a region suitable - * for copying to userspace - * @name: A string which is used in /proc/slabinfo to identify this cache. - * @size: The size of objects to be created in this cache. - * @align: The required alignment for the objects. - * @flags: SLAB flags - * @useroffset: Usercopy region offset - * @usersize: Usercopy region size - * @ctor: A constructor for the objects. - * - * Cannot be called within a interrupt, but can be interrupted. - * The @ctor is run when new pages are allocated by the cache. - * - * The flags are - * - * %SLAB_POISON - Poison the slab with a known test pattern (a5a5a5a5) - * to catch references to uninitialised memory. - * - * %SLAB_RED_ZONE - Insert `Red` zones around the allocated memory to check - * for buffer overruns. - * - * %SLAB_HWCACHE_ALIGN - Align the objects in this cache to a hardware - * cacheline. This can be beneficial if you're counting cycles as closely - * as davem. - * - * Return: a pointer to the cache on success, NULL on failure. - */ -struct kmem_cache * -kmem_cache_create_usercopy(const char *name, unsigned int size, - unsigned int align, slab_flags_t flags, - unsigned int useroffset, unsigned int usersize, - void (*ctor)(void *)) -{ - struct kmem_cache_args kmem_args = { - .align = align, - .ctor = ctor, - .useroffset = useroffset, - .usersize = usersize, - }; - - return __kmem_cache_create_args(name, size, &kmem_args, flags); -} -EXPORT_SYMBOL(kmem_cache_create_usercopy); - /** * __kmem_cache_create - Create a cache. * @name: A string which is used in /proc/slabinfo to identify this cache. From patchwork Thu Sep 5 07:56:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13791878 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E502192D8F for ; Thu, 5 Sep 2024 07:58:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523087; cv=none; b=otw71nbBw4YtS+n/IkecLaD7NDzGqmz3EyGSIl+gjPrfEaLFIuixthJ1y/zObNQOoRg24RcfDgtl+BCrBUqxXqkJjhY9P7HlWy5ZORKKd6SoB/eVAflVN74tP6otSTmTUPIPyfB1AlEie+c9+E0dYwG0iuCnCMlBU0KkVoIMMxM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523087; c=relaxed/simple; bh=rwJKoUBiD4opsF3q3lc+9pnVXWY01qThhoh7PngENwk=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=WjuxKIHaozJFH7Uhl4BzMKaLto0b7D/+vQxkMz1nZSbinD5yWsd6+XRpLy8EBBWmE6HkDmWNknOnr21rMhL8pQ6zZXeM2X/gTvbxVZi6ZFxI4r8PPcx1gzpdx9kjlK1YoOJe7I21iCbB0taxjy7PbdKRJmKZ8ysHQe1Ar5Q7xj4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UqxXrGSk; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UqxXrGSk" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7F903C4CEC9; Thu, 5 Sep 2024 07:58:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725523086; bh=rwJKoUBiD4opsF3q3lc+9pnVXWY01qThhoh7PngENwk=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=UqxXrGSksmpwRA6ISWIfcsy2RBc0SFe/JRNtZg49zhcODYuHDz3xFWj156OUJxLZ+ wlgA48rZ4aVjDeDgAisVGRtKKDZSAUgVWYuH02G8IAvR1flVT5sBrD/kRxow7t9U/b avyDBgagBqJmDSz3qCSe51VQ7h6c2PS6AnttGq5P2P8rnS3PPs8BJMOKtM7B0O91yS TrSImZBnPOaVASgyaAjqy93Uugn9w9qq7b1L9P0Er6SKg16c3pVSoYGQMouoQRQjrO 0FMi80Rd1VmLqkkLYKgzSdFQJeQW52mEZPHMSWtDuXWaiDK9hDfmx/JgEGNqxTI1tB U71j7fw3uPvdg== From: Christian Brauner Date: Thu, 05 Sep 2024 09:56:59 +0200 Subject: [PATCH v4 16/17] slab: make __kmem_cache_create() static inline Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240905-work-kmem_cache_args-v4-16-ed45d5380679@kernel.org> References: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> In-Reply-To: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=2885; i=brauner@kernel.org; h=from:subject:message-id; bh=rwJKoUBiD4opsF3q3lc+9pnVXWY01qThhoh7PngENwk=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdTPEuPu2j8kRPI0DSfXdC71rLqwZ7b+4XeG9n8f8WI 9OWKwblHaUsDGJcDLJiiiwO7Sbhcst5KjYbZWrAzGFlAhnCwMUpABMRyWf4K+S264SiRNnN4pCy otf59c/vPTdSWrb90XZ2I713D2RXZjIyzJ4mt6dx8kq790bb0i95+X2PfXjqj82S4I29cxZ9uaN 6mhsA X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Make __kmem_cache_create() a static inline function. Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Roman Gushchin --- include/linux/slab.h | 13 ++++++++++--- mm/slab_common.c | 38 -------------------------------------- 2 files changed, 10 insertions(+), 41 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index d744397aa46f..597f6913cc0f 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -261,10 +261,17 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, unsigned int object_size, struct kmem_cache_args *args, slab_flags_t flags); +static inline struct kmem_cache * +__kmem_cache_create(const char *name, unsigned int size, unsigned int align, + slab_flags_t flags, void (*ctor)(void *)) +{ + struct kmem_cache_args kmem_args = { + .align = align, + .ctor = ctor, + }; -struct kmem_cache *__kmem_cache_create(const char *name, unsigned int size, - unsigned int align, slab_flags_t flags, - void (*ctor)(void *)); + return __kmem_cache_create_args(name, size, &kmem_args, flags); +} /** * kmem_cache_create_usercopy - Create a cache with a region suitable diff --git a/mm/slab_common.c b/mm/slab_common.c index 3477a3918afd..30000dcf0736 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -337,44 +337,6 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, } EXPORT_SYMBOL(__kmem_cache_create_args); -/** - * __kmem_cache_create - Create a cache. - * @name: A string which is used in /proc/slabinfo to identify this cache. - * @size: The size of objects to be created in this cache. - * @align: The required alignment for the objects. - * @flags: SLAB flags - * @ctor: A constructor for the objects. - * - * Cannot be called within a interrupt, but can be interrupted. - * The @ctor is run when new pages are allocated by the cache. - * - * The flags are - * - * %SLAB_POISON - Poison the slab with a known test pattern (a5a5a5a5) - * to catch references to uninitialised memory. - * - * %SLAB_RED_ZONE - Insert `Red` zones around the allocated memory to check - * for buffer overruns. - * - * %SLAB_HWCACHE_ALIGN - Align the objects in this cache to a hardware - * cacheline. This can be beneficial if you're counting cycles as closely - * as davem. - * - * Return: a pointer to the cache on success, NULL on failure. - */ -struct kmem_cache *__kmem_cache_create(const char *name, unsigned int size, - unsigned int align, slab_flags_t flags, - void (*ctor)(void *)) -{ - struct kmem_cache_args kmem_args = { - .align = align, - .ctor = ctor, - }; - - return __kmem_cache_create_args(name, size, &kmem_args, flags); -} -EXPORT_SYMBOL(__kmem_cache_create); - static struct kmem_cache *kmem_buckets_cache __ro_after_init; /** From patchwork Thu Sep 5 07:57:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13791879 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 41B7D193422 for ; Thu, 5 Sep 2024 07:58:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523091; cv=none; b=cELPeyIlcFEqZ+PeQN7X7Joqm2L8cdsDXH1aS/Tf8wD5PJJ10sL/ZqQP6xG77ZAHs70HVyMwAFTXo6vK8OssqO6DFDNMEuXJXwX5BueGJIhLyw0Hh0FgdR/bTQ8olPYrstn1rCXGe9QAcXYp3WADCzNbSmuQ8fkQ8lBhP53B84U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725523091; c=relaxed/simple; bh=EjTjGH3N7I+He4i3zTdEnIsNUEnQ9qSuzD1HUDVnA88=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=kTfwcAzmZzirQrp8DmjkmsL9KlvxMzqOAzFBv4N+O7J1jubsKJxO84Cgw/kLdyKjDjdqjkaWAbSlD8n2/fW48rt86iByPBZbabTkDAkvOwBQR6YjVALjybzmfwzS+TJ0puyJ0WFAUZ5U47zq+/Zk2LjYw4ESY0M2W5kY5Wn0yRc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=n/cii1Y9; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="n/cii1Y9" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6815AC4CEC3; Thu, 5 Sep 2024 07:58:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725523090; bh=EjTjGH3N7I+He4i3zTdEnIsNUEnQ9qSuzD1HUDVnA88=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=n/cii1Y9UaMO6oLLzq+PA4Vy0/MwZIQeJ9yLODD3MvOMqqhPRxWm1x7GIX25GI/qp G690aWfdS5p10x5FCW5PqvhbTqhZefA0icmgivaU150EthEStFU1AgYtFd3GeaOJQy U4EScFhr20WPqHmcHBwGz1opcb326Vs6aCqxqTaMdv5qaB+uU16A1bpHtGHEOJoNG8 cG/Y6jAnwZWHfrKLMKWP4gsjwHZdhNZHV1UgcQkVza8kALbZknFpAhxn9Vbp65HgHn YDHbbJ5pm08OwI8mAypTWkqRLG7PQy+k8l1qGm1E1kBRtOq1owOgJziQNM69rdMVQo PdkjcvlwIrOyg== From: Christian Brauner Date: Thu, 05 Sep 2024 09:57:00 +0200 Subject: [PATCH v4 17/17] io_uring: port to struct kmem_cache_args Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240905-work-kmem_cache_args-v4-17-ed45d5380679@kernel.org> References: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> In-Reply-To: <20240905-work-kmem_cache_args-v4-0-ed45d5380679@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=1730; i=brauner@kernel.org; h=from:subject:message-id; bh=EjTjGH3N7I+He4i3zTdEnIsNUEnQ9qSuzD1HUDVnA88=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdTPHhj1L7r/Bk2qwz7j/NNA3ZPkds8Gb9+f4l61uHu nD93VXpHaUsDGJcDLJiiiwO7Sbhcst5KjYbZWrAzGFlAhnCwMUpABPxOc/wV4Cf57l0b/FcH1+T VFkn/6fcDXo3btpERvZv8f637gjjBob/SW8yPtZ1fvr0p+QH91+O+GOvnQR7eBceb745USDVSoK VDQA= X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Port req_cachep to struct kmem_cache_args. Reviewed-by: Kees Cook Reviewed-by: Jens Axboe Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Vlastimil Babka Signed-off-by: Christian Brauner --- io_uring/io_uring.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 3942db160f18..d9d721d1424e 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3638,6 +3638,11 @@ SYSCALL_DEFINE2(io_uring_setup, u32, entries, static int __init io_uring_init(void) { + struct kmem_cache_args kmem_args = { + .useroffset = offsetof(struct io_kiocb, cmd.data), + .usersize = sizeof_field(struct io_kiocb, cmd.data), + }; + #define __BUILD_BUG_VERIFY_OFFSET_SIZE(stype, eoffset, esize, ename) do { \ BUILD_BUG_ON(offsetof(stype, ename) != eoffset); \ BUILD_BUG_ON(sizeof_field(stype, ename) != esize); \ @@ -3722,12 +3727,9 @@ static int __init io_uring_init(void) * range, and HARDENED_USERCOPY will complain if we haven't * correctly annotated this range. */ - req_cachep = kmem_cache_create_usercopy("io_kiocb", - sizeof(struct io_kiocb), 0, - SLAB_HWCACHE_ALIGN | SLAB_PANIC | - SLAB_ACCOUNT | SLAB_TYPESAFE_BY_RCU, - offsetof(struct io_kiocb, cmd.data), - sizeof_field(struct io_kiocb, cmd.data), NULL); + req_cachep = kmem_cache_create("io_kiocb", sizeof(struct io_kiocb), &kmem_args, + SLAB_HWCACHE_ALIGN | SLAB_PANIC | SLAB_ACCOUNT | + SLAB_TYPESAFE_BY_RCU); io_buf_cachep = KMEM_CACHE(io_buffer, SLAB_HWCACHE_ALIGN | SLAB_PANIC | SLAB_ACCOUNT);