From patchwork Wed Sep 4 10:21:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13790362 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8E53CA0ED3 for ; Wed, 4 Sep 2024 10:22:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B6286B00BF; Wed, 4 Sep 2024 06:22:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 53D346B00CD; Wed, 4 Sep 2024 06:22:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 404BC6B00D0; Wed, 4 Sep 2024 06:22:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 200516B00BF for ; Wed, 4 Sep 2024 06:22:30 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B6ADF80F93 for ; Wed, 4 Sep 2024 10:22:29 +0000 (UTC) X-FDA: 82526666418.18.3E8A89F Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf25.hostedemail.com (Postfix) with ESMTP id D957CA0015 for ; Wed, 4 Sep 2024 10:22:27 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=us9HEmIz; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf25.hostedemail.com: domain of brauner@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=brauner@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725445324; a=rsa-sha256; cv=none; b=iEzXyDSRy9ldFz+3erw2Q1tidTPNiEYg9KeXX7xGwonwVqem39gv3nus8BQuz1bLrXWY4+ f/ztMzL55m3zDxpX+jy71udjs6PZkQB+ShTPIC8sSPCJTPkFOZEu2zz5b4i0q4+ABBNUx8 GAiEyeKYt7mHjJH0V2R/Tfyzm9VRZVQ= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=us9HEmIz; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf25.hostedemail.com: domain of brauner@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=brauner@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725445324; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=W0IBUcs4bR+e/ACSTkKoJTzE3j3GJoZfBKXsggNrr5w=; b=33Jp7+HSL3oWzIzBPPqNzEYYFoEfIVZa7j2/tVhvl14oznp05Zs9fuf9lH+iBq2zhB5bkN FZH7/tMGpn2d+dyaxuTYiCS2T0IxijDk8Cpdi6ZEK/OphlJeC/5dtXPKrjEhwEu/mmMEFb PajE6o9tWGS3mjUFcawLRCmHEsCCKEk= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 9052F5C56EA; Wed, 4 Sep 2024 10:22:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 35D50C4CEC6; Wed, 4 Sep 2024 10:22:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725445346; bh=8hORSdr9O/EqElT2ajhJQ8gEdIqY6bU8s9Nue9pitwk=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=us9HEmIzCgJ1DP9lrROrQhAb2zX+L5BEnnhdKkWq6pq0V3MhZk2lZ4bevheiquKzS 60XaBJaYfYppDbcqI/WDixM0L7NIaJA4M8LKXgTWMK0fphczOVIiFOsf9Atupsb1bj UFfnioh3A2OD/lsX+87sFGf2cMJ/Xtal5GknHUeK0aBeZV2Hc3G2OFN09CIPtZSCNH Bf9Xglj3Y62QKfyzei6gxD5Ilr4phgEjtoQX5dDDD2zqvwnJ9Fxs8c2MqhIwTYArAf Q8EafrAHwYLvHu3mDnBU/gu2RKAk59ffY6BBz7AxxxC9klbM/s1gJ+x4RApnjLJkVA iLHmH+bMcC5Gw== From: Christian Brauner Date: Wed, 04 Sep 2024 12:21:14 +0200 Subject: [PATCH v3 09/17] slab: remove rcu_freeptr_offset from struct kmem_cache MIME-Version: 1.0 Message-Id: <20240904-work-kmem_cache_args-v3-9-05db2179a8c2@kernel.org> References: <20240904-work-kmem_cache_args-v3-0-05db2179a8c2@kernel.org> In-Reply-To: <20240904-work-kmem_cache_args-v3-0-05db2179a8c2@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=3911; i=brauner@kernel.org; h=from:subject:message-id; bh=8hORSdr9O/EqElT2ajhJQ8gEdIqY6bU8s9Nue9pitwk=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaTdMNk1Y2pnvP31qKVpOcJ5do+eigrNe3hBOWNSeo3uI obMvRKBHaUsDGJcDLJiiiwO7Sbhcst5KjYbZWrAzGFlAhnCwMUpABNRMWFkeMCSwrpzXcROLyPf JzwXoxOe3hR/ovf8iI7sgtCN83NWVjMytJ4XPf57hZ7huRP6s2Z5+YTtMrRZIt6vpv7B7bDZisW +fAA= X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 X-Rspam-User: X-Rspamd-Queue-Id: D957CA0015 X-Rspamd-Server: rspam01 X-Stat-Signature: j5pq713iait3b9cro1he3ushbme5i1zh X-HE-Tag: 1725445347-908484 X-HE-Meta: U2FsdGVkX1/2R1v9r5s5TdPsuubq3dCa0SWQQGDdcxLHyixccJOAy3Eln8v2U/2dSnrnGV3m5778uDVBO+H9W8Z12G9ErvccMk1QvtQDbDqBunOw+Qvn8/jtLHcQNFqHpoQqO4YmiUsyGVIFwPFFMILvYO9e0PBNx9c9Y64W82i2wP7e368GN9g9ilG6H7pSWkY+GoU8bsIfJkTrbyfJW5ChHRWeXJB8c5XsuNDzRh3cf7VTzICCaf+Fg8Nngqh12tJpHvywNd9lBM+fm+4kuwojyu/tA1+y05NsV0x98XFwAliKj05Hjxk7d3otwRVPfvro1turB3+ZUE7AspULweySOwS6N9rvoEY0K9SQAhyRtkVNy1roSJE7QbhZmxCedQ4qhswJYwbZUb0dBCfgRHfYh8UaXa+uC53c7C1KG2wFDDkXwgj9YLeIbGvf43XWBIpYjsJWXE0FPVIgKq4p1Lq+qukfGl8bBeaUCRX10oeEv8/CftwLMJxMsWNIE9YHc05nQSZUM+w1avGHmoXngo/qMGhOsNJA5qdJuMp8SE4hE6itb7E7Ncm96usuWUvxn7hnFMLeLXzHj5vYcg4TD7oygYEw6UWpHAuQ0v3cU4LYP8Z+l2bbZD9aRT2DMz9NQyiNknppdVociZjWejXBLWpZRY2bv6l5sjqHP9fsxNgeiSJ9Ir2Zxt2tYNfrZfLhDt7rQAfjhxI4JrmXWw7VuhGkxn5uo5FVLA6yXXyG4bZHmZD1ucMQRE0RSCZzKd47AdZZUE+Ji4XKeclTYTo2A9b+3wssmNo8mYerMzuTw+N54H2wKf/xeCsHE7uYDQNziqwAb+Lq4wXfyxYupE7ZFwrIgmDcwxOrq/huV5fzkEPXkn1D1jErAre+KWtyXU9kpwWNvZuLH8mg0mT+H2onvzA3dmz1PptQXAhJwFn8Tjq9bzjzjs1AE+54rHWYw5ebDSn8qxfQYLwkjtmkwQn UMx7x7Da fvvfSPqBfFPGq5ig/AuW+sHR9mpeK4tr2RAsHI9dDCr6mi51fFklcc/dVQkyjOaF/80+KHepo9GH3pg4P3FyRj5C+hSQ2KDUbk5yyGy3dJhSqbIumNfVCPVpDR6gExmq47QqEH3eHG6NJjmTVZNYS148ONXiGn7xuGeD4kN5eBavS9OwKex0u83E8NS5gmkS9jnppNd5E+UaCpl91IidEDY2aJ8e9zXQ/KK07mfTMv2sbsso9U1rhYyfgndI/omTz32zwExtWN/2DYXFSlzt61a7F8kKTu7GgTmdsoKWHcOEsgjvk4b8hWeQXFS3P2k9N2Vt1fzR902LnH7ZFGoqLiC7AXnD2KWJFsRpJM+zsBYn387fGPZVoB5DdGzwHB0URJpK6mgIyWyjikcseeLOMROVWWg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Pass down struct kmem_cache_args to calculate_sizes() so we can use args->{use}_freeptr_offset directly. This allows us to remove ->rcu_freeptr_offset from struct kmem_cache. Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Vlastimil Babka Signed-off-by: Christian Brauner --- mm/slab.h | 2 -- mm/slub.c | 25 +++++++------------------ 2 files changed, 7 insertions(+), 20 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index c7a4e0fc3cf1..36ac38e21fcb 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -261,8 +261,6 @@ struct kmem_cache { unsigned int object_size; /* Object size without metadata */ struct reciprocal_value reciprocal_size; unsigned int offset; /* Free pointer offset */ - /* Specific free pointer requested (if not UINT_MAX) */ - unsigned int rcu_freeptr_offset; #ifdef CONFIG_SLUB_CPU_PARTIAL /* Number of per cpu partial objects to keep around */ unsigned int cpu_partial; diff --git a/mm/slub.c b/mm/slub.c index 4719b60215b8..a23c7036cd61 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3916,8 +3916,7 @@ static void *__slab_alloc_node(struct kmem_cache *s, * If the object has been wiped upon free, make sure it's fully initialized by * zeroing out freelist pointer. * - * Note that we also wipe custom freelist pointers specified via - * s->rcu_freeptr_offset. + * Note that we also wipe custom freelist pointers. */ static __always_inline void maybe_wipe_obj_freeptr(struct kmem_cache *s, void *obj) @@ -5141,17 +5140,11 @@ static void set_cpu_partial(struct kmem_cache *s) #endif } -/* Was a valid freeptr offset requested? */ -static inline bool has_freeptr_offset(const struct kmem_cache *s) -{ - return s->rcu_freeptr_offset != UINT_MAX; -} - /* * calculate_sizes() determines the order and the distribution of data within * a slab object. */ -static int calculate_sizes(struct kmem_cache *s) +static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s) { slab_flags_t flags = s->flags; unsigned int size = s->object_size; @@ -5192,7 +5185,7 @@ static int calculate_sizes(struct kmem_cache *s) */ s->inuse = size; - if (((flags & SLAB_TYPESAFE_BY_RCU) && !has_freeptr_offset(s)) || + if (((flags & SLAB_TYPESAFE_BY_RCU) && !args->use_freeptr_offset) || (flags & SLAB_POISON) || s->ctor || ((flags & SLAB_RED_ZONE) && (s->object_size < sizeof(void *) || slub_debug_orig_size(s)))) { @@ -5214,8 +5207,8 @@ static int calculate_sizes(struct kmem_cache *s) */ s->offset = size; size += sizeof(void *); - } else if ((flags & SLAB_TYPESAFE_BY_RCU) && has_freeptr_offset(s)) { - s->offset = s->rcu_freeptr_offset; + } else if ((flags & SLAB_TYPESAFE_BY_RCU) && args->use_freeptr_offset) { + s->offset = args->freeptr_offset; } else { /* * Store freelist pointer near middle of object to keep @@ -5856,10 +5849,6 @@ int do_kmem_cache_create(struct kmem_cache *s, const char *name, #ifdef CONFIG_SLAB_FREELIST_HARDENED s->random = get_random_long(); #endif - if (args->use_freeptr_offset) - s->rcu_freeptr_offset = args->freeptr_offset; - else - s->rcu_freeptr_offset = UINT_MAX; s->align = args->align; s->ctor = args->ctor; #ifdef CONFIG_HARDENED_USERCOPY @@ -5867,7 +5856,7 @@ int do_kmem_cache_create(struct kmem_cache *s, const char *name, s->usersize = args->usersize; #endif - if (!calculate_sizes(s)) + if (!calculate_sizes(args, s)) goto out; if (disable_higher_order_debug) { /* @@ -5877,7 +5866,7 @@ int do_kmem_cache_create(struct kmem_cache *s, const char *name, if (get_order(s->size) > get_order(s->object_size)) { s->flags &= ~DEBUG_METADATA_FLAGS; s->offset = 0; - if (!calculate_sizes(s)) + if (!calculate_sizes(args, s)) goto out; } }