From patchwork Tue Nov 16 00:16:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12620731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E1E9C433FE for ; Tue, 16 Nov 2021 00:17:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F0D1E614C8 for ; Tue, 16 Nov 2021 00:17:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org F0D1E614C8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 605D46B00A2; Mon, 15 Nov 2021 19:16:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C27A36B00A3; Mon, 15 Nov 2021 19:16:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 740A26B00A1; Mon, 15 Nov 2021 19:16:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0119.hostedemail.com [216.40.44.119]) by kanga.kvack.org (Postfix) with ESMTP id D34F36B00A7 for ; Mon, 15 Nov 2021 19:16:43 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 94CAB1815BBFC for ; Tue, 16 Nov 2021 00:16:43 +0000 (UTC) X-FDA: 78812877486.24.CE8F30F Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf25.hostedemail.com (Postfix) with ESMTP id D7EEEB000193 for ; Tue, 16 Nov 2021 00:16:29 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 1CED61FD73; Tue, 16 Nov 2021 00:16:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1637021802; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4HEc0a+5kUJeYeUUKFV/Jr2mozOhZtMjtJch9JnNS5g=; b=RFFv6xjO4bZgywTvtmKvMDcoNMdUf32NsSKwkpGOyGTTT3Wy1lKfpLMWp5WGoF7/iOM7hy 3CE4C6s8vnKmlST8nZ1LyIjR4brvXigRHoaau4RuNorRtva4aJ8Knz72gEWHMGgD8MQrv6 kwJ82B9TkJLCh2jl6gDCmvQ4yQ4cemE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1637021802; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4HEc0a+5kUJeYeUUKFV/Jr2mozOhZtMjtJch9JnNS5g=; b=PnwVtA/S1JTi/D3xnxvp2NY92JvAMHA0uXxUOI0hCJol0508QkHg2x71i7xFx3peLtIHmP lHb8vv5pbc5hiFBg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id DDC0113F72; Tue, 16 Nov 2021 00:16:41 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 0AONNWn4kmFjXAAAMHmgww (envelope-from ); Tue, 16 Nov 2021 00:16:41 +0000 From: Vlastimil Babka To: Matthew Wilcox , linux-mm@kvack.org, Christoph Lameter , David Rientjes , Joonsoo Kim , Pekka Enberg Cc: Vlastimil Babka , Alexander Potapenko , Marco Elver , Dmitry Vyukov , kasan-dev@googlegroups.com Subject: [RFC PATCH 30/32] mm/sl*b: Differentiate struct slab fields by sl*b implementations Date: Tue, 16 Nov 2021 01:16:26 +0100 Message-Id: <20211116001628.24216-31-vbabka@suse.cz> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211116001628.24216-1-vbabka@suse.cz> References: <20211116001628.24216-1-vbabka@suse.cz> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4583; h=from:subject; bh=HpvxF6blYd9Gxl8D0B/0lpH3ep8bdHVJ8W1r0M/ydds=; b=owEBbQGS/pANAwAIAeAhynPxiakQAcsmYgBhkvhXodE76eu6sNsRldmo9f4ZC/NuS3j9IsK5x5/Q lJgFbd+JATMEAAEIAB0WIQSNS5MBqTXjGL5IXszgIcpz8YmpEAUCYZL4VwAKCRDgIcpz8YmpEMxSCA Ck/Bodg7xN+SrNWGfwBc0nMLXkVJZUg7QYlOMYrbg14Wb7jdiq4aWhvJ3MkwpN4FUj6qBfxkfNf30X f73MrCXMmKfS63JbXwd0aPcM9ANzTYT31Cys3tiZC8cdObSIq+5iJhxzxlwe24Vh7BW3QbX2+fiTF5 NKXYxdR21dc057mZ/MRTmxIalYYM+5rcFEmUss/Liq8uVAaMMQzuIlSAc4/4DAUl2/y5Nyzp0gIK6s tAwWcynBsk07XBA8OwaI530/l903irks6siXta4ruUOV8hdAq2CZPFAjzpKqchwsegJnJRAmvta9P2 GJQZpz1QcgMKz19hJf3O5SHJeS0zzx X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Stat-Signature: xsco67bchg1gtkb3jfo8ok9pprzztfmp Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=RFFv6xjO; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="PnwVtA/S"; spf=pass (imf25.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: D7EEEB000193 X-HE-Tag: 1637021789-223695 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With a struct slab definition separate from struct page, we can go further and define only fields that the chosen sl*b implementation uses. This means everything between __page_flags and __page_refcount placeholders now depends on the chosen CONFIG_SL*B. Some fields exist in all implementations (slab_list) but can be part of a union in some, so it's simpler to repeat them than complicate the definition with ifdefs even more. The patch doesn't change physical offsets of the fields, although it could be done later - for example it's now clear that tighter packing in SLOB could be possible. This should also prevent accidental use of fields that don't exist in given implementation. Before this patch virt_to_cache() and and cache_from_obj() was visible for SLOB (albeit not used), although it relies on the slab_cache field that isn't set by SLOB. With this patch it's now a compile error, so these functions are now hidden behind #ifndef CONFIG_SLOB. Signed-off-by: Vlastimil Babka Cc: Alexander Potapenko (maintainer:KFENCE) Cc: Marco Elver (maintainer:KFENCE) Cc: Dmitry Vyukov (reviewer:KFENCE) Cc: Tested-by: Marco Elver --- mm/kfence/core.c | 9 +++++---- mm/slab.h | 46 ++++++++++++++++++++++++++++++++++++---------- 2 files changed, 41 insertions(+), 14 deletions(-) diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 4eb60cf5ff8b..46103a7628a6 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -427,10 +427,11 @@ static void *kfence_guarded_alloc(struct kmem_cache *cache, size_t size, gfp_t g /* Set required slab fields. */ slab = virt_to_slab((void *)meta->addr); slab->slab_cache = cache; - if (IS_ENABLED(CONFIG_SLUB)) - slab->objects = 1; - if (IS_ENABLED(CONFIG_SLAB)) - slab->s_mem = addr; +#if defined(CONFIG_SLUB) + slab->objects = 1; +#elif defined (CONFIG_SLAB) + slab->s_mem = addr; +#endif /* Memory initialization. */ for_each_canary(meta, set_canary_byte); diff --git a/mm/slab.h b/mm/slab.h index 58b65e5e5d49..10a9ee195249 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -8,9 +8,24 @@ /* Reuses the bits in struct page */ struct slab { unsigned long __page_flags; + +#if defined(CONFIG_SLAB) + + union { + struct list_head slab_list; + struct rcu_head rcu_head; + }; + struct kmem_cache *slab_cache; + void *freelist; /* array of free object indexes */ + void * s_mem; /* first object */ + unsigned int active; + +#elif defined(CONFIG_SLUB) + union { struct list_head slab_list; - struct { /* Partial pages */ + struct rcu_head rcu_head; + struct { struct slab *next; #ifdef CONFIG_64BIT int slabs; /* Nr of slabs left */ @@ -18,25 +33,32 @@ struct slab { short int slabs; #endif }; - struct rcu_head rcu_head; }; - struct kmem_cache *slab_cache; /* not slob */ + struct kmem_cache *slab_cache; /* Double-word boundary */ void *freelist; /* first free object */ union { - void *s_mem; /* slab: first object */ - unsigned long counters; /* SLUB */ - struct { /* SLUB */ + unsigned long counters; + struct { unsigned inuse:16; unsigned objects:15; unsigned frozen:1; }; }; + unsigned int __unused; + +#elif defined(CONFIG_SLOB) + + struct list_head slab_list; + void * __unused_1; + void *freelist; /* first free block */ + void * __unused_2; + int units; + +#else +#error "Unexpected slab allocator configured" +#endif - union { - unsigned int active; /* SLAB */ - int units; /* SLOB */ - }; atomic_t __page_refcount; #ifdef CONFIG_MEMCG unsigned long memcg_data; @@ -47,7 +69,9 @@ struct slab { static_assert(offsetof(struct page, pg) == offsetof(struct slab, sl)) SLAB_MATCH(flags, __page_flags); SLAB_MATCH(compound_head, slab_list); /* Ensure bit 0 is clear */ +#ifndef CONFIG_SLOB SLAB_MATCH(rcu_head, rcu_head); +#endif SLAB_MATCH(_refcount, __page_refcount); #ifdef CONFIG_MEMCG SLAB_MATCH(memcg_data, memcg_data); @@ -623,6 +647,7 @@ static inline void memcg_slab_free_hook(struct kmem_cache *s, } #endif /* CONFIG_MEMCG_KMEM */ +#ifndef CONFIG_SLOB static inline struct kmem_cache *virt_to_cache(const void *obj) { struct slab *slab; @@ -669,6 +694,7 @@ static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) print_tracking(cachep, x); return cachep; } +#endif /* CONFIG_SLOB */ static inline size_t slab_ksize(const struct kmem_cache *s) {