From patchwork Fri Aug 9 07:33:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13758475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27979C52D73 for ; Fri, 9 Aug 2024 07:33:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA5E56B009A; Fri, 9 Aug 2024 03:33:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D0DCE6B00A3; Fri, 9 Aug 2024 03:33:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96AA86B00A0; Fri, 9 Aug 2024 03:33:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4921C6B009E for ; Fri, 9 Aug 2024 03:33:13 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D7EFB140A32 for ; Fri, 9 Aug 2024 07:33:12 +0000 (UTC) X-FDA: 82431891024.09.8A07AAF Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf09.hostedemail.com (Postfix) with ESMTP id 1692614000A for ; Fri, 9 Aug 2024 07:33:10 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="o4q8IAk/"; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf09.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723188737; a=rsa-sha256; cv=none; b=ydKh7AaPO9Mos5paB+KUmrm7cYnYgZFDPi/1I0X7LBUVDQGXZ/XR5Of9c6dSYVlalm8v+5 FQ/YnMDhDFmZIk8Dy5zeJ+GT4/58OY/ig78sVILj7cN1lEQ2VPF8hpcKip4tIh8O7tqd+c bv1QJkEzqvGYb25fXE7sMSpUODOzdtg= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="o4q8IAk/"; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf09.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723188737; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Nh7dt7bQuU6LNtnXjDqc3kjt5MiWMUuLcZZcyTOBVhU=; b=onUp/XOpByfSkX3hityv23SLfikJ21FqLUWaV8MUD5QjbPZnXGpJHtI3BM7UKdqGLwgUgr aCAa19eK78ZGRVM0g0HE+9+0DacvelGDn6KlhSSExlfbym1kFtf3ZnPmu27/Cr1oKSYNZs oemA67AMc1Iqx2DMCtiygRj2UpYcMqE= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 062536165F; Fri, 9 Aug 2024 07:33:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A8720C4AF0F; Fri, 9 Aug 2024 07:33:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723188789; bh=Iabo073v2g0BX5idKAlpNK0sS3Na+mbsF9YuoIGYIVE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=o4q8IAk/qlgqcTCea3BBJdIf8hDOC1XjjLxhcI8dkm1ljzbm+iszrHUacqp7lGQQq jGNBVMkdhPRB0YBHgDKEbwNi4qZIaA7IOQMRouJlraAAcY8ZANl/rh82rIf2q7a7Sk RGDqTlGc0UkZl79HBkzQlQPhNfE3UMMSY31PiSUNZY49KE4SM+jd9QH+P+ipVYPvlr PHnohauWkmIwt2xg5wEQ8yc8x/rD4IETjsdn1fVLzbTajnMZKrEL/FtYFbIOrNxBUu jCkle8QjxtskMzf4qXmks1dSVdfJo3x06Hf3Uia/frmvQwd2b0c1VatOnwxcL02pJN EqUeRe1J4h/QQ== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Suren Baghdasaryan , Kent Overstreet , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Jann Horn , Matteo Rizzo , jvoisin , Xiu Jianfeng , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH 4/5] alloc_tag: Track fixed vs dynamic sized kmalloc calls Date: Fri, 9 Aug 2024 00:33:05 -0700 Message-Id: <20240809073309.2134488-4-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240809072532.work.266-kees@kernel.org> References: <20240809072532.work.266-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8513; i=kees@kernel.org; h=from:subject; bh=Iabo073v2g0BX5idKAlpNK0sS3Na+mbsF9YuoIGYIVE=; b=owGbwMvMwCVmps19z/KJym7G02pJDGlbjxla/OkOZpqjmTXXV1pxl+T8wjPTGhjbs13unfB2Y DSfGRnZUcrCIMbFICumyBJk5x7n4vG2Pdx9riLMHFYmkCEMXJwCMBFZZ0aG2zfLDrTphUWynG79 HvZ6UuCs3+Fnq13vX2MPMGJ4Pkd5ESPDrh3z73lW79405Y3IHS2OfPYiTYNZKYUfe4ubfrw2nb2 WGQA= X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 1692614000A X-Stat-Signature: bwjd7jjegkuof7gmnqx1j7cj3d46t8iw X-Rspam-User: X-HE-Tag: 1723188790-242937 X-HE-Meta: U2FsdGVkX18I3EmSkRSQbLngBk13iH1nJA3RF3vSGPTPJcIwWnvGo1fg1ENSqG29bC2mYQMmuLmrnYxhR41WhTBisz+vCN77pq5fCAivWsjutPFryXeAe/eX0r/C/UJBVnLseBxVywLm376ixEmjWJC/pWA3sJ1rwwVV3h087lVXHOgR+gCVV2fAg/LC1EBQdCOchSuOqeyAbDh4bGHyZrsziTYlxXTw+uZe5tq89pD98928SdybyVTMguIcnv9W9gMgvYVR3f0sTYJJ9o8FW96HSm09Bntyly1D1foBrhBPiKYZ7oWGTFjYxDixL4w7dkeZfDlkzMeHprK/IkZcP5pqet2vsZdJF03HS/KjvrHclzyN3KIEeOaB87YCUSyD8nUvxK1EDlADV2v5XD1wAvLh29zotruVvBgomSCz7Qa2oMKYznRk6HdsmORB2s2agJEEQdA7LjyJDrTaGaU7kID0H/jrBSd2fV+YXFUAw9nus4Mxq33w3XQntCOfy0zXjf4S9vYFs9UYrUT5+EUIYHGzwFLOyrY+QoTQBUZhh0opCp9hLUhmKRq9bfo7vdz3Acml6hVXnEwIipPkP/YYndNljm0+xNKKDHEoC+s3HcXc3fDa1K3WeAdujMQUUYM8U+U2Zd4+PNtw1lvccJaEkGZxAV/JqsbrVicETcoyayNL5qVlzqOMUNR2rsoUX4AVDK0vAP7NLmf3SNlcdP36ymH3v7Dw+hoWcqmm7LiDCtZwj0oibZThqaE3v+WdRl/Zl6twVbdjI0hXLBumE5/XY/xXXu2ZjOdpCkXlmKc1K+7epYPlg1omz72lEQDOeRaU7SLQHjh7wps+x+F5bDrlnv9w5UB5VODKdudXX6WR7T6eghCDDDvXOfSWN+r+hTdiwMPaSmt595RlDXnGQuDJFzflnl9Hq3amw6xwLz1J7YCWaZ0ljBFVAKSreUSLRgNf0Kkn5qFc/ieYu5uxBEN T2vcjOqO mVAnS+WqBIt0JsvoyYL2Zna9bXqjr/+6q17JErXNwYJko1cJ4/j7h4++Aqi3uW6E8WerPClXCkyQzQWWXIwhTlu2+G66XoGSfO2Dk92d3gOeVsofOwFY2PdB8cSftjXt0KIuW90feQiO0u9JhxplvjHKvFwDPDysN7HCph4BUBK4zyqxOEJr41voDb5hoZOQphggVBrP6+XqKa/dXKqyCXZJPfHcROoGuBQnkkjZUm1/gZSyrIKjDHvRg3jKj6qJoZwpwCj+Jh32eBCdelEnBtwFFDo+4cDbe/ZDEqtG+yTkYq1/smJ5f9Dm6YwFAgQz/QGYHD1R0ldEWiTfX/7wfDkKnFaS7SiClvOTMqnp2w4S1QXxD3fR11SQ6X5cVstGeP2EEtFfkq+DWus5hNFXx4NoYKdRnTBqbcHwCjGLZe9ODfBsbVw17ZKObJyvILIKM/sRf+yoh0RsWslH/PF5BKw5GjkbKw+91MfJKJ9ZDIl+vsqLt/e/NLue4h6PaI4Iv15RxRVXCrDZcjsfEBhIoOZ7AZ9dClUUVU5h8Y221n8axWTGzVbF8/l9Q45WtTAJrEzRxX3i/8Z0txXRO5h+QYL4+94KuOw3+/DXXQ9Hh0nShsPe1wyjWuVM1Ola27BvPU5MGv7aHez+bU9I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For slab allocations, record whether the call site is using a fixed size (i.e. compile time constant) or a dynamic size. Report the results in /proc/allocinfo. Improvements needed: - examine realloc routines for needed coverage Signed-off-by: Kees Cook --- Cc: Suren Baghdasaryan Cc: Kent Overstreet Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/alloc_tag.h | 30 ++++++++++++++++++++++++++---- include/linux/slab.h | 16 ++++++++-------- lib/alloc_tag.c | 8 ++++++++ mm/Kconfig | 8 ++++++++ 4 files changed, 50 insertions(+), 12 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 8c61ccd161ba..f5d8c5849b82 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -20,6 +20,19 @@ struct alloc_tag_counters { u64 calls; }; +#ifdef CONFIG_SLAB_PER_SITE +struct alloc_meta { + /* 0 means non-slab, SIZE_MAX means dynamic, and everything else is fixed-size. */ + size_t sized; +}; +#define ALLOC_META_INIT(_size) { \ + .sized = (__builtin_constant_p(_size) ? (_size) : SIZE_MAX), \ + } +#else +struct alloc_meta { }; +#define ALLOC_META_INIT(_size) { } +#endif + /* * An instance of this structure is created in a special ELF section at every * allocation callsite. At runtime, the special section is treated as @@ -27,6 +40,7 @@ struct alloc_tag_counters { */ struct alloc_tag { struct codetag ct; + struct alloc_meta meta; struct alloc_tag_counters __percpu *counters; } __aligned(8); @@ -74,19 +88,21 @@ static inline struct alloc_tag *ct_to_alloc_tag(struct codetag *ct) */ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag); -#define DEFINE_ALLOC_TAG(_alloc_tag) \ +#define DEFINE_ALLOC_TAG(_alloc_tag, _meta_init) \ static struct alloc_tag _alloc_tag __used __aligned(8) \ __section("alloc_tags") = { \ .ct = CODE_TAG_INIT, \ + .meta = _meta_init, \ .counters = &_shared_alloc_tag }; #else /* ARCH_NEEDS_WEAK_PER_CPU */ -#define DEFINE_ALLOC_TAG(_alloc_tag) \ +#define DEFINE_ALLOC_TAG(_alloc_tag, _meta_init) \ static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \ static struct alloc_tag _alloc_tag __used __aligned(8) \ __section("alloc_tags") = { \ .ct = CODE_TAG_INIT, \ + .meta = _meta_init, \ .counters = &_alloc_tag_cntr }; #endif /* ARCH_NEEDS_WEAK_PER_CPU */ @@ -191,7 +207,7 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) #else /* CONFIG_MEM_ALLOC_PROFILING */ -#define DEFINE_ALLOC_TAG(_alloc_tag) +#define DEFINE_ALLOC_TAG(_alloc_tag, _meta_init) static inline bool mem_alloc_profiling_enabled(void) { return false; } static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) {} @@ -210,8 +226,14 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} #define alloc_hooks(_do_alloc) \ ({ \ - DEFINE_ALLOC_TAG(_alloc_tag); \ + DEFINE_ALLOC_TAG(_alloc_tag, { }); \ alloc_hooks_tag(&_alloc_tag, _do_alloc); \ }) +#define alloc_sized_hooks(_do_alloc, _size, ...) \ +({ \ + DEFINE_ALLOC_TAG(_alloc_tag, ALLOC_META_INIT(_size)); \ + alloc_hooks_tag(&_alloc_tag, _do_alloc(_size, __VA_ARGS__)); \ +}) + #endif /* _LINUX_ALLOC_TAG_H */ diff --git a/include/linux/slab.h b/include/linux/slab.h index 86cb61a0102c..314d24c79e05 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -541,7 +541,7 @@ static_assert(PAGE_SHIFT <= 20); */ void *kmem_cache_alloc_noprof(struct kmem_cache *cachep, gfp_t flags) __assume_slab_alignment __malloc; -#define kmem_cache_alloc(...) alloc_hooks(kmem_cache_alloc_noprof(__VA_ARGS__)) +#define kmem_cache_alloc(...) alloc_hooks(kmem_cache_alloc_noprof(__VA_ARGS__)) void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags) __assume_slab_alignment __malloc; @@ -685,7 +685,7 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f } return __kmalloc_noprof(size, flags); } -#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__)) +#define kmalloc(size, ...) alloc_sized_hooks(kmalloc_noprof, size, __VA_ARGS__) #define kmem_buckets_alloc(_b, _size, _flags) \ alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE)) @@ -708,7 +708,7 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf } return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node); } -#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__)) +#define kmalloc_node(size, ...) alloc_sized_hooks(kmalloc_node_noprof, size, __VA_ARGS__) /** * kmalloc_array - allocate memory for an array. @@ -726,7 +726,7 @@ static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t siz return kmalloc_noprof(bytes, flags); return kmalloc_noprof(bytes, flags); } -#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__)) +#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__)) /** * krealloc_array - reallocate memory for an array. @@ -761,8 +761,8 @@ void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flag unsigned long caller) __alloc_size(1); #define kmalloc_node_track_caller_noprof(size, flags, node, caller) \ __kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, caller) -#define kmalloc_node_track_caller(...) \ - alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_)) +#define kmalloc_node_track_caller(size, ...) \ + alloc_sized_hooks(kmalloc_node_track_caller_noprof, size, __VA_ARGS__, _RET_IP_) /* * kmalloc_track_caller is a special version of kmalloc that records the @@ -807,13 +807,13 @@ static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags) { return kmalloc_noprof(size, flags | __GFP_ZERO); } -#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__)) +#define kzalloc(size, ...) alloc_sized_hooks(kzalloc_noprof, size, __VA_ARGS__) #define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node) void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) __alloc_size(1); #define kvmalloc_node_noprof(size, flags, node) \ __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node) -#define kvmalloc_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) +#define kvmalloc_node(size, ...) alloc_sized_hooks(kvmalloc_node_noprof, size, __VA_ARGS__) #define kvmalloc(_size, _flags) kvmalloc_node(_size, _flags, NUMA_NO_NODE) #define kvmalloc_noprof(_size, _flags) kvmalloc_node_noprof(_size, _flags, NUMA_NO_NODE) diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 81e5f9a70f22..6d2cb72bf269 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -78,6 +78,14 @@ static void alloc_tag_to_text(struct seq_buf *out, struct codetag *ct) seq_buf_printf(out, "%12lli %8llu ", bytes, counter.calls); codetag_to_text(out, ct); +#ifdef CONFIG_SLAB_PER_SITE + seq_buf_putc(out, ' '); + seq_buf_printf(out, "size:%s(%zu) slab:%s", + tag->meta.sized == 0 ? "non-slab" : + tag->meta.sized == SIZE_MAX ? "dynamic" : "fixed", + tag->meta.sized == SIZE_MAX ? 0 : tag->meta.sized, + tag->meta.cache ? "ready" : "unused"); +#endif seq_buf_putc(out, ' '); seq_buf_putc(out, '\n'); } diff --git a/mm/Kconfig b/mm/Kconfig index b72e7d040f78..855c63c3270d 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -296,6 +296,14 @@ config SLAB_BUCKETS If unsure, say Y. +config SLAB_PER_SITE + bool "Separate slab allocations by call size" + depends on !SLUB_TINY + default SLAB_FREELIST_HARDENED + select SLAB_BUCKETS + help + Track sizes of kmalloc() call sites. + config SLUB_STATS default n bool "Enable performance statistics"