From patchwork Mon Apr 4 16:41:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12800565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DE91C433EF for ; Mon, 4 Apr 2022 16:41:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 770F16B0074; Mon, 4 Apr 2022 12:41:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FA676B0075; Mon, 4 Apr 2022 12:41:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 573456B0078; Mon, 4 Apr 2022 12:41:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0245.hostedemail.com [216.40.44.245]) by kanga.kvack.org (Postfix) with ESMTP id 4959F6B0074 for ; Mon, 4 Apr 2022 12:41:32 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 12A43ACF7F for ; Mon, 4 Apr 2022 16:41:22 +0000 (UTC) X-FDA: 79319762004.30.B92A6FA Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf29.hostedemail.com (Postfix) with ESMTP id 5B0A6120029 for ; Mon, 4 Apr 2022 16:41:21 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 1FA1D210F4; Mon, 4 Apr 2022 16:41:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1649090480; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Msvh6m+t5rYoB11gb9lJtAm2YmxtlamKtBfXfIOctHA=; b=tlOMGI7kNSNyy2XuejrBGFf836Qy3z44eiz/mZrTq/6EQLL8eEbwOBjEmkdymnHKzzl7I+ xg/MpwhH6iO+cOh6iLRJzp4+I9UgBPv6jG1Fuyof3z/QdP2HRknjDjBeYO+NOfHTPgIplf BZaZyWwz+Fzfk6IjO7o2gNtz4TwDExw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1649090480; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Msvh6m+t5rYoB11gb9lJtAm2YmxtlamKtBfXfIOctHA=; b=sMZ/KEKgA+a21s2WOtHvaiFSM98e28VjkRdAIhCVcH9KoppIuFRW+5wPIwkKTxNicqWQMO fo1IelcSJgf646Dw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id E0FBC13A89; Mon, 4 Apr 2022 16:41:19 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id GLNTNq8fS2LjfAAAMHmgww (envelope-from ); Mon, 04 Apr 2022 16:41:19 +0000 From: Vlastimil Babka To: David Rientjes , Christoph Lameter , Joonsoo Kim , Pekka Enberg , Roman Gushchin Cc: Andrew Morton , linux-mm@kvack.org, patches@lists.linux.dev, linux-kernel@vger.kernel.org, Oliver Glitta , Marco Elver , Mike Rapoport , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Imran Khan , Vlastimil Babka Subject: [PATCH v3 1/6] lib/stackdepot: allow requesting early initialization dynamically Date: Mon, 4 Apr 2022 18:41:07 +0200 Message-Id: <20220404164112.18372-2-vbabka@suse.cz> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220404164112.18372-1-vbabka@suse.cz> References: <20220404164112.18372-1-vbabka@suse.cz> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7886; h=from:subject; bh=g13raqQ+rnG6rknZ2M+4fuKo89TMDIRw3HFuYCzSfK0=; b=owEBbQGS/pANAwAIAeAhynPxiakQAcsmYgBiSx+jaLlmGkFJ39Gr8G3pJl5+AE10txtNwpy9PXCs wtDx4X6JATMEAAEIAB0WIQSNS5MBqTXjGL5IXszgIcpz8YmpEAUCYksfowAKCRDgIcpz8YmpEGy9B/ 4ye/zmHC7nUHoPB5v5Z7YbJbuhVSD6qAUUS3grCPUqL+fy7gCmlAmYGuHShuU9+cZYEdJi54QV5bvs j0UsAgImm33hsWBUSlBtveKek6TtHefdzgefKxlDo8EGtA7CaHfffAFGHhEWJ/7OKv1wPlo3nwU9S1 FtULwgfLmU0GD/me12NCb22pFYJfr4/5QD34m9HBu/vxo+YHMNd6HbNHlWB1I1HFrzml7s8cOY5JkZ QpopD9eqRDO8Xhh9Bcmo0Wopbrd3gHQbQSMl8VbaG4C7yWz5cHcxFRqMnHZldbh1i2ouG3sGL2+P9k DkhMxTEEo9dBtz12zcBamASZce5ml9 X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=tlOMGI7k; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="sMZ/KEKg"; spf=pass (imf29.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 5B0A6120029 X-Stat-Signature: n5jkajxfcsaomkrdk8qh3g9trex53m6j X-HE-Tag: 1649090481-11851 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In a later patch we want to add stackdepot support for object owner tracking in slub caches, which is enabled by slub_debug boot parameter. This creates a bootstrap problem as some caches are created early in boot when slab_is_available() is false and thus stack_depot_init() tries to use memblock. But, as reported by Hyeonggon Yoo [1] we are already beyond memblock_free_all(). Ideally memblock allocation should fail, yet it succeeds, but later the system crashes, which is a separately handled issue. To resolve this boostrap issue in a robust way, this patch adds another way to request stack_depot_early_init(), which happens at a well-defined point of time. In addition to build-time CONFIG_STACKDEPOT_ALWAYS_INIT, code that's e.g. processing boot parameters (which happens early enough) can call a new function stack_depot_want_early_init(), which sets a flag that stack_depot_early_init() will check. In this patch we also convert page_owner to this approach. While it doesn't have the bootstrap issue as slub, it's also a functionality enabled by a boot param and can thus request stack_depot_early_init() with memblock allocation instead of later initialization with kvmalloc(). As suggested by Mike, make stack_depot_early_init() only attempt memblock allocation and stack_depot_init() only attempt kvmalloc(). Also change the latter to kvcalloc(). In both cases we can lose the explicit array zeroing, which the allocations do already. As suggested by Marco, provide empty implementations of the init functions for !CONFIG_STACKDEPOT builds to simplify the callers. [1] https://lore.kernel.org/all/YhnUcqyeMgCrWZbd@ip-172-31-19-208.ap-northeast-1.compute.internal/ Reported-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Suggested-by: Mike Rapoport Suggested-by: Marco Elver Signed-off-by: Vlastimil Babka Reviewed-by: Marco Elver Reviewed-and-tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Mike Rapoport Acked-by: David Rientjes --- include/linux/stackdepot.h | 26 ++++++++++++--- lib/stackdepot.c | 66 +++++++++++++++++++++++++------------- mm/page_owner.c | 9 ++++-- 3 files changed, 72 insertions(+), 29 deletions(-) diff --git a/include/linux/stackdepot.h b/include/linux/stackdepot.h index 17f992fe6355..bc2797955de9 100644 --- a/include/linux/stackdepot.h +++ b/include/linux/stackdepot.h @@ -20,18 +20,36 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, gfp_t gfp_flags, bool can_alloc); /* - * Every user of stack depot has to call this during its own init when it's - * decided that it will be calling stack_depot_save() later. + * Every user of stack depot has to call stack_depot_init() during its own init + * when it's decided that it will be calling stack_depot_save() later. This is + * recommended for e.g. modules initialized later in the boot process, when + * slab_is_available() is true. * * The alternative is to select STACKDEPOT_ALWAYS_INIT to have stack depot * enabled as part of mm_init(), for subsystems where it's known at compile time * that stack depot will be used. + * + * Another alternative is to call stack_depot_want_early_init(), when the + * decision to use stack depot is taken e.g. when evaluating kernel boot + * parameters, which precedes the enablement point in mm_init(). + * + * stack_depot_init() and stack_depot_want_early_init() can be called regardless + * of CONFIG_STACKDEPOT and are no-op when disabled. The actual save/fetch/print + * functions should only be called from code that makes sure CONFIG_STACKDEPOT + * is enabled. */ +#ifdef CONFIG_STACKDEPOT int stack_depot_init(void); -#ifdef CONFIG_STACKDEPOT_ALWAYS_INIT -static inline int stack_depot_early_init(void) { return stack_depot_init(); } +void __init stack_depot_want_early_init(void); + +/* This is supposed to be called only from mm_init() */ +int __init stack_depot_early_init(void); #else +static inline int stack_depot_init(void) { return 0; } + +static inline void stack_depot_want_early_init(void) { } + static inline int stack_depot_early_init(void) { return 0; } #endif diff --git a/lib/stackdepot.c b/lib/stackdepot.c index bf5ba9af0500..6c4644c9ed44 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -66,6 +66,9 @@ struct stack_record { unsigned long entries[]; /* Variable-sized array of entries. */ }; +static bool __stack_depot_want_early_init __initdata = IS_ENABLED(CONFIG_STACKDEPOT_ALWAYS_INIT); +static bool __stack_depot_early_init_passed __initdata; + static void *stack_slabs[STACK_ALLOC_MAX_SLABS]; static int depot_index; @@ -162,38 +165,57 @@ static int __init is_stack_depot_disabled(char *str) } early_param("stack_depot_disable", is_stack_depot_disabled); -/* - * __ref because of memblock_alloc(), which will not be actually called after - * the __init code is gone, because at that point slab_is_available() is true - */ -__ref int stack_depot_init(void) +void __init stack_depot_want_early_init(void) +{ + /* Too late to request early init now */ + WARN_ON(__stack_depot_early_init_passed); + + __stack_depot_want_early_init = true; +} + +int __init stack_depot_early_init(void) +{ + size_t size; + + /* This is supposed to be called only once, from mm_init() */ + if (WARN_ON(__stack_depot_early_init_passed)) + return 0; + + __stack_depot_early_init_passed = true; + + if (!__stack_depot_want_early_init || stack_depot_disable) + return 0; + + pr_info("Stack Depot early init allocating hash table with memblock_alloc\n"); + size = (STACK_HASH_SIZE * sizeof(struct stack_record *)); + stack_table = memblock_alloc(size, SMP_CACHE_BYTES); + + if (!stack_table) { + pr_err("Stack Depot hash table allocation failed, disabling\n"); + stack_depot_disable = true; + return -ENOMEM; + } + + return 0; +} + +int stack_depot_init(void) { static DEFINE_MUTEX(stack_depot_init_mutex); + int ret = 0; mutex_lock(&stack_depot_init_mutex); if (!stack_depot_disable && !stack_table) { - size_t size = (STACK_HASH_SIZE * sizeof(struct stack_record *)); - int i; - - if (slab_is_available()) { - pr_info("Stack Depot allocating hash table with kvmalloc\n"); - stack_table = kvmalloc(size, GFP_KERNEL); - } else { - pr_info("Stack Depot allocating hash table with memblock_alloc\n"); - stack_table = memblock_alloc(size, SMP_CACHE_BYTES); - } - if (stack_table) { - for (i = 0; i < STACK_HASH_SIZE; i++) - stack_table[i] = NULL; - } else { + pr_info("Stack Depot allocating hash table with kvcalloc\n"); + stack_table = kvcalloc(STACK_HASH_SIZE, sizeof(struct stack_record *), GFP_KERNEL); + if (!stack_table) { pr_err("Stack Depot hash table allocation failed, disabling\n"); stack_depot_disable = true; - mutex_unlock(&stack_depot_init_mutex); - return -ENOMEM; + ret = -ENOMEM; } } mutex_unlock(&stack_depot_init_mutex); - return 0; + return ret; } EXPORT_SYMBOL_GPL(stack_depot_init); diff --git a/mm/page_owner.c b/mm/page_owner.c index fb3a05fdebdb..2743062e92c2 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -45,7 +45,12 @@ static void init_early_allocated_pages(void); static int __init early_page_owner_param(char *buf) { - return kstrtobool(buf, &page_owner_enabled); + int ret = kstrtobool(buf, &page_owner_enabled); + + if (page_owner_enabled) + stack_depot_want_early_init(); + + return ret; } early_param("page_owner", early_page_owner_param); @@ -83,8 +88,6 @@ static __init void init_page_owner(void) if (!page_owner_enabled) return; - stack_depot_init(); - register_dummy_stack(); register_failure_stack(); register_early_stack(); From patchwork Mon Apr 4 16:41:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12800566 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 737B5C433F5 for ; Mon, 4 Apr 2022 16:42:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C52E26B0075; Mon, 4 Apr 2022 12:41:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C01E46B0078; Mon, 4 Apr 2022 12:41:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA2606B007B; Mon, 4 Apr 2022 12:41:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.a.hostedemail.com [64.99.140.24]) by kanga.kvack.org (Postfix) with ESMTP id 95EEA6B0078 for ; Mon, 4 Apr 2022 12:41:32 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 60E0560ADB for ; Mon, 4 Apr 2022 16:41:22 +0000 (UTC) X-FDA: 79319762004.09.52B35C4 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf25.hostedemail.com (Postfix) with ESMTP id B7217A001B for ; Mon, 4 Apr 2022 16:41:21 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 52825210FC; Mon, 4 Apr 2022 16:41:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1649090480; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iaE2YvQWImFfSU0vpFQAVEvFm93nlCxWB9zb2gu0sPw=; b=Aq4DwyFirzDpf643d6LaNTryRKgg7O+NS/BhX6awzui5v67ahbF167x2uJ4IVhlqqB1Qhi I546Ma+WaIhbgy0P6wzM9k3qdJHuuc4qMYKDuWzkGSApqOUr5eG9GjQnQnEfJqldefGCdX 0n/qQKfOFnFxjqL/wSYI4V4vxsTBTy0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1649090480; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iaE2YvQWImFfSU0vpFQAVEvFm93nlCxWB9zb2gu0sPw=; b=eK5U25dTPANluiZzRfIbMte9s4p29IJQoGQ84fGRIy3pMoj0hICHeDPHIVww0a44zz0/Yp aIy7jm9OOTEOSADg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 228CB12FC5; Mon, 4 Apr 2022 16:41:20 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id wOviB7AfS2LjfAAAMHmgww (envelope-from ); Mon, 04 Apr 2022 16:41:20 +0000 From: Vlastimil Babka To: David Rientjes , Christoph Lameter , Joonsoo Kim , Pekka Enberg , Roman Gushchin Cc: Andrew Morton , linux-mm@kvack.org, patches@lists.linux.dev, linux-kernel@vger.kernel.org, Oliver Glitta , Marco Elver , Mike Rapoport , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Imran Khan , Vlastimil Babka Subject: [PATCH v3 2/6] mm/slub: move struct track init out of set_track() Date: Mon, 4 Apr 2022 18:41:08 +0200 Message-Id: <20220404164112.18372-3-vbabka@suse.cz> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220404164112.18372-1-vbabka@suse.cz> References: <20220404164112.18372-1-vbabka@suse.cz> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2040; h=from:subject; bh=zs3eo+rtqVSox6il95y1GSalLVpmki26s9979QfrOMU=; b=owEBbQGS/pANAwAIAeAhynPxiakQAcsmYgBiSx+kjPN+TrSQnc6TZ653XW3JCx/kwZAczK0EOz/c Ron6iIiJATMEAAEIAB0WIQSNS5MBqTXjGL5IXszgIcpz8YmpEAUCYksfpAAKCRDgIcpz8YmpEG23B/ 9nhTJhIIp33l1sFVO1vFJMQhAgm86MyxIu8FsrjB2606GdpWRVtTw2z5YRfxr9SPAT+ReeGrYbV2jH nRw1BKaEn2uaI7dmtMJbS8A02vJdQ81IYbuog734gHEBPcnwHGknA3EWHFvfoB+FuEWvVj8eYVTePg koaBMTKgjQf6C0oUAob9QxCeZNMZxFRYMyPk2BYwHsYT9cCKcq6URNkfdXkWZQn9BQaSvQDkwS90Vm 2BQDZOmCZkevFucPzODgaZ/2gBDNw6L0TUnwn3B8Fx3rrfyVFyDUB4K04DudJn1hy2SdwNvCOay3LY ejOVai4fUxxfY4gg4sqxTUB8XJXgxh X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Aq4DwyFi; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=eK5U25dT; spf=pass (imf25.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Stat-Signature: tpagf6b5i3c4wqg7hre1wiypn7c1gs66 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: B7217A001B X-HE-Tag: 1649090481-948490 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: set_track() either zeroes out the struct track or fills it, depending on the addr parameter. This is unnecessary as there's only one place that calls it for the initialization - init_tracking(). We can simply do the zeroing there, with a single memset() that covers both TRACK_ALLOC and TRACK_FREE as they are adjacent. Signed-off-by: Vlastimil Babka Reviewed-and-tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Acked-by: David Rientjes --- mm/slub.c | 32 +++++++++++++++----------------- 1 file changed, 15 insertions(+), 17 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 74d92aa4a3a2..cd4fd0159911 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -729,34 +729,32 @@ static void set_track(struct kmem_cache *s, void *object, { struct track *p = get_track(s, object, alloc); - if (addr) { #ifdef CONFIG_STACKTRACE - unsigned int nr_entries; + unsigned int nr_entries; - metadata_access_enable(); - nr_entries = stack_trace_save(kasan_reset_tag(p->addrs), - TRACK_ADDRS_COUNT, 3); - metadata_access_disable(); + metadata_access_enable(); + nr_entries = stack_trace_save(kasan_reset_tag(p->addrs), + TRACK_ADDRS_COUNT, 3); + metadata_access_disable(); - if (nr_entries < TRACK_ADDRS_COUNT) - p->addrs[nr_entries] = 0; + if (nr_entries < TRACK_ADDRS_COUNT) + p->addrs[nr_entries] = 0; #endif - p->addr = addr; - p->cpu = smp_processor_id(); - p->pid = current->pid; - p->when = jiffies; - } else { - memset(p, 0, sizeof(struct track)); - } + p->addr = addr; + p->cpu = smp_processor_id(); + p->pid = current->pid; + p->when = jiffies; } static void init_tracking(struct kmem_cache *s, void *object) { + struct track *p; + if (!(s->flags & SLAB_STORE_USER)) return; - set_track(s, object, TRACK_FREE, 0UL); - set_track(s, object, TRACK_ALLOC, 0UL); + p = get_track(s, object, TRACK_ALLOC); + memset(p, 0, 2*sizeof(struct track)); } static void print_track(const char *s, struct track *t, unsigned long pr_time) From patchwork Mon Apr 4 16:41:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12800568 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D68CCC433F5 for ; Mon, 4 Apr 2022 16:42:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 268E26B0081; Mon, 4 Apr 2022 12:41:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A3B68D0002; Mon, 4 Apr 2022 12:41:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E9C3B6B0080; Mon, 4 Apr 2022 12:41:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0176.hostedemail.com [216.40.44.176]) by kanga.kvack.org (Postfix) with ESMTP id C182D6B007B for ; Mon, 4 Apr 2022 12:41:32 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 82B95ACF43 for ; Mon, 4 Apr 2022 16:41:22 +0000 (UTC) X-FDA: 79319762004.24.8A2736C Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf23.hostedemail.com (Postfix) with ESMTP id BDCF7140040 for ; Mon, 4 Apr 2022 16:41:21 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 86147210FD; Mon, 4 Apr 2022 16:41:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1649090480; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jpdRyaAI44RLfX/6gtHESfD63hZ9rkC3wwwCds7xcYg=; b=WUaMCf4+SIZAa7qgDV19Emis9/gV04ss1pFTH2GTghWsDT2TL3ZLsmxl9YS+iGRBXB99kh hCNxKYGBj0ZRdriGiC6ouHD2ycYGJrNFpOMnjbEKrhHOe/KVib0wfLZ/8SI8fNdPj2al7c c+myLykvduVuKQnL6zFseQc5zdx5m68= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1649090480; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jpdRyaAI44RLfX/6gtHESfD63hZ9rkC3wwwCds7xcYg=; b=OJvO09OaFxMg/K6X3fl4crIlKTfYnDkiliURiU/JbuH/GgsVKOfzdq5zIjk5VLKArQTDq7 nuivuyGF2v9UOUCQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 55FFD13A89; Mon, 4 Apr 2022 16:41:20 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id sDdtFLAfS2LjfAAAMHmgww (envelope-from ); Mon, 04 Apr 2022 16:41:20 +0000 From: Vlastimil Babka To: David Rientjes , Christoph Lameter , Joonsoo Kim , Pekka Enberg , Roman Gushchin Cc: Andrew Morton , linux-mm@kvack.org, patches@lists.linux.dev, linux-kernel@vger.kernel.org, Oliver Glitta , Marco Elver , Mike Rapoport , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Imran Khan , Vlastimil Babka Subject: [PATCH v3 3/6] mm/slub: use stackdepot to save stack trace in objects Date: Mon, 4 Apr 2022 18:41:09 +0200 Message-Id: <20220404164112.18372-4-vbabka@suse.cz> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220404164112.18372-1-vbabka@suse.cz> References: <20220404164112.18372-1-vbabka@suse.cz> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7688; i=vbabka@suse.cz; h=from:subject; bh=U1uM2D7lNWodW4RBkGX+ViWCyz7KIYcT6hc4NWYCO/k=; b=owEBbQGS/pANAwAIAeAhynPxiakQAcsmYgBiSx+kNJewmzPQWpQjcKCSVCqUh4VDue97pxQIl0By c4M79m2JATMEAAEIAB0WIQSNS5MBqTXjGL5IXszgIcpz8YmpEAUCYksfpAAKCRDgIcpz8YmpEB5jB/ 9eld+0oGKhvE7J+KbV5jdhgBU3yOm9u8wkzwgDLGKZYN1co9zlcvNXYDNMpCxzVnzenDaQemO1HFlH oOvwXs3bpn8/FZHr6qKVeD06Wy8+YG4nWpTog2AGhxAndCjhkuones0q1/ZQSsmizxAAlpJOR23CRV i4yJx1dPms1KuKCOvB/8OK5e2tI3yJTD/o3Vv1SQw2JPMzHjD77IKL1jJyW48KWVa3v9WPorkUCTq6 asC0fLqpi7kWThrZE2t4NdtExO6wHffnewYPcc/zlbr6NlHNvU3K4kMFg37Js/T5rxwWB2tsT2y96A pyFQNKOk30A/Jd5exyhn/AH0ig7bQG X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Stat-Signature: 5xzwbonusorq4y3ti5rrjqn5ddtt3q4i X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: BDCF7140040 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=WUaMCf4+; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=OJvO09Oa; dmarc=none; spf=pass (imf23.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Rspam-User: X-HE-Tag: 1649090481-579274 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Oliver Glitta Many stack traces are similar so there are many similar arrays. Stackdepot saves each unique stack only once. Replace field addrs in struct track with depot_stack_handle_t handle. Use stackdepot to save stack trace. The benefits are smaller memory overhead and possibility to aggregate per-cache statistics in the following patch using the stackdepot handle instead of matching stacks manually. [ vbabka@suse.cz: rebase to 5.17-rc1 and adjust accordingly ] This was initially merged as commit 788691464c29 and reverted by commit ae14c63a9f20 due to several issues, that should now be fixed. The problem of unconditional memory overhead by stackdepot has been addressed by commit 2dba5eb1c73b ("lib/stackdepot: allow optional init and stack_table allocation by kvmalloc()"), so the dependency on stackdepot will result in extra memory usage only when a slab cache tracking is actually enabled, and not for all CONFIG_SLUB_DEBUG builds. The build failures on some architectures were also addressed, and the reported issue with xfs/433 test did not reproduce on 5.17-rc1 with this patch. Signed-off-by: Oliver Glitta Signed-off-by: Vlastimil Babka Reviewed-and-tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: David Rientjes Cc: Christoph Lameter Cc: Pekka Enberg Cc: Joonsoo Kim Acked-by: David Rientjes --- init/Kconfig | 1 + lib/Kconfig.debug | 1 + mm/slab_common.c | 5 ++++ mm/slub.c | 71 ++++++++++++++++++++++++++--------------------- 4 files changed, 47 insertions(+), 31 deletions(-) diff --git a/init/Kconfig b/init/Kconfig index ddcbefe535e9..adc57f989d87 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -1875,6 +1875,7 @@ config SLUB_DEBUG default y bool "Enable SLUB debugging support" if EXPERT depends on SLUB && SYSFS + select STACKDEPOT if STACKTRACE_SUPPORT help SLUB has extensive debug support features. Disabling these can result in significant savings in code size. This also disables diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 075cd25363ac..78d6139111cd 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -709,6 +709,7 @@ config DEBUG_SLAB config SLUB_DEBUG_ON bool "SLUB debugging on by default" depends on SLUB && SLUB_DEBUG + select STACKDEPOT_ALWAYS_INIT if STACKTRACE_SUPPORT default n help Boot with debugging on by default. SLUB boots by default with diff --git a/mm/slab_common.c b/mm/slab_common.c index 6ee64d6208b3..73943479a2b7 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -24,6 +24,7 @@ #include #include #include +#include #define CREATE_TRACE_POINTS #include @@ -314,9 +315,13 @@ kmem_cache_create_usercopy(const char *name, * If no slub_debug was enabled globally, the static key is not yet * enabled by setup_slub_debug(). Enable it if the cache is being * created with any of the debugging flags passed explicitly. + * It's also possible that this is the first cache created with + * SLAB_STORE_USER and we should init stack_depot for it. */ if (flags & SLAB_DEBUG_FLAGS) static_branch_enable(&slub_debug_enabled); + if (flags & SLAB_STORE_USER) + stack_depot_init(); #endif mutex_lock(&slab_mutex); diff --git a/mm/slub.c b/mm/slub.c index cd4fd0159911..98c1450c23f0 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include #include @@ -264,8 +265,8 @@ static inline bool kmem_cache_has_cpu_partial(struct kmem_cache *s) #define TRACK_ADDRS_COUNT 16 struct track { unsigned long addr; /* Called from address */ -#ifdef CONFIG_STACKTRACE - unsigned long addrs[TRACK_ADDRS_COUNT]; /* Called from address */ +#ifdef CONFIG_STACKDEPOT + depot_stack_handle_t handle; #endif int cpu; /* Was running on cpu */ int pid; /* Pid context */ @@ -724,22 +725,19 @@ static struct track *get_track(struct kmem_cache *s, void *object, return kasan_reset_tag(p + alloc); } -static void set_track(struct kmem_cache *s, void *object, +static void noinline set_track(struct kmem_cache *s, void *object, enum track_item alloc, unsigned long addr) { struct track *p = get_track(s, object, alloc); -#ifdef CONFIG_STACKTRACE +#ifdef CONFIG_STACKDEPOT + unsigned long entries[TRACK_ADDRS_COUNT]; unsigned int nr_entries; - metadata_access_enable(); - nr_entries = stack_trace_save(kasan_reset_tag(p->addrs), - TRACK_ADDRS_COUNT, 3); - metadata_access_disable(); - - if (nr_entries < TRACK_ADDRS_COUNT) - p->addrs[nr_entries] = 0; + nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 3); + p->handle = stack_depot_save(entries, nr_entries, GFP_NOWAIT); #endif + p->addr = addr; p->cpu = smp_processor_id(); p->pid = current->pid; @@ -759,20 +757,19 @@ static void init_tracking(struct kmem_cache *s, void *object) static void print_track(const char *s, struct track *t, unsigned long pr_time) { + depot_stack_handle_t handle __maybe_unused; + if (!t->addr) return; pr_err("%s in %pS age=%lu cpu=%u pid=%d\n", s, (void *)t->addr, pr_time - t->when, t->cpu, t->pid); -#ifdef CONFIG_STACKTRACE - { - int i; - for (i = 0; i < TRACK_ADDRS_COUNT; i++) - if (t->addrs[i]) - pr_err("\t%pS\n", (void *)t->addrs[i]); - else - break; - } +#ifdef CONFIG_STACKDEPOT + handle = READ_ONCE(t->handle); + if (handle) + stack_depot_print(handle); + else + pr_err("object allocation/free stack trace missing\n"); #endif } @@ -1532,6 +1529,8 @@ static int __init setup_slub_debug(char *str) global_slub_debug_changed = true; } else { slab_list_specified = true; + if (flags & SLAB_STORE_USER) + stack_depot_want_early_init(); } } @@ -1549,6 +1548,8 @@ static int __init setup_slub_debug(char *str) } out: slub_debug = global_flags; + if (slub_debug & SLAB_STORE_USER) + stack_depot_want_early_init(); if (slub_debug != 0 || slub_debug_string) static_branch_enable(&slub_debug_enabled); else @@ -4342,18 +4343,26 @@ void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab) objp = fixup_red_left(s, objp); trackp = get_track(s, objp, TRACK_ALLOC); kpp->kp_ret = (void *)trackp->addr; -#ifdef CONFIG_STACKTRACE - for (i = 0; i < KS_ADDRS_COUNT && i < TRACK_ADDRS_COUNT; i++) { - kpp->kp_stack[i] = (void *)trackp->addrs[i]; - if (!kpp->kp_stack[i]) - break; - } +#ifdef CONFIG_STACKDEPOT + { + depot_stack_handle_t handle; + unsigned long *entries; + unsigned int nr_entries; + + handle = READ_ONCE(trackp->handle); + if (handle) { + nr_entries = stack_depot_fetch(handle, &entries); + for (i = 0; i < KS_ADDRS_COUNT && i < nr_entries; i++) + kpp->kp_stack[i] = (void *)entries[i]; + } - trackp = get_track(s, objp, TRACK_FREE); - for (i = 0; i < KS_ADDRS_COUNT && i < TRACK_ADDRS_COUNT; i++) { - kpp->kp_free_stack[i] = (void *)trackp->addrs[i]; - if (!kpp->kp_free_stack[i]) - break; + trackp = get_track(s, objp, TRACK_FREE); + handle = READ_ONCE(trackp->handle); + if (handle) { + nr_entries = stack_depot_fetch(handle, &entries); + for (i = 0; i < KS_ADDRS_COUNT && i < nr_entries; i++) + kpp->kp_free_stack[i] = (void *)entries[i]; + } } #endif #endif From patchwork Mon Apr 4 16:41:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12800572 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F988C433F5 for ; Mon, 4 Apr 2022 16:45:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C39CB8D0001; Mon, 4 Apr 2022 12:41:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF0C88D0005; Mon, 4 Apr 2022 12:41:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6820D8D0002; Mon, 4 Apr 2022 12:41:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.28]) by kanga.kvack.org (Postfix) with ESMTP id 175E98D0001 for ; Mon, 4 Apr 2022 12:41:33 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C080B23D80 for ; Mon, 4 Apr 2022 16:41:22 +0000 (UTC) X-FDA: 79319762088.04.6B689A5 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf16.hostedemail.com (Postfix) with ESMTP id 025F7180037 for ; Mon, 4 Apr 2022 16:41:21 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id B766F21100; Mon, 4 Apr 2022 16:41:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1649090480; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FcNvCr26VJpdX7oyGqii89Cwpfix4yCPyYMQI1flsJw=; b=x9FaROwQP77uLqIKZOoSlI3OTwBphZaHdP0q2puFfZdaqsGYbCiQdR7rIijJrNVkueGJoA bgjmv+bFQuNclKeIBDN8sc1LfF3Sk03dM5UJU0HVfDwKK9RZU0zd9YujqzIGCOqEdkfyaa T6kXGFzBiFPuq7f0/BXIlmjeu13ELiA= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1649090480; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FcNvCr26VJpdX7oyGqii89Cwpfix4yCPyYMQI1flsJw=; b=d49e9uB+Jyc2beWWaOwHjc46S4kPYXvSYW9aLKpvhxbDVqZox1KFT/E3SB/JoclUd7Qo+E whqFaXTb9ppELdBA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 88E7D12FC5; Mon, 4 Apr 2022 16:41:20 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id gG3EILAfS2LjfAAAMHmgww (envelope-from ); Mon, 04 Apr 2022 16:41:20 +0000 From: Vlastimil Babka To: David Rientjes , Christoph Lameter , Joonsoo Kim , Pekka Enberg , Roman Gushchin Cc: Andrew Morton , linux-mm@kvack.org, patches@lists.linux.dev, linux-kernel@vger.kernel.org, Oliver Glitta , Marco Elver , Mike Rapoport , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Imran Khan , Vlastimil Babka Subject: [PATCH v3 4/6] mm/slub: distinguish and print stack traces in debugfs files Date: Mon, 4 Apr 2022 18:41:10 +0200 Message-Id: <20220404164112.18372-5-vbabka@suse.cz> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220404164112.18372-1-vbabka@suse.cz> References: <20220404164112.18372-1-vbabka@suse.cz> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2691; i=vbabka@suse.cz; h=from:subject; bh=TYBaV9Bpt1W9CRWxPFEGgUeXoCCN9uIpTjzA+mehKao=; b=owEBbQGS/pANAwAIAeAhynPxiakQAcsmYgBiSx+l4rujDPLMqAaci99S+uU3nVbrVpdi1Mha9UXq y+iVjpGJATMEAAEIAB0WIQSNS5MBqTXjGL5IXszgIcpz8YmpEAUCYksfpQAKCRDgIcpz8YmpEOHICA C1HKyC20o4z3ebpWfjUW2hruI+y8Bq8UBSziz87nv9/7++lyUuk63UBiQGd8O3XB/7ODR83ord0qQ+ ksExeB3w9cLOl0iV1K2DFclkVMgjCoFnKSvatK4bAwW4hnaX62t2LJZL+sm6XEA4KtuQ6BUcgoIAzM 3vquw0pYZ2lBPHTHoU6geu+EI53qoSEcKl6U+ShepJ3tX6iK8Kr9RHl9IF2ihs4r9Sl3p8FH0nIaWM pDZieOmyJ7sT4tG0psuJ6TlppyoP69c57vH/z4WTbX8HLbBFOijhcZTAYlOf3fXETVPKR5P0X5PBvd LHMZK37e2KomEkgwq26UnDtHMqDfS8 X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspam-User: X-Stat-Signature: ez4dh143f17pb8tk7qjs168bpbmmywew Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=x9FaROwQ; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=d49e9uB+; spf=pass (imf16.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 025F7180037 X-HE-Tag: 1649090481-792958 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Oliver Glitta Aggregate objects in slub cache by unique stack trace in addition to caller address when producing contents of debugfs files alloc_traces and free_traces in debugfs. Also add the stack traces to the debugfs output. This makes it much more useful to e.g. debug memory leaks. Signed-off-by: Oliver Glitta Signed-off-by: Vlastimil Babka Reviewed-and-tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 98c1450c23f0..f2e550e1adf0 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -5064,6 +5064,7 @@ EXPORT_SYMBOL(validate_slab_cache); */ struct location { + depot_stack_handle_t handle; unsigned long count; unsigned long addr; long long sum_time; @@ -5116,9 +5117,13 @@ static int add_location(struct loc_track *t, struct kmem_cache *s, { long start, end, pos; struct location *l; - unsigned long caddr; + unsigned long caddr, chandle; unsigned long age = jiffies - track->when; + depot_stack_handle_t handle = 0; +#ifdef CONFIG_STACKDEPOT + handle = READ_ONCE(track->handle); +#endif start = -1; end = t->count; @@ -5133,7 +5138,8 @@ static int add_location(struct loc_track *t, struct kmem_cache *s, break; caddr = t->loc[pos].addr; - if (track->addr == caddr) { + chandle = t->loc[pos].handle; + if ((track->addr == caddr) && (handle == chandle)) { l = &t->loc[pos]; l->count++; @@ -5158,6 +5164,8 @@ static int add_location(struct loc_track *t, struct kmem_cache *s, if (track->addr < caddr) end = pos; + else if (track->addr == caddr && handle < chandle) + end = pos; else start = pos; } @@ -5180,6 +5188,7 @@ static int add_location(struct loc_track *t, struct kmem_cache *s, l->max_time = age; l->min_pid = track->pid; l->max_pid = track->pid; + l->handle = handle; cpumask_clear(to_cpumask(l->cpus)); cpumask_set_cpu(track->cpu, to_cpumask(l->cpus)); nodes_clear(l->nodes); @@ -6089,6 +6098,21 @@ static int slab_debugfs_show(struct seq_file *seq, void *v) seq_printf(seq, " nodes=%*pbl", nodemask_pr_args(&l->nodes)); +#ifdef CONFIG_STACKDEPOT + { + depot_stack_handle_t handle; + unsigned long *entries; + unsigned int nr_entries, j; + + handle = READ_ONCE(l->handle); + if (handle) { + nr_entries = stack_depot_fetch(handle, &entries); + seq_puts(seq, "\n"); + for (j = 0; j < nr_entries; j++) + seq_printf(seq, " %pS\n", (void *)entries[j]); + } + } +#endif seq_puts(seq, "\n"); } From patchwork Mon Apr 4 16:41:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12800571 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 626F9C433EF for ; Mon, 4 Apr 2022 16:44:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A56F38D0003; Mon, 4 Apr 2022 12:41:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 98F958D0001; Mon, 4 Apr 2022 12:41:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B0116B0078; Mon, 4 Apr 2022 12:41:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0062.hostedemail.com [216.40.44.62]) by kanga.kvack.org (Postfix) with ESMTP id 07F116B007B for ; Mon, 4 Apr 2022 12:41:33 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id C182C824C420 for ; Mon, 4 Apr 2022 16:41:22 +0000 (UTC) X-FDA: 79319762004.30.111ED33 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf04.hostedemail.com (Postfix) with ESMTP id 406B54004B for ; Mon, 4 Apr 2022 16:41:22 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id E843121110; Mon, 4 Apr 2022 16:41:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1649090480; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8KmIpmO8YqXFupMJe70SE4ST4S5v6JXlyjUwSJMcLP8=; b=TVTTVGBGcLRn7CeaaMh2C17Xjn0XKxhFq614E6Xpexe+8p9ZeTX0735WHz7q4R2JFLpE8G mJBXgopXf/XUbIdCE/nCMEfHl0PjzmeTEKz21nE4L2f82bNuUz7rVa/nKJj6NefkB6K0QZ 9WuQc9MeADaearI10OPax4gwG//IRq4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1649090480; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8KmIpmO8YqXFupMJe70SE4ST4S5v6JXlyjUwSJMcLP8=; b=ATZhY4H6/PBegfP9r6vtIlKmb29/JpHLT/dEFJTF/Wa0cHS4scHRvlIq7ObkFlJXwLcX45 7i4sjjq+qiHdJvDw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id B8C1713A89; Mon, 4 Apr 2022 16:41:20 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id gPx5LLAfS2LjfAAAMHmgww (envelope-from ); Mon, 04 Apr 2022 16:41:20 +0000 From: Vlastimil Babka To: David Rientjes , Christoph Lameter , Joonsoo Kim , Pekka Enberg , Roman Gushchin Cc: Andrew Morton , linux-mm@kvack.org, patches@lists.linux.dev, linux-kernel@vger.kernel.org, Oliver Glitta , Marco Elver , Mike Rapoport , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Imran Khan , Vlastimil Babka Subject: [PATCH v3 5/6] mm/slub: sort debugfs output by frequency of stack traces Date: Mon, 4 Apr 2022 18:41:11 +0200 Message-Id: <20220404164112.18372-6-vbabka@suse.cz> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220404164112.18372-1-vbabka@suse.cz> References: <20220404164112.18372-1-vbabka@suse.cz> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1585; i=vbabka@suse.cz; h=from:subject; bh=VWFnnyqmJ3+H/KNgJI+ywlf8rteatjMmjGdFv2ZRZAw=; b=owEBbQGS/pANAwAIAeAhynPxiakQAcsmYgBiSx+mYi5UGPiEavlh5JxH0UiWzCBiobPIgKlRfrvu b+85Wi2JATMEAAEIAB0WIQSNS5MBqTXjGL5IXszgIcpz8YmpEAUCYksfpgAKCRDgIcpz8YmpEEOdB/ 9jpr5TWCnpmgEh6Z2MrAz9Xj9T/goJ6+Zgnoln8bhAP12iksWj7XdGTFTK8rwf05YRYaIfHeI1+Vvl LD1DPs0SJFixbeWW0DB2A0EniuqfNgsJ7liONNz1xxioOnkZ8AJFXS8kDyJ2wCjgGLmbMc7I9RSg9/ CuzAJ4CD2x1YoP1GV9e9Gfk0UnaxONR5i0UQZDUU9LXbOx3lOpDUUU129+9gplokFWIFz9PoHtbIXu E95Md+iEzYsS3tUN9gecEJLHPESQsDYrZculV45wdlwyFYLK8VycXMun1fd5I4bPo9pIcEBFNN+g17 juUwhdYTPOxnJHR3nf8kpPtCI7HZMB X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspam-User: Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=TVTTVGBG; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=ATZhY4H6; spf=pass (imf04.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 406B54004B X-Stat-Signature: 8hxnoaash86ph9hqku8fu9mtmbt5q1sf X-HE-Tag: 1649090482-988764 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Oliver Glitta Sort the output of debugfs alloc_traces and free_traces by the frequency of allocation/freeing stack traces. Most frequently used stack traces will be printed first, e.g. for easier memory leak debugging. Signed-off-by: Oliver Glitta Signed-off-by: Vlastimil Babka Reviewed-and-tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Acked-by: David Rientjes --- mm/slub.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/mm/slub.c b/mm/slub.c index f2e550e1adf0..2963dc123336 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include @@ -6137,6 +6138,17 @@ static void *slab_debugfs_next(struct seq_file *seq, void *v, loff_t *ppos) return NULL; } +static int cmp_loc_by_count(const void *a, const void *b, const void *data) +{ + struct location *loc1 = (struct location *)a; + struct location *loc2 = (struct location *)b; + + if (loc1->count > loc2->count) + return -1; + else + return 1; +} + static void *slab_debugfs_start(struct seq_file *seq, loff_t *ppos) { struct loc_track *t = seq->private; @@ -6198,6 +6210,10 @@ static int slab_debug_trace_open(struct inode *inode, struct file *filep) spin_unlock_irqrestore(&n->list_lock, flags); } + /* Sort locations by count */ + sort_r(t->loc, t->count, sizeof(struct location), + cmp_loc_by_count, NULL, NULL); + bitmap_free(obj_map); return 0; } From patchwork Mon Apr 4 16:41:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12800570 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8AA24C433F5 for ; Mon, 4 Apr 2022 16:44:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7CF716B007E; Mon, 4 Apr 2022 12:41:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F9DF6B007D; Mon, 4 Apr 2022 12:41:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 267136B007E; Mon, 4 Apr 2022 12:41:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0084.hostedemail.com [216.40.44.84]) by kanga.kvack.org (Postfix) with ESMTP id 062036B0078 for ; Mon, 4 Apr 2022 12:41:33 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id C2FEDA7992 for ; Mon, 4 Apr 2022 16:41:22 +0000 (UTC) X-FDA: 79319762004.29.EA2EB13 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf27.hostedemail.com (Postfix) with ESMTP id 4542D40041 for ; Mon, 4 Apr 2022 16:41:22 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 2E1961F390; Mon, 4 Apr 2022 16:41:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1649090481; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bei0JZDnlsbNqP4J+qIzBne1zsUAj2g0RJqQ6r2Qiqw=; b=i0ObrEE+b2RebhFzMnlcKdL0BqgE865TCRWCuhUnAv17qBxbGvIcJplUrQaMTs8YblHn+x JVYozogrIS2Flhw1EeDSlj69/ISgAxaOJXo8Mn306ngj4wFWuSk/ION7LdkcmFuho1kt6R AVD9SLP//1+twgSKqikQFnTX4ieiAgM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1649090481; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bei0JZDnlsbNqP4J+qIzBne1zsUAj2g0RJqQ6r2Qiqw=; b=pjR4WFPCNtrEDWu4FIUpx1x9XcuuyDgEQPSsLzovOSperD09sJK2XCWlVGiXuHNbagn1dx JIOhyrVxO5bavqCQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id EC0B112FC5; Mon, 4 Apr 2022 16:41:20 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id yPnrOLAfS2LjfAAAMHmgww (envelope-from ); Mon, 04 Apr 2022 16:41:20 +0000 From: Vlastimil Babka To: David Rientjes , Christoph Lameter , Joonsoo Kim , Pekka Enberg , Roman Gushchin Cc: Andrew Morton , linux-mm@kvack.org, patches@lists.linux.dev, linux-kernel@vger.kernel.org, Oliver Glitta , Marco Elver , Mike Rapoport , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Imran Khan , Vlastimil Babka , Jonathan Corbet , Randy Dunlap , linux-doc@vger.kernel.org Subject: [PATCH v3 6/6] slab, documentation: add description of debugfs files for SLUB caches Date: Mon, 4 Apr 2022 18:41:12 +0200 Message-Id: <20220404164112.18372-7-vbabka@suse.cz> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220404164112.18372-1-vbabka@suse.cz> References: <20220404164112.18372-1-vbabka@suse.cz> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3418; i=vbabka@suse.cz; h=from:subject; bh=aiEHhTlwaPsxqCPnZoF+zACTkJmNyiCa8U4FMIlm4hU=; b=owEBbQGS/pANAwAIAeAhynPxiakQAcsmYgBiSx+n9eEtd8BOf/2A9Gbbt9oR1Y/79g1fBU4U6ywZ pM9StUiJATMEAAEIAB0WIQSNS5MBqTXjGL5IXszgIcpz8YmpEAUCYksfpwAKCRDgIcpz8YmpEECtB/ 9deLfjoG4HXeZchPe/4/vX8Z2LlMjZ83wdxrA0d76J0Ln8c9FtzXuvRCjAhEdYj2UCH80jXW7fMBWF NDqQnwSChm4yVLEyZ9tpycx4qYoRFuoLCug2IVu6YKqglXKFFdMelZXfeqQT7YszkiDYbUTrfmShc/ fZp5U2dZ32SnH3TCbuM+x9Pe8K1tl9FEX0d4pl7xGrHArnPkyB3cA2wLoFleb0C/ZnuJ8R8cCuZQfG 9VHsYtj7qeRn1sgF6cqJ/ycMZf443uHFR6U/7Up+st7X33szxhjIQPP0hto8IlaHBeI2PS+sRU8viJ kT0X6iWUw5xO/W47QkjQckQ7Ozq0xe X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Stat-Signature: nm3b1491zn7f8d7kidhu9qo4pnzfs1xu Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=i0ObrEE+; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=pjR4WFPC; spf=pass (imf27.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 4542D40041 X-HE-Tag: 1649090482-803171 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Oliver Glitta Add description of debugfs files alloc_traces and free_traces to SLUB cache documentation. [ vbabka@suse.cz: some rewording ] Signed-off-by: Oliver Glitta Signed-off-by: Vlastimil Babka Cc: Jonathan Corbet Cc: Randy Dunlap Cc: linux-doc@vger.kernel.org Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Acked-by: Mike Rapoport Acked-by: David Rientjes --- Documentation/vm/slub.rst | 64 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) diff --git a/Documentation/vm/slub.rst b/Documentation/vm/slub.rst index d3028554b1e9..43063ade737a 100644 --- a/Documentation/vm/slub.rst +++ b/Documentation/vm/slub.rst @@ -384,5 +384,69 @@ c) Execute ``slabinfo-gnuplot.sh`` in '-t' mode, passing all of the 40,60`` range will plot only samples collected between 40th and 60th seconds). + +DebugFS files for SLUB +====================== + +For more information about current state of SLUB caches with the user tracking +debug option enabled, debugfs files are available, typically under +/sys/kernel/debug/slab// (created only for caches with enabled user +tracking). There are 2 types of these files with the following debug +information: + +1. alloc_traces:: + + Prints information about unique allocation traces of the currently + allocated objects. The output is sorted by frequency of each trace. + + Information in the output: + Number of objects, allocating function, minimal/average/maximal jiffies since alloc, + pid range of the allocating processes, cpu mask of allocating cpus, and stack trace. + + Example::: + + 1085 populate_error_injection_list+0x97/0x110 age=166678/166680/166682 pid=1 cpus=1:: + __slab_alloc+0x6d/0x90 + kmem_cache_alloc_trace+0x2eb/0x300 + populate_error_injection_list+0x97/0x110 + init_error_injection+0x1b/0x71 + do_one_initcall+0x5f/0x2d0 + kernel_init_freeable+0x26f/0x2d7 + kernel_init+0xe/0x118 + ret_from_fork+0x22/0x30 + + +2. free_traces:: + + Prints information about unique freeing traces of the currently allocated + objects. The freeing traces thus come from the previous life-cycle of the + objects and are reported as not available for objects allocated for the first + time. The output is sorted by frequency of each trace. + + Information in the output: + Number of objects, freeing function, minimal/average/maximal jiffies since free, + pid range of the freeing processes, cpu mask of freeing cpus, and stack trace. + + Example::: + + 1980 age=4294912290 pid=0 cpus=0 + 51 acpi_ut_update_ref_count+0x6a6/0x782 age=236886/237027/237772 pid=1 cpus=1 + kfree+0x2db/0x420 + acpi_ut_update_ref_count+0x6a6/0x782 + acpi_ut_update_object_reference+0x1ad/0x234 + acpi_ut_remove_reference+0x7d/0x84 + acpi_rs_get_prt_method_data+0x97/0xd6 + acpi_get_irq_routing_table+0x82/0xc4 + acpi_pci_irq_find_prt_entry+0x8e/0x2e0 + acpi_pci_irq_lookup+0x3a/0x1e0 + acpi_pci_irq_enable+0x77/0x240 + pcibios_enable_device+0x39/0x40 + do_pci_enable_device.part.0+0x5d/0xe0 + pci_enable_device_flags+0xfc/0x120 + pci_enable_device+0x13/0x20 + virtio_pci_probe+0x9e/0x170 + local_pci_probe+0x48/0x80 + pci_device_probe+0x105/0x1c0 + Christoph Lameter, May 30, 2007 Sergey Senozhatsky, October 23, 2015