From patchwork Mon May 1 16:54:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227667 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 722E6C77B73 for ; Mon, 1 May 2023 16:56:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 10EFC900006; Mon, 1 May 2023 12:56:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C11A900002; Mon, 1 May 2023 12:56:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA270900006; Mon, 1 May 2023 12:56:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DAF43900002 for ; Mon, 1 May 2023 12:56:25 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B6C4EADA5B for ; Mon, 1 May 2023 16:56:25 +0000 (UTC) X-FDA: 80742289530.22.A6A167A Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf29.hostedemail.com (Postfix) with ESMTP id D9C19120015 for ; Mon, 1 May 2023 16:56:23 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=ebS+MuTL; spf=pass (imf29.hostedemail.com: domain of 3N-9PZAYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3N-9PZAYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960183; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bfMxO5ATd5n4S/fh5B/Nudv/nxDkBrDhDPER315pa0U=; b=wdQvCRndGxjXXQfzN5O7pwU6jy0CCYMUC6a5DyKxOFmWC/SwUjiHd1LUrEPH2Rx9Vlw9cQ uBT4SerOzJ97EpmoAq6fllO/Fwr5T6gA34sQztscY7o/l1tZYnjnMxql3iYLXE5L9GsDi2 M21phL2wgL8B0b0yMr3CNgbSUhtEkFk= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=ebS+MuTL; spf=pass (imf29.hostedemail.com: domain of 3N-9PZAYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3N-9PZAYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960183; a=rsa-sha256; cv=none; b=kmB/LQNVxbAjNPlR2puQn60BIjbqNC1rSQhI7irzYQcGJTbT/xIBpnV9k/HOBOjJfpv7jS HoWP4STETI0XazTaGzDWCNqMM17QQq+qJfj+nqTYAKjGGmsBVZzKNv1+FnyilpY9FBhbAO EOpJ8BX57SCW5BJztC9RwRYAcpQVHJ8= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-b9a792ff423so5508728276.1 for ; Mon, 01 May 2023 09:56:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960183; x=1685552183; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=bfMxO5ATd5n4S/fh5B/Nudv/nxDkBrDhDPER315pa0U=; b=ebS+MuTL+cQNWT4eZEJvTQOcTARn+QnzuJah4/SMgcb7+qkoUNzC7ex8MrpVKhZRXc KgeXlio4zrfdxmOppnmcCQe6TR+hOuBdIc9ihcKiKIEVRbsGHe8ObZbrrM4utujRh9wG TqBTS5rCEI/Jizsrf15UbPFacomNmio9VSdaY4o+amTMj91xth7X0uCHsGdXYXQ52AxU vnIxBaA8u69160qtwfedxuqqybJKyd5xlJm5XUDfPQCWKc/CaKvS4YCnPYvKKCkeigKq aazNcodiJ2//C8yradLEHTJBg1grANJmustwDAaYwDmuE541j3ivI8Z4B8/7N+gU15xY 5tZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960183; x=1685552183; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bfMxO5ATd5n4S/fh5B/Nudv/nxDkBrDhDPER315pa0U=; b=he1lpvw2qzUi1mfIqj8vigw+jN7bBt4clCD5etiqKg75vbLBSSOBH3VDRYomKSLB5Q u1k/yAFQI/FxzmL9v7zs3OZ7NAjw6k3XmJdvELWA+n218mQJCy9JfgKV3iS/9w2chC1H zVobUh4Sw16sPFBiFgBd27mYAU//TiD8Ir3Ofr6Ztbnd/KkZHbZGn2ibqMxxB3dBtL4v 0AJVOXTWiivGvcOEdTb/eIz4dBzemqxIAxt8GapKUfkOX401N3xF7G3Cq72b/B+u3SA/ VFA5a05z6RJysn934kK04KkTJ9qgA5sSiYu2yo39LiMCtKP0zkYrUNMpEn+fkC0VA6Cm h9kw== X-Gm-Message-State: AC+VfDwB9utf3sLa4cV4xW1WJGxhchbX4FDQoE0demFgviEdpPXH+iXu EjcFDCQUadGYv1xvTdqPiFg6t0sTILI= X-Google-Smtp-Source: ACHHUZ6mXAn8JgUNXvyGZqHJw9Jv8uWK4VnvTdJsjQhuuRu+wtW4LhhvfSOs0fXoyYvuKXJ26U7UG2NCZPo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:8046:0:b0:b9d:c866:d92d with SMTP id a6-20020a258046000000b00b9dc866d92dmr3485899ybn.1.1682960183189; Mon, 01 May 2023 09:56:23 -0700 (PDT) Date: Mon, 1 May 2023 09:54:43 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-34-surenb@google.com> Subject: [PATCH 33/40] move stack capture functionality into a separate function for reuse From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: abmd8te4szrta9ebzmk9qz5ngb6zq7fr X-Rspam-User: X-Rspamd-Queue-Id: D9C19120015 X-Rspamd-Server: rspam06 X-HE-Tag: 1682960183-59543 X-HE-Meta: U2FsdGVkX1/NA92+SMAIoFlhR+Kh+zMuuEvyw4Jo9qMw7fX9AKkX88vEwByd/MR88Sm9Bk/QLcF1dOzBpHQwb6XQdwUXN2zCo/+faSu+8wwX+ZFHzVwVMhdCgCjr9E6aQxyOspJTMKq7bZ31iF5LsI9tPrnklna6arV6xxwKtDeRFSNX+A4iYxWjBc2QA5t4wURmnDz1OrjMcmH1iTtmnym8AL2JHw5xDJjwpnk6eCXlp7GDYGzGRfvZaCuoSZdqxnVjny46BP6+ixfhSfR+PVZyhQV6UmHy7mVu/TACMJBg7A1gpnAq0/7jIDnnwtATlICJ1HsQCpOz9b8lo22//wLA2qXzUlYGWYPxVU9gggqIIX8UYmitFnpPyKrCJiu8iXLsQ5CtvS33u/NDc08fM/pVDjmYV7F9E8rYEjMG9/a3e983rCNn+ixWAaxHhlkohj3esjs8G23WuwSXIwdVwzA9qyHBMl0BAvDFNSFIeApxJVwQQeusbJ2kz74IipcrJ2sVWroXimWsP7YPE7i0MCx+NZ07D66UBJhDuBT1L+RkYIwU9CnO0181ujK7SF7molbKidiEx9AwECy2WZmOr75S6kF7wWD45L7be9g3aMwpO4XtZqoVBjy9Bpx8Ev7xhEP0GG9lQB7WIGdyIRIOx/zuPULbEUYHM2PsQfufQnQf/DMipRy2Il3fOgINOGN2wjO8VNts4GKnp8u9m9xDeenuQvh3L8C7VRMFX4wvciv9o/3BDVQIGqPhHchuhju8zsuVEBwbvZKrwK4wNxsus7gPf7ycbiX6dcWto8cn1/WyC8dm1eTccJW9SKog3LalLxV8RgCe2DFhgZFgCpONQGKFztokBsl+h9DVTnoH1zwGqrCRR5rQaYC8BhFODxMHmn/8FDwRwMRl35u+PXW7BK69ouHoSqOU1em3bG2rFLlQr1N5/aaNjN4lZ5J9qjZJC792bd6GUpZELzVRQB5 cV8EfKw0 K5qcPEdg096PJEqrnfmfHuHd9U/xURsrNOP33CRTrH7zXFIls/4uM3MP0lfLMQGbQFXofX2Q97zlEHMtqoXVIsQKYXLMHXv/H0gK21oN9RPM8ecWJh5zGUbiqnt2F2uPJg72V66zWBAQ60m/iggvsjzKXcfac5lq3GiZwGg21mG3KlePLR9+ybg61XQcgmdzLEuDiXHoIl3ersW/iFaBbddvEFT4UdYvSgcfAc9OzXtmgysbdO7wbJKy+fgiMB9kfeK10kpu90hC7r+9dkVY4WpyB5vj3QFci9Xfl51oaQnq1UpWDB9lBzTQ9G6WlzlakNAW3WAzFxWrA9GzlYrmM8QPWEcbweC5QvJuMjdZc2gcGVq/n4tPYut7mA4Zoef3PiAiagpDY8tDW+sBfY6sc1SubXiiPOcJY+/ev/sV1yUrjVhVA1BBEwsxyqxJohaSbZmXVdF4AfRj5jB12+4t/t7UvyPkn8DjLksbiPJwSXm0Fga5qlHJ6LP2ykRclLZ0GZqG/T2VmcQWI2W6GorKaIFE/C+oABIi6mkP9c/+E0UeiGyPi5ucg/mJyqHzeB2Xu2rTRyp+VCutpL6L3K0209WY0olK4EOGBXz+yhc0NWBIsQnOU2B/1dsJDdl9DeTXVEsd4M8cObATCVbMO3/kCLxATonQ3DJjqr7I3mgchRpNL2csiSMN88NXyMQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make save_stack() function part of stackdepot API to be used outside of page_owner. Also rename task_struct's in_page_owner to in_capture_stack flag to better convey the wider use of this flag. Signed-off-by: Suren Baghdasaryan --- include/linux/sched.h | 6 ++-- include/linux/stackdepot.h | 16 +++++++++ lib/stackdepot.c | 68 ++++++++++++++++++++++++++++++++++++++ mm/page_owner.c | 52 ++--------------------------- 4 files changed, 90 insertions(+), 52 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 33708bf8f191..6eca46ab6d78 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -942,9 +942,9 @@ struct task_struct { /* Stalled due to lack of memory */ unsigned in_memstall:1; #endif -#ifdef CONFIG_PAGE_OWNER - /* Used by page_owner=on to detect recursion in page tracking. */ - unsigned in_page_owner:1; +#ifdef CONFIG_STACKDEPOT + /* Used by stack_depot_capture_stack to detect recursion. */ + unsigned in_capture_stack:1; #endif #ifdef CONFIG_EVENTFD /* Recursion prevention for eventfd_signal() */ diff --git a/include/linux/stackdepot.h b/include/linux/stackdepot.h index e58306783d8e..baf7e80cf449 100644 --- a/include/linux/stackdepot.h +++ b/include/linux/stackdepot.h @@ -164,4 +164,20 @@ depot_stack_handle_t __must_check stack_depot_set_extra_bits( */ unsigned int stack_depot_get_extra_bits(depot_stack_handle_t handle); +/** + * stack_depot_capture_init - Initialize stack depot capture mechanism + * + * Return: Stack depot initialization status + */ +bool stack_depot_capture_init(void); + +/** + * stack_depot_capture_stack - Capture current stack trace into stack depot + * + * @flags: Allocation GFP flags + * + * Return: Handle of the stack trace stored in depot, 0 on failure + */ +depot_stack_handle_t stack_depot_capture_stack(gfp_t flags); + #endif diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 2f5aa851834e..c7e5e22fcb16 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -539,3 +539,71 @@ unsigned int stack_depot_get_extra_bits(depot_stack_handle_t handle) return parts.extra; } EXPORT_SYMBOL(stack_depot_get_extra_bits); + +static depot_stack_handle_t recursion_handle; +static depot_stack_handle_t failure_handle; + +static __always_inline depot_stack_handle_t create_custom_stack(void) +{ + unsigned long entries[4]; + unsigned int nr_entries; + + nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 0); + return stack_depot_save(entries, nr_entries, GFP_KERNEL); +} + +static noinline void register_recursion_stack(void) +{ + recursion_handle = create_custom_stack(); +} + +static noinline void register_failure_stack(void) +{ + failure_handle = create_custom_stack(); +} + +bool stack_depot_capture_init(void) +{ + static DEFINE_MUTEX(stack_depot_capture_init_mutex); + static bool utility_stacks_ready; + + mutex_lock(&stack_depot_capture_init_mutex); + if (!utility_stacks_ready) { + register_recursion_stack(); + register_failure_stack(); + utility_stacks_ready = true; + } + mutex_unlock(&stack_depot_capture_init_mutex); + + return utility_stacks_ready; +} + +/* TODO: teach stack_depot_capture_stack to use off stack temporal storage */ +#define CAPTURE_STACK_DEPTH (16) + +depot_stack_handle_t stack_depot_capture_stack(gfp_t flags) +{ + unsigned long entries[CAPTURE_STACK_DEPTH]; + depot_stack_handle_t handle; + unsigned int nr_entries; + + /* + * Avoid recursion. + * + * Sometimes page metadata allocation tracking requires more + * memory to be allocated: + * - when new stack trace is saved to stack depot + * - when backtrace itself is calculated (ia64) + */ + if (current->in_capture_stack) + return recursion_handle; + current->in_capture_stack = 1; + + nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 2); + handle = stack_depot_save(entries, nr_entries, flags); + if (!handle) + handle = failure_handle; + + current->in_capture_stack = 0; + return handle; +} diff --git a/mm/page_owner.c b/mm/page_owner.c index 8b6086c666e6..9fafbc290d5b 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -15,12 +15,6 @@ #include "internal.h" -/* - * TODO: teach PAGE_OWNER_STACK_DEPTH (__dump_page_owner and save_stack) - * to use off stack temporal storage - */ -#define PAGE_OWNER_STACK_DEPTH (16) - struct page_owner { unsigned short order; short last_migrate_reason; @@ -37,8 +31,6 @@ struct page_owner { static bool page_owner_enabled __initdata; DEFINE_STATIC_KEY_FALSE(page_owner_inited); -static depot_stack_handle_t dummy_handle; -static depot_stack_handle_t failure_handle; static depot_stack_handle_t early_handle; static void init_early_allocated_pages(void); @@ -68,16 +60,6 @@ static __always_inline depot_stack_handle_t create_dummy_stack(void) return stack_depot_save(entries, nr_entries, GFP_KERNEL); } -static noinline void register_dummy_stack(void) -{ - dummy_handle = create_dummy_stack(); -} - -static noinline void register_failure_stack(void) -{ - failure_handle = create_dummy_stack(); -} - static noinline void register_early_stack(void) { early_handle = create_dummy_stack(); @@ -88,8 +70,7 @@ static __init void init_page_owner(void) if (!page_owner_enabled) return; - register_dummy_stack(); - register_failure_stack(); + stack_depot_capture_init(); register_early_stack(); static_branch_enable(&page_owner_inited); init_early_allocated_pages(); @@ -107,33 +88,6 @@ static inline struct page_owner *get_page_owner(struct page_ext *page_ext) return (void *)page_ext + page_owner_ops.offset; } -static noinline depot_stack_handle_t save_stack(gfp_t flags) -{ - unsigned long entries[PAGE_OWNER_STACK_DEPTH]; - depot_stack_handle_t handle; - unsigned int nr_entries; - - /* - * Avoid recursion. - * - * Sometimes page metadata allocation tracking requires more - * memory to be allocated: - * - when new stack trace is saved to stack depot - * - when backtrace itself is calculated (ia64) - */ - if (current->in_page_owner) - return dummy_handle; - current->in_page_owner = 1; - - nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 2); - handle = stack_depot_save(entries, nr_entries, flags); - if (!handle) - handle = failure_handle; - - current->in_page_owner = 0; - return handle; -} - void __reset_page_owner(struct page *page, unsigned short order) { int i; @@ -146,7 +100,7 @@ void __reset_page_owner(struct page *page, unsigned short order) if (unlikely(!page_ext)) return; - handle = save_stack(GFP_NOWAIT | __GFP_NOWARN); + handle = stack_depot_capture_stack(GFP_NOWAIT | __GFP_NOWARN); for (i = 0; i < (1 << order); i++) { __clear_bit(PAGE_EXT_OWNER_ALLOCATED, &page_ext->flags); page_owner = get_page_owner(page_ext); @@ -189,7 +143,7 @@ noinline void __set_page_owner(struct page *page, unsigned short order, struct page_ext *page_ext; depot_stack_handle_t handle; - handle = save_stack(gfp_mask); + handle = stack_depot_capture_stack(gfp_mask); page_ext = page_ext_get(page); if (unlikely(!page_ext))