From patchwork Sun May 28 05:17:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Rostedt X-Patchwork-Id: 13257755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62F64C77B7C for ; Sun, 28 May 2023 05:26:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229445AbjE1F0w (ORCPT ); Sun, 28 May 2023 01:26:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45196 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229473AbjE1F0s (ORCPT ); Sun, 28 May 2023 01:26:48 -0400 Received: from mammoth.local.home (cpe-172-100-189-27.stny.res.rr.com [172.100.189.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id D4235E4; Sat, 27 May 2023 22:26:45 -0700 (PDT) Received: by mammoth.local.home (Postfix, from userid 5657) id B95EB300A58; Sun, 28 May 2023 01:17:53 -0400 (EDT) From: Steven Rostedt To: LKML , Linux trace kernel Cc: Masami Hiramatsu , Mark Rutland , "Steven Rostedt (Google)" Subject: [PATCH 1/5] tracing: Move setting of tracing_selftest_running out of register_tracer() Date: Sun, 28 May 2023 01:17:38 -0400 Message-Id: <20230528051742.1325503-2-rostedt@goodmis.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230528051742.1325503-1-rostedt@goodmis.org> References: <20230528051742.1325503-1-rostedt@goodmis.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org From: "Steven Rostedt (Google)" The variables tracing_selftest_running and tracing_selftest_disabled are only used for when CONFIG_FTRACE_STARTUP_TEST is enabled. Make them only visible within the selftest code. The setting of those variables are in the register_tracer() call, and set in a location where they do not need to be. Create a wrapper around run_tracer_selftest() called do_run_tracer_selftest() which sets those variables, and have register_tracer() call that instead. Having those variables only set within the CONFIG_FTRACE_STARTUP_TEST scope gets rid of them (and also the ability to remove testing against them) when the startup tests are not enabled (most cases). Signed-off-by: Steven Rostedt (Google) --- kernel/trace/trace.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index 81801dc31784..87e5920b141f 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -2041,6 +2041,17 @@ static int run_tracer_selftest(struct tracer *type) return 0; } +static int do_run_tracer_selftest(struct tracer *type) +{ + int ret; + + tracing_selftest_running = true; + ret = run_tracer_selftest(type); + tracing_selftest_running = false; + + return ret; +} + static __init int init_trace_selftests(void) { struct trace_selftests *p, *n; @@ -2092,6 +2103,10 @@ static inline int run_tracer_selftest(struct tracer *type) { return 0; } +static inline int do_run_tracer_selftest(struct tracer *type) +{ + return 0; +} #endif /* CONFIG_FTRACE_STARTUP_TEST */ static void add_tracer_options(struct trace_array *tr, struct tracer *t); @@ -2127,8 +2142,6 @@ int __init register_tracer(struct tracer *type) mutex_lock(&trace_types_lock); - tracing_selftest_running = true; - for (t = trace_types; t; t = t->next) { if (strcmp(type->name, t->name) == 0) { /* already found */ @@ -2157,7 +2170,7 @@ int __init register_tracer(struct tracer *type) /* store the tracer for __set_tracer_option */ type->flags->trace = type; - ret = run_tracer_selftest(type); + ret = do_run_tracer_selftest(type); if (ret < 0) goto out; @@ -2166,7 +2179,6 @@ int __init register_tracer(struct tracer *type) add_tracer_options(&global_trace, type); out: - tracing_selftest_running = false; mutex_unlock(&trace_types_lock); if (ret || !default_bootup_tracer) From patchwork Sun May 28 05:17:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Rostedt X-Patchwork-Id: 13257752 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36ADBC7EE2F for ; Sun, 28 May 2023 05:26:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229485AbjE1F0u (ORCPT ); Sun, 28 May 2023 01:26:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45188 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229456AbjE1F0r (ORCPT ); Sun, 28 May 2023 01:26:47 -0400 Received: from mammoth.local.home (cpe-172-100-189-27.stny.res.rr.com [172.100.189.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id AEE8CDC; Sat, 27 May 2023 22:26:40 -0700 (PDT) Received: by mammoth.local.home (Postfix, from userid 5657) id C1A87300A64; Sun, 28 May 2023 01:17:53 -0400 (EDT) From: Steven Rostedt To: LKML , Linux trace kernel Cc: Masami Hiramatsu , Mark Rutland , "Steven Rostedt (Google)" Subject: [PATCH 2/5] tracing: Have tracer selftests call cond_resched() before running Date: Sun, 28 May 2023 01:17:39 -0400 Message-Id: <20230528051742.1325503-3-rostedt@goodmis.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230528051742.1325503-1-rostedt@goodmis.org> References: <20230528051742.1325503-1-rostedt@goodmis.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org From: "Steven Rostedt (Google)" As there are more and more internal selftests being added to the Linux kernel (KSAN, lockdep, etc) the selftests are taking longer to run when these are enabled. Add a cond_resched() to the calling of do_run_tracer_selftest() to force a schedule if NEED_RESCHED is set, otherwise the soft lockup watchdog may trigger on boot up. Signed-off-by: Steven Rostedt (Google) --- kernel/trace/trace.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index 87e5920b141f..70f2b511b9cd 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -2045,6 +2045,13 @@ static int do_run_tracer_selftest(struct tracer *type) { int ret; + /* + * Tests can take a long time, especially if they are run one after the + * other, as does happen during bootup when all the tracers are + * registered. This could cause the soft lockup watchdog to trigger. + */ + cond_resched(); + tracing_selftest_running = true; ret = run_tracer_selftest(type); tracing_selftest_running = false; From patchwork Sun May 28 05:17:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Rostedt X-Patchwork-Id: 13257753 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BAB9C77B7E for ; Sun, 28 May 2023 05:26:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229483AbjE1F0t (ORCPT ); Sun, 28 May 2023 01:26:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229437AbjE1F0r (ORCPT ); Sun, 28 May 2023 01:26:47 -0400 Received: from mammoth.local.home (cpe-172-100-189-27.stny.res.rr.com [172.100.189.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id EC282E1; Sat, 27 May 2023 22:26:40 -0700 (PDT) Received: by mammoth.local.home (Postfix, from userid 5657) id C8B2D300CB6; Sun, 28 May 2023 01:17:53 -0400 (EDT) From: Steven Rostedt To: LKML , Linux trace kernel Cc: Masami Hiramatsu , Mark Rutland , "Steven Rostedt (Google)" Subject: [PATCH 3/5] tracing: Make tracing_selftest_running/delete nops when not used Date: Sun, 28 May 2023 01:17:40 -0400 Message-Id: <20230528051742.1325503-4-rostedt@goodmis.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230528051742.1325503-1-rostedt@goodmis.org> References: <20230528051742.1325503-1-rostedt@goodmis.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org From: "Steven Rostedt (Google)" There's no reason to test the condition variables tracing_selftest_running or tracing_selftest_delete when tracing selftests are not enabled. Make them define 0s when not the selftests are not configured in. Signed-off-by: Steven Rostedt (Google) --- kernel/trace/trace.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index 70f2b511b9cd..004f5f99e943 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -60,6 +60,7 @@ */ bool ring_buffer_expanded; +#ifdef CONFIG_FTRACE_STARTUP_TEST /* * We need to change this state when a selftest is running. * A selftest will lurk into the ring-buffer to count the @@ -75,7 +76,6 @@ static bool __read_mostly tracing_selftest_running; */ bool __read_mostly tracing_selftest_disabled; -#ifdef CONFIG_FTRACE_STARTUP_TEST void __init disable_tracing_selftest(const char *reason) { if (!tracing_selftest_disabled) { @@ -83,6 +83,9 @@ void __init disable_tracing_selftest(const char *reason) pr_info("Ftrace startup test is disabled due to %s\n", reason); } } +#else +#define tracing_selftest_running 0 +#define tracing_selftest_disabled 0 #endif /* Pipe tracepoints to printk */ From patchwork Sun May 28 05:17:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Rostedt X-Patchwork-Id: 13257754 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57EC3C7EE31 for ; Sun, 28 May 2023 05:26:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229486AbjE1F0v (ORCPT ); Sun, 28 May 2023 01:26:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45182 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229472AbjE1F0r (ORCPT ); Sun, 28 May 2023 01:26:47 -0400 Received: from mammoth.local.home (cpe-172-100-189-27.stny.res.rr.com [172.100.189.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id B7A33DE; Sat, 27 May 2023 22:26:40 -0700 (PDT) Received: by mammoth.local.home (Postfix, from userid 5657) id CF898300CD5; Sun, 28 May 2023 01:17:53 -0400 (EDT) From: Steven Rostedt To: LKML , Linux trace kernel Cc: Masami Hiramatsu , Mark Rutland , "Steven Rostedt (Google)" Subject: [PATCH 4/5] tracing: Only make selftest conditionals affect the global_trace Date: Sun, 28 May 2023 01:17:41 -0400 Message-Id: <20230528051742.1325503-5-rostedt@goodmis.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230528051742.1325503-1-rostedt@goodmis.org> References: <20230528051742.1325503-1-rostedt@goodmis.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org From: "Steven Rostedt (Google)" The tracing_selftest_running and tracing_selftest_disabled variables were to keep trace_printk() and other writes from affecting the tracing selftests, as the tracing selftests would examine the ring buffer to see if it contained what it expected or not. trace_printk() and friends could add to the ring buffer and cause the selftests to fail (and then disable the tracer that was being tested). To keep that from happening, these variables were added and would keep trace_printk() and friends from writing to the ring buffer while the tests were going on. But this was only the top level ring buffer (owned by the global_trace instance). There is no reason to prevent writing into ring buffers of other instances via the trace_array_printk() and friends. For the functions that could be used by other instances, check if the global_trace is the tracer instance that is being written to before deciding to not allow the write. Signed-off-by: Steven Rostedt (Google) --- kernel/trace/trace.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index 004f5f99e943..64a4dde073ef 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -1054,7 +1054,10 @@ int __trace_array_puts(struct trace_array *tr, unsigned long ip, if (!(tr->trace_flags & TRACE_ITER_PRINTK)) return 0; - if (unlikely(tracing_selftest_running || tracing_disabled)) + if (unlikely(tracing_selftest_running && tr == &global_trace)) + return 0; + + if (unlikely(tracing_disabled)) return 0; alloc = sizeof(*entry) + size + 2; /* possible \n added */ @@ -3512,7 +3515,7 @@ __trace_array_vprintk(struct trace_buffer *buffer, unsigned int trace_ctx; char *tbuffer; - if (tracing_disabled || tracing_selftest_running) + if (tracing_disabled) return 0; /* Don't pollute graph traces with trace_vprintk internals */ @@ -3560,6 +3563,9 @@ __printf(3, 0) int trace_array_vprintk(struct trace_array *tr, unsigned long ip, const char *fmt, va_list args) { + if (tracing_selftest_running && tr == &global_trace) + return 0; + return __trace_array_vprintk(tr->array_buffer.buffer, ip, fmt, args); } From patchwork Sun May 28 05:17:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Rostedt X-Patchwork-Id: 13257751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BDE7C77B7C for ; Sun, 28 May 2023 05:26:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229481AbjE1F0s (ORCPT ); Sun, 28 May 2023 01:26:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45176 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229470AbjE1F0r (ORCPT ); Sun, 28 May 2023 01:26:47 -0400 X-Greylist: delayed 525 seconds by postgrey-1.37 at lindbergh.monkeyblade.net; Sat, 27 May 2023 22:26:40 PDT Received: from mammoth.local.home (cpe-172-100-189-27.stny.res.rr.com [172.100.189.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 94AC7D8; Sat, 27 May 2023 22:26:40 -0700 (PDT) Received: by mammoth.local.home (Postfix, from userid 5657) id D650C30113E; Sun, 28 May 2023 01:17:53 -0400 (EDT) From: Steven Rostedt To: LKML , Linux trace kernel Cc: Masami Hiramatsu , Mark Rutland , "Steven Rostedt (Google)" Subject: [PATCH 5/5] tracing: Have function_graph selftest call cond_resched() Date: Sun, 28 May 2023 01:17:42 -0400 Message-Id: <20230528051742.1325503-6-rostedt@goodmis.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230528051742.1325503-1-rostedt@goodmis.org> References: <20230528051742.1325503-1-rostedt@goodmis.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org From: "Steven Rostedt (Google)" When all kernel debugging is enabled (lockdep, KSAN, etc), the function graph enabling and disabling can take several seconds to complete. The function_graph selftest enables and disables function graph tracing several times. With full debugging enabled, the soft lockup watchdog was triggering because the selftest was running without ever scheduling. Add cond_resched() throughout the test to make sure it does not trigger the soft lockup detector. Signed-off-by: Steven Rostedt (Google) --- kernel/trace/trace_selftest.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c index a931d9aaea26..529590499b1f 100644 --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -848,6 +848,12 @@ trace_selftest_startup_function_graph(struct tracer *trace, } #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + /* + * These tests can take some time to run. Make sure on non PREEMPT + * kernels, we do not trigger the softlockup detector. + */ + cond_resched(); + tracing_reset_online_cpus(&tr->array_buffer); set_graph_array(tr); @@ -869,6 +875,8 @@ trace_selftest_startup_function_graph(struct tracer *trace, if (ret) goto out; + cond_resched(); + ret = register_ftrace_graph(&fgraph_ops); if (ret) { warn_failed_init_tracer(trace, ret); @@ -891,6 +899,8 @@ trace_selftest_startup_function_graph(struct tracer *trace, if (ret) goto out; + cond_resched(); + tracing_start(); if (!ret && !count) {