From patchwork Sun Jun 2 03:38:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Rostedt X-Patchwork-Id: 13682677 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1AC013F9EC; Sun, 2 Jun 2024 03:37:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717299448; cv=none; b=XvwdRLJ7fqUzFaw77XR2FROPSf+NYIN6gkIyAFbaHpJnUDrqMxE+5wx9qjos2IZZuOxVjJVwplzL7aE9jDiaCyJEYqoBINn2P2/eLo4+ue7L9eJigQoRIWDJ65dnncHJXRSXauopXls1vYzfSjEXBV4abLl6BWRfl9PjGr9vyuk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717299448; c=relaxed/simple; bh=e6OMmkD4s/RHL4/tJKjYQJ+V2EKgDceFAbZtocCTnk4=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=pPSFW3JGVoA3SZGjXOuAkPmCztG0ZL1XRJV70CAp1inamySTMSrOipew04FgIl/hDlith3X+4QV6r9TibYaIiY1nT59xCXdkv0TptA+dOq/iCiyvs6yeTcVybDeXs4LEjcVK210nmj3kZ09AoenmvcjDBPi3OE1S6XsGp2fMGDU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 01821C3277B; Sun, 2 Jun 2024 03:37:28 +0000 (UTC) Received: from rostedt by gandalf with local (Exim 4.97) (envelope-from ) id 1sDc3H-000000094VR-1GpN; Sat, 01 Jun 2024 23:38:35 -0400 Message-ID: <20240602033835.158064307@goodmis.org> User-Agent: quilt/0.68 Date: Sat, 01 Jun 2024 23:38:09 -0400 From: Steven Rostedt To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Mathieu Desnoyers , Andrew Morton , Alexei Starovoitov , Florent Revest , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v2 25/27] function_graph: Use static_call and branch to optimize return function References: <20240602033744.563858532@goodmis.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Steven Rostedt (Google)" In most cases function graph is used by a single user. Instead of calling a loop to call function graph callbacks in this case, call the function return callback directly. Use the static_key that is set when the function graph tracer has less than 2 callbacks registered. It will do the direct call in that case, and will do the loop over all callers when there are 2 or more callbacks registered. Signed-off-by: Steven Rostedt (Google) --- kernel/trace/fgraph.c | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 7c3b0261b1bb..4bf91eebbb08 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -514,6 +514,7 @@ static struct fgraph_ops fgraph_stub = { static struct fgraph_ops *fgraph_direct_gops = &fgraph_stub; DEFINE_STATIC_CALL(fgraph_func, ftrace_graph_entry_stub); +DEFINE_STATIC_CALL(fgraph_retfunc, ftrace_graph_ret_stub); DEFINE_STATIC_KEY_TRUE(fgraph_do_direct); /** @@ -808,13 +809,21 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs bitmap = get_bitmap_bits(current, offset); - for_each_set_bit(i, &bitmap, sizeof(bitmap) * BITS_PER_BYTE) { - struct fgraph_ops *gops = fgraph_array[i]; +#ifdef CONFIG_HAVE_STATIC_CALL + if (static_branch_likely(&fgraph_do_direct)) { + if (test_bit(fgraph_direct_gops->idx, &bitmap)) + static_call(fgraph_retfunc)(&trace, fgraph_direct_gops); + } else +#endif + { + for_each_set_bit(i, &bitmap, sizeof(bitmap) * BITS_PER_BYTE) { + struct fgraph_ops *gops = fgraph_array[i]; - if (gops == &fgraph_stub) - continue; + if (gops == &fgraph_stub) + continue; - gops->retfunc(&trace, gops); + gops->retfunc(&trace, gops); + } } /* @@ -1232,17 +1241,20 @@ static void init_task_vars(int idx) static void ftrace_graph_enable_direct(bool enable_branch) { trace_func_graph_ent_t func = NULL; + trace_func_graph_ret_t retfunc = NULL; int i; for_each_set_bit(i, &fgraph_array_bitmask, sizeof(fgraph_array_bitmask) * BITS_PER_BYTE) { func = fgraph_array[i]->entryfunc; + retfunc = fgraph_array[i]->retfunc; fgraph_direct_gops = fgraph_array[i]; } if (WARN_ON_ONCE(!func)) return; static_call_update(fgraph_func, func); + static_call_update(fgraph_retfunc, retfunc); if (enable_branch) static_branch_disable(&fgraph_do_direct); } @@ -1252,6 +1264,7 @@ static void ftrace_graph_disable_direct(bool disable_branch) if (disable_branch) static_branch_disable(&fgraph_do_direct); static_call_update(fgraph_func, ftrace_graph_entry_stub); + static_call_update(fgraph_retfunc, ftrace_graph_ret_stub); fgraph_direct_gops = &fgraph_stub; }