[RFC,bpf-next,0/3] bpf: BPF runtime hooks: BPF debug mode (tracing all kfuncs and helpers calls)

Message

Juntong Deng Feb. 17, 2025, 5:17 p.m. UTC

Last week, I proposed non-intrusive BPF runtime hooks for implementing
automatic resource release in watchdog [0].

I realized this weekend that the idea of non-intrusive BPF runtime 
hooks can have more application scenarios, and that it is a
general infrastructure.

For example, we can implement BPF version of strace based on BPF runtime
hooks to help us diagnose and debug bpf programs.

We can add BPF debug mode. When a bpf program is in debug mode, all
calls to kfuncs and helpers will be traced and output to the trace
ring buffer.

We can also record timestamps for each call to analyze the performance
of a bpf program, making it easier for us to optimize the performance
of a bpf program.

This patch series is a proof-of-concept implementation of BPF version of
strace based on BPF runtime hooks.

Note that this patch series is developed based on BPF runtime hooks and
therefore cannot be applied separately.

In this patch series I used trace_printk directly in the BPF debug mode
hook for proof-of-concept purposes.

In actual implementation, the BPF debug mode hook should only be
responsible for recording information, and parsing and outputting
information should be done in another thread to avoid affecting
the performance of the bpf program.

We may also need to add a new FTRACE_ENTRY to customize the format of
the event output.

We always use bpf program to trace/analyze/diagnose kernel/applications,
and now we can also trace/analyze/diagnose bpf programs via
BPF debug mode.

This will be helpful for us to debug/optimize complex bpf programs.

(An interesting idea is that maybe one day we will also be able to
attach bpf programs to the BPF debug mode hook. We use a bpf program
to trace other bpf programs in the kernel to help us improve the
performance of other bpf programs. Of course, the bpf program attached
to the debug mode hook cannot enter debug mode.)

[0]: https://lore.kernel.org/bpf/[e-mail]/T/#u

The following is a simple test result (removed trace_printk prefix):

[456.900891] bpf_iter_num_new(ffffa91a02103c78,1,3) = 0
[456.903768] bpf_iter_num_next(ffffa91a02103c78) = ffffa91a02103c78
[456.904849] bpf_task_from_pid(1) = ffff8bf4c1320000
[456.905904] bpf_probe_read_kernel(ffffa91a02103c74,4,ffff8bf4c1320690) = 0
[456.905971] bpf_task_release(ffff8bf4c1320000)
[456.906893] bpf_iter_num_next(ffffa91a02103c78) = ffffa91a02103c78
[456.907751] bpf_task_from_pid(2) = ffff8bf4c1320fc0
[456.908635] bpf_probe_read_kernel(ffffa91a02103c74,4,ffff8bf4c1321650) = 0
[456.908656] bpf_task_release(ffff8bf4c1320fc0)
[456.909570] bpf_iter_num_next(ffffa91a02103c78) = 0
[456.910425] bpf_iter_num_destroy(ffffa91a02103c78)
[456.911604] bpf_cpumask_create() = ffff8bf4c4042a88
[456.912600] bpf_cpumask_release(ffff8bf4c4042a88)

Any feedback is welcome.

Many thanks.

Signed-off-by: Juntong Deng <[e-mail]>

Juntong Deng (3):
  bpf: Add BPF debug mode
  bpf: Add bpf_runtime_kfunc_tracing_hook
  selftests/bpf: Add test case for demonstrating BPF debug mode.

 arch/x86/net/bpf_jit_comp.c                   |   2 +-
 include/linux/bpf.h                           |   3 +-
 include/linux/btf.h                           |   4 +-
 include/uapi/linux/bpf.h                      |   1 +
 kernel/bpf/btf.c                              |  72 +++++++++-
 kernel/bpf/syscall.c                          |   4 +-
 tools/include/uapi/linux/bpf.h                |   2 +
 tools/lib/bpf/libbpf.c                        |   6 +
 tools/lib/bpf/libbpf.h                        |   2 +
 tools/testing/selftests/bpf_debug/Makefile    | 136 ++++++++++++++++++
 tools/testing/selftests/bpf_debug/debug.bpf.c |  39 +++++
 tools/testing/selftests/bpf_debug/debug.c     |  24 ++++
 12 files changed, 290 insertions(+), 5 deletions(-)
 create mode 100644 tools/testing/selftests/bpf_debug/Makefile
 create mode 100644 tools/testing/selftests/bpf_debug/debug.bpf.c
 create mode 100644 tools/testing/selftests/bpf_debug/debug.c