| Message ID | 20250224140151.667679-1-jolsa@kernel.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | uprobes: Add support to optimize usdt probes on x86_64 | expand |
* Jiri Olsa <jolsa@kernel.org> wrote: > hi, > this patchset adds support to optimize usdt probes on top of 5-byte > nop instruction. > > The generic approach (optimize all uprobes) is hard due to emulating > possible multiple original instructions and its related issues. The > usdt case, which stores 5-byte nop seems much easier, so starting > with that. > > The basic idea is to replace breakpoint exception with syscall which > is faster on x86_64. For more details please see changelog of patch 8. > > The run_bench_uprobes.sh benchmark triggers uprobe (on top of different > original instructions) in a loop and counts how many of those happened > per second (the unit below is million loops). > > There's big speed up if you consider current usdt implementation > (uprobe-nop) compared to proposed usdt (uprobe-nop5): > > # ./benchs/run_bench_uprobes.sh > > usermode-count : 818.386 ± 1.886M/s > syscall-count : 8.923 ± 0.003M/s > --> uprobe-nop : 3.086 ± 0.005M/s > uprobe-push : 2.751 ± 0.001M/s > uprobe-ret : 1.481 ± 0.000M/s > --> uprobe-nop5 : 4.016 ± 0.002M/s > uretprobe-nop : 1.712 ± 0.008M/s > uretprobe-push : 1.616 ± 0.001M/s > uretprobe-ret : 1.052 ± 0.000M/s > uretprobe-nop5 : 2.015 ± 0.000M/s So I had to dig into patch #12 to see the magnitude of the speedup: # current: # usermode-count : 818.836 ± 2.842M/s # syscall-count : 8.917 ± 0.003M/s # uprobe-nop : 3.056 ± 0.013M/s # uprobe-push : 2.903 ± 0.002M/s # uprobe-ret : 1.533 ± 0.001M/s # --> uprobe-nop5 : 1.492 ± 0.000M/s # uretprobe-nop : 1.783 ± 0.000M/s # uretprobe-push : 1.672 ± 0.001M/s # uretprobe-ret : 1.067 ± 0.002M/s # --> uretprobe-nop5 : 1.052 ± 0.000M/s # # after the change: # # usermode-count : 818.386 ± 1.886M/s # syscall-count : 8.923 ± 0.003M/s # uprobe-nop : 3.086 ± 0.005M/s # uprobe-push : 2.751 ± 0.001M/s # uprobe-ret : 1.481 ± 0.000M/s # --> uprobe-nop5 : 4.016 ± 0.002M/s # uretprobe-nop : 1.712 ± 0.008M/s # uretprobe-push : 1.616 ± 0.001M/s # uretprobe-ret : 1.052 ± 0.000M/s # --> uretprobe-nop5 : 2.015 ± 0.000M/s That's a +169% and a +91% speedup - pretty darn impressive! Thanks, Ingo
hi, this patchset adds support to optimize usdt probes on top of 5-byte nop instruction. The generic approach (optimize all uprobes) is hard due to emulating possible multiple original instructions and its related issues. The usdt case, which stores 5-byte nop seems much easier, so starting with that. The basic idea is to replace breakpoint exception with syscall which is faster on x86_64. For more details please see changelog of patch 8. The run_bench_uprobes.sh benchmark triggers uprobe (on top of different original instructions) in a loop and counts how many of those happened per second (the unit below is million loops). There's big speed up if you consider current usdt implementation (uprobe-nop) compared to proposed usdt (uprobe-nop5): # ./benchs/run_bench_uprobes.sh usermode-count : 818.386 ± 1.886M/s syscall-count : 8.923 ± 0.003M/s --> uprobe-nop : 3.086 ± 0.005M/s uprobe-push : 2.751 ± 0.001M/s uprobe-ret : 1.481 ± 0.000M/s --> uprobe-nop5 : 4.016 ± 0.002M/s uretprobe-nop : 1.712 ± 0.008M/s uretprobe-push : 1.616 ± 0.001M/s uretprobe-ret : 1.052 ± 0.000M/s uretprobe-nop5 : 2.015 ± 0.000M/s rfc v2 changes: - make uretprobe work properly with optimized uprobe - make the uprobe optimized code x86_64 specific [Peter] - rework the verify function logic, using it now as callback - fix find_nearest_page to include [PAGE_SIZE, ... ] area [Andrii] - try lockless vma lookup in in_uprobe_trampoline [Peter] - do per partes instructions update using int3 like in text_poke_bp_batch [David] - map uprobe trampoline via single global page [Thomas] - keep track of uprobes per mm_struct pending todo (follow ups): - use PROCMAP_QUERY in tests - alloc 'struct uprobes_state' for mm_struct only when needed [Andrii] - seccomp change for new uprobe syscall (same as for uretprobe) thanks, jirka --- Jiri Olsa (18): uprobes: Rename arch_uretprobe_trampoline function uprobes: Make copy_from_page global uprobes: Move ref_ctr_offset update out of uprobe_write_opcode uprobes: Add uprobe_write function uprobes: Add nbytes argument to uprobe_write_opcode uprobes: Add orig argument to uprobe_write and uprobe_write_opcode uprobes: Add swbp argument to arch_uretprobe_hijack_return_addr uprobes/x86: Add uprobe syscall to speed up uprobe uprobes/x86: Add mapping for optimized uprobe trampolines uprobes/x86: Add mm_uprobe objects to track uprobes within mm uprobes/x86: Add support to emulate nop5 instruction uprobes/x86: Add support to optimize uprobes selftests/bpf: Reorg the uprobe_syscall test function selftests/bpf: Use 5-byte nop for x86 usdt probes selftests/bpf: Add uprobe/usdt syscall tests selftests/bpf: Add hit/attach/detach race optimized uprobe test selftests/bpf: Add uprobe syscall sigill signal test selftests/bpf: Add 5-byte nop uprobe trigger bench arch/arm/probes/uprobes/core.c | 4 +- arch/arm64/kernel/probes/uprobes.c | 2 +- arch/csky/kernel/probes/uprobes.c | 2 +- arch/loongarch/kernel/uprobes.c | 2 +- arch/mips/kernel/uprobes.c | 2 +- arch/powerpc/kernel/uprobes.c | 2 +- arch/riscv/kernel/probes/uprobes.c | 2 +- arch/s390/kernel/uprobes.c | 2 +- arch/sparc/kernel/uprobes.c | 2 +- arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/include/asm/uprobes.h | 6 ++ arch/x86/kernel/uprobes.c | 530 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- include/linux/syscalls.h | 2 + include/linux/uprobes.h | 23 +++- kernel/events/uprobes.c | 147 +++++++++++++++++-------- kernel/fork.c | 1 + kernel/sys_ni.c | 1 + tools/testing/selftests/bpf/bench.c | 12 +++ tools/testing/selftests/bpf/benchs/bench_trigger.c | 42 ++++++++ tools/testing/selftests/bpf/benchs/run_bench_uprobes.sh | 2 +- tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c | 342 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++--- tools/testing/selftests/bpf/progs/uprobe_syscall_executed.c | 34 +++++- tools/testing/selftests/bpf/sdt.h | 9 +- 23 files changed, 1093 insertions(+), 79 deletions(-)