| Message ID | 1421064563-77831-1-git-send-email-wangnan0@huawei.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
(2015/01/12 21:09), Wang Nan wrote: > Original code failed to disarm the probed instruction after > > echo 0 > /sys/kernel/debug/kprobes/enabled > > if OPTPROBE is enabled. > > This patch checks kprobes_all_disarmed in unoptimized_kprobe(). > Looks good :) Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Thank you! > Signed-off-by: Wang Nan <wangnan0@huawei.com> > --- > kernel/kprobes.c | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/kernel/kprobes.c b/kernel/kprobes.c > index 9471710..f16936b 100644 > --- a/kernel/kprobes.c > +++ b/kernel/kprobes.c > @@ -630,6 +630,9 @@ static void unoptimize_kprobe(struct kprobe *p, bool force) > { > struct optimized_kprobe *op; > > + if (kprobes_all_disarmed) > + return; > + > if (!kprobe_aggrprobe(p) || kprobe_disarmed(p)) > return; /* This is not an optprobe nor optimized */ > >
Hi Masami Hiramatsu, I can't find this patch and '[PATCH] kprobes: bugfix: checks kprobes_all_disarmed in unoptimized_kprobe().' in current mainline. How do these patches get there? Should they be merged into Russell King's tree first? Thank you! On 2015/1/12 20:52, Masami Hiramatsu wrote: > (2015/01/12 21:09), Wang Nan wrote: >> Original code failed to disarm the probed instruction after >> >> echo 0 > /sys/kernel/debug/kprobes/enabled >> >> if OPTPROBE is enabled. >> >> This patch checks kprobes_all_disarmed in unoptimized_kprobe(). >> > > Looks good :) > > Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> > > Thank you! > >> Signed-off-by: Wang Nan <wangnan0@huawei.com> >> --- >> kernel/kprobes.c | 3 +++ >> 1 file changed, 3 insertions(+) >> >> diff --git a/kernel/kprobes.c b/kernel/kprobes.c >> index 9471710..f16936b 100644 >> --- a/kernel/kprobes.c >> +++ b/kernel/kprobes.c >> @@ -630,6 +630,9 @@ static void unoptimize_kprobe(struct kprobe *p, bool force) >> { >> struct optimized_kprobe *op; >> >> + if (kprobes_all_disarmed) >> + return; >> + >> if (!kprobe_aggrprobe(p) || kprobe_disarmed(p)) >> return; /* This is not an optprobe nor optimized */ >> >> > >
Hi Wang, I've found a problem on this patch, since kprobes calls unoptioize_kprobe with kprobes_all_disarmed=true when trying to disable all kprobes, this cause a serious problem. Moreover, I couldn't reproduce your reported bug on my 3.19-rc4 kernel. Could you test it again? Unless I could reproduce this bug, I'd like to keep this uncommitted. Thank you, (2015/01/19 12:04), Wang Nan wrote: > Hi Masami Hiramatsu, > > I can't find this patch and '[PATCH] kprobes: bugfix: checks kprobes_all_disarmed > in unoptimized_kprobe().' in current mainline. How do these patches get there? > Should they be merged into Russell King's tree first? > > Thank you! > > On 2015/1/12 20:52, Masami Hiramatsu wrote: >> (2015/01/12 21:09), Wang Nan wrote: >>> Original code failed to disarm the probed instruction after >>> >>> echo 0 > /sys/kernel/debug/kprobes/enabled >>> >>> if OPTPROBE is enabled. >>> >>> This patch checks kprobes_all_disarmed in unoptimized_kprobe(). >>> >> >> Looks good :) >> >> Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> >> >> Thank you! >> >>> Signed-off-by: Wang Nan <wangnan0@huawei.com> >>> --- >>> kernel/kprobes.c | 3 +++ >>> 1 file changed, 3 insertions(+) >>> >>> diff --git a/kernel/kprobes.c b/kernel/kprobes.c >>> index 9471710..f16936b 100644 >>> --- a/kernel/kprobes.c >>> +++ b/kernel/kprobes.c >>> @@ -630,6 +630,9 @@ static void unoptimize_kprobe(struct kprobe *p, bool force) >>> { >>> struct optimized_kprobe *op; >>> >>> + if (kprobes_all_disarmed) >>> + return; >>> + >>> if (!kprobe_aggrprobe(p) || kprobe_disarmed(p)) >>> return; /* This is not an optprobe nor optimized */ >>> >>> >> >> > > >
On 2015/1/19 17:05, Masami Hiramatsu wrote: > Hi Wang, > > I've found a problem on this patch, since kprobes calls unoptioize_kprobe > with kprobes_all_disarmed=true when trying to disable all kprobes, this > cause a serious problem. > > Moreover, I couldn't reproduce your reported bug on my 3.19-rc4 kernel. > Could you test it again? > I tested it again based on 3.19-rc5 and found that the problem still exists. My testing is based on QEMU. First I tested my kprobeopt for ARM, then on x86_64. The test results are pasted at the bottom of this mail. Commands after 'gdb attaches to QEMU' is my actions on a gdb console attached to QEMU; commands after 'inside virtual machine' is what I do in Linux run under QEMU. ** ARM result ** ------ gdb attaches to QEMU ------- (gdb) x/3i sys_open 0xc013619c <SyS_open>: mov r12, sp <--- *original insn* 0xc01361a0 <SyS_open+4>: push {r11, r12, lr, pc} 0xc01361a4 <SyS_open+8>: sub r11, r12, #4 ------ inside virtual machine ------- # echo 'p:myprobe sys_open' > /sys/kernel/debug/tracing/kprobe_events # echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable ------ gdb attaches to QEMU ------- cpu_v7_do_idle () at /home/w00229757/kernel-hydrogen/arch/arm/mm/proc-v7.S:74 74 ret lr (gdb) x/3i sys_open 0xc013619c <SyS_open>: b 0xbf000000 <--- *optimized* 0xc01361a0 <SyS_open+4>: push {r11, r12, lr, pc} 0xc01361a4 <SyS_open+8>: sub r11, r12, #4 (gdb) c ------ inside virtual machine ------- # echo 0 > /sys/kernel/debug/kprobes/enabled ------ gdb attaches to QEMU ------- cpu_v7_do_idle () at /home/w00229757/kernel-hydrogen/arch/arm/mm/proc-v7.S:74 74 ret lr (gdb) x/3i sys_open 0xc013619c <SyS_open>: ; <UNDEFINED> instruction: 0xe7f001f8 <--- *BREAKPOINT* 0xc01361a0 <SyS_open+4>: push {r11, r12, lr, pc} 0xc01361a4 <SyS_open+8>: sub r11, r12, #4 (gdb) c ** x86_64 result ** ------ gdb attaches to QEMU ------- (gdb) x/10i sys_open 0xffffffff81184fe0 <SyS_open>: data32 data32 data32 xchg %ax,%ax 0xffffffff81184fe5 <SyS_open+5>: push %rbp 0xffffffff81184fe6 <SyS_open+6>: movzwl %dx,%ecx 0xffffffff81184fe9 <SyS_open+9>: mov %esi,%edx 0xffffffff81184feb <SyS_open+11>: mov %rsp,%rbp 0xffffffff81184fee <SyS_open+14>: mov %rdi,%rsi 0xffffffff81184ff1 <SyS_open+17>: or $0x80,%dh 0xffffffff81184ff4 <SyS_open+20>: mov $0xffffff9c,%edi 0xffffffff81184ff9 <SyS_open+25>: callq 0xffffffff81184da0 <do_sys_open> 0xffffffff81184ffe <SyS_open+30>: pop %rbp (gdb) c Continuing ------ inside virtual machine ------- # echo 'p:myprobe sys_open+20' > /sys/kernel/debug/tracing/kprobe_events # echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable ------ gdb attaches to QEMU ------- (gdb) x/10i sys_open 0xffffffff81184fe0 <SyS_open>: data32 data32 data32 xchg %ax,%ax 0xffffffff81184fe5 <SyS_open+5>: push %rbp 0xffffffff81184fe6 <SyS_open+6>: movzwl %dx,%ecx 0xffffffff81184fe9 <SyS_open+9>: mov %esi,%edx 0xffffffff81184feb <SyS_open+11>: mov %rsp,%rbp 0xffffffff81184fee <SyS_open+14>: mov %rdi,%rsi 0xffffffff81184ff1 <SyS_open+17>: or $0x80,%dh 0xffffffff81184ff4 <SyS_open+20>: jmpq 0xffffffffa0002000 <--- *optimized* 0xffffffff81184ff9 <SyS_open+25>: callq 0xffffffff81184da0 <do_sys_open> 0xffffffff81184ffe <SyS_open+30>: pop %rbp (gdb) c Continuing. ------ inside virtual machine ------- # echo 0 > /sys/kernel/debug/kprobes/enabled ------ gdb attaches to QEMU ------- (gdb) x/10i sys_open 0xffffffff81184fe0 <SyS_open>: data32 data32 data32 xchg %ax,%ax 0xffffffff81184fe5 <SyS_open+5>: push %rbp 0xffffffff81184fe6 <SyS_open+6>: movzwl %dx,%ecx 0xffffffff81184fe9 <SyS_open+9>: mov %esi,%edx 0xffffffff81184feb <SyS_open+11>: mov %rsp,%rbp 0xffffffff81184fee <SyS_open+14>: mov %rdi,%rsi 0xffffffff81184ff1 <SyS_open+17>: or $0x80,%dh 0xffffffff81184ff4 <SyS_open+20>: int3 <-- **BREAKPOINT** 0xffffffff81184ff5 <SyS_open+21>: pushfq 0xffffffff81184ff6 <SyS_open+22>: (bad) (gdb)
(2015/01/19 20:21), Wang Nan wrote: > On 2015/1/19 17:05, Masami Hiramatsu wrote: >> Hi Wang, >> >> I've found a problem on this patch, since kprobes calls unoptioize_kprobe >> with kprobes_all_disarmed=true when trying to disable all kprobes, this >> cause a serious problem. >> >> Moreover, I couldn't reproduce your reported bug on my 3.19-rc4 kernel. >> Could you test it again? >> > > I tested it again based on 3.19-rc5 and found that the problem still exists. > My testing is based on QEMU. > > First I tested my kprobeopt for ARM, then on x86_64. The test results are pasted > at the bottom of this mail. Commands after 'gdb attaches to QEMU' is my actions > on a gdb console attached to QEMU; commands after 'inside virtual machine' is > what I do in Linux run under QEMU. Thank you for the reporting. So, now I know what happened, the problem is "debugfs/kprobes/enabled doesn't work correctly on optimized kprobes". Please make update the patch description. I also reproduced the bug without gdb. Here is the log. ---- [root@localhost ~]# cd /sys/kernel/debug/tracing/ [root@localhost tracing]# echo p do_fork+5 > kprobe_events # setup new event [root@localhost tracing]# echo $$ > set_ftrace_pid # trace only this process [root@localhost tracing]# echo 1 > events/kprobes/p_do_fork_5/enable # enable it [root@localhost tracing]# cat trace # check the trace data # tracer: nop # # entries-in-buffer/entries-written: 1/1 #P:8 # # _-----=> irqs-off # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | bash-3883 [006] d... 279.799023: p_do_fork_5: (do_fork+0x5/0x360) # OK, now tracing [root@localhost tracing]# cat ../kprobes/list ffffffff810bc1c5 k do_fork+0x5 [OPTIMIZED] # and it is actually optimized [root@localhost tracing]# echo 0 > ../kprobes/enabled # disable *ALL* kprobes [root@localhost tracing]# echo > trace # clear events [root@localhost tracing]# cat trace # this should show empty buffer # tracer: nop # # entries-in-buffer/entries-written: 1/1 #P:8 # # _-----=> irqs-off # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | bash-3883 [006] d... 337.770785: p_do_fork_5: (do_fork+0x5/0x360) # But still tracing! [root@localhost tracing]# cat trace # Check again # tracer: nop # # entries-in-buffer/entries-written: 2/2 #P:8 # # _-----=> irqs-off # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | bash-3883 [006] d... 337.770785: p_do_fork_5: (do_fork+0x5/0x360) bash-3883 [006] d... 345.592178: p_do_fork_5: (do_fork+0x5/0x360) # We are tracing!! So, after global disabling kprobes, ALL kprobes event should be disabled, but not. OK, I think your first patch is better than the second one, but not enough. What we should do is use kprobes_all_disarmed for force option like below. unoptimize_kprobe(p, kprobes_all_disarmed); /* Try to unoptimize */ We also would better to check the flag in unregistering path for skipping unneeded disarming process when kprobes globally disarmed. Thank you,
On 2015/1/19 20:45, Masami Hiramatsu wrote: > (2015/01/19 20:21), Wang Nan wrote: >> On 2015/1/19 17:05, Masami Hiramatsu wrote: >>> Hi Wang, >>> >>> I've found a problem on this patch, since kprobes calls unoptioize_kprobe >>> with kprobes_all_disarmed=true when trying to disable all kprobes, this >>> cause a serious problem. >>> >>> Moreover, I couldn't reproduce your reported bug on my 3.19-rc4 kernel. >>> Could you test it again? >>> >> >> I tested it again based on 3.19-rc5 and found that the problem still exists. >> My testing is based on QEMU. >> >> First I tested my kprobeopt for ARM, then on x86_64. The test results are pasted >> at the bottom of this mail. Commands after 'gdb attaches to QEMU' is my actions >> on a gdb console attached to QEMU; commands after 'inside virtual machine' is >> what I do in Linux run under QEMU. > > Thank you for the reporting. > So, now I know what happened, the problem is "debugfs/kprobes/enabled doesn't work > correctly on optimized kprobes". Please make update the patch description. > > I also reproduced the bug without gdb. > Here is the log. > > ---- > [root@localhost ~]# cd /sys/kernel/debug/tracing/ > [root@localhost tracing]# echo p do_fork+5 > kprobe_events # setup new event > [root@localhost tracing]# echo $$ > set_ftrace_pid # trace only this process > [root@localhost tracing]# echo 1 > events/kprobes/p_do_fork_5/enable # enable it > [root@localhost tracing]# cat trace # check the trace data > # tracer: nop > # > # entries-in-buffer/entries-written: 1/1 #P:8 > # > # _-----=> irqs-off > # / _----=> need-resched > # | / _---=> hardirq/softirq > # || / _--=> preempt-depth > # ||| / delay > # TASK-PID CPU# |||| TIMESTAMP FUNCTION > # | | | |||| | | > bash-3883 [006] d... 279.799023: p_do_fork_5: (do_fork+0x5/0x360) # OK, now tracing > [root@localhost tracing]# cat ../kprobes/list > ffffffff810bc1c5 k do_fork+0x5 [OPTIMIZED] # and it is actually optimized > [root@localhost tracing]# echo 0 > ../kprobes/enabled # disable *ALL* kprobes > [root@localhost tracing]# echo > trace # clear events > [root@localhost tracing]# cat trace # this should show empty buffer > # tracer: nop > # > # entries-in-buffer/entries-written: 1/1 #P:8 > # > # _-----=> irqs-off > # / _----=> need-resched > # | / _---=> hardirq/softirq > # || / _--=> preempt-depth > # ||| / delay > # TASK-PID CPU# |||| TIMESTAMP FUNCTION > # | | | |||| | | > bash-3883 [006] d... 337.770785: p_do_fork_5: (do_fork+0x5/0x360) # But still tracing! > [root@localhost tracing]# cat trace # Check again > # tracer: nop > # > # entries-in-buffer/entries-written: 2/2 #P:8 > # > # _-----=> irqs-off > # / _----=> need-resched > # | / _---=> hardirq/softirq > # || / _--=> preempt-depth > # ||| / delay > # TASK-PID CPU# |||| TIMESTAMP FUNCTION > # | | | |||| | | > bash-3883 [006] d... 337.770785: p_do_fork_5: (do_fork+0x5/0x360) > bash-3883 [006] d... 345.592178: p_do_fork_5: (do_fork+0x5/0x360) # We are tracing!! > > So, after global disabling kprobes, ALL kprobes event should be disabled, but not. > > OK, I think your first patch is better than the second one, but not enough. > What we should do is use kprobes_all_disarmed for force option like below. > > unoptimize_kprobe(p, kprobes_all_disarmed); /* Try to unoptimize */ > > We also would better to check the flag in unregistering path for skipping unneeded > disarming process when kprobes globally disarmed. > > Thank you, > Thanks to your quick reply. I'll post an improved v1 patch tomorrow.
diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 9471710..f16936b 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -630,6 +630,9 @@ static void unoptimize_kprobe(struct kprobe *p, bool force) { struct optimized_kprobe *op; + if (kprobes_all_disarmed) + return; + if (!kprobe_aggrprobe(p) || kprobe_disarmed(p)) return; /* This is not an optprobe nor optimized */
Original code failed to disarm the probed instruction after echo 0 > /sys/kernel/debug/kprobes/enabled if OPTPROBE is enabled. This patch checks kprobes_all_disarmed in unoptimized_kprobe(). Signed-off-by: Wang Nan <wangnan0@huawei.com> --- kernel/kprobes.c | 3 +++ 1 file changed, 3 insertions(+)