| Message ID | 1455053182-31404-1-git-send-email-yang.shi@linaro.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Tue, Feb 09, 2016 at 01:26:22PM -0800, Yang Shi wrote: > dump_backtrace may be called in kthread context, which is not bound to a single > cpu, i.e. khungtaskd, then calling smp_processor_id may trigger the below bug > report: If we're preemptible here, it means that our irq_stack_ptr is potentially bogus. Whilst this isn't an issue for kthreads, it does feel like we could make this slightly more robust in the face of potential frame corruption. Maybe just zero the IRQ stack pointer if we're in preemptible context? Will > BUG: using smp_processor_id() in preemptible [00000000] code: khungtaskd/71 > caller is debug_smp_processor_id+0x1c/0x28 > CPU: 3 PID: 71 Comm: khungtaskd Not tainted 4.5.0-rc1 #144 > Hardware name: Freescale Layerscape 2085a RDB Board (DT) > Call trace: > [<ffffffc00008d120>] dump_backtrace+0x0/0x290 > [<ffffffc00008d3d4>] show_stack+0x24/0x30 > [<ffffffc0006bfd1c>] dump_stack+0x8c/0xd8 > [<ffffffc0006fe56c>] check_preemption_disabled+0x184/0x188 > [<ffffffc0006fe58c>] debug_smp_processor_id+0x1c/0x28 > [<ffffffc00008d1a4>] dump_backtrace+0x84/0x290 > [<ffffffc00008d3d4>] show_stack+0x24/0x30 > [<ffffffc00012aa5c>] sched_show_task+0x16c/0x280 > [<ffffffc0001f3df8>] watchdog+0x560/0x708 > [<ffffffc000111bb8>] kthread+0x1b0/0x1d0 > [<ffffffc0000864d0>] ret_from_fork+0x10/0x40 > > Replace it to raw version to prevent from the race condition. > > Signed-off-by: Yang Shi <yang.shi@linaro.org> > --- > arch/arm64/kernel/traps.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c > index cbedd72..9abe236 100644 > --- a/arch/arm64/kernel/traps.c > +++ b/arch/arm64/kernel/traps.c > @@ -146,7 +146,7 @@ static void dump_instr(const char *lvl, struct pt_regs *regs) > static void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk) > { > struct stackframe frame; > - unsigned long irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id()); > + unsigned long irq_stack_ptr = IRQ_STACK_PTR(raw_smp_processor_id()); > int skip; > > pr_debug("%s(regs = %p tsk = %p)\n", __func__, regs, tsk); > -- > 2.0.2 >
On 10/02/16 10:29, Will Deacon wrote: > On Tue, Feb 09, 2016 at 01:26:22PM -0800, Yang Shi wrote: >> dump_backtrace may be called in kthread context, which is not bound to a single >> cpu, i.e. khungtaskd, then calling smp_processor_id may trigger the below bug >> report: > > If we're preemptible here, it means that our irq_stack_ptr is potentially > bogus. Whilst this isn't an issue for kthreads, it does feel like we > could make this slightly more robust in the face of potential frame > corruption. Maybe just zero the IRQ stack pointer if we're in preemptible > context? Switching between stacks is only valid if we are tracing ourselves while on the irq_stack, we should probably prevent it for other tasks too. Something like (untested): --------------------- if (tsk == current && in_atomic()) irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id()); else irq_stack_ptr = 0; --------------------- This would work when we trace ourselves while on the irq_stack, but break* tracing a running task on a remote cpu (khungtaskd doesn't do this). The same fix would apply to unwind_frame(), we have 'tsk' in both functions. Thoughts? James * If this were to ever happen, we would fail to switch to the original stack if tracing a remote irq stack, and print the wrong exception args. I'm not aware of any users of this, (panic() sends an IPI).
On Wed, Feb 10, 2016 at 11:52:31AM +0000, James Morse wrote: > On 10/02/16 10:29, Will Deacon wrote: > > On Tue, Feb 09, 2016 at 01:26:22PM -0800, Yang Shi wrote: > >> dump_backtrace may be called in kthread context, which is not bound to a single > >> cpu, i.e. khungtaskd, then calling smp_processor_id may trigger the below bug > >> report: > > > > If we're preemptible here, it means that our irq_stack_ptr is potentially > > bogus. Whilst this isn't an issue for kthreads, it does feel like we > > could make this slightly more robust in the face of potential frame > > corruption. Maybe just zero the IRQ stack pointer if we're in preemptible > > context? > > Switching between stacks is only valid if we are tracing ourselves while on the > irq_stack, we should probably prevent it for other tasks too. > > Something like (untested): > --------------------- > if (tsk == current && in_atomic()) > irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id()); > else > irq_stack_ptr = 0; > --------------------- > > This would work when we trace ourselves while on the irq_stack, but break* > tracing a running task on a remote cpu (khungtaskd doesn't do this). > > The same fix would apply to unwind_frame(), we have 'tsk' in both functions. > > Thoughts? in_atomic is a misnomer: https://lwn.net/Articles/274695/ ;) So we might be better off zeroing the pointer if tsk != current || preemptible(). But yeah, I think we're in general agreement about this. Will
On 2/10/2016 4:10 AM, Will Deacon wrote: > On Wed, Feb 10, 2016 at 11:52:31AM +0000, James Morse wrote: >> On 10/02/16 10:29, Will Deacon wrote: >>> On Tue, Feb 09, 2016 at 01:26:22PM -0800, Yang Shi wrote: >>>> dump_backtrace may be called in kthread context, which is not bound to a single >>>> cpu, i.e. khungtaskd, then calling smp_processor_id may trigger the below bug >>>> report: >>> >>> If we're preemptible here, it means that our irq_stack_ptr is potentially >>> bogus. Whilst this isn't an issue for kthreads, it does feel like we >>> could make this slightly more robust in the face of potential frame >>> corruption. Maybe just zero the IRQ stack pointer if we're in preemptible >>> context? >> >> Switching between stacks is only valid if we are tracing ourselves while on the >> irq_stack, we should probably prevent it for other tasks too. >> >> Something like (untested): >> --------------------- >> if (tsk == current && in_atomic()) >> irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id()); One follow up question, is it possible to have both tsk != current and on_irq_stack is true at the same time? If it is possible, this may be a problem in unwind_frame called by profile_pc which has tsk being NULL. Thanks, Yang >> else >> irq_stack_ptr = 0; >> --------------------- >> >> This would work when we trace ourselves while on the irq_stack, but break* >> tracing a running task on a remote cpu (khungtaskd doesn't do this). >> >> The same fix would apply to unwind_frame(), we have 'tsk' in both functions. >> >> Thoughts? > > in_atomic is a misnomer: > > https://lwn.net/Articles/274695/ > > ;) > > So we might be better off zeroing the pointer if tsk != current || > preemptible(). But yeah, I think we're in general agreement about this. > > Will >
Hi! On 10/02/16 18:12, Shi, Yang wrote: > On 2/10/2016 4:10 AM, Will Deacon wrote: >> On Wed, Feb 10, 2016 at 11:52:31AM +0000, James Morse wrote: >>> On 10/02/16 10:29, Will Deacon wrote: >>>> On Tue, Feb 09, 2016 at 01:26:22PM -0800, Yang Shi wrote: >>>>> dump_backtrace may be called in kthread context, which is not bound to a >>>>> single >>>>> cpu, i.e. khungtaskd, then calling smp_processor_id may trigger the below bug >>>>> report: >>>> >>>> If we're preemptible here, it means that our irq_stack_ptr is potentially >>>> bogus. Whilst this isn't an issue for kthreads, it does feel like we >>>> could make this slightly more robust in the face of potential frame >>>> corruption. Maybe just zero the IRQ stack pointer if we're in preemptible >>>> context? >>> >>> Switching between stacks is only valid if we are tracing ourselves while on the >>> irq_stack, we should probably prevent it for other tasks too. >>> >>> Something like (untested): >>> --------------------- >>> if (tsk == current && in_atomic()) >>> irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id()); > > One follow up question, is it possible to have both tsk != current and > on_irq_stack is true at the same time? No. If you are tracing an irq stack, it must be your own stack. If this weren't the case, it would be the stack of a running task on a remote CPU, and you would be racing with the remote CPU changing the values you are reading. Fortunately nothing tries to do this. (The third case would be tracing a sleeping irq stack - this doesn't happen either, as we switch back to the original stack before calling schedule()). > If it is possible, this may be a problem > in unwind_frame called by profile_pc which has tsk being NULL. Ah, well spotted. I guess there should also be a != NULL comparison thrown into the mix. I don't think it will be a problem for profile_pc() as it should always find a !in_lock_functions() frame before it needs to switch stack, (which we are preventing it from doing). If this ever did happen, it will return 0. Thanks, James
On 2/11/2016 2:41 AM, James Morse wrote: > Hi! > > On 10/02/16 18:12, Shi, Yang wrote: >> On 2/10/2016 4:10 AM, Will Deacon wrote: >>> On Wed, Feb 10, 2016 at 11:52:31AM +0000, James Morse wrote: >>>> On 10/02/16 10:29, Will Deacon wrote: >>>>> On Tue, Feb 09, 2016 at 01:26:22PM -0800, Yang Shi wrote: >>>>>> dump_backtrace may be called in kthread context, which is not bound to a >>>>>> single >>>>>> cpu, i.e. khungtaskd, then calling smp_processor_id may trigger the below bug >>>>>> report: >>>>> >>>>> If we're preemptible here, it means that our irq_stack_ptr is potentially >>>>> bogus. Whilst this isn't an issue for kthreads, it does feel like we >>>>> could make this slightly more robust in the face of potential frame >>>>> corruption. Maybe just zero the IRQ stack pointer if we're in preemptible >>>>> context? >>>> >>>> Switching between stacks is only valid if we are tracing ourselves while on the >>>> irq_stack, we should probably prevent it for other tasks too. >>>> >>>> Something like (untested): >>>> --------------------- >>>> if (tsk == current && in_atomic()) >>>> irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id()); >> >> One follow up question, is it possible to have both tsk != current and >> on_irq_stack is true at the same time? > > No. If you are tracing an irq stack, it must be your own stack. > > If this weren't the case, it would be the stack of a running task on a remote > CPU, and you would be racing with the remote CPU changing the values you are > reading. Fortunately nothing tries to do this. > > (The third case would be tracing a sleeping irq stack - this doesn't happen > either, as we switch back to the original stack before calling schedule()). > > >> If it is possible, this may be a problem >> in unwind_frame called by profile_pc which has tsk being NULL. > > Ah, well spotted. I guess there should also be a != NULL comparison thrown into > the mix. I don't think it will be a problem for profile_pc() as it should always > find a !in_lock_functions() frame before it needs to switch stack, (which we are > preventing it from doing). If this ever did happen, it will return 0. Thanks for the elaboration. I changed the logic a little bit to: if (tsk == current && !preemptible()) irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id()); else irq_stack_ptr = 0; In this way, the NULL pointer will be covered by "else" too. v2 patch will be sent out soon once I'm done some smoke testing. Yang > > > Thanks, > > James >
diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index cbedd72..9abe236 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -146,7 +146,7 @@ static void dump_instr(const char *lvl, struct pt_regs *regs) static void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk) { struct stackframe frame; - unsigned long irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id()); + unsigned long irq_stack_ptr = IRQ_STACK_PTR(raw_smp_processor_id()); int skip; pr_debug("%s(regs = %p tsk = %p)\n", __func__, regs, tsk);
dump_backtrace may be called in kthread context, which is not bound to a single cpu, i.e. khungtaskd, then calling smp_processor_id may trigger the below bug report: BUG: using smp_processor_id() in preemptible [00000000] code: khungtaskd/71 caller is debug_smp_processor_id+0x1c/0x28 CPU: 3 PID: 71 Comm: khungtaskd Not tainted 4.5.0-rc1 #144 Hardware name: Freescale Layerscape 2085a RDB Board (DT) Call trace: [<ffffffc00008d120>] dump_backtrace+0x0/0x290 [<ffffffc00008d3d4>] show_stack+0x24/0x30 [<ffffffc0006bfd1c>] dump_stack+0x8c/0xd8 [<ffffffc0006fe56c>] check_preemption_disabled+0x184/0x188 [<ffffffc0006fe58c>] debug_smp_processor_id+0x1c/0x28 [<ffffffc00008d1a4>] dump_backtrace+0x84/0x290 [<ffffffc00008d3d4>] show_stack+0x24/0x30 [<ffffffc00012aa5c>] sched_show_task+0x16c/0x280 [<ffffffc0001f3df8>] watchdog+0x560/0x708 [<ffffffc000111bb8>] kthread+0x1b0/0x1d0 [<ffffffc0000864d0>] ret_from_fork+0x10/0x40 Replace it to raw version to prevent from the race condition. Signed-off-by: Yang Shi <yang.shi@linaro.org> --- arch/arm64/kernel/traps.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)