| Message ID | 20210516040018.128105-2-madvenka@linux.microsoft.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | arm64: Stack trace reliability checks in the unwinder | expand |
On Sat, May 15, 2021 at 11:00:17PM -0500, madvenka@linux.microsoft.com wrote: > Other reliability checks will be added in the future. ... > + frame->reliable = true; > + All these checks are good checks but as you say there's more stuff that we need to add (like your patch 2 here) so I'm slightly nervous about actually setting the reliable flag here without even a comment. Equally well there's no actual use of this until arch_stack_walk_reliable() gets implemented so it's not like it's causing any problems and it gives us the structure to start building up the rest of the checks. The other thing I guess is the question of if we want to bother flagging frames as unrelaible when we return an error; I don't see an issue with it and it may turn out to make it easier to do something in the future so I'm fine with that.
On 5/21/21 11:11 AM, Mark Brown wrote: > On Sat, May 15, 2021 at 11:00:17PM -0500, madvenka@linux.microsoft.com wrote: > >> Other reliability checks will be added in the future. > > ... > >> + frame->reliable = true; >> + > > All these checks are good checks but as you say there's more stuff that > we need to add (like your patch 2 here) so I'm slightly nervous about > actually setting the reliable flag here without even a comment. Equally > well there's no actual use of this until arch_stack_walk_reliable() gets > implemented so it's not like it's causing any problems and it gives us > the structure to start building up the rest of the checks. > OK. So how about changing the field from a flag to an enum that says exactly what happened with the frame? enum { FRAME_NORMAL = 0, FRAME_UNALIGNED, FRAME_NOT_ACCESSIBLE, FRAME_RECURSION, FRAME_GRAPH_ERROR, FRAME_INVALID_TEXT_ADDRESS, FRAME_UNRELIABLE_FUNCTION, FRAME_NUM_STATUS, } frame_status; struct stackframe { ... enum frame_status status; }; unwind_frame() { frame->status = FRAME_NORMAL; Then, for each situation, change the status appropriately. } Eventually, arch_stack_walk_reliable() could just declare the stack trace as unreliable if status != FRAME_NORMAL. Also, the caller can get an exact idea of why the stack trace failed. Is that acceptable? > The other thing I guess is the question of if we want to bother flagging > frames as unrelaible when we return an error; I don't see an issue with > it and it may turn out to make it easier to do something in the future > so I'm fine with that Initially, I thought that there is no need to flag it for errors. But Josh had a comment that the stack trace is indeed unreliable on errors. Again, the word unreliable is the one causing the problem. The above enum-based solution addresses Josh's comment as well. Let me know if this is good. Thanks! Madhavan
On Fri, May 21, 2021 at 12:23:52PM -0500, Madhavan T. Venkataraman wrote: > On 5/21/21 11:11 AM, Mark Brown wrote: > > On Sat, May 15, 2021 at 11:00:17PM -0500, madvenka@linux.microsoft.com wrote: > >> + frame->reliable = true; > > All these checks are good checks but as you say there's more stuff that > > we need to add (like your patch 2 here) so I'm slightly nervous about > OK. So how about changing the field from a flag to an enum that says exactly > what happened with the frame? TBH I think the code is fine, or rather will be fine when it gets as far as actually being used - this was more a comment about when we flip this switch. > Also, the caller can get an exact idea of why the stack trace failed. I'm not sure anything other than someone debugging things will care enough to get the code out and then decode it so it seems like it'd be more trouble than it's worth, we're unlikely to be logging the code as standard. > > The other thing I guess is the question of if we want to bother flagging > > frames as unrelaible when we return an error; I don't see an issue with > > it and it may turn out to make it easier to do something in the future > > so I'm fine with that > Initially, I thought that there is no need to flag it for errors. But Josh > had a comment that the stack trace is indeed unreliable on errors. Again, the > word unreliable is the one causing the problem. My understanding there is that arch_stack_walk_reliable() should be returning an error if either the unwinder detected an error or if any frame in the stack is flagged as unreliable so from the point of view of users it's just looking at the error code, it's more that there's no need for arch_stack_walk_reliable() to consider the reliability information if an error has been detected and nothing else looks at the reliability information. Like I say we may come up with some use for the flag in error cases in future so I'm not opposed to keeping the accounting there.
On 5/21/21 12:42 PM, Mark Brown wrote: > On Fri, May 21, 2021 at 12:23:52PM -0500, Madhavan T. Venkataraman wrote: >> On 5/21/21 11:11 AM, Mark Brown wrote: >>> On Sat, May 15, 2021 at 11:00:17PM -0500, madvenka@linux.microsoft.com wrote: > >>>> + frame->reliable = true; > >>> All these checks are good checks but as you say there's more stuff that >>> we need to add (like your patch 2 here) so I'm slightly nervous about > >> OK. So how about changing the field from a flag to an enum that says exactly >> what happened with the frame? > > TBH I think the code is fine, or rather will be fine when it gets as far > as actually being used - this was more a comment about when we flip this > switch. > OK. >> Also, the caller can get an exact idea of why the stack trace failed. > > I'm not sure anything other than someone debugging things will care > enough to get the code out and then decode it so it seems like it'd be > more trouble than it's worth, we're unlikely to be logging the code as > standard. > OK. >>> The other thing I guess is the question of if we want to bother flagging >>> frames as unrelaible when we return an error; I don't see an issue with >>> it and it may turn out to make it easier to do something in the future >>> so I'm fine with that > >> Initially, I thought that there is no need to flag it for errors. But Josh >> had a comment that the stack trace is indeed unreliable on errors. Again, the >> word unreliable is the one causing the problem. > > My understanding there is that arch_stack_walk_reliable() should be > returning an error if either the unwinder detected an error or if any > frame in the stack is flagged as unreliable so from the point of view of > users it's just looking at the error code, it's more that there's no > need for arch_stack_walk_reliable() to consider the reliability > information if an error has been detected and nothing else looks at the > reliability information. > > Like I say we may come up with some use for the flag in error cases in > future so I'm not opposed to keeping the accounting there. > So, should I leave it the way it is now? Or should I not set reliable = false for errors? Which one do you prefer? Josh, Are you OK with not flagging reliable = false for errors in unwind_frame()? Madhavan
On Fri, May 21, 2021 at 12:47:13PM -0500, Madhavan T. Venkataraman wrote: > On 5/21/21 12:42 PM, Mark Brown wrote: > > Like I say we may come up with some use for the flag in error cases in > > future so I'm not opposed to keeping the accounting there. > So, should I leave it the way it is now? Or should I not set reliable = false > for errors? Which one do you prefer? > Josh, > Are you OK with not flagging reliable = false for errors in unwind_frame()? I think it's fine to leave it as it is.
On Fri, May 21, 2021 at 06:53:18PM +0100, Mark Brown wrote: > On Fri, May 21, 2021 at 12:47:13PM -0500, Madhavan T. Venkataraman wrote: > > On 5/21/21 12:42 PM, Mark Brown wrote: > > > > Like I say we may come up with some use for the flag in error cases in > > > future so I'm not opposed to keeping the accounting there. > > > So, should I leave it the way it is now? Or should I not set reliable = false > > for errors? Which one do you prefer? > > > Josh, > > > Are you OK with not flagging reliable = false for errors in unwind_frame()? > > I think it's fine to leave it as it is. Either way works for me, but if you remove those 'reliable = false' statements for stack corruption then, IIRC, the caller would still have some confusion between the end of stack error (-ENOENT) and the other errors (-EINVAL). So the caller would have to know that -ENOENT really means success. Which, to me, seems kind of flaky. BTW, not sure if you've seen what we do in x86, but we have a 'frame->error' which gets set for an error, and which is cumulative across frames. So non-fatal reliable-type errors don't necessarily have to stop the unwind. The end result is the same as your patch, but it seems less confusing to me because the 'error' is cumulative. But that might be personal preference and I'd defer to the arm64 folks.
On 5/21/21 1:48 PM, Josh Poimboeuf wrote: > On Fri, May 21, 2021 at 06:53:18PM +0100, Mark Brown wrote: >> On Fri, May 21, 2021 at 12:47:13PM -0500, Madhavan T. Venkataraman wrote: >>> On 5/21/21 12:42 PM, Mark Brown wrote: >> >>>> Like I say we may come up with some use for the flag in error cases in >>>> future so I'm not opposed to keeping the accounting there. >> >>> So, should I leave it the way it is now? Or should I not set reliable = false >>> for errors? Which one do you prefer? >> >>> Josh, >> >>> Are you OK with not flagging reliable = false for errors in unwind_frame()? >> >> I think it's fine to leave it as it is. > > Either way works for me, but if you remove those 'reliable = false' > statements for stack corruption then, IIRC, the caller would still have > some confusion between the end of stack error (-ENOENT) and the other > errors (-EINVAL). > I will leave it the way it is. That is, I will do reliable = false on errors like you suggested. > So the caller would have to know that -ENOENT really means success. > Which, to me, seems kind of flaky. > Actually, that is why -ENOENT was introduced - to indicate successful stack trace termination. A return value of 0 is for continuing with the stack trace. A non-zero value is for terminating the stack trace. So, either we return a positive value (say 1) to indicate successful termination. Or, we return -ENOENT to say no more stack frames left. I guess -ENOENT was chosen. > BTW, not sure if you've seen what we do in x86, but we have a > 'frame->error' which gets set for an error, and which is cumulative > across frames. So non-fatal reliable-type errors don't necessarily have > to stop the unwind. The end result is the same as your patch, but it > seems less confusing to me because the 'error' is cumulative. But that > might be personal preference and I'd defer to the arm64 folks. > OK. I will wait to see if any arm64 folks have an opinion on this. I am fine with any approach. Madhavan
On Fri, May 21, 2021 at 01:59:16PM -0500, Madhavan T. Venkataraman wrote: > > > On 5/21/21 1:48 PM, Josh Poimboeuf wrote: > > On Fri, May 21, 2021 at 06:53:18PM +0100, Mark Brown wrote: > >> On Fri, May 21, 2021 at 12:47:13PM -0500, Madhavan T. Venkataraman wrote: > >>> On 5/21/21 12:42 PM, Mark Brown wrote: > >> > >>>> Like I say we may come up with some use for the flag in error cases in > >>>> future so I'm not opposed to keeping the accounting there. > >> > >>> So, should I leave it the way it is now? Or should I not set reliable = false > >>> for errors? Which one do you prefer? > >> > >>> Josh, > >> > >>> Are you OK with not flagging reliable = false for errors in unwind_frame()? > >> > >> I think it's fine to leave it as it is. > > > > Either way works for me, but if you remove those 'reliable = false' > > statements for stack corruption then, IIRC, the caller would still have > > some confusion between the end of stack error (-ENOENT) and the other > > errors (-EINVAL). > > > > I will leave it the way it is. That is, I will do reliable = false on errors > like you suggested. > > > So the caller would have to know that -ENOENT really means success. > > Which, to me, seems kind of flaky. > > > > Actually, that is why -ENOENT was introduced - to indicate successful > stack trace termination. A return value of 0 is for continuing with > the stack trace. A non-zero value is for terminating the stack trace. > > So, either we return a positive value (say 1) to indicate successful > termination. Or, we return -ENOENT to say no more stack frames left. > I guess -ENOENT was chosen. I see. So it's a tri-state return value, and frame->reliable is intended to be a private interface not checked by the callers. That makes sense, and probably fine, it's just perhaps a bit nonstandard compared to most Linux interfaces.
On Fri, May 21, 2021 at 02:11:45PM -0500, Josh Poimboeuf wrote: > On Fri, May 21, 2021 at 01:59:16PM -0500, Madhavan T. Venkataraman wrote: > > > > > > On 5/21/21 1:48 PM, Josh Poimboeuf wrote: > > > On Fri, May 21, 2021 at 06:53:18PM +0100, Mark Brown wrote: > > >> On Fri, May 21, 2021 at 12:47:13PM -0500, Madhavan T. Venkataraman wrote: > > >>> On 5/21/21 12:42 PM, Mark Brown wrote: > > >> > > >>>> Like I say we may come up with some use for the flag in error cases in > > >>>> future so I'm not opposed to keeping the accounting there. > > >> > > >>> So, should I leave it the way it is now? Or should I not set reliable = false > > >>> for errors? Which one do you prefer? > > >> > > >>> Josh, > > >> > > >>> Are you OK with not flagging reliable = false for errors in unwind_frame()? > > >> > > >> I think it's fine to leave it as it is. > > > > > > Either way works for me, but if you remove those 'reliable = false' > > > statements for stack corruption then, IIRC, the caller would still have > > > some confusion between the end of stack error (-ENOENT) and the other > > > errors (-EINVAL). > > > > > > > I will leave it the way it is. That is, I will do reliable = false on errors > > like you suggested. > > > > > So the caller would have to know that -ENOENT really means success. > > > Which, to me, seems kind of flaky. > > > > > > > Actually, that is why -ENOENT was introduced - to indicate successful > > stack trace termination. A return value of 0 is for continuing with > > the stack trace. A non-zero value is for terminating the stack trace. > > > > So, either we return a positive value (say 1) to indicate successful > > termination. Or, we return -ENOENT to say no more stack frames left. > > I guess -ENOENT was chosen. > > I see. So it's a tri-state return value, and frame->reliable is > intended to be a private interface not checked by the callers. Or is frame->reliable supposed to be checked after all? Looking at the code again, I'm not sure. Either way it would be good to document the interface more clearly in a comment above the function.
On 5/21/21 2:16 PM, Josh Poimboeuf wrote: > On Fri, May 21, 2021 at 02:11:45PM -0500, Josh Poimboeuf wrote: >> On Fri, May 21, 2021 at 01:59:16PM -0500, Madhavan T. Venkataraman wrote: >>> >>> >>> On 5/21/21 1:48 PM, Josh Poimboeuf wrote: >>>> On Fri, May 21, 2021 at 06:53:18PM +0100, Mark Brown wrote: >>>>> On Fri, May 21, 2021 at 12:47:13PM -0500, Madhavan T. Venkataraman wrote: >>>>>> On 5/21/21 12:42 PM, Mark Brown wrote: >>>>> >>>>>>> Like I say we may come up with some use for the flag in error cases in >>>>>>> future so I'm not opposed to keeping the accounting there. >>>>> >>>>>> So, should I leave it the way it is now? Or should I not set reliable = false >>>>>> for errors? Which one do you prefer? >>>>> >>>>>> Josh, >>>>> >>>>>> Are you OK with not flagging reliable = false for errors in unwind_frame()? >>>>> >>>>> I think it's fine to leave it as it is. >>>> >>>> Either way works for me, but if you remove those 'reliable = false' >>>> statements for stack corruption then, IIRC, the caller would still have >>>> some confusion between the end of stack error (-ENOENT) and the other >>>> errors (-EINVAL). >>>> >>> >>> I will leave it the way it is. That is, I will do reliable = false on errors >>> like you suggested. >>> >>>> So the caller would have to know that -ENOENT really means success. >>>> Which, to me, seems kind of flaky. >>>> >>> >>> Actually, that is why -ENOENT was introduced - to indicate successful >>> stack trace termination. A return value of 0 is for continuing with >>> the stack trace. A non-zero value is for terminating the stack trace. >>> >>> So, either we return a positive value (say 1) to indicate successful >>> termination. Or, we return -ENOENT to say no more stack frames left. >>> I guess -ENOENT was chosen. >> >> I see. So it's a tri-state return value, and frame->reliable is >> intended to be a private interface not checked by the callers. > > Or is frame->reliable supposed to be checked after all? Looking at the > code again, I'm not sure. > > Either way it would be good to document the interface more clearly in a > comment above the function. > So, arch_stack_walk_reliable() would do this: start_backtrace(frame); while (...) { if (!frame->reliable) return error; consume_entry(...); ret = unwind_frame(...); if (ret) break; } if (ret == -ENOENT) return success; return error; Something like that. I will add a comment about all of this in the unwinder. Thanks! Madhavan
On Fri, May 21, 2021 at 02:41:56PM -0500, Madhavan T. Venkataraman wrote: > > Or is frame->reliable supposed to be checked after all? Looking at the > > code again, I'm not sure. > > > > Either way it would be good to document the interface more clearly in a > > comment above the function. > > > > So, arch_stack_walk_reliable() would do this: > > start_backtrace(frame); > > while (...) { > if (!frame->reliable) > return error; > > consume_entry(...); > > ret = unwind_frame(...); > > if (ret) > break; > } > > if (ret == -ENOENT) > return success; > return error; > > > Something like that. I see. So basically there are six possible combinations of return states: 1) No error frame->reliable 2) No error !frame->reliable 3) -ENOENT frame->reliable 5) -ENOENT !frame->reliable (doesn't happen in practice) 4) Other error frame->reliable (doesn't happen in practice) 6) Other error !frame->reliable On x86 we have fewer combinations: 1) No error state->error 2) No error !state->error 3) Error state->error 4) Error !state->error (doesn't happen in practice) I think the x86 interface seems more robust, because it's more narrow and has fewer edge cases. Also it doesn't have to distinguish between error enums, which can get hairy if a downstream callee happens to return -ENOENT for a different reason.
On 5/21/21 12:53 PM, Mark Brown wrote: > On Fri, May 21, 2021 at 12:47:13PM -0500, Madhavan T. Venkataraman wrote: >> On 5/21/21 12:42 PM, Mark Brown wrote: > >>> Like I say we may come up with some use for the flag in error cases in >>> future so I'm not opposed to keeping the accounting there. > >> So, should I leave it the way it is now? Or should I not set reliable = false >> for errors? Which one do you prefer? > >> Josh, > >> Are you OK with not flagging reliable = false for errors in unwind_frame()? > > I think it's fine to leave it as it is. > OK. I will address the comments so far and send out v5. Thanks. Madhavan
diff --git a/arch/arm64/include/asm/stacktrace.h b/arch/arm64/include/asm/stacktrace.h index eb29b1fe8255..f1eab6b029f7 100644 --- a/arch/arm64/include/asm/stacktrace.h +++ b/arch/arm64/include/asm/stacktrace.h @@ -49,6 +49,8 @@ struct stack_info { * * @graph: When FUNCTION_GRAPH_TRACER is selected, holds the index of a * replacement lr value in the ftrace graph stack. + * + * @reliable: Is this stack frame reliable? */ struct stackframe { unsigned long fp; @@ -59,6 +61,7 @@ struct stackframe { #ifdef CONFIG_FUNCTION_GRAPH_TRACER int graph; #endif + bool reliable; }; extern int unwind_frame(struct task_struct *tsk, struct stackframe *frame); @@ -169,6 +172,7 @@ static inline void start_backtrace(struct stackframe *frame, bitmap_zero(frame->stacks_done, __NR_STACK_TYPES); frame->prev_fp = 0; frame->prev_type = STACK_TYPE_UNKNOWN; + frame->reliable = true; } #endif /* __ASM_STACKTRACE_H */ diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c index d55bdfb7789c..d38232cab3ee 100644 --- a/arch/arm64/kernel/stacktrace.c +++ b/arch/arm64/kernel/stacktrace.c @@ -44,21 +44,29 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame) unsigned long fp = frame->fp; struct stack_info info; + frame->reliable = true; + /* Terminal record; nothing to unwind */ if (!fp) return -ENOENT; - if (fp & 0xf) + if (fp & 0xf) { + frame->reliable = false; return -EINVAL; + } if (!tsk) tsk = current; - if (!on_accessible_stack(tsk, fp, &info)) + if (!on_accessible_stack(tsk, fp, &info)) { + frame->reliable = false; return -EINVAL; + } - if (test_bit(info.type, frame->stacks_done)) + if (test_bit(info.type, frame->stacks_done)) { + frame->reliable = false; return -EINVAL; + } /* * As stacks grow downward, any valid record on the same stack must be @@ -74,8 +82,10 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame) * stack. */ if (info.type == frame->prev_type) { - if (fp <= frame->prev_fp) + if (fp <= frame->prev_fp) { + frame->reliable = false; return -EINVAL; + } } else { set_bit(frame->prev_type, frame->stacks_done); } @@ -100,14 +110,29 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame) * So replace it to an original value. */ ret_stack = ftrace_graph_get_ret_stack(tsk, frame->graph++); - if (WARN_ON_ONCE(!ret_stack)) + if (WARN_ON_ONCE(!ret_stack)) { + frame->reliable = false; return -EINVAL; + } frame->pc = ret_stack->ret; } #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ frame->pc = ptrauth_strip_insn_pac(frame->pc); + /* + * Check the return PC for conditions that make unwinding unreliable. + * In each case, mark the stack trace as such. + */ + + /* + * Make sure that the return address is a proper kernel text address. + * A NULL or invalid return address probably means there's some + * generated code which __kernel_text_address() doesn't know about. + */ + if (!__kernel_text_address(frame->pc)) + frame->reliable = false; + return 0; } NOKPROBE_SYMBOL(unwind_frame);