[v2,1/4] stacktrace: add interface based on shadow call stack

Commit Message

[e-mail] March 23, 2022, 3:32 p.m. UTC

From: Andrey Konovalov <[e-mail]>

Add a new interface stack_trace_save_shadow() for collecting stack traces
by copying frames from the Shadow Call Stack.

Collecting stack traces this way is significantly faster: boot time
of a defconfig build with KASAN enabled gets descreased by ~30%.

The few patches following this one add an implementation of
stack_trace_save_shadow() for arm64.

The implementation of the added interface is not meant to use
stack_trace_consume_fn to avoid making a function call for each
collected frame to further improve performance.

Signed-off-by: Andrey Konovalov <[e-mail]>
---
 arch/Kconfig               |  6 ++++++
 include/linux/stacktrace.h | 15 +++++++++++++++
 kernel/stacktrace.c        | 21 +++++++++++++++++++++
 3 files changed, 42 insertions(+)

Comments

Andrew Morton March 25, 2022, 8:46 p.m. UTC | #1

On Wed, 23 Mar 2022 16:32:52 +0100 [e-mail] wrote:

> From: Andrey Konovalov <[e-mail]>
> 
> Add a new interface stack_trace_save_shadow() for collecting stack traces
> by copying frames from the Shadow Call Stack.
> 
> Collecting stack traces this way is significantly faster: boot time
> of a defconfig build with KASAN enabled gets descreased by ~30%.
> 
> The few patches following this one add an implementation of
> stack_trace_save_shadow() for arm64.
> 
> The implementation of the added interface is not meant to use
> stack_trace_consume_fn to avoid making a function call for each
> collected frame to further improve performance.
> 
> ...
>
> @@ -108,4 +111,16 @@ static inline int stack_trace_save_tsk_reliable(struct task_struct *tsk,
>  }
>  #endif
>  
> +#if defined(CONFIG_STACKTRACE) && defined(CONFIG_HAVE_SHADOW_STACKTRACE)
> +int stack_trace_save_shadow(unsigned long *store, unsigned int size,
> +			    unsigned int skipnr);
> +#else
> +static inline int stack_trace_save_shadow(unsigned long *store,
> +					  unsigned int size,
> +					  unsigned int skipnr)
> +{
> +	return -ENOSYS;
> +}
> +#endif

checkpatch sayeth "WARNING: ENOSYS means 'invalid syscall nr' and
nothing else".

checkpatch also picked up a typo in a changelog.  Useful thing to run,
is checkpatch.

Andrey Konovalov March 29, 2022, 6:36 p.m. UTC | #2

On Fri, Mar 25, 2022 at 9:46 PM Andrew Morton <[e-mail]> wrote:
>
> On Wed, 23 Mar 2022 16:32:52 +0100 [e-mail] wrote:
>
> > From: Andrey Konovalov <[e-mail]>
> >
> > Add a new interface stack_trace_save_shadow() for collecting stack traces
> > by copying frames from the Shadow Call Stack.
> >
> > Collecting stack traces this way is significantly faster: boot time
> > of a defconfig build with KASAN enabled gets descreased by ~30%.
> >
> > The few patches following this one add an implementation of
> > stack_trace_save_shadow() for arm64.
> >
> > The implementation of the added interface is not meant to use
> > stack_trace_consume_fn to avoid making a function call for each
> > collected frame to further improve performance.
> >
> > ...
> >
> > @@ -108,4 +111,16 @@ static inline int stack_trace_save_tsk_reliable(struct task_struct *tsk,
> >  }
> >  #endif
> >
> > +#if defined(CONFIG_STACKTRACE) && defined(CONFIG_HAVE_SHADOW_STACKTRACE)
> > +int stack_trace_save_shadow(unsigned long *store, unsigned int size,
> > +                         unsigned int skipnr);
> > +#else
> > +static inline int stack_trace_save_shadow(unsigned long *store,
> > +                                       unsigned int size,
> > +                                       unsigned int skipnr)
> > +{
> > +     return -ENOSYS;
> > +}
> > +#endif
>
> checkpatch sayeth "WARNING: ENOSYS means 'invalid syscall nr' and
> nothing else".

This is done deliberately to mimic stack_trace_save_tsk_reliable().
I'll mention this in the changelog.

> checkpatch also picked up a typo in a changelog.  Useful thing to run,
> is checkpatch.

My bad, I ran it on diff instead of the patch. Will fix in v3. Thanks!

Mark Rutland March 31, 2022, 9:19 a.m. UTC | #3

On Wed, Mar 23, 2022 at 04:32:52PM +0100, [e-mail] wrote:
> From: Andrey Konovalov <[e-mail]>
> 
> Add a new interface stack_trace_save_shadow() for collecting stack traces
> by copying frames from the Shadow Call Stack.
> 
> Collecting stack traces this way is significantly faster: boot time
> of a defconfig build with KASAN enabled gets descreased by ~30%.

Hmm... just to check, do ou know if that's just because of hte linear copy, or
because we're skipping other work we have to do in the regular stacktrace?

> The few patches following this one add an implementation of
> stack_trace_save_shadow() for arm64.
> 
> The implementation of the added interface is not meant to use
> stack_trace_consume_fn to avoid making a function call for each
> collected frame to further improve performance.

... because we could easily provide an inline-optimized stack copy *without*
having to write a distinct unwinder, and I'd *really* like to avoid having a
bunch of distinct unwinders for arm64, as it really hinders maintenance. We're
working on fixing/improving the arm64 unwinder for things like
RELIABLE_STACKTRACE, and I know that some of that work is non-trivial to make
work with an SCS-based unwind rather than an FP-based unwind, and/or will
undermine the saving anyway.

> 
> Signed-off-by: Andrey Konovalov <[e-mail]>
> ---
>  arch/Kconfig               |  6 ++++++
>  include/linux/stacktrace.h | 15 +++++++++++++++
>  kernel/stacktrace.c        | 21 +++++++++++++++++++++
>  3 files changed, 42 insertions(+)
> 
> diff --git a/arch/Kconfig b/arch/Kconfig
> index e12a4268c01d..207c1679c53a 100644
> --- a/arch/Kconfig
> +++ b/arch/Kconfig
> @@ -1041,6 +1041,12 @@ config HAVE_RELIABLE_STACKTRACE
>  	  arch_stack_walk_reliable() function which only returns a stack trace
>  	  if it can guarantee the trace is reliable.
>  
> +config HAVE_SHADOW_STACKTRACE
> +	bool
> +	help
> +	  If this is set, the architecture provides the arch_stack_walk_shadow()
> +	  function, which collects the stack trace from the shadow call stack.
> +
>  config HAVE_ARCH_HASH
>  	bool
>  	default n
> diff --git a/include/linux/stacktrace.h b/include/linux/stacktrace.h
> index 97455880ac41..b74d1e42e157 100644
> --- a/include/linux/stacktrace.h
> +++ b/include/linux/stacktrace.h
> @@ -60,6 +60,9 @@ int arch_stack_walk_reliable(stack_trace_consume_fn consume_entry, void *cookie,
>  
>  void arch_stack_walk_user(stack_trace_consume_fn consume_entry, void *cookie,
>  			  const struct pt_regs *regs);
> +
> +int arch_stack_walk_shadow(unsigned long *store, unsigned int size,
> +			   unsigned int skipnr);
>  #endif /* CONFIG_ARCH_STACKWALK */
>  
>  #ifdef CONFIG_STACKTRACE
> @@ -108,4 +111,16 @@ static inline int stack_trace_save_tsk_reliable(struct task_struct *tsk,
>  }
>  #endif
>  
> +#if defined(CONFIG_STACKTRACE) && defined(CONFIG_HAVE_SHADOW_STACKTRACE)
> +int stack_trace_save_shadow(unsigned long *store, unsigned int size,
> +			    unsigned int skipnr);
> +#else
> +static inline int stack_trace_save_shadow(unsigned long *store,
> +					  unsigned int size,
> +					  unsigned int skipnr)
> +{
> +	return -ENOSYS;
> +}
> +#endif
> +
>  #endif /* __LINUX_STACKTRACE_H */
> diff --git a/kernel/stacktrace.c b/kernel/stacktrace.c
> index 9ed5ce989415..fe305861fd55 100644
> --- a/kernel/stacktrace.c
> +++ b/kernel/stacktrace.c
> @@ -237,6 +237,27 @@ unsigned int stack_trace_save_user(unsigned long *store, unsigned int size)
>  }
>  #endif
>  
> +#ifdef CONFIG_HAVE_SHADOW_STACKTRACE
> +/**
> + * stack_trace_save_shadow - Save a stack trace based on shadow call stack
> + * @store:	Pointer to the storage array
> + * @size:	Size of the storage array
> + * @skipnr:	Number of entries to skip at the start of the stack trace
> + *
> + * Return: Number of trace entries stored.
> + */
> +int stack_trace_save_shadow(unsigned long *store, unsigned int size,
> +			    unsigned int skipnr)
> +{
> +	/*
> +	 * Do not use stack_trace_consume_fn to avoid making a function
> +	 * call for each collected frame to improve performance.
> +	 * Skip + 1 frame to skip stack_trace_save_shadow.
> +	 */
> +	return arch_stack_walk_shadow(store, size, skipnr + 1);
> +}
> +#endif

If we really need this, can we make it an __always_inline in a header so that
we can avoid the skip? Generally the skipping is problematic due to
inlining/outlining and LTO, and I'd like to avoid adding more of it
unnecessarily.

Thanks,
Mark.

> +
>  #else /* CONFIG_ARCH_STACKWALK */
>  
>  /*
> -- 
> 2.25.1
>

Andrey Konovalov April 5, 2022, 3:37 p.m. UTC | #4

On Thu, Mar 31, 2022 at 11:19 AM Mark Rutland <[e-mail]> wrote:
>
> > Collecting stack traces this way is significantly faster: boot time
> > of a defconfig build with KASAN enabled gets descreased by ~30%.
>
> Hmm... just to check, do ou know if that's just because of hte linear copy, or
> because we're skipping other work we have to do in the regular stacktrace?

No, I haven't looked into this.

> > The implementation of the added interface is not meant to use
> > stack_trace_consume_fn to avoid making a function call for each
> > collected frame to further improve performance.
>
> ... because we could easily provide an inline-optimized stack copy *without*
> having to write a distinct unwinder, and I'd *really* like to avoid having a
> bunch of distinct unwinders for arm64, as it really hinders maintenance. We're
> working on fixing/improving the arm64 unwinder for things like
> RELIABLE_STACKTRACE, and I know that some of that work is non-trivial to make
> work with an SCS-based unwind rather than an FP-based unwind, and/or will
> undermine the saving anyway.

Responded on the cover letter wrt this.

> > +int stack_trace_save_shadow(unsigned long *store, unsigned int size,
> > +                         unsigned int skipnr)
> > +{
> > +     /*
> > +      * Do not use stack_trace_consume_fn to avoid making a function
> > +      * call for each collected frame to improve performance.
> > +      * Skip + 1 frame to skip stack_trace_save_shadow.
> > +      */
> > +     return arch_stack_walk_shadow(store, size, skipnr + 1);
> > +}
> > +#endif
>
> If we really need this, can we make it an __always_inline in a header so that
> we can avoid the skip? Generally the skipping is problematic due to
> inlining/outlining and LTO, and I'd like to avoid adding more of it
> unnecessarily.

Yes, I think this should work.

However, if we keep the implementation in mm/kasan, this integration
will not be required.

Thanks!

Patch

diff --git a/arch/Kconfig b/arch/Kconfig
index e12a4268c01d..207c1679c53a 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -1041,6 +1041,12 @@  config HAVE_RELIABLE_STACKTRACE
 	  arch_stack_walk_reliable() function which only returns a stack trace
 	  if it can guarantee the trace is reliable.
 
+config HAVE_SHADOW_STACKTRACE
+	bool
+	help
+	  If this is set, the architecture provides the arch_stack_walk_shadow()
+	  function, which collects the stack trace from the shadow call stack.
+
 config HAVE_ARCH_HASH
 	bool
 	default n
diff --git a/include/linux/stacktrace.h b/include/linux/stacktrace.h
index 97455880ac41..b74d1e42e157 100644
--- a/include/linux/stacktrace.h
+++ b/include/linux/stacktrace.h
@@ -60,6 +60,9 @@  int arch_stack_walk_reliable(stack_trace_consume_fn consume_entry, void *cookie,
 
 void arch_stack_walk_user(stack_trace_consume_fn consume_entry, void *cookie,
 			  const struct pt_regs *regs);
+
+int arch_stack_walk_shadow(unsigned long *store, unsigned int size,
+			   unsigned int skipnr);
 #endif /* CONFIG_ARCH_STACKWALK */
 
 #ifdef CONFIG_STACKTRACE
@@ -108,4 +111,16 @@  static inline int stack_trace_save_tsk_reliable(struct task_struct *tsk,
 }
 #endif
 
+#if defined(CONFIG_STACKTRACE) && defined(CONFIG_HAVE_SHADOW_STACKTRACE)
+int stack_trace_save_shadow(unsigned long *store, unsigned int size,
+			    unsigned int skipnr);
+#else
+static inline int stack_trace_save_shadow(unsigned long *store,
+					  unsigned int size,
+					  unsigned int skipnr)
+{
+	return -ENOSYS;
+}
+#endif
+
 #endif /* __LINUX_STACKTRACE_H */
diff --git a/kernel/stacktrace.c b/kernel/stacktrace.c
index 9ed5ce989415..fe305861fd55 100644
--- a/kernel/stacktrace.c
+++ b/kernel/stacktrace.c
@@ -237,6 +237,27 @@  unsigned int stack_trace_save_user(unsigned long *store, unsigned int size)
 }
 #endif
 
+#ifdef CONFIG_HAVE_SHADOW_STACKTRACE
+/**
+ * stack_trace_save_shadow - Save a stack trace based on shadow call stack
+ * @store:	Pointer to the storage array
+ * @size:	Size of the storage array
+ * @skipnr:	Number of entries to skip at the start of the stack trace
+ *
+ * Return: Number of trace entries stored.
+ */
+int stack_trace_save_shadow(unsigned long *store, unsigned int size,
+			    unsigned int skipnr)
+{
+	/*
+	 * Do not use stack_trace_consume_fn to avoid making a function
+	 * call for each collected frame to improve performance.
+	 * Skip + 1 frame to skip stack_trace_save_shadow.
+	 */
+	return arch_stack_walk_shadow(store, size, skipnr + 1);
+}
+#endif
+
 #else /* CONFIG_ARCH_STACKWALK */
 
 /*