| Message ID | 20200610163135.17364-7-vbabka@suse.cz (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | slub_debug fixes and improvements | expand |
On Wed, Jun 10, 2020 at 06:31:32PM +0200, Vlastimil Babka wrote: > One advantage of CONFIG_SLUB_DEBUG is that a generic distro kernel can be built > with the option enabled, but it's inactive until simply enabled on boot, > without rebuilding the kernel. With a static key, we can further eliminate the > overhead of checking whether a cache has a particular debug flag enabled if we > know that there are no such caches (slub_debug was not enabled during boot). We > use the same mechanism also for e.g. page_owner, debug_pagealloc or kmemcg > functionality. > > This patch introduces the static key and makes the general check for per-cache > debug flags kmem_cache_debug() use it. This benefits several call sites, > including (slow path but still rather frequent) __slab_free(). The next patches > will add more uses. > > Signed-off-by: Vlastimil Babka <vbabka@suse.cz> Acked-by: Roman Gushchin <guro@fb.com> Thanks! > --- > mm/slub.c | 16 +++++++++++++--- > 1 file changed, 13 insertions(+), 3 deletions(-) > > diff --git a/mm/slub.c b/mm/slub.c > index 8dd2925448ae..24d3e5f832aa 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -114,13 +114,21 @@ > * the fast path and disables lockless freelists. > */ > > +#ifdef CONFIG_SLUB_DEBUG > +#ifdef CONFIG_SLUB_DEBUG_ON > +DEFINE_STATIC_KEY_TRUE(slub_debug_enabled); > +#else > +DEFINE_STATIC_KEY_FALSE(slub_debug_enabled); > +#endif > +#endif > + > static inline int kmem_cache_debug(struct kmem_cache *s) > { > #ifdef CONFIG_SLUB_DEBUG > - return unlikely(s->flags & SLAB_DEBUG_FLAGS); > -#else > - return 0; > + if (static_branch_unlikely(&slub_debug_enabled)) > + return s->flags & SLAB_DEBUG_FLAGS; > #endif > + return 0; > } > > void *fixup_red_left(struct kmem_cache *s, void *p) > @@ -1389,6 +1397,8 @@ static int __init setup_slub_debug(char *str) > slub_debug_string = saved_str; > } > out: > + if (slub_debug != 0 || slub_debug_string) > + static_branch_enable(&slub_debug_enabled); > if ((static_branch_unlikely(&init_on_alloc) || > static_branch_unlikely(&init_on_free)) && > (slub_debug & SLAB_POISON)) > -- > 2.26.2 >
On Wed, Jun 10, 2020 at 06:31:32PM +0200, Vlastimil Babka wrote: > One advantage of CONFIG_SLUB_DEBUG is that a generic distro kernel can be built > with the option enabled, but it's inactive until simply enabled on boot, > without rebuilding the kernel. With a static key, we can further eliminate the > overhead of checking whether a cache has a particular debug flag enabled if we > know that there are no such caches (slub_debug was not enabled during boot). We > use the same mechanism also for e.g. page_owner, debug_pagealloc or kmemcg > functionality. > > This patch introduces the static key and makes the general check for per-cache > debug flags kmem_cache_debug() use it. This benefits several call sites, > including (slow path but still rather frequent) __slab_free(). The next patches > will add more uses. > > Signed-off-by: Vlastimil Babka <vbabka@suse.cz> > --- > mm/slub.c | 16 +++++++++++++--- > 1 file changed, 13 insertions(+), 3 deletions(-) > > diff --git a/mm/slub.c b/mm/slub.c > index 8dd2925448ae..24d3e5f832aa 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -114,13 +114,21 @@ > * the fast path and disables lockless freelists. > */ > > +#ifdef CONFIG_SLUB_DEBUG > +#ifdef CONFIG_SLUB_DEBUG_ON > +DEFINE_STATIC_KEY_TRUE(slub_debug_enabled); > +#else > +DEFINE_STATIC_KEY_FALSE(slub_debug_enabled); > +#endif > +#endif Yet another potential future user of DEFINE_STATIC_KEY_MAYBE()[1], once I get it landed. :) Reviewed-by: Kees Cook <keescook@chromium.org> [1] https://lore.kernel.org/lkml/20200324203231.64324-2-keescook@chromium.org/
diff --git a/mm/slub.c b/mm/slub.c index 8dd2925448ae..24d3e5f832aa 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -114,13 +114,21 @@ * the fast path and disables lockless freelists. */ +#ifdef CONFIG_SLUB_DEBUG +#ifdef CONFIG_SLUB_DEBUG_ON +DEFINE_STATIC_KEY_TRUE(slub_debug_enabled); +#else +DEFINE_STATIC_KEY_FALSE(slub_debug_enabled); +#endif +#endif + static inline int kmem_cache_debug(struct kmem_cache *s) { #ifdef CONFIG_SLUB_DEBUG - return unlikely(s->flags & SLAB_DEBUG_FLAGS); -#else - return 0; + if (static_branch_unlikely(&slub_debug_enabled)) + return s->flags & SLAB_DEBUG_FLAGS; #endif + return 0; } void *fixup_red_left(struct kmem_cache *s, void *p) @@ -1389,6 +1397,8 @@ static int __init setup_slub_debug(char *str) slub_debug_string = saved_str; } out: + if (slub_debug != 0 || slub_debug_string) + static_branch_enable(&slub_debug_enabled); if ((static_branch_unlikely(&init_on_alloc) || static_branch_unlikely(&init_on_free)) && (slub_debug & SLAB_POISON))
One advantage of CONFIG_SLUB_DEBUG is that a generic distro kernel can be built with the option enabled, but it's inactive until simply enabled on boot, without rebuilding the kernel. With a static key, we can further eliminate the overhead of checking whether a cache has a particular debug flag enabled if we know that there are no such caches (slub_debug was not enabled during boot). We use the same mechanism also for e.g. page_owner, debug_pagealloc or kmemcg functionality. This patch introduces the static key and makes the general check for per-cache debug flags kmem_cache_debug() use it. This benefits several call sites, including (slow path but still rather frequent) __slab_free(). The next patches will add more uses. Signed-off-by: Vlastimil Babka <vbabka@suse.cz> --- mm/slub.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)