| Message ID | 20180530001204.183758-1-shakeelb@google.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Tue, May 29, 2018 at 05:12:04PM -0700, Shakeel Butt wrote: > The memcg kmem cache creation and deactivation (SLUB only) is > asynchronous. If a root kmem cache is destroyed whose memcg cache is in > the process of creation or deactivation, the kernel may crash. > > Example of one such crash: > general protection fault: 0000 [#1] SMP PTI > CPU: 1 PID: 1721 Comm: kworker/14:1 Not tainted 4.17.0-smp > ... > Workqueue: memcg_kmem_cache kmemcg_deactivate_workfn > RIP: 0010:has_cpu_slab > ... > Call Trace: > ? on_each_cpu_cond > __kmem_cache_shrink > kmemcg_cache_deact_after_rcu > kmemcg_deactivate_workfn > process_one_work > worker_thread > kthread > ret_from_fork+0x35/0x40 > > To fix this race, on root kmem cache destruction, mark the cache as > dying and flush the workqueue used for memcg kmem cache creation and > deactivation. > @@ -845,6 +862,8 @@ void kmem_cache_destroy(struct kmem_cache *s) > if (unlikely(!s)) > return; > > + flush_memcg_workqueue(s); > + This should definitely help against async memcg_kmem_cache_create(), but I'm afraid it doesn't eliminate the race with async destruction, unfortunately, because the latter uses call_rcu_sched(): memcg_deactivate_kmem_caches __kmem_cache_deactivate slab_deactivate_memcg_cache_rcu_sched call_rcu_sched kmem_cache_destroy shutdown_memcg_caches shutdown_cache memcg_deactivate_rcufn <dereference destroyed cache> Can we somehow flush those pending rcu requests?
On Sat, Jun 9, 2018 at 3:20 AM Vladimir Davydov <vdavydov.dev@gmail.com> wrote: > > On Tue, May 29, 2018 at 05:12:04PM -0700, Shakeel Butt wrote: > > The memcg kmem cache creation and deactivation (SLUB only) is > > asynchronous. If a root kmem cache is destroyed whose memcg cache is in > > the process of creation or deactivation, the kernel may crash. > > > > Example of one such crash: > > general protection fault: 0000 [#1] SMP PTI > > CPU: 1 PID: 1721 Comm: kworker/14:1 Not tainted 4.17.0-smp > > ... > > Workqueue: memcg_kmem_cache kmemcg_deactivate_workfn > > RIP: 0010:has_cpu_slab > > ... > > Call Trace: > > ? on_each_cpu_cond > > __kmem_cache_shrink > > kmemcg_cache_deact_after_rcu > > kmemcg_deactivate_workfn > > process_one_work > > worker_thread > > kthread > > ret_from_fork+0x35/0x40 > > > > To fix this race, on root kmem cache destruction, mark the cache as > > dying and flush the workqueue used for memcg kmem cache creation and > > deactivation. > > > @@ -845,6 +862,8 @@ void kmem_cache_destroy(struct kmem_cache *s) > > if (unlikely(!s)) > > return; > > > > + flush_memcg_workqueue(s); > > + > > This should definitely help against async memcg_kmem_cache_create(), > but I'm afraid it doesn't eliminate the race with async destruction, > unfortunately, because the latter uses call_rcu_sched(): > > memcg_deactivate_kmem_caches > __kmem_cache_deactivate > slab_deactivate_memcg_cache_rcu_sched > call_rcu_sched > kmem_cache_destroy > shutdown_memcg_caches > shutdown_cache > memcg_deactivate_rcufn > <dereference destroyed cache> > > Can we somehow flush those pending rcu requests? You are right and thanks for catching that. Now I am wondering if synchronize_sched() just before flush_workqueue() should be enough. Otherwise we might have to replace call_sched_rcu with synchronize_sched() in kmemcg_deactivate_workfn which I would not prefer as that would holdup the kmem_cache workqueue. +Paul Paul, we have a situation something similar to the following pseudo code. CPU0: lock(l) if (!flag) call_rcu_sched(callback); unlock(l) ------ CPU1: lock(l) flag = true unlock(l) synchronize_sched() ------ If CPU0 has called already called call_rchu_sched(callback) then later if CPU1 calls synchronize_sched(). Is there any guarantee that on return from synchronize_sched(), the rcu callback scheduled by CPU0 has already been executed? thanks, Shakeel
On Sun, Jun 10, 2018 at 07:52:50AM -0700, Shakeel Butt wrote: > On Sat, Jun 9, 2018 at 3:20 AM Vladimir Davydov <vdavydov.dev@gmail.com> wrote: > > > > On Tue, May 29, 2018 at 05:12:04PM -0700, Shakeel Butt wrote: > > > The memcg kmem cache creation and deactivation (SLUB only) is > > > asynchronous. If a root kmem cache is destroyed whose memcg cache is in > > > the process of creation or deactivation, the kernel may crash. > > > > > > Example of one such crash: > > > general protection fault: 0000 [#1] SMP PTI > > > CPU: 1 PID: 1721 Comm: kworker/14:1 Not tainted 4.17.0-smp > > > ... > > > Workqueue: memcg_kmem_cache kmemcg_deactivate_workfn > > > RIP: 0010:has_cpu_slab > > > ... > > > Call Trace: > > > ? on_each_cpu_cond > > > __kmem_cache_shrink > > > kmemcg_cache_deact_after_rcu > > > kmemcg_deactivate_workfn > > > process_one_work > > > worker_thread > > > kthread > > > ret_from_fork+0x35/0x40 > > > > > > To fix this race, on root kmem cache destruction, mark the cache as > > > dying and flush the workqueue used for memcg kmem cache creation and > > > deactivation. > > > > > @@ -845,6 +862,8 @@ void kmem_cache_destroy(struct kmem_cache *s) > > > if (unlikely(!s)) > > > return; > > > > > > + flush_memcg_workqueue(s); > > > + > > > > This should definitely help against async memcg_kmem_cache_create(), > > but I'm afraid it doesn't eliminate the race with async destruction, > > unfortunately, because the latter uses call_rcu_sched(): > > > > memcg_deactivate_kmem_caches > > __kmem_cache_deactivate > > slab_deactivate_memcg_cache_rcu_sched > > call_rcu_sched > > kmem_cache_destroy > > shutdown_memcg_caches > > shutdown_cache > > memcg_deactivate_rcufn > > <dereference destroyed cache> > > > > Can we somehow flush those pending rcu requests? > > You are right and thanks for catching that. Now I am wondering if > synchronize_sched() just before flush_workqueue() should be enough. > Otherwise we might have to replace call_sched_rcu with > synchronize_sched() in kmemcg_deactivate_workfn which I would not > prefer as that would holdup the kmem_cache workqueue. > > +Paul > > Paul, we have a situation something similar to the following pseudo code. > > CPU0: > lock(l) > if (!flag) > call_rcu_sched(callback); > unlock(l) > ------ > CPU1: > lock(l) > flag = true > unlock(l) > synchronize_sched() > ------ > > If CPU0 has called already called call_rchu_sched(callback) then later > if CPU1 calls synchronize_sched(). Is there any guarantee that on > return from synchronize_sched(), the rcu callback scheduled by CPU0 > has already been executed? No. There is no such guarantee. You instead want rcu_barrier_sched(), which waits for the callbacks from all prior invocations of call_rcu_sched() to be invoked. Please note that synchronize_sched() is -not- sufficient. It is only guaranteed to wait for a grace period, not necessarily for all prior callbacks. This goes both directions because if there are no callbacks in the system, then rcu_barrier_sched() is within its rights to return immediately. So please make sure you use each of synchronize_sched() and rcu_barrier_sched() to do the job that it was intended to do! ;-) If your lock(l) is shorthand for spin_lock(&l), it looks to me like you actually only need rcu_barrier_sched(): CPU0: spin_lock(&l); if (!flag) call_rcu_sched(callback); spin_unlock(&l); CPU1: spin_lock(&l); flag = true; spin_unlock(&l); /* At this point, no more callbacks will be registered. */ rcu_barrier_sched(); /* At this point, all registered callbacks will have been invoked. */ On the other hand, if your "lock(l)" was instead shorthand for rcu_read_lock_sched(), then you need -both- synchronize_sched() -and- rcu_barrier(). And even then, you will be broken in -rt kernels. (Which might or might not be a concern, depending on whether your code matters to -rt kernels. Make sense? Thanx, Paul
On Sun, Jun 10, 2018 at 9:32 AM Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote: > > On Sun, Jun 10, 2018 at 07:52:50AM -0700, Shakeel Butt wrote: > > On Sat, Jun 9, 2018 at 3:20 AM Vladimir Davydov <vdavydov.dev@gmail.com> wrote: > > > > > > On Tue, May 29, 2018 at 05:12:04PM -0700, Shakeel Butt wrote: > > > > The memcg kmem cache creation and deactivation (SLUB only) is > > > > asynchronous. If a root kmem cache is destroyed whose memcg cache is in > > > > the process of creation or deactivation, the kernel may crash. > > > > > > > > Example of one such crash: > > > > general protection fault: 0000 [#1] SMP PTI > > > > CPU: 1 PID: 1721 Comm: kworker/14:1 Not tainted 4.17.0-smp > > > > ... > > > > Workqueue: memcg_kmem_cache kmemcg_deactivate_workfn > > > > RIP: 0010:has_cpu_slab > > > > ... > > > > Call Trace: > > > > ? on_each_cpu_cond > > > > __kmem_cache_shrink > > > > kmemcg_cache_deact_after_rcu > > > > kmemcg_deactivate_workfn > > > > process_one_work > > > > worker_thread > > > > kthread > > > > ret_from_fork+0x35/0x40 > > > > > > > > To fix this race, on root kmem cache destruction, mark the cache as > > > > dying and flush the workqueue used for memcg kmem cache creation and > > > > deactivation. > > > > > > > @@ -845,6 +862,8 @@ void kmem_cache_destroy(struct kmem_cache *s) > > > > if (unlikely(!s)) > > > > return; > > > > > > > > + flush_memcg_workqueue(s); > > > > + > > > > > > This should definitely help against async memcg_kmem_cache_create(), > > > but I'm afraid it doesn't eliminate the race with async destruction, > > > unfortunately, because the latter uses call_rcu_sched(): > > > > > > memcg_deactivate_kmem_caches > > > __kmem_cache_deactivate > > > slab_deactivate_memcg_cache_rcu_sched > > > call_rcu_sched > > > kmem_cache_destroy > > > shutdown_memcg_caches > > > shutdown_cache > > > memcg_deactivate_rcufn > > > <dereference destroyed cache> > > > > > > Can we somehow flush those pending rcu requests? > > > > You are right and thanks for catching that. Now I am wondering if > > synchronize_sched() just before flush_workqueue() should be enough. > > Otherwise we might have to replace call_sched_rcu with > > synchronize_sched() in kmemcg_deactivate_workfn which I would not > > prefer as that would holdup the kmem_cache workqueue. > > > > +Paul > > > > Paul, we have a situation something similar to the following pseudo code. > > > > CPU0: > > lock(l) > > if (!flag) > > call_rcu_sched(callback); > > unlock(l) > > ------ > > CPU1: > > lock(l) > > flag = true > > unlock(l) > > synchronize_sched() > > ------ > > > > If CPU0 has called already called call_rchu_sched(callback) then later > > if CPU1 calls synchronize_sched(). Is there any guarantee that on > > return from synchronize_sched(), the rcu callback scheduled by CPU0 > > has already been executed? > > No. There is no such guarantee. > > You instead want rcu_barrier_sched(), which waits for the callbacks from > all prior invocations of call_rcu_sched() to be invoked. > > Please note that synchronize_sched() is -not- sufficient. It is only > guaranteed to wait for a grace period, not necessarily for all prior > callbacks. This goes both directions because if there are no callbacks > in the system, then rcu_barrier_sched() is within its rights to return > immediately. > > So please make sure you use each of synchronize_sched() and > rcu_barrier_sched() to do the job that it was intended to do! ;-) > > If your lock(l) is shorthand for spin_lock(&l), it looks to me like you > actually only need rcu_barrier_sched(): > > CPU0: > spin_lock(&l); > if (!flag) > call_rcu_sched(callback); > spin_unlock(&l); > > CPU1: > spin_lock(&l); > flag = true; > spin_unlock(&l); > /* At this point, no more callbacks will be registered. */ > rcu_barrier_sched(); > /* At this point, all registered callbacks will have been invoked. */ > > On the other hand, if your "lock(l)" was instead shorthand for > rcu_read_lock_sched(), then you need -both- synchronize_sched() -and- > rcu_barrier(). And even then, you will be broken in -rt kernels. > (Which might or might not be a concern, depending on whether your code > matters to -rt kernels. > > Make sense? > Thanks a lot, that was really helpful. The lock is actually mutex_lock. So, I think rcu_barrier_sched() should be sufficient. Shakeel
On Sun, Jun 10, 2018 at 10:40:17AM -0700, Shakeel Butt wrote: > On Sun, Jun 10, 2018 at 9:32 AM Paul E. McKenney > <paulmck@linux.vnet.ibm.com> wrote: > > > > On Sun, Jun 10, 2018 at 07:52:50AM -0700, Shakeel Butt wrote: > > > On Sat, Jun 9, 2018 at 3:20 AM Vladimir Davydov <vdavydov.dev@gmail.com> wrote: > > > > > > > > On Tue, May 29, 2018 at 05:12:04PM -0700, Shakeel Butt wrote: > > > > > The memcg kmem cache creation and deactivation (SLUB only) is > > > > > asynchronous. If a root kmem cache is destroyed whose memcg cache is in > > > > > the process of creation or deactivation, the kernel may crash. > > > > > > > > > > Example of one such crash: > > > > > general protection fault: 0000 [#1] SMP PTI > > > > > CPU: 1 PID: 1721 Comm: kworker/14:1 Not tainted 4.17.0-smp > > > > > ... > > > > > Workqueue: memcg_kmem_cache kmemcg_deactivate_workfn > > > > > RIP: 0010:has_cpu_slab > > > > > ... > > > > > Call Trace: > > > > > ? on_each_cpu_cond > > > > > __kmem_cache_shrink > > > > > kmemcg_cache_deact_after_rcu > > > > > kmemcg_deactivate_workfn > > > > > process_one_work > > > > > worker_thread > > > > > kthread > > > > > ret_from_fork+0x35/0x40 > > > > > > > > > > To fix this race, on root kmem cache destruction, mark the cache as > > > > > dying and flush the workqueue used for memcg kmem cache creation and > > > > > deactivation. > > > > > > > > > @@ -845,6 +862,8 @@ void kmem_cache_destroy(struct kmem_cache *s) > > > > > if (unlikely(!s)) > > > > > return; > > > > > > > > > > + flush_memcg_workqueue(s); > > > > > + > > > > > > > > This should definitely help against async memcg_kmem_cache_create(), > > > > but I'm afraid it doesn't eliminate the race with async destruction, > > > > unfortunately, because the latter uses call_rcu_sched(): > > > > > > > > memcg_deactivate_kmem_caches > > > > __kmem_cache_deactivate > > > > slab_deactivate_memcg_cache_rcu_sched > > > > call_rcu_sched > > > > kmem_cache_destroy > > > > shutdown_memcg_caches > > > > shutdown_cache > > > > memcg_deactivate_rcufn > > > > <dereference destroyed cache> > > > > > > > > Can we somehow flush those pending rcu requests? > > > > > > You are right and thanks for catching that. Now I am wondering if > > > synchronize_sched() just before flush_workqueue() should be enough. > > > Otherwise we might have to replace call_sched_rcu with > > > synchronize_sched() in kmemcg_deactivate_workfn which I would not > > > prefer as that would holdup the kmem_cache workqueue. > > > > > > +Paul > > > > > > Paul, we have a situation something similar to the following pseudo code. > > > > > > CPU0: > > > lock(l) > > > if (!flag) > > > call_rcu_sched(callback); > > > unlock(l) > > > ------ > > > CPU1: > > > lock(l) > > > flag = true > > > unlock(l) > > > synchronize_sched() > > > ------ > > > > > > If CPU0 has called already called call_rchu_sched(callback) then later > > > if CPU1 calls synchronize_sched(). Is there any guarantee that on > > > return from synchronize_sched(), the rcu callback scheduled by CPU0 > > > has already been executed? > > > > No. There is no such guarantee. > > > > You instead want rcu_barrier_sched(), which waits for the callbacks from > > all prior invocations of call_rcu_sched() to be invoked. > > > > Please note that synchronize_sched() is -not- sufficient. It is only > > guaranteed to wait for a grace period, not necessarily for all prior > > callbacks. This goes both directions because if there are no callbacks > > in the system, then rcu_barrier_sched() is within its rights to return > > immediately. > > > > So please make sure you use each of synchronize_sched() and > > rcu_barrier_sched() to do the job that it was intended to do! ;-) > > > > If your lock(l) is shorthand for spin_lock(&l), it looks to me like you > > actually only need rcu_barrier_sched(): > > > > CPU0: > > spin_lock(&l); > > if (!flag) > > call_rcu_sched(callback); > > spin_unlock(&l); > > > > CPU1: > > spin_lock(&l); > > flag = true; > > spin_unlock(&l); > > /* At this point, no more callbacks will be registered. */ > > rcu_barrier_sched(); > > /* At this point, all registered callbacks will have been invoked. */ > > > > On the other hand, if your "lock(l)" was instead shorthand for > > rcu_read_lock_sched(), then you need -both- synchronize_sched() -and- > > rcu_barrier(). And even then, you will be broken in -rt kernels. > > (Which might or might not be a concern, depending on whether your code > > matters to -rt kernels. > > > > Make sense? > > Thanks a lot, that was really helpful. The lock is actually > mutex_lock. So, I think rcu_barrier_sched() should be sufficient. Yes, with either spin_lock() or mutex_lock(), this should work. Mutual exclusion and all that. ;-) Thanx, Paul
diff --git a/include/linux/slab.h b/include/linux/slab.h index 9ebe659bd4a5..71c5467d99c1 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -658,6 +658,7 @@ struct memcg_cache_params { struct memcg_cache_array __rcu *memcg_caches; struct list_head __root_caches_node; struct list_head children; + bool dying; }; struct { struct mem_cgroup *memcg; diff --git a/mm/slab_common.c b/mm/slab_common.c index b0dd9db1eb2f..a3496375f5ea 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -136,6 +136,7 @@ void slab_init_memcg_params(struct kmem_cache *s) s->memcg_params.root_cache = NULL; RCU_INIT_POINTER(s->memcg_params.memcg_caches, NULL); INIT_LIST_HEAD(&s->memcg_params.children); + s->memcg_params.dying = false; } static int init_memcg_params(struct kmem_cache *s, @@ -608,7 +609,7 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg, * The memory cgroup could have been offlined while the cache * creation work was pending. */ - if (memcg->kmem_state != KMEM_ONLINE) + if (memcg->kmem_state != KMEM_ONLINE || root_cache->memcg_params.dying) goto out_unlock; idx = memcg_cache_id(memcg); @@ -712,6 +713,9 @@ void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, WARN_ON_ONCE(s->memcg_params.deact_fn)) return; + if (s->memcg_params.root_cache->memcg_params.dying) + return; + /* pin memcg so that @s doesn't get destroyed in the middle */ css_get(&s->memcg_params.memcg->css); @@ -823,11 +827,24 @@ static int shutdown_memcg_caches(struct kmem_cache *s) return -EBUSY; return 0; } + +static void flush_memcg_workqueue(struct kmem_cache *s) +{ + mutex_lock(&slab_mutex); + s->memcg_params.dying = true; + mutex_unlock(&slab_mutex); + + flush_workqueue(memcg_kmem_cache_wq); +} #else static inline int shutdown_memcg_caches(struct kmem_cache *s) { return 0; } + +static inline void flush_memcg_workqueue(struct kmem_cache *s) +{ +} #endif /* CONFIG_MEMCG && !CONFIG_SLOB */ void slab_kmem_cache_release(struct kmem_cache *s) @@ -845,6 +862,8 @@ void kmem_cache_destroy(struct kmem_cache *s) if (unlikely(!s)) return; + flush_memcg_workqueue(s); + get_online_cpus(); get_online_mems();
The memcg kmem cache creation and deactivation (SLUB only) is asynchronous. If a root kmem cache is destroyed whose memcg cache is in the process of creation or deactivation, the kernel may crash. Example of one such crash: general protection fault: 0000 [#1] SMP PTI CPU: 1 PID: 1721 Comm: kworker/14:1 Not tainted 4.17.0-smp ... Workqueue: memcg_kmem_cache kmemcg_deactivate_workfn RIP: 0010:has_cpu_slab ... Call Trace: ? on_each_cpu_cond __kmem_cache_shrink kmemcg_cache_deact_after_rcu kmemcg_deactivate_workfn process_one_work worker_thread kthread ret_from_fork+0x35/0x40 To fix this race, on root kmem cache destruction, mark the cache as dying and flush the workqueue used for memcg kmem cache creation and deactivation. Signed-off-by: Shakeel Butt <shakeelb@google.com> --- Changelog since v2: - Instead of refcount, flush the workqueue Changelog since v1: - Added more documentation to the code - Renamed fields to be more readable --- include/linux/slab.h | 1 + mm/slab_common.c | 21 ++++++++++++++++++++- 2 files changed, 21 insertions(+), 1 deletion(-)