| Message ID | 201405292334.EAG00503.FLOOJFStHVQMFO@I-love.SAKURA.ne.jp (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Thu, May 29, 2014 at 11:34:59PM +0900, Tetsuo Handa wrote: > Tetsuo Handa wrote: > > Konrad Rzeszutek Wilk wrote: > > > On Sat, May 24, 2014 at 11:22:09PM +0900, Tetsuo Handa wrote: > > > > Hello. > > > > > > > > I tried to test whether it is OK (from point of view of reentrant) to use > > > > mutex_lock() or mutex_lock_killable() inside shrinker functions when shrinker > > > > functions do memory allocation, for drivers/gpu/drm/ttm/ttm_page_alloc_dma.c is > > > > doing memory allocation with mutex lock held inside ttm_dma_pool_shrink_scan(). > > > > > > > > If I compile a test module shown below which mimics extreme case of what > > > > ttm_dma_pool_shrink_scan() will do > > > > > > And ttm_pool_shrink_scan. > > > > I don't know why but ttm_pool_shrink_scan() does not take mutex. > > > Well, it seems to me that ttm_pool_shrink_scan() not taking mutex is a bug > which could lead to stack overflow if kmalloc() in ttm_page_pool_free() > triggered recursion. > > shrink_slab() > => ttm_pool_shrink_scan() > => ttm_page_pool_free() > => kmalloc(GFP_KERNEL) > => shrink_slab() > => ttm_pool_shrink_scan() > => ttm_page_pool_free() > => kmalloc(GFP_KERNEL) > > Maybe shrink_slab() should be updated not to call same shrinker in parallel? > > Also, it seems to me that ttm_dma_pool_shrink_scan() has potential division > by 0 bug as described below. Is this patch correct? Looks OK. I would need to test it first. Could you send both patches to me please so I can just test them and queue them up together? Thank you! > ---------- > >From 4a65744a300e14e5e202c5f13ba2759e1e797d29 Mon Sep 17 00:00:00 2001 > From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> > Date: Thu, 29 May 2014 18:25:42 +0900 > Subject: [PATCH] gpu/drm/ttm: Use mutex_trylock() for shrinker functions. > > I can observe that RHEL7 environment stalls with 100% CPU usage when a > certain type of memory pressure is given. While the shrinker functions > are called by shrink_slab() before the OOM killer is triggered, the stall > lasts for many minutes. > > One of reasons of this stall is that > ttm_dma_pool_shrink_count()/ttm_dma_pool_shrink_scan() are called and > are blocked at mutex_lock(&_manager->lock). GFP_KERNEL allocation with > _manager->lock held causes someone (including kswapd) to deadlock when > these functions are called due to memory pressure. This patch changes > "mutex_lock();" to "if (!mutex_trylock()) return ...;" in order to > avoid deadlock. > > At the same time, this patch fixes potential division by 0 due to > unconditionally doing "% _manager->npools". This is because > list_empty(&_manager->pools) being false does not guarantee that > _manager->npools != 0 after taking the _manager->lock because > _manager->npools is updated under the _manager->lock. > > At the same time, this patch moves updating of start_pool variable > in order to avoid skipping when choosing a pool to shrink in > round-robin style. The start_pool is changed from "atomic_t" to > "unsigned int" because it is now updated under the _manager->lock. > > Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> > Cc: stable <stable@kernel.org> [3.3+] > --- > drivers/gpu/drm/ttm/ttm_page_alloc_dma.c | 15 ++++++++++----- > 1 file changed, 10 insertions(+), 5 deletions(-) > > diff --git a/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c b/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c > index fb8259f..5e332b4 100644 > --- a/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c > +++ b/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c > @@ -1004,9 +1004,9 @@ EXPORT_SYMBOL_GPL(ttm_dma_unpopulate); > static unsigned long > ttm_dma_pool_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) > { > - static atomic_t start_pool = ATOMIC_INIT(0); > + static unsigned int start_pool; > unsigned idx = 0; > - unsigned pool_offset = atomic_add_return(1, &start_pool); > + unsigned pool_offset; > unsigned shrink_pages = sc->nr_to_scan; > struct device_pools *p; > unsigned long freed = 0; > @@ -1014,8 +1014,11 @@ ttm_dma_pool_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) > if (list_empty(&_manager->pools)) > return SHRINK_STOP; > > - mutex_lock(&_manager->lock); > - pool_offset = pool_offset % _manager->npools; > + if (!mutex_trylock(&_manager->lock)) > + return SHRINK_STOP; > + if (!_manager->npools) > + goto out; > + pool_offset = ++start_pool % _manager->npools; > list_for_each_entry(p, &_manager->pools, pools) { > unsigned nr_free; > > @@ -1034,6 +1037,7 @@ ttm_dma_pool_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) > p->pool->dev_name, p->pool->name, current->pid, > nr_free, shrink_pages); > } > +out: > mutex_unlock(&_manager->lock); > return freed; > } > @@ -1044,7 +1048,8 @@ ttm_dma_pool_shrink_count(struct shrinker *shrink, struct shrink_control *sc) > struct device_pools *p; > unsigned long count = 0; > > - mutex_lock(&_manager->lock); > + if (!mutex_trylock(&_manager->lock)) > + return 0; > list_for_each_entry(p, &_manager->pools, pools) > count += p->pool->npages_free; > mutex_unlock(&_manager->lock); > -- > 1.8.3.1
diff --git a/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c b/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c index fb8259f..5e332b4 100644 --- a/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c +++ b/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c @@ -1004,9 +1004,9 @@ EXPORT_SYMBOL_GPL(ttm_dma_unpopulate); static unsigned long ttm_dma_pool_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) { - static atomic_t start_pool = ATOMIC_INIT(0); + static unsigned int start_pool; unsigned idx = 0; - unsigned pool_offset = atomic_add_return(1, &start_pool); + unsigned pool_offset; unsigned shrink_pages = sc->nr_to_scan; struct device_pools *p; unsigned long freed = 0; @@ -1014,8 +1014,11 @@ ttm_dma_pool_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) if (list_empty(&_manager->pools)) return SHRINK_STOP; - mutex_lock(&_manager->lock); - pool_offset = pool_offset % _manager->npools; + if (!mutex_trylock(&_manager->lock)) + return SHRINK_STOP; + if (!_manager->npools) + goto out; + pool_offset = ++start_pool % _manager->npools; list_for_each_entry(p, &_manager->pools, pools) { unsigned nr_free; @@ -1034,6 +1037,7 @@ ttm_dma_pool_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) p->pool->dev_name, p->pool->name, current->pid, nr_free, shrink_pages); } +out: mutex_unlock(&_manager->lock); return freed; } @@ -1044,7 +1048,8 @@ ttm_dma_pool_shrink_count(struct shrinker *shrink, struct shrink_control *sc) struct device_pools *p; unsigned long count = 0; - mutex_lock(&_manager->lock); + if (!mutex_trylock(&_manager->lock)) + return 0; list_for_each_entry(p, &_manager->pools, pools) count += p->pool->npages_free; mutex_unlock(&_manager->lock);