| Message ID | 20221123001303.533968-1-dmitry.osipenko@collabora.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [v1] drm/scheduler: Fix lockup in drm_sched_entity_kill() | expand |
Am 23.11.22 um 01:13 schrieb Dmitry Osipenko: > The drm_sched_entity_kill() is invoked twice by drm_sched_entity_destroy() > while userspace process is exiting or being killed. First time it's invoked > when sched entity is flushed and second time when entity is released. This > causes a lockup within wait_for_completion(entity_idle) due to how completion > API works. > > Calling wait_for_completion() more times than complete() was invoked is a > error condition that causes lockup because completion internally uses > counter for complete/wait calls. The complete_all() must be used instead > in such cases. > > This patch fixes lockup of Panfrost driver that is reproducible by killing > any application in a middle of 3d drawing operation. > > Fixes: 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini") > Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Oh, good point. Reviewed-by: Christian König <christian.koenig@amd.com> > --- > drivers/gpu/drm/scheduler/sched_entity.c | 2 +- > drivers/gpu/drm/scheduler/sched_main.c | 4 ++-- > 2 files changed, 3 insertions(+), 3 deletions(-) > > diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c > index fe09e5be79bd..15d04a0ec623 100644 > --- a/drivers/gpu/drm/scheduler/sched_entity.c > +++ b/drivers/gpu/drm/scheduler/sched_entity.c > @@ -81,7 +81,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, > init_completion(&entity->entity_idle); > > /* We start in an idle state. */ > - complete(&entity->entity_idle); > + complete_all(&entity->entity_idle); > > spin_lock_init(&entity->rq_lock); > spsc_queue_init(&entity->job_queue); > diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c > index 6ce04c2e90c0..857ec20be9e8 100644 > --- a/drivers/gpu/drm/scheduler/sched_main.c > +++ b/drivers/gpu/drm/scheduler/sched_main.c > @@ -1026,7 +1026,7 @@ static int drm_sched_main(void *param) > sched_job = drm_sched_entity_pop_job(entity); > > if (!sched_job) { > - complete(&entity->entity_idle); > + complete_all(&entity->entity_idle); > continue; > } > > @@ -1037,7 +1037,7 @@ static int drm_sched_main(void *param) > > trace_drm_run_job(sched_job, entity); > fence = sched->ops->run_job(sched_job); > - complete(&entity->entity_idle); > + complete_all(&entity->entity_idle); > drm_sched_fence_scheduled(s_fence); > > if (!IS_ERR_OR_NULL(fence)) {
On 11/23/22 03:13, Dmitry Osipenko wrote: > The drm_sched_entity_kill() is invoked twice by drm_sched_entity_destroy() > while userspace process is exiting or being killed. First time it's invoked > when sched entity is flushed and second time when entity is released. This > causes a lockup within wait_for_completion(entity_idle) due to how completion > API works. > > Calling wait_for_completion() more times than complete() was invoked is a > error condition that causes lockup because completion internally uses > counter for complete/wait calls. The complete_all() must be used instead > in such cases. > > This patch fixes lockup of Panfrost driver that is reproducible by killing > any application in a middle of 3d drawing operation. > > Fixes: 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini") > Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> > --- > drivers/gpu/drm/scheduler/sched_entity.c | 2 +- > drivers/gpu/drm/scheduler/sched_main.c | 4 ++-- > 2 files changed, 3 insertions(+), 3 deletions(-) > > diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c > index fe09e5be79bd..15d04a0ec623 100644 > --- a/drivers/gpu/drm/scheduler/sched_entity.c > +++ b/drivers/gpu/drm/scheduler/sched_entity.c > @@ -81,7 +81,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, > init_completion(&entity->entity_idle); > > /* We start in an idle state. */ > - complete(&entity->entity_idle); > + complete_all(&entity->entity_idle); > > spin_lock_init(&entity->rq_lock); > spsc_queue_init(&entity->job_queue); > diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c > index 6ce04c2e90c0..857ec20be9e8 100644 > --- a/drivers/gpu/drm/scheduler/sched_main.c > +++ b/drivers/gpu/drm/scheduler/sched_main.c > @@ -1026,7 +1026,7 @@ static int drm_sched_main(void *param) > sched_job = drm_sched_entity_pop_job(entity); > > if (!sched_job) { > - complete(&entity->entity_idle); > + complete_all(&entity->entity_idle); > continue; > } > > @@ -1037,7 +1037,7 @@ static int drm_sched_main(void *param) > > trace_drm_run_job(sched_job, entity); > fence = sched->ops->run_job(sched_job); > - complete(&entity->entity_idle); > + complete_all(&entity->entity_idle); > drm_sched_fence_scheduled(s_fence); > > if (!IS_ERR_OR_NULL(fence)) { Applied to drm-misc-next
Hi Dmitry / Christian, thanks for the fix!
(And thanks Melissa for pointing that, saving me lots of time in
research heh)
Is this fix planned to be released on 6.2-rc cycle? I've just tested it
on Steam Deck, and it resolved a lockup observed (since v6.2-rc1) -
exactly the same thing mentioned in the commit message.
FWIW:
Tested-By: Guilherme G. Piccoli <gpiccoli@igalia.com> # Steam Deck
Cheers,
Guilherme
Hi, On 12/27/22 22:28, Guilherme G. Piccoli wrote: > Hi Dmitry / Christian, thanks for the fix! > > (And thanks Melissa for pointing that, saving me lots of time in > research heh) > > Is this fix planned to be released on 6.2-rc cycle? I've just tested it > on Steam Deck, and it resolved a lockup observed (since v6.2-rc1) - > exactly the same thing mentioned in the commit message. > > FWIW: > Tested-By: Guilherme G. Piccoli <gpiccoli@igalia.com> # Steam Deck I'll push the patch to misc-fixes as soon as it will be rebased on 6.2-rc. Thanks! Best regards, Dmitry
On 29/12/2022 07:15, Dmitry Osipenko wrote: > [...] > I'll push the patch to misc-fixes as soon as it will be rebased on > 6.2-rc. Thanks! > > Best regards, > Dmitry > Thank you Dmitry, much appreciated! Cheers, Guilherme
On 11/23/22 03:13, Dmitry Osipenko wrote: > The drm_sched_entity_kill() is invoked twice by drm_sched_entity_destroy() > while userspace process is exiting or being killed. First time it's invoked > when sched entity is flushed and second time when entity is released. This > causes a lockup within wait_for_completion(entity_idle) due to how completion > API works. > > Calling wait_for_completion() more times than complete() was invoked is a > error condition that causes lockup because completion internally uses > counter for complete/wait calls. The complete_all() must be used instead > in such cases. > > This patch fixes lockup of Panfrost driver that is reproducible by killing > any application in a middle of 3d drawing operation. > > Fixes: 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini") > Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> > --- > drivers/gpu/drm/scheduler/sched_entity.c | 2 +- > drivers/gpu/drm/scheduler/sched_main.c | 4 ++-- > 2 files changed, 3 insertions(+), 3 deletions(-) > > diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c > index fe09e5be79bd..15d04a0ec623 100644 > --- a/drivers/gpu/drm/scheduler/sched_entity.c > +++ b/drivers/gpu/drm/scheduler/sched_entity.c > @@ -81,7 +81,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, > init_completion(&entity->entity_idle); > > /* We start in an idle state. */ > - complete(&entity->entity_idle); > + complete_all(&entity->entity_idle); > > spin_lock_init(&entity->rq_lock); > spsc_queue_init(&entity->job_queue); > diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c > index 6ce04c2e90c0..857ec20be9e8 100644 > --- a/drivers/gpu/drm/scheduler/sched_main.c > +++ b/drivers/gpu/drm/scheduler/sched_main.c > @@ -1026,7 +1026,7 @@ static int drm_sched_main(void *param) > sched_job = drm_sched_entity_pop_job(entity); > > if (!sched_job) { > - complete(&entity->entity_idle); > + complete_all(&entity->entity_idle); > continue; > } > > @@ -1037,7 +1037,7 @@ static int drm_sched_main(void *param) > > trace_drm_run_job(sched_job, entity); > fence = sched->ops->run_job(sched_job); > - complete(&entity->entity_idle); > + complete_all(&entity->entity_idle); > drm_sched_fence_scheduled(s_fence); > > if (!IS_ERR_OR_NULL(fence)) { Applied to drm-misc-next-fixes
diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c index fe09e5be79bd..15d04a0ec623 100644 --- a/drivers/gpu/drm/scheduler/sched_entity.c +++ b/drivers/gpu/drm/scheduler/sched_entity.c @@ -81,7 +81,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, init_completion(&entity->entity_idle); /* We start in an idle state. */ - complete(&entity->entity_idle); + complete_all(&entity->entity_idle); spin_lock_init(&entity->rq_lock); spsc_queue_init(&entity->job_queue); diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 6ce04c2e90c0..857ec20be9e8 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -1026,7 +1026,7 @@ static int drm_sched_main(void *param) sched_job = drm_sched_entity_pop_job(entity); if (!sched_job) { - complete(&entity->entity_idle); + complete_all(&entity->entity_idle); continue; } @@ -1037,7 +1037,7 @@ static int drm_sched_main(void *param) trace_drm_run_job(sched_job, entity); fence = sched->ops->run_job(sched_job); - complete(&entity->entity_idle); + complete_all(&entity->entity_idle); drm_sched_fence_scheduled(s_fence); if (!IS_ERR_OR_NULL(fence)) {
The drm_sched_entity_kill() is invoked twice by drm_sched_entity_destroy() while userspace process is exiting or being killed. First time it's invoked when sched entity is flushed and second time when entity is released. This causes a lockup within wait_for_completion(entity_idle) due to how completion API works. Calling wait_for_completion() more times than complete() was invoked is a error condition that causes lockup because completion internally uses counter for complete/wait calls. The complete_all() must be used instead in such cases. This patch fixes lockup of Panfrost driver that is reproducible by killing any application in a middle of 3d drawing operation. Fixes: 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini") Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> --- drivers/gpu/drm/scheduler/sched_entity.c | 2 +- drivers/gpu/drm/scheduler/sched_main.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)