| Message ID | 1505205930-2445-1-git-send-email-houtao1@huawei.com (mailing list archive) |
|---|---|
| State | Accepted, archived |
| Headers | show |
On Tue, Sep 12, 2017 at 04:45:30PM +0800, Hou Tao wrote: > A umount hang is possible when a race occurs between the umount > process and the xfsaild kthread. The following sequences outline > the race: > > xfsaild: kthread_should_stop() > => return false, so xfsaild continue > > umount: set_bit(KTHREAD_SHOULD_STOP, &kthread->flags) > => by kthread_stop() > umount: wake_up_process() > => because xfsaild is still running, so 0 is returned > > xfsaild: __set_current_state(TASK_INTERRUPTIBLE) > xfsaild: schedule() > => now, xfsaild will wait indefinitely > > umount: wait_for_completion() > => and umount will hang > > To fix that, we need to check kthread_should_stop() after we set > the task state, so the xfsaild will either see the stop bit and > exit or the task state is reset to runnable by wake_up_process() > such that it isn't scheduled out indefinitely and detects the stop > bit at the next iteration. > > Reviewed-by: Christoph Hellwig <hch@lst.de> > Reviewed-by: Brian Foster <bfoster@redhat.com> > Signed-off-by: Hou Tao <houtao1@huawei.com> Looks ok, I guess, but just to reiterate what I said in another email 15 minutes ago, can this be turned into a regression test? --D > --- > v2: > * comment updates suggested by Brain Foster > v1: > * http://www.spinics.net/lists/linux-xfs/msg10285.html > --- > fs/xfs/xfs_trans_ail.c | 21 ++++++++++++++++++--- > 1 file changed, 18 insertions(+), 3 deletions(-) > > diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c > index 9056c0f..2d77d9c 100644 > --- a/fs/xfs/xfs_trans_ail.c > +++ b/fs/xfs/xfs_trans_ail.c > @@ -499,11 +499,26 @@ xfsaild( > current->flags |= PF_MEMALLOC; > set_freezable(); > > - while (!kthread_should_stop()) { > + while (1) { > if (tout && tout <= 20) > - __set_current_state(TASK_KILLABLE); > + set_current_state(TASK_KILLABLE); > else > - __set_current_state(TASK_INTERRUPTIBLE); > + set_current_state(TASK_INTERRUPTIBLE); > + > + /* > + * Check kthread_should_stop() after we set the task state > + * to guarantee that we either see the stop bit and exit or > + * the task state is reset to runnable such that it's not > + * scheduled out indefinitely and detects the stop bit at > + * next iteration. > + * > + * A memory barrier is included in above task state set to > + * serialize again kthread_stop(). > + */ > + if (kthread_should_stop()) { > + __set_current_state(TASK_RUNNING); > + break; > + } > > spin_lock(&ailp->xa_lock); > > -- > 2.5.0 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Darrick, On 2017/9/19 2:52, Darrick J. Wong wrote: > On Tue, Sep 12, 2017 at 04:45:30PM +0800, Hou Tao wrote: >> A umount hang is possible when a race occurs between the umount >> process and the xfsaild kthread. The following sequences outline >> the race: >> >> xfsaild: kthread_should_stop() >> => return false, so xfsaild continue >> >> umount: set_bit(KTHREAD_SHOULD_STOP, &kthread->flags) >> => by kthread_stop() >> umount: wake_up_process() >> => because xfsaild is still running, so 0 is returned >> >> xfsaild: __set_current_state(TASK_INTERRUPTIBLE) >> xfsaild: schedule() >> => now, xfsaild will wait indefinitely >> >> umount: wait_for_completion() >> => and umount will hang >> >> To fix that, we need to check kthread_should_stop() after we set >> the task state, so the xfsaild will either see the stop bit and >> exit or the task state is reset to runnable by wake_up_process() >> such that it isn't scheduled out indefinitely and detects the stop >> bit at the next iteration. >> >> Reviewed-by: Christoph Hellwig <hch@lst.de> >> Reviewed-by: Brian Foster <bfoster@redhat.com> >> Signed-off-by: Hou Tao <houtao1@huawei.com> > > Looks ok, I guess, but just to reiterate what I said in another email > 15 minutes ago, can this be turned into a regression test? It's difficult to create an always-happened test for the race, and I had test the patch by adding artificial delays (as suggested by Brian Foster) in kernel source code. I also have tried to reproduce the problem by lifting up the schedule class/priority of the umount process and lifting down the schedule class/priorityof the xfsaild kthread, but still can not reproduce the problem, so any ideas or suggestions ? Regard > --D > >> --- >> v2: >> * comment updates suggested by Brain Foster >> v1: >> * http://www.spinics.net/lists/linux-xfs/msg10285.html >> --- >> fs/xfs/xfs_trans_ail.c | 21 ++++++++++++++++++--- >> 1 file changed, 18 insertions(+), 3 deletions(-) >> >> diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c >> index 9056c0f..2d77d9c 100644 >> --- a/fs/xfs/xfs_trans_ail.c >> +++ b/fs/xfs/xfs_trans_ail.c >> @@ -499,11 +499,26 @@ xfsaild( >> current->flags |= PF_MEMALLOC; >> set_freezable(); >> >> - while (!kthread_should_stop()) { >> + while (1) { >> if (tout && tout <= 20) >> - __set_current_state(TASK_KILLABLE); >> + set_current_state(TASK_KILLABLE); >> else >> - __set_current_state(TASK_INTERRUPTIBLE); >> + set_current_state(TASK_INTERRUPTIBLE); >> + >> + /* >> + * Check kthread_should_stop() after we set the task state >> + * to guarantee that we either see the stop bit and exit or >> + * the task state is reset to runnable such that it's not >> + * scheduled out indefinitely and detects the stop bit at >> + * next iteration. >> + * >> + * A memory barrier is included in above task state set to >> + * serialize again kthread_stop(). >> + */ >> + if (kthread_should_stop()) { >> + __set_current_state(TASK_RUNNING); >> + break; >> + } >> >> spin_lock(&ailp->xa_lock); >> >> -- >> 2.5.0 >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > . > -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Sep 22, 2017 at 06:57:00PM +0800, Hou Tao wrote: > Hi Darrick, > > On 2017/9/19 2:52, Darrick J. Wong wrote: > > On Tue, Sep 12, 2017 at 04:45:30PM +0800, Hou Tao wrote: > >> A umount hang is possible when a race occurs between the umount > >> process and the xfsaild kthread. The following sequences outline > >> the race: > >> > >> xfsaild: kthread_should_stop() > >> => return false, so xfsaild continue > >> > >> umount: set_bit(KTHREAD_SHOULD_STOP, &kthread->flags) > >> => by kthread_stop() > >> umount: wake_up_process() > >> => because xfsaild is still running, so 0 is returned > >> > >> xfsaild: __set_current_state(TASK_INTERRUPTIBLE) > >> xfsaild: schedule() > >> => now, xfsaild will wait indefinitely > >> > >> umount: wait_for_completion() > >> => and umount will hang > >> > >> To fix that, we need to check kthread_should_stop() after we set > >> the task state, so the xfsaild will either see the stop bit and > >> exit or the task state is reset to runnable by wake_up_process() > >> such that it isn't scheduled out indefinitely and detects the stop > >> bit at the next iteration. > >> > >> Reviewed-by: Christoph Hellwig <hch@lst.de> > >> Reviewed-by: Brian Foster <bfoster@redhat.com> > >> Signed-off-by: Hou Tao <houtao1@huawei.com> > > > > Looks ok, I guess, but just to reiterate what I said in another email > > 15 minutes ago, can this be turned into a regression test? > It's difficult to create an always-happened test for the race, and > I had test the patch by adding artificial delays (as suggested by Brian Foster) in > kernel source code. I also have tried to reproduce the problem by lifting up the > schedule class/priority of the umount process and lifting down the schedule > class/priorityof the xfsaild kthread, but still can not reproduce the problem, > so any ideas or suggestions ? Do the artificial delays make it so that the race happens reliably? If so, we could just add a debug knob to inject delays, which you could then call from a xfstest script. --D > > Regard > > > > --D > > > >> --- > >> v2: > >> * comment updates suggested by Brain Foster > >> v1: > >> * http://www.spinics.net/lists/linux-xfs/msg10285.html > >> --- > >> fs/xfs/xfs_trans_ail.c | 21 ++++++++++++++++++--- > >> 1 file changed, 18 insertions(+), 3 deletions(-) > >> > >> diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c > >> index 9056c0f..2d77d9c 100644 > >> --- a/fs/xfs/xfs_trans_ail.c > >> +++ b/fs/xfs/xfs_trans_ail.c > >> @@ -499,11 +499,26 @@ xfsaild( > >> current->flags |= PF_MEMALLOC; > >> set_freezable(); > >> > >> - while (!kthread_should_stop()) { > >> + while (1) { > >> if (tout && tout <= 20) > >> - __set_current_state(TASK_KILLABLE); > >> + set_current_state(TASK_KILLABLE); > >> else > >> - __set_current_state(TASK_INTERRUPTIBLE); > >> + set_current_state(TASK_INTERRUPTIBLE); > >> + > >> + /* > >> + * Check kthread_should_stop() after we set the task state > >> + * to guarantee that we either see the stop bit and exit or > >> + * the task state is reset to runnable such that it's not > >> + * scheduled out indefinitely and detects the stop bit at > >> + * next iteration. > >> + * > >> + * A memory barrier is included in above task state set to > >> + * serialize again kthread_stop(). > >> + */ > >> + if (kthread_should_stop()) { > >> + __set_current_state(TASK_RUNNING); > >> + break; > >> + } > >> > >> spin_lock(&ailp->xa_lock); > >> > >> -- > >> 2.5.0 > >> > >> -- > >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > >> the body of a message to majordomo@vger.kernel.org > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > . > > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c index 9056c0f..2d77d9c 100644 --- a/fs/xfs/xfs_trans_ail.c +++ b/fs/xfs/xfs_trans_ail.c @@ -499,11 +499,26 @@ xfsaild( current->flags |= PF_MEMALLOC; set_freezable(); - while (!kthread_should_stop()) { + while (1) { if (tout && tout <= 20) - __set_current_state(TASK_KILLABLE); + set_current_state(TASK_KILLABLE); else - __set_current_state(TASK_INTERRUPTIBLE); + set_current_state(TASK_INTERRUPTIBLE); + + /* + * Check kthread_should_stop() after we set the task state + * to guarantee that we either see the stop bit and exit or + * the task state is reset to runnable such that it's not + * scheduled out indefinitely and detects the stop bit at + * next iteration. + * + * A memory barrier is included in above task state set to + * serialize again kthread_stop(). + */ + if (kthread_should_stop()) { + __set_current_state(TASK_RUNNING); + break; + } spin_lock(&ailp->xa_lock);
A umount hang is possible when a race occurs between the umount process and the xfsaild kthread. The following sequences outline the race: xfsaild: kthread_should_stop() => return false, so xfsaild continue umount: set_bit(KTHREAD_SHOULD_STOP, &kthread->flags) => by kthread_stop() umount: wake_up_process() => because xfsaild is still running, so 0 is returned xfsaild: __set_current_state(TASK_INTERRUPTIBLE) xfsaild: schedule() => now, xfsaild will wait indefinitely umount: wait_for_completion() => and umount will hang To fix that, we need to check kthread_should_stop() after we set the task state, so the xfsaild will either see the stop bit and exit or the task state is reset to runnable by wake_up_process() such that it isn't scheduled out indefinitely and detects the stop bit at the next iteration. Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Hou Tao <houtao1@huawei.com> --- v2: * comment updates suggested by Brain Foster v1: * http://www.spinics.net/lists/linux-xfs/msg10285.html --- fs/xfs/xfs_trans_ail.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)