From patchwork Thu Oct 26 07:37:52 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hou Tao X-Patchwork-Id: 10027543 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B0B2860567 for ; Thu, 26 Oct 2017 07:32:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9EB8028D37 for ; Thu, 26 Oct 2017 07:32:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9320728D47; Thu, 26 Oct 2017 07:32:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB2ED28D45 for ; Thu, 26 Oct 2017 07:32:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751305AbdJZHcW (ORCPT ); Thu, 26 Oct 2017 03:32:22 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:9508 "EHLO szxga05-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751166AbdJZHcV (ORCPT ); Thu, 26 Oct 2017 03:32:21 -0400 Received: from 172.30.72.59 (EHLO DGGEMS410-HUB.china.huawei.com) ([172.30.72.59]) by dggrg05-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id DJY42450; Thu, 26 Oct 2017 15:31:59 +0800 (CST) Received: from huawei.com (10.175.124.28) by DGGEMS410-HUB.china.huawei.com (10.3.19.210) with Microsoft SMTP Server id 14.3.361.1; Thu, 26 Oct 2017 15:31:35 +0800 From: Hou Tao To: CC: , , Subject: [PATCH 2/2] xfs: test for umount hang caused by the pending dquota log item in AIL Date: Thu, 26 Oct 2017 15:37:52 +0800 Message-ID: <1509003472-24191-2-git-send-email-houtao1@huawei.com> X-Mailer: git-send-email 2.7.5 In-Reply-To: <1509003472-24191-1-git-send-email-houtao1@huawei.com> References: <1509003472-24191-1-git-send-email-houtao1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.124.28] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A010205.59F18F70.0042, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 7efc79fc18acd94ff656e6bc1258794b Sender: linux-xfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When the first writeback and the retried writeback of dquota buffer get the same IO error, XFS will let xfsaild to restart the writeback and xfs_qm_dqflush_done() will not be invoked. xfsaild will try to re-push the quota log item in AIL, the push will return early everytime after checking xfs_dqflock_nowait(), and xfsaild will try to push it again. IOWs, AIL will never be empty, and the umount process will wait for the drain of AIL, so the umount process hangs. Signed-off-by: Hou Tao --- tests/xfs/999 | 169 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/999.out | 2 + tests/xfs/group | 1 + 3 files changed, 172 insertions(+) create mode 100755 tests/xfs/999 create mode 100644 tests/xfs/999.out diff --git a/tests/xfs/999 b/tests/xfs/999 new file mode 100755 index 0000000..4b89899 --- /dev/null +++ b/tests/xfs/999 @@ -0,0 +1,169 @@ +#! /bin/bash +# FS QA Test No. 999 +# +# Test for XFS umount hang problem caused by the unceasing push +# of dquot log item in AIL. Because xfs_qm_dqflush_done() will +# not be invoked, so each time xfsaild initiates the push, +# the push will return early after checking xfs_dqflock_nowait(). +# +# xfs_qm_dqflush_done() should be invoked by xfs_buf_do_callbacks(). +# However after the first write and the retried write of dquota buffer +# get the same IO error, XFS will let xfsaild to restart the write and +# xfs_buf_do_callbacks() will not be inovked. +# +# This test emulates the write error by using dm-flakey. The log +# area of the XFS filesystem is excluded from the range covered by +# dm-flakey, so the XFS will not be shutdown prematurely. +# +#----------------------------------------------------------------------- +# Copyright (c) 2017 Huawei Technologies Co., Ltd. All Rights Reserved. +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +#----------------------------------------------------------------------- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* + sysctl -w fs.xfs.xfssyncd_centisecs=3000 >/dev/null 2>&1 + _unmount_flakey >/dev/null 2>&1 + _cleanup_flakey > /dev/null 2>&1 +} + +_get_xfs_scratch_sb_field() +{ + local field=$1 + + echo $(_scratch_xfs_db -r -c "sb 0" -c "print $field" | \ + awk -v field=$field '$0 ~ field {print $3}') +} + +# inject IO write error for the XFS filesystem except its log section +_make_xfs_scratch_flakey_table() +{ + local opt="0 1 1 error_writes" + local dev=${SCRATCH_DEV} + local dev_sz=$(blockdev --getsz $dev) + + if [ "${USE_EXTERNAL}" = "yes" -a ! -z "$SCRATCH_LOGDEV" ]; then + echo "0 ${dev_sz} flakey $dev 0 ${opt}" + return + fi + + local blk_sz=$(_get_xfs_scratch_sb_field blocksize) + local log_ofs=$(_get_xfs_scratch_sb_field logstart) + local log_sz=$(_get_xfs_scratch_sb_field logblocks) + local table="" + local ofs=0 + local sz + + let "log_ofs *= blk_sz / 512" + let "log_sz *= blk_sz / 512" + + if [ "$ofs" -lt "${log_ofs}" ]; then + let "sz = log_ofs - ofs" + table="$ofs $sz flakey $dev $ofs ${opt}" + fi + + table="$table\n${log_ofs} ${log_sz} linear $dev ${log_ofs}" + + let "ofs = log_ofs + log_sz" + if [ "$ofs" -lt "${dev_sz}" ]; then + let "sz = dev_sz - ofs" + table="$table\n$ofs $sz flakey $dev $ofs ${opt}" + fi + + echo -e $table +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/dmflakey +. ./common/xfs +. ./common/quota + +_supported_fs xfs +_supported_os Linux + +# due to the injection of write IO error, the fs will be inconsistent +_require_scratch_nocheck +_require_dm_target linear +_require_dm_target flakey +_require_user +_require_xfs_quota +_require_freeze + +rm -f $seqres.full + +echo "Silence is golden" + +_scratch_mkfs_xfs >/dev/null 2>&1 + +_init_flakey + +# no error will be injected +FLAKEY_TABLE="0 $(blockdev --getsz $SCRATCH_DEV) linear $SCRATCH_DEV 0" +_load_flakey_table $FLAKEY_ALLOW_WRITES +$DMSETUP_PROG info >> $seqres.full +$DMSETUP_PROG table >> $seqres.full + +# shorten the time waiting for the push of ail items +sysctl -w fs.xfs.xfssyncd_centisecs=100 >/dev/null 2>&1 + +_qmount_option "usrquota" +_mount_flakey + +# We need to set the quota limitation twice, and inject the write error +# after the second setting. If we try to inject the write error after +# the first setting, the initialization of the dquota buffer will get +# IO error and also be retried, and during the umount process the +# write will be ended, and xfs_qm_dqflush_done() will be inovked, and +# the umount will exit normally. +$XFS_QUOTA_PROG -x -c "limit -u isoft=500 fsgqa" $SCRATCH_MNT +$XFS_QUOTA_PROG -x -c "report -ih" $SCRATCH_MNT >> $seqres.full + +# ensure the initialization of the dquota buffer is done +xfs_freeze -f $SCRATCH_MNT +xfs_freeze -u $SCRATCH_MNT + +$XFS_QUOTA_PROG -x -c "limit -u isoft=400 fsgqa" $SCRATCH_MNT +$XFS_QUOTA_PROG -x -c "report -ih" $SCRATCH_MNT >> $seqres.full + +# inject write IO error +FLAKEY_TABLE=$(_make_xfs_scratch_flakey_table) +_load_flakey_table $FLAKEY_ALLOW_WRITES +$DMSETUP_PROG info >> $seqres.full +$DMSETUP_PROG table >> $seqres.full + +sync +# wait for the push of the dquota log item in AIL and +# the completion of the retried write of dquota buffer +sleep 2 + +_unmount_flakey + +_cleanup_flakey + +# success, all done +status=0 +exit diff --git a/tests/xfs/999.out b/tests/xfs/999.out new file mode 100644 index 0000000..3b276ca --- /dev/null +++ b/tests/xfs/999.out @@ -0,0 +1,2 @@ +QA output created by 999 +Silence is golden diff --git a/tests/xfs/group b/tests/xfs/group index b439842..d11cbfd 100644 --- a/tests/xfs/group +++ b/tests/xfs/group @@ -431,3 +431,4 @@ 431 auto quick dangerous 432 auto quick dir metadata 433 auto quick attr +999 quota dangerous