From patchwork Thu Nov 10 16:27:22 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zorro Lang X-Patchwork-Id: 9421439 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 964BF601C0 for ; Thu, 10 Nov 2016 16:27:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7E93228916 for ; Thu, 10 Nov 2016 16:27:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7388C2894B; Thu, 10 Nov 2016 16:27:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 11EE328916 for ; Thu, 10 Nov 2016 16:27:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934808AbcKJQ1d (ORCPT ); Thu, 10 Nov 2016 11:27:33 -0500 Received: from mx1.redhat.com ([209.132.183.28]:41104 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S934617AbcKJQ1c (ORCPT ); Thu, 10 Nov 2016 11:27:32 -0500 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6C995C04B321; Thu, 10 Nov 2016 16:27:32 +0000 (UTC) Received: from localhost.localdomain (vpn1-5-198.pek2.redhat.com [10.72.5.198]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id uAAGRUEc015109; Thu, 10 Nov 2016 11:27:31 -0500 From: Zorro Lang To: fstests@vger.kernel.org Cc: linux-xfs@vger.kernel.org Subject: [PATCH] shared/006: improve the speed of case running Date: Fri, 11 Nov 2016 00:27:22 +0800 Message-Id: <1478795242-14022-1-git-send-email-zlang@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Thu, 10 Nov 2016 16:27:32 +0000 (UTC) Sender: fstests-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP There're three problems of this case: 1. Thousands of threads will be created to create lots of files, then kernel need to waste lots of system resource to schedule these threads. Some poor performance machines will take long long time on that. 2. Per thread try to create 1000 files by run 1000 times "echo >file". For the 1st problem, I limit 2 threads per cpu, and the maximum is 20. For the 2nd problem, use "sed 1 1000 | xargs touch" to instead of the old way. With this change, this case can run over in 2 mins on my x86_64 virtual machine with 1 cpu and 1G memory. Before that, it was still running even a quarter passed. Signed-off-by: Zorro Lang --- Hi, The performance of this case affect the test time of xfstests, especially on poor performance VM. I always doubt it hangs there, because it has run too long time. After this improvement: It ran 105s on my virtual machine with 1 cpu and 1G memory. It ran 60s on my real machine with 8 cpu and 64G memory. The difference of "for ((i=0;i<1000;i++)); echo -n > file$i;done" and "touch file{1..1000}" is: The 1st one will run 1000 times execve, open, close and so on. The execve() will take much time, especially on VM. But the 2nd one will run once execve, 1000 times open and once close. open() take much less time than execve(). Too many threads really waste too much time. For example, on my VM, when I use $((ncpus * 2)) threads to run this case, it ran 100s. But if I use $((ncpus * 4)) threads, the time increase to 130s. So too many threads is not helpful, in contrast it wastes more time. Thanks, Zorro tests/shared/006 | 42 ++++++++++++++++++++++++++++-------------- 1 file changed, 28 insertions(+), 14 deletions(-) diff --git a/tests/shared/006 b/tests/shared/006 index 6a237c9..42cd34d 100755 --- a/tests/shared/006 +++ b/tests/shared/006 @@ -43,13 +43,16 @@ create_file() { local dir=$1 local nr_file=$2 - local prefix=$3 - local i=0 - while [ $i -lt $nr_file ]; do - echo -n > $dir/${prefix}_${i} - let i=$i+1 - done + if [ ! -d $dir ]; then + mkdir -p $dir + fi + + if [ ${nr_file} -gt 0 ]; then + pushd $dir >/dev/null + seq 1 $nr_file | xargs touch + popd >/dev/null + fi } # get standard environment, filters and checks @@ -61,6 +64,9 @@ _supported_fs ext4 ext3 ext2 xfs _supported_os Linux _require_scratch +_require_test_program "feature" + +ncpus=`$here/src/feature -o` rm -f $seqres.full echo "Silence is golden" @@ -68,19 +74,27 @@ echo "Silence is golden" _scratch_mkfs_sized $((1024 * 1024 * 1024)) >>$seqres.full 2>&1 _scratch_mount -i=0 free_inode=`_get_free_inode $SCRATCH_MNT` file_per_dir=1000 -loop=$((free_inode / file_per_dir + 1)) -mkdir -p $SCRATCH_MNT/testdir - -echo "Create $((loop * file_per_dir)) files in $SCRATCH_MNT/testdir" >>$seqres.full -while [ $i -lt $loop ]; do - create_file $SCRATCH_MNT/testdir $file_per_dir $i >>$seqres.full 2>&1 & - let i=$i+1 +num_dirs=$(( free_inode / (file_per_dir + 1) )) +num_threads=$(( ncpus * 2 )) +[ $num_threads -gt 20 ] && num_threads=20 +loop=$(( num_dirs / num_threads )) + +echo "Create $((loop * num_threads)) dirs and $file_per_dir files per dir in $SCRATCH_MNT" >>$seqres.full +for ((i=0; i>$seqres.full + create_file $SCRATCH_MNT/testdir_$i_$j $((free_inode - 1)) +fi + # log inode status in $seqres.full for debug purpose echo "Inode status after taking all inodes" >>$seqres.full $DF_PROG -i $SCRATCH_MNT >>$seqres.full