From patchwork Sat Jun 23 18:00:28 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zorro Lang X-Patchwork-Id: 10483809 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A695660532 for ; Sat, 23 Jun 2018 18:00:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A06DD28A75 for ; Sat, 23 Jun 2018 18:00:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 93E0828A91; Sat, 23 Jun 2018 18:00:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2271828A75 for ; Sat, 23 Jun 2018 18:00:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751734AbeFWSAm (ORCPT ); Sat, 23 Jun 2018 14:00:42 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47596 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751721AbeFWSAl (ORCPT ); Sat, 23 Jun 2018 14:00:41 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 49B2C8314C; Sat, 23 Jun 2018 18:00:41 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-12-42.pek2.redhat.com [10.72.12.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9BAA2111AF00; Sat, 23 Jun 2018 18:00:39 +0000 (UTC) From: Zorro Lang To: fstests@vger.kernel.org Cc: linux-xfs@vger.kernel.org Subject: [PATCH v3 2/3] xfstests: iterate dedupe integrity test Date: Sun, 24 Jun 2018 02:00:28 +0800 Message-Id: <20180623180029.24497-2-zlang@redhat.com> In-Reply-To: <20180623180029.24497-1-zlang@redhat.com> References: <20180623180029.24497-1-zlang@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Sat, 23 Jun 2018 18:00:41 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Sat, 23 Jun 2018 18:00:41 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'zlang@redhat.com' RCPT:'' Sender: fstests-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This case does dedupe on a dir, then copy the dir to next dir. Dedupe the next dir again, then copy this dir to next again, and dedupe again ... At the end, verify the data in the last dir is still same with the first one. Signed-off-by: Zorro Lang --- V3 update description about SPDX-License Thanks, Zorro tests/shared/009 | 104 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/shared/009.out | 4 ++ tests/shared/group | 1 + 3 files changed, 109 insertions(+) create mode 100755 tests/shared/009 create mode 100644 tests/shared/009.out diff --git a/tests/shared/009 b/tests/shared/009 new file mode 100755 index 00000000..933d3098 --- /dev/null +++ b/tests/shared/009 @@ -0,0 +1,104 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2018 Red Hat Inc. All Rights Reserved. +# +# FS QA Test 009 +# +# Iterate dedupe integrity test. Copy an original data0 several +# times (d0 -> d1, d1 -> d2, ... dn-1 -> dn), dedupe dataN everytime +# before copy. At last, verify dataN same with data0. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter +. ./common/reflink + +# remove previous $seqres.full before test +rm -f $seqres.full + +# real QA test starts here + +# duperemove only supports btrfs and xfs (with reflink feature). +# Add other filesystems if it supports more later. +_supported_fs xfs btrfs +_supported_os Linux +_require_scratch_dedupe +_require_command "$DUPEREMOVE_PROG" duperemove + +_scratch_mkfs > $seqres.full 2>&1 +_scratch_mount >> $seqres.full 2>&1 + +function iterate_dedup_verify() +{ + local src=$srcdir + local dest=$dupdir/1 + + for ((index = 1; index <= times; index++)); do + cp -a $src $dest + find $dest -type f -exec md5sum {} \; \ + > $md5file$index + # Make some noise + $FSSTRESS_PROG $fsstress_opts -d $noisedir \ + -n 200 -p $((5 * LOAD_FACTOR)) >/dev/null 2>&1 + # Too many output, so only save error output + $DUPEREMOVE_PROG -dr --dedupe-options=same $dupdir \ + >/dev/null 2>$seqres.full + md5sum -c --quiet $md5file$index + src=$dest + dest=$dupdir/$((index + 1)) + done +} + +srcdir=$SCRATCH_MNT/src +dupdir=$SCRATCH_MNT/dup +noisedir=$dupdir/noise +mkdir $srcdir $dupdir +mkdir $dupdir/noise + +md5file=${tmp}.md5sum + +fsstress_opts="-w -r" +# Create some files to be original data +$FSSTRESS_PROG $fsstress_opts -d $srcdir \ + -n 500 -p $((5 * LOAD_FACTOR)) >/dev/null 2>&1 + +# Calculate how many test cycles will be run +src_size=`du -ks $srcdir | awk '{print $1}'` +free_size=`df -kP $SCRATCH_MNT | grep -v Filesystem | awk '{print $4}'` +times=$((free_size / src_size)) +if [ $times -gt $((4 * TIME_FACTOR)) ]; then + times=$((4 * TIME_FACTOR)) +fi + +echo "= Do dedup and verify =" +iterate_dedup_verify + +# Use the last checksum file to verify the original data +sed -e s#dup/$times#src#g $md5file$times > $md5file +echo "= Backwords verify =" +md5sum -c --quiet $md5file + +# read from the disk also doesn't show mutations. +_scratch_cycle_mount +echo "= Verify after cycle mount =" +for ((index = 1; index <= times; index++)); do + md5sum -c --quiet $md5file$index +done + +status=0 +exit diff --git a/tests/shared/009.out b/tests/shared/009.out new file mode 100644 index 00000000..44a78ba3 --- /dev/null +++ b/tests/shared/009.out @@ -0,0 +1,4 @@ +QA output created by 009 += Do dedup and verify = += Backwords verify = += Verify after cycle mount = diff --git a/tests/shared/group b/tests/shared/group index 49ffa8dd..9c484794 100644 --- a/tests/shared/group +++ b/tests/shared/group @@ -11,6 +11,7 @@ 006 auto enospc 007 dangerous_fuzzers 008 auto stress dedupe +009 auto stress dedupe 032 mkfs auto quick 272 auto enospc rw 289 auto quick