| Message ID | 20191029055713.28191-1-cgxu519@mykernel.net (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | overlay/066: adjust test file size && add more test patterns | expand |
On Tue, Oct 29, 2019 at 7:57 AM Chengguang Xu <cgxu519@mykernel.net> wrote: > Can you please send the patch as plain/text. Your mailer has sent it with quoted printable encoding and git am fails to apply the patch: https://lore.kernel.org/fstests/20191029055713.28191-1-cgxu519@mykernel.net/raw > Make many small holes in 10M test file seems not very > helpful for test coverage and it takes too much time > on creating test files. So in order to improve test > speed we adjust test file size to (10 * iosize) for > iosize aligned hole files meanwhile add more test > patterns for small random holes and small empty file. > > Signed-off-by: Chengguang Xu <cgxu519@mykernel.net> > --- > tests/overlay/066 | 89 +++++++++++++++++++++++++++++++++++------------ > 1 file changed, 67 insertions(+), 22 deletions(-) > > diff --git a/tests/overlay/066 b/tests/overlay/066 > index 285a5aff..fb8f5e5c 100755 > --- a/tests/overlay/066 > +++ b/tests/overlay/066 > @@ -40,49 +40,82 @@ _require_scratch > # Remove all files from previous tests > _scratch_mkfs > > -# We have totally 14 test files in this test. > +# We have totally 16 test files in this test. > # The detail as below: > -# 1 empty file(10M) + 2^0(K)..2^11(K) hole size files(each 10M) + 1 random hole size file(100M). > +# 1 small empty file 4K > +# 1 big empty file 4M > +# 1 small random hole file 10M > +# 1 big random hole file 100M > +# > +# 12 files with variant iosize aligned holes. > +# 2^0(K)..2^11(K) hole size files(file size = 10 * iosize) > # > # Considering both upper and lower fs will fill zero when copy-up > # hole area in the file, this test at least requires double disk > # space of the sum of above test files' size. > > -_require_fs_space $OVL_BASE_SCRATCH_MNT $(((10*1024*13 + 100*1024*1) * 2)) > +_require_fs_space $OVL_BASE_SCRATCH_MNT $((((4) + (4096) + (10 * 1024) \ > + + (100 * 1024) + (10 * (1 + 2048) * 12 / 2)) * 2)) > > lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER > upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER > testfile="copyup_sparse_test" > > -# Create a completely empty hole file(10M). > -file_size=10240 > -$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_empty_holefile" \ > +# Create a small completely empty hole file(4K). > +file_size=4 > +$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_empty_small_holefile" \ > + >>$seqres.full > + > +# Create a big completely empty hole file(4M). > +file_size=4096 > +$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_empty_big_holefile" \ > >>$seqres.full > > -# Create 2^0(K)..2^11(K) hole size test files(each 10M). > +# Create 2^0(K)..2^11(K) hole size test files(file size = 10 * iosize). > # > # The pattern is like below, both hole and data are equal to > # iosize except last hole. > # > # |-- hole --|-- data --| ... |-- data --|-- hole --| > > -iosize=1 > +min_iosize=1 > max_iosize=2048 > -file_size=10240 > -max_pos=`expr $file_size - $max_iosize` > +iosize=$min_iosize > > while [ $iosize -le $max_iosize ]; do > + file_size=$(($iosize * 10)) > + max_pos=$(($file_size - $iosize)) > pos=$iosize > $XFS_IO_PROG -fc "truncate ${file_size}K" \ > "${lowerdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full > while [ $pos -lt $max_pos ]; do > $XFS_IO_PROG -fc "pwrite ${pos}K ${iosize}K" \ > "${lowerdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full > - pos=`expr $pos + $iosize + $iosize` > + pos=$(($pos + $iosize * 2)) > done > - iosize=`expr $iosize + $iosize` > + iosize=$(($iosize * 2)) > done > > +# Create test file with many random small holes(hole size is between 4K and 512K), > +# total file size is 10M. > + > +pos=4 > +max_pos=9216 > +file_size=10240 > +min_hole=4 > +max_hole=512 > + > +$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_random_small_holefile" \ > + >>$seqres.full > + > +while [ $pos -le $max_pos ]; do > + iosize=$(($RANDOM % ($max_hole - $min_hole) + $min_hole)) > + $XFS_IO_PROG -fc "pwrite ${pos}K ${iosize}K" \ > + "${lowerdir}/${testfile}_random_small_holefile" >>$seqres.full > + pos=$(($pos + $iosize * 2)) > +done > + > + > # Create test file with many random holes(hole size is between 1M and 5M), > # total file size is 100M. > > @@ -92,14 +125,14 @@ file_size=102400 > min_hole=1024 > max_hole=5120 > > -$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_random_holefile" \ > +$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_random_big_holefile" \ > >>$seqres.full > > while [ $pos -le $max_pos ]; do > iosize=$(($RANDOM % ($max_hole - $min_hole) + $min_hole)) > $XFS_IO_PROG -fc "pwrite ${pos}K ${iosize}K" \ > - "${lowerdir}/${testfile}_random_holefile" >>$seqres.full > - pos=`expr $pos + $iosize + $iosize` > + "${lowerdir}/${testfile}_random_big_holefile" >>$seqres.full > + pos=$(($pos + $iosize * 2)) > done > > _scratch_mount > @@ -112,18 +145,30 @@ done > echo "Silence is golden" > > # Check all copy-up files in upper layer. > -iosize=1 > -while [ $iosize -le 2048 ]; do > +min_iosize=1 > +max_iosize=2048 My intention was that you use those "constants" defined above when creating the files, not that you re-define them when verifying the files. > +iosize=$min_iosize > + > +while [ $iosize -le $max_iosize ]; do > diff "${lowerdir}/${testfile}_iosize${iosize}K_holefile" \ > "${upperdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full ||\ > echo "${upperdir}/${testfile}_iosize${iosize}K_holefile" copy up failed! > - iosize=`expr $iosize + $iosize` > + iosize=$(($iosize * 2)) > done > > -diff "${lowerdir}/${testfile}_empty_holefile" "${upperdir}/${testfile}_empty_holefile" \ > - >>$seqres.full || echo "${upperdir}/${testfile}_empty_holefile" copy up failed! > -diff "${lowerdir}/${testfile}_random_holefile" "${upperdir}/${testfile}_random_holefile" \ > - >>$seqres.full || echo "${upperdir}/${testfile}_random_holefile" copy up failed! > +# Check empty hole files > +diff "${lowerdir}/${testfile}_empty_small_holefile" "${upperdir}/${testfile}_empty_small_holefile" \ > + >>$seqres.full || echo "${upperdir}/${testfile}_empty_small_holefile" copy up failed! > + > +diff "${lowerdir}/${testfile}_empty_big_holefile" "${upperdir}/${testfile}_empty_big_holefile" \ > + >>$seqres.full || echo "${upperdir}/${testfile}_empty_big_holefile" copy up failed! > + > +# Check random hole files > +diff "${lowerdir}/${testfile}_random_small_holefile" "${upperdir}/${testfile}_random_small_holefile" \ > + >>$seqres.full || echo "${upperdir}/${testfile}_random_small_holefile" copy up failed! > + > +diff "${lowerdir}/${testfile}_random_big_holefile" "${upperdir}/${testfile}_random_big_holefile" \ > + >>$seqres.full || echo "${upperdir}/${testfile}_random_big_holefile" copy up failed! > This would be much nicer with for name in empty_small empty_big random_small random_big; do ... And I would still like to test the patch before I ACK it, so please send the patch as attachment or re-send in plain text encoding. Thanks, Amir.
---- 在 星期二, 2019-10-29 16:32:32 Amir Goldstein <amir73il@gmail.com> 撰写 ---- > On Tue, Oct 29, 2019 at 7:57 AM Chengguang Xu <cgxu519@mykernel.net> wrote: > > > > Can you please send the patch as plain/text. > Your mailer has sent it with quoted printable encoding and git am > fails to apply the patch: > https://lore.kernel.org/fstests/20191029055713.28191-1-cgxu519@mykernel.net/raw > Sorry for that, I'm not clear for the reason, so I send you the patch in attachment first. > > Make many small holes in 10M test file seems not very > > helpful for test coverage and it takes too much time > > on creating test files. So in order to improve test > > speed we adjust test file size to (10 * iosize) for > > iosize aligned hole files meanwhile add more test > > patterns for small random holes and small empty file. > > > > Signed-off-by: Chengguang Xu <cgxu519@mykernel.net> > > --- > > tests/overlay/066 | 89 +++++++++++++++++++++++++++++++++++------------ > > 1 file changed, 67 insertions(+), 22 deletions(-) > > > > diff --git a/tests/overlay/066 b/tests/overlay/066 > > index 285a5aff..fb8f5e5c 100755 > > --- a/tests/overlay/066 > > +++ b/tests/overlay/066 > > @@ -40,49 +40,82 @@ _require_scratch > > # Remove all files from previous tests > > _scratch_mkfs > > > > -# We have totally 14 test files in this test. > > +# We have totally 16 test files in this test. > > # The detail as below: > > -# 1 empty file(10M) + 2^0(K)..2^11(K) hole size files(each 10M) + 1 random hole size file(100M). > > +# 1 small empty file 4K > > +# 1 big empty file 4M > > +# 1 small random hole file 10M > > +# 1 big random hole file 100M > > +# > > +# 12 files with variant iosize aligned holes. > > +# 2^0(K)..2^11(K) hole size files(file size = 10 * iosize) > > # > > # Considering both upper and lower fs will fill zero when copy-up > > # hole area in the file, this test at least requires double disk > > # space of the sum of above test files' size. > > > > -_require_fs_space $OVL_BASE_SCRATCH_MNT $(((10*1024*13 + 100*1024*1) * 2)) > > +_require_fs_space $OVL_BASE_SCRATCH_MNT $((((4) + (4096) + (10 * 1024) \ > > + + (100 * 1024) + (10 * (1 + 2048) * 12 / 2)) * 2)) > > > > lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER > > upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER > > testfile="copyup_sparse_test" > > > > -# Create a completely empty hole file(10M). > > -file_size=10240 > > -$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_empty_holefile" \ > > +# Create a small completely empty hole file(4K). > > +file_size=4 > > +$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_empty_small_holefile" \ > > + >>$seqres.full > > + > > +# Create a big completely empty hole file(4M). > > +file_size=4096 > > +$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_empty_big_holefile" \ > > >>$seqres.full > > > > -# Create 2^0(K)..2^11(K) hole size test files(each 10M). > > +# Create 2^0(K)..2^11(K) hole size test files(file size = 10 * iosize). > > # > > # The pattern is like below, both hole and data are equal to > > # iosize except last hole. > > # > > # |-- hole --|-- data --| ... |-- data --|-- hole --| > > > > -iosize=1 > > +min_iosize=1 > > max_iosize=2048 > > -file_size=10240 > > -max_pos=`expr $file_size - $max_iosize` > > +iosize=$min_iosize > > > > while [ $iosize -le $max_iosize ]; do > > + file_size=$(($iosize * 10)) > > + max_pos=$(($file_size - $iosize)) > > pos=$iosize > > $XFS_IO_PROG -fc "truncate ${file_size}K" \ > > "${lowerdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full > > while [ $pos -lt $max_pos ]; do > > $XFS_IO_PROG -fc "pwrite ${pos}K ${iosize}K" \ > > "${lowerdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full > > - pos=`expr $pos + $iosize + $iosize` > > + pos=$(($pos + $iosize * 2)) > > done > > - iosize=`expr $iosize + $iosize` > > + iosize=$(($iosize * 2)) > > done > > > > +# Create test file with many random small holes(hole size is between 4K and 512K), > > +# total file size is 10M. > > + > > +pos=4 > > +max_pos=9216 > > +file_size=10240 > > +min_hole=4 > > +max_hole=512 > > + > > +$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_random_small_holefile" \ > > + >>$seqres.full > > + > > +while [ $pos -le $max_pos ]; do > > + iosize=$(($RANDOM % ($max_hole - $min_hole) + $min_hole)) > > + $XFS_IO_PROG -fc "pwrite ${pos}K ${iosize}K" \ > > + "${lowerdir}/${testfile}_random_small_holefile" >>$seqres.full > > + pos=$(($pos + $iosize * 2)) > > +done > > + > > + > > # Create test file with many random holes(hole size is between 1M and 5M), > > # total file size is 100M. > > > > @@ -92,14 +125,14 @@ file_size=102400 > > min_hole=1024 > > max_hole=5120 > > > > -$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_random_holefile" \ > > +$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_random_big_holefile" \ > > >>$seqres.full > > > > while [ $pos -le $max_pos ]; do > > iosize=$(($RANDOM % ($max_hole - $min_hole) + $min_hole)) > > $XFS_IO_PROG -fc "pwrite ${pos}K ${iosize}K" \ > > - "${lowerdir}/${testfile}_random_holefile" >>$seqres.full > > - pos=`expr $pos + $iosize + $iosize` > > + "${lowerdir}/${testfile}_random_big_holefile" >>$seqres.full > > + pos=$(($pos + $iosize * 2)) > > done > > > > _scratch_mount > > @@ -112,18 +145,30 @@ done > > echo "Silence is golden" > > > > # Check all copy-up files in upper layer. > > -iosize=1 > > -while [ $iosize -le 2048 ]; do > > +min_iosize=1 > > +max_iosize=2048 > > My intention was that you use those "constants" defined above when creating > the files, not that you re-define them when verifying the files. Maybe there is a risk that iosize is changed unexpectedly between creating first test files and before verifying them in the future. Now I think better solution is just compare all test files in lower/upper dir by one diff command. > > > +iosize=$min_iosize > > + > > +while [ $iosize -le $max_iosize ]; do > > diff "${lowerdir}/${testfile}_iosize${iosize}K_holefile" \ > > "${upperdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full ||\ > > echo "${upperdir}/${testfile}_iosize${iosize}K_holefile" copy up failed! > > - iosize=`expr $iosize + $iosize` > > + iosize=$(($iosize * 2)) > > done > > > > -diff "${lowerdir}/${testfile}_empty_holefile" "${upperdir}/${testfile}_empty_holefile" \ > > - >>$seqres.full || echo "${upperdir}/${testfile}_empty_holefile" copy up failed! > > -diff "${lowerdir}/${testfile}_random_holefile" "${upperdir}/${testfile}_random_holefile" \ > > - >>$seqres.full || echo "${upperdir}/${testfile}_random_holefile" copy up failed! > > +# Check empty hole files > > +diff "${lowerdir}/${testfile}_empty_small_holefile" "${upperdir}/${testfile}_empty_small_holefile" \ > > + >>$seqres.full || echo "${upperdir}/${testfile}_empty_small_holefile" copy up failed! > > + > > +diff "${lowerdir}/${testfile}_empty_big_holefile" "${upperdir}/${testfile}_empty_big_holefile" \ > > + >>$seqres.full || echo "${upperdir}/${testfile}_empty_big_holefile" copy up failed! > > + > > +# Check random hole files > > +diff "${lowerdir}/${testfile}_random_small_holefile" "${upperdir}/${testfile}_random_small_holefile" \ > > + >>$seqres.full || echo "${upperdir}/${testfile}_random_small_holefile" copy up failed! > > + > > +diff "${lowerdir}/${testfile}_random_big_holefile" "${upperdir}/${testfile}_random_big_holefile" \ > > + >>$seqres.full || echo "${upperdir}/${testfile}_random_big_holefile" copy up failed! > > > > This would be much nicer with > > for name in empty_small empty_big random_small random_big; do > ... > > > And I would still like to test the patch before I ACK it, so please > send the patch > as attachment or re-send in plain text encoding. > > Thanks, > Amir. >
On Tue, Oct 29, 2019 at 1:17 PM Chengguang Xu <cgxu519@mykernel.net> wrote: > > ---- 在 星期二, 2019-10-29 16:32:32 Amir Goldstein <amir73il@gmail.com> 撰写 ---- > > On Tue, Oct 29, 2019 at 7:57 AM Chengguang Xu <cgxu519@mykernel.net> wrote: > > > > > > > Can you please send the patch as plain/text. > > Your mailer has sent it with quoted printable encoding and git am > > fails to apply the patch: > > https://lore.kernel.org/fstests/20191029055713.28191-1-cgxu519@mykernel.net/raw > > > > Sorry for that, I'm not clear for the reason, so I send you the patch in attachment first. > FWIW I am using git-send-email... > > > > Make many small holes in 10M test file seems not very > > > helpful for test coverage and it takes too much time > > > on creating test files. So in order to improve test > > > speed we adjust test file size to (10 * iosize) for > > > iosize aligned hole files meanwhile add more test > > > patterns for small random holes and small empty file. > > > > > > Signed-off-by: Chengguang Xu <cgxu519@mykernel.net> > > > --- > > > tests/overlay/066 | 89 +++++++++++++++++++++++++++++++++++------------ > > > 1 file changed, 67 insertions(+), 22 deletions(-) > > > > > > diff --git a/tests/overlay/066 b/tests/overlay/066 > > > index 285a5aff..fb8f5e5c 100755 > > > --- a/tests/overlay/066 > > > +++ b/tests/overlay/066 > > > @@ -40,49 +40,82 @@ _require_scratch > > > # Remove all files from previous tests > > > _scratch_mkfs > > > > > > -# We have totally 14 test files in this test. > > > +# We have totally 16 test files in this test. > > > # The detail as below: > > > -# 1 empty file(10M) + 2^0(K)..2^11(K) hole size files(each 10M) + 1 random hole size file(100M). > > > +# 1 small empty file 4K > > > +# 1 big empty file 4M > > > +# 1 small random hole file 10M > > > +# 1 big random hole file 100M > > > +# > > > +# 12 files with variant iosize aligned holes. > > > +# 2^0(K)..2^11(K) hole size files(file size = 10 * iosize) > > > # > > > # Considering both upper and lower fs will fill zero when copy-up > > > # hole area in the file, this test at least requires double disk > > > # space of the sum of above test files' size. > > > > > > -_require_fs_space $OVL_BASE_SCRATCH_MNT $(((10*1024*13 + 100*1024*1) * 2)) > > > +_require_fs_space $OVL_BASE_SCRATCH_MNT $((((4) + (4096) + (10 * 1024) \ > > > + + (100 * 1024) + (10 * (1 + 2048) * 12 / 2)) * 2)) > > > > > > lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER > > > upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER > > > testfile="copyup_sparse_test" > > > > > > -# Create a completely empty hole file(10M). > > > -file_size=10240 > > > -$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_empty_holefile" \ > > > +# Create a small completely empty hole file(4K). > > > +file_size=4 > > > +$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_empty_small_holefile" \ > > > + >>$seqres.full > > > + > > > +# Create a big completely empty hole file(4M). > > > +file_size=4096 > > > +$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_empty_big_holefile" \ > > > >>$seqres.full > > > > > > -# Create 2^0(K)..2^11(K) hole size test files(each 10M). > > > +# Create 2^0(K)..2^11(K) hole size test files(file size = 10 * iosize). > > > # > > > # The pattern is like below, both hole and data are equal to > > > # iosize except last hole. > > > # > > > # |-- hole --|-- data --| ... |-- data --|-- hole --| > > > > > > -iosize=1 > > > +min_iosize=1 > > > max_iosize=2048 > > > -file_size=10240 > > > -max_pos=`expr $file_size - $max_iosize` > > > +iosize=$min_iosize > > > > > > while [ $iosize -le $max_iosize ]; do > > > + file_size=$(($iosize * 10)) > > > + max_pos=$(($file_size - $iosize)) > > > pos=$iosize > > > $XFS_IO_PROG -fc "truncate ${file_size}K" \ > > > "${lowerdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full > > > while [ $pos -lt $max_pos ]; do > > > $XFS_IO_PROG -fc "pwrite ${pos}K ${iosize}K" \ > > > "${lowerdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full > > > - pos=`expr $pos + $iosize + $iosize` > > > + pos=$(($pos + $iosize * 2)) > > > done > > > - iosize=`expr $iosize + $iosize` > > > + iosize=$(($iosize * 2)) > > > done > > > > > > +# Create test file with many random small holes(hole size is between 4K and 512K), > > > +# total file size is 10M. > > > + > > > +pos=4 > > > +max_pos=9216 > > > +file_size=10240 > > > +min_hole=4 > > > +max_hole=512 > > > + > > > +$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_random_small_holefile" \ > > > + >>$seqres.full > > > + > > > +while [ $pos -le $max_pos ]; do > > > + iosize=$(($RANDOM % ($max_hole - $min_hole) + $min_hole)) > > > + $XFS_IO_PROG -fc "pwrite ${pos}K ${iosize}K" \ > > > + "${lowerdir}/${testfile}_random_small_holefile" >>$seqres.full > > > + pos=$(($pos + $iosize * 2)) > > > +done > > > + > > > + > > > # Create test file with many random holes(hole size is between 1M and 5M), > > > # total file size is 100M. > > > > > > @@ -92,14 +125,14 @@ file_size=102400 > > > min_hole=1024 > > > max_hole=5120 > > > > > > -$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_random_holefile" \ > > > +$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_random_big_holefile" \ > > > >>$seqres.full > > > > > > while [ $pos -le $max_pos ]; do > > > iosize=$(($RANDOM % ($max_hole - $min_hole) + $min_hole)) > > > $XFS_IO_PROG -fc "pwrite ${pos}K ${iosize}K" \ > > > - "${lowerdir}/${testfile}_random_holefile" >>$seqres.full > > > - pos=`expr $pos + $iosize + $iosize` > > > + "${lowerdir}/${testfile}_random_big_holefile" >>$seqres.full > > > + pos=$(($pos + $iosize * 2)) > > > done > > > > > > _scratch_mount > > > @@ -112,18 +145,30 @@ done > > > echo "Silence is golden" > > > > > > # Check all copy-up files in upper layer. > > > -iosize=1 > > > -while [ $iosize -le 2048 ]; do > > > +min_iosize=1 > > > +max_iosize=2048 > > > > My intention was that you use those "constants" defined above when creating > > the files, not that you re-define them when verifying the files. > > Maybe there is a risk that iosize is changed unexpectedly between creating first test files > and before verifying them in the future. Now I think better solution is just compare all > test files in lower/upper dir by one diff command. > Agreed. diff -qr "${lowerdir}/" "${upperdir}/" Would be enough and provide enough details in 066.bad should the diff fail Thanks, Amir.
On Tue, Oct 29, 2019 at 1:58 PM Amir Goldstein <amir73il@gmail.com> wrote: > > On Tue, Oct 29, 2019 at 1:17 PM Chengguang Xu <cgxu519@mykernel.net> wrote: > > > > ---- 在 星期二, 2019-10-29 16:32:32 Amir Goldstein <amir73il@gmail.com> 撰写 ---- > > > On Tue, Oct 29, 2019 at 7:57 AM Chengguang Xu <cgxu519@mykernel.net> wrote: > > > > > > > > > > Can you please send the patch as plain/text. > > > Your mailer has sent it with quoted printable encoding and git am > > > fails to apply the patch: > > > https://lore.kernel.org/fstests/20191029055713.28191-1-cgxu519@mykernel.net/raw > > > > > > > Sorry for that, I'm not clear for the reason, so I send you the patch in attachment first. > > > OK, I can verify that test runs quick (5s) on my VM. But there is one more issue that I think needs to be addressed, either in this fix patch or in a follow up patch. If the test ever fails on some run with a specific random holes sequence, it is going to be quite hard for reporter to report this sequence or for developers to reproduce the same random sequence. One way to fix this is to output the sequence of commands that generated the random files to full output, so it can be used as a recipe for reproducer in case of a failure. If you do that, please make sure that the recipe for creating small random file is clearly separated from recipe for creating big random file. Thanks, Amir.
---- 在 星期二, 2019-10-29 20:32:43 Amir Goldstein <amir73il@gmail.com> 撰写 ---- > On Tue, Oct 29, 2019 at 1:58 PM Amir Goldstein <amir73il@gmail.com> wrote: > > > > On Tue, Oct 29, 2019 at 1:17 PM Chengguang Xu <cgxu519@mykernel.net> wrote: > > > > > > ---- 在 星期二, 2019-10-29 16:32:32 Amir Goldstein <amir73il@gmail.com> 撰写 ---- > > > > On Tue, Oct 29, 2019 at 7:57 AM Chengguang Xu <cgxu519@mykernel.net> wrote: > > > > > > > > > > > > > Can you please send the patch as plain/text. > > > > Your mailer has sent it with quoted printable encoding and git am > > > > fails to apply the patch: > > > > https://lore.kernel.org/fstests/20191029055713.28191-1-cgxu519@mykernel.net/raw > > > > > > > > > > Sorry for that, I'm not clear for the reason, so I send you the patch in attachment first. > > > > > > > OK, I can verify that test runs quick (5s) on my VM. > > But there is one more issue that I think needs to be addressed, either > in this fix patch or in a follow up patch. > > If the test ever fails on some run with a specific random holes sequence, > it is going to be quite hard for reporter to report this sequence or for > developers to reproduce the same random sequence. IMO, it's not so hard as you thought, I prefer to use filefrag to check it. I think below tidy info is very clear and easy to understand what had happened. [root@hades ovl-lower]# filefrag -k -e copyup_sparse_test_random_small_holefile Filesystem type is: 58465342 File size of copyup_sparse_test_random_small_holefile is 10485760 (10240 blocks of 1024 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 4.. 411: 2625148.. 2625555: 408: 4: 1: 816.. 1259: 2626172.. 2626615: 444: 2625960: 2: 1696.. 1783: 2627196.. 2627283: 88: 2627052: 3: 1872.. 2207: 2627372.. 2627707: 336: 4: 2544.. 3019: 2629244.. 2629719: 476: 2628044: 5: 3496.. 3599: 2629720.. 2629823: 104: 6: 3704.. 3819: 2629928.. 2630043: 116: 7: 3936.. 3959: 2630044.. 2630067: 24: 8: 3980.. 4487: 2631292.. 2631799: 508: 2630088: 9: 4992.. 5235: 2631800.. 2632043: 244: 10: 5472.. 5715: 2632044.. 2632287: 244: 11: 5956.. 6355: 2633340.. 2633739: 400: 2632528: 12: 6752.. 6787: 2633740.. 2633775: 36: 13: 6820.. 6907: 2633808.. 2633895: 88: 14: 6996.. 7447: 2633896.. 2634347: 452: 15: 7900.. 8211: 2637436.. 2637747: 312: 2634800: 16: 8516.. 8867: 2638052.. 2638403: 352: 17: 9216.. 9703: 2638752.. 2639239: 488: last copyup_sparse_test_random_small_holefile: 7 extents found Thanks, Chengguang > > One way to fix this is to output the sequence of commands that > generated the random files to full output, so it can be used as a recipe > for reproducer in case of a failure. > > If you do that, please make sure that the recipe for creating > small random file is clearly separated from recipe for creating > big random file. > > Thanks, > Amir. >
On Wed, Oct 30, 2019 at 6:46 AM Chengguang Xu <cgxu519@mykernel.net> wrote: > > ---- 在 星期二, 2019-10-29 20:32:43 Amir Goldstein <amir73il@gmail.com> 撰写 ---- > > On Tue, Oct 29, 2019 at 1:58 PM Amir Goldstein <amir73il@gmail.com> wrote: > > > > > > On Tue, Oct 29, 2019 at 1:17 PM Chengguang Xu <cgxu519@mykernel.net> wrote: > > > > > > > > ---- 在 星期二, 2019-10-29 16:32:32 Amir Goldstein <amir73il@gmail.com> 撰写 ---- > > > > > On Tue, Oct 29, 2019 at 7:57 AM Chengguang Xu <cgxu519@mykernel.net> wrote: > > > > > > > > > > > > > > > > Can you please send the patch as plain/text. > > > > > Your mailer has sent it with quoted printable encoding and git am > > > > > fails to apply the patch: > > > > > https://lore.kernel.org/fstests/20191029055713.28191-1-cgxu519@mykernel.net/raw > > > > > > > > > > > > > Sorry for that, I'm not clear for the reason, so I send you the patch in attachment first. > > > > > > > > > > > OK, I can verify that test runs quick (5s) on my VM. > > > > But there is one more issue that I think needs to be addressed, either > > in this fix patch or in a follow up patch. > > > > If the test ever fails on some run with a specific random holes sequence, > > it is going to be quite hard for reporter to report this sequence or for > > developers to reproduce the same random sequence. > > IMO, it's not so hard as you thought, I prefer to use filefrag to check it. > > I think below tidy info is very clear and easy to understand what had happened. > > [root@hades ovl-lower]# filefrag -k -e copyup_sparse_test_random_small_holefile > Filesystem type is: 58465342 > File size of copyup_sparse_test_random_small_holefile is 10485760 (10240 blocks of 1024 bytes) > ext: logical_offset: physical_offset: length: expected: flags: > 0: 4.. 411: 2625148.. 2625555: 408: 4: > 1: 816.. 1259: 2626172.. 2626615: 444: 2625960: > 2: 1696.. 1783: 2627196.. 2627283: 88: 2627052: > 3: 1872.. 2207: 2627372.. 2627707: 336: > 4: 2544.. 3019: 2629244.. 2629719: 476: 2628044: > 5: 3496.. 3599: 2629720.. 2629823: 104: > 6: 3704.. 3819: 2629928.. 2630043: 116: > 7: 3936.. 3959: 2630044.. 2630067: 24: > 8: 3980.. 4487: 2631292.. 2631799: 508: 2630088: > 9: 4992.. 5235: 2631800.. 2632043: 244: > 10: 5472.. 5715: 2632044.. 2632287: 244: > 11: 5956.. 6355: 2633340.. 2633739: 400: 2632528: > 12: 6752.. 6787: 2633740.. 2633775: 36: > 13: 6820.. 6907: 2633808.. 2633895: 88: > 14: 6996.. 7447: 2633896.. 2634347: 452: > 15: 7900.. 8211: 2637436.. 2637747: 312: 2634800: > 16: 8516.. 8867: 2638052.. 2638403: 352: > 17: 9216.. 9703: 2638752.. 2639239: 488: last > copyup_sparse_test_random_small_holefile: 7 extents found > There is a difference between understanding what happened and reproducing, but there is no reason to choose one method over the other. As a developer, when I get a bug report I would rather have both an easy reproducer and all the postmortem information available. Therefore, please echo xfs_io commands, at least for creation of random files to full log AND filefrag info, at least for the random files to full log. Thanks, Amir.
---- 在 星期三, 2019-10-30 13:33:59 Amir Goldstein <amir73il@gmail.com> 撰写 ---- > On Wed, Oct 30, 2019 at 6:46 AM Chengguang Xu <cgxu519@mykernel.net> wrote: > > > > ---- 在 星期二, 2019-10-29 20:32:43 Amir Goldstein <amir73il@gmail.com> 撰写 ---- > > > On Tue, Oct 29, 2019 at 1:58 PM Amir Goldstein <amir73il@gmail.com> wrote: > > > > > > > > On Tue, Oct 29, 2019 at 1:17 PM Chengguang Xu <cgxu519@mykernel.net> wrote: > > > > > > > > > > ---- 在 星期二, 2019-10-29 16:32:32 Amir Goldstein <amir73il@gmail.com> 撰写 ---- > > > > > > On Tue, Oct 29, 2019 at 7:57 AM Chengguang Xu <cgxu519@mykernel.net> wrote: > > > > > > > > > > > > > > > > > > > Can you please send the patch as plain/text. > > > > > > Your mailer has sent it with quoted printable encoding and git am > > > > > > fails to apply the patch: > > > > > > https://lore.kernel.org/fstests/20191029055713.28191-1-cgxu519@mykernel.net/raw > > > > > > > > > > > > > > > > Sorry for that, I'm not clear for the reason, so I send you the patch in attachment first. > > > > > > > > > > > > > > > OK, I can verify that test runs quick (5s) on my VM. > > > > > > But there is one more issue that I think needs to be addressed, either > > > in this fix patch or in a follow up patch. > > > > > > If the test ever fails on some run with a specific random holes sequence, > > > it is going to be quite hard for reporter to report this sequence or for > > > developers to reproduce the same random sequence. > > > > IMO, it's not so hard as you thought, I prefer to use filefrag to check it. > > > > I think below tidy info is very clear and easy to understand what had happened. > > > > [root@hades ovl-lower]# filefrag -k -e copyup_sparse_test_random_small_holefile > > Filesystem type is: 58465342 > > File size of copyup_sparse_test_random_small_holefile is 10485760 (10240 blocks of 1024 bytes) > > ext: logical_offset: physical_offset: length: expected: flags: > > 0: 4.. 411: 2625148.. 2625555: 408: 4: > > 1: 816.. 1259: 2626172.. 2626615: 444: 2625960: > > 2: 1696.. 1783: 2627196.. 2627283: 88: 2627052: > > 3: 1872.. 2207: 2627372.. 2627707: 336: > > 4: 2544.. 3019: 2629244.. 2629719: 476: 2628044: > > 5: 3496.. 3599: 2629720.. 2629823: 104: > > 6: 3704.. 3819: 2629928.. 2630043: 116: > > 7: 3936.. 3959: 2630044.. 2630067: 24: > > 8: 3980.. 4487: 2631292.. 2631799: 508: 2630088: > > 9: 4992.. 5235: 2631800.. 2632043: 244: > > 10: 5472.. 5715: 2632044.. 2632287: 244: > > 11: 5956.. 6355: 2633340.. 2633739: 400: 2632528: > > 12: 6752.. 6787: 2633740.. 2633775: 36: > > 13: 6820.. 6907: 2633808.. 2633895: 88: > > 14: 6996.. 7447: 2633896.. 2634347: 452: > > 15: 7900.. 8211: 2637436.. 2637747: 312: 2634800: > > 16: 8516.. 8867: 2638052.. 2638403: 352: > > 17: 9216.. 9703: 2638752.. 2639239: 488: last > > copyup_sparse_test_random_small_holefile: 7 extents found > > > > There is a difference between understanding what happened and > reproducing, but there is no reason to choose one method over > the other. > > As a developer, when I get a bug report I would rather have both > an easy reproducer and all the postmortem information available. > Therefore, please echo xfs_io commands, at least for creation of > random files to full log AND filefrag info, at least for the random > files to full log. > Actually, xfs_io itself will leave detail information for write operation (pos+write size) See below, IMO, it is almost no difference compare to echo xfs_io command. So I just added title for those write scenarios in v2. --- iosize=2048K hole test write scenarios --- (This is what I added in v2) wrote 2097152/2097152 bytes at offset 2097152 2 MiB, 512 ops; 0.0007 sec (2.732 GiB/sec and 716083.9161 ops/sec) wrote 2097152/2097152 bytes at offset 6291456 2 MiB, 512 ops; 0.0006 sec (2.889 GiB/sec and 757396.4497 ops/sec) wrote 2097152/2097152 bytes at offset 10485760 2 MiB, 512 ops; 0.0007 sec (2.728 GiB/sec and 715083.7989 ops/sec) wrote 2097152/2097152 bytes at offset 14680064 2 MiB, 512 ops; 0.0007 sec (2.778 GiB/sec and 728307.2546 ops/sec) Thanks, Chengguang
> > There is a difference between understanding what happened and > > reproducing, but there is no reason to choose one method over > > the other. > > > > As a developer, when I get a bug report I would rather have both > > an easy reproducer and all the postmortem information available. > > Therefore, please echo xfs_io commands, at least for creation of > > random files to full log AND filefrag info, at least for the random > > files to full log. > > > > Actually, xfs_io itself will leave detail information for write operation (pos+write size) > See below, IMO, it is almost no difference compare to echo xfs_io command. > So I just added title for those write scenarios in v2. > > --- > iosize=2048K hole test write scenarios --- (This is what I added in v2) > > wrote 2097152/2097152 bytes at offset 2097152 > 2 MiB, 512 ops; 0.0007 sec (2.732 GiB/sec and 716083.9161 ops/sec) > wrote 2097152/2097152 bytes at offset 6291456 > 2 MiB, 512 ops; 0.0006 sec (2.889 GiB/sec and 757396.4497 ops/sec) > wrote 2097152/2097152 bytes at offset 10485760 > 2 MiB, 512 ops; 0.0007 sec (2.728 GiB/sec and 715083.7989 ops/sec) > wrote 2097152/2097152 bytes at offset 14680064 > 2 MiB, 512 ops; 0.0007 sec (2.778 GiB/sec and 728307.2546 ops/sec) > It's good that you added the titles, but not enough IMO. It is a social engineering issue, not a technical one. It is *nicer* for a test to provide a reproducer, than to provide information that needs to be converted to a reproducer by a developer. And the main reason to be *nicer* in this case, is that it is zero effort for the test writer to provide the report in the form of reproducer, simply by echoing the xfs_io commands via a helper, (e.g. do_io). Please understand that most of xfstests are reproducers by themselves. Some random tests (like fsx/fsstress) leave a recorded log of operations to be used to reproduce a failure. Some random xfstests log the random seed to log, so it can be used to reproduce. In any case, leaving a *simple* one to reproduce a failure is essential. Thanks, Amir.
diff --git a/tests/overlay/066 b/tests/overlay/066 index 285a5aff..fb8f5e5c 100755 --- a/tests/overlay/066 +++ b/tests/overlay/066 @@ -40,49 +40,82 @@ _require_scratch # Remove all files from previous tests _scratch_mkfs -# We have totally 14 test files in this test. +# We have totally 16 test files in this test. # The detail as below: -# 1 empty file(10M) + 2^0(K)..2^11(K) hole size files(each 10M) + 1 random hole size file(100M). +# 1 small empty file 4K +# 1 big empty file 4M +# 1 small random hole file 10M +# 1 big random hole file 100M +# +# 12 files with variant iosize aligned holes. +# 2^0(K)..2^11(K) hole size files(file size = 10 * iosize) # # Considering both upper and lower fs will fill zero when copy-up # hole area in the file, this test at least requires double disk # space of the sum of above test files' size. -_require_fs_space $OVL_BASE_SCRATCH_MNT $(((10*1024*13 + 100*1024*1) * 2)) +_require_fs_space $OVL_BASE_SCRATCH_MNT $((((4) + (4096) + (10 * 1024) \ + + (100 * 1024) + (10 * (1 + 2048) * 12 / 2)) * 2)) lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER testfile="copyup_sparse_test" -# Create a completely empty hole file(10M). -file_size=10240 -$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_empty_holefile" \ +# Create a small completely empty hole file(4K). +file_size=4 +$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_empty_small_holefile" \ + >>$seqres.full + +# Create a big completely empty hole file(4M). +file_size=4096 +$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_empty_big_holefile" \ >>$seqres.full -# Create 2^0(K)..2^11(K) hole size test files(each 10M). +# Create 2^0(K)..2^11(K) hole size test files(file size = 10 * iosize). # # The pattern is like below, both hole and data are equal to # iosize except last hole. # # |-- hole --|-- data --| ... |-- data --|-- hole --| -iosize=1 +min_iosize=1 max_iosize=2048 -file_size=10240 -max_pos=`expr $file_size - $max_iosize` +iosize=$min_iosize while [ $iosize -le $max_iosize ]; do + file_size=$(($iosize * 10)) + max_pos=$(($file_size - $iosize)) pos=$iosize $XFS_IO_PROG -fc "truncate ${file_size}K" \ "${lowerdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full while [ $pos -lt $max_pos ]; do $XFS_IO_PROG -fc "pwrite ${pos}K ${iosize}K" \ "${lowerdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full - pos=`expr $pos + $iosize + $iosize` + pos=$(($pos + $iosize * 2)) done - iosize=`expr $iosize + $iosize` + iosize=$(($iosize * 2)) done +# Create test file with many random small holes(hole size is between 4K and 512K), +# total file size is 10M. + +pos=4 +max_pos=9216 +file_size=10240 +min_hole=4 +max_hole=512 + +$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_random_small_holefile" \ + >>$seqres.full + +while [ $pos -le $max_pos ]; do + iosize=$(($RANDOM % ($max_hole - $min_hole) + $min_hole)) + $XFS_IO_PROG -fc "pwrite ${pos}K ${iosize}K" \ + "${lowerdir}/${testfile}_random_small_holefile" >>$seqres.full + pos=$(($pos + $iosize * 2)) +done + + # Create test file with many random holes(hole size is between 1M and 5M), # total file size is 100M. @@ -92,14 +125,14 @@ file_size=102400 min_hole=1024 max_hole=5120 -$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_random_holefile" \ +$XFS_IO_PROG -fc "truncate ${file_size}K" "${lowerdir}/${testfile}_random_big_holefile" \ >>$seqres.full while [ $pos -le $max_pos ]; do iosize=$(($RANDOM % ($max_hole - $min_hole) + $min_hole)) $XFS_IO_PROG -fc "pwrite ${pos}K ${iosize}K" \ - "${lowerdir}/${testfile}_random_holefile" >>$seqres.full - pos=`expr $pos + $iosize + $iosize` + "${lowerdir}/${testfile}_random_big_holefile" >>$seqres.full + pos=$(($pos + $iosize * 2)) done _scratch_mount @@ -112,18 +145,30 @@ done echo "Silence is golden" # Check all copy-up files in upper layer. -iosize=1 -while [ $iosize -le 2048 ]; do +min_iosize=1 +max_iosize=2048 +iosize=$min_iosize + +while [ $iosize -le $max_iosize ]; do diff "${lowerdir}/${testfile}_iosize${iosize}K_holefile" \ "${upperdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full ||\ echo "${upperdir}/${testfile}_iosize${iosize}K_holefile" copy up failed! - iosize=`expr $iosize + $iosize` + iosize=$(($iosize * 2)) done -diff "${lowerdir}/${testfile}_empty_holefile" "${upperdir}/${testfile}_empty_holefile" \ - >>$seqres.full || echo "${upperdir}/${testfile}_empty_holefile" copy up failed! -diff "${lowerdir}/${testfile}_random_holefile" "${upperdir}/${testfile}_random_holefile" \ - >>$seqres.full || echo "${upperdir}/${testfile}_random_holefile" copy up failed! +# Check empty hole files +diff "${lowerdir}/${testfile}_empty_small_holefile" "${upperdir}/${testfile}_empty_small_holefile" \ + >>$seqres.full || echo "${upperdir}/${testfile}_empty_small_holefile" copy up failed! + +diff "${lowerdir}/${testfile}_empty_big_holefile" "${upperdir}/${testfile}_empty_big_holefile" \ + >>$seqres.full || echo "${upperdir}/${testfile}_empty_big_holefile" copy up failed! + +# Check random hole files +diff "${lowerdir}/${testfile}_random_small_holefile" "${upperdir}/${testfile}_random_small_holefile" \ + >>$seqres.full || echo "${upperdir}/${testfile}_random_small_holefile" copy up failed! + +diff "${lowerdir}/${testfile}_random_big_holefile" "${upperdir}/${testfile}_random_big_holefile" \ + >>$seqres.full || echo "${upperdir}/${testfile}_random_big_holefile" copy up failed! # success, all done status=0
Make many small holes in 10M test file seems not very helpful for test coverage and it takes too much time on creating test files. So in order to improve test speed we adjust test file size to (10 * iosize) for iosize aligned hole files meanwhile add more test patterns for small random holes and small empty file. Signed-off-by: Chengguang Xu <cgxu519@mykernel.net> --- tests/overlay/066 | 89 +++++++++++++++++++++++++++++++++++------------ 1 file changed, 67 insertions(+), 22 deletions(-)