diff mbox series

[2/2] btrfs: test repair with corrupted sectors interleaved over multiple mirrors

Message ID 20220520164743.4023665-3-hch@lst.de (mailing list archive)
State New, archived
Headers show
Series [1/2] btrfs: test repair with sectors corrupted in multiple mirrors | expand

Commit Message

Christoph Hellwig May 20, 2022, 4:47 p.m. UTC
Test that repair handles the case where it needs to read from more than
a single mirror on the raid1c3 profile and needs to take turns over the
mirrors to recover data for the whole read.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 tests/btrfs/266     | 145 ++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/266.out | 109 +++++++++++++++++++++++++++++++++
 2 files changed, 254 insertions(+)
 create mode 100755 tests/btrfs/266
 create mode 100644 tests/btrfs/266.out

Comments

Anand Jain May 22, 2022, 1:13 a.m. UTC | #1
On 5/20/22 22:17, Christoph Hellwig wrote:
> Test that repair handles the case where it needs to read from more than
> a single mirror on the raid1c3 profile and needs to take turns over the
> mirrors to recover data for the whole read.
> 
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
>   tests/btrfs/266     | 145 ++++++++++++++++++++++++++++++++++++++++++++
>   tests/btrfs/266.out | 109 +++++++++++++++++++++++++++++++++
>   2 files changed, 254 insertions(+)
>   create mode 100755 tests/btrfs/266
>   create mode 100644 tests/btrfs/266.out
> 
> diff --git a/tests/btrfs/266 b/tests/btrfs/266
> new file mode 100755
> index 00000000..24c2b5fd
> --- /dev/null
> +++ b/tests/btrfs/266
> @@ -0,0 +1,145 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2017 Liu Bo.  All Rights Reserved.
> +# Copyright (c) 2022 Christoph Hellwig.
> +#
> +# FS QA Test 266
> +#
> +# Test that btrfs raid repair on a raid1c3 profile can repair interleaving
> +# errors on all mirrors.
> +#
> +
> +. ./common/preamble
> +_begin_fstest auto quick read_repair
> +
> +# Import common functions.
> +. ./common/filter
> +
> +# real QA test starts here
> +
> +_supported_fs btrfs
> +_require_scratch_dev_pool 3
> +
> +BTRFS_MAP_LOGICAL_PROG=$(type -P btrfs-map-logical)
> +
> +_require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical
> +_require_command "$FILEFRAG_PROG" filefrag
> +_require_odirect
> +# Overwriting data is forbidden on a zoned block device
> +_require_non_zoned_device "${SCRATCH_DEV}"
> +
> +get_physical()
> +{
> +	local logical=$1
> +	local stripe=$2
> +
> +	$BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV >> $seqres.full 2>&1
> +	$BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \
> +		$AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$6 }"
> +}
> +			
> +get_device_path()
> +{
> +	local logical=$1
> +	local stripe=$2
> +
> +	$BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \
> +		$AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$8 }"
> +}
> +

Can wrap into a helper in common/btrfs.

> +_scratch_dev_pool_get 3
> +# step 1, create a raid1 btrfs which contains one 128k file.
> +echo "step 1......mkfs.btrfs"
> +
> +mkfs_opts="-d raid1c3 -b 1G"
> +_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
> +
> +# make sure data is written to the start position of the data chunk
> +_scratch_mount $(_btrfs_no_v1_cache_opt)
> +
> +$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" \
> +	"$SCRATCH_MNT/foobar" | \
> +	_filter_xfs_io_offset
> +
> +# ensure btrfs-map-logical sees the tree updates
> +sync
> +
> +# step 2, corrupt 4k in each copy
> +echo "step 2......corrupt file extent"
> +
> +${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full
> +logical=`${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar | _filter_filefrag | cut -d '#' -f 1`
> +
> +physical1=$(get_physical ${logical} 1)
> +devpath1=$(get_device_path ${logical} 1)
> +
> +physical2=$(get_physical ${logical} 2)
> +devpath2=$(get_device_path ${logical} 2)
> +
> +physical3=$(get_physical ${logical} 3)
> +devpath3=$(get_device_path ${logical} 3)
> +
> +_scratch_unmount
> +



> +$XFS_IO_PROG -d -c "pwrite -S 0xbd -b 4K $physical3 4K" $devpath3 \
> +	> /dev/null
> +
> +$XFS_IO_PROG -d -c "pwrite -S 0xba -b 4K $physical1 8K" $devpath1 \
> +	> /dev/null
> +
> +$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 4K $((physical2 + 4096)) 8K" $devpath2 \
> +	> /dev/null
> +
> +$XFS_IO_PROG -d -c "pwrite -S 0xbc -b 4K $((physical3 + (2 * 4096))) 8K"  \
> +	$devpath3 > /dev/null
> +

Could you please make this test case compatible with the 64K sectorsize 
configs too?




> +_scratch_mount
> +
> +# step 3, 128k dio read (this read can repair bad copy)
> +echo "step 3......repair the bad copy"
> +
> +# since raid1c3 consists of three copies, and the bad copy was put on stripe #1
> +# while the good copy lies the other stripes, the bad copy only gets accessed
> +# when the reader's pid % 3 is 1
> +while true; do
> +	echo 3 > /proc/sys/vm/drop_caches
> +	$XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" > /dev/null &
> +	pid=$!
> +	wait
> +	if [ $((pid % 3)) == 1 ]; then
> +	    break
> +	fi
> +done
> +while true; do
> +	echo 3 > /proc/sys/vm/drop_caches
> +	$XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" > /dev/null &
> +	pid=$!
> +	wait
> +	if [ $((pid % 3)) == 2 ]; then
> +	    break
> +	fi
> +done
> +while true; do
> +	echo 3 > /proc/sys/vm/drop_caches
> +	$XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" > /dev/null &
> +	pid=$!
> +	wait
> +	if [ $((pid % 3)) == 0 ]; then
> +	    break
> +	fi
> +done
> +

Same here. They can wrap into a helper.

Thanks, Anand

> +_scratch_unmount
> +
> +echo "step 4......check if the repair works"
> +$XFS_IO_PROG -d -c "pread -v -b 512 $physical1 512" $devpath1 |\
> +	_filter_xfs_io_offset
> +$XFS_IO_PROG -d -c "pread -v -b 512 $physical2 512" $devpath2 |\
> +	_filter_xfs_io_offset
> +$XFS_IO_PROG -d -c "pread -v -b 512 $physical3 512" $devpath3 |\
> +	_filter_xfs_io_offset
> +
> +_scratch_dev_pool_put
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/266.out b/tests/btrfs/266.out
> new file mode 100644
> index 00000000..243d1e1d
> --- /dev/null
> +++ b/tests/btrfs/266.out
> @@ -0,0 +1,109 @@
> +QA output created by 266
> +step 1......mkfs.btrfs
> +wrote 131072/131072 bytes
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +step 2......corrupt file extent
> +step 3......repair the bad copy
> +step 4......check if the repair works
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +read 512/512 bytes
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +read 512/512 bytes
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
> +read 512/512 bytes
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
Qu Wenruo May 22, 2022, 6:31 a.m. UTC | #2
On 2022/5/22 09:13, Anand Jain wrote:
> On 5/20/22 22:17, Christoph Hellwig wrote:
>> Test that repair handles the case where it needs to read from more than
>> a single mirror on the raid1c3 profile and needs to take turns over the
>> mirrors to recover data for the whole read.
>>
>> Signed-off-by: Christoph Hellwig <hch@lst.de>
>> ---
>>   tests/btrfs/266     | 145 ++++++++++++++++++++++++++++++++++++++++++++
>>   tests/btrfs/266.out | 109 +++++++++++++++++++++++++++++++++
>>   2 files changed, 254 insertions(+)
>>   create mode 100755 tests/btrfs/266
>>   create mode 100644 tests/btrfs/266.out
>>
>> diff --git a/tests/btrfs/266 b/tests/btrfs/266
>> new file mode 100755
>> index 00000000..24c2b5fd
>> --- /dev/null
>> +++ b/tests/btrfs/266
>> @@ -0,0 +1,145 @@
>> +#! /bin/bash
>> +# SPDX-License-Identifier: GPL-2.0
>> +# Copyright (c) 2017 Liu Bo.  All Rights Reserved.
>> +# Copyright (c) 2022 Christoph Hellwig.
>> +#
>> +# FS QA Test 266
>> +#
>> +# Test that btrfs raid repair on a raid1c3 profile can repair
>> interleaving
>> +# errors on all mirrors.
>> +#
>> +
>> +. ./common/preamble
>> +_begin_fstest auto quick read_repair
>> +
>> +# Import common functions.
>> +. ./common/filter
>> +
>> +# real QA test starts here
>> +
>> +_supported_fs btrfs
>> +_require_scratch_dev_pool 3
>> +
>> +BTRFS_MAP_LOGICAL_PROG=$(type -P btrfs-map-logical)
>> +
>> +_require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical
>> +_require_command "$FILEFRAG_PROG" filefrag
>> +_require_odirect
>> +# Overwriting data is forbidden on a zoned block device
>> +_require_non_zoned_device "${SCRATCH_DEV}"
>> +
>> +get_physical()
>> +{
>> +    local logical=$1
>> +    local stripe=$2
>> +
>> +    $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV >> $seqres.full
>> 2>&1
>> +    $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \
>> +        $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$6 }"
>> +}
>> +
>> +get_device_path()
>> +{
>> +    local logical=$1
>> +    local stripe=$2
>> +
>> +    $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \
>> +        $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$8 }"
>> +}
>> +
>
> Can wrap into a helper in common/btrfs.
>
>> +_scratch_dev_pool_get 3
>> +# step 1, create a raid1 btrfs which contains one 128k file.
>> +echo "step 1......mkfs.btrfs"
>> +
>> +mkfs_opts="-d raid1c3 -b 1G"
>> +_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
>> +
>> +# make sure data is written to the start position of the data chunk
>> +_scratch_mount $(_btrfs_no_v1_cache_opt)
>> +
>> +$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" \
>> +    "$SCRATCH_MNT/foobar" | \
>> +    _filter_xfs_io_offset
>> +
>> +# ensure btrfs-map-logical sees the tree updates
>> +sync
>> +
>> +# step 2, corrupt 4k in each copy
>> +echo "step 2......corrupt file extent"
>> +
>> +${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full
>> +logical=`${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar | _filter_filefrag |
>> cut -d '#' -f 1`
>> +
>> +physical1=$(get_physical ${logical} 1)
>> +devpath1=$(get_device_path ${logical} 1)
>> +
>> +physical2=$(get_physical ${logical} 2)
>> +devpath2=$(get_device_path ${logical} 2)
>> +
>> +physical3=$(get_physical ${logical} 3)
>> +devpath3=$(get_device_path ${logical} 3)
>> +
>> +_scratch_unmount
>> +
>
>
>
>> +$XFS_IO_PROG -d -c "pwrite -S 0xbd -b 4K $physical3 4K" $devpath3 \
>> +    > /dev/null
>> +
>> +$XFS_IO_PROG -d -c "pwrite -S 0xba -b 4K $physical1 8K" $devpath1 \
>> +    > /dev/null
>> +
>> +$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 4K $((physical2 + 4096)) 8K"
>> $devpath2 \
>> +    > /dev/null
>> +
>> +$XFS_IO_PROG -d -c "pwrite -S 0xbc -b 4K $((physical3 + (2 * 4096)))
>> 8K"  \
>> +    $devpath3 > /dev/null
>> +
>
> Could you please make this test case compatible with the 64K sectorsize
> configs too?
>
>
>
>
>> +_scratch_mount
>> +
>> +# step 3, 128k dio read (this read can repair bad copy)
>> +echo "step 3......repair the bad copy"
>> +
>> +# since raid1c3 consists of three copies, and the bad copy was put on
>> stripe #1
>> +# while the good copy lies the other stripes, the bad copy only gets
>> accessed
>> +# when the reader's pid % 3 is 1
>> +while true; do
>> +    echo 3 > /proc/sys/vm/drop_caches
>> +    $XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" >
>> /dev/null &
>> +    pid=$!
>> +    wait
>> +    if [ $((pid % 3)) == 1 ]; then
>> +        break
>> +    fi
>> +done
>> +while true; do
>> +    echo 3 > /proc/sys/vm/drop_caches
>> +    $XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" >
>> /dev/null &
>> +    pid=$!
>> +    wait
>> +    if [ $((pid % 3)) == 2 ]; then
>> +        break
>> +    fi
>> +done
>> +while true; do
>> +    echo 3 > /proc/sys/vm/drop_caches
>> +    $XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" >
>> /dev/null &
>> +    pid=$!
>> +    wait
>> +    if [ $((pid % 3)) == 0 ]; then
>> +        break
>> +    fi
>> +done
>> +
>
> Same here. They can wrap into a helper.

In fact, we should recheck the way we handle the read on specific mirror
behavior.

For a lot of runs, btrfs/14[01] can take super long time (over 100s) to
really trigger the data read on expected mirror.

For a lot of runs, the read just got the same pid again and again.

Can't we at least add some noise at background to make the pid more random?

Thanks,
Qu
>
> Thanks, Anand
>
>> +_scratch_unmount
>> +
>> +echo "step 4......check if the repair works"
>> +$XFS_IO_PROG -d -c "pread -v -b 512 $physical1 512" $devpath1 |\
>> +    _filter_xfs_io_offset
>> +$XFS_IO_PROG -d -c "pread -v -b 512 $physical2 512" $devpath2 |\
>> +    _filter_xfs_io_offset
>> +$XFS_IO_PROG -d -c "pread -v -b 512 $physical3 512" $devpath3 |\
>> +    _filter_xfs_io_offset
>> +
>> +_scratch_dev_pool_put
>> +# success, all done
>> +status=0
>> +exit
>> diff --git a/tests/btrfs/266.out b/tests/btrfs/266.out
>> new file mode 100644
>> index 00000000..243d1e1d
>> --- /dev/null
>> +++ b/tests/btrfs/266.out
>> @@ -0,0 +1,109 @@
>> +QA output created by 266
>> +step 1......mkfs.btrfs
>> +wrote 131072/131072 bytes
>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>> +step 2......corrupt file extent
>> +step 3......repair the bad copy
>> +step 4......check if the repair works
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +read 512/512 bytes
>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +read 512/512 bytes
>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>> ................
>> +read 512/512 bytes
>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>
Anand Jain May 22, 2022, 12:56 p.m. UTC | #3
On 5/22/22 12:01, Qu Wenruo wrote:
> 
> 
> On 2022/5/22 09:13, Anand Jain wrote:
>> On 5/20/22 22:17, Christoph Hellwig wrote:
>>> Test that repair handles the case where it needs to read from more than
>>> a single mirror on the raid1c3 profile and needs to take turns over the
>>> mirrors to recover data for the whole read.
>>>
>>> Signed-off-by: Christoph Hellwig <hch@lst.de>
>>> ---
>>>   tests/btrfs/266     | 145 ++++++++++++++++++++++++++++++++++++++++++++
>>>   tests/btrfs/266.out | 109 +++++++++++++++++++++++++++++++++
>>>   2 files changed, 254 insertions(+)
>>>   create mode 100755 tests/btrfs/266
>>>   create mode 100644 tests/btrfs/266.out
>>>
>>> diff --git a/tests/btrfs/266 b/tests/btrfs/266
>>> new file mode 100755
>>> index 00000000..24c2b5fd
>>> --- /dev/null
>>> +++ b/tests/btrfs/266
>>> @@ -0,0 +1,145 @@
>>> +#! /bin/bash
>>> +# SPDX-License-Identifier: GPL-2.0
>>> +# Copyright (c) 2017 Liu Bo.  All Rights Reserved.
>>> +# Copyright (c) 2022 Christoph Hellwig.
>>> +#
>>> +# FS QA Test 266
>>> +#
>>> +# Test that btrfs raid repair on a raid1c3 profile can repair
>>> interleaving
>>> +# errors on all mirrors.
>>> +#
>>> +
>>> +. ./common/preamble
>>> +_begin_fstest auto quick read_repair
>>> +
>>> +# Import common functions.
>>> +. ./common/filter
>>> +
>>> +# real QA test starts here
>>> +
>>> +_supported_fs btrfs
>>> +_require_scratch_dev_pool 3
>>> +
>>> +BTRFS_MAP_LOGICAL_PROG=$(type -P btrfs-map-logical)
>>> +
>>> +_require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical
>>> +_require_command "$FILEFRAG_PROG" filefrag
>>> +_require_odirect
>>> +# Overwriting data is forbidden on a zoned block device
>>> +_require_non_zoned_device "${SCRATCH_DEV}"
>>> +
>>> +get_physical()
>>> +{
>>> +    local logical=$1
>>> +    local stripe=$2
>>> +
>>> +    $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV >> $seqres.full
>>> 2>&1
>>> +    $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \
>>> +        $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$6 }"
>>> +}
>>> +
>>> +get_device_path()
>>> +{
>>> +    local logical=$1
>>> +    local stripe=$2
>>> +
>>> +    $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \
>>> +        $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$8 }"
>>> +}
>>> +
>>
>> Can wrap into a helper in common/btrfs.
>>
>>> +_scratch_dev_pool_get 3
>>> +# step 1, create a raid1 btrfs which contains one 128k file.
>>> +echo "step 1......mkfs.btrfs"
>>> +
>>> +mkfs_opts="-d raid1c3 -b 1G"
>>> +_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
>>> +
>>> +# make sure data is written to the start position of the data chunk
>>> +_scratch_mount $(_btrfs_no_v1_cache_opt)
>>> +
>>> +$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" \
>>> +    "$SCRATCH_MNT/foobar" | \
>>> +    _filter_xfs_io_offset
>>> +
>>> +# ensure btrfs-map-logical sees the tree updates
>>> +sync
>>> +
>>> +# step 2, corrupt 4k in each copy
>>> +echo "step 2......corrupt file extent"
>>> +
>>> +${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full
>>> +logical=`${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar | _filter_filefrag |
>>> cut -d '#' -f 1`
>>> +
>>> +physical1=$(get_physical ${logical} 1)
>>> +devpath1=$(get_device_path ${logical} 1)
>>> +
>>> +physical2=$(get_physical ${logical} 2)
>>> +devpath2=$(get_device_path ${logical} 2)
>>> +
>>> +physical3=$(get_physical ${logical} 3)
>>> +devpath3=$(get_device_path ${logical} 3)
>>> +
>>> +_scratch_unmount
>>> +
>>
>>
>>
>>> +$XFS_IO_PROG -d -c "pwrite -S 0xbd -b 4K $physical3 4K" $devpath3 \
>>> +    > /dev/null
>>> +
>>> +$XFS_IO_PROG -d -c "pwrite -S 0xba -b 4K $physical1 8K" $devpath1 \
>>> +    > /dev/null
>>> +
>>> +$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 4K $((physical2 + 4096)) 8K"
>>> $devpath2 \
>>> +    > /dev/null
>>> +
>>> +$XFS_IO_PROG -d -c "pwrite -S 0xbc -b 4K $((physical3 + (2 * 4096)))
>>> 8K"  \
>>> +    $devpath3 > /dev/null
>>> +
>>
>> Could you please make this test case compatible with the 64K sectorsize
>> configs too?
>>
>>
>>
>>
>>> +_scratch_mount
>>> +
>>> +# step 3, 128k dio read (this read can repair bad copy)
>>> +echo "step 3......repair the bad copy"
>>> +
>>> +# since raid1c3 consists of three copies, and the bad copy was put on
>>> stripe #1
>>> +# while the good copy lies the other stripes, the bad copy only gets
>>> accessed
>>> +# when the reader's pid % 3 is 1
>>> +while true; do
>>> +    echo 3 > /proc/sys/vm/drop_caches
>>> +    $XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" >
>>> /dev/null &
>>> +    pid=$!
>>> +    wait
>>> +    if [ $((pid % 3)) == 1 ]; then
>>> +        break
>>> +    fi
>>> +done
>>> +while true; do
>>> +    echo 3 > /proc/sys/vm/drop_caches
>>> +    $XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" >
>>> /dev/null &
>>> +    pid=$!
>>> +    wait
>>> +    if [ $((pid % 3)) == 2 ]; then
>>> +        break
>>> +    fi
>>> +done
>>> +while true; do
>>> +    echo 3 > /proc/sys/vm/drop_caches
>>> +    $XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" >
>>> /dev/null &
>>> +    pid=$!
>>> +    wait
>>> +    if [ $((pid % 3)) == 0 ]; then
>>> +        break
>>> +    fi
>>> +done
>>> +
>>
>> Same here. They can wrap into a helper.
> 
> In fact, we should recheck the way we handle the read on specific mirror
> behavior.
> 
> For a lot of runs, btrfs/14[01] can take super long time (over 100s) to
> really trigger the data read on expected mirror.
> 
> For a lot of runs, the read just got the same pid again and again.
> 
> Can't we at least add some noise at background to make the pid more random?
> 


  Agreed this method is unreliable. But there is no other choice.
  Unless we integrated [1] patch in the ML.

  [1] btrfs: introduce new read_policy device

  [1] is more reliable. You can set which mirrored device to read.

Thanks, Anand


> Thanks,
> Qu
>>
>> Thanks, Anand
>>
>>> +_scratch_unmount
>>> +
>>> +echo "step 4......check if the repair works"
>>> +$XFS_IO_PROG -d -c "pread -v -b 512 $physical1 512" $devpath1 |\
>>> +    _filter_xfs_io_offset
>>> +$XFS_IO_PROG -d -c "pread -v -b 512 $physical2 512" $devpath2 |\
>>> +    _filter_xfs_io_offset
>>> +$XFS_IO_PROG -d -c "pread -v -b 512 $physical3 512" $devpath3 |\
>>> +    _filter_xfs_io_offset
>>> +
>>> +_scratch_dev_pool_put
>>> +# success, all done
>>> +status=0
>>> +exit
>>> diff --git a/tests/btrfs/266.out b/tests/btrfs/266.out
>>> new file mode 100644
>>> index 00000000..243d1e1d
>>> --- /dev/null
>>> +++ b/tests/btrfs/266.out
>>> @@ -0,0 +1,109 @@
>>> +QA output created by 266
>>> +step 1......mkfs.btrfs
>>> +wrote 131072/131072 bytes
>>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>>> +step 2......corrupt file extent
>>> +step 3......repair the bad copy
>>> +step 4......check if the repair works
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +read 512/512 bytes
>>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +read 512/512 bytes
>>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +read 512/512 bytes
>>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>>
Christoph Hellwig May 22, 2022, 1 p.m. UTC | #4
On Sun, May 22, 2022 at 06:26:26PM +0530, Anand Jain wrote:
>  Agreed this method is unreliable. But there is no other choice.
>  Unless we integrated [1] patch in the ML.
>
>  [1] btrfs: introduce new read_policy device
>
>  [1] is more reliable. You can set which mirrored device to read.

I actually thought about something like this as the reliable way
to hit a mirror.  Besides testing I could also thing of some other
use cases for it where you want to avoid a mirror by default for some
reason (e.g. because it is network attached store while the other
one is local).
diff mbox series

Patch

diff --git a/tests/btrfs/266 b/tests/btrfs/266
new file mode 100755
index 00000000..24c2b5fd
--- /dev/null
+++ b/tests/btrfs/266
@@ -0,0 +1,145 @@ 
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2017 Liu Bo.  All Rights Reserved.
+# Copyright (c) 2022 Christoph Hellwig.
+#
+# FS QA Test 266
+#
+# Test that btrfs raid repair on a raid1c3 profile can repair interleaving
+# errors on all mirrors.
+#
+
+. ./common/preamble
+_begin_fstest auto quick read_repair
+
+# Import common functions.
+. ./common/filter
+
+# real QA test starts here
+
+_supported_fs btrfs
+_require_scratch_dev_pool 3
+
+BTRFS_MAP_LOGICAL_PROG=$(type -P btrfs-map-logical)
+
+_require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical
+_require_command "$FILEFRAG_PROG" filefrag
+_require_odirect
+# Overwriting data is forbidden on a zoned block device
+_require_non_zoned_device "${SCRATCH_DEV}"
+
+get_physical()
+{
+	local logical=$1
+	local stripe=$2
+
+	$BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV >> $seqres.full 2>&1
+	$BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \
+		$AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$6 }"
+}
+			
+get_device_path()
+{
+	local logical=$1
+	local stripe=$2
+
+	$BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \
+		$AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$8 }"
+}
+
+_scratch_dev_pool_get 3
+# step 1, create a raid1 btrfs which contains one 128k file.
+echo "step 1......mkfs.btrfs"
+
+mkfs_opts="-d raid1c3 -b 1G"
+_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
+
+# make sure data is written to the start position of the data chunk
+_scratch_mount $(_btrfs_no_v1_cache_opt)
+
+$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" \
+	"$SCRATCH_MNT/foobar" | \
+	_filter_xfs_io_offset
+
+# ensure btrfs-map-logical sees the tree updates
+sync
+
+# step 2, corrupt 4k in each copy
+echo "step 2......corrupt file extent"
+
+${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full
+logical=`${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar | _filter_filefrag | cut -d '#' -f 1`
+
+physical1=$(get_physical ${logical} 1)
+devpath1=$(get_device_path ${logical} 1)
+
+physical2=$(get_physical ${logical} 2)
+devpath2=$(get_device_path ${logical} 2)
+
+physical3=$(get_physical ${logical} 3)
+devpath3=$(get_device_path ${logical} 3)
+
+_scratch_unmount
+
+$XFS_IO_PROG -d -c "pwrite -S 0xbd -b 4K $physical3 4K" $devpath3 \
+	> /dev/null
+
+$XFS_IO_PROG -d -c "pwrite -S 0xba -b 4K $physical1 8K" $devpath1 \
+	> /dev/null
+
+$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 4K $((physical2 + 4096)) 8K" $devpath2 \
+	> /dev/null
+
+$XFS_IO_PROG -d -c "pwrite -S 0xbc -b 4K $((physical3 + (2 * 4096))) 8K"  \
+	$devpath3 > /dev/null
+
+_scratch_mount
+
+# step 3, 128k dio read (this read can repair bad copy)
+echo "step 3......repair the bad copy"
+
+# since raid1c3 consists of three copies, and the bad copy was put on stripe #1
+# while the good copy lies the other stripes, the bad copy only gets accessed
+# when the reader's pid % 3 is 1
+while true; do
+	echo 3 > /proc/sys/vm/drop_caches
+	$XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" > /dev/null &
+	pid=$!
+	wait
+	if [ $((pid % 3)) == 1 ]; then
+	    break
+	fi
+done
+while true; do
+	echo 3 > /proc/sys/vm/drop_caches
+	$XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" > /dev/null &
+	pid=$!
+	wait
+	if [ $((pid % 3)) == 2 ]; then
+	    break
+	fi
+done
+while true; do
+	echo 3 > /proc/sys/vm/drop_caches
+	$XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" > /dev/null &
+	pid=$!
+	wait
+	if [ $((pid % 3)) == 0 ]; then
+	    break
+	fi
+done
+
+_scratch_unmount
+
+echo "step 4......check if the repair works"
+$XFS_IO_PROG -d -c "pread -v -b 512 $physical1 512" $devpath1 |\
+	_filter_xfs_io_offset
+$XFS_IO_PROG -d -c "pread -v -b 512 $physical2 512" $devpath2 |\
+	_filter_xfs_io_offset
+$XFS_IO_PROG -d -c "pread -v -b 512 $physical3 512" $devpath3 |\
+	_filter_xfs_io_offset
+
+_scratch_dev_pool_put
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/266.out b/tests/btrfs/266.out
new file mode 100644
index 00000000..243d1e1d
--- /dev/null
+++ b/tests/btrfs/266.out
@@ -0,0 +1,109 @@ 
+QA output created by 266
+step 1......mkfs.btrfs
+wrote 131072/131072 bytes
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+step 2......corrupt file extent
+step 3......repair the bad copy
+step 4......check if the repair works
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+read 512/512 bytes
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+read 512/512 bytes
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+read 512/512 bytes
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)